From dieter at handshake.de  Mon Nov  1 12:18:43 2021
From: dieter at handshake.de (Dieter Maurer)
Date: Mon, 1 Nov 2021 17:18:43 +0100
Subject: Python C API: how to mark a type as subclass of another type
In-Reply-To: <CABbU2U9EbBES+CU3OwXqY_9T0LVT6rYUFzy3e-rebGQ53zHOyw@mail.gmail.com>
References: <CABbU2U9EbBES+CU3OwXqY_9T0LVT6rYUFzy3e-rebGQ53zHOyw@mail.gmail.com>
Message-ID: <24960.4963.376344.239230@ixdm.fritz.box>

Marco Sulla wrote at 2021-10-31 23:59 +0100:
>I have two types declared as
>
>PyTypeObject PyX_Type = {
>    PyVarObject_HEAD_INIT(&PyType_Type, 0)
>
>etc.
>
>How can I mark one of the types as subclass of the other one? I tried
>to use tp_base but it didn't work.

Read the "Python/C Api" documentation. Watch out for `tp_base`.

From shishaozhong at gmail.com  Mon Nov  1 19:02:23 2021
From: shishaozhong at gmail.com (Shaozhong SHI)
Date: Mon, 1 Nov 2021 23:02:23 +0000
Subject: System, configuration and Python performance
Message-ID: <CA+i5JwZKkTvdDHxDPVP0LYbv_4xHFmMKktVxX0qmgWVhbYHgUg@mail.gmail.com>

How to configure to improve Python performance in a system like the
following:

Windows 10

System

Processor Intel(R) Core(TM) i7-9700 CPU @3.60GHz, 3.60 GHz
Installed memory (RAM) 32.0 GB (31.8 GB usable)
System type: 64-bit Operating System, x64-based processor

I found that the Python script was runnig slowly and wanted to find out
what is going on and what activities it is doing.

I opened the Task Manager and found that there is not much CPU usage.

Do I need to do something like configuration to improve Python's
performance?

Regards,

David

From python at mrabarnett.plus.com  Mon Nov  1 20:04:27 2021
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 2 Nov 2021 00:04:27 +0000
Subject: System, configuration and Python performance
In-Reply-To: <CA+i5JwZKkTvdDHxDPVP0LYbv_4xHFmMKktVxX0qmgWVhbYHgUg@mail.gmail.com>
References: <CA+i5JwZKkTvdDHxDPVP0LYbv_4xHFmMKktVxX0qmgWVhbYHgUg@mail.gmail.com>
Message-ID: <367adc73-c522-7802-58c0-87f3e0fea691@mrabarnett.plus.com>

On 2021-11-01 23:02, Shaozhong SHI wrote:
> How to configure to improve Python performance in a system like the
> following:
> 
> Windows 10
> 
> System
> 
> Processor Intel(R) Core(TM) i7-9700 CPU @3.60GHz, 3.60 GHz
> Installed memory (RAM) 32.0 GB (31.8 GB usable)
> System type: 64-bit Operating System, x64-based processor
> 
> I found that the Python script was runnig slowly and wanted to find out
> what is going on and what activities it is doing.
> 
> I opened the Task Manager and found that there is not much CPU usage.
> 
> Do I need to do something like configuration to improve Python's
> performance?
> 
If CPU usage is low, then that isn't the cause of the slowness.

What about disk usage?

What about network usage?

If it's communicating across the internet, then it might be waiting for 
the other end. If that's the case, then there's probably not much you 
can do about it.

From shishaozhong at gmail.com  Mon Nov  1 20:20:12 2021
From: shishaozhong at gmail.com (Shaozhong SHI)
Date: Tue, 2 Nov 2021 00:20:12 +0000
Subject: System, configuration and Python performance
In-Reply-To: <367adc73-c522-7802-58c0-87f3e0fea691@mrabarnett.plus.com>
References: <CA+i5JwZKkTvdDHxDPVP0LYbv_4xHFmMKktVxX0qmgWVhbYHgUg@mail.gmail.com>
 <367adc73-c522-7802-58c0-87f3e0fea691@mrabarnett.plus.com>
Message-ID: <CA+i5JwZvd0GKNtmfXUBTZDHCrBcRYze7v2HtjA-ik_1=iNEu5A@mail.gmail.com>

On Tue, 2 Nov 2021 at 00:09, MRAB <python at mrabarnett.plus.com> wrote:

> On 2021-11-01 23:02, Shaozhong SHI wrote:
> > How to configure to improve Python performance in a system like the
> > following:
> >
> > Windows 10
> >
> > System
> >
> > Processor Intel(R) Core(TM) i7-9700 CPU @3.60GHz, 3.60 GHz
> > Installed memory (RAM) 32.0 GB (31.8 GB usable)
> > System type: 64-bit Operating System, x64-based processor
> >
> > I found that the Python script was runnig slowly and wanted to find out
> > what is going on and what activities it is doing.
> >
> > I opened the Task Manager and found that there is not much CPU usage.
> >
> > Do I need to do something like configuration to improve Python's
> > performance?
> >
> If CPU usage is low, then that isn't the cause of the slowness.
>
> What about disk usage?
>
> What about network usage?
>
> If it's communicating across the internet, then it might be waiting for
> the other end. If that's the case, then there's probably not much you
> can do about it.
>

Both disk usage and network usage are very low as well.

It is checking out responses of internet pages with given URLs.

Regards,

David




> --
> https://mail.python.org/mailman/listinfo/python-list
>

From shishaozhong at gmail.com  Mon Nov  1 20:23:47 2021
From: shishaozhong at gmail.com (Shaozhong SHI)
Date: Tue, 2 Nov 2021 00:23:47 +0000
Subject: System, configuration and Python performance
In-Reply-To: <CA+i5JwZvd0GKNtmfXUBTZDHCrBcRYze7v2HtjA-ik_1=iNEu5A@mail.gmail.com>
References: <CA+i5JwZKkTvdDHxDPVP0LYbv_4xHFmMKktVxX0qmgWVhbYHgUg@mail.gmail.com>
 <367adc73-c522-7802-58c0-87f3e0fea691@mrabarnett.plus.com>
 <CA+i5JwZvd0GKNtmfXUBTZDHCrBcRYze7v2HtjA-ik_1=iNEu5A@mail.gmail.com>
Message-ID: <CA+i5JwbXQ=TiLBzpaDjc8EVDXzRMk5CLTvQXwMBuKxu2ciRvRQ@mail.gmail.com>

On Tue, 2 Nov 2021 at 00:20, Shaozhong SHI <shishaozhong at gmail.com> wrote:

>
>
> On Tue, 2 Nov 2021 at 00:09, MRAB <python at mrabarnett.plus.com> wrote:
>
>> On 2021-11-01 23:02, Shaozhong SHI wrote:
>> > How to configure to improve Python performance in a system like the
>> > following:
>> >
>> > Windows 10
>> >
>> > System
>> >
>> > Processor Intel(R) Core(TM) i7-9700 CPU @3.60GHz, 3.60 GHz
>> > Installed memory (RAM) 32.0 GB (31.8 GB usable)
>> > System type: 64-bit Operating System, x64-based processor
>> >
>> > I found that the Python script was runnig slowly and wanted to find out
>> > what is going on and what activities it is doing.
>> >
>> > I opened the Task Manager and found that there is not much CPU usage.
>> >
>> > Do I need to do something like configuration to improve Python's
>> > performance?
>> >
>> If CPU usage is low, then that isn't the cause of the slowness.
>>
>> What about disk usage?
>>
>> What about network usage?
>>
>> If it's communicating across the internet, then it might be waiting for
>> the other end. If that's the case, then there's probably not much you
>> can do about it.
>>
>
> Both disk usage and network usage are very low as well.
>
> It is checking out responses of internet pages with given URLs.
>
> It is checking out whether each url is valid or not.
>
>
>
>> --
>> https://mail.python.org/mailman/listinfo/python-list
>>
>

From rosuav at gmail.com  Mon Nov  1 20:23:47 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 2 Nov 2021 11:23:47 +1100
Subject: System, configuration and Python performance
In-Reply-To: <CA+i5JwZvd0GKNtmfXUBTZDHCrBcRYze7v2HtjA-ik_1=iNEu5A@mail.gmail.com>
References: <CA+i5JwZKkTvdDHxDPVP0LYbv_4xHFmMKktVxX0qmgWVhbYHgUg@mail.gmail.com>
 <367adc73-c522-7802-58c0-87f3e0fea691@mrabarnett.plus.com>
 <CA+i5JwZvd0GKNtmfXUBTZDHCrBcRYze7v2HtjA-ik_1=iNEu5A@mail.gmail.com>
Message-ID: <CAPTjJmoUOJVP-RWd_PXYqg-5ijy+-7GumYGEwtDzu2c+f7z_eA@mail.gmail.com>

On Tue, Nov 2, 2021 at 11:21 AM Shaozhong SHI <shishaozhong at gmail.com> wrote:
>
> On Tue, 2 Nov 2021 at 00:09, MRAB <python at mrabarnett.plus.com> wrote:
>
> > On 2021-11-01 23:02, Shaozhong SHI wrote:
> > > How to configure to improve Python performance in a system like the
> > > following:
> > >
> > > Windows 10
> > >
> > > System
> > >
> > > Processor Intel(R) Core(TM) i7-9700 CPU @3.60GHz, 3.60 GHz
> > > Installed memory (RAM) 32.0 GB (31.8 GB usable)
> > > System type: 64-bit Operating System, x64-based processor
> > >
> > > I found that the Python script was runnig slowly and wanted to find out
> > > what is going on and what activities it is doing.
> > >
> > > I opened the Task Manager and found that there is not much CPU usage.
> > >
> > > Do I need to do something like configuration to improve Python's
> > > performance?
> > >
> > If CPU usage is low, then that isn't the cause of the slowness.
> >
> > What about disk usage?
> >
> > What about network usage?
> >
> > If it's communicating across the internet, then it might be waiting for
> > the other end. If that's the case, then there's probably not much you
> > can do about it.
> >
>
> Both disk usage and network usage are very low as well.
>
> It is checking out responses of internet pages with given URLs.
>

That's almost certainly waiting for the server, then. There's nothing
you can do - in Python or any other language - to speed that up.

What you MAY be able to do is parallelize - do multiple requests at
once - but not all servers will let you hammer them.

ChrisA

From elbarbun at gmail.com  Tue Nov  2 08:43:15 2021
From: elbarbun at gmail.com (Marco Sulla)
Date: Tue, 2 Nov 2021 13:43:15 +0100
Subject: Python C API: how to mark a type as subclass of another type
In-Reply-To: <24960.4963.376344.239230@ixdm.fritz.box>
References: <CABbU2U9EbBES+CU3OwXqY_9T0LVT6rYUFzy3e-rebGQ53zHOyw@mail.gmail.com>
 <24960.4963.376344.239230@ixdm.fritz.box>
Message-ID: <CABbU2U-zHmUeqHvdz5ESSvbY4hnV4mJdCANf01vbZacAPeXYHQ@mail.gmail.com>

I already added the address of the type to tp_base, but it does not work.

On Mon, 1 Nov 2021 at 17:18, Dieter Maurer <dieter at handshake.de> wrote:
>
> Marco Sulla wrote at 2021-10-31 23:59 +0100:
> >I have two types declared as
> >
> >PyTypeObject PyX_Type = {
> >    PyVarObject_HEAD_INIT(&PyType_Type, 0)
> >
> >etc.
> >
> >How can I mark one of the types as subclass of the other one? I tried
> >to use tp_base but it didn't work.
>
> Read the "Python/C Api" documentation. Watch out for `tp_base`.

From robin at reportlab.com  Tue Nov  2 08:55:17 2021
From: robin at reportlab.com (Robin Becker)
Date: Tue, 2 Nov 2021 12:55:17 +0000
Subject: lxml parsing with validation and target?
Message-ID: <0723b9fd-62a8-cb87-98e7-d0966dfb4e98@everest.reportlab.co.uk>

I'm having a problem using lxml.etree to make a treebuilding parser that validates; I have test code where invalid xml 
is detected and an error raised when the line below target=ET.TreeBuilder(), is commented out.

The validation error looks as expected >  python tlxml.py invalid.rml
> re.compile('^.*(?:\\W|\\b)(?P<fn>dynamic_rml\\.dtd|rml\\.dtd|rml_0_2\\.dtd|rml_0_3\\.dtd|rml_1_0\\.dtd)$', re.MULTILINE)
> Resolving url='../rml.dtd' context=<lxml.etree._ParserContext object at 0x7f66103273c0> dtdPath='rml.dtd'
> Traceback (most recent call last):
>   File "/home/robin/devel/reportlab/REPOS/rlextra/tmp/tlxml.py", line 78, in <module>
>     tree = ET.parse(sys.argv[1],parser)
>   File "src/lxml/etree.pyx", line 3521, in lxml.etree.parse
>   File "src/lxml/parser.pxi", line 1859, in lxml.etree._parseDocument
>   File "src/lxml/parser.pxi", line 1885, in lxml.etree._parseDocumentFromURL
>   File "src/lxml/parser.pxi", line 1789, in lxml.etree._parseDocFromFile
>   File "src/lxml/parser.pxi", line 1177, in lxml.etree._BaseParser._parseDocFromFile
>   File "src/lxml/parser.pxi", line 615, in lxml.etree._ParserContext._handleParseResultDoc
>   File "src/lxml/parser.pxi", line 725, in lxml.etree._handleParseResult
>   File "src/lxml/parser.pxi", line 654, in lxml.etree._raiseParseError
>   File "invalid.rml", line 23
> lxml.etree.XMLSyntaxError: No declaration for attribute x of element place1, line 23, column 55

when I have the target=etree.TreeBuilder() active the validation does not work and the tree is formed and passed to the 
primitive tuple tree builder so the output looks like
> $ python tlxml.py invalid.rml 
> Resolving url='../rml.dtd' context=<lxml.etree._TargetParserContext object at 0x7f73d7b159c0> dtdPath='rml.dtd'
> ('document',
>  {'filename': 'test_000_simple.pdf', 'invariant': '1'},
>  ['\n\n',
>   ('stylesheet',
 > ........
>        None,
>        44),
>       '\n    \t\t\n    \t\t'],
>      40),
>     '\n'],
>    35),
>   '\n\n'],
>  2)

If I use the standard example EchoTarget the validation also fails. So I assume that the target argument makes the 
validation fail. Is there a way to get validation to work with a target?

The code is
######################################################################################################
from pprint import pprint
from lxml import etree as ET
import sys, os, re
from rlextra.rml2pdf.rml2pdf import CompatibleDTDNames as rmlDTDPat
rmlDTDPat = re.compile('^.*(?:\\W|\\b)(?P<fn>%s)$' % '|'.join((re.escape(_) for _ in rmlDTDPat)),re.M)

class TT:
     def __init__(self):
         pass

     def __call__(self,e):
         return (e.tag,e.attrib or None,self.content(e),e.sourceline)

     def content(self,e):
         t = e.text
         if len(e)==0 and t is None:
             return t
         else:
             r = [].append
             if t is not None: r(t)
             for c in e:
                 r(self(c))
                 t = c.tail
                 if t is not None:
                     r(t)
             return r.__self__

class RMLDTDResolver(ET.Resolver):
     __dtds = None
     def resolve(self, url, id, context):
         m = rmlDTDPat.match(url)
         if m:
             if self.__dtds is None:
                 from rlextra import rml2pdf
                 self.__dtds = {}
                 for fn in ('rml.dtd','dynamic_rml.dtd'):
                     with open(os.path.join(os.path.dirname(rml2pdf.__file__),fn),'r') as _:
                         self.__dtds[fn] = _.read()
             fn = m.group('fn')
             dtdPath = 'rml.dtd' if fn.startswith('rml') else 'dynamic.dtd'
             print(f"Resolving url={url!r} context={context!r} {dtdPath=}")
             return self.resolve_string(
                     self.__dtds[dtdPath],
                     context,
                     )
         else:
             return None

parser = ET.XMLParser(
                     load_dtd=True,
                     dtd_validation=True,
                     attribute_defaults=True,
                     no_network=True,
                     remove_comments=True,
                     remove_pis=True,
                     strip_cdata=True,
                     resolve_entities=True,
                     target=ET.TreeBuilder(),   #if commented the parser validates
                     )
parser.resolvers.add(RMLDTDResolver())
tree = ET.parse(sys.argv[1],parser)
pprint(TT()(tree))
######################################################################################################

From elbarbun at gmail.com  Tue Nov  2 09:00:20 2021
From: elbarbun at gmail.com (Marco Sulla)
Date: Tue, 2 Nov 2021 14:00:20 +0100
Subject: Python C API: how to mark a type as subclass of another type
In-Reply-To: <CABbU2U-zHmUeqHvdz5ESSvbY4hnV4mJdCANf01vbZacAPeXYHQ@mail.gmail.com>
References: <CABbU2U9EbBES+CU3OwXqY_9T0LVT6rYUFzy3e-rebGQ53zHOyw@mail.gmail.com>
 <24960.4963.376344.239230@ixdm.fritz.box>
 <CABbU2U-zHmUeqHvdz5ESSvbY4hnV4mJdCANf01vbZacAPeXYHQ@mail.gmail.com>
Message-ID: <CABbU2U_Z4oxZjo5WADW7_jO1mvSwLg2s6w8tg69oKVbtuTsSfQ@mail.gmail.com>

*ahem* evidently I didn't check the right package..... it works like a charme :D

On Tue, 2 Nov 2021 at 13:43, Marco Sulla <elbarbun at gmail.com> wrote:
>
> I already added the address of the type to tp_base, but it does not work.
>
> On Mon, 1 Nov 2021 at 17:18, Dieter Maurer <dieter at handshake.de> wrote:
> >
> > Marco Sulla wrote at 2021-10-31 23:59 +0100:
> > >I have two types declared as
> > >
> > >PyTypeObject PyX_Type = {
> > >    PyVarObject_HEAD_INIT(&PyType_Type, 0)
> > >
> > >etc.
> > >
> > >How can I mark one of the types as subclass of the other one? I tried
> > >to use tp_base but it didn't work.
> >
> > Read the "Python/C Api" documentation. Watch out for `tp_base`.

From dallasdisabilityattorney at gmail.com  Tue Nov  2 09:22:26 2021
From: dallasdisabilityattorney at gmail.com (dallasdisabi...@gmail.com)
Date: Tue, 2 Nov 2021 06:22:26 -0700 (PDT)
Subject: Python/django app design
Message-ID: <a9f558b6-5e4d-41c9-b46e-3a27baf4bbcdn@googlegroups.com>

I am seeking to create a django application.  The user would upload a text file that includes medical records and other unstructured data.  I want to create a word document report based upon the information in the files.

A key feature is to look for specific predefined text.  So with regard to medical records, I want to zero in a MRI, or X-results.  Is this in the area of machine learning?  Or am a creating a database of terms like "MRI", etc, that I want to scan for ?

From dieter at handshake.de  Tue Nov  2 12:15:17 2021
From: dieter at handshake.de (Dieter Maurer)
Date: Tue, 2 Nov 2021 17:15:17 +0100
Subject: Python C API: how to mark a type as subclass of another type
In-Reply-To: <CABbU2U-zHmUeqHvdz5ESSvbY4hnV4mJdCANf01vbZacAPeXYHQ@mail.gmail.com>
References: <CABbU2U9EbBES+CU3OwXqY_9T0LVT6rYUFzy3e-rebGQ53zHOyw@mail.gmail.com>
 <24960.4963.376344.239230@ixdm.fritz.box>
 <CABbU2U-zHmUeqHvdz5ESSvbY4hnV4mJdCANf01vbZacAPeXYHQ@mail.gmail.com>
Message-ID: <24961.25621.620802.606029@ixdm.fritz.box>

Marco Sulla wrote at 2021-11-2 13:43 +0100:
>I already added the address of the type to tp_base, but it does not work.

It worked for me in `dm.incrementalsearch`.

Maybe, you need a special value for `tp_flags`.

Look at the examples.

From loris.bennett at fu-berlin.de  Wed Nov  3 08:36:07 2021
From: loris.bennett at fu-berlin.de (Loris Bennett)
Date: Wed, 03 Nov 2021 13:36:07 +0100
Subject: argparse - 3rd arg only valid for one of two mutually exclusive args?
Message-ID: <87o871h0w8.fsf@hornfels.zedat.fu-berlin.de>

Hi,

With argparse's add_mutually_exclusive_group() I can add mutually
exclusive args, but how do I deal with a 3rd arg which only makes sense
for one of the mutually exclusive args?

More generally I suppose I am interested in having something like

  [ --foo (--foobar) | --bar (--barfoo) ]

if that makes it any clearer.

I have seen subcommand suggested as a way of doing this, but that
doesn't seem like a very good fit for my use-case.  Is there a more
argument-orientated alternative?

Cheers,

Loris

-- 
This signature is currently under construction.

From robin at reportlab.com  Wed Nov  3 10:59:29 2021
From: robin at reportlab.com (Robin Becker)
Date: Wed, 3 Nov 2021 14:59:29 +0000
Subject: lxml parsing with validation and target?
In-Reply-To: <0723b9fd-62a8-cb87-98e7-d0966dfb4e98@everest.reportlab.co.uk>
References: <0723b9fd-62a8-cb87-98e7-d0966dfb4e98@everest.reportlab.co.uk>
Message-ID: <f0687e81-bf3a-e072-edea-d4f0a8641f9b@everest.reportlab.co.uk>

On 02/11/2021 12:55, Robin Becker wrote:
> I'm having a problem using lxml.etree to make a treebuilding parser that validates; I have test code where invalid xml 
> is detected and an error raised when the line below target=ET.TreeBuilder(), is commented out.
> 
.........

I managed to overcome this problem by utilizing the non-targeted parser with returns an _ElementTree object. I can then 
convert to tuple tree using code like this

> class TT:
> 	def __init__(self):
> 		pass
> 
> 	def __call__(self,tree):
> 		if not tree: return
> 		return self.maketuple(next(tree.iter()))
> 
> 	def maketuple(self,e):
> 		return (e.tag,e.attrib or None,self.content(e),e.sourceline)
> 
> 	def content(self,e):
> 		t = e.text
> 		kids = e.getchildren()
> 		if len(kids)==0 and t is None:
> 			return t
> 		else:
> 			r = [].append
> 			if t is not None: r(t)
> 			for c in kids:
> 				r(self.maketuple(c))
> 				t = c.tail
> 				if t is not None:
> 					r(t)
> 			return r.__self__


-- 
Robin Becker

From arj.python at gmail.com  Thu Nov  4 03:41:35 2021
From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer)
Date: Thu, 4 Nov 2021 11:41:35 +0400
Subject: First Ever FlaskCon Tickets Sales
Message-ID: <CADrxXXkDg1sikte+A+f3pQvK5twAqL12N7zYpmLkH2eDPHMsCA@mail.gmail.com>

Greetings list,

I count my events-related posts to the mailing list.

But i'd like to make an exception for FlaskCon this year.

As the first ever tickets sales is ongoing:
https://ti.to/flaskcon/flaskcon-2021

Python fans, please forward / tweet the message.

Any questions, ping me!

Kind Regards,

Abdur-Rahmaan Janhangeer
about <https://compileralchemy.github.io/> | blog
<https://www.pythonkitchen.com>
github <https://github.com/Abdur-RahmaanJ>
Mauritius

From ast at invalid  Thu Nov  4 03:57:14 2021
From: ast at invalid (ast)
Date: Thu, 4 Nov 2021 08:57:14 +0100
Subject: Recursion on list
Message-ID: <6183925b$0$8912$426a74cc@news.free.fr>

 > li = []
 > li.append(li)
 > li
[[...]]

 >li[0][0][0][0]
[[...]]

That's funny

From ast at invalid  Thu Nov  4 07:43:23 2021
From: ast at invalid (ast)
Date: Thu, 4 Nov 2021 12:43:23 +0100
Subject: Syntax not understood
Message-ID: <6183c75c$0$29473$426a74cc@news.free.fr>

Hello

In this function

def get4(srcpages):
     scale = 0.5
     srcpages = PageMerge() + srcpages
     x_increment, y_increment = (scale * i for i in srcpages.xobj_box[2:])
     for i, page in enumerate(srcpages):
         page.scale(scale)
         page.x = x_increment if i & 1 else 0
         page.y = 0 if i & 2 else y_increment
     return srcpages.render()

found here

https://www.blog.pythonlibrary.org/2018/06/06/creating-and-manipulating-pdfs-with-pdfrw/

I do not understand this line:

x_increment, y_increment = (scale * i for i in srcpages.xobj_box[2:])

(scale * i for i in srcpages.xobj_box[2:]) is a generator, a single
object, it should not be possible to unpack it into 2 variables.

x, y = 1 generates an error
x, y = (i for i in range(10)) too

but not

x_increment, y_increment = (scale * i for i in srcpages.xobj_box[2:])

why ?

From pieter-l at vanoostrum.org  Thu Nov  4 08:11:37 2021
From: pieter-l at vanoostrum.org (Pieter van Oostrum)
Date: Thu, 04 Nov 2021 13:11:37 +0100
Subject: Recursion on list
References: <6183925b$0$8912$426a74cc@news.free.fr>
Message-ID: <m24k8sruh2.fsf@vanoostrum.org>

ast <ast at invalid> writes:

>> li = []
>> li.append(li)
>> li
> [[...]]
>
>>li[0][0][0][0]
> [[...]]
>
> That's funny
>

You made a list whose only element is itself.

In [1]: li = []

In [3]: li.append(li)

In [4]: li[0] is li
Out[4]: True

-- 
Pieter van Oostrum <pieter at vanoostrum.org>
www: http://pieter.vanoostrum.org/
PGP key: [8DAE142BE17999C4]

From pkpearson at nowhere.invalid  Thu Nov  4 11:00:59 2021
From: pkpearson at nowhere.invalid (Peter Pearson)
Date: 4 Nov 2021 15:00:59 GMT
Subject: Recursion on list
References: <6183925b$0$8912$426a74cc@news.free.fr>
Message-ID: <iuiatbF1hu1U1@mid.individual.net>

On Thu, 4 Nov 2021 08:57:14 +0100, ast <ast at invalid> wrote:
> > li = []
> > li.append(li)
> > li
> [[...]]
>
> >li[0][0][0][0]
> [[...]]
>
> That's funny

After the coming AI upheaval, such cruelty to machines will
be considered punishable and not funny.


-- 
To email me, substitute nowhere->runbox, invalid->com.

From ast at invalid  Thu Nov  4 12:09:24 2021
From: ast at invalid (ast)
Date: Thu, 4 Nov 2021 17:09:24 +0100
Subject: Syntax not understood
In-Reply-To: <assignment-20211104164019@ram.dialup.fu-berlin.de>
References: <6183c75c$0$29473$426a74cc@news.free.fr>
 <assignment-20211104164019@ram.dialup.fu-berlin.de>
Message-ID: <618405b5$0$4998$426a34cc@news.free.fr>

Le 04/11/2021 ? 16:41, Stefan Ram a ?crit?:
> ast <ast at invalid> writes:
>> (scale * i for i in srcpages.xobj_box[2:]) is a generator, a single
>> object, it should not be possible to unpack it into 2 variables.
> 
>    But the value of the right-hand side /always/ is a single object!
> 
>    A syntax of an assignment statement that has been simplified
>    by me but is sufficient for this post is:
> 
> target list = source expression
> 
>    . The evaluation of the source expression yields an object.
> 
>    If the target list is not a single target, that object must
>    be an iterable with the same number of items as there are targets
>    in the target list, and the items are assigned, from left to
>    right, to the corresponding targets.
> 
>    A generator object /is/ an iterable, and, here, it apparently
>    yields exactly two items.
> 
> 

understood

It is like:

x, y = (i for i in range(2))

thx

From david at lowryduda.com  Thu Nov  4 14:36:48 2021
From: david at lowryduda.com (David Lowry-Duda)
Date: Thu, 4 Nov 2021 14:36:48 -0400
Subject: Syntax not understood
In-Reply-To: <6183c75c$0$29473$426a74cc@news.free.fr>
References: <6183c75c$0$29473$426a74cc@news.free.fr>
Message-ID: <YYQoQJEB1qTa/k7j@icerm-dld>

> x_increment, y_increment = (scale * i for i in srcpages.xobj_box[2:])
> 
> (scale * i for i in srcpages.xobj_box[2:]) is a generator, a single
> object, it should not be possible to unpack it into 2 variables.

If you know the exact number of values in the generator, you can do 
this. Here is an oversimplified example.

  l = [0, 1, 2, 3, 4, 5]
  a, b = (elem * 10 for elem in l[:4])
  print(a, b) # prints 40 50

This is very fragile code and I would recommend against using it.

- DLD

From 2QdxY4RzWzUUiLuE at potatochowder.com  Thu Nov  4 15:21:58 2021
From: 2QdxY4RzWzUUiLuE at potatochowder.com (2QdxY4RzWzUUiLuE at potatochowder.com)
Date: Thu, 4 Nov 2021 12:21:58 -0700
Subject: Syntax not understood
In-Reply-To: <YYQoQJEB1qTa/k7j@icerm-dld>
References: <6183c75c$0$29473$426a74cc@news.free.fr>
 <YYQoQJEB1qTa/k7j@icerm-dld>
Message-ID: <YYQy1sEOudjiA2oH@scrozzle>

On 2021-11-04 at 14:36:48 -0400,
David Lowry-Duda <david at lowryduda.com> wrote:

> > x_increment, y_increment = (scale * i for i in srcpages.xobj_box[2:])
> > 
> > (scale * i for i in srcpages.xobj_box[2:]) is a generator, a single
> > object, it should not be possible to unpack it into 2 variables.
> 
> If you know the exact number of values in the generator, you can do 
> this. Here is an oversimplified example.
> 
>   l = [0, 1, 2, 3, 4, 5]
>   a, b = (elem * 10 for elem in l[:4])
>   print(a, b) # prints 40 50
> 
> This is very fragile code and I would recommend against using it.

How is that any more fragile than any other operation that destructs (or
doesn't) a tuple?

Is the following fragile:

    quotient, remainder = divmod(numerator, denominator)

Would you surround it with try/except, or always unpack it defensively?

    result = divmod(numerator, denominator)
    if len(result) == 2:
        quotient, remainder = result
    else:
        raise SomeException("divmod didn't work")

If I know where a generator (or an iterator, or a tuple, or a list) came
from, and it's documented to contain/yield a known quantity of values,
then why is it fragile to depend on that?

On the other hand, yes, if I build code that depends on a lot of
unstated relationships between values of questionable (or unknown)
origin, then I am definitely asking for trouble.

From rosuav at gmail.com  Thu Nov  4 15:28:34 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 5 Nov 2021 06:28:34 +1100
Subject: Syntax not understood
In-Reply-To: <YYQy1sEOudjiA2oH@scrozzle>
References: <6183c75c$0$29473$426a74cc@news.free.fr>
 <YYQoQJEB1qTa/k7j@icerm-dld> <YYQy1sEOudjiA2oH@scrozzle>
Message-ID: <CAPTjJmqzrj494GQ=L_8S6DR3Gcjdc7=ObNkUtJJdJ2PTvbopCQ@mail.gmail.com>

On Fri, Nov 5, 2021 at 6:23 AM <2QdxY4RzWzUUiLuE at potatochowder.com> wrote:
>
> On 2021-11-04 at 14:36:48 -0400,
> David Lowry-Duda <david at lowryduda.com> wrote:
>
> > > x_increment, y_increment = (scale * i for i in srcpages.xobj_box[2:])
> > >
> > > (scale * i for i in srcpages.xobj_box[2:]) is a generator, a single
> > > object, it should not be possible to unpack it into 2 variables.
> >
> > If you know the exact number of values in the generator, you can do
> > this. Here is an oversimplified example.
> >
> >   l = [0, 1, 2, 3, 4, 5]
> >   a, b = (elem * 10 for elem in l[:4])
> >   print(a, b) # prints 40 50
> >
> > This is very fragile code and I would recommend against using it.
>
> How is that any more fragile than any other operation that destructs (or
> doesn't) a tuple?
>

The only part that's fragile, in my opinion, is (from the initial
post) that it's using removal syntax to slice off some, which is
fragile against the original input length:

x_increment, y_increment = (scale * i for i in srcpages.xobj_box[2:])

But since it says "box", I would start by assuming that it has four
elements. (They might be x1,y1,x2,y2 or x,y,w,h but it'll almost
always be four.) So it's not TOO fragile, when working with boxes, but
it is a little less clear that thing[2:] will always give exactly two
results.

ChrisA

From 2QdxY4RzWzUUiLuE at potatochowder.com  Thu Nov  4 15:44:20 2021
From: 2QdxY4RzWzUUiLuE at potatochowder.com (2QdxY4RzWzUUiLuE at potatochowder.com)
Date: Thu, 4 Nov 2021 12:44:20 -0700
Subject: Syntax not understood
In-Reply-To: <CAPTjJmqzrj494GQ=L_8S6DR3Gcjdc7=ObNkUtJJdJ2PTvbopCQ@mail.gmail.com>
References: <6183c75c$0$29473$426a74cc@news.free.fr>
 <YYQoQJEB1qTa/k7j@icerm-dld> <YYQy1sEOudjiA2oH@scrozzle>
 <CAPTjJmqzrj494GQ=L_8S6DR3Gcjdc7=ObNkUtJJdJ2PTvbopCQ@mail.gmail.com>
Message-ID: <YYQ4FMqkD4xwW3Qw@scrozzle>

On 2021-11-05 at 06:28:34 +1100,
Chris Angelico <rosuav at gmail.com> wrote:

> On Fri, Nov 5, 2021 at 6:23 AM <2QdxY4RzWzUUiLuE at potatochowder.com> wrote:
> >
> > On 2021-11-04 at 14:36:48 -0400,
> > David Lowry-Duda <david at lowryduda.com> wrote:
> >
> > > > x_increment, y_increment = (scale * i for i in srcpages.xobj_box[2:])
> > > >
> > > > (scale * i for i in srcpages.xobj_box[2:]) is a generator, a single
> > > > object, it should not be possible to unpack it into 2 variables.
> > >
> > > If you know the exact number of values in the generator, you can do
> > > this. Here is an oversimplified example.
> > >
> > >   l = [0, 1, 2, 3, 4, 5]
> > >   a, b = (elem * 10 for elem in l[:4])
> > >   print(a, b) # prints 40 50
> > >
> > > This is very fragile code and I would recommend against using it.
> >
> > How is that any more fragile than any other operation that destructs (or
> > doesn't) a tuple?
> >
> 
> The only part that's fragile, in my opinion, is (from the initial
> post) that it's using removal syntax to slice off some, which is
> fragile against the original input length:
> 
> x_increment, y_increment = (scale * i for i in srcpages.xobj_box[2:])
> 
> But since it says "box", I would start by assuming that it has four
> elements. (They might be x1,y1,x2,y2 or x,y,w,h but it'll almost
> always be four.) So it's not TOO fragile, when working with boxes, but
> it is a little less clear that thing[2:] will always give exactly two
> results.

I won't disagree that most boxes are going to be described by four
elements, in which case slicing off the first two *should* leave two
more (presumably x_increment and y_increment).

So if you're saying that all that's missing is a description of
srcpages.xobj_box, then I agree.  :-)  But such is life on a mailing
list where posters don't always give their questions the context they
deserve.

From rosuav at gmail.com  Thu Nov  4 16:02:20 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 5 Nov 2021 07:02:20 +1100
Subject: Syntax not understood
In-Reply-To: <YYQ4FMqkD4xwW3Qw@scrozzle>
References: <6183c75c$0$29473$426a74cc@news.free.fr>
 <YYQoQJEB1qTa/k7j@icerm-dld> <YYQy1sEOudjiA2oH@scrozzle>
 <CAPTjJmqzrj494GQ=L_8S6DR3Gcjdc7=ObNkUtJJdJ2PTvbopCQ@mail.gmail.com>
 <YYQ4FMqkD4xwW3Qw@scrozzle>
Message-ID: <CAPTjJmo0uBvTZzxdox5C4t6zF+e6fhnFhx=b+6rwD1RD7qDKZQ@mail.gmail.com>

On Fri, Nov 5, 2021 at 6:45 AM <2QdxY4RzWzUUiLuE at potatochowder.com> wrote:
> > But since it says "box", I would start by assuming that it has four
> > elements. (They might be x1,y1,x2,y2 or x,y,w,h but it'll almost
> > always be four.) So it's not TOO fragile, when working with boxes, but
> > it is a little less clear that thing[2:] will always give exactly two
> > results.
>
> I won't disagree that most boxes are going to be described by four
> elements, in which case slicing off the first two *should* leave two
> more (presumably x_increment and y_increment).

Exactly. To an experienced Python programmer, it's obvious that that
line of code is scaling the dimensions and assigning them. But if you
aren't comfortable with that, then it isn't so obvious.

Perhaps it'd be better to spell it as [-2:], which will clearly give
two results regardless of the length of the input (unless it's too
short). But that's minor IMO.

> So if you're saying that all that's missing is a description of
> srcpages.xobj_box, then I agree.  :-)  But such is life on a mailing
> list where posters don't always give their questions the context they
> deserve.

Indeed :)

ChrisA

From praneshkumarpj at gmail.com  Fri Nov  5 04:20:52 2021
From: praneshkumarpj at gmail.com (pranesh kumar)
Date: Fri, 5 Nov 2021 13:50:52 +0530
Subject: Vnev issue - AttributeError: module 'sysconfig' has no
 attribute '_get_default_scheme'. Did you mean: 'get_default_scheme'?
Message-ID: <150E7253-E4FF-4249-8007-A433E7247B25@hxcore.ol>



   Hi

   Facing problem in creating virtual environment and in importing modules in
   pycharm community version 2021.2.2.

   Trying to import Open CV in pycharm but not able to complete

   When creating new virtual environment error I got is "AttributeError:
   module 'sysconfig' has no attribute '_get_default_scheme'. Did you mean:
   'get_default_scheme'?"



   Pranesh Kumar

   9789411538



   Your inputs could be so grateful on this issue

   Sent from [1]Mail for Windows





References

   Visible links
   1. https://go.microsoft.com/fwlink/?LinkId=550986

From torque.india at gmail.com  Fri Nov  5 14:31:04 2021
From: torque.india at gmail.com (OmPs)
Date: Sat, 6 Nov 2021 00:01:04 +0530
Subject: Vnev issue - AttributeError: module 'sysconfig' has no attribute
 '_get_default_scheme'. Did you mean: 'get_default_scheme'?
In-Reply-To: <150E7253-E4FF-4249-8007-A433E7247B25@hxcore.ol>
References: <150E7253-E4FF-4249-8007-A433E7247B25@hxcore.ol>
Message-ID: <CAFo8oMBbyN_qA2Yu8co=nOcvB=scDGsU1Agm2HdBHcN3bRqvYg@mail.gmail.com>

How are you creating the virtualenv. Creating virtualenv is pretty
straightforward. Please share what you are doing?

On Fri, 5 Nov 2021 at 10:15 PM, pranesh kumar <praneshkumarpj at gmail.com>
wrote:

>
>
>    Hi
>
>    Facing problem in creating virtual environment and in importing modules
> in
>    pycharm community version 2021.2.2.
>
>    Trying to import Open CV in pycharm but not able to complete
>
>    When creating new virtual environment error I got is "AttributeError:
>    module 'sysconfig' has no attribute '_get_default_scheme'. Did you mean:
>    'get_default_scheme'?"
>
>
>
>    Pranesh Kumar
>
>    9789411538
>
>
>
>    Your inputs could be so grateful on this issue
>
>    Sent from [1]Mail for Windows
>
>
>
>
>
> References
>
>    Visible links
>    1. https://go.microsoft.com/fwlink/?LinkId=550986
> --
> https://mail.python.org/mailman/listinfo/python-list
>

From lizzyhollins99 at gmail.com  Fri Nov  5 16:08:08 2021
From: lizzyhollins99 at gmail.com (Betty Hollinshead)
Date: Fri, 5 Nov 2021 13:08:08 -0700 (PDT)
Subject: Strange Change Of Module Name
Message-ID: <6e1aba78-9d32-4991-a2da-87bf78c1a34an@googlegroups.com>

python3-3.10.0-1.fc35.x86_64
alacarte-3.36.0-6.fc35.noarch

Someone in the Python world changed the name of an import.
Once upon a time the import (in alacarte) it looked like this:
     from collections import Sequence
     
In a standard Fedora 35 install a few days ago, alacarte crashes.
....because the import (now) needs to look like this:
     from collections.abc import Sequence
     
Am I alone in wondering about this sort of stupid change of name? 

L.

From rosuav at gmail.com  Fri Nov  5 17:52:55 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 6 Nov 2021 08:52:55 +1100
Subject: Strange Change Of Module Name
In-Reply-To: <6e1aba78-9d32-4991-a2da-87bf78c1a34an@googlegroups.com>
References: <6e1aba78-9d32-4991-a2da-87bf78c1a34an@googlegroups.com>
Message-ID: <CAPTjJmrhqMBmC5i3p1cXF_YkQEDDvbj-hkegHJLsDmvwcpX78g@mail.gmail.com>

On Sat, Nov 6, 2021 at 8:42 AM Betty Hollinshead
<lizzyhollins99 at gmail.com> wrote:
>
> python3-3.10.0-1.fc35.x86_64
> alacarte-3.36.0-6.fc35.noarch
>
> Someone in the Python world changed the name of an import.
> Once upon a time the import (in alacarte) it looked like this:
>      from collections import Sequence
>
> In a standard Fedora 35 install a few days ago, alacarte crashes.
> ....because the import (now) needs to look like this:
>      from collections.abc import Sequence
>
> Am I alone in wondering about this sort of stupid change of name?
>

Calling it "stupid" isn't a great way to get a useful discussion.

I would say that alacarte has been buggy for the better part of a
decade, and you're only now getting an actual error. Here's the change
- it wasn't a rename at all:

https://docs.python.org/3/whatsnew/3.9.html#you-should-check-for-deprecationwarning-in-your-code

The "collections.Sequence" alias has been kept for backward
compatibility, but as of 3.10, has been removed. The correct spelling,
"collections.abc.Sequence", has been valid since at least 2012 (that's
when it started emitting a warning - don't know if it was available
longer and I don't have a Python 3.2 to test). So either alacarte
hasn't been updated in many many years, or nobody has bothered to
check the warnings.

Maybe, rather than calling the Python devs "stupid", you should
consider sending pull requests to projects you use, fixing things that
are raising warnings?

ChrisA

From lukasz at langa.pl  Fri Nov  5 19:16:15 2021
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Sat, 6 Nov 2021 00:16:15 +0100
Subject: [RELEASE] Python 3.9.8 and 3.11.0a2 are now available
Message-ID: <7B794818-D45A-4CF5-A611-D85ADC8182E9@langa.pl>

Tcl/Tk updates

With the recent release of macOS 12 Monterey, we noticed that tkinter file dialogs are failing to show up on this new operating system, including in our built-in IDLE. Thanks to rapid help from the Tk team, and Marc Culler in particular, we were able to fix the issue by bundling Python 3.9.8 and Python 3.11.0a2 with a fixed Tcl/Tk version. In 3.9.8 it?s a patched 8.6.11 release while 3.11.0a2 is rocking the bleeding-edge 8.6.12rc1.

Since the issue also affected our latest stable version, 3.10.0, an updated macOS installer for this version was issued. You can recognize it by the post2 version appendix: python-3.10.0post2-macos11.pkg <https://www.python.org/ftp/python/3.10.0/python-3.10.0post2-macos11.pkg>. We didn?t have to bump the version number of Python itself since there are no Python source differences between this package and the original 3.10.0. The only difference is the bundled patched Tcl/Tk library.

Initially the original 3.10.0 installer was removed from the website after all URLs got updated to point to the patched version but it turned out this breaks some workflows <https://discuss.python.org/t/disappearing-macos-packages-on-python-org/11737> so the patched installer is now also available under the original filename.

 <https://discuss.python.org/t/python-3-9-8-and-3-11-0a2-are-now-available/11763#python-398-2>Python 3.9.8

Get it here: https://www.python.org/downloads/release/python-398/ <https://www.python.org/downloads/release/python-398/>

Python 3.9.8 is the seventh maintenance release of the legacy 3.9 series. Python 3.10 is now the latest feature release series of Python 3. Get the latest release of 3.10.x here <https://python.org/downloads/>.
There?s been 202 commits since 3.9.7 which shows that there?s still considerable interest in improving Python 3.9. To compare, at the same stage of the release cycle Python 3.8 received over 25% fewer commits. See the changelog <https://docs.python.org/release/3.9.8/whatsnew/changelog.html> for details on what changed.

On macOS, the default installer is now the new universal2 variant. It?s compatible with Mac OS X 10.9 and newer, including macOS 11 Big Sur and macOS 12 Monterey. You may need to upgrade third-party components, like pip, to the newest versions. You may experience differences in behavior in IDLE and other Tk-based applications due to using the newest version of Tk. As always, if you encounter problems when using this installer variant, please check https://bugs.python.org <https://bugs.python.org/> for existing reports and for opening new issues.

The next Python 3.9 maintenance release will be 3.9.9, currently scheduled for 2022-01-03.

 <https://discuss.python.org/t/python-3-9-8-and-3-11-0a2-are-now-available/11763#python-3110a2-3>Python 3.11.0a2

Get it here: https://www.python.org/downloads/release/python-3110a2/ <https://www.python.org/downloads/release/python-3110a2/>

Python 3.11 is still in development. This release, 3.11.0a2 is the second of seven planned alpha releases.
Alpha releases are intended to make it easier to test the current state of new features and bug fixes and to test the release process.

During the alpha phase, features may be added up until the start of the beta phase (2022-05-06) and, if necessary, may be modified or deleted up until the release candidate phase (2022-08-01). Please keep in mind that this is a preview release and its use is not recommended for production environments.

Many new features for Python 3.11 are still being planned and written. Among the new major new features and changes so far:

PEP 657 <https://www.python.org/dev/peps/pep-0657/> ? Include Fine-Grained Error Locations in Tracebacks
The Faster CPython Project 1 <https://github.com/faster-cpython> is already yielding some exciting results: this version of CPython 3.11 is ~12% faster on the geometric mean of the PyPerformance benchmarks <>, compared to 3.10.0.
(Hey, fellow core developer, if a feature you find important is missing from this list, let Pablo know <mailto:pablogsal at python.org>.)
The next pre-release of Python 3.11 will be 3.11.0a3, currently scheduled for Monday, 2021-12-06.

 <https://discuss.python.org/t/python-3-9-8-and-3-11-0a2-are-now-available/11763#we-hope-you-enjoy-the-new-releases-4>We hope you enjoy the new releases

Your friendly release team,
Ned Deily @nad <https://discuss.python.org/u/nad>
Pablo Galindo Salgado @pablogsal <https://discuss.python.org/u/pablogsal>
Steve Dower @steve.dower <https://discuss.python.org/u/steve.dower>
?ukasz Langa @ambv <https://discuss.python.org/u/ambv>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <https://mail.python.org/pipermail/python-list/attachments/20211106/955ab699/attachment.sig>

From nt_mahmood at yahoo.com  Sat Nov  6 12:16:15 2021
From: nt_mahmood at yahoo.com (Mahmood Naderan)
Date: Sat, 6 Nov 2021 16:16:15 +0000 (UTC)
Subject: Problem with concatenating two dataframes
References: <1670105892.779098.1636215375137.ref@mail.yahoo.com>
Message-ID: <1670105892.779098.1636215375137@mail.yahoo.com>

In the following code, I am trying to create some key-value pairs in a dictionary where the first element is a name and the second element is a dataframe.

# Creating a dictionary
data = {'Value':[0,0,0]}
kernel_df = pd.DataFrame(data, index=['M1','M2','M3'])
dict = {'dummy':kernel_df}
# dummy??->??????????Value
#?????????????? M1??????0
#?????????????? M2??????0
#?????????????? M3??????0


Then I read a file and create some batches and compare the name in the batch with the stored names in dictionary. If it doesn't exist, a new key-value (name and dataframe) is created. Otherwise, the Value column is appended to the existing dataframe.


df = pd.read_csv('test.batch.csv')
print(df)
for i in range(0, len(df), 3):
????print("\n------BATCH BEGIN")
????batch_df = df.iloc[i:i+3]
????name = batch_df.loc[i].at["Name"]
????values = batch_df.loc[:,["Value"]]
????print(name)
????print(values)
????print("------BATCH END")
????if name in dict:
????????# Append values to the existing key
??????? dict[name] = pd.concat( dict[name],values )?? #### ERROR 
????else:
????????# Create a new pair in dictionary
????????dict[name] = values;



As you can see in the output, the join statement has error.



 ?? ID Name Metric??Value
0?? 0?? K1???? M1???? 10
1?? 0?? K1???? M2??????5
2?? 0?? K1???? M3???? 10
3?? 1?? K2???? M1???? 20
4?? 1?? K2???? M2???? 10
5?? 1?? K2???? M3???? 15
6?? 2?? K1???? M1??????2
7?? 2?? K1???? M2??????2
8?? 2?? K1???? M3??????2

------BATCH BEGIN
K1
?? Value
0???? 10
1??????5
2???? 10
------BATCH END

------BATCH BEGIN
K2
?? Value
3???? 20
4???? 10
5???? 15
------BATCH END

------BATCH BEGIN
K1
?? Value
6??????2
7??????2
8??????2
------BATCH END




As it reaches the contact() statement, I get this error:

TypeError: first argument must be an iterable of pandas objects, you passed an object of type "DataFrame"


Based on the definition I wrote in the beginning of the code, "dict[name]" should be a dataframe. Isn't that?

How can I fix that?



Regards,
Mahmood

From python at mrabarnett.plus.com  Sat Nov  6 15:50:56 2021
From: python at mrabarnett.plus.com (MRAB)
Date: Sat, 6 Nov 2021 19:50:56 +0000
Subject: Problem with concatenating two dataframes
In-Reply-To: <1670105892.779098.1636215375137@mail.yahoo.com>
References: <1670105892.779098.1636215375137.ref@mail.yahoo.com>
 <1670105892.779098.1636215375137@mail.yahoo.com>
Message-ID: <737b7bd8-b718-2c4a-ee02-da077172bf4d@mrabarnett.plus.com>

On 2021-11-06 16:16, Mahmood Naderan via Python-list wrote:
> In the following code, I am trying to create some key-value pairs in a dictionary where the first element is a name and the second element is a dataframe.
> 
> # Creating a dictionary
> data = {'Value':[0,0,0]}
> kernel_df = pd.DataFrame(data, index=['M1','M2','M3'])
> dict = {'dummy':kernel_df}
> # dummy??->??????????Value
> #?????????????? M1??????0
> #?????????????? M2??????0
> #?????????????? M3??????0
> 
> 
> Then I read a file and create some batches and compare the name in the batch with the stored names in dictionary. If it doesn't exist, a new key-value (name and dataframe) is created. Otherwise, the Value column is appended to the existing dataframe.
> 
> 
> df = pd.read_csv('test.batch.csv')
> print(df)
> for i in range(0, len(df), 3):
>  ????print("\n------BATCH BEGIN")
>  ????batch_df = df.iloc[i:i+3]
>  ????name = batch_df.loc[i].at["Name"]
>  ????values = batch_df.loc[:,["Value"]]
>  ????print(name)
>  ????print(values)
>  ????print("------BATCH END")
>  ????if name in dict:
>  ????????# Append values to the existing key
>  ??????? dict[name] = pd.concat( dict[name],values )?? #### ERROR
>  ????else:
>  ????????# Create a new pair in dictionary
>  ????????dict[name] = values;
> 
> 
> 
> As you can see in the output, the join statement has error.
> 
> 
> 
>   ?? ID Name Metric??Value
> 0?? 0?? K1???? M1???? 10
> 1?? 0?? K1???? M2??????5
> 2?? 0?? K1???? M3???? 10
> 3?? 1?? K2???? M1???? 20
> 4?? 1?? K2???? M2???? 10
> 5?? 1?? K2???? M3???? 15
> 6?? 2?? K1???? M1??????2
> 7?? 2?? K1???? M2??????2
> 8?? 2?? K1???? M3??????2
> 
> ------BATCH BEGIN
> K1
>  ?? Value
> 0???? 10
> 1??????5
> 2???? 10
> ------BATCH END
> 
> ------BATCH BEGIN
> K2
>  ?? Value
> 3???? 20
> 4???? 10
> 5???? 15
> ------BATCH END
> 
> ------BATCH BEGIN
> K1
>  ?? Value
> 6??????2
> 7??????2
> 8??????2
> ------BATCH END
> 
> 
> 
> 
> As it reaches the contact() statement, I get this error:
> 
> TypeError: first argument must be an iterable of pandas objects, you passed an object of type "DataFrame"
> 
> 
> Based on the definition I wrote in the beginning of the code, "dict[name]" should be a dataframe. Isn't that?
> 
> How can I fix that?
> 
You're trying to concatenate by passing the 2 items as the first 2 
arguments to pd.concat, but I think that you're supposed to pass them as 
an _iterable_, e.g. a list, as the first argument to pd.concat.

Try this instead:

     dict[name] = pd.concat([dict[name], values])

From nt_mahmood at yahoo.com  Sat Nov  6 16:12:13 2021
From: nt_mahmood at yahoo.com (Mahmood Naderan)
Date: Sat, 6 Nov 2021 20:12:13 +0000 (UTC)
Subject: Problem with concatenating two dataframes
In-Reply-To: <737b7bd8-b718-2c4a-ee02-da077172bf4d@mrabarnett.plus.com>
References: <1670105892.779098.1636215375137.ref@mail.yahoo.com>
 <1670105892.779098.1636215375137@mail.yahoo.com>
 <737b7bd8-b718-2c4a-ee02-da077172bf4d@mrabarnett.plus.com>
Message-ID: <1170888568.656209.1636229533285@mail.yahoo.com>

>Try this instead:
>
>
>? ? dict[name] = pd.concat([dict[name], values])


OK. That fixed the problem, however, I see that they are concatenated vertically. How can I change that to horizontal? The printed dictionary in the end looks like


{'dummy':???? Value
M1????? 0
M2????? 0
M3????? 0, 'K1':??? Value
0???? 10
1????? 5
2???? 10
6????? 2
7????? 2
8????? 2, 'K2':??? Value
3???? 20
4???? 10
5???? 15}



For K1, there should be three rows and two columns labeled as Value.




Regards,
Mahmood






From python at mrabarnett.plus.com  Sat Nov  6 16:37:51 2021
From: python at mrabarnett.plus.com (MRAB)
Date: Sat, 6 Nov 2021 20:37:51 +0000
Subject: Problem with concatenating two dataframes
In-Reply-To: <1170888568.656209.1636229533285@mail.yahoo.com>
References: <1670105892.779098.1636215375137.ref@mail.yahoo.com>
 <1670105892.779098.1636215375137@mail.yahoo.com>
 <737b7bd8-b718-2c4a-ee02-da077172bf4d@mrabarnett.plus.com>
 <1170888568.656209.1636229533285@mail.yahoo.com>
Message-ID: <d1bf9117-9867-0606-d082-11e8fe3abf0a@mrabarnett.plus.com>


On 2021-11-06 20:12, Mahmood Naderan wrote:
> >Try this instead:
> >
> >
> >? ? dict[name] = pd.concat([dict[name], values])
>
>
> OK. That fixed the problem, however, I see that they are concatenated vertically. How can I change that to horizontal? The printed dictionary in the end looks like
>
>
> {'dummy':???? Value
> M1????? 0
> M2????? 0
> M3????? 0, 'K1':??? Value
> 0???? 10
> 1????? 5
> 2???? 10
> 6????? 2
> 7????? 2
> 8????? 2, 'K2':??? Value
> 3???? 20
> 4???? 10
> 5???? 15}
>
>
>
> For K1, there should be three rows and two columns labeled as Value.
>
The second argument of pd.concat is 'axis', which defaults to 0. Try 
using 1 instead of 0.

From nt_mahmood at yahoo.com  Sat Nov  6 18:01:01 2021
From: nt_mahmood at yahoo.com (Mahmood Naderan)
Date: Sat, 6 Nov 2021 22:01:01 +0000 (UTC)
Subject: Problem with concatenating two dataframes
In-Reply-To: <d1bf9117-9867-0606-d082-11e8fe3abf0a@mrabarnett.plus.com>
References: <1670105892.779098.1636215375137.ref@mail.yahoo.com>
 <1670105892.779098.1636215375137@mail.yahoo.com>
 <737b7bd8-b718-2c4a-ee02-da077172bf4d@mrabarnett.plus.com>
 <1170888568.656209.1636229533285@mail.yahoo.com>
 <d1bf9117-9867-0606-d082-11e8fe3abf0a@mrabarnett.plus.com>
Message-ID: <1371877181.810127.1636236061880@mail.yahoo.com>

>The second argument of pd.concat is 'axis', which defaults to 0. Try
>using 1 instead of 0.


Unfortunately, that doesn't help...


dict[name] = pd.concat( [dict[name],values], axis=1 )



{'dummy':???? Value
M1??????0
M2??????0
M3??????0, 'K1':????Value??Value
0?? 10.0????NaN
1????5.0????NaN
2?? 10.0????NaN
6????NaN????2.0
7????NaN????2.0
8????NaN????2.0, 'K2':????Value
3???? 20
4???? 10
5???? 15}



Regards,
Mahmood




From Marco.Sulla.Python at gmail.com  Mon Nov  8 02:42:42 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Mon, 8 Nov 2021 08:42:42 +0100
Subject: Py_IS_TYPE(op, &PyDict_Type) does not work on MacOS
Message-ID: <CABbU2U-+06OrRNzZunGshQ9CRK5K=8BSCAJF7kDvy5UreZ793A@mail.gmail.com>

As you can read here:

https://github.com/Marco-Sulla/python-frozendict/issues/37

Py_IS_TYPE(op, &PyDict_Type) did not work on MacOS. I had to use PyDict_Check.

Why don't I have this problem with Linux?

PS: since I'm creating a modified version of dict, I copied the dict
internal source and I link against them. Maybe the problem is
correlated.

From barry at barrys-emacs.org  Mon Nov  8 13:30:43 2021
From: barry at barrys-emacs.org (Barry)
Date: Mon, 8 Nov 2021 18:30:43 +0000
Subject: Py_IS_TYPE(op, &PyDict_Type) does not work on MacOS
In-Reply-To: <CABbU2U-+06OrRNzZunGshQ9CRK5K=8BSCAJF7kDvy5UreZ793A@mail.gmail.com>
References: <CABbU2U-+06OrRNzZunGshQ9CRK5K=8BSCAJF7kDvy5UreZ793A@mail.gmail.com>
Message-ID: <078EC0D0-E084-40C1-9679-A18C77446527@barrys-emacs.org>



> On 8 Nov 2021, at 07:45, Marco Sulla <Marco.Sulla.Python at gmail.com> wrote:
> 
> ?As you can read here:
> 
> https://github.com/Marco-Sulla/python-frozendict/issues/37
> 
> Py_IS_TYPE(op, &PyDict_Type) did not work on MacOS. I had to use PyDict_Check.
> 
> Why don't I have this problem with Linux?
> 
> PS: since I'm creating a modified version of dict, I copied the dict
> internal source and I link against them. Maybe the problem is
> correlated.

You can see what I did for PyCXX at https://sourceforge.net/p/cxx/code/HEAD/tree/trunk/CXX/Src/IndirectPythonInterface.cxx

See the _DictCheck and ends up using PyObject_IsInstance.

My guess is that use PyDict_Check is a good and better for the future.

Barry

> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

From Joseph.Schachner at Teledyne.com  Mon Nov  8 14:29:42 2021
From: Joseph.Schachner at Teledyne.com (Schachner, Joseph)
Date: Mon, 8 Nov 2021 19:29:42 +0000
Subject: Problem with concatenating two dataframes
In-Reply-To: <1371877181.810127.1636236061880@mail.yahoo.com>
References: <1670105892.779098.1636215375137.ref@mail.yahoo.com>
 <1670105892.779098.1636215375137@mail.yahoo.com>
 <737b7bd8-b718-2c4a-ee02-da077172bf4d@mrabarnett.plus.com>
 <1170888568.656209.1636229533285@mail.yahoo.com>
 <d1bf9117-9867-0606-d082-11e8fe3abf0a@mrabarnett.plus.com>
 <1371877181.810127.1636236061880@mail.yahoo.com>
Message-ID: <BN8PR14MB28513EBD0792E10D54F19DBDF5919@BN8PR14MB2851.namprd14.prod.outlook.com>

The problem I see here is use of Pandas.   I know I have he losing opinion, but people who use Python to load Panadas and then only use Pandas are missing out on the functionality of Python.

I'll bet you could code combining this data in pure Python, into one dictionary. In fact I'd be shocked if you couldn't do it.

---- Joseph S.


Teledyne Confidential; Commercially Sensitive Business Data

-----Original Message-----
From: Mahmood Naderan <nt_mahmood at yahoo.com> 
Sent: Saturday, November 6, 2021 6:01 PM
To: python-list at python.org; MRAB <python at mrabarnett.plus.com>
Subject: Re: Problem with concatenating two dataframes

>The second argument of pd.concat is 'axis', which defaults to 0. Try 
>using 1 instead of 0.


Unfortunately, that doesn't help...


dict[name] = pd.concat( [dict[name],values], axis=1 )



{'dummy':???? Value
M1??????0
M2??????0
M3??????0, 'K1':????Value??Value
0?? 10.0????NaN
1????5.0????NaN
2?? 10.0????NaN
6????NaN????2.0
7????NaN????2.0
8????NaN????2.0, 'K2':????Value
3???? 20
4???? 10
5???? 15}



Regards,
Mahmood




From Marco.Sulla.Python at gmail.com  Wed Nov 10 11:40:55 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Wed, 10 Nov 2021 17:40:55 +0100
Subject: Py_IS_TYPE(op, &PyDict_Type) does not work on MacOS
In-Reply-To: <078EC0D0-E084-40C1-9679-A18C77446527@barrys-emacs.org>
References: <CABbU2U-+06OrRNzZunGshQ9CRK5K=8BSCAJF7kDvy5UreZ793A@mail.gmail.com>
 <078EC0D0-E084-40C1-9679-A18C77446527@barrys-emacs.org>
Message-ID: <CABbU2U8EqRawbUwGNdM==mSJmOULypqJhZum2m1oFtsQXrk2FA@mail.gmail.com>

Indeed now I use PyDict_Check, but anyway it's very strange that
Py_IS_TYPE(op, &PyDict_Type) does not work only on MacOS.

On Mon, 8 Nov 2021 at 19:30, Barry <barry at barrys-emacs.org> wrote:
>
>
>
> On 8 Nov 2021, at 07:45, Marco Sulla <Marco.Sulla.Python at gmail.com> wrote:
>
> ?As you can read here:
>
> https://github.com/Marco-Sulla/python-frozendict/issues/37
>
> Py_IS_TYPE(op, &PyDict_Type) did not work on MacOS. I had to use PyDict_Check.
>
> Why don't I have this problem with Linux?
>
> PS: since I'm creating a modified version of dict, I copied the dict
> internal source and I link against them. Maybe the problem is
> correlated.
>
>
> You can see what I did for PyCXX at https://sourceforge.net/p/cxx/code/HEAD/tree/trunk/CXX/Src/IndirectPythonInterface.cxx
>
> See the _DictCheck and ends up using PyObject_IsInstance.
>
> My guess is that use PyDict_Check is a good and better for the future.
>
> Barry
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>

From praneshkumarpj at gmail.com  Wed Nov 10 12:21:49 2021
From: praneshkumarpj at gmail.com (pranesh kumar)
Date: Wed, 10 Nov 2021 22:51:49 +0530
Subject: Vnev issue - AttributeError: module 'sysconfig' has no
 attribute '_get_default_scheme'. Did you mean: 'get_default_scheme'?
In-Reply-To: <CAFo8oMBbyN_qA2Yu8co=nOcvB=scDGsU1Agm2HdBHcN3bRqvYg@mail.gmail.com>
References: <150E7253-E4FF-4249-8007-A433E7247B25@hxcore.ol>,
 <CAFo8oMBbyN_qA2Yu8co=nOcvB=scDGsU1Agm2HdBHcN3bRqvYg@mail.gmail.com>
Message-ID: <A0704FD9-1C6E-4BC5-8F5F-63AFDB5129E4@hxcore.ol>

   Hi

   Shared a google sheet with details

   [1]https://docs.google.com/presentation/d/1W9wH08dfzA1XP5sqBBNj0bk0YPPGVG1z0-WQ3yt1NUU/edit?usp=sharing



   after importing OpenCV

   When typing "cv2." All functions related to the OpenCV module should show
   up which is not happening

   How to fix this issue.



   Thanks in advance

   Regards

   Pranesh Kumar

   9789411538



   Sent from [2]Mail for Windows



   From: [3]OmPs
   Sent: 06 November 2021 00:01
   To: [4]pranesh kumar
   Cc: [5]python-list at python.org
   Subject: Re: Vnev issue - AttributeError: module 'sysconfig' has no
   attribute '_get_default_scheme'. Did you mean: 'get_default_scheme'?



   How are you creating the virtualenv. Creating virtualenv is pretty
   straightforward. Please share what you are doing?



   On Fri, 5 Nov 2021 at 10:15 PM, pranesh kumar
   <[6]praneshkumarpj at gmail.com> wrote:

      Hi

      Facing problem in creating virtual environment and in importing modules
   in
      pycharm community version 2021.2.2.

      Trying to import Open CV in pycharm but not able to complete

      When creating new virtual environment error I got is "AttributeError:
      module 'sysconfig' has no attribute '_get_default_scheme'. Did you
   mean:
      'get_default_scheme'?"

      Pranesh Kumar

      9789411538

      Your inputs could be so grateful on this issue

      Sent from [1]Mail for Windows

   References

      Visible links
      1. [7]https://go.microsoft.com/fwlink/?LinkId=550986
   --
   [8]https://mail.python.org/mailman/listinfo/python-list



References

   Visible links
   1. https://docs.google.com/presentation/d/1W9wH08dfzA1XP5sqBBNj0bk0YPPGVG1z0-WQ3yt1NUU/edit?usp=sharing
   2. https://go.microsoft.com/fwlink/?LinkId=550986
   3. mailto:torque.india at gmail.com
   4. mailto:praneshkumarpj at gmail.com
   5. mailto:python-list at python.org
   6. mailto:praneshkumarpj at gmail.com
   7. https://go.microsoft.com/fwlink/?LinkId=550986
   8. https://mail.python.org/mailman/listinfo/python-list

From vijay.karavadra at marutitech.com  Wed Nov 10 00:16:03 2021
From: vijay.karavadra at marutitech.com (Vijay Karavadra)
Date: Wed, 10 Nov 2021 10:46:03 +0530
Subject: Breaking new relic format on a new line character in FileHandler
 appender
Message-ID: <CABeRhcojoyA63zHT1C607xTigJ=DpgLcujDEvJ+hdvcPDCQfxA@mail.gmail.com>

Hello Team,

I'm trying to add logs in the new relic platform from a python application.
For that, I've to add logs in a local file in a specific format which is

'{"log.level":"%(levelname)s", "log.entity.name":"my-service-name",
"message":"%(message)s"}'


This works fine in normal scenario and generates the below type of
line in log file and logs are added to new relic with expected
properties set like log level, entity name, message etc.

{"log.level":"INFO", "log.entity.name":"my-service-name", "message":"test"}


This issue occurs when a new line is present in the message property of the
log i.e. on any error we have a message like "Traceback (most recent call
last):" which represents an error traceback after this line with a new line
character. In this case, multiple logs are added instead of a single log,
my expected formatting breaks and logging properties are not set in the new
relic.

Sample breaking log:

{"log.level":"ERROR", "log.entity.name":"my-service-name",
"message":"[Errno 2] No such file or directory:
'E:\\comms-nlp-service\\src\\GoogleAuthentication\\xyz.json' -> Traceback
(most recent call last):
  File "E:\comms-nlp-service\src\app\nlp\carrier_cancel_apiai.py", line 31,
in query
    credentials =
service_account.Credentials.from_service_account_file(filename, scopes
=(scope,))
  File "C:\Python34\lib\site-packages\google\oauth2\service_account.py",
line 209, in from_service_account_file
    filename, require=['client_email', 'token_uri'])
  File
"C:\Python34\lib\site-packages\google\auth\_service_account_info.py", line
71, in from_filename
    with io.open(filename, 'r', encoding='utf-8') as json_file:
FileNotFoundError: [Errno 2] No such file or directory:
'E:\\comms-nlp-service\\src\\GoogleAuthentication\\ xyz  .json'
"}

Any suggestions on this would be a great help.

-- 
Thanks,
Vijay Karavadra

From p_d_a_s_i_l_v_a_ns at nonetnoaddress.pt  Wed Nov 10 16:37:52 2021
From: p_d_a_s_i_l_v_a_ns at nonetnoaddress.pt (Paulo da Silva)
Date: Wed, 10 Nov 2021 21:37:52 +0000
Subject: Avoid nested SIGINT handling
Message-ID: <smhe3h$12pi$1@gioia.aioe.org>

Hi!

How do I handle a SIGINT (or any other signal) avoid nesting?

Does this work?

class STATUS:
	InInt=False

def SIGINT_handler(sn,f):
	if STATUS.InInt: return
	STATUS.InInt=True
	process_int()
	STATUS.InInt=False

Thanks for any suggestions.
Paulo

From jon+usenet at unequivocal.eu  Wed Nov 10 16:55:05 2021
From: jon+usenet at unequivocal.eu (Jon Ribbens)
Date: Wed, 10 Nov 2021 21:55:05 -0000 (UTC)
Subject: Avoid nested SIGINT handling
References: <smhe3h$12pi$1@gioia.aioe.org>
Message-ID: <slrnsooftp.5h7.jon+usenet@raven.unequivocal.eu>

On 2021-11-10, Paulo da Silva <p_d_a_s_i_l_v_a_ns at nonetnoaddress.pt> wrote:
> Hi!
>
> How do I handle a SIGINT (or any other signal) avoid nesting?

I don't think you need to. Python will only call signal handlers in
the main thread, so a handler can't be executed while another handler
is running anyway.

From p_d_a_s_i_l_v_a_ns at nonetnoaddress.pt  Wed Nov 10 20:37:14 2021
From: p_d_a_s_i_l_v_a_ns at nonetnoaddress.pt (Paulo da Silva)
Date: Thu, 11 Nov 2021 01:37:14 +0000
Subject: Avoid nested SIGINT handling
References: <smhe3h$12pi$1@gioia.aioe.org>
 <slrnsooftp.5h7.jon+usenet@raven.unequivocal.eu>
Message-ID: <smhs4b$1q9$1@gioia.aioe.org>

?s 21:55 de 10/11/21, Jon Ribbens escreveu:
> On 2021-11-10, Paulo da Silva <p_d_a_s_i_l_v_a_ns at nonetnoaddress.pt> wrote:
>> Hi!
>>
>> How do I handle a SIGINT (or any other signal) avoid nesting?
> 
> I don't think you need to. Python will only call signal handlers in
> the main thread, so a handler can't be executed while another handler
> is running anyway.
> 

Do you mean that if I issue a ctrl+c while the previous one is
"processing" it is held until, at least, the "processing" returns?

From rosuav at gmail.com  Thu Nov 11 01:19:42 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 11 Nov 2021 17:19:42 +1100
Subject: Breaking new relic format on a new line character in FileHandler
 appender
In-Reply-To: <CABeRhcojoyA63zHT1C607xTigJ=DpgLcujDEvJ+hdvcPDCQfxA@mail.gmail.com>
References: <CABeRhcojoyA63zHT1C607xTigJ=DpgLcujDEvJ+hdvcPDCQfxA@mail.gmail.com>
Message-ID: <CAPTjJmoGg6DRXCZR5LsL9KpW7QfcA=XSvBbDQW8HqHnZuAkaMA@mail.gmail.com>

On Thu, Nov 11, 2021 at 5:00 PM Vijay Karavadra via Python-list
<python-list at python.org> wrote:
>
> Hello Team,
>
> I'm trying to add logs in the new relic platform from a python application.
> For that, I've to add logs in a local file in a specific format which is
>
> '{"log.level":"%(levelname)s", "log.entity.name":"my-service-name",
> "message":"%(message)s"}'
>

Looks like JSON to me. Have you considered using the json module to format it?

json.dumps({"log.level":levelname, "log.entity.name":"my-service-name",
"message":message})

ChrisA

From rosuav at gmail.com  Thu Nov 11 01:22:15 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 11 Nov 2021 17:22:15 +1100
Subject: Avoid nested SIGINT handling
In-Reply-To: <slrnsooftp.5h7.jon+usenet@raven.unequivocal.eu>
References: <smhe3h$12pi$1@gioia.aioe.org>
 <slrnsooftp.5h7.jon+usenet@raven.unequivocal.eu>
Message-ID: <CAPTjJmqgcFs8cPHS5ZQnVEixDQAgiJb94VutdvGspnGDszjVZQ@mail.gmail.com>

On Thu, Nov 11, 2021 at 5:01 PM Jon Ribbens via Python-list
<python-list at python.org> wrote:
>
> On 2021-11-10, Paulo da Silva <p_d_a_s_i_l_v_a_ns at nonetnoaddress.pt> wrote:
> > Hi!
> >
> > How do I handle a SIGINT (or any other signal) avoid nesting?
>
> I don't think you need to. Python will only call signal handlers in
> the main thread, so a handler can't be executed while another handler
> is running anyway.

Threads aren't the point here - signals happen immediately.

Would it be easier to catch KeyboardInterrupt and do your processing
there, rather than actually catching SIGINT?

I'd recommend just trying what you have, and seeing if it's reentrant.
My suspicion is that it isn't, on a technical level (the Python
function will be queued for when it's safe to call it - probably after
the next bytecode instruction), but that your own code will still need
to worry about reentrancy.

ChrisA

From p_d_a_s_i_l_v_a_ns at nonetnoaddress.pt  Thu Nov 11 12:18:52 2021
From: p_d_a_s_i_l_v_a_ns at nonetnoaddress.pt (Paulo da Silva)
Date: Thu, 11 Nov 2021 17:18:52 +0000
Subject: Avoid nested SIGINT handling
References: <smhe3h$12pi$1@gioia.aioe.org>
 <slrnsooftp.5h7.jon+usenet@raven.unequivocal.eu>
 <CAPTjJmqgcFs8cPHS5ZQnVEixDQAgiJb94VutdvGspnGDszjVZQ@mail.gmail.com>
 <mailman.196.1636611749.23718.python-list@python.org>
Message-ID: <smjj9s$loi$1@gioia.aioe.org>

?s 06:22 de 11/11/21, Chris Angelico escreveu:
> On Thu, Nov 11, 2021 at 5:01 PM Jon Ribbens via Python-list
> <python-list at python.org> wrote:
>>
>> On 2021-11-10, Paulo da Silva <p_d_a_s_i_l_v_a_ns at nonetnoaddress.pt> wrote:
>>> Hi!
>>>
>>> How do I handle a SIGINT (or any other signal) avoid nesting?
>>
>> I don't think you need to. Python will only call signal handlers in
>> the main thread, so a handler can't be executed while another handler
>> is running anyway.
> 
> Threads aren't the point here - signals happen immediately.
> 
> Would it be easier to catch KeyboardInterrupt and do your processing
> there, rather than actually catching SIGINT?
> 
> I'd recommend just trying what you have, and seeing if it's reentrant.
> My suspicion is that it isn't, on a technical level (the Python
> function will be queued for when it's safe to call it - probably after
> the next bytecode instruction), but that your own code will still need
> to worry about reentrancy.
> 
OK, thank you

From python at mrabarnett.plus.com  Thu Nov 11 11:51:24 2021
From: python at mrabarnett.plus.com (MRAB)
Date: Thu, 11 Nov 2021 16:51:24 +0000
Subject: Breaking new relic format on a new line character in FileHandler
 appender
In-Reply-To: <CABeRhcojoyA63zHT1C607xTigJ=DpgLcujDEvJ+hdvcPDCQfxA@mail.gmail.com>
References: <CABeRhcojoyA63zHT1C607xTigJ=DpgLcujDEvJ+hdvcPDCQfxA@mail.gmail.com>
Message-ID: <a7d00861-c99b-03bc-f433-a71108c1dadc@mrabarnett.plus.com>

On 2021-11-10 05:16, Vijay Karavadra via Python-list wrote:
> Hello Team,
> 
> I'm trying to add logs in the new relic platform from a python application.
> For that, I've to add logs in a local file in a specific format which is
> 
> '{"log.level":"%(levelname)s", "log.entity.name":"my-service-name",
> "message":"%(message)s"}'
> 
> 
> This works fine in normal scenario and generates the below type of
> line in log file and logs are added to new relic with expected
> properties set like log level, entity name, message etc.
> 
> {"log.level":"INFO", "log.entity.name":"my-service-name", "message":"test"}
> 
> 
> This issue occurs when a new line is present in the message property of the
> log i.e. on any error we have a message like "Traceback (most recent call
> last):" which represents an error traceback after this line with a new line
> character. In this case, multiple logs are added instead of a single log,
> my expected formatting breaks and logging properties are not set in the new
> relic.
> 
> Sample breaking log:
> 
> {"log.level":"ERROR", "log.entity.name":"my-service-name",
> "message":"[Errno 2] No such file or directory:
> 'E:\\comms-nlp-service\\src\\GoogleAuthentication\\xyz.json' -> Traceback
> (most recent call last):
>    File "E:\comms-nlp-service\src\app\nlp\carrier_cancel_apiai.py", line 31,
> in query
>      credentials =
> service_account.Credentials.from_service_account_file(filename, scopes
> =(scope,))
>    File "C:\Python34\lib\site-packages\google\oauth2\service_account.py",
> line 209, in from_service_account_file
>      filename, require=['client_email', 'token_uri'])
>    File
> "C:\Python34\lib\site-packages\google\auth\_service_account_info.py", line
> 71, in from_filename
>      with io.open(filename, 'r', encoding='utf-8') as json_file:
> FileNotFoundError: [Errno 2] No such file or directory:
> 'E:\\comms-nlp-service\\src\\GoogleAuthentication\\ xyz  .json'
> "}
> 
> Any suggestions on this would be a great help.
> 
How about formatting using 'r' or 'a' instead of 's'?

'{"log.level": "%(levelname)s", "log.entity.name": "my-service-name", 
"message": "%(message)r"}'

From Marco.Sulla.Python at gmail.com  Fri Nov 12 09:30:01 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Fri, 12 Nov 2021 15:30:01 +0100
Subject: Unable to compile my C Extension on Windows: unresolved external link
 errors
Message-ID: <CABbU2U-d0f=mD5DXxgnDr=qc=p-XsFfi83YCd=vcFEXURHJymw@mail.gmail.com>

I have no problem compiling my extension under Linux and MacOS. Under
Windows, I get a lot of

Error LNK2001: unresolved external symbol PyErr_SetObject

and so on.

I post the part of my setup.py about the C Extension:

extra_compile_args = ["-DPY_SSIZE_T_CLEAN", "-DPy_BUILD_CORE"]
undef_macros = []

setuptools.Extension(
    ext1_fullname,
    sources = cpython_sources,
    include_dirs = cpython_include_dirs,
    extra_compile_args = extra_compile_args,
    undef_macros = undef_macros,
)

Here is the full code:
https://github.com/Marco-Sulla/python-frozendict/blob/master/setup.py

Steps to reproduce: I installed python3.10 and VS compiler on my
Windows 10 machine, then I created a venv, activated it, run

pip install -U pip setuptools wheel

and then

python setup.py bdist_wheel

From gisle.vanem at gmail.com  Fri Nov 12 09:53:37 2021
From: gisle.vanem at gmail.com (Gisle Vanem)
Date: Fri, 12 Nov 2021 15:53:37 +0100
Subject: Unable to compile my C Extension on Windows: unresolved external
 link errors
In-Reply-To: <CABbU2U-d0f=mD5DXxgnDr=qc=p-XsFfi83YCd=vcFEXURHJymw@mail.gmail.com>
References: <CABbU2U-d0f=mD5DXxgnDr=qc=p-XsFfi83YCd=vcFEXURHJymw@mail.gmail.com>
Message-ID: <e8f1169e-8ea9-420b-5503-2befa5495b88@gmail.com>

Marco Sulla wrote:

> Error LNK2001: unresolved external symbol PyErr_SetObject
> 
> and so on.
> 
> I post the part of my setup.py about the C Extension:
> 
> extra_compile_args = ["-DPY_SSIZE_T_CLEAN", "-DPy_BUILD_CORE"]

Shouldn't this be "-DPy_BUILD_CORE_MODULE"?


-- 
--gv

From Marco.Sulla.Python at gmail.com  Fri Nov 12 11:26:30 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Fri, 12 Nov 2021 17:26:30 +0100
Subject: Unable to compile my C Extension on Windows: unresolved external
 link errors
In-Reply-To: <e8f1169e-8ea9-420b-5503-2befa5495b88@gmail.com>
References: <CABbU2U-d0f=mD5DXxgnDr=qc=p-XsFfi83YCd=vcFEXURHJymw@mail.gmail.com>
 <e8f1169e-8ea9-420b-5503-2befa5495b88@gmail.com>
Message-ID: <CABbU2U8Pp7KOVhzrHACVhx8ye4bx7XernrQ8GLjY+VFKv6ttHQ@mail.gmail.com>

On Fri, 12 Nov 2021 at 15:55, Gisle Vanem <gisle.vanem at gmail.com> wrote:
> Marco Sulla wrote:
> > Error LNK2001: unresolved external symbol PyErr_SetObject
> >
> > and so on.
> >
> > I post the part of my setup.py about the C Extension:
> >
> > extra_compile_args = ["-DPY_SSIZE_T_CLEAN", "-DPy_BUILD_CORE"]
>
> Shouldn't this be "-DPy_BUILD_CORE_MODULE"?

I tried it, but now I get three

error C2099: initializer is not a constant

when I try to compile dictobject.c. Yes, my extension needs dictobject.

From rosuav at gmail.com  Fri Nov 12 11:36:44 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 13 Nov 2021 03:36:44 +1100
Subject: Unable to compile my C Extension on Windows: unresolved external
 link errors
In-Reply-To: <CABbU2U8Pp7KOVhzrHACVhx8ye4bx7XernrQ8GLjY+VFKv6ttHQ@mail.gmail.com>
References: <CABbU2U-d0f=mD5DXxgnDr=qc=p-XsFfi83YCd=vcFEXURHJymw@mail.gmail.com>
 <e8f1169e-8ea9-420b-5503-2befa5495b88@gmail.com>
 <CABbU2U8Pp7KOVhzrHACVhx8ye4bx7XernrQ8GLjY+VFKv6ttHQ@mail.gmail.com>
Message-ID: <CAPTjJmqDPku3x_QcuV0OXcfzKEGyN+sww-JseMBwsfo-fw6OLg@mail.gmail.com>

On Sat, Nov 13, 2021 at 3:28 AM Marco Sulla
<Marco.Sulla.Python at gmail.com> wrote:
>
> On Fri, 12 Nov 2021 at 15:55, Gisle Vanem <gisle.vanem at gmail.com> wrote:
> > Marco Sulla wrote:
> > > Error LNK2001: unresolved external symbol PyErr_SetObject
> > >
> > > and so on.
> > >
> > > I post the part of my setup.py about the C Extension:
> > >
> > > extra_compile_args = ["-DPY_SSIZE_T_CLEAN", "-DPy_BUILD_CORE"]
> >
> > Shouldn't this be "-DPy_BUILD_CORE_MODULE"?
>
> I tried it, but now I get three
>
> error C2099: initializer is not a constant
>
> when I try to compile dictobject.c. Yes, my extension needs dictobject.

Are you sure that you really need Py_BUILD_CORE? Here's what the code
has to say about it:

https://github.com/python/cpython/blob/main/Include/pyport.h#L17

Py_BUILD_CORE: Build Python core. Give access to Python internals, but
should not be used by third-party modules.

If your extension truly needs to reach into the internals of CPython,
then you're going to have to figure a lot of things out for yourself,
including how to make sure it works on every platform. A more
productive approach would be to stick to the actual API:

https://docs.python.org/3/c-api/stable.html

What is it that doesn't work without Py_BUILD_CORE?

ChrisA

From Marco.Sulla.Python at gmail.com  Fri Nov 12 15:01:07 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Fri, 12 Nov 2021 21:01:07 +0100
Subject: Unable to compile my C Extension on Windows: unresolved external
 link errors
In-Reply-To: <CAPTjJmqDPku3x_QcuV0OXcfzKEGyN+sww-JseMBwsfo-fw6OLg@mail.gmail.com>
References: <CABbU2U-d0f=mD5DXxgnDr=qc=p-XsFfi83YCd=vcFEXURHJymw@mail.gmail.com>
 <e8f1169e-8ea9-420b-5503-2befa5495b88@gmail.com>
 <CABbU2U8Pp7KOVhzrHACVhx8ye4bx7XernrQ8GLjY+VFKv6ttHQ@mail.gmail.com>
 <CAPTjJmqDPku3x_QcuV0OXcfzKEGyN+sww-JseMBwsfo-fw6OLg@mail.gmail.com>
Message-ID: <CABbU2U_e4BqukXXzq=97Wxiph6m4EsNpk-tHCbPi5DUMp18YMg@mail.gmail.com>

Chris? Maybe I'm dreaming X-D

On Fri, 12 Nov 2021 at 17:38, Chris Angelico <rosuav at gmail.com> wrote:
> Are you sure that you really need Py_BUILD_CORE?

Yes, because I need the internal functions of `dict`. So I need to
compile also dictobject.c and include it. So I need that flag.

This is the code:

https://github.com/Marco-Sulla/python-frozendict.git

On Linux and MacOS it works like a charme. On Windows, it seems it does not find
python3.lib. I also added its path to the PATH variable

From rosuav at gmail.com  Fri Nov 12 15:07:36 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 13 Nov 2021 07:07:36 +1100
Subject: Unable to compile my C Extension on Windows: unresolved external
 link errors
In-Reply-To: <CABbU2U_e4BqukXXzq=97Wxiph6m4EsNpk-tHCbPi5DUMp18YMg@mail.gmail.com>
References: <CABbU2U-d0f=mD5DXxgnDr=qc=p-XsFfi83YCd=vcFEXURHJymw@mail.gmail.com>
 <e8f1169e-8ea9-420b-5503-2befa5495b88@gmail.com>
 <CABbU2U8Pp7KOVhzrHACVhx8ye4bx7XernrQ8GLjY+VFKv6ttHQ@mail.gmail.com>
 <CAPTjJmqDPku3x_QcuV0OXcfzKEGyN+sww-JseMBwsfo-fw6OLg@mail.gmail.com>
 <CABbU2U_e4BqukXXzq=97Wxiph6m4EsNpk-tHCbPi5DUMp18YMg@mail.gmail.com>
Message-ID: <CAPTjJmqSDxbvH6fGYEG6fKRQ1DoE6+CMGpUPoKURX3bVE6a_tg@mail.gmail.com>

On Sat, Nov 13, 2021 at 7:01 AM Marco Sulla
<Marco.Sulla.Python at gmail.com> wrote:
> On Fri, 12 Nov 2021 at 17:38, Chris Angelico <rosuav at gmail.com> wrote:
> > Are you sure that you really need Py_BUILD_CORE?
>
> Yes, because I need the internal functions of `dict`. So I need to
> compile also dictobject.c and include it. So I need that flag.
>
> This is the code:
>
> https://github.com/Marco-Sulla/python-frozendict.git
>

Ah, gotcha.

Unfortunately that does mean you're delving deep into internals, and a
lot of stuff that isn't designed for extensions to use. So my best
recommendation is: dig even deeper into internals, and duplicate how
the core is doing things (maybe including another header or
something). It may be that, by declaring Py_BUILD_CORE, you're getting
a macro version of that instead of the normal exported function.

ChrisA

From arj.python at gmail.com  Fri Nov 12 16:51:17 2021
From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer)
Date: Sat, 13 Nov 2021 01:51:17 +0400
Subject: Getting Directory of Command Line Entry Point For Packages
Message-ID: <CADrxXXkzOTpoW5BbVZFJUSwbHnVnb2rn=EAr=D-=SpjT=WPZ_Q@mail.gmail.com>

Greetings list,

Let's say i created a package named miaw

miaw also has a cli command called miaw

miaw prints files and folders in the directory it is called in

except that when miaw is used, it prints the files and folders in
site-packages

This is an analogy for  a package i have.

Well forgetting about the lines above, how do i get the path from
which miaw the command is called from?

Thanks

Kind Regards,

Abdur-Rahmaan Janhangeer
about <https://compileralchemy.github.io/> | blog
<https://www.pythonkitchen.com>
github <https://github.com/Abdur-RahmaanJ>
Mauritius

From DomainAdmin at DancesWithMice.info  Fri Nov 12 17:02:55 2021
From: DomainAdmin at DancesWithMice.info (David L Neil)
Date: Sat, 13 Nov 2021 11:02:55 +1300
Subject: Getting Directory of Command Line Entry Point For Packages
In-Reply-To: <CADrxXXkzOTpoW5BbVZFJUSwbHnVnb2rn=EAr=D-=SpjT=WPZ_Q@mail.gmail.com>
References: <CADrxXXkzOTpoW5BbVZFJUSwbHnVnb2rn=EAr=D-=SpjT=WPZ_Q@mail.gmail.com>
Message-ID: <42eaf0dd-409d-fe8d-3ee4-1e80ca225f0a@DancesWithMice.info>

On 13/11/2021 10.51, Abdur-Rahmaan Janhangeer wrote:
> Greetings list,
> 
> Let's say i created a package named miaw
> 
> miaw also has a cli command called miaw
> 
> miaw prints files and folders in the directory it is called in
> 
> except that when miaw is used, it prints the files and folders in
> site-packages
> 
> This is an analogy for  a package i have.
> 
> Well forgetting about the lines above, how do i get the path from
> which miaw the command is called from?

try:

file_path = __file__
print( file_path )

and process file_path as-required.
-- 
Regards =dn

From PythonList at DancesWithMice.info  Fri Nov 12 17:05:33 2021
From: PythonList at DancesWithMice.info (dn)
Date: Sat, 13 Nov 2021 11:05:33 +1300
Subject: Getting Directory of Command Line Entry Point For Packages
In-Reply-To: <CADrxXXkzOTpoW5BbVZFJUSwbHnVnb2rn=EAr=D-=SpjT=WPZ_Q@mail.gmail.com>
References: <CADrxXXkzOTpoW5BbVZFJUSwbHnVnb2rn=EAr=D-=SpjT=WPZ_Q@mail.gmail.com>
Message-ID: <ff324aff-83a8-28ab-ad55-ee28ba334c13@DancesWithMice.info>

On 13/11/2021 10.51, Abdur-Rahmaan Janhangeer wrote:
> Greetings list,
> 
> Let's say i created a package named miaw
> 
> miaw also has a cli command called miaw
> 
> miaw prints files and folders in the directory it is called in
> 
> except that when miaw is used, it prints the files and folders in
> site-packages
> 
> This is an analogy for  a package i have.
> 
> Well forgetting about the lines above, how do i get the path from
> which miaw the command is called from?

try:

file_path = __file__
print( file_path )

and process file_path as-required.

-- 
Regards,
=dn

From Marco.Sulla.Python at gmail.com  Fri Nov 12 17:50:31 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Fri, 12 Nov 2021 23:50:31 +0100
Subject: Unable to compile my C Extension on Windows: unresolved external
 link errors
In-Reply-To: <CAPTjJmqSDxbvH6fGYEG6fKRQ1DoE6+CMGpUPoKURX3bVE6a_tg@mail.gmail.com>
References: <CABbU2U-d0f=mD5DXxgnDr=qc=p-XsFfi83YCd=vcFEXURHJymw@mail.gmail.com>
 <e8f1169e-8ea9-420b-5503-2befa5495b88@gmail.com>
 <CABbU2U8Pp7KOVhzrHACVhx8ye4bx7XernrQ8GLjY+VFKv6ttHQ@mail.gmail.com>
 <CAPTjJmqDPku3x_QcuV0OXcfzKEGyN+sww-JseMBwsfo-fw6OLg@mail.gmail.com>
 <CABbU2U_e4BqukXXzq=97Wxiph6m4EsNpk-tHCbPi5DUMp18YMg@mail.gmail.com>
 <CAPTjJmqSDxbvH6fGYEG6fKRQ1DoE6+CMGpUPoKURX3bVE6a_tg@mail.gmail.com>
Message-ID: <CABbU2U_uEeA+0mE0FygzicOhnqS3wNb75rk3gSPPjgU-DHJnHg@mail.gmail.com>

On Fri, 12 Nov 2021 at 21:09, Chris Angelico <rosuav at gmail.com> wrote:
>
> On Sat, Nov 13, 2021 at 7:01 AM Marco Sulla
> <Marco.Sulla.Python at gmail.com> wrote:
> > On Fri, 12 Nov 2021 at 17:38, Chris Angelico <rosuav at gmail.com> wrote:
> > > Are you sure that you really need Py_BUILD_CORE?
> >
> > Yes, because I need the internal functions of `dict`. So I need to
> > compile also dictobject.c and include it. So I need that flag.
> >
> > This is the code:
> >
> > https://github.com/Marco-Sulla/python-frozendict.git
> >
>
> Ah, gotcha.
>
> Unfortunately that does mean you're delving deep into internals, and a
> lot of stuff that isn't designed for extensions to use. So my best
> recommendation is: dig even deeper into internals, and duplicate how
> the core is doing things (maybe including another header or
> something). It may be that, by declaring Py_BUILD_CORE, you're getting
> a macro version of that instead of the normal exported function.

I've not understood what I have to do in practice.... but anyway, as I
said, Py_BUILD_CORE works on Linux and MacOS. And it works also on
Windows. Indeed dictobject.c is compiled. The only problem is in the
linking phase, when the two objects should be linked in one library,
_the_ library. It seems that on Windows it doesn't find python3.lib,
even if I put it in the path. So I get the `unresolved external link`
errors.

From greg.ewing at canterbury.ac.nz  Fri Nov 12 18:37:14 2021
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 13 Nov 2021 12:37:14 +1300
Subject: Getting Directory of Command Line Entry Point For Packages
In-Reply-To: <mailman.2.1636753892.15285.python-list@python.org>
References: <CADrxXXkzOTpoW5BbVZFJUSwbHnVnb2rn=EAr=D-=SpjT=WPZ_Q@mail.gmail.com>
 <mailman.2.1636753892.15285.python-list@python.org>
Message-ID: <iv8c5dF8mbbU1@mid.individual.net>

On 13/11/21 10:51 am, Abdur-Rahmaan Janhangeer wrote:
> ow do i get the path from
> which miaw the command is called from?

What exactly do you mean by "called from"?

If you want the user's working directory, os.getcwd() will
give you that.

If you want something else, you'll have to give us more
details.

-- 
Greg

From dallakyans at gmail.com  Fri Nov 12 22:40:23 2021
From: dallakyans at gmail.com (Sarkis Dallakian)
Date: Fri, 12 Nov 2021 19:40:23 -0800
Subject: Looking to Partner with Python Developer with Apple M1 Chip
In-Reply-To: <CAAWj0RCLSsksRL78RXOrKmpFzcx7ovLgarDjYx3pTsUJhdntWw@mail.gmail.com>
References: <CAAWj0RCLSsksRL78RXOrKmpFzcx7ovLgarDjYx3pTsUJhdntWw@mail.gmail.com>
Message-ID: <CAAWj0RCDMD5QpfreNjqxgcjQzSFEL37+pTyntf0s7yuG6nUuEQ@mail.gmail.com>

 Greetings,

I'm looking for someone who has the latest Apple with M1 Chip and who can
partner with me to build a software I wrote called PyRx. This is based on
wxPython and can provide you with a steady source of income.

Please email me if you are interested to discuss the details.

Thank you,
Sarkis
PS: Sorry if this is not the right place to post this message.

From arj.python at gmail.com  Sat Nov 13 01:23:57 2021
From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer)
Date: Sat, 13 Nov 2021 10:23:57 +0400
Subject: Getting Directory of Command Line Entry Point For Packages
In-Reply-To: <iv8c5dF8mbbU1@mid.individual.net>
References: <CADrxXXkzOTpoW5BbVZFJUSwbHnVnb2rn=EAr=D-=SpjT=WPZ_Q@mail.gmail.com>
 <mailman.2.1636753892.15285.python-list@python.org>
 <iv8c5dF8mbbU1@mid.individual.net>
Message-ID: <CADrxXXkMCq_AaGY=qzndHqDpX1fY=HdJQovm6=Aqc6JbJTsHvQ@mail.gmail.com>

Greetings,

Well since sometimes i have this: https://github.com/shopyo/shopyo
Old versions worked as we are using it for FlaskCon
<https://github.com/flaskcon/traveller>, even newer versions
until sometimes ago.

shopyo has a copy of the project which is a flask app in site-packages.
upon using shopyo new, it creates a copy of the project in the current
folder
then on running say shopyo run, it just passes commandline args flask run
under the hood

Only thing is that os.getcwd is giving the path of site-packages and not
the directory from
which the command is run from. See the run command's source here:
https://github.com/shopyo/shopyo/blob/19dc2ee03ff0a6a0dca3237f80a11478ee2dbe46/shopyo/api/cmd_helper.py#L313

Same goes for other commands. I wanted to know how to get the directory
from which the
command is called from. Thanks


Else i assume you are the author of this article
<https://python-history.blogspot.com/2009/04/and-snake-attacks.html> as i
was compiling a list of articles
<https://www.pythonkitchen.com/python-engineering-articles/>
and guess that it might be.

Kind Regards,

Abdur-Rahmaan Janhangeer
about <https://compileralchemy.github.io/> | blog
<https://www.pythonkitchen.com>
github <https://github.com/Abdur-RahmaanJ>
Mauritius

From barry at barrys-emacs.org  Sat Nov 13 03:55:23 2021
From: barry at barrys-emacs.org (Barry)
Date: Sat, 13 Nov 2021 08:55:23 +0000
Subject: Getting Directory of Command Line Entry Point For Packages
In-Reply-To: <CADrxXXkMCq_AaGY=qzndHqDpX1fY=HdJQovm6=Aqc6JbJTsHvQ@mail.gmail.com>
References: <CADrxXXkMCq_AaGY=qzndHqDpX1fY=HdJQovm6=Aqc6JbJTsHvQ@mail.gmail.com>
Message-ID: <06D60114-FC50-4803-B1AD-B581DC4C31BA@barrys-emacs.org>



> On 13 Nov 2021, at 06:26, Abdur-Rahmaan Janhangeer <arj.python at gmail.com> wrote:
> 
> Only thing is that os.getcwd is giving the path of site-packages and not
> the directory from
> which the command is run from.

In which case the code is doing a os.chdir() before the call to os.getwd().
You need to call getwd() before that happens.

Barry

From barry at barrys-emacs.org  Sat Nov 13 04:00:04 2021
From: barry at barrys-emacs.org (Barry)
Date: Sat, 13 Nov 2021 09:00:04 +0000
Subject: Unable to compile my C Extension on Windows: unresolved external
 link errors
In-Reply-To: <CABbU2U_uEeA+0mE0FygzicOhnqS3wNb75rk3gSPPjgU-DHJnHg@mail.gmail.com>
References: <CABbU2U_uEeA+0mE0FygzicOhnqS3wNb75rk3gSPPjgU-DHJnHg@mail.gmail.com>
Message-ID: <2DA56A2E-EA0E-42B7-832E-EF09C1CF162B@barrys-emacs.org>



> On 12 Nov 2021, at 22:53, Marco Sulla <Marco.Sulla.Python at gmail.com> wrote:
> 
> It seems that on Windows it doesn't find python3.lib,
> even if I put it in the path. So I get the `unresolved external link`
> errors.

I think you need the python310.lib (not sure of file name) to get to the internal symbols.

You can use the objdump(?) utility to check that the symbols are in the lib.

Barry



From barry at barrys-emacs.org  Sat Nov 13 05:10:47 2021
From: barry at barrys-emacs.org (Barry Scott)
Date: Sat, 13 Nov 2021 10:10:47 +0000
Subject: Unable to compile my C Extension on Windows: unresolved external
 link errors
In-Reply-To: <2DA56A2E-EA0E-42B7-832E-EF09C1CF162B@barrys-emacs.org>
References: <CABbU2U_uEeA+0mE0FygzicOhnqS3wNb75rk3gSPPjgU-DHJnHg@mail.gmail.com>
 <2DA56A2E-EA0E-42B7-832E-EF09C1CF162B@barrys-emacs.org>
Message-ID: <1D6456BF-AA3F-425A-AF2B-3E66A0868BA3@barrys-emacs.org>



> On 13 Nov 2021, at 09:00, Barry <barry at barrys-emacs.org> wrote:
> 
> 
> 
>> On 12 Nov 2021, at 22:53, Marco Sulla <Marco.Sulla.Python at gmail.com> wrote:
>> 
>> It seems that on Windows it doesn't find python3.lib,
>> even if I put it in the path. So I get the `unresolved external link`
>> errors.
> 
> I think you need the python310.lib (not sure of file name) to get to the internal symbols.

Another thing that you will need to check is that the symbols you are after have been
exposed in the DLL at all. Being external in the source is not enough they also have to
listed in the .DLL's def file ( is that the right term?) as well.

If its not clear yet, you are going to have to read a lot or source code and understand
the tool chain used on Windows to solve this.


> 
> You can use the objdump(?) utility to check that the symbols are in the lib.
> 
> Barry

Barry


From arj.python at gmail.com  Sat Nov 13 05:54:31 2021
From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer)
Date: Sat, 13 Nov 2021 14:54:31 +0400
Subject: Getting Directory of Command Line Entry Point For Packages
In-Reply-To: <06D60114-FC50-4803-B1AD-B581DC4C31BA@barrys-emacs.org>
References: <CADrxXXkMCq_AaGY=qzndHqDpX1fY=HdJQovm6=Aqc6JbJTsHvQ@mail.gmail.com>
 <06D60114-FC50-4803-B1AD-B581DC4C31BA@barrys-emacs.org>
Message-ID: <CADrxXXm7CF2-T0y5iVu6PObX18H0LzFtv=5OS3YNqntenm0HWQ@mail.gmail.com>

Greetings,

This is what I am trying to do:
How to get the getcwd of the directory of where the command is run and not
that
of the file where the cli entrypoint is found. Having the user enter the
absolute path as a cli
argument does not sound nice.

Kind Regards,

Abdur-Rahmaan Janhangeer
about <https://compileralchemy.github.io/> | blog
<https://www.pythonkitchen.com>
github <https://github.com/Abdur-RahmaanJ>
Mauritius

From Marco.Sulla.Python at gmail.com  Sat Nov 13 06:17:54 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Sat, 13 Nov 2021 12:17:54 +0100
Subject: Unable to compile my C Extension on Windows: unresolved external
 link errors
In-Reply-To: <1D6456BF-AA3F-425A-AF2B-3E66A0868BA3@barrys-emacs.org>
References: <CABbU2U_uEeA+0mE0FygzicOhnqS3wNb75rk3gSPPjgU-DHJnHg@mail.gmail.com>
 <2DA56A2E-EA0E-42B7-832E-EF09C1CF162B@barrys-emacs.org>
 <1D6456BF-AA3F-425A-AF2B-3E66A0868BA3@barrys-emacs.org>
Message-ID: <CABbU2U-RyBjiq4xK+J+3_JZN7yb+7T_J66=XpkJJzkXNdkf+6g@mail.gmail.com>

..... Sorry, the problem is I downloaded the 32 bit version of VS
compiler and 64 bit version of Python......

On Sat, 13 Nov 2021 at 11:10, Barry Scott <barry at barrys-emacs.org> wrote:
>
>
>
> > On 13 Nov 2021, at 09:00, Barry <barry at barrys-emacs.org> wrote:
> >
> >
> >
> >> On 12 Nov 2021, at 22:53, Marco Sulla <Marco.Sulla.Python at gmail.com> wrote:
> >>
> >> It seems that on Windows it doesn't find python3.lib,
> >> even if I put it in the path. So I get the `unresolved external link`
> >> errors.
> >
> > I think you need the python310.lib (not sure of file name) to get to the internal symbols.
>
> Another thing that you will need to check is that the symbols you are after have been
> exposed in the DLL at all. Being external in the source is not enough they also have to
> listed in the .DLL's def file ( is that the right term?) as well.
>
> If its not clear yet, you are going to have to read a lot or source code and understand
> the tool chain used on Windows to solve this.
>
>
> >
> > You can use the objdump(?) utility to check that the symbols are in the lib.
> >
> > Barry
>
> Barry
>

From mgogala at yahoo.com  Sat Nov 13 00:13:20 2021
From: mgogala at yahoo.com (Mladen Gogala)
Date: Sat, 13 Nov 2021 05:13:20 -0000 (UTC)
Subject: Avoid nested SIGINT handling
References: <smhe3h$12pi$1@gioia.aioe.org>
 <slrnsooftp.5h7.jon+usenet@raven.unequivocal.eu>
 <CAPTjJmqgcFs8cPHS5ZQnVEixDQAgiJb94VutdvGspnGDszjVZQ@mail.gmail.com>
 <mailman.196.1636611749.23718.python-list@python.org>
Message-ID: <smnhhg$a7g$1@gioia.aioe.org>

On Thu, 11 Nov 2021 17:22:15 +1100, Chris Angelico wrote:

> Threads aren't the point here - signals happen immediately.

Actually, signals are not delivered immediately. Signals are delivered 
the next time the process gets its turn on CPU. The process scheduler 
will make process runnable and the process will check for any pending 
signals first and will execute the handler. It is possible to have 
several SIGINT signals pending, for instance when I nervously press ctrl-
C several times. However, signals are not processed as a part of the 
normal flow of the process and are processed sequentially.. When the 
process finds a pending signal, it executes the registered signal 
handler. It's always the same signal handler, unless signal handler is 
changed within the signal handler. After the signals are delivered, the 
process continues its normal operation until its CPU quantum expires or 
until initiates a synchronous I/O operation, as is the case with all 
normal read operations.
BTW, that's the case on both Unix/Linux systems and Windows systems.



-- 
Mladen Gogala
Database Consultant
https://dbwhisperer.wordpress.com

From greg.ewing at canterbury.ac.nz  Sat Nov 13 02:11:06 2021
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 13 Nov 2021 20:11:06 +1300
Subject: Getting Directory of Command Line Entry Point For Packages
In-Reply-To: <mailman.1.1636784652.19479.python-list@python.org>
References: <CADrxXXkzOTpoW5BbVZFJUSwbHnVnb2rn=EAr=D-=SpjT=WPZ_Q@mail.gmail.com>
 <mailman.2.1636753892.15285.python-list@python.org>
 <iv8c5dF8mbbU1@mid.individual.net>
 <CADrxXXkMCq_AaGY=qzndHqDpX1fY=HdJQovm6=Aqc6JbJTsHvQ@mail.gmail.com>
 <mailman.1.1636784652.19479.python-list@python.org>
Message-ID: <iv96ocFda4lU1@mid.individual.net>

On 13/11/21 7:23 pm, Abdur-Rahmaan Janhangeer wrote:
> os.getcwd is giving the path of site-packages and not
> the directory from
> which the command is run from.

Something must be changing the working directory before getcwd
is called. Once that happens there's no way I know of to find
out what it was. You'll have to capture it earlier in the process
somehow and pass it to where it's needed.

-- 
Greg

From rosuav at gmail.com  Sat Nov 13 16:58:41 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 14 Nov 2021 08:58:41 +1100
Subject: Avoid nested SIGINT handling
In-Reply-To: <smnhhg$a7g$1@gioia.aioe.org>
References: <smhe3h$12pi$1@gioia.aioe.org>
 <slrnsooftp.5h7.jon+usenet@raven.unequivocal.eu>
 <CAPTjJmqgcFs8cPHS5ZQnVEixDQAgiJb94VutdvGspnGDszjVZQ@mail.gmail.com>
 <mailman.196.1636611749.23718.python-list@python.org>
 <smnhhg$a7g$1@gioia.aioe.org>
Message-ID: <CAPTjJmpBBjP=Ue6KvFBoWn3gwq8MWNaQ6=1Zr394wCxfxB4xvA@mail.gmail.com>

On Sun, Nov 14, 2021 at 4:42 AM Mladen Gogala via Python-list
<python-list at python.org> wrote:
>
> On Thu, 11 Nov 2021 17:22:15 +1100, Chris Angelico wrote:
>
> > Threads aren't the point here - signals happen immediately.
>
> Actually, signals are not delivered immediately. Signals are delivered
> the next time the process gets its turn on CPU. The process scheduler
> will make process runnable and the process will check for any pending
> signals first and will execute the handler. It is possible to have
> several SIGINT signals pending, for instance when I nervously press ctrl-
> C several times. However, signals are not processed as a part of the
> normal flow of the process and are processed sequentially.. When the
> process finds a pending signal, it executes the registered signal
> handler. It's always the same signal handler, unless signal handler is
> changed within the signal handler. After the signals are delivered, the
> process continues its normal operation until its CPU quantum expires or
> until initiates a synchronous I/O operation, as is the case with all
> normal read operations.
> BTW, that's the case on both Unix/Linux systems and Windows systems.
>

Maybe at the C level, but none of that happens in Python :)

There's a very very small signal handler in CPython that just sets a
flag and then handles things separately.

ChrisA

From eryksun at gmail.com  Sat Nov 13 17:41:45 2021
From: eryksun at gmail.com (Eryk Sun)
Date: Sat, 13 Nov 2021 16:41:45 -0600
Subject: Avoid nested SIGINT handling
In-Reply-To: <smnhhg$a7g$1@gioia.aioe.org>
References: <smhe3h$12pi$1@gioia.aioe.org>
 <slrnsooftp.5h7.jon+usenet@raven.unequivocal.eu>
 <CAPTjJmqgcFs8cPHS5ZQnVEixDQAgiJb94VutdvGspnGDszjVZQ@mail.gmail.com>
 <mailman.196.1636611749.23718.python-list@python.org>
 <smnhhg$a7g$1@gioia.aioe.org>
Message-ID: <CACL+1aufzpBkG3N7mEgV26J0B7mvNFV2Kud1D6RUXWStYLwiJA@mail.gmail.com>

On 11/12/21, Mladen Gogala via Python-list <python-list at python.org> wrote:
> On Thu, 11 Nov 2021 17:22:15 +1100, Chris Angelico wrote:
>
>> Threads aren't the point here - signals happen immediately.
>
> [snip: description of POSIX signals]
>
> BTW, that's the case on both Unix/Linux systems and Windows systems.

Windows is quite different. The base NT system has nothing that's
closely analogous to POSIX signals. Its asynchronous procedure call
(APC) capability is roughly similar to POSIX signals. Every thread has
two APC queues, one for kernel-mode APCs and one for user-mode APCs.
Unlike a POSIX signal, queuing an APC requires the address of a
function in the process of the target thread. This requires well-known
or registered function targets. User-mode APCs can be queued to a
thread via QueueUserAPC(). Unless forced, they execute when the thread
does an alertable wait.

Another similar feature is a window message. A thread that uses any
window-manager function is converted to a GUI thread with a message
queue. A thread's window message queue is typically processed during a
GetMessage() or PeekMessage() call. A message can be queued directly
to a thread via PostThreadMessage(), provided it's a GUI thread.

There's an overlap between Windows exceptions and POSIX signals.
Exceptions get raised either from the kernel or RaiseException().
Exceptions have stack based handlers that are set via Microsoft's
extended C __try/__except and __try/__finally statements. Process-wide
exception handlers can also be set via AddVectoredExceptionHandler()
and SetUnhandledExceptionFilter().

Windows also has console control events, which the console host
(conhost.exe or openconsole.exe) sends to process groups that are
connected to the hosted session (i.e. a TUI window/tab on a GUI
desktop). The console host generates these events in response to
typing Ctrl+C (cancel) or Ctrl+Break in the window/tab, closing the
window/tab, user logoff, and system shutdown.

    CTRL_C_EVENT (0)
    CTRL_BREAK_EVENT (1)
    CTRL_CLOSE_EVENT (2)
    CTRL_LOGOFF_EVENT (5)
    CTRL_SHUTDOWN_EVENT (6)

The Ctrl+C and Ctrl+Break events can also be generated for a process
group via GenerateConsoleCtrlEvent(ctrlEvent, processGroupId).

There's no thread or process queue for console control events. To
deliver an event, the console host sends a message to the Windows
session server (csrss.exe) that contains the control event number and
one or more target processes. The session server creates a new thread
in each target process. For example, if the user presses Ctrl+C five
times sequentially in the console window, csrss.exe creates five
threads in every process that's attached to the console. This thread
injection is so different from POSIX signals that I hesitate to group
it with signal-like features.

A console control thread begins executing at the CtrlRoutine()
function in kernelbase.dll. It sequentially iterates the list of
registered handlers. Each is called with the given control event. If a
handler returns TRUE (i.e. handled), the control thread exits. If no
handler returns TRUE, the default handler is called, which exits the
process with the exit code STATUS_CONTROL_C_EXIT (0xC000013A). A
inheritable flag can be set to disable CTRL_C_EVENT, such that
CtrlRoutine() basically does nothing for this event.

For CTRL_CLOSE_EVENT, CTRL_LOGOFF_EVENT, and CTRL_SHUTDOWN_EVENT, the
process must be terminated after the configured timeout, which is 5
seconds in a default setup. The process can exit gracefully of its own
accord before the timeout.

The C runtime library in Windows emulates some signals that are
required by the language standard.

The following two are emulated for C raise() and abort():

    SIGABRT - abort, abnormal termination
    SIGTERM - terminate

The following three are based on an exception handler:

    SIGSEGV - segmentation violation
    SIGILL - illegal/invalid instruction
    SIGFPE - floating point exception

The following two are based on a console control event handler
(SIGBREAK is non-standard):

    SIGINT - interrupt
    SIGBREAK - break, close, logoff, shutdown

For SIGINT and SIGBREAK, the C runtime uses CTRL_C_EVENT and CTRL_BREAK_EVENT.

It also maps SIGBREAK to CTRL_CLOSE_EVENT, CTRL_LOGOFF_EVENT, and
CTRL_SHUTDOWN_EVENT.  But Python's C signal handler just sets a flag
and returns, so SIGBREAK due to those events can't be handled with
Python's signal module. The process gets terminated long before the
main thread can call Python's registered SIGBREAK handler.

From nt_mahmood at yahoo.com  Sun Nov 14 04:01:51 2021
From: nt_mahmood at yahoo.com (Mahmood Naderan)
Date: Sun, 14 Nov 2021 09:01:51 +0000 (UTC)
Subject: Returning the index of a row in dataframe
References: <1093639769.406029.1636880511424.ref@mail.yahoo.com>
Message-ID: <1093639769.406029.1636880511424@mail.yahoo.com>

Hi

In the following dataframe, I want to get the index string by specifying the row 
number which is the same as value column.


???????????????????? Value
??? global loads?????? 0
??? global stores????? 1
??? local loads??????? 2


For example, `df.iloc[1].index.name` should return "global stores" but the 
output is `None`. Any idea about that?


Regards,
Mahmood

From Marco.Sulla.Python at gmail.com  Sun Nov 14 05:38:56 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Sun, 14 Nov 2021 11:38:56 +0100
Subject: Unable to compile my C Extension on Windows: unresolved external
 link errors
In-Reply-To: <CABbU2U-RyBjiq4xK+J+3_JZN7yb+7T_J66=XpkJJzkXNdkf+6g@mail.gmail.com>
References: <CABbU2U_uEeA+0mE0FygzicOhnqS3wNb75rk3gSPPjgU-DHJnHg@mail.gmail.com>
 <2DA56A2E-EA0E-42B7-832E-EF09C1CF162B@barrys-emacs.org>
 <1D6456BF-AA3F-425A-AF2B-3E66A0868BA3@barrys-emacs.org>
 <CABbU2U-RyBjiq4xK+J+3_JZN7yb+7T_J66=XpkJJzkXNdkf+6g@mail.gmail.com>
Message-ID: <CABbU2U91ncetcGS0iObOURK4-1PX1MFfoBM_ErweUJFJgfHwUA@mail.gmail.com>

Okay, now the problem seems to be another: I get the same "unresolved
external link" errors, but only for internal functions.

This seems quite normal. The public .lib does not expose the internals
of Python.
The strange fact is: why can I compile it on Linux and MacOS? Their
external libraries expose the internal functions?

Anyway, is there a way to compile Python on Windows in such a way that
I get a shared library that exposes all the functions?

On Sat, 13 Nov 2021 at 12:17, Marco Sulla <Marco.Sulla.Python at gmail.com> wrote:
>
> ..... Sorry, the problem is I downloaded the 32 bit version of VS
> compiler and 64 bit version of Python......
>
> On Sat, 13 Nov 2021 at 11:10, Barry Scott <barry at barrys-emacs.org> wrote:
> >
> >
> >
> > > On 13 Nov 2021, at 09:00, Barry <barry at barrys-emacs.org> wrote:
> > >
> > >
> > >
> > >> On 12 Nov 2021, at 22:53, Marco Sulla <Marco.Sulla.Python at gmail.com> wrote:
> > >>
> > >> It seems that on Windows it doesn't find python3.lib,
> > >> even if I put it in the path. So I get the `unresolved external link`
> > >> errors.
> > >
> > > I think you need the python310.lib (not sure of file name) to get to the internal symbols.
> >
> > Another thing that you will need to check is that the symbols you are after have been
> > exposed in the DLL at all. Being external in the source is not enough they also have to
> > listed in the .DLL's def file ( is that the right term?) as well.
> >
> > If its not clear yet, you are going to have to read a lot or source code and understand
> > the tool chain used on Windows to solve this.
> >
> >
> > >
> > > You can use the objdump(?) utility to check that the symbols are in the lib.
> > >
> > > Barry
> >
> > Barry
> >

From nt_mahmood at yahoo.com  Sun Nov 14 10:24:19 2021
From: nt_mahmood at yahoo.com (Mahmood Naderan)
Date: Sun, 14 Nov 2021 15:24:19 +0000 (UTC)
Subject: Returning the index of a row in dataframe
In-Reply-To: <CANy1k1ghZBxJiq0H92OaghFsZyczQTmZXBoeonTiHNFBDKgb7g@mail.gmail.com>
References: <1093639769.406029.1636880511424.ref@mail.yahoo.com>
 <1093639769.406029.1636880511424@mail.yahoo.com>
 <CANy1k1ghZBxJiq0H92OaghFsZyczQTmZXBoeonTiHNFBDKgb7g@mail.gmail.com>
Message-ID: <800661040.347189.1636903459073@mail.yahoo.com>

>>> df.iloc[1].name


Correct.... I also see that 'df.index[1]' works fine.
Thanks.


Regards,
Mahmood




From nt_mahmood at yahoo.com  Sun Nov 14 10:41:01 2021
From: nt_mahmood at yahoo.com (Mahmood Naderan)
Date: Sun, 14 Nov 2021 15:41:01 +0000 (UTC)
Subject: Using astype(int) for strings with thousand separator
References: <1880036131.424082.1636904461851.ref@mail.yahoo.com>
Message-ID: <1880036131.424082.1636904461851@mail.yahoo.com>

Hi

While reading a csv file, some cells have values like '1,024' which I mean they contains thousand separator ','. Therefore, when I want to process them with 

? row = df.iloc[0].astype(int)

I get the following error

? ValueError: invalid literal for int() with base 10: '1,024'


How can I fix that? Any idea?



Regards,
Mahmood

From barry at barrys-emacs.org  Sun Nov 14 10:42:45 2021
From: barry at barrys-emacs.org (Barry Scott)
Date: Sun, 14 Nov 2021 15:42:45 +0000
Subject: Unable to compile my C Extension on Windows: unresolved external
 link errors
In-Reply-To: <CABbU2U91ncetcGS0iObOURK4-1PX1MFfoBM_ErweUJFJgfHwUA@mail.gmail.com>
References: <CABbU2U_uEeA+0mE0FygzicOhnqS3wNb75rk3gSPPjgU-DHJnHg@mail.gmail.com>
 <2DA56A2E-EA0E-42B7-832E-EF09C1CF162B@barrys-emacs.org>
 <1D6456BF-AA3F-425A-AF2B-3E66A0868BA3@barrys-emacs.org>
 <CABbU2U-RyBjiq4xK+J+3_JZN7yb+7T_J66=XpkJJzkXNdkf+6g@mail.gmail.com>
 <CABbU2U91ncetcGS0iObOURK4-1PX1MFfoBM_ErweUJFJgfHwUA@mail.gmail.com>
Message-ID: <5E127DC8-AE26-4EFC-8293-9E9FDB39475B@barrys-emacs.org>

Sorry iPad sent the message before it was complete...

> On 14 Nov 2021, at 10:38, Marco Sulla <Marco.Sulla.Python at gmail.com> wrote:
> 
> Okay, now the problem seems to be another: I get the same "unresolved
> external link" errors, but only for internal functions.
> 
> This seems quite normal. The public .lib does not expose the internals
> of Python.
> The strange fact is: why can I compile it on Linux and MacOS? Their
> external libraries expose the internal functions?

Windows is not Linux is not macOS,
The toolchain on each OS has its own strengths, weaknesses and quirks.

On Windows DLLs only allow access to the symbols that are explicitly listed to be access.
On macOS .dynlib and Unix .so its being extern that does this.

> 
> Anyway, is there a way to compile Python on Windows in such a way that
> I get a shared library that exposes all the functions?

Yes you can do your own build of python that exposes the symbols you want.
But that build will be private to you and will not allow others to use your work
(on the assumption that they will not use your private build of python).

Maybe you could copy the code that you want and add it to your code?
Change any conflicting symbols of course.

Barry

> 
> On Sat, 13 Nov 2021 at 12:17, Marco Sulla <Marco.Sulla.Python at gmail.com> wrote:
>> 
>> ..... Sorry, the problem is I downloaded the 32 bit version of VS
>> compiler and 64 bit version of Python......
>> 
>> On Sat, 13 Nov 2021 at 11:10, Barry Scott <barry at barrys-emacs.org> wrote:
>>> 
>>> 
>>> 
>>>> On 13 Nov 2021, at 09:00, Barry <barry at barrys-emacs.org> wrote:
>>>> 
>>>> 
>>>> 
>>>>> On 12 Nov 2021, at 22:53, Marco Sulla <Marco.Sulla.Python at gmail.com> wrote:
>>>>> 
>>>>> It seems that on Windows it doesn't find python3.lib,
>>>>> even if I put it in the path. So I get the `unresolved external link`
>>>>> errors.
>>>> 
>>>> I think you need the python310.lib (not sure of file name) to get to the internal symbols.
>>> 
>>> Another thing that you will need to check is that the symbols you are after have been
>>> exposed in the DLL at all. Being external in the source is not enough they also have to
>>> listed in the .DLL's def file ( is that the right term?) as well.
>>> 
>>> If its not clear yet, you are going to have to read a lot or source code and understand
>>> the tool chain used on Windows to solve this.
>>> 
>>> 
>>>> 
>>>> You can use the objdump(?) utility to check that the symbols are in the lib.
>>>> 
>>>> Barry
>>> 
>>> Barry
>>> 
> 


From barry at barrys-emacs.org  Sun Nov 14 11:00:38 2021
From: barry at barrys-emacs.org (Barry Scott)
Date: Sun, 14 Nov 2021 16:00:38 +0000
Subject: Using astype(int) for strings with thousand separator
In-Reply-To: <1880036131.424082.1636904461851@mail.yahoo.com>
References: <1880036131.424082.1636904461851.ref@mail.yahoo.com>
 <1880036131.424082.1636904461851@mail.yahoo.com>
Message-ID: <F523C36A-658B-4EB0-B833-3100624CB2C8@barrys-emacs.org>



> On 14 Nov 2021, at 15:41, Mahmood Naderan via Python-list <python-list at python.org> wrote:
> 
> Hi
> 
> While reading a csv file, some cells have values like '1,024' which I mean they contains thousand separator ','. Therefore, when I want to process them with 
> 
>   row = df.iloc[0].astype(int)

remove the "," from the sting seems the simplest things to do.

Use string's replace() to remove the comma.

Barry

> 
> I get the following error
> 
>   ValueError: invalid literal for int() with base 10: '1,024'
> 
> 
> How can I fix that? Any idea?
> 
> 
> 
> Regards,
> Mahmood
> -- 
> https://mail.python.org/mailman/listinfo/python-list


From Marco.Sulla.Python at gmail.com  Sun Nov 14 13:57:21 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Sun, 14 Nov 2021 19:57:21 +0100
Subject: Unable to compile my C Extension on Windows: unresolved external
 link errors
In-Reply-To: <5E127DC8-AE26-4EFC-8293-9E9FDB39475B@barrys-emacs.org>
References: <CABbU2U_uEeA+0mE0FygzicOhnqS3wNb75rk3gSPPjgU-DHJnHg@mail.gmail.com>
 <2DA56A2E-EA0E-42B7-832E-EF09C1CF162B@barrys-emacs.org>
 <1D6456BF-AA3F-425A-AF2B-3E66A0868BA3@barrys-emacs.org>
 <CABbU2U-RyBjiq4xK+J+3_JZN7yb+7T_J66=XpkJJzkXNdkf+6g@mail.gmail.com>
 <CABbU2U91ncetcGS0iObOURK4-1PX1MFfoBM_ErweUJFJgfHwUA@mail.gmail.com>
 <5E127DC8-AE26-4EFC-8293-9E9FDB39475B@barrys-emacs.org>
Message-ID: <CABbU2U9HJFBExVv5du0xQTG7nkMh7Xq1sGOBWwfoyMKfDgbSZQ@mail.gmail.com>

On Sun, 14 Nov 2021 at 16:42, Barry Scott <barry at barrys-emacs.org> wrote:
>
> Sorry iPad sent the message before it was complete...
>
> > On 14 Nov 2021, at 10:38, Marco Sulla <Marco.Sulla.Python at gmail.com> wrote:
> >
> > Okay, now the problem seems to be another: I get the same "unresolved
> > external link" errors, but only for internal functions.
> >
> > This seems quite normal. The public .lib does not expose the internals
> > of Python.
> > The strange fact is: why can I compile it on Linux and MacOS? Their
> > external libraries expose the internal functions?
>
> Windows is not Linux is not macOS,
> The toolchain on each OS has its own strengths, weaknesses and quirks.
>
> On Windows DLLs only allow access to the symbols that are explicitly listed to be access.

Where are those symbols listed?

> On macOS .dynlib and Unix .so its being extern that does this.

And extern is the default. I understand now.

> Maybe you could copy the code that you want and add it to your code?
> Change any conflicting symbols of course.

It's quite hard. I have to compile dictobject.c, which needs a lot of
internal functions. And I suppose that every internal function may
require 1 or more other internal functions.

I have other two other solutions:
* compile a whole python DLL with the symbols I need and link against
it. I have to put this DLL in my code, which is ugly.
* drop the support of the C Extension for Windows users and make for
them the slow, pure py version only.

Since my interest in Windows now is near to zero, I think I'll opt for
the third for now.

From tjol at tjol.eu  Sun Nov 14 11:27:22 2021
From: tjol at tjol.eu (Thomas Jollans)
Date: Sun, 14 Nov 2021 17:27:22 +0100
Subject: Using astype(int) for strings with thousand separator
In-Reply-To: <1880036131.424082.1636904461851@mail.yahoo.com>
References: <1880036131.424082.1636904461851.ref@mail.yahoo.com>
 <1880036131.424082.1636904461851@mail.yahoo.com>
Message-ID: <71cc6081-4eb1-380a-8f30-3cecad10218c@tjol.eu>

On 14.11.21 16:41, Mahmood Naderan via Python-list wrote:
> Hi
>
> While reading a csv file, some cells have values like '1,024' which I mean they contains thousand separator ','. Therefore, when I want to process them with
>
>  ? row = df.iloc[0].astype(int)


If you are reading a CSV with pandas.read_csv, that function takes a 
parameter 'thousands' that should do what you're looking for (see 
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html)

>
> I get the following error
>
>  ? ValueError: invalid literal for int() with base 10: '1,024'
>
>
> How can I fix that? Any idea?
>



From martin.schoon at gmail.com  Sun Nov 14 16:09:24 2021
From: martin.schoon at gmail.com (Martin =?UTF-8?Q?Sch=C3=B6=C3=B6n?=)
Date: 14 Nov 2021 21:09:24 GMT
Subject: Alternatives to Jupyter Notebook
References: <CA+i5JwbMjb9n-ysbdsqk+UW9x6Hwni7LbBcsdUTcQnE-NDRxxA@mail.gmail.com>
 <mailman.985.1634744845.4164.python-list@python.org>
Message-ID: <ivdc84F77llU1@mid.individual.net>

Den 2021-10-20 skrev Shaozhong SHI <shishaozhong at gmail.com>:
>
> My Jupyter notebook becomes unresponsive in browsers.
>
Odd, I never had any problems like that. I use Firefox on Linux.

> Are there alternatives to read, edit and run Jupyter Notebook?
>
I know some people use emacs orgmode. I have never tried it
myself and do not know how well it works.

/Martin

From drsalists at gmail.com  Sun Nov 14 19:22:53 2021
From: drsalists at gmail.com (Dan Stromberg)
Date: Sun, 14 Nov 2021 16:22:53 -0800
Subject: Alternatives to Jupyter Notebook
In-Reply-To: <CA+i5JwbMjb9n-ysbdsqk+UW9x6Hwni7LbBcsdUTcQnE-NDRxxA@mail.gmail.com>
References: <CA+i5JwbMjb9n-ysbdsqk+UW9x6Hwni7LbBcsdUTcQnE-NDRxxA@mail.gmail.com>
Message-ID: <CAGGBd_oJzRUZ28DvV44F9nGzkmK_XDF5OttZpctYsJRCvDfijg@mail.gmail.com>

Does this help?


On Wed, Oct 20, 2021 at 8:48 AM Shaozhong SHI <shishaozhong at gmail.com>
wrote:

> Hello,
>
> Is anyone familiar with alternatives to Jupyter Notebook.
>
> My Jupyter notebook becomes unresponsive in browsers.
>
> Are there alternatives to read, edit and run Jupyter Notebook?
>
> Regards,
>
> David
> --
> https://mail.python.org/mailman/listinfo/python-list
>

From drsalists at gmail.com  Sun Nov 14 19:23:46 2021
From: drsalists at gmail.com (Dan Stromberg)
Date: Sun, 14 Nov 2021 16:23:46 -0800
Subject: Alternatives to Jupyter Notebook
In-Reply-To: <CAGGBd_oJzRUZ28DvV44F9nGzkmK_XDF5OttZpctYsJRCvDfijg@mail.gmail.com>
References: <CA+i5JwbMjb9n-ysbdsqk+UW9x6Hwni7LbBcsdUTcQnE-NDRxxA@mail.gmail.com>
 <CAGGBd_oJzRUZ28DvV44F9nGzkmK_XDF5OttZpctYsJRCvDfijg@mail.gmail.com>
Message-ID: <CAGGBd_rW+nHLN9deTdAbjE-kMnQ9FnPsEwATLfNoHhXyZj-uBQ@mail.gmail.com>

Sorry, gmail got in a hurry to send my message :)

Does this help? https://lwn.net/Articles/855875/

On Sun, Nov 14, 2021 at 4:22 PM Dan Stromberg <drsalists at gmail.com> wrote:

>
> Does this help?
>
>
> On Wed, Oct 20, 2021 at 8:48 AM Shaozhong SHI <shishaozhong at gmail.com>
> wrote:
>
>> Hello,
>>
>> Is anyone familiar with alternatives to Jupyter Notebook.
>>
>> My Jupyter notebook becomes unresponsive in browsers.
>>
>> Are there alternatives to read, edit and run Jupyter Notebook?
>>
>> Regards,
>>
>> David
>> --
>> https://mail.python.org/mailman/listinfo/python-list
>>
>

From paige.bailey at alumni.rice.edu  Sun Nov 14 18:24:12 2021
From: paige.bailey at alumni.rice.edu (Paige Bailey)
Date: Sun, 14 Nov 2021 17:24:12 -0600
Subject: Alternatives to Jupyter Notebook
In-Reply-To: <ivdc84F77llU1@mid.individual.net>
References: <CA+i5JwbMjb9n-ysbdsqk+UW9x6Hwni7LbBcsdUTcQnE-NDRxxA@mail.gmail.com>
 <mailman.985.1634744845.4164.python-list@python.org>
 <ivdc84F77llU1@mid.individual.net>
Message-ID: <CAOj92wYeV9cQ8inp=tJ3SZ_v1A0f0VZYHWem1xV+cNbrxBB+Ow@mail.gmail.com>

If you're experiencing issues with Jupyter, perhaps try JupyterLab:
JupyterLab Documentation ? JupyterLab 3.2.3 documentation
<https://jupyterlab.readthedocs.io/en/stable/>

Or VS Code, which is also open-source and has support for Jupyter notebooks:
Working with Jupyter Notebooks in Visual Studio Code
<https://code.visualstudio.com/docs/datascience/jupyter-notebooks>

On Sun, Nov 14, 2021 at 5:20 PM Martin Sch??n <martin.schoon at gmail.com>
wrote:

> Den 2021-10-20 skrev Shaozhong SHI <shishaozhong at gmail.com>:
> >
> > My Jupyter notebook becomes unresponsive in browsers.
> >
> Odd, I never had any problems like that. I use Firefox on Linux.
>
> > Are there alternatives to read, edit and run Jupyter Notebook?
> >
> I know some people use emacs orgmode. I have never tried it
> myself and do not know how well it works.
>
> /Martin
> --
> https://mail.python.org/mailman/listinfo/python-list
>

From nt_mahmood at yahoo.com  Mon Nov 15 12:08:05 2021
From: nt_mahmood at yahoo.com (Mahmood Naderan)
Date: Mon, 15 Nov 2021 17:08:05 +0000 (UTC)
Subject: Using astype(int) for strings with thousand separator
In-Reply-To: <71cc6081-4eb1-380a-8f30-3cecad10218c@tjol.eu>
References: <1880036131.424082.1636904461851.ref@mail.yahoo.com>
 <1880036131.424082.1636904461851@mail.yahoo.com>
 <71cc6081-4eb1-380a-8f30-3cecad10218c@tjol.eu>
Message-ID: <1911165963.604589.1636996085769@mail.yahoo.com>

> (see https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html)

Got it. Thanks.


Regards,
Mahmood



From loris.bennett at fu-berlin.de  Mon Nov 15 07:20:29 2021
From: loris.bennett at fu-berlin.de (Loris Bennett)
Date: Mon, 15 Nov 2021 13:20:29 +0100
Subject: Alternatives to Jupyter Notebook
References: <CA+i5JwbMjb9n-ysbdsqk+UW9x6Hwni7LbBcsdUTcQnE-NDRxxA@mail.gmail.com>
 <mailman.985.1634744845.4164.python-list@python.org>
 <ivdc84F77llU1@mid.individual.net>
Message-ID: <87h7cdbofm.fsf@hornfels.zedat.fu-berlin.de>

Martin Sch??n <martin.schoon at gmail.com> writes:

> Den 2021-10-20 skrev Shaozhong SHI <shishaozhong at gmail.com>:
>>
>> My Jupyter notebook becomes unresponsive in browsers.
>>
> Odd, I never had any problems like that. I use Firefox on Linux.
>
>> Are there alternatives to read, edit and run Jupyter Notebook?
>>
> I know some people use emacs orgmode. I have never tried it
> myself and do not know how well it works.

I don't know Jupyter well enough to know whether Org mode is a real
alternative.  However, with Org mode you can combine text and code in
multiple languages, run the code within Emacs and then export the
results in various formats such as PDF and HTML.

I use it to collate monthly statistics by running shell scripts on a
remove server to generate data in tables within the Org file, running R
code on the data to generate plots from the data.  I finally export the
whole report to a PDF.  This is all done within a single local Emacs
instance.

YMMV.

Cheers,

Loris

-- 
This signature is currently under construction.

From ast at invalid  Mon Nov 15 08:10:39 2021
From: ast at invalid (ast)
Date: Mon, 15 Nov 2021 14:10:39 +0100
Subject: One line sort
Message-ID: <61925c51$0$28601$426a34cc@news.free.fr>

A curiosity:

q = lambda x: x and q([i for i in x[1:] if i < x[0]]) + [x[0]] + q([i 
for i in x[1:] if i >= x[0]])

 >>> q([7, 5, 9, 0])
[0, 5, 7, 9]

From rosuav at gmail.com  Mon Nov 15 14:03:13 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 16 Nov 2021 06:03:13 +1100
Subject: One line sort
In-Reply-To: <61925c51$0$28601$426a34cc@news.free.fr>
References: <61925c51$0$28601$426a34cc@news.free.fr>
Message-ID: <CAPTjJmoy55xeOWTPNpSMUM2_9n1EaiQ-uUK078=_JimbKMRUvQ@mail.gmail.com>

On Tue, Nov 16, 2021 at 5:58 AM ast <ast at invalid> wrote:
>
> A curiosity:
>
> q = lambda x: x and q([i for i in x[1:] if i < x[0]]) + [x[0]] + q([i
> for i in x[1:] if i >= x[0]])
>
>  >>> q([7, 5, 9, 0])
> [0, 5, 7, 9]

A nicely obfuscated quicksort. Although the "one line" part falls down
a bit if it has to get wrapped for the mailing list post.

(Horribly inefficient, but it's clear what the intention is.)

ChrisA

From eryksun at gmail.com  Mon Nov 15 14:50:37 2021
From: eryksun at gmail.com (Eryk Sun)
Date: Mon, 15 Nov 2021 13:50:37 -0600
Subject: Unable to compile my C Extension on Windows: unresolved external
 link errors
In-Reply-To: <CABbU2U9HJFBExVv5du0xQTG7nkMh7Xq1sGOBWwfoyMKfDgbSZQ@mail.gmail.com>
References: <CABbU2U_uEeA+0mE0FygzicOhnqS3wNb75rk3gSPPjgU-DHJnHg@mail.gmail.com>
 <2DA56A2E-EA0E-42B7-832E-EF09C1CF162B@barrys-emacs.org>
 <1D6456BF-AA3F-425A-AF2B-3E66A0868BA3@barrys-emacs.org>
 <CABbU2U-RyBjiq4xK+J+3_JZN7yb+7T_J66=XpkJJzkXNdkf+6g@mail.gmail.com>
 <CABbU2U91ncetcGS0iObOURK4-1PX1MFfoBM_ErweUJFJgfHwUA@mail.gmail.com>
 <5E127DC8-AE26-4EFC-8293-9E9FDB39475B@barrys-emacs.org>
 <CABbU2U9HJFBExVv5du0xQTG7nkMh7Xq1sGOBWwfoyMKfDgbSZQ@mail.gmail.com>
Message-ID: <CACL+1asxHLLiR070HtawpM0Hd5aOv-6ezBXviGeJCZvV0wGw9w@mail.gmail.com>

On 11/14/21, Marco Sulla <Marco.Sulla.Python at gmail.com> wrote:
> On Sun, 14 Nov 2021 at 16:42, Barry Scott <barry at barrys-emacs.org> wrote:
>
>> On macOS .dynlib and Unix .so its being extern that does this.
>
> And extern is the default. I understand now.

Per Include/exports.h and Include/pyport.h, Python should be built in
Unix with "default" visibility (per global/local binding) for the API
PyAPI_FUNC(RTYPE) and PyAPI_DATA(RTYPE) symbols. Everything else with
global binding will be hidden via the "-fvisibility=hidden" compiler
option that's configured in the makefile. For example:

    $ readelf -s Objects/dictobject.o | grep HIDDEN | cut -b 40-
    GLOBAL HIDDEN     4 _pydict_global_version
    GLOBAL HIDDEN     1 _PyDict_ClearFreeList
    GLOBAL HIDDEN     1 _PyDict_Fini
    GLOBAL HIDDEN     1 _PyDictKeys_StringLookup
    GLOBAL HIDDEN     9 _Py_dict_lookup
    GLOBAL HIDDEN     1 _PyDict_GetItemHint
    GLOBAL HIDDEN     1 _PyDict_LoadGlobal
    GLOBAL HIDDEN     1 _PyDict_Pop_KnownHash
    GLOBAL HIDDEN     1 _PyDict_FromKeys
    GLOBAL HIDDEN     1 _PyDict_KeysSize
    GLOBAL HIDDEN     1 _PyDict_NewKeysForClass
    GLOBAL HIDDEN     1 _PyObject_InitializeDict
    GLOBAL HIDDEN     1 _PyObject_MakeDictFromIns
    GLOBAL HIDDEN     1 _PyObject_StoreInstanceAt
    GLOBAL HIDDEN     1 _PyObject_GetInstanceAttr
    GLOBAL HIDDEN     1 _PyObject_IsInstanceDictE
    GLOBAL HIDDEN     1 _PyObject_VisitInstanceAt
    GLOBAL HIDDEN     1 _PyObject_ClearInstanceAt
    GLOBAL HIDDEN     1 _PyObject_FreeInstanceAtt
    GLOBAL HIDDEN     1 _PyObjectDict_SetItem
    GLOBAL HIDDEN     1 _PyDictKeys_DecRef
    GLOBAL HIDDEN     1 _PyDictKeys_GetVersionFor

These hidden symbols get linked in the executable or shared-object
with local binding. For example:

    $ readelf -s python | grep _PyDict_FromKeys | cut -b 40-
    LOCAL  DEFAULT   16 _PyDict_FromKeys

I suggest testing your project with Python built as a shared library,
i.e. --enable-shared. The local binding on internal symbols may be a
problem in this case.

From PythonList at DancesWithMice.info  Mon Nov 15 15:49:52 2021
From: PythonList at DancesWithMice.info (dn)
Date: Tue, 16 Nov 2021 09:49:52 +1300
Subject: OT: Alternatives to Jupyter Notebook
In-Reply-To: <CAOj92wYeV9cQ8inp=tJ3SZ_v1A0f0VZYHWem1xV+cNbrxBB+Ow@mail.gmail.com>
References: <CA+i5JwbMjb9n-ysbdsqk+UW9x6Hwni7LbBcsdUTcQnE-NDRxxA@mail.gmail.com>
 <mailman.985.1634744845.4164.python-list@python.org>
 <ivdc84F77llU1@mid.individual.net>
 <CAOj92wYeV9cQ8inp=tJ3SZ_v1A0f0VZYHWem1xV+cNbrxBB+Ow@mail.gmail.com>
Message-ID: <98283afb-bda5-3e38-72e1-12341285d4c3@DancesWithMice.info>

On 15/11/2021 12.24, Paige Bailey wrote:
> If you're experiencing issues with Jupyter, perhaps try JupyterLab:
> JupyterLab Documentation ? JupyterLab 3.2.3 documentation
> <https://jupyterlab.readthedocs.io/en/stable/>
> 
> Or VS Code, which is also open-source and has support for Jupyter notebooks:
> Working with Jupyter Notebooks in Visual Studio Code
> <https://code.visualstudio.com/docs/datascience/jupyter-notebooks>


Which description, depends heavily upon one's definition of
"open-source", surely?

There are major differences between countries, cultures, and languages.
The typical US-understanding narrows-down to "free", as-in $0; and
relegates further meaning of the word, referring to "freedom" (in
French: "libre" - more often translated into English as "liberty").

In relation to software, the question may become: freedom from what?

There is a practice termed "open washing". This is attempting to portray
something as "open", when it may not be so - according to other people's
understandings. Marketing people often skirt definitions and confuse
with their own portrayals, eg what is "virgin olive oil", do "Kiwi
fruit" really come from New Zealand, is it actually "sourdough" bread or
merely flavored, can it be "balsamic vinegar" if it is made with
caramel, is it "champagne" if the grapes were grown somewhere else?

IT-Developers are generally more precise when choosing names and jargon
- even then, some are more accurate/strict than others! Did MS-Windows
really "come free" with that computer, or did some of the $money you
paid go to Microsoft rather than the hardware manufacturer? Do you "own"
your cell-phone/computer or does the Apple contract forbid you from
mending/augmenting it - because they still 'control' it (or you?). Did
you "buy" your Internet Domain Name, or have you in-reality
rented/leased such?

Yes, VS-Code is built upon an 'open' core. Thereafter Microsoft added
"telemetry" (ie "ET, phone home" - for their own purposes(!)), and other
proprietary additions ("featuritus" leads to "better"?). Further, it is
officially only available from their 'store'. Is this "open"? Is it
$free? Are you free from 'surveillance'? Is the product as a whole,
"open-source"? Are you free (legally unencumbered) to modify (or even
read) the source-code of any/all MSFT 'improvements'?

VSCodium claims to be an "open source" version/equivalent. Its
IDE-functionality is (all(?)) the same, but eschews MSFT-only 'features'
- including editor-sharing across the cloud (to my profound regret!).

The 'product' names are very similar.

They are both fundamental and invaluable tools for (Python) developers -
millions of us can't be wrong!

Either one could help the OP, exactly the advice intended by the above post!


Web.Refs:
https://vscodium.com/
https://www.reddit.com/r/privacy/comments/n7l444/vscode_vs_vscodium/

Further philosophy:
- "Libert?, ?galit?, fraternit? " as a catch-cry of the French
Revolution (translates to: liberty, equality, brotherhood)
- becoming "created equal" and "Life, Liberty, and the Pursuit of
Happiness" in the (US) Declaration of Independence
- "some...more equal than others" purported corruption of 'purity'
illustrated in Orwell's "Animal Farm"
-- 
Regards,
=dn

From lukasz at langa.pl  Mon Nov 15 17:12:31 2021
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Mon, 15 Nov 2021 23:12:31 +0100
Subject: [RELEASE] Python 3.9.9 hotfix release is now available
Message-ID: <7023BF6B-A407-4B39-A549-EEDA8440E401@langa.pl>

Get it here: https://www.python.org/downloads/release/python-399/ <https://www.python.org/downloads/release/python-399/>
Python 3.9.9 is the eighth maintenance release of the legacy 3.9 series. Python 3.10 is now the latest feature release series of Python 3. Get the latest release of 3.10.x here <https://python.org/downloads/>.

3.9.9 was released out of schedule as a hotfix for an argparse regression in Python 3.9.8 which caused complex command-line tools to fail recognizing sub-commands properly. Details in BPO-45235 <https://bugs.python.org/issue45235>. There are only three other bugfixes in this release compared to 3.9.8. See the changelog <https://docs.python.org/release/3.9.9/whatsnew/changelog.html> for details on what changed.

Upgrading to 3.9.9 is highly recommended if you?re running Python 3.9.8.

The next Python 3.9 maintenance release will be 3.9.10, currently scheduled for 2022-01-03.

 <https://discuss.python.org/t/python-3-9-9-hotfix-release-is-now-available/11978#we-apologize-for-the-inconvenience-1>We apologize for the inconvenience

?and still hope you?ll enjoy the new release!

Your friendly release team,
Ned Deily @nad <https://discuss.python.org/u/nad>
Steve Dower @steve.dower <https://discuss.python.org/u/steve.dower>
?ukasz Langa @ambv <https://discuss.python.org/u/ambv>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <https://mail.python.org/pipermail/python-list/attachments/20211115/6a355fa1/attachment.sig>

From arj.python at gmail.com  Tue Nov 16 08:04:05 2021
From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer)
Date: Tue, 16 Nov 2021 17:04:05 +0400
Subject: Advantages of Default Factory in Dataclasses
Message-ID: <CADrxXXksfD+nxVT0pPyHjX6jtkqfKDYvyTf11P7eT+Bbtbu81A@mail.gmail.com>

Greetings list,

A simple question: why do we need field(default_factory ) in dataclasses?

Why not make that field as an attribute return a function?

Useful implementation examples / use cases appreciated.

Kind Regards,

Abdur-Rahmaan Janhangeer
about <https://compileralchemy.github.io/> | blog
<https://www.pythonkitchen.com>
github <https://github.com/Abdur-RahmaanJ>
Mauritius

From pbryan at anode.ca  Tue Nov 16 10:17:26 2021
From: pbryan at anode.ca (Paul Bryan)
Date: Tue, 16 Nov 2021 07:17:26 -0800
Subject: Advantages of Default Factory in Dataclasses
In-Reply-To: <CADrxXXksfD+nxVT0pPyHjX6jtkqfKDYvyTf11P7eT+Bbtbu81A@mail.gmail.com>
References: <CADrxXXksfD+nxVT0pPyHjX6jtkqfKDYvyTf11P7eT+Bbtbu81A@mail.gmail.com>
Message-ID: <4765df8e2335cee6e90bd737c5cebed113696d23.camel@anode.ca>

On Tue, 2021-11-16 at 17:04 +0400, Abdur-Rahmaan Janhangeer wrote:

> A simple question: why do we need field(default_factory ) in
> dataclasses?

To initialize a default value when a new instance of the dataclass is
created. For example, if you want a field to default to a dict. A new
dict is created for each instance of the dataclass created.

> Why not make that field as an attribute return a function?

I do not understand the question.

> Useful implementation examples / use cases appreciated.

Any case where a you want a dataclass field to default to a mutable
value. Examples: dicts, lists, other dataclasses.

Paul


From PythonList at DancesWithMice.info  Tue Nov 16 14:49:34 2021
From: PythonList at DancesWithMice.info (dn)
Date: Wed, 17 Nov 2021 08:49:34 +1300
Subject: Advantages of Default Factory in Dataclasses
In-Reply-To: <CADrxXXksfD+nxVT0pPyHjX6jtkqfKDYvyTf11P7eT+Bbtbu81A@mail.gmail.com>
References: <CADrxXXksfD+nxVT0pPyHjX6jtkqfKDYvyTf11P7eT+Bbtbu81A@mail.gmail.com>
Message-ID: <98c8f6e9-792d-8dea-643d-b5de1a199aeb@DancesWithMice.info>

On 17/11/2021 02.04, Abdur-Rahmaan Janhangeer wrote:
> A simple question: why do we need field(default_factory ) in dataclasses?
> 
> Why not make that field as an attribute return a function?
> 
> Useful implementation examples / use cases appreciated.


It's an interesting question: We could define a list (or whatever)
within __post_init__()!

Greater minds than mine may care to comment on the theory that something
defined as a dataclasses.field will be allocated as part of the class
constructor, ie lower-cost. Whereas, something defined post-init will
require extra process-time and storage-allocation. (?)

Doesn't it already return a (evaluated) function? Remember: one is not
limited to Python built-in or PSL types. Thus:

def countdown():
    return [ 10, 9, 8, 7, ..., "blast-off" ]

...
    launch_commentary:list = field( default_factory=countdown )


The use of default-values for mutables is something of a Python
'gotcha'. My use-case is that some wise-soul's decision to do things
this way prevents me from falling into that debugging "slough of despond".
(does such qualify as a "use case"?)
-- 
Regards,
=dn

From david at lowryduda.com  Tue Nov 16 18:00:05 2021
From: david at lowryduda.com (David Lowry-Duda)
Date: Tue, 16 Nov 2021 18:00:05 -0500
Subject: Advantages of Default Factory in Dataclasses
In-Reply-To: <CADrxXXksfD+nxVT0pPyHjX6jtkqfKDYvyTf11P7eT+Bbtbu81A@mail.gmail.com>
References: <CADrxXXksfD+nxVT0pPyHjX6jtkqfKDYvyTf11P7eT+Bbtbu81A@mail.gmail.com>
Message-ID: <YZQ39Zd0tEhhdEfL@icerm-dld>

On Tue, Nov 16, 2021 at 05:04:05PM +0400, Abdur-Rahmaan Janhangeer wrote:
> A simple question: why do we need field(default_factory ) in 
> dataclasses?

For the same reason that the following code doesn't do what some people 
might expect it to:

```python
def add_to(elem, inlist=[]):
    inlist.append(elem)
    return inlist

list1 = add_to(1)
list2 = add_to(2)
print(list1)  # prints [1]
print(list2)  # prints [1, 2], potentially confusing
```

Mutable attributes are created once, upon definition. Reusing the same 
function (or instantiating objects of the same class) and modifying this 
attribute can lead to possibly unexpected side effects as above.

- DLD

From auriocus at gmx.de  Tue Nov 16 02:15:38 2021
From: auriocus at gmx.de (Christian Gollwitzer)
Date: Tue, 16 Nov 2021 08:15:38 +0100
Subject: One line sort
In-Reply-To: <61925c51$0$28601$426a34cc@news.free.fr>
References: <61925c51$0$28601$426a34cc@news.free.fr>
Message-ID: <smvlqr$hqt$1@dont-email.me>

Am 15.11.21 um 14:10 schrieb ast:
> A curiosity:
> 
> q = lambda x: x and q([i for i in x[1:] if i < x[0]]) + [x[0]] + q([i 
> for i in x[1:] if i >= x[0]])
> 
>  >>> q([7, 5, 9, 0])
> [0, 5, 7, 9]

That seems to be a translation of the classic Haskell quicksort example:

qsort [] = []
qsort (x:xs) = qsort [a | a <- xs, a < x]
                       ++ [x] ++
                qsort [b | b <- xs, b >= x]


The Haskell version is a bit clearer IMHO due to the pattern matching, 
but that results in a 2 liner :)

	Christian

From martin.schoon at gmail.com  Tue Nov 16 04:49:46 2021
From: martin.schoon at gmail.com (Martin =?UTF-8?Q?Sch=C3=B6=C3=B6n?=)
Date: 16 Nov 2021 09:49:46 GMT
Subject: Alternatives to Jupyter Notebook
References: <CA+i5JwbMjb9n-ysbdsqk+UW9x6Hwni7LbBcsdUTcQnE-NDRxxA@mail.gmail.com>
 <mailman.985.1634744845.4164.python-list@python.org>
 <ivdc84F77llU1@mid.individual.net>
 <87h7cdbofm.fsf@hornfels.zedat.fu-berlin.de>
Message-ID: <ivhd5qF2bjU1@mid.individual.net>

Den 2021-11-15 skrev Loris Bennett <loris.bennett at fu-berlin.de>:
> Martin Sch??n <martin.schoon at gmail.com> writes:
>
>> Den 2021-10-20 skrev Shaozhong SHI <shishaozhong at gmail.com>:
>>>
>>> My Jupyter notebook becomes unresponsive in browsers.
>>>
>> Odd, I never had any problems like that. I use Firefox on Linux.
>>
>>> Are there alternatives to read, edit and run Jupyter Notebook?
>>>
>> I know some people use emacs orgmode. I have never tried it
>> myself and do not know how well it works.
>
> I don't know Jupyter well enough to know whether Org mode is a real
> alternative.  However, with Org mode you can combine text and code in
> multiple languages, run the code within Emacs and then export the
> results in various formats such as PDF and HTML.
>
I realise I was not making myself clear. Orgmode is great. Full stop.

What I don't have experience of is combining orgmode with Jupyter.

/Martin

From alan at csail.mit.edu  Tue Nov 16 18:24:43 2021
From: alan at csail.mit.edu (Alan Bawden)
Date: Tue, 16 Nov 2021 18:24:43 -0500
Subject: Advantages of Default Factory in Dataclasses
References: <CADrxXXksfD+nxVT0pPyHjX6jtkqfKDYvyTf11P7eT+Bbtbu81A@mail.gmail.com>
 <YZQ39Zd0tEhhdEfL@icerm-dld>
 <mailman.28.1637104197.19479.python-list@python.org>
Message-ID: <86lf1nhef8.fsf@williamsburg.bawden.org>

David Lowry-Duda <david at lowryduda.com> writes:

   ...

   For the same reason that the following code doesn't do what some people 
   might expect it to:

   ```python
   def add_to(elem, inlist=[]):
       inlist.append(elem)
       return inlist

   list1 = add_to(1)
   list2 = add_to(2)
   print(list1)  # prints [1]
   print(list2)  # prints [1, 2], potentially confusing
   ```

Not only does it not print what "most people" expect.  It also doesn't
print what _you_ expect!  (But you made your point.)

From skip.montanaro at gmail.com  Wed Nov 17 08:19:24 2021
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Wed, 17 Nov 2021 07:19:24 -0600
Subject: Proliferation of Python packaging formats
Message-ID: <CANc-5UynNv8L+2SP=n1-gKcnF3=dYW6eArEdRbBT3HxHaQYhgg@mail.gmail.com>

Is the proliferation of packaging formats in Python as nutzo as this author
believes?

https://drewdevault.com/2021/11/16/Python-stop-screwing-distros-over.html

Asking because I've never been in the business of releasing "retail" Python
applications or packages.

Skip

From rosuav at gmail.com  Wed Nov 17 09:44:32 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 18 Nov 2021 01:44:32 +1100
Subject: Proliferation of Python packaging formats
In-Reply-To: <CANc-5UynNv8L+2SP=n1-gKcnF3=dYW6eArEdRbBT3HxHaQYhgg@mail.gmail.com>
References: <CANc-5UynNv8L+2SP=n1-gKcnF3=dYW6eArEdRbBT3HxHaQYhgg@mail.gmail.com>
Message-ID: <CAPTjJmqDPE88_ySSgA0ALf=Q0Ka75SaV+g9tA7XhqQN107AC8Q@mail.gmail.com>

On Thu, Nov 18, 2021 at 12:20 AM Skip Montanaro
<skip.montanaro at gmail.com> wrote:
>
> Is the proliferation of packaging formats in Python as nutzo as this author
> believes?
>
> https://drewdevault.com/2021/11/16/Python-stop-screwing-distros-over.html
>
> Asking because I've never been in the business of releasing "retail" Python
> applications or packages.
>

It's a problem if you feel the need to release in every single
possible way. The XKCD in that example is a showcase of what happens
when you attempt to install in every possible way, which is usually a
consequence of everyone releasing in just one way.

He's blaming the PSF, but that's not really fair. The PSF has never
said "hey everyone, make sure you package for conda as well as posting
on PyPI". Part of the proliferation comes because there are multiple
completely independent distributions, and in the XKCD shown there, he
has installed Python at least five times from completely different
sources (the OS-provided one, two from Homebrew, one from Anaconda,
and one from python.org), and maybe installed pip into one of those as
well. Duh, of course that's going to cause problems.

Here's the reality: OS-provided Pythons are always going to exist, and
they will always have a place (because I should be able to say "apt
install <X>" without having to worry about whether that's going to
download from PyPI), and they'll never have 100% of everything exactly
how you want it, so you will always have more than one place where
packages can be installed, more than one package format. And it's not
the PSF's fault, nor the PSF's responsibility.

The massive complexities come when people aren't satisfied with the
status quo, and create a new thing. The blog mentions tox, flit,
conda, and poetry, all of which are third-party distributions or
package managers. Is the PSF supposed to tell people "don't create new
package managers for any Python code"? Seriously?

Blog dismissed as whining.

ChrisA

From ast at invalid  Wed Nov 17 06:12:29 2021
From: ast at invalid (ast)
Date: Wed, 17 Nov 2021 12:12:29 +0100
Subject: Symbolic links on Windows
Message-ID: <6194e3a0$0$8880$426a74cc@news.free.fr>

Hello,

It seems that symbolic links on Windows are not
well reconized by modules os or pathlib.

I have a file named json.txt on my destop. With a
drag and drop right click on it I create a link
automatically named: json.txt - Raccourci.lnk

Then:

 >>> from pathlib import Path

 >>> p2 = Path('C:/Users/jm/desktop/json.txt - Raccourci.lnk')

 >>> p2
WindowsPath('C:/Users/jm/desktop/json.txt - Raccourci.lnk')

 >>> p2.exists()
True

 >>> p2.is_file()
True

 >>> p2.is_symlink()
False

With this last command I was expecting True and for
p2_is_file() I was expecting False



With os, it's the same

import os

 >>> os.path.exists(p2)
True

 >>> os.path.isfile(p2)
True

os.path.islink(p2)
False


What's wrong plz ?

From ast at invalid  Wed Nov 17 08:41:58 2021
From: ast at invalid (ast)
Date: Wed, 17 Nov 2021 14:41:58 +0100
Subject: Symbolic links on Windows
In-Reply-To: <6194f115$0$4977$426a74cc@news.free.fr>
References: <6194e3a0$0$8880$426a74cc@news.free.fr>
 <6194f115$0$4977$426a74cc@news.free.fr>
Message-ID: <619506a8$0$6452$426a74cc@news.free.fr>

Le 17/11/2021 ? 13:10, Python a ?crit?:
> ast wrote:
>> Hello,
>>
>> It seems that symbolic links on Windows are not
>> well reconized by modules os or pathlib.
>>
>> I have a file named json.txt on my destop. With a
>> drag and drop right click on it I create a link
>> automatically named: json.txt - Raccourci.lnk
> 
> This is not a symbolic link.
> 
> Symbolic links were introduced in Windows Vista/Windows
> Server 2008. A .lnk file is just a regular file containing
> the path of the target (you can open it and read it if you
> wish) that is interpreted in a specific way by the graphical
> Shell and some other applications.
> 
> 

Yes you are right

On Windows symbolic links are created in a cmd tool:
e.g

mklink json2.txt json.txt

And it is something different than shortcut files (suffix .lnk)

From jon+usenet at unequivocal.eu  Wed Nov 17 09:10:19 2021
From: jon+usenet at unequivocal.eu (Jon Ribbens)
Date: Wed, 17 Nov 2021 14:10:19 -0000 (UTC)
Subject: Proliferation of Python packaging formats
References: <CANc-5UynNv8L+2SP=n1-gKcnF3=dYW6eArEdRbBT3HxHaQYhgg@mail.gmail.com>
 <mailman.29.1637155179.19479.python-list@python.org>
Message-ID: <slrnspa3ab.5h7.jon+usenet@raven.unequivocal.eu>

On 2021-11-17, Skip Montanaro <skip.montanaro at gmail.com> wrote:
> Is the proliferation of packaging formats in Python as nutzo as this author
> believes?
>
> https://drewdevault.com/2021/11/16/Python-stop-screwing-distros-over.html
>
> Asking because I've never been in the business of releasing "retail" Python
> applications or packages.

Well the first paragraph is ridiculous. I've never heard of half of the
things he lists as being necessary to deal with, and half the remainder
are just words relating to packages, i.e. you could make a similar list
for any language.

The other major problem with that post is that it gives no examples
or even clues as to what the author's actual problem is...

On the other hand it is true that Python's packaging system is confusing
and very badly documented, and I doubt the vast majority of people have
any idea of what the difference between 'distutils' and 'setuptools' is
(I certainly don't), except inasmuch as 'setuptools' (and 'wheel') is
something you have to remember to manually update after creating a
virtual environment before installing your actual packages.

It's also true that you have to remember with Python that you basically
cannot use 'pip' to install anything globally as it will interfere with
the operating system's packaging. You must use virtual envs for
everything, or the operating system's provided packages.

Also PEP 518's choice of TOML is absolutely risible, a language about
which the only positive thing can be said is that it's not as bad as
YAML, and for which Python doesn't even have a built-in parser -
something that should have absolutely ruled it out as an option.

From mats at wichmann.us  Wed Nov 17 15:57:58 2021
From: mats at wichmann.us (Mats Wichmann)
Date: Wed, 17 Nov 2021 13:57:58 -0700
Subject: Proliferation of Python packaging formats
In-Reply-To: <CAPTjJmqDPE88_ySSgA0ALf=Q0Ka75SaV+g9tA7XhqQN107AC8Q@mail.gmail.com>
References: <CANc-5UynNv8L+2SP=n1-gKcnF3=dYW6eArEdRbBT3HxHaQYhgg@mail.gmail.com>
 <CAPTjJmqDPE88_ySSgA0ALf=Q0Ka75SaV+g9tA7XhqQN107AC8Q@mail.gmail.com>
Message-ID: <78072033-c685-a912-08bc-779f688ac5fc@wichmann.us>


On 11/17/21 07:44, Chris Angelico wrote:
> On Thu, Nov 18, 2021 at 12:20 AM Skip Montanaro
> <skip.montanaro at gmail.com> wrote:
>>
>> Is the proliferation of packaging formats in Python as nutzo as this author
>> believes?
>>
>> https://drewdevault.com/2021/11/16/Python-stop-screwing-distros-over.html
>>
>> Asking because I've never been in the business of releasing "retail" Python
>> applications or packages.
>>
> 
> It's a problem if you feel the need to release in every single
> possible way. The XKCD in that example is a showcase of what happens
> when you attempt to install in every possible way, which is usually a
> consequence of everyone releasing in just one way.
> 
> He's blaming the PSF, but that's not really fair. The PSF has never
> said "hey everyone, make sure you package for conda as well as posting
> on PyPI". Part of the proliferation comes because there are multiple
> completely independent distributions, and in the XKCD shown there, he
> has installed Python at least five times from completely different
> sources (the OS-provided one, two from Homebrew, one from Anaconda,
> and one from python.org), and maybe installed pip into one of those as
> well. Duh, of course that's going to cause problems.

There are people working on the problems, but the author doesn't seem to 
like the progress being made. Here's one jumping point for those 
interested who haven't otherwise followed this stuff:

https://discuss.python.org/t/graceful-cooperation-between-external-and-python-package-managers-pep-668/10302

Some of the problems in Python ecosystem, which categorically does *not* 
just consist of machines running a Linux distro, are in fact caused by 
the choices the Linux distros make in packaging. The model of "get 
everything you need from your distro's packages" doesn't come close to 
cutting it these days, so there has to be coexistence.  Anecdotally, I 
have a module I use in a couple of projects and I forgot that on my main 
machine I actually had that supplied by a Fedora package (silly me), and 
surprise - it's been dropped in F35.

Yes, it's true that the PyPA does not set itself up to be the final 
picker of winners, so this comment isn't just off the wall:

"P.S. PEP-517 and 518 are a start, but are very disappointing in how 
little they address distro problems. These PEPs are designed to tolerate 
the proliferation of build systems, which is exactly what needs to stop. 
Python ought to stop trying to avoid hurting anyone?s feelings and pick 
one."

That doesn't seem likely to happen, and in fact there's probably 
strength in having multiple implementations of a PEP (in other worlds, 
like W3C, multiple implementations are mandatory before a standard can 
be accepted).

From nt_mahmood at yahoo.com  Thu Nov 18 04:49:25 2021
From: nt_mahmood at yahoo.com (Mahmood Naderan)
Date: Thu, 18 Nov 2021 09:49:25 +0000 (UTC)
Subject: get_axes not present?
References: <1263450234.1243976.1637228965678.ref@mail.yahoo.com>
Message-ID: <1263450234.1243976.1637228965678@mail.yahoo.com>

Hi
I am using the following versions


>>> import matplotlib
>>> print(matplotlib. __version__)
3.3.4
>>> import pandas as pd
>>> print(pd.__version__)
1.2.3
>>> import sys
>>> sys.version_info
sys.version_info(major=3, minor=8, micro=10, releaselevel='final', serial=0)



In my code, I use axes in Pandas plot() like this (note that I omit some variables in this snippet to highlight the problem):



def plot_dataframe(df, cnt, axes):
??? plt.subplot(2, 1, 1)
??? ax1 = row.plot( fontsize=font_size, linewidth=line_width, markersize=marker_size, marker='o', title='Raw values', label=cnt, ax=axes[0] )



def plot_kernels(my_dict2):
??? fig,axes = plt.subplots(2,1, figsize=(20, 15))
??? should_plot = plot_dataframe(df, cnt, axes=axes)
??? for ax in axes:
??????? ax.legend()
??? plt.show()



However, I get this error:


Traceback (most recent call last):
? File "process_csv.py", line 174, in <module>
??? plot_kernels( my_dict2 )
? File "process_csv.py", line 62, in plot_kernels
??? should_plot = plot_dataframe(df, cnt, axes=axes)
? File "process_csv.py", line 34, in plot_dataframe
??? ax1 = row.plot( fontsize=font_size, linewidth=line_width, markersize=marker_size, marker='o', title='Raw values', label=cnt, ax=axes[0] )
? File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_core.py", line 955, in __call__
??? return plot_backend.plot(data, kind=kind, **kwargs)
? File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/__init__.py", line 61, in plot
??? plot_obj.generate()
? File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/core.py", line 283, in generate
??? self._adorn_subplots()
? File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/core.py", line 483, in _adorn_subplots
??? all_axes = self._get_subplots()
? File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/core.py", line 903, in _get_subplots
??? ax for ax in self.axes[0].get_figure().get_axes() if isinstance(ax, Subplot)
AttributeError: 'NoneType' object has no attribute 'get_axes'




I guess there is a mismatch between versions. Is there any workaround for that?




Regards,
Mahmood

From david at lowryduda.com  Thu Nov 18 10:43:10 2021
From: david at lowryduda.com (David Lowry-Duda)
Date: Thu, 18 Nov 2021 10:43:10 -0500
Subject: Advantages of Default Factory in Dataclasses
In-Reply-To: <86lf1nhef8.fsf@williamsburg.bawden.org>
References: <CADrxXXksfD+nxVT0pPyHjX6jtkqfKDYvyTf11P7eT+Bbtbu81A@mail.gmail.com>
 <YZQ39Zd0tEhhdEfL@icerm-dld>
 <mailman.28.1637104197.19479.python-list@python.org>
 <86lf1nhef8.fsf@williamsburg.bawden.org>
Message-ID: <YZZ0jr4oOZ0CGklg@icerm-dld>

On Tue, Nov 16, 2021 at 06:24:43PM -0500, Alan Bawden wrote:
>    ```python
>    def add_to(elem, inlist=[]):
>        inlist.append(elem)
>        return inlist
> 
>    list1 = add_to(1)
>    list2 = add_to(2)
>    print(list1)  # prints [1]
>    print(list2)  # prints [1, 2], potentially confusing
>    ```
> 
> Not only does it not print what "most people" expect.  It also doesn't
> print what _you_ expect!  (But you made your point.)

Haha, you're right. I would guess that I reordered the statements when I 
quickly checked this in the interpreter. But indeed, both list1 and 
list2 point to inlist, and thus are the same. Whoops!

- DLD

From mats at wichmann.us  Thu Nov 18 10:58:08 2021
From: mats at wichmann.us (Mats Wichmann)
Date: Thu, 18 Nov 2021 08:58:08 -0700
Subject: get_axes not present?
In-Reply-To: <1263450234.1243976.1637228965678@mail.yahoo.com>
References: <1263450234.1243976.1637228965678.ref@mail.yahoo.com>
 <1263450234.1243976.1637228965678@mail.yahoo.com>
Message-ID: <500ef579-bd07-6a5c-b652-3b4dad8e4ba4@wichmann.us>

On 11/18/21 02:49, Mahmood Naderan via Python-list wrote:

>  ? File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/core.py", line 903, in _get_subplots
>  ??? ax for ax in self.axes[0].get_figure().get_axes() if isinstance(ax, Subplot)
> AttributeError: 'NoneType' object has no attribute 'get_axes'
> 
> 
> 
> 
> I guess there is a mismatch between versions. Is there any workaround for that?

It's not saying get_axes doesn't exist because of version skew, it's 
saying that the object returned by the call to the left of it 
(get_figure()) returned None, and None doesn't have methods....

Something isn't set up right, but you'll have to trace that through.


From nt_mahmood at yahoo.com  Thu Nov 18 12:54:43 2021
From: nt_mahmood at yahoo.com (Mahmood Naderan)
Date: Thu, 18 Nov 2021 17:54:43 +0000 (UTC)
Subject: get_axes not present?
In-Reply-To: <500ef579-bd07-6a5c-b652-3b4dad8e4ba4@wichmann.us>
References: <1263450234.1243976.1637228965678.ref@mail.yahoo.com>
 <1263450234.1243976.1637228965678@mail.yahoo.com>
 <500ef579-bd07-6a5c-b652-3b4dad8e4ba4@wichmann.us>
Message-ID: <203139843.1328468.1637258083048@mail.yahoo.com>

>It's not saying get_axes doesn't exist because of version skew, it's
>saying that the object returned by the call to the left of it
>(get_figure()) returned None, and None doesn't have methods....
>
>Something isn't set up right, but you'll have to trace that through.



Do you think the following statement is correct?

ax1 = row.plot( fontsize=font_size, 
??????????????????????? linewidth=line_width, 
??????????????????????? markersize=marker_size, 
??????????????????????? marker='o', 
??????????????????????? title='Raw values', 
??????????????????????? label=cnt, 
??????????????????????? ax=axes[0] )
ax1.set_ylabel( yax_label, fontsize=font_size )


As you can see I put the result of plot() to ax1 and then use some functions, e.g. set_ylabel().

On the other hand, I have specified `label` and `ax` in plot(), too.



Regards,
Mahmood


From torriem at gmail.com  Thu Nov 18 14:28:28 2021
From: torriem at gmail.com (Michael Torrie)
Date: Thu, 18 Nov 2021 12:28:28 -0700
Subject: get_axes not present?
In-Reply-To: <203139843.1328468.1637258083048@mail.yahoo.com>
References: <1263450234.1243976.1637228965678.ref@mail.yahoo.com>
 <1263450234.1243976.1637228965678@mail.yahoo.com>
 <500ef579-bd07-6a5c-b652-3b4dad8e4ba4@wichmann.us>
 <203139843.1328468.1637258083048@mail.yahoo.com>
Message-ID: <afc3960c-c7cf-c09a-703b-57be45793db4@gmail.com>

On 11/18/21 10:54 AM, Mahmood Naderan via Python-list wrote:
> As you can see I put the result of plot() to ax1 and then use some functions, e.g. set_ylabel().

And what is the result of plot()?  Is it a valid object, or is it None?

From * at eli.users.panix.com  Wed Nov 17 15:34:14 2021
From: * at eli.users.panix.com (Eli the Bearded)
Date: Wed, 17 Nov 2021 20:34:14 -0000 (UTC)
Subject: news to me today: RIP Aahz
Message-ID: <eli$2111171533@qaz.wtf>

Aahz, co-author of Python for Dummies with Stef Maruch, recently passed
away.

Tiny death notice (with name typo) from the wilds of the Internet:

http://file770.com/pixel-scroll-10-15-21-i-know-what-pixel-you-scrolled-last-summer/

    (12) AAHZ MARUCH (1967-2021). [Item by James Davis Nicoll.] Python
    programmer, whose fannish activities date back at least as far as
    classic USENET (alt.poly and other groups), died October 14
    following several years of ill health. Survived by partner Steph
    Maruch.

    Editor's postscript: Alan Prince Winston earlier this year described
    him as "an unstoppable-seeming guy" who "became a contra and square
    dance caller and choreographer despite really severe hearing
    impairment."

I met Aahz once. He always wanted to be a mononym person, and used his
partner's surname only reluctantly.

Elijah
------
Aahz's rule6 website seems to be held by a squatter now


From sjlukacs at gmail.com  Wed Nov 17 16:31:23 2021
From: sjlukacs at gmail.com (lucas)
Date: Wed, 17 Nov 2021 13:31:23 -0800 (PST)
Subject: import question
Message-ID: <f8f5d859-df62-4923-adfd-a2017d3024e6n@googlegroups.com>

hello one and all,

are there any other ways to import a module or package other then the "import" or "from...import..." statements?  i ask because i'm allowing programming on my web2py website and i don't want any accessing packages like os or sys.

thank you in advance and have a great day, lucas

From framstag at rus.uni-stuttgart.de  Thu Nov 18 07:21:57 2021
From: framstag at rus.uni-stuttgart.de (Ulli Horlacher)
Date: Thu, 18 Nov 2021 12:21:57 +0000 (UTC)
Subject: several issues with pyinstaller on Windows 10
Message-ID: <sn5gh5$47b$1@news2.informatik.uni-stuttgart.de>

On Windows 10 I have installed:

P:\W10>python --version
Python 3.10.0

P:\W10>pyinstaller --version
4.7

I can compile a VERY simple Python program:

P:\W10>type argv.pv
#!/usr/bin/python3

import sys

for a in sys.argv: print("["+a+"]")


P:\W10>pyinstaller.exe --onefile tcpbm.py

But I can run it only once:

P:\W10\dist>argv a b
[dist\argv]
[a]
[b]

P:\W10\dist>argv a b
The process cannot access the file because it is being used by another process.

P:\W10\dist>argv 1 2 zzz
The process cannot access the file because it is being used by another process.


Compiling a little more complex program (with net IO) fails:

P:\W10>pyinstaller.exe --onefile tcpbm.py
140 INFO: PyInstaller: 4.7
140 INFO: Python: 3.10.0
171 INFO: Platform: Windows-10-10.0.19041-SP0
171 INFO: wrote P:\W10\tcpbm.spec
171 INFO: UPX is not available.
203 INFO: Extending PYTHONPATH with paths
['P:\\W10']
531 INFO: checking Analysis
547 INFO: Building Analysis because Analysis-00.toc is non existent
547 INFO: Initializing module dependency graph...
562 INFO: Caching module graph hooks...
593 INFO: Analyzing base_library.zip ...
4656 INFO: Processing pre-find module path hook distutils from 'C:\\Users\\admin\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\PyInstaller\\hooks\\pre_find_module_path\\hook-distutils.py'.
4656 INFO: distutils: retargeting to non-venv dir 'C:\\Users\\admin\\AppData\\Local\\Programs\\Python\\Python310\\lib'
5781 INFO: Caching module dependency graph...
6031 INFO: running Analysis Analysis-00.toc
6031 INFO: Adding Microsoft.Windows.Common-Controls to dependent assemblies of final executable
  required by C:\Users\admin\AppData\Local\Programs\Python\Python310\python.exe
6187 INFO: Analyzing P:\W10\tcpbm.py
6250 INFO: Processing module hooks...
6250 INFO: Loading module hook 'hook-difflib.py' from 'C:\\Users\\admin\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\PyInstaller\\hooks'...
6250 INFO: Loading module hook 'hook-distutils.py' from 'C:\\Users\\admin\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\PyInstaller\\hooks'...
6250 INFO: Loading module hook 'hook-distutils.util.py' from 'C:\\Users\\admin\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\PyInstaller\\hooks'...
6250 INFO: Loading module hook 'hook-encodings.py' from 'C:\\Users\\admin\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\PyInstaller\\hooks'...
6390 INFO: Loading module hook 'hook-heapq.py' from 'C:\\Users\\admin\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\PyInstaller\\hooks'...
6390 INFO: Loading module hook 'hook-lib2to3.py' from 'C:\\Users\\admin\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\PyInstaller\\hooks'...
6484 INFO: Loading module hook 'hook-multiprocessing.util.py' from 'C:\\Users\\admin\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\PyInstaller\\hooks'...
6484 INFO: Loading module hook 'hook-pickle.py' from 'C:\\Users\\admin\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\PyInstaller\\hooks'...
6484 INFO: Loading module hook 'hook-sysconfig.py' from 'C:\\Users\\admin\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\PyInstaller\\hooks'...
6484 INFO: Loading module hook 'hook-xml.etree.cElementTree.py' from 'C:\\Users\\admin\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\PyInstaller\\hooks'...
6500 INFO: Loading module hook 'hook-xml.py' from 'C:\\Users\\admin\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\PyInstaller\\hooks'...
6546 INFO: Loading module hook 'hook-_tkinter.py' from 'C:\\Users\\admin\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\PyInstaller\\hooks'...
6734 INFO: checking Tree
6734 INFO: Building Tree because Tree-00.toc is non existent
6749 INFO: Building Tree Tree-00.toc
6890 INFO: checking Tree
6890 INFO: Building Tree because Tree-01.toc is non existent
6890 INFO: Building Tree Tree-01.toc
7031 INFO: checking Tree
7031 INFO: Building Tree because Tree-02.toc is non existent
7031 INFO: Building Tree Tree-02.toc
7062 INFO: Looking for ctypes DLLs
7078 INFO: Analyzing run-time hooks ...
7093 INFO: Including run-time hook 'C:\\Users\\admin\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\PyInstaller\\hooks\\rthooks\\pyi_rth_pkgutil.py'
7093 INFO: Including run-time hook 'C:\\Users\\admin\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\PyInstaller\\hooks\\rthooks\\pyi_rth_multiprocessing.py'
7109 INFO: Including run-time hook 'C:\\Users\\admin\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\PyInstaller\\hooks\\rthooks\\pyi_rth_inspect.py'
7109 INFO: Looking for dynamic libraries
7546 INFO: Looking for eggs
7546 INFO: Using Python library
C:\Users\admin\AppData\Local\Programs\Python\Python310\python310.dll
7562 INFO: Found binding redirects:
[]
7562 INFO: Warnings written to P:\W10\build\tcpbm\warn-tcpbm.txt
7624 INFO: Graph cross-reference written to P:\W10\build\tcpbm\xref-tcpbm.html
7640 INFO: checking PYZ
7640 INFO: Building PYZ because PYZ-00.toc is non existent
7640 INFO: Building PYZ (ZlibArchive) P:\W10\build\tcpbm\PYZ-00.pyz
8125 INFO: Building PYZ (ZlibArchive) P:\W10\build\tcpbm\PYZ-00.pyz completed successfully.
8155 INFO: checking PKG
8155 INFO: Building PKG because PKG-00.toc is non existent
8155 INFO: Building PKG (CArchive) tcpbm.pkg
9890 INFO: Building PKG (CArchive) tcpbm.pkg completed successfully.
9890 INFO: Bootloader C:\Users\admin\AppData\Local\Programs\Python\Python310\lib\site-packages\PyInstaller\bootloader\Windows-64bit\run.exe
9890 INFO: checking EXE
9890 INFO: Building EXE because EXE-00.toc is non existent
9890 INFO: Building EXE from EXE-00.toc
9890 INFO: Copying bootloader EXE to P:\W10\dist\tcpbm.exe
9921 INFO: Copying icon to EXE
9921 INFO: Copying icons from ['C:\\Users\\admin\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\PyInstaller\\bootloader\\images\\icon-console.ico']
9937 INFO: Writing RT_GROUP_ICON 0 resource with 104 bytes
9937 INFO: Writing RT_ICON 1 resource with 3752 bytes
9937 INFO: Writing RT_ICON 2 resource with 2216 bytes
9937 INFO: Writing RT_ICON 3 resource with 1384 bytes
9937 INFO: Writing RT_ICON 4 resource with 37019 bytes
9937 INFO: Writing RT_ICON 5 resource with 9640 bytes
9937 INFO: Writing RT_ICON 6 resource with 4264 bytes
9937 INFO: Writing RT_ICON 7 resource with 1128 bytes
9968 INFO: Copying 0 resources to EXE
9968 INFO: Emedding manifest in EXE
9968 INFO: Updating manifest in P:\W10\dist\tcpbm.exe
10203 INFO: Updating resource type 24 name 1 language 0
Traceback (most recent call last):
  File "C:\Users\admin\AppData\Local\Programs\Python\Python310\lib\site-packages\win32ctypes\pywin32\pywintypes.py", line 35, in pywin32error
    yield
  File "C:\Users\admin\AppData\Local\Programs\Python\Python310\lib\site-packages\win32ctypes\pywin32\win32api.py", line 229, in EndUpdateResource
    _resource._EndUpdateResource(handle, discard)
  File "C:\Users\admin\AppData\Local\Programs\Python\Python310\lib\site-packages\win32ctypes\core\ctypes\_util.py", line 64, in check_false
    raise make_error(function, function_name)
OSError: [WinError 110] The system cannot open the device or file specified.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\admin\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\admin\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\admin\AppData\Local\Programs\Python\Python310\Scripts\pyinstaller.exe\__main__.py", line 7, in <module>
  File "C:\Users\admin\AppData\Local\Programs\Python\Python310\lib\site-packages\PyInstaller\__main__.py", line 124, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "C:\Users\admin\AppData\Local\Programs\Python\Python310\lib\site-packages\PyInstaller\__main__.py", line 58, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "C:\Users\admin\AppData\Local\Programs\Python\Python310\lib\site-packages\PyInstaller\building\build_main.py", line 782, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
  File "C:\Users\admin\AppData\Local\Programs\Python\Python310\lib\site-packages\PyInstaller\building\build_main.py", line 714, in build
    exec(code, spec_namespace)
  File "P:\W10\tcpbm.spec", line 23, in <module>
    exe = EXE(pyz,
  File "C:\Users\admin\AppData\Local\Programs\Python\Python310\lib\site-packages\PyInstaller\building\api.py", line 507, in __init__
    self.__postinit__()
  File "C:\Users\admin\AppData\Local\Programs\Python\Python310\lib\site-packages\PyInstaller\building\datastruct.py", line 155, in __postinit__
    self.assemble()
  File "C:\Users\admin\AppData\Local\Programs\Python\Python310\lib\site-packages\PyInstaller\building\api.py", line 666, in assemble
    self.manifest.update_resources(self.name, [1])
  File "C:\Users\admin\AppData\Local\Programs\Python\Python310\lib\site-packages\PyInstaller\utils\win32\winmanifest.py", line 902, in update_resources
    UpdateManifestResourcesFromXML(dstpath, self.toprettyxml().encode("UTF-8"), names, languages)
  File "C:\Users\admin\AppData\Local\Programs\Python\Python310\lib\site-packages\PyInstaller\utils\win32\winmanifest.py", line 990, in UpdateManifestResourcesFromXML
    winresource.UpdateResources(dstpath, xmlstr, RT_MANIFEST, names or [name], languages or [0, "*"])
  File "C:\Users\admin\AppData\Local\Programs\Python\Python310\lib\site-packages\PyInstaller\utils\win32\winresource.py", line 189, in UpdateResources
    win32api.EndUpdateResource(hdst, 0)
  File "C:\Users\admin\AppData\Local\Programs\Python\Python310\lib\site-packages\win32ctypes\pywin32\win32api.py", line 228, in EndUpdateResource
    with _pywin32error():
  File "C:\Users\admin\AppData\Local\Programs\Python\Python310\lib\contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "C:\Users\admin\AppData\Local\Programs\Python\Python310\lib\site-packages\win32ctypes\pywin32\pywintypes.py", line 37, in pywin32error
    raise error(exception.winerror, exception.function, exception.strerror)
win32ctypes.pywin32.pywintypes.error: (110, 'EndUpdateResourceW', 'The system cannot open the device or file specified.')


For my Windows knowledge is near zero (I am an UNIX guy) I cannot
interpret these error messages. What is going wrong?


The source code is:

https://fex.rus.uni-stuttgart.de/fop/rA3oDJfN/tcpbm.py


-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher at tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/

From rosuav at gmail.com  Thu Nov 18 15:20:56 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 19 Nov 2021 07:20:56 +1100
Subject: import question
In-Reply-To: <f8f5d859-df62-4923-adfd-a2017d3024e6n@googlegroups.com>
References: <f8f5d859-df62-4923-adfd-a2017d3024e6n@googlegroups.com>
Message-ID: <CAPTjJmq0K+9W9AvHVJfpOiZCw0SO8jN_ga1sTO1UgbgQ5FTdaA@mail.gmail.com>

On Fri, Nov 19, 2021 at 7:09 AM lucas <sjlukacs at gmail.com> wrote:
>
> hello one and all,
>
> are there any other ways to import a module or package other then the "import" or "from...import..." statements?  i ask because i'm allowing programming on my web2py website and i don't want any accessing packages like os or sys.
>
> thank you in advance and have a great day, lucas
>

Yes, there are many. For starters, the importlib module can do
anything that importing can do, as can the __import__ function. Plus,
with Python code, you could open the file, read from it, and exec it.
There are myriad ways to fetch up code, and it's even possible to
break out of a sandbox without ever using a single underscore.

If you're trying to make a Python-in-Python sandbox, I recommend not.
Instead, use an OS-level sandbox (a chroot, probably some sort of CPU
usage limiting, etc), and use that to guard the entire Python process.
Python-in-Python will basically *never* be secure.

ChrisA

From Marco.Sulla.Python at gmail.com  Thu Nov 18 17:30:35 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Thu, 18 Nov 2021 23:30:35 +0100
Subject: How to support annotations for a custom type in a C extension?
In-Reply-To: <si7jla$136h$1@ciao.gmane.io>
References: <CABbU2U-XzTKOW2MfbYnEQKDKj9pMfZ8PAHrjLDX__e2FsHFp5w@mail.gmail.com>
 <74a8da49-6966-62e1-d481-db350b69bed2@mrabarnett.plus.com>
 <si4vf7$12b9$1@ciao.gmane.io>
 <0638d578-f322-0b59-3af5-3d0927c1eeb2@mrabarnett.plus.com>
 <si7jla$136h$1@ciao.gmane.io>
Message-ID: <CABbU2U_ssCABSkG9hRSBOU4gQaCkmmjy=UsqK9YgQ45VdHd7dA@mail.gmail.com>

It works. Thanks a lot.

On Sun, 19 Sept 2021 at 19:23, Serhiy Storchaka <storchaka at gmail.com> wrote:
>
> 19.09.21 05:59, MRAB ????:
> > On 2021-09-18 16:09, Serhiy Storchaka wrote:
> >> "(PyCFunction)" is redundant, Py_GenericAlias already has the right
> >> type. Overuse of casting to PyCFunction can hide actual bugs.
> >>
> > I borrowed that from listobject.c, which does have the cast.
>
> Fixed. https://github.com/python/cpython/pull/28450
>
> --
> https://mail.python.org/mailman/listinfo/python-list

From drsalists at gmail.com  Thu Nov 18 19:24:28 2021
From: drsalists at gmail.com (Dan Stromberg)
Date: Thu, 18 Nov 2021 16:24:28 -0800
Subject: import question
In-Reply-To: <CAPTjJmq0K+9W9AvHVJfpOiZCw0SO8jN_ga1sTO1UgbgQ5FTdaA@mail.gmail.com>
References: <f8f5d859-df62-4923-adfd-a2017d3024e6n@googlegroups.com>
 <CAPTjJmq0K+9W9AvHVJfpOiZCw0SO8jN_ga1sTO1UgbgQ5FTdaA@mail.gmail.com>
Message-ID: <CAGGBd_qUiSAnXV=b2j20Mu-aLMa0kdc3v-Jr3hiM1v+qzOFRUQ@mail.gmail.com>

On Thu, Nov 18, 2021 at 12:21 PM Chris Angelico <rosuav at gmail.com> wrote:

> If you're trying to make a Python-in-Python sandbox, I recommend not.
> Instead, use an OS-level sandbox (a chroot, probably some sort of CPU
> usage limiting, etc), and use that to guard the entire Python process.
> Python-in-Python will basically *never* be secure.
>

Good advice to not try to sandbox python.

But chroot can sometimes be broken out of.  It isn't a cure-all.

From grant.b.edwards at gmail.com  Thu Nov 18 20:59:29 2021
From: grant.b.edwards at gmail.com (Grant Edwards)
Date: Thu, 18 Nov 2021 17:59:29 -0800 (PST)
Subject: import question
References: <f8f5d859-df62-4923-adfd-a2017d3024e6n@googlegroups.com>
Message-ID: <61970501.1c69fb81.780e1.5bcd@mx.google.com>

On 2021-11-17, lucas <sjlukacs at gmail.com> wrote:

> are there any other ways to import a module or package other then
> the "import" or "from...import..." statements?  i ask because i'm
> allowing programming on my web2py website and i don't want any
> accessing packages like os or sys.

Safely allowing people to enter/upload and then execute Python code is
very difficult. From my brief research into that question a little
while (a year or two) ago, the answer was that can't really be done in
any general way. IIRC, some promising work was done in PyPy to address
this problem, but the sandbox stuff never got moved to Py3?

--
Grant

From eryksun at gmail.com  Thu Nov 18 21:12:04 2021
From: eryksun at gmail.com (Eryk Sun)
Date: Thu, 18 Nov 2021 20:12:04 -0600
Subject: several issues with pyinstaller on Windows 10
In-Reply-To: <sn5gh5$47b$1@news2.informatik.uni-stuttgart.de>
References: <sn5gh5$47b$1@news2.informatik.uni-stuttgart.de>
Message-ID: <CACL+1auVSFeW5_Z+hTG7VVGOftHUysao8OOZ9XehZCjgWFT05w@mail.gmail.com>

On 11/18/21, Ulli Horlacher <framstag at rus.uni-stuttgart.de> wrote:
>
> P:\W10\dist>argv a b
> The process cannot access the file because it is being used by another
> process.

Try searching for open handles for "argv.exe" using Sysinternals
Process Explorer [1]. Terminate the offending process.

Since you're inexperienced with Windows, here's a brief explanation of
ERROR_SHARING_VIOLATION (32).

File objects that are opened for filesystem files have a
read-write-delete sharing mode if the open requests any
read-write-delete data access, but not if the open only has metadata
access (e.g. timestamps, ID, attributes, extended attributes,
security). Note that delete access includes the right to rename a
file.

The share mode flags and associated data access rights are as follows:

    FILE_SHARE_READ
        - FILE_READ_DATA | FILE_EXECUTE
    FILE_SHARE_WRITE
        - FILE_WRITE_DATA | FILE_APPEND_DATA
    FILE_SHARE_DELETE
        - DELETE

For example, if an existing open for a file has any data access and
doesn't share read access, then trying to open the file with execute
access will fail as a sharing violation. The security context of the
open request doesn't matter. For example, SYSTEM and administrators
aren't privileged to bypass the sharing mode. It can only be bypassed
from the kernel.

Unfortunately the system error message for ERROR_SHARING_VIOLATION is
misleading. The sharing mode has nothing to do with processes. It's
strictly a function of the File objects that are opened for the file.

Python's open() and os.open() functions share read and write access,
but they do not share delete access. For example, overlapping calls to
open('spam.txt', 'w') are allowed. For os.stat(), the share mode of
existing opens doesn't matter because it opens the file with metadata
access only.

Sharing data access can get messy. Each open has its own file pointer
in the associated OS file object. For example, say an open writes 10
bytes and flushes the buffer. Then a second open(..., 'w') call
overwrites the file, truncating it to 0 bytes. When the original open
writes to the file again, the OS will back fill the file with 10 null
bytes.

Windows also provides byte-range shared and exclusive locking that can
exceed the current size of the file. A byte-range lock doesn't prevent
opening the file with read, write, or delete access, and it doesn't
prevent deleting the file. It causes write or read-write operations on
the range to fail with ERROR_LOCK_VIOLATION (33).

> win32ctypes.pywin32.pywintypes.error: (110, 'EndUpdateResourceW',
> 'The system cannot open the device or file specified.')

This is likely due to a sharing violation. EndUpdateResourceW() [2]
requires exclusive access to the target file. If its internal open
fails for any reason, it maps all errors to this generic
ERROR_OPEN_FAILED (110) error code.

---
[1] https://docs.microsoft.com/en-us/sysinternals/downloads/process-explorer
[2] https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-endupdateresourcew

From rosuav at gmail.com  Thu Nov 18 21:18:19 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 19 Nov 2021 13:18:19 +1100
Subject: import question
In-Reply-To: <CAGGBd_qUiSAnXV=b2j20Mu-aLMa0kdc3v-Jr3hiM1v+qzOFRUQ@mail.gmail.com>
References: <f8f5d859-df62-4923-adfd-a2017d3024e6n@googlegroups.com>
 <CAPTjJmq0K+9W9AvHVJfpOiZCw0SO8jN_ga1sTO1UgbgQ5FTdaA@mail.gmail.com>
 <CAGGBd_qUiSAnXV=b2j20Mu-aLMa0kdc3v-Jr3hiM1v+qzOFRUQ@mail.gmail.com>
Message-ID: <CAPTjJmqd=YHuTPZxvXq_JfSY1wvW09TG8P9irQCHe1RCuj0R2g@mail.gmail.com>

On Fri, Nov 19, 2021 at 11:24 AM Dan Stromberg <drsalists at gmail.com> wrote:
>
>
> On Thu, Nov 18, 2021 at 12:21 PM Chris Angelico <rosuav at gmail.com> wrote:
>>
>> If you're trying to make a Python-in-Python sandbox, I recommend not.
>> Instead, use an OS-level sandbox (a chroot, probably some sort of CPU
>> usage limiting, etc), and use that to guard the entire Python process.
>> Python-in-Python will basically *never* be secure.
>
>
> Good advice to not try to sandbox python.
>
> But chroot can sometimes be broken out of.  It isn't a cure-all.
>

That's true, but it's way better than attempting Python-in-Python
sandboxing. In any case, all the options worth investigating will be
at the OS level.

(Or maybe higher, but I can't imagine it being practical to create
individual VMs for each client who comes to the web site.)

ChrisA

From rene.silva.valdes at gmail.com  Thu Nov 18 21:16:32 2021
From: rene.silva.valdes at gmail.com (=?UTF-8?Q?Ren=C3=A9_Silva_Vald=C3=A9s?=)
Date: Thu, 18 Nov 2021 23:16:32 -0300
Subject: Unexpected behaviour of math.floor, round and int functions (rounding)
Message-ID: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>

Hello, I would like to report the following issue:

Working with floats i noticed that:

int(23.99999999999999/12) returns 1, and
int(23.999999999999999/12) returns 2

This implies that int() function is rounding, which doesn't appear to be
expected (documentation doesn't say anything about it). Looking further i
noticed that

0.5+0.49999999999999994 returns 1. This seems to be related to double
numbers' operations in C language, where 0.49999999999999994 is the
greatest floating-point value less than 0.5. Counting on this several
examples can be deduced, like:

round(0+0.49999999999999994) returns 0, and
round(1+0.49999999999999994) returns 2

This seems to be a known issue in Java (see reference)


Reference:

https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6430675

I hope this information is helpful, thanks in advance for reading this,

Kind regards,

Ren?

From 2QdxY4RzWzUUiLuE at potatochowder.com  Thu Nov 18 21:40:31 2021
From: 2QdxY4RzWzUUiLuE at potatochowder.com (2QdxY4RzWzUUiLuE at potatochowder.com)
Date: Thu, 18 Nov 2021 20:40:31 -0600
Subject: Unexpected behaviour of math.floor, round and int functions
 (rounding)
In-Reply-To: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
Message-ID: <YZcOn5iN65pgLqrk@scrozzle>

On 2021-11-18 at 23:16:32 -0300,
Ren? Silva Vald?s <rene.silva.valdes at gmail.com> wrote:

> Hello, I would like to report the following issue:
> 
> Working with floats i noticed that:
> 
> int(23.99999999999999/12) returns 1, and
> int(23.999999999999999/12) returns 2
> 
> This implies that int() function is rounding ...

It's not int() that's doing the rounding; that second numerator is being
rounded before being divided by 12:

    Python 3.9.7 (default, Oct 10 2021, 15:13:22) 
    [GCC 11.1.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> 23.999999999999999
    24.0
    >>> (23.999999999999999).hex()
    '0x1.8000000000000p+4'


From python at mrabarnett.plus.com  Thu Nov 18 21:51:04 2021
From: python at mrabarnett.plus.com (MRAB)
Date: Fri, 19 Nov 2021 02:51:04 +0000
Subject: Unexpected behaviour of math.floor, round and int functions
 (rounding)
In-Reply-To: <YZcOn5iN65pgLqrk@scrozzle>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
Message-ID: <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>

On 2021-11-19 02:40, 2QdxY4RzWzUUiLuE at potatochowder.com wrote:
> On 2021-11-18 at 23:16:32 -0300,
> Ren? Silva Vald?s <rene.silva.valdes at gmail.com> wrote:
> 
>> Hello, I would like to report the following issue:
>> 
>> Working with floats i noticed that:
>> 
>> int(23.99999999999999/12) returns 1, and
>> int(23.999999999999999/12) returns 2
>> 
>> This implies that int() function is rounding ...
> 
> It's not int() that's doing the rounding; that second numerator is being
> rounded before being divided by 12:
> 
>      Python 3.9.7 (default, Oct 10 2021, 15:13:22)
>      [GCC 11.1.0] on linux
>      Type "help", "copyright", "credits" or "license" for more information.
>      >>> 23.999999999999999
>      24.0
>      >>> (23.999999999999999).hex()
>      '0x1.8000000000000p+4'
> 
I think this is a bit clearer because it shows that it's not just being 
rounded for display:

Python 3.10.0 (tags/v3.10.0:b494f59, Oct  4 2021, 19:00:18) [MSC v.1929 
64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
 >>> 23.99999999999999 == 24
False
 >>> 23.999999999999999 == 24
True

From drsalists at gmail.com  Thu Nov 18 23:00:10 2021
From: drsalists at gmail.com (Dan Stromberg)
Date: Thu, 18 Nov 2021 20:00:10 -0800
Subject: import question
In-Reply-To: <CAPTjJmqd=YHuTPZxvXq_JfSY1wvW09TG8P9irQCHe1RCuj0R2g@mail.gmail.com>
References: <f8f5d859-df62-4923-adfd-a2017d3024e6n@googlegroups.com>
 <CAPTjJmq0K+9W9AvHVJfpOiZCw0SO8jN_ga1sTO1UgbgQ5FTdaA@mail.gmail.com>
 <CAGGBd_qUiSAnXV=b2j20Mu-aLMa0kdc3v-Jr3hiM1v+qzOFRUQ@mail.gmail.com>
 <CAPTjJmqd=YHuTPZxvXq_JfSY1wvW09TG8P9irQCHe1RCuj0R2g@mail.gmail.com>
Message-ID: <CAGGBd_pENf3-25YyEZww-eOp9j78OKvk3QbORktoT8qTQpeL6A@mail.gmail.com>

On Thu, Nov 18, 2021 at 6:19 PM Chris Angelico <rosuav at gmail.com> wrote:

> On Fri, Nov 19, 2021 at 11:24 AM Dan Stromberg <drsalists at gmail.com>
> wrote:
> >
> >
> > On Thu, Nov 18, 2021 at 12:21 PM Chris Angelico <rosuav at gmail.com>
> wrote:
> >>
> >> If you're trying to make a Python-in-Python sandbox, I recommend not.
> >> Instead, use an OS-level sandbox (a chroot, probably some sort of CPU
> >> usage limiting, etc), and use that to guard the entire Python process.
> >> Python-in-Python will basically *never* be secure.
> >
> >
> > Good advice to not try to sandbox python.
> >
> > But chroot can sometimes be broken out of.  It isn't a cure-all.
> >
>
> That's true, but it's way better than attempting Python-in-Python
> sandboxing. In any case, all the options worth investigating will be
> at the OS level.
>
> (Or maybe higher, but I can't imagine it being practical to create
> individual VMs for each client who comes to the web site.)
>

Actually, there are ports of CPython and Micropython that run inside a web
browser over WASM.  Going with one of these might be safer.

From rosuav at gmail.com  Thu Nov 18 23:25:10 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 19 Nov 2021 15:25:10 +1100
Subject: import question
In-Reply-To: <CAGGBd_pENf3-25YyEZww-eOp9j78OKvk3QbORktoT8qTQpeL6A@mail.gmail.com>
References: <f8f5d859-df62-4923-adfd-a2017d3024e6n@googlegroups.com>
 <CAPTjJmq0K+9W9AvHVJfpOiZCw0SO8jN_ga1sTO1UgbgQ5FTdaA@mail.gmail.com>
 <CAGGBd_qUiSAnXV=b2j20Mu-aLMa0kdc3v-Jr3hiM1v+qzOFRUQ@mail.gmail.com>
 <CAPTjJmqd=YHuTPZxvXq_JfSY1wvW09TG8P9irQCHe1RCuj0R2g@mail.gmail.com>
 <CAGGBd_pENf3-25YyEZww-eOp9j78OKvk3QbORktoT8qTQpeL6A@mail.gmail.com>
Message-ID: <CAPTjJmrWhphjyQwRBGNTcD+8xHnrTRoBSvC2PSphB=t5N-gi_Q@mail.gmail.com>

On Fri, Nov 19, 2021 at 3:00 PM Dan Stromberg <drsalists at gmail.com> wrote:
>
>
> On Thu, Nov 18, 2021 at 6:19 PM Chris Angelico <rosuav at gmail.com> wrote:
>>
>> On Fri, Nov 19, 2021 at 11:24 AM Dan Stromberg <drsalists at gmail.com> wrote:
>> >
>> >
>> > On Thu, Nov 18, 2021 at 12:21 PM Chris Angelico <rosuav at gmail.com> wrote:
>> >>
>> >> If you're trying to make a Python-in-Python sandbox, I recommend not.
>> >> Instead, use an OS-level sandbox (a chroot, probably some sort of CPU
>> >> usage limiting, etc), and use that to guard the entire Python process.
>> >> Python-in-Python will basically *never* be secure.
>> >
>> >
>> > Good advice to not try to sandbox python.
>> >
>> > But chroot can sometimes be broken out of.  It isn't a cure-all.
>> >
>>
>> That's true, but it's way better than attempting Python-in-Python
>> sandboxing. In any case, all the options worth investigating will be
>> at the OS level.
>>
>> (Or maybe higher, but I can't imagine it being practical to create
>> individual VMs for each client who comes to the web site.)
>
>
> Actually, there are ports of CPython and Micropython that run inside a web browser over WASM.  Going with one of these might be safer.
>

Hmm, interesting point. I'd mentally ruled out the in-browser options
since the performance hit is usually far too costly, but if this is
basically an educational site, it MAY be sufficient (people won't need
spectacular performance when they're just learning the basics).

ChrisA

From mats at wichmann.us  Fri Nov 19 10:13:04 2021
From: mats at wichmann.us (Mats Wichmann)
Date: Fri, 19 Nov 2021 08:13:04 -0700
Subject: import question
In-Reply-To: <CAGGBd_pENf3-25YyEZww-eOp9j78OKvk3QbORktoT8qTQpeL6A@mail.gmail.com>
References: <f8f5d859-df62-4923-adfd-a2017d3024e6n@googlegroups.com>
 <CAPTjJmq0K+9W9AvHVJfpOiZCw0SO8jN_ga1sTO1UgbgQ5FTdaA@mail.gmail.com>
 <CAGGBd_qUiSAnXV=b2j20Mu-aLMa0kdc3v-Jr3hiM1v+qzOFRUQ@mail.gmail.com>
 <CAPTjJmqd=YHuTPZxvXq_JfSY1wvW09TG8P9irQCHe1RCuj0R2g@mail.gmail.com>
 <CAGGBd_pENf3-25YyEZww-eOp9j78OKvk3QbORktoT8qTQpeL6A@mail.gmail.com>
Message-ID: <85a62fc3-beeb-65c0-9a51-da7bb2612690@wichmann.us>

On 11/18/21 21:00, Dan Stromberg wrote:
> On Thu, Nov 18, 2021 at 6:19 PM Chris Angelico <rosuav at gmail.com> wrote:
> 
>> On Fri, Nov 19, 2021 at 11:24 AM Dan Stromberg <drsalists at gmail.com>
>> wrote:
>>>
>>>
>>> On Thu, Nov 18, 2021 at 12:21 PM Chris Angelico <rosuav at gmail.com>
>> wrote:
>>>>
>>>> If you're trying to make a Python-in-Python sandbox, I recommend not.
>>>> Instead, use an OS-level sandbox (a chroot, probably some sort of CPU
>>>> usage limiting, etc), and use that to guard the entire Python process.
>>>> Python-in-Python will basically *never* be secure.
>>>
>>>
>>> Good advice to not try to sandbox python.
>>>
>>> But chroot can sometimes be broken out of.  It isn't a cure-all.
>>>
>>
>> That's true, but it's way better than attempting Python-in-Python
>> sandboxing. In any case, all the options worth investigating will be
>> at the OS level.
>>
>> (Or maybe higher, but I can't imagine it being practical to create
>> individual VMs for each client who comes to the web site.)
>>
> 
> Actually, there are ports of CPython and Micropython that run inside a web
> browser over WASM.  Going with one of these might be safer.

indeed... see pyodide

https://github.com/pyodide/pyodide



From nt_mahmood at yahoo.com  Fri Nov 19 12:38:30 2021
From: nt_mahmood at yahoo.com (Mahmood Naderan)
Date: Fri, 19 Nov 2021 17:38:30 +0000 (UTC)
Subject: get_axes not present?
In-Reply-To: <afc3960c-c7cf-c09a-703b-57be45793db4@gmail.com>
References: <1263450234.1243976.1637228965678.ref@mail.yahoo.com>
 <1263450234.1243976.1637228965678@mail.yahoo.com>
 <500ef579-bd07-6a5c-b652-3b4dad8e4ba4@wichmann.us>
 <203139843.1328468.1637258083048@mail.yahoo.com>
 <afc3960c-c7cf-c09a-703b-57be45793db4@gmail.com>
Message-ID: <476561180.1556266.1637343510565@mail.yahoo.com>

>And what is the result of plot()?? Is it a valid object, or is it None?

Well the error happens on the plot() line. I tried to print some information like this:


??? print("axes=", axes)
??? print("axes[0]=", axes[0])
??? print("cnt=", cnt)
??? print("row=", row)
??? ax1 = row.plot( fontsize=font_size, linewidth=line_width, markersize=marker_size, marker='o', title='Raw values', label=cnt, ax=axes[0] )



The output looks like


axes= [<AxesSubplot:> <AxesSubplot:>]
axes[0]= AxesSubplot(0.125,0.53;0.775x0.35)
cnt= 1
row= 1?????? 278528
2?????? 278528
3?????? 278528
4?????? 278528
5?????? 278528
???????? ...
5604??? 278528
5605??? 278528
5606??? 278528
5607??? 278528
5608??? 278528
Name: 4, Length: 5608, dtype: int64
Traceback (most recent call last):
? File "process_csv.py", line 178, in <module>
??? plot_kernels( my_dict2 )
? File "process_csv.py", line 66, in plot_kernels
??? should_plot = plot_dataframe(df, cnt, axes)
? File "process_csv.py", line 38, in plot_dataframe
??? ax1 = row.plot( fontsize=font_size, linewidth=line_width, markersize=marker_size, marker='o', title='Raw values', label=cnt, ax=axes[0] )
? File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_core.py", line 955, in __call__
??? return plot_backend.plot(data, kind=kind, **kwargs)
? File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/__init__.py", line 61, in plot
??? plot_obj.generate()
? File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/core.py", line 283, in generate
??? self._adorn_subplots()
? File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/core.py", line 483, in _adorn_subplots
??? all_axes = self._get_subplots()
? File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/core.py", line 903, in _get_subplots
??? ax for ax in self.axes[0].get_figure().get_axes() if isinstance(ax, Subplot)
AttributeError: 'NoneType' object has no attribute 'get_axes'



The error is weird. I have stick at this error...
Any thoughts on that?


Regards,
Mahmood




From Marco.Sulla.Python at gmail.com  Fri Nov 19 12:48:39 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Fri, 19 Nov 2021 18:48:39 +0100
Subject: pytest segfault, not with -v
Message-ID: <CABbU2U_cTjL031iQ3S7BtAQM4Yc=rhRF6NzK=_C1kOoe+JwjPA@mail.gmail.com>

I have a battery of tests done with pytest. My tests break with a
segfault if I run them normally. If I run them using pytest -v, the
segfault does not happen.

What could cause this quantical phenomenon?

From ast at invalid  Fri Nov 19 06:43:07 2021
From: ast at invalid (ast)
Date: Fri, 19 Nov 2021 12:43:07 +0100
Subject: Unexpected behaviour of math.floor, round and int functions
 (rounding)
In-Reply-To: <mailman.45.1637290453.19479.python-list@python.org>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
Message-ID: <61978dcc$0$20254$426a34cc@news.free.fr>

Le 19/11/2021 ? 03:51, MRAB a ?crit?:
> On 2021-11-19 02:40, 2QdxY4RzWzUUiLuE at potatochowder.com wrote:
>> On 2021-11-18 at 23:16:32 -0300,
>> Ren? Silva Vald?s <rene.silva.valdes at gmail.com> wrote:
>>
>>> Hello, I would like to report the following issue:
>>>
>>> Working with floats i noticed that:
>>>
>>> int(23.99999999999999/12) returns 1, and
>>> int(23.999999999999999/12) returns 2
>>>
>>> This implies that int() function is rounding ...
>>
>> It's not int() that's doing the rounding; that second numerator is being
>> rounded before being divided by 12:
>>
>> ???? Python 3.9.7 (default, Oct 10 2021, 15:13:22)
>> ???? [GCC 11.1.0] on linux
>> ???? Type "help", "copyright", "credits" or "license" for more 
>> information.
>> ???? >>> 23.999999999999999
>> ???? 24.0
>> ???? >>> (23.999999999999999).hex()
>> ???? '0x1.8000000000000p+4'
>>
> I think this is a bit clearer because it shows that it's not just being 
> rounded for display:
> 
> Python 3.10.0 (tags/v3.10.0:b494f59, Oct? 4 2021, 19:00:18) [MSC v.1929 
> 64 bit (AMD64)] on win32
> Type "help", "copyright", "credits" or "license" for more information.
>  >>> 23.99999999999999 == 24
> False
>  >>> 23.999999999999999 == 24
> True

 >>> 0.3 + 0.3 + 0.3 == 0.9
False

From ast at invalid  Fri Nov 19 06:47:06 2021
From: ast at invalid (ast)
Date: Fri, 19 Nov 2021 12:47:06 +0100
Subject: Unexpected behaviour of math.floor, round and int functions
 (rounding)
In-Reply-To: <61978dcc$0$20254$426a34cc@news.free.fr>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
Message-ID: <61978ebb$0$8914$426a34cc@news.free.fr>

Le 19/11/2021 ? 12:43, ast a ?crit?:
> Le 19/11/2021 ? 03:51, MRAB a ?crit?:
>> On 2021-11-19 02:40, 2QdxY4RzWzUUiLuE at potatochowder.com wrote:
>>> On 2021-11-18 at 23:16:32 -0300,
>>> Ren? Silva Vald?s <rene.silva.valdes at gmail.com> wrote:
>>>
>>>> Hello, I would like to report the following issue:
>>>>
>>>> Working with floats i noticed that:
>>>>
>>>> int(23.99999999999999/12) returns 1, and
>>>> int(23.999999999999999/12) returns 2
>>>>
>>>> This implies that int() function is rounding ...
>>>
>>> It's not int() that's doing the rounding; that second numerator is being
>>> rounded before being divided by 12:
>>>
>>> ???? Python 3.9.7 (default, Oct 10 2021, 15:13:22)
>>> ???? [GCC 11.1.0] on linux
>>> ???? Type "help", "copyright", "credits" or "license" for more 
>>> information.
>>> ???? >>> 23.999999999999999
>>> ???? 24.0
>>> ???? >>> (23.999999999999999).hex()
>>> ???? '0x1.8000000000000p+4'
>>>
>> I think this is a bit clearer because it shows that it's not just 
>> being rounded for display:
>>
>> Python 3.10.0 (tags/v3.10.0:b494f59, Oct? 4 2021, 19:00:18) [MSC 
>> v.1929 64 bit (AMD64)] on win32
>> Type "help", "copyright", "credits" or "license" for more information.
>> ?>>> 23.99999999999999 == 24
>> False
>> ?>>> 23.999999999999999 == 24
>> True
> 
>  >>> 0.3 + 0.3 + 0.3 == 0.9
> False

Better use math.isclose to test equality between 2 floats

 >>> import math
 >>> math.isclose(0.3 + 0.3 + 0.3, 0.9)
True

From framstag at rus.uni-stuttgart.de  Fri Nov 19 07:09:26 2021
From: framstag at rus.uni-stuttgart.de (Ulli Horlacher)
Date: Fri, 19 Nov 2021 12:09:26 +0000 (UTC)
Subject: getting source code line of error?
Message-ID: <sn845l$pc1$1@news2.informatik.uni-stuttgart.de>

I am trying to get the source code line of the last error.
I know traceback.format_exc() but this contains much more information, e.g.:

Traceback (most recent call last):
  File "./error.py", line 18, in main
    x=1/0
ZeroDivisionError: division by zero

I could extract the source code line with re.search(), but is there an
easier way?


I have:

      exc_type,exc_str,exc_tb = sys.exc_info()
      fname = exc_tb.tb_frame.f_code.co_filename
      line = exc_tb.tb_lineno
      print('%s in %s line %d' % (exc_str,fname,line))

But I also want to output the line itself, not only its number.

-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher at tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/

From sjlukacs at gmail.com  Fri Nov 19 09:38:39 2021
From: sjlukacs at gmail.com (lucas)
Date: Fri, 19 Nov 2021 06:38:39 -0800 (PST)
Subject: import question
In-Reply-To: <mailman.47.1637295922.19479.python-list@python.org>
References: <f8f5d859-df62-4923-adfd-a2017d3024e6n@googlegroups.com>
 <CAPTjJmq0K+9W9AvHVJfpOiZCw0SO8jN_ga1sTO1UgbgQ5FTdaA@mail.gmail.com>
 <CAGGBd_qUiSAnXV=b2j20Mu-aLMa0kdc3v-Jr3hiM1v+qzOFRUQ@mail.gmail.com>
 <CAPTjJmqd=YHuTPZxvXq_JfSY1wvW09TG8P9irQCHe1RCuj0R2g@mail.gmail.com>
 <CAPTjJmrWhphjyQwRBGNTcD+8xHnrTRoBSvC2PSphB=t5N-gi_Q@mail.gmail.com>
 <CAGGBd_pENf3-25YyEZww-eOp9j78OKvk3QbORktoT8qTQpeL6A@mail.gmail.com>
 <mailman.47.1637295922.19479.python-list@python.org>
Message-ID: <12a111ae-04e4-4a24-bd95-62044a68942fn@googlegroups.com>

ok.  all good advice.  thank you for that.  and with all that I've decided what to do.

I'm going to close off any server-side python access so that I don't expose my server or the file system to vulnerabilities and/or wonton attacks.  I am building a site for education and what I will configure is allow students to setup and save their projects on the server but only direct them to program in client-side Brython, which is a javascript translation of python for browsers, hence "Brython" or "browser python".  my server will provide the javascript files for Brython and its standard libraries and any processing of the student's projects will be directly on the client-side.  this way there is no access to the server or cpu or memory management problems.  the server will simply server html and Brython-based text, i.e., static pages, to the client browser and the browser will process and interact with the Brython directly.  

overall, the server will stay secure and the students can learn python through Brython.  sound, right?  Lucas

From Marco.Sulla.Python at gmail.com  Fri Nov 19 14:13:52 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Fri, 19 Nov 2021 20:13:52 +0100
Subject: getting source code line of error?
In-Reply-To: <sn845l$pc1$1@news2.informatik.uni-stuttgart.de>
References: <sn845l$pc1$1@news2.informatik.uni-stuttgart.de>
Message-ID: <CABbU2U8xXY29rA3GBCPOVkJak7M3Tm0QK5FdDerCSvxeO=oX2w@mail.gmail.com>

Have you tried the logger module and the format options?

On Fri, 19 Nov 2021 at 19:09, Ulli Horlacher
<framstag at rus.uni-stuttgart.de> wrote:
>
> I am trying to get the source code line of the last error.
> I know traceback.format_exc() but this contains much more information, e.g.:
>
> Traceback (most recent call last):
>   File "./error.py", line 18, in main
>     x=1/0
> ZeroDivisionError: division by zero
>
> I could extract the source code line with re.search(), but is there an
> easier way?
>
>
> I have:
>
>       exc_type,exc_str,exc_tb = sys.exc_info()
>       fname = exc_tb.tb_frame.f_code.co_filename
>       line = exc_tb.tb_lineno
>       print('%s in %s line %d' % (exc_str,fname,line))
>
> But I also want to output the line itself, not only its number.
>
> --
> Ullrich Horlacher              Server und Virtualisierung
> Rechenzentrum TIK
> Universitaet Stuttgart         E-Mail: horlacher at tik.uni-stuttgart.de
> Allmandring 30a                Tel:    ++49-711-68565868
> 70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
> --
> https://mail.python.org/mailman/listinfo/python-list

From python at mrabarnett.plus.com  Fri Nov 19 14:33:07 2021
From: python at mrabarnett.plus.com (MRAB)
Date: Fri, 19 Nov 2021 19:33:07 +0000
Subject: pytest segfault, not with -v
In-Reply-To: <CABbU2U_cTjL031iQ3S7BtAQM4Yc=rhRF6NzK=_C1kOoe+JwjPA@mail.gmail.com>
References: <CABbU2U_cTjL031iQ3S7BtAQM4Yc=rhRF6NzK=_C1kOoe+JwjPA@mail.gmail.com>
Message-ID: <255c2cb8-b5ea-6e32-d6eb-1765af3967d5@mrabarnett.plus.com>

On 2021-11-19 17:48, Marco Sulla wrote:
> I have a battery of tests done with pytest. My tests break with a
> segfault if I run them normally. If I run them using pytest -v, the
> segfault does not happen.
> 
> What could cause this quantical phenomenon?
> 
Are you testing an extension that you're compiling? That kind of problem 
can occur if there's an uninitialised variable or incorrect reference 
counting (Py_INCREF/Py_DECREF).

From PythonList at DancesWithMice.info  Fri Nov 19 14:38:12 2021
From: PythonList at DancesWithMice.info (dn)
Date: Sat, 20 Nov 2021 08:38:12 +1300
Subject: import question
In-Reply-To: <12a111ae-04e4-4a24-bd95-62044a68942fn@googlegroups.com>
References: <f8f5d859-df62-4923-adfd-a2017d3024e6n@googlegroups.com>
 <CAPTjJmq0K+9W9AvHVJfpOiZCw0SO8jN_ga1sTO1UgbgQ5FTdaA@mail.gmail.com>
 <CAGGBd_qUiSAnXV=b2j20Mu-aLMa0kdc3v-Jr3hiM1v+qzOFRUQ@mail.gmail.com>
 <CAPTjJmqd=YHuTPZxvXq_JfSY1wvW09TG8P9irQCHe1RCuj0R2g@mail.gmail.com>
 <CAPTjJmrWhphjyQwRBGNTcD+8xHnrTRoBSvC2PSphB=t5N-gi_Q@mail.gmail.com>
 <CAGGBd_pENf3-25YyEZww-eOp9j78OKvk3QbORktoT8qTQpeL6A@mail.gmail.com>
 <mailman.47.1637295922.19479.python-list@python.org>
 <12a111ae-04e4-4a24-bd95-62044a68942fn@googlegroups.com>
Message-ID: <d19efdbf-bf5c-cb85-24ef-b16df1e9e998@DancesWithMice.info>

On 20/11/2021 03.38, lucas wrote:
> ok.  all good advice.  thank you for that.  and with all that I've decided what to do.
> 
> I'm going to close off any server-side python access so that I don't expose my server or the file system to vulnerabilities and/or wonton attacks.  I am building a site for education and what I will configure is allow students to setup and save their projects on the server but only direct them to program in client-side Brython, which is a javascript translation of python for browsers, hence "Brython" or "browser python".  my server will provide the javascript files for Brython and its standard libraries and any processing of the student's projects will be directly on the client-side.  this way there is no access to the server or cpu or memory management problems.  the server will simply server html and Brython-based text, i.e., static pages, to the client browser and the browser will process and interact with the Brython directly.  
> 
> overall, the server will stay secure and the students can learn python through Brython.  sound, right?  Lucas


Alternately, 'stand on the shoulders of giants' and consider
https://pythontutor.com/visualize.html#mode=edit

This has the additional value for your trainees of showing a visual
execution of their code. They can see step-by-step how Python/the
computer interprets their (?perfect) instruction and exactly where
things fall-over - with-out the added complication/cognitive-load of
having to master a debugger!

If you're still determined to invest a lot of time, it looks as if Phil
has invested a lot of time, more recently, in widening the range of
languages, (which is perhaps why(?)) the system has fallen behind in
Python release.
-- 
Regards,
=dn

From mats at wichmann.us  Fri Nov 19 14:52:12 2021
From: mats at wichmann.us (Mats Wichmann)
Date: Fri, 19 Nov 2021 12:52:12 -0700
Subject: Unexpected behaviour of math.floor, round and int functions
 (rounding)
In-Reply-To: <YZcOn5iN65pgLqrk@scrozzle>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
Message-ID: <1d20a6d2-757f-4565-fc51-6984fa9958ef@wichmann.us>

On 11/18/21 19:40, 2QdxY4RzWzUUiLuE at potatochowder.com wrote:
> On 2021-11-18 at 23:16:32 -0300,
> Ren? Silva Vald?s <rene.silva.valdes at gmail.com> wrote:
> 
>> Hello, I would like to report the following issue:
>>
>> Working with floats i noticed that:
>>
>> int(23.99999999999999/12) returns 1, and
>> int(23.999999999999999/12) returns 2
>>
>> This implies that int() function is rounding ...
> 
> It's not int() that's doing the rounding; that second numerator is being
> rounded before being divided by 12:
> 
>      Python 3.9.7 (default, Oct 10 2021, 15:13:22)
>      [GCC 11.1.0] on linux
>      Type "help", "copyright", "credits" or "license" for more information.
>      >>> 23.999999999999999
>      24.0
>      >>> (23.999999999999999).hex()
>      '0x1.8000000000000p+4'
> 

The documentation has a fair bit to say on the subject of floating 
point.  I never remember the precise link, but fortunately it's pretty 
easy to search for:

https://docs.python.org/3/tutorial/floatingpoint.html


From rosuav at gmail.com  Fri Nov 19 15:17:53 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 20 Nov 2021 07:17:53 +1100
Subject: Unexpected behaviour of math.floor,
 round and int functions (rounding)
In-Reply-To: <61978dcc$0$20254$426a34cc@news.free.fr>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
Message-ID: <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>

On Sat, Nov 20, 2021 at 5:08 AM ast <ast at invalid> wrote:
>
> Le 19/11/2021 ? 03:51, MRAB a ?crit :
> > On 2021-11-19 02:40, 2QdxY4RzWzUUiLuE at potatochowder.com wrote:
> >> On 2021-11-18 at 23:16:32 -0300,
> >> Ren? Silva Vald?s <rene.silva.valdes at gmail.com> wrote:
> >>
> >>> Hello, I would like to report the following issue:
> >>>
> >>> Working with floats i noticed that:
> >>>
> >>> int(23.99999999999999/12) returns 1, and
> >>> int(23.999999999999999/12) returns 2
> >>>
> >>> This implies that int() function is rounding ...
> >>
> >> It's not int() that's doing the rounding; that second numerator is being
> >> rounded before being divided by 12:
> >>
> >>      Python 3.9.7 (default, Oct 10 2021, 15:13:22)
> >>      [GCC 11.1.0] on linux
> >>      Type "help", "copyright", "credits" or "license" for more
> >> information.
> >>      >>> 23.999999999999999
> >>      24.0
> >>      >>> (23.999999999999999).hex()
> >>      '0x1.8000000000000p+4'
> >>
> > I think this is a bit clearer because it shows that it's not just being
> > rounded for display:
> >
> > Python 3.10.0 (tags/v3.10.0:b494f59, Oct  4 2021, 19:00:18) [MSC v.1929
> > 64 bit (AMD64)] on win32
> > Type "help", "copyright", "credits" or "license" for more information.
> >  >>> 23.99999999999999 == 24
> > False
> >  >>> 23.999999999999999 == 24
> > True
>
>  >>> 0.3 + 0.3 + 0.3 == 0.9
> False

That's because 0.3 is not 3/10. It's not because floats are
"unreliable" or "inaccurate". It's because the ones you're entering
are not what you think they are.

When will people understand this?

(Probably never. Sigh.)

ChrisA

From PythonList at DancesWithMice.info  Fri Nov 19 15:38:01 2021
From: PythonList at DancesWithMice.info (dn)
Date: Sat, 20 Nov 2021 09:38:01 +1300
Subject: Unexpected behaviour of math.floor, round and int functions
 (rounding)
In-Reply-To: <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
Message-ID: <7cf55d6a-e704-004a-c5ed-afe2bdc9a34b@DancesWithMice.info>

On 20/11/2021 09.17, Chris Angelico wrote:
> On Sat, Nov 20, 2021 at 5:08 AM ast <ast at invalid> wrote:
>> Le 19/11/2021 ? 03:51, MRAB a ?crit :
>>> On 2021-11-19 02:40, 2QdxY4RzWzUUiLuE at potatochowder.com wrote:
>>>> On 2021-11-18 at 23:16:32 -0300,
>>>> Ren? Silva Vald?s <rene.silva.valdes at gmail.com> wrote:
>>>>> Working with floats i noticed that:
>>>>> int(23.99999999999999/12) returns 1, and
>>>>> int(23.999999999999999/12) returns 2


Has the OP (now) realised that the observation is a "feature" not a
"bug"? It is one of the difficulties of representing small numbers or
numerical-components in binary - there are many decimal values which
cannot be accurately-expressed in binary - exactly as noted.

...
>>  >>> 0.3 + 0.3 + 0.3 == 0.9
>> False
> 
> That's because 0.3 is not 3/10. It's not because floats are
> "unreliable" or "inaccurate". It's because the ones you're entering
> are not what you think they are.
> 
> When will people understand this?
> (Probably never. Sigh.)


Am not aware of any institution which teaches the inner-workings of a
CPU/ALU/FPU/GPU in a general programming class, ie "Programming" and
particularly "Coding", have diverged from "Computer Science" - in at
least this respect.

As well as the approximations involved in trying to maintain
decimal-numbers (floats/floating-point numbers), and particularly values
to the 'right' of a decimal-point; we had to study?suffer classes in
"Numerical Analysis" and be able to gauge the declining accuracy and
precision of sundry calculations. A skill disappearing as fast as
slide-rules!?

This 'pool of ignorance' is particularly noticeable in folk who have
come 'up' through the 'CodeCamp'/'BootCamp' approach to training. On the
other hand, if one is not intending to 'get into' a scientific or highly
mathematical branch of computing/Python, eg commercial applications
using (only) Decimal (or int), the average web-app, and similar; why
bother?
-- 
Regards,
=dn

From Marco.Sulla.Python at gmail.com  Fri Nov 19 16:11:22 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Fri, 19 Nov 2021 22:11:22 +0100
Subject: frozenset can be altered by |=
Message-ID: <CABbU2U_79XPx1T6EGWEniYkN8_m0oOO6mxx+HNbaLqg_wmT-Gw@mail.gmail.com>

(venv_3_10) marco at buzz:~$ python
Python 3.10.0 (heads/3.10-dirty:f6e8b80d20, Nov 18 2021, 19:16:18)
[GCC 10.1.1 20200718] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> a = frozenset((3, 4))
>>> a
frozenset({3, 4})
>>> a |= {5,}
>>> a
frozenset({3, 4, 5})

From rosuav at gmail.com  Fri Nov 19 16:16:31 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 20 Nov 2021 08:16:31 +1100
Subject: frozenset can be altered by |=
In-Reply-To: <CABbU2U_79XPx1T6EGWEniYkN8_m0oOO6mxx+HNbaLqg_wmT-Gw@mail.gmail.com>
References: <CABbU2U_79XPx1T6EGWEniYkN8_m0oOO6mxx+HNbaLqg_wmT-Gw@mail.gmail.com>
Message-ID: <CAPTjJmq=4+yEhibTJoNA3QVUNR1HmScgnQgx3NtQRpgG4aNY1Q@mail.gmail.com>

On Sat, Nov 20, 2021 at 8:13 AM Marco Sulla
<Marco.Sulla.Python at gmail.com> wrote:
>
> (venv_3_10) marco at buzz:~$ python
> Python 3.10.0 (heads/3.10-dirty:f6e8b80d20, Nov 18 2021, 19:16:18)
> [GCC 10.1.1 20200718] on linux
> Type "help", "copyright", "credits" or "license" for more information.
> >>> a = frozenset((3, 4))
> >>> a
> frozenset({3, 4})
> >>> a |= {5,}
> >>> a
> frozenset({3, 4, 5})

That's the same as how "x = 4; x += 1" can "alter" four into five.

>>> a = frozenset((3, 4))
>>> id(a), a
(140545764976096, frozenset({3, 4}))
>>> a |= {5,}
>>> id(a), a
(140545763014944, frozenset({3, 4, 5}))

It's a different frozenset.

ChrisA

From rosuav at gmail.com  Fri Nov 19 16:17:22 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 20 Nov 2021 08:17:22 +1100
Subject: frozenset can be altered by |=
In-Reply-To: <CAPTjJmq=4+yEhibTJoNA3QVUNR1HmScgnQgx3NtQRpgG4aNY1Q@mail.gmail.com>
References: <CABbU2U_79XPx1T6EGWEniYkN8_m0oOO6mxx+HNbaLqg_wmT-Gw@mail.gmail.com>
 <CAPTjJmq=4+yEhibTJoNA3QVUNR1HmScgnQgx3NtQRpgG4aNY1Q@mail.gmail.com>
Message-ID: <CAPTjJmo6wAjeAV=T88Zk2mto2J3GDmdh4ST+i3MMvpEFHuXFCQ@mail.gmail.com>

On Sat, Nov 20, 2021 at 8:16 AM Chris Angelico <rosuav at gmail.com> wrote:
>
> On Sat, Nov 20, 2021 at 8:13 AM Marco Sulla
> <Marco.Sulla.Python at gmail.com> wrote:
> >
> > (venv_3_10) marco at buzz:~$ python
> > Python 3.10.0 (heads/3.10-dirty:f6e8b80d20, Nov 18 2021, 19:16:18)
> > [GCC 10.1.1 20200718] on linux
> > Type "help", "copyright", "credits" or "license" for more information.
> > >>> a = frozenset((3, 4))
> > >>> a
> > frozenset({3, 4})
> > >>> a |= {5,}
> > >>> a
> > frozenset({3, 4, 5})
>
> That's the same as how "x = 4; x += 1" can "alter" four into five.
>
> >>> a = frozenset((3, 4))
> >>> id(a), a
> (140545764976096, frozenset({3, 4}))
> >>> a |= {5,}
> >>> id(a), a
> (140545763014944, frozenset({3, 4, 5}))
>
> It's a different frozenset.
>

Oh, even better test:

>>> a = frozenset((3, 4)); b = a
>>> id(a), a, id(b), b
(140602825123296, frozenset({3, 4}), 140602825123296, frozenset({3, 4}))
>>> a |= {5,}
>>> id(a), a, id(b), b
(140602825254144, frozenset({3, 4, 5}), 140602825123296, frozenset({3, 4}))

ChrisA

From rosuav at gmail.com  Fri Nov 19 16:21:18 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 20 Nov 2021 08:21:18 +1100
Subject: Unexpected behaviour of math.floor,
 round and int functions (rounding)
In-Reply-To: <7cf55d6a-e704-004a-c5ed-afe2bdc9a34b@DancesWithMice.info>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <7cf55d6a-e704-004a-c5ed-afe2bdc9a34b@DancesWithMice.info>
Message-ID: <CAPTjJmoFwaa6xcWuHdxPcwBYe_rf85AX7ESfOT8Km2cK1m8y0g@mail.gmail.com>

On Sat, Nov 20, 2021 at 7:39 AM dn via Python-list
<python-list at python.org> wrote:
> >>  >>> 0.3 + 0.3 + 0.3 == 0.9
> >> False
> >
> > That's because 0.3 is not 3/10. It's not because floats are
> > "unreliable" or "inaccurate". It's because the ones you're entering
> > are not what you think they are.
> >
> > When will people understand this?
> > (Probably never. Sigh.)
>
>
> Am not aware of any institution which teaches the inner-workings of a
> CPU/ALU/FPU/GPU in a general programming class, ie "Programming" and
> particularly "Coding", have diverged from "Computer Science" - in at
> least this respect.
>

I think what I find annoying about this sort of thing is that people
triumphantly announce that the computer is WRONG. It's the numeric
equivalent of XKCD 169, and people get smug for the exact same reason,
and perhaps unfortunately, do not get their arms cut off.

"Computer, give me a number as close as possible to three tenths."

"HAH! THAT ISN'T THREE TENTHS! Hah you suck!"

*slash*

ChrisA

From Marco.Sulla.Python at gmail.com  Fri Nov 19 16:37:30 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Fri, 19 Nov 2021 22:37:30 +0100
Subject: frozenset can be altered by |=
In-Reply-To: <CAPTjJmo6wAjeAV=T88Zk2mto2J3GDmdh4ST+i3MMvpEFHuXFCQ@mail.gmail.com>
References: <CABbU2U_79XPx1T6EGWEniYkN8_m0oOO6mxx+HNbaLqg_wmT-Gw@mail.gmail.com>
 <CAPTjJmq=4+yEhibTJoNA3QVUNR1HmScgnQgx3NtQRpgG4aNY1Q@mail.gmail.com>
 <CAPTjJmo6wAjeAV=T88Zk2mto2J3GDmdh4ST+i3MMvpEFHuXFCQ@mail.gmail.com>
Message-ID: <CABbU2U_3RQXqEwTmOwOMwANjgDZ19wwYMYNCw_SVziD0-inhvA@mail.gmail.com>

Mh. Now I'm thinking that I've done

a = "Marco "
a += "Sulla"

many times without bothering.

On Fri, 19 Nov 2021 at 22:22, Chris Angelico <rosuav at gmail.com> wrote:
>
> On Sat, Nov 20, 2021 at 8:16 AM Chris Angelico <rosuav at gmail.com> wrote:
> >
> > On Sat, Nov 20, 2021 at 8:13 AM Marco Sulla
> > <Marco.Sulla.Python at gmail.com> wrote:
> > >
> > > (venv_3_10) marco at buzz:~$ python
> > > Python 3.10.0 (heads/3.10-dirty:f6e8b80d20, Nov 18 2021, 19:16:18)
> > > [GCC 10.1.1 20200718] on linux
> > > Type "help", "copyright", "credits" or "license" for more information.
> > > >>> a = frozenset((3, 4))
> > > >>> a
> > > frozenset({3, 4})
> > > >>> a |= {5,}
> > > >>> a
> > > frozenset({3, 4, 5})
> >
> > That's the same as how "x = 4; x += 1" can "alter" four into five.
> >
> > >>> a = frozenset((3, 4))
> > >>> id(a), a
> > (140545764976096, frozenset({3, 4}))
> > >>> a |= {5,}
> > >>> id(a), a
> > (140545763014944, frozenset({3, 4, 5}))
> >
> > It's a different frozenset.
> >
>
> Oh, even better test:
>
> >>> a = frozenset((3, 4)); b = a
> >>> id(a), a, id(b), b
> (140602825123296, frozenset({3, 4}), 140602825123296, frozenset({3, 4}))
> >>> a |= {5,}
> >>> id(a), a, id(b), b
> (140602825254144, frozenset({3, 4, 5}), 140602825123296, frozenset({3, 4}))
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list

From python at mrabarnett.plus.com  Fri Nov 19 16:50:24 2021
From: python at mrabarnett.plus.com (MRAB)
Date: Fri, 19 Nov 2021 21:50:24 +0000
Subject: frozenset can be altered by |=
In-Reply-To: <CABbU2U_79XPx1T6EGWEniYkN8_m0oOO6mxx+HNbaLqg_wmT-Gw@mail.gmail.com>
References: <CABbU2U_79XPx1T6EGWEniYkN8_m0oOO6mxx+HNbaLqg_wmT-Gw@mail.gmail.com>
Message-ID: <b1f75dad-8a59-0af1-71b5-031d86b96477@mrabarnett.plus.com>

On 2021-11-19 21:11, Marco Sulla wrote:
> (venv_3_10) marco at buzz:~$ python
> Python 3.10.0 (heads/3.10-dirty:f6e8b80d20, Nov 18 2021, 19:16:18)
> [GCC 10.1.1 20200718] on linux
> Type "help", "copyright", "credits" or "license" for more information.
>>>> a = frozenset((3, 4))
>>>> a
> frozenset({3, 4})
>>>> a |= {5,}
>>>> a
> frozenset({3, 4, 5})
> 
I'll counter with:

Python 3.10.0 (tags/v3.10.0:b494f59, Oct  4 2021, 19:00:18) [MSC v.1929 
64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
 >>> a = frozenset((3, 4))
 >>> a
frozenset({3, 4})
 >>> b = a
 >>> a |= {5,}
 >>> a
frozenset({3, 4, 5})
 >>> b
frozenset({3, 4})


frozenset doesn't support mutation, so:

     a |= {5,}

falls back to:

     a = a | {5,}


The same kind of thing happens with tuples:

 >>> a = (3, 4)
 >>> a
(3, 4)
 >>> a += (5,)
 >>> a
(3, 4, 5)

From ben.usenet at bsb.me.uk  Fri Nov 19 16:55:27 2021
From: ben.usenet at bsb.me.uk (Ben Bacarisse)
Date: Fri, 19 Nov 2021 21:55:27 +0000
Subject: Unexpected behaviour of math.floor,
 round and int functions (rounding)
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
Message-ID: <87fsrrkdyo.fsf@bsb.me.uk>

Chris Angelico <rosuav at gmail.com> writes:

> On Sat, Nov 20, 2021 at 5:08 AM ast <ast at invalid> wrote:

>>  >>> 0.3 + 0.3 + 0.3 == 0.9
>> False
>
> That's because 0.3 is not 3/10. It's not because floats are
> "unreliable" or "inaccurate". It's because the ones you're entering
> are not what you think they are.
>
> When will people understand this?
>
> (Probably never. Sigh.)

Most people understand what's going on when it's explained to them.  And
I think that being initially baffled is not unreasonable.  After all,
almost everyone comes to computers after learning that 3/10 can be
written as 0.3.  And Python "plays along" with the fiction to some
extent.  0.3 prints as 0.3, 3/10 prints as 0.3 and 0.3 == 3/10 is True.

The language (a language, not Python) could tell you that you were not
getting the value you asked for.  Every 0.3 could come with a warning
that 0.3 can not be represented exactly as a floating point value.  To
avoid the warning, the programmer would write ~0.3 meaning, exactly, the
binary (or whatever the base really is) floating point number closest to
0.3.

-- 
Ben.

From rosuav at gmail.com  Fri Nov 19 17:15:38 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 20 Nov 2021 09:15:38 +1100
Subject: Unexpected behaviour of math.floor,
 round and int functions (rounding)
In-Reply-To: <87fsrrkdyo.fsf@bsb.me.uk>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
Message-ID: <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>

On Sat, Nov 20, 2021 at 9:07 AM Ben Bacarisse <ben.usenet at bsb.me.uk> wrote:
>
> Chris Angelico <rosuav at gmail.com> writes:
>
> > On Sat, Nov 20, 2021 at 5:08 AM ast <ast at invalid> wrote:
>
> >>  >>> 0.3 + 0.3 + 0.3 == 0.9
> >> False
> >
> > That's because 0.3 is not 3/10. It's not because floats are
> > "unreliable" or "inaccurate". It's because the ones you're entering
> > are not what you think they are.
> >
> > When will people understand this?
> >
> > (Probably never. Sigh.)
>
> Most people understand what's going on when it's explained to them.  And
> I think that being initially baffled is not unreasonable.  After all,
> almost everyone comes to computers after learning that 3/10 can be
> written as 0.3.  And Python "plays along" with the fiction to some
> extent.  0.3 prints as 0.3, 3/10 prints as 0.3 and 0.3 == 3/10 is True.

In grade school, we learn that not everything can be written that way,
and 1/3 isn't actually equal to 0.3333333333. Yet somehow people
understand that computers speak binary ("have you learned to count
yet, or are you still on zeroes and ones?" -- insult to a machine
empire, in Stellaris), but don't seem to appreciate that floats are
absolutely accurate and reliable, just in binary.

But lack of knowledge is never a problem. (Or rather, it's a solvable
problem, and I'm always happy to explain things to people.) The
problem is when, following that lack of understanding, people assume
that floats are "unreliable" or "inaccurate", and that you should
never ever compare two floats for equality, because they're never
"really equal". That's what leads to horrible coding practices and
badly-defined "approximately equal" checks that cause far more harm
than a simple misunderstanding ever could on its own.

ChrisA

From PythonList at DancesWithMice.info  Fri Nov 19 17:21:27 2021
From: PythonList at DancesWithMice.info (dn)
Date: Sat, 20 Nov 2021 11:21:27 +1300
Subject: Unexpected behaviour of math.floor, round and int functions
 (rounding)
In-Reply-To: <CAPTjJmoFwaa6xcWuHdxPcwBYe_rf85AX7ESfOT8Km2cK1m8y0g@mail.gmail.com>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <7cf55d6a-e704-004a-c5ed-afe2bdc9a34b@DancesWithMice.info>
 <CAPTjJmoFwaa6xcWuHdxPcwBYe_rf85AX7ESfOT8Km2cK1m8y0g@mail.gmail.com>
Message-ID: <4577d67e-3ba7-ae2a-61a8-29f9f513e22d@DancesWithMice.info>

On 20/11/2021 10.21, Chris Angelico wrote:
> On Sat, Nov 20, 2021 at 7:39 AM dn via Python-list
> <python-list at python.org> wrote:
>>>>  >>> 0.3 + 0.3 + 0.3 == 0.9
>>>> False
>>>
>>> That's because 0.3 is not 3/10. It's not because floats are
>>> "unreliable" or "inaccurate". It's because the ones you're entering
>>> are not what you think they are.
>>>
>>> When will people understand this?
>>> (Probably never. Sigh.)
>>
>>
>> Am not aware of any institution which teaches the inner-workings of a
>> CPU/ALU/FPU/GPU in a general programming class, ie "Programming" and
>> particularly "Coding", have diverged from "Computer Science" - in at
>> least this respect.
>>
> 
> I think what I find annoying about this sort of thing is that people
> triumphantly announce that the computer is WRONG. It's the numeric
> equivalent of XKCD 169, and people get smug for the exact same reason,
> and perhaps unfortunately, do not get their arms cut off.


I guess I'm a little more familiar with the 'arrogant' side of this
phenomenon because in formal courses, eg uni, I'd predict at least one
such personality per group of 50~60 students!

However, cultural-components should be considered. There is no
requirement that the OP possess an advanced command of the English
language! (nor of Python, nor of computers/computing, ...) - see OP's name!


That said, it can be frustrating for those of us who see particular
problems over-and-over. If it were evidenced in a course, I would be
addressing a short-coming in the training materials - but there is no
one Python-course to suit us all...

Another behavior is to assume that because 'I' learned something
years-ago, so should 'everyone else'. Even superficial reflection
reveals that this just isn't true - particularly when we have an
education system which is largely based upon 'date of manufacture'!
(just because I'm older than you doesn't make me 'smarter' - although I
am definitely better-looking!)


The problem is not limited to the limitations of (hah!) floating-point
arithmetic. We've seen other questions 'here' and on "Tutor", which
illustrate lack(s) of understanding of basic relationships between
Python (software) and the hardware which runs its instructions, eg the
basic simplicity of using a 'paper computer' for debugging, the order of
precedence, that the RHS must happen before we can consider anything on
the LHS, ... I've met 'graduates' who have never even peered 'under the
hood' of a computer!

(Incidentally, I recently recommended against employing one such
otherwise extremely promising ("on paper" and 'in person') candidate.
Quizzed by A.N.Other on the panel (who was suitably-impressed), the
justification was an apparent total lack of curiosity - a personality
essential for problem-solvers! (IMHO)


Another one of my 'pet-peeves' (one of the many! Although, perhaps not
to the extent of chopping-off people's arms - no matter how Monty
Python-ic that might be, Sir Knight; is the lack of
spelling/typing/proof-reading skills evidenced by many. Criticising a
(school) teacher of my acquaintance for exactly this, I was bluntly told
"but you knew what I meant" and thus he felt "communication" had been
achieved. Um, yes, er, quite true - but at what effort, and who's effort?

I find this phenomenon in computer-people particularly fascinating,
given that Python (insert any other language-name here) is very
particular and finds "far" instead of "for" far-from acceptable (hah!
again!). Does this mean that their coding-technique is to
(almost-literally) throw stuff at Python and have the computer 'find'
all the typos and spelling-errors? (and if so, is this good use of
time/good technique?)

Thus, and on this, the members of the aforementioned panel had agreed
without discussion: any documents forwarded by a candidate which
contained spelling mistakes were 'binned' (trashed) without the usual
acceptance/tolerance/mercy one would apply to work-documents.


Training/education takes time and costs money. Accordingly, we have this
constant 'battle' between wanting to educate/acquire knowledge versus
the cost-benefit of some fact/skill which may never/rarely be used - and
the 'factory component' of choosing what is good for a group/for 'the
average trainee' cf what is so for the individual.

Entries on the back of a post-card...
-- 
Regards,
=dn

From Marco.Sulla.Python at gmail.com  Fri Nov 19 18:44:56 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Sat, 20 Nov 2021 00:44:56 +0100
Subject: pytest segfault, not with -v
In-Reply-To: <255c2cb8-b5ea-6e32-d6eb-1765af3967d5@mrabarnett.plus.com>
References: <CABbU2U_cTjL031iQ3S7BtAQM4Yc=rhRF6NzK=_C1kOoe+JwjPA@mail.gmail.com>
 <255c2cb8-b5ea-6e32-d6eb-1765af3967d5@mrabarnett.plus.com>
Message-ID: <CABbU2U-cWQbJBog-k6vr9pMPpmoY+viHdc9s1DEaihSwaNcB5A@mail.gmail.com>

On Fri, 19 Nov 2021 at 20:38, MRAB <python at mrabarnett.plus.com> wrote:
>
> On 2021-11-19 17:48, Marco Sulla wrote:
> > I have a battery of tests done with pytest. My tests break with a
> > segfault if I run them normally. If I run them using pytest -v, the
> > segfault does not happen.
> >
> > What could cause this quantical phenomenon?
> >
> Are you testing an extension that you're compiling? That kind of problem
> can occur if there's an uninitialised variable or incorrect reference
> counting (Py_INCREF/Py_DECREF).

Ok, I know. But why can't it be reproduced if I do pytest -v? This way
I don't know which test fails.
Furthermore I noticed that if I remove the __pycache__ dir of tests,
pytest does not crash, until I re-ran it with the __pycache__ dir
present.
This way is very hard for me to understand what caused the segfault.
I'm starting to think pytest is not good for testing C extensions.

From python at mrabarnett.plus.com  Fri Nov 19 19:44:26 2021
From: python at mrabarnett.plus.com (MRAB)
Date: Sat, 20 Nov 2021 00:44:26 +0000
Subject: pytest segfault, not with -v
In-Reply-To: <CABbU2U-cWQbJBog-k6vr9pMPpmoY+viHdc9s1DEaihSwaNcB5A@mail.gmail.com>
References: <CABbU2U_cTjL031iQ3S7BtAQM4Yc=rhRF6NzK=_C1kOoe+JwjPA@mail.gmail.com>
 <255c2cb8-b5ea-6e32-d6eb-1765af3967d5@mrabarnett.plus.com>
 <CABbU2U-cWQbJBog-k6vr9pMPpmoY+viHdc9s1DEaihSwaNcB5A@mail.gmail.com>
Message-ID: <e5d32ffa-eedd-715e-7306-e6c099258193@mrabarnett.plus.com>

On 2021-11-19 23:44, Marco Sulla wrote:
> On Fri, 19 Nov 2021 at 20:38, MRAB <python at mrabarnett.plus.com> wrote:
>>
>> On 2021-11-19 17:48, Marco Sulla wrote:
>> > I have a battery of tests done with pytest. My tests break with a
>> > segfault if I run them normally. If I run them using pytest -v, the
>> > segfault does not happen.
>> >
>> > What could cause this quantical phenomenon?
>> >
>> Are you testing an extension that you're compiling? That kind of problem
>> can occur if there's an uninitialised variable or incorrect reference
>> counting (Py_INCREF/Py_DECREF).
> 
> Ok, I know. But why can't it be reproduced if I do pytest -v? This way
> I don't know which test fails.
> Furthermore I noticed that if I remove the __pycache__ dir of tests,
> pytest does not crash, until I re-ran it with the __pycache__ dir
> present.
> This way is very hard for me to understand what caused the segfault.
> I'm starting to think pytest is not good for testing C extensions.
> 
If there are too few Py_INCREF or too many Py_DECREF, it'll free the 
object too soon, and whether or when that will cause a segfault will 
depend on whatever other code is running. That's the nature of the 
beast: it's unpredictable!

You could try running each of the tests in a loop to see which one 
causes a segfault. (Trying several in a loop will let you narrow it down 
more quickly.)

pytest et al. are good for testing behaviour, but not for narrowing down 
segfaults.

From ben.usenet at bsb.me.uk  Fri Nov 19 20:24:32 2021
From: ben.usenet at bsb.me.uk (Ben Bacarisse)
Date: Sat, 20 Nov 2021 01:24:32 +0000
Subject: Unexpected behaviour of math.floor,
 round and int functions (rounding)
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
 <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>
 <mailman.63.1637360151.19479.python-list@python.org>
Message-ID: <874k87k4a7.fsf@bsb.me.uk>

Chris Angelico <rosuav at gmail.com> writes:

> On Sat, Nov 20, 2021 at 9:07 AM Ben Bacarisse <ben.usenet at bsb.me.uk> wrote:
>>
>> Chris Angelico <rosuav at gmail.com> writes:
>>
>> > On Sat, Nov 20, 2021 at 5:08 AM ast <ast at invalid> wrote:
>>
>> >>  >>> 0.3 + 0.3 + 0.3 == 0.9
>> >> False
>> >
>> > That's because 0.3 is not 3/10. It's not because floats are
>> > "unreliable" or "inaccurate". It's because the ones you're entering
>> > are not what you think they are.
>> >
>> > When will people understand this?
>> >
>> > (Probably never. Sigh.)
>>
>> Most people understand what's going on when it's explained to them.  And
>> I think that being initially baffled is not unreasonable.  After all,
>> almost everyone comes to computers after learning that 3/10 can be
>> written as 0.3.  And Python "plays along" with the fiction to some
>> extent.  0.3 prints as 0.3, 3/10 prints as 0.3 and 0.3 == 3/10 is True.
>
> In grade school, we learn that not everything can be written that way,
> and 1/3 isn't actually equal to 0.3333333333.

Yes.  We learn early on that 0.3333333333 means 3333333333/10000000000.
We don't learn that 0.3333333333 is a special notation for machines that
have something called "binary floating point hardware" that does not
mean 3333333333/10000000000.  That has to be learned later.  And every
generation has to learn it afresh.

> Yet somehow people
> understand that computers speak binary ("have you learned to count
> yet, or are you still on zeroes and ones?" -- insult to a machine
> empire, in Stellaris), but don't seem to appreciate that floats are
> absolutely accurate and reliable, just in binary.

Yes, agreed, but I was not commenting on the odd (and incorrect) view
that floating point operations are not reliable and well-defined, but on
the reasonable assumption that a clever programming language might take
0.3 to mean what I was taught it meant in grade school.

As an old hand, I know it won't (in most languages), and I know why it
won't, and I know why that's usually the right design choice for the
language.  But I can also appreciate that it's by no means obvious that,
to a beginner, "binary" implies the particular kind of representation
that makes 0.3 not mean 3/10.  After all, the rational 3/10 can be
represented exactly in binary in many different ways.

> But lack of knowledge is never a problem. (Or rather, it's a solvable
> problem, and I'm always happy to explain things to people.) The
> problem is when, following that lack of understanding, people assume
> that floats are "unreliable" or "inaccurate", and that you should
> never ever compare two floats for equality, because they're never
> "really equal". That's what leads to horrible coding practices and
> badly-defined "approximately equal" checks that cause far more harm
> than a simple misunderstanding ever could on its own.

Agreed.  Often, the "explanations" just make things worse.

-- 
Ben.

From rosuav at gmail.com  Fri Nov 19 20:51:04 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 20 Nov 2021 12:51:04 +1100
Subject: Unexpected behaviour of math.floor,
 round and int functions (rounding)
In-Reply-To: <874k87k4a7.fsf@bsb.me.uk>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
 <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>
 <mailman.63.1637360151.19479.python-list@python.org>
 <874k87k4a7.fsf@bsb.me.uk>
Message-ID: <CAPTjJmohqdKMP4srxWQZmu4Xn0+M2t7J8UtL9CDtNQRuOZ8N0A@mail.gmail.com>

On Sat, Nov 20, 2021 at 12:43 PM Ben Bacarisse <ben.usenet at bsb.me.uk> wrote:
>
> Chris Angelico <rosuav at gmail.com> writes:
>
> > On Sat, Nov 20, 2021 at 9:07 AM Ben Bacarisse <ben.usenet at bsb.me.uk> wrote:
> >>
> >> Chris Angelico <rosuav at gmail.com> writes:
> >>
> >> > On Sat, Nov 20, 2021 at 5:08 AM ast <ast at invalid> wrote:
> >>
> >> >>  >>> 0.3 + 0.3 + 0.3 == 0.9
> >> >> False
> >> >
> >> > That's because 0.3 is not 3/10. It's not because floats are
> >> > "unreliable" or "inaccurate". It's because the ones you're entering
> >> > are not what you think they are.
> >> >
> >> > When will people understand this?
> >> >
> >> > (Probably never. Sigh.)
> >>
> >> Most people understand what's going on when it's explained to them.  And
> >> I think that being initially baffled is not unreasonable.  After all,
> >> almost everyone comes to computers after learning that 3/10 can be
> >> written as 0.3.  And Python "plays along" with the fiction to some
> >> extent.  0.3 prints as 0.3, 3/10 prints as 0.3 and 0.3 == 3/10 is True.
> >
> > In grade school, we learn that not everything can be written that way,
> > and 1/3 isn't actually equal to 0.3333333333.
>
> Yes.  We learn early on that 0.3333333333 means 3333333333/10000000000.
> We don't learn that 0.3333333333 is a special notation for machines that
> have something called "binary floating point hardware" that does not
> mean 3333333333/10000000000.  That has to be learned later.  And every
> generation has to learn it afresh.

But you learn that it isn't the same as 1/3. That's my point. You
already understand that it is *impossible* to write out 1/3 in
decimal. Is it such a stretch to discover that you cannot write 3/10
in binary?

Every generation has to learn about repeating fractions, but most of
us learn them in grade school. Every generation learns that computers
talk in binary. Yet, putting those two concepts together seems beyond
many people, to the point that they feel that floating point can't be
trusted.

> Yes, agreed, but I was not commenting on the odd (and incorrect) view
> that floating point operations are not reliable and well-defined, but on
> the reasonable assumption that a clever programming language might take
> 0.3 to mean what I was taught it meant in grade school.

It does mean exactly what it meant in grade school, just as 1/3 means
exactly what it meant in grade school. Now try to represent 1/3 on a
blackboard, as a decimal fraction. If that's impossible, does it mean
that 1/3 doesn't mean 1/3, or that 1/3 can't be represented?

> > But lack of knowledge is never a problem. (Or rather, it's a solvable
> > problem, and I'm always happy to explain things to people.) The
> > problem is when, following that lack of understanding, people assume
> > that floats are "unreliable" or "inaccurate", and that you should
> > never ever compare two floats for equality, because they're never
> > "really equal". That's what leads to horrible coding practices and
> > badly-defined "approximately equal" checks that cause far more harm
> > than a simple misunderstanding ever could on its own.
>
> Agreed.  Often, the "explanations" just make things worse.
>

When they're based on a fear of floats, yes. Explanations like "never
use == with floats because 0.1+0.2!=0.3" are worse than useless,
because they create that fear in a way that creates awful cargo-cult
programming practices.

If someone does something in Python, gets a weird result, and comes to
the list saying "I don't understand this", that's not a problem. We
get it all the time with mutables. Recent question about frozensets
appearing to be mutable, same thing. I have no problem with someone
humbly asking "what's happening?", based on an internal assumption
that there's a reason things are the way they are. For some reason,
floats don't get that same respect from many people.

ChrisA

From ben.usenet at bsb.me.uk  Fri Nov 19 22:25:53 2021
From: ben.usenet at bsb.me.uk (Ben Bacarisse)
Date: Sat, 20 Nov 2021 03:25:53 +0000
Subject: Unexpected behaviour of math.floor,
 round and int functions (rounding)
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
 <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>
 <mailman.63.1637360151.19479.python-list@python.org>
 <874k87k4a7.fsf@bsb.me.uk>
 <CAPTjJmohqdKMP4srxWQZmu4Xn0+M2t7J8UtL9CDtNQRuOZ8N0A@mail.gmail.com>
 <mailman.67.1637373077.19479.python-list@python.org>
Message-ID: <87y25jik3i.fsf@bsb.me.uk>

Chris Angelico <rosuav at gmail.com> writes:

> On Sat, Nov 20, 2021 at 12:43 PM Ben Bacarisse <ben.usenet at bsb.me.uk> wrote:
>>
>> Chris Angelico <rosuav at gmail.com> writes:
>>
>> > On Sat, Nov 20, 2021 at 9:07 AM Ben Bacarisse <ben.usenet at bsb.me.uk> wrote:
>> >>
>> >> Chris Angelico <rosuav at gmail.com> writes:
>> >>
>> >> > On Sat, Nov 20, 2021 at 5:08 AM ast <ast at invalid> wrote:
>> >>
>> >> >>  >>> 0.3 + 0.3 + 0.3 == 0.9
>> >> >> False
>> >> >
>> >> > That's because 0.3 is not 3/10. It's not because floats are
>> >> > "unreliable" or "inaccurate". It's because the ones you're entering
>> >> > are not what you think they are.
>> >> >
>> >> > When will people understand this?
>> >> >
>> >> > (Probably never. Sigh.)
>> >>
>> >> Most people understand what's going on when it's explained to them.  And
>> >> I think that being initially baffled is not unreasonable.  After all,
>> >> almost everyone comes to computers after learning that 3/10 can be
>> >> written as 0.3.  And Python "plays along" with the fiction to some
>> >> extent.  0.3 prints as 0.3, 3/10 prints as 0.3 and 0.3 == 3/10 is True.
>> >
>> > In grade school, we learn that not everything can be written that way,
>> > and 1/3 isn't actually equal to 0.3333333333.
>>
>> Yes.  We learn early on that 0.3333333333 means 3333333333/10000000000.
>> We don't learn that 0.3333333333 is a special notation for machines that
>> have something called "binary floating point hardware" that does not
>> mean 3333333333/10000000000.  That has to be learned later.  And every
>> generation has to learn it afresh.
>
> But you learn that it isn't the same as 1/3. That's my point. You
> already understand that it is *impossible* to write out 1/3 in
> decimal. Is it such a stretch to discover that you cannot write 3/10
> in binary?
>
> Every generation has to learn about repeating fractions, but most of
> us learn them in grade school. Every generation learns that computers
> talk in binary. Yet, putting those two concepts together seems beyond
> many people, to the point that they feel that floating point can't be
> trusted.

Binary is a bit of a red herring here.  It's the floating point format
that needs to be understood.  Three tenths can be represented in many
binary formats, and even decimal floating point will have some surprises
for the novice.

>> Yes, agreed, but I was not commenting on the odd (and incorrect) view
>> that floating point operations are not reliable and well-defined, but on
>> the reasonable assumption that a clever programming language might take
>> 0.3 to mean what I was taught it meant in grade school.
>
> It does mean exactly what it meant in grade school, just as 1/3 means
> exactly what it meant in grade school. Now try to represent 1/3 on a
> blackboard, as a decimal fraction. If that's impossible, does it mean
> that 1/3 doesn't mean 1/3, or that 1/3 can't be represented?

As you know, it is possible, but let's say we outlaw any finite notation
for repeated digits...  Why should I convert 1/3 to this particular
apparently unsuitable representation?  I will write 1/3 and manipulate
that number using factional notation.

The novice programmer might similarly expect that when they write 0.3,
the program will manipulate that number as the faction it clearly is.
They may well be surprised by the fact that it must get put into a
format that can't represent what those three characters mean, just as I
would be surprised if you insisted I write 1/3 as a finite decimal (with
no repeat notation).

I'm not saying your analogy would not help someone understand, but you
first have to explain why 0.3 is not treated as three tenths -- why I
(to use your analogy) must not keep 1/3 as a proper fraction, but I must
instead write it using a finite number of decimal digits.  Neither is,
in my view, obvious to the beginner.

>> > But lack of knowledge is never a problem. (Or rather, it's a solvable
>> > problem, and I'm always happy to explain things to people.) The
>> > problem is when, following that lack of understanding, people assume
>> > that floats are "unreliable" or "inaccurate", and that you should
>> > never ever compare two floats for equality, because they're never
>> > "really equal". That's what leads to horrible coding practices and
>> > badly-defined "approximately equal" checks that cause far more harm
>> > than a simple misunderstanding ever could on its own.
>>
>> Agreed.  Often, the "explanations" just make things worse.
>
> When they're based on a fear of floats, yes. Explanations like "never
> use == with floats because 0.1+0.2!=0.3" are worse than useless,
> because they create that fear in a way that creates awful cargo-cult
> programming practices.
>
> If someone does something in Python, gets a weird result, and comes to
> the list saying "I don't understand this", that's not a problem. We
> get it all the time with mutables. Recent question about frozensets
> appearing to be mutable, same thing. I have no problem with someone
> humbly asking "what's happening?", based on an internal assumption
> that there's a reason things are the way they are. For some reason,
> floats don't get that same respect from many people.

On all this, I agree.  As a former numerical analyst, I want maximal
respect for all the various floating-point representations!

-- 
Ben.

From rosuav at gmail.com  Sat Nov 20 00:18:12 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 20 Nov 2021 16:18:12 +1100
Subject: Unexpected behaviour of math.floor,
 round and int functions (rounding)
In-Reply-To: <87y25jik3i.fsf@bsb.me.uk>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
 <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>
 <mailman.63.1637360151.19479.python-list@python.org>
 <874k87k4a7.fsf@bsb.me.uk>
 <CAPTjJmohqdKMP4srxWQZmu4Xn0+M2t7J8UtL9CDtNQRuOZ8N0A@mail.gmail.com>
 <mailman.67.1637373077.19479.python-list@python.org>
 <87y25jik3i.fsf@bsb.me.uk>
Message-ID: <CAPTjJmr-Sik-NEzm1m17uXUMXSf8ijT9x=BvX9PYSoW6x=Nf0w@mail.gmail.com>

On Sat, Nov 20, 2021 at 3:41 PM Ben Bacarisse <ben.usenet at bsb.me.uk> wrote:
>
> Chris Angelico <rosuav at gmail.com> writes:
>
> > On Sat, Nov 20, 2021 at 12:43 PM Ben Bacarisse <ben.usenet at bsb.me.uk> wrote:
> >>
> >> Chris Angelico <rosuav at gmail.com> writes:
> >>
> >> > On Sat, Nov 20, 2021 at 9:07 AM Ben Bacarisse <ben.usenet at bsb.me.uk> wrote:
> >> >>
> >> >> Chris Angelico <rosuav at gmail.com> writes:
> >> >>
> >> >> > On Sat, Nov 20, 2021 at 5:08 AM ast <ast at invalid> wrote:
> >> >>
> >> >> >>  >>> 0.3 + 0.3 + 0.3 == 0.9
> >> >> >> False
> >> >> >
> >> >> > That's because 0.3 is not 3/10. It's not because floats are
> >> >> > "unreliable" or "inaccurate". It's because the ones you're entering
> >> >> > are not what you think they are.
> >> >> >
> >> >> > When will people understand this?
> >> >> >
> >> >> > (Probably never. Sigh.)
> >> >>
> >> >> Most people understand what's going on when it's explained to them.  And
> >> >> I think that being initially baffled is not unreasonable.  After all,
> >> >> almost everyone comes to computers after learning that 3/10 can be
> >> >> written as 0.3.  And Python "plays along" with the fiction to some
> >> >> extent.  0.3 prints as 0.3, 3/10 prints as 0.3 and 0.3 == 3/10 is True.
> >> >
> >> > In grade school, we learn that not everything can be written that way,
> >> > and 1/3 isn't actually equal to 0.3333333333.
> >>
> >> Yes.  We learn early on that 0.3333333333 means 3333333333/10000000000.
> >> We don't learn that 0.3333333333 is a special notation for machines that
> >> have something called "binary floating point hardware" that does not
> >> mean 3333333333/10000000000.  That has to be learned later.  And every
> >> generation has to learn it afresh.
> >
> > But you learn that it isn't the same as 1/3. That's my point. You
> > already understand that it is *impossible* to write out 1/3 in
> > decimal. Is it such a stretch to discover that you cannot write 3/10
> > in binary?
> >
> > Every generation has to learn about repeating fractions, but most of
> > us learn them in grade school. Every generation learns that computers
> > talk in binary. Yet, putting those two concepts together seems beyond
> > many people, to the point that they feel that floating point can't be
> > trusted.
>
> Binary is a bit of a red herring here.  It's the floating point format
> that needs to be understood.  Three tenths can be represented in many
> binary formats, and even decimal floating point will have some surprises
> for the novice.

Not completely a red herring; binary floating-point as used in Python
(IEEE double-precision) is defined as a binary mantissa and a scale,
just as "blackboard arithmetic" is generally defined as a decimal
mantissa and a scale. (At least, I don't think I've ever seen anyone
doing arithmetic on a blackboard in hex or octal.)

> >> Yes, agreed, but I was not commenting on the odd (and incorrect) view
> >> that floating point operations are not reliable and well-defined, but on
> >> the reasonable assumption that a clever programming language might take
> >> 0.3 to mean what I was taught it meant in grade school.
> >
> > It does mean exactly what it meant in grade school, just as 1/3 means
> > exactly what it meant in grade school. Now try to represent 1/3 on a
> > blackboard, as a decimal fraction. If that's impossible, does it mean
> > that 1/3 doesn't mean 1/3, or that 1/3 can't be represented?
>
> As you know, it is possible, but let's say we outlaw any finite notation
> for repeated digits...  Why should I convert 1/3 to this particular
> apparently unsuitable representation?  I will write 1/3 and manipulate
> that number using factional notation.

If you want that, the fractions module is there for you. And again,
grade school, we learned about ratios as well as decimals (or vulgar
fractions and decimal fractions). They have different tradeoffs. For
instance, I learned pi as both 22/7 and 3.14, because sometimes it'd
be convenient to use the rational form and other times the decimal.

> The novice programmer might similarly expect that when they write 0.3,
> the program will manipulate that number as the faction it clearly is.
> They may well be surprised by the fact that it must get put into a
> format that can't represent what those three characters mean, just as I
> would be surprised if you insisted I write 1/3 as a finite decimal (with
> no repeat notation).

Except that 0.3 isn't written as a fraction, it's written as a decimal.

> I'm not saying your analogy would not help someone understand, but you
> first have to explain why 0.3 is not treated as three tenths -- why I
> (to use your analogy) must not keep 1/3 as a proper fraction, but I must
> instead write it using a finite number of decimal digits.  Neither is,
> in my view, obvious to the beginner.

Try adding 1/3 + e; either you have to convert 1/3 to a decimal, or
find a rational approximation for e (there aren't any really good ones
but 193/71 seems promising - that's 2.7183, close enough) and then go
to the work of rational addition. No, more likely you'll go for a
finite number of decimal digits.

> >> > But lack of knowledge is never a problem. (Or rather, it's a solvable
> >> > problem, and I'm always happy to explain things to people.) The
> >> > problem is when, following that lack of understanding, people assume
> >> > that floats are "unreliable" or "inaccurate", and that you should
> >> > never ever compare two floats for equality, because they're never
> >> > "really equal". That's what leads to horrible coding practices and
> >> > badly-defined "approximately equal" checks that cause far more harm
> >> > than a simple misunderstanding ever could on its own.
> >>
> >> Agreed.  Often, the "explanations" just make things worse.
> >
> > When they're based on a fear of floats, yes. Explanations like "never
> > use == with floats because 0.1+0.2!=0.3" are worse than useless,
> > because they create that fear in a way that creates awful cargo-cult
> > programming practices.
> >
> > If someone does something in Python, gets a weird result, and comes to
> > the list saying "I don't understand this", that's not a problem. We
> > get it all the time with mutables. Recent question about frozensets
> > appearing to be mutable, same thing. I have no problem with someone
> > humbly asking "what's happening?", based on an internal assumption
> > that there's a reason things are the way they are. For some reason,
> > floats don't get that same respect from many people.
>
> On all this, I agree.  As a former numerical analyst, I want maximal
> respect for all the various floating-point representations!
>

Yeah - or if not the full meaning of the representation, at least the
fact that it involves no more than 53 bits of "number", and all the
consequences of that.

Oh, and at least once, everyone should play with a stringified number
system, where you actually truly store the string representation and
do arithmetic on it. It has very interesting consequences that really
make you appreciate IEEE binary floating-point.

ChrisA

From drsalists at gmail.com  Sat Nov 20 11:23:21 2021
From: drsalists at gmail.com (Dan Stromberg)
Date: Sat, 20 Nov 2021 08:23:21 -0800
Subject: pytest segfault, not with -v
In-Reply-To: <CABbU2U_cTjL031iQ3S7BtAQM4Yc=rhRF6NzK=_C1kOoe+JwjPA@mail.gmail.com>
References: <CABbU2U_cTjL031iQ3S7BtAQM4Yc=rhRF6NzK=_C1kOoe+JwjPA@mail.gmail.com>
Message-ID: <CAGGBd_onbD8jTSSh3X3c8D64uA=1wcGM9MkhzCuXqJTq6LWqWg@mail.gmail.com>

On Fri, Nov 19, 2021 at 9:49 AM Marco Sulla <Marco.Sulla.Python at gmail.com>
wrote:

> I have a battery of tests done with pytest. My tests break with a
> segfault if I run them normally. If I run them using pytest -v, the
> segfault does not happen.
>
> What could cause this quantical phenomenon?
>

Pure python code shouldn't do this, unless you're using ctypes or similar
(which arguably isn't pure python).

But C extension modules sure can.  See:
https://stromberg.dnsalias.org/~strombrg/checking-early.html .  It uses
Fortran to make its point, but the same thing very much applies to C.

BTW, if you're using C extension modules, the troublesome one doesn't
necessarily have to be one you wrote. It could be a dependency created by
someone else too.

From Marco.Sulla.Python at gmail.com  Sat Nov 20 12:40:01 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Sat, 20 Nov 2021 18:40:01 +0100
Subject: pytest segfault, not with -v
In-Reply-To: <e5d32ffa-eedd-715e-7306-e6c099258193@mrabarnett.plus.com>
References: <CABbU2U_cTjL031iQ3S7BtAQM4Yc=rhRF6NzK=_C1kOoe+JwjPA@mail.gmail.com>
 <255c2cb8-b5ea-6e32-d6eb-1765af3967d5@mrabarnett.plus.com>
 <CABbU2U-cWQbJBog-k6vr9pMPpmoY+viHdc9s1DEaihSwaNcB5A@mail.gmail.com>
 <e5d32ffa-eedd-715e-7306-e6c099258193@mrabarnett.plus.com>
Message-ID: <CABbU2U8z=r3vgukN==fT9dFJN0d1fTLboB8ek+jukG7ASYKzqg@mail.gmail.com>

Indeed I have introduced a command line parameter in my bench.py
script that simply specifies the number of times the benchmarks are
performed. This way I have a sort of segfault checker.

But I don't bench any part of the library. I suppose I have to create
a separate script that does a simple loop for all the cases, and
remove the optional parameter from bench. How boring.
PS: is there a way to monitor the Python consumed memory inside Python
itself? In this way I could also trap memory leaks.

On Sat, 20 Nov 2021 at 01:46, MRAB <python at mrabarnett.plus.com> wrote:
>
> On 2021-11-19 23:44, Marco Sulla wrote:
> > On Fri, 19 Nov 2021 at 20:38, MRAB <python at mrabarnett.plus.com> wrote:
> >>
> >> On 2021-11-19 17:48, Marco Sulla wrote:
> >> > I have a battery of tests done with pytest. My tests break with a
> >> > segfault if I run them normally. If I run them using pytest -v, the
> >> > segfault does not happen.
> >> >
> >> > What could cause this quantical phenomenon?
> >> >
> >> Are you testing an extension that you're compiling? That kind of problem
> >> can occur if there's an uninitialised variable or incorrect reference
> >> counting (Py_INCREF/Py_DECREF).
> >
> > Ok, I know. But why can't it be reproduced if I do pytest -v? This way
> > I don't know which test fails.
> > Furthermore I noticed that if I remove the __pycache__ dir of tests,
> > pytest does not crash, until I re-ran it with the __pycache__ dir
> > present.
> > This way is very hard for me to understand what caused the segfault.
> > I'm starting to think pytest is not good for testing C extensions.
> >
> If there are too few Py_INCREF or too many Py_DECREF, it'll free the
> object too soon, and whether or when that will cause a segfault will
> depend on whatever other code is running. That's the nature of the
> beast: it's unpredictable!
>
> You could try running each of the tests in a loop to see which one
> causes a segfault. (Trying several in a loop will let you narrow it down
> more quickly.)
>
> pytest et al. are good for testing behaviour, but not for narrowing down
> segfaults.
> --
> https://mail.python.org/mailman/listinfo/python-list

From Marco.Sulla.Python at gmail.com  Sat Nov 20 12:45:16 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Sat, 20 Nov 2021 18:45:16 +0100
Subject: No right operator in tp_as_number?
Message-ID: <CABbU2U8CzKoXCTyOqxOnd4L86GmxupJQADSDGiLbR9qvX4tA8g@mail.gmail.com>

I checked the documentation:
https://docs.python.org/3/c-api/typeobj.html#number-structs
and it seems that, in the Python C API, the right operators do not exist.
For example, there is nb_add, that in Python is __add__, but there's
no nb_right_add, that in Python is __radd__

Am I missing something?

From python at mrabarnett.plus.com  Sat Nov 20 12:55:33 2021
From: python at mrabarnett.plus.com (MRAB)
Date: Sat, 20 Nov 2021 17:55:33 +0000
Subject: pytest segfault, not with -v
In-Reply-To: <CABbU2U8z=r3vgukN==fT9dFJN0d1fTLboB8ek+jukG7ASYKzqg@mail.gmail.com>
References: <CABbU2U_cTjL031iQ3S7BtAQM4Yc=rhRF6NzK=_C1kOoe+JwjPA@mail.gmail.com>
 <255c2cb8-b5ea-6e32-d6eb-1765af3967d5@mrabarnett.plus.com>
 <CABbU2U-cWQbJBog-k6vr9pMPpmoY+viHdc9s1DEaihSwaNcB5A@mail.gmail.com>
 <e5d32ffa-eedd-715e-7306-e6c099258193@mrabarnett.plus.com>
 <CABbU2U8z=r3vgukN==fT9dFJN0d1fTLboB8ek+jukG7ASYKzqg@mail.gmail.com>
Message-ID: <08f263b7-5e26-2dfb-4bf6-1f09470c28b0@mrabarnett.plus.com>

On 2021-11-20 17:40, Marco Sulla wrote:
> Indeed I have introduced a command line parameter in my bench.py
> script that simply specifies the number of times the benchmarks are
> performed. This way I have a sort of segfault checker.
> 
> But I don't bench any part of the library. I suppose I have to create
> a separate script that does a simple loop for all the cases, and
> remove the optional parameter from bench. How boring.
> PS: is there a way to monitor the Python consumed memory inside Python
> itself? In this way I could also trap memory leaks.
> 
I'm on Windows 10, so I debug in Microsoft Visual Studio. I also have a 
look at the memory usage in Task Manager. If the program uses more 
memory when there are more iterations, then that's a sign of a memory 
leak. For some objects I'd look at the reference count to see if it's 
increasing or decreasing for each iteration when it should be constant 
over time.

> On Sat, 20 Nov 2021 at 01:46, MRAB <python at mrabarnett.plus.com> wrote:
>>
>> On 2021-11-19 23:44, Marco Sulla wrote:
>> > On Fri, 19 Nov 2021 at 20:38, MRAB <python at mrabarnett.plus.com> wrote:
>> >>
>> >> On 2021-11-19 17:48, Marco Sulla wrote:
>> >> > I have a battery of tests done with pytest. My tests break with a
>> >> > segfault if I run them normally. If I run them using pytest -v, the
>> >> > segfault does not happen.
>> >> >
>> >> > What could cause this quantical phenomenon?
>> >> >
>> >> Are you testing an extension that you're compiling? That kind of problem
>> >> can occur if there's an uninitialised variable or incorrect reference
>> >> counting (Py_INCREF/Py_DECREF).
>> >
>> > Ok, I know. But why can't it be reproduced if I do pytest -v? This way
>> > I don't know which test fails.
>> > Furthermore I noticed that if I remove the __pycache__ dir of tests,
>> > pytest does not crash, until I re-ran it with the __pycache__ dir
>> > present.
>> > This way is very hard for me to understand what caused the segfault.
>> > I'm starting to think pytest is not good for testing C extensions.
>> >
>> If there are too few Py_INCREF or too many Py_DECREF, it'll free the
>> object too soon, and whether or when that will cause a segfault will
>> depend on whatever other code is running. That's the nature of the
>> beast: it's unpredictable!
>>
>> You could try running each of the tests in a loop to see which one
>> causes a segfault. (Trying several in a loop will let you narrow it down
>> more quickly.)
>>
>> pytest et al. are good for testing behaviour, but not for narrowing down
>> segfaults.
> 

From Marco.Sulla.Python at gmail.com  Sat Nov 20 13:07:32 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Sat, 20 Nov 2021 19:07:32 +0100
Subject: pytest segfault, not with -v
In-Reply-To: <08f263b7-5e26-2dfb-4bf6-1f09470c28b0@mrabarnett.plus.com>
References: <CABbU2U_cTjL031iQ3S7BtAQM4Yc=rhRF6NzK=_C1kOoe+JwjPA@mail.gmail.com>
 <255c2cb8-b5ea-6e32-d6eb-1765af3967d5@mrabarnett.plus.com>
 <CABbU2U-cWQbJBog-k6vr9pMPpmoY+viHdc9s1DEaihSwaNcB5A@mail.gmail.com>
 <e5d32ffa-eedd-715e-7306-e6c099258193@mrabarnett.plus.com>
 <CABbU2U8z=r3vgukN==fT9dFJN0d1fTLboB8ek+jukG7ASYKzqg@mail.gmail.com>
 <08f263b7-5e26-2dfb-4bf6-1f09470c28b0@mrabarnett.plus.com>
Message-ID: <CABbU2U_hj_OtW1u3Jk5-3c3kzYtU187RQjmX_-vbsf3kgmC10A@mail.gmail.com>

I know how to check the refcounts, but I don't know how to check the
memory usage, since it's not a program, it's a simple library. Is
there not a way to check inside Python the memory usage? I have to use
a bash script (I'm on Linux)?

On Sat, 20 Nov 2021 at 19:00, MRAB <python at mrabarnett.plus.com> wrote:
>
> On 2021-11-20 17:40, Marco Sulla wrote:
> > Indeed I have introduced a command line parameter in my bench.py
> > script that simply specifies the number of times the benchmarks are
> > performed. This way I have a sort of segfault checker.
> >
> > But I don't bench any part of the library. I suppose I have to create
> > a separate script that does a simple loop for all the cases, and
> > remove the optional parameter from bench. How boring.
> > PS: is there a way to monitor the Python consumed memory inside Python
> > itself? In this way I could also trap memory leaks.
> >
> I'm on Windows 10, so I debug in Microsoft Visual Studio. I also have a
> look at the memory usage in Task Manager. If the program uses more
> memory when there are more iterations, then that's a sign of a memory
> leak. For some objects I'd look at the reference count to see if it's
> increasing or decreasing for each iteration when it should be constant
> over time.
>
> > On Sat, 20 Nov 2021 at 01:46, MRAB <python at mrabarnett.plus.com> wrote:
> >>
> >> On 2021-11-19 23:44, Marco Sulla wrote:
> >> > On Fri, 19 Nov 2021 at 20:38, MRAB <python at mrabarnett.plus.com> wrote:
> >> >>
> >> >> On 2021-11-19 17:48, Marco Sulla wrote:
> >> >> > I have a battery of tests done with pytest. My tests break with a
> >> >> > segfault if I run them normally. If I run them using pytest -v, the
> >> >> > segfault does not happen.
> >> >> >
> >> >> > What could cause this quantical phenomenon?
> >> >> >
> >> >> Are you testing an extension that you're compiling? That kind of problem
> >> >> can occur if there's an uninitialised variable or incorrect reference
> >> >> counting (Py_INCREF/Py_DECREF).
> >> >
> >> > Ok, I know. But why can't it be reproduced if I do pytest -v? This way
> >> > I don't know which test fails.
> >> > Furthermore I noticed that if I remove the __pycache__ dir of tests,
> >> > pytest does not crash, until I re-ran it with the __pycache__ dir
> >> > present.
> >> > This way is very hard for me to understand what caused the segfault.
> >> > I'm starting to think pytest is not good for testing C extensions.
> >> >
> >> If there are too few Py_INCREF or too many Py_DECREF, it'll free the
> >> object too soon, and whether or when that will cause a segfault will
> >> depend on whatever other code is running. That's the nature of the
> >> beast: it's unpredictable!
> >>
> >> You could try running each of the tests in a loop to see which one
> >> causes a segfault. (Trying several in a loop will let you narrow it down
> >> more quickly.)
> >>
> >> pytest et al. are good for testing behaviour, but not for narrowing down
> >> segfaults.
> >
> --
> https://mail.python.org/mailman/listinfo/python-list

From python at mrabarnett.plus.com  Sat Nov 20 13:35:20 2021
From: python at mrabarnett.plus.com (MRAB)
Date: Sat, 20 Nov 2021 18:35:20 +0000
Subject: No right operator in tp_as_number?
In-Reply-To: <CABbU2U8CzKoXCTyOqxOnd4L86GmxupJQADSDGiLbR9qvX4tA8g@mail.gmail.com>
References: <CABbU2U8CzKoXCTyOqxOnd4L86GmxupJQADSDGiLbR9qvX4tA8g@mail.gmail.com>
Message-ID: <9b672dcb-a4ff-1153-2b5d-58ffc0351b4c@mrabarnett.plus.com>

On 2021-11-20 17:45, Marco Sulla wrote:
> I checked the documentation:
> https://docs.python.org/3/c-api/typeobj.html#number-structs
> and it seems that, in the Python C API, the right operators do not exist.
> For example, there is nb_add, that in Python is __add__, but there's
> no nb_right_add, that in Python is __radd__
> 
> Am I missing something?
> 
A quick Google came up with this:

Python's __radd__ doesn't work for C-defined types
https://stackoverflow.com/questions/18794169/pythons-radd-doesnt-work-for-c-defined-types

It's about Python 2.7, but the principle is the same.

From drsalists at gmail.com  Sat Nov 20 13:59:55 2021
From: drsalists at gmail.com (Dan Stromberg)
Date: Sat, 20 Nov 2021 10:59:55 -0800
Subject: pytest segfault, not with -v
In-Reply-To: <CABbU2U_hj_OtW1u3Jk5-3c3kzYtU187RQjmX_-vbsf3kgmC10A@mail.gmail.com>
References: <CABbU2U_cTjL031iQ3S7BtAQM4Yc=rhRF6NzK=_C1kOoe+JwjPA@mail.gmail.com>
 <255c2cb8-b5ea-6e32-d6eb-1765af3967d5@mrabarnett.plus.com>
 <CABbU2U-cWQbJBog-k6vr9pMPpmoY+viHdc9s1DEaihSwaNcB5A@mail.gmail.com>
 <e5d32ffa-eedd-715e-7306-e6c099258193@mrabarnett.plus.com>
 <CABbU2U8z=r3vgukN==fT9dFJN0d1fTLboB8ek+jukG7ASYKzqg@mail.gmail.com>
 <08f263b7-5e26-2dfb-4bf6-1f09470c28b0@mrabarnett.plus.com>
 <CABbU2U_hj_OtW1u3Jk5-3c3kzYtU187RQjmX_-vbsf3kgmC10A@mail.gmail.com>
Message-ID: <CAGGBd_p2ozk5tV8AGS6XSA4Hy+D3iFcoXCwxksc6yCfqr8iqsA@mail.gmail.com>

On Sat, Nov 20, 2021 at 10:09 AM Marco Sulla <Marco.Sulla.Python at gmail.com>
wrote:

> I know how to check the refcounts, but I don't know how to check the
> memory usage, since it's not a program, it's a simple library. Is
> there not a way to check inside Python the memory usage? I have to use
> a bash script (I'm on Linux)?
>

ps auxww
...can show you how much memory is in use for the entire process.

It's commonly combined with grep, like:
ps auxww | head -1
ps auxww | grep my-program-name

Have a look at the %MEM, VSZ and RSS columns.

But being out of memory doesn't necessarily lead to a segfault - it can (EG
if a malloc failed, and some C programmer neglected to do decent error
checking), but an OOM kill is more likely.

From drsalists at gmail.com  Sat Nov 20 14:22:06 2021
From: drsalists at gmail.com (Dan Stromberg)
Date: Sat, 20 Nov 2021 11:22:06 -0800
Subject: pytest segfault, not with -v
In-Reply-To: <CAGGBd_p2ozk5tV8AGS6XSA4Hy+D3iFcoXCwxksc6yCfqr8iqsA@mail.gmail.com>
References: <CABbU2U_cTjL031iQ3S7BtAQM4Yc=rhRF6NzK=_C1kOoe+JwjPA@mail.gmail.com>
 <255c2cb8-b5ea-6e32-d6eb-1765af3967d5@mrabarnett.plus.com>
 <CABbU2U-cWQbJBog-k6vr9pMPpmoY+viHdc9s1DEaihSwaNcB5A@mail.gmail.com>
 <e5d32ffa-eedd-715e-7306-e6c099258193@mrabarnett.plus.com>
 <CABbU2U8z=r3vgukN==fT9dFJN0d1fTLboB8ek+jukG7ASYKzqg@mail.gmail.com>
 <08f263b7-5e26-2dfb-4bf6-1f09470c28b0@mrabarnett.plus.com>
 <CABbU2U_hj_OtW1u3Jk5-3c3kzYtU187RQjmX_-vbsf3kgmC10A@mail.gmail.com>
 <CAGGBd_p2ozk5tV8AGS6XSA4Hy+D3iFcoXCwxksc6yCfqr8iqsA@mail.gmail.com>
Message-ID: <CAGGBd_paTU1hdnbEA6hwUhSNfAQVA8mp9iV4xjgymrM_78s2OA@mail.gmail.com>

On Sat, Nov 20, 2021 at 10:59 AM Dan Stromberg <drsalists at gmail.com> wrote:

>
>
> On Sat, Nov 20, 2021 at 10:09 AM Marco Sulla <Marco.Sulla.Python at gmail.com>
> wrote:
>
>> I know how to check the refcounts, but I don't know how to check the
>> memory usage, since it's not a program, it's a simple library. Is
>> there not a way to check inside Python the memory usage? I have to use
>> a bash script (I'm on Linux)?
>>
>
> ps auxww
> ...can show you how much memory is in use for the entire process.
>
> It's commonly combined with grep, like:
> ps auxww | head -1
> ps auxww | grep my-program-name
>
> Have a look at the %MEM, VSZ and RSS columns.
>
> But being out of memory doesn't necessarily lead to a segfault - it can
> (EG if a malloc failed, and some C programmer neglected to do decent error
> checking), but an OOM kill is more likely.
>

The above can be used to detect a leak in the _process_.

Once it's been established (if it's established) that the process is
getting oversized, you can sometimes see where the memory is going with:
https://www.fugue.co/blog/diagnosing-and-fixing-memory-leaks-in-python.html

But again, a memory leak isn't necessarily going to lead to a segfault.

From ben.usenet at bsb.me.uk  Sat Nov 20 05:35:25 2021
From: ben.usenet at bsb.me.uk (Ben Bacarisse)
Date: Sat, 20 Nov 2021 10:35:25 +0000
Subject: Unexpected behaviour of math.floor,
 round and int functions (rounding)
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
 <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>
 <mailman.63.1637360151.19479.python-list@python.org>
 <874k87k4a7.fsf@bsb.me.uk>
 <CAPTjJmohqdKMP4srxWQZmu4Xn0+M2t7J8UtL9CDtNQRuOZ8N0A@mail.gmail.com>
 <mailman.67.1637373077.19479.python-list@python.org>
 <87y25jik3i.fsf@bsb.me.uk>
 <CAPTjJmr-Sik-NEzm1m17uXUMXSf8ijT9x=BvX9PYSoW6x=Nf0w@mail.gmail.com>
 <mailman.68.1637385505.19479.python-list@python.org>
Message-ID: <87sfvri07m.fsf@bsb.me.uk>

Chris Angelico <rosuav at gmail.com> writes:

> On Sat, Nov 20, 2021 at 3:41 PM Ben Bacarisse <ben.usenet at bsb.me.uk> wrote:
>>
>> Chris Angelico <rosuav at gmail.com> writes:
>>
>> > On Sat, Nov 20, 2021 at 12:43 PM Ben Bacarisse <ben.usenet at bsb.me.uk> wrote:
>> >>
>> >> Chris Angelico <rosuav at gmail.com> writes:
>> >>
>> >> > On Sat, Nov 20, 2021 at 9:07 AM Ben Bacarisse <ben.usenet at bsb.me.uk> wrote:
>> >> >>
>> >> >> Chris Angelico <rosuav at gmail.com> writes:
>> >> >>
>> >> >> > On Sat, Nov 20, 2021 at 5:08 AM ast <ast at invalid> wrote:
>> >> >>
>> >> >> >>  >>> 0.3 + 0.3 + 0.3 == 0.9
>> >> >> >> False
>> >> >> >
>> >> >> > That's because 0.3 is not 3/10. It's not because floats are
>> >> >> > "unreliable" or "inaccurate". It's because the ones you're entering
>> >> >> > are not what you think they are.
>> >> >> >
>> >> >> > When will people understand this?
>> >> >> >
>> >> >> > (Probably never. Sigh.)
>> >> >>
>> >> >> Most people understand what's going on when it's explained to them.  And
>> >> >> I think that being initially baffled is not unreasonable.  After all,
>> >> >> almost everyone comes to computers after learning that 3/10 can be
>> >> >> written as 0.3.  And Python "plays along" with the fiction to some
>> >> >> extent.  0.3 prints as 0.3, 3/10 prints as 0.3 and 0.3 == 3/10 is True.
>> >> >
>> >> > In grade school, we learn that not everything can be written that way,
>> >> > and 1/3 isn't actually equal to 0.3333333333.
>> >>
>> >> Yes.  We learn early on that 0.3333333333 means 3333333333/10000000000.
>> >> We don't learn that 0.3333333333 is a special notation for machines that
>> >> have something called "binary floating point hardware" that does not
>> >> mean 3333333333/10000000000.  That has to be learned later.  And every
>> >> generation has to learn it afresh.
>> >
>> > But you learn that it isn't the same as 1/3. That's my point. You
>> > already understand that it is *impossible* to write out 1/3 in
>> > decimal. Is it such a stretch to discover that you cannot write 3/10
>> > in binary?
>> >
>> > Every generation has to learn about repeating fractions, but most of
>> > us learn them in grade school. Every generation learns that computers
>> > talk in binary. Yet, putting those two concepts together seems beyond
>> > many people, to the point that they feel that floating point can't be
>> > trusted.
>>
>> Binary is a bit of a red herring here.  It's the floating point format
>> that needs to be understood.  Three tenths can be represented in many
>> binary formats, and even decimal floating point will have some surprises
>> for the novice.
>
> Not completely a red herring; binary floating-point as used in Python
> (IEEE double-precision) is defined as a binary mantissa and a scale,
> just as "blackboard arithmetic" is generally defined as a decimal
> mantissa and a scale. (At least, I don't think I've ever seen anyone
> doing arithmetic on a blackboard in hex or octal.)

You seem to be agreeing with me.  It's the floating point part that is
the issue, not the base itself.

>> >> Yes, agreed, but I was not commenting on the odd (and incorrect) view
>> >> that floating point operations are not reliable and well-defined, but on
>> >> the reasonable assumption that a clever programming language might take
>> >> 0.3 to mean what I was taught it meant in grade school.
>> >
>> > It does mean exactly what it meant in grade school, just as 1/3 means
>> > exactly what it meant in grade school. Now try to represent 1/3 on a
>> > blackboard, as a decimal fraction. If that's impossible, does it mean
>> > that 1/3 doesn't mean 1/3, or that 1/3 can't be represented?
>>
>> As you know, it is possible, but let's say we outlaw any finite notation
>> for repeated digits...  Why should I convert 1/3 to this particular
>> apparently unsuitable representation?  I will write 1/3 and manipulate
>> that number using factional notation.
>
> If you want that, the fractions module is there for you.

Yes, I know.  The only point of disagreement (as far as can see) is
that literals like 0.3 appears to be confusing for beginners.  You think
they should know that "binary" (which may be all they know about
computers and numbers) means fixed-width binary floating point (or at
least might imply a format that can't represent three tenths), where I
think it's not unreasonable for them to suppose that 0.3 is manipulated
as the rational number it so clearly is.

> And again,
> grade school, we learned about ratios as well as decimals (or vulgar
> fractions and decimal fractions). They have different tradeoffs. For
> instance, I learned pi as both 22/7 and 3.14, because sometimes it'd
> be convenient to use the rational form and other times the decimal.
>
>> The novice programmer might similarly expect that when they write 0.3,
>> the program will manipulate that number as the faction it clearly is.
>> They may well be surprised by the fact that it must get put into a
>> format that can't represent what those three characters mean, just as I
>> would be surprised if you insisted I write 1/3 as a finite decimal (with
>> no repeat notation).
>
> Except that 0.3 isn't written as a fraction, it's written as a
> decimal.

Are you worrying about the term I used?  0.3 is a rational number.  It
is the way to write a particular fraction in decimal.  I think the only
point of disagreement is that you hope or expect that anyone writing
three tenths as 0.3 should expect that that literal might be converted
to some collection of bits that can't represent it.  I think it's not
unreasonable for a beginner to think otherwise.

>> I'm not saying your analogy would not help someone understand, but you
>> first have to explain why 0.3 is not treated as three tenths -- why I
>> (to use your analogy) must not keep 1/3 as a proper fraction, but I must
>> instead write it using a finite number of decimal digits.  Neither is,
>> in my view, obvious to the beginner.
>
> Try adding 1/3 + e; either you have to convert 1/3 to a decimal, or
> find a rational approximation for e (there aren't any really good ones
> but 193/71 seems promising - that's 2.7183, close enough) and then go
> to the work of rational addition. No, more likely you'll go for a
> finite number of decimal digits.

Of course, but I'm not sure how this adds to the discussion.  e is not
like three tenths.  Most programming languages give us a way to write
what looks like exactly three tenths, but most don't give us a way write
a literal that looks like it means e exactly.

-- 
Ben.

From framstag at rus.uni-stuttgart.de  Sat Nov 20 14:15:32 2021
From: framstag at rus.uni-stuttgart.de (Ulli Horlacher)
Date: Sat, 20 Nov 2021 19:15:32 +0000 (UTC)
Subject: getting source code line of error?
References: <sn845l$pc1$1@news2.informatik.uni-stuttgart.de>
 <CABbU2U8xXY29rA3GBCPOVkJak7M3Tm0QK5FdDerCSvxeO=oX2w@mail.gmail.com>
 <mailman.51.1637349272.19479.python-list@python.org>
 <traceback-20211119203816@ram.dialup.fu-berlin.de>
 <lineno-20211120195208@ram.dialup.fu-berlin.de>
Message-ID: <snbhgk$kqd$1@news2.informatik.uni-stuttgart.de>

Stefan Ram <ram at zedat.fu-berlin.de> wrote:
> ram at zedat.fu-berlin.de (Stefan Ram) writes:
> >except Exception as inst:
> >    print( traceback.format_exc() )
> 
>   More to the point of getting the line number:

As I wrote in my initial posting:
I already have the line number. I am looking for the source code line! 

So far I use:

      m = re.search(r'\n\s*(.+)\n.*\n$',traceback.format_exc())
      if m: print('%s %s' % (prefix,m.group(1)))

-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher at tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/

From avigross at verizon.net  Sat Nov 20 16:30:42 2021
From: avigross at verizon.net (Avi Gross)
Date: Sat, 20 Nov 2021 16:30:42 -0500
Subject: Unexpected behaviour of math.floor,
 round and int functions (rounding)
In-Reply-To: <87sfvri07m.fsf@bsb.me.uk>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
 <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>
 <mailman.63.1637360151.19479.python-list@python.org>
 <874k87k4a7.fsf@bsb.me.uk>
 <CAPTjJmohqdKMP4srxWQZmu4Xn0+M2t7J8UtL9CDtNQRuOZ8N0A@mail.gmail.com>
 <mailman.67.1637373077.19479.python-list@python.org>
 <87y25jik3i.fsf@bsb.me.uk>
 <CAPTjJmr-Sik-NEzm1m17uXUMXSf8ijT9x=BvX9PYSoW6x=Nf0w@mail.gmail.com>
 <mailman.68.1637385505.19479.python-list@python.org>
 <87sfvri07m.fsf@bsb.me.uk>
Message-ID: <03c901d7de55$df8dd640$9ea982c0$@verizon.net>

This discussion gets tiresome for some.

Mathematics is a pristine world that is NOT the real world. It handles
near-infinities fairly gracefully but many things in the real world break
down because our reality is not infinitely divisible and some parts are
neither contiguous nor fixed but in some sense wavy and probabilistic or
worse.

So in any computer, or computer language, we have realities to deal with
when someone asks for say the square root of 2 or other transcendental
numbers like pi or e or things like the sin(x) as often they are numbers
which in decimal require an infinite number of digits and in many cases do
not repeat. Something as simple as the fractions for 1/7, in decimal, has an
interesting repeating pattern but is otherwise infinite.

.142857142857142857 ... ->> 1/7
.285714285714285714 ... ->> 2/7
.428571 ...
.571428 ...
.714285 ...
.857142 ...

No matter how many bits you set aside, you cannot capture such numbers
exactly IN BASE 10.

You may be able to capture some such things in another base but then yet
others cannot be seen in various other bases. I suspect someone has
considered a data type that stores results in arbitrary bases and delays
evaluation as late as possible, but even those cannot handle many numbers.

So the reality is that most computer programming is ultimately BINARY as in
BASE 2. At some level almost anything is rounded and imprecise. About all we
want to guarantee is that any rounding or truncation done is as consistent
as possible so every time you ask for pi or the square root of 2, you get
the same result stored as bits. BUT if you ask a slightly different
question, why expect the same results? sqrt(2) operates on the number 2. But
sqrt(6*(1/3)) first evaluates 1/3 and stores it as bits then multiplies it
by the bit representation of 6 and stores a result which then is handed to
sqrt() and if the bits are not identical, there is no guarantee that the
result is identical.

I will say this. Python has perhaps an improved handling of large integers.
Many languages have an assortment of integer sizes you can use such as 16
bits or 32 or 64 and possibly many others including using 8 or 1bits for
limited cases. But for larger numbers, there is a problem where the result
overflows what can be shown in that many bits and the result either is seen
as an error or worse, as a smaller number where some of the overflow bits
are thrown away. Python has indefinite length integers that work fine. But
if I take a real number with the same value and do a similar operation, I
get what I consider a truncated result:

>>> 256**40
2135987035920910082395021706169552114602704522356652769947041607822219725780
640550022962086936576
>>> 256.0**40
2.13598703592091e+96

That is because Python has not chosen to implement a default floating point
method that allows larger storage formats that could preserve more digits.

Could we design a more flexible storage form? I suspect we could BUT it
would not solve certain problems. I mean Consider these two squarings:

>>> .123456789123456789 * .123456789123456789
0.015241578780673677
>>> 123456789123456789 * 123456789123456789
15241578780673678515622620750190521

Clearly a fuller answer to the first part, based on the second, is
.015241578780673678515622620750190521

So one way to implement such extended functionality might be to have an
object that has a storage of the decimal part of something as an extended
integer variation along with storage of other parts like the exponent. SOME
operations would then use the integer representation and then be converted
back as needed. But such an item would not conform to existing standards and
would not trivially be integrated everywhere a normal floating point is
expected and thus may be truncated in many cases or have to be converted
before use.

But even such an object faces a serious problem as asking for a fraction
like 1/7 might lead to an infinite regress as the computer keeps lengthening
the data representation indefinitely. It has to be terminated eventually and
some of the examples shown where the whole does not seem to be the same
when viewed several ways, would still show the anomalies some invoke.

Do note pure Mathematics is just as confusing at times. The number
.99999999... where the dot-dot-dot notation means go on forever, is
mathematically equivalent to the number 1 as is any infinite series that
asymptotically approaches 1 as in 

	1/2 + 1/4 + 1/8 + ... + 1/(2**N) + ...

It is not seen by many students how continually appending a 9 can ever be
the same as a number like 1.00000 since every single digit is always not a
match. But the mathematical theorems about limits are now well understood
and in the limit as N approaches infinity, the two come to mean the same
thing. 

Python is a tool. More specifically, it is a changing platform that hosts
many additional tools. For the moment the tools are built on bits which are
both very precise but also cannot finitely represent everything. Maybe if we
develop decent quantum computers and use QBITS, we might have a wider range
of what can be stored and used by programs using a language that can handle
the superpositions involved. But, I suspect even that kind of machine might
still not handle some infinities well as I suspect our real world, quantum
features and all, at some levels reduces to probabilities that at some level
are not exactly the same each time.

So, what should be stressed, and often is, is to use tools available that
let you compare numbers for being nearly equal. I have seen some where the
size in bits of the machine storage method is used  to determine if numbers
are equal until within a few bits of the end of the representation. But
other places should be noted such as a hill-climbing algorithm that is
looking for a peak or valley, cannot be expected to converge to exactly what
you want and you may settle for it getting close enough as in 0.1%, or stop
if it does not improve in the last hundred iterations. The Mathematics may
require a precise answer, such as a point around which the slope of a curve
is zero, but the algorithm, especially when used with storage of variables
limited to some precision, may not reliably be able to zero in on a much
better answer so again, not a test for actual equality is needed, but one
close enough to likely be good enough.

I note how unamused I was when making a small table in EXCEL (Note, not
Python) of credit card numbers and balances when I saw the darn credit card
numbers were too long and a number like:

4195032150199578 

was displayed by EXCEL as:

4195032150199570

It looks like I just missed having significant stored digits and EXCEL
reconstructed it by filling in a zero for the missing extra. The problem is
I had to check balances sometimes and copy/paste generated the wrong number
to use. I ended up storing the number as text using '4195032150199578 as I
was not doing anything mathematical with it and this allowed me to keep all
the digits as text strings can be quite long.

But does this mean EXCEL is useless (albeit some thing so) or that the tool
can only be used up to some extent and beyond that, can (silently) mislead
you?

Having said all that, this reminds me a bit about the Y2K issues where
somehow nobody thought much about what happens when the year 2000 arrives
and someone 103 years old becomes 3 again as only the final two digits of
the year are stored. We now have the ability to make computers with
increased speed and memory and so on and I wonder if anyone has tried to
make a standard for say a 256-byte storage for multiple-precision floating
point that holds lots more digits of precision as well as allowing truly
huge exponents. Of course, it may not be practical to have computers that
have registers and circuitry that can multiply two such numbers in a very
few cycles, and it may be done in stages in thousands of cycles, so use of
something big like that might not be a good default.



From rosuav at gmail.com  Sat Nov 20 17:04:31 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 21 Nov 2021 09:04:31 +1100
Subject: Unexpected behaviour of math.floor,
 round and int functions (rounding)
In-Reply-To: <87sfvri07m.fsf@bsb.me.uk>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
 <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>
 <mailman.63.1637360151.19479.python-list@python.org>
 <874k87k4a7.fsf@bsb.me.uk>
 <CAPTjJmohqdKMP4srxWQZmu4Xn0+M2t7J8UtL9CDtNQRuOZ8N0A@mail.gmail.com>
 <mailman.67.1637373077.19479.python-list@python.org>
 <87y25jik3i.fsf@bsb.me.uk>
 <CAPTjJmr-Sik-NEzm1m17uXUMXSf8ijT9x=BvX9PYSoW6x=Nf0w@mail.gmail.com>
 <mailman.68.1637385505.19479.python-list@python.org>
 <87sfvri07m.fsf@bsb.me.uk>
Message-ID: <CAPTjJmpoMsyYT+YjZunYvLtPdikWrSfXtDSiQcLe+7-rvTeG+A@mail.gmail.com>

On Sun, Nov 21, 2021 at 6:51 AM Ben Bacarisse <ben.usenet at bsb.me.uk> wrote:
>
> Chris Angelico <rosuav at gmail.com> writes:
>
> > On Sat, Nov 20, 2021 at 3:41 PM Ben Bacarisse <ben.usenet at bsb.me.uk> wrote:
> >>
> >> Chris Angelico <rosuav at gmail.com> writes:
> >>
> >> > On Sat, Nov 20, 2021 at 12:43 PM Ben Bacarisse <ben.usenet at bsb.me.uk> wrote:
> >> >>
> >> >> Chris Angelico <rosuav at gmail.com> writes:
> >> >>
> >> >> > On Sat, Nov 20, 2021 at 9:07 AM Ben Bacarisse <ben.usenet at bsb.me.uk> wrote:
> >> >> >>
> >> >> >> Chris Angelico <rosuav at gmail.com> writes:
> >> >> >>
> >> >> >> > On Sat, Nov 20, 2021 at 5:08 AM ast <ast at invalid> wrote:
> >> >> >>
> >> >> >> >>  >>> 0.3 + 0.3 + 0.3 == 0.9
> >> >> >> >> False
> >> >> >> >
> >> >> >> > That's because 0.3 is not 3/10. It's not because floats are
> >> >> >> > "unreliable" or "inaccurate". It's because the ones you're entering
> >> >> >> > are not what you think they are.
> >> >> >> >
> >> >> >> > When will people understand this?
> >> >> >> >
> >> >> >> > (Probably never. Sigh.)
> >> >> >>
> >> >> >> Most people understand what's going on when it's explained to them.  And
> >> >> >> I think that being initially baffled is not unreasonable.  After all,
> >> >> >> almost everyone comes to computers after learning that 3/10 can be
> >> >> >> written as 0.3.  And Python "plays along" with the fiction to some
> >> >> >> extent.  0.3 prints as 0.3, 3/10 prints as 0.3 and 0.3 == 3/10 is True.
> >> >> >
> >> >> > In grade school, we learn that not everything can be written that way,
> >> >> > and 1/3 isn't actually equal to 0.3333333333.
> >> >>
> >> >> Yes.  We learn early on that 0.3333333333 means 3333333333/10000000000.
> >> >> We don't learn that 0.3333333333 is a special notation for machines that
> >> >> have something called "binary floating point hardware" that does not
> >> >> mean 3333333333/10000000000.  That has to be learned later.  And every
> >> >> generation has to learn it afresh.
> >> >
> >> > But you learn that it isn't the same as 1/3. That's my point. You
> >> > already understand that it is *impossible* to write out 1/3 in
> >> > decimal. Is it such a stretch to discover that you cannot write 3/10
> >> > in binary?
> >> >
> >> > Every generation has to learn about repeating fractions, but most of
> >> > us learn them in grade school. Every generation learns that computers
> >> > talk in binary. Yet, putting those two concepts together seems beyond
> >> > many people, to the point that they feel that floating point can't be
> >> > trusted.
> >>
> >> Binary is a bit of a red herring here.  It's the floating point format
> >> that needs to be understood.  Three tenths can be represented in many
> >> binary formats, and even decimal floating point will have some surprises
> >> for the novice.
> >
> > Not completely a red herring; binary floating-point as used in Python
> > (IEEE double-precision) is defined as a binary mantissa and a scale,
> > just as "blackboard arithmetic" is generally defined as a decimal
> > mantissa and a scale. (At least, I don't think I've ever seen anyone
> > doing arithmetic on a blackboard in hex or octal.)
>
> You seem to be agreeing with me.  It's the floating point part that is
> the issue, not the base itself.

Mostly, but all the problems come from people expecting decimal floats
when they're using binary floats.

> >> >> Yes, agreed, but I was not commenting on the odd (and incorrect) view
> >> >> that floating point operations are not reliable and well-defined, but on
> >> >> the reasonable assumption that a clever programming language might take
> >> >> 0.3 to mean what I was taught it meant in grade school.
> >> >
> >> > It does mean exactly what it meant in grade school, just as 1/3 means
> >> > exactly what it meant in grade school. Now try to represent 1/3 on a
> >> > blackboard, as a decimal fraction. If that's impossible, does it mean
> >> > that 1/3 doesn't mean 1/3, or that 1/3 can't be represented?
> >>
> >> As you know, it is possible, but let's say we outlaw any finite notation
> >> for repeated digits...  Why should I convert 1/3 to this particular
> >> apparently unsuitable representation?  I will write 1/3 and manipulate
> >> that number using factional notation.
> >
> > If you want that, the fractions module is there for you.
>
> Yes, I know.  The only point of disagreement (as far as can see) is
> that literals like 0.3 appears to be confusing for beginners.  You think
> they should know that "binary" (which may be all they know about
> computers and numbers) means fixed-width binary floating point (or at
> least might imply a format that can't represent three tenths), where I
> think it's not unreasonable for them to suppose that 0.3 is manipulated
> as the rational number it so clearly is.

Rationals are mostly irrelevant. We don't use int/int for most
purposes. When you're comparing number systems between the way people
write them and the way computers do, the difference isn't "0.3" and
"3/10". If people are prepared to switch their thinking to rationals
instead of decimals, then sure, the computer can represent those
precisely; but that has different tradeoffs.

> > And again,
> > grade school, we learned about ratios as well as decimals (or vulgar
> > fractions and decimal fractions). They have different tradeoffs. For
> > instance, I learned pi as both 22/7 and 3.14, because sometimes it'd
> > be convenient to use the rational form and other times the decimal.
> >
> >> The novice programmer might similarly expect that when they write 0.3,
> >> the program will manipulate that number as the faction it clearly is.
> >> They may well be surprised by the fact that it must get put into a
> >> format that can't represent what those three characters mean, just as I
> >> would be surprised if you insisted I write 1/3 as a finite decimal (with
> >> no repeat notation).
> >
> > Except that 0.3 isn't written as a fraction, it's written as a
> > decimal.
>
> Are you worrying about the term I used?  0.3 is a rational number.  It
> is the way to write a particular fraction in decimal.  I think the only
> point of disagreement is that you hope or expect that anyone writing
> three tenths as 0.3 should expect that that literal might be converted
> to some collection of bits that can't represent it.  I think it's not
> unreasonable for a beginner to think otherwise.

0.3 is a "rational number" but it's not being written as a fraction.
And most people aren't going to think of it as "3/10". The problem
here isn't mathematics, it's people, so the point isn't whether a
number could be considered rational in the abstract.

> >> I'm not saying your analogy would not help someone understand, but you
> >> first have to explain why 0.3 is not treated as three tenths -- why I
> >> (to use your analogy) must not keep 1/3 as a proper fraction, but I must
> >> instead write it using a finite number of decimal digits.  Neither is,
> >> in my view, obvious to the beginner.
> >
> > Try adding 1/3 + e; either you have to convert 1/3 to a decimal, or
> > find a rational approximation for e (there aren't any really good ones
> > but 193/71 seems promising - that's 2.7183, close enough) and then go
> > to the work of rational addition. No, more likely you'll go for a
> > finite number of decimal digits.
>
> Of course, but I'm not sure how this adds to the discussion.  e is not
> like three tenths.  Most programming languages give us a way to write
> what looks like exactly three tenths, but most don't give us a way write
> a literal that looks like it means e exactly.
>

That's, once again, because people. People want a way to say "give me
a value that's as close as possible to 0.3" and so that's what we get.
And then they get caught out by the fact that it isn't what they want
it to be.

ChrisA

From rosuav at gmail.com  Sat Nov 20 17:17:09 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 21 Nov 2021 09:17:09 +1100
Subject: Unexpected behaviour of math.floor,
 round and int functions (rounding)
In-Reply-To: <03c901d7de55$df8dd640$9ea982c0$@verizon.net>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
 <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>
 <mailman.63.1637360151.19479.python-list@python.org>
 <874k87k4a7.fsf@bsb.me.uk>
 <CAPTjJmohqdKMP4srxWQZmu4Xn0+M2t7J8UtL9CDtNQRuOZ8N0A@mail.gmail.com>
 <mailman.67.1637373077.19479.python-list@python.org>
 <87y25jik3i.fsf@bsb.me.uk>
 <CAPTjJmr-Sik-NEzm1m17uXUMXSf8ijT9x=BvX9PYSoW6x=Nf0w@mail.gmail.com>
 <mailman.68.1637385505.19479.python-list@python.org>
 <87sfvri07m.fsf@bsb.me.uk>
 <03c901d7de55$df8dd640$9ea982c0$@verizon.net>
Message-ID: <CAPTjJmp2SqZayWrRguqRi-o-_4dHnwptwgmbpF0N7tvk7HOK9g@mail.gmail.com>

On Sun, Nov 21, 2021 at 8:32 AM Avi Gross via Python-list
<python-list at python.org> wrote:
>
> This discussion gets tiresome for some.
>
> Mathematics is a pristine world that is NOT the real world. It handles
> near-infinities fairly gracefully but many things in the real world break
> down because our reality is not infinitely divisible and some parts are
> neither contiguous nor fixed but in some sense wavy and probabilistic or
> worse.

But the purity of mathematics isn't the problem. The problem is
people's expectations around computers. (The problem is ALWAYS
people's expectations.)

> So in any computer, or computer language, we have realities to deal with
> when someone asks for say the square root of 2 or other transcendental
> numbers like pi or e or things like the sin(x) as often they are numbers
> which in decimal require an infinite number of digits and in many cases do
> not repeat. Something as simple as the fractions for 1/7, in decimal, has an
> interesting repeating pattern but is otherwise infinite.
>
> .142857142857142857 ... ->> 1/7
> .285714285714285714 ... ->> 2/7
> .428571 ...
> .571428 ...
> .714285 ...
> .857142 ...
>
> No matter how many bits you set aside, you cannot capture such numbers
> exactly IN BASE 10.

Right, and people understand this. Yet as soon as you switch from base
10 to base 2, it becomes impossible for people to understand that 1/5
now becomes the exact same thing: an infinitely repeating expansion
for the rational number.

> You may be able to capture some such things in another base but then yet
> others cannot be seen in various other bases. I suspect someone has
> considered a data type that stores results in arbitrary bases and delays
> evaluation as late as possible, but even those cannot handle many numbers.

More likely it would just store rationals as rationals - or, in other
words, fractions.Fraction().

> So the reality is that most computer programming is ultimately BINARY as in
> BASE 2. At some level almost anything is rounded and imprecise. About all we
> want to guarantee is that any rounding or truncation done is as consistent
> as possible so every time you ask for pi or the square root of 2, you get
> the same result stored as bits. BUT if you ask a slightly different
> question, why expect the same results? sqrt(2) operates on the number 2. But
> sqrt(6*(1/3)) first evaluates 1/3 and stores it as bits then multiplies it
> by the bit representation of 6 and stores a result which then is handed to
> sqrt() and if the bits are not identical, there is no guarantee that the
> result is identical.

This is what I take issue with. Binary doesn't mean "rounded and
imprecise". It means "base two". People get stroppy at a computer's
inability to represent 0.3 correctly, because they think that it
should be perfectly obvious what that value is. Nobody's bothered by
sqrt(2) not being precise, but they're very much bothered by 1/10 not
"working".

> Do note pure Mathematics is just as confusing at times. The number
> .99999999... where the dot-dot-dot notation means go on forever, is
> mathematically equivalent to the number 1 as is any infinite series that
> asymptotically approaches 1 as in
>
>         1/2 + 1/4 + 1/8 + ... + 1/(2**N) + ...
>
> It is not seen by many students how continually appending a 9 can ever be
> the same as a number like 1.00000 since every single digit is always not a
> match. But the mathematical theorems about limits are now well understood
> and in the limit as N approaches infinity, the two come to mean the same
> thing.

Mathematics is confusing. That's not a problem. To be quite frank, the
real world is far more confusing than the pristine beauty that we have
inside a computer. The problem isn't the difference between reality
and mathematics, or between reality and computers, or anything like
that; the problem, as always, is between people's expectations and
what computers do.

Tell me: if a is equal to b and b is equal to c, is a equal to c?
Mathematicians say "of course it is". Engineers say "there's no way
you can rely on that". Computer programmers side with whoever makes
most sense right this instant.

> So, what should be stressed, and often is, is to use tools available that
> let you compare numbers for being nearly equal.

No. No no no no no. You don't need to use a "nearly equal" comparison
just because floats are "inaccurate". It isn't like that. It's this
exact misinformation that I am trying to fight, because floats are NOT
inaccurate. They're just in binary, same as everything that computers
do.

> I note how unamused I was when making a small table in EXCEL (Note, not
> Python) of credit card numbers and balances when I saw the darn credit card
> numbers were too long and a number like:
>
> 4195032150199578
>
> was displayed by EXCEL as:
>
> 4195032150199570
>
> It looks like I just missed having significant stored digits and EXCEL
> reconstructed it by filling in a zero for the missing extra. The problem is
> I had to check balances sometimes and copy/paste generated the wrong number
> to use. I ended up storing the number as text using '4195032150199578 as I
> was not doing anything mathematical with it and this allowed me to keep all
> the digits as text strings can be quite long.
>
> But does this mean EXCEL is useless (albeit some thing so) or that the tool
> can only be used up to some extent and beyond that, can (silently) mislead
> you?

Oh, Excel is moronic in plenty of other ways.

https://www.youtube.com/watch?v=yb2zkxHDfUE

> Having said all that, this reminds me a bit about the Y2K issues where
> somehow nobody thought much about what happens when the year 2000 arrives
> and someone 103 years old becomes 3 again as only the final two digits of
> the year are stored. We now have the ability to make computers with
> increased speed and memory and so on and I wonder if anyone has tried to
> make a standard for say a 256-byte storage for multiple-precision floating
> point that holds lots more digits of precision as well as allowing truly
> huge exponents. Of course, it may not be practical to have computers that
> have registers and circuitry that can multiply two such numbers in a very
> few cycles, and it may be done in stages in thousands of cycles, so use of
> something big like that might not be a good default.
>

Yes, you could use 80-bit floats, 128-bit floats, or 256-bit floats,
but that won't change the fact that 0.3 can't be represented precisely
in binary, nor will it change the fact that 0.5 *can*. If people can't
think in binary, they won't think in binary with more bits either.

ChrisA

From grant.b.edwards at gmail.com  Sat Nov 20 17:21:21 2021
From: grant.b.edwards at gmail.com (Grant Edwards)
Date: Sat, 20 Nov 2021 14:21:21 -0800 (PST)
Subject: Unexpected behaviour of math.floor, round and int functions
 (rounding)
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
 <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>
 <mailman.63.1637360151.19479.python-list@python.org>
 <874k87k4a7.fsf@bsb.me.uk>
 <CAPTjJmohqdKMP4srxWQZmu4Xn0+M2t7J8UtL9CDtNQRuOZ8N0A@mail.gmail.com>
Message-ID: <619974e1.1c69fb81.b7d99.bd43@mx.google.com>

On 2021-11-20, Chris Angelico <rosuav at gmail.com> wrote:

> But you learn that it isn't the same as 1/3. That's my point. You
> already understand that it is *impossible* to write out 1/3 in
> decimal. Is it such a stretch to discover that you cannot write 3/10
> in binary?

For many people, it seems to be.

There are plenty of people trying to write code who don't even under
the concept of different bases.

I remember trying to explain the concept of CPU registers, stacks,
interrupts, and binary representations to VAX/VMS FORTRAN programmers
and getting absolutely nowhere.

Years later, I went through the same exercise with a bunch of Windows
C++ programmers, and they seemed similarly baffled.

Perhaps I was just a bad teacher.
 
--
Grant


From grant.b.edwards at gmail.com  Sat Nov 20 17:23:37 2021
From: grant.b.edwards at gmail.com (Grant Edwards)
Date: Sat, 20 Nov 2021 14:23:37 -0800 (PST)
Subject: Unexpected behaviour of math.floor, round and int functions
 (rounding)
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
 <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>
 <mailman.63.1637360151.19479.python-list@python.org>
 <874k87k4a7.fsf@bsb.me.uk>
 <CAPTjJmohqdKMP4srxWQZmu4Xn0+M2t7J8UtL9CDtNQRuOZ8N0A@mail.gmail.com>
 <mailman.67.1637373077.19479.python-list@python.org>
 <87y25jik3i.fsf@bsb.me.uk>
 <CAPTjJmr-Sik-NEzm1m17uXUMXSf8ijT9x=BvX9PYSoW6x=Nf0w@mail.gmail.com>
 <mailman.68.1637385505.19479.python-list@python.org>
 <87sfvri07m.fsf@bsb.me.uk>
Message-ID: <61997569.1c69fb81.5eb3c.b508@mx.google.com>

On 2021-11-20, Ben Bacarisse <ben.usenet at bsb.me.uk> wrote:

> You seem to be agreeing with me.  It's the floating point part that is
> the issue, not the base itself.

No, it's the base. Floating point can't represent 3/10 _because_ it's
base 2 floating point. Floating point in base 10 doesn't have any
problem representing 3/10.

--
Grant

From rosuav at gmail.com  Sat Nov 20 17:27:24 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 21 Nov 2021 09:27:24 +1100
Subject: Unexpected behaviour of math.floor,
 round and int functions (rounding)
In-Reply-To: <619974e1.1c69fb81.b7d99.bd43@mx.google.com>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
 <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>
 <mailman.63.1637360151.19479.python-list@python.org>
 <874k87k4a7.fsf@bsb.me.uk>
 <CAPTjJmohqdKMP4srxWQZmu4Xn0+M2t7J8UtL9CDtNQRuOZ8N0A@mail.gmail.com>
 <619974e1.1c69fb81.b7d99.bd43@mx.google.com>
Message-ID: <CAPTjJmrW=UG81OH=HJqqyPFNTZXW4EdXa-HySg18Cu4rQruw3A@mail.gmail.com>

On Sun, Nov 21, 2021 at 9:22 AM Grant Edwards <grant.b.edwards at gmail.com> wrote:
>
> On 2021-11-20, Chris Angelico <rosuav at gmail.com> wrote:
>
> > But you learn that it isn't the same as 1/3. That's my point. You
> > already understand that it is *impossible* to write out 1/3 in
> > decimal. Is it such a stretch to discover that you cannot write 3/10
> > in binary?
>
> For many people, it seems to be.
>
> There are plenty of people trying to write code who don't even under
> the concept of different bases.
>
> I remember trying to explain the concept of CPU registers, stacks,
> interrupts, and binary representations to VAX/VMS FORTRAN programmers
> and getting absolutely nowhere.
>
> Years later, I went through the same exercise with a bunch of Windows
> C++ programmers, and they seemed similarly baffled.
>
> Perhaps I was just a bad teacher.
>

And to some extent, that's not really surprising; not everyone can
think the way other people do, and not everyone can think the way
computers do. But it seems that, in this one specific case, there's a
massive tendency to (a) misunderstand, and then (b) belligerently
assume that the computer acts the way they want it to act. And then
sometimes (c) get really annoyed at the computer for not being a
person, and start the cargo cult practice of "always use a
nearly-equal function instead of testing for equality", which we've
seen in this exact thread.

That's what I take issue with: the smug "0.1 + 0.2 != 0.3, therefore
computers are wrong" people, and the extremely unhelpful "never use ==
with floats" people.

ChrisA

From avigross at verizon.net  Sat Nov 20 17:59:43 2021
From: avigross at verizon.net (Avi Gross)
Date: Sat, 20 Nov 2021 17:59:43 -0500
Subject: Unexpected behaviour of math.floor,
 round and int functions (rounding)
In-Reply-To: <CAPTjJmp2SqZayWrRguqRi-o-_4dHnwptwgmbpF0N7tvk7HOK9g@mail.gmail.com>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
 <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>
 <mailman.63.1637360151.19479.python-list@python.org>
 <874k87k4a7.fsf@bsb.me.uk>
 <CAPTjJmohqdKMP4srxWQZmu4Xn0+M2t7J8UtL9CDtNQRuOZ8N0A@mail.gmail.com>
 <mailman.67.1637373077.19479.python-list@python.org>
 <87y25jik3i.fsf@bsb.me.uk>
 <CAPTjJmr-Sik-NEzm1m17uXUMXSf8ijT9x=BvX9PYSoW6x=Nf0w@mail.gmail.com>
 <mailman.68.1637385505.19479.python-list@python.org>
 <87sfvri07m.fsf@bsb.me.uk> <03c901d7de55$df8dd640$9ea982c0$@verizon.net>
 <CAPTjJmp2SqZayWrRguqRi-o-_4dHnwptwgmbpF0N7tvk7HOK9g@mail.gmail.com>
Message-ID: <03f601d7de62$4eac6080$ec052180$@verizon.net>

Chris,

I generally agree with your comments albeit I might take a different slant.

What I meant is that people who learn mathematics (as I and many here
obviously did) can come away with idealized ideas that they then expect to
be replicable everywhere. But there are grey lines along the way where some
mathematical proofs do weird things like IGNORE parts of a calculation by
suggesting they are going to zero much faster than other parts and then wave
a mathematical wand about what happens when they approach a limit like zero
and voila, we just "proved" that the derivative of X**2 is 2*X or the more
general derivative of A*(X**N) is N*A*(X**(N-1)) and then extend that to N
being negative or fractional or a transcendental number and beyond.

Computers generally use finite methods, sometimes too finite. Yes, the
problem is not Mathematics as a field. It is how humans often generalize or
analogize from one area into something a bit different. I do not agree with
any suggestion that a series of bits that encodes a result that is rounded
or truncated is CORRECT. A representation of 0.3 in a binary version of some
floating point format is not technically correct. Storing it as 3/10 and
carefully later multiplying it by 20 and then carefully canceling part will
result in exactly 6. While storing it digitally and then multiplying it in
registers or whatever by 20 may get a result slightly different than the
storage representation of 6.0000000000... and that is a fact and risk we
generally are willing to take.

But consider a different example. If I have a filesystem or URL or anything
that does not care about whether parts are in upper or lower case, then
"filename" and "FILENAME" and many variations like "fIlEnAmE" are all
assumed to mean the same thing. A program may even simply store all of them
in the same way as all uppercase. But when you ask to compare two versions
with a function where case matters, they all test as unequal! So there are
ways to ask for a comparison that is approximately equal given the
constraints that case does not matter:

>>> alpha="Hello"
>>> beta="hELLO"
>>> alpha == beta
False
>>> alpha.lower() == beta.lower()
True

I see no reason why a comparison canot be done like this in cases you are
concerned with small errors creeping in:

>>> from math import isclose
>>> isclose(1, .9999999999999999999999)
True
>>> isclose(1, .9999999999)
True
>>> isclose(1, .999)
False

I will agree with you that binary is not any more imprecise than base 10.
Computer hardware is much easier to design though that works with binary. 

So floats by themselves are not inaccurate but realistically the results of
operations ARE. I mean if I ask a long number to be stored that does not
fully fit, it is often silently truncated and what the storage location now
represent accurately is not my number but the shorter version that is at the
limit of tolerance. But consider another analogy often encountered in
mathematics.

If I measure several numbers in the real world such as weight and height and
temperature and so on, some are considered accurate only to a limited number
of digits. Your weight on a standard digital scale may well be 189.8 but if
I add a feather or subtract one, the reading may well shift to one unit up
or down. Heck, the same person measured just minutes later may shift. If I
used a deluxe scale that measures to more decimal places, it may get hard to
get the exact same number twice in a row as just taking a deeper breath may
make a change. 

So what happens if I measure a box in three dimensions to the nearest .1
inch and decide it is 10.1 by 20.2 by 30.3 inches? What is the volume,
ignoring pesky details about the width of the cardboard or whatever?

A straightforward multiplication yields 4141.606 cubic inches. You may have
been told to round that to something like 4141.6 because the potential error
in each measure cannot result in more precision. In reality, you might even
calculate two sets of numbers assuming the true width may have been a tad
more or less and come up with the volume being BETWEEN a somewhat smaller
number and a somewhat larger number.

I claim a similar issue plagues using a computer to deal with stored
numbers, perhaps not stored 100% perfectly as discussed, and doing
calculations. The result often comes out more precisely than warranted. I
suspect there are modules out there that might do multi-step calculations
where at each step, numbers generated with extra precision are throttled
back so the extra precision is set to zeroes after rounding to avoid the
small increments adding up. Others may just do the calculations and keep
track and remove extra precision at the end.

And again, this is not because the implementation of numbers is in any way
wrong but because a real-world situation requires the humans to sort of dial
back how they are used and not over-reach.

So comparing for close-enough inequality is not necessarily a reflection on
floats but on the design not accommodating the precision needed or perhaps
on the algorithm used not necessarily being expected to reach a certain
level.



-----Original Message-----
From: Python-list <python-list-bounces+avigross=verizon.net at python.org> On
Behalf Of Chris Angelico
Sent: Saturday, November 20, 2021 5:17 PM
To: python-list at python.org
Subject: Re: Unexpected behaviour of math.floor, round and int functions
(rounding)

On Sun, Nov 21, 2021 at 8:32 AM Avi Gross via Python-list
<python-list at python.org> wrote:
>
> This discussion gets tiresome for some.
>
> Mathematics is a pristine world that is NOT the real world. It handles 
> near-infinities fairly gracefully but many things in the real world 
> break down because our reality is not infinitely divisible and some 
> parts are neither contiguous nor fixed but in some sense wavy and 
> probabilistic or worse.

But the purity of mathematics isn't the problem. The problem is people's
expectations around computers. (The problem is ALWAYS people's
expectations.)

> So in any computer, or computer language, we have realities to deal 
> with when someone asks for say the square root of 2 or other 
> transcendental numbers like pi or e or things like the sin(x) as often 
> they are numbers which in decimal require an infinite number of digits 
> and in many cases do not repeat. Something as simple as the fractions 
> for 1/7, in decimal, has an interesting repeating pattern but is otherwise
infinite.
>
> .142857142857142857 ... ->> 1/7
> .285714285714285714 ... ->> 2/7
> .428571 ...
> .571428 ...
> .714285 ...
> .857142 ...
>
> No matter how many bits you set aside, you cannot capture such numbers 
> exactly IN BASE 10.

Right, and people understand this. Yet as soon as you switch from base
10 to base 2, it becomes impossible for people to understand that 1/5 now
becomes the exact same thing: an infinitely repeating expansion for the
rational number.

> You may be able to capture some such things in another base but then 
> yet others cannot be seen in various other bases. I suspect someone 
> has considered a data type that stores results in arbitrary bases and 
> delays evaluation as late as possible, but even those cannot handle many
numbers.

More likely it would just store rationals as rationals - or, in other words,
fractions.Fraction().

> So the reality is that most computer programming is ultimately BINARY 
> as in BASE 2. At some level almost anything is rounded and imprecise. 
> About all we want to guarantee is that any rounding or truncation done 
> is as consistent as possible so every time you ask for pi or the 
> square root of 2, you get the same result stored as bits. BUT if you 
> ask a slightly different question, why expect the same results? 
> sqrt(2) operates on the number 2. But
> sqrt(6*(1/3)) first evaluates 1/3 and stores it as bits then 
> multiplies it by the bit representation of 6 and stores a result which 
> then is handed to
> sqrt() and if the bits are not identical, there is no guarantee that 
> the result is identical.

This is what I take issue with. Binary doesn't mean "rounded and imprecise".
It means "base two". People get stroppy at a computer's inability to
represent 0.3 correctly, because they think that it should be perfectly
obvious what that value is. Nobody's bothered by
sqrt(2) not being precise, but they're very much bothered by 1/10 not
"working".

> Do note pure Mathematics is just as confusing at times. The number 
> .99999999... where the dot-dot-dot notation means go on forever, is 
> mathematically equivalent to the number 1 as is any infinite series 
> that asymptotically approaches 1 as in
>
>         1/2 + 1/4 + 1/8 + ... + 1/(2**N) + ...
>
> It is not seen by many students how continually appending a 9 can ever 
> be the same as a number like 1.00000 since every single digit is 
> always not a match. But the mathematical theorems about limits are now 
> well understood and in the limit as N approaches infinity, the two 
> come to mean the same thing.

Mathematics is confusing. That's not a problem. To be quite frank, the real
world is far more confusing than the pristine beauty that we have inside a
computer. The problem isn't the difference between reality and mathematics,
or between reality and computers, or anything like that; the problem, as
always, is between people's expectations and what computers do.

Tell me: if a is equal to b and b is equal to c, is a equal to c?
Mathematicians say "of course it is". Engineers say "there's no way you can
rely on that". Computer programmers side with whoever makes most sense right
this instant.

> So, what should be stressed, and often is, is to use tools available 
> that let you compare numbers for being nearly equal.

No. No no no no no. You don't need to use a "nearly equal" comparison just
because floats are "inaccurate". It isn't like that. It's this exact
misinformation that I am trying to fight, because floats are NOT inaccurate.
They're just in binary, same as everything that computers do.

> I note how unamused I was when making a small table in EXCEL (Note, 
> not
> Python) of credit card numbers and balances when I saw the darn credit 
> card numbers were too long and a number like:
>
> 4195032150199578
>
> was displayed by EXCEL as:
>
> 4195032150199570
>
> It looks like I just missed having significant stored digits and EXCEL 
> reconstructed it by filling in a zero for the missing extra. The 
> problem is I had to check balances sometimes and copy/paste generated 
> the wrong number to use. I ended up storing the number as text using 
> '4195032150199578 as I was not doing anything mathematical with it and 
> this allowed me to keep all the digits as text strings can be quite long.
>
> But does this mean EXCEL is useless (albeit some thing so) or that the 
> tool can only be used up to some extent and beyond that, can 
> (silently) mislead you?

Oh, Excel is moronic in plenty of other ways.

https://www.youtube.com/watch?v=yb2zkxHDfUE

> Having said all that, this reminds me a bit about the Y2K issues where 
> somehow nobody thought much about what happens when the year 2000 
> arrives and someone 103 years old becomes 3 again as only the final 
> two digits of the year are stored. We now have the ability to make 
> computers with increased speed and memory and so on and I wonder if 
> anyone has tried to make a standard for say a 256-byte storage for 
> multiple-precision floating point that holds lots more digits of 
> precision as well as allowing truly huge exponents. Of course, it may 
> not be practical to have computers that have registers and circuitry 
> that can multiply two such numbers in a very few cycles, and it may be 
> done in stages in thousands of cycles, so use of something big like that
might not be a good default.
>

Yes, you could use 80-bit floats, 128-bit floats, or 256-bit floats, but
that won't change the fact that 0.3 can't be represented precisely in
binary, nor will it change the fact that 0.5 *can*. If people can't think in
binary, they won't think in binary with more bits either.

ChrisA
--
https://mail.python.org/mailman/listinfo/python-list


From rob.cliffe at btinternet.com  Sat Nov 20 18:18:30 2021
From: rob.cliffe at btinternet.com (Rob Cliffe)
Date: Sat, 20 Nov 2021 23:18:30 +0000
Subject: Unexpected behaviour of math.floor, round and int functions
 (rounding)
In-Reply-To: <03f601d7de62$4eac6080$ec052180$@verizon.net>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
 <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>
 <mailman.63.1637360151.19479.python-list@python.org>
 <874k87k4a7.fsf@bsb.me.uk>
 <CAPTjJmohqdKMP4srxWQZmu4Xn0+M2t7J8UtL9CDtNQRuOZ8N0A@mail.gmail.com>
 <mailman.67.1637373077.19479.python-list@python.org>
 <87y25jik3i.fsf@bsb.me.uk>
 <CAPTjJmr-Sik-NEzm1m17uXUMXSf8ijT9x=BvX9PYSoW6x=Nf0w@mail.gmail.com>
 <mailman.68.1637385505.19479.python-list@python.org>
 <87sfvri07m.fsf@bsb.me.uk> <03c901d7de55$df8dd640$9ea982c0$@verizon.net>
 <CAPTjJmp2SqZayWrRguqRi-o-_4dHnwptwgmbpF0N7tvk7HOK9g@mail.gmail.com>
 <03f601d7de62$4eac6080$ec052180$@verizon.net>
Message-ID: <e9cde1af-7f36-1187-0a9f-2283db306a79@btinternet.com>



On 20/11/2021 22:59, Avi Gross via Python-list wrote:
> there are grey lines along the way where some
> mathematical proofs do weird things like IGNORE parts of a calculation by
> suggesting they are going to zero much faster than other parts and then wave
> a mathematical wand about what happens when they approach a limit like zero
> and voila, we just "proved" that the derivative of X**2 is 2*X or the more
> general derivative of A*(X**N) is N*A*(X**(N-1)) and then extend that to N
> being negative or fractional or a transcendental number and beyond.
>
>
 ??? You seem to be maligning mathematicians.
 ??? What you say was true in the time of Newton, Leibniz and Bishop 
Berkeley, but analysis was made completely rigorous by the efforts of 
Weierstrass and others.? There are no "grey lines".? Proofs do not 
"suggest", they PROVE (else they are not proofs, they are plain wrong).? 
It is not the fault of mathematicians (or mathematics) if some people 
produce sloppy hand-wavy "proofs" as justification for their conclusions.
 ??? I am absolutely sure you know all this, but your post does not read 
as if you do.? And it could give a mistaken impression to a 
non-mathematician.? I think we have had enough denigration of experts.
Best
Rob Cliffe




From rosuav at gmail.com  Sat Nov 20 18:22:38 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 21 Nov 2021 10:22:38 +1100
Subject: Unexpected behaviour of math.floor,
 round and int functions (rounding)
In-Reply-To: <03f601d7de62$4eac6080$ec052180$@verizon.net>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
 <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>
 <mailman.63.1637360151.19479.python-list@python.org>
 <874k87k4a7.fsf@bsb.me.uk>
 <CAPTjJmohqdKMP4srxWQZmu4Xn0+M2t7J8UtL9CDtNQRuOZ8N0A@mail.gmail.com>
 <mailman.67.1637373077.19479.python-list@python.org>
 <87y25jik3i.fsf@bsb.me.uk>
 <CAPTjJmr-Sik-NEzm1m17uXUMXSf8ijT9x=BvX9PYSoW6x=Nf0w@mail.gmail.com>
 <mailman.68.1637385505.19479.python-list@python.org>
 <87sfvri07m.fsf@bsb.me.uk>
 <03c901d7de55$df8dd640$9ea982c0$@verizon.net>
 <CAPTjJmp2SqZayWrRguqRi-o-_4dHnwptwgmbpF0N7tvk7HOK9g@mail.gmail.com>
 <03f601d7de62$4eac6080$ec052180$@verizon.net>
Message-ID: <CAPTjJmr-8ABoMOUVQTwJBD13SS8cc01615=2wr1jJe-A7PP1+A@mail.gmail.com>

On Sun, Nov 21, 2021 at 10:01 AM Avi Gross via Python-list
<python-list at python.org> wrote:
> Computers generally use finite methods, sometimes too finite. Yes, the
> problem is not Mathematics as a field. It is how humans often generalize or
> analogize from one area into something a bit different. I do not agree with
> any suggestion that a series of bits that encodes a result that is rounded
> or truncated is CORRECT. A representation of 0.3 in a binary version of some
> floating point format is not technically correct. Storing it as 3/10 and
> carefully later multiplying it by 20 and then carefully canceling part will
> result in exactly 6. While storing it digitally and then multiplying it in
> registers or whatever by 20 may get a result slightly different than the
> storage representation of 6.0000000000... and that is a fact and risk we
> generally are willing to take.

Do you accept that storing the floating point value 1/4, then
multiplying by 20, will give precisely 5? Because that is
*guaranteed*. You don't have to expect a result "slightly different"
from 5, it will be absolutely exactly five:

>>> (1/4) * 20 == 5.0
True

This is what I'm talking about. Some numbers can be represented
perfectly, others can't. If you try to represent the square root of
two as a decimal number, then multiply it by itself, you won't get
back precisely 2, because you can't have written out the *exact*
square root of two. But you most certainly CAN write "1.875" on a
piece of paper, and it really truly does exactly mean fifteen eighths.
And you can write that number as a binary float, too, and it'll mean
the exact same value.

> But consider a different example. If I have a filesystem or URL or anything
> that does not care about whether parts are in upper or lower case, then
> "filename" and "FILENAME" and many variations like "fIlEnAmE" are all
> assumed to mean the same thing. A program may even simply store all of them
> in the same way as all uppercase. But when you ask to compare two versions
> with a function where case matters, they all test as unequal! So there are
> ways to ask for a comparison that is approximately equal given the
> constraints that case does not matter:

A URL has distinct parts to it: the domain has some precise folding
done (most notably case folding), the path does not, and you can
consider "http://example.com:80/foo" to be the same as
"http://example.com/foo" because 80 is the default port.

> >>> alpha="Hello"
> >>> beta="hELLO"
> >>> alpha == beta
> False
> >>> alpha.lower() == beta.lower()
> True
>

That's a terrible way to compare URLs, because it's both too sloppy
AND too strict at the same time. But if you have a URL representation
tool, it should be able to consider two things equal.

Floats are representations of numbers that can be compared for
equality if they truly represent the same number. The value 3/6 is
precisely equal to the value 7/14:

>>> 3/6 == 7/14
True

You don't need an "approximately equal" function here. They are the
same value. They are equal.

> I see no reason why a comparison canot be done like this in cases you are
> concerned with small errors creeping in:
>
> >>> from math import isclose
> >>> isclose(1, .9999999999999999999999)
> True
> >>> isclose(1, .9999999999)
> True
> >>> isclose(1, .999)
> False

This is exactly the problem though: HOW close counts as equal? The
only way to answer that question is to know the accuracy of your
inputs, and the operations done.

> So floats by themselves are not inaccurate but realistically the results of
> operations ARE. I mean if I ask a long number to be stored that does not
> fully fit, it is often silently truncated and what the storage location now
> represent accurately is not my number but the shorter version that is at the
> limit of tolerance. But consider another analogy often encountered in
> mathematics.

Not true. Operations are often perfectly accurate.

> If I measure several numbers in the real world such as weight and height and
> temperature and so on, some are considered accurate only to a limited number
> of digits. Your weight on a standard digital scale may well be 189.8 but if
> I add a feather or subtract one, the reading may well shift to one unit up
> or down. Heck, the same person measured just minutes later may shift. If I
> used a deluxe scale that measures to more decimal places, it may get hard to
> get the exact same number twice in a row as just taking a deeper breath may
> make a change.
>
> So what happens if I measure a box in three dimensions to the nearest .1
> inch and decide it is 10.1 by 20.2 by 30.3 inches? What is the volume,
> ignoring pesky details about the width of the cardboard or whatever?
>
> A straightforward multiplication yields 4141.606 cubic inches. You may have
> been told to round that to something like 4141.6 because the potential error
> in each measure cannot result in more precision. In reality, you might even
> calculate two sets of numbers assuming the true width may have been a tad
> more or less and come up with the volume being BETWEEN a somewhat smaller
> number and a somewhat larger number.

If those initial figures were accurate to three digits, you should
round it to 4140 cubic inches, because that's all the accuracy you
have. (Or, if you prefer, 4140 +/- 5.)

> I claim a similar issue plagues using a computer to deal with stored
> numbers, perhaps not stored 100% perfectly as discussed, and doing
> calculations. The result often comes out more precisely than warranted. I
> suspect there are modules out there that might do multi-step calculations
> where at each step, numbers generated with extra precision are throttled
> back so the extra precision is set to zeroes after rounding to avoid the
> small increments adding up. Others may just do the calculations and keep
> track and remove extra precision at the end.

When your input values aren't accurate, your output won't be accurate.
That's something the computer can never know. When you store the
number 3602879701896397/36028797018963968, did you actually mean that
number, or did you mean some other number that's kinda close to it? If
you don't tell the computer, it's going to assume that you wanted
exactly that number.

> And again, this is not because the implementation of numbers is in any way
> wrong but because a real-world situation requires the humans to sort of dial
> back how they are used and not over-reach.
>
> So comparing for close-enough inequality is not necessarily a reflection on
> floats but on the design not accommodating the precision needed or perhaps
> on the algorithm used not necessarily being expected to reach a certain
> level.

And close-enough equality is the correct thing to do when you know
exactly what the accuracy of your inputs is. If you need to be
completely rigorous about it, you'd have to store every number as a
range (so you might say that your input length is "10.05 to 10.15" or
"10.1, error 0.5") and do all arithmetic on those ranges. What you'd
find is that some operations widen the ranges and others don't. The
trouble is, that's not actually all that useful; Fermi estimates are
far more accurate than they seem like they "should be" because the
balance of probability is in favour of errors cancelling out, at least
partially.

ChrisA

From ben.usenet at bsb.me.uk  Sat Nov 20 17:44:28 2021
From: ben.usenet at bsb.me.uk (Ben Bacarisse)
Date: Sat, 20 Nov 2021 22:44:28 +0000
Subject: Unexpected behaviour of math.floor,
 round and int functions (rounding)
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
 <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>
 <mailman.63.1637360151.19479.python-list@python.org>
 <874k87k4a7.fsf@bsb.me.uk>
 <CAPTjJmohqdKMP4srxWQZmu4Xn0+M2t7J8UtL9CDtNQRuOZ8N0A@mail.gmail.com>
 <mailman.67.1637373077.19479.python-list@python.org>
 <87y25jik3i.fsf@bsb.me.uk>
 <CAPTjJmr-Sik-NEzm1m17uXUMXSf8ijT9x=BvX9PYSoW6x=Nf0w@mail.gmail.com>
 <mailman.68.1637385505.19479.python-list@python.org>
 <87sfvri07m.fsf@bsb.me.uk>
 <61997569.1c69fb81.5eb3c.b508@mx.google.com>
 <mailman.81.1637447020.19479.python-list@python.org>
Message-ID: <87lf1ih2gj.fsf@bsb.me.uk>

Grant Edwards <grant.b.edwards at gmail.com> writes:

> On 2021-11-20, Ben Bacarisse <ben.usenet at bsb.me.uk> wrote:
>
>> You seem to be agreeing with me.  It's the floating point part that is
>> the issue, not the base itself.
>
> No, it's the base. Floating point can't represent 3/10 _because_ it's
> base 2 floating point. Floating point in base 10 doesn't have any
> problem representing 3/10.

Every base has the same problem for some numbers.  It's the floating
point part that causes the problem.

Binary and decimal stand out because we write a lot of decimals in
source code and computers use binary, but if decimal floating point were
common (as it increasingly is) different fractions would become the oft
quoted "surprise" results.

-- 
Ben.

From paolo.cantore at t-online.de  Sat Nov 20 18:20:02 2021
From: paolo.cantore at t-online.de (Paolo G. Cantore)
Date: Sun, 21 Nov 2021 00:20:02 +0100
Subject: getting source code line of error?
In-Reply-To: <snbhgk$kqd$1@news2.informatik.uni-stuttgart.de>
References: <sn845l$pc1$1@news2.informatik.uni-stuttgart.de>
 <CABbU2U8xXY29rA3GBCPOVkJak7M3Tm0QK5FdDerCSvxeO=oX2w@mail.gmail.com>
 <mailman.51.1637349272.19479.python-list@python.org>
 <traceback-20211119203816@ram.dialup.fu-berlin.de>
 <lineno-20211120195208@ram.dialup.fu-berlin.de>
 <snbhgk$kqd$1@news2.informatik.uni-stuttgart.de>
Message-ID: <snbvr3$j4m$1@dont-email.me>

Am 20.11.21 um 20:15 schrieb Ulli Horlacher:
> Stefan Ram <ram at zedat.fu-berlin.de> wrote:
>> ram at zedat.fu-berlin.de (Stefan Ram) writes:
>>> except Exception as inst:
>>>     print( traceback.format_exc() )
>>
>>    More to the point of getting the line number:
> 
> As I wrote in my initial posting:
> I already have the line number. I am looking for the source code line!
> 
> So far I use:
> 
>        m = re.search(r'\n\s*(.+)\n.*\n$',traceback.format_exc())
>        if m: print('%s %s' % (prefix,m.group(1)))
> 
Stefan Ram's solution missed only the line content. Here it is.


import sys
import traceback

try:
     1/0
except ZeroDivisionError as exception:
     tr = traceback.TracebackException.from_exception( exception )
     x = tr.stack[0]
     print("Exception %s in line %s: %s" % (exception, x.lineno, x.line))


The traceback object does not only contain the lineno but also the 
content of the offending line.

--------------
Paolo

From rosuav at gmail.com  Sat Nov 20 18:57:55 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 21 Nov 2021 10:57:55 +1100
Subject: Unexpected behaviour of math.floor,
 round and int functions (rounding)
In-Reply-To: <87lf1ih2gj.fsf@bsb.me.uk>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
 <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>
 <mailman.63.1637360151.19479.python-list@python.org>
 <874k87k4a7.fsf@bsb.me.uk>
 <CAPTjJmohqdKMP4srxWQZmu4Xn0+M2t7J8UtL9CDtNQRuOZ8N0A@mail.gmail.com>
 <mailman.67.1637373077.19479.python-list@python.org>
 <87y25jik3i.fsf@bsb.me.uk>
 <CAPTjJmr-Sik-NEzm1m17uXUMXSf8ijT9x=BvX9PYSoW6x=Nf0w@mail.gmail.com>
 <mailman.68.1637385505.19479.python-list@python.org>
 <87sfvri07m.fsf@bsb.me.uk>
 <61997569.1c69fb81.5eb3c.b508@mx.google.com>
 <mailman.81.1637447020.19479.python-list@python.org>
 <87lf1ih2gj.fsf@bsb.me.uk>
Message-ID: <CAPTjJmoUMyOB8NUB6xN1y6W0GUGkYKOS5gxTYSjsi+93+XuBwg@mail.gmail.com>

On Sun, Nov 21, 2021 at 10:55 AM Ben Bacarisse <ben.usenet at bsb.me.uk> wrote:
>
> Grant Edwards <grant.b.edwards at gmail.com> writes:
>
> > On 2021-11-20, Ben Bacarisse <ben.usenet at bsb.me.uk> wrote:
> >
> >> You seem to be agreeing with me.  It's the floating point part that is
> >> the issue, not the base itself.
> >
> > No, it's the base. Floating point can't represent 3/10 _because_ it's
> > base 2 floating point. Floating point in base 10 doesn't have any
> > problem representing 3/10.
>
> Every base has the same problem for some numbers.  It's the floating
> point part that causes the problem.
>
> Binary and decimal stand out because we write a lot of decimals in
> source code and computers use binary, but if decimal floating point were
> common (as it increasingly is) different fractions would become the oft
> quoted "surprise" results.
>

And if decimal floating point were common, other "surprise" behaviour
would be cited, like how x < y and (x+y)/2 < x.

ChrisA

From avigross at verizon.net  Sat Nov 20 19:38:20 2021
From: avigross at verizon.net (Avi Gross)
Date: Sat, 20 Nov 2021 19:38:20 -0500
Subject: Unexpected behaviour of math.floor,
 round and int functions (rounding)
In-Reply-To: <61997569.1c69fb81.5eb3c.b508@mx.google.com>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
 <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>
 <mailman.63.1637360151.19479.python-list@python.org>
 <874k87k4a7.fsf@bsb.me.uk>
 <CAPTjJmohqdKMP4srxWQZmu4Xn0+M2t7J8UtL9CDtNQRuOZ8N0A@mail.gmail.com>
 <mailman.67.1637373077.19479.python-list@python.org>
 <87y25jik3i.fsf@bsb.me.uk>
 <CAPTjJmr-Sik-NEzm1m17uXUMXSf8ijT9x=BvX9PYSoW6x=Nf0w@mail.gmail.com>
 <mailman.68.1637385505.19479.python-list@python.org>
 <87sfvri07m.fsf@bsb.me.uk> <61997569.1c69fb81.5eb3c.b508@mx.google.com>
Message-ID: <041a01d7de70$15b9a3b0$412ceb10$@verizon.net>

Can I suggest a way to look at it, Grant?

In base 10, we represent all numbers as the (possibly infinite) sum of ten
raised to some integral power.

123 is 3 times 1 (ten to the zero power) plus
2 times 10 (ten to the one power) plus
1 times 100 (ten to the two power)

123.456 just extends this with
4 times 1/10 (ten to the minus one power) plus
5 times 1/100 (10**-2) plus
6 time 1/1000 (10**-3)

In binary, all the powers are not powers of 10 but powers of two.

So IF you wrote something like 111 it means 1 times 1 plus 1 times 2 plus 1
times 4 or 7. A zero anywhere just skips a 2 to that power. If you added a
decimal point to make 111.111 the latter part would be 1/2 plus 1/4 plus 1/8
or 7/8 which combined might be 7 and 7/8. So any fractions of the form
something over 2**N can be made easily and almost everything else cannot be
made in finite stretches. How would you make 2/3 or 3 /10?

But the opposite is something true. In decimal, to make the above it becomes
7.875 and to make other fractions of the kind, you need more and more As it
happens, all such base-2 compatible streams can be made because each is in
some sense a divide by two.

7/16 = 1/2 * .875 = .4375
7/32 = 1/2 * .4375 = .21875

and so on. But this ability is a special case artifact caused by a terminal
digit 5 always being able to be divided in tow to make a 25 a unit longer
and then again and again. Note 2 and 5 are factors of 10.  In the more
general case, this fails. In base 7, 3/7 is written easily as 0.3 but the
same fraction in decimal is a repeating copy of .428571... which never
terminates. A number like 3/7 + 4/49 + 5/343 generally cannot be written in
base 7 but the opposite is also true that only a approximation of numbers in
base 2 or base 10 can ever be written. I am, of course, talking about the
part to the right of the decimal. Integers to the left can be written in any
base. It is fractional parts that can end up being nonrepeating.

What about pi and e and the square root of 2? I suspect all of them have an
infinite sequence with no real repetition (over long enough stretches) in
any base! I mean an integer base, of course. The constant e in base e is
just 1.

As has been hammered home, computers have generally always dealt in one or
more combined on/off or Boolean idea so deep down they tend to have binary
circuits. At one point, programmers sometimes used base 8, octal, to group
three binary digits together as in setting flags for a file's permissions,
may use 01, 02 and 04 to be OR'ed with the current value to turn on
read/write/execute bits, or a combination like 7 (1+2+4) to set all of them
at once. And, again, for some purposes, base 16 (hexadecimal) is often used
with numerals extended to include a-f to represent a nibble or half byte, as
in some programs that let you set colors or whatever. But they are just a
convenience as ultimately they are used as binary for most purposes. In high
school, for a while, and just for fun, I annoyed one teacher by doing much
of my math in base 32 leaving them very perplexed as to how I got the
answers right. As far as I know, nobody seriously uses any bases not already
a power of two even for intermediate steps, outside of some interesting
stuff in number theory.

I think there have been attempts to use a decimal representation in some
accounting packages or database applications that allow any decimal numbers
to be faithfully represented and used in calculations. Generally this is not
a very efficient process but it can handle 0.3 albeit still have no way to
deal with transcendental numbers.

As such, since this is a Python Forum let me add you can get limited support
for some of this using the decimal module:

https://www.askpython.com/python-modules/python-decimal-module

But I doubt Python can be said to do things worse than just about any other
computer language when storing and using floating point. As hammered in
repeatedly, it is doing whatever is allowed in binary and many things just
cannot easily or at all be done in binary.

Let me leave you with Egyptian mathematics. Their use of fractions, WAY BACK
WHEN, only had the concept of a reciprocal of an integer. As in for any
integer N, there was a fraction of 1/N. They had a concept of 1/3 but not of
2/3 or 4/9.

So they added reciprocals to make any more complex fractions. To make 2/3
they added 1/2 plus 1/6 for example.

Since they were not stuck with any one base, all kinds of such combined
fractions could be done but of course the square root of 2 or pi were a bit
beyond them and for similar reasons.

https://en.wikipedia.org/wiki/Egyptian_fraction

My point is there are many ways humans can choose to play with numbers and
not all of them can easily do the same thing. Roman Numerals were (and
remain) a horror to do much mathematics with and especially when they play
games based on whether a symbol like X is to the left or right of another
like C as XC is 90 and CX is 110.

To do programming learn the rules that only what can be represented in
binary has a chance to ...





-----Original Message-----
From: Python-list <python-list-bounces+avigross=verizon.net at python.org> On
Behalf Of Grant Edwards
Sent: Saturday, November 20, 2021 5:24 PM
To: python-list at python.org
Subject: Re: Unexpected behaviour of math.floor, round and int functions
(rounding)

On 2021-11-20, Ben Bacarisse <ben.usenet at bsb.me.uk> wrote:

> You seem to be agreeing with me.  It's the floating point part that is 
> the issue, not the base itself.

No, it's the base. Floating point can't represent 3/10 _because_ it's base 2
floating point. Floating point in base 10 doesn't have any problem
representing 3/10.

--
Grant
--
https://mail.python.org/mailman/listinfo/python-list


From rosuav at gmail.com  Sat Nov 20 20:02:45 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 21 Nov 2021 12:02:45 +1100
Subject: Unexpected behaviour of math.floor,
 round and int functions (rounding)
In-Reply-To: <041a01d7de70$15b9a3b0$412ceb10$@verizon.net>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
 <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>
 <mailman.63.1637360151.19479.python-list@python.org>
 <874k87k4a7.fsf@bsb.me.uk>
 <CAPTjJmohqdKMP4srxWQZmu4Xn0+M2t7J8UtL9CDtNQRuOZ8N0A@mail.gmail.com>
 <mailman.67.1637373077.19479.python-list@python.org>
 <87y25jik3i.fsf@bsb.me.uk>
 <CAPTjJmr-Sik-NEzm1m17uXUMXSf8ijT9x=BvX9PYSoW6x=Nf0w@mail.gmail.com>
 <mailman.68.1637385505.19479.python-list@python.org>
 <87sfvri07m.fsf@bsb.me.uk>
 <61997569.1c69fb81.5eb3c.b508@mx.google.com>
 <041a01d7de70$15b9a3b0$412ceb10$@verizon.net>
Message-ID: <CAPTjJmqHBrOZsF8xsiQuWhPJ+xJ5FWWmuQbt9UBkyNVjYyc60Q@mail.gmail.com>

On Sun, Nov 21, 2021 at 11:39 AM Avi Gross via Python-list
<python-list at python.org> wrote:
>
> Can I suggest a way to look at it, Grant?
>
> In base 10, we represent all numbers as the (possibly infinite) sum of ten
> raised to some integral power.

Not infinite. If you allow an infinite sequence of digits, you create
numerous paradoxes, not to mention the need for infinite storage.

> 123 is 3 times 1 (ten to the zero power) plus
> 2 times 10 (ten to the one power) plus
> 1 times 100 (ten to the two power)
>
> 123.456 just extends this with
> 4 times 1/10 (ten to the minus one power) plus
> 5 times 1/100 (10**-2) plus
> 6 time 1/1000 (10**-3)
>
> In binary, all the powers are not powers of 10 but powers of two.
>
> So IF you wrote something like 111 it means 1 times 1 plus 1 times 2 plus 1
> times 4 or 7. A zero anywhere just skips a 2 to that power. If you added a
> decimal point to make 111.111 the latter part would be 1/2 plus 1/4 plus 1/8
> or 7/8 which combined might be 7 and 7/8. So any fractions of the form
> something over 2**N can be made easily and almost everything else cannot be
> made in finite stretches. How would you make 2/3 or 3 /10?

Right, this is exactly how place value works.

> But the opposite is something true. In decimal, to make the above it becomes
> 7.875 and to make other fractions of the kind, you need more and more As it
> happens, all such base-2 compatible streams can be made because each is in
> some sense a divide by two.
>
> 7/16 = 1/2 * .875 = .4375
> 7/32 = 1/2 * .4375 = .21875
>
> and so on. But this ability is a special case artifact caused by a terminal
> digit 5 always being able to be divided in tow to make a 25 a unit longer
> and then again and again. Note 2 and 5 are factors of 10.  In the more
> general case, this fails. In base 7, 3/7 is written easily as 0.3 but the
> same fraction in decimal is a repeating copy of .428571... which never
> terminates. A number like 3/7 + 4/49 + 5/343 generally cannot be written in
> base 7 but the opposite is also true that only a approximation of numbers in
> base 2 or base 10 can ever be written. I am, of course, talking about the
> part to the right of the decimal. Integers to the left can be written in any
> base. It is fractional parts that can end up being nonrepeating.

If you have a number with a finite binary representation, you can
guarantee that it can be represented finitely in decimal too.
Infinitely repeating expansions come from denominators that are
coprime with the numeric base.

> What about pi and e and the square root of 2? I suspect all of them have an
> infinite sequence with no real repetition (over long enough stretches) in
> any base! I mean an integer base, of course. The constant e in base e is
> just 1.

More than "suspect". This has been proven. That's what transcendental means.

I don't think "base e" means the same thing that "base ten" does.
(Normally you'd talk about a base e *logarithm*, which is a completely
different concept.) But if you try to work with a transcendental base
like that, it would be impossible to represent any integer finitely.

(Side point: There are other representations that have different
implications about what repeats and what doesn't. For instance, the
decimal expansion for a square root doesn't repeat, but the continued
fraction for the same square root will. For instance, 7**0.5 is
2;1,1,1,4,1,1,1,4... with an infinitely repeating four-element unit.)

> I think there have been attempts to use a decimal representation in some
> accounting packages or database applications that allow any decimal numbers
> to be faithfully represented and used in calculations. Generally this is not
> a very efficient process but it can handle 0.3 albeit still have no way to
> deal with transcendental numbers.

Fixed point has been around for a long time (the simplest example
being "work in cents and use integers"), but actual decimal
floating-point is quite unusual. Some databases support it, and REXX
used that as its only numeric form, but it's not hugely popular.

> Let me leave you with Egyptian mathematics. Their use of fractions, WAY BACK
> WHEN, only had the concept of a reciprocal of an integer. As in for any
> integer N, there was a fraction of 1/N. They had a concept of 1/3 but not of
> 2/3 or 4/9.
>
> So they added reciprocals to make any more complex fractions. To make 2/3
> they added 1/2 plus 1/6 for example.
>
> Since they were not stuck with any one base, all kinds of such combined
> fractions could be done but of course the square root of 2 or pi were a bit
> beyond them and for similar reasons.
>
> https://en.wikipedia.org/wiki/Egyptian_fraction

It's interesting as a curiosity, but it makes arithmetic extremely difficult.

> My point is there are many ways humans can choose to play with numbers and
> not all of them can easily do the same thing. Roman Numerals were (and
> remain) a horror to do much mathematics with and especially when they play
> games based on whether a symbol like X is to the left or right of another
> like C as XC is 90 and CX is 110.

Of course, there are myriad ways to do things. And each one has
implications. Which makes it even more surprising that, when someone
sits down at a computer and asks it to do arithmetic, they can't
handle the different implications.

ChrisA

From grant.b.edwards at gmail.com  Sat Nov 20 20:18:21 2021
From: grant.b.edwards at gmail.com (Grant Edwards)
Date: Sat, 20 Nov 2021 17:18:21 -0800 (PST)
Subject: Unexpected behaviour of math.floor, round and int functions
 (rounding)
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
 <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>
 <mailman.63.1637360151.19479.python-list@python.org>
 <874k87k4a7.fsf@bsb.me.uk>
 <CAPTjJmohqdKMP4srxWQZmu4Xn0+M2t7J8UtL9CDtNQRuOZ8N0A@mail.gmail.com>
 <mailman.67.1637373077.19479.python-list@python.org>
 <87y25jik3i.fsf@bsb.me.uk>
 <CAPTjJmr-Sik-NEzm1m17uXUMXSf8ijT9x=BvX9PYSoW6x=Nf0w@mail.gmail.com>
 <mailman.68.1637385505.19479.python-list@python.org>
 <87sfvri07m.fsf@bsb.me.uk> <61997569.1c69fb81.5eb3c.b508@mx.google.com>
 <041a01d7de70$15b9a3b0$412ceb10$@verizon.net>
 <CAPTjJmqHBrOZsF8xsiQuWhPJ+xJ5FWWmuQbt9UBkyNVjYyc60Q@mail.gmail.com>
Message-ID: <61999e5d.1c69fb81.a7898.c7e0@mx.google.com>

On 2021-11-21, Chris Angelico <rosuav at gmail.com> wrote:

>> I think there have been attempts to use a decimal representation in some
>> accounting packages or database applications that allow any decimal numbers
>> to be faithfully represented and used in calculations. Generally this is not
>> a very efficient process but it can handle 0.3 albeit still have no way to
>> deal with transcendental numbers.
>
> Fixed point has been around for a long time (the simplest example
> being "work in cents and use integers"), but actual decimal
> floating-point is quite unusual. Some databases support it, and REXX
> used that as its only numeric form, but it's not hugely popular.

My recollection is that it was quite common back in the days before FP
hardware was "a thing" on small computers. CPM and DOS compilers for
various languages often gave the user a choice between binary FP and
decimal (BCD) FP.

If you were doing accounting you chose decimal. If you were doing
science, you chose binary (better range and precision for the same
number of bits of storage).

Once binary FP hardware became available, decimal FP support was
abandoned.

--
Grant

From avigross at verizon.net  Sat Nov 20 20:54:53 2021
From: avigross at verizon.net (Avi Gross)
Date: Sat, 20 Nov 2021 20:54:53 -0500
Subject: Unexpected behaviour of math.floor,
 round and int functions (rounding)
In-Reply-To: <e9cde1af-7f36-1187-0a9f-2283db306a79@btinternet.com>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
 <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>
 <mailman.63.1637360151.19479.python-list@python.org>
 <874k87k4a7.fsf@bsb.me.uk>
 <CAPTjJmohqdKMP4srxWQZmu4Xn0+M2t7J8UtL9CDtNQRuOZ8N0A@mail.gmail.com>
 <mailman.67.1637373077.19479.python-list@python.org>
 <87y25jik3i.fsf@bsb.me.uk>
 <CAPTjJmr-Sik-NEzm1m17uXUMXSf8ijT9x=BvX9PYSoW6x=Nf0w@mail.gmail.com>
 <mailman.68.1637385505.19479.python-list@python.org>
 <87sfvri07m.fsf@bsb.me.uk> <03c901d7de55$df8dd640$9ea982c0$@verizon.net>
 <CAPTjJmp2SqZayWrRguqRi-o-_4dHnwptwgmbpF0N7tvk7HOK9g@mail.gmail.com>
 <03f601d7de62$4eac6080$ec052180$@verizon.net> <e9cde1af-7f36-1187-0
 a9f-2283db306a79@btinternet.com>
Message-ID: <049401d7de7a$c7560a00$56021e00$@verizon.net>

Not at all, Robb. I am not intending to demean Mathematicians as one of my degrees is in that subject and I liked it. I mean that some things in mathematics are not as intuitive to people when they first encounter them, let alone those who never see them and then marvel at results and have expectations.

The example I gave, is NOW, indeed on quite firm footing but for quite a while was not.

What we have in this forum recently is people taking pot shots at aspects of Python where in a similar way, they know not what is actually happening and insist it be some other way. Some people also assume that an email message work any way they want and post things to a text-only group that other cannot see or become badly formatted or complain why a very large attachment makes a message be rejected. They also expect SPAM checkers to be perfect and never reject valid messages and so on.

Things are what they are, not what we wish them to be. And many kinds of pure mathematics live in a Platonic world and must be used with care. Calculus is NOT on a firm footing when any of the ideas in it are violated. A quantum Mechanical universe at a deep level does not have continuity so continuous functions may not really exist and there can be no such thing as an infinitesimal smaller than any epsilon and so on. Much of what we see at that level includes things like a probabilistic view of an electron cloud forming the probability that an electron (which is not a mathematical point) is at any moment at a particular location around an atom. But some like the p-orbital have a sort of 3-D figure eight shape (sort of a pair of teardrops) where there is a plane between the two halves with a mathematically zero probability of the electron ever being there. Yet, quantum tunneling effects let it dross through that plane without actually ever being in the plane because various kinds of quantum jumps in a very wiggly space-time fabric can and will happen in a way normal mathematics may not predict or allow. 

Which brings me back to the python analogy of algorithms implemented that gradually zoom in on an answer you might view as a local maximum or minimum. It may be that with infinite precision calculations, you might zoom in ever closer to the optimal answer where the tangent to such a curve has slope zero. Your program would never halt though if the condition was that it be exactly at that point to an infinite number of decimal places. This is a place I do not agree that the concept of being near the answer (or in this case being near zero) is not a good enough heuristic solution. There are many iterative problems (and recursive ones) where  a close-enough condition is adequate. Some libraries incorporated into languages like Python use an infinite series to calculate something like sin(x) and many other such things, including potentially e and pi and various roots. Many of them can safely stop after N significant digits are locked into place, and especially when all available significant digits are locked. Running them further gains nothing much. So code like:

(previous_estimate - current_estimate) == 0

may be a bad idea compared to something like:

abs(previous_estimate - current_estimate) < epsilon

No disrespect to mathematics intended. My understanding is that mathematics can only be used validly if all underlying axioms are assumed to be true. When (as in the real world or computer programs) some axioms are violated, watch out. Matrix multiplication does not have a symmetry so A*B in general is not the same as B*A and even worse, may be a matrix of a different dimension. A 4x2 matrix and a 2x4 matrix can result in either a 2x2 or 4x4 for example. The violation of that rule may bother some people but is not really an issue as any mathematics that has an axiom for say an abelian group, simply is not expected to apply for a non-abelian case.




-----Original Message-----
From: Python-list <python-list-bounces+avigross=verizon.net at python.org> On Behalf Of Rob Cliffe via Python-list
Sent: Saturday, November 20, 2021 6:19 PM
To: 
Subject: Re: Unexpected behaviour of math.floor, round and int functions (rounding)



On 20/11/2021 22:59, Avi Gross via Python-list wrote:
> there are grey lines along the way where some mathematical proofs do 
> weird things like IGNORE parts of a calculation by suggesting they are 
> going to zero much faster than other parts and then wave a 
> mathematical wand about what happens when they approach a limit like 
> zero and voila, we just "proved" that the derivative of X**2 is 2*X or 
> the more general derivative of A*(X**N) is N*A*(X**(N-1)) and then 
> extend that to N being negative or fractional or a transcendental number and beyond.
>
>
     You seem to be maligning mathematicians.
     What you say was true in the time of Newton, Leibniz and Bishop Berkeley, but analysis was made completely rigorous by the efforts of Weierstrass and others.  There are no "grey lines".  Proofs do not "suggest", they PROVE (else they are not proofs, they are plain wrong). It is not the fault of mathematicians (or mathematics) if some people produce sloppy hand-wavy "proofs" as justification for their conclusions.
     I am absolutely sure you know all this, but your post does not read as if you do.  And it could give a mistaken impression to a non-mathematician.  I think we have had enough denigration of experts.
Best
Rob Cliffe



--
https://mail.python.org/mailman/listinfo/python-list


From rosuav at gmail.com  Sat Nov 20 21:01:21 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 21 Nov 2021 13:01:21 +1100
Subject: Unexpected behaviour of math.floor,
 round and int functions (rounding)
In-Reply-To: <049401d7de7a$c7560a00$56021e00$@verizon.net>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
 <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>
 <mailman.63.1637360151.19479.python-list@python.org>
 <874k87k4a7.fsf@bsb.me.uk>
 <CAPTjJmohqdKMP4srxWQZmu4Xn0+M2t7J8UtL9CDtNQRuOZ8N0A@mail.gmail.com>
 <mailman.67.1637373077.19479.python-list@python.org>
 <87y25jik3i.fsf@bsb.me.uk>
 <CAPTjJmr-Sik-NEzm1m17uXUMXSf8ijT9x=BvX9PYSoW6x=Nf0w@mail.gmail.com>
 <mailman.68.1637385505.19479.python-list@python.org>
 <87sfvri07m.fsf@bsb.me.uk>
 <03c901d7de55$df8dd640$9ea982c0$@verizon.net>
 <CAPTjJmp2SqZayWrRguqRi-o-_4dHnwptwgmbpF0N7tvk7HOK9g@mail.gmail.com>
 <03f601d7de62$4eac6080$ec052180$@verizon.net>
 <e9cde1af-7f36-1187-0a9f-2283db306a79@btinternet.com>
 <049401d7de7a$c7560a00$56021e00$@verizon.net>
Message-ID: <CAPTjJmoKSzN+H-a3A11Cwe-OywSzK51kv1huu8SXVqgGdvLGmQ@mail.gmail.com>

On Sun, Nov 21, 2021 at 12:56 PM Avi Gross via Python-list
<python-list at python.org> wrote:
>
> Not at all, Robb. I am not intending to demean Mathematicians as one of my degrees is in that subject and I liked it. I mean that some things in mathematics are not as intuitive to people when they first encounter them, let alone those who never see them and then marvel at results and have expectations.
>
> The example I gave, is NOW, indeed on quite firm footing but for quite a while was not.
>
> What we have in this forum recently is people taking pot shots at aspects of Python where in a similar way, they know not what is actually happening and insist it be some other way. Some people also assume that an email message work any way they want and post things to a text-only group that other cannot see or become badly formatted or complain why a very large attachment makes a message be rejected. They also expect SPAM checkers to be perfect and never reject valid messages and so on.
>
> Things are what they are, not what we wish them to be. And many kinds of pure mathematics live in a Platonic world and must be used with care. Calculus is NOT on a firm footing when any of the ideas in it are violated. A quantum Mechanical universe at a deep level does not have continuity so continuous functions may not really exist and there can be no such thing as an infinitesimal smaller than any epsilon and so on. Much of what we see at that level includes things like a probabilistic view of an electron cloud forming the probability that an electron (which is not a mathematical point) is at any moment at a particular location around an atom. But some like the p-orbital have a sort of 3-D figure eight shape (sort of a pair of teardrops) where there is a plane between the two halves with a mathematically zero probability of the electron ever being there. Yet, quantum tunneling effects let it dross through that plane without actually ever being in the plane because various kinds of
>  quantum jumps in a very wiggly space-time fabric can and will happen in a way normal mathematics may not predict or allow.
>
> Which brings me back to the python analogy of algorithms implemented that gradually zoom in on an answer you might view as a local maximum or minimum. It may be that with infinite precision calculations, you might zoom in ever closer to the optimal answer where the tangent to such a curve has slope zero. Your program would never halt though if the condition was that it be exactly at that point to an infinite number of decimal places. This is a place I do not agree that the concept of being near the answer (or in this case being near zero) is not a good enough heuristic solution. There are many iterative problems (and recursive ones) where  a close-enough condition is adequate. Some libraries incorporated into languages like Python use an infinite series to calculate something like sin(x) and many other such things, including potentially e and pi and various roots. Many of them can safely stop after N significant digits are locked into place, and especially when all available signific
>  ant digits are locked. Running them further gains nothing much. So code like:
>
> (previous_estimate - current_estimate) == 0
>
> may be a bad idea compared to something like:
>
> abs(previous_estimate - current_estimate) < epsilon
>
> No disrespect to mathematics intended. My understanding is that mathematics can only be used validly if all underlying axioms are assumed to be true. When (as in the real world or computer programs) some axioms are violated, watch out. Matrix multiplication does not have a symmetry so A*B in general is not the same as B*A and even worse, may be a matrix of a different dimension. A 4x2 matrix and a 2x4 matrix can result in either a 2x2 or 4x4 for example. The violation of that rule may bother some people but is not really an issue as any mathematics that has an axiom for say an abelian group, simply is not expected to apply for a non-abelian case.
>

All of this is true, but utterly irrelevant to floating-point. If your
algorithm is inherently based on repeated estimates (Newton's method,
for instance), then you can iterate until you're "happy enough" with
the result. That's fine. But that is nothing whatsoever to do with the
complaint that 0.1+0.2!=0.3 or that you should "never use == with
floats" or any of those claims. It's as relevant as saying that my
ruler claims to be 30cm long but is actually nearly 310mm long, and
therefore the centimeter is an inherently unreliable unit and anything
measured in it should be treated as an estimate.

ChrisA

From rob.cliffe at btinternet.com  Sat Nov 20 21:18:56 2021
From: rob.cliffe at btinternet.com (Rob Cliffe)
Date: Sun, 21 Nov 2021 02:18:56 +0000
Subject: Unexpected behaviour of math.floor, round and int functions
 (rounding)
In-Reply-To: <CAPTjJmqHBrOZsF8xsiQuWhPJ+xJ5FWWmuQbt9UBkyNVjYyc60Q@mail.gmail.com>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
 <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>
 <mailman.63.1637360151.19479.python-list@python.org>
 <874k87k4a7.fsf@bsb.me.uk>
 <CAPTjJmohqdKMP4srxWQZmu4Xn0+M2t7J8UtL9CDtNQRuOZ8N0A@mail.gmail.com>
 <mailman.67.1637373077.19479.python-list@python.org>
 <87y25jik3i.fsf@bsb.me.uk>
 <CAPTjJmr-Sik-NEzm1m17uXUMXSf8ijT9x=BvX9PYSoW6x=Nf0w@mail.gmail.com>
 <mailman.68.1637385505.19479.python-list@python.org>
 <87sfvri07m.fsf@bsb.me.uk> <61997569.1c69fb81.5eb3c.b508@mx.google.com>
 <041a01d7de70$15b9a3b0$412ceb10$@verizon.net>
 <CAPTjJmqHBrOZsF8xsiQuWhPJ+xJ5FWWmuQbt9UBkyNVjYyc60Q@mail.gmail.com>
Message-ID: <7f35cd5f-12ae-2755-ae03-6182895a6d1e@btinternet.com>



On 21/11/2021 01:02, Chris Angelico wrote:
>
> If you have a number with a finite binary representation, you can
> guarantee that it can be represented finitely in decimal too.
> Infinitely repeating expansions come from denominators that are
> coprime with the numeric base.
>
>
Not quite, e.g. 1/14 is a repeating decimal but 14 and 10 are not coprime.
I believe it is correct to say that infinitely recurring expansions 
occur when the denominator is divisible by a prime that does not divide 
the base.
Rob Cliffe

From avigross at verizon.net  Sat Nov 20 21:36:35 2021
From: avigross at verizon.net (Avi Gross)
Date: Sat, 20 Nov 2021 21:36:35 -0500
Subject: Unexpected behaviour of math.floor,
 round and int functions (rounding)
In-Reply-To: <CAPTjJmr-8ABoMOUVQTwJBD13SS8cc01615=2wr1jJe-A7PP1+A@mail.gmail.com>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
 <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>
 <mailman.63.1637360151.19479.python-list@python.org>
 <874k87k4a7.fsf@bsb.me.uk>
 <CAPTjJmohqdKMP4srxWQZmu4Xn0+M2t7J8UtL9CDtNQRuOZ8N0A@mail.gmail.com>
 <mailman.67.1637373077.19479.python-list@python.org>
 <87y25jik3i.fsf@bsb.me.uk>
 <CAPTjJmr-Sik-NEzm1m17uXUMXSf8ijT9x=BvX9PYSoW6x=Nf0w@mail.gmail.com>
 <mailman.68.1637385505.19479.python-list@python.org>
 <87sfvri07m.fsf@bsb.me.uk> <03c901d7de55$df8dd640$9ea982c0$@verizon.net>
 <CAPTjJmp2SqZayWrRguqRi-o-_4dHnwptwgmbpF0N7tvk7HOK9g@mail.gmail.com>
 <03f601d7de62$4eac6080$ec052180$@verizo
 n.net> <CAPTjJmr-8ABoMOUVQTwJBD13SS8cc01615=2wr1jJe-A7PP1+A@mail.gmail.com>
Message-ID: <049e01d7de80$9a9d01c0$cfd70540$@verizon.net>

Chris,

You know I am going to fully agree with you that within some bounds, any combination of numbers that can accurately be represented will continue to be adequately represented under some operations like addition and subtraction and multiplication up to any point where they do not overflow (or underflow) the storage mechanism.

Division may be problematic and especially division by zero. 

But bring in any number that is not fully and accurately representable, and it can poison everything much in the way including an NA poisons any attempts to take a sum or mean. Any calculation that includes an e is an example.

Of course there is not much in computing that necessarily relies on representable numbers and especially not when the numbers are dynamically gotten as in from a file or user and already not quite what is representable. I can even imagine a situation where some fraction is represented in a double and then "copied" into a regular singular float and some of it is lost/truncated. 

I get your point about URL's but I was really focused at that point on filenames as an example on systems where they are not case sensitive. Some programming languages had a similar concept. Yes, you can have URL with more complex comparison functions needed including when something lengthens them or whatever. In one weird sense, as in you GET TO THE SAME PAGE, any URL that redirects you to another might be considered synonymous even if the two look nothing at all alike.

To continue, I do not mean to give the impression that comparing representable numbers with == is generally wrong. I am saying there are places where there may be good reasons for the alternative.

I can imagine an algorithm that starts with representable numbers and maybe at each stage continues to generate representable numbers, such as one of the hill climbing algorithms I am talking about. It may end up overshooting a bit past the peak and next round overshooting back to the other side and getting stuck in a loop. One way out is to keep track of past locations and abort when the cycle is seen to be repeating. Another is to leave when the result seems close enough. 

However, my comments about over/underflow may apply here as enough iterations with representable numbers may at some point result in the kind of rounding error that warps the results of further calculations.

I note some of your argument is the valid difference between when your knowledge of the input numbers is uncertain and what the computer does with them. Yes, my measures of the height/width/depth may be uncertain and it is not the fault of a python program if it multiplies them to provide an exact answer as if in a mathematical world where numbers are normally precise. I am saying that the human using the program needs external info before they use the answer. In my example, I would note the rule that when dealing with numbers that are only significant to some number of digits, the final calculation should often be rounded down according to some rules. So instead of printing out the volume as 4140.606, the program may call some function like round() as in round(10.1*20.2*30.3, 1) so it displays 4141.6 instead. The Python language does what you ask and not what you do not ask. 

Now a statistical program or perhaps an AI or Machine Learning program I write, might actually care about the probabilistic effects. I often create graphs that include perhaps a smoothed curve of some kind that approximates the points in the data as well as a light gray ribbon that represents some kind of error bands above and below and which suggest the line not be taken too seriously and there may be something like a 95% chance the true values are within the gray zone an even some chance they may be beyond it in an even lighter series of gray (color is not the issue) zones representing a 1% chance or even less. 

Such approaches apply if the measurement errors are assume to be as much as .1 inches for each measure independently. The smallest volume would be (10.1 - 0.1)*(20.2 - 0.1)*(30.3 - 0.1) = 6070.2

The largest possible volume if all my measures were off by that amount in the other direction would be:

(10.1 + 0.1)*(20.2 + 0.1)*(30.3 + 0.1) = 6294.6 

The above are truncated to one significant digit. 

The Python program evaluates all the above representable numbers perfectly, albeit I doubt they are all representable in binary. But for human purposes, the actual answer for a volume has some uncertainty built-in to the method of measurement and perhaps others such as the inner sides of the box may not be perfectly flat or the angles things join at may not be precisely 90 degrees and filling it with something like oranges may not fit as much more if you enlarge it a tad as they may not stack much better from a minor change.

Python is not to blame in these cases if not programmed well enough. And I suggest the often minor errors introduced by a representation being not quite right in the last available decimal representation places (binary underneath) may be much smaller than these other introduced errors in the reality of such a superficially simple calculation.

>From my side of this discussion, I do not see much we basically disagree on, albeit we may word some ideas differently. I think I may opt out of further comments unless something new is mentioned and it does relate to Python or similar programming languages. I am spending too much time today on this one and another in an R mailing list and other things elsewhere so my main plans for the day have fallen behind ?

Avi

-----Original Message-----
From: Python-list <python-list-bounces+avigross=verizon.net at python.org> On Behalf Of Chris Angelico
Sent: Saturday, November 20, 2021 6:23 PM
To: python-list at python.org
Subject: Re: Unexpected behaviour of math.floor, round and int functions (rounding)

On Sun, Nov 21, 2021 at 10:01 AM Avi Gross via Python-list <python-list at python.org> wrote:
> Computers generally use finite methods, sometimes too finite. Yes, the 
> problem is not Mathematics as a field. It is how humans often 
> generalize or analogize from one area into something a bit different. 
> I do not agree with any suggestion that a series of bits that encodes 
> a result that is rounded or truncated is CORRECT. A representation of 
> 0.3 in a binary version of some floating point format is not 
> technically correct. Storing it as 3/10 and carefully later 
> multiplying it by 20 and then carefully canceling part will result in 
> exactly 6. While storing it digitally and then multiplying it in 
> registers or whatever by 20 may get a result slightly different than 
> the storage representation of 6.0000000000... and that is a fact and risk we generally are willing to take.

Do you accept that storing the floating point value 1/4, then multiplying by 20, will give precisely 5? Because that is *guaranteed*. You don't have to expect a result "slightly different"
from 5, it will be absolutely exactly five:

>>> (1/4) * 20 == 5.0
True

This is what I'm talking about. Some numbers can be represented perfectly, others can't. If you try to represent the square root of two as a decimal number, then multiply it by itself, you won't get back precisely 2, because you can't have written out the *exact* square root of two. But you most certainly CAN write "1.875" on a piece of paper, and it really truly does exactly mean fifteen eighths.
And you can write that number as a binary float, too, and it'll mean the exact same value.

> But consider a different example. If I have a filesystem or URL or 
> anything that does not care about whether parts are in upper or lower 
> case, then "filename" and "FILENAME" and many variations like 
> "fIlEnAmE" are all assumed to mean the same thing. A program may even 
> simply store all of them in the same way as all uppercase. But when 
> you ask to compare two versions with a function where case matters, 
> they all test as unequal! So there are ways to ask for a comparison 
> that is approximately equal given the constraints that case does not matter:

A URL has distinct parts to it: the domain has some precise folding done (most notably case folding), the path does not, and you can consider "http://example.com:80/foo" to be the same as "http://example.com/foo" because 80 is the default port.

> >>> alpha="Hello"
> >>> beta="hELLO"
> >>> alpha == beta
> False
> >>> alpha.lower() == beta.lower()
> True
>

That's a terrible way to compare URLs, because it's both too sloppy AND too strict at the same time. But if you have a URL representation tool, it should be able to consider two things equal.

Floats are representations of numbers that can be compared for equality if they truly represent the same number. The value 3/6 is precisely equal to the value 7/14:

>>> 3/6 == 7/14
True

You don't need an "approximately equal" function here. They are the same value. They are equal.

> I see no reason why a comparison canot be done like this in cases you 
> are concerned with small errors creeping in:
>
> >>> from math import isclose
> >>> isclose(1, .9999999999999999999999)
> True
> >>> isclose(1, .9999999999)
> True
> >>> isclose(1, .999)
> False

This is exactly the problem though: HOW close counts as equal? The only way to answer that question is to know the accuracy of your inputs, and the operations done.

> So floats by themselves are not inaccurate but realistically the 
> results of operations ARE. I mean if I ask a long number to be stored 
> that does not fully fit, it is often silently truncated and what the 
> storage location now represent accurately is not my number but the 
> shorter version that is at the limit of tolerance. But consider 
> another analogy often encountered in mathematics.

Not true. Operations are often perfectly accurate.

> If I measure several numbers in the real world such as weight and 
> height and temperature and so on, some are considered accurate only to 
> a limited number of digits. Your weight on a standard digital scale 
> may well be 189.8 but if I add a feather or subtract one, the reading 
> may well shift to one unit up or down. Heck, the same person measured 
> just minutes later may shift. If I used a deluxe scale that measures 
> to more decimal places, it may get hard to get the exact same number 
> twice in a row as just taking a deeper breath may make a change.
>
> So what happens if I measure a box in three dimensions to the nearest 
> .1 inch and decide it is 10.1 by 20.2 by 30.3 inches? What is the 
> volume, ignoring pesky details about the width of the cardboard or whatever?
>
> A straightforward multiplication yields 4141.606 cubic inches. You may 
> have been told to round that to something like 4141.6 because the 
> potential error in each measure cannot result in more precision. In 
> reality, you might even calculate two sets of numbers assuming the 
> true width may have been a tad more or less and come up with the 
> volume being BETWEEN a somewhat smaller number and a somewhat larger number.

If those initial figures were accurate to three digits, you should round it to 4140 cubic inches, because that's all the accuracy you have. (Or, if you prefer, 4140 +/- 5.)

> I claim a similar issue plagues using a computer to deal with stored 
> numbers, perhaps not stored 100% perfectly as discussed, and doing 
> calculations. The result often comes out more precisely than 
> warranted. I suspect there are modules out there that might do 
> multi-step calculations where at each step, numbers generated with 
> extra precision are throttled back so the extra precision is set to 
> zeroes after rounding to avoid the small increments adding up. Others 
> may just do the calculations and keep track and remove extra precision at the end.

When your input values aren't accurate, your output won't be accurate.
That's something the computer can never know. When you store the number 3602879701896397/36028797018963968, did you actually mean that number, or did you mean some other number that's kinda close to it? If you don't tell the computer, it's going to assume that you wanted exactly that number.

> And again, this is not because the implementation of numbers is in any 
> way wrong but because a real-world situation requires the humans to 
> sort of dial back how they are used and not over-reach.
>
> So comparing for close-enough inequality is not necessarily a 
> reflection on floats but on the design not accommodating the precision 
> needed or perhaps on the algorithm used not necessarily being expected 
> to reach a certain level.

And close-enough equality is the correct thing to do when you know exactly what the accuracy of your inputs is. If you need to be completely rigorous about it, you'd have to store every number as a range (so you might say that your input length is "10.05 to 10.15" or "10.1, error 0.5") and do all arithmetic on those ranges. What you'd find is that some operations widen the ranges and others don't. The trouble is, that's not actually all that useful; Fermi estimates are far more accurate than they seem like they "should be" because the balance of probability is in favour of errors cancelling out, at least partially.

ChrisA
--
https://mail.python.org/mailman/listinfo/python-list


From rosuav at gmail.com  Sat Nov 20 21:40:28 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 21 Nov 2021 13:40:28 +1100
Subject: Unexpected behaviour of math.floor,
 round and int functions (rounding)
In-Reply-To: <7f35cd5f-12ae-2755-ae03-6182895a6d1e@btinternet.com>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
 <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>
 <mailman.63.1637360151.19479.python-list@python.org>
 <874k87k4a7.fsf@bsb.me.uk>
 <CAPTjJmohqdKMP4srxWQZmu4Xn0+M2t7J8UtL9CDtNQRuOZ8N0A@mail.gmail.com>
 <mailman.67.1637373077.19479.python-list@python.org>
 <87y25jik3i.fsf@bsb.me.uk>
 <CAPTjJmr-Sik-NEzm1m17uXUMXSf8ijT9x=BvX9PYSoW6x=Nf0w@mail.gmail.com>
 <mailman.68.1637385505.19479.python-list@python.org>
 <87sfvri07m.fsf@bsb.me.uk>
 <61997569.1c69fb81.5eb3c.b508@mx.google.com>
 <041a01d7de70$15b9a3b0$412ceb10$@verizon.net>
 <CAPTjJmqHBrOZsF8xsiQuWhPJ+xJ5FWWmuQbt9UBkyNVjYyc60Q@mail.gmail.com>
 <7f35cd5f-12ae-2755-ae03-6182895a6d1e@btinternet.com>
Message-ID: <CAPTjJmpjhPDgL6xD8UJn0kzYTwVCYvSJ53vrAo9xGDJKRbUnUQ@mail.gmail.com>

On Sun, Nov 21, 2021 at 1:20 PM Rob Cliffe via Python-list
<python-list at python.org> wrote:
>
>
>
> On 21/11/2021 01:02, Chris Angelico wrote:
> >
> > If you have a number with a finite binary representation, you can
> > guarantee that it can be represented finitely in decimal too.
> > Infinitely repeating expansions come from denominators that are
> > coprime with the numeric base.
> >
> >
> Not quite, e.g. 1/14 is a repeating decimal but 14 and 10 are not coprime.
> I believe it is correct to say that infinitely recurring expansions
> occur when the denominator is divisible by a prime that does not divide
> the base.
> Rob Cliffe

True, my bad. I can't remember if there's a term for that, but your
description is correct.

ChrisA

From avigross at verizon.net  Sat Nov 20 22:09:23 2021
From: avigross at verizon.net (Avi Gross)
Date: Sat, 20 Nov 2021 22:09:23 -0500
Subject: Unexpected behaviour of math.floor,
 round and int functions (rounding)
In-Reply-To: <CAPTjJmqHBrOZsF8xsiQuWhPJ+xJ5FWWmuQbt9UBkyNVjYyc60Q@mail.gmail.com>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
 <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>
 <mailman.63.1637360151.19479.python-list@python.org>
 <874k87k4a7.fsf@bsb.me.uk>
 <CAPTjJmohqdKMP4srxWQZmu4Xn0+M2t7J8UtL9CDtNQRuOZ8N0A@mail.gmail.com>
 <mailman.67.1637373077.19479.python-list@python.org>
 <87y25jik3i.fsf@bsb.me.uk>
 <CAPTjJmr-Sik-NEzm1m17uXUMXSf8ijT9x=BvX9PYSoW6x=Nf0w@mail.gmail.com>
 <mailman.68.1637385505.19479.python-list@python.org>
 <87sfvri07m.fsf@bsb.me.uk> <61997569.1c69fb81.5eb3c.b508@mx.google.com>
 <041a01d7de70$15b9a3b0$412ceb10$@verizon.net>
 <CAPTjJmqHBrOZsF8xsiQuWhPJ+xJ5FWWmuQbt9UBkyNVjYyc60Q@mail.gmail
 .com>
Message-ID: <04ce01d7de85$2f4fbb10$8def3130$@verizon.net>

Sorry Chris,

I was talking mathematically where a number like pi or like 1/7 conceptually
have an infinite number of digits needed that are added to a growing sum
using ever smaller powers of 10, in the decimal case.

In programming, and in the binary storage, the number of such is clearly
limited.

Is there any official limit on the maximum size of a python integer other
than available memory?

And replying sparsely, yes, pretty much nothing can be represent completely
in base e other than integral multiples of e, perhaps. No other numbers,
especially integers,  can be linear combinations of e or e raised to an
integral power.

Having said that, if you throw in another transcendental called pi and
expand to include the complex number i, then you can weirdly combine them
another way to make -1. I am sure you have seen equations like:

e**(pi*i) +1 = 0

By extension, you can make any integer by adding multiple such entities
together.

On another point, an indefinitely continued repeated fraction is sort of
similar to indefinitely summed series. Both can exist and demonstrate a
regularity when the actual digits of the number seemingly show no patterns.



-----Original Message-----
From: Python-list <python-list-bounces+avigross=verizon.net at python.org> On
Behalf Of Chris Angelico
Sent: Saturday, November 20, 2021 8:03 PM
To: python-list at python.org
Subject: Re: Unexpected behaviour of math.floor, round and int functions
(rounding)

On Sun, Nov 21, 2021 at 11:39 AM Avi Gross via Python-list
<python-list at python.org> wrote:
>
> Can I suggest a way to look at it, Grant?
>
> In base 10, we represent all numbers as the (possibly infinite) sum of 
> ten raised to some integral power.

Not infinite. If you allow an infinite sequence of digits, you create
numerous paradoxes, not to mention the need for infinite storage.

> 123 is 3 times 1 (ten to the zero power) plus
> 2 times 10 (ten to the one power) plus
> 1 times 100 (ten to the two power)
>
> 123.456 just extends this with
> 4 times 1/10 (ten to the minus one power) plus
> 5 times 1/100 (10**-2) plus
> 6 time 1/1000 (10**-3)
>
> In binary, all the powers are not powers of 10 but powers of two.
>
> So IF you wrote something like 111 it means 1 times 1 plus 1 times 2 
> plus 1 times 4 or 7. A zero anywhere just skips a 2 to that power. If 
> you added a decimal point to make 111.111 the latter part would be 1/2 
> plus 1/4 plus 1/8 or 7/8 which combined might be 7 and 7/8. So any 
> fractions of the form something over 2**N can be made easily and 
> almost everything else cannot be made in finite stretches. How would you
make 2/3 or 3 /10?

Right, this is exactly how place value works.

> But the opposite is something true. In decimal, to make the above it 
> becomes
> 7.875 and to make other fractions of the kind, you need more and more 
> As it happens, all such base-2 compatible streams can be made because 
> each is in some sense a divide by two.
>
> 7/16 = 1/2 * .875 = .4375
> 7/32 = 1/2 * .4375 = .21875
>
> and so on. But this ability is a special case artifact caused by a 
> terminal digit 5 always being able to be divided in tow to make a 25 a 
> unit longer and then again and again. Note 2 and 5 are factors of 10.  
> In the more general case, this fails. In base 7, 3/7 is written easily 
> as 0.3 but the same fraction in decimal is a repeating copy of 
> .428571... which never terminates. A number like 3/7 + 4/49 + 5/343 
> generally cannot be written in base 7 but the opposite is also true 
> that only a approximation of numbers in base 2 or base 10 can ever be 
> written. I am, of course, talking about the part to the right of the 
> decimal. Integers to the left can be written in any base. It is fractional
parts that can end up being nonrepeating.

If you have a number with a finite binary representation, you can guarantee
that it can be represented finitely in decimal too.
Infinitely repeating expansions come from denominators that are coprime with
the numeric base.

> What about pi and e and the square root of 2? I suspect all of them 
> have an infinite sequence with no real repetition (over long enough 
> stretches) in any base! I mean an integer base, of course. The 
> constant e in base e is just 1.

More than "suspect". This has been proven. That's what transcendental means.

I don't think "base e" means the same thing that "base ten" does.
(Normally you'd talk about a base e *logarithm*, which is a completely
different concept.) But if you try to work with a transcendental base like
that, it would be impossible to represent any integer finitely.

(Side point: There are other representations that have different
implications about what repeats and what doesn't. For instance, the decimal
expansion for a square root doesn't repeat, but the continued fraction for
the same square root will. For instance, 7**0.5 is 2;1,1,1,4,1,1,1,4... with
an infinitely repeating four-element unit.)

> I think there have been attempts to use a decimal representation in 
> some accounting packages or database applications that allow any 
> decimal numbers to be faithfully represented and used in calculations. 
> Generally this is not a very efficient process but it can handle 0.3 
> albeit still have no way to deal with transcendental numbers.

Fixed point has been around for a long time (the simplest example being
"work in cents and use integers"), but actual decimal floating-point is
quite unusual. Some databases support it, and REXX used that as its only
numeric form, but it's not hugely popular.

> Let me leave you with Egyptian mathematics. Their use of fractions, 
> WAY BACK WHEN, only had the concept of a reciprocal of an integer. As 
> in for any integer N, there was a fraction of 1/N. They had a concept 
> of 1/3 but not of
> 2/3 or 4/9.
>
> So they added reciprocals to make any more complex fractions. To make 
> 2/3 they added 1/2 plus 1/6 for example.
>
> Since they were not stuck with any one base, all kinds of such 
> combined fractions could be done but of course the square root of 2 or 
> pi were a bit beyond them and for similar reasons.
>
> https://en.wikipedia.org/wiki/Egyptian_fraction

It's interesting as a curiosity, but it makes arithmetic extremely
difficult.

> My point is there are many ways humans can choose to play with numbers 
> and not all of them can easily do the same thing. Roman Numerals were 
> (and
> remain) a horror to do much mathematics with and especially when they 
> play games based on whether a symbol like X is to the left or right of 
> another like C as XC is 90 and CX is 110.

Of course, there are myriad ways to do things. And each one has
implications. Which makes it even more surprising that, when someone sits
down at a computer and asks it to do arithmetic, they can't handle the
different implications.

ChrisA
--
https://mail.python.org/mailman/listinfo/python-list


From greg.ewing at canterbury.ac.nz  Sun Nov 21 00:39:19 2021
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 21 Nov 2021 18:39:19 +1300
Subject: Unexpected behaviour of math.floor, round and int functions
 (rounding)
In-Reply-To: <mailman.89.1637457504.19479.python-list@python.org>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
 <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>
 <mailman.63.1637360151.19479.python-list@python.org>
 <874k87k4a7.fsf@bsb.me.uk>
 <CAPTjJmohqdKMP4srxWQZmu4Xn0+M2t7J8UtL9CDtNQRuOZ8N0A@mail.gmail.com>
 <mailman.67.1637373077.19479.python-list@python.org>
 <87y25jik3i.fsf@bsb.me.uk>
 <CAPTjJmr-Sik-NEzm1m17uXUMXSf8ijT9x=BvX9PYSoW6x=Nf0w@mail.gmail.com>
 <mailman.68.1637385505.19479.python-list@python.org>
 <87sfvri07m.fsf@bsb.me.uk> <61997569.1c69fb81.5eb3c.b508@mx.google.com>
 <041a01d7de70$15b9a3b0$412ceb10$@verizon.net>
 <CAPTjJmqHBrOZsF8xsiQuWhPJ+xJ5FWWmuQbt9UBkyNVjYyc60Q@mail.gmail.com>
 <61999e5d.1c69fb81.a7898.c7e0@mx.google.com>
 <mailman.89.1637457504.19479.python-list@python.org>
Message-ID: <ivu4c9FdmtbU1@mid.individual.net>

On 21/11/21 2:18 pm, Grant Edwards wrote:
> My recollection is that it was quite common back in the days before FP
> hardware was "a thing" on small computers. CPM and DOS compilers for
> various languages often gave the user a choice between binary FP and
> decimal (BCD) FP.

It's also very common for handheld calculators to work in decimal.
Most of HP's classic calculators used a CPU that was specifically
designed for doing BCD arithmetic, and many versions of it didn't
even have a way of doing arithmetic in binary!

-- 
Greg

From dieter at handshake.de  Sun Nov 21 01:14:13 2021
From: dieter at handshake.de (Dieter Maurer)
Date: Sun, 21 Nov 2021 07:14:13 +0100
Subject: pytest segfault, not with -v
In-Reply-To: <CABbU2U_hj_OtW1u3Jk5-3c3kzYtU187RQjmX_-vbsf3kgmC10A@mail.gmail.com>
References: <CABbU2U_cTjL031iQ3S7BtAQM4Yc=rhRF6NzK=_C1kOoe+JwjPA@mail.gmail.com>
 <255c2cb8-b5ea-6e32-d6eb-1765af3967d5@mrabarnett.plus.com>
 <CABbU2U-cWQbJBog-k6vr9pMPpmoY+viHdc9s1DEaihSwaNcB5A@mail.gmail.com>
 <e5d32ffa-eedd-715e-7306-e6c099258193@mrabarnett.plus.com>
 <CABbU2U8z=r3vgukN==fT9dFJN0d1fTLboB8ek+jukG7ASYKzqg@mail.gmail.com>
 <08f263b7-5e26-2dfb-4bf6-1f09470c28b0@mrabarnett.plus.com>
 <CABbU2U_hj_OtW1u3Jk5-3c3kzYtU187RQjmX_-vbsf3kgmC10A@mail.gmail.com>
Message-ID: <24985.58293.668991.392376@ixdm.fritz.box>

Marco Sulla wrote at 2021-11-20 19:07 +0100:
>I know how to check the refcounts, but I don't know how to check the
>memory usage, since it's not a program, it's a simple library. Is
>there not a way to check inside Python the memory usage? I have to use
>a bash script (I'm on Linux)?

If Python was compiled appropriately (with "PYMALLOG_DEBUG"), `sys` contains
the function `_debugmallocstats` which prints details
about Python's memory allocation and free lists.

I was not able to compile Python 2.7 in this way. But the (system) Python 3.6
of Ubuntu was compiled appropriately.


Note that memory leaks usually do not cause segfaults (unless the application
runs out of memory due to the leak).

Your observation shows (apparently) non-deterministic behavior. In those cases,
minor differences (e.g. with/without "-v") can significantly change
the behavior (e.g. segfault or not). Memory management bugs (releasing memory
still in use) are a primary cause for this kind of behavior in Python
applications.

From barry at barrys-emacs.org  Sun Nov 21 08:30:30 2021
From: barry at barrys-emacs.org (Barry)
Date: Sun, 21 Nov 2021 13:30:30 +0000
Subject: pytest segfault, not with -v
In-Reply-To: <CABbU2U-cWQbJBog-k6vr9pMPpmoY+viHdc9s1DEaihSwaNcB5A@mail.gmail.com>
References: <CABbU2U-cWQbJBog-k6vr9pMPpmoY+viHdc9s1DEaihSwaNcB5A@mail.gmail.com>
Message-ID: <F97D3B6E-2F97-4A6F-BF14-B05EC46EA571@barrys-emacs.org>

I would run the whole set of tests under gdb and wait for the segv to happen.
You may find that an isolated test will pass. Sometimes it is a sequence of test
and lead to the segv.

Barry

> On 19 Nov 2021, at 23:48, Marco Sulla <Marco.Sulla.Python at gmail.com> wrote:
> 
> ?On Fri, 19 Nov 2021 at 20:38, MRAB <python at mrabarnett.plus.com> wrote:
>> 
>>> On 2021-11-19 17:48, Marco Sulla wrote:
>>> I have a battery of tests done with pytest. My tests break with a
>>> segfault if I run them normally. If I run them using pytest -v, the
>>> segfault does not happen.
>>> 
>>> What could cause this quantical phenomenon?
>>> 
>> Are you testing an extension that you're compiling? That kind of problem
>> can occur if there's an uninitialised variable or incorrect reference
>> counting (Py_INCREF/Py_DECREF).
> 
> Ok, I know. But why can't it be reproduced if I do pytest -v? This way
> I don't know which test fails.
> Furthermore I noticed that if I remove the __pycache__ dir of tests,
> pytest does not crash, until I re-ran it with the __pycache__ dir
> present.
> This way is very hard for me to understand what caused the segfault.
> I'm starting to think pytest is not good for testing C extensions.
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 


From torriem at gmail.com  Sun Nov 21 10:09:51 2021
From: torriem at gmail.com (Michael Torrie)
Date: Sun, 21 Nov 2021 08:09:51 -0700
Subject: get_axes not present?
In-Reply-To: <476561180.1556266.1637343510565@mail.yahoo.com>
References: <1263450234.1243976.1637228965678.ref@mail.yahoo.com>
 <1263450234.1243976.1637228965678@mail.yahoo.com>
 <500ef579-bd07-6a5c-b652-3b4dad8e4ba4@wichmann.us>
 <203139843.1328468.1637258083048@mail.yahoo.com>
 <afc3960c-c7cf-c09a-703b-57be45793db4@gmail.com>
 <476561180.1556266.1637343510565@mail.yahoo.com>
Message-ID: <a54553d9-a557-f40e-24a7-6dfce16d7eb2@gmail.com>

On 11/19/21 10:38 AM, Mahmood Naderan wrote:
>> And what is the result of plot()?? Is it a valid object, or is it None?
> 
> Well the error happens on the plot() line. I tried to print some information like this:
> <snipped incomplete code>
> Any thoughts on that?

It's not really possible for us to know what is happening since none of
us are in front of your computer with your code in front of us.  I
cannot run any of your posted code excerpts.  The best way to get
assistance here on the list is to create a minimal, self-contained,
run-able, example program that you can post in its entirety here that
demonstrates the issue. Otherwise all anyone can do is make guesses.

Did you read through the exception message?  It is providing a lot of
information. The message is saying that matplotlib is trying to call
.get_figure() on a self.axes list item, but that is not a valid object,
but is None. The question is, why is that list full of Nones?  Farther
up the traceback it notes that on line 66 of process_csv.py, you make a
call to plot_data() passing it the axes. Perhaps there's something wrong
with the axes object or list you are passing in there.  Did you print
that out to make sure that df, cnt, and axes all contain valid things?

All I can tell you is to use the standard debugging techniques you would
use for any problem.  Use either a debugger or print()'s to print out
the contents and types of each variable to verify they contain the data
you think they do.  If you have nested function calls, break them out
separately so you can examine the inputs and outputs of each of them.
If necessary, trace your way through the matplotlib code, but almost
certainly you'll find the problem is in bad data you are passing to
matplotlib.

From grant.b.edwards at gmail.com  Sun Nov 21 10:58:24 2021
From: grant.b.edwards at gmail.com (Grant Edwards)
Date: Sun, 21 Nov 2021 07:58:24 -0800 (PST)
Subject: Unexpected behaviour of math.floor, round and int functions
 (rounding)
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
 <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>
 <mailman.63.1637360151.19479.python-list@python.org>
 <874k87k4a7.fsf@bsb.me.uk>
 <CAPTjJmohqdKMP4srxWQZmu4Xn0+M2t7J8UtL9CDtNQRuOZ8N0A@mail.gmail.com>
 <mailman.67.1637373077.19479.python-list@python.org>
 <87y25jik3i.fsf@bsb.me.uk>
 <CAPTjJmr-Sik-NEzm1m17uXUMXSf8ijT9x=BvX9PYSoW6x=Nf0w@mail.gmail.com>
 <mailman.68.1637385505.19479.python-list@python.org>
 <87sfvri07m.fsf@bsb.me.uk> <61997569.1c69fb81.5eb3c.b508@mx.google.com>
 <041a01d7de70$15b9a3b0$412ceb10$@verizon.net>
 <CAPTjJmqHBrOZsF8xsiQuWhPJ+xJ5FWWmuQbt9UBkyNVjYyc60Q@mail.gmail.com>
 <61999e5d.1c69fb81.a7898.c7e0@mx.google.com>
 <mailman.89.1637457504.19479.python-list@python.org>
 <ivu4c9FdmtbU1@mid.individual.net>
Message-ID: <619a6ca0.1c69fb81.d15e2.977b@mx.google.com>

On 2021-11-21, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> On 21/11/21 2:18 pm, Grant Edwards wrote:
>> My recollection is that it was quite common back in the days before FP
>> hardware was "a thing" on small computers. CPM and DOS compilers for
>> various languages often gave the user a choice between binary FP and
>> decimal (BCD) FP.
>
> It's also very common for handheld calculators to work in decimal.
> Most of HP's classic calculators used a CPU that was specifically
> designed for doing BCD arithmetic, and many versions of it didn't
> even have a way of doing arithmetic in binary!

Yep, IIRC, it was a 4 bit processor because 4 bits is what it takes to
represent one decimal digit. The original Intel ?Processor was a also
4-bit processor designed around BCD rather than base-2 arithmatic
operations.

--
Grant




From nt_mahmood at yahoo.com  Sun Nov 21 11:39:14 2021
From: nt_mahmood at yahoo.com (Mahmood Naderan)
Date: Sun, 21 Nov 2021 16:39:14 +0000 (UTC)
Subject: get_axes not present?
In-Reply-To: <a54553d9-a557-f40e-24a7-6dfce16d7eb2@gmail.com>
References: <1263450234.1243976.1637228965678.ref@mail.yahoo.com>
 <1263450234.1243976.1637228965678@mail.yahoo.com>
 <500ef579-bd07-6a5c-b652-3b4dad8e4ba4@wichmann.us>
 <203139843.1328468.1637258083048@mail.yahoo.com>
 <afc3960c-c7cf-c09a-703b-57be45793db4@gmail.com>
 <476561180.1556266.1637343510565@mail.yahoo.com>
 <a54553d9-a557-f40e-24a7-6dfce16d7eb2@gmail.com>
Message-ID: <661335163.1498134.1637512754986@mail.yahoo.com>

>The best way to get
>assistance here on the list is to create a minimal, self-contained,
>run-able, example program that you can post in its entirety here that
>demonstrates the issue.


I created a sample code with input. Since the code processes a csv file to group input rows, I also included those in this minimal code but those preprocesses are not buggy. In this sample code, I used print() to print necessary information. The error exists in the plot function. I tested the dictionary build before that and it is fine.


Code is available at https://pastebin.com/giAnjJDV? and the input file (test.batch.csv) is available https://pastebin.com/Hdp4Wt9B 

The run command is "python3 test.py". With the versions specified in my system, here is the full output:





$ python3 test.py
Reading file...
matplotlib version =? 3.3.4
pandas version =? 1.2.3
sys version sys.version_info(major=3, minor=8, micro=10, releaselevel='final', serial=0)
Original dictionary =? {'dummy':???? Value
M1????? 0
M2????? 0
M3????? 0, 'K1::foo(bar::z(x,u))':??? Value? Value
0???? 10????? 2
1????? 5????? 2
2???? 10????? 2, 'K2::foo()':??? Value
0???? 20
1???? 10
2???? 15, 'K3::foo(baar::y(z,u))':??? Value
0???? 12
1???? 13
2???? 14, 'K3::foo(bar::y(z,u))':??? Value
0????? 6
1????? 7
2????? 8}
New dictionary for plot =? {'dummy':???? Value
M1????? 0
M2????? 0
M3????? 0, 'K1::foo(bar::z(x,u))':??? Value? Value
0???? 10????? 2
1????? 5????? 2
2???? 10????? 2, 'K3::foo(bar::y(z,u))':??? Value
0????? 6
1????? 7
2????? 8}
Key is? K1::foo(bar::z(x,u))? -> df is???? Value? Value
0???? 10????? 2
1????? 5????? 2
2???? 10????? 2
axes= [<AxesSubplot:> <AxesSubplot:>]
axes[0]= AxesSubplot(0.125,0.53;0.775x0.35)
cnt= 1
row= 1??? 10
2???? 2
Name: 0, dtype: int64
Traceback (most recent call last):
? File "test.py", line 74, in <module>
??? plot_kernels(my_dict2)
? File "test.py", line 52, in plot_kernels
??? plot_dataframe(df, cnt, axes)
? File "test.py", line 36, in plot_dataframe
??? ax1 = row.plot(label=cnt, ax=axes[0], marker='o')?? # Line chart
? File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_core.py", line 955, in __call__
??? return plot_backend.plot(data, kind=kind, **kwargs)
? File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/__init__.py", line 61, in plot
??? plot_obj.generate()
? File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/core.py", line 283, in generate
??? self._adorn_subplots()
? File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/core.py", line 483, in _adorn_subplots
??? all_axes = self._get_subplots()
? File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/core.py", line 903, in _get_subplots
??? ax for ax in self.axes[0].get_figure().get_axes() if isinstance(ax, Subplot)
AttributeError: 'NoneType' object has no attribute 'get_axes'



I am pretty sure that there is a version mismatch because on a system with Pandas 1.3.3 the output should be like https://imgur.com/a/LZ9eAzl

Any feedback is appreciated.


Regards,
Mahmood


From rosuav at gmail.com  Sun Nov 21 12:17:46 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 22 Nov 2021 04:17:46 +1100
Subject: get_axes not present?
In-Reply-To: <661335163.1498134.1637512754986@mail.yahoo.com>
References: <1263450234.1243976.1637228965678.ref@mail.yahoo.com>
 <1263450234.1243976.1637228965678@mail.yahoo.com>
 <500ef579-bd07-6a5c-b652-3b4dad8e4ba4@wichmann.us>
 <203139843.1328468.1637258083048@mail.yahoo.com>
 <afc3960c-c7cf-c09a-703b-57be45793db4@gmail.com>
 <476561180.1556266.1637343510565@mail.yahoo.com>
 <a54553d9-a557-f40e-24a7-6dfce16d7eb2@gmail.com>
 <661335163.1498134.1637512754986@mail.yahoo.com>
Message-ID: <CAPTjJmrFNnZG-pOrOSk5rGc0=RzRkbgZ+-R0OwS3twffODcAPg@mail.gmail.com>

On Mon, Nov 22, 2021 at 3:40 AM Mahmood Naderan via Python-list
<python-list at python.org> wrote:
>   File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/core.py", line 903, in _get_subplots
>     ax for ax in self.axes[0].get_figure().get_axes() if isinstance(ax, Subplot)
> AttributeError: 'NoneType' object has no attribute 'get_axes'
>

My reading of this is that you're trying to do something that depends
on having an associated figure, but you haven't specified the figure.
Look at documentation and examples for the functions you're calling
and see if one of them depends on a figure.

Your example isn't minimal enough for me to be able to pin it down any
better than that, though.

ChrisA

From python at mrabarnett.plus.com  Sun Nov 21 12:44:06 2021
From: python at mrabarnett.plus.com (MRAB)
Date: Sun, 21 Nov 2021 17:44:06 +0000
Subject: get_axes not present?
In-Reply-To: <661335163.1498134.1637512754986@mail.yahoo.com>
References: <1263450234.1243976.1637228965678.ref@mail.yahoo.com>
 <1263450234.1243976.1637228965678@mail.yahoo.com>
 <500ef579-bd07-6a5c-b652-3b4dad8e4ba4@wichmann.us>
 <203139843.1328468.1637258083048@mail.yahoo.com>
 <afc3960c-c7cf-c09a-703b-57be45793db4@gmail.com>
 <476561180.1556266.1637343510565@mail.yahoo.com>
 <a54553d9-a557-f40e-24a7-6dfce16d7eb2@gmail.com>
 <661335163.1498134.1637512754986@mail.yahoo.com>
Message-ID: <661572c9-4d3b-7a45-6e45-e5bf86b3a8ee@mrabarnett.plus.com>

On 2021-11-21 16:39, Mahmood Naderan via Python-list wrote:
>>The best way to get
>>assistance here on the list is to create a minimal, self-contained,
>>run-able, example program that you can post in its entirety here that
>>demonstrates the issue.
> 
> 
> I created a sample code with input. Since the code processes a csv file to group input rows, I also included those in this minimal code but those preprocesses are not buggy. In this sample code, I used print() to print necessary information. The error exists in the plot function. I tested the dictionary build before that and it is fine.
> 
> 
> Code is available at https://pastebin.com/giAnjJDV? and the input file (test.batch.csv) is available https://pastebin.com/Hdp4Wt9B
> 
> The run command is "python3 test.py". With the versions specified in my system, here is the full output:
> 
> 
> 
> 
> 
> $ python3 test.py
> Reading file...
> matplotlib version =? 3.3.4
> pandas version =? 1.2.3
> sys version sys.version_info(major=3, minor=8, micro=10, releaselevel='final', serial=0)
> Original dictionary =? {'dummy':???? Value
> M1????? 0
> M2????? 0
> M3????? 0, 'K1::foo(bar::z(x,u))':??? Value? Value
> 0???? 10????? 2
> 1????? 5????? 2
> 2???? 10????? 2, 'K2::foo()':??? Value
> 0???? 20
> 1???? 10
> 2???? 15, 'K3::foo(baar::y(z,u))':??? Value
> 0???? 12
> 1???? 13
> 2???? 14, 'K3::foo(bar::y(z,u))':??? Value
> 0????? 6
> 1????? 7
> 2????? 8}
> New dictionary for plot =? {'dummy':???? Value
> M1????? 0
> M2????? 0
> M3????? 0, 'K1::foo(bar::z(x,u))':??? Value? Value
> 0???? 10????? 2
> 1????? 5????? 2
> 2???? 10????? 2, 'K3::foo(bar::y(z,u))':??? Value
> 0????? 6
> 1????? 7
> 2????? 8}
> Key is? K1::foo(bar::z(x,u))? -> df is???? Value? Value
> 0???? 10????? 2
> 1????? 5????? 2
> 2???? 10????? 2
> axes= [<AxesSubplot:> <AxesSubplot:>]
> axes[0]= AxesSubplot(0.125,0.53;0.775x0.35)
> cnt= 1
> row= 1??? 10
> 2???? 2
> Name: 0, dtype: int64
> Traceback (most recent call last):
>  ? File "test.py", line 74, in <module>
>  ??? plot_kernels(my_dict2)
>  ? File "test.py", line 52, in plot_kernels
>  ??? plot_dataframe(df, cnt, axes)
>  ? File "test.py", line 36, in plot_dataframe
>  ??? ax1 = row.plot(label=cnt, ax=axes[0], marker='o')?? # Line chart
>  ? File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_core.py", line 955, in __call__
>  ??? return plot_backend.plot(data, kind=kind, **kwargs)
>  ? File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/__init__.py", line 61, in plot
>  ??? plot_obj.generate()
>  ? File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/core.py", line 283, in generate
>  ??? self._adorn_subplots()
>  ? File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/core.py", line 483, in _adorn_subplots
>  ??? all_axes = self._get_subplots()
>  ? File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/core.py", line 903, in _get_subplots
>  ??? ax for ax in self.axes[0].get_figure().get_axes() if isinstance(ax, Subplot)
> AttributeError: 'NoneType' object has no attribute 'get_axes'
> 
> 
> 
> I am pretty sure that there is a version mismatch because on a system with Pandas 1.3.3 the output should be like https://imgur.com/a/LZ9eAzl
> 
> Any feedback is appreciated.
> 
I installed the latest pandas, although on Python 3.10, and the script 
worked without a problem.

From arj.python at gmail.com  Sun Nov 21 12:51:33 2021
From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer)
Date: Sun, 21 Nov 2021 21:51:33 +0400
Subject: Advantages of Default Factory in Dataclasses
In-Reply-To: <4765df8e2335cee6e90bd737c5cebed113696d23.camel@anode.ca>
References: <CADrxXXksfD+nxVT0pPyHjX6jtkqfKDYvyTf11P7eT+Bbtbu81A@mail.gmail.com>
 <4765df8e2335cee6e90bd737c5cebed113696d23.camel@anode.ca>
Message-ID: <CADrxXXmA6gSxN_XLVLY2EpAoaVZ4nP97QfVCMCXLUreSGi67aA@mail.gmail.com>

On Tue, Nov 16, 2021 at 7:17 PM Paul Bryan <pbryan at anode.ca> wrote:

> On Tue, 2021-11-16 at 17:04 +0400, Abdur-Rahmaan Janhangeer wrote:
>
> A simple question: why do we need field(default_factory ) in dataclasses?
>
>
> To initialize a default value when a new instance of the dataclass is
> created. For example, if you want a field to default to a dict. A new dict
> is created for each instance of the dataclass created.
>


Why not have an attribute which returns a deep copy of a dict?

Like the only advantage of default factory is copying whatever we specify?

From pbryan at anode.ca  Sun Nov 21 13:25:23 2021
From: pbryan at anode.ca (Paul Bryan)
Date: Sun, 21 Nov 2021 10:25:23 -0800
Subject: Advantages of Default Factory in Dataclasses
In-Reply-To: <CADrxXXmA6gSxN_XLVLY2EpAoaVZ4nP97QfVCMCXLUreSGi67aA@mail.gmail.com>
References: <CADrxXXksfD+nxVT0pPyHjX6jtkqfKDYvyTf11P7eT+Bbtbu81A@mail.gmail.com>
 <4765df8e2335cee6e90bd737c5cebed113696d23.camel@anode.ca>
 <CADrxXXmA6gSxN_XLVLY2EpAoaVZ4nP97QfVCMCXLUreSGi67aA@mail.gmail.com>
Message-ID: <050b83f489b6e110cee739812b753f399b8cfb6f.camel@anode.ca>

On Sun, 2021-11-21 at 21:51 +0400, Abdur-Rahmaan Janhangeer wrote:
> 
> On Tue, Nov 16, 2021 at 7:17 PM Paul Bryan <pbryan at anode.ca> wrote:
> > On Tue, 2021-11-16 at 17:04 +0400, Abdur-Rahmaan Janhangeer wrote:
> > 
> > > A simple question: why do we need field(default_factory ) in
> > > dataclasses?
> > 
> > 
> > To initialize a default value when a new instance of the dataclass
> > is created. For example, if you want a field to default to a dict.
> > A new dict is created for each instance of the dataclass created.
> > 
> 
> 
> Why not have an attribute which returns a deep copy of a dict?

You can certainly write a default factory to return a deep copy. I'm
not understanding your question about the attribute though. Attribute
in what object? What might the code look like using an attribute?

> Like the only advantage of default factory is copying whatever we
> specify??

The advantage of the default factory is that it can generate a value at
the time a data class is initialized.


From hjp-python at hjp.at  Sun Nov 21 13:23:33 2021
From: hjp-python at hjp.at (Peter J. Holzer)
Date: Sun, 21 Nov 2021 19:23:33 +0100
Subject: Unexpected behaviour of math.floor, round and int functions
 (rounding)
In-Reply-To: <61978dcc$0$20254$426a34cc@news.free.fr>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
Message-ID: <20211121182333.qhdzulxecmmqsj7w@hjp.at>

On 2021-11-19 12:43:07 +0100, ast wrote:
> Le 19/11/2021 ? 03:51, MRAB a ?crit?:
> > On 2021-11-19 02:40, 2QdxY4RzWzUUiLuE at potatochowder.com wrote:
> > > On 2021-11-18 at 23:16:32 -0300,
> > > Ren? Silva Vald?s <rene.silva.valdes at gmail.com> wrote:
> > > > Hello, I would like to report the following issue:
> > > > 
> > > > Working with floats i noticed that:
> > > > 
> > > > int(23.99999999999999/12) returns 1, and
> > > > int(23.999999999999999/12) returns 2
> > > > 
> > > > This implies that int() function is rounding ...
[...]
> > Python 3.10.0 (tags/v3.10.0:b494f59, Oct? 4 2021, 19:00:18) [MSC v.1929
> > 64 bit (AMD64)] on win32
> > Type "help", "copyright", "credits" or "license" for more information.
> >  >>> 23.99999999999999 == 24
> > False
> >  >>> 23.999999999999999 == 24
> > True
> 
> >>> 0.3 + 0.3 + 0.3 == 0.9
> False

Fascinating. The OP ran into the fact that FP numbers have a limited
number of digits in the mantissa (completely independent of the base).
Someone else mentions 0.3 and everybody takes off on that tangent.

        hp

-- 
   _  | Peter J. Holzer    | Story must make more sense than reality.
|_|_) |                    |
| |   | hjp at hjp.at         |    -- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |       challenge!"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://mail.python.org/pipermail/python-list/attachments/20211121/41a195b3/attachment.sig>

From hjp-python at hjp.at  Sun Nov 21 13:40:45 2021
From: hjp-python at hjp.at (Peter J. Holzer)
Date: Sun, 21 Nov 2021 19:40:45 +0100
Subject: Unexpected behaviour of math.floor, round and int functions
 (rounding)
In-Reply-To: <87y25jik3i.fsf@bsb.me.uk>
References: <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
 <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>
 <mailman.63.1637360151.19479.python-list@python.org>
 <874k87k4a7.fsf@bsb.me.uk>
 <CAPTjJmohqdKMP4srxWQZmu4Xn0+M2t7J8UtL9CDtNQRuOZ8N0A@mail.gmail.com>
 <mailman.67.1637373077.19479.python-list@python.org>
 <87y25jik3i.fsf@bsb.me.uk>
Message-ID: <20211121184045.c2vux4zjdtehtwqx@hjp.at>

On 2021-11-20 03:25:53 +0000, Ben Bacarisse wrote:
> Chris Angelico <rosuav at gmail.com> writes:
> 
> > It does mean exactly what it meant in grade school, just as 1/3 means
> > exactly what it meant in grade school. Now try to represent 1/3 on a
> > blackboard, as a decimal fraction. If that's impossible, does it mean
> > that 1/3 doesn't mean 1/3, or that 1/3 can't be represented?
> 
> As you know, it is possible, but let's say we outlaw any finite notation
> for repeated digits...  Why should I convert 1/3 to this particular
> apparently unsuitable representation?

Because you want to use tools which require that particular
representation? Like for example a pocket calculator?

> I will write 1/3 and manipulate that number using factional notation.

On paper, maybe. But if after a few more steps you have fractions like
37645 / 9537654, you might reconsider that choice.

In a program? Yes, there are cases where you really want to use
fractions. That's why fractions.Fraction exists in Python (and similar
datatypes in many other programming languages). But they have their
limits, too (no ? or ?2) and for most problems you don't need them. I
don't actually think I ever used fractions.Fraction in my 7 years of
Python programming. (I think I used Math::BigRat in Perl, but I've been
programming in Perl for a lot longer.)

        hp

-- 
   _  | Peter J. Holzer    | Story must make more sense than reality.
|_|_) |                    |
| |   | hjp at hjp.at         |    -- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |       challenge!"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://mail.python.org/pipermail/python-list/attachments/20211121/c0311577/attachment.sig>

From rosuav at gmail.com  Sun Nov 21 13:43:48 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 22 Nov 2021 05:43:48 +1100
Subject: Unexpected behaviour of math.floor,
 round and int functions (rounding)
In-Reply-To: <20211121184045.c2vux4zjdtehtwqx@hjp.at>
References: <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
 <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>
 <mailman.63.1637360151.19479.python-list@python.org>
 <874k87k4a7.fsf@bsb.me.uk>
 <CAPTjJmohqdKMP4srxWQZmu4Xn0+M2t7J8UtL9CDtNQRuOZ8N0A@mail.gmail.com>
 <mailman.67.1637373077.19479.python-list@python.org>
 <87y25jik3i.fsf@bsb.me.uk> <20211121184045.c2vux4zjdtehtwqx@hjp.at>
Message-ID: <CAPTjJmrGF6-Ci0utkP+TRymkMpSE9M+A8g6vMELyEcXVWhsjhA@mail.gmail.com>

On Mon, Nov 22, 2021 at 5:42 AM Peter J. Holzer <hjp-python at hjp.at> wrote:
> (I think I used Math::BigRat in Perl, but I've been
> programming in Perl for a lot longer.)
>

Rodents Of Unusual Size? I don't think they exist...

ChrisA

From danialmeidac at gmail.com  Sun Nov 21 13:36:23 2021
From: danialmeidac at gmail.com (Daniel Eduardo Almeida Correa)
Date: Sun, 21 Nov 2021 13:36:23 -0500
Subject: doubt About import machine
Message-ID: <CABBN8GnSpUs_dQ0YgZaMeesa7xhgpUWhhyLv8_K7qazwO2xjfg@mail.gmail.com>

Hello, I'm trying to use the machine library in python 3.10 version, but I
can't import it with the pip install machine, could you tell me a way to
solve it  or a python version compatible with the library? Thank you a lot
for your answer.

From jfong at ms4.hinet.net  Sun Nov 21 03:30:18 2021
From: jfong at ms4.hinet.net (Jach Feng)
Date: Sun, 21 Nov 2021 00:30:18 -0800 (PST)
Subject: frozenset can be altered by |=
In-Reply-To: <mailman.57.1637356320.19479.python-list@python.org>
References: <CABbU2U_79XPx1T6EGWEniYkN8_m0oOO6mxx+HNbaLqg_wmT-Gw@mail.gmail.com>
 <mailman.57.1637356320.19479.python-list@python.org>
Message-ID: <1ac403e0-e068-4921-901d-0175c44d73bfn@googlegroups.com>

Marco Sulla ? 2021?11?20? ?????5:12:19 [UTC+8] ??????
> (venv_3_10) marco at buzz:~$ python 
> Python 3.10.0 (heads/3.10-dirty:f6e8b80d20, Nov 18 2021, 19:16:18) 
> [GCC 10.1.1 20200718] on linux 
> Type "help", "copyright", "credits" or "license" for more information. 
> >>> a = frozenset((3, 4)) 
> >>> a 
> frozenset({3, 4}) 
> >>> a |= {5,} 
> >>> a 
> frozenset({3, 4, 5})
There is no CONSTANT in Pyhton as other languages does. The only way to have it is using special naming convention, such as in all capital letters, to remind yourself not to re-assign it:-)

--Jach

From hjp-python at hjp.at  Sun Nov 21 13:59:20 2021
From: hjp-python at hjp.at (Peter J. Holzer)
Date: Sun, 21 Nov 2021 19:59:20 +0100
Subject: Unexpected behaviour of math.floor, round and int functions
 (rounding)
In-Reply-To: <CAPTjJmoUMyOB8NUB6xN1y6W0GUGkYKOS5gxTYSjsi+93+XuBwg@mail.gmail.com>
References: <CAPTjJmohqdKMP4srxWQZmu4Xn0+M2t7J8UtL9CDtNQRuOZ8N0A@mail.gmail.com>
 <mailman.67.1637373077.19479.python-list@python.org>
 <87y25jik3i.fsf@bsb.me.uk>
 <CAPTjJmr-Sik-NEzm1m17uXUMXSf8ijT9x=BvX9PYSoW6x=Nf0w@mail.gmail.com>
 <mailman.68.1637385505.19479.python-list@python.org>
 <87sfvri07m.fsf@bsb.me.uk>
 <61997569.1c69fb81.5eb3c.b508@mx.google.com>
 <mailman.81.1637447020.19479.python-list@python.org>
 <87lf1ih2gj.fsf@bsb.me.uk>
 <CAPTjJmoUMyOB8NUB6xN1y6W0GUGkYKOS5gxTYSjsi+93+XuBwg@mail.gmail.com>
Message-ID: <20211121185920.idkohal6o6iwpmyr@hjp.at>

On 2021-11-21 10:57:55 +1100, Chris Angelico wrote:
> And if decimal floating point were common, other "surprise" behaviour
> would be cited, like how x < y and (x+y)/2 < x.

Yup. Took me a bit to find an example, but this can happen. My HP-48
calculator uses a mantissa of 12 decimal digits.

666666666666 + 666666666667 = 1333333333330
1333333333330 / 2 = 666666666665

666666666665 < 666666666666. QED.

        hp


-- 
   _  | Peter J. Holzer    | Story must make more sense than reality.
|_|_) |                    |
| |   | hjp at hjp.at         |    -- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |       challenge!"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://mail.python.org/pipermail/python-list/attachments/20211121/4444e31b/attachment.sig>

From hjp-python at hjp.at  Sun Nov 21 14:04:04 2021
From: hjp-python at hjp.at (Peter J. Holzer)
Date: Sun, 21 Nov 2021 20:04:04 +0100
Subject: Unexpected behaviour of math.floor, round and int functions
 (rounding)
In-Reply-To: <CAPTjJmrGF6-Ci0utkP+TRymkMpSE9M+A8g6vMELyEcXVWhsjhA@mail.gmail.com>
References: <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
 <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>
 <mailman.63.1637360151.19479.python-list@python.org>
 <874k87k4a7.fsf@bsb.me.uk>
 <CAPTjJmohqdKMP4srxWQZmu4Xn0+M2t7J8UtL9CDtNQRuOZ8N0A@mail.gmail.com>
 <mailman.67.1637373077.19479.python-list@python.org>
 <87y25jik3i.fsf@bsb.me.uk> <20211121184045.c2vux4zjdtehtwqx@hjp.at>
 <CAPTjJmrGF6-Ci0utkP+TRymkMpSE9M+A8g6vMELyEcXVWhsjhA@mail.gmail.com>
Message-ID: <20211121190404.say7qaavhgh74vvw@hjp.at>

On 2021-11-22 05:43:48 +1100, Chris Angelico wrote:
> On Mon, Nov 22, 2021 at 5:42 AM Peter J. Holzer <hjp-python at hjp.at> wrote:
> > (I think I used Math::BigRat in Perl, but I've been
> > programming in Perl for a lot longer.)
> 
> Rodents Of Unusual Size? I don't think they exist...

https://www.girlgeniusonline.com/comic.php?date=20210614 ;-)

        hp

-- 
   _  | Peter J. Holzer    | Story must make more sense than reality.
|_|_) |                    |
| |   | hjp at hjp.at         |    -- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |       challenge!"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://mail.python.org/pipermail/python-list/attachments/20211121/427fc7f0/attachment.sig>

From nt_mahmood at yahoo.com  Sun Nov 21 14:11:57 2021
From: nt_mahmood at yahoo.com (Mahmood Naderan)
Date: Sun, 21 Nov 2021 19:11:57 +0000 (UTC)
Subject: get_axes not present?
In-Reply-To: <661572c9-4d3b-7a45-6e45-e5bf86b3a8ee@mrabarnett.plus.com>
References: <1263450234.1243976.1637228965678.ref@mail.yahoo.com>
 <1263450234.1243976.1637228965678@mail.yahoo.com>
 <500ef579-bd07-6a5c-b652-3b4dad8e4ba4@wichmann.us>
 <203139843.1328468.1637258083048@mail.yahoo.com>
 <afc3960c-c7cf-c09a-703b-57be45793db4@gmail.com>
 <476561180.1556266.1637343510565@mail.yahoo.com>
 <a54553d9-a557-f40e-24a7-6dfce16d7eb2@gmail.com>
 <661335163.1498134.1637512754986@mail.yahoo.com>
 <661572c9-4d3b-7a45-6e45-e5bf86b3a8ee@mrabarnett.plus.com>
Message-ID: <1350778971.1819772.1637521917360@mail.yahoo.com>

>I installed the latest pandas, although on Python 3.10, and the script
>worked without a problem.


Yes as I wrote it works with 1.3.3 but mine is 1.2.3.
I am trying to keep the current version because of the possible future consequences. In the end maybe I have to upgrade the pandas.


Regards,
Mahmood




From nt_mahmood at yahoo.com  Sun Nov 21 14:21:52 2021
From: nt_mahmood at yahoo.com (Mahmood Naderan)
Date: Sun, 21 Nov 2021 19:21:52 +0000 (UTC)
Subject: get_axes not present?
In-Reply-To: <CAPTjJmrFNnZG-pOrOSk5rGc0=RzRkbgZ+-R0OwS3twffODcAPg@mail.gmail.com>
References: <1263450234.1243976.1637228965678.ref@mail.yahoo.com>
 <1263450234.1243976.1637228965678@mail.yahoo.com>
 <500ef579-bd07-6a5c-b652-3b4dad8e4ba4@wichmann.us>
 <203139843.1328468.1637258083048@mail.yahoo.com>
 <afc3960c-c7cf-c09a-703b-57be45793db4@gmail.com>
 <476561180.1556266.1637343510565@mail.yahoo.com>
 <a54553d9-a557-f40e-24a7-6dfce16d7eb2@gmail.com>
 <661335163.1498134.1637512754986@mail.yahoo.com>
 <CAPTjJmrFNnZG-pOrOSk5rGc0=RzRkbgZ+-R0OwS3twffODcAPg@mail.gmail.com>
Message-ID: <1544875209.283662.1637522512809@mail.yahoo.com>


>Your example isn't minimal enough for me to be able to pin it down any
>better than that, though.

Chris,
I was able to simply it even further. Please look at this:


$ cat test.batch.csv
Value,Value
10,2
5,2
10,2


$ cat test.py
import pandas as pd
import csv,sys
import matplotlib
import matplotlib.pyplot as plt

df = pd.read_csv('test.batch.csv')
print(df)

def plot_dataframe(df, cnt, axes):
??? df.columns = range(1, len(df.columns)+1)?? # Ignore the column header
??? row = df.iloc[0].astype(int)? # First row in the dataframe
??? plt.subplot(2, 1, 1)
??? print("axes=", axes)
??? print("axes[0]=", axes[0])
??? print("cnt=", cnt)
??? print("row=", row)
??? ax1 = row.plot(label=cnt, ax=axes[0], marker='o')?? # Line chart
??? ax1.set_ylabel( 'test', fontsize=15 )
??? plt.subplot(2, 1, 2)
??? df2 = row.value_counts()
??? df2.reindex().plot(kind='bar', label=cnt, ax=axes[1])?? # Histogram

def plot_kernels(df):
??? fig,axes = plt.subplots(2,1, figsize=(20, 15))
??? cnt=1
??? plot_dataframe(df, cnt, axes)
??? cnt = cnt + 1
??? for ax in axes:
??????? ax.legend()
??? plt.show()

print("matplotlib version = ",? matplotlib.__version__)
print("pandas version = ", pd.__version__)
print("sys version", sys.version_info)

plot_kernels(df)




And the output is


$ python3 test.py
?? Value? Value.1
0???? 10??????? 2
1????? 5??????? 2
2???? 10??????? 2
matplotlib version =? 3.3.4
pandas version =? 1.2.3
sys version sys.version_info(major=3, minor=8, micro=10, releaselevel='final', serial=0)
axes= [<AxesSubplot:> <AxesSubplot:>]
axes[0]= AxesSubplot(0.125,0.53;0.775x0.35)
cnt= 1
row= 1??? 10
2???? 2
Name: 0, dtype: int64
Traceback (most recent call last):
? File "test.py", line 41, in <module>
??? plot_kernels(df)
? File "test.py", line 29, in plot_kernels
??? plot_dataframe(df, cnt, axes)
? File "test.py", line 19, in plot_dataframe
??? ax1 = row.plot(label=cnt, ax=axes[0], marker='o')?? # Line chart
? File "/home/mnaderan/.local/lib/python3.8/site-packages/pandas/plotting/_core.py", line 955, in __call__
??? return plot_backend.plot(data, kind=kind, **kwargs)
? File "/home/mnaderan/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/__init__.py", line 61, in plot
??? plot_obj.generate()
? File "/home/mnaderan/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/core.py", line 283, in generate
??? self._adorn_subplots()
? File "/home/mnaderan/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/core.py", line 483, in _adorn_subplots
??? all_axes = self._get_subplots()
? File "/home/mnaderan/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/core.py", line 903, in _get_subplots
??? ax for ax in self.axes[0].get_figure().get_axes() if isinstance(ax, Subplot)
AttributeError: 'NoneType' object has no attribute 'get_axes'



Any idea about that?



Regards,
Mahmood


From python at mrabarnett.plus.com  Sun Nov 21 14:56:51 2021
From: python at mrabarnett.plus.com (MRAB)
Date: Sun, 21 Nov 2021 19:56:51 +0000
Subject: doubt About import machine
In-Reply-To: <CABBN8GnSpUs_dQ0YgZaMeesa7xhgpUWhhyLv8_K7qazwO2xjfg@mail.gmail.com>
References: <CABBN8GnSpUs_dQ0YgZaMeesa7xhgpUWhhyLv8_K7qazwO2xjfg@mail.gmail.com>
Message-ID: <8a5a1f96-fda9-86a0-7dbb-73ca09026561@mrabarnett.plus.com>

On 2021-11-21 18:36, Daniel Eduardo Almeida Correa wrote:
> Hello, I'm trying to use the machine library in python 3.10 version, but I
> can't import it with the pip install machine, could you tell me a way to
> solve it  or a python version compatible with the library? Thank you a lot
> for your answer.
> 
Are you talking about the "machine" module of MicroPython? I believe 
that's specific to MicroPython.

From pfeiffer at cs.nmsu.edu  Sun Nov 21 17:34:10 2021
From: pfeiffer at cs.nmsu.edu (Joe Pfeiffer)
Date: Sun, 21 Nov 2021 15:34:10 -0700
Subject: doubt About import machine
References: <CABBN8GnSpUs_dQ0YgZaMeesa7xhgpUWhhyLv8_K7qazwO2xjfg@mail.gmail.com>
 <mailman.108.1637520892.19479.python-list@python.org>
Message-ID: <1b4k85kujh.fsf@pfeifferfamily.net>

Daniel Eduardo Almeida Correa <danialmeidac at gmail.com> writes:

> Hello, I'm trying to use the machine library in python 3.10 version, but I
> can't import it with the pip install machine, could you tell me a way to
> solve it  or a python version compatible with the library? Thank you a lot
> for your answer.

The "machine" package on pypi appears to only be an example of best
practices for a python package, and doesn't actually contain anything
useful.  Installing it doesn't actually give you a package called
"machine" that you can import; it lets you import something called
"sample" instead, which contains

def main():
    """Entry point for the application script"""
    print("Call your main application code here")

as its __init__()

I'll note that I'm sort of a beginner with python so I may be speaking
out of turn, but it strikes me as really unlikely that the "best
practice" for a package named "machine" would put its code in a
directory named "sample".

Following the home page link for the project leads to a 404 on github

From greg.ewing at canterbury.ac.nz  Sun Nov 21 20:02:01 2021
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 22 Nov 2021 14:02:01 +1300
Subject: Unexpected behaviour of math.floor, round and int functions
 (rounding)
In-Reply-To: <mailman.99.1637510307.19479.python-list@python.org>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <87fsrrkdyo.fsf@bsb.me.uk>
 <CAPTjJmpQr4Mua+S2aAGibNaNj_oYatNJn-pqVZykcjAzou+d2Q@mail.gmail.com>
 <mailman.63.1637360151.19479.python-list@python.org>
 <874k87k4a7.fsf@bsb.me.uk>
 <CAPTjJmohqdKMP4srxWQZmu4Xn0+M2t7J8UtL9CDtNQRuOZ8N0A@mail.gmail.com>
 <mailman.67.1637373077.19479.python-list@python.org>
 <87y25jik3i.fsf@bsb.me.uk>
 <CAPTjJmr-Sik-NEzm1m17uXUMXSf8ijT9x=BvX9PYSoW6x=Nf0w@mail.gmail.com>
 <mailman.68.1637385505.19479.python-list@python.org>
 <87sfvri07m.fsf@bsb.me.uk> <61997569.1c69fb81.5eb3c.b508@mx.google.com>
 <041a01d7de70$15b9a3b0$412ceb10$@verizon.net>
 <CAPTjJmqHBrOZsF8xsiQuWhPJ+xJ5FWWmuQbt9UBkyNVjYyc60Q@mail.gmail.com>
 <61999e5d.1c69fb81.a7898.c7e0@mx.google.com>
 <mailman.89.1637457504.19479.python-list@python.org>
 <ivu4c9FdmtbU1@mid.individual.net>
 <619a6ca0.1c69fb81.d15e2.977b@mx.google.com>
 <mailman.99.1637510307.19479.python-list@python.org>
Message-ID: <j008g9Fq5g7U1@mid.individual.net>

On 22/11/21 4:58 am, Grant Edwards wrote:
> Yep, IIRC, it was a 4 bit processor because 4 bits is what it takes to
> represent one decimal digit.

That was the Saturn, first used in the HP-71B. The original
architecture (known as the "Nut")was weirder than that. It operated
serially on 56 bit words (14 BCD digits), and the instructions
had options for operating on various fields of a floating-point
number (mantissa, exponent, sign, etc.)

-- 
Greg

From nt_mahmood at yahoo.com  Mon Nov 22 04:03:53 2021
From: nt_mahmood at yahoo.com (Mahmood Naderan)
Date: Mon, 22 Nov 2021 09:03:53 +0000 (UTC)
Subject: About get_axes() in Pandas 1.2.3
References: <1435065197.1906229.1637571833565.ref@mail.yahoo.com>
Message-ID: <1435065197.1906229.1637571833565@mail.yahoo.com>

Hi

I asked a question some days ago, but due to the lack of minimal producing code, the topic got a bit messy. So, I have decided to ask it in a new topic with a clear minimum code.

With Pandas 1.2.3 and Matplotlib 3.3.4, the following plot() functions returns error and I don't know what is wrong with that.



import pandas as pd
import csv,sys
import matplotlib
import matplotlib.pyplot as plt

df = pd.read_csv('test.batch.csv')
print(df)

print("matplotlib version = ",??matplotlib.__version__)
print("pandas version = ", pd.__version__)
print("sys version", sys.version_info)

fig,axes = plt.subplots(2,1, figsize=(20, 15))
df.columns = range(1, len(df.columns)+1)?? # Ignore the column header
row = df.iloc[0].astype(int)??# First row in the dataframe
plt.subplot(2, 1, 1)
print("axes=", axes)
print("axes[0]=", axes[0])
print("row=", row)????
ax1 = row.plot(ax=axes[0])?? # Line chart???? <-- ERROR
ax1.set_ylabel( 'test' )
plt.subplot(2, 1, 2)
df2 = row.value_counts()
df2.reindex().plot(kind='bar', ax=axes[1])?? # Histogram

plt.show()




The output is



$ cat test.batch.csv
Value,Value
10,2
5,2
10,2

$ python3 test.py
?? Value??Value.1
0???? 10????????2
1??????5????????2
2???? 10????????2
matplotlib version =??3.3.4
pandas version =??1.2.3
sys version sys.version_info(major=3, minor=8, micro=10, releaselevel='final', serial=0)
axes= [<AxesSubplot:> <AxesSubplot:>]
axes[0]= AxesSubplot(0.125,0.53;0.775x0.35)
row= 1????10
2???? 2
Name: 0, dtype: int64
Traceback (most recent call last):
??File "test.py", line 20, in <module>
????ax1 = row.plot(ax=axes[0])?? # Line chart
??File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_core.py", line 955, in __call__
????return plot_backend.plot(data, kind=kind, **kwargs)
??File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/__init__.py", line 61, in plot
????plot_obj.generate()
??File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/core.py", line 283, in generate
????self._adorn_subplots()
??File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/core.py", line 483, in _adorn_subplots
????all_axes = self._get_subplots()
??File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/core.py", line 903, in _get_subplots
????ax for ax in self.axes[0].get_figure().get_axes() if isinstance(ax, Subplot)
AttributeError: 'NoneType' object has no attribute 'get_axes'



Although the plot() crashes, I see that row and axes variables are valid. So, I wonder what is the workaround for this code without upgrading? Pandas or Matplotlib. Any idea?



Regards,
Mahmood

From David.Raymond at tomtom.com  Mon Nov 22 08:51:13 2021
From: David.Raymond at tomtom.com (David Raymond)
Date: Mon, 22 Nov 2021 13:51:13 +0000
Subject: frozenset can be altered by |=
In-Reply-To: <CAPTjJmq=4+yEhibTJoNA3QVUNR1HmScgnQgx3NtQRpgG4aNY1Q@mail.gmail.com>
References: <CABbU2U_79XPx1T6EGWEniYkN8_m0oOO6mxx+HNbaLqg_wmT-Gw@mail.gmail.com>
 <CAPTjJmq=4+yEhibTJoNA3QVUNR1HmScgnQgx3NtQRpgG4aNY1Q@mail.gmail.com>
Message-ID: <AM0PR07MB57000CEF0CEABBD27EA941DA879F9@AM0PR07MB5700.eurprd07.prod.outlook.com>

>> (venv_3_10) marco at buzz:~$ python
>> Python 3.10.0 (heads/3.10-dirty:f6e8b80d20, Nov 18 2021, 19:16:18)
>> [GCC 10.1.1 20200718] on linux
>> Type "help", "copyright", "credits" or "license" for more information.
>> >>> a = frozenset((3, 4))
>> >>> a
>> frozenset({3, 4})
>> >>> a |= {5,}
>> >>> a
>> frozenset({3, 4, 5})
> 
> That's the same as how "x = 4; x += 1" can "alter" four into five.
> 
> >>> a = frozenset((3, 4))
> >>> id(a), a
> (140545764976096, frozenset({3, 4}))
> >>> a |= {5,}
> >>> id(a), a
> (140545763014944, frozenset({3, 4, 5}))
> 
> It's a different frozenset.
> 
> ChrisA

Another possible option is instead of

a |= {5,}

change it to

a.update({5,})

If a is a regular set it will update the original object, and if a is a frozenset it will raise an AttributeError. Which may not be what you want, but at least it won't quietly do something you weren't expecting.

It is a little confusing since the docs list this in a section that says they don't apply to frozensets, and lists the two versions next to each other as the same thing.

https://docs.python.org/3.9/library/stdtypes.html#set-types-set-frozenset

The following table lists operations available for set that do not apply to immutable instances of frozenset:

update(*others)
set |= other | ...

    Update the set, adding elements from all others.

From rosuav at gmail.com  Mon Nov 22 08:57:39 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 23 Nov 2021 00:57:39 +1100
Subject: frozenset can be altered by |=
In-Reply-To: <AM0PR07MB57000CEF0CEABBD27EA941DA879F9@AM0PR07MB5700.eurprd07.prod.outlook.com>
References: <CABbU2U_79XPx1T6EGWEniYkN8_m0oOO6mxx+HNbaLqg_wmT-Gw@mail.gmail.com>
 <CAPTjJmq=4+yEhibTJoNA3QVUNR1HmScgnQgx3NtQRpgG4aNY1Q@mail.gmail.com>
 <AM0PR07MB57000CEF0CEABBD27EA941DA879F9@AM0PR07MB5700.eurprd07.prod.outlook.com>
Message-ID: <CAPTjJmpprmCdjOJBz5FVecV6KqKFRC6Y7bOsNoTP+W-t-k3vew@mail.gmail.com>

On Tue, Nov 23, 2021 at 12:52 AM David Raymond <David.Raymond at tomtom.com> wrote:
> It is a little confusing since the docs list this in a section that says they don't apply to frozensets, and lists the two versions next to each other as the same thing.
>
> https://docs.python.org/3.9/library/stdtypes.html#set-types-set-frozenset
>
> The following table lists operations available for set that do not apply to immutable instances of frozenset:
>
> update(*others)
> set |= other | ...
>
>     Update the set, adding elements from all others.

Yeah, it's a little confusing, but at the language level, something
that doesn't support |= will implicitly support it using the expanded
version:

a |= b
a = a | b

and in the section above, you can see that frozensets DO support the
Or operator.

By not having specific behaviour on the |= operator, frozensets
implicitly fall back on this default.

ChrisA

From ast at invalid  Mon Nov 22 09:39:23 2021
From: ast at invalid (ast)
Date: Mon, 22 Nov 2021 15:39:23 +0100
Subject: copy.copy
Message-ID: <619bab9d$0$8899$426a34cc@news.free.fr>

Hi,

 >>> a = 6
 >>> b = 6
 >>> a is b
True

ok, we all know that Python creates a sole instance
with small integers, but:

 >>> import copy
 >>> b = copy.copy(a)
 >>> a is b
True

I was expecting False

From jon+usenet at unequivocal.eu  Mon Nov 22 10:02:05 2021
From: jon+usenet at unequivocal.eu (Jon Ribbens)
Date: Mon, 22 Nov 2021 15:02:05 -0000 (UTC)
Subject: copy.copy
References: <619bab9d$0$8899$426a34cc@news.free.fr>
Message-ID: <slrnspnc7d.383v.jon+usenet@raven.unequivocal.eu>

On 2021-11-22, ast <ast at invalid> wrote:
> Hi,
>
> >>> a = 6
> >>> b = 6
> >>> a is b
> True
>
> ok, we all know that Python creates a sole instance
> with small integers, but:
>
> >>> import copy
> >>> b = copy.copy(a)
> >>> a is b
> True
>
> I was expecting False

Why did you expect False?

For immutable types, copy(foo) just returns foo.

From ast at invalid  Mon Nov 22 10:29:41 2021
From: ast at invalid (ast)
Date: Mon, 22 Nov 2021 16:29:41 +0100
Subject: copy.copy
In-Reply-To: <slrnspnc7d.383v.jon+usenet@raven.unequivocal.eu>
References: <619bab9d$0$8899$426a34cc@news.free.fr>
 <slrnspnc7d.383v.jon+usenet@raven.unequivocal.eu>
Message-ID: <619bb765$0$8916$426a34cc@news.free.fr>

Le 22/11/2021 ? 16:02, Jon Ribbens a ?crit?:
> On 2021-11-22, ast <ast at invalid> wrote:

> 
> For immutable types, copy(foo) just returns foo.
> 

ok, thx

From torriem at gmail.com  Mon Nov 22 12:13:03 2021
From: torriem at gmail.com (Michael Torrie)
Date: Mon, 22 Nov 2021 10:13:03 -0700
Subject: About get_axes() in Pandas 1.2.3
In-Reply-To: <1435065197.1906229.1637571833565@mail.yahoo.com>
References: <1435065197.1906229.1637571833565.ref@mail.yahoo.com>
 <1435065197.1906229.1637571833565@mail.yahoo.com>
Message-ID: <d45ba6b2-2444-7aa0-77b6-010dc9c9874b@gmail.com>

On 11/22/21 2:03 AM, Mahmood Naderan via Python-list wrote:
> Hi
> 
> I asked a question some days ago, but due to the lack of minimal
> producing code, the topic got a bit messy. So, I have decided to ask
> it in a new topic with a clear minimum code.
> import pandas as pd
> import csv,sys
> import matplotlib
> import matplotlib.pyplot as plt
> 
> df = pd.read_csv('test.batch.csv')
> print(df)
> 
> print("matplotlib version = ",??matplotlib.__version__)
> print("pandas version = ", pd.__version__)
> print("sys version", sys.version_info)
> 
> fig,axes = plt.subplots(2,1, figsize=(20, 15))
               ^^^^^^^^^^^^^
I can help you narrow it down a bit. The problem actually occurs inside
this function call somehow. You can verify this by doing this:

fig,axes = plt.subplots(2,1, figsize=(20, 15))
print ("axes[0].get_figure()=",axes[0].get_figure())

You'll find that get_figure() is returning None, when it should be
returning Figure(2000x1500). So plt.subplots is not doing something
properly which was corrected at some point. Oddly enough, with pandas
1.1.4 and matplotlib 3.2.2 (which is what my system has by default),
there is no error, although the graph is blank.

In my venv, when I upgrade matplotlib from 3.3.4 to 3.5, the problem
also goes away.  3.4.0 also works.

Honestly your solution is going to be to provide a virtual environment
with your script.  That way you can bundle the appropriate dependencies
without modifying anything on the host system.


From nt_mahmood at yahoo.com  Mon Nov 22 13:14:43 2021
From: nt_mahmood at yahoo.com (Mahmood Naderan)
Date: Mon, 22 Nov 2021 18:14:43 +0000 (UTC)
Subject: About get_axes() in Pandas 1.2.3
In-Reply-To: <d45ba6b2-2444-7aa0-77b6-010dc9c9874b@gmail.com>
References: <1435065197.1906229.1637571833565.ref@mail.yahoo.com>
 <1435065197.1906229.1637571833565@mail.yahoo.com>
 <d45ba6b2-2444-7aa0-77b6-010dc9c9874b@gmail.com>
Message-ID: <264730935.107521.1637604883030@mail.yahoo.com>

>I can help you narrow it down a bit. The problem actually occurs inside
>this function call somehow. You can verify this by doing this:
>
>
>fig,axes = plt.subplots(2,1, figsize=(20, 15))
>
>print ("axes[0].get_figure()=",axes[0].get_figure())
>
>You'll find that get_figure() is returning None, when it should be
>returning Figure(2000x1500). So plt.subplots is not doing something
>properly which was corrected at some point. Oddly enough, with pandas
>1.1.4 and matplotlib 3.2.2 (which is what my system has by default),
>there is no error, although the graph is blank.
>
>In my venv, when I upgrade matplotlib from 3.3.4 to 3.5, the problem
>also goes away.? 3.4.0 also works.
>
>Honestly your solution is going to be to provide a virtual environment
>with your script.? That way you can bundle the appropriate dependencies
>without modifying anything on the host system.



Thanks for the feedback. You are right.
I agree that virtualenv is the most safest method at this time.


Regards,
Mahmood



From Marco.Sulla.Python at gmail.com  Mon Nov 22 13:54:22 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Mon, 22 Nov 2021 19:54:22 +0100
Subject: frozenset can be altered by |=
In-Reply-To: <CAPTjJmpprmCdjOJBz5FVecV6KqKFRC6Y7bOsNoTP+W-t-k3vew@mail.gmail.com>
References: <CABbU2U_79XPx1T6EGWEniYkN8_m0oOO6mxx+HNbaLqg_wmT-Gw@mail.gmail.com>
 <CAPTjJmq=4+yEhibTJoNA3QVUNR1HmScgnQgx3NtQRpgG4aNY1Q@mail.gmail.com>
 <AM0PR07MB57000CEF0CEABBD27EA941DA879F9@AM0PR07MB5700.eurprd07.prod.outlook.com>
 <CAPTjJmpprmCdjOJBz5FVecV6KqKFRC6Y7bOsNoTP+W-t-k3vew@mail.gmail.com>
Message-ID: <CABbU2U-rDQr0xkySeMsArbnox32TMdPO28SnbWtjbRhU7xzERA@mail.gmail.com>

Yes, and you do this regularly. Indeed integers, for example, are immutables and

a = 0
a += 1

is something you do dozens of times, and you simply don't think that
another object is created and substituted for the variable named `a`.

On Mon, 22 Nov 2021 at 14:59, Chris Angelico <rosuav at gmail.com> wrote:
>
> On Tue, Nov 23, 2021 at 12:52 AM David Raymond <David.Raymond at tomtom.com> wrote:
> > It is a little confusing since the docs list this in a section that says they don't apply to frozensets, and lists the two versions next to each other as the same thing.
> >
> > https://docs.python.org/3.9/library/stdtypes.html#set-types-set-frozenset
> >
> > The following table lists operations available for set that do not apply to immutable instances of frozenset:
> >
> > update(*others)
> > set |= other | ...
> >
> >     Update the set, adding elements from all others.
>
> Yeah, it's a little confusing, but at the language level, something
> that doesn't support |= will implicitly support it using the expanded
> version:
>
> a |= b
> a = a | b
>
> and in the section above, you can see that frozensets DO support the
> Or operator.
>
> By not having specific behaviour on the |= operator, frozensets
> implicitly fall back on this default.
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list

From arnaud at sphaero.org  Tue Nov 23 07:07:30 2021
From: arnaud at sphaero.org (Arnaud Loonstra)
Date: Tue, 23 Nov 2021 13:07:30 +0100
Subject: Embedding Python crash on PyTuple_New
Message-ID: <d0b57569-ef0d-3ab4-9918-9cd498f1e658@sphaero.org>

Hi,

I've got Python embedded successfully in a program up until now as I'm 
now running into weird GC related segfaults. I'm currently trying to 
debug this but my understanding of CPython limits me here.

I'm creating a Tuple in C but it crashes on creating it after a while. 
It doesn't make sense which makes me wonder something else must be 
happening? Could be it just crashes here because the GC is cleaning up 
stuff completely unrelated to the allocation of the new tuple? How can I 
troubleshoot this?

I've got CPython compiled with  --with-valgrind --without-pymalloc 
--with-pydebug

In C I'm creating a tuple with the following method:

static PyObject *
s_py_zosc_tuple(pythonactor_t *self, zosc_t *oscmsg)
{
     assert(self);
     assert(oscmsg);
     char *format = zosc_format(oscmsg);

     PyObject *rettuple = PyTuple_New((Py_ssize_t) strlen(format) );

It segfaults here (frame 16) after 320 times (consistently) 


1   __GI_raise             raise.c          49   0x7ffff72c4e71
2   __GI_abort             abort.c          79   0x7ffff72ae536
3   fatal_error            pylifecycle.c    2183 0x7ffff7d84b4f
4   Py_FatalError          pylifecycle.c    2193 0x7ffff7d878b2
5   _PyObject_AssertFailed object.c         2200 0x7ffff7c93cf2
6   visit_decref           gcmodule.c       378  0x7ffff7dadfd5
7   tupletraverse          tupleobject.c    623  0x7ffff7ca3e81
8   subtract_refs          gcmodule.c       406  0x7ffff7dad340
9   collect                gcmodule.c       1054 0x7ffff7dae838
10  collect_with_callback  gcmodule.c       1240 0x7ffff7daf17b
11  collect_generations    gcmodule.c       1262 0x7ffff7daf3f6
12  _PyObject_GC_Alloc     gcmodule.c       1977 0x7ffff7daf4f2
13  _PyObject_GC_Malloc    gcmodule.c       1987 0x7ffff7dafebc
14  _PyObject_GC_NewVar    gcmodule.c       2016 0x7ffff7daffa5
15  PyTuple_New            tupleobject.c    118  0x7ffff7ca4da7
16  s_py_zosc_tuple        pythonactor.c    366  0x55555568cc82
17  pythonactor_socket     pythonactor.c    664  0x55555568dac7
18  pythonactor_handle_msg pythonactor.c    862  0x55555568e472
19  pythonactor_handler    pythonactor.c    828  0x55555568e2e2
20  sphactor_actor_run     sphactor_actor.c 855  0x5555558cb268
... <More>

Any pointer really appreciated.

Rg,

Arnaud

From arnaud at sphaero.org  Tue Nov 23 09:20:19 2021
From: arnaud at sphaero.org (Arnaud Loonstra)
Date: Tue, 23 Nov 2021 15:20:19 +0100
Subject: Embedding Python crash on PyTuple_New
In-Reply-To: <d0b57569-ef0d-3ab4-9918-9cd498f1e658@sphaero.org>
References: <d0b57569-ef0d-3ab4-9918-9cd498f1e658@sphaero.org>
Message-ID: <254bc208-244c-218d-9fd8-e944f5576cb9@sphaero.org>

On 23-11-2021 13:07, Arnaud Loonstra wrote:
> Hi,
> 
> I've got Python embedded successfully in a program up until now as I'm 
> now running into weird GC related segfaults. I'm currently trying to 
> debug this but my understanding of CPython limits me here.
> 
> I'm creating a Tuple in C but it crashes on creating it after a while. 
> It doesn't make sense which makes me wonder something else must be 
> happening? Could be it just crashes here because the GC is cleaning up 
> stuff completely unrelated to the allocation of the new tuple? How can I 
> troubleshoot this?
> 
> I've got CPython compiled with? --with-valgrind --without-pymalloc 
> --with-pydebug
> 
> In C I'm creating a tuple with the following method:
> 
> static PyObject *
> s_py_zosc_tuple(pythonactor_t *self, zosc_t *oscmsg)
> {
>  ??? assert(self);
>  ??? assert(oscmsg);
>  ??? char *format = zosc_format(oscmsg);
> 
>  ??? PyObject *rettuple = PyTuple_New((Py_ssize_t) strlen(format) );
> 
> It segfaults here (frame 16) after 320 times (consistently)
> 
> 1?? __GI_raise???????????? raise.c????????? 49?? 0x7ffff72c4e71
> 2?? __GI_abort???????????? abort.c????????? 79?? 0x7ffff72ae536
> 3?? fatal_error??????????? pylifecycle.c??? 2183 0x7ffff7d84b4f
> 4?? Py_FatalError????????? pylifecycle.c??? 2193 0x7ffff7d878b2
> 5?? _PyObject_AssertFailed object.c???????? 2200 0x7ffff7c93cf2
> 6?? visit_decref?????????? gcmodule.c?????? 378? 0x7ffff7dadfd5
> 7?? tupletraverse????????? tupleobject.c??? 623? 0x7ffff7ca3e81
> 8?? subtract_refs????????? gcmodule.c?????? 406? 0x7ffff7dad340
> 9?? collect??????????????? gcmodule.c?????? 1054 0x7ffff7dae838
> 10? collect_with_callback? gcmodule.c?????? 1240 0x7ffff7daf17b
> 11? collect_generations??? gcmodule.c?????? 1262 0x7ffff7daf3f6
> 12? _PyObject_GC_Alloc???? gcmodule.c?????? 1977 0x7ffff7daf4f2
> 13? _PyObject_GC_Malloc??? gcmodule.c?????? 1987 0x7ffff7dafebc
> 14? _PyObject_GC_NewVar??? gcmodule.c?????? 2016 0x7ffff7daffa5
> 15? PyTuple_New??????????? tupleobject.c??? 118? 0x7ffff7ca4da7
> 16? s_py_zosc_tuple??????? pythonactor.c??? 366? 0x55555568cc82
> 17? pythonactor_socket???? pythonactor.c??? 664? 0x55555568dac7
> 18? pythonactor_handle_msg pythonactor.c??? 862? 0x55555568e472
> 19? pythonactor_handler??? pythonactor.c??? 828? 0x55555568e2e2
> 20? sphactor_actor_run???? sphactor_actor.c 855? 0x5555558cb268
> ... <More>
> 
> Any pointer really appreciated.

I've found enabling PYTHONTRACEMALLOC=1 in the environment gives me a 
pointer to where to offending block was allocated.

I'm calling this method from C:

18    def handleSocket(self, addr, data, type, name, uuid):
19        if addr == "/pulse":
20            self.lampval += 1
21            return (addr, [0,0])

Modules/gcmodule.c:108: gc_decref: Assertion "gc_get_refs(g) > 0" 
failed: refcount is too small
Memory block allocated at (most recent call first):
   File "/home/arnaud/src/build-gazebosc-Desktop-Debug/bin/lampen.py", 
line 21

object address  : 0x7fffd81154c0
object refcount : 1
object type     : 0x7ffff7f3df20
object type name: list
object repr     : [117, 0]

Fatal Python error: _PyObject_AssertFailed
Python runtime state: initialized

Current thread 0x00007ffff2481640 (most recent call first):
   File "/home/arnaud/src/build-gazebosc-Desktop-Debug/bin/lampen.py", 
line 21 in handleSocket

Thread 0x00007ffff5d288c0 (most recent call first):
<no Python frame>

Now it clearly says the refcount is 1 so I'm puzzling what it means?

Rg,

Arnaud


From python at mrabarnett.plus.com  Tue Nov 23 09:34:57 2021
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 23 Nov 2021 14:34:57 +0000
Subject: Embedding Python crash on PyTuple_New
In-Reply-To: <d0b57569-ef0d-3ab4-9918-9cd498f1e658@sphaero.org>
References: <d0b57569-ef0d-3ab4-9918-9cd498f1e658@sphaero.org>
Message-ID: <4eb693c6-fd5f-7c54-246d-0b081a0488d5@mrabarnett.plus.com>

On 2021-11-23 12:07, Arnaud Loonstra wrote:
> Hi,
> 
> I've got Python embedded successfully in a program up until now as I'm
> now running into weird GC related segfaults. I'm currently trying to
> debug this but my understanding of CPython limits me here.
> 
> I'm creating a Tuple in C but it crashes on creating it after a while.
> It doesn't make sense which makes me wonder something else must be
> happening? Could be it just crashes here because the GC is cleaning up
> stuff completely unrelated to the allocation of the new tuple? How can I
> troubleshoot this?
> 
> I've got CPython compiled with  --with-valgrind --without-pymalloc
> --with-pydebug
> 
> In C I'm creating a tuple with the following method:
> 
> static PyObject *
> s_py_zosc_tuple(pythonactor_t *self, zosc_t *oscmsg)
> {
>       assert(self);
>       assert(oscmsg);
>       char *format = zosc_format(oscmsg);
> 
>       PyObject *rettuple = PyTuple_New((Py_ssize_t) strlen(format) );
> 
> It segfaults here (frame 16) after 320 times (consistently)
> 
> 
> 1   __GI_raise             raise.c          49   0x7ffff72c4e71
> 2   __GI_abort             abort.c          79   0x7ffff72ae536
> 3   fatal_error            pylifecycle.c    2183 0x7ffff7d84b4f
> 4   Py_FatalError          pylifecycle.c    2193 0x7ffff7d878b2
> 5   _PyObject_AssertFailed object.c         2200 0x7ffff7c93cf2
> 6   visit_decref           gcmodule.c       378  0x7ffff7dadfd5
> 7   tupletraverse          tupleobject.c    623  0x7ffff7ca3e81
> 8   subtract_refs          gcmodule.c       406  0x7ffff7dad340
> 9   collect                gcmodule.c       1054 0x7ffff7dae838
> 10  collect_with_callback  gcmodule.c       1240 0x7ffff7daf17b
> 11  collect_generations    gcmodule.c       1262 0x7ffff7daf3f6
> 12  _PyObject_GC_Alloc     gcmodule.c       1977 0x7ffff7daf4f2
> 13  _PyObject_GC_Malloc    gcmodule.c       1987 0x7ffff7dafebc
> 14  _PyObject_GC_NewVar    gcmodule.c       2016 0x7ffff7daffa5
> 15  PyTuple_New            tupleobject.c    118  0x7ffff7ca4da7
> 16  s_py_zosc_tuple        pythonactor.c    366  0x55555568cc82
> 17  pythonactor_socket     pythonactor.c    664  0x55555568dac7
> 18  pythonactor_handle_msg pythonactor.c    862  0x55555568e472
> 19  pythonactor_handler    pythonactor.c    828  0x55555568e2e2
> 20  sphactor_actor_run     sphactor_actor.c 855  0x5555558cb268
> ... <More>
> 
> Any pointer really appreciated.
> 
You're creating a tuple that'll have the same number of members as the 
length of a string? That looks strange to me.

How are you setting the tuple's members?

From arnaud at sphaero.org  Tue Nov 23 09:44:00 2021
From: arnaud at sphaero.org (Arnaud Loonstra)
Date: Tue, 23 Nov 2021 15:44:00 +0100
Subject: Embedding Python crash on PyTuple_New
In-Reply-To: <4eb693c6-fd5f-7c54-246d-0b081a0488d5@mrabarnett.plus.com>
References: <d0b57569-ef0d-3ab4-9918-9cd498f1e658@sphaero.org>
 <4eb693c6-fd5f-7c54-246d-0b081a0488d5@mrabarnett.plus.com>
Message-ID: <c827faf9-0871-fde3-83a0-fa19576b3c62@sphaero.org>

On 23-11-2021 15:34, MRAB wrote:
> On 2021-11-23 12:07, Arnaud Loonstra wrote:
>> Hi,
>>
>> I've got Python embedded successfully in a program up until now as I'm
>> now running into weird GC related segfaults. I'm currently trying to
>> debug this but my understanding of CPython limits me here.
>>
>> I'm creating a Tuple in C but it crashes on creating it after a while.
>> It doesn't make sense which makes me wonder something else must be
>> happening? Could be it just crashes here because the GC is cleaning up
>> stuff completely unrelated to the allocation of the new tuple? How can I
>> troubleshoot this?
>>
>> I've got CPython compiled with? --with-valgrind --without-pymalloc
>> --with-pydebug
>>
>> In C I'm creating a tuple with the following method:
>>
>> static PyObject *
>> s_py_zosc_tuple(pythonactor_t *self, zosc_t *oscmsg)
>> {
>> ????? assert(self);
>> ????? assert(oscmsg);
>> ????? char *format = zosc_format(oscmsg);
>>
>> ????? PyObject *rettuple = PyTuple_New((Py_ssize_t) strlen(format) );
>>
>> It segfaults here (frame 16) after 320 times (consistently)
>>
>>
>> 1?? __GI_raise???????????? raise.c????????? 49?? 0x7ffff72c4e71
>> 2?? __GI_abort???????????? abort.c????????? 79?? 0x7ffff72ae536
>> 3?? fatal_error??????????? pylifecycle.c??? 2183 0x7ffff7d84b4f
>> 4?? Py_FatalError????????? pylifecycle.c??? 2193 0x7ffff7d878b2
>> 5?? _PyObject_AssertFailed object.c???????? 2200 0x7ffff7c93cf2
>> 6?? visit_decref?????????? gcmodule.c?????? 378? 0x7ffff7dadfd5
>> 7?? tupletraverse????????? tupleobject.c??? 623? 0x7ffff7ca3e81
>> 8?? subtract_refs????????? gcmodule.c?????? 406? 0x7ffff7dad340
>> 9?? collect??????????????? gcmodule.c?????? 1054 0x7ffff7dae838
>> 10? collect_with_callback? gcmodule.c?????? 1240 0x7ffff7daf17b
>> 11? collect_generations??? gcmodule.c?????? 1262 0x7ffff7daf3f6
>> 12? _PyObject_GC_Alloc???? gcmodule.c?????? 1977 0x7ffff7daf4f2
>> 13? _PyObject_GC_Malloc??? gcmodule.c?????? 1987 0x7ffff7dafebc
>> 14? _PyObject_GC_NewVar??? gcmodule.c?????? 2016 0x7ffff7daffa5
>> 15? PyTuple_New??????????? tupleobject.c??? 118? 0x7ffff7ca4da7
>> 16? s_py_zosc_tuple??????? pythonactor.c??? 366? 0x55555568cc82
>> 17? pythonactor_socket???? pythonactor.c??? 664? 0x55555568dac7
>> 18? pythonactor_handle_msg pythonactor.c??? 862? 0x55555568e472
>> 19? pythonactor_handler??? pythonactor.c??? 828? 0x55555568e2e2
>> 20? sphactor_actor_run???? sphactor_actor.c 855? 0x5555558cb268
>> ... <More>
>>
>> Any pointer really appreciated.
>>
> You're creating a tuple that'll have the same number of members as the 
> length of a string? That looks strange to me.
> 
> How are you setting the tuple's members?

It's from a serialisation format called OSC. The string describes the 
type of bytes, every character is a type.

I'm creating the tuple as follows:

PyObject *rettuple = PyTuple_New((Py_ssize_t) strlen(format) );

Then I iterate the OSC message using the format string, (just showing 
handling an int (i))

     char type = '0';
     Py_ssize_t pos = 0;
     const void *data =  zosc_first(oscmsg, &type);
     while(data)
     {
         switch (type)
         {
         case('i'):
         {
             int32_t val = 9;
             int rc = zosc_pop_int32(oscmsg, &val);
             assert(rc == 0);
             PyObject *o = PyLong_FromLong((long)val);
             assert( o );
             rc = PyTuple_SetItem(rettuple, pos, o);
             assert(rc == 0);
             break;
         }

Full code is here:

https://github.com/hku-ect/gazebosc/blob/822452dfa27201db274d37ce09e835d98fe500b2/Actors/pythonactor.c#L360


From python at mrabarnett.plus.com  Tue Nov 23 10:17:25 2021
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 23 Nov 2021 15:17:25 +0000
Subject: Embedding Python crash on PyTuple_New
In-Reply-To: <c827faf9-0871-fde3-83a0-fa19576b3c62@sphaero.org>
References: <d0b57569-ef0d-3ab4-9918-9cd498f1e658@sphaero.org>
 <4eb693c6-fd5f-7c54-246d-0b081a0488d5@mrabarnett.plus.com>
 <c827faf9-0871-fde3-83a0-fa19576b3c62@sphaero.org>
Message-ID: <d6c7a17a-b24c-9a45-a94e-7bbc3d8fb1c1@mrabarnett.plus.com>

On 2021-11-23 14:44, Arnaud Loonstra wrote:
> On 23-11-2021 15:34, MRAB wrote:
>> On 2021-11-23 12:07, Arnaud Loonstra wrote:
>>> Hi,
>>>
>>> I've got Python embedded successfully in a program up until now as I'm
>>> now running into weird GC related segfaults. I'm currently trying to
>>> debug this but my understanding of CPython limits me here.
>>>
>>> I'm creating a Tuple in C but it crashes on creating it after a while.
>>> It doesn't make sense which makes me wonder something else must be
>>> happening? Could be it just crashes here because the GC is cleaning up
>>> stuff completely unrelated to the allocation of the new tuple? How can I
>>> troubleshoot this?
>>>
>>> I've got CPython compiled with? --with-valgrind --without-pymalloc
>>> --with-pydebug
>>>
>>> In C I'm creating a tuple with the following method:
>>>
>>> static PyObject *
>>> s_py_zosc_tuple(pythonactor_t *self, zosc_t *oscmsg)
>>> {
>>> ????? assert(self);
>>> ????? assert(oscmsg);
>>> ????? char *format = zosc_format(oscmsg);
>>>
>>> ????? PyObject *rettuple = PyTuple_New((Py_ssize_t) strlen(format) );
>>>
>>> It segfaults here (frame 16) after 320 times (consistently)
>>>
>>>
>>> 1?? __GI_raise???????????? raise.c????????? 49?? 0x7ffff72c4e71
>>> 2?? __GI_abort???????????? abort.c????????? 79?? 0x7ffff72ae536
>>> 3?? fatal_error??????????? pylifecycle.c??? 2183 0x7ffff7d84b4f
>>> 4?? Py_FatalError????????? pylifecycle.c??? 2193 0x7ffff7d878b2
>>> 5?? _PyObject_AssertFailed object.c???????? 2200 0x7ffff7c93cf2
>>> 6?? visit_decref?????????? gcmodule.c?????? 378? 0x7ffff7dadfd5
>>> 7?? tupletraverse????????? tupleobject.c??? 623? 0x7ffff7ca3e81
>>> 8?? subtract_refs????????? gcmodule.c?????? 406? 0x7ffff7dad340
>>> 9?? collect??????????????? gcmodule.c?????? 1054 0x7ffff7dae838
>>> 10? collect_with_callback? gcmodule.c?????? 1240 0x7ffff7daf17b
>>> 11? collect_generations??? gcmodule.c?????? 1262 0x7ffff7daf3f6
>>> 12? _PyObject_GC_Alloc???? gcmodule.c?????? 1977 0x7ffff7daf4f2
>>> 13? _PyObject_GC_Malloc??? gcmodule.c?????? 1987 0x7ffff7dafebc
>>> 14? _PyObject_GC_NewVar??? gcmodule.c?????? 2016 0x7ffff7daffa5
>>> 15? PyTuple_New??????????? tupleobject.c??? 118? 0x7ffff7ca4da7
>>> 16? s_py_zosc_tuple??????? pythonactor.c??? 366? 0x55555568cc82
>>> 17? pythonactor_socket???? pythonactor.c??? 664? 0x55555568dac7
>>> 18? pythonactor_handle_msg pythonactor.c??? 862? 0x55555568e472
>>> 19? pythonactor_handler??? pythonactor.c??? 828? 0x55555568e2e2
>>> 20? sphactor_actor_run???? sphactor_actor.c 855? 0x5555558cb268
>>> ... <More>
>>>
>>> Any pointer really appreciated.
>>>
>> You're creating a tuple that'll have the same number of members as the 
>> length of a string? That looks strange to me.
>> 
>> How are you setting the tuple's members?
> 
> It's from a serialisation format called OSC. The string describes the
> type of bytes, every character is a type.
> 
> I'm creating the tuple as follows:
> 
> PyObject *rettuple = PyTuple_New((Py_ssize_t) strlen(format) );
> 
> Then I iterate the OSC message using the format string, (just showing
> handling an int (i))
> 
>       char type = '0';
>       Py_ssize_t pos = 0;
>       const void *data =  zosc_first(oscmsg, &type);
>       while(data)
>       {
>           switch (type)
>           {
>           case('i'):
>           {
>               int32_t val = 9;
>               int rc = zosc_pop_int32(oscmsg, &val);
>               assert(rc == 0);
>               PyObject *o = PyLong_FromLong((long)val);
>               assert( o );
>               rc = PyTuple_SetItem(rettuple, pos, o);
>               assert(rc == 0);
>               break;
>           }
> 
> Full code is here:
> 
> https://github.com/hku-ect/gazebosc/blob/822452dfa27201db274d37ce09e835d98fe500b2/Actors/pythonactor.c#L360
> 
Looking at that code, you have:

     PyObject *o = Py_BuildValue("s#", str, 1);

what I'd check is the type of the 1 that you're passing. Wouldn't the 
compiler assume that it's an int?

The format string tells the function to expect a Py_ssize_t, but how 
would the compiler know that?

From python at mrabarnett.plus.com  Tue Nov 23 10:37:42 2021
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 23 Nov 2021 15:37:42 +0000
Subject: Embedding Python crash on PyTuple_New
In-Reply-To: <d6c7a17a-b24c-9a45-a94e-7bbc3d8fb1c1@mrabarnett.plus.com>
References: <d0b57569-ef0d-3ab4-9918-9cd498f1e658@sphaero.org>
 <4eb693c6-fd5f-7c54-246d-0b081a0488d5@mrabarnett.plus.com>
 <c827faf9-0871-fde3-83a0-fa19576b3c62@sphaero.org>
 <d6c7a17a-b24c-9a45-a94e-7bbc3d8fb1c1@mrabarnett.plus.com>
Message-ID: <f7711e63-a9a7-eb7a-b2b1-0c7ebd9829af@mrabarnett.plus.com>

On 2021-11-23 15:17, MRAB wrote:
> On 2021-11-23 14:44, Arnaud Loonstra wrote:
>> On 23-11-2021 15:34, MRAB wrote:
>>> On 2021-11-23 12:07, Arnaud Loonstra wrote:
>>>> Hi,
>>>>
>>>> I've got Python embedded successfully in a program up until now as I'm
>>>> now running into weird GC related segfaults. I'm currently trying to
>>>> debug this but my understanding of CPython limits me here.
>>>>
>>>> I'm creating a Tuple in C but it crashes on creating it after a while.
>>>> It doesn't make sense which makes me wonder something else must be
>>>> happening? Could be it just crashes here because the GC is cleaning up
>>>> stuff completely unrelated to the allocation of the new tuple? How can I
>>>> troubleshoot this?
>>>>
>>>> I've got CPython compiled with? --with-valgrind --without-pymalloc
>>>> --with-pydebug
>>>>
>>>> In C I'm creating a tuple with the following method:
>>>>
>>>> static PyObject *
>>>> s_py_zosc_tuple(pythonactor_t *self, zosc_t *oscmsg)
>>>> {
>>>> ????? assert(self);
>>>> ????? assert(oscmsg);
>>>> ????? char *format = zosc_format(oscmsg);
>>>>
>>>> ????? PyObject *rettuple = PyTuple_New((Py_ssize_t) strlen(format) );
>>>>
>>>> It segfaults here (frame 16) after 320 times (consistently)
>>>>
>>>>
>>>> 1?? __GI_raise???????????? raise.c????????? 49?? 0x7ffff72c4e71
>>>> 2?? __GI_abort???????????? abort.c????????? 79?? 0x7ffff72ae536
>>>> 3?? fatal_error??????????? pylifecycle.c??? 2183 0x7ffff7d84b4f
>>>> 4?? Py_FatalError????????? pylifecycle.c??? 2193 0x7ffff7d878b2
>>>> 5?? _PyObject_AssertFailed object.c???????? 2200 0x7ffff7c93cf2
>>>> 6?? visit_decref?????????? gcmodule.c?????? 378? 0x7ffff7dadfd5
>>>> 7?? tupletraverse????????? tupleobject.c??? 623? 0x7ffff7ca3e81
>>>> 8?? subtract_refs????????? gcmodule.c?????? 406? 0x7ffff7dad340
>>>> 9?? collect??????????????? gcmodule.c?????? 1054 0x7ffff7dae838
>>>> 10? collect_with_callback? gcmodule.c?????? 1240 0x7ffff7daf17b
>>>> 11? collect_generations??? gcmodule.c?????? 1262 0x7ffff7daf3f6
>>>> 12? _PyObject_GC_Alloc???? gcmodule.c?????? 1977 0x7ffff7daf4f2
>>>> 13? _PyObject_GC_Malloc??? gcmodule.c?????? 1987 0x7ffff7dafebc
>>>> 14? _PyObject_GC_NewVar??? gcmodule.c?????? 2016 0x7ffff7daffa5
>>>> 15? PyTuple_New??????????? tupleobject.c??? 118? 0x7ffff7ca4da7
>>>> 16? s_py_zosc_tuple??????? pythonactor.c??? 366? 0x55555568cc82
>>>> 17? pythonactor_socket???? pythonactor.c??? 664? 0x55555568dac7
>>>> 18? pythonactor_handle_msg pythonactor.c??? 862? 0x55555568e472
>>>> 19? pythonactor_handler??? pythonactor.c??? 828? 0x55555568e2e2
>>>> 20? sphactor_actor_run???? sphactor_actor.c 855? 0x5555558cb268
>>>> ... <More>
>>>>
>>>> Any pointer really appreciated.
>>>>
>>> You're creating a tuple that'll have the same number of members as the 
>>> length of a string? That looks strange to me.
>>> 
>>> How are you setting the tuple's members?
>> 
>> It's from a serialisation format called OSC. The string describes the
>> type of bytes, every character is a type.
>> 
>> I'm creating the tuple as follows:
>> 
>> PyObject *rettuple = PyTuple_New((Py_ssize_t) strlen(format) );
>> 
>> Then I iterate the OSC message using the format string, (just showing
>> handling an int (i))
>> 
>>       char type = '0';
>>       Py_ssize_t pos = 0;
>>       const void *data =  zosc_first(oscmsg, &type);
>>       while(data)
>>       {
>>           switch (type)
>>           {
>>           case('i'):
>>           {
>>               int32_t val = 9;
>>               int rc = zosc_pop_int32(oscmsg, &val);
>>               assert(rc == 0);
>>               PyObject *o = PyLong_FromLong((long)val);
>>               assert( o );
>>               rc = PyTuple_SetItem(rettuple, pos, o);
>>               assert(rc == 0);
>>               break;
>>           }
>> 
>> Full code is here:
>> 
>> https://github.com/hku-ect/gazebosc/blob/822452dfa27201db274d37ce09e835d98fe500b2/Actors/pythonactor.c#L360
>> 
> Looking at that code, you have:
> 
>       PyObject *o = Py_BuildValue("s#", str, 1);
> 
> what I'd check is the type of the 1 that you're passing. Wouldn't the
> compiler assume that it's an int?
> 
> The format string tells the function to expect a Py_ssize_t, but how
> would the compiler know that?
> 
Looking at https://www.mankier.com/3/zosc, it says for 'T' and 'F' "(no 
value required)", but you're doing:

     int rc = zosc_pop_bool(oscmsg, &bl);

If no value is required, is there a bool there to be popped?

From ast at invalid  Tue Nov 23 06:53:56 2021
From: ast at invalid (ast)
Date: Tue, 23 Nov 2021 12:53:56 +0100
Subject: Unexpected behaviour of math.floor, round and int functions
 (rounding)
In-Reply-To: <mailman.55.1637353085.19479.python-list@python.org>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
Message-ID: <619cd655$0$1349$426a74cc@news.free.fr>

Le 19/11/2021 ? 21:17, Chris Angelico a ?crit?:
> On Sat, Nov 20, 2021 at 5:08 AM ast <ast at invalid> wrote:
>>
>> Le 19/11/2021 ? 03:51, MRAB a ?crit :
>>> On 2021-11-19 02:40, 2QdxY4RzWzUUiLuE at potatochowder.com wrote:
>>>> On 2021-11-18 at 23:16:32 -0300,
>>>> Ren? Silva Vald?s <rene.silva.valdes at gmail.com> wrote:
>>>>

>>
>>   >>> 0.3 + 0.3 + 0.3 == 0.9
>> False
> 
> That's because 0.3 is not 3/10. It's not because floats are
> "unreliable" or "inaccurate". It's because the ones you're entering
> are not what you think they are.
> 
> When will people understand this?
> 
> (Probably never. Sigh.)
> 
> ChrisA
> 

I posted that to make people aware of danger of float comparison,
not because I was not understanding what happened.

We can see there is a difference on the lsb, due to rounding.

 >>> (0.3+0.3+0.3).hex()
'0x1.cccccccccccccp-1'
 >>> 0.9.hex()
'0x1.ccccccccccccdp-1'
 >>>

An isclose() function is provided in module math to do float
comparison safely.

 >>> math.isclose(0.3+0.3+0.3, 0.9)
True

From nuu.05.05 at gmail.com  Tue Nov 23 10:17:37 2021
From: nuu.05.05 at gmail.com (nuu.05.05 at gmail.com)
Date: Tue, 23 Nov 2021 20:17:37 +0500
Subject: Subprocess Connection error
Message-ID: <DEFDB46F-C346-456E-94E8-EC47D1EEB968@hxcore.ol>

   Hello,

   ?

   I have been 3.8.10 version of python until I ran the code for tkinter and
   I started getting subprocess connection error. After this I haven?t been
   able to open python for use as I get this error upon startup.

   ?

   Please help resolve the matter soon.

   ?

   Best regards,

   Nuha

   ?

From arnaud at sphaero.org  Tue Nov 23 11:04:32 2021
From: arnaud at sphaero.org (Arnaud Loonstra)
Date: Tue, 23 Nov 2021 17:04:32 +0100
Subject: Embedding Python crash on PyTuple_New
In-Reply-To: <f7711e63-a9a7-eb7a-b2b1-0c7ebd9829af@mrabarnett.plus.com>
References: <d0b57569-ef0d-3ab4-9918-9cd498f1e658@sphaero.org>
 <4eb693c6-fd5f-7c54-246d-0b081a0488d5@mrabarnett.plus.com>
 <c827faf9-0871-fde3-83a0-fa19576b3c62@sphaero.org>
 <d6c7a17a-b24c-9a45-a94e-7bbc3d8fb1c1@mrabarnett.plus.com>
 <f7711e63-a9a7-eb7a-b2b1-0c7ebd9829af@mrabarnett.plus.com>
Message-ID: <8788bcb1-5e02-b241-3533-43c40d363906@sphaero.org>

On 23-11-2021 16:37, MRAB wrote:
> On 2021-11-23 15:17, MRAB wrote:
>> On 2021-11-23 14:44, Arnaud Loonstra wrote:
>>> On 23-11-2021 15:34, MRAB wrote:
>>>> On 2021-11-23 12:07, Arnaud Loonstra wrote:
>>>>> Hi,
>>>>>
>>>>> I've got Python embedded successfully in a program up until now as I'm
>>>>> now running into weird GC related segfaults. I'm currently trying to
>>>>> debug this but my understanding of CPython limits me here.
>>>>>
>>>>> I'm creating a Tuple in C but it crashes on creating it after a while.
>>>>> It doesn't make sense which makes me wonder something else must be
>>>>> happening? Could be it just crashes here because the GC is cleaning up
>>>>> stuff completely unrelated to the allocation of the new tuple? How 
>>>>> can I
>>>>> troubleshoot this?
>>>>>
>>>>> I've got CPython compiled with? --with-valgrind --without-pymalloc
>>>>> --with-pydebug
>>>>>
>>>>> In C I'm creating a tuple with the following method:
>>>>>
>>>>> static PyObject *
>>>>> s_py_zosc_tuple(pythonactor_t *self, zosc_t *oscmsg)
>>>>> {
>>>>> ????? assert(self);
>>>>> ????? assert(oscmsg);
>>>>> ????? char *format = zosc_format(oscmsg);
>>>>>
>>>>> ????? PyObject *rettuple = PyTuple_New((Py_ssize_t) strlen(format) );
>>>>>
>>>>> It segfaults here (frame 16) after 320 times (consistently)
>>>>>
>>>>>
>>>>> 1?? __GI_raise???????????? raise.c????????? 49?? 0x7ffff72c4e71
>>>>> 2?? __GI_abort???????????? abort.c????????? 79?? 0x7ffff72ae536
>>>>> 3?? fatal_error??????????? pylifecycle.c??? 2183 0x7ffff7d84b4f
>>>>> 4?? Py_FatalError????????? pylifecycle.c??? 2193 0x7ffff7d878b2
>>>>> 5?? _PyObject_AssertFailed object.c???????? 2200 0x7ffff7c93cf2
>>>>> 6?? visit_decref?????????? gcmodule.c?????? 378? 0x7ffff7dadfd5
>>>>> 7?? tupletraverse????????? tupleobject.c??? 623? 0x7ffff7ca3e81
>>>>> 8?? subtract_refs????????? gcmodule.c?????? 406? 0x7ffff7dad340
>>>>> 9?? collect??????????????? gcmodule.c?????? 1054 0x7ffff7dae838
>>>>> 10? collect_with_callback? gcmodule.c?????? 1240 0x7ffff7daf17b
>>>>> 11? collect_generations??? gcmodule.c?????? 1262 0x7ffff7daf3f6
>>>>> 12? _PyObject_GC_Alloc???? gcmodule.c?????? 1977 0x7ffff7daf4f2
>>>>> 13? _PyObject_GC_Malloc??? gcmodule.c?????? 1987 0x7ffff7dafebc
>>>>> 14? _PyObject_GC_NewVar??? gcmodule.c?????? 2016 0x7ffff7daffa5
>>>>> 15? PyTuple_New??????????? tupleobject.c??? 118? 0x7ffff7ca4da7
>>>>> 16? s_py_zosc_tuple??????? pythonactor.c??? 366? 0x55555568cc82
>>>>> 17? pythonactor_socket???? pythonactor.c??? 664? 0x55555568dac7
>>>>> 18? pythonactor_handle_msg pythonactor.c??? 862? 0x55555568e472
>>>>> 19? pythonactor_handler??? pythonactor.c??? 828? 0x55555568e2e2
>>>>> 20? sphactor_actor_run???? sphactor_actor.c 855? 0x5555558cb268
>>>>> ... <More>
>>>>>
>>>>> Any pointer really appreciated.
>>>>>
>>>> You're creating a tuple that'll have the same number of members as 
>>>> the length of a string? That looks strange to me.
>>>>
>>>> How are you setting the tuple's members?
>>>
>>> It's from a serialisation format called OSC. The string describes the
>>> type of bytes, every character is a type.
>>>
>>> I'm creating the tuple as follows:
>>>
>>> PyObject *rettuple = PyTuple_New((Py_ssize_t) strlen(format) );
>>>
>>> Then I iterate the OSC message using the format string, (just showing
>>> handling an int (i))
>>>
>>> ????? char type = '0';
>>> ????? Py_ssize_t pos = 0;
>>> ????? const void *data =? zosc_first(oscmsg, &type);
>>> ????? while(data)
>>> ????? {
>>> ????????? switch (type)
>>> ????????? {
>>> ????????? case('i'):
>>> ????????? {
>>> ????????????? int32_t val = 9;
>>> ????????????? int rc = zosc_pop_int32(oscmsg, &val);
>>> ????????????? assert(rc == 0);
>>> ????????????? PyObject *o = PyLong_FromLong((long)val);
>>> ????????????? assert( o );
>>> ????????????? rc = PyTuple_SetItem(rettuple, pos, o);
>>> ????????????? assert(rc == 0);
>>> ????????????? break;
>>> ????????? }
>>>
>>> Full code is here:
>>>
>>> https://github.com/hku-ect/gazebosc/blob/822452dfa27201db274d37ce09e835d98fe500b2/Actors/pythonactor.c#L360 
>>>
>>>
>> Looking at that code, you have:
>>
>> ????? PyObject *o = Py_BuildValue("s#", str, 1);
>>
>> what I'd check is the type of the 1 that you're passing. Wouldn't the
>> compiler assume that it's an int?
>>
>> The format string tells the function to expect a Py_ssize_t, but how
>> would the compiler know that?
>>
> Looking at https://www.mankier.com/3/zosc, it says for 'T' and 'F' "(no 
> value required)", but you're doing:
> 
>  ??? int rc = zosc_pop_bool(oscmsg, &bl);
> 
> If no value is required, is there a bool there to be popped?

The value is only required to set a user provided boolean to the value 
in the message. There's no boolean value encoded in the message, only 
the T and F in the format string.

With regards to the Py_BuildValue("s#", str, 1);, that's a valid point. 
I'll fix that. However in the segfaults I'm testing that code is not 
touched.

I'm now testing different parts of the code to see if it runs stable. 
I've found it runs stable if I do not process the returned tuple.

PyObject *pReturn = PyObject_CallMethod(self->pyinstance,
                     "handleSocket", "sOsss",
                     oscaddress,
                     py_osctuple,
                     ev->type, ev->name, strdup(ev->uuid));
Py_XINCREF(pReturn);

https://github.com/hku-ect/gazebosc/blob/505b30c46bf3f78d188c3f575c80e294d3db7e5d/Actors/pythonactor.c#L673

and a bit further in the code I convert the Python tuple to an OSC message:

zosc_t *retosc = s_py_zosc(pAddress, pData);

https://github.com/hku-ect/gazebosc/blob/505b30c46bf3f78d188c3f575c80e294d3db7e5d/Actors/pythonactor.c#L732

If I change that line to:

zosc_t *retosc = zosc_create("/temp", "ii", 32, 64);

It runs stable.

I would turn my attention to s_py_zosc function but I'm not sure. Since 
the errors are GC related it could caused anywhere?

https://github.com/hku-ect/gazebosc/blob/505b30c46bf3f78d188c3f575c80e294d3db7e5d/Actors/pythonactor.c#L286 


From rosuav at gmail.com  Tue Nov 23 11:08:12 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 24 Nov 2021 03:08:12 +1100
Subject: Unexpected behaviour of math.floor,
 round and int functions (rounding)
In-Reply-To: <619cd655$0$1349$426a74cc@news.free.fr>
References: <CAF_USHuJ0bV7X9hWiciKwBL6=1zFCHfrFbPeV6KuVRnvFbwvmA@mail.gmail.com>
 <YZcOn5iN65pgLqrk@scrozzle>
 <ed1e8d09-2a5d-c0b8-baed-146ec13a00f7@mrabarnett.plus.com>
 <mailman.45.1637290453.19479.python-list@python.org>
 <61978dcc$0$20254$426a34cc@news.free.fr>
 <CAPTjJmoCc9HfB-KNdW6SK=JmGqCgGFGf-HnYtOc0uZZ9uNeVLg@mail.gmail.com>
 <mailman.55.1637353085.19479.python-list@python.org>
 <619cd655$0$1349$426a74cc@news.free.fr>
Message-ID: <CAPTjJmoPXPotK88Ljqz633WFqMM88ihVQi7_o43ziTZopLb-rg@mail.gmail.com>

On Wed, Nov 24, 2021 at 3:04 AM ast <ast at invalid> wrote:
>
> Le 19/11/2021 ? 21:17, Chris Angelico a ?crit :
> > On Sat, Nov 20, 2021 at 5:08 AM ast <ast at invalid> wrote:
> >>
> >> Le 19/11/2021 ? 03:51, MRAB a ?crit :
> >>> On 2021-11-19 02:40, 2QdxY4RzWzUUiLuE at potatochowder.com wrote:
> >>>> On 2021-11-18 at 23:16:32 -0300,
> >>>> Ren? Silva Vald?s <rene.silva.valdes at gmail.com> wrote:
> >>>>
>
> >>
> >>   >>> 0.3 + 0.3 + 0.3 == 0.9
> >> False
> >
> > That's because 0.3 is not 3/10. It's not because floats are
> > "unreliable" or "inaccurate". It's because the ones you're entering
> > are not what you think they are.
> >
> > When will people understand this?
> >
> > (Probably never. Sigh.)
> >
> > ChrisA
> >
>
> I posted that to make people aware of danger of float comparison,
> not because I was not understanding what happened.

And I posted to show that equality is not the problem, and that float
comparison is not dangerous.

> We can see there is a difference on the lsb, due to rounding.
>
>  >>> (0.3+0.3+0.3).hex()
> '0x1.cccccccccccccp-1'
>  >>> 0.9.hex()
> '0x1.ccccccccccccdp-1'
>  >>>
>
> An isclose() function is provided in module math to do float
> comparison safely.
>
>  >>> math.isclose(0.3+0.3+0.3, 0.9)
> True

This is why isclose() was so controversial: it is very very easy to
misuse it. Like this.

ChrisA

From python at mrabarnett.plus.com  Tue Nov 23 12:31:22 2021
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 23 Nov 2021 17:31:22 +0000
Subject: Embedding Python crash on PyTuple_New
In-Reply-To: <8788bcb1-5e02-b241-3533-43c40d363906@sphaero.org>
References: <d0b57569-ef0d-3ab4-9918-9cd498f1e658@sphaero.org>
 <4eb693c6-fd5f-7c54-246d-0b081a0488d5@mrabarnett.plus.com>
 <c827faf9-0871-fde3-83a0-fa19576b3c62@sphaero.org>
 <d6c7a17a-b24c-9a45-a94e-7bbc3d8fb1c1@mrabarnett.plus.com>
 <f7711e63-a9a7-eb7a-b2b1-0c7ebd9829af@mrabarnett.plus.com>
 <8788bcb1-5e02-b241-3533-43c40d363906@sphaero.org>
Message-ID: <1d070d88-8b19-c1ea-c700-4a47876fe871@mrabarnett.plus.com>

On 2021-11-23 16:04, Arnaud Loonstra wrote:
> On 23-11-2021 16:37, MRAB wrote:
>> On 2021-11-23 15:17, MRAB wrote:
>>> On 2021-11-23 14:44, Arnaud Loonstra wrote:
>>>> On 23-11-2021 15:34, MRAB wrote:
>>>>> On 2021-11-23 12:07, Arnaud Loonstra wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I've got Python embedded successfully in a program up until now as I'm
>>>>>> now running into weird GC related segfaults. I'm currently trying to
>>>>>> debug this but my understanding of CPython limits me here.
>>>>>>
>>>>>> I'm creating a Tuple in C but it crashes on creating it after a while.
>>>>>> It doesn't make sense which makes me wonder something else must be
>>>>>> happening? Could be it just crashes here because the GC is cleaning up
>>>>>> stuff completely unrelated to the allocation of the new tuple? How 
>>>>>> can I
>>>>>> troubleshoot this?
>>>>>>
>>>>>> I've got CPython compiled with? --with-valgrind --without-pymalloc
>>>>>> --with-pydebug
>>>>>>
>>>>>> In C I'm creating a tuple with the following method:
>>>>>>
>>>>>> static PyObject *
>>>>>> s_py_zosc_tuple(pythonactor_t *self, zosc_t *oscmsg)
>>>>>> {
>>>>>> ????? assert(self);
>>>>>> ????? assert(oscmsg);
>>>>>> ????? char *format = zosc_format(oscmsg);
>>>>>>
>>>>>> ????? PyObject *rettuple = PyTuple_New((Py_ssize_t) strlen(format) );
>>>>>>
>>>>>> It segfaults here (frame 16) after 320 times (consistently)
>>>>>>
>>>>>>
>>>>>> 1?? __GI_raise???????????? raise.c????????? 49?? 0x7ffff72c4e71
>>>>>> 2?? __GI_abort???????????? abort.c????????? 79?? 0x7ffff72ae536
>>>>>> 3?? fatal_error??????????? pylifecycle.c??? 2183 0x7ffff7d84b4f
>>>>>> 4?? Py_FatalError????????? pylifecycle.c??? 2193 0x7ffff7d878b2
>>>>>> 5?? _PyObject_AssertFailed object.c???????? 2200 0x7ffff7c93cf2
>>>>>> 6?? visit_decref?????????? gcmodule.c?????? 378? 0x7ffff7dadfd5
>>>>>> 7?? tupletraverse????????? tupleobject.c??? 623? 0x7ffff7ca3e81
>>>>>> 8?? subtract_refs????????? gcmodule.c?????? 406? 0x7ffff7dad340
>>>>>> 9?? collect??????????????? gcmodule.c?????? 1054 0x7ffff7dae838
>>>>>> 10? collect_with_callback? gcmodule.c?????? 1240 0x7ffff7daf17b
>>>>>> 11? collect_generations??? gcmodule.c?????? 1262 0x7ffff7daf3f6
>>>>>> 12? _PyObject_GC_Alloc???? gcmodule.c?????? 1977 0x7ffff7daf4f2
>>>>>> 13? _PyObject_GC_Malloc??? gcmodule.c?????? 1987 0x7ffff7dafebc
>>>>>> 14? _PyObject_GC_NewVar??? gcmodule.c?????? 2016 0x7ffff7daffa5
>>>>>> 15? PyTuple_New??????????? tupleobject.c??? 118? 0x7ffff7ca4da7
>>>>>> 16? s_py_zosc_tuple??????? pythonactor.c??? 366? 0x55555568cc82
>>>>>> 17? pythonactor_socket???? pythonactor.c??? 664? 0x55555568dac7
>>>>>> 18? pythonactor_handle_msg pythonactor.c??? 862? 0x55555568e472
>>>>>> 19? pythonactor_handler??? pythonactor.c??? 828? 0x55555568e2e2
>>>>>> 20? sphactor_actor_run???? sphactor_actor.c 855? 0x5555558cb268
>>>>>> ... <More>
>>>>>>
>>>>>> Any pointer really appreciated.
>>>>>>
>>>>> You're creating a tuple that'll have the same number of members as 
>>>>> the length of a string? That looks strange to me.
>>>>>
>>>>> How are you setting the tuple's members?
>>>>
>>>> It's from a serialisation format called OSC. The string describes the
>>>> type of bytes, every character is a type.
>>>>
>>>> I'm creating the tuple as follows:
>>>>
>>>> PyObject *rettuple = PyTuple_New((Py_ssize_t) strlen(format) );
>>>>
>>>> Then I iterate the OSC message using the format string, (just showing
>>>> handling an int (i))
>>>>
>>>> ????? char type = '0';
>>>> ????? Py_ssize_t pos = 0;
>>>> ????? const void *data =? zosc_first(oscmsg, &type);
>>>> ????? while(data)
>>>> ????? {
>>>> ????????? switch (type)
>>>> ????????? {
>>>> ????????? case('i'):
>>>> ????????? {
>>>> ????????????? int32_t val = 9;
>>>> ????????????? int rc = zosc_pop_int32(oscmsg, &val);
>>>> ????????????? assert(rc == 0);
>>>> ????????????? PyObject *o = PyLong_FromLong((long)val);
>>>> ????????????? assert( o );
>>>> ????????????? rc = PyTuple_SetItem(rettuple, pos, o);
>>>> ????????????? assert(rc == 0);
>>>> ????????????? break;
>>>> ????????? }
>>>>
>>>> Full code is here:
>>>>
>>>> https://github.com/hku-ect/gazebosc/blob/822452dfa27201db274d37ce09e835d98fe500b2/Actors/pythonactor.c#L360 
>>>>
>>>>
>>> Looking at that code, you have:
>>>
>>> ????? PyObject *o = Py_BuildValue("s#", str, 1);
>>>
>>> what I'd check is the type of the 1 that you're passing. Wouldn't the
>>> compiler assume that it's an int?
>>>
>>> The format string tells the function to expect a Py_ssize_t, but how
>>> would the compiler know that?
>>>
>> Looking at https://www.mankier.com/3/zosc, it says for 'T' and 'F' "(no 
>> value required)", but you're doing:
>> 
>>  ??? int rc = zosc_pop_bool(oscmsg, &bl);
>> 
>> If no value is required, is there a bool there to be popped?
> 
> The value is only required to set a user provided boolean to the value
> in the message. There's no boolean value encoded in the message, only
> the T and F in the format string.
> 
> With regards to the Py_BuildValue("s#", str, 1);, that's a valid point.
> I'll fix that. However in the segfaults I'm testing that code is not
> touched.

You can use "C" as a format string for Py_BuildValue to convert a C int 
representing a character to a Python string.

> I'm now testing different parts of the code to see if it runs stable.
> I've found it runs stable if I do not process the returned tuple.
> 
> PyObject *pReturn = PyObject_CallMethod(self->pyinstance,
>                       "handleSocket", "sOsss",
>                       oscaddress,
>                       py_osctuple,
>                       ev->type, ev->name, strdup(ev->uuid));
> Py_XINCREF(pReturn);
> 
Why the Py_XINCREF? PyObject_CallMethod returns a new reference. The 
Py_DECREF that you do later won't destroy the object because of that 
additional Py_XINCREF, so that's a memory leak.

> https://github.com/hku-ect/gazebosc/blob/505b30c46bf3f78d188c3f575c80e294d3db7e5d/Actors/pythonactor.c#L673
> 
> and a bit further in the code I convert the Python tuple to an OSC message:
> 
> zosc_t *retosc = s_py_zosc(pAddress, pData);
> 
> https://github.com/hku-ect/gazebosc/blob/505b30c46bf3f78d188c3f575c80e294d3db7e5d/Actors/pythonactor.c#L732
> 
> If I change that line to:
> 
> zosc_t *retosc = zosc_create("/temp", "ii", 32, 64);
> 
> It runs stable.
> 
> I would turn my attention to s_py_zosc function but I'm not sure. Since
> the errors are GC related it could caused anywhere?
> 
Basically, yes, but I won't be surprised if it was due to too few 
INCREFs or too many DECREFs somewhere.

> https://github.com/hku-ect/gazebosc/blob/505b30c46bf3f78d188c3f575c80e294d3db7e5d/Actors/pythonactor.c#L286
> 
Incidentally, in s_py_zosc_tuple, you're not doing "assert(rc == 0);" 
after "after zosc_pop_float" or "zosc_pop_double".

From drsalists at gmail.com  Tue Nov 23 12:34:21 2021
From: drsalists at gmail.com (Dan Stromberg)
Date: Tue, 23 Nov 2021 09:34:21 -0800
Subject: Subprocess Connection error
In-Reply-To: <DEFDB46F-C346-456E-94E8-EC47D1EEB968@hxcore.ol>
References: <DEFDB46F-C346-456E-94E8-EC47D1EEB968@hxcore.ol>
Message-ID: <CAGGBd_ruF2PwsrU9Bp4377+GMKUpMuLTT+TQr2FE3_sRgVL3Sw@mail.gmail.com>

Hi.

It's important to include the full text of whatever error messages you're
getting.

If you can, you should also include your code, or better: include an
http://sscce.org/

Can you start the python interpreter from bash, powershell or cmd.exe? If
not, please again include the full text of any error messages you get.

BTW, it's important to cut and paste TEXT - don't just send screenshots.

Good luck.

On Tue, Nov 23, 2021 at 8:03 AM <nuu.05.05 at gmail.com> wrote:

>    Hello,
>
>
>
>    I have been 3.8.10 version of python until I ran the code for tkinter
> and
>    I started getting subprocess connection error. After this I haven?t been
>    able to open python for use as I get this error upon startup.
>
>
>
>    Please help resolve the matter soon.
>
>
>
>    Best regards,
>
>    Nuha
>
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>

From python at mrabarnett.plus.com  Tue Nov 23 12:44:07 2021
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 23 Nov 2021 17:44:07 +0000
Subject: Embedding Python crash on PyTuple_New
In-Reply-To: <1d070d88-8b19-c1ea-c700-4a47876fe871@mrabarnett.plus.com>
References: <d0b57569-ef0d-3ab4-9918-9cd498f1e658@sphaero.org>
 <4eb693c6-fd5f-7c54-246d-0b081a0488d5@mrabarnett.plus.com>
 <c827faf9-0871-fde3-83a0-fa19576b3c62@sphaero.org>
 <d6c7a17a-b24c-9a45-a94e-7bbc3d8fb1c1@mrabarnett.plus.com>
 <f7711e63-a9a7-eb7a-b2b1-0c7ebd9829af@mrabarnett.plus.com>
 <8788bcb1-5e02-b241-3533-43c40d363906@sphaero.org>
 <1d070d88-8b19-c1ea-c700-4a47876fe871@mrabarnett.plus.com>
Message-ID: <8bf13799-ebcc-5110-3944-c0ad7e9b933c@mrabarnett.plus.com>

On 2021-11-23 17:31, MRAB wrote:
> On 2021-11-23 16:04, Arnaud Loonstra wrote:
[snip]
>> 
>> I would turn my attention to s_py_zosc function but I'm not sure. Since
>> the errors are GC related it could caused anywhere?
>> 
> Basically, yes, but I won't be surprised if it was due to too few
> INCREFs or too many DECREFs somewhere.
> 
>> https://github.com/hku-ect/gazebosc/blob/505b30c46bf3f78d188c3f575c80e294d3db7e5d/Actors/pythonactor.c#L286
>> 
> Incidentally, in s_py_zosc_tuple, you're not doing "assert(rc == 0);"
> after "after zosc_pop_float" or "zosc_pop_double".
>
Here's something interesting in s_py_zosc:

             PyTypeObject* type = Py_TYPE(item);
             const char * typename = _PyType_Name(type);
             if (streq(typename, "c_int"))
             {
                 zsys_warning("Unsupported ctypes.c_int until we find a 
way to access the value :S");
             }
             else
                 zsys_warning("unsupported python type");

             Py_DECREF(type);

According to the docs, Py_TYPE returns a borrowed reference, but you're 
DECREFing it. If you call s_py_zosc enough times, the reference count of 
the type will drop to 0. I think that might be the problem.

From arnaud at sphaero.org  Tue Nov 23 15:25:35 2021
From: arnaud at sphaero.org (Arnaud Loonstra)
Date: Tue, 23 Nov 2021 21:25:35 +0100
Subject: Embedding Python crash on PyTuple_New
In-Reply-To: <1d070d88-8b19-c1ea-c700-4a47876fe871@mrabarnett.plus.com>
References: <d0b57569-ef0d-3ab4-9918-9cd498f1e658@sphaero.org>
 <4eb693c6-fd5f-7c54-246d-0b081a0488d5@mrabarnett.plus.com>
 <c827faf9-0871-fde3-83a0-fa19576b3c62@sphaero.org>
 <d6c7a17a-b24c-9a45-a94e-7bbc3d8fb1c1@mrabarnett.plus.com>
 <f7711e63-a9a7-eb7a-b2b1-0c7ebd9829af@mrabarnett.plus.com>
 <8788bcb1-5e02-b241-3533-43c40d363906@sphaero.org>
 <1d070d88-8b19-c1ea-c700-4a47876fe871@mrabarnett.plus.com>
Message-ID: <b575c5da-7354-57e5-039b-3b1c05861ffa@sphaero.org>

On 23-11-2021 18:31, MRAB wrote:
> On 2021-11-23 16:04, Arnaud Loonstra wrote:
>> On 23-11-2021 16:37, MRAB wrote:
>>> On 2021-11-23 15:17, MRAB wrote:
>>>> On 2021-11-23 14:44, Arnaud Loonstra wrote:
>>>>> On 23-11-2021 15:34, MRAB wrote:
>>>>>> On 2021-11-23 12:07, Arnaud Loonstra wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I've got Python embedded successfully in a program up until now 
>>>>>>> as I'm
>>>>>>> now running into weird GC related segfaults. I'm currently trying to
>>>>>>> debug this but my understanding of CPython limits me here.
>>>>>>>
>>>>>>> I'm creating a Tuple in C but it crashes on creating it after a 
>>>>>>> while.
>>>>>>> It doesn't make sense which makes me wonder something else must be
>>>>>>> happening? Could be it just crashes here because the GC is 
>>>>>>> cleaning up
>>>>>>> stuff completely unrelated to the allocation of the new tuple? 
>>>>>>> How can I
>>>>>>> troubleshoot this?
>>>>>>>
>>>>>>> I've got CPython compiled with? --with-valgrind --without-pymalloc
>>>>>>> --with-pydebug
>>>>>>>
>>>>>>> In C I'm creating a tuple with the following method:
>>>>>>>
>>>>>>> static PyObject *
>>>>>>> s_py_zosc_tuple(pythonactor_t *self, zosc_t *oscmsg)
>>>>>>> {
>>>>>>> ????? assert(self);
>>>>>>> ????? assert(oscmsg);
>>>>>>> ????? char *format = zosc_format(oscmsg);
>>>>>>>
>>>>>>> ????? PyObject *rettuple = PyTuple_New((Py_ssize_t) 
>>>>>>> strlen(format) );
>>>>>>>
>>>>>>> It segfaults here (frame 16) after 320 times (consistently)
>>>>>>>
>>>>>>>
>>>>>>> 1?? __GI_raise???????????? raise.c????????? 49?? 0x7ffff72c4e71
>>>>>>> 2?? __GI_abort???????????? abort.c????????? 79?? 0x7ffff72ae536
>>>>>>> 3?? fatal_error??????????? pylifecycle.c??? 2183 0x7ffff7d84b4f
>>>>>>> 4?? Py_FatalError????????? pylifecycle.c??? 2193 0x7ffff7d878b2
>>>>>>> 5?? _PyObject_AssertFailed object.c???????? 2200 0x7ffff7c93cf2
>>>>>>> 6?? visit_decref?????????? gcmodule.c?????? 378? 0x7ffff7dadfd5
>>>>>>> 7?? tupletraverse????????? tupleobject.c??? 623? 0x7ffff7ca3e81
>>>>>>> 8?? subtract_refs????????? gcmodule.c?????? 406? 0x7ffff7dad340
>>>>>>> 9?? collect??????????????? gcmodule.c?????? 1054 0x7ffff7dae838
>>>>>>> 10? collect_with_callback? gcmodule.c?????? 1240 0x7ffff7daf17b
>>>>>>> 11? collect_generations??? gcmodule.c?????? 1262 0x7ffff7daf3f6
>>>>>>> 12? _PyObject_GC_Alloc???? gcmodule.c?????? 1977 0x7ffff7daf4f2
>>>>>>> 13? _PyObject_GC_Malloc??? gcmodule.c?????? 1987 0x7ffff7dafebc
>>>>>>> 14? _PyObject_GC_NewVar??? gcmodule.c?????? 2016 0x7ffff7daffa5
>>>>>>> 15? PyTuple_New??????????? tupleobject.c??? 118? 0x7ffff7ca4da7
>>>>>>> 16? s_py_zosc_tuple??????? pythonactor.c??? 366? 0x55555568cc82
>>>>>>> 17? pythonactor_socket???? pythonactor.c??? 664? 0x55555568dac7
>>>>>>> 18? pythonactor_handle_msg pythonactor.c??? 862? 0x55555568e472
>>>>>>> 19? pythonactor_handler??? pythonactor.c??? 828? 0x55555568e2e2
>>>>>>> 20? sphactor_actor_run???? sphactor_actor.c 855? 0x5555558cb268
>>>>>>> ... <More>
>>>>>>>
>>>>>>> Any pointer really appreciated.
>>>>>>>
>>>>>> You're creating a tuple that'll have the same number of members as 
>>>>>> the length of a string? That looks strange to me.
>>>>>>
>>>>>> How are you setting the tuple's members?
>>>>>
>>>>> It's from a serialisation format called OSC. The string describes the
>>>>> type of bytes, every character is a type.
>>>>>
>>>>> I'm creating the tuple as follows:
>>>>>
>>>>> PyObject *rettuple = PyTuple_New((Py_ssize_t) strlen(format) );
>>>>>
>>>>> Then I iterate the OSC message using the format string, (just showing
>>>>> handling an int (i))
>>>>>
>>>>> ????? char type = '0';
>>>>> ????? Py_ssize_t pos = 0;
>>>>> ????? const void *data =? zosc_first(oscmsg, &type);
>>>>> ????? while(data)
>>>>> ????? {
>>>>> ????????? switch (type)
>>>>> ????????? {
>>>>> ????????? case('i'):
>>>>> ????????? {
>>>>> ????????????? int32_t val = 9;
>>>>> ????????????? int rc = zosc_pop_int32(oscmsg, &val);
>>>>> ????????????? assert(rc == 0);
>>>>> ????????????? PyObject *o = PyLong_FromLong((long)val);
>>>>> ????????????? assert( o );
>>>>> ????????????? rc = PyTuple_SetItem(rettuple, pos, o);
>>>>> ????????????? assert(rc == 0);
>>>>> ????????????? break;
>>>>> ????????? }
>>>>>
>>>>> Full code is here:
>>>>>
>>>>> https://github.com/hku-ect/gazebosc/blob/822452dfa27201db274d37ce09e835d98fe500b2/Actors/pythonactor.c#L360 
>>>>>
>>>>>
>>>> Looking at that code, you have:
>>>>
>>>> ????? PyObject *o = Py_BuildValue("s#", str, 1);
>>>>
>>>> what I'd check is the type of the 1 that you're passing. Wouldn't the
>>>> compiler assume that it's an int?
>>>>
>>>> The format string tells the function to expect a Py_ssize_t, but how
>>>> would the compiler know that?
>>>>
>>> Looking at https://www.mankier.com/3/zosc, it says for 'T' and 'F' 
>>> "(no value required)", but you're doing:
>>>
>>> ???? int rc = zosc_pop_bool(oscmsg, &bl);
>>>
>>> If no value is required, is there a bool there to be popped?
>>
>> The value is only required to set a user provided boolean to the value
>> in the message. There's no boolean value encoded in the message, only
>> the T and F in the format string.
>>
>> With regards to the Py_BuildValue("s#", str, 1);, that's a valid point.
>> I'll fix that. However in the segfaults I'm testing that code is not
>> touched.
> 
> You can use "C" as a format string for Py_BuildValue to convert a C int 
> representing a character to a Python string.
> 
>> I'm now testing different parts of the code to see if it runs stable.
>> I've found it runs stable if I do not process the returned tuple.
>>
>> PyObject *pReturn = PyObject_CallMethod(self->pyinstance,
>> ????????????????????? "handleSocket", "sOsss",
>> ????????????????????? oscaddress,
>> ????????????????????? py_osctuple,
>> ????????????????????? ev->type, ev->name, strdup(ev->uuid));
>> Py_XINCREF(pReturn);
>>
> Why the Py_XINCREF? PyObject_CallMethod returns a new reference. The 
> Py_DECREF that you do later won't destroy the object because of that 
> additional Py_XINCREF, so that's a memory leak.
> 
>> https://github.com/hku-ect/gazebosc/blob/505b30c46bf3f78d188c3f575c80e294d3db7e5d/Actors/pythonactor.c#L673 
>>
>>
>> and a bit further in the code I convert the Python tuple to an OSC 
>> message:
>>
>> zosc_t *retosc = s_py_zosc(pAddress, pData);
>>
>> https://github.com/hku-ect/gazebosc/blob/505b30c46bf3f78d188c3f575c80e294d3db7e5d/Actors/pythonactor.c#L732 
>>
>>
>> If I change that line to:
>>
>> zosc_t *retosc = zosc_create("/temp", "ii", 32, 64);
>>
>> It runs stable.
>>
>> I would turn my attention to s_py_zosc function but I'm not sure. Since
>> the errors are GC related it could caused anywhere?
>>
> Basically, yes, but I won't be surprised if it was due to too few 
> INCREFs or too many DECREFs somewhere.
> 
>> https://github.com/hku-ect/gazebosc/blob/505b30c46bf3f78d188c3f575c80e294d3db7e5d/Actors/pythonactor.c#L286 
>>
>>
> Incidentally, in s_py_zosc_tuple, you're not doing "assert(rc == 0);" 
> after "after zosc_pop_float" or "zosc_pop_double".

Thanks for those pointers! I think your intuition is right. I might have 
found the bugger. In s_py_zosc I call Py_DECREF on pAddress and pData.
However they are acquired by PyTuple_GetItem which returns a borrowed 
reference. I think pAddress and pData are then also 'decrefed' when the 
pReturn tuple which contains pAddress and pData is 'decrefed'?

I'm testing it now but it's running stable for a while now.

Preparing a PR: https://github.com/hku-ect/gazebosc/pull/181

From framstag at rus.uni-stuttgart.de  Tue Nov 23 15:44:03 2021
From: framstag at rus.uni-stuttgart.de (Ulli Horlacher)
Date: Tue, 23 Nov 2021 20:44:03 +0000 (UTC)
Subject: pyinstaller: icon not visable on desktop?
Message-ID: <snjjqj$m1m$1@news2.informatik.uni-stuttgart.de>

When I compile a python program with pyinstaller, I get an executable:

pyinstaller --onefile --icon fex.ico fextasy.py

But the executables icon is only visable within the windows file browser, 
not on the desktop:

https://fex.flupp.org/fop/V2ioMZcl/X-20211123213712.png

There is only the default pyinstaller icon.

Why?


-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher at tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/

From python at mrabarnett.plus.com  Tue Nov 23 19:46:05 2021
From: python at mrabarnett.plus.com (MRAB)
Date: Wed, 24 Nov 2021 00:46:05 +0000
Subject: Embedding Python crash on PyTuple_New
In-Reply-To: <b575c5da-7354-57e5-039b-3b1c05861ffa@sphaero.org>
References: <d0b57569-ef0d-3ab4-9918-9cd498f1e658@sphaero.org>
 <4eb693c6-fd5f-7c54-246d-0b081a0488d5@mrabarnett.plus.com>
 <c827faf9-0871-fde3-83a0-fa19576b3c62@sphaero.org>
 <d6c7a17a-b24c-9a45-a94e-7bbc3d8fb1c1@mrabarnett.plus.com>
 <f7711e63-a9a7-eb7a-b2b1-0c7ebd9829af@mrabarnett.plus.com>
 <8788bcb1-5e02-b241-3533-43c40d363906@sphaero.org>
 <1d070d88-8b19-c1ea-c700-4a47876fe871@mrabarnett.plus.com>
 <b575c5da-7354-57e5-039b-3b1c05861ffa@sphaero.org>
Message-ID: <d341afa3-b3ae-ee2e-25e9-ed76a0273704@mrabarnett.plus.com>

On 2021-11-23 20:25, Arnaud Loonstra wrote:
> On 23-11-2021 18:31, MRAB wrote:
>> On 2021-11-23 16:04, Arnaud Loonstra wrote:
>>> On 23-11-2021 16:37, MRAB wrote:
>>>> On 2021-11-23 15:17, MRAB wrote:
>>>>> On 2021-11-23 14:44, Arnaud Loonstra wrote:
>>>>>> On 23-11-2021 15:34, MRAB wrote:
>>>>>>> On 2021-11-23 12:07, Arnaud Loonstra wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I've got Python embedded successfully in a program up until now 
>>>>>>>> as I'm
>>>>>>>> now running into weird GC related segfaults. I'm currently trying to
>>>>>>>> debug this but my understanding of CPython limits me here.
>>>>>>>>
>>>>>>>> I'm creating a Tuple in C but it crashes on creating it after a 
>>>>>>>> while.
>>>>>>>> It doesn't make sense which makes me wonder something else must be
>>>>>>>> happening? Could be it just crashes here because the GC is 
>>>>>>>> cleaning up
>>>>>>>> stuff completely unrelated to the allocation of the new tuple? 
>>>>>>>> How can I
>>>>>>>> troubleshoot this?
>>>>>>>>
>>>>>>>> I've got CPython compiled with? --with-valgrind --without-pymalloc
>>>>>>>> --with-pydebug
>>>>>>>>
>>>>>>>> In C I'm creating a tuple with the following method:
>>>>>>>>
>>>>>>>> static PyObject *
>>>>>>>> s_py_zosc_tuple(pythonactor_t *self, zosc_t *oscmsg)
>>>>>>>> {
>>>>>>>> ????? assert(self);
>>>>>>>> ????? assert(oscmsg);
>>>>>>>> ????? char *format = zosc_format(oscmsg);
>>>>>>>>
>>>>>>>> ????? PyObject *rettuple = PyTuple_New((Py_ssize_t) 
>>>>>>>> strlen(format) );
>>>>>>>>
>>>>>>>> It segfaults here (frame 16) after 320 times (consistently)
>>>>>>>>
>>>>>>>>
>>>>>>>> 1?? __GI_raise???????????? raise.c????????? 49?? 0x7ffff72c4e71
>>>>>>>> 2?? __GI_abort???????????? abort.c????????? 79?? 0x7ffff72ae536
>>>>>>>> 3?? fatal_error??????????? pylifecycle.c??? 2183 0x7ffff7d84b4f
>>>>>>>> 4?? Py_FatalError????????? pylifecycle.c??? 2193 0x7ffff7d878b2
>>>>>>>> 5?? _PyObject_AssertFailed object.c???????? 2200 0x7ffff7c93cf2
>>>>>>>> 6?? visit_decref?????????? gcmodule.c?????? 378? 0x7ffff7dadfd5
>>>>>>>> 7?? tupletraverse????????? tupleobject.c??? 623? 0x7ffff7ca3e81
>>>>>>>> 8?? subtract_refs????????? gcmodule.c?????? 406? 0x7ffff7dad340
>>>>>>>> 9?? collect??????????????? gcmodule.c?????? 1054 0x7ffff7dae838
>>>>>>>> 10? collect_with_callback? gcmodule.c?????? 1240 0x7ffff7daf17b
>>>>>>>> 11? collect_generations??? gcmodule.c?????? 1262 0x7ffff7daf3f6
>>>>>>>> 12? _PyObject_GC_Alloc???? gcmodule.c?????? 1977 0x7ffff7daf4f2
>>>>>>>> 13? _PyObject_GC_Malloc??? gcmodule.c?????? 1987 0x7ffff7dafebc
>>>>>>>> 14? _PyObject_GC_NewVar??? gcmodule.c?????? 2016 0x7ffff7daffa5
>>>>>>>> 15? PyTuple_New??????????? tupleobject.c??? 118? 0x7ffff7ca4da7
>>>>>>>> 16? s_py_zosc_tuple??????? pythonactor.c??? 366? 0x55555568cc82
>>>>>>>> 17? pythonactor_socket???? pythonactor.c??? 664? 0x55555568dac7
>>>>>>>> 18? pythonactor_handle_msg pythonactor.c??? 862? 0x55555568e472
>>>>>>>> 19? pythonactor_handler??? pythonactor.c??? 828? 0x55555568e2e2
>>>>>>>> 20? sphactor_actor_run???? sphactor_actor.c 855? 0x5555558cb268
>>>>>>>> ... <More>
>>>>>>>>
>>>>>>>> Any pointer really appreciated.
>>>>>>>>
>>>>>>> You're creating a tuple that'll have the same number of members as 
>>>>>>> the length of a string? That looks strange to me.
>>>>>>>
>>>>>>> How are you setting the tuple's members?
>>>>>>
>>>>>> It's from a serialisation format called OSC. The string describes the
>>>>>> type of bytes, every character is a type.
>>>>>>
>>>>>> I'm creating the tuple as follows:
>>>>>>
>>>>>> PyObject *rettuple = PyTuple_New((Py_ssize_t) strlen(format) );
>>>>>>
>>>>>> Then I iterate the OSC message using the format string, (just showing
>>>>>> handling an int (i))
>>>>>>
>>>>>> ????? char type = '0';
>>>>>> ????? Py_ssize_t pos = 0;
>>>>>> ????? const void *data =? zosc_first(oscmsg, &type);
>>>>>> ????? while(data)
>>>>>> ????? {
>>>>>> ????????? switch (type)
>>>>>> ????????? {
>>>>>> ????????? case('i'):
>>>>>> ????????? {
>>>>>> ????????????? int32_t val = 9;
>>>>>> ????????????? int rc = zosc_pop_int32(oscmsg, &val);
>>>>>> ????????????? assert(rc == 0);
>>>>>> ????????????? PyObject *o = PyLong_FromLong((long)val);
>>>>>> ????????????? assert( o );
>>>>>> ????????????? rc = PyTuple_SetItem(rettuple, pos, o);
>>>>>> ????????????? assert(rc == 0);
>>>>>> ????????????? break;
>>>>>> ????????? }
>>>>>>
>>>>>> Full code is here:
>>>>>>
>>>>>> https://github.com/hku-ect/gazebosc/blob/822452dfa27201db274d37ce09e835d98fe500b2/Actors/pythonactor.c#L360 
>>>>>>
>>>>>>
>>>>> Looking at that code, you have:
>>>>>
>>>>> ????? PyObject *o = Py_BuildValue("s#", str, 1);
>>>>>
>>>>> what I'd check is the type of the 1 that you're passing. Wouldn't the
>>>>> compiler assume that it's an int?
>>>>>
>>>>> The format string tells the function to expect a Py_ssize_t, but how
>>>>> would the compiler know that?
>>>>>
>>>> Looking at https://www.mankier.com/3/zosc, it says for 'T' and 'F' 
>>>> "(no value required)", but you're doing:
>>>>
>>>> ???? int rc = zosc_pop_bool(oscmsg, &bl);
>>>>
>>>> If no value is required, is there a bool there to be popped?
>>>
>>> The value is only required to set a user provided boolean to the value
>>> in the message. There's no boolean value encoded in the message, only
>>> the T and F in the format string.
>>>
>>> With regards to the Py_BuildValue("s#", str, 1);, that's a valid point.
>>> I'll fix that. However in the segfaults I'm testing that code is not
>>> touched.
>> 
>> You can use "C" as a format string for Py_BuildValue to convert a C int 
>> representing a character to a Python string.
>> 
>>> I'm now testing different parts of the code to see if it runs stable.
>>> I've found it runs stable if I do not process the returned tuple.
>>>
>>> PyObject *pReturn = PyObject_CallMethod(self->pyinstance,
>>> ????????????????????? "handleSocket", "sOsss",
>>> ????????????????????? oscaddress,
>>> ????????????????????? py_osctuple,
>>> ????????????????????? ev->type, ev->name, strdup(ev->uuid));
>>> Py_XINCREF(pReturn);
>>>
>> Why the Py_XINCREF? PyObject_CallMethod returns a new reference. The 
>> Py_DECREF that you do later won't destroy the object because of that 
>> additional Py_XINCREF, so that's a memory leak.
>> 
>>> https://github.com/hku-ect/gazebosc/blob/505b30c46bf3f78d188c3f575c80e294d3db7e5d/Actors/pythonactor.c#L673 
>>>
>>>
>>> and a bit further in the code I convert the Python tuple to an OSC 
>>> message:
>>>
>>> zosc_t *retosc = s_py_zosc(pAddress, pData);
>>>
>>> https://github.com/hku-ect/gazebosc/blob/505b30c46bf3f78d188c3f575c80e294d3db7e5d/Actors/pythonactor.c#L732 
>>>
>>>
>>> If I change that line to:
>>>
>>> zosc_t *retosc = zosc_create("/temp", "ii", 32, 64);
>>>
>>> It runs stable.
>>>
>>> I would turn my attention to s_py_zosc function but I'm not sure. Since
>>> the errors are GC related it could caused anywhere?
>>>
>> Basically, yes, but I won't be surprised if it was due to too few 
>> INCREFs or too many DECREFs somewhere.
>> 
>>> https://github.com/hku-ect/gazebosc/blob/505b30c46bf3f78d188c3f575c80e294d3db7e5d/Actors/pythonactor.c#L286 
>>>
>>>
>> Incidentally, in s_py_zosc_tuple, you're not doing "assert(rc == 0);" 
>> after "after zosc_pop_float" or "zosc_pop_double".
> 
> Thanks for those pointers! I think your intuition is right. I might have
> found the bugger. In s_py_zosc I call Py_DECREF on pAddress and pData.
> However they are acquired by PyTuple_GetItem which returns a borrowed
> reference. I think pAddress and pData are then also 'decrefed' when the
> pReturn tuple which contains pAddress and pData is 'decrefed'?
> 
Yes, members of a container are DECREFed when the container is destroyed.

It's bad practice for a function to DECREF its arguments unless the 
function's sole purpose is cleanup because the function won't know where 
the arguments came from.

> I'm testing it now but it's running stable for a while now.
> 
> Preparing a PR: https://github.com/hku-ect/gazebosc/pull/181
> 


From arnaud at sphaero.org  Wed Nov 24 02:59:15 2021
From: arnaud at sphaero.org (Arnaud Loonstra)
Date: Wed, 24 Nov 2021 08:59:15 +0100
Subject: Embedding Python crash on PyTuple_New
In-Reply-To: <d341afa3-b3ae-ee2e-25e9-ed76a0273704@mrabarnett.plus.com>
References: <d0b57569-ef0d-3ab4-9918-9cd498f1e658@sphaero.org>
 <4eb693c6-fd5f-7c54-246d-0b081a0488d5@mrabarnett.plus.com>
 <c827faf9-0871-fde3-83a0-fa19576b3c62@sphaero.org>
 <d6c7a17a-b24c-9a45-a94e-7bbc3d8fb1c1@mrabarnett.plus.com>
 <f7711e63-a9a7-eb7a-b2b1-0c7ebd9829af@mrabarnett.plus.com>
 <8788bcb1-5e02-b241-3533-43c40d363906@sphaero.org>
 <1d070d88-8b19-c1ea-c700-4a47876fe871@mrabarnett.plus.com>
 <b575c5da-7354-57e5-039b-3b1c05861ffa@sphaero.org>
 <d341afa3-b3ae-ee2e-25e9-ed76a0273704@mrabarnett.plus.com>
Message-ID: <19876e9f-3b51-851e-9339-e36fb35028c1@sphaero.org>


On 24-11-2021 01:46, MRAB wrote:
> On 2021-11-23 20:25, Arnaud Loonstra wrote:
>> On 23-11-2021 18:31, MRAB wrote:
>>> On 2021-11-23 16:04, Arnaud Loonstra wrote:
>>>> On 23-11-2021 16:37, MRAB wrote:
>>>>> On 2021-11-23 15:17, MRAB wrote:
>>>>>> On 2021-11-23 14:44, Arnaud Loonstra wrote:
>>>>>>> On 23-11-2021 15:34, MRAB wrote:
>>>>>>>> On 2021-11-23 12:07, Arnaud Loonstra wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I've got Python embedded successfully in a program up until now 
>>>>>>>>> as I'm
>>>>>>>>> now running into weird GC related segfaults. I'm currently 
>>>>>>>>> trying to
>>>>>>>>> debug this but my understanding of CPython limits me here.
>>>>>>>>>
>>>>>>>>> I'm creating a Tuple in C but it crashes on creating it after a 
>>>>>>>>> while.
>>>>>>>>> It doesn't make sense which makes me wonder something else must be
>>>>>>>>> happening? Could be it just crashes here because the GC is 
>>>>>>>>> cleaning up
>>>>>>>>> stuff completely unrelated to the allocation of the new tuple? 
>>>>>>>>> How can I
>>>>>>>>> troubleshoot this?
>>>>>>>>>
>>>>>>>>> I've got CPython compiled with? --with-valgrind --without-pymalloc
>>>>>>>>> --with-pydebug
>>>>>>>>>
>>>>>>>>> In C I'm creating a tuple with the following method:
>>>>>>>>>
>>>>>>>>> static PyObject *
>>>>>>>>> s_py_zosc_tuple(pythonactor_t *self, zosc_t *oscmsg)
>>>>>>>>> {
>>>>>>>>> ????? assert(self);
>>>>>>>>> ????? assert(oscmsg);
>>>>>>>>> ????? char *format = zosc_format(oscmsg);
>>>>>>>>>
>>>>>>>>> ????? PyObject *rettuple = PyTuple_New((Py_ssize_t) 
>>>>>>>>> strlen(format) );
>>>>>>>>>
>>>>>>>>> It segfaults here (frame 16) after 320 times (consistently)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 1?? __GI_raise???????????? raise.c????????? 49?? 0x7ffff72c4e71
>>>>>>>>> 2?? __GI_abort???????????? abort.c????????? 79?? 0x7ffff72ae536
>>>>>>>>> 3?? fatal_error??????????? pylifecycle.c??? 2183 0x7ffff7d84b4f
>>>>>>>>> 4?? Py_FatalError????????? pylifecycle.c??? 2193 0x7ffff7d878b2
>>>>>>>>> 5?? _PyObject_AssertFailed object.c???????? 2200 0x7ffff7c93cf2
>>>>>>>>> 6?? visit_decref?????????? gcmodule.c?????? 378? 0x7ffff7dadfd5
>>>>>>>>> 7?? tupletraverse????????? tupleobject.c??? 623? 0x7ffff7ca3e81
>>>>>>>>> 8?? subtract_refs????????? gcmodule.c?????? 406? 0x7ffff7dad340
>>>>>>>>> 9?? collect??????????????? gcmodule.c?????? 1054 0x7ffff7dae838
>>>>>>>>> 10? collect_with_callback? gcmodule.c?????? 1240 0x7ffff7daf17b
>>>>>>>>> 11? collect_generations??? gcmodule.c?????? 1262 0x7ffff7daf3f6
>>>>>>>>> 12? _PyObject_GC_Alloc???? gcmodule.c?????? 1977 0x7ffff7daf4f2
>>>>>>>>> 13? _PyObject_GC_Malloc??? gcmodule.c?????? 1987 0x7ffff7dafebc
>>>>>>>>> 14? _PyObject_GC_NewVar??? gcmodule.c?????? 2016 0x7ffff7daffa5
>>>>>>>>> 15? PyTuple_New??????????? tupleobject.c??? 118? 0x7ffff7ca4da7
>>>>>>>>> 16? s_py_zosc_tuple??????? pythonactor.c??? 366? 0x55555568cc82
>>>>>>>>> 17? pythonactor_socket???? pythonactor.c??? 664? 0x55555568dac7
>>>>>>>>> 18? pythonactor_handle_msg pythonactor.c??? 862? 0x55555568e472
>>>>>>>>> 19? pythonactor_handler??? pythonactor.c??? 828? 0x55555568e2e2
>>>>>>>>> 20? sphactor_actor_run???? sphactor_actor.c 855? 0x5555558cb268
>>>>>>>>> ... <More>
>>>>>>>>>
>>>>>>>>> Any pointer really appreciated.

[snip]

>>>>
>>> Basically, yes, but I won't be surprised if it was due to too few 
>>> INCREFs or too many DECREFs somewhere.
>>>
>>>> https://github.com/hku-ect/gazebosc/blob/505b30c46bf3f78d188c3f575c80e294d3db7e5d/Actors/pythonactor.c#L286 
>>>>
>>>>
>>> Incidentally, in s_py_zosc_tuple, you're not doing "assert(rc == 0);" 
>>> after "after zosc_pop_float" or "zosc_pop_double".
>>
>> Thanks for those pointers! I think your intuition is right. I might have
>> found the bugger. In s_py_zosc I call Py_DECREF on pAddress and pData.
>> However they are acquired by PyTuple_GetItem which returns a borrowed
>> reference. I think pAddress and pData are then also 'decrefed' when the
>> pReturn tuple which contains pAddress and pData is 'decrefed'?
>>
> Yes, members of a container are DECREFed when the container is destroyed.
> 
> It's bad practice for a function to DECREF its arguments unless the 
> function's sole purpose is cleanup because the function won't know where 
> the arguments came from.
> 

I'm finding it out now. What strikes me was how hard it was to debug 
this. I think it was caused because I INCREFed the return object. I 
guess I did that to workaround the wrong DECREF data in the return 
object. However that caused a hell to debug. I'm really curious what the 
best practices are for debugging embedded CPython.

Thanks big time for your feedback!

From cspealma at redhat.com  Wed Nov 24 09:51:44 2021
From: cspealma at redhat.com (Calvin Spealman)
Date: Wed, 24 Nov 2021 09:51:44 -0500
Subject: pyinstaller: icon not visable on desktop?
In-Reply-To: <snjjqj$m1m$1@news2.informatik.uni-stuttgart.de>
References: <snjjqj$m1m$1@news2.informatik.uni-stuttgart.de>
Message-ID: <CACo5Rz5xoTuCtaXu=0ezYhqh_7B7VO9m=PVCP5U8HisowEs9-A@mail.gmail.com>

What version of Windows is this on? That's probably going to be a useful
bit of information, because there could be differences there to how EXE
icons are handled.

On Tue, Nov 23, 2021 at 4:18 PM Ulli Horlacher <
framstag at rus.uni-stuttgart.de> wrote:

> When I compile a python program with pyinstaller, I get an executable:
>
> pyinstaller --onefile --icon fex.ico fextasy.py
>
> But the executables icon is only visable within the windows file browser,
> not on the desktop:
>
> https://fex.flupp.org/fop/V2ioMZcl/X-20211123213712.png
>
> There is only the default pyinstaller icon.
>
> Why?
>
>
> --
> Ullrich Horlacher              Server und Virtualisierung
> Rechenzentrum TIK
> Universitaet Stuttgart         E-Mail: horlacher at tik.uni-stuttgart.de
> Allmandring 30a                Tel:    ++49-711-68565868
> 70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
> --
> https://mail.python.org/mailman/listinfo/python-list
>
>

-- 

CALVIN SPEALMAN

SENIOR QUALITY ENGINEER

calvin.spealman at redhat.com  M: +1.336.210.5107
[image: https://red.ht/sig] <https://red.ht/sig>
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>

From nospam_2021 at efbe.prima.de  Wed Nov 24 03:40:41 2021
From: nospam_2021 at efbe.prima.de (nospam_2021 at efbe.prima.de)
Date: Wed, 24 Nov 2021 09:40:41 +0100
Subject: pyinstaller: icon not visable on desktop?
In-Reply-To: <snjjqj$m1m$1@news2.informatik.uni-stuttgart.de>
References: <snjjqj$m1m$1@news2.informatik.uni-stuttgart.de>
Message-ID: <j06c49Fp9iU1@mid.individual.net>

Am 23.11.21 um 21:44 schrieb Ulli Horlacher:
> When I compile a python program with pyinstaller, I get an executable:
> 
> pyinstaller --onefile --icon fex.ico fextasy.py
> 
> But the executables icon is only visable within the windows file browser, 
> not on the desktop:
> 
> https://fex.flupp.org/fop/V2ioMZcl/X-20211123213712.png
> 

I had to postprocess the exe with resourcehacker freeware:
http://angusj.com/resourcehacker/

"C:\Program Files (x86)\Resource Hacker\ResourceHacker.exe" -open
skydiaA.exe -save skydia.exe -action addskip -res Mainicon.ico -mask
ICONGROUP,MAINICON,

creates skydia.exe with the "wish icon".
skydiaA.exe has the default icon.

HTH
Frank



From python at mrabarnett.plus.com  Wed Nov 24 13:15:43 2021
From: python at mrabarnett.plus.com (MRAB)
Date: Wed, 24 Nov 2021 18:15:43 +0000
Subject: Embedding Python crash on PyTuple_New
In-Reply-To: <19876e9f-3b51-851e-9339-e36fb35028c1@sphaero.org>
References: <d0b57569-ef0d-3ab4-9918-9cd498f1e658@sphaero.org>
 <4eb693c6-fd5f-7c54-246d-0b081a0488d5@mrabarnett.plus.com>
 <c827faf9-0871-fde3-83a0-fa19576b3c62@sphaero.org>
 <d6c7a17a-b24c-9a45-a94e-7bbc3d8fb1c1@mrabarnett.plus.com>
 <f7711e63-a9a7-eb7a-b2b1-0c7ebd9829af@mrabarnett.plus.com>
 <8788bcb1-5e02-b241-3533-43c40d363906@sphaero.org>
 <1d070d88-8b19-c1ea-c700-4a47876fe871@mrabarnett.plus.com>
 <b575c5da-7354-57e5-039b-3b1c05861ffa@sphaero.org>
 <d341afa3-b3ae-ee2e-25e9-ed76a0273704@mrabarnett.plus.com>
 <19876e9f-3b51-851e-9339-e36fb35028c1@sphaero.org>
Message-ID: <02626ade-9ec0-8a30-b500-637cf527361a@mrabarnett.plus.com>

On 2021-11-24 07:59, Arnaud Loonstra wrote:
> 
> On 24-11-2021 01:46, MRAB wrote:
>> On 2021-11-23 20:25, Arnaud Loonstra wrote:
>>> On 23-11-2021 18:31, MRAB wrote:
>>>> On 2021-11-23 16:04, Arnaud Loonstra wrote:
>>>>> On 23-11-2021 16:37, MRAB wrote:
>>>>>> On 2021-11-23 15:17, MRAB wrote:
>>>>>>> On 2021-11-23 14:44, Arnaud Loonstra wrote:
>>>>>>>> On 23-11-2021 15:34, MRAB wrote:
>>>>>>>>> On 2021-11-23 12:07, Arnaud Loonstra wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I've got Python embedded successfully in a program up until now 
>>>>>>>>>> as I'm
>>>>>>>>>> now running into weird GC related segfaults. I'm currently 
>>>>>>>>>> trying to
>>>>>>>>>> debug this but my understanding of CPython limits me here.
>>>>>>>>>>
>>>>>>>>>> I'm creating a Tuple in C but it crashes on creating it after a 
>>>>>>>>>> while.
>>>>>>>>>> It doesn't make sense which makes me wonder something else must be
>>>>>>>>>> happening? Could be it just crashes here because the GC is 
>>>>>>>>>> cleaning up
>>>>>>>>>> stuff completely unrelated to the allocation of the new tuple? 
>>>>>>>>>> How can I
>>>>>>>>>> troubleshoot this?
>>>>>>>>>>
>>>>>>>>>> I've got CPython compiled with? --with-valgrind --without-pymalloc
>>>>>>>>>> --with-pydebug
>>>>>>>>>>
>>>>>>>>>> In C I'm creating a tuple with the following method:
>>>>>>>>>>
>>>>>>>>>> static PyObject *
>>>>>>>>>> s_py_zosc_tuple(pythonactor_t *self, zosc_t *oscmsg)
>>>>>>>>>> {
>>>>>>>>>> ????? assert(self);
>>>>>>>>>> ????? assert(oscmsg);
>>>>>>>>>> ????? char *format = zosc_format(oscmsg);
>>>>>>>>>>
>>>>>>>>>> ????? PyObject *rettuple = PyTuple_New((Py_ssize_t) 
>>>>>>>>>> strlen(format) );
>>>>>>>>>>
>>>>>>>>>> It segfaults here (frame 16) after 320 times (consistently)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 1?? __GI_raise???????????? raise.c????????? 49?? 0x7ffff72c4e71
>>>>>>>>>> 2?? __GI_abort???????????? abort.c????????? 79?? 0x7ffff72ae536
>>>>>>>>>> 3?? fatal_error??????????? pylifecycle.c??? 2183 0x7ffff7d84b4f
>>>>>>>>>> 4?? Py_FatalError????????? pylifecycle.c??? 2193 0x7ffff7d878b2
>>>>>>>>>> 5?? _PyObject_AssertFailed object.c???????? 2200 0x7ffff7c93cf2
>>>>>>>>>> 6?? visit_decref?????????? gcmodule.c?????? 378? 0x7ffff7dadfd5
>>>>>>>>>> 7?? tupletraverse????????? tupleobject.c??? 623? 0x7ffff7ca3e81
>>>>>>>>>> 8?? subtract_refs????????? gcmodule.c?????? 406? 0x7ffff7dad340
>>>>>>>>>> 9?? collect??????????????? gcmodule.c?????? 1054 0x7ffff7dae838
>>>>>>>>>> 10? collect_with_callback? gcmodule.c?????? 1240 0x7ffff7daf17b
>>>>>>>>>> 11? collect_generations??? gcmodule.c?????? 1262 0x7ffff7daf3f6
>>>>>>>>>> 12? _PyObject_GC_Alloc???? gcmodule.c?????? 1977 0x7ffff7daf4f2
>>>>>>>>>> 13? _PyObject_GC_Malloc??? gcmodule.c?????? 1987 0x7ffff7dafebc
>>>>>>>>>> 14? _PyObject_GC_NewVar??? gcmodule.c?????? 2016 0x7ffff7daffa5
>>>>>>>>>> 15? PyTuple_New??????????? tupleobject.c??? 118? 0x7ffff7ca4da7
>>>>>>>>>> 16? s_py_zosc_tuple??????? pythonactor.c??? 366? 0x55555568cc82
>>>>>>>>>> 17? pythonactor_socket???? pythonactor.c??? 664? 0x55555568dac7
>>>>>>>>>> 18? pythonactor_handle_msg pythonactor.c??? 862? 0x55555568e472
>>>>>>>>>> 19? pythonactor_handler??? pythonactor.c??? 828? 0x55555568e2e2
>>>>>>>>>> 20? sphactor_actor_run???? sphactor_actor.c 855? 0x5555558cb268
>>>>>>>>>> ... <More>
>>>>>>>>>>
>>>>>>>>>> Any pointer really appreciated.
> 
> [snip]
> 
>>>>>
>>>> Basically, yes, but I won't be surprised if it was due to too few 
>>>> INCREFs or too many DECREFs somewhere.
>>>>
>>>>> https://github.com/hku-ect/gazebosc/blob/505b30c46bf3f78d188c3f575c80e294d3db7e5d/Actors/pythonactor.c#L286 
>>>>>
>>>>>
>>>> Incidentally, in s_py_zosc_tuple, you're not doing "assert(rc == 0);" 
>>>> after "after zosc_pop_float" or "zosc_pop_double".
>>>
>>> Thanks for those pointers! I think your intuition is right. I might have
>>> found the bugger. In s_py_zosc I call Py_DECREF on pAddress and pData.
>>> However they are acquired by PyTuple_GetItem which returns a borrowed
>>> reference. I think pAddress and pData are then also 'decrefed' when the
>>> pReturn tuple which contains pAddress and pData is 'decrefed'?
>>>
>> Yes, members of a container are DECREFed when the container is destroyed.
>> 
>> It's bad practice for a function to DECREF its arguments unless the 
>> function's sole purpose is cleanup because the function won't know where 
>> the arguments came from.
>> 
> 
> I'm finding it out now. What strikes me was how hard it was to debug
> this. I think it was caused because I INCREFed the return object. I
> guess I did that to workaround the wrong DECREF data in the return
> object. However that caused a hell to debug. I'm really curious what the
> best practices are for debugging embedded CPython.
> 
> Thanks big time for your feedback!
> 
What I do when writing the code is add comments showing what variables 
refer to an object at that point in the code, each suffixed with "+" if 
it owns a reference and/or "?" if it could be NULL.

Example 1:

     //>
     PyObject *my_tuple = PyTuple_New(count);
     //> my_tuple+?
     if (!my_tuple)
          goto error;
     //> my_tuple+

"//>" means that there are no variables that point to an object.

"//> my_tuple+?" means that "my_tuple" points to an object and it owns a 
reference, but it might be NULL.

"//> my_tuple+" means that "my_tuple" points to an object and it owns a 
reference.

Example 2:

     //>
     PyObject *my_item = PyList_New(my_list, index);
     //> my_tuple?
     if (!my_tuple)
          goto error;
     //> my_tuple

"//>" means that there are no variables that point to an object.

"//> my_tuple?" means that "my_tuple" points to an object, but it might 
be NULL.

"//> my_tuple" means that "my_tuple" points to an object.


From framstag at rus.uni-stuttgart.de  Wed Nov 24 14:17:25 2021
From: framstag at rus.uni-stuttgart.de (Ulli Horlacher)
Date: Wed, 24 Nov 2021 19:17:25 +0000 (UTC)
Subject: pyinstaller: icon not visable on desktop?
References: <snjjqj$m1m$1@news2.informatik.uni-stuttgart.de>
 <CACo5Rz5xoTuCtaXu=0ezYhqh_7B7VO9m=PVCP5U8HisowEs9-A@mail.gmail.com>
 <mailman.1.1637765527.9869.python-list@python.org>
Message-ID: <snm344$ahp$1@news2.informatik.uni-stuttgart.de>

Calvin Spealman <cspealma at redhat.com> wrote:
> What version of Windows is this on? That's probably going to be a useful
> bit of information, because there could be differences there to how EXE
> icons are handled.
> 
> On Tue, Nov 23, 2021 at 4:18 PM Ulli Horlacher <
> framstag at rus.uni-stuttgart.de> wrote:
> 
> > When I compile a python program with pyinstaller, I get an executable:
> >
> > pyinstaller --onefile --icon fex.ico fextasy.py
> >
> > But the executables icon is only visable within the windows file browser,
> > not on the desktop:
> >
> > https://fex.flupp.org/fop/V2ioMZcl/X-20211123213712.png

Python says: Windows 10.0.19041


-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher at tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/

From jenkris at tutanota.com  Wed Nov 24 17:42:38 2021
From: jenkris at tutanota.com (jenkris at tutanota.com)
Date: Wed, 24 Nov 2021 23:42:38 +0100 (CET)
Subject: Eventfd with epoll BlockingIOError
Message-ID: <MpJ6MZm--B-2@tutanota.com>

I have a C program that uses fork-execv to run Python 3.10 in a child process, and I am using eventfd with epoll for IPC between them.? The eventfd file descriptor is created in C and passed to Python through execv.? Once the Python child process starts I print the file descriptor to verify that it is correct (it is).? 

In this scenario C will write to the eventfd at intervals and Python will read the eventfd and take action based on the value in the eventfd.? But in the Python while True loop I get "BlockingIOError: [Errno 11] Resource temporarily unavailable" then with each new read it prints "Failed epoll_wait Bad file descriptor."?

This is the Python code:

#!/usr/bin/python3
import sys
import os
import select

print("Inside Python")

event_fd = int(sys.argv[3])

print("Eventfd received by Python")
print(event_fd)

ep = select.epoll(-1)
ep.register(event_fd, select.EPOLLIN | select.EPOLLOUT)

event_write_value = 100

while True:

??? print("Waiting in Python for event")
??? ep.poll(timeout=None, maxevents=- 1)
??? v = os.eventfd_read(event_fd)

??? if v != 99:
??????? print("found")
??????? print(v)
??????? os.eventfd_write(event_fd, event_write_value)

??? if v == 99:
??????? os.close(event_fd)

This is the C code that writes to Python, then waits for Python to write back:

ssize_t epoll_write(int event_fd, int epoll_fd, struct epoll_event * event_struc, int action_code)
{
?? int64_t ewbuf[1];
?? ewbuf[0] = (int64_t)action_code;
?? int maxevents = 1;
?? int timeout = -1;

?? fprintf(stdout, " Writing to Python \n%d", event_fd);

?? write(event_fd, &ewbuf, 8);

??? if (epoll_wait(epoll_fd, event_struc, maxevents, timeout) == -1)
??? {
??????? fprintf(stderr, "Failed epoll_wait %s\n", strerror(errno));
??? }

??? ssize_t rdval = read(event_fd, &ewbuf, 8);???

??? fprintf(stdout, " Received from Python \n%ld", rdval);

??? return 0;
}

This is the screen output when I run with gdb:

????????? Inside Python
Eventfd received by Python
5
Waiting in Python for event
Traceback (most recent call last):
? File "/usr/local/lib/python3.10/runpy.py", line 196, in 	_run_module_as_main
??? return _run_code(code, main_globals, None,
? File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
??? exec(code, run_globals)
? File "/opt/P01_SH/NPC_CPython.py", line 36, in <module>
??? v = os.eventfd_read(event_fd)
BlockingIOError: [Errno 11] Resource temporarily unavailable
Writing to Python
5 Received from Python
8 Writing to Python
Failed epoll_wait Bad file descriptor
5 Received from Python
8 Writing to Python
Failed epoll_wait Bad file descriptor
5 Received from Python
-1time taken 0.000548
Failed to close epoll file descriptor
Unlink_shm status: Bad file descriptor
fn() took 0.000648 seconds to execute
[Inferior 1 (process 12618) exited normally]
(gdb)

So my question is why do I get "BlockingIOError: [Errno 11] Resource temporarily unavailable" and "Failed epoll_wait Bad file descriptor" from Python??

-- 
 Sent with Tutanota, the secure & ad-free mailbox. 

From barry at barrys-emacs.org  Thu Nov 25 03:47:46 2021
From: barry at barrys-emacs.org (Barry)
Date: Thu, 25 Nov 2021 08:47:46 +0000
Subject: Eventfd with epoll BlockingIOError
In-Reply-To: <MpJ6MZm--B-2@tutanota.com>
References: <MpJ6MZm--B-2@tutanota.com>
Message-ID: <F5592CF0-5F56-4AA4-94E1-61BB7B67DA4C@barrys-emacs.org>



> On 24 Nov 2021, at 23:09, Jen via Python-list <python-list at python.org> wrote:
> 
> ?I have a C program that uses fork-execv to run Python 3.10 in a child process, and I am using eventfd with epoll for IPC between them.  The eventfd file descriptor is created in C and passed to Python through execv.  Once the Python child process starts I print the file descriptor to verify that it is correct (it is).  
> 
> In this scenario C will write to the eventfd at intervals and Python will read the eventfd and take action based on the value in the eventfd.  But in the Python while True loop I get "BlockingIOError: [Errno 11] Resource temporarily unavailable" then with each new read it prints "Failed epoll_wait Bad file descriptor." 
> 
> This is the Python code:
> 
> #!/usr/bin/python3
> import sys
> import os
> import select
> 
> print("Inside Python")
> 
> event_fd = int(sys.argv[3])
> 
> print("Eventfd received by Python")
> print(event_fd)
> 
> ep = select.epoll(-1)
> ep.register(event_fd, select.EPOLLIN | select.EPOLLOUT)
> 
> event_write_value = 100
> 
> while True:
> 
>     print("Waiting in Python for event")
>     ep.poll(timeout=None, maxevents=- 1)
>     v = os.eventfd_read(event_fd)
> 
>     if v != 99:
>         print("found")
>         print(v)
>         os.eventfd_write(event_fd, event_write_value)
> 
>     if v == 99:
>         os.close(event_fd)
> 
> This is the C code that writes to Python, then waits for Python to write back:
> 
> ssize_t epoll_write(int event_fd, int epoll_fd, struct epoll_event * event_struc, int action_code)
> {
>    int64_t ewbuf[1];
>    ewbuf[0] = (int64_t)action_code;
>    int maxevents = 1;
>    int timeout = -1;
> 
>    fprintf(stdout, " Writing to Python \n%d", event_fd);
> 
>    write(event_fd, &ewbuf, 8);
> 
>     if (epoll_wait(epoll_fd, event_struc, maxevents, timeout) == -1)
>     {
>         fprintf(stderr, "Failed epoll_wait %s\n", strerror(errno));
>     }
> 
>     ssize_t rdval = read(event_fd, &ewbuf, 8);   
> 
>     fprintf(stdout, " Received from Python \n%ld", rdval);
> 
>     return 0;
> }
> 
> This is the screen output when I run with gdb:
> 
>           Inside Python
> Eventfd received by Python
> 5
> Waiting in Python for event
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.10/runpy.py", line 196, in    _run_module_as_main
>     return _run_code(code, main_globals, None,
>   File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
>     exec(code, run_globals)
>   File "/opt/P01_SH/NPC_CPython.py", line 36, in <module>
>     v = os.eventfd_read(event_fd)
> BlockingIOError: [Errno 11] Resource temporarily unavailable
> Writing to Python
> 5 Received from Python
> 8 Writing to Python
> Failed epoll_wait Bad file descriptor
> 5 Received from Python
> 8 Writing to Python
> Failed epoll_wait Bad file descriptor
> 5 Received from Python
> -1time taken 0.000548
> Failed to close epoll file descriptor
> Unlink_shm status: Bad file descriptor
> fn() took 0.000648 seconds to execute
> [Inferior 1 (process 12618) exited normally]
> (gdb)
> 
> So my question is why do I get "BlockingIOError: [Errno 11] Resource temporarily unavailable" and "Failed epoll_wait Bad file descriptor" from Python? 

Is the fd set to non blocking? Try use fcntl to set the fd non blocking.
I am not sure if the non block state is inherited on a fork.

Do you have a C version of this code that works as a test?
Try running the python version under strace to see exactly what system calls are used.

Barry

> 
> -- 
> Sent with Tutanota, the secure & ad-free mailbox. 
> -- 
> https://mail.python.org/mailman/listinfo/python-list


From framstag at rus.uni-stuttgart.de  Thu Nov 25 04:20:47 2021
From: framstag at rus.uni-stuttgart.de (Ulli Horlacher)
Date: Thu, 25 Nov 2021 09:20:47 +0000 (UTC)
Subject: pyinstaller wrong classified as Windows virus
Message-ID: <snnkhf$mve$1@news2.informatik.uni-stuttgart.de>

When I compile my programs with pyinstaller, Windows classifies them as
virus and even deletes them!

pyinstaller.exe --onefile --noconsole -i fex.ico fextasy.py
187 INFO: PyInstaller: 4.7
187 INFO: Python: 3.10.0
218 INFO: Platform: Windows-10-10.0.19041-SP0
218 INFO: wrote P:\W10\fextasy.spec
(...)
14392 INFO: Copying 0 resources to EXE
14392 INFO: Emedding manifest in EXE
14392 INFO: Updating manifest in P:\W10\dist\fextasy.exe
14533 INFO: Updating resource type 24 name 1 language 0
14579 INFO: Appending PKG archive to EXE
18836 INFO: Building EXE from EXE-00.toc completed successfully.

https://fex.flupp.org/fop/ylds7Y9d/X-20211125101112.png

What can I do?

Rebooting does not help :-}

-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher at tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/

From dan at djph.net  Thu Nov 25 06:01:46 2021
From: dan at djph.net (Dan Purgert)
Date: Thu, 25 Nov 2021 11:01:46 -0000 (UTC)
Subject: pyinstaller wrong classified as Windows virus
References: <snnkhf$mve$1@news2.informatik.uni-stuttgart.de>
Message-ID: <slrnspur9f.51o.dan@djph.net>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Ulli Horlacher wrote:
> When I compile my programs with pyinstaller, Windows classifies them as
> virus and even deletes them!
> [...]
> What can I do?

Stop writing viruses ;)

Have you tried compiling from a different machine? Maybe there's
something broken on the one that's flagging them. 

Alternatively, maybe run updates?

> Rebooting does not help :-}
If testing from a different machine works ... format and reinstall might
be the fastest fix for the affected machine.

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEE3asj+xn6fYUcweBnbWVw5UznKGAFAmGfbS8ACgkQbWVw5Uzn
KGBvQA//YuSJChhmj5fXzkOHD5W+horNsyS3psmFjQzp/opeeF+tqfkeDNXyUwkb
+YxDgq7KDvpaUbaMiYtmzs0fZ6fLaWPHA6pJFS3+DS8Wu/EoPmHyfe6bw0OOFiBI
AB8Zr8Y7e1GntyE/HQ53+outTxDwpTStU+MTyqTjfMDRYl3NyYhnb0rbt0aTQWP0
LzYC6HM5g2EFTrfiAImuoJisu5aofw90QjJEqlRn+MQNcFm8bvAdXpWQ6fp0iNRx
IbXhb+HoL/WNy9sxeWoBRMrNY9wu1lgBDI8IPIu8m+rz13WgIX/l2CMbgxiXW3bN
WbKa4UZ1z7NB2sYhhJ9D0V4pUukFae2A9RaDV6aDC5Vm7ZNQ7eLuEsbjgXABP/qc
0OZODTUN/KYSlMa6E5azdxR5um4jQ85hcsKjtOr7UqwMurrzRL9KxSgsZVlj5FzY
vCxCw1aoqD8WBxunOLnwKd8Fonch0/Ov8a6IA1ZKxx3VbDlY3ZgVMpqYtjzw+wgH
hMD2Int0fJtiC+TBl18P7s47q5RlXbA7mCrjEXuJGdIbZ6K9SxMjWnWycCcVwNHC
LrAKWiA2xOSNcJzpXYN8U2TMsYZCLdLeuNyR5T8QLTvHokQTYuTba2M2EYraQt73
oWQsu47ws/IIAQ/EXOBGMcenhE3wTPHT/36a6udOAQktHzbqKGg=
=nm2S
-----END PGP SIGNATURE-----

-- 
|_|O|_| Github: https://github.com/dpurgert
|_|_|O| PGP: DDAB 23FB 19FA 7D85 1CC1  E067 6D65 70E5 4CE7 2860
|O|O|O| Former PGP: 05CA 9A50 3F2E 1335 4DC5  4AEE 8E11 DDF3 1279 A281

From framstag at rus.uni-stuttgart.de  Thu Nov 25 07:41:06 2021
From: framstag at rus.uni-stuttgart.de (Ulli Horlacher)
Date: Thu, 25 Nov 2021 12:41:06 +0000 (UTC)
Subject: pyinstaller wrong classified as Windows virus
References: <snnkhf$mve$1@news2.informatik.uni-stuttgart.de>
 <slrnspur9f.51o.dan@djph.net>
Message-ID: <sno092$prd$1@news2.informatik.uni-stuttgart.de>

Dan Purgert <dan at djph.net> wrote:

> > When I compile my programs with pyinstaller, Windows classifies them as
> > virus and even deletes them!
> > [...]
> 
> Have you tried compiling from a different machine? Maybe there's
> something broken on the one that's flagging them. 

I have only this Windows installation.

But meanwhile I can compile my program and Windows does not complain any
more about Viruses, though I have not changed anything in my source code!


> Alternatively, maybe run updates?

I have done this before without any success (in respect to the wrong virus
report).

-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher at tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/

From barry at barrys-emacs.org  Thu Nov 25 09:34:31 2021
From: barry at barrys-emacs.org (Barry Scott)
Date: Thu, 25 Nov 2021 14:34:31 +0000
Subject: Eventfd with epoll BlockingIOError
In-Reply-To: <MpJ6MZm--B-2@tutanota.com>
References: <MpJ6MZm--B-2@tutanota.com>
Message-ID: <A48057B9-1031-4DC8-9FC9-E59D2DA14BB4@barrys-emacs.org>



> On 24 Nov 2021, at 22:42, Jen via Python-list <python-list at python.org> wrote:
> 
> I have a C program that uses fork-execv to run Python 3.10 in a child process, and I am using eventfd with epoll for IPC between them.  The eventfd file descriptor is created in C and passed to Python through execv.  Once the Python child process starts I print the file descriptor to verify that it is correct (it is).  
> 
> In this scenario C will write to the eventfd at intervals and Python will read the eventfd and take action based on the value in the eventfd.  But in the Python while True loop I get "BlockingIOError: [Errno 11] Resource temporarily unavailable" then with each new read it prints "Failed epoll_wait Bad file descriptor." 
> 
> This is the Python code:
> 
> #!/usr/bin/python3
> import sys
> import os
> import select
> 
> print("Inside Python")
> 
> event_fd = int(sys.argv[3])
> 
> print("Eventfd received by Python")
> print(event_fd)
> 
> ep = select.epoll(-1)
> ep.register(event_fd, select.EPOLLIN | select.EPOLLOUT)

This says tell me if I can read or write to the event_fd.
write will be allowed until the kernel buffers are full.

Usually you only add EPOLLOUT if you have data to write.
In this case do not set EPOLLOUT.

And if you know that you will never fill the kernel buffers then you
do not need to bother polling for write.

> 
> event_write_value = 100
> 
> while True:
> 
>     print("Waiting in Python for event")
>     ep.poll(timeout=None, maxevents=- 1)

You have to get the result of the poll() and process the list of entries that are returned.

You must check that POLLIN is set before attempting the read.


>     v = os.eventfd_read(event_fd)

Will raise EWOULDBLOCK because there is no data available to read.

Here is the docs from python:

poll.poll([timeout]) <file:///Library/Frameworks/Python.framework/Versions/3.9/Resources/English.lproj/Documentation/library/select.html?highlight=select#select.poll.poll>
Polls the set of registered file descriptors, and returns a possibly-empty list containing (fd, event) 2-tuples for the descriptors that have events or errors to report. fd is the file descriptor, and event is a bitmask with bits set for the reported events for that descriptor ? POLLIN for waiting input, POLLOUT to indicate that the descriptor can be written to, and so forth. An empty list indicates that the call timed out and no file descriptors had any events to report. If timeout is given, it specifies the length of time in milliseconds which the system will wait for events before returning. If timeout is omitted, negative, or None <file:///Library/Frameworks/Python.framework/Versions/3.9/Resources/English.lproj/Documentation/library/constants.html#None>, the call will block until there is an event for this poll object.

You end up with code like this:

for fd_event in ep.poll():
	fd, event == fd_event
	if (event&select.POLLIN) != 0 and fd == event_fd:
		v = os.eventfd_read(event_fd)

> 
>     if v != 99:
>         print("found")
>         print(v)
>         os.eventfd_write(event_fd, event_write_value)
> 
>     if v == 99:
>         os.close(event_fd)
> 
> This is the C code that writes to Python, then waits for Python to write back:
> 
> ssize_t epoll_write(int event_fd, int epoll_fd, struct epoll_event * event_struc, int action_code)
> {
>    int64_t ewbuf[1];
>    ewbuf[0] = (int64_t)action_code;
>    int maxevents = 1;
>    int timeout = -1;
> 
>    fprintf(stdout, " Writing to Python \n%d", event_fd);
> 
>    write(event_fd, &ewbuf, 8);
> 
>     if (epoll_wait(epoll_fd, event_struc, maxevents, timeout) == -1)
>     {
>         fprintf(stderr, "Failed epoll_wait %s\n", strerror(errno));
>     }
> 
>     ssize_t rdval = read(event_fd, &ewbuf, 8);   
> 
>     fprintf(stdout, " Received from Python \n%ld", rdval);
> 
>     return 0;
> }
> 
> This is the screen output when I run with gdb:
> 
>           Inside Python
> Eventfd received by Python
> 5
> Waiting in Python for event
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.10/runpy.py", line 196, in 	_run_module_as_main
>     return _run_code(code, main_globals, None,
>   File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
>     exec(code, run_globals)
>   File "/opt/P01_SH/NPC_CPython.py", line 36, in <module>
>     v = os.eventfd_read(event_fd)
> BlockingIOError: [Errno 11] Resource temporarily unavailable

Expected as there you have not checked that there is data to read.
Check for POLLIN being set.

> Writing to Python
> 5 Received from Python
> 8 Writing to Python
> Failed epoll_wait Bad file descriptor
> 5 Received from Python
> 8 Writing to Python
> Failed epoll_wait Bad file descriptor
> 5 Received from Python
> -1time taken 0.000548
> Failed to close epoll file descriptor
> Unlink_shm status: Bad file descriptor
> fn() took 0.000648 seconds to execute
> [Inferior 1 (process 12618) exited normally]
> (gdb)
> 
> So my question is why do I get "BlockingIOError: [Errno 11] Resource temporarily unavailable" and "Failed epoll_wait Bad file descriptor" from Python? 

If your protocol is not trivia you should implement a state machine to know what to do at each event.

Barry

> 
> -- 
> Sent with Tutanota, the secure & ad-free mailbox. 
> -- 
> https://mail.python.org/mailman/listinfo/python-list


From barry at barrys-emacs.org  Thu Nov 25 09:38:27 2021
From: barry at barrys-emacs.org (Barry Scott)
Date: Thu, 25 Nov 2021 14:38:27 +0000
Subject: pyinstaller wrong classified as Windows virus
In-Reply-To: <snnkhf$mve$1@news2.informatik.uni-stuttgart.de>
References: <snnkhf$mve$1@news2.informatik.uni-stuttgart.de>
Message-ID: <253A025C-CA28-44FF-BB2D-5CA286D01012@barrys-emacs.org>



> On 25 Nov 2021, at 09:20, Ulli Horlacher <framstag at rus.uni-stuttgart.de> wrote:
> 
> When I compile my programs with pyinstaller, Windows classifies them as
> virus and even deletes them!

Microsoft will fix the malware detection if you provide the info they need.

Submit false positive info to: https://www.microsoft.com/security/portal/submission/submit.aspx <https://www.microsoft.com/security/portal/submission/submit.aspx>

I have done this for a false positive in the past and they do resolve the false positive.

Barry


> 
> pyinstaller.exe --onefile --noconsole -i fex.ico fextasy.py
> 187 INFO: PyInstaller: 4.7
> 187 INFO: Python: 3.10.0
> 218 INFO: Platform: Windows-10-10.0.19041-SP0
> 218 INFO: wrote P:\W10\fextasy.spec
> (...)
> 14392 INFO: Copying 0 resources to EXE
> 14392 INFO: Emedding manifest in EXE
> 14392 INFO: Updating manifest in P:\W10\dist\fextasy.exe
> 14533 INFO: Updating resource type 24 name 1 language 0
> 14579 INFO: Appending PKG archive to EXE
> 18836 INFO: Building EXE from EXE-00.toc completed successfully.
> 
> https://fex.flupp.org/fop/ylds7Y9d/X-20211125101112.png
> 
> What can I do?
> 
> Rebooting does not help :-}
> 
> -- 
> Ullrich Horlacher              Server und Virtualisierung
> Rechenzentrum TIK         
> Universitaet Stuttgart         E-Mail: horlacher at tik.uni-stuttgart.de
> Allmandring 30a                Tel:    ++49-711-68565868
> 70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 


From torriem at gmail.com  Thu Nov 25 10:11:14 2021
From: torriem at gmail.com (Michael Torrie)
Date: Thu, 25 Nov 2021 08:11:14 -0700
Subject: pyinstaller wrong classified as Windows virus
In-Reply-To: <snnkhf$mve$1@news2.informatik.uni-stuttgart.de>
References: <snnkhf$mve$1@news2.informatik.uni-stuttgart.de>
Message-ID: <bef653be-a764-a297-077a-55921baafd59@gmail.com>

On 11/25/21 2:20 AM, Ulli Horlacher wrote:
> When I compile my programs with pyinstaller, Windows classifies them as
> virus and even deletes them!
> 
> pyinstaller.exe --onefile --noconsole -i fex.ico fextasy.py
> 187 INFO: PyInstaller: 4.7
> 187 INFO: Python: 3.10.0
> 218 INFO: Platform: Windows-10-10.0.19041-SP0
> 218 INFO: wrote P:\W10\fextasy.spec
> (...)
> 14392 INFO: Copying 0 resources to EXE
> 14392 INFO: Emedding manifest in EXE
> 14392 INFO: Updating manifest in P:\W10\dist\fextasy.exe
> 14533 INFO: Updating resource type 24 name 1 language 0
> 14579 INFO: Appending PKG archive to EXE
> 18836 INFO: Building EXE from EXE-00.toc completed successfully.
> 
> https://fex.flupp.org/fop/ylds7Y9d/X-20211125101112.png
> 
> What can I do?

False positive virus detection is pretty common with pyinstaller from
what I can see on the Googles.  It's actually very common problem with
less-popular compilers and languages too. Not sure what it is that trips
them all up.

Submit your exe to virustotal.com and then the only real solution is to
submit it to each major antivirus vendor as a false positive and hope
things get changed.

From framstag at rus.uni-stuttgart.de  Thu Nov 25 08:41:49 2021
From: framstag at rus.uni-stuttgart.de (Ulli Horlacher)
Date: Thu, 25 Nov 2021 13:41:49 +0000 (UTC)
Subject: pyinstaller wrong classified as Windows virus
References: <snnkhf$mve$1@news2.informatik.uni-stuttgart.de>
 <slrnspur9f.51o.dan@djph.net>
 <sno092$prd$1@news2.informatik.uni-stuttgart.de>
Message-ID: <sno3qt$r1b$1@news2.informatik.uni-stuttgart.de>

Ulli Horlacher <framstag at rus.uni-stuttgart.de> wrote:
> Dan Purgert <dan at djph.net> wrote:
> 
> > > When I compile my programs with pyinstaller, Windows classifies them as
> > > virus and even deletes them!
> > > [...]
> > 
> > Have you tried compiling from a different machine? Maybe there's
> > something broken on the one that's flagging them. 
> 
> I have only this Windows installation.
> 
> But meanwhile I can compile my program and Windows does not complain any
> more about Viruses, though I have not changed anything in my source code!

And now the bad virus reports are back. With the SAME sourcecode!
It is totally erratic.

-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher at tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/

From framstag at rus.uni-stuttgart.de  Thu Nov 25 11:08:17 2021
From: framstag at rus.uni-stuttgart.de (Ulli Horlacher)
Date: Thu, 25 Nov 2021 16:08:17 +0000 (UTC)
Subject: pyinstaller wrong classified as Windows virus
References: <snnkhf$mve$1@news2.informatik.uni-stuttgart.de>
 <253A025C-CA28-44FF-BB2D-5CA286D01012@barrys-emacs.org>
 <mailman.6.1637851110.9869.python-list@python.org>
Message-ID: <snocdh$t49$1@news2.informatik.uni-stuttgart.de>

Barry Scott <barry at barrys-emacs.org> wrote:
> 
> 
> > On 25 Nov 2021, at 09:20, Ulli Horlacher <framstag at rus.uni-stuttgart.de> wrote:
> > 
> > When I compile my programs with pyinstaller, Windows classifies them as
> > virus and even deletes them!
> 
> Microsoft will fix the malware detection if you provide the info they need.
> 
> Submit false positive info to: https://www.microsoft.com/security/portal/submission/submit.aspx <https://www.microsoft.com/security/portal/submission/submit.aspx>

I cannot submit my executables, because the Windows Virus scannners
deletes them as soon as I compile my program!

And I need a Microsoft login to submit a file!
I do not have such a login.


-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher at tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/

From rosuav at gmail.com  Thu Nov 25 11:53:54 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 26 Nov 2021 03:53:54 +1100
Subject: pyinstaller wrong classified as Windows virus
In-Reply-To: <sno3qt$r1b$1@news2.informatik.uni-stuttgart.de>
References: <snnkhf$mve$1@news2.informatik.uni-stuttgart.de>
 <slrnspur9f.51o.dan@djph.net> <sno092$prd$1@news2.informatik.uni-stuttgart.de>
 <sno3qt$r1b$1@news2.informatik.uni-stuttgart.de>
Message-ID: <CAPTjJmq=w3iFDWjzNFAfpmAC=5kFZm1C8=Ex7Cwo2iwDVY_LHA@mail.gmail.com>

On Fri, Nov 26, 2021 at 3:49 AM Ulli Horlacher
<framstag at rus.uni-stuttgart.de> wrote:
>
> Ulli Horlacher <framstag at rus.uni-stuttgart.de> wrote:
> > Dan Purgert <dan at djph.net> wrote:
> >
> > > > When I compile my programs with pyinstaller, Windows classifies them as
> > > > virus and even deletes them!
> > > > [...]
> > >
> > > Have you tried compiling from a different machine? Maybe there's
> > > something broken on the one that's flagging them.
> >
> > I have only this Windows installation.
> >
> > But meanwhile I can compile my program and Windows does not complain any
> > more about Viruses, though I have not changed anything in my source code!
>
> And now the bad virus reports are back. With the SAME sourcecode!
> It is totally erratic.
>

Your Python source code is only a very small part of the code.
Unfortunately, if you're not going to go to the effort of getting your
executables signed and added to the full ecosystem, this sort of
hassle is going to be eternal. It's yet another way that turning
Python scripts into executables is a ridiculous amount of hassle. Yet
another reason to just distribute .py files.

ChrisA

From framstag at rus.uni-stuttgart.de  Thu Nov 25 12:10:02 2021
From: framstag at rus.uni-stuttgart.de (Ulli Horlacher)
Date: Thu, 25 Nov 2021 17:10:02 +0000 (UTC)
Subject: pyinstaller wrong classified as Windows virus
References: <snnkhf$mve$1@news2.informatik.uni-stuttgart.de>
 <slrnspur9f.51o.dan@djph.net>
 <sno092$prd$1@news2.informatik.uni-stuttgart.de>
 <sno3qt$r1b$1@news2.informatik.uni-stuttgart.de>
 <CAPTjJmq=w3iFDWjzNFAfpmAC=5kFZm1C8=Ex7Cwo2iwDVY_LHA@mail.gmail.com>
 <mailman.8.1637859246.9869.python-list@python.org>
Message-ID: <snog1a$u4v$1@news2.informatik.uni-stuttgart.de>

Chris Angelico <rosuav at gmail.com> wrote:

> Unfortunately, if you're not going to go to the effort of getting your
> executables signed

I cannot sign my executables (how can I do it anyway?), because Windows
deletes my executable as soon as I have compiled them! They exist only
for a few seconds and then they are gone.


> another reason to just distribute .py files.

I cannot do that because my users do not have Python installed and they
are not allowed to do it.

-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher at tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/

From rosuav at gmail.com  Thu Nov 25 12:21:54 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 26 Nov 2021 04:21:54 +1100
Subject: pyinstaller wrong classified as Windows virus
In-Reply-To: <snog1a$u4v$1@news2.informatik.uni-stuttgart.de>
References: <snnkhf$mve$1@news2.informatik.uni-stuttgart.de>
 <slrnspur9f.51o.dan@djph.net> <sno092$prd$1@news2.informatik.uni-stuttgart.de>
 <sno3qt$r1b$1@news2.informatik.uni-stuttgart.de>
 <CAPTjJmq=w3iFDWjzNFAfpmAC=5kFZm1C8=Ex7Cwo2iwDVY_LHA@mail.gmail.com>
 <mailman.8.1637859246.9869.python-list@python.org>
 <snog1a$u4v$1@news2.informatik.uni-stuttgart.de>
Message-ID: <CAPTjJmrdM=puoFg7aPEtbRi73NYN+6WHWb=FKHvqwK7pUcx+FQ@mail.gmail.com>

On Fri, Nov 26, 2021 at 4:18 AM Ulli Horlacher
<framstag at rus.uni-stuttgart.de> wrote:
>
> Chris Angelico <rosuav at gmail.com> wrote:
>
> > Unfortunately, if you're not going to go to the effort of getting your
> > executables signed
>
> I cannot sign my executables (how can I do it anyway?), because Windows
> deletes my executable as soon as I have compiled them! They exist only
> for a few seconds and then they are gone.
>
>
> > another reason to just distribute .py files.
>
> I cannot do that because my users do not have Python installed and they
> are not allowed to do it.
>

Are they really allowed to install your unsigned executables but are
not allowed to install Python from a known and trusted source?

If there's some bizarre loophole that allows them to run completely
untrusted binary code, but not to run legitimate code that can be
fetched from a variety of trusted places (including python.org, the
Windows store, etc), then I'm afraid you're on your own, and will
probably need to play around with the exact loophole to figure out
what is going to be permitted.

Alternatively, just go find the person who decides what gets
installed, and request a Python interpreter to be added to the
permitted list. That's probably easier, and it's certainly going to be
better long-term.

ChrisA

From barry at barrys-emacs.org  Thu Nov 25 12:42:32 2021
From: barry at barrys-emacs.org (Barry)
Date: Thu, 25 Nov 2021 17:42:32 +0000
Subject: pyinstaller wrong classified as Windows virus
In-Reply-To: <snocdh$t49$1@news2.informatik.uni-stuttgart.de>
References: <snocdh$t49$1@news2.informatik.uni-stuttgart.de>
Message-ID: <A49FB449-C45E-4A45-A017-ECF0A5387B6C@barrys-emacs.org>



> On 25 Nov 2021, at 16:51, Ulli Horlacher <framstag at rus.uni-stuttgart.de> wrote:
> 
> ?Barry Scott <barry at barrys-emacs.org> wrote:
>> 
>> 
>>>> On 25 Nov 2021, at 09:20, Ulli Horlacher <framstag at rus.uni-stuttgart.de> wrote:
>>> 
>>> When I compile my programs with pyinstaller, Windows classifies them as
>>> virus and even deletes them!
>> 
>> Microsoft will fix the malware detection if you provide the info they need.
>> 
>> Submit false positive info to: https://www.microsoft.com/security/portal/submission/submit.aspx <https://www.microsoft.com/security/portal/submission/submit.aspx>
> 
> I cannot submit my executables, because the Windows Virus scannners
> deletes them as soon as I compile my program!

You should be able to tell the software to quarantine instead of delete.
Or even disable while you build one instance.
I am assuming you have admin control.

> And I need a Microsoft login to submit a file!
> I do not have such a login.
It?s free to get an account. How badly do you want to fix this?

> 
> 
> -- 
> Ullrich Horlacher              Server und Virtualisierung
> Rechenzentrum TIK         
> Universitaet Stuttgart         E-Mail: horlacher at tik.uni-stuttgart.de
> Allmandring 30a                Tel:    ++49-711-68565868
> 70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 


From Richard at Damon-Family.org  Thu Nov 25 12:48:47 2021
From: Richard at Damon-Family.org (Richard Damon)
Date: Thu, 25 Nov 2021 12:48:47 -0500
Subject: pyinstaller wrong classified as Windows virus
In-Reply-To: <CAPTjJmrdM=puoFg7aPEtbRi73NYN+6WHWb=FKHvqwK7pUcx+FQ@mail.gmail.com>
References: <snnkhf$mve$1@news2.informatik.uni-stuttgart.de>
 <slrnspur9f.51o.dan@djph.net>
 <sno092$prd$1@news2.informatik.uni-stuttgart.de>
 <sno3qt$r1b$1@news2.informatik.uni-stuttgart.de>
 <CAPTjJmq=w3iFDWjzNFAfpmAC=5kFZm1C8=Ex7Cwo2iwDVY_LHA@mail.gmail.com>
 <mailman.8.1637859246.9869.python-list@python.org>
 <snog1a$u4v$1@news2.informatik.uni-stuttgart.de>
 <CAPTjJmrdM=puoFg7aPEtbRi73NYN+6WHWb=FKHvqwK7pUcx+FQ@mail.gmail.com>
Message-ID: <cd2599b2-9428-26ee-7de9-9ecafc2adff1@Damon-Family.org>

On 11/25/21 12:21 PM, Chris Angelico wrote:
> On Fri, Nov 26, 2021 at 4:18 AM Ulli Horlacher
> <framstag at rus.uni-stuttgart.de> wrote:
>> Chris Angelico <rosuav at gmail.com> wrote:
>>
>>> Unfortunately, if you're not going to go to the effort of getting your
>>> executables signed
>> I cannot sign my executables (how can I do it anyway?), because Windows
>> deletes my executable as soon as I have compiled them! They exist only
>> for a few seconds and then they are gone.
>>
>>
>>> another reason to just distribute .py files.
>> I cannot do that because my users do not have Python installed and they
>> are not allowed to do it.
>>
> Are they really allowed to install your unsigned executables but are
> not allowed to install Python from a known and trusted source?
>
> If there's some bizarre loophole that allows them to run completely
> untrusted binary code, but not to run legitimate code that can be
> fetched from a variety of trusted places (including python.org, the
> Windows store, etc), then I'm afraid you're on your own, and will
> probably need to play around with the exact loophole to figure out
> what is going to be permitted.
>
> Alternatively, just go find the person who decides what gets
> installed, and request a Python interpreter to be added to the
> permitted list. That's probably easier, and it's certainly going to be
> better long-term.
>
> ChrisA

My first guess is it isn't so much what is 'allowed' but what can be 
easily done.

On a somewhat locked down computer, the user does not have admin rights, 
so needs to get 'IT' to run any installers that need admin permissions 
to run.

And EXE that just needs to be copied to the computer and rhen just RUN, 
doesn't need IT to 'install' it (they just can't put it into Program 
Files, but that isn't really that important for programs that don't need 
an installer.

Likely, just copying an EXE file from an outside source may still be 
against the rules (and needs approval), but some think if they can do it 
and no one complains, it must be ok. On the other hand, they may have 
given approval, knowing the source.

-- 
Richard Damon


From rosuav at gmail.com  Thu Nov 25 13:00:49 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 26 Nov 2021 05:00:49 +1100
Subject: pyinstaller wrong classified as Windows virus
In-Reply-To: <cd2599b2-9428-26ee-7de9-9ecafc2adff1@Damon-Family.org>
References: <snnkhf$mve$1@news2.informatik.uni-stuttgart.de>
 <slrnspur9f.51o.dan@djph.net> <sno092$prd$1@news2.informatik.uni-stuttgart.de>
 <sno3qt$r1b$1@news2.informatik.uni-stuttgart.de>
 <CAPTjJmq=w3iFDWjzNFAfpmAC=5kFZm1C8=Ex7Cwo2iwDVY_LHA@mail.gmail.com>
 <mailman.8.1637859246.9869.python-list@python.org>
 <snog1a$u4v$1@news2.informatik.uni-stuttgart.de>
 <CAPTjJmrdM=puoFg7aPEtbRi73NYN+6WHWb=FKHvqwK7pUcx+FQ@mail.gmail.com>
 <cd2599b2-9428-26ee-7de9-9ecafc2adff1@Damon-Family.org>
Message-ID: <CAPTjJmpCUR7hO4pSZa2ed1iXR6-=VQEQzjMiT0pa8t0JMf94ow@mail.gmail.com>

On Fri, Nov 26, 2021 at 4:50 AM Richard Damon <Richard at damon-family.org> wrote:
>
> On 11/25/21 12:21 PM, Chris Angelico wrote:
> > On Fri, Nov 26, 2021 at 4:18 AM Ulli Horlacher
> > <framstag at rus.uni-stuttgart.de> wrote:
> >> Chris Angelico <rosuav at gmail.com> wrote:
> >>
> >>> Unfortunately, if you're not going to go to the effort of getting your
> >>> executables signed
> >> I cannot sign my executables (how can I do it anyway?), because Windows
> >> deletes my executable as soon as I have compiled them! They exist only
> >> for a few seconds and then they are gone.
> >>
> >>
> >>> another reason to just distribute .py files.
> >> I cannot do that because my users do not have Python installed and they
> >> are not allowed to do it.
> >>
> > Are they really allowed to install your unsigned executables but are
> > not allowed to install Python from a known and trusted source?
> >
> > If there's some bizarre loophole that allows them to run completely
> > untrusted binary code, but not to run legitimate code that can be
> > fetched from a variety of trusted places (including python.org, the
> > Windows store, etc), then I'm afraid you're on your own, and will
> > probably need to play around with the exact loophole to figure out
> > what is going to be permitted.
> >
> > Alternatively, just go find the person who decides what gets
> > installed, and request a Python interpreter to be added to the
> > permitted list. That's probably easier, and it's certainly going to be
> > better long-term.
> >
> > ChrisA
>
> My first guess is it isn't so much what is 'allowed' but what can be
> easily done.
>
> On a somewhat locked down computer, the user does not have admin rights,
> so needs to get 'IT' to run any installers that need admin permissions
> to run.

Can someone confirm that it's still possible to run the Python
installer without admin rights, for a per-user installation? It always
used to be possible, but I haven't checked.

> Likely, just copying an EXE file from an outside source may still be
> against the rules (and needs approval), but some think if they can do it
> and no one complains, it must be ok. On the other hand, they may have
> given approval, knowing the source.

Maybe. I would still consider it unlikely that you can run an EXE from
an arbitrary source, but can't run a trusted installer from a known
source. You're right that admin perms would be harder, but that
shouldn't stop you from installing Python.

Also, obligatory XKCD: https://xkcd.com/1200/

ChrisA

From torriem at gmail.com  Thu Nov 25 15:36:00 2021
From: torriem at gmail.com (Michael Torrie)
Date: Thu, 25 Nov 2021 13:36:00 -0700
Subject: pyinstaller wrong classified as Windows virus
In-Reply-To: <snocdh$t49$1@news2.informatik.uni-stuttgart.de>
References: <snnkhf$mve$1@news2.informatik.uni-stuttgart.de>
 <253A025C-CA28-44FF-BB2D-5CA286D01012@barrys-emacs.org>
 <mailman.6.1637851110.9869.python-list@python.org>
 <snocdh$t49$1@news2.informatik.uni-stuttgart.de>
Message-ID: <8d93f53d-89c6-78c7-da59-fed607a8ea36@gmail.com>

On 11/25/21 9:08 AM, Ulli Horlacher wrote:
> I cannot submit my executables, because the Windows Virus scannners
> deletes them as soon as I compile my program!

Add an exclusion rule to your machine. While this is not an option for
your end users, this will certainly allow you to work on the problem,
submitting the exe to the various virus vendors.

> And I need a Microsoft login to submit a file!
> I do not have such a login.

I sympathize.  But if you want to develop for Windows, you might just
have to get one.


From torriem at gmail.com  Thu Nov 25 15:36:25 2021
From: torriem at gmail.com (Michael Torrie)
Date: Thu, 25 Nov 2021 13:36:25 -0700
Subject: pyinstaller wrong classified as Windows virus
In-Reply-To: <snocdh$t49$1@news2.informatik.uni-stuttgart.de>
References: <snnkhf$mve$1@news2.informatik.uni-stuttgart.de>
 <253A025C-CA28-44FF-BB2D-5CA286D01012@barrys-emacs.org>
 <mailman.6.1637851110.9869.python-list@python.org>
 <snocdh$t49$1@news2.informatik.uni-stuttgart.de>
Message-ID: <959abf3f-cfca-6fa6-b114-b78906f59207@gmail.com>

On 11/25/21 9:08 AM, Ulli Horlacher wrote:

> I cannot submit my executables, because the Windows Virus scannners
> deletes them as soon as I compile my program!

I forgot to post this link:
https://support.microsoft.com/en-us/windows/add-an-exclusion-to-windows-security-811816c0-4dfd-af4a-47e4-c301afe13b26

From mats at wichmann.us  Thu Nov 25 15:52:06 2021
From: mats at wichmann.us (Mats Wichmann)
Date: Thu, 25 Nov 2021 13:52:06 -0700
Subject: pyinstaller wrong classified as Windows virus
In-Reply-To: <CAPTjJmpCUR7hO4pSZa2ed1iXR6-=VQEQzjMiT0pa8t0JMf94ow@mail.gmail.com>
References: <snnkhf$mve$1@news2.informatik.uni-stuttgart.de>
 <slrnspur9f.51o.dan@djph.net>
 <sno092$prd$1@news2.informatik.uni-stuttgart.de>
 <sno3qt$r1b$1@news2.informatik.uni-stuttgart.de>
 <CAPTjJmq=w3iFDWjzNFAfpmAC=5kFZm1C8=Ex7Cwo2iwDVY_LHA@mail.gmail.com>
 <mailman.8.1637859246.9869.python-list@python.org>
 <snog1a$u4v$1@news2.informatik.uni-stuttgart.de>
 <CAPTjJmrdM=puoFg7aPEtbRi73NYN+6WHWb=FKHvqwK7pUcx+FQ@mail.gmail.com>
 <cd2599b2-9428-26ee-7de9-9ecafc2adff1@Damon-Family.org>
 <CAPTjJmpCUR7hO4pSZa2ed1iXR6-=VQEQzjMiT0pa8t0JMf94ow@mail.gmail.com>
Message-ID: <dc8cf499-749d-56a0-9262-d55ae3bb6bcb@wichmann.us>

On 11/25/21 11:00, Chris Angelico wrote:

> Can someone confirm that it's still possible to run the Python
> installer without admin rights, for a per-user installation? It always
> used to be possible, but I haven't checked.

You only need admin rights for some special cases.  While Win7 was still 
supported, you needed to run the (included) installer for vcredist of a 
different version than the one that comes with Win7, and that needed 
admin rights (unfortunately, the Python installer didn't prompt for that 
and so the vcredist install failed silently, leaving a broken install if 
you didn't already have it from other means - but that's all in the past 
now).  It's possible you also need it for the Python Launcher, since 
that goes into a "system location" (not sure about that one).  And if 
you asked for an install for "all users" that requires admin rights.

From rosuav at gmail.com  Thu Nov 25 16:08:10 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 26 Nov 2021 08:08:10 +1100
Subject: pyinstaller wrong classified as Windows virus
In-Reply-To: <dc8cf499-749d-56a0-9262-d55ae3bb6bcb@wichmann.us>
References: <snnkhf$mve$1@news2.informatik.uni-stuttgart.de>
 <slrnspur9f.51o.dan@djph.net> <sno092$prd$1@news2.informatik.uni-stuttgart.de>
 <sno3qt$r1b$1@news2.informatik.uni-stuttgart.de>
 <CAPTjJmq=w3iFDWjzNFAfpmAC=5kFZm1C8=Ex7Cwo2iwDVY_LHA@mail.gmail.com>
 <mailman.8.1637859246.9869.python-list@python.org>
 <snog1a$u4v$1@news2.informatik.uni-stuttgart.de>
 <CAPTjJmrdM=puoFg7aPEtbRi73NYN+6WHWb=FKHvqwK7pUcx+FQ@mail.gmail.com>
 <cd2599b2-9428-26ee-7de9-9ecafc2adff1@Damon-Family.org>
 <CAPTjJmpCUR7hO4pSZa2ed1iXR6-=VQEQzjMiT0pa8t0JMf94ow@mail.gmail.com>
 <dc8cf499-749d-56a0-9262-d55ae3bb6bcb@wichmann.us>
Message-ID: <CAPTjJmrLdkuTeF3atqp0qoAff6i=A6XJiV-V8G5xBcuCR_ZN6g@mail.gmail.com>

On Fri, Nov 26, 2021 at 7:53 AM Mats Wichmann <mats at wichmann.us> wrote:
>
> On 11/25/21 11:00, Chris Angelico wrote:
>
> > Can someone confirm that it's still possible to run the Python
> > installer without admin rights, for a per-user installation? It always
> > used to be possible, but I haven't checked.
>
> You only need admin rights for some special cases.  While Win7 was still
> supported, you needed to run the (included) installer for vcredist of a
> different version than the one that comes with Win7, and that needed
> admin rights (unfortunately, the Python installer didn't prompt for that
> and so the vcredist install failed silently, leaving a broken install if
> you didn't already have it from other means - but that's all in the past
> now).  It's possible you also need it for the Python Launcher, since
> that goes into a "system location" (not sure about that one).  And if
> you asked for an install for "all users" that requires admin rights.

That's what I thought, yeah. So even on a system you don't own, where
you're not allowed to get admin privileges, it should still be
possible to install Python just fine.

Would be worth testing (if someone has a fresh Windows system around)
to see if it's possible to set up the "double click on .py file to run
it" association without admin privileges. That's the only part that
might be a limitation.

ChrisA

From jenkris at tutanota.com  Thu Nov 25 17:29:07 2021
From: jenkris at tutanota.com (jenkris at tutanota.com)
Date: Thu, 25 Nov 2021 23:29:07 +0100 (CET)
Subject: Eventfd with epoll BlockingIOError
In-Reply-To: <A48057B9-1031-4DC8-9FC9-E59D2DA14BB4@barrys-emacs.org>
References: <MpJ6MZm--B-2@tutanota.com>
 <A48057B9-1031-4DC8-9FC9-E59D2DA14BB4@barrys-emacs.org>
Message-ID: <MpOCrN6--3-2@tutanota.com>

Thanks very much for your reply.? 

I am now getting a single event returned in Python, but it's not the right event, as I'll explain below.?

I rearranged the Python code based on your comments:

#!/usr/bin/python3
import sys
import os
import select

print("Inside Python")

event_fd = int(sys.argv[3])

print("Eventfd received by Python")
print(event_fd)

event_write_value = 100

ep = select.epoll(-1)
ep.register(event_fd, select.EPOLLIN | select.EPOLLOUT )

os.set_blocking(event_fd, False)

#__________

print("Starting poll loop")

for fd_event in ep.poll():
??? print("Python fd_event")
??? print(fd_event)
??? fd_received = fd_event[0]
??? event_received = fd_event[1]

You advised to leave off select.EPOLLOUT from the line ep.register(event_fd, select.EPOLLIN | select.EPOLLOUT ) -- which makes sense because I'm not waiting for that event -- but without it both processes freeze in the for loop (below print("Starting poll loop")) so we never receive an EPOLLIN event.? So I included it, and here is the screen output from gdb:

Inside Python
Eventfd received by Python
5
Everything OK in Python
Starting poll loop
Python fd_event
(5, 4)
Writing to Python
5 Received from Python
8 Writing to Python
Failed epoll_wait Bad file descriptor
5 Received from Python
8 Writing to Python
Failed epoll_wait Bad file descriptor
5 Received from Python
-1time taken 0.000629
Failed to close epoll file descriptor
Unlink_shm status: Bad file descriptor
fn() took 0.000717 seconds to execute
[Inferior 1 (process 26718) exited normally]
(gdb) q

The Python fd_event tuple is 5, 4 -- 5 is the correct file descriptor and 4 is an EPOLLOUT event, which is not what I want.?

The eventfd is created in C as nonblocking:

int eventfd_initialize() {
? int efd = eventfd(0, EFD_NONBLOCK);
? return efd; }

When C writes it calls epoll_wait:

ssize_t epoll_write(int event_fd, int epoll_fd, struct epoll_event * event_struc, int action_code)
{
?? int64_t ewbuf[1];
?? ewbuf[0] = (int64_t)action_code;
?? int maxevents = 1;
?? int timeout = -1;

?? fprintf(stdout, " Writing to Python \n%d", event_fd);

??? write(event_fd, &ewbuf, 8);

??? if (epoll_wait(epoll_fd, event_struc, maxevents, timeout) == -1)
??? {
??????? fprintf(stderr, "Failed epoll_wait %s\n", strerror(errno));
??? }

??? ssize_t rdval = read(event_fd, &ewbuf, 8);???

??? fprintf(stdout, " Received from Python \n%ld", rdval);

??? return 0;
}

The C side initializes its epoll this way:

int epoll_initialize(int efd, int64_t * output_array)
{
? struct epoll_event ev = {};
? int epoll_fd = epoll_create1(0);

? struct epoll_event * ptr_ev = &ev;
??
? if(epoll_fd == -1)
? {
??? fprintf(stderr, "Failed to create epoll file descriptor\n");
??? return 1;
? }

? ev.events = EPOLLIN | EPOLLOUT;
? ev.data.fd = efd; //was 0

? if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, efd, &ev) == -1)
? {
????? fprintf(stderr, "Failed to add file descriptor to epoll\n");
????? close(epoll_fd);
????? return 1;
? }

? output_array[0] = epoll_fd;
? output_array[1] = (int64_t)ptr_ev; //&ev;

? return 0;
}

Technically C is not waiting for an EPOLLIN event, but again without it both processes freeze unless either C or Python includes both events.? So that appears to be where the problem is.?

The Linux epoll man page says, "epoll_wait waits for I/O events, blocking the calling thread if no events are currently available."?? https://man7.org/linux/man-pages/man7/epoll.7.html.? That may be the clue to why both processes freeze when I poll on only one event in each one.?

Thanks for any ideas based on this update, and thanks again for your earlier reply.?

Jen


-- 
 Sent with Tutanota, the secure & ad-free mailbox. 



Nov 25, 2021, 06:34 by barry at barrys-emacs.org:

>
>
>
>> On 24 Nov 2021, at 22:42, Jen via Python-list <>> python-list at python.org>> > wrote:
>>
>> I have a C program that uses fork-execv to run Python 3.10 in a child process, and I am using eventfd with epoll for IPC between them.? The eventfd file descriptor is created in C and passed to Python through execv.? Once the Python child process starts I print the file descriptor to verify that it is correct (it is).? 
>>
>> In this scenario C will write to the eventfd at intervals and Python will read the eventfd and take action based on the value in the eventfd.? But in the Python while True loop I get "BlockingIOError: [Errno 11] Resource temporarily unavailable" then with each new read it prints "Failed epoll_wait Bad file descriptor."?
>>
>> This is the Python code:
>>
>> #!/usr/bin/python3
>> import sys
>> import os
>> import select
>>
>> print("Inside Python")
>>
>> event_fd = int(sys.argv[3])
>>
>>
>> print("Eventfd received by Python")
>> print(event_fd)
>>
>> ep = select.epoll(-1)
>> ep.register(event_fd, select.EPOLLIN | select.EPOLLOUT)
>>
>
> This says tell me if I can read or write to the event_fd.
> write will be allowed until the kernel buffers are full.
>
> Usually you only add EPOLLOUT if you have data to write.
> In this case do not set EPOLLOUT.
>
> And if you know that you will never fill the kernel buffers then you
> do not need to bother polling for write.
>
>
>>
>> event_write_value = 100
>>
>> while True:
>>
>> ??? print("Waiting in Python for event")
>> ??? ep.poll(timeout=None, maxevents=- 1)
>>
>
> You have to get the result of the poll() and process the list of entries that are returned.
>
> You must check that POLLIN is set before attempting the read.
>
>
>
>> ? ? v = os.eventfd_read(event_fd)
>>
>
> Will raise EWOULDBLOCK because there is no data available to read.
>
> Here is the docs from python:
>
> poll.> poll> (> [> timeout> ]> )
>
> Polls the set of registered file descriptors, and returns a possibly-empty listcontaining > (fd,>  > event)>  2-tuples for the descriptors that have events orerrors to report. > fd>  is the file descriptor, and > event>  is a bitmask withbits set for the reported events for that descriptor ? > POLLIN>  forwaiting input, > POLLOUT>  to indicate that the descriptor can be writtento, and so forth. An empty list indicates that the call timed out and no filedescriptors had any events to report. If > timeout>  is given, it specifies thelength of time in milliseconds which the system will wait for events beforereturning. If > timeout>  is omitted, negative, or > None <>> , the call willblock until there is an event for this poll object.
>
> You end up with code like this:
>
> for fd_event in ep.poll():
> fd, event == fd_event
> if (event&select.POLLIN) != 0 and fd == event_fd:
> v = os.eventfd_read(event_fd)
>
>
>>
>> ??? if v != 99:
>> ??????? print("found")
>> ??????? print(v)
>>
>> ??????? os.eventfd_write(event_fd, event_write_value)
>>
>> ??? if v == 99:
>> ??????? os.close(event_fd)
>>
>> This is the C code that writes to Python, then waits for Python to write back:
>>
>> ssize_t epoll_write(int event_fd, int epoll_fd, struct epoll_event * event_struc, int action_code)
>> {
>> ?? int64_t ewbuf[1];
>> ?? ewbuf[0] = (int64_t)action_code;
>> ?? int maxevents = 1;
>> ?? int timeout = -1;
>>
>> ?? fprintf(stdout, " Writing to Python \n%d", event_fd);
>>
>> ?? write(event_fd, &ewbuf, 8);
>>
>> ??? if (epoll_wait(epoll_fd, event_struc, maxevents, timeout) == -1)
>> ??? {
>> ??????? fprintf(stderr, "Failed epoll_wait %s\n", strerror(errno));
>> ??? }
>>
>> ??? ssize_t rdval = read(event_fd, &ewbuf, 8);???
>>
>> ??? fprintf(stdout, " Received from Python \n%ld", rdval);
>>
>> ??? return 0;
>> }
>>
>> This is the screen output when I run with gdb:
>>
>> ????????? Inside Python
>> Eventfd received by Python
>> 5
>> Waiting in Python for event
>> Traceback (most recent call last):
>> ? File "/usr/local/lib/python3.10/runpy.py", line 196, in >>  >> _run_module_as_main
>> ??? return _run_code(code, main_globals, None,
>> ? File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
>> ??? exec(code, run_globals)
>> ? File "/opt/P01_SH/NPC_CPython.py", line 36, in <module>
>> ??? v = os.eventfd_read(event_fd)
>> BlockingIOError: [Errno 11] Resource temporarily unavailable
>>
>
> Expected as there you have not checked that there is data to read.
> Check for POLLIN being set.
>
>
>> Writing to Python
>> 5 Received from Python
>> 8 Writing to Python
>> Failed epoll_wait Bad file descriptor
>> 5 Received from Python
>> 8 Writing to Python
>> Failed epoll_wait Bad file descriptor
>> 5 Received from Python
>> -1time taken 0.000548
>> Failed to close epoll file descriptor
>> Unlink_shm status: Bad file descriptor
>> fn() took 0.000648 seconds to execute
>> [Inferior 1 (process 12618) exited normally]
>> (gdb)
>>
>> So my question is why do I get "BlockingIOError: [Errno 11] Resource temporarily unavailable" and "Failed epoll_wait Bad file descriptor" from Python??
>>
>
> If your protocol is not trivia you should implement a state machine to know what to do at each event.
>
> Barry
>
>
>>
>> -- 
>>  Sent with Tutanota, the secure & ad-free mailbox. 
>> -- 
>> https://mail.python.org/mailman/listinfo/python-list
>>


From avigross at verizon.net  Thu Nov 25 18:17:12 2021
From: avigross at verizon.net (Avi Gross)
Date: Thu, 25 Nov 2021 18:17:12 -0500
Subject: pyinstaller wrong classified as Windows virus
In-Reply-To: <snog1a$u4v$1@news2.informatik.uni-stuttgart.de>
References: <snnkhf$mve$1@news2.informatik.uni-stuttgart.de>
 <slrnspur9f.51o.dan@djph.net>
 <sno092$prd$1@news2.informatik.uni-stuttgart.de>
 <sno3qt$r1b$1@news2.informatik.uni-stuttgart.de>
 <CAPTjJmq=w3iFDWjzNFAfpmAC=5kFZm1C8=Ex7Cwo2iwDVY_LHA@mail.gmail.com>
 <mailman.8.1637859246.9869.python-list@python.org>
 <snog1a$u4v$1@news2.informatik.uni-stuttgart.de>
Message-ID: <007701d7e252$9402b500$bc081f00$@verizon.net>

I am not sure what your real problem is, Ulli, but many antivirus programs
can be TEMPORARILY shut off. Not highly recommended, of course, but if you
properly disable it on a newly rebooted system running little, and it still
happens, then something else may be going on.

If one recognizes your code a potentially having a virus, it may be for an
assortment of reasons such as a table it contains to look at position N in
the executable for an exact match with some bit-string. If so, one potential
fix is a slight change in the code that compiles a bit differently like
x=sin(30) or other filler. 

But consider another possibility that your compiler software is compromised
and actually placing something into everything it compiles. Is this
happening to only one set of code?

You can think of other similar experiments based on your setup that may help
you debug and perhaps your problem is something else entirely.


-----Original Message-----
From: Python-list <python-list-bounces+avigross=verizon.net at python.org> On
Behalf Of Ulli Horlacher
Sent: Thursday, November 25, 2021 12:10 PM
To: python-list at python.org
Subject: Re: pyinstaller wrong classified as Windows virus

Chris Angelico <rosuav at gmail.com> wrote:

> Unfortunately, if you're not going to go to the effort of getting your 
> executables signed

I cannot sign my executables (how can I do it anyway?), because Windows
deletes my executable as soon as I have compiled them! They exist only for a
few seconds and then they are gone.


> another reason to just distribute .py files.

I cannot do that because my users do not have Python installed and they are
not allowed to do it.

-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher at tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
--
https://mail.python.org/mailman/listinfo/python-list


From frank at chagford.com  Fri Nov 26 04:17:18 2021
From: frank at chagford.com (Frank Millman)
Date: Fri, 26 Nov 2021 11:17:18 +0200
Subject: Negative subscripts
Message-ID: <96741f1a-72b3-6551-39b3-f1862c5a80b8@chagford.com>

Hi all

In my program I have a for-loop like this -

 >>> for item in x[:-y]:
...??? [do stuff]

'y' may or may not be 0. If it is 0 I want to process the entire list 
'x', but of course -0 equals 0, so it returns an empty list.

In theory I can say

 >>> for item in x[:-y] if y else x:
...??? [do stuff]

But in my actual program, both x and y are fairly long expressions, so 
the result is pretty ugly.

Are there any other techniques anyone can suggest, or is the only 
alternative to use if...then...else to cater for y = 0?

Thanks

Frank Millman

From antoon.pardon at vub.be  Fri Nov 26 04:49:40 2021
From: antoon.pardon at vub.be (Antoon Pardon)
Date: Fri, 26 Nov 2021 10:49:40 +0100
Subject: Negative subscripts
In-Reply-To: <96741f1a-72b3-6551-39b3-f1862c5a80b8@chagford.com>
References: <96741f1a-72b3-6551-39b3-f1862c5a80b8@chagford.com>
Message-ID: <f0bfafe6-e1db-14f2-8ded-c59f47367633@vub.be>

Op 26/11/2021 om 10:17 schreef Frank Millman:
> Hi all
>
> In my program I have a for-loop like this -
>
> >>> for item in x[:-y]:
> ...??? [do stuff]
>
> 'y' may or may not be 0. If it is 0 I want to process the entire list 
> 'x', but of course -0 equals 0, so it returns an empty list.
>
> In theory I can say
>
> >>> for item in x[:-y] if y else x:
> ...??? [do stuff]
>
> But in my actual program, both x and y are fairly long expressions, so 
> the result is pretty ugly.
>
> Are there any other techniques anyone can suggest, or is the only 
> alternative to use if...then...else to cater for y = 0?

It depends. Since you are talking about x being a fairly long expression,
how about adapting that expression so that is produces the reverse of
what is now produced and then use the [y:] slice?

You can also adapt the y expression so that instead of y it produces
len(x) - y.

You may consider writing a reverse_view function, that takes a list
as argument and produces something that behaves as the reversed list
without actually reversing the list and then as in the first option
use the [y:] slice on that.

-- 
Antoon Pardon.

From rob.cliffe at btinternet.com  Fri Nov 26 06:09:03 2021
From: rob.cliffe at btinternet.com (Rob Cliffe)
Date: Fri, 26 Nov 2021 11:09:03 +0000
Subject: Negative subscripts
In-Reply-To: <96741f1a-72b3-6551-39b3-f1862c5a80b8@chagford.com>
References: <96741f1a-72b3-6551-39b3-f1862c5a80b8@chagford.com>
Message-ID: <ee198109-7f76-f987-8f41-7f18080de51e@btinternet.com>

You could evaluate y separately:

yval = <long expression>
for item in x[:-yval] if yval else x:
 ??? [do stuff]

or you could do it using the walrus operator:

for item in x[:-yval] if (yval := <long expression>) else x:
 ??? [do stuff]

or, perhaps simplest, you could do

for item in x[:-y or None]: # a value of None for a slice argument means 
"don't slice here"
 ??? [do stuff]

Best wishes
Rob Cliffe

On 26/11/2021 09:17, Frank Millman wrote:
> Hi all
>
> In my program I have a for-loop like this -
>
> >>> for item in x[:-y]:
> ...??? [do stuff]
>
> 'y' may or may not be 0. If it is 0 I want to process the entire list 
> 'x', but of course -0 equals 0, so it returns an empty list.
>
> In theory I can say
>
> >>> for item in x[:-y] if y else x:
> ...??? [do stuff]
>
> But in my actual program, both x and y are fairly long expressions, so 
> the result is pretty ugly.
>
> Are there any other techniques anyone can suggest, or is the only 
> alternative to use if...then...else to cater for y = 0?
>
> Thanks
>
> Frank Millman


From rosuav at gmail.com  Fri Nov 26 06:24:16 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 26 Nov 2021 22:24:16 +1100
Subject: Negative subscripts
In-Reply-To: <ee198109-7f76-f987-8f41-7f18080de51e@btinternet.com>
References: <96741f1a-72b3-6551-39b3-f1862c5a80b8@chagford.com>
 <ee198109-7f76-f987-8f41-7f18080de51e@btinternet.com>
Message-ID: <CAPTjJmp6DCt230qTyoGtZkv9ofHoq0NUCZWz9s8Esdsj=O4VUQ@mail.gmail.com>

On Fri, Nov 26, 2021 at 10:11 PM Rob Cliffe via Python-list
<python-list at python.org> wrote:
> or, perhaps simplest, you could do
>
> for item in x[:-y or None]: # a value of None for a slice argument means
> "don't slice here"
>      [do stuff]
>

This is the one I'd recommend. If you're negating a slice like this,
just add "or None" to handle negative zero as the other end of the
list.

Be aware that this will still slice even if you're going to use the
whole list, unlike the "x[-y] if y else x" approach, which will
iterate over the original list. That might be important.

ChrisA

From frank at chagford.com  Fri Nov 26 08:40:09 2021
From: frank at chagford.com (Frank Millman)
Date: Fri, 26 Nov 2021 15:40:09 +0200
Subject: Negative subscripts
In-Reply-To: <96741f1a-72b3-6551-39b3-f1862c5a80b8@chagford.com>
References: <96741f1a-72b3-6551-39b3-f1862c5a80b8@chagford.com>
Message-ID: <1ed3404c-3eaf-bbdc-56a1-5b4494f495bb@chagford.com>

On 2021-11-26 11:17 AM, Frank Millman wrote:
> Hi all
> 
> In my program I have a for-loop like this -
> 
>  >>> for item in x[:-y]:
> ...??? [do stuff]
> 
> 'y' may or may not be 0. If it is 0 I want to process the entire list 
> 'x', but of course -0 equals 0, so it returns an empty list.
> 
> In theory I can say
> 
>  >>> for item in x[:-y] if y else x:
> ...??? [do stuff]
> 
> But in my actual program, both x and y are fairly long expressions, so 
> the result is pretty ugly.
> 
> Are there any other techniques anyone can suggest, or is the only 
> alternative to use if...then...else to cater for y = 0?
> 

Thanks for all the replies. A selection of neat ideas for me to choose from.

Much appreciated.

Frank


From PythonList at DancesWithMice.info  Fri Nov 26 16:24:23 2021
From: PythonList at DancesWithMice.info (dn)
Date: Sat, 27 Nov 2021 10:24:23 +1300
Subject: Negative subscripts
In-Reply-To: <96741f1a-72b3-6551-39b3-f1862c5a80b8@chagford.com>
References: <96741f1a-72b3-6551-39b3-f1862c5a80b8@chagford.com>
Message-ID: <ba4a976c-5829-03ac-2e3f-cf670f00c3f7@DancesWithMice.info>

On 26/11/2021 22.17, Frank Millman wrote:
> In my program I have a for-loop like this -
> 
>>>> for item in x[:-y]:
> ...??? [do stuff]
> 
> 'y' may or may not be 0. If it is 0 I want to process the entire list
> 'x', but of course -0 equals 0, so it returns an empty list.
...

> But in my actual program, both x and y are fairly long expressions, so
> the result is pretty ugly.
> 
> Are there any other techniques anyone can suggest, or is the only
> alternative to use if...then...else to cater for y = 0?


Here's hoping that this good-looking boy has not missed the point about
"ugly".

Is this the problem, re-stated to be fully explicit, with (almost)
zero-complexity?  [Zen of Python]

>>> x = [ "a", "b", "c", "d", "e" ]
>>> for y in [ 1, 2, 3, 4, 5 ]:
...     print( y, x[ :-y ] )
...
1 ['a', 'b', 'c', 'd']
2 ['a', 'b', 'c']
3 ['a', 'b']
4 ['a']
5 []

Whilst the code 'works', it lacks the 'zero-case' of what is not to be
included - which is an overly-complicated way of saying: misses the
'all-case'.

As observed, if we add a zero to the slice/loop-control that fails
logically (and we get more religion: "the first shall be last, and the
last shall be first". [Christian Bible])

>>> for y in [ 0, 1, 2, 3, 4, 5 ]:
...     print( y, x[ :-y ] )
...
0 []
1 ['a', 'b', 'c', 'd']
2 ['a', 'b', 'c']
3 ['a', 'b']
4 ['a']
5 []

The problem, as expressed, is that whilst there is a negative of 1, ie
-1, and other positive-integers, there is no (material) negative of 0,
ie 0 == -0 == +0.


(if the following seems pedantic and 'talking down', it is because I'm
filleting a tutorial prepared for Python-Apprentices)


Remember that 'salvation' may lie in the somewhat mysterious rules of
slicing's default values. According to the Python rule-book (not exactly
a religious text, but we must tend it with some fervor... [Built-in
Types] https://docs.python.org/3/library/stdtypes.html?highlight=slice)

?
s[i] ith item of s, origin 0	(3)

s[i:j] slice of s from i to j	(3)(4)

s[i:j:k] slice of s from i to j with step k	(3)(5)

...

(3) If i or j is negative, the index is relative to the end of sequence
s: len(s) + i or len(s) + j is substituted. But note that -0 is still 0.
?

Which anticipates the problem 'here', and only rubs salt into the wound.

Pressing on...

?
(4) The slice of s from i to j is defined as the sequence of items with
index k such that i <= k < j. If i or j is greater than len(s), use
len(s). If i is omitted or None, use 0. If j is omitted or None, use
len(s). If i is greater than or equal to j, the slice is empty.


(5) The slice of s from i to j with step k is defined as the sequence of
items with index x = i + n*k such that 0 <= n < (j-i)/k. In other words,
the indices are i, i+k, i+2*k, i+3*k and so on, stopping when j is
reached (but never including j). When k is positive, i and j are reduced
to len(s) if they are greater. When k is negative, i and j are reduced
to len(s) - 1 if they are greater. If i or j are omitted or None, they
become ?end? values (which end depends on the sign of k). Note, k cannot
be zero. If k is None, it is treated like 1.
?

...and so we see the open-closed nature of Python's ranges and
range-alike objects!


The 'secret' lies in the 'sins of omission'. As above, if either the
'start' or the 'stop' attribute is omitted, then it is treated as None, ie

x[ :-y ] becomes x[ None:-y ]

Thereafter we look at how None is interpreted. In the above example, the
'start' None becomes zero, thus:

x[ :-y ] becomes x[ None:-y ] and thus x[ 0:-y ]


The pertinent topic is the other end of the range. A 'stop' None will
become len( x ), thus putting it together with the 'start' points, the
short-hand form of a copy-operation is:

x[ : ] which becomes x[ None:None ] and thus x[ 0:len( x ) ]


Let's combine that with the idea of negative indexes/indices.
Referring-back to (3) (above).

If all of x is to be included: x[ :] is the way to go - but another way
to express that is:

x[ :len( x ) ]


If the last item in x is to be excluded: x[ :-1 ]

If the last two items ... excluded: x[ :-2 ]

(etc)

and another way to write those two specific examples is

x[ :len( x ) - 1 ]  and  x[ :len( x ) - 2 ]


and while we are taking things 'away' from the end of the iterable, the
missing x[ :len( x ) ]

is the same as x[ :len( x ) - 0 ]


Thus, by now, you've leaped ahead of me, to:

>>> for y in [ 0, 1, 2, 3, 4, 5 ]:
...     print( y, x[ :len( x ) - y ] )
...
0 ['a', 'b', 'c', 'd', 'e']
1 ['a', 'b', 'c', 'd']
2 ['a', 'b', 'c']
3 ['a', 'b']
4 ['a']
5 []

and yes, if computing y is expensive/ugly, for extra-credit, calculate
the 'stop' value outside/prior-to the for-loop!
-- 
Regards,
=dn

From framstag at rus.uni-stuttgart.de  Thu Nov 25 12:58:10 2021
From: framstag at rus.uni-stuttgart.de (Ulli Horlacher)
Date: Thu, 25 Nov 2021 17:58:10 +0000 (UTC)
Subject: pyinstaller wrong classified as Windows virus
References: <snnkhf$mve$1@news2.informatik.uni-stuttgart.de>
 <slrnspur9f.51o.dan@djph.net>
 <sno092$prd$1@news2.informatik.uni-stuttgart.de>
 <sno3qt$r1b$1@news2.informatik.uni-stuttgart.de>
 <CAPTjJmq=w3iFDWjzNFAfpmAC=5kFZm1C8=Ex7Cwo2iwDVY_LHA@mail.gmail.com>
 <mailman.8.1637859246.9869.python-list@python.org>
 <snog1a$u4v$1@news2.informatik.uni-stuttgart.de>
 <CAPTjJmrdM=puoFg7aPEtbRi73NYN+6WHWb=FKHvqwK7pUcx+FQ@mail.gmail.com>
 <cd2599b2-9428-26ee-7de9-9ecafc2adff1@Damon-Family.org>
 <mailman.11.1637862528.9869.python-list@python.org>
Message-ID: <snoiri$usa$1@news2.informatik.uni-stuttgart.de>

Richard Damon <Richard at damon-family.org> wrote:

> On a somewhat locked down computer, the user does not have admin rights, 
> so needs to get 'IT' to run any installers that need admin permissions 
> to run.
> 
> And EXE that just needs to be copied to the computer and rhen just RUN, 
> doesn't need IT to 'install' it (they just can't put it into Program 
> Files, but that isn't really that important for programs that don't need 
> an installer.

That's the situation for many of my users.


-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher at tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/

From framstag at rus.uni-stuttgart.de  Fri Nov 26 02:13:30 2021
From: framstag at rus.uni-stuttgart.de (Ulli Horlacher)
Date: Fri, 26 Nov 2021 07:13:30 +0000 (UTC)
Subject: pyinstaller wrong classified as Windows virus
References: <snnkhf$mve$1@news2.informatik.uni-stuttgart.de>
 <slrnspur9f.51o.dan@djph.net>
 <sno092$prd$1@news2.informatik.uni-stuttgart.de>
 <sno3qt$r1b$1@news2.informatik.uni-stuttgart.de>
 <CAPTjJmq=w3iFDWjzNFAfpmAC=5kFZm1C8=Ex7Cwo2iwDVY_LHA@mail.gmail.com>
 <mailman.8.1637859246.9869.python-list@python.org>
 <snog1a$u4v$1@news2.informatik.uni-stuttgart.de>
 <007701d7e252$9402b500$bc081f00$@verizon.net>
 <mailman.18.1637882241.9869.python-list@python.org>
Message-ID: <snq1eq$amo$1@news2.informatik.uni-stuttgart.de>

Avi Gross <avigross at verizon.net> wrote:

> I am not sure what your real problem is, Ulli, but many antivirus programs
> can be TEMPORARILY shut off.

Meanwhile I found this configuration option.
But this does not help me much, because my programs must run on other
Windows PCs of other users and they cannot disable the default Windows
virus scanner.

I for myself does not use Windows at all, I just use it to compile my
programs.


> If one recognizes your code a potentially having a virus, it may be for an
> assortment of reasons such as a table it contains to look at position N in
> the executable for an exact match with some bit-string. If so, one potential
> fix is a slight change in the code that compiles a bit differently like
> x=sin(30) or other filler. 

I do not know what and where the virus scanning is complaining about.
It simple says virus threat found and then it deletes my executables.
It is the default virus scanner of Windows 10, I have not installed any
additional software (besides Python and cygwin).


> But consider another possibility that your compiler software is compromised

Then https://www.python.org/ftp/python/3.10.0/python-3.10.0-amd64.exe
is infected. I doubt this.

                                          
> Is this happening to only one set of code?

This is happening SOMETIMES, not always. With the SAME source code. When I
call pyinstaller often enough, then the virus scanner is quiet. In about 1
of 20 compile runs.



-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher at tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/

From edmondo.giovannozzi at gmail.com  Fri Nov 26 04:28:54 2021
From: edmondo.giovannozzi at gmail.com (Edmondo Giovannozzi)
Date: Fri, 26 Nov 2021 01:28:54 -0800 (PST)
Subject: pyinstaller wrong classified as Windows virus
In-Reply-To: <snq1eq$amo$1@news2.informatik.uni-stuttgart.de>
References: <snnkhf$mve$1@news2.informatik.uni-stuttgart.de>
 <slrnspur9f.51o.dan@djph.net> <sno092$prd$1@news2.informatik.uni-stuttgart.de>
 <sno3qt$r1b$1@news2.informatik.uni-stuttgart.de>
 <CAPTjJmq=w3iFDWjzNFAfpmAC=5kFZm1C8=Ex7Cwo2iwDVY_LHA@mail.gmail.com>
 <mailman.8.1637859246.9869.python-list@python.org>
 <snog1a$u4v$1@news2.informatik.uni-stuttgart.de>
 <007701d7e252$9402b500$bc081f00$@verizon.net>
 <mailman.18.1637882241.9869.python-list@python.org>
 <snq1eq$amo$1@news2.informatik.uni-stuttgart.de>
Message-ID: <cf0c2017-5bc2-469f-923e-86efdc5d5107n@googlegroups.com>

Il giorno venerd? 26 novembre 2021 alle 08:13:50 UTC+1 Ulli Horlacher ha scritto:
> Avi Gross <avig... at verizon.net> wrote: 
> 
> > I am not sure what your real problem is, Ulli, but many antivirus programs 
> > can be TEMPORARILY shut off.
> Meanwhile I found this configuration option. 
> But this does not help me much, because my programs must run on other 
> Windows PCs of other users and they cannot disable the default Windows 
> virus scanner. 
> 
> I for myself does not use Windows at all, I just use it to compile my 
> programs.
> > If one recognizes your code a potentially having a virus, it may be for an 
> > assortment of reasons such as a table it contains to look at position N in 
> > the executable for an exact match with some bit-string. If so, one potential 
> > fix is a slight change in the code that compiles a bit differently like 
> > x=sin(30) or other filler.
> I do not know what and where the virus scanning is complaining about. 
> It simple says virus threat found and then it deletes my executables. 
> It is the default virus scanner of Windows 10, I have not installed any 
> additional software (besides Python and cygwin).
> > But consider another possibility that your compiler software is compromised
> Then https://www.python.org/ftp/python/3.10.0/python-3.10.0-amd64.exe 
> is infected. I doubt this.
> > Is this happening to only one set of code?
> This is happening SOMETIMES, not always. With the SAME source code. When I 
> call pyinstaller often enough, then the virus scanner is quiet. In about 1 
> of 20 compile runs.
> -- 
> Ullrich Horlacher Server und Virtualisierung 
> Rechenzentrum TIK 
> Universitaet Stuttgart E-Mail: horl... at tik.uni-stuttgart.de 
> Allmandring 30a Tel: ++49-711-68565868 
> 70569 Stuttgart (Germany) WWW: http://www.tik.uni-stuttgart.de/


You can try to download winpython: https://github.com/winpython/winpython/releases
It is an executable, but you don't need to execute it as it is a 7zip compressed archive.
You may run it or use directly 7zip to decompress it, the result will be the same. 
 
Then you have a full python installation that don't need to be installed. 
You may try to put your program there and give the users that directory.


From edmondo.giovannozzi at gmail.com  Fri Nov 26 04:36:04 2021
From: edmondo.giovannozzi at gmail.com (Edmondo Giovannozzi)
Date: Fri, 26 Nov 2021 01:36:04 -0800 (PST)
Subject: Negative subscripts
In-Reply-To: <mailman.19.1637918605.9869.python-list@python.org>
References: <96741f1a-72b3-6551-39b3-f1862c5a80b8@chagford.com>
 <mailman.19.1637918605.9869.python-list@python.org>
Message-ID: <6f673c52-2c89-4d28-bd74-b697b98d3054n@googlegroups.com>

Il giorno venerd? 26 novembre 2021 alle 10:23:46 UTC+1 Frank Millman ha scritto:
> Hi all 
> 
> In my program I have a for-loop like this - 
> 
> >>> for item in x[:-y]: 
> ...    [do stuff] 
> 
> 'y' may or may not be 0. If it is 0 I want to process the entire list 
> 'x', but of course -0 equals 0, so it returns an empty list. 
> 
> In theory I can say 
> 
> >>> for item in x[:-y] if y else x: 
> ...    [do stuff] 
> 
> But in my actual program, both x and y are fairly long expressions, so 
> the result is pretty ugly. 
> 
> Are there any other techniques anyone can suggest, or is the only 
> alternative to use if...then...else to cater for y = 0? 
> 
> Thanks 
> 
> Frank Millman

First assign your long expressions to variables x and y and then you may write:

for item in x[:len(x) - y]:
   ...


From framstag at rus.uni-stuttgart.de  Fri Nov 26 07:32:39 2021
From: framstag at rus.uni-stuttgart.de (Ulli Horlacher)
Date: Fri, 26 Nov 2021 12:32:39 +0000 (UTC)
Subject: pyinstaller wrong classified as Windows virus
References: <snnkhf$mve$1@news2.informatik.uni-stuttgart.de>
 <slrnspur9f.51o.dan@djph.net>
 <sno092$prd$1@news2.informatik.uni-stuttgart.de>
 <sno3qt$r1b$1@news2.informatik.uni-stuttgart.de>
 <CAPTjJmq=w3iFDWjzNFAfpmAC=5kFZm1C8=Ex7Cwo2iwDVY_LHA@mail.gmail.com>
 <mailman.8.1637859246.9869.python-list@python.org>
 <snog1a$u4v$1@news2.informatik.uni-stuttgart.de>
 <007701d7e252$9402b500$bc081f00$@verizon.net>
 <mailman.18.1637882241.9869.python-list@python.org>
 <snq1eq$amo$1@news2.informatik.uni-stuttgart.de>
 <cf0c2017-5bc2-469f-923e-86efdc5d5107n@googlegroups.com>
Message-ID: <snqk57$ffe$1@news2.informatik.uni-stuttgart.de>

Edmondo Giovannozzi <edmondo.giovannozzi at gmail.com> wrote:

> You can try to download winpython: https://github.com/winpython/winpython/releases
> It is an executable, but you don't need to execute it as it is a 7zip compressed archive.
> You may run it or use directly 7zip to decompress it, the result will be the same. 
>  
> Then you have a full python installation that don't need to be installed. 
> You may try to put your program there and give the users that directory.

I do not understand it quite...
(How) can I add PySimpleGUI (*) and my program to this (self-extracting?)
package? 


https://sourceforge.net/projects/winpython/files/WinPython_3.10/3.10.0.1/
633 MB! I have a lot of users with internet speed < 200 kB/s

With pyinstaller my program executables are are around 10 MB.


(*) My programs depend on:
    https://pysimplegui.readthedocs.io/en/latest/

-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher at tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/

From pieter-l at vanoostrum.org  Fri Nov 26 09:08:18 2021
From: pieter-l at vanoostrum.org (Pieter van Oostrum)
Date: Fri, 26 Nov 2021 15:08:18 +0100
Subject: Negative subscripts
References: <96741f1a-72b3-6551-39b3-f1862c5a80b8@chagford.com>
 <mailman.19.1637918605.9869.python-list@python.org>
Message-ID: <m2ilwfm2lp.fsf@vanoostrum.org>

Frank Millman <frank at chagford.com> writes:

> Hi all
>
> In my program I have a for-loop like this -
>
>>>> for item in x[:-y]:
> ...??? [do stuff]
>
> 'y' may or may not be 0. If it is 0 I want to process the entire list
> 'x', but of course -0 equals 0, so it returns an empty list.
>
> In theory I can say
>
>>>> for item in x[:-y] if y else x:
> ...??? [do stuff]
>
> But in my actual program, both x and y are fairly long expressions, so
> the result is pretty ugly.
>
> Are there any other techniques anyone can suggest, or is the only
> alternative to use if...then...else to cater for y = 0?

If you put it in a function with x and y as parameters, then both x and y are just a simple identifier inside the function.
And then you can then even eliminate the if with

for item in x[:len(x)-y]:
-- 
Pieter van Oostrum <pieter at vanoostrum.org>
www: http://pieter.vanoostrum.org/
PGP key: [8DAE142BE17999C4]

From peter.heitzer at rz.uni-regensburg.de  Fri Nov 26 10:30:52 2021
From: peter.heitzer at rz.uni-regensburg.de (Peter Heitzer)
Date: 26 Nov 2021 15:30:52 GMT
Subject: pyinstaller wrong classified as Windows virus
References: <snnkhf$mve$1@news2.informatik.uni-stuttgart.de>
 <CAPTjJmq=w3iFDWjzNFAfpmAC=5kFZm1C8=Ex7Cwo2iwDVY_LHA@mail.gmail.com>
 <mailman.8.1637859246.9869.python-list@python.org>
 <snog1a$u4v$1@news2.informatik.uni-stuttgart.de>
 <007701d7e252$9402b500$bc081f00$@verizon.net>
 <mailman.18.1637882241.9869.python-list@python.org>
 <snq1eq$amo$1@news2.informatik.uni-stuttgart.de>
 <cf0c2017-5bc2-469f-923e-86efdc5d5107n@googlegroups.com>
 <snqk57$ffe$1@news2.informatik.uni-stuttgart.de>
Message-ID: <j0cctcF5fp3U1@mid.individual.net>

Ulli Horlacher <framstag at rus.uni-stuttgart.de> wrote:
>Edmondo Giovannozzi <edmondo.giovannozzi at gmail.com> wrote:

>> You can try to download winpython: https://github.com/winpython/winpython/releases
>> It is an executable, but you don't need to execute it as it is a 7zip compressed archive.
>> You may run it or use directly 7zip to decompress it, the result will be the same. 
>>  
>> Then you have a full python installation that don't need to be installed. 
>> You may try to put your program there and give the users that directory.

>I do not understand it quite...
>(How) can I add PySimpleGUI (*) and my program to this (self-extracting?)
>package? 


>https://sourceforge.net/projects/winpython/files/WinPython_3.10/3.10.0.1/
>633 MB! I have a lot of users with internet speed < 200 kB/s

>With pyinstaller my program executables are are around 10 MB.
I would suggest to build your own installer, for example using NSIS
https://nsis.sourceforge.io
It is not very complicated. If you include the normal Python installation
and the additional packages and your scripts the whole installer will not
be greater as about 30 MB. 
If you want I can mail you a sample config file you can start with.

-- 
Dipl.-Inform(FH) Peter Heitzer, peter.heitzer at rz.uni-regensburg.de

From petermwale47 at gmail.com  Fri Nov 26 15:38:44 2021
From: petermwale47 at gmail.com (Peter Mwale)
Date: Fri, 26 Nov 2021 22:38:44 +0200
Subject: Failure to Display Top menu
Message-ID: <CAGihD5xiNq9v1hq8-sBZsYHhnao-b7_+s8uM6PD6mu_rnxc=6g@mail.gmail.com>

Hello, my python 3.10 shell is not displaying the top menu. What should I
do?

-- 
Peter Mwale
Principal Economist
Ministry of Gender, Social Welfare and Community Development
Private Bag 320
Lilongwe 3
Cell: +265 998 605 723
Phone: +265 1 788 888
Fax: +265 1 788 093
E-mail: petermwale47 at gmail.com

From greg.ewing at canterbury.ac.nz  Fri Nov 26 18:37:06 2021
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 27 Nov 2021 12:37:06 +1300
Subject: Negative subscripts
In-Reply-To: <mailman.23.1637934336.9869.python-list@python.org>
References: <96741f1a-72b3-6551-39b3-f1862c5a80b8@chagford.com>
 <1ed3404c-3eaf-bbdc-56a1-5b4494f495bb@chagford.com>
 <mailman.23.1637934336.9869.python-list@python.org>
Message-ID: <j0d9d2Fb5s2U1@mid.individual.net>

> On 2021-11-26 11:17 AM, Frank Millman wrote:
>> Are there any other techniques anyone can suggest, or is the only 
>> alternative to use if...then...else to cater for y = 0?

x[:-y or None]

Seems to work:

 >>> l
['a', 'b', 'c', 'd', 'e']
 >>> def f(x): return l[:-x or None]
...
 >>> f(3)
['a', 'b']
 >>> f(2)
['a', 'b', 'c']
 >>> f(1)
['a', 'b', 'c', 'd']
 >>> f(0)
['a', 'b', 'c', 'd', 'e']

-- 
Greg


From frank at chagford.com  Sat Nov 27 01:11:51 2021
From: frank at chagford.com (Frank Millman)
Date: Sat, 27 Nov 2021 08:11:51 +0200
Subject: Negative subscripts
In-Reply-To: <ba4a976c-5829-03ac-2e3f-cf670f00c3f7@DancesWithMice.info>
References: <96741f1a-72b3-6551-39b3-f1862c5a80b8@chagford.com>
 <ba4a976c-5829-03ac-2e3f-cf670f00c3f7@DancesWithMice.info>
Message-ID: <1c37f528-250c-85b5-5e98-2ccde350438b@chagford.com>

On 2021-11-26 11:24 PM, dn via Python-list wrote:
> On 26/11/2021 22.17, Frank Millman wrote:
>> In my program I have a for-loop like this -
>>
>>>>> for item in x[:-y]:
>> ...??? [do stuff]
>>
>> 'y' may or may not be 0. If it is 0 I want to process the entire list
>> 'x', but of course -0 equals 0, so it returns an empty list.
> ...
> 

[...]

That was an interesting read - thanks for spelling it out.

> 
>>>> for y in [ 0, 1, 2, 3, 4, 5 ]:
> ...     print( y, x[ :len( x ) - y ] )
> ...
> 0 ['a', 'b', 'c', 'd', 'e']
> 1 ['a', 'b', 'c', 'd']
> 2 ['a', 'b', 'c']
> 3 ['a', 'b']
> 4 ['a']
> 5 []
> 
> and yes, if computing y is expensive/ugly, for extra-credit, calculate
> the 'stop' value outside/prior-to the for-loop!
> 

Ignoring the 'ugly' for the moment, what if computing y is expensive?

To check this, I will restate the example to more closely match my use case.

 >>> x = [1, 2, 3, 4, 5, 6, 7]
 >>> y = [5, 4, 3]
 >>> z = []
 >>>
 >>> for i in x[ : len(x) - len(y) ]:
...   i
...
1
2
3
4
 >>>
 >>> for i in x[ : len(x) - len(z) ]:
...   i
...
1
2
3
4
5
6
7
 >>>

So it works perfectly (not that I had any doubts).

But what if it is expensive to compute y? Or to rephrase it, is y 
computed on every iteration, or only on the first one?

Without knowing the internals, it is not possible to tell just by 
looking at it. But there is a technique I learned from Peter Otten 
(haven't heard from him for a while - hope he is still around).

 >>> def lng(lst):
...   print(f'*{len(lst)}*')
...   return len(lst)
...
 >>>
 >>> for i in x[ : lng(x) - lng(y) ]:
...   i
...
*7*
*3*
1
2
3
4
 >>>
 >>> for i in x[ : lng(x) - lng(z) ]:
...   i
...
*7*
*0*
1
2
3
4
5
6
7
 >>>

 From this it is clear that y is only computed once, when the loop is 
started. Therefore I think it follows that there is no need to 
pre-compute y.

Hope this makes sense.

Frank

From PythonList at DancesWithMice.info  Sat Nov 27 01:15:55 2021
From: PythonList at DancesWithMice.info (dn)
Date: Sat, 27 Nov 2021 19:15:55 +1300
Subject: Friday Finking: Docstrings and DataClasses
Message-ID: <4a362bd2-29b3-57f1-d124-a8f1b6943d9b@DancesWithMice.info>

How have you updated your (team's) standards and conventions for
docstrings, when using dataclasses?


NB the question is specifically class-related. Whereas many of the
examples 'here' are of functions; a docstring is a docstring, is a
docstring.


The original word on the subject is/was "PEP 257 -- Docstring
Conventions" (https://www.python.org/dev/peps/pep-0257/) and included
suggestions such as:

def complex(real=0.0, imag=0.0):
    """Form a complex number.

    Keyword arguments:
    real -- the real part (default 0.0)
    imag -- the imaginary part (default 0.0)
    """
    if imag == 0.0 and real == 0.0:
        return complex_zero
    ...

The key information is the purpose of the function. We also find a
re-statement of the parameter names, their default-values, an intimation
of their types, and an indication of their purpose.

NB no mention of the output/return-value's specifics.


This format plays-nicely with the Python help() system and related.


Barely one year later the topic was advanced, with the more detailed
"PEP 287 -- reStructuredText Docstring Format"
(https://www.python.org/dev/peps/pep-0287/).

This idea is better-illustrated in "The Sphinx docstring format"
(https://sphinx-rtd-tutorial.readthedocs.io/en/latest/docstrings.html).

That offers a template:

"""[Summary]

:param [ParamName]: [ParamDescription], defaults to [DefaultParamVal]
:type [ParamName]: [ParamType](, optional)
...
:raises [ErrorType]: [ErrorDescription]
...
:return: [ReturnDescription]
:rtype: [ReturnType]
"""

...with practical-example docstrings written for "the SimpleBleDevice
class, which is defined within our simpleble module":

class SimpleBleDevice(object):
    """This is a conceptual class representation of a simple BLE device
    (GATT Server). It is essentially an extended combination of the
    :class:`bluepy.btle.Peripheral` and :class:`bluepy.btle.ScanEntry`
classes

    :param client: A handle to the :class:`simpleble.SimpleBleClient` client
        object that detected the device
    :type client: class:`simpleble.SimpleBleClient`
    :param addr: Device MAC address, defaults to None
    :type addr: str, optional
    :param addrType: Device address type - one of ADDR_TYPE_PUBLIC or
        ADDR_TYPE_RANDOM, defaults to ADDR_TYPE_PUBLIC
    :type addrType: str, optional
    :param iface: Bluetooth interface number (0 = /dev/hci0) used for the
        connection, defaults to 0
    :type iface: int, optional
    :param data: A list of tuples (adtype, description, value)
containing the
        AD type code, human-readable description and value for all available
        advertising data items, defaults to None
    :type data: list, optional
    :param rssi: Received Signal Strength Indication for the last received
        broadcast from the device. This is an integer value measured in dB,
        where 0 dB is the maximum (theoretical) signal strength, and more
        negative numbers indicate a weaker signal, defaults to 0
    :type rssi: int, optional
    :param connectable: `True` if the device supports connections, and
`False`
        otherwise (typically used for advertising ?beacons?).,
        defaults to `False`
    :type connectable: bool, optional
    :param updateCount: Integer count of the number of advertising packets
        received from the device so far, defaults to 0
    :type updateCount: int, optional
    """

(apologies: email word-wrap likely wrecks their neat formatting)


We've probably reached the point-in-time when this 'grumpy old man'
bemoans that such formulaic approaches tend to lead to slavish habits -
which become vices when they do nothing to improve the intended
objectives of "readability" (or worse detract from it), ie the docstring
equivalent of:

counter += 1  # add one to counter


Management advice: if you set a standard/requirement, such will be met -
without regard for other measures of quality, ie be careful what you
wish for!


Why the negative attitude? An example published in "Python Docstrings :
How to document your Python code ?"
(https://amiradata.com/python-docstrings/):

def setType(self,type):
    """Change the pokemon type.

        The parameter value is stored in the type
        variable of the pokemon class.

        :param type: type pokemon
        :type type: string
        :return: no value
        :rtype: none
        """

    self.type = type
    return

    return self.type

NB are you also amused by the inclusion of a question-mark in the
article's title?

The use of the name "type", alongside the Python term "type", adds to my
confusion (because I didn't/don't even know that there is more than one
type/kind/flavor of Pokemon (mea culpa!)). To say nothing of the
multiple return statements and non-PEP-008 naming.


Fast-forward a good 15 years and today we have 'typing' (many PEP-refs)
and "PEP 557 -- Data Classes"
(https://www.python.org/dev/peps/pep-0557/), which combined to become
the "Data Classes" library in the PSL
(https://docs.python.org/3/library/dataclasses.html)

Now we can write:

@dataclass
class InventoryItem:
    """Class for keeping track of an item in inventory."""
    name: str
    unit_price: float
    quantity_on_hand: int = 0

The docstring itself still covers the purpose of the function.

In this case, there is no re-statement of the parameter names (which
would otherwise have appeared in __init__()), because their declaration
is immediately beneath the docstring.

As are any default-values.

The parameter-typing has become a compulsory part of the code (cf
added-documentation).

All we appear to be missing is an indication of each parameter's purpose
- over-and-above any meaning which can be gleaned from the careful
choice of attribute-name.

Plus, if we add return-value typing to a function def, then that need is
similarly satisfied.

Yes, I'm a strong user of typing (but not always)...


Finally, to the strong-arm tactics that linters seem to have become:

I'm amused (ok, it irritates) that one of the linters built-in or
plugged-into my IDE does not like in-line comments. No place, no time!@
Accordingly, the following results in criticism:

    unit_price: float  # the cost of a single unit in NZD
    quantity_on_hand: int = 0  # excludes items reserved by Sales Dept

Yet, such seems a quite-reasonable (updated for dataclasses) approach.


Do you agree?

Do you not bother with docstring style at all?

Have you stuck with PEP-257, after all these years?

Do you (still) use the Sphinx/RST format for docstrings, even though it
seems even more repetitive and wordy?

Have you updated your conventions/style-manual to acknowledge
dataclasses? If so, how?
-- 
Regards,
=dn

From rosuav at gmail.com  Sat Nov 27 01:34:15 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 27 Nov 2021 17:34:15 +1100
Subject: pyinstaller wrong classified as Windows virus
In-Reply-To: <snoiri$usa$1@news2.informatik.uni-stuttgart.de>
References: <snnkhf$mve$1@news2.informatik.uni-stuttgart.de>
 <slrnspur9f.51o.dan@djph.net> <sno092$prd$1@news2.informatik.uni-stuttgart.de>
 <sno3qt$r1b$1@news2.informatik.uni-stuttgart.de>
 <CAPTjJmq=w3iFDWjzNFAfpmAC=5kFZm1C8=Ex7Cwo2iwDVY_LHA@mail.gmail.com>
 <mailman.8.1637859246.9869.python-list@python.org>
 <snog1a$u4v$1@news2.informatik.uni-stuttgart.de>
 <CAPTjJmrdM=puoFg7aPEtbRi73NYN+6WHWb=FKHvqwK7pUcx+FQ@mail.gmail.com>
 <cd2599b2-9428-26ee-7de9-9ecafc2adff1@Damon-Family.org>
 <mailman.11.1637862528.9869.python-list@python.org>
 <snoiri$usa$1@news2.informatik.uni-stuttgart.de>
Message-ID: <CAPTjJmrEjiPJzQ9SkaDsFew-poQRVAisuENqwbwL6cDZujsmmQ@mail.gmail.com>

On Sat, Nov 27, 2021 at 4:47 PM Ulli Horlacher
<framstag at rus.uni-stuttgart.de> wrote:
>
> Richard Damon <Richard at damon-family.org> wrote:
>
> > On a somewhat locked down computer, the user does not have admin rights,
> > so needs to get 'IT' to run any installers that need admin permissions
> > to run.
> >
> > And EXE that just needs to be copied to the computer and rhen just RUN,
> > doesn't need IT to 'install' it (they just can't put it into Program
> > Files, but that isn't really that important for programs that don't need
> > an installer.
>
> That's the situation for many of my users.
>

Several of us pointed out that you do not need admin rights to install
Python. Did you consider that?

ChrisA

From PythonList at DancesWithMice.info  Sat Nov 27 03:20:15 2021
From: PythonList at DancesWithMice.info (dn)
Date: Sat, 27 Nov 2021 21:20:15 +1300
Subject: Negative subscripts
In-Reply-To: <1c37f528-250c-85b5-5e98-2ccde350438b@chagford.com>
References: <96741f1a-72b3-6551-39b3-f1862c5a80b8@chagford.com>
 <ba4a976c-5829-03ac-2e3f-cf670f00c3f7@DancesWithMice.info>
 <1c37f528-250c-85b5-5e98-2ccde350438b@chagford.com>
Message-ID: <c21483f1-9d14-968d-7d75-aa1ca3f90cc0@DancesWithMice.info>

On 27/11/2021 19.11, Frank Millman wrote:
> On 2021-11-26 11:24 PM, dn via Python-list wrote:
>> On 26/11/2021 22.17, Frank Millman wrote:
>>> In my program I have a for-loop like this -
>>>
>>>>>> for item in x[:-y]:
>>> ...??? [do stuff]
>>>
>>> 'y' may or may not be 0. If it is 0 I want to process the entire list
>>> 'x', but of course -0 equals 0, so it returns an empty list.
>> ...
>>
> 
> [...]
> 
> That was an interesting read - thanks for spelling it out.
> 
>>
>>>>> for y in [ 0, 1, 2, 3, 4, 5 ]:
>> ...???? print( y, x[ :len( x ) - y ] )
>> ...
>> 0 ['a', 'b', 'c', 'd', 'e']
>> 1 ['a', 'b', 'c', 'd']
>> 2 ['a', 'b', 'c']
>> 3 ['a', 'b']
>> 4 ['a']
>> 5 []
>>
>> and yes, if computing y is expensive/ugly, for extra-credit, calculate
>> the 'stop' value outside/prior-to the for-loop!
>>
> 
> Ignoring the 'ugly' for the moment, what if computing y is expensive?
> 
> To check this, I will restate the example to more closely match my use
> case.
> 
>>>> x = [1, 2, 3, 4, 5, 6, 7]
>>>> y = [5, 4, 3]
>>>> z = []
>>>>
>>>> for i in x[ : len(x) - len(y) ]:
> ...?? i
> ...
> 1
> 2
> 3
> 4
>>>>
>>>> for i in x[ : len(x) - len(z) ]:
> ...?? i
> ...
> 1
> 2
> 3
> 4
> 5
> 6
> 7
>>>>
> 
> So it works perfectly (not that I had any doubts).
> 
> But what if it is expensive to compute y? Or to rephrase it, is y
> computed on every iteration, or only on the first one?
> 
> Without knowing the internals, it is not possible to tell just by
> looking at it. But there is a technique I learned from Peter Otten
> (haven't heard from him for a while - hope he is still around).
> 
>>>> def lng(lst):
> ...?? print(f'*{len(lst)}*')
> ...?? return len(lst)
> ...
>>>>
>>>> for i in x[ : lng(x) - lng(y) ]:
> ...?? i
> ...
> *7*
> *3*
> 1
> 2
> 3
> 4
>>>>
>>>> for i in x[ : lng(x) - lng(z) ]:
> ...?? i
> ...
> *7*
> *0*
> 1
> 2
> 3
> 4
> 5
> 6
> 7
>>>>
> 
> From this it is clear that y is only computed once, when the loop is
> started. Therefore I think it follows that there is no need to
> pre-compute y.
> 
> Hope this makes sense.

Perfect sense (== @Peter sense).


To confirm:

?8.3. The for statement?

The for statement is used to iterate over the elements of a sequence
(such as a string, tuple or list) or other iterable object:

for_stmt ::=  "for" target_list "in" expression_list ":" suite
              ["else" ":" suite]

The expression list is evaluated once; it should yield an iterable
object. An iterator is created for the result of the expression_list.
The suite is then executed once for each item provided by the iterator,
in the order returned by the iterator. Each item in turn is assigned to
the target list using the standard rules for assignments (see Assignment
statements), and then the suite is executed. When the items are
exhausted (which is immediately when the sequence is empty or an
iterator raises a StopIteration exception), the suite in the else
clause, if present, is executed, and the loop terminates.
?
https://docs.python.org/3/reference/compound_stmts.html#the-for-statement


That said, I'm wondering if all is strictly true when the
expression_list is a generator, which by definition features
lazy-execution rather than up-front list production. So, things may
depend upon the full-nature of the application.


Assuming, as-per these simple examples, there is no advantage to
pre-computation, it may be more readable to 'prepare' the control-values
separately, thereby simplifying the (appearance of the) for-statement.


The problem at this level, are the terms: "fairly long", "pretty ugly"*,
and "expensive". There are various tactics which might be brought to
bear, but will only be applicable in specific situations. So, too little
information, too much speculation...

For example if there is repetition inside the loop, where a particular
x[ i ] is related to something multiple times - perhaps in  multiple
executions of the loop, 'memoisation' may save recalculation -
particularly if this happens inside a nested-loop. This has nothing to
do with the for-loop itself. It trades memory-space for execution-time -
one "expense" for another. However, only you can tell if it, or any
other idea, might be relevant in this application...


* can you make up your mind? Is it pretty, or ugly?
-- 
Regards,
=dn

From rosuav at gmail.com  Sat Nov 27 03:23:00 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 27 Nov 2021 19:23:00 +1100
Subject: Negative subscripts
In-Reply-To: <c21483f1-9d14-968d-7d75-aa1ca3f90cc0@DancesWithMice.info>
References: <96741f1a-72b3-6551-39b3-f1862c5a80b8@chagford.com>
 <ba4a976c-5829-03ac-2e3f-cf670f00c3f7@DancesWithMice.info>
 <1c37f528-250c-85b5-5e98-2ccde350438b@chagford.com>
 <c21483f1-9d14-968d-7d75-aa1ca3f90cc0@DancesWithMice.info>
Message-ID: <CAPTjJmoHcEB5GJnEpmq-B18qz6eX2u8_qLhGkSKv0kr-V7cYcw@mail.gmail.com>

On Sat, Nov 27, 2021 at 7:21 PM dn via Python-list
<python-list at python.org> wrote:
> The expression list is evaluated once; it should yield an iterable
> object. An iterator is created for the result of the expression_list.
> The suite is then executed once for each item provided by the iterator,
> in the order returned by the iterator. Each item in turn is assigned to
> the target list using the standard rules for assignments (see Assignment
> statements), and then the suite is executed. When the items are
> exhausted (which is immediately when the sequence is empty or an
> iterator raises a StopIteration exception), the suite in the else
> clause, if present, is executed, and the loop terminates.
> ?
> https://docs.python.org/3/reference/compound_stmts.html#the-for-statement
>
>
> That said, I'm wondering if all is strictly true when the
> expression_list is a generator, which by definition features
> lazy-execution rather than up-front list production. So, things may
> depend upon the full-nature of the application.

Yes, it is. Evaluating a generator expression gives you a generator
object, which is an iterable.

ChrisA

From auriocus at gmx.de  Sat Nov 27 05:36:47 2021
From: auriocus at gmx.de (Christian Gollwitzer)
Date: Sat, 27 Nov 2021 11:36:47 +0100
Subject: Failure to Display Top menu
In-Reply-To: <mailman.25.1637991960.9869.python-list@python.org>
References: <CAGihD5xiNq9v1hq8-sBZsYHhnao-b7_+s8uM6PD6mu_rnxc=6g@mail.gmail.com>
 <mailman.25.1637991960.9869.python-list@python.org>
Message-ID: <snt1nv$3kh$1@dont-email.me>

Am 26.11.21 um 21:38 schrieb Peter Mwale:
> Hello, my python 3.10 shell is not displaying the top menu. What should I
> do?
> 

You should explain, what you do exactly. The Python interpreter does not 
have a menu.

a) What platform are you on? Windows, macOS, Linux?

b) How did ou start Python and what was the expectation?

	Christian

From barry at barrys-emacs.org  Sat Nov 27 14:24:09 2021
From: barry at barrys-emacs.org (Barry Scott)
Date: Sat, 27 Nov 2021 19:24:09 +0000
Subject: Eventfd with epoll BlockingIOError
In-Reply-To: <MpOCrN6--3-2@tutanota.com>
References: <MpJ6MZm--B-2@tutanota.com>
 <A48057B9-1031-4DC8-9FC9-E59D2DA14BB4@barrys-emacs.org>
 <MpOCrN6--3-2@tutanota.com>
Message-ID: <F16D6E5A-71C9-42C6-88C3-64883B7E84EB@barrys-emacs.org>


> On 25 Nov 2021, at 22:29, jenkris at tutanota.com wrote:
> 
> Thanks very much for your reply.  
> 
> I am now getting a single event returned in Python, but it's not the right event, as I'll explain below. 
> 
> I rearranged the Python code based on your comments:
> 
> #!/usr/bin/python3
> import sys
> import os
> import select
> 
> print("Inside Python")
> 
> event_fd = int(sys.argv[3])
> 
> print("Eventfd received by Python")
> print(event_fd)
> 
> event_write_value = 100
> 
> ep = select.epoll(-1)
> ep.register(event_fd, select.EPOLLIN | select.EPOLLOUT )
> 
> os.set_blocking(event_fd, False)
> 
> #__________
> 
> print("Starting poll loop")
> 
> for fd_event in ep.poll():
>     print("Python fd_event")
>     print(fd_event)
>     fd_received = fd_event[0]
>     event_received = fd_event[1]
> 
> You advised to leave off select.EPOLLOUT from the line ep.register(event_fd, select.EPOLLIN | select.EPOLLOUT ) -- which makes sense because I'm not waiting for that event -- but without it both processes freeze in the for loop (below print("Starting poll loop")) so we never receive an EPOLLIN event.  So I included it, and here is the screen output from gdb:
> 
> Inside Python
> Eventfd received by Python
> 5
> Everything OK in Python
> Starting poll loop
> Python fd_event
> (5, 4)
> Writing to Python
> 5 Received from Python
> 8 Writing to Python
> Failed epoll_wait Bad file descriptor
> 5 Received from Python
> 8 Writing to Python
> Failed epoll_wait Bad file descriptor
> 5 Received from Python
> -1time taken 0.000629
> Failed to close epoll file descriptor
> Unlink_shm status: Bad file descriptor
> fn() took 0.000717 seconds to execute
> [Inferior 1 (process 26718) exited normally]
> (gdb) q
> 
> The Python fd_event tuple is 5, 4 -- 5 is the correct file descriptor and 4 is an EPOLLOUT event, which is not what I want. 
> 
> The eventfd is created in C as nonblocking:
> 
> int eventfd_initialize() {
>   int efd = eventfd(0, EFD_NONBLOCK);

So it is not a semaphore as EFD_SEMAPHORE is not set.
Thus you intend to send arbitrary 64 int values and know that the value of 0

>   return efd; }
> 
> When C writes it calls epoll_wait:
> 
> ssize_t epoll_write(int event_fd, int epoll_fd, struct epoll_event * event_struc, int action_code)
> {
>    int64_t ewbuf[1];

The type is wrong its uint64_t.

>    ewbuf[0] = (int64_t)action_code;
>    int maxevents = 1;
>    int timeout = -1;
> 
>    fprintf(stdout, " Writing to Python \n%d", event_fd);
> 
>     write(event_fd, &ewbuf, 8);

> 
>     if (epoll_wait(epoll_fd, event_struc, maxevents, timeout) == -1)

Why is this here? Surely you need the python code to unblock because action_code is != 0.

>     {
>         fprintf(stderr, "Failed epoll_wait %s\n", strerror(errno));
>     }
> 
>     ssize_t rdval = read(event_fd, &ewbuf, 8);   

Why are you reading here? That is what you expect the python code to do.

Do you want to know if python has read the value written? If so then wait for the fd to be writeable...

> 
>     fprintf(stdout, " Received from Python \n%ld", rdval);

How do you know if the value read is from python?
Why isn't it the value you wrote above?

If you want two ways communications you need 2 eventfd's I'd guess.
One to allow you to send a >0 64 bit int to python.
One to allow python to send a >0 64 bit int to C.

--

What is the problem you are solving?

Is eventfd even the right solution to that problem?

Maybe you would be better off using Unix Domain Sockets that allow
you to send and receive a block of bytes.

Barry



> 
>     return 0;
> }
> 
> The C side initializes its epoll this way:
> 
> int epoll_initialize(int efd, int64_t * output_array)
> {
>   struct epoll_event ev = {};
>   int epoll_fd = epoll_create1(0);
> 
>   struct epoll_event * ptr_ev = &ev;
>   
>   if(epoll_fd == -1)
>   {
>     fprintf(stderr, "Failed to create epoll file descriptor\n");
>     return 1;
>   }
> 
>   ev.events = EPOLLIN | EPOLLOUT;
>   ev.data.fd = efd; //was 0
> 
>   if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, efd, &ev) == -1)
>   {
>       fprintf(stderr, "Failed to add file descriptor to epoll\n");
>       close(epoll_fd);
>       return 1;
>   }
> 
>   output_array[0] = epoll_fd;
>   output_array[1] = (int64_t)ptr_ev; //&ev;
> 
>   return 0;
> }
> 
> Technically C is not waiting for an EPOLLIN event, but again without it both processes freeze unless either C or Python includes both events.  So that appears to be where the problem is. 
> 
> The Linux epoll man page says, "epoll_wait waits for I/O events, blocking the calling thread if no events are currently available."   https://man7.org/linux/man-pages/man7/epoll.7.html <https://man7.org/linux/man-pages/man7/epoll.7.html>.  That may be the clue to why both processes freeze when I poll on only one event in each one. 
> 
> Thanks for any ideas based on this update, and thanks again for your earlier reply. 
> 
> Jen
> 
> 
> -- 
> Sent with Tutanota, the secure & ad-free mailbox. 
> 
> 
> 
> Nov 25, 2021, 06:34 by barry at barrys-emacs.org:
> 
> 
>> On 24 Nov 2021, at 22:42, Jen via Python-list <python-list at python.org <mailto:python-list at python.org>> wrote:
>> 
>> I have a C program that uses fork-execv to run Python 3.10 in a child process, and I am using eventfd with epoll for IPC between them.  The eventfd file descriptor is created in C and passed to Python through execv.  Once the Python child process starts I print the file descriptor to verify that it is correct (it is).  
>> 
>> In this scenario C will write to the eventfd at intervals and Python will read the eventfd and take action based on the value in the eventfd.  But in the Python while True loop I get "BlockingIOError: [Errno 11] Resource temporarily unavailable" then with each new read it prints "Failed epoll_wait Bad file descriptor." 
>> 
>> This is the Python code:
>> 
>> #!/usr/bin/python3
>> import sys
>> import os
>> import select
>> 
>> print("Inside Python")
>> 
>> event_fd = int(sys.argv[3])
>> 
>> print("Eventfd received by Python")
>> print(event_fd)
>> 
>> ep = select.epoll(-1)
>> ep.register(event_fd, select.EPOLLIN | select.EPOLLOUT)
> 
> This says tell me if I can read or write to the event_fd.
> write will be allowed until the kernel buffers are full.
> 
> Usually you only add EPOLLOUT if you have data to write.
> In this case do not set EPOLLOUT.
> 
> And if you know that you will never fill the kernel buffers then you
> do not need to bother polling for write.
> 
>> 
>> event_write_value = 100
>> 
>> while True:
>> 
>>     print("Waiting in Python for event")
>>     ep.poll(timeout=None, maxevents=- 1)
> 
> You have to get the result of the poll() and process the list of entries that are returned.
> 
> You must check that POLLIN is set before attempting the read.
> 
> 
>>     v = os.eventfd_read(event_fd)
> 
> Will raise EWOULDBLOCK because there is no data available to read.
> 
> Here is the docs from python:
> 
> poll.poll([timeout]) <>
> Polls the set of registered file descriptors, and returns a possibly-empty list containing (fd, event) 2-tuples for the descriptors that have events or errors to report. fd is the file descriptor, and event is a bitmask with bits set for the reported events for that descriptor ? POLLIN for waiting input, POLLOUT to indicate that the descriptor can be written to, and so forth. An empty list indicates that the call timed out and no file descriptors had any events to report. If timeout is given, it specifies the length of time in milliseconds which the system will wait for events before returning. If timeout is omitted, negative, or None <>, the call will block until there is an event for this poll object.
> 
> You end up with code like this:
> 
> for fd_event in ep.poll():
> fd, event == fd_event
> if (event&select.POLLIN) != 0 and fd == event_fd:
> v = os.eventfd_read(event_fd)
> 
>> 
>>     if v != 99:
>>         print("found")
>>         print(v)
>>         os.eventfd_write(event_fd, event_write_value)
>> 
>>     if v == 99:
>>         os.close(event_fd)
>> 
>> This is the C code that writes to Python, then waits for Python to write back:
>> 
>> ssize_t epoll_write(int event_fd, int epoll_fd, struct epoll_event * event_struc, int action_code)
>> {
>>    int64_t ewbuf[1];
>>    ewbuf[0] = (int64_t)action_code;
>>    int maxevents = 1;
>>    int timeout = -1;
>> 
>>    fprintf(stdout, " Writing to Python \n%d", event_fd);
>> 
>>    write(event_fd, &ewbuf, 8);
>> 
>>     if (epoll_wait(epoll_fd, event_struc, maxevents, timeout) == -1)
>>     {
>>         fprintf(stderr, "Failed epoll_wait %s\n", strerror(errno));
>>     }
>> 
>>     ssize_t rdval = read(event_fd, &ewbuf, 8);   
>> 
>>     fprintf(stdout, " Received from Python \n%ld", rdval);
>> 
>>     return 0;
>> }
>> 
>> This is the screen output when I run with gdb:
>> 
>>           Inside Python
>> Eventfd received by Python
>> 5
>> Waiting in Python for event
>> Traceback (most recent call last):
>>   File "/usr/local/lib/python3.10/runpy.py", line 196, in  _run_module_as_main
>>     return _run_code(code, main_globals, None,
>>   File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
>>     exec(code, run_globals)
>>   File "/opt/P01_SH/NPC_CPython.py", line 36, in <module>
>>     v = os.eventfd_read(event_fd)
>> BlockingIOError: [Errno 11] Resource temporarily unavailable
> 
> Expected as there you have not checked that there is data to read.
> Check for POLLIN being set.
> 
>> Writing to Python
>> 5 Received from Python
>> 8 Writing to Python
>> Failed epoll_wait Bad file descriptor
>> 5 Received from Python
>> 8 Writing to Python
>> Failed epoll_wait Bad file descriptor
>> 5 Received from Python
>> -1time taken 0.000548
>> Failed to close epoll file descriptor
>> Unlink_shm status: Bad file descriptor
>> fn() took 0.000648 seconds to execute
>> [Inferior 1 (process 12618) exited normally]
>> (gdb)
>> 
>> So my question is why do I get "BlockingIOError: [Errno 11] Resource temporarily unavailable" and "Failed epoll_wait Bad file descriptor" from Python? 
> 
> If your protocol is not trivia you should implement a state machine to know what to do at each event.
> 
> Barry
> 
>> 
>> -- 
>> Sent with Tutanota, the secure & ad-free mailbox. 
>> -- 
>> https://mail.python.org/mailman/listinfo/python-list <https://mail.python.org/mailman/listinfo/python-list>
> 
> 


From framstag at rus.uni-stuttgart.de  Sat Nov 27 13:27:23 2021
From: framstag at rus.uni-stuttgart.de (Ulli Horlacher)
Date: Sat, 27 Nov 2021 18:27:23 +0000 (UTC)
Subject: CERTIFICATE_VERIFY_FAILED Windows only?
Message-ID: <snttab$a1o$1@news2.informatik.uni-stuttgart.de>

My program uses https and runs fine on Linux, but on Windows it crashes:

W10dev:/cygdrive/p: python fextasy.py -D
DEBUG(fextasy.py): verbose=0
DEBUG(fextasy.py): User-Agent: fextasy-20211127_1806 Windows 10.0.19041
DEBUG(fextasy.py): TCPCONNECT to fex.flupp.org:443
Traceback (most recent call last):
  File "P:\fextasy.py", line 1351, in <module>
    main()
  File "P:\fextasy.py", line 232, in main
    file = fexget('')
  File "P:\fextasy.py", line 622, in fexget
    if not http_connect(server,port):
  File "P:\fextasy.py", line 956, in http_connect
    if not tcp_connect(server,port): return
  File "P:\fextasy.py", line 973, in tcp_connect
    sock = context.wrap_socket(sock,server_hostname=host)
  File "C:\Users\admin\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 512, in wrap_socket
    return self.sslsocket_class._create(
  File "C:\Users\admin\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 1070, in _create
    self.do_handshake()
  File "C:\Users\admin\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 1341, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_s
sl.c:997)


The sourcecode here is:

def tcp_connect(host,port):
  global sock

  message(f"D:TCPCONNECT to {host}:{port}")
  try:
    sock = socket.create_connection((host,port))
  except socket.error as e:
    message("E:cannot connect to %s:%d - %s" % (host,port,e.strerror))
    return False
  sock.settimeout(timeout)
  if port == 443:
    context = ssl.create_default_context()
    sock = context.wrap_socket(sock,server_hostname=host)
  return True



Google chrome and firefox both say the certifacte is valid:

https://fex.flupp.org/fop/U4xC4kz8/X-20211127192031.png

https://fex.flupp.org/fop/mBabXKSz/X-20211127192416.png

Why does Python complain (only on Windows!)?



-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher at tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/

From rosuav at gmail.com  Sat Nov 27 16:59:22 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 28 Nov 2021 08:59:22 +1100
Subject: CERTIFICATE_VERIFY_FAILED Windows only?
In-Reply-To: <snttab$a1o$1@news2.informatik.uni-stuttgart.de>
References: <snttab$a1o$1@news2.informatik.uni-stuttgart.de>
Message-ID: <CAPTjJmqOobui8NNiE+W2gKSDTJpuqz8tkTS4Do49+LCT+=1zQg@mail.gmail.com>

On Sun, Nov 28, 2021 at 6:38 AM Ulli Horlacher
<framstag at rus.uni-stuttgart.de> wrote:
>
> My program uses https and runs fine on Linux, but on Windows it crashes:
>
> Google chrome and firefox both say the certifacte is valid:
>
> https://fex.flupp.org/fop/U4xC4kz8/X-20211127192031.png
>
> https://fex.flupp.org/fop/mBabXKSz/X-20211127192416.png
>
> Why does Python complain (only on Windows!)?
>

What version of Python is it, and where did you install it from? On
some versions, Python will use Microsoft's provided certificate store.
One solution may be to fetch Mozilla's root certs from PyPI:

https://pypi.org/project/certifi/

ChrisA

From arj.python at gmail.com  Sun Nov 28 08:08:51 2021
From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer)
Date: Sun, 28 Nov 2021 17:08:51 +0400
Subject: print('\N{flag: Mauritius}') not supported in py3.9
Message-ID: <CADrxXXkQcFMJhAt5AGt=BL=QqHCgJ7qbx=ts0zR8H9s2L1ZMdg@mail.gmail.com>

Mike Driscoll printed this on Twitter

>>> print('\N{Sauropod}')
?

Using py3.9 i got the above.

I found the whole CLDR short name here:
https://unicode.org/emoji/charts/full-emoji-list.html

However when i do

>>> print('\N{flag: Mauritius}')
  File "<stdin>", line 1
    print('\N{flag: Mauritius}')
                               ^
i get

SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in
position 0-18: unknown Unicode character name

So is it that Python3.9 does not support it or what is the issue here?
Thanks

Kind Regards,

Abdur-Rahmaan Janhangeer
about <https://compileralchemy.github.io/> | blog
<https://www.pythonkitchen.com>
github <https://github.com/Abdur-RahmaanJ>
Mauritius

From rosuav at gmail.com  Sun Nov 28 08:18:40 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 29 Nov 2021 00:18:40 +1100
Subject: print('\N{flag: Mauritius}') not supported in py3.9
In-Reply-To: <CADrxXXkQcFMJhAt5AGt=BL=QqHCgJ7qbx=ts0zR8H9s2L1ZMdg@mail.gmail.com>
References: <CADrxXXkQcFMJhAt5AGt=BL=QqHCgJ7qbx=ts0zR8H9s2L1ZMdg@mail.gmail.com>
Message-ID: <CAPTjJmoxxRU=-uA0_RJnocZFm6WG0+OpZXFM3TDX2Tgcr1=WBg@mail.gmail.com>

On Mon, Nov 29, 2021 at 12:10 AM Abdur-Rahmaan Janhangeer
<arj.python at gmail.com> wrote:
> I found the whole CLDR short name here:
> https://unicode.org/emoji/charts/full-emoji-list.html
>
> However when i do
>
> >>> print('\N{flag: Mauritius}')
>   File "<stdin>", line 1
>     print('\N{flag: Mauritius}')
>                                ^
> i get
>
> SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in
> position 0-18: unknown Unicode character name
>
> So is it that Python3.9 does not support it or what is the issue here?

Flags are actually constructed from multiple codepoints. What you want
is to insert each codepoint separately. You can see them listed in the
second column of the table you linked to.

>>> "\U0001F1F2\U0001F1FA"
'??'

To do this with names, you need the names of those two codepoints:

'\U0001f1f2' REGIONAL INDICATOR SYMBOL LETTER M
'\U0001f1fa' REGIONAL INDICATOR SYMBOL LETTER U

>>> "\N{REGIONAL INDICATOR SYMBOL LETTER M}\N{REGIONAL INDICATOR SYMBOL LETTER U}"
'??'

ChrisA

From arj.python at gmail.com  Sun Nov 28 08:29:07 2021
From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer)
Date: Sun, 28 Nov 2021 17:29:07 +0400
Subject: print('\N{flag: Mauritius}') not supported in py3.9
In-Reply-To: <CAPTjJmoxxRU=-uA0_RJnocZFm6WG0+OpZXFM3TDX2Tgcr1=WBg@mail.gmail.com>
References: <CADrxXXkQcFMJhAt5AGt=BL=QqHCgJ7qbx=ts0zR8H9s2L1ZMdg@mail.gmail.com>
 <CAPTjJmoxxRU=-uA0_RJnocZFm6WG0+OpZXFM3TDX2Tgcr1=WBg@mail.gmail.com>
Message-ID: <CADrxXXkXEyuEo9ciW=puFAtPqAz1sc9hWTX7MCoU0n9tGkh9Uw@mail.gmail.com>

Greetings,

I get you but why do the short names work for some and not for
others?

Kind Regards,

Abdur-Rahmaan Janhangeer
about <https://compileralchemy.github.io/> | blog
<https://www.pythonkitchen.com>
github <https://github.com/Abdur-RahmaanJ>
Mauritius

From rosuav at gmail.com  Sun Nov 28 09:02:40 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 29 Nov 2021 01:02:40 +1100
Subject: print('\N{flag: Mauritius}') not supported in py3.9
In-Reply-To: <CADrxXXkXEyuEo9ciW=puFAtPqAz1sc9hWTX7MCoU0n9tGkh9Uw@mail.gmail.com>
References: <CADrxXXkQcFMJhAt5AGt=BL=QqHCgJ7qbx=ts0zR8H9s2L1ZMdg@mail.gmail.com>
 <CAPTjJmoxxRU=-uA0_RJnocZFm6WG0+OpZXFM3TDX2Tgcr1=WBg@mail.gmail.com>
 <CADrxXXkXEyuEo9ciW=puFAtPqAz1sc9hWTX7MCoU0n9tGkh9Uw@mail.gmail.com>
Message-ID: <CAPTjJmrBD_zyg7kryTZqSHA+j7naKmVtzEW6PGJYs_p109yVvw@mail.gmail.com>

On Mon, Nov 29, 2021 at 12:29 AM Abdur-Rahmaan Janhangeer
<arj.python at gmail.com> wrote:
>
> Greetings,
>
> I get you but why do the short names work for some and not for
> others?
>

Which ones work? The ones that can be identified by a single
codepoint? Look at the specification for Python's \N escapes.

ChrisA

From arj.python at gmail.com  Sun Nov 28 11:16:56 2021
From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer)
Date: Sun, 28 Nov 2021 20:16:56 +0400
Subject: print('\N{flag: Mauritius}') not supported in py3.9
In-Reply-To: <CAPTjJmrBD_zyg7kryTZqSHA+j7naKmVtzEW6PGJYs_p109yVvw@mail.gmail.com>
References: <CADrxXXkQcFMJhAt5AGt=BL=QqHCgJ7qbx=ts0zR8H9s2L1ZMdg@mail.gmail.com>
 <CAPTjJmoxxRU=-uA0_RJnocZFm6WG0+OpZXFM3TDX2Tgcr1=WBg@mail.gmail.com>
 <CADrxXXkXEyuEo9ciW=puFAtPqAz1sc9hWTX7MCoU0n9tGkh9Uw@mail.gmail.com>
 <CAPTjJmrBD_zyg7kryTZqSHA+j7naKmVtzEW6PGJYs_p109yVvw@mail.gmail.com>
Message-ID: <CADrxXXmfceg1ROwB4CB_YcnQRw8EOgvDK1PExdEEBTyvs=2FFQ@mail.gmail.com>

Greetings,


But why is it so?

From framstag at rus.uni-stuttgart.de  Sat Nov 27 19:27:52 2021
From: framstag at rus.uni-stuttgart.de (Ulli Horlacher)
Date: Sun, 28 Nov 2021 00:27:52 +0000 (UTC)
Subject: CERTIFICATE_VERIFY_FAILED Windows only?
References: <snttab$a1o$1@news2.informatik.uni-stuttgart.de>
 <CAPTjJmqOobui8NNiE+W2gKSDTJpuqz8tkTS4Do49+LCT+=1zQg@mail.gmail.com>
 <mailman.32.1638050375.9869.python-list@python.org>
Message-ID: <snuie8$fb4$1@news2.informatik.uni-stuttgart.de>

Chris Angelico <rosuav at gmail.com> wrote:
> On Sun, Nov 28, 2021 at 6:38 AM Ulli Horlacher
> <framstag at rus.uni-stuttgart.de> wrote:
> >
> > My program uses https and runs fine on Linux, but on Windows it crashes:
> >
> > Google chrome and firefox both say the certifacte is valid:
> >
> > https://fex.flupp.org/fop/U4xC4kz8/X-20211127192031.png
> >
> > https://fex.flupp.org/fop/mBabXKSz/X-20211127192416.png
> >
> > Why does Python complain (only on Windows!)?
> >
> 
> What version of Python is it, and where did you install it from?

https://www.python.org/ftp/python/3.10.0/python-3.10.0-amd64.exe


> On some versions, Python will use Microsoft's provided certificate store.
> One solution may be to fetch Mozilla's root certs from PyPI:
> 
> https://pypi.org/project/certifi/

C:\Users\admin>pip install certifi
Collecting certifi
  Downloading certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Installing collected packages: certifi
Successfully installed certifi-2021.10.8


Great!
Now my program runs without CERTIFICATE_VERIFY_FAILED


-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher at tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/

From cl at isbd.net  Sun Nov 28 09:20:13 2021
From: cl at isbd.net (Chris Green)
Date: Sun, 28 Nov 2021 14:20:13 +0000
Subject: Why is there no get_body() in mailbox.Maildir?
Message-ID: <tpfc7i-n9kc1.ln1@esprimo.zbmc.eu>

I have the following error:-

    chris at esprimo$ ./showmail.py boating/Odin/moorings/angers "01 Oct 2009 18:03:00"
    Traceback (most recent call last):
      File "/home/chris/dev/cgi-bin/./showmail.py", line 24, in <module>
        print(msg.get_body('plain',))
    AttributeError: 'MaildirMessage' object has no attribute 'get_body'


It's being returned by the following (rather trivial) Python program:-

    #!/usr/bin/python3 
    # 
    # 
    # replacement for complex Zend/PHP E-Mail message extraction 
    # 
    import mailbox
    import email
    import sys
    import os

    mbroot = "/home/chris/mail/folder"

    mbname = os.path.join(mbroot, sys.argv[1])

    mb = mailbox.Maildir(mbname)
    # 
    # 
    # run through the messages in the mailbox looking for date/time match 
    # 
    for msg in mb:
        # print(msg.get("Subject")) 
        # print(msg.get("Date")) 
        if sys.argv[2] in  msg.get("Date"):                  
            print(msg.get_body('plain',)) 
            exit(0) 
    exit(1) 

I thought MaildirMessage inherited from email.message, it has inherited the
get() method so why not the get_body() method?  I'm running Python 3.9.7
on xubuntu Linux 21.10.

Is there another way of doing this or do I have to sort through the message
parts and stuff?  It sounded like get_body() would make things easier.


-- 
Chris Green
?

From cl at isbd.net  Sun Nov 28 11:14:42 2021
From: cl at isbd.net (Chris Green)
Date: Sun, 28 Nov 2021 16:14:42 +0000
Subject: How to decode ISO8859-1 in Python 3?
Message-ID: <igmc7i-ttcd1.ln1@esprimo.zbmc.eu>

This sounds as if it should be trivial but searching only seems to
produce ways ofd doing it in Python 2.

I have some text files which are ISO8859-1 encoded and I want to output
them to screen using Python.

-- 
Chris Green
?

From PythonList at DancesWithMice.info  Sun Nov 28 16:08:29 2021
From: PythonList at DancesWithMice.info (dn)
Date: Mon, 29 Nov 2021 10:08:29 +1300
Subject: print('\N{flag: Mauritius}') not supported in py3.9
In-Reply-To: <CAPTjJmoxxRU=-uA0_RJnocZFm6WG0+OpZXFM3TDX2Tgcr1=WBg@mail.gmail.com>
References: <CADrxXXkQcFMJhAt5AGt=BL=QqHCgJ7qbx=ts0zR8H9s2L1ZMdg@mail.gmail.com>
 <CAPTjJmoxxRU=-uA0_RJnocZFm6WG0+OpZXFM3TDX2Tgcr1=WBg@mail.gmail.com>
Message-ID: <c4f62875-5c4e-055e-cc5d-7a555e25e88f@DancesWithMice.info>

On 29/11/2021 02.18, Chris Angelico wrote:
> On Mon, Nov 29, 2021 at 12:10 AM Abdur-Rahmaan Janhangeer
> <arj.python at gmail.com> wrote:
> 
> Flags are actually constructed from multiple codepoints. What you want
> is to insert each codepoint separately. You can see them listed in the
> second column of the table you linked to.
> 
>>>> "\U0001F1F2\U0001F1FA"
> '??'
> 
> To do this with names, you need the names of those two codepoints:
> 
> '\U0001f1f2' REGIONAL INDICATOR SYMBOL LETTER M
> '\U0001f1fa' REGIONAL INDICATOR SYMBOL LETTER U
> 
>>>> "\N{REGIONAL INDICATOR SYMBOL LETTER M}\N{REGIONAL INDICATOR SYMBOL LETTER U}"
> '??'


Don't use Emojis that often. The colored circles (U+1F534 etc) display
in full, glorious, technicolor.

However, when trying the above, with our local flag in (Fedora Linux,
Gnome) Terminal or PyCharm's Run terminal; the two letters "N" and "Z"
are shown with dotted-outlines. Similarly, the Mauritius' flag is shown
as "M" and "U".

Whereas here in email (Thunderbird) or in a web-browser, the flags
appear, as desired.

Is this a terminal short-coming (locale charmap -> UTF-8 - which brings
to mind the old UCS-4 questions), a font issue, or what (to fix)?
-- 
Regards,
=dn

From cl at isbd.net  Sun Nov 28 11:53:57 2021
From: cl at isbd.net (Chris Green)
Date: Sun, 28 Nov 2021 16:53:57 +0000
Subject: How to decode ISO8859-1 in Python 3?
References: <igmc7i-ttcd1.ln1@esprimo.zbmc.eu>
 <read-iso8859-1-20211128173223@ram.dialup.fu-berlin.de>
Message-ID: <5qoc7i-s7jd1.ln1@esprimo.zbmc.eu>

Stefan Ram <ram at zedat.fu-berlin.de> wrote:
> Chris Green <cl at isbd.net> writes:
> >I have some text files which are ISO8859-1 encoded and I want to output
> >them to screen using Python.
> 
>   Well, the first attempt would be:
> 
> import pathlib
> name = r"C:\example.txt"
> encoding = r"iso8859-1"
> path = pathlib.Path( name )
> with path.open( encoding=encoding, errors="ignore" )as file:
>    print( file.read() )
> 
>   , but if you write to a console which needs a special encoding,
>   you might need:
> 
> sys.stdout.buffer.write( file.read().encode( 'your screen encoding' ))
> 
>   (which is not always possible) instead of "print( file.read())", 
>   and "import sys" at the start of the script.
> 
Thanks!  It's rather messy isn't it!  :-)

I think I'll try converting the files before letting Python loose on
them, iconv should manage that.

-- 
Chris Green
?

From cl at isbd.net  Sun Nov 28 11:52:11 2021
From: cl at isbd.net (Chris Green)
Date: Sun, 28 Nov 2021 16:52:11 +0000
Subject: Why is there no get_body() in mailbox.Maildir?
References: <tpfc7i-n9kc1.ln1@esprimo.zbmc.eu>
 <get_body-20211128171128@ram.dialup.fu-berlin.de>
Message-ID: <rmoc7i-s7jd1.ln1@esprimo.zbmc.eu>

Stefan Ram <ram at zedat.fu-berlin.de> wrote:
> Chris Green <cl at isbd.net> writes:
> >I thought MaildirMessage inherited from email.message, 
> 
>   It inherits from mailbox.Message which inherits from
>   email.message.Message which has no "get_body" method.
> 
So the documentation at:-
    https://docs.python.org/3/library/email.message.html

is very confusing then.  It has a get_body method listed in the
email.message.EmailMessage class.

Where is email.message.Message documented?


>   email.message.MIMEPart has a get_body method.
>  
>   Also check out: get_payload.
> 
> 
-- 
Chris Green
?

From rosuav at gmail.com  Sun Nov 28 17:19:16 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 29 Nov 2021 09:19:16 +1100
Subject: print('\N{flag: Mauritius}') not supported in py3.9
In-Reply-To: <c4f62875-5c4e-055e-cc5d-7a555e25e88f@DancesWithMice.info>
References: <CADrxXXkQcFMJhAt5AGt=BL=QqHCgJ7qbx=ts0zR8H9s2L1ZMdg@mail.gmail.com>
 <CAPTjJmoxxRU=-uA0_RJnocZFm6WG0+OpZXFM3TDX2Tgcr1=WBg@mail.gmail.com>
 <c4f62875-5c4e-055e-cc5d-7a555e25e88f@DancesWithMice.info>
Message-ID: <CAPTjJmoaM74QUXpqtOV_BHiYrJyfdpreEoHdYzjveeZSOvaofA@mail.gmail.com>

On Mon, Nov 29, 2021 at 8:10 AM dn via Python-list
<python-list at python.org> wrote:
> However, when trying the above, with our local flag in (Fedora Linux,
> Gnome) Terminal or PyCharm's Run terminal; the two letters "N" and "Z"
> are shown with dotted-outlines. Similarly, the Mauritius' flag is shown
> as "M" and "U".
>
> Whereas here in email (Thunderbird) or in a web-browser, the flags
> appear, as desired.
>
> Is this a terminal short-coming (locale charmap -> UTF-8 - which brings
> to mind the old UCS-4 questions), a font issue, or what (to fix)?

Probably a font issue. Not many fonts support the flags.

ChrisA

From cs at cskk.id.au  Sun Nov 28 18:06:00 2021
From: cs at cskk.id.au (Cameron Simpson)
Date: Mon, 29 Nov 2021 10:06:00 +1100
Subject: print('\N{flag: Mauritius}') not supported in py3.9
In-Reply-To: <CAPTjJmoaM74QUXpqtOV_BHiYrJyfdpreEoHdYzjveeZSOvaofA@mail.gmail.com>
References: <CAPTjJmoaM74QUXpqtOV_BHiYrJyfdpreEoHdYzjveeZSOvaofA@mail.gmail.com>
Message-ID: <YaQLWEaV3+KWBgzl@cskk.homeip.net>

On 29Nov2021 09:19, Chris Angelico <rosuav at gmail.com> wrote:
>On Mon, Nov 29, 2021 at 8:10 AM dn via Python-list
><python-list at python.org> wrote:
>> However, when trying the above, with our local flag in (Fedora Linux,
>> Gnome) Terminal or PyCharm's Run terminal; the two letters "N" and "Z"
>> are shown with dotted-outlines. Similarly, the Mauritius' flag is shown
>> as "M" and "U".
>>
>> Whereas here in email (Thunderbird) or in a web-browser, the flags
>> appear, as desired.
>>
>> Is this a terminal short-coming (locale charmap -> UTF-8 - which brings
>> to mind the old UCS-4 questions), a font issue, or what (to fix)?
>
>Probably a font issue. Not many fonts support the flags.

Agree about the font support. Some terminal emulators make an effort to 
have fallback fonts for when your preferred font lacks a glyph. IIRC 
urxvt is such a terminal on Linux.

Cheers,
Cameron Simpson <cs at cskk.id.au>

From tony.flury at btinternet.com  Sun Nov 28 18:52:05 2021
From: tony.flury at btinternet.com (Tony Flury)
Date: Sun, 28 Nov 2021 23:52:05 +0000
Subject: pyinstaller wrong classified as Windows virus
In-Reply-To: <snog1a$u4v$1@news2.informatik.uni-stuttgart.de>
References: <snnkhf$mve$1@news2.informatik.uni-stuttgart.de>
 <slrnspur9f.51o.dan@djph.net>
 <sno092$prd$1@news2.informatik.uni-stuttgart.de>
 <sno3qt$r1b$1@news2.informatik.uni-stuttgart.de>
 <CAPTjJmq=w3iFDWjzNFAfpmAC=5kFZm1C8=Ex7Cwo2iwDVY_LHA@mail.gmail.com>
 <mailman.8.1637859246.9869.python-list@python.org>
 <snog1a$u4v$1@news2.informatik.uni-stuttgart.de>
Message-ID: <1a2f8847-49b5-c2ee-943f-cef46b9d5460@btinternet.com>

Have you tried using Nuitka - rather than pyInstalller - it means you 
distribute a single executable and the Python run time library (which 
they probably have already), and it has the advantage that it is a bit 
quicker than standard python.

Rather than bundle the source code and interpreter in single executable, 
Nuitka actually compiles the Python source code to native machine code 
(via a set of C files), and? this native executable uses the Python 
runtime library to implement the python features.It does rely on you 
having a Windows C compiler available.


On 25/11/2021 17:10, Ulli Horlacher wrote:
> Chris Angelico <rosuav at gmail.com> wrote:
>
>> Unfortunately, if you're not going to go to the effort of getting your
>> executables signed
> I cannot sign my executables (how can I do it anyway?), because Windows
> deletes my executable as soon as I have compiled them! They exist only
> for a few seconds and then they are gone.
>
>
>> another reason to just distribute .py files.
> I cannot do that because my users do not have Python installed and they
> are not allowed to do it.
>

From anthony.flury at btinternet.com  Sun Nov 28 18:53:20 2021
From: anthony.flury at btinternet.com (anthony.flury)
Date: Sun, 28 Nov 2021 23:53:20 +0000
Subject: pyinstaller wrong classified as Windows virus
In-Reply-To: <snog1a$u4v$1@news2.informatik.uni-stuttgart.de>
References: <snnkhf$mve$1@news2.informatik.uni-stuttgart.de>
 <slrnspur9f.51o.dan@djph.net>
 <sno092$prd$1@news2.informatik.uni-stuttgart.de>
 <sno3qt$r1b$1@news2.informatik.uni-stuttgart.de>
 <CAPTjJmq=w3iFDWjzNFAfpmAC=5kFZm1C8=Ex7Cwo2iwDVY_LHA@mail.gmail.com>
 <mailman.8.1637859246.9869.python-list@python.org>
 <snog1a$u4v$1@news2.informatik.uni-stuttgart.de>
Message-ID: <e0c30b15-d8e1-e0b0-40ae-d5e48bb4cedf@btinternet.com>

Have you tried using Nuitka - rather than pyInstalller - it means you 
distribute a single executable and the Python run time library (which 
they probably have already), and it has the advantage that it is a bit 
quicker than standard python.

Rather than bundle the source code and interpreter in single executable, 
Nuitka actually compiles the Python source code to native machine code 
(via a set of C files), and? this native executable uses the Python 
runtime library to implement the python features.It does rely on you 
having a Windows C compiler available.

On 25/11/2021 17:10, Ulli Horlacher wrote:
> Chris Angelico <rosuav at gmail.com> wrote:
>
>> Unfortunately, if you're not going to go to the effort of getting your
>> executables signed
> I cannot sign my executables (how can I do it anyway?), because Windows
> deletes my executable as soon as I have compiled them! They exist only
> for a few seconds and then they are gone.
>
>
>> another reason to just distribute .py files.
> I cannot do that because my users do not have Python installed and they
> are not allowed to do it.
>
-- 
Anthony Flury
*Moble*: +44 07743 282707
*Home*: +44 (0)1206 391294
*email*: anthony.flury at btinternet.com <mailto:anthony.flury at btinternet.com>

From anthony.flury at btinternet.com  Sun Nov 28 18:58:12 2021
From: anthony.flury at btinternet.com (anthony.flury)
Date: Sun, 28 Nov 2021 23:58:12 +0000
Subject: pyinstaller wrong classified as Windows virus
In-Reply-To: <snq1eq$amo$1@news2.informatik.uni-stuttgart.de>
References: <snnkhf$mve$1@news2.informatik.uni-stuttgart.de>
 <slrnspur9f.51o.dan@djph.net>
 <sno092$prd$1@news2.informatik.uni-stuttgart.de>
 <sno3qt$r1b$1@news2.informatik.uni-stuttgart.de>
 <CAPTjJmq=w3iFDWjzNFAfpmAC=5kFZm1C8=Ex7Cwo2iwDVY_LHA@mail.gmail.com>
 <mailman.8.1637859246.9869.python-list@python.org>
 <snog1a$u4v$1@news2.informatik.uni-stuttgart.de>
 <007701d7e252$9402b500$bc081f00$@verizon.net>
 <mailman.18.1637882241.9869.python-list@python.org>
 <snq1eq$amo$1@news2.informatik.uni-stuttgart.de>
Message-ID: <3f687d0a-48ce-febb-76b7-17d19f3ad1a5@btinternet.com>


On 26/11/2021 07:13, Ulli Horlacher wrote
>> But consider another possibility that your compiler software is compromised
> Then https://www.python.org/ftp/python/3.10.0/python-3.10.0-amd64.exe
> is infected. I doubt this.

But you aren't using python3.10 to 'compile' the code to the executable 
that windows complains about: you are using pyinstaller, which if memory 
serves is a 3rd party application.

I assume that you have no problem running the script without pyinstaller ?

so Might pyinstaller be compromised in some way ?


>                                            
>> Is this happening to only one set of code?
> This is happening SOMETIMES, not always. With the SAME source code. When I
> call pyinstaller often enough, then the virus scanner is quiet. In about 1
> of 20 compile runs.
>
>
>
-- 
Anthony Flury
*Moble*: +44 07743 282707
*Home*: +44 (0)1206 391294
*email*: anthony.flury at btinternet.com <mailto:anthony.flury at btinternet.com>

From PythonList at DancesWithMice.info  Mon Nov 29 04:25:41 2021
From: PythonList at DancesWithMice.info (dn)
Date: Mon, 29 Nov 2021 22:25:41 +1300
Subject: print('\N{flag: Mauritius}') not supported in py3.9
In-Reply-To: <YaQLWEaV3+KWBgzl@cskk.homeip.net>
References: <CAPTjJmoaM74QUXpqtOV_BHiYrJyfdpreEoHdYzjveeZSOvaofA@mail.gmail.com>
 <YaQLWEaV3+KWBgzl@cskk.homeip.net>
Message-ID: <acaee9f0-b582-28ef-326a-bdda74b39a9c@DancesWithMice.info>

On 29/11/2021 12.06, Cameron Simpson wrote:
> On 29Nov2021 09:19, Chris Angelico <rosuav at gmail.com> wrote:
>> On Mon, Nov 29, 2021 at 8:10 AM dn via Python-list
>> <python-list at python.org> wrote:
>>> However, when trying the above, with our local flag in (Fedora Linux,
>>> Gnome) Terminal or PyCharm's Run terminal; the two letters "N" and "Z"
>>> are shown with dotted-outlines. Similarly, the Mauritius' flag is shown
>>> as "M" and "U".
>>>
>>> Whereas here in email (Thunderbird) or in a web-browser, the flags
>>> appear, as desired.
>>>
>>> Is this a terminal short-coming (locale charmap -> UTF-8 - which brings
>>> to mind the old UCS-4 questions), a font issue, or what (to fix)?
>>
>> Probably a font issue. Not many fonts support the flags.
> 
> Agree about the font support. Some terminal emulators make an effort to 
> have fallback fonts for when your preferred font lacks a glyph. IIRC 
> urxvt is such a terminal on Linux.


Not sure about this. Most other applications on this PC will display the
two countries' flags, as desired, eg Writer, web-browser, even xed
(basic text editor).

Accordingly, took @Cameron's advice. Leading to:

Gnome Terminal: won't display "\U0001F1F3\U0001F1FF" (etc)
Terminator: won't display
Tabby: doesn't seem to load from (rpm) repo
RoxTerm: no choice of fonts, won't display
rxvt: won't compile, gave-up fighting unfamiliar requirements
Terminology: offers choice of fonts, but still fails

Kitty: works!


Kitty is not something I've come-across before. Its write-up says
?
Kitty is a free, open-source, and fast, feature-rich, GPU accelerated
terminal emulator for Linux, that supports all present-day terminal
features, such as Unicode, true color, text formatting, bold/italic
fonts, tiling of multiple windows and tabs, etc.

Kitty is written in C and Python programming languages, and it is one of
few terminal emulators with GPU support
?


Yes, the one that 'works', is using the same fonts as (say) Writer, and
the original (Gnome) Terminal that fails.


Please don't take this as a scientific survey. I didn't spend any time
investigating - either the s/w worked or it didn't! However, a terminal
is doing a simple job (at the user-level), so there's not much to them
in terms of knobs to twiddle or levers to pull.
-- 
Regards,
=dn

From framstag at rus.uni-stuttgart.de  Mon Nov 29 02:46:11 2021
From: framstag at rus.uni-stuttgart.de (Ulli Horlacher)
Date: Mon, 29 Nov 2021 07:46:11 +0000 (UTC)
Subject: pyinstaller wrong classified as Windows virus
References: <snnkhf$mve$1@news2.informatik.uni-stuttgart.de>
 <slrnspur9f.51o.dan@djph.net>
 <sno092$prd$1@news2.informatik.uni-stuttgart.de>
 <sno3qt$r1b$1@news2.informatik.uni-stuttgart.de>
 <CAPTjJmq=w3iFDWjzNFAfpmAC=5kFZm1C8=Ex7Cwo2iwDVY_LHA@mail.gmail.com>
 <mailman.8.1637859246.9869.python-list@python.org>
 <snog1a$u4v$1@news2.informatik.uni-stuttgart.de>
 <007701d7e252$9402b500$bc081f00$@verizon.net>
 <mailman.18.1637882241.9869.python-list@python.org>
 <snq1eq$amo$1@news2.informatik.uni-stuttgart.de>
 <3f687d0a-48ce-febb-76b7-17d19f3ad1a5@btinternet.com>
 <mailman.43.1638144039.9869.python-list@python.org>
Message-ID: <so20g3$b5n$1@news2.informatik.uni-stuttgart.de>

anthony.flury <anthony.flury at btinternet.com> wrote:
> 
> On 26/11/2021 07:13, Ulli Horlacher wrote
> >> But consider another possibility that your compiler software is compromised
> > Then https://www.python.org/ftp/python/3.10.0/python-3.10.0-amd64.exe
> > is infected. I doubt this.
> 
> But you aren't using python3.10 to 'compile' the code to the executable 
> that windows complains about: you are using pyinstaller, which if memory 
> serves is a 3rd party application.
> 
> I assume that you have no problem running the script without pyinstaller ?
> 
> so Might pyinstaller be compromised in some way ?

Then is the pip repository compromised. I doubt this, too.
I have installed pyinstaller with: pip intall pyinstaller


-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher at tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/

From barry at barrys-emacs.org  Mon Nov 29 13:57:19 2021
From: barry at barrys-emacs.org (Barry)
Date: Mon, 29 Nov 2021 18:57:19 +0000
Subject: pyinstaller wrong classified as Windows virus
In-Reply-To: <3f687d0a-48ce-febb-76b7-17d19f3ad1a5@btinternet.com>
References: <3f687d0a-48ce-febb-76b7-17d19f3ad1a5@btinternet.com>
Message-ID: <28173055-EE61-45FB-8569-40EC0CE812CD@barrys-emacs.org>



> On 29 Nov 2021, at 00:03, anthony.flury via Python-list <python-list at python.org> wrote:
> 
> ?
> On 26/11/2021 07:13, Ulli Horlacher wrote
>>> But consider another possibility that your compiler software is compromised
>> Then https://www.python.org/ftp/python/3.10.0/python-3.10.0-amd64.exe
>> is infected. I doubt this.
> 
> But you aren't using python3.10 to 'compile' the code to the executable that windows complains about: you are using pyinstaller, which if memory serves is a 3rd party application.
> 
> I assume that you have no problem running the script without pyinstaller ?
> 
> so Might pyinstaller be compromised in some way ?

Not likely.

On windows pyinstall, and other tools like it, create .exe files on windows.
I would guess it?s that .exe that is triggering the malware detector false positive.

Barry
> 
> 
>>                                           
>>> Is this happening to only one set of code?
>> This is happening SOMETIMES, not always. With the SAME source code. When I
>> call pyinstaller often enough, then the virus scanner is quiet. In about 1
>> of 20 compile runs.
>> 
>> 
>> 
> -- 
> Anthony Flury
> *Moble*: +44 07743 282707
> *Home*: +44 (0)1206 391294
> *email*: anthony.flury at btinternet.com <mailto:anthony.flury at btinternet.com>
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 


From bschollnick at schollnick.net  Mon Nov 29 14:04:03 2021
From: bschollnick at schollnick.net (Benjamin Schollnick)
Date: Mon, 29 Nov 2021 14:04:03 -0500
Subject: pyinstaller wrong classified as Windows virus
In-Reply-To: <28173055-EE61-45FB-8569-40EC0CE812CD@barrys-emacs.org>
References: <3f687d0a-48ce-febb-76b7-17d19f3ad1a5@btinternet.com>
 <28173055-EE61-45FB-8569-40EC0CE812CD@barrys-emacs.org>
Message-ID: <0024700A-3D66-4146-BC37-356180B1B8FB@schollnick.net>

Windows Defender has a setting to also use ?Reputation Scoring?.

What that simply means is that WDef will report back a hash to microsoft which is then checked to see if it is known.  If it is known, then it has a reputation and based off that reputation Defender will either allow it to run or not.

But if there is no reputation (eg no one has ever run it), that?s suspicious.  And that?s what you are running into.  

You can submit the EXE to the defender team, which should allow it to operate properly without any issue.

	- Benjamin



> On Nov 29, 2021, at 1:57 PM, Barry <barry at barrys-emacs.org> wrote:
> 
> 
> 
>> On 29 Nov 2021, at 00:03, anthony.flury via Python-list <python-list at python.org> wrote:
>> 
>> ?
>> On 26/11/2021 07:13, Ulli Horlacher wrote
>>>> But consider another possibility that your compiler software is compromised
>>> Then https://www.python.org/ftp/python/3.10.0/python-3.10.0-amd64.exe
>>> is infected. I doubt this.
>> 
>> But you aren't using python3.10 to 'compile' the code to the executable that windows complains about: you are using pyinstaller, which if memory serves is a 3rd party application.
>> 
>> I assume that you have no problem running the script without pyinstaller ?
>> 
>> so Might pyinstaller be compromised in some way ?
> 
> Not likely.
> 
> On windows pyinstall, and other tools like it, create .exe files on windows.
> I would guess it?s that .exe that is triggering the malware detector false positive.
> 
> Barry
>> 
>> 
>>> 
>>>> Is this happening to only one set of code?
>>> This is happening SOMETIMES, not always. With the SAME source code. When I
>>> call pyinstaller often enough, then the virus scanner is quiet. In about 1
>>> of 20 compile runs.
>>> 
>>> 
>>> 
>> -- 
>> Anthony Flury
>> *Moble*: +44 07743 282707
>> *Home*: +44 (0)1206 391294
>> *email*: anthony.flury at btinternet.com <mailto:anthony.flury at btinternet.com>
>> -- 
>> https://mail.python.org/mailman/listinfo/python-list
>> 
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list


From petermwale47 at gmail.com  Mon Nov 29 13:43:31 2021
From: petermwale47 at gmail.com (Peter Mwale)
Date: Mon, 29 Nov 2021 20:43:31 +0200
Subject: Failure to Display Top menu
In-Reply-To: <snt1nv$3kh$1@dont-email.me>
References: <CAGihD5xiNq9v1hq8-sBZsYHhnao-b7_+s8uM6PD6mu_rnxc=6g@mail.gmail.com>
 <mailman.25.1637991960.9869.python-list@python.org>
 <snt1nv$3kh$1@dont-email.me>
Message-ID: <CAGihD5z8TcmEqRdLFO7unZHiwR55DVKnGxRy40RkfejdT2-NnQ@mail.gmail.com>

Hello Christian,
Am referring to menu marked in picture below. It's not coming on my window.


On Sat, Nov 27, 2021, 20:19 Christian Gollwitzer <auriocus at gmx.de> wrote:

> Am 26.11.21 um 21:38 schrieb Peter Mwale:
> > Hello, my python 3.10 shell is not displaying the top menu. What should I
> > do?
> >
>
> You should explain, what you do exactly. The Python interpreter does not
> have a menu.
>
> a) What platform are you on? Windows, macOS, Linux?
>
> b) How did ou start Python and what was the expectation?
>
>         Christian
> --
> https://mail.python.org/mailman/listinfo/python-list
>

From PythonList at DancesWithMice.info  Mon Nov 29 14:23:17 2021
From: PythonList at DancesWithMice.info (dn)
Date: Tue, 30 Nov 2021 08:23:17 +1300
Subject: Failure to Display Top menu
In-Reply-To: <CAGihD5z8TcmEqRdLFO7unZHiwR55DVKnGxRy40RkfejdT2-NnQ@mail.gmail.com>
References: <CAGihD5xiNq9v1hq8-sBZsYHhnao-b7_+s8uM6PD6mu_rnxc=6g@mail.gmail.com>
 <mailman.25.1637991960.9869.python-list@python.org>
 <snt1nv$3kh$1@dont-email.me>
 <CAGihD5z8TcmEqRdLFO7unZHiwR55DVKnGxRy40RkfejdT2-NnQ@mail.gmail.com>
Message-ID: <27f97f30-aba5-f42d-283d-f5b499dfd650@DancesWithMice.info>

On 30/11/2021 07.43, Peter Mwale wrote:
> Hello Christian,
> Am referring to menu marked in picture below. It's not coming on my window.
> 
> 
> On Sat, Nov 27, 2021, 20:19 Christian Gollwitzer <auriocus at gmx.de> wrote:
> 
>> Am 26.11.21 um 21:38 schrieb Peter Mwale:
>>> Hello, my python 3.10 shell is not displaying the top menu. What should I
>>> do?
>>>
>>
>> You should explain, what you do exactly. The Python interpreter does not
>> have a menu.
>>
>> a) What platform are you on? Windows, macOS, Linux?
>>
>> b) How did ou start Python and what was the expectation?


Sadly, this mailing list will not forward non-text attachments (for
security reasons).

Please start at https://docs.python.org/3/using/index.html with
particular attention to the chapter on MS-Windows.

This should answer your question.

If not, at least we will have some common terminology to be able to
express and solve any remaining problem...
-- 
Regards,
=dn

From cspealma at redhat.com  Mon Nov 29 14:24:14 2021
From: cspealma at redhat.com (Calvin Spealman)
Date: Mon, 29 Nov 2021 14:24:14 -0500
Subject: Failure to Display Top menu
In-Reply-To: <CAGihD5z8TcmEqRdLFO7unZHiwR55DVKnGxRy40RkfejdT2-NnQ@mail.gmail.com>
References: <CAGihD5xiNq9v1hq8-sBZsYHhnao-b7_+s8uM6PD6mu_rnxc=6g@mail.gmail.com>
 <mailman.25.1637991960.9869.python-list@python.org>
 <snt1nv$3kh$1@dont-email.me>
 <CAGihD5z8TcmEqRdLFO7unZHiwR55DVKnGxRy40RkfejdT2-NnQ@mail.gmail.com>
Message-ID: <CACo5Rz4FBYYSjy9geqek4N8wXPKAxJmEo7xOKqdq6QbLGe9NPw@mail.gmail.com>

This mailing list does not allow attachments. If you're trying to share a
screenshot, post it to imgur or something like that and share a URL here.

On Mon, Nov 29, 2021 at 2:12 PM Peter Mwale <petermwale47 at gmail.com> wrote:

> Hello Christian,
> Am referring to menu marked in picture below. It's not coming on my window.
>
>
> On Sat, Nov 27, 2021, 20:19 Christian Gollwitzer <auriocus at gmx.de> wrote:
>
> > Am 26.11.21 um 21:38 schrieb Peter Mwale:
> > > Hello, my python 3.10 shell is not displaying the top menu. What
> should I
> > > do?
> > >
> >
> > You should explain, what you do exactly. The Python interpreter does not
> > have a menu.
> >
> > a) What platform are you on? Windows, macOS, Linux?
> >
> > b) How did ou start Python and what was the expectation?
> >
> >         Christian
> > --
> > https://mail.python.org/mailman/listinfo/python-list
> >
> --
> https://mail.python.org/mailman/listinfo/python-list
>
>

-- 

CALVIN SPEALMAN

SENIOR QUALITY ENGINEER

calvin.spealman at redhat.com  M: +1.336.210.5107
[image: https://red.ht/sig] <https://red.ht/sig>
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>

From mats at wichmann.us  Mon Nov 29 14:34:02 2021
From: mats at wichmann.us (Mats Wichmann)
Date: Mon, 29 Nov 2021 12:34:02 -0700
Subject: pyinstaller wrong classified as Windows virus
In-Reply-To: <0024700A-3D66-4146-BC37-356180B1B8FB@schollnick.net>
References: <3f687d0a-48ce-febb-76b7-17d19f3ad1a5@btinternet.com>
 <28173055-EE61-45FB-8569-40EC0CE812CD@barrys-emacs.org>
 <0024700A-3D66-4146-BC37-356180B1B8FB@schollnick.net>
Message-ID: <5355b916-a2e5-c646-4331-960da1658474@wichmann.us>

On 11/29/21 12:04, Benjamin Schollnick wrote:
> Windows Defender has a setting to also use ?Reputation Scoring?.
> 
> What that simply means is that WDef will report back a hash to microsoft which is then checked to see if it is known.  If it is known, then it has a reputation and based off that reputation Defender will either allow it to run or not.
> 
> But if there is no reputation (eg no one has ever run it), that?s suspicious.  And that?s what you are running into.
> 
> You can submit the EXE to the defender team, which should allow it to operate properly without any issue.
> 
> 	- Benjamin

sure... "that's suspicious". Unless you're a developer compiling your 
own code. In which case every fresh build will be something "unknown". 
You have to set every setting you can find to "developer mode" to help 
with this kind of thing, and sometimes it's still not enough.



From bschollnick at schollnick.net  Mon Nov 29 14:41:00 2021
From: bschollnick at schollnick.net (Benjamin Schollnick)
Date: Mon, 29 Nov 2021 14:41:00 -0500
Subject: pyinstaller wrong classified as Windows virus
In-Reply-To: <5355b916-a2e5-c646-4331-960da1658474@wichmann.us>
References: <3f687d0a-48ce-febb-76b7-17d19f3ad1a5@btinternet.com>
 <28173055-EE61-45FB-8569-40EC0CE812CD@barrys-emacs.org>
 <0024700A-3D66-4146-BC37-356180B1B8FB@schollnick.net>
 <5355b916-a2e5-c646-4331-960da1658474@wichmann.us>
Message-ID: <696E0E6A-9E5D-4F29-96F8-B8ABA708864F@schollnick.net>

>> Windows Defender has a setting to also use ?Reputation Scoring?.
>> What that simply means is that WDef will report back a hash to microsoft which is then checked to see if it is known.  If it is known, then it has a reputation and based off that reputation Defender will either allow it to run or not.
>> But if there is no reputation (eg no one has ever run it), that?s suspicious.  And that?s what you are running into.
>> You can submit the EXE to the defender team, which should allow it to operate properly without any issue.
>> 	- Benjamin
> 
> sure... "that's suspicious". Unless you're a developer compiling your own code. In which case every fresh build will be something "unknown". You have to set every setting you can find to "developer mode" to help with this kind of thing, and sometimes it's still not enough.

Understandable, and Steve Gibson (GRC.com <http://grc.com/>, creator of Spinrite) has mentioned that he has had this problem with every Beta of his applications.

I agree completely with you, but that?s how Microsoft has set things up.

	- Benjamin



From PythonList at DancesWithMice.info  Mon Nov 29 14:44:03 2021
From: PythonList at DancesWithMice.info (dn)
Date: Tue, 30 Nov 2021 08:44:03 +1300
Subject: Negative subscripts
In-Reply-To: <CAPTjJmoHcEB5GJnEpmq-B18qz6eX2u8_qLhGkSKv0kr-V7cYcw@mail.gmail.com>
References: <96741f1a-72b3-6551-39b3-f1862c5a80b8@chagford.com>
 <ba4a976c-5829-03ac-2e3f-cf670f00c3f7@DancesWithMice.info>
 <1c37f528-250c-85b5-5e98-2ccde350438b@chagford.com>
 <c21483f1-9d14-968d-7d75-aa1ca3f90cc0@DancesWithMice.info>
 <CAPTjJmoHcEB5GJnEpmq-B18qz6eX2u8_qLhGkSKv0kr-V7cYcw@mail.gmail.com>
Message-ID: <90500dcf-fb29-4923-6004-d16e338068b8@DancesWithMice.info>



On 27/11/2021 21.23, Chris Angelico wrote:
> On Sat, Nov 27, 2021 at 7:21 PM dn via Python-list
> <python-list at python.org> wrote:
>> The expression list is evaluated once; it should yield an iterable
>> object. An iterator is created for the result of the expression_list.
>> The suite is then executed once for each item provided by the iterator,
>> in the order returned by the iterator. Each item in turn is assigned to
>> the target list using the standard rules for assignments (see Assignment
>> statements), and then the suite is executed. When the items are
>> exhausted (which is immediately when the sequence is empty or an
>> iterator raises a StopIteration exception), the suite in the else
>> clause, if present, is executed, and the loop terminates.
>> ?
>> https://docs.python.org/3/reference/compound_stmts.html#the-for-statement
>>
>>
>> That said, I'm wondering if all is strictly true when the
>> expression_list is a generator, which by definition features
>> lazy-execution rather than up-front list production. So, things may
>> depend upon the full-nature of the application.
> 
> Yes, it is. Evaluating a generator expression gives you a generator
> object, which is an iterable.


You (and the text) are correct - the expression list is "evaluated once"
and produces a generator object. For a particular understanding of
"evaluate".

However, as described, all that has been "evaluated" is a
generator-object. Unlike (say) a list's iterator, a generator only
eventually produces a 'list' - and each time the generator is
called-upon to yield the next value, that value has to be "evaluated",
ie there's some further evaluation loop-by-loop.

Further, that the values-returned can be amended during the life of the
generator (in ways anticipated and unanticipated by coder and
interpreter alike). Thus, it seems that the "list" doesn't actually
exist, as in, hasn't been "evaluated" as data-values, when the loop is
enacted.

What has been 'evaluated' are the terms by which the looping will
proceed, and terminate.


-- 
Regards,
=dn

From rosuav at gmail.com  Mon Nov 29 14:50:00 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 30 Nov 2021 06:50:00 +1100
Subject: Negative subscripts
In-Reply-To: <90500dcf-fb29-4923-6004-d16e338068b8@DancesWithMice.info>
References: <96741f1a-72b3-6551-39b3-f1862c5a80b8@chagford.com>
 <ba4a976c-5829-03ac-2e3f-cf670f00c3f7@DancesWithMice.info>
 <1c37f528-250c-85b5-5e98-2ccde350438b@chagford.com>
 <c21483f1-9d14-968d-7d75-aa1ca3f90cc0@DancesWithMice.info>
 <CAPTjJmoHcEB5GJnEpmq-B18qz6eX2u8_qLhGkSKv0kr-V7cYcw@mail.gmail.com>
 <90500dcf-fb29-4923-6004-d16e338068b8@DancesWithMice.info>
Message-ID: <CAPTjJmpGCKnUsheCft8pyoyF9RECvw9WxH_WjVFzBwqxp0tgCA@mail.gmail.com>

On Tue, Nov 30, 2021 at 6:45 AM dn via Python-list
<python-list at python.org> wrote:
>
>
>
> On 27/11/2021 21.23, Chris Angelico wrote:
> > On Sat, Nov 27, 2021 at 7:21 PM dn via Python-list
> > <python-list at python.org> wrote:
> >> The expression list is evaluated once; it should yield an iterable
> >> object. An iterator is created for the result of the expression_list.
> >> The suite is then executed once for each item provided by the iterator,
> >> in the order returned by the iterator. Each item in turn is assigned to
> >> the target list using the standard rules for assignments (see Assignment
> >> statements), and then the suite is executed. When the items are
> >> exhausted (which is immediately when the sequence is empty or an
> >> iterator raises a StopIteration exception), the suite in the else
> >> clause, if present, is executed, and the loop terminates.
> >> ?
> >> https://docs.python.org/3/reference/compound_stmts.html#the-for-statement
> >>
> >>
> >> That said, I'm wondering if all is strictly true when the
> >> expression_list is a generator, which by definition features
> >> lazy-execution rather than up-front list production. So, things may
> >> depend upon the full-nature of the application.
> >
> > Yes, it is. Evaluating a generator expression gives you a generator
> > object, which is an iterable.
>
>
> You (and the text) are correct - the expression list is "evaluated once"
> and produces a generator object. For a particular understanding of
> "evaluate".
>
> However, as described, all that has been "evaluated" is a
> generator-object. Unlike (say) a list's iterator, a generator only
> eventually produces a 'list' - and each time the generator is
> called-upon to yield the next value, that value has to be "evaluated",
> ie there's some further evaluation loop-by-loop.
>
> Further, that the values-returned can be amended during the life of the
> generator (in ways anticipated and unanticipated by coder and
> interpreter alike). Thus, it seems that the "list" doesn't actually
> exist, as in, hasn't been "evaluated" as data-values, when the loop is
> enacted.
>
> What has been 'evaluated' are the terms by which the looping will
> proceed, and terminate.
>

That's true of ALL iterators though. If you get a list iterator, and
while you're stepping through it, the underlying list changes, you'll
see those changes. Nothing gets "snapshot" unless you explicitly
request it (eg by slicing the list first).

ChrisA

From jenkris at tutanota.com  Mon Nov 29 15:34:08 2021
From: jenkris at tutanota.com (Jen Kris)
Date: Mon, 29 Nov 2021 21:34:08 +0100 (CET)
Subject: Python child process in while True loop blocks parent
Message-ID: <MphOu5i--3-2@tutanota.com>

I have a C program that forks to create a child process and uses execv to call a Python program.? The Python program communicates with the parent process (in C) through a FIFO pipe monitored with epoll().? 

The Python child process is in a while True loop, which is intended to keep it running while the parent process proceeds, and perform functions for the C program only at intervals when the parent sends data to the child -- similar to a daemon process.?

The C process writes to its end of the pipe and the child process reads it, but then the child process continues to loop, thereby blocking the parent.?

This is the Python code:

#!/usr/bin/python3
import os
import select

#Open the named pipes
pr = os.open('/tmp/Pipe_01', os.O_RDWR)
pw = os.open('/tmp/Pipe_02', os.O_RDWR)

ep = select.epoll(-1)
ep.register(pr, select.EPOLLIN)

while True:

??? events = ep.poll(timeout=2.5, maxevents=-1)
??? #events = ep.poll(timeout=None, maxevents=-1)

??? print("child is looping")

??? for fileno, event in events:
??????? print("Python fileno")
??????? print(fileno)
??????? print("Python event")
??????? print(event)
??????? v = os.read(pr,64)
??????? print("Pipe value")
??????? print(v)

The child process correctly receives the signal from ep.poll and correctly reads the data in the pipe, but then it continues looping.? For example, when I put in a timeout:

child is looping
Python fileno
4
Python event
1
Pipe value
b'10\x00'
child is looping
child is looping

That suggests that a while True loop is not the right thing to do in this case.? My question is, what type of process loop is best for this situation?? The multiprocessing, asyncio and subprocess libraries are very extensive, and it would help if someone could suggest the best alternative for what I am doing here.?

Thanks very much for any ideas.?



From cs at cskk.id.au  Mon Nov 29 16:10:54 2021
From: cs at cskk.id.au (Cameron Simpson)
Date: Tue, 30 Nov 2021 08:10:54 +1100
Subject: Python child process in while True loop blocks parent
In-Reply-To: <MphOu5i--3-2@tutanota.com>
References: <MphOu5i--3-2@tutanota.com>
Message-ID: <YaVB3oBgreBZDaO9@cskk.homeip.net>

On 29Nov2021 21:34, Jen Kris <jenkris at tutanota.com> wrote:
>I have a C program that forks to create a child process and uses execv to call a Python program.? The Python program communicates with the parent process (in C) through a FIFO pipe monitored with epoll().?
>
>The Python child process is in a while True loop, which is intended to keep it running while the parent process proceeds, and perform functions for the C program only at intervals when the parent sends data to the child -- similar to a daemon process.?
>
>The C process writes to its end of the pipe and the child process reads it, but then the child process continues to loop, thereby blocking the parent.?

It seems to me that the child Python process never writes anything back 
to the parent. If the parent is waiting for some response, of course it 
will be blocked.

Personally I wouldn't be using epoll at all. I'd just read data from pr, 
act on it, and write back to pw. That way the child blocks waiting for 
the parent istead of polling. You only want epoll if you're either 
polling something _while_ you do other work, or monitoring more than one 
thing. A plain read-from-one-pipe, work, write-back-to-the-other does 
not need it.

Cheers,
Cameron Simpson <cs at cskk.id.au>

From cs at cskk.id.au  Mon Nov 29 16:19:02 2021
From: cs at cskk.id.au (Cameron Simpson)
Date: Tue, 30 Nov 2021 08:19:02 +1100
Subject: print('\N{flag: Mauritius}') not supported in py3.9
In-Reply-To: <acaee9f0-b582-28ef-326a-bdda74b39a9c@DancesWithMice.info>
References: <acaee9f0-b582-28ef-326a-bdda74b39a9c@DancesWithMice.info>
Message-ID: <YaVDxsWMb6QqORoF@cskk.homeip.net>

On 29Nov2021 22:25, DL Neil <PythonList at DancesWithMice.info> wrote:
>>> Probably a font issue. Not many fonts support the flags.
>>
>> Agree about the font support. Some terminal emulators make an effort to
>> have fallback fonts for when your preferred font lacks a glyph. IIRC
>> urxvt is such a terminal on Linux.
>
>Not sure about this. Most other applications on this PC will display the
>two countries' flags, as desired, eg Writer, web-browser, even xed
>(basic text editor).

Seem Stefan Ram's advice, which points out that you can tell if this is 
a font problem (no flag glyph) or a combining problem (2 glyphs 
presented instead of one). I had not considered that.

>rxvt: won't compile, gave-up fighting unfamiliar requirements

See if there's a package for urxvt, which was the "unicode" flavour of 
rxvt (long ago - rxvt if probably supposed to be unicode capable these 
days, surely).

>Kitty: works!

Yay!

>Kitty is not something I've come-across before. Its write-up says
>?
>Kitty is a free, open-source, and fast, feature-rich, GPU accelerated
>terminal emulator for Linux, that supports all present-day terminal
>features, such as Unicode, true color, text formatting, bold/italic
>fonts, tiling of multiple windows and tabs, etc.

A tiling terminal emulator can be a great thing. I'm on a Mac with 
iTerm, which:
- has tabs
- has panes (split the view into multiple panels, each running a 
  terminal)

My personal dev desktop tends to use a full screen iTerm split 
vertically into 2 panes: an editor on the left (vim, itself split 
vertically into 2 vim windows) and a shell on the right; sometimes 
several shells (right hand pane further split horizontally).

Then, since I tend to keep per-branch checkouts around, tabs for the 
things I'm working on, each configured as above. Then I just switch tabs 
for the different areas.

Cheers,
Cameron Simpson <cs at cskk.id.au>

From Marco.Sulla.Python at gmail.com  Mon Nov 29 16:53:48 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Mon, 29 Nov 2021 22:53:48 +0100
Subject: frozenset can be altered by |=
In-Reply-To: <CABbU2U-rDQr0xkySeMsArbnox32TMdPO28SnbWtjbRhU7xzERA@mail.gmail.com>
References: <CABbU2U_79XPx1T6EGWEniYkN8_m0oOO6mxx+HNbaLqg_wmT-Gw@mail.gmail.com>
 <CAPTjJmq=4+yEhibTJoNA3QVUNR1HmScgnQgx3NtQRpgG4aNY1Q@mail.gmail.com>
 <AM0PR07MB57000CEF0CEABBD27EA941DA879F9@AM0PR07MB5700.eurprd07.prod.outlook.com>
 <CAPTjJmpprmCdjOJBz5FVecV6KqKFRC6Y7bOsNoTP+W-t-k3vew@mail.gmail.com>
 <CABbU2U-rDQr0xkySeMsArbnox32TMdPO28SnbWtjbRhU7xzERA@mail.gmail.com>
Message-ID: <CABbU2U_+t+Gg0EXaz3hc6V5xBG=TooCzMazWLLboM8fgSc1u5w@mail.gmail.com>

I must say that I'm reading the documentation now, and it's a bit
confusing. In the docs, inplace operators as |= should not work. They
are listed under the set-only functions and operators. But, as we saw,
this is not completely true: they work but they don't mutate the
original object. The same for += and *= that are listed under `list`
only.

On Mon, 22 Nov 2021 at 19:54, Marco Sulla <Marco.Sulla.Python at gmail.com> wrote:
>
> Yes, and you do this regularly. Indeed integers, for example, are immutables and
>
> a = 0
> a += 1
>
> is something you do dozens of times, and you simply don't think that
> another object is created and substituted for the variable named `a`.
>
> On Mon, 22 Nov 2021 at 14:59, Chris Angelico <rosuav at gmail.com> wrote:
> >
> > On Tue, Nov 23, 2021 at 12:52 AM David Raymond <David.Raymond at tomtom.com> wrote:
> > > It is a little confusing since the docs list this in a section that says they don't apply to frozensets, and lists the two versions next to each other as the same thing.
> > >
> > > https://docs.python.org/3.9/library/stdtypes.html#set-types-set-frozenset
> > >
> > > The following table lists operations available for set that do not apply to immutable instances of frozenset:
> > >
> > > update(*others)
> > > set |= other | ...
> > >
> > >     Update the set, adding elements from all others.
> >
> > Yeah, it's a little confusing, but at the language level, something
> > that doesn't support |= will implicitly support it using the expanded
> > version:
> >
> > a |= b
> > a = a | b
> >
> > and in the section above, you can see that frozensets DO support the
> > Or operator.
> >
> > By not having specific behaviour on the |= operator, frozensets
> > implicitly fall back on this default.
> >
> > ChrisA
> > --
> > https://mail.python.org/mailman/listinfo/python-list

From PythonList at DancesWithMice.info  Mon Nov 29 16:59:30 2021
From: PythonList at DancesWithMice.info (dn)
Date: Tue, 30 Nov 2021 10:59:30 +1300
Subject: print('\N{flag: Mauritius}') not supported in py3.9
In-Reply-To: <YaVDxsWMb6QqORoF@cskk.homeip.net>
References: <acaee9f0-b582-28ef-326a-bdda74b39a9c@DancesWithMice.info>
 <YaVDxsWMb6QqORoF@cskk.homeip.net>
Message-ID: <a511b1cf-c634-9302-fa9b-8fbe5d4ae050@DancesWithMice.info>

On 30/11/2021 10.19, Cameron Simpson wrote:
> On 29Nov2021 22:25, DL Neil <PythonList at DancesWithMice.info> wrote:
>>>> Probably a font issue. Not many fonts support the flags.
>>>
>>> Agree about the font support. Some terminal emulators make an effort to
>>> have fallback fonts for when your preferred font lacks a glyph. IIRC
>>> urxvt is such a terminal on Linux.
>>
>> Not sure about this. Most other applications on this PC will display the
>> two countries' flags, as desired, eg Writer, web-browser, even xed
>> (basic text editor).
> 
> Seem Stefan Ram's advice, which points out that you can tell if this is 
> a font problem (no flag glyph) or a combining problem (2 glyphs 
> presented instead of one). I had not considered that.

@Stefan's advice preceded by report that two glyphs were printed (not
one): "the two letters "N" and "Z" are shown with dotted-outlines"
(I think, the UTF-16 decoding)


>> rxvt: won't compile, gave-up fighting unfamiliar requirements
> 
> See if there's a package for urxvt, which was the "unicode" flavour of 
> rxvt (long ago - rxvt if probably supposed to be unicode capable these 
> days, surely).

Fedora names it as rxvt-unicode.
Installed v9.26
Text is too small for these old eyes.
No menu bar and no context menus.
Unable to copy-paste (the flag codes) into that window.
Removed.


>> Kitty: works!
> 
> Yay!
> 
>> Kitty is not something I've come-across before. Its write-up says
>> ?
>> Kitty is a free, open-source, and fast, feature-rich, GPU accelerated
>> terminal emulator for Linux, that supports all present-day terminal
>> features, such as Unicode, true color, text formatting, bold/italic
>> fonts, tiling of multiple windows and tabs, etc.
> 
> A tiling terminal emulator can be a great thing. I'm on a Mac with 
> iTerm, which:
> - has tabs
> - has panes (split the view into multiple panels, each running a 
>   terminal)
> 
> My personal dev desktop tends to use a full screen iTerm split 
> vertically into 2 panes: an editor on the left (vim, itself split 
> vertically into 2 vim windows) and a shell on the right; sometimes 
> several shells (right hand pane further split horizontally).
> 
> Then, since I tend to keep per-branch checkouts around, tabs for the 
> things I'm working on, each configured as above. Then I just switch tabs 
> for the different areas.


Yes, a good way to work. Neatly explained.

I choose not to even try to remember the difficulties of working with
small screens over-laying one another and 'getting lost' in the pile!

My desktop/dev.svr features two screens: one in 'portrait mode' and the
other 'landscape'. The former is very good for code-listings. The latter
for execution-display, pytest reports, etc. As you describe, each can be
sub-divided when needs-arise. A neat feature is that "Tool Windows" can
be tucked-away, and only 'pulled-out' when required, eg am not looking
at Sourcery's assessments right now (unless it highlights the commission
of some 'crime' as I type) but will check prior to (or as part of) git
commit. The Tool Window's name/label in the 'dock' also forms a reminder
that I shouldn't (totally) neglect my responsibilities!

These are managed within the IDE - sadly, PyCharm's terminal/console
fails the 'flag test', as reported yesterday. (am not sure if one might
be able to select which terminal emulator to use ...)

I'm a lazy toad. Thus the idea that the IDE will allow me to 'press a
(single) button' to repeat the last-run test/execute the code, without
me having to commit a Save and to jump between panels/windows/screens,
is seductive.

Don't tell my trainees! Every course sees three or four who 'cry for
help' because making some change in their choice of editor/IDE does not
result in similar within the web-browser. Did you forget to save the
source, Luke? That's not to say there won't be considerably more who
manage to diagnose the problem without admitting such to 'the outside
world'!


There are times when there is no need to (wait quite a while to) boot-up
a whole IDE, eg running a utility program. I've nominated Kitty as
Fedora's default terminal. We'll see how it goes with work-loads beyond
raising the flag...
Salute!
-- 
Regards,
=dn

From rosuav at gmail.com  Mon Nov 29 17:01:22 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 30 Nov 2021 09:01:22 +1100
Subject: frozenset can be altered by |=
In-Reply-To: <CABbU2U_+t+Gg0EXaz3hc6V5xBG=TooCzMazWLLboM8fgSc1u5w@mail.gmail.com>
References: <CABbU2U_79XPx1T6EGWEniYkN8_m0oOO6mxx+HNbaLqg_wmT-Gw@mail.gmail.com>
 <CAPTjJmq=4+yEhibTJoNA3QVUNR1HmScgnQgx3NtQRpgG4aNY1Q@mail.gmail.com>
 <AM0PR07MB57000CEF0CEABBD27EA941DA879F9@AM0PR07MB5700.eurprd07.prod.outlook.com>
 <CAPTjJmpprmCdjOJBz5FVecV6KqKFRC6Y7bOsNoTP+W-t-k3vew@mail.gmail.com>
 <CABbU2U-rDQr0xkySeMsArbnox32TMdPO28SnbWtjbRhU7xzERA@mail.gmail.com>
 <CABbU2U_+t+Gg0EXaz3hc6V5xBG=TooCzMazWLLboM8fgSc1u5w@mail.gmail.com>
Message-ID: <CAPTjJmr2cQEuD=q1rLfx8-LwgTrfuJz5FTXYaPB9HB-a1Ef4wQ@mail.gmail.com>

On Tue, Nov 30, 2021 at 8:55 AM Marco Sulla
<Marco.Sulla.Python at gmail.com> wrote:
>
> I must say that I'm reading the documentation now, and it's a bit
> confusing. In the docs, inplace operators as |= should not work. They
> are listed under the set-only functions and operators. But, as we saw,
> this is not completely true: they work but they don't mutate the
> original object. The same for += and *= that are listed under `list`
> only.
>

Previously explained here:

> > On Mon, 22 Nov 2021 at 14:59, Chris Angelico <rosuav at gmail.com> wrote:
> > >
> > > Yeah, it's a little confusing, but at the language level, something
> > > that doesn't support |= will implicitly support it using the expanded
> > > version:
> > >
> > > a |= b
> > > a = a | b
> > >
> > > and in the section above, you can see that frozensets DO support the
> > > Or operator.
> > >
> > > By not having specific behaviour on the |= operator, frozensets
> > > implicitly fall back on this default.
> > >

The docs explicitly show that inplace operators are defined for the
mutable set, and not defined for the immutable frozenset. Therefore,
using an inplace operator on a frozenset uses the standard language
behavior of using the binary operator, then reassigning back to the
left operand.

This is a language feature and applies to everything. You've seen it
plenty of times with integers, and probably strings too. A frozenset
behaves the same way that anything else does.

ChrisA

From barry at barrys-emacs.org  Mon Nov 29 17:12:13 2021
From: barry at barrys-emacs.org (Barry)
Date: Mon, 29 Nov 2021 22:12:13 +0000
Subject: Python child process in while True loop blocks parent
In-Reply-To: <MphOu5i--3-2@tutanota.com>
References: <MphOu5i--3-2@tutanota.com>
Message-ID: <B05343F2-30B5-470C-8CFC-EBFA7C5EFD6A@barrys-emacs.org>



> On 29 Nov 2021, at 20:36, Jen Kris via Python-list <python-list at python.org> wrote:
> 
> ?I have a C program that forks to create a child process and uses execv to call a Python program.  The Python program communicates with the parent process (in C) through a FIFO pipe monitored with epoll().  
> 
> The Python child process is in a while True loop, which is intended to keep it running while the parent process proceeds, and perform functions for the C program only at intervals when the parent sends data to the child -- similar to a daemon process. 
> 
> The C process writes to its end of the pipe and the child process reads it, but then the child process continues to loop, thereby blocking the parent. 
> 
> This is the Python code:
> 
> #!/usr/bin/python3
> import os
> import select
> 
> #Open the named pipes
> pr = os.open('/tmp/Pipe_01', os.O_RDWR)
Why open rdwr if you are only going to read the pipe?
> pw = os.open('/tmp/Pipe_02', os.O_RDWR)
Only need to open for write.
> 
> ep = select.epoll(-1)
> ep.register(pr, select.EPOLLIN)

Is the only thing that the child does this:
1. Read message from pr
2. Process message
3. Write result to pw.
4. Loop from 1

If so as Cameron said you do not need to worry about the poll.
Do you plan for the child to become more complex?

> 
> while True:
> 
>     events = ep.poll(timeout=2.5, maxevents=-1)
>     #events = ep.poll(timeout=None, maxevents=-1)
> 
>     print("child is looping")
> 
>     for fileno, event in events:
>         print("Python fileno")
>         print(fileno)
>         print("Python event")
>         print(event)
>         v = os.read(pr,64)
>         print("Pipe value")
>         print(v)
> 
> The child process correctly receives the signal from ep.poll and correctly reads the data in the pipe, but then it continues looping.  For example, when I put in a timeout:
> 
> child is looping
> Python fileno
> 4
> Python event
> 1
> Pipe value
> b'10\x00'
The C code does not need to write a 0 bytes at the end.
I assume the 0 is from the end of a C string.
UDS messages have a length.
In the C just write 2 byes in the case.

Barry

> child is looping
> child is looping
> 
> That suggests that a while True loop is not the right thing to do in this case.  My question is, what type of process loop is best for this situation?  The multiprocessing, asyncio and subprocess libraries are very extensive, and it would help if someone could suggest the best alternative for what I am doing here. 
> 
> Thanks very much for any ideas. 
> 
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list


From barry at barrys-emacs.org  Mon Nov 29 17:14:36 2021
From: barry at barrys-emacs.org (Barry)
Date: Mon, 29 Nov 2021 22:14:36 +0000
Subject: Python child process in while True loop blocks parent
In-Reply-To: <MphOu5i--3-2@tutanota.com>
References: <MphOu5i--3-2@tutanota.com>
Message-ID: <38F95640-D94B-4906-B27A-112401BAAAEB@barrys-emacs.org>



> On 29 Nov 2021, at 20:36, Jen Kris via Python-list <python-list at python.org> wrote:
> 
> ?I have a C program that forks to create a child process and uses execv to call a Python program.  The Python program communicates with the parent process (in C) through a FIFO pipe monitored with epoll().  
> 
> The Python child process is in a while True loop, which is intended to keep it running while the parent process proceeds, and perform functions for the C program only at intervals when the parent sends data to the child -- similar to a daemon process. 
> 
> The C process writes to its end of the pipe and the child process reads it, but then the child process continues to loop, thereby blocking the parent. 
> 
> This is the Python code:
> 
> #!/usr/bin/python3
> import os
> import select
> 
> #Open the named pipes
> pr = os.open('/tmp/Pipe_01', os.O_RDWR)
> pw = os.open('/tmp/Pipe_02', os.O_RDWR)

You will need to set the fd?s to non blocking on parent and child.
Otherwise the parent will block on its write until the child reads the message.

Barry
> 
> ep = select.epoll(-1)
> ep.register(pr, select.EPOLLIN)
> 
> while True:
> 
>     events = ep.poll(timeout=2.5, maxevents=-1)
>     #events = ep.poll(timeout=None, maxevents=-1)
> 
>     print("child is looping")
> 
>     for fileno, event in events:
>         print("Python fileno")
>         print(fileno)
>         print("Python event")
>         print(event)
>         v = os.read(pr,64)
>         print("Pipe value")
>         print(v)
> 
> The child process correctly receives the signal from ep.poll and correctly reads the data in the pipe, but then it continues looping.  For example, when I put in a timeout:
> 
> child is looping
> Python fileno
> 4
> Python event
> 1
> Pipe value
> b'10\x00'
> child is looping
> child is looping
> 
> That suggests that a while True loop is not the right thing to do in this case.  My question is, what type of process loop is best for this situation?  The multiprocessing, asyncio and subprocess libraries are very extensive, and it would help if someone could suggest the best alternative for what I am doing here. 
> 
> Thanks very much for any ideas. 
> 
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list


From jenkris at tutanota.com  Mon Nov 29 17:31:51 2021
From: jenkris at tutanota.com (Jen Kris)
Date: Mon, 29 Nov 2021 23:31:51 +0100 (CET)
Subject: Python child process in while True loop blocks parent
In-Reply-To: <B05343F2-30B5-470C-8CFC-EBFA7C5EFD6A@barrys-emacs.org>
References: <MphOu5i--3-2@tutanota.com>
 <B05343F2-30B5-470C-8CFC-EBFA7C5EFD6A@barrys-emacs.org>
Message-ID: <MphoqOf--3-2@tutanota.com>

Thanks to you and Cameron for your replies.? The C side has an epoll_ctl set, but no event loop to handle it yet.? I'm putting that in now with a pipe write in Python-- as Cameron pointed out that is the likely source of blocking on C.? The pipes are opened as rdwr in Python because that's nonblocking by default.? The child will become more complex, but not in a way that affects polling.? And thanks for the tip about the c-string termination.?



Nov 29, 2021, 14:12 by barry at barrys-emacs.org:

>
>
>> On 29 Nov 2021, at 20:36, Jen Kris via Python-list <python-list at python.org> wrote:
>>
>> ?I have a C program that forks to create a child process and uses execv to call a Python program.  The Python program communicates with the parent process (in C) through a FIFO pipe monitored with epoll(). 
>>
>> The Python child process is in a while True loop, which is intended to keep it running while the parent process proceeds, and perform functions for the C program only at intervals when the parent sends data to the child -- similar to a daemon process. 
>>
>> The C process writes to its end of the pipe and the child process reads it, but then the child process continues to loop, thereby blocking the parent. 
>>
>> This is the Python code:
>>
>> #!/usr/bin/python3
>> import os
>> import select
>>
>> #Open the named pipes
>> pr = os.open('/tmp/Pipe_01', os.O_RDWR)
>>
> Why open rdwr if you are only going to read the pipe?
>
>> pw = os.open('/tmp/Pipe_02', os.O_RDWR)
>>
> Only need to open for write.
>
>>
>> ep = select.epoll(-1)
>> ep.register(pr, select.EPOLLIN)
>>
>
> Is the only thing that the child does this:
> 1. Read message from pr
> 2. Process message
> 3. Write result to pw.
> 4. Loop from 1
>
> If so as Cameron said you do not need to worry about the poll.
> Do you plan for the child to become more complex?
>
>>
>> while True:
>>
>>  events = ep.poll(timeout=2.5, maxevents=-1)
>>  #events = ep.poll(timeout=None, maxevents=-1)
>>
>>  print("child is looping")
>>
>>  for fileno, event in events:
>>  print("Python fileno")
>>  print(fileno)
>>  print("Python event")
>>  print(event)
>>  v = os.read(pr,64)
>>  print("Pipe value")
>>  print(v)
>>
>> The child process correctly receives the signal from ep.poll and correctly reads the data in the pipe, but then it continues looping.  For example, when I put in a timeout:
>>
>> child is looping
>> Python fileno
>> 4
>> Python event
>> 1
>> Pipe value
>> b'10\x00'
>>
> The C code does not need to write a 0 bytes at the end.
> I assume the 0 is from the end of a C string.
> UDS messages have a length.
> In the C just write 2 byes in the case.
>
> Barry
>
>> child is looping
>> child is looping
>>
>> That suggests that a while True loop is not the right thing to do in this case.  My question is, what type of process loop is best for this situation?  The multiprocessing, asyncio and subprocess libraries are very extensive, and it would help if someone could suggest the best alternative for what I am doing here. 
>>
>> Thanks very much for any ideas. 
>>
>>
>> -- 
>> https://mail.python.org/mailman/listinfo/python-list
>>


From cs at cskk.id.au  Mon Nov 29 18:31:24 2021
From: cs at cskk.id.au (Cameron Simpson)
Date: Tue, 30 Nov 2021 10:31:24 +1100
Subject: print('\N{flag: Mauritius}') not supported in py3.9
In-Reply-To: <a511b1cf-c634-9302-fa9b-8fbe5d4ae050@DancesWithMice.info>
References: <a511b1cf-c634-9302-fa9b-8fbe5d4ae050@DancesWithMice.info>
Message-ID: <YaVizKu2y3/ml75h@cskk.homeip.net>

On 30Nov2021 10:59, DL Neil <PythonList at DancesWithMice.info> wrote:
>Fedora names it as rxvt-unicode.
>Installed v9.26
>Text is too small for these old eyes.

Fair enough. There are resources, but not worth it unless you really 
want the app.

>No menu bar and no context menus.

Um, yes. The (hardware, serial) terminals we had a uni didn't have such 
niceties, and therefore we should not want them.

>Unable to copy-paste (the flag codes) into that window.
>Removed.

Fair enough.

>I choose not to even try to remember the difficulties of working with
>small screens over-laying one another and 'getting lost' in the pile!

Aye. Tiling is very nice.

>I'm a lazy toad. Thus the idea that the IDE will allow me to 'press a
>(single) button' to repeat the last-run test/execute the code, without
>me having to commit a Save and to jump between panels/windows/screens,
>is seductive.

I once had a vi macro bound to ";" for "^W:!!^M", which autosaves the 
current file and reran the last shell command. Used it a lot in the 
single-terminal days. I unbound it several years ago though.

>I've nominated Kitty as
>Fedora's default terminal. We'll see how it goes with work-loads beyond
>raising the flag...

I'd like to hear how that goes down the track. If I find myself on a 
Linux desktop again a good terminal emulator would be very welcome.

Cheers,
Cameron Simpson <cs at cskk.id.au>

From Richard at Damon-Family.org  Mon Nov 29 20:39:40 2021
From: Richard at Damon-Family.org (Richard Damon)
Date: Mon, 29 Nov 2021 20:39:40 -0500
Subject: frozenset can be altered by |=
In-Reply-To: <CAPTjJmr2cQEuD=q1rLfx8-LwgTrfuJz5FTXYaPB9HB-a1Ef4wQ@mail.gmail.com>
References: <CABbU2U_79XPx1T6EGWEniYkN8_m0oOO6mxx+HNbaLqg_wmT-Gw@mail.gmail.com>
 <CAPTjJmq=4+yEhibTJoNA3QVUNR1HmScgnQgx3NtQRpgG4aNY1Q@mail.gmail.com>
 <AM0PR07MB57000CEF0CEABBD27EA941DA879F9@AM0PR07MB5700.eurprd07.prod.outlook.com>
 <CAPTjJmpprmCdjOJBz5FVecV6KqKFRC6Y7bOsNoTP+W-t-k3vew@mail.gmail.com>
 <CABbU2U-rDQr0xkySeMsArbnox32TMdPO28SnbWtjbRhU7xzERA@mail.gmail.com>
 <CABbU2U_+t+Gg0EXaz3hc6V5xBG=TooCzMazWLLboM8fgSc1u5w@mail.gmail.com>
 <CAPTjJmr2cQEuD=q1rLfx8-LwgTrfuJz5FTXYaPB9HB-a1Ef4wQ@mail.gmail.com>
Message-ID: <2bfd366f-9ba0-a024-204c-01c33e4112d6@Damon-Family.org>

On 11/29/21 5:01 PM, Chris Angelico wrote:
> On Tue, Nov 30, 2021 at 8:55 AM Marco Sulla
> <Marco.Sulla.Python at gmail.com> wrote:
>> I must say that I'm reading the documentation now, and it's a bit
>> confusing. In the docs, inplace operators as |= should not work. They
>> are listed under the set-only functions and operators. But, as we saw,
>> this is not completely true: they work but they don't mutate the
>> original object. The same for += and *= that are listed under `list`
>> only.
>>
> Previously explained here:
>
>>> On Mon, 22 Nov 2021 at 14:59, Chris Angelico <rosuav at gmail.com> wrote:
>>>> Yeah, it's a little confusing, but at the language level, something
>>>> that doesn't support |= will implicitly support it using the expanded
>>>> version:
>>>>
>>>> a |= b
>>>> a = a | b
>>>>
>>>> and in the section above, you can see that frozensets DO support the
>>>> Or operator.
>>>>
>>>> By not having specific behaviour on the |= operator, frozensets
>>>> implicitly fall back on this default.
>>>>
> The docs explicitly show that inplace operators are defined for the
> mutable set, and not defined for the immutable frozenset. Therefore,
> using an inplace operator on a frozenset uses the standard language
> behavior of using the binary operator, then reassigning back to the
> left operand.
>
> This is a language feature and applies to everything. You've seen it
> plenty of times with integers, and probably strings too. A frozenset
> behaves the same way that anything else does.
>
> ChrisA

I suppose the question comes down to is it worth adding a reminder in 
the description of the inplace operators that if a type doesn't support 
the inplace operator, it is automatically converted into the equivalent 
assignment with the binary operator?

-- 
Richard Damon


From rosuav at gmail.com  Mon Nov 29 20:44:51 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 30 Nov 2021 12:44:51 +1100
Subject: frozenset can be altered by |=
In-Reply-To: <2bfd366f-9ba0-a024-204c-01c33e4112d6@Damon-Family.org>
References: <CABbU2U_79XPx1T6EGWEniYkN8_m0oOO6mxx+HNbaLqg_wmT-Gw@mail.gmail.com>
 <CAPTjJmq=4+yEhibTJoNA3QVUNR1HmScgnQgx3NtQRpgG4aNY1Q@mail.gmail.com>
 <AM0PR07MB57000CEF0CEABBD27EA941DA879F9@AM0PR07MB5700.eurprd07.prod.outlook.com>
 <CAPTjJmpprmCdjOJBz5FVecV6KqKFRC6Y7bOsNoTP+W-t-k3vew@mail.gmail.com>
 <CABbU2U-rDQr0xkySeMsArbnox32TMdPO28SnbWtjbRhU7xzERA@mail.gmail.com>
 <CABbU2U_+t+Gg0EXaz3hc6V5xBG=TooCzMazWLLboM8fgSc1u5w@mail.gmail.com>
 <CAPTjJmr2cQEuD=q1rLfx8-LwgTrfuJz5FTXYaPB9HB-a1Ef4wQ@mail.gmail.com>
 <2bfd366f-9ba0-a024-204c-01c33e4112d6@Damon-Family.org>
Message-ID: <CAPTjJmoLaQTXvpXfG6=MVLE0B+3BuKuCg_FkU-feoXyGxKHNJA@mail.gmail.com>

On Tue, Nov 30, 2021 at 12:41 PM Richard Damon <Richard at damon-family.org> wrote:
>
> On 11/29/21 5:01 PM, Chris Angelico wrote:
> > On Tue, Nov 30, 2021 at 8:55 AM Marco Sulla
> > <Marco.Sulla.Python at gmail.com> wrote:
> >> I must say that I'm reading the documentation now, and it's a bit
> >> confusing. In the docs, inplace operators as |= should not work. They
> >> are listed under the set-only functions and operators. But, as we saw,
> >> this is not completely true: they work but they don't mutate the
> >> original object. The same for += and *= that are listed under `list`
> >> only.
> >>
> > Previously explained here:
> >
> >>> On Mon, 22 Nov 2021 at 14:59, Chris Angelico <rosuav at gmail.com> wrote:
> >>>> Yeah, it's a little confusing, but at the language level, something
> >>>> that doesn't support |= will implicitly support it using the expanded
> >>>> version:
> >>>>
> >>>> a |= b
> >>>> a = a | b
> >>>>
> >>>> and in the section above, you can see that frozensets DO support the
> >>>> Or operator.
> >>>>
> >>>> By not having specific behaviour on the |= operator, frozensets
> >>>> implicitly fall back on this default.
> >>>>
> > The docs explicitly show that inplace operators are defined for the
> > mutable set, and not defined for the immutable frozenset. Therefore,
> > using an inplace operator on a frozenset uses the standard language
> > behavior of using the binary operator, then reassigning back to the
> > left operand.
> >
> > This is a language feature and applies to everything. You've seen it
> > plenty of times with integers, and probably strings too. A frozenset
> > behaves the same way that anything else does.
> >
> > ChrisA
>
> I suppose the question comes down to is it worth adding a reminder in
> the description of the inplace operators that if a type doesn't support
> the inplace operator, it is automatically converted into the equivalent
> assignment with the binary operator?
>

My view is: no, because you'd have to put that reminder on every
single object in Python. The details are here, and apply to all Python
code, not to any particular type:

https://docs.python.org/3/reference/simple_stmts.html#augmented-assignment-statements

ChrisA

From rosuav at gmail.com  Tue Nov 30 01:24:47 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 30 Nov 2021 17:24:47 +1100
Subject: A decade or so of Python programming,
 and I've never thought to "for-elif"
Message-ID: <CAPTjJmqtdjqmJ7jiOB4z1G2oMv9btiLcFqn4v9gnqbHtxGJwNA@mail.gmail.com>

for ns in namespaces:
    if name in ns:
        print("Found!")
        break
elif name.isupper():
    print("All-caps name that wasn't found")

This actually doesn't work. I have been programming in Python for well
over a decade, and never before been in a situation where this would
be useful.

As YAGNIs go, this is right up there.

(For the record, since this was the last thing in the function, I just
made the break a return. Alternatively, an extra indentation level
"else: if name.isupper():" wouldn't have been that terrible.)

Your random piece of amusement for today.

ChrisA

From barry at barrys-emacs.org  Tue Nov 30 14:42:09 2021
From: barry at barrys-emacs.org (Barry Scott)
Date: Tue, 30 Nov 2021 19:42:09 +0000
Subject: Python child process in while True loop blocks parent
In-Reply-To: <MphoqOf--3-2@tutanota.com>
References: <MphOu5i--3-2@tutanota.com>
 <B05343F2-30B5-470C-8CFC-EBFA7C5EFD6A@barrys-emacs.org>
 <MphoqOf--3-2@tutanota.com>
Message-ID: <7A3CE5EB-C53A-404D-BF32-77CF0243BF3D@barrys-emacs.org>



> On 29 Nov 2021, at 22:31, Jen Kris <jenkris at tutanota.com> wrote:
> 
> Thanks to you and Cameron for your replies.  The C side has an epoll_ctl set, but no event loop to handle it yet.  I'm putting that in now with a pipe write in Python-- as Cameron pointed out that is the likely source of blocking on C.  The pipes are opened as rdwr in Python because that's nonblocking by default.  The child will become more complex, but not in a way that affects polling.  And thanks for the tip about the c-string termination. 
> 

flags is a bit mask. You say its BLOCKing by not setting os.O_NONBLOCK.
You should not use O_RDWR when you only need O_RDONLY access or only O_WRONLY access.

You may find

	man 2 open

useful to understand in detail what is behind os.open().

Barry



> 
> 
> Nov 29, 2021, 14:12 by barry at barrys-emacs.org:
> 
> On 29 Nov 2021, at 20:36, Jen Kris via Python-list <python-list at python.org> wrote:
> 
> ?I have a C program that forks to create a child process and uses execv to call a Python program. The Python program communicates with the parent process (in C) through a FIFO pipe monitored with epoll(). 
> 
> The Python child process is in a while True loop, which is intended to keep it running while the parent process proceeds, and perform functions for the C program only at intervals when the parent sends data to the child -- similar to a daemon process. 
> 
> The C process writes to its end of the pipe and the child process reads it, but then the child process continues to loop, thereby blocking the parent. 
> 
> This is the Python code:
> 
> #!/usr/bin/python3
> import os
> import select
> 
> #Open the named pipes
> pr = os.open('/tmp/Pipe_01', os.O_RDWR)
> Why open rdwr if you are only going to read the pipe?
> pw = os.open('/tmp/Pipe_02', os.O_RDWR)
> Only need to open for write.
> 
> ep = select.epoll(-1)
> ep.register(pr, select.EPOLLIN)
> 
> Is the only thing that the child does this:
> 1. Read message from pr
> 2. Process message
> 3. Write result to pw.
> 4. Loop from 1
> 
> If so as Cameron said you do not need to worry about the poll.
> Do you plan for the child to become more complex?
> 
> while True:
> 
> events = ep.poll(timeout=2.5, maxevents=-1)
> #events = ep.poll(timeout=None, maxevents=-1)
> 
> print("child is looping")
> 
> for fileno, event in events:
> print("Python fileno")
> print(fileno)
> print("Python event")
> print(event)
> v = os.read(pr,64)
> print("Pipe value")
> print(v)
> 
> The child process correctly receives the signal from ep.poll and correctly reads the data in the pipe, but then it continues looping. For example, when I put in a timeout:
> 
> child is looping
> Python fileno
> 4
> Python event
> 1
> Pipe value
> b'10\x00'
> The C code does not need to write a 0 bytes at the end.
> I assume the 0 is from the end of a C string.
> UDS messages have a length.
> In the C just write 2 byes in the case.
> 
> Barry
> child is looping
> child is looping
> 
> That suggests that a while True loop is not the right thing to do in this case. My question is, what type of process loop is best for this situation? The multiprocessing, asyncio and subprocess libraries are very extensive, and it would help if someone could suggest the best alternative for what I am doing here. 
> 
> Thanks very much for any ideas. 
> 
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list
>