From chris.barker at noaa.gov  Sun Jul  1 01:18:09 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Sat, 30 Jun 2018 22:18:09 -0700
Subject: [Python-ideas] grouping / dict of lists
In-Reply-To: <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
Message-ID: <CALGmxEJ7O9hTkmWn=nr5eeA7GY=1Lc6gOCuZtHMY-tvxV+dFGA@mail.gmail.com>

On Fri, Jun 29, 2018 at 10:53 AM, Michael Selik <mike at selik.org> wrote:

> I've drafted a PEP for an easier way to construct groups of elements from
> a sequence. https://github.com/selik/peps/blob/master/pep-9999.rst
>
> I'm really warming to the:

Alternate: collections.Grouping

version -- I really like this as a kind of custom mapping, rather than
"just a function" (or alternate constructor) -- and I like your point that
it can have a bit of functionality built in other than on construction.

But I think it should be more like the other collection classes -- i.e. a
general purpose class that can be used for grouping, but also used more
general-purpose-y as well. That way people can do their "custom" stuff (key
function, etc.) with comprehensions.

The big differences are a custom __setitem__:

    def __setitem__(self, key, value):
        self.setdefault(key, []).append(value)

And the __init__ and update would take an iterable of (key, value) pairs,
rather than a single sequence.

This would get away from the itertools.groupby approach, which I find kinda
awkward:

* How often do you have your data in a single sequence?

* Do you need your keys (and values!) to be sortable???)

* Do we really want folks to have to be writing custom key functions and/or
lambdas for really simple stuff?

* and you may need to "transform" both your keys and values

I've enclosed an example implementation, borrowing heavily from Michael's
code.

The test code has a couple examples of use, but I'll put them here for the
sake of discussion.

Michael had:

Grouping('AbBa', key=c.casefold))

with my code, that would be:

Grouping(((c.casefold(), c) for c in 'AbBa'))

Note that the key function is applied outside the Grouping object, it
doesn't need to know anything about it -- and then users can use an
expression in a comprehension rather than a key function.

This looks a tad clumsier with my approach, but this is a pretty contrived
example -- in the more common case [*], you'd be writing a bunch of
lambdas, etc, and I'm not sure there is a way to get the values customized
as well, if you want that. (without applying a map later on)

Here is the example that the OP posted that kicked off this thread:

In [37]: student_school_list = [('Fred', 'SchoolA'),
    ...:                        ('Bob', 'SchoolB'),
    ...:                        ('Mary', 'SchoolA'),
    ...:                        ('Jane', 'SchoolB'),
    ...:                        ('Nancy', 'SchoolC'),
    ...:                        ]

In [38]: Grouping(((item[1], item[0]) for item in student_school_list))
Out[38]: Grouping({'SchoolA': ['Fred', 'Mary'],
                   'SchoolB': ['Bob', 'Jane'],
                   'SchoolC': ['Nancy']})

or

In [40]: Grouping((reversed(item) for item in student_school_list))
Out[40]: Grouping({'SchoolA': ['Fred', 'Mary'], 'SchoolB': ['Bob', 'Jane'],
'SchoolC': ['Nancy']})

(note that if those keys and values were didn't have to be reversed, you
could just pass the list in raw.

I really like how I can use a generator expression and simple expressions
to transform the data in the way I need, rather than having to make key
functions.

And with Michael's approach, I think you'd need to call .map() after
generating the grouping -- a much klunkier way to do it. (and you'd get
plain dict rather than a Grouping that you could add stuff too later...)

I'm sure there are ways to improve my code, and maybe Grouping isn't the
best name, but I think something like this would be a nice addition to the
collections module.

-CHB

[*] -- before making any decisions about the best API, it would probably be
a good idea to collect examples of the kind of data that people really do
need to group like this. Does it come in (key, value) pairs naturally? or
in one big sequence with a key function that's easy to write? who knows
without examples of real world use cases.

I will show one "real world" example here:

In my Python classes, I like to use Dave Thomas' trigrams: "code kata":

http://codekata.com/kata/kata14-tom-swift-under-the-milkwood/

A key piece of this is building up a data structure with word pairs, and a
list of all the words that follow the pair in a piece of text.

This is a nice exercise to help people think about how to use dicts, etc.
Currently the most clean code uses .setdefault:

    word_pairs = {}
    # loop through the words
    # (rare case where using the index to loop is easiest)
    for i in range(len(words) - 2):  # minus 2, 'cause you need a pair
        pair = tuple(words[i:i + 2])  # a tuple so it can be a key in the
dict
        follower = words[i + 2]
        word_pairs.setdefault(pair, []).append(follower)

if this were done with my Grouping class, it would be:

In [53]: word_pairs = Grouping()

In [54]: for i in range(len(words) - 2):
    ...:     pair = tuple(words[i:i + 2])  # a tuple so it can be a key in
the dict
    ...:     follower = words[i + 2]
    ...:     word_pairs[pair] = follower
    ...:

In [55]: word_pairs
Out[55]: Grouping({('I', 'wish'): ['I', 'I'], ('wish', 'I'): ['may',
'might'], ('I', 'may'): ['I'], ('may', 'I'): ['wish']})

Not that different, really, but it saves folks from having to find and
understand setdefault. But you could also make it into a generator
expression like so:

In [56]: Grouping(((w1, w2), w3) for w1, w2, w3, in zip(words[:],
words[1:], words[2:]))
Out[56]: Grouping({('I', 'wish'): ['I', 'I'], ('wish', 'I'): ['may',
'might'], ('I', 'may'): ['I'], ('may', 'I'): ['wish']})

which I think is pretty slick. And satisfies the OP's desire for a
comprehension-like approach, rather than the:

- create an empty dict
- loop through the iterable
- use setdefault in the loop

approach.

> As a teacher, I've found that grouping is one of the most awkward tasks
for beginners to learn in Python. While this proposal requires
understanding a key-function, in my experience that's easier to teach than
the nuances of setdefault or defaultdict.

well, yes, and no -- as above, I use an example of this in teaching so that
I CAN teach the nuances of setdefault -- or at least dicts themselves (most
student use a if key in dict" construct before I tell them about setdefault)

So if you are teaching, say data analysis with Python -- it might be nice
to have this builtin, but if you are teaching "programming with Python" I'd
probably encourage them to do it by hand first anyway :-)

> Defaultdict requires passing a factory function or class, similar to a
key-function. Setdefault is awkwardly named and requires a discussion of
references and mutability.

I agree that the naming is awkward, but I haven't found confusion with
references an mutabilty from this --though I do keep hammering those points
throughout the class anyway :-)

and my approach doesn't require any key functions either :-)




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180630/64261996/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: grouper.py
Type: text/x-python-script
Size: 3429 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180630/64261996/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test_grouper.py
Type: text/x-python-script
Size: 2420 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180630/64261996/attachment-0003.bin>

From ncoghlan at gmail.com  Sun Jul  1 02:54:13 2018
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 1 Jul 2018 16:54:13 +1000
Subject: [Python-ideas] grouping / dict of lists
In-Reply-To: <CALGmxEJ7O9hTkmWn=nr5eeA7GY=1Lc6gOCuZtHMY-tvxV+dFGA@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CALGmxEJ7O9hTkmWn=nr5eeA7GY=1Lc6gOCuZtHMY-tvxV+dFGA@mail.gmail.com>
Message-ID: <CADiSq7fHuUoj2rdDKJN4dGYkqdBZQmL-MYkordvrbdjsump-PA@mail.gmail.com>

On 1 July 2018 at 15:18, Chris Barker via Python-ideas
<python-ideas at python.org> wrote:
> On Fri, Jun 29, 2018 at 10:53 AM, Michael Selik <mike at selik.org> wrote:
>>
>> I've drafted a PEP for an easier way to construct groups of elements from
>> a sequence. https://github.com/selik/peps/blob/master/pep-9999.rst
>>
> I'm really warming to the:
>
> Alternate: collections.Grouping
>
> version -- I really like this as a kind of custom mapping, rather than "just
> a function" (or alternate constructor) -- and I like your point that it can
> have a bit of functionality built in other than on construction.
>
> But I think it should be more like the other collection classes -- i.e. a
> general purpose class that can be used for grouping, but also used more
> general-purpose-y as well. That way people can do their "custom" stuff (key
> function, etc.) with comprehensions.
>
> The big differences are a custom __setitem__:
>
>     def __setitem__(self, key, value):
>         self.setdefault(key, []).append(value)
>
> And the __init__ and update would take an iterable of (key, value) pairs,
> rather than a single sequence.
>
> This would get away from the itertools.groupby approach, which I find kinda
> awkward:
>
> * How often do you have your data in a single sequence?
>
> * Do you need your keys (and values!) to be sortable???)
>
> * Do we really want folks to have to be writing custom key functions and/or
> lambdas for really simple stuff?
>
> * and you may need to "transform" both your keys and values
>
> I've enclosed an example implementation, borrowing heavily from Michael's
> code.
>
> The test code has a couple examples of use, but I'll put them here for the
> sake of discussion.
>
> Michael had:
>
> Grouping('AbBa', key=c.casefold))
>
> with my code, that would be:
>
> Grouping(((c.casefold(), c) for c in 'AbBa'))
>
> Note that the key function is applied outside the Grouping object, it
> doesn't need to know anything about it -- and then users can use an
> expression in a comprehension rather than a key function.
>
> This looks a tad clumsier with my approach, but this is a pretty contrived
> example -- in the more common case [*], you'd be writing a bunch of lambdas,
> etc, and I'm not sure there is a way to get the values customized as well,
> if you want that. (without applying a map later on)
>
> Here is the example that the OP posted that kicked off this thread:
>
> In [37]: student_school_list = [('Fred', 'SchoolA'),
>     ...:                        ('Bob', 'SchoolB'),
>     ...:                        ('Mary', 'SchoolA'),
>     ...:                        ('Jane', 'SchoolB'),
>     ...:                        ('Nancy', 'SchoolC'),
>     ...:                        ]
>
> In [38]: Grouping(((item[1], item[0]) for item in student_school_list))
> Out[38]: Grouping({'SchoolA': ['Fred', 'Mary'],
>                    'SchoolB': ['Bob', 'Jane'],
>                    'SchoolC': ['Nancy']})

Unpacking and repacking the tuple would also work:

    Grouping(((school, student) for student, school in student_school_list))

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From levkivskyi at gmail.com  Sun Jul  1 09:02:20 2018
From: levkivskyi at gmail.com (Ivan Levkivskyi)
Date: Sun, 1 Jul 2018 14:02:20 +0100
Subject: [Python-ideas] Add a __cite__ method for scientific packages
In-Reply-To: <CAPJVwBnUUS3Wz-NSsdUGxxp3Nr0gPhzoRbRaJq+1rxgp3JMwJQ@mail.gmail.com>
References: <CADit4boE-L+OOxKLpmyOzeS55F=VC+ifLgXxEpQNz1XgcFOu_g@mail.gmail.com>
 <CAPJVwBnUUS3Wz-NSsdUGxxp3Nr0gPhzoRbRaJq+1rxgp3JMwJQ@mail.gmail.com>
Message-ID: <CAOMjWkm6e31d6c_5==t7rBEYu2jusNB3WAxJhknQRR4Mutkkbw@mail.gmail.com>

On 28 June 2018 at 01:19, Nathaniel Smith <njs at pobox.com> wrote:

> On Wed, Jun 27, 2018 at 2:20 PM, Andrei Kucharavy
> <andrei.kucharavy at gmail.com> wrote:
> > To remediate to that situation, I suggest a __citation__ method
> associated
> > to each package installation and import. Called from the __main__,
> > __citation__() would scan __citation__ of all imported packages and
> return
> > the list of all relevant top-level citations associated to the packages.
> >
> > As a scientific package developer working in academia, the problem is
> quite
> > serious, and the solution seems relatively straightforward.
> >
> > What does Python core team think about addition and long-term
> maintenance of
> > such a feature to the import and setup mechanisms? What do other users
> and
> > scientific package developers think of such a mechanism for citations
> > retrieval?
>
> This is indeed a serious problem. I suspect python-ideas isn't the
> best venue for addressing it though ? there's nothing here that needs
> changes to the Python interpreter itself (I think), and the people who
> understand this problem the best and who are most affected by it,
> mostly aren't here.
>

I actually think the opposite. If this is not fixed in a PEP it will stay
in the current state.
Writing a PEP (and officially accepting it) for this purpose will give a
signal that it is a standard practice
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180701/4cbf1f77/attachment.html>

From levkivskyi at gmail.com  Sun Jul  1 09:03:33 2018
From: levkivskyi at gmail.com (Ivan Levkivskyi)
Date: Sun, 1 Jul 2018 14:03:33 +0100
Subject: [Python-ideas] Should nested classes in an Enum be Enum members?
In-Reply-To: <5B33A33F.30207@stoneleaf.us>
References: <5B33A33F.30207@stoneleaf.us>
Message-ID: <CAOMjWk=bFncXoCmAfHjZEo+VpN3fSYbnx7dwzhrg-1siUULZOg@mail.gmail.com>

Replying to the list this time.

On 27 June 2018 at 15:46, Ethan Furman <ethan at stoneleaf.us> wrote:

> [...]
> So I'm asking the community:  What real-world examples can you offer for
> either behavior?  Cases where nested classes should be enum members, and
> cases where nested classes should not be members.
>

I wanted few times to make an enum of enums. For example:

class  Method(Enum):
    Powell = 1
    Newton_CG = 2
    <few more>
    class Trust(Enum):
        Constr = 3
        Exact = 4
        <few more>

So that one can write:

minimize(..., method=Method.Powell)
minimize(..., method=Method.Trust.Exact)  # this currently fails

--
Ivan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180701/138d3335/attachment.html>

From mertz at gnosis.cx  Sun Jul  1 09:43:53 2018
From: mertz at gnosis.cx (David Mertz)
Date: Sun, 1 Jul 2018 09:43:53 -0400
Subject: [Python-ideas] Add a __cite__ method for scientific packages
In-Reply-To: <CAOMjWkm6e31d6c_5==t7rBEYu2jusNB3WAxJhknQRR4Mutkkbw@mail.gmail.com>
References: <CADit4boE-L+OOxKLpmyOzeS55F=VC+ifLgXxEpQNz1XgcFOu_g@mail.gmail.com>
 <CAPJVwBnUUS3Wz-NSsdUGxxp3Nr0gPhzoRbRaJq+1rxgp3JMwJQ@mail.gmail.com>
 <CAOMjWkm6e31d6c_5==t7rBEYu2jusNB3WAxJhknQRR4Mutkkbw@mail.gmail.com>
Message-ID: <CAEbHw4Y61oAT7a7rdRC+ShBzbNNsv0VamO1CGg+mTfaSSx+=mg@mail.gmail.com>

I think a __citation__ *method* is a bad idea. This yells out "attribute"
to me. A function or two that parses those attributes in some manner is a
better idea... And there's no reason that function or two need to be
dunders. There's also no reason they need to be in the standard library...
There might be many citation/writing applications that process the data to
their own needs.

But assuming there is an attribute, WHAT goes inside it? Is it a string?
And if so, in what markup format? Is it a dictionary? A list? A custom
class? Does some wrapper function deal with different formats. Does the
wrapper also scan for __author__, __copyright__, and friends?

We also need to decide what __citation__ is an attribute OF. Only modules?
Classes? Methods? Functions? All of the above? If multiple, how are the
attributes at different places synthesized or processed? Can one object
have multiple citations (e.g. what if a class or method implements multiple
algorithms depending on a switch... Or depending on the shape of the data
being processed? The different algorithms might need different citations).

These are all questions that could have good answers. But I don't know what
the answers are. I've worked in scientific computing for a good while, but
not as an academic. And when I was an academic it wasn't in scientific
computing. This list is not mostly composed of the relevant experts. Those
are the authors and users of SciPy and statsmodels, and scikit-learn, and
xarray, and Tensorflow, and astropy, and so on.

There's absolutely nothing in the idea that requires a change in Python,
and Python developers or users are not, as such, the relevant experts. In
the future, AFTER there is widespread acceptance of what goes on a
__citation__ attribute, it would be easy and obvious to add minimal support
in Python itself for displaying citation content. But this is the wrong
group to mandate what the actual academic needs are here.

On Sun, Jul 1, 2018, 9:07 AM Ivan Levkivskyi <levkivskyi at gmail.com> wrote:

> On 28 June 2018 at 01:19, Nathaniel Smith <njs at pobox.com> wrote:
>
>> On Wed, Jun 27, 2018 at 2:20 PM, Andrei Kucharavy
>> <andrei.kucharavy at gmail.com> wrote:
>> > To remediate to that situation, I suggest a __citation__ method
>> associated
>> > to each package installation and import. Called from the __main__,
>> > __citation__() would scan __citation__ of all imported packages and
>> return
>> > the list of all relevant top-level citations associated to the packages.
>>
>
> I actually think the opposite. If this is not fixed in a PEP it will stay
> in the current state.
> Writing a PEP (and officially accepting it) for this purpose will give a
> signal that it is a standard practice
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180701/3ddbb7b3/attachment-0001.html>

From prometheus235 at gmail.com  Sun Jul  1 12:02:16 2018
From: prometheus235 at gmail.com (Nick Timkovich)
Date: Sun, 1 Jul 2018 11:02:16 -0500
Subject: [Python-ideas] Add a __cite__ method for scientific packages
In-Reply-To: <CA+U3c5af_PW=MZVMJ7RZW76LJb0+cPCBV1H+sakqk-86T5hXig@mail.gmail.com>
References: <CADit4boE-L+OOxKLpmyOzeS55F=VC+ifLgXxEpQNz1XgcFOu_g@mail.gmail.com>
 <CAP7+vJ+F6ZtQp_JnnXfxjYtW64fhUDSc5dPe+5Oc-_a7s1LcoQ@mail.gmail.com>
 <CAJXewOnPc1=CnT94FDLVHJ-E+MqNEfpYXMFPyH8-P99801SN+Q@mail.gmail.com>
 <VI1P194MB00794E873FB3D72F0F166D119B4F0@VI1P194MB0079.EURP194.PROD.OUTLOOK.COM>
 <CALGmxE+3NNw=A2jeWwF6yrA-yaz+MhoeS+cMuTj3AqZc-jVMhQ@mail.gmail.com>
 <CADit4bp=Z+Fy6ZMvibUoHVR8_8o5fMoSH7y8AOvZL6DTo_B-vA@mail.gmail.com>
 <20180629001724.GO14437@ando.pearwood.info>
 <067d01d40f5d$403ea710$c0bbf530$@sdamon.com>
 <CAEUL7mHDM4QQzA6ER9hq_d0rZbyhjUrJA=MWCJSu8_V+mkwUYw@mail.gmail.com>
 <069001d40f61$62103260$26309720$@sdamon.com>
 <CAJXewOko-kdWzUfj+wEkz958YVdDKDy=cp39ZFNbyR_mJZO6Qw@mail.gmail.com>
 <CADit4bqcPh8--sgnzUmrctfpMX-cWR3MnLFGNrbObczXhsf62A@mail.gmail.com>
 <CA+U3c5af_PW=MZVMJ7RZW76LJb0+cPCBV1H+sakqk-86T5hXig@mail.gmail.com>
Message-ID: <CAHkxivdOpSdq=m=uErSb0KZEJ2b5Cr0zc0nW2VYNQJJyEmSZig@mail.gmail.com>

On Fri, Jun 29, 2018 at 8:58 PM, Matt Arcidy <marcidy at gmail.com> wrote:

> It seems like the very small percentage of academic users whose careers
> depend on this cannot resolve the political issue of forming a standards
> body.
>
> I don't see how externalizing the standard development will help.  Kudos
> for shortcutting the process in a practical way to just get it done,  but
> this just puts core devs in the middle of silly academic spats.  A language
> endorsed citation method isn't a 'correct' method, and without the broad
> consensus that currently doesn't exist, this becomes _your_ method, a
> picked winner but ultimately a lightning rod for bored tenured professors
> with personal axes to grind.  If this were about implementing an existing
> correct method I'm sure a grad student would be tasked with it for an
> afternoon.
>
>
[...]  Just create a jstor style git server where obeying the citation
> protocol is mandatory.
>

I don't know if it constitutes a standards body, but there are a couple
journals out there that are meant to serve as mechanisms for turning a repo
into a published/citable thing, they might be good to look at for prior art
as well as to what metadata should be included:

* https://joss.theoj.org/about (sponsored by NumFOCUS)
* https://www.journals.elsevier.com/softwarex/

>From an abstract level, however, citing code requires that it's published
in some form, which is very strongly related to packaging, so I think such
discussions would best revolve around there. Maybe rolling something into
pkg_resources that could pull out a short citation string from some package
metadata (a hypothetical `pkg_resources.get_distribution("numpy").citation`
that could be wrapped by some helper function if desired)? The actual
mechanism to convert metadata into something in the repo (a dunder cite
string in the root module, a separate metadata file, etc.) into the package
metadata isn't as important as rolling said metadata into something part of
the distribution package like the version or long_description fields. Once
the schema of the citation data is defined, you could add it to the
metadata spec (outgrowth of PEP-566)
https://packaging.python.org/specifications/core-metadata/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180701/7502f6ab/attachment.html>

From ethan at stoneleaf.us  Sun Jul  1 15:47:44 2018
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 01 Jul 2018 12:47:44 -0700
Subject: [Python-ideas] Should nested classes in an Enum be Enum members?
In-Reply-To: <CAOMjWk=bFncXoCmAfHjZEo+VpN3fSYbnx7dwzhrg-1siUULZOg@mail.gmail.com>
References: <5B33A33F.30207@stoneleaf.us>
 <CAOMjWk=bFncXoCmAfHjZEo+VpN3fSYbnx7dwzhrg-1siUULZOg@mail.gmail.com>
Message-ID: <5B392FE0.8000908@stoneleaf.us>

On 07/01/2018 06:03 AM, Ivan Levkivskyi wrote:> On 27 June 2018 at 15:46, Ethan Furman wrote:

>> [...]
>> So I'm asking the community:  What real-world examples can you offer for either behavior?  Cases where nested
>> classes should be enum members, and cases where nested classes should not be members.
>
> I wanted few times to make an enum of enums. For example:
>
> class  Method(Enum):
>      Powell = 1
>      Newton_CG = 2
>      <few more>
>      class Trust(Enum):
>          Constr = 3
>          Exact = 4
>          <few more>
>
> So that one can write:
>
> minimize(..., method=Method.Powell)
> minimize(..., method=Method.Trust.Exact)  # this currently fails

In such a case, would you want/expect for

--> list(Method)

to return

[<Powel: 1>, <Newton_CG: 2>, ..., <Trust: -something->]

?

--
~Ethan~


From tritium-list at sdamon.com  Sun Jul  1 16:58:37 2018
From: tritium-list at sdamon.com (Alex Walters)
Date: Sun, 1 Jul 2018 16:58:37 -0400
Subject: [Python-ideas] Add a __cite__ method for scientific packages
In-Reply-To: <CAHkxivdOpSdq=m=uErSb0KZEJ2b5Cr0zc0nW2VYNQJJyEmSZig@mail.gmail.com>
References: <CADit4boE-L+OOxKLpmyOzeS55F=VC+ifLgXxEpQNz1XgcFOu_g@mail.gmail.com>
 <CAP7+vJ+F6ZtQp_JnnXfxjYtW64fhUDSc5dPe+5Oc-_a7s1LcoQ@mail.gmail.com>
 <CAJXewOnPc1=CnT94FDLVHJ-E+MqNEfpYXMFPyH8-P99801SN+Q@mail.gmail.com>
 <VI1P194MB00794E873FB3D72F0F166D119B4F0@VI1P194MB0079.EURP194.PROD.OUTLOOK.COM>
 <CALGmxE+3NNw=A2jeWwF6yrA-yaz+MhoeS+cMuTj3AqZc-jVMhQ@mail.gmail.com>
 <CADit4bp=Z+Fy6ZMvibUoHVR8_8o5fMoSH7y8AOvZL6DTo_B-vA@mail.gmail.com>
 <20180629001724.GO14437@ando.pearwood.info>
 <067d01d40f5d$403ea710$c0bbf530$@sdamon.com>
 <CAEUL7mHDM4QQzA6ER9hq_d0rZbyhjUrJA=MWCJSu8_V+mkwUYw@mail.gmail.com>
 <069001d40f61$62103260$26309720$@sdamon.com>
 <CAJXewOko-kdWzUfj+wEkz958YVdDKDy=cp39ZFNbyR_mJZO6Qw@mail.gmail.com>
 <CADit4bqcPh8--sgnzUmrctfpMX-cWR3MnLFGNrbObczXhsf62A@mail.gmail.com>
 <CA+U3c5af_PW=MZVMJ7RZW76LJb0+cPCBV1H+sakqk-86T5hXig@mail.gmail.com>
 <CAHkxivdOpSdq=m=uErSb0KZEJ2b5Cr0zc0nW2VYNQJJyEmSZig@mail.gmail.com>
Message-ID: <1cb6e01d4117e$48646ec0$d92d4c40$@sdamon.com>

> -----Original Message-----
> From: Python-ideas <python-ideas-bounces+tritium-
> list=sdamon.com at python.org> On Behalf Of Nick Timkovich
> Sent: Sunday, July 1, 2018 12:02 PM
> To: Matt Arcidy <marcidy at gmail.com>
> Cc: python-ideas <python-ideas at python.org>
> Subject: Re: [Python-ideas] Add a __cite__ method for scientific packages

> 
> From an abstract level, however, citing code requires that it's published in
> some form, which is very strongly related to packaging, so I think such
> discussions would best revolve around there. Maybe rolling something into
> pkg_resources that could pull out a short citation string from some package
> metadata (a hypothetical `pkg_resources.get_distribution("numpy").citation`
> that could be wrapped by some helper function if desired)? The actual
> mechanism to convert metadata into something in the repo (a dunder cite
> string in the root module, a separate metadata file, etc.) into the package
> metadata isn't as important as rolling said metadata into something part of
> the distribution package like the version or long_description fields. Once the
> schema of the citation data is defined, you could add it to the metadata spec
> (outgrowth of PEP-566) https://packaging.python.org/specifications/core-
> metadata/

Putting citation information into pyproject.toml makes a lot more sense than putting it in the modules themselves, where they would have to be introspected to be extracted.

* It puts zero burden on the core developers
* It puts near zero burden on the distutils special interest group
* It doesn't consume names from the package namespace
* It's just a TOML file -  you can add sections to it willy-nilly
* It's just a TOML file - there's libraries in almost all ecosystems to handle it.

Nothing has to go into the core metadata specification unless part of your suggestion is that Pypi show the citations.  I don't think that is a good idea for the scope of Pypi and the workload of the warehouse developers.  I don't think it's too much to ask for the scientific community to figure out the solution that works for most people before bringing it back here.  I also don't think its out of scope to suggest taking this to SciPy - yes, not everything depends on SciPy, but you don't need everything, you just momentum.


From levkivskyi at gmail.com  Sun Jul  1 17:32:38 2018
From: levkivskyi at gmail.com (Ivan Levkivskyi)
Date: Sun, 1 Jul 2018 22:32:38 +0100
Subject: [Python-ideas] Should nested classes in an Enum be Enum members?
In-Reply-To: <5B392FE0.8000908@stoneleaf.us>
References: <5B33A33F.30207@stoneleaf.us>
 <CAOMjWk=bFncXoCmAfHjZEo+VpN3fSYbnx7dwzhrg-1siUULZOg@mail.gmail.com>
 <5B392FE0.8000908@stoneleaf.us>
Message-ID: <CAOMjWkmKvqhWcFfbWuwnURPObCcsURZ0gLwVki-rKwsS8MCzSA@mail.gmail.com>

On 1 July 2018 at 20:47, Ethan Furman <ethan at stoneleaf.us> wrote:

> On 07/01/2018 06:03 AM, Ivan Levkivskyi wrote:> On 27 June 2018 at 15:46,
> Ethan Furman wrote:
>
> [...]
>>> So I'm asking the community:  What real-world examples can you offer for
>>> either behavior?  Cases where nested
>>> classes should be enum members, and cases where nested classes should
>>> not be members.
>>>
>>
>> I wanted few times to make an enum of enums. For example:
>>
>> class  Method(Enum):
>>      Powell = 1
>>      Newton_CG = 2
>>      <few more>
>>      class Trust(Enum):
>>          Constr = 3
>>          Exact = 4
>>          <few more>
>>
>> So that one can write:
>>
>> minimize(..., method=Method.Powell)
>> minimize(..., method=Method.Trust.Exact)  # this currently fails
>>
>
> In such a case, would you want/expect for
>
> --> list(Method)
>
> to return
>
> [<Powel: 1>, <Newton_CG: 2>, ..., <Trust: -something->]
>
> ?
>

I am fine with what `list(Method)` does currently:

[<Method.Powell: 1>, <Method.Newton_CG: 2>, <Method.Trust: <enum 'Trust'>>]

I think it is intuitive that the nested enums are _not_ automatically
flattened. Also it is easy to write a helper that
manually flattens nested enums, but it would be tedious to group them back
if they are already flattened.

The only thing that I find a bit unintuitive is that one needs to write
`Method.Trust.value.Exact` instead of just
`Method.Trust.Exact`.

Maybe his can be allowed by adding a `__getattr__` to `Method` (by the
metaclass) that will check if a value is an enum and
delegate the attribute access.

--
Ivan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180701/71b5294a/attachment-0001.html>

From chris.barker at noaa.gov  Sun Jul  1 20:45:51 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Sun, 1 Jul 2018 17:45:51 -0700
Subject: [Python-ideas] Fwd:  grouping / dict of lists
In-Reply-To: <CALGmxEJH6pn5AeXFv8gEeM=sLQFSixpwqCt5REQoJh9zhjcbcA@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CALGmxEJ7O9hTkmWn=nr5eeA7GY=1Lc6gOCuZtHMY-tvxV+dFGA@mail.gmail.com>
 <CAOMjWkkst4XJv_mYZUZ+XymwKB1XtWKx1tAWG3DbbgasfDxQHg@mail.gmail.com>
 <CALGmxEJH6pn5AeXFv8gEeM=sLQFSixpwqCt5REQoJh9zhjcbcA@mail.gmail.com>
Message-ID: <CALGmxELhj0HRq7h7Zk4YYNNreu-EvMTWddU-qHw2xJaXZ4rb_g@mail.gmail.com>

Ivan,

Did you mean this to go to the list? I hope so, as I've cc-d it this time
:-)

On Sun, Jul 1, 2018 at 1:20 AM, Ivan Levkivskyi <levkivskyi at gmail.com>
wrote:

> On 1 July 2018 at 06:18, Chris Barker via Python-ideas <
> python-ideas at python.org> wrote:
>
>> I'm really warming to the:
>>
>> Alternate: collections.Grouping
>>
>> version -- I really like this as a kind of custom mapping, rather than
>> "just a function" (or alternate constructor) --
>>
>
I wanted the group to be represented as a set, not a list. I however
> understand that list may be more common. Can we design an API that
> would make this configurable? Something like:
>
>     from collections import Grouping
>
>     deps = Grouping(set)  # list can be the default
>     deps.update(other_deps)  # uses set.update or list.extend for every key
>     deps.add(trigger, target)  # uses set.add or list.append
>

yeah, I thought about that too -- Michael was using set() in some of his
examples.

But the question is -- do we have a single switchable version or just too
classes?


> Probably allowing an arbitrary collection for values is to general/hard.
>

maybe not -- if we had the criteria that you pass in any collection you
wanted, as long as it had either an .append() or .add()method, it would be
pretty easy to do with duck typing magic.

Sure -- a user could make a mess easily enough by passing in a weird custom
class, but so what? Using something other than a set or list would be a "at
your own risk" thing anyway.


> Maybe we can just add a flag `unique=True` to the constructor, that will
> cause using sets instead of lists for groups?
>

That's another, more robust, but less flexible option.

Stay tuned for a prototype, if I can get it done fast....

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180701/c2cd3d60/attachment.html>

From mertz at gnosis.cx  Sun Jul  1 22:12:16 2018
From: mertz at gnosis.cx (David Mertz)
Date: Sun, 1 Jul 2018 22:12:16 -0400
Subject: [Python-ideas] Fwd: grouping / dict of lists
In-Reply-To: <CALGmxELhj0HRq7h7Zk4YYNNreu-EvMTWddU-qHw2xJaXZ4rb_g@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CALGmxEJ7O9hTkmWn=nr5eeA7GY=1Lc6gOCuZtHMY-tvxV+dFGA@mail.gmail.com>
 <CAOMjWkkst4XJv_mYZUZ+XymwKB1XtWKx1tAWG3DbbgasfDxQHg@mail.gmail.com>
 <CALGmxEJH6pn5AeXFv8gEeM=sLQFSixpwqCt5REQoJh9zhjcbcA@mail.gmail.com>
 <CALGmxELhj0HRq7h7Zk4YYNNreu-EvMTWddU-qHw2xJaXZ4rb_g@mail.gmail.com>
Message-ID: <CAEbHw4bnkpfqSHvXvDA6Cya102LyW3b765qq7uC0U0Jx8VYQbw@mail.gmail.com>

Michael changed from set to list at my urging. A list is more general. A
groupby in Pandas or SQL does not enforce uniqueness, but DOES preserve
order. I think the PEP is not fully updated, but it's a list everywhere in
the proposal itself, just not in the "old techniques."

Moreover,  Michael gives example of "casting" the Grouping to a dictionary
with either sets or Counters as values. Both are useful, and both can be
derived from list. But you cannot go backwards from either to the list. The
transformation is simple and obvious, and can be included in eventual
documentation.

It really is better to construct the collection using lists?in the fully
general manner?and then only throw away the generality when that
appropriate.

On Sun, Jul 1, 2018, 8:47 PM Chris Barker via Python-ideas <
python-ideas at python.org> wrote:

>
>
> Ivan,
>
> Did you mean this to go to the list? I hope so, as I've cc-d it this time
> :-)
>
> On Sun, Jul 1, 2018 at 1:20 AM, Ivan Levkivskyi <levkivskyi at gmail.com>
> wrote:
>
>> On 1 July 2018 at 06:18, Chris Barker via Python-ideas <
>> python-ideas at python.org> wrote:
>>
>>> I'm really warming to the:
>>>
>>> Alternate: collections.Grouping
>>>
>>> version -- I really like this as a kind of custom mapping, rather than
>>> "just a function" (or alternate constructor) --
>>>
>>
> I wanted the group to be represented as a set, not a list. I however
>> understand that list may be more common. Can we design an API that
>> would make this configurable? Something like:
>>
>>     from collections import Grouping
>>
>>     deps = Grouping(set)  # list can be the default
>>     deps.update(other_deps)  # uses set.update or list.extend for every
>> key
>>     deps.add(trigger, target)  # uses set.add or list.append
>>
>
> yeah, I thought about that too -- Michael was using set() in some of his
> examples.
>
> But the question is -- do we have a single switchable version or just too
> classes?
>
>
>> Probably allowing an arbitrary collection for values is to general/hard.
>>
>
> maybe not -- if we had the criteria that you pass in any collection you
> wanted, as long as it had either an .append() or .add()method, it would
> be pretty easy to do with duck typing magic.
>
> Sure -- a user could make a mess easily enough by passing in a weird
> custom class, but so what? Using something other than a set or list would
> be a "at your own risk" thing anyway.
>
>
>> Maybe we can just add a flag `unique=True` to the constructor, that will
>> cause using sets instead of lists for groups?
>>
>
> That's another, more robust, but less flexible option.
>
> Stay tuned for a prototype, if I can get it done fast....
>
> -CHB
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180701/ee84a3f8/attachment.html>

From mertz at gnosis.cx  Sun Jul  1 22:28:51 2018
From: mertz at gnosis.cx (David Mertz)
Date: Sun, 1 Jul 2018 22:28:51 -0400
Subject: [Python-ideas] Fwd: grouping / dict of lists
In-Reply-To: <CAEbHw4bnkpfqSHvXvDA6Cya102LyW3b765qq7uC0U0Jx8VYQbw@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CALGmxEJ7O9hTkmWn=nr5eeA7GY=1Lc6gOCuZtHMY-tvxV+dFGA@mail.gmail.com>
 <CAOMjWkkst4XJv_mYZUZ+XymwKB1XtWKx1tAWG3DbbgasfDxQHg@mail.gmail.com>
 <CALGmxEJH6pn5AeXFv8gEeM=sLQFSixpwqCt5REQoJh9zhjcbcA@mail.gmail.com>
 <CALGmxELhj0HRq7h7Zk4YYNNreu-EvMTWddU-qHw2xJaXZ4rb_g@mail.gmail.com>
 <CAEbHw4bnkpfqSHvXvDA6Cya102LyW3b765qq7uC0U0Jx8VYQbw@mail.gmail.com>
Message-ID: <CAEbHw4aYef6GAizSvmBtp2f1NFfuKhOaUe72fNug8Gp0wivjvg@mail.gmail.com>

Oh, it looks like he has modified the PEP and taken out the examples of
conversion. That's too bad, hopefully they'll be added back.

But it's pretty simple. Whether my idea of collections.Grouping is adapted
or whether a function/classmethod grouping() produces a plain dictionary,
the casting would be the same:

{k:set(v) for k,v in deps.items()}

{k:Counter(v) for k,v in deps.items()}



On Sun, Jul 1, 2018, 10:12 PM David Mertz <mertz at gnosis.cx> wrote:

> Michael changed from set to list at my urging. A list is more general. A
> groupby in Pandas or SQL does not enforce uniqueness, but DOES preserve
> order. I think the PEP is not fully updated, but it's a list everywhere in
> the proposal itself, just not in the "old techniques."
>
> Moreover,  Michael gives example of "casting" the Grouping to a dictionary
> with either sets or Counters as values. Both are useful, and both can be
> derived from list. But you cannot go backwards from either to the list. The
> transformation is simple and obvious, and can be included in eventual
> documentation.
>
> It really is better to construct the collection using lists?in the fully
> general manner?and then only throw away the generality when that
> appropriate.
>
> On Sun, Jul 1, 2018, 8:47 PM Chris Barker via Python-ideas <
> python-ideas at python.org> wrote:
>
>>
>>
>> Ivan,
>>
>> Did you mean this to go to the list? I hope so, as I've cc-d it this time
>> :-)
>>
>> On Sun, Jul 1, 2018 at 1:20 AM, Ivan Levkivskyi <levkivskyi at gmail.com>
>> wrote:
>>
>>> On 1 July 2018 at 06:18, Chris Barker via Python-ideas <
>>> python-ideas at python.org> wrote:
>>>
>>>> I'm really warming to the:
>>>>
>>>> Alternate: collections.Grouping
>>>>
>>>> version -- I really like this as a kind of custom mapping, rather than
>>>> "just a function" (or alternate constructor) --
>>>>
>>>
>> I wanted the group to be represented as a set, not a list. I however
>>> understand that list may be more common. Can we design an API that
>>> would make this configurable? Something like:
>>>
>>>     from collections import Grouping
>>>
>>>     deps = Grouping(set)  # list can be the default
>>>     deps.update(other_deps)  # uses set.update or list.extend for every
>>> key
>>>     deps.add(trigger, target)  # uses set.add or list.append
>>>
>>
>> yeah, I thought about that too -- Michael was using set() in some of his
>> examples.
>>
>> But the question is -- do we have a single switchable version or just too
>> classes?
>>
>>
>>> Probably allowing an arbitrary collection for values is to general/hard.
>>>
>>
>> maybe not -- if we had the criteria that you pass in any collection you
>> wanted, as long as it had either an .append() or .add()method, it would
>> be pretty easy to do with duck typing magic.
>>
>> Sure -- a user could make a mess easily enough by passing in a weird
>> custom class, but so what? Using something other than a set or list would
>> be a "at your own risk" thing anyway.
>>
>>
>>> Maybe we can just add a flag `unique=True` to the constructor, that will
>>> cause using sets instead of lists for groups?
>>>
>>
>> That's another, more robust, but less flexible option.
>>
>> Stay tuned for a prototype, if I can get it done fast....
>>
>> -CHB
>>
>>
>> --
>>
>> Christopher Barker, Ph.D.
>> Oceanographer
>>
>> Emergency Response Division
>> NOAA/NOS/OR&R            (206) 526-6959   voice
>> 7600 Sand Point Way NE   (206) 526-6329   fax
>> Seattle, WA  98115       (206) 526-6317   main reception
>>
>> Chris.Barker at noaa.gov
>>
>>
>>
>> --
>>
>> Christopher Barker, Ph.D.
>> Oceanographer
>>
>> Emergency Response Division
>> NOAA/NOS/OR&R            (206) 526-6959   voice
>> 7600 Sand Point Way NE   (206) 526-6329   fax
>> Seattle, WA  98115       (206) 526-6317   main reception
>>
>> Chris.Barker at noaa.gov
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180701/a41e86f5/attachment-0001.html>

From chris.barker at noaa.gov  Mon Jul  2 00:36:40 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Sun, 1 Jul 2018 21:36:40 -0700
Subject: [Python-ideas] Fwd: grouping / dict of lists
In-Reply-To: <CAEbHw4aYef6GAizSvmBtp2f1NFfuKhOaUe72fNug8Gp0wivjvg@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CALGmxEJ7O9hTkmWn=nr5eeA7GY=1Lc6gOCuZtHMY-tvxV+dFGA@mail.gmail.com>
 <CAOMjWkkst4XJv_mYZUZ+XymwKB1XtWKx1tAWG3DbbgasfDxQHg@mail.gmail.com>
 <CALGmxEJH6pn5AeXFv8gEeM=sLQFSixpwqCt5REQoJh9zhjcbcA@mail.gmail.com>
 <CALGmxELhj0HRq7h7Zk4YYNNreu-EvMTWddU-qHw2xJaXZ4rb_g@mail.gmail.com>
 <CAEbHw4bnkpfqSHvXvDA6Cya102LyW3b765qq7uC0U0Jx8VYQbw@mail.gmail.com>
 <CAEbHw4aYef6GAizSvmBtp2f1NFfuKhOaUe72fNug8Gp0wivjvg@mail.gmail.com>
Message-ID: <CALGmxEKNLJhetAoO8q5p7i64odgHmkNkfwpCX6CKEq5_q+gM1A@mail.gmail.com>

On Sun, Jul 1, 2018 at 7:28 PM, David Mertz <mertz at gnosis.cx> wrote:

> But it's pretty simple. Whether my idea of collections.Grouping is adapted
> or whether a function/classmethod grouping() produces a plain dictionary,
>

or my custom class...


> the casting would be the same:
>
> {k:set(v) for k,v in deps.items()}
>
> {k:Counter(v) for k,v in deps.items()}
>

hmm, makes we wonder if it would make sense to update my implementation to
allow mapping types as well for the collection --now we'd be getting really
magic....

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180701/50b6e7b5/attachment.html>

From chris.barker at noaa.gov  Mon Jul  2 00:31:18 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Sun, 1 Jul 2018 21:31:18 -0700
Subject: [Python-ideas] Fwd: grouping / dict of lists
In-Reply-To: <CAEbHw4bnkpfqSHvXvDA6Cya102LyW3b765qq7uC0U0Jx8VYQbw@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CALGmxEJ7O9hTkmWn=nr5eeA7GY=1Lc6gOCuZtHMY-tvxV+dFGA@mail.gmail.com>
 <CAOMjWkkst4XJv_mYZUZ+XymwKB1XtWKx1tAWG3DbbgasfDxQHg@mail.gmail.com>
 <CALGmxEJH6pn5AeXFv8gEeM=sLQFSixpwqCt5REQoJh9zhjcbcA@mail.gmail.com>
 <CALGmxELhj0HRq7h7Zk4YYNNreu-EvMTWddU-qHw2xJaXZ4rb_g@mail.gmail.com>
 <CAEbHw4bnkpfqSHvXvDA6Cya102LyW3b765qq7uC0U0Jx8VYQbw@mail.gmail.com>
Message-ID: <CALGmxELauHJw7wX3m7NabviRfZkmD_kaoK8nbfRUYZ3ExcXgzQ@mail.gmail.com>

On Sun, Jul 1, 2018 at 7:12 PM, David Mertz <mertz at gnosis.cx> wrote:

> Michael changed from set to list at my urging. A list is more general. A
> groupby in Pandas or SQL does not enforce uniqueness, but DOES preserve
> order.
>

<snip>

It really is better to construct the collection using lists?in the fully
general manner?and then only throw away the generality when that
appropriate.

well, yes -- if there were only one option, then list is pretty obvious.

but whether converting to sets after the fact is just as good or not -- I
don't think so.

It's only just as good if you think of it as a one-time operation --
process a bunch of data all at once, and get back a dict with the results.
But I'm thinking of it in a different way:

Create a custom class derived from dict that you can add stuff to at any
time --much more like the current examples in the collections module.

If you simply want a groupby function that returns a regular dict, then you
need a utility function (or a few), not a new class.

If you are making a class that enforces the the values to be a collection
of items, then list is the obvious default, but of someone wants a set --
they want it built in to the class, not converted after the fact.

I've extended my prototype to do just that:

class Grouping(dict):
   ...
    def __init__(self, iterable=(), *, collection=list):

"collection" is a class that's either a Mutable Sequence (has .append and
.extend methods) or Set (has .add and .update methods).

Once you create a Grouping instance, the collection class you pass in is
used everywhere.

I've put the prototype up on gitHub if anyone wants to take a look, try it
out, suggest changes, etc:

https://github.com/PythonCHB/grouper

(and enclosed here)

Note that I am NOT proposing this particular implementation or names, or
anything. I welcome feedback on the implementation, API and naming scheme,
but it would be great if we could all be clear on whether the critique is
of the idea or of the implementation.

This particular implementation uses pretty hack meta-class magic (or the
type constructor anyway) -- if something set-like is passed in, it creates
a subclass that adds .append and .extend methods, so that the rest of the
code doesn't have to special case. Not sure if that's a good idea, it feels
pretty kludgy -- but kinda fun to write.

It also needs more test cases and example use cases for sure.

And before we go much farther with this discussion, it would be great to
see some more real-world use cases, if anyone has some in mind.

-CHB

-------
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180701/5be65cd6/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: grouper.py
Type: text/x-python-script
Size: 4171 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180701/5be65cd6/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test_grouper.py
Type: text/x-python-script
Size: 2914 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180701/5be65cd6/attachment-0003.bin>

From chris.barker at noaa.gov  Mon Jul  2 01:34:20 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Sun, 1 Jul 2018 22:34:20 -0700
Subject: [Python-ideas] Fwd: grouping / dict of lists
In-Reply-To: <CALGmxEKNLJhetAoO8q5p7i64odgHmkNkfwpCX6CKEq5_q+gM1A@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CALGmxEJ7O9hTkmWn=nr5eeA7GY=1Lc6gOCuZtHMY-tvxV+dFGA@mail.gmail.com>
 <CAOMjWkkst4XJv_mYZUZ+XymwKB1XtWKx1tAWG3DbbgasfDxQHg@mail.gmail.com>
 <CALGmxEJH6pn5AeXFv8gEeM=sLQFSixpwqCt5REQoJh9zhjcbcA@mail.gmail.com>
 <CALGmxELhj0HRq7h7Zk4YYNNreu-EvMTWddU-qHw2xJaXZ4rb_g@mail.gmail.com>
 <CAEbHw4bnkpfqSHvXvDA6Cya102LyW3b765qq7uC0U0Jx8VYQbw@mail.gmail.com>
 <CAEbHw4aYef6GAizSvmBtp2f1NFfuKhOaUe72fNug8Gp0wivjvg@mail.gmail.com>
 <CALGmxEKNLJhetAoO8q5p7i64odgHmkNkfwpCX6CKEq5_q+gM1A@mail.gmail.com>
Message-ID: <CALGmxEJHfSr9PyUgjHhSp+qs3Q+wzH3-C5-hHY0rQzSX+oCz5g@mail.gmail.com>

On Sun, Jul 1, 2018 at 9:36 PM, Chris Barker <chris.barker at noaa.gov> wrote:

> hmm, makes we wonder if it would make sense to update my implementation to
> allow mapping types as well for the collection
>

general mapping types don't make sense -- but I added Counter. Which is a
pretty special case, so I think it probably makes that case that it should
just always be a list, and you can convert to others later. Though maybe
list, set and Counter are the ones you'd want to use ????

Again, real world use cases are needed!

-CHB

code here:

https://github.com/PythonCHB/grouper/blob/master/grouper/grouper.py


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180701/16ca342d/attachment.html>

From mike at selik.org  Mon Jul  2 03:43:17 2018
From: mike at selik.org (Michael Selik)
Date: Mon, 2 Jul 2018 00:43:17 -0700
Subject: [Python-ideas] grouping / dict of lists
In-Reply-To: <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CAP7+vJJ6vN9H8esj8tS+bSKsFELh8q8ywYHAW8Ti_upEJHf1vg@mail.gmail.com>
 <CADqi797HXzHcUnrNLp7r+pFEM5fcApiKHV4s=P5-D6mXHYYv-g@mail.gmail.com>
 <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
Message-ID: <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>

I made some heavy revisions to the PEP. Linking again for convenience.
https://github.com/selik/peps/blob/master/pep-9999.rst

Replying to Guido, Nick, David, Chris, and Ivan in 4 sections below.


[Guido]
On Fri, Jun 29, 2018 at 11:25 PM Guido van Rossum <guido at python.org> wrote:

> On Fri, Jun 29, 2018 at 3:23 PM Michael Selik <mike at selik.org> wrote:
>
>> On Fri, Jun 29, 2018 at 2:43 PM Guido van Rossum <guido at python.org>
>> wrote:
>>
>>> On a quick skim I see nothing particularly objectionable or
>>> controversial in your PEP, except I'm unclear why it needs to be a class
>>> method on `dict`.
>>>
>>
>> Since it constructs a basic dict, I thought it belongs best as a dict
>> constructor like dict.fromkeys. It seemed to match other classmethods like
>> datetime.now.
>>
>
> It doesn't strike me as important enough. Surely not every stdlib function
> that returns a fresh dict needs to be a class method on dict!
>

Thinking back, I may have chosen the name "groupby" first, following
`itertools.groupby`, SQL, and other languages, and I wanted to make a clear
distinction from `itertools.groupby`. Putting it on the `dict` namespace
clarified that it's returning a dict.

However, naming it `grouping` allows it to be a stand-alone function.

But I still think it is much better off as a helper function in itertools.
>
I considered placing it in the itertools module, but decided against
>> because it doesn't return an iterator. I'm open to that if that's the
>> consensus.
>>
>
> You'll never get consensus on anything here, but you have my blessing for
> this without consensus.
>

That feels like a success, but I'm going to be a bit more ambitious and try
to persuade you that `grouping` belongs in the built-ins. I revised my
draft to streamline the examples and make a clearer comparison with
existing tools.


[Nick]
On Sat, Jun 30, 2018 at 2:01 AM Nick Coghlan <ncoghlan at gmail.com> wrote:

> I'm not sure if the draft was updated since [Guido] looked at it, but it


> does mention that one benefit of the collections.Grouping approach is
> being able to add native support for mapping a callable across every
> individual item in the collection (ignoring the group structure), as
> well as for applying aggregate functions to reduce the groups to
> single values in a standard dict.
>
> Delegating those operations to the container API that way then means
> that other libraries can expose classes that implement the grouping
> API, but with a completely different backend storage model.
>

While it'd be nice to create a standard interface as you point out, my
primary goal is to create an "obvious" way for both beginners and experts
to group, classify, categorize, bucket, demultiplex, taxonomize, etc. I
started revising the PEP last night and found myself getting carried away
with adding methods to the Grouping class that were more distracting than
useful. Since the most important thing is to make this as accessible and
easy as possible, I re-focused the proposal on the core idea of grouping.


[Ivan, Chris, David]
On Sun, Jul 1, 2018 at 7:29 PM David Mertz <mertz at gnosis.cx> wrote:

> {k:set(v) for k,v in deps.items()}
> {k:Counter(v) for k,v in deps.items()}
>

I had dropped those specific examples in favor of generically "func(g)",
but added them back. Your discussion with Ivan and Chris showed that it was
useful to be specific.


[Chris]
On Sat, Jun 30, 2018 at 10:18 PM Chris Barker <chris.barker at noaa.gov> wrote:

> I'm really warming to the:
> Alternate: collections.Grouping
> version -- I really like this as a kind of custom mapping, rather than
> "just a function" (or alternate constructor) -- and I like your point that
> it can have a bit of functionality built in other than on construction.
>

I moved ``collections.Grouping`` to the "Rejected Alternatives" section,
but that's more like a "personal 2nd choices" instead of "rejected".

[...]
> __init__ and update would take an iterable of (key, value) pairs, rather
> than a single sequence.
>

I added a better demonstration in the PEP for handling that kind of input.
You have one of two strategies with my proposed function.

Either create a reverse lookup dict:
    d = {v: k for k, v in items}
    grouping(d, key=lambda k: d[k])

Or discard the keys after grouping:
    groups = grouping(items, key=lambda t: t[0])
    groups = {k: [v for _, v in g] for k, g in groups.items()}

While thinking of examples for this PEP, it's tempting to use
overly-simplified data. In practice, instead of (key, value) pairs, it's
usually either individual values or n-tuple rows. In the latter case,
sometimes the key should be dropped from the row when grouping, sometimes
kept in the row, and sometimes the key must be computed from multiple
values within the row.


[...] building up a data structure with word pairs, and a list of all the
> words that follow the pair in a piece of text. [...example code...]
>

I provided a similar example in my first draft, showing the creation of a
Markov chain data structure. A few folks gave the feedback that it was more
distracting from the PEP than useful. It's still there in the "stateful
key-function" example, but it's now just a few lines.

[...] if you are teaching, say data analysis with Python -- it might be
> nice to have this builtin, but if you are teaching "programming with
> Python" I'd probably encourage them to do it by hand first anyway :-)
>

I agree, but users in both cases will appreciate the proposed built-in.


On Sun, Jul 1, 2018 at 10:35 PM Chris Barker <chris.barker at noaa.gov> wrote:

> Though maybe list, set and Counter are the [aggregation collections] you'd
> want to use?
>

I've been searching the standard library and popular community libraries
for use of setdefault, defaultdict, groupby, and the word "group" or
"groups" periodically over the past year or so. I admit I haven't been as
systematic as maybe I should have been, but I feel like I've been pretty
thorough.

The majority of grouping uses a list. A significant portion use a set. A
handful use a Counter. And that's basically it. Sometimes there's a
specialized container class, but they are generally composed of a list,
set, or Counter. There may have been other types, but if it was
interesting, I think I'd have written down an example of it in my notes.

Most other languages with a similar tool have decided to return a mapping
of lists or the equivalent for that language. If we make that choice, we're
in good company.


[...]
> before making any decisions about the best API, it would probably be a
> good idea to collect examples of the kind of data that people really do
> need to group like this. Does it come in (key, value) pairs naturally? or
> in one big sequence with a key function that's easy to write? who knows
> without examples of real world use cases.
>

It may not come across in the PEP how much research I've put into this.
I'll some time to compile the evidence, but I'm confident that it's more
common to need a key-function than to have (key, value) pairs. I'll get
back to you soon(ish) with data.


-- Michael

PS. Not to bikeshed, but a Grouper is a kind of fish. :-)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180702/558c3153/attachment-0001.html>

From nicolas.rolin at tiime.fr  Mon Jul  2 05:31:47 2018
From: nicolas.rolin at tiime.fr (Nicolas Rolin)
Date: Mon, 2 Jul 2018 11:31:47 +0200
Subject: [Python-ideas] grouping / dict of lists
In-Reply-To: <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CAP7+vJJ6vN9H8esj8tS+bSKsFELh8q8ywYHAW8Ti_upEJHf1vg@mail.gmail.com>
 <CADqi797HXzHcUnrNLp7r+pFEM5fcApiKHV4s=P5-D6mXHYYv-g@mail.gmail.com>
 <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
Message-ID: <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>

I think the current default quite weird, as it pretty much account to a
count() of each key (which can be useful, but not really what I except from
a grouping). I would prefer a default that might return an error to a
default that says ok and output something that is not what I might want.
For example the default could be such that grouping unpack tuples (key,
value) from the iterator and do what's expected with it (group value by
key). It is quite reasonable, and you have one example with (key, value) in
your example, and no example with the current default. It also allows to
use syntax of the kind

>grouping((food_type, food_name for food_type, food_name in foods))

which is pretty nice to have.

-- 
Nicolas Rolin



2018-07-02 9:43 GMT+02:00 Michael Selik <mike at selik.org>:

> I made some heavy revisions to the PEP. Linking again for convenience.
> https://github.com/selik/peps/blob/master/pep-9999.rst
>
> Replying to Guido, Nick, David, Chris, and Ivan in 4 sections below.
>
>
> [Guido]
> On Fri, Jun 29, 2018 at 11:25 PM Guido van Rossum <guido at python.org>
> wrote:
>
>> On Fri, Jun 29, 2018 at 3:23 PM Michael Selik <mike at selik.org> wrote:
>>
>>> On Fri, Jun 29, 2018 at 2:43 PM Guido van Rossum <guido at python.org>
>>> wrote:
>>>
>>>> On a quick skim I see nothing particularly objectionable or
>>>> controversial in your PEP, except I'm unclear why it needs to be a class
>>>> method on `dict`.
>>>>
>>>
>>> Since it constructs a basic dict, I thought it belongs best as a dict
>>> constructor like dict.fromkeys. It seemed to match other classmethods like
>>> datetime.now.
>>>
>>
>> It doesn't strike me as important enough. Surely not every stdlib
>> function that returns a fresh dict needs to be a class method on dict!
>>
>
> Thinking back, I may have chosen the name "groupby" first, following
> `itertools.groupby`, SQL, and other languages, and I wanted to make a clear
> distinction from `itertools.groupby`. Putting it on the `dict` namespace
> clarified that it's returning a dict.
>
> However, naming it `grouping` allows it to be a stand-alone function.
>
> But I still think it is much better off as a helper function in itertools.
>>
> I considered placing it in the itertools module, but decided against
>>> because it doesn't return an iterator. I'm open to that if that's the
>>> consensus.
>>>
>>
>> You'll never get consensus on anything here, but you have my blessing for
>> this without consensus.
>>
>
> That feels like a success, but I'm going to be a bit more ambitious and
> try to persuade you that `grouping` belongs in the built-ins. I revised my
> draft to streamline the examples and make a clearer comparison with
> existing tools.
>
>
> [Nick]
> On Sat, Jun 30, 2018 at 2:01 AM Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> I'm not sure if the draft was updated since [Guido] looked at it, but it
>
>
>> does mention that one benefit of the collections.Grouping approach is
>> being able to add native support for mapping a callable across every
>> individual item in the collection (ignoring the group structure), as
>> well as for applying aggregate functions to reduce the groups to
>> single values in a standard dict.
>>
>> Delegating those operations to the container API that way then means
>> that other libraries can expose classes that implement the grouping
>> API, but with a completely different backend storage model.
>>
>
> While it'd be nice to create a standard interface as you point out, my
> primary goal is to create an "obvious" way for both beginners and experts
> to group, classify, categorize, bucket, demultiplex, taxonomize, etc. I
> started revising the PEP last night and found myself getting carried away
> with adding methods to the Grouping class that were more distracting than
> useful. Since the most important thing is to make this as accessible and
> easy as possible, I re-focused the proposal on the core idea of grouping.
>
>
> [Ivan, Chris, David]
> On Sun, Jul 1, 2018 at 7:29 PM David Mertz <mertz at gnosis.cx> wrote:
>
>> {k:set(v) for k,v in deps.items()}
>> {k:Counter(v) for k,v in deps.items()}
>>
>
> I had dropped those specific examples in favor of generically "func(g)",
> but added them back. Your discussion with Ivan and Chris showed that it was
> useful to be specific.
>
>
> [Chris]
> On Sat, Jun 30, 2018 at 10:18 PM Chris Barker <chris.barker at noaa.gov>
> wrote:
>
>> I'm really warming to the:
>> Alternate: collections.Grouping
>> version -- I really like this as a kind of custom mapping, rather than
>> "just a function" (or alternate constructor) -- and I like your point that
>> it can have a bit of functionality built in other than on construction.
>>
>
> I moved ``collections.Grouping`` to the "Rejected Alternatives" section,
> but that's more like a "personal 2nd choices" instead of "rejected".
>
> [...]
>> __init__ and update would take an iterable of (key, value) pairs, rather
>> than a single sequence.
>>
>
> I added a better demonstration in the PEP for handling that kind of input.
> You have one of two strategies with my proposed function.
>
> Either create a reverse lookup dict:
>     d = {v: k for k, v in items}
>     grouping(d, key=lambda k: d[k])
>
> Or discard the keys after grouping:
>     groups = grouping(items, key=lambda t: t[0])
>     groups = {k: [v for _, v in g] for k, g in groups.items()}
>
> While thinking of examples for this PEP, it's tempting to use
> overly-simplified data. In practice, instead of (key, value) pairs, it's
> usually either individual values or n-tuple rows. In the latter case,
> sometimes the key should be dropped from the row when grouping, sometimes
> kept in the row, and sometimes the key must be computed from multiple
> values within the row.
>
>
> [...] building up a data structure with word pairs, and a list of all the
>> words that follow the pair in a piece of text. [...example code...]
>>
>
> I provided a similar example in my first draft, showing the creation of a
> Markov chain data structure. A few folks gave the feedback that it was more
> distracting from the PEP than useful. It's still there in the "stateful
> key-function" example, but it's now just a few lines.
>
> [...] if you are teaching, say data analysis with Python -- it might be
>> nice to have this builtin, but if you are teaching "programming with
>> Python" I'd probably encourage them to do it by hand first anyway :-)
>>
>
> I agree, but users in both cases will appreciate the proposed built-in.
>
>
> On Sun, Jul 1, 2018 at 10:35 PM Chris Barker <chris.barker at noaa.gov>
> wrote:
>
>> Though maybe list, set and Counter are the [aggregation collections]
>> you'd want to use?
>>
>
> I've been searching the standard library and popular community libraries
> for use of setdefault, defaultdict, groupby, and the word "group" or
> "groups" periodically over the past year or so. I admit I haven't been as
> systematic as maybe I should have been, but I feel like I've been pretty
> thorough.
>
> The majority of grouping uses a list. A significant portion use a set. A
> handful use a Counter. And that's basically it. Sometimes there's a
> specialized container class, but they are generally composed of a list,
> set, or Counter. There may have been other types, but if it was
> interesting, I think I'd have written down an example of it in my notes.
>
> Most other languages with a similar tool have decided to return a mapping
> of lists or the equivalent for that language. If we make that choice, we're
> in good company.
>
>
> [...]
>> before making any decisions about the best API, it would probably be a
>> good idea to collect examples of the kind of data that people really do
>> need to group like this. Does it come in (key, value) pairs naturally? or
>> in one big sequence with a key function that's easy to write? who knows
>> without examples of real world use cases.
>>
>
> It may not come across in the PEP how much research I've put into this.
> I'll some time to compile the evidence, but I'm confident that it's more
> common to need a key-function than to have (key, value) pairs. I'll get
> back to you soon(ish) with data.
>
>
> -- Michael
>
> PS. Not to bikeshed, but a Grouper is a kind of fish. :-)
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>


-- 

--
*Nicolas Rolin* | Data Scientist
+ 33 631992617 - nicolas.rolin at tiime.fr <prenom.nom at tiime.fr>


*15 rue Auber, **75009 Paris*
*www.tiime.fr <http://www.tiime.fr>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180702/2123393e/attachment-0001.html>

From mike at selik.org  Mon Jul  2 05:52:03 2018
From: mike at selik.org (Michael Selik)
Date: Mon, 2 Jul 2018 02:52:03 -0700
Subject: [Python-ideas] grouping / dict of lists
In-Reply-To: <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CAP7+vJJ6vN9H8esj8tS+bSKsFELh8q8ywYHAW8Ti_upEJHf1vg@mail.gmail.com>
 <CADqi797HXzHcUnrNLp7r+pFEM5fcApiKHV4s=P5-D6mXHYYv-g@mail.gmail.com>
 <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
Message-ID: <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>

On Mon, Jul 2, 2018 at 2:32 AM Nicolas Rolin <nicolas.rolin at tiime.fr> wrote:

> I think the current default quite weird, as it pretty much account to a
> count() of each key (which can be useful, but not really what I except from
> a grouping). I would prefer a default that might return an error to a
> default that says ok and output something that is not what I might want.
> For example the default could be such that grouping unpack tuples (key,
> value) from the iterator and do what's expected with it (group value by
> key). It is quite reasonable, and you have one example with (key, value) in
> your example, and no example with the current default. It also allows to
> use syntax of the kind
>
> >grouping((food_type, food_name for food_type, food_name in foods))
>
> which is pretty nice to have.
>

I'm of two minds on this point. First, I agree that it'd be nice to handle
the (key, value) pair case more elegantly. It comes to mind often when
writing examples, even if proportionally less in practice.

Second, I'll paraphrase "Jakob's Law of the Internet User Experience" --
users spend most of their time using *other* functions. Because
itertools.groupby and other functions in Python established a standard for
the behavior of key-functions, I want to keep that standard.

Third, some classes might have a rich equality method that allows many
interesting values to all wind up in the same group even if using the
default "identity" key-function.

Thanks for the suggestion. I'll include it in the PEP, at least for
documenting all reasonable options.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180702/d6d103b2/attachment.html>

From mike at selik.org  Mon Jul  2 06:14:40 2018
From: mike at selik.org (Michael Selik)
Date: Mon, 2 Jul 2018 03:14:40 -0700
Subject: [Python-ideas] grouping / dict of lists
In-Reply-To: <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CAP7+vJJ6vN9H8esj8tS+bSKsFELh8q8ywYHAW8Ti_upEJHf1vg@mail.gmail.com>
 <CADqi797HXzHcUnrNLp7r+pFEM5fcApiKHV4s=P5-D6mXHYYv-g@mail.gmail.com>
 <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
Message-ID: <CADqi797egicvtVLFtuVTpY5paB9J_XeAXs0cx6Rjhhk57LoK6A@mail.gmail.com>

On Mon, Jul 2, 2018 at 2:52 AM Michael Selik <mike at selik.org> wrote:

> On Mon, Jul 2, 2018 at 2:32 AM Nicolas Rolin <nicolas.rolin at tiime.fr>
> wrote:
>
>> I think the current default quite weird, as it pretty much account to a
>> count() of each key (which can be useful, but not really what I except from
>> a grouping). I would prefer a default that might return an error to a
>> default that says ok and output something that is not what I might want.
>> For example the default could be such that grouping unpack tuples (key,
>> value) from the iterator and do what's expected with it (group value by
>> key). It is quite reasonable, and you have one example with (key, value) in
>> your example, and no example with the current default. It also allows to
>> use syntax of the kind
>>
>> >grouping((food_type, food_name for food_type, food_name in foods))
>>
>> which is pretty nice to have.
>>
>
> I'm of two minds on this point. First, I agree that it'd be nice to handle
> the (key, value) pair case more elegantly. It comes to mind often when
> writing examples, even if proportionally less in practice.
>
> Second, I'll paraphrase "Jakob's Law of the Internet User Experience" --
> users spend most of their time using *other* functions. Because
> itertools.groupby and other functions in Python established a standard for
> the behavior of key-functions, I want to keep that standard.
>
> Third, some classes might have a rich equality method that allows many
> interesting values to all wind up in the same group even if using the
> default "identity" key-function.
>
> Thanks for the suggestion. I'll include it in the PEP, at least for
> documenting all reasonable options.
>


It might not be pure (does not default to identity key-function), but it
sure seems practical.

::

    from itertools import groupby as _groupby
    from operator import itemgetter

    def grouping(iterable, key=itemgetter(0)):
        '''
        Group elements of an iterable into a dict of lists.

        The ``key`` is a function computing a key value for each
        element.  Each key corresponds to a group -- a list of elements
        in the same order as encountered.

        By default, the key-function gets the 0th index of each element.

            .>>> grouping(['apple', 'banana', 'aardvark'])
            {'a': ['apple', 'aardvark'], 'b': ['banana']}

        '''
        groups = {}
        for k, g in _groupby(iterable, key):
            groups.setdefault(k, []).extend(g)
        return groups
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180702/04683b89/attachment.html>

From nicolas.rolin at tiime.fr  Mon Jul  2 06:14:57 2018
From: nicolas.rolin at tiime.fr (Nicolas Rolin)
Date: Mon, 2 Jul 2018 12:14:57 +0200
Subject: [Python-ideas] grouping / dict of lists
In-Reply-To: <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CAP7+vJJ6vN9H8esj8tS+bSKsFELh8q8ywYHAW8Ti_upEJHf1vg@mail.gmail.com>
 <CADqi797HXzHcUnrNLp7r+pFEM5fcApiKHV4s=P5-D6mXHYYv-g@mail.gmail.com>
 <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
Message-ID: <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>

My question would be : does it have to be a key function ? Can't we just
remove the "key" argument ?
Because for pretty much all the given examples, I would find my default as
readable and nearly as short as the "key" syntax :

> grouping(words, key=len)
grouping((len(word), word for word in words))

>grouping(names, key=itemgetter(0))
grouping((name_initial, name_initial+_name for name_initial, *_name in
names)

>grouping(contacts, key=itemgetter('city')
grouping((contact['city'], contact for contact in contacts)

>grouping(employees, key=itemgetter('department'))
grouping((employee['department'], employee for employee in employees)

>grouping(os.listdir('.'), key=lambda filepath:
os.path.splitext(filepath)[1])
grouping((os.path.splitext(filepath)[1]), filepath for filepath in
os.listdir('.'))

>grouping(transactions, key=lambda v: 'debit' if v > 0 else 'credit')
grouping(('debit' if v > 0 else 'credit', transaction_amount for
transaction_amount in transactions))

The code is slightly more verbose, but it is akin to filter(iterable,
function) vs (i for i in iterable if function(i)).

-- 
Nicolas Rolin

2018-07-02 11:52 GMT+02:00 Michael Selik <mike at selik.org>:

> On Mon, Jul 2, 2018 at 2:32 AM Nicolas Rolin <nicolas.rolin at tiime.fr>
> wrote:
>
>> I think the current default quite weird, as it pretty much account to a
>> count() of each key (which can be useful, but not really what I except from
>> a grouping). I would prefer a default that might return an error to a
>> default that says ok and output something that is not what I might want.
>> For example the default could be such that grouping unpack tuples (key,
>> value) from the iterator and do what's expected with it (group value by
>> key). It is quite reasonable, and you have one example with (key, value) in
>> your example, and no example with the current default. It also allows to
>> use syntax of the kind
>>
>> >grouping((food_type, food_name for food_type, food_name in foods))
>>
>> which is pretty nice to have.
>>
>
> I'm of two minds on this point. First, I agree that it'd be nice to handle
> the (key, value) pair case more elegantly. It comes to mind often when
> writing examples, even if proportionally less in practice.
>
> Second, I'll paraphrase "Jakob's Law of the Internet User Experience" --
> users spend most of their time using *other* functions. Because
> itertools.groupby and other functions in Python established a standard for
> the behavior of key-functions, I want to keep that standard.
>
> Third, some classes might have a rich equality method that allows many
> interesting values to all wind up in the same group even if using the
> default "identity" key-function.
>
> Thanks for the suggestion. I'll include it in the PEP, at least for
> documenting all reasonable options.
>



-- 

--
*Nicolas Rolin* | Data Scientist
+ 33 631992617 - nicolas.rolin at tiime.fr <prenom.nom at tiime.fr>


*15 rue Auber, **75009 Paris*
*www.tiime.fr <http://www.tiime.fr>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180702/61faf761/attachment-0001.html>

From steve at pearwood.info  Mon Jul  2 06:47:47 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 2 Jul 2018 20:47:47 +1000
Subject: [Python-ideas] grouping / dict of lists
In-Reply-To: <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CAP7+vJJ6vN9H8esj8tS+bSKsFELh8q8ywYHAW8Ti_upEJHf1vg@mail.gmail.com>
 <CADqi797HXzHcUnrNLp7r+pFEM5fcApiKHV4s=P5-D6mXHYYv-g@mail.gmail.com>
 <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
Message-ID: <20180702104747.GZ14437@ando.pearwood.info>

On Mon, Jul 02, 2018 at 02:52:03AM -0700, Michael Selik wrote:

> Third, some classes might have a rich equality method that allows many
> interesting values to all wind up in the same group even if using the
> default "identity" key-function.

I would expect an identity key function to group by *identity* (is), not 
equality.

But I would expect the default grouper to group by *equality*.


-- 
Steve

From chris.barker at noaa.gov  Mon Jul  2 11:48:28 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Mon, 2 Jul 2018 08:48:28 -0700
Subject: [Python-ideas] grouping / dict of lists
In-Reply-To: <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CAP7+vJJ6vN9H8esj8tS+bSKsFELh8q8ywYHAW8Ti_upEJHf1vg@mail.gmail.com>
 <CADqi797HXzHcUnrNLp7r+pFEM5fcApiKHV4s=P5-D6mXHYYv-g@mail.gmail.com>
 <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
Message-ID: <CALGmxELLvdpOniH8yohfZygx1Fb7nfT=ESeHrJF27++5dM9ohA@mail.gmail.com>

On Fri, Jun 29, 2018 at 11:25 PM, Guido van Rossum <guido at python.org> wrote:


> Hm, this actually feels heavier to me. But then again I never liked or
> understood the need for Counter --
>

actually, me neither -- and partly because it's too lightweight -- that is,
it's still a regular dict, and you pretty much have to know that to use it.
That it, it provides a nice counting constructor, but after that, it's just
a key:integer dict :-)

But in this case, I think there is more of an argument for a custom class
-- if al it were was a dict with a custom constructor (and update) method,
then yeah, better to have a function. But there is more tha could be built
on top of a grouping class, one that happened to be a dict under the hod,
but really s its own thing, with a handful of interfaces and methods that
are specific to it.

More detail elsewhere  in the discussion.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180702/e031f941/attachment.html>

From mike at selik.org  Mon Jul  2 12:39:00 2018
From: mike at selik.org (Michael Selik)
Date: Mon, 2 Jul 2018 09:39:00 -0700
Subject: [Python-ideas] grouping / dict of lists
In-Reply-To: <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CAP7+vJJ6vN9H8esj8tS+bSKsFELh8q8ywYHAW8Ti_upEJHf1vg@mail.gmail.com>
 <CADqi797HXzHcUnrNLp7r+pFEM5fcApiKHV4s=P5-D6mXHYYv-g@mail.gmail.com>
 <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
Message-ID: <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>

>
> On Mon, Jul 2, 2018 at 2:32 AM Nicolas Rolin <nicolas.rolin at tiime.fr>
>> wrote:
>>
>>> For example the default could be such that grouping unpack tuples (key,
>>> value) from the iterator and do what's expected with it (group value by
>>> key). It is quite reasonable, and you have one example with (key, value) in
>>> your example, and no example with the current default.
>>>
>>
On Mon, Jul 2, 2018 at 3:22 AM Nicolas Rolin <nicolas.rolin at tiime.fr> wrote:

> My question would be: does it have to be a key function? Can't we just
> remove the "key" argument?
>

In the examples or from the parameters? A key function is necessary to
support a wide variety of uses.


Because for pretty much all the given examples, I would find my default as
> readable and nearly as short as the "key" syntax :
>
> > grouping(words, key=len)
> grouping((len(word), word for word in words))
>

I think the fact that you misplaced a closing parenthesis demonstrates how
the key-function pattern can be more clear.


The code is slightly more verbose, but it is akin to filter(iterable,
> function) vs (i for i in iterable if function(i)).
>

Sometimes I prefer ``map`` and sometimes I prefer a list comprehension. It
usually hinges on whether I think the reader might get confused over what
one of the elements is. If so, I like to write out the comprehension to
provide that extra variable name for clarity.

I'd write:
    map(len, words)

But I'd also write
    [len(fullname) for fullname in contacts]

I appreciate that defaulting the grouping key-function to ``itemgetter(0)``
would enable a pleasant flexibility for people to make that same choice for
each use. I haven't fully come around to that, yet, because so many other
tools use the equality function as the default.



On Mon, Jul 2, 2018 at 3:48 AM Steven D'Aprano <steve at pearwood.info> wrote:

> On Mon, Jul 02, 2018 at 02:52:03AM -0700, Michael Selik wrote:
> > Third, some classes might have a rich equality method that allows many
> > interesting values to all wind up in the same group even if using the
> > default "identity" key-function.
>
> I would expect an identity key function to group by *identity* (is), not
> equality. But I would expect the default grouper to group by *equality*.


Yep, I should have been saying "equality function" instead of "identity
function." Thanks for the clarification.

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180702/6f6f5950/attachment.html>

From guido at python.org  Mon Jul  2 14:26:02 2018
From: guido at python.org (Guido van Rossum)
Date: Mon, 2 Jul 2018 11:26:02 -0700
Subject: [Python-ideas] grouping / dict of lists
In-Reply-To: <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CAP7+vJJ6vN9H8esj8tS+bSKsFELh8q8ywYHAW8Ti_upEJHf1vg@mail.gmail.com>
 <CADqi797HXzHcUnrNLp7r+pFEM5fcApiKHV4s=P5-D6mXHYYv-g@mail.gmail.com>
 <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
Message-ID: <CAP7+vJ+yVLLwQrq9N4mgrBdXxcMzQ0W6BnPDQFtvCTzSSuk_2w@mail.gmail.com>

On Mon, Jul 2, 2018 at 12:50 AM Michael Selik <mike at selik.org> wrote:

>
> [Guido]
>
>> You'll never get consensus on anything here, but you have my blessing for
>> this without consensus.
>>
>
> That feels like a success, but I'm going to be a bit more ambitious and
> try to persuade you that `grouping` belongs in the built-ins. I revised my
> draft to streamline the examples and make a clearer comparison with
> existing tools.
>

Sorry, I'm not biting. This will not be a builtin nor a method on a
builtin. I'm going to mute this thread because it's getting too noisy.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180702/df6ddf8c/attachment.html>

From chris.barker at noaa.gov  Tue Jul  3 02:50:49 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Mon, 2 Jul 2018 23:50:49 -0700
Subject: [Python-ideas] grouping / dict of lists
In-Reply-To: <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CAP7+vJJ6vN9H8esj8tS+bSKsFELh8q8ywYHAW8Ti_upEJHf1vg@mail.gmail.com>
 <CADqi797HXzHcUnrNLp7r+pFEM5fcApiKHV4s=P5-D6mXHYYv-g@mail.gmail.com>
 <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
Message-ID: <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>

On Mon, Jul 2, 2018 at 9:39 AM, Michael Selik <mike at selik.org> wrote:

> On Mon, Jul 2, 2018 at 2:32 AM Nicolas Rolin <nicolas.rolin at tiime.fr>
>>> wrote:
>>>
>>>> For example the default could be such that grouping unpack tuples (key,
>>>> value) from the iterator and do what's expected with it (group value by
>>>> key). It is quite reasonable, and you have one example with (key, value) in
>>>> your example, and no example with the current default.
>>>>
>>>
I agree, the default should do something that has some chance of being
useful on its own, and ideally, the most "common" use, if we can identify
that.


> On Mon, Jul 2, 2018 at 3:22 AM Nicolas Rolin <nicolas.rolin at tiime.fr>
> wrote:
>
>> My question would be: does it have to be a key function? Can't we just
>> remove the "key" argument?
>>
>
> In the examples or from the parameters? A key function is necessary to
> support a wide variety of uses.
>

not if you have the expectation of an iterable of (key, value) pairs as the
input -- then any processing required to get a different key happens before
hand, allowing folks to use comprehension syntax.

as so: :-)

Because for pretty much all the given examples, I would find my default as
>> readable and nearly as short as the "key" syntax :
>>
>> > grouping(words, key=len)
>> grouping((len(word), word for word in words))
>>
>
> I think the fact that you misplaced a closing parenthesis demonstrates how
> the key-function pattern can be more clear.
>

I think it demonstrates that you shouldn't post untested code :-) -- the
missing paren is a syntax error -- it would be caught right away in real
life.

>
> The code is slightly more verbose, but it is akin to filter(iterable,
>> function) vs (i for i in iterable if function(i)).
>>
>
> Sometimes I prefer ``map`` and sometimes I prefer a list comprehension.
>

That is a "problem" with python: there are two ways to do things like map
and filter, and one way is not always the clearest choice.

But I wonder if map and filter would exist if they didn't pre-date
comprehensions..... That aside, the comprehension approach is pretty well
liked and useful. And almost always prefer it -- an expression is simple on
the eyes to me :-)

But when it's really a win is when you don't have a handy built-in function
to do what you want, even though it's simple expression.

With the map, filter, key approach, you have to write a bunch of little
utility functions or lambdas, which can really clutter up the code.

If so, I like to write out the comprehension to provide that extra variable
> name for clarity.
>
> I'd write:
>     map(len, words)
>
> But I'd also write
>     [len(fullname) for fullname in contacts]
>

A key (pun intended) issue is that passing functions around looks so neat
and  clean when it's a simple built in function like "len" -- but if not,
the it gets uglier, like:

map(lambda name: name.first_name, all_names)

vs

[name.first_name for nam in all names]

I really like the comprehension form much better when what you really want
is a simple expression like an attribute access or index or simple
calculation, or ....

I appreciate that defaulting the grouping key-function to ``itemgetter(0)``
> would enable a pleasant flexibility for people to make that same choice for
> each use. I haven't fully come around to that, yet, because so many other
> tools use the equality function as the default.
>

well, kinda, but maybe those aren't "pythonic" :-)

(and yes, itemgetter() is probably a better option than lambda in many
cases -- but I always forget that exists)

I started out in this topic answering a question about how to do a grouping
for a list of tuples, in that case the OP wanted a comprehension. I don't
think there's a good way to get a direct comprehension, but I do think we
can make a class of function that takes something you could build with a
comprehension.

And I took a look at itertools.groupby, and found it very, very awkward,
ultimately arriving at:

student_school_list =  [('Fred', 'SchoolA'),
                        ('Bob', 'SchoolB'),
                        ('Mary', 'SchoolA'),
                        ('Jane', 'SchoolB'),
                        ('Nancy', 'SchoolC')]

grouped = {a:[t[0] for t in b] for a,b in groupby(sorted(student_school_list,
key=lambda t: t[1]), key=lambda t: t[1])}

{'SchoolA': ['Fred', 'Mary'], 'SchoolB': ['Bob', 'Jane'], 'SchoolC':
['Nancy']}

So why is that so painful?

("it" is itertools.groupby)

a) it  returns an iterable of tuples, so to get a dict you need to do the
dict comp
b) it requires the data to be sorted -- so you ned to sort it first
c) I need to provide a key function to sort by
d) I need to provide (the same, always?) key function to group by
e) when I make the dict, I need to make the list, and use an expression to
get the value I want.
f) because I need those key functions, I need to use lambda for what could
be a simple expression


So the current proposal in the PEP makes that a lot better:

a) It makes a dict, so that step is done
b) It doesn't require the data to be sorted

but:

d) I still need the key function to do anything useful
e) If my data aren't clean, I need to do some post-processing to get the
value I want.

So -- the above example with the (current version of the) PEP's function:

In [35]: grouping(student_school_list, key=lambda t: t[1])
Out[35]:
{'SchoolA': [('Fred', 'SchoolA'), ('Mary', 'SchoolA')],
 'SchoolB': [('Bob', 'SchoolB'), ('Jane', 'SchoolB')],
 'SchoolC': [('Nancy', 'SchoolC')]}

Darn! that's not right -- I need to clean up the values, too:

gr = { k: [t[0] for t in l] for k, l in gr.items()}

OK, but pretty painful really, so I guess I should clean it up first, but I
can't actually see any clean way to do that! Am I missing something?

Ah -- I see it in your PEP: "Sequences of values that are already paired
with their keys can be easily transformed after grouping." -- sorry that
code is not "easily" -- having to write that kind of code makes this whole
thing pretty much useless compared to using, say, setdefault() in the first
place.

One option would be to add a value function, to unpack the value, analogous
to the key function:

In [44]: gr = grouping(student_school_list, key=lambda t: t[1],
value=lambda t: t[0])

Out[45]: {'SchoolA': ['Fred', 'Mary'], 'SchoolB': ['Bob', 'Jane'],
'SchoolC': ['Nancy']}

That's pretty slick.

However, I still much prefer an API that assumes an iterator of (key,value)
pairs:


def grouping(iterable):
    groups = {}
    for k, g in iterable:
        groups.setdefault(k, []).append(value)
        return groups

(easy to impliment :-) )

And now you get something that "just works" for at least one case:

In [54]: def grouping(iterable):
    ...:     groups = {}
    ...:     for key, value in iterable:
    ...:         groups.setdefault(key, []).append(value)
    ...:     return groups
    ...:

In [55]: school_student_list
Out[55]:
[('SchoolA', 'Fred'),
 ('SchoolB', 'Bob'),
 ('SchoolA', 'Mary'),
 ('SchoolB', 'Jane'),
 ('SchoolC', 'Nancy')]

In [56]: grouping(school_student_list)
Out[56]: {'SchoolA': ['Fred', 'Mary'], 'SchoolB': ['Bob', 'Jane'],
'SchoolC': ['Nancy']}

And if you need to massage the data you can do so with a generator
expression:

In [58]: grouping((reversed(t) for t in student_school_list))

Out[58]: {'SchoolA': ['Fred', 'Mary'], 'SchoolB': ['Bob', 'Jane'],
'SchoolC': ['Nancy']}

And here are the examples from the PEP:
(untested -- I may hav missed some brackets, etc)

# Group words by length:


grouping(((len(word), word) for word in words))

# Group names by first initial:

grouping((name[0], name) for name in names))

# Group people by city:

grouping((contact.city, contact) for contact in contacts)

# Group employees by department:

grouping((employee['department'] for employee in employees)

# Group files by extension:

grouping((os.path.splitext(filepath)[1] for filepath in os.listdir('.')))

# Group transactions by type:

grouping(( 'debit' if v > 0 else 'credit' for v in transactions))

# Invert a dictionary, d, without discarding repeated values:

grouping(((v, k) for v, k in d.items()))

So that was an interesting exercise -- many of those are a bit clearer (or
more compact) with the key function. But I also notice a pattern -- all
thos examples fit very well into the key function pattern:

you want the entire item stored in your iterable.
you want to group by some quality of the item itself.

Perhaps those ARE the most common use cases -- I honestly don't know, but
from an earlier post:

" In practice, instead of (key, value) pairs, it's usually either
individual values or n-tuple rows. In the latter case, sometimes the key
should be dropped from the row when grouping, sometimes kept in the row,
and sometimes the key must be computed from multiple values within the row."

It seems that the comprehension style I'm suggesting would be a win for
the  case of n-tuple rows.

Doing this exercise has expanded my view, so I suggest that:

- keep the key function optional parameter.
- add a value function optional parameter. -- it really makes any case
where you don't want to store the whole item a lot easier.

- Have the default key function be itemgetter(0) and the default value
function be itemgetter(1) (or some similar way to implement default support
for processing an iterable of (key, value) pairs.

Having no value function and an equality default for the key function may
be "common", but it's also pretty much useless -- why have a useless
default?

Thinking this through now I do see that having key and value default to to
the pair method means that if you specify  key function, you will probably
have to specify a value function as well -- so maybe that's not ideal.

hmm.


A couple other comments:

implementation detail: Do you gain anything by using the itertools groupby?
over, say:

groups = {}
for item in iterable:
    groups.setdefault(key(item), []).append(item)

Final point:

I still prefer the class idea over a utility function, because:

* with a class, you can ad stuff to the grouping later:

a_grouping['key'] = value

or maybe a_grouping.add(item)

* with a class, you can add utility methods -- I kinda liked that in your
original PEP.

I see this in the section about a custom class: "Merging groupings is not a
one-liner," -- I'm pretty sure the update() method on my prototype was a
merge operation -- so very easy :-) -- and another argument for a custom
class.

final thought about custon class. If you want to be abel to do:

a_grouping['key'] = value

then that really reinforces the "natural" mapping of keys to values -- it's
kind of another way to think about this -- rather than thinking about it as
"groupby" function that creates a dict, think of it as a dict-like object,
that, instead of writing over an existing key, accumulates the multiple
values.

which means you really want the "normal" constructor to take an iterable of
(key, value) pairs, to make it more dict-like.

-CHB














-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180702/0b2a12c3/attachment-0001.html>

From chris.barker at noaa.gov  Tue Jul  3 03:46:42 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Tue, 3 Jul 2018 00:46:42 -0700
Subject: [Python-ideas] grouping / dict of lists
In-Reply-To: <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CAP7+vJJ6vN9H8esj8tS+bSKsFELh8q8ywYHAW8Ti_upEJHf1vg@mail.gmail.com>
 <CADqi797HXzHcUnrNLp7r+pFEM5fcApiKHV4s=P5-D6mXHYYv-g@mail.gmail.com>
 <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
Message-ID: <CALGmxEJDF2DENht69br9PT3rOYNuqX-idVr1ebhTFcbFDQ6bLw@mail.gmail.com>

On Mon, Jul 2, 2018 at 11:50 PM, Chris Barker <chris.barker at noaa.gov> wrote:

> - keep the key function optional parameter.
> - add a value function optional parameter. -- it really makes any case
> where you don't want to store the whole item a lot easier.
>
> - Have the default key function be itemgetter(0) and the default value
> function be itemgetter(1) (or some similar way to implement default support
> for processing an iterable of (key, value) pairs.
>
> Having no value function and an equality default for the key function may
> be "common", but it's also pretty much useless -- why have a useless
> default?
>
> Thinking this through now I do see that having key and value default to to
> the pair method means that if you specify  key function, you will probably
> have to specify a value function as well -- so maybe that's not ideal.
>

OK, I prototyped a class solution that defaults to key, value pairs, but
you can specify a key and/or value function. and with convoluted logic, if
you specify just a key, then the value defaults to the entire item. So
folks will pretty much get what they expect.

I think it's actually pretty slick -- best of both worlds?

Code here:

https://github.com/PythonCHB/grouper/blob/master/grouper/grouper.py

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180703/ef7ebd5e/attachment.html>

From mertz at gnosis.cx  Tue Jul  3 08:58:05 2018
From: mertz at gnosis.cx (David Mertz)
Date: Tue, 3 Jul 2018 08:58:05 -0400
Subject: [Python-ideas] grouping / dict of lists
In-Reply-To: <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CAP7+vJJ6vN9H8esj8tS+bSKsFELh8q8ywYHAW8Ti_upEJHf1vg@mail.gmail.com>
 <CADqi797HXzHcUnrNLp7r+pFEM5fcApiKHV4s=P5-D6mXHYYv-g@mail.gmail.com>
 <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
Message-ID: <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>

On Tue, Jul 3, 2018 at 2:52 AM Chris Barker via Python-ideas <
python-ideas at python.org> wrote:

> I'd write:
>>
>     map(len, words)
>>
>> But I'd also write
>>     [len(fullname) for fullname in contacts]
>>
>
> map(lambda name: name.first_name, all_names)
> vs
> [name.first_name for nam in all names]
>
> I really like the comprehension form much better when what you really want
> is a simple expression like an attribute access or index or simple
> calculation, or ....
>

Why not `map(attrgetter('first_name'), all_names)`?


> In [56]: grouping(school_student_list)
> Out[56]: {'SchoolA': ['Fred', 'Mary'], 'SchoolB': ['Bob', 'Jane'],
> 'SchoolC': ['Nancy']}
>

This one case is definitely nice. However...

And here are the examples from the PEP:
> (untested -- I may hav missed some brackets, etc)
>

What you've missed, in *several* examples is the value part of the tuple in
your API. You've pulled out the key, and forgotten to include anything in
the actual groups.  I have a hunch that if your API were used, this would
be a common pitfall.

I think this argues against your API and for Michael's that simply deals
with "sequences of groupable things."  That's much more like what one deals
with in SQL, and is familiar that way.  If the things grouped are compound
object such as dictionaries, objects with common attributes, named tuples,
etc. then the list of things in a group usually *does not* want the
grouping attribute removed.


> grouping(((len(word), word) for word in words))
> grouping((name[0], name) for name in names))
> grouping((contact.city, contact) for contact in contacts)
>

Good so far, but a lot of redundancy in always spelling tuple of
`(derived-key, object)`.


> grouping((employee['department'] for employee in employees)
> grouping((os.path.splitext(filepath)[1] for filepath in os.listdir('.')))
> grouping(('debit' if v > 0 else 'credit' for v in transactions))
>

And here you forget about the object itself 3 times in a row (or also
forget some derived "value" that you might want in your other comments).


> grouping(((v, k) for v, k in d.items()))
>

This is nice, and spelled correctly.


> So that was an interesting exercise -- many of those are a bit clearer (or
> more compact) with the key function. But I also notice a pattern -- all
> those examples fit very well into the key function pattern:
>

Yep.

I also think that the row-style "list of data" where you want to discard
the key from the values is nicely spelled (in the PEP) as:

INDEX = 0
grouping(sequence, key=lambda row: row.pop(INDEX))

groups = {}
> for item in iterable:
>     groups.setdefault(key(item), []).append(item)
>

I agree this seems better as an implementation.


> I still prefer the class idea over a utility function, because:
> * with a class, you can ad stuff to the grouping later:
>
> a_grouping['key'] = value
>
> or maybe a_grouping.add(item)
> * with a class, you can add utility methods -- I kinda liked that in your
> original PEP.
>

I agree still (after all, I proposed it to Michael).  But this seems minor,
and Guido seems not to like `collections` that much (or at least he
commented on not using Counter ... which I personally love to use and to
teach).

That said, a 'grouping()' function seems fine to me also... with a couple
utility functions (that need not be builtin, or even standard library
necessarily) in place of methods.  A lot of what methods would do can
easily be done using comprehensions as well, some examples are shown in the
PEP.


-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180703/98173c9b/attachment.html>

From mertz at gnosis.cx  Tue Jul  3 09:23:07 2018
From: mertz at gnosis.cx (David Mertz)
Date: Tue, 3 Jul 2018 09:23:07 -0400
Subject: [Python-ideas] Where should grouping() live (was: grouping / dict
 of lists)
In-Reply-To: <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CAP7+vJJ6vN9H8esj8tS+bSKsFELh8q8ywYHAW8Ti_upEJHf1vg@mail.gmail.com>
 <CADqi797HXzHcUnrNLp7r+pFEM5fcApiKHV4s=P5-D6mXHYYv-g@mail.gmail.com>
 <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
Message-ID: <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>

Guido said he has mooted this discussion, so it's probably not reaching
him.  It took one thousand fewer messages for him to stop following this
than with PEP 572, for some reason :-).

But before putting it on auto-archive, the BDFL said (1) NO GO on getting a
new builtin; (2) NO OBJECTION to putting it in itertools.

My problem with the second idea is that *I* find it very wrong to have
something in itertools that does not return an iterator.  It wrecks the
combinatorial algebra of the module.

That said, it's easy to fix... and I believe independently useful.  Just
make grouping() a generator function rather than a plain function.  This
lets us get an incremental grouping of an iterable.  This can be useful if
the iterable is slow or infinite, but the partial groupings are useful in
themselves.

Python 3.7.0 (default, Jun 28 2018, 07:39:16)
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from grouping import grouping
>>> grouped = grouping('AbBa', key=str.casefold)
>>> for dct in grouped: print(dct)
...
{'a': ['A']}
{'a': ['A'], 'b': ['b']}
{'a': ['A'], 'b': ['b', 'B']}
{'a': ['A', 'a'], 'b': ['b', 'B']}


This isn't so useful for the concrete sequence, but for this it would be
great:

for grouped in grouping(data_over_wire()):

    process_partial_groups(grouped)


The implementation need not and should not rely on "pre-grouping" with
itertools.groupby:

def grouping(iterable, key=None):
    groups = {}
    key = key or (lambda x: x)
    for item in iterable:
        groups.setdefault(key(item), []).append(item)
        yield groups



-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180703/b9d7b68a/attachment-0001.html>

From mertz at gnosis.cx  Tue Jul  3 09:24:30 2018
From: mertz at gnosis.cx (David Mertz)
Date: Tue, 3 Jul 2018 09:24:30 -0400
Subject: [Python-ideas] Where should grouping() live (was: grouping /
 dict of lists)
In-Reply-To: <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CAP7+vJJ6vN9H8esj8tS+bSKsFELh8q8ywYHAW8Ti_upEJHf1vg@mail.gmail.com>
 <CADqi797HXzHcUnrNLp7r+pFEM5fcApiKHV4s=P5-D6mXHYYv-g@mail.gmail.com>
 <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
Message-ID: <CAEbHw4bFxOKB-h1uUsWo2hYKQ84Y86UYL3AVsLj-wp7dMacwFg@mail.gmail.com>

On Tue, Jul 3, 2018 at 9:23 AM David Mertz <mertz at gnosis.cx> wrote:

> Guido said he has mooted this discussion, so it's probably not reaching
> him.
>

I meant 'muted'.  Hopefully he hasn't 'mooted' it.

-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180703/0fc57f03/attachment.html>

From mike at selik.org  Tue Jul  3 10:02:04 2018
From: mike at selik.org (Michael Selik)
Date: Tue, 3 Jul 2018 07:02:04 -0700
Subject: [Python-ideas] Where should grouping() live (was: grouping /
 dict of lists)
In-Reply-To: <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CAP7+vJJ6vN9H8esj8tS+bSKsFELh8q8ywYHAW8Ti_upEJHf1vg@mail.gmail.com>
 <CADqi797HXzHcUnrNLp7r+pFEM5fcApiKHV4s=P5-D6mXHYYv-g@mail.gmail.com>
 <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
Message-ID: <CADqi795_RJWA0=dbg=8TRpCwZJooEGY1-uu4Qq5Q4tBXq3gfyw@mail.gmail.com>

I'd prefer to simply write an example for the documentation or clarify the
existing ones, then add good answers to StackOverflow questions.


On Tue, Jul 3, 2018, 6:23 AM David Mertz <mertz at gnosis.cx> wrote:

> Guido said he has mooted this discussion, so it's probably not reaching
> him.  It took one thousand fewer messages for him to stop following this
> than with PEP 572, for some reason :-).
>
> But before putting it on auto-archive, the BDFL said (1) NO GO on getting
> a new builtin; (2) NO OBJECTION to putting it in itertools.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180703/da42d54c/attachment.html>

From nicolas.rolin at tiime.fr  Tue Jul  3 10:12:14 2018
From: nicolas.rolin at tiime.fr (Nicolas Rolin)
Date: Tue, 3 Jul 2018 16:12:14 +0200
Subject: [Python-ideas] grouping / dict of lists
In-Reply-To: <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CAP7+vJJ6vN9H8esj8tS+bSKsFELh8q8ywYHAW8Ti_upEJHf1vg@mail.gmail.com>
 <CADqi797HXzHcUnrNLp7r+pFEM5fcApiKHV4s=P5-D6mXHYYv-g@mail.gmail.com>
 <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
Message-ID: <CAOmnVUBp9vJjTp6v17LHy2C=7GzR63fb4ARZW2Dco022uTDpfA@mail.gmail.com>

2018-07-03 14:58 GMT+02:00 David Mertz <mertz at gnosis.cx>:

> On Tue, Jul 3, 2018 at 2:52 AM Chris Barker via Python-ideas <pytho
> <python-ideas at python.org>
>
> What you've missed, in *several* examples is the value part of the tuple
> in your API. You've pulled out the key, and forgotten to include anything
> in the actual groups.  I have a hunch that if your API were used, this
> would be a common pitfall.
>
> I think this argues against your API and for Michael's that simply deals
> with "sequences of groupable things."  That's much more like what one deals
> with in SQL, and is familiar that way.  If the things grouped are compound
> object such as dictionaries, objects with common attributes, named tuples,
> etc. then the list of things in a group usually *does not* want the
> grouping attribute removed.
>


I agree the examples have lisp-level of brackets. However by using the fact
tuples don't need brackets and the fact we can use a list instead of an
iterable (the grouper will have to stock the whole object in memory anyway,
and if it is really big, use itertools.groupby who is designed exactly for
that)
For example
grouping(((len(word), word) for word in words))
can be written
grouping([len(word), word for word in words])

which is less "bracket issue prone".
The main advantage of having this syntax is that it gives a definition very
close to the one of a dict comprehension, which is nice considering want we
obtain is a dict (without that feature I'm not sure I will never attempt to
use this function).
And that allows us to have the same construction syntax as a dictionary,
with an iterable of (key, value) pair (
https://docs.python.org/3.7/library/stdtypes.html#dict).



>  So that was an interesting exercise -- many of those are a bit clearer
>> (or more compact) with the key function. But I also notice a pattern -- all
>> those examples fit very well into the key function pattern:
>>
>
> Yep.
>

Well those were the examples used to showcase the keyfunction in the PEP.
This is as bad as it gets for the "initialization by comprehension" syntax.



> I agree still (after all, I proposed it to Michael).  But this seems
> minor, and Guido seems not to like `collections` that much (or at least he
> commented on not using Counter ... which I personally love to use and to
> teach).
>

Actually counter is very very close to grouping (replace the append  with
starting value [] in the for loop by a += with starting value 0 and
groupping becomes a counter), so adding it to collections makes the more
sense by a long shot.

As far as I'm concerned, CHB semantics and syntax for the groupper object
does everything that is needed, and even a little bit too much.
It could be called AppendDict and just accept a (key, value) interable in
input, and instead of doing dict[key] = value as a dict does, does
dict[key] = [value] if key not in dict else dict[key] + [value]  (and
should be coded in C I guess)

-- 
Nicolas Rolin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180703/0b7eea18/attachment.html>

From steve at pearwood.info  Tue Jul  3 11:24:14 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 4 Jul 2018 01:24:14 +1000
Subject: [Python-ideas] Where should grouping() live (was: grouping /
 dict of lists)
In-Reply-To: <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
References: <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
Message-ID: <20180703152413.GF14437@ando.pearwood.info>

On Tue, Jul 03, 2018 at 09:23:07AM -0400, David Mertz wrote:

> But before putting it on auto-archive, the BDFL said (1) NO GO on getting a
> new builtin; (2) NO OBJECTION to putting it in itertools.
> 
> My problem with the second idea is that *I* find it very wrong to have
> something in itertools that does not return an iterator.  It wrecks the
> combinatorial algebra of the module.

That seems like a reasonable objection to me.


> That said, it's easy to fix... and I believe independently useful.  Just
> make grouping() a generator function rather than a plain function.  This
> lets us get an incremental grouping of an iterable.

We already have something which lazily groups an iterable, returning 
groups as they are seen: groupby.

What makes grouping() different from groupby() is that it accumulates 
ALL of the subgroups rather than just consecutive subgroupings. To make 
it clear with a simulated example (ignoring the keys for brevity):

groupby("aaAAbbCaAB", key=str.upper)
=> groups "aaAA", "bb", "C", "aA", "B"

grouping("aaAAbbCaAB", key=str.upper)
=> groups "aaAAaA", "bbB", "C"

So grouping() cannot even begin returning values until it has processed 
the entire data set. In that regard, it is like sorted() -- it cannot be 
lazy, it is a fundamentally eager operation.

I propose that a better name which indicates the non-lazy nature of this 
function is *grouped* rather than grouping, like sorted().

As for where it belongs, perhaps the collections module is the least 
worst fit.


> This can be useful if
> the iterable is slow or infinite, but the partial groupings are useful in
> themselves.

Under what circumstances would the partial groupings be useful? Given 
the example above:

grouping("aaAAbbCaAB", key=str.upper)

when would you want to see the accumulated partial groups?

# again, ignoring the keys for brevity
"aaAA"
"aaAA", "bb"
"aaAA", "bb", "C"
"aaAAaA", "bb", "C"
"aaAAaA", "bbB", "C"


I don't see any practical use for this -- if you start processing the 
partial groupings immediately, you end up double-processing some 
of the items; if you wait until the last, what's the point of the 
intermediate values?

As you say yourself:

> This isn't so useful for the concrete sequence, but for this it would be
> great:
> 
> for grouped in grouping(data_over_wire()):
>     process_partial_groups(grouped)

And that demonstrated exactly why this would be a terrible bug magnet, 
suckering people into doing what you just did, and ending up processing 
values more than once.

To avoid that, your process_partial_groups would need to remember which 
values it has seen before for each key it has seen before.


-- 
Steve

From steve at pearwood.info  Tue Jul  3 11:33:39 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 4 Jul 2018 01:33:39 +1000
Subject: [Python-ideas] grouping / dict of lists
In-Reply-To: <CAOmnVUBp9vJjTp6v17LHy2C=7GzR63fb4ARZW2Dco022uTDpfA@mail.gmail.com>
References: <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAOmnVUBp9vJjTp6v17LHy2C=7GzR63fb4ARZW2Dco022uTDpfA@mail.gmail.com>
Message-ID: <20180703153338.GG14437@ando.pearwood.info>

On Tue, Jul 03, 2018 at 04:12:14PM +0200, Nicolas Rolin wrote:

> I agree the examples have lisp-level of brackets. However by using the fact
> tuples don't need brackets and the fact we can use a list instead of an
> iterable (the grouper will have to stock the whole object in memory anyway,
> and if it is really big, use itertools.groupby who is designed exactly for
> that)
> For example
> grouping(((len(word), word) for word in words))
> can be written
> grouping([len(word), word for word in words])
> 
> which is less "bracket issue prone".

Did you try this? It is a syntax error. Generator expressions must be 
surrounded by round brackets:

    grouping([len(word), (word for word in words)])

Or perhaps you meant this:

    grouping([(len(word), word) for word in words])

but now it seems pointless to use a list comprehension instead of a 
generator expression:

    grouping((len(word), word) for word in words)

but why are we using key values by hand when grouping ought to do it for 
us, as Michael Selik's version does?

    grouping(words, key=len)



-- 
Steve

From steve at pearwood.info  Tue Jul  3 11:47:07 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 4 Jul 2018 01:47:07 +1000
Subject: [Python-ideas] grouping / dict of lists
In-Reply-To: <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
Message-ID: <20180703154707.GH14437@ando.pearwood.info>

On Fri, Jun 29, 2018 at 10:53:34AM -0700, Michael Selik wrote:
> Hello,
> 
> I've drafted a PEP for an easier way to construct groups of elements from a
> sequence. https://github.com/selik/peps/blob/master/pep-9999.rst

Seems useful, but I suggest that since it has to process the entire data 
set eagerly, the name ought to be grouped() following the precedent set by 
sorted().

I also suggest using keyfunc as the second parameter, following the same 
convention as itertools.groupby. That gives this possible implementation:

def grouped(iterable, keyfunc=None):
    groups = {}
    for k, g in itertools.groupby(iterable, keyfunc):
        groups.setdefault(k, []).extend(g)
    return groups


Since Guido has ruled out making this a built-in, there's no really 
comfortable place in the standard library for it:

- it doesn't return an iterator (since it is eager, it would
  be pointless to yield key/items pairs instead of just 
  returning the dict), so itertools is not a good fit;

- it doesn't return a specialist class, so collections is not
  a good fit;

- there's currently no "useful utilities which aren't useful
  enough to be built-in" module.


I fear that this proposal will fall into that awkward position of being 
doomed by not having somewhere to put it.

(Your suggestion to consider this an alternate constructor of dicts 
seems more sensible all the time... but again Guido disagrees.)



-- 
Steve

From chris.barker at noaa.gov  Tue Jul  3 12:15:47 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Tue, 3 Jul 2018 09:15:47 -0700
Subject: [Python-ideas] Where should grouping() live (was: grouping /
 dict of lists)
In-Reply-To: <20180703152413.GF14437@ando.pearwood.info>
References: <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
 <20180703152413.GF14437@ando.pearwood.info>
Message-ID: <CALGmxE+Wk7WCqMEWb-1EHOjZHzdeykKRhyag3Fw0X+tDHr40Ew@mail.gmail.com>

On Tue, Jul 3, 2018 at 8:24 AM, Steven D'Aprano <steve at pearwood.info> wrote:

> On Tue, Jul 03, 2018 at 09:23:07AM -0400, David Mertz wrote:
>


> > My problem with the second idea is that *I* find it very wrong to have
> > something in itertools that does not return an iterator.  It wrecks the
> > combinatorial algebra of the module.
>

hmm -- that seems to be a pretty pedantic approach -- practicality beats
purity, after all :-)

I think we should first decide if a grouping() function is a useful
addition to the standard library (after all:  "not every two line function
needs to in the stdlib"), and f so, then we can find a home for it.

personally, I'm wondering if a "dicttools" or something module would make
sense -- I imagine there are all sorts of other handy utilities for working
with dicts that could go there. (though, yeah, we'd want to actually have a
handful of these before creating a new module :-) )

> That said, it's easy to fix... and I believe independently useful.  Just
> > make grouping() a generator function rather than a plain function.  This
> > lets us get an incremental grouping of an iterable.
>
> We already have something which lazily groups an iterable, returning
> groups as they are seen: groupby.
>
> What makes grouping() different from groupby() is that it accumulates
> ALL of the subgroups rather than just consecutive subgroupings.


well, yeah, but it wont actually get you those until you exhaust the
iterator -- so while it's different than itertools.groupby, it is different
than itertools.groupby(sorted(iterable))?

In short, this wouldn't really solve the problems that itertools.groupby
has for this sort of task -- so what's the point?

 > As for where it belongs, perhaps the collections module is the least
worst fit.

That depends some on whether we go with a simple function, in which case
collections is a pretty bad fit (but maybe still the least worse).

Personally I still like the idea of having this be special type of dict,
rather than "just a function" -- and then it's really obvious where to put
it :-)

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180703/579b4e95/attachment.html>

From chris.barker at noaa.gov  Tue Jul  3 13:33:55 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Tue, 3 Jul 2018 10:33:55 -0700
Subject: [Python-ideas] grouping / dict of lists
In-Reply-To: <20180703153338.GG14437@ando.pearwood.info>
References: <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAOmnVUBp9vJjTp6v17LHy2C=7GzR63fb4ARZW2Dco022uTDpfA@mail.gmail.com>
 <20180703153338.GG14437@ando.pearwood.info>
Message-ID: <CALGmxEJBYFEoPOFo7aVSN=Xy=sgSMnrF0k8pnt=VhJ6cvhS_4Q@mail.gmail.com>

On Tue, Jul 3, 2018 at 8:33 AM, Steven D'Aprano <steve at pearwood.info> wrote:

> but why are we using key values by hand when grouping ought to do it for
>
us, as Michael Selik's version does?
>
>     grouping(words, key=len)


because supplying a key function is sometimes cleaner, and sometimes uglier
than building up a comprehension -- which I think comes down to:

1) taste (style?)

2) whether the key function is as simple as the expression

3) whether you ned to transform the value in any way.

This argument is pretty much the same as whether you should use a
comprehension or map:

map(len, words)

vs

(len(word) for word in words)

In that case, map() looks cleaner and easier, but when you have something
less simple:

map(operator.attrgetter('something'), some_objects)

vs

(object.something for object in some_objects)

I like the comprehension better.

add a filter, and comps really get nicer -- after all they were added to
the language for a reason.

Then when you add the additional complication of needing to "transform" the
value as well, it's easy to do with the comprehension, but there is no way
to do it with only a key function.

I think the "confilct" here is that Micheal started with  a bunch of
examples that area ll well suited to the key_function approach, and Nicolas
started with a use-case that is better suited to the comprehension /
(key,value) approach.

However, while the key, value approach can be reasonably (if a bit klunky)
used everywhere the key function approach can, the opposite is not true
(for when the value needs to be transformed as well.

But in the spirit of "Python has both map and comprehensions", I say let's
use both!

* The default behavior is to process a (key.value) pair.

* A key function can be provided in which case it is used, and the value is
the full item.

* A value function can be provided, in which case, it is used to "process"
the value.

If this is too confusing an interface, we could forget the value function,
and folks would have to use the (key, value) interface if they need to
transform the value.

What makes no sense to me is having the identify function as the default
key (and yes, it is the identity function, it would return the actual
object, or not be there at all) -- the grouping would be done by the hash
of key after passing through the key function).

That's because having a default that is (almost) completely useless  makes
no sense -- it might as well be a required parameter.

(unless there was a value function as well, in which case, it's not a
completely useless default).

- CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180703/66681e57/attachment.html>

From chris.barker at noaa.gov  Tue Jul  3 13:36:12 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Tue, 3 Jul 2018 10:36:12 -0700
Subject: [Python-ideas] Where should grouping() live (was: grouping /
 dict of lists)
In-Reply-To: <CALGmxE+Wk7WCqMEWb-1EHOjZHzdeykKRhyag3Fw0X+tDHr40Ew@mail.gmail.com>
References: <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
 <20180703152413.GF14437@ando.pearwood.info>
 <CALGmxE+Wk7WCqMEWb-1EHOjZHzdeykKRhyag3Fw0X+tDHr40Ew@mail.gmail.com>
Message-ID: <CALGmxEJchSnE44ez4-Dw2vUU8es08pSSXT=fCJVNkEdB6L9nCg@mail.gmail.com>

It seems a really stupid reason to make this choice, but:

If we make a Grouping class, it has an obvious home in the collections
module

If we make a grouping (or grouped) function, we don't know where to put it

But since I like the Grouping class idea anyway, it's one more reason...

-CHB


On Tue, Jul 3, 2018 at 9:15 AM, Chris Barker <chris.barker at noaa.gov> wrote:

> On Tue, Jul 3, 2018 at 8:24 AM, Steven D'Aprano <steve at pearwood.info>
> wrote:
>
>> On Tue, Jul 03, 2018 at 09:23:07AM -0400, David Mertz wrote:
>>
>
>
>> > My problem with the second idea is that *I* find it very wrong to have
>> > something in itertools that does not return an iterator.  It wrecks the
>> > combinatorial algebra of the module.
>>
>
> hmm -- that seems to be a pretty pedantic approach -- practicality beats
> purity, after all :-)
>
> I think we should first decide if a grouping() function is a useful
> addition to the standard library (after all:  "not every two line function
> needs to in the stdlib"), and f so, then we can find a home for it.
>
> personally, I'm wondering if a "dicttools" or something module would make
> sense -- I imagine there are all sorts of other handy utilities for working
> with dicts that could go there. (though, yeah, we'd want to actually have a
> handful of these before creating a new module :-) )
>
> > That said, it's easy to fix... and I believe independently useful.  Just
>> > make grouping() a generator function rather than a plain function.  This
>> > lets us get an incremental grouping of an iterable.
>>
>> We already have something which lazily groups an iterable, returning
>> groups as they are seen: groupby.
>>
>> What makes grouping() different from groupby() is that it accumulates
>> ALL of the subgroups rather than just consecutive subgroupings.
>
>
> well, yeah, but it wont actually get you those until you exhaust the
> iterator -- so while it's different than itertools.groupby, it is different
> than itertools.groupby(sorted(iterable))?
>
> In short, this wouldn't really solve the problems that itertools.groupby
> has for this sort of task -- so what's the point?
>
>  > As for where it belongs, perhaps the collections module is the least
> worst fit.
>
> That depends some on whether we go with a simple function, in which case
> collections is a pretty bad fit (but maybe still the least worse).
>
> Personally I still like the idea of having this be special type of dict,
> rather than "just a function" -- and then it's really obvious where to put
> it :-)
>
> -CHB
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180703/d5769a0d/attachment-0001.html>

From mertz at gnosis.cx  Tue Jul  3 15:01:10 2018
From: mertz at gnosis.cx (David Mertz)
Date: Tue, 3 Jul 2018 15:01:10 -0400
Subject: [Python-ideas] Where should grouping() live (was: grouping /
 dict of lists)
In-Reply-To: <CALGmxE+Wk7WCqMEWb-1EHOjZHzdeykKRhyag3Fw0X+tDHr40Ew@mail.gmail.com>
References: <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
 <20180703152413.GF14437@ando.pearwood.info>
 <CALGmxE+Wk7WCqMEWb-1EHOjZHzdeykKRhyag3Fw0X+tDHr40Ew@mail.gmail.com>
Message-ID: <CAEbHw4bU3hFq1H5wTAhRE4dYizcfVwj2bXb3XuiRi=hpmPbYsA@mail.gmail.com>

I admit a hypothetical itertools.grouping that returned incrementally built
dictionaries doesn't fill any simple need I have often encountered.  I can
be hand-wavy about "stateful bucketing of streams" and looking at
windowing/tails, but I don't have a clean and simple example where I need
this.  The "run to exhaustion" interface has more obvious uses (albeit,
they *must* be technically a subset of the incremental ones).

I think I will also concede that in incrementally built and yielded
dictionary isn't *really* in the spirit of itertools either.  I suppose
tee() can grow unboundedly if only one tine is utilized... but in general,
itertools is meant to provide iterators that keep memory usage limited to a
few elements in memory at a time (yes, groupby, takewhile, or dropwhile
have pathological cases that could be unbounded... but usually they're not).

So maybe we really do need a dicttools or mappingtools module, with this as
the first function to put inside it.

... but I STILL like a new collections.Grouping (or collections.Grouper)
the best.  It might overcome Guido's reluctance... and what goes there is
really delegated by him, not his own baby.

On Tue, Jul 3, 2018 at 12:19 PM Chris Barker via Python-ideas <
python-ideas at python.org> wrote:

> On Tue, Jul 3, 2018 at 8:24 AM, Steven D'Aprano <steve at pearwood.info>
> wrote:
>
>> On Tue, Jul 03, 2018 at 09:23:07AM -0400, David Mertz wrote:
>>
>
>
>> > My problem with the second idea is that *I* find it very wrong to have
>> > something in itertools that does not return an iterator.  It wrecks the
>> > combinatorial algebra of the module.
>>
>
> hmm -- that seems to be a pretty pedantic approach -- practicality beats
> purity, after all :-)
>
> I think we should first decide if a grouping() function is a useful
> addition to the standard library (after all:  "not every two line function
> needs to in the stdlib"), and f so, then we can find a home for it.
>
> personally, I'm wondering if a "dicttools" or something module would make
> sense -- I imagine there are all sorts of other handy utilities for working
> with dicts that could go there. (though, yeah, we'd want to actually have a
> handful of these before creating a new module :-) )
>
> > That said, it's easy to fix... and I believe independently useful.  Just
>> > make grouping() a generator function rather than a plain function.  This
>> > lets us get an incremental grouping of an iterable.
>>
>> We already have something which lazily groups an iterable, returning
>> groups as they are seen: groupby.
>>
>> What makes grouping() different from groupby() is that it accumulates
>> ALL of the subgroups rather than just consecutive subgroupings.
>
>
> well, yeah, but it wont actually get you those until you exhaust the
> iterator -- so while it's different than itertools.groupby, it is different
> than itertools.groupby(sorted(iterable))?
>
> In short, this wouldn't really solve the problems that itertools.groupby
> has for this sort of task -- so what's the point?
>
>  > As for where it belongs, perhaps the collections module is the least
> worst fit.
>
> That depends some on whether we go with a simple function, in which case
> collections is a pretty bad fit (but maybe still the least worse).
>
> Personally I still like the idea of having this be special type of dict,
> rather than "just a function" -- and then it's really obvious where to put
> it :-)
>
> -CHB
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180703/737b8c49/attachment.html>

From chris.barker at noaa.gov  Tue Jul  3 16:12:28 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Tue, 3 Jul 2018 13:12:28 -0700
Subject: [Python-ideas] Where should grouping() live (was: grouping /
 dict of lists)
In-Reply-To: <CAEbHw4bU3hFq1H5wTAhRE4dYizcfVwj2bXb3XuiRi=hpmPbYsA@mail.gmail.com>
References: <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
 <20180703152413.GF14437@ando.pearwood.info>
 <CALGmxE+Wk7WCqMEWb-1EHOjZHzdeykKRhyag3Fw0X+tDHr40Ew@mail.gmail.com>
 <CAEbHw4bU3hFq1H5wTAhRE4dYizcfVwj2bXb3XuiRi=hpmPbYsA@mail.gmail.com>
Message-ID: <CALGmxEJFuXhCXwvunJsTYAWw5fMsj1zaHn2LvA9PF=dkAR4uuw@mail.gmail.com>

On Tue, Jul 3, 2018 at 12:01 PM, David Mertz <mertz at gnosis.cx> wrote:

> ... but I STILL like a new collections.Grouping (or collections.Grouper)
> the best.
>

me too.


> It might overcome Guido's reluctance... and what goes there is really
> delegated by him, not his own baby.
>

Is collections anyone in particular's baby? like itertools "belongs" to
Raymond?

-CHB




> On Tue, Jul 3, 2018 at 12:19 PM Chris Barker via Python-ideas <
> python-ideas at python.org> wrote:
>
>> On Tue, Jul 3, 2018 at 8:24 AM, Steven D'Aprano <steve at pearwood.info>
>> wrote:
>>
>>> On Tue, Jul 03, 2018 at 09:23:07AM -0400, David Mertz wrote:
>>>
>>
>>
>>> > My problem with the second idea is that *I* find it very wrong to have
>>> > something in itertools that does not return an iterator.  It wrecks the
>>> > combinatorial algebra of the module.
>>>
>>
>> hmm -- that seems to be a pretty pedantic approach -- practicality beats
>> purity, after all :-)
>>
>> I think we should first decide if a grouping() function is a useful
>> addition to the standard library (after all:  "not every two line function
>> needs to in the stdlib"), and f so, then we can find a home for it.
>>
>> personally, I'm wondering if a "dicttools" or something module would make
>> sense -- I imagine there are all sorts of other handy utilities for working
>> with dicts that could go there. (though, yeah, we'd want to actually have a
>> handful of these before creating a new module :-) )
>>
>> > That said, it's easy to fix... and I believe independently useful.  Just
>>> > make grouping() a generator function rather than a plain function.
>>> This
>>> > lets us get an incremental grouping of an iterable.
>>>
>>> We already have something which lazily groups an iterable, returning
>>> groups as they are seen: groupby.
>>>
>>> What makes grouping() different from groupby() is that it accumulates
>>> ALL of the subgroups rather than just consecutive subgroupings.
>>
>>
>> well, yeah, but it wont actually get you those until you exhaust the
>> iterator -- so while it's different than itertools.groupby, it is different
>> than itertools.groupby(sorted(iterable))?
>>
>> In short, this wouldn't really solve the problems that itertools.groupby
>> has for this sort of task -- so what's the point?
>>
>>  > As for where it belongs, perhaps the collections module is the least
>> worst fit.
>>
>> That depends some on whether we go with a simple function, in which case
>> collections is a pretty bad fit (but maybe still the least worse).
>>
>> Personally I still like the idea of having this be special type of dict,
>> rather than "just a function" -- and then it's really obvious where to put
>> it :-)
>>
>> -CHB
>>
>>
>> --
>>
>> Christopher Barker, Ph.D.
>> Oceanographer
>>
>> Emergency Response Division
>> NOAA/NOS/OR&R            (206) 526-6959   voice
>> 7600 Sand Point Way NE
>> <https://maps.google.com/?q=7600+Sand+Point+Way+NE&entry=gmail&source=g>
>>   (206) 526-6329   fax
>> Seattle, WA  98115       (206) 526-6317   main reception
>>
>> Chris.Barker at noaa.gov
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> --
> Keeping medicines from the bloodstreams of the sick; food
> from the bellies of the hungry; books from the hands of the
> uneducated; technology from the underdeveloped; and putting
> advocates of freedom in prisons.  Intellectual property is
> to the 21st century what the slave trade was to the 16th.
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180703/9f92ed55/attachment-0001.html>

From greg.ewing at canterbury.ac.nz  Tue Jul  3 18:20:49 2018
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 04 Jul 2018 10:20:49 +1200
Subject: [Python-ideas] grouping / dict of lists
In-Reply-To: <CAOmnVUBp9vJjTp6v17LHy2C=7GzR63fb4ARZW2Dco022uTDpfA@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CAP7+vJJ6vN9H8esj8tS+bSKsFELh8q8ywYHAW8Ti_upEJHf1vg@mail.gmail.com>
 <CADqi797HXzHcUnrNLp7r+pFEM5fcApiKHV4s=P5-D6mXHYYv-g@mail.gmail.com>
 <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAOmnVUBp9vJjTp6v17LHy2C=7GzR63fb4ARZW2Dco022uTDpfA@mail.gmail.com>
Message-ID: <5B3BF6C1.7020005@canterbury.ac.nz>

Nicolas Rolin wrote:

> grouping(((len(word), word) for word in words))

That actually has one more level of parens than are needed,
you can just write

    grouping((len(word), word) for word in words)

-- 
Greg

From greg.ewing at canterbury.ac.nz  Tue Jul  3 18:35:04 2018
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 04 Jul 2018 10:35:04 +1200
Subject: [Python-ideas] Where should grouping() live
In-Reply-To: <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CAP7+vJJ6vN9H8esj8tS+bSKsFELh8q8ywYHAW8Ti_upEJHf1vg@mail.gmail.com>
 <CADqi797HXzHcUnrNLp7r+pFEM5fcApiKHV4s=P5-D6mXHYYv-g@mail.gmail.com>
 <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
Message-ID: <5B3BFA18.7020400@canterbury.ac.nz>

David Mertz wrote:
> Just 
> make grouping() a generator function rather than a plain function.  This 
> lets us get an incremental grouping of an iterable.  This can be useful 
> if the iterable is slow or infinite, but the partial groupings are 
> useful in themselves.

Do you have any real-world examples? I'm having trouble
thinking of any.

Even if there are a few, it seems like the vast majority of
the time you *won't* want the intermediate groupings, just
the final one, and then what do you do? It would be annoying
to have to write code to exhaust the iterator just to get the
result you're after.

Also, were you intending it to return a series of independent
objects, or does it just return the same object every time,
adding stuff to it? The former would be very inefficient for
the majority of uses, whereas the latter doesn't seem to
be in keeping with the spirit of itertools.

This idea seems like purity beating practicality to me.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Tue Jul  3 18:44:17 2018
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 04 Jul 2018 10:44:17 +1200
Subject: [Python-ideas] Where should grouping() live
In-Reply-To: <20180703152413.GF14437@ando.pearwood.info>
References: <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
 <20180703152413.GF14437@ando.pearwood.info>
Message-ID: <5B3BFC41.8040207@canterbury.ac.nz>

Steven D'Aprano wrote:
> I propose that a better name which indicates the non-lazy nature of this 
> function is *grouped* rather than grouping, like sorted().

+1

> As for where it belongs, perhaps the collections module is the least 
> worst fit.

But then there's the equally strong purist argument that it's
not a data type, just a function.

Unless we *make* it a data type. Then not only would it fit
well in collections, it would also make it fairly easy to do
incremental grouping if you really wanted that.

Usual case:

    g = groupdict((key(val), val) for val in things)

Incremental case:

    g = groupdict()
    for key(val), val in things:
       g.add(key, val)
       process_partial_grouping(g)

-- 
Greg

From python at mrabarnett.plus.com  Tue Jul  3 18:53:01 2018
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 3 Jul 2018 23:53:01 +0100
Subject: [Python-ideas] grouping / dict of lists
In-Reply-To: <5B3BF6C1.7020005@canterbury.ac.nz>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CAP7+vJJ6vN9H8esj8tS+bSKsFELh8q8ywYHAW8Ti_upEJHf1vg@mail.gmail.com>
 <CADqi797HXzHcUnrNLp7r+pFEM5fcApiKHV4s=P5-D6mXHYYv-g@mail.gmail.com>
 <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAOmnVUBp9vJjTp6v17LHy2C=7GzR63fb4ARZW2Dco022uTDpfA@mail.gmail.com>
 <5B3BF6C1.7020005@canterbury.ac.nz>
Message-ID: <4fcf192f-9625-566a-ea37-8c970f796e06@mrabarnett.plus.com>

On 2018-07-03 23:20, Greg Ewing wrote:
> Nicolas Rolin wrote:
> 
>> grouping(((len(word), word) for word in words))
> 
> That actually has one more level of parens than are needed,
> you can just write
> 
>      grouping((len(word), word) for word in words)
> 
FWIW, here's my opinion.

I much prefer something like:

     grouped(words, key=len)

I think that building an iterable of 2-tuples to pass to 'grouped' is 
much like following a decorate-sort-undecorate pattern when sorting, or 
something similar when using 'min' or 'max'. Passing an iterable of 
items and optionally a key function is simpler, IMHO.

Why would you pass 2-tuples, anyway?

Maybe it's because 'grouped' returns a dict and a dict can be built from 
an iterable of 2-tuples, but that's OK because a dict needs key/value pairs.

When 'Counter' was being proposed, it was suggested that one could be 
created from an iterable of 2-tuples, which sort of made sense because a 
Counter is like a dict, but, then, how would you count 2-tuples?

Fortunately, Counter counts items, so you can do things like:

     counts = Counter(list_of_words)

I think it's the same thing here.

'grouped' returns a dict, so passing 2-tuples initially seems 
reasonable, but, as in the case with Counter, I think it would be a mistake.

It would be nice to be able to say:

     grouped(words, key=str.casefold)

rather than:

     grouped((word.casefold(), word) for word in words)

It would match the pattern of sorted, min and max.

From greg.ewing at canterbury.ac.nz  Tue Jul  3 19:22:40 2018
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 04 Jul 2018 11:22:40 +1200
Subject: [Python-ideas] grouping / dict of lists
In-Reply-To: <4fcf192f-9625-566a-ea37-8c970f796e06@mrabarnett.plus.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi797HXzHcUnrNLp7r+pFEM5fcApiKHV4s=P5-D6mXHYYv-g@mail.gmail.com>
 <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAOmnVUBp9vJjTp6v17LHy2C=7GzR63fb4ARZW2Dco022uTDpfA@mail.gmail.com>
 <5B3BF6C1.7020005@canterbury.ac.nz>
 <4fcf192f-9625-566a-ea37-8c970f796e06@mrabarnett.plus.com>
Message-ID: <5B3C0540.7030600@canterbury.ac.nz>

MRAB wrote:
> I think that building an iterable of 2-tuples to pass to 'grouped' is 
> much like following a decorate-sort-undecorate pattern when sorting, or 
> something similar when using 'min' or 'max'. Passing an iterable of 
> items and optionally a key function is simpler, IMHO.

It should certainly be an option, but I don't think it
should be the only one. Like with map() vs. comprehensions,
sometimes one way is more convenient, sometimes the other.

-- 
Greg

From steve at pearwood.info  Tue Jul  3 20:21:03 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 4 Jul 2018 10:21:03 +1000
Subject: [Python-ideas] Where should grouping() live
In-Reply-To: <5B3BFC41.8040207@canterbury.ac.nz>
References: <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
 <20180703152413.GF14437@ando.pearwood.info>
 <5B3BFC41.8040207@canterbury.ac.nz>
Message-ID: <20180704002102.GJ14437@ando.pearwood.info>

On Wed, Jul 04, 2018 at 10:44:17AM +1200, Greg Ewing wrote:
> Steven D'Aprano wrote:
> >I propose that a better name which indicates the non-lazy nature of this 
> >function is *grouped* rather than grouping, like sorted().
> 
> +1
> 
> >As for where it belongs, perhaps the collections module is the least 
> >worst fit.
> 
> But then there's the equally strong purist argument that it's
> not a data type, just a function.

Yes, I realised that after I posted my earlier comment.


> Unless we *make* it a data type. Then not only would it fit
> well in collections, it would also make it fairly easy to do
> incremental grouping if you really wanted that.
> 
> Usual case:
> 
>    g = groupdict((key(val), val) for val in things)


How does groupdict differ from regular defaultdicts, aside from the 
slightly different constructor?


> Incremental case:
> 
>    g = groupdict()
>    for key(val), val in things:
>       g.add(key, val)
>       process_partial_grouping(g)

I don't think that syntax works. I get:

SyntaxError: can't assign to function call


Even if it did work, it's hardly any simpler than

    d = defaultdict(list)
    for val in things:
        d[key(val)].append(val)

But then Counter is hardly any simpler than a regular dict too.




-- 
Steve

From steve at pearwood.info  Tue Jul  3 21:32:02 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 4 Jul 2018 11:32:02 +1000
Subject: [Python-ideas] grouping / dict of lists
In-Reply-To: <CALGmxEJBYFEoPOFo7aVSN=Xy=sgSMnrF0k8pnt=VhJ6cvhS_4Q@mail.gmail.com>
References: <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAOmnVUBp9vJjTp6v17LHy2C=7GzR63fb4ARZW2Dco022uTDpfA@mail.gmail.com>
 <20180703153338.GG14437@ando.pearwood.info>
 <CALGmxEJBYFEoPOFo7aVSN=Xy=sgSMnrF0k8pnt=VhJ6cvhS_4Q@mail.gmail.com>
Message-ID: <20180704013202.GK14437@ando.pearwood.info>

On Tue, Jul 03, 2018 at 10:33:55AM -0700, Chris Barker wrote:
> On Tue, Jul 3, 2018 at 8:33 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> 
> > but why are we using key values by hand when grouping ought to do it for
> >> us, as Michael Selik's version does?
> >
> >     grouping(words, key=len)
> 
> 
> because supplying a key function is sometimes cleaner, and sometimes uglier
> than building up a comprehension -- which I think comes down to:
> 
> 1) taste (style?)
> 
> 2) whether the key function is as simple as the expression
> 
> 3) whether you ned to transform the value in any way.


Of course you can prepare the sequence any way you like, but these are 
not equivalent:

    grouping(words, keyfunc=len)

    grouping((len(word), word) for word in words)

The first groups words by their length; the second groups pairs of 
(length, word) tuples by equality.


py> grouping("a bb ccc d ee fff".split(), keyfunc=len)
{1: ['a', 'd'], 2: ['bb', 'ee'], 3: ['ccc', 'fff']}

py> grouping((len(w), w) for w in "a bb ccc d ee fff".split())
{(3, 'ccc'): [(3, 'ccc')], (1, 'd'): [(1, 'd')], (2, 'ee'): [(2, 'ee')], 
(3, 'fff'): [(3, 'fff')], (1, 'a'): [(1, 'a')], (2, 'bb'): [(2, 'bb')]}


Don't worry, it wasn't obvious to me at 1am (my local time) either :-)



-- 
Steve

From chris.barker at noaa.gov  Wed Jul  4 01:09:51 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Tue, 3 Jul 2018 22:09:51 -0700
Subject: [Python-ideas] Where should grouping() live
In-Reply-To: <20180704002102.GJ14437@ando.pearwood.info>
References: <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
 <20180703152413.GF14437@ando.pearwood.info>
 <5B3BFC41.8040207@canterbury.ac.nz>
 <20180704002102.GJ14437@ando.pearwood.info>
Message-ID: <CALGmxEKHPcqY2rTJvz0T2nXv2PeL7Vu7URVkvz_QemyVDRDJDw@mail.gmail.com>

So this ended up a long post, so the TL;DR

* There are types of data well suited to the key function approach, and
other data not so well suited to it. If you want to support the not as well
suited use cases, you should have a value function as well and/or take a
(key, value) pair.

* There are some nice advantages in flexibility to having a Grouping class,
rather than simply a function.

So: I propose a best of all worlds version: a Grouping class (subclass of
dict):

* The constructor takes an iterable of (key, value) pairs by default.

* The constructor takes an optional key_func -- when not None, it is used
to determine the keys in the iterable instead.

* The constructor also takes a value_func -- when specified, it processes
the items to determine the values.

* a_grouping[key] = value

  adds the value to the list corresponding to the key.

* a_grouping.add(item) -- applies the key_func and value_func to add a new
value to the appropriate group.

Prototype code here:

https://github.com/PythonCHB/grouper

Now the lengthy commentary and examples:

On Tue, Jul 3, 2018 at 5:21 PM, Steven D'Aprano <steve at pearwood.info> wrote:

> On Wed, Jul 04, 2018 at 10:44:17AM +1200, Greg Ewing wrote:
> > Steven D'Aprano wrote:
>
> > Unless we *make* it a data type. Then not only would it fit
> > well in collections, it would also make it fairly easy to do
> > incremental grouping if you really wanted that.


indeed -- one of motivations for my prototype:

https://github.com/PythonCHB/grouper

(Did none of my messages get to this list??)


> > Usual case:
> >
> >    g = groupdict((key(val), val) for val in things)
>
>
> How does groupdict differ from regular defaultdicts, aside from the
> slightly different constructor?
>

* You don't need to declare the defaultdict (and what the default is) first

* You don't need to call .append() yourself

* It can have a custom .init() and .update()

* It can have a .add() method

* It can (optionally) use a key function.

* And you can have other methods that do useful things with the groupings.

   >    g = groupdict()

> >    for key(val), val in things:
> >       g.add(key, val)
> >       process_partial_grouping(g)
>
> I don't think that syntax works. I get:
>
> SyntaxError: can't assign to function call
>

looks like untested code :-)

with my prototype it would be:

g = groupdict()
for key, val in things:
    g[key] = val
process_partial_grouping(g)

(this assumes your things are (key, value) pairs)

Again, IF you data are a sequence of items, and the value is the item
itself, and the key is a simple function of the item, THEN the key function
method makes more sense, which for the incremental adding of data would be:

g = groupdict(key_fun=a_fun)
for thing in things:
    g.add(thing)
process_partial_grouping(g)

Even if it did work, it's hardly any simpler than
>
>     d = defaultdict(list)
>     for val in things:
>         d[key(val)].append(val)
>
> But then Counter is hardly any simpler than a regular dict too.
>

exactly -- and counter is actually a little annoyingly too much like a
regular dict, in my mind :-)

In the latest version of my prototype, the __init__  expects a (key, value)
pair by default, but you can also pass in a key_func, and then it will
process the iterable passes in as (key_func(item), item) pairs.

And the update() method will also use the key_func if one was provided.

So a best of both worlds -- pick your API.

In this thread, and in the PEP, there various ways of accomplishing this
task presented -- none of them (except using a raw itertools.groupby in
some cases) is all that onerous.

But I do think a custom function or even better, custom class, would create
a "one obvious" way to do a common manipulation.

A final (repeated) point:

Some data are better suited to a (key, value) pair style, and some to a key
function style. All of the examples in the PEP are well suited to the key
function style. But the example that kicked off this discussion was about
data already in (key, value) pairs (actual in that case, (value, key) pairs.

And there are other examples. Here's a good one for how one might want to
use a Grouping dict more like a regular dict -- of maybe like a simple
function constructor:

(code in: https://github.com/PythonCHB/grouper/blob/master/examples/
trigrams.py)

#!/usr/bin/env python3

"""
Demo of processing "trigrams" from Dave Thomas' Coding Kata site:

http://codekata.com/kata/kata14-tom-swift-under-the-milkwood/

This is only addressing the part of the problem of building up the trigrams.

This is showing various ways of doing it with the Grouping object.
"""

from grouper import Grouping
from operator import itemgetter

words = "I wish I may I wish I might".split()

# using setdefault with a regular dict:
# how I might do it without a Grouping class
trigrams = {}
for i in range(len(words) - 2):
    pair = tuple(words[i:i + 2])
    follower = words[i + 2]
    trigrams.setdefault(pair, []).append(follower)

print(trigrams)

# using a Grouping with a regular loop:

trigrams = Grouping()
for i in range(len(words) - 2):
    pair = tuple(words[i:i + 2])
    follower = words[i + 2]
    trigrams[pair] = follower

print(trigrams)

# using a Grouping with zip

trigrams = Grouping()
for w1, w2, w3 in zip(words[:], words[1:], words[2:]):
    trigrams[(w1, w2)] = w3

print(trigrams)

# Now we can do it one expression:

trigrams = Grouping(((w1, w2), w3)
                    for w1, w2, w3 in zip(words[:], words[1:], words[2:]))
print(trigrams)

# Now with the key function:
# in this case it needs to be in a sequence, so we can't use a simple loop

trigrams = Grouping(zip(words[:], words[1:], words[2:]),
                    key_fun=itemgetter(0, 1))

print(trigrams)

# Darn! that got the key right, but the value is not right.
# we can post process:
trigrams = {key: [t[2] for t in value] for key, value in trigrams.items()}

print(trigrams)

# But THAT is a lot harder to wrap your head around than the original
setdefault() loop!
# And it mixes key function style and comprehension style -- so no good.

# Adding a value_func helps a lot:
trigrams = Grouping(zip(words[:], words[1:], words[2:]),
                    key_fun=itemgetter(0, 1),
                    value_fun=itemgetter(2))

print(trigrams)

#that works fine, but I, at least, find it klunkier than the comprehensions
style

# Finally, we can use a regular loop with the functions

trigrams = Grouping(key_fun=itemgetter(0, 1),
                    value_fun=itemgetter(2))
for triple in zip(words[:], words[1:], words[2:]):
    trigrams.add(triple)

print(trigrams)

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180703/10166120/attachment.html>

From mike at selik.org  Wed Jul  4 01:46:19 2018
From: mike at selik.org (Michael Selik)
Date: Tue, 3 Jul 2018 22:46:19 -0700
Subject: [Python-ideas] grouping / dict of lists
In-Reply-To: <20180704013202.GK14437@ando.pearwood.info>
References: <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAOmnVUBp9vJjTp6v17LHy2C=7GzR63fb4ARZW2Dco022uTDpfA@mail.gmail.com>
 <20180703153338.GG14437@ando.pearwood.info>
 <CALGmxEJBYFEoPOFo7aVSN=Xy=sgSMnrF0k8pnt=VhJ6cvhS_4Q@mail.gmail.com>
 <20180704013202.GK14437@ando.pearwood.info>
Message-ID: <CADqi797fK2218821K73fRTBKmHe7CCEObWV-Eb-PXXrDc4-y8A@mail.gmail.com>

On Tue, Jul 3, 2018, 6:32 PM Steven D'Aprano <steve at pearwood.info> wrote:

> On Tue, Jul 03, 2018 at 10:33:55AM -0700, Chris Barker wrote:
> > On Tue, Jul 3, 2018 at 8:33 AM, Steven D'Aprano <steve at pearwood.info>
> wrote:
> >
> > > but why are we using key values by hand when grouping ought to do it
> for
> > >> us, as Michael Selik's version does?
> > >
> > >     grouping(words, key=len)
> >
> >
> > because supplying a key function is sometimes cleaner, and sometimes
> uglier
> > than building up a comprehension -- which I think comes down to:
> >
> > 1) taste (style?)
> >
> > 2) whether the key function is as simple as the expression
> >
> > 3) whether you ned to transform the value in any way.
>
>
> Of course you can prepare the sequence any way you like, but these are
> not equivalent:
>
>     grouping(words, keyfunc=len)
>
>     grouping((len(word), word) for word in words)
>
> The first groups words by their length; the second groups pairs of
> (length, word) tuples by equality.
>
>
> py> grouping("a bb ccc d ee fff".split(), keyfunc=len)
> {1: ['a', 'd'], 2: ['bb', 'ee'], 3: ['ccc', 'fff']}
>
> py> grouping((len(w), w) for w in "a bb ccc d ee fff".split())
> {(3, 'ccc'): [(3, 'ccc')], (1, 'd'): [(1, 'd')], (2, 'ee'): [(2, 'ee')],
> (3, 'fff'): [(3, 'fff')], (1, 'a'): [(1, 'a')], (2, 'bb'): [(2, 'bb')]}
>

This handles the case that someone is passing in n-tuple rows and wants to
keep the rows unchanged.

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180703/5500fb93/attachment-0001.html>

From levkivskyi at gmail.com  Wed Jul  4 06:08:05 2018
From: levkivskyi at gmail.com (Ivan Levkivskyi)
Date: Wed, 4 Jul 2018 11:08:05 +0100
Subject: [Python-ideas] Where should grouping() live (was: grouping /
 dict of lists)
In-Reply-To: <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CAP7+vJJ6vN9H8esj8tS+bSKsFELh8q8ywYHAW8Ti_upEJHf1vg@mail.gmail.com>
 <CADqi797HXzHcUnrNLp7r+pFEM5fcApiKHV4s=P5-D6mXHYYv-g@mail.gmail.com>
 <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
Message-ID: <CAOMjWkmNCM-s-BQSb1Dp+KbufnTSBg=dB+6awmOAKj-roaijow@mail.gmail.com>

Replying to the question in subject, I think it would be better in
collections as a class.
Having it just as a function doesn't  buy much, because one can do the same
with three lines and a defaultdict.
However, if this is a class it can support adding new elements, merge the
groupeddicts, etc.

--
Ivan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180704/ef397746/attachment.html>

From steve at pearwood.info  Wed Jul  4 06:25:02 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 4 Jul 2018 20:25:02 +1000
Subject: [Python-ideas] Where should grouping() live (was: grouping /
 dict of lists)
In-Reply-To: <CAOMjWkmNCM-s-BQSb1Dp+KbufnTSBg=dB+6awmOAKj-roaijow@mail.gmail.com>
References: <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
 <CAOMjWkmNCM-s-BQSb1Dp+KbufnTSBg=dB+6awmOAKj-roaijow@mail.gmail.com>
Message-ID: <20180704102502.GO14437@ando.pearwood.info>

On Wed, Jul 04, 2018 at 11:08:05AM +0100, Ivan Levkivskyi wrote:
> Replying to the question in subject, I think it would be better in
> collections as a class.
> Having it just as a function doesn't  buy much, because one can do the same
> with three lines and a defaultdict.
> However, if this is a class it can support adding new elements, merge the
> groupeddicts, etc.

defaultdicts support adding new elements, and they have an update method 
same as regular dicts :-)



-- 
Steve

From levkivskyi at gmail.com  Wed Jul  4 06:36:07 2018
From: levkivskyi at gmail.com (Ivan Levkivskyi)
Date: Wed, 4 Jul 2018 11:36:07 +0100
Subject: [Python-ideas] Where should grouping() live (was: grouping /
 dict of lists)
In-Reply-To: <20180704102502.GO14437@ando.pearwood.info>
References: <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
 <CAOMjWkmNCM-s-BQSb1Dp+KbufnTSBg=dB+6awmOAKj-roaijow@mail.gmail.com>
 <20180704102502.GO14437@ando.pearwood.info>
Message-ID: <CAOMjWk=Qe39EcK-PirbU-54LM0_Qquic2dbkmmJApz4dcuA=+A@mail.gmail.com>

On 4 July 2018 at 11:25, Steven D'Aprano <steve at pearwood.info> wrote:

> On Wed, Jul 04, 2018 at 11:08:05AM +0100, Ivan Levkivskyi wrote:
> > Replying to the question in subject, I think it would be better in
> > collections as a class.
> > Having it just as a function doesn't  buy much, because one can do the
> same
> > with three lines and a defaultdict.
> > However, if this is a class it can support adding new elements, merge the
> > groupeddicts, etc.
>
> defaultdicts support adding new elements, and they have an update method
> same as regular dicts :-)
>

Except that updating will not do what I want. Merging two groupeddicts is
not just `one.update(other)`
Moreover, using just an update with regular dicts will do something
bug-prone, it will add every group
from `other` as an element to the corresponding group in `one`.

--
Ivan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180704/7feca1fd/attachment.html>

From songofacandy at gmail.com  Wed Jul  4 06:53:22 2018
From: songofacandy at gmail.com (INADA Naoki)
Date: Wed, 4 Jul 2018 19:53:22 +0900
Subject: [Python-ideas] Where should grouping() live (was: grouping /
 dict of lists)
In-Reply-To: <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CAP7+vJJ6vN9H8esj8tS+bSKsFELh8q8ywYHAW8Ti_upEJHf1vg@mail.gmail.com>
 <CADqi797HXzHcUnrNLp7r+pFEM5fcApiKHV4s=P5-D6mXHYYv-g@mail.gmail.com>
 <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
Message-ID: <CAEfz+TxsBJKQ6=JbiBnTbCgz=xpeOmLpLn-kFsFUdXvmvrsazw@mail.gmail.com>

I'm -1 on adding it in stdlib.

But if it happens, I'm -1 on functools and collections.
They are used very much.  Every Python tool import them regardless how much
of their contents are used.

On the other hand, itertools contains random stuff very rarely used.

If you really want to add it in collections, I suggests
from collections.groupdict import GroupDict.

Regards,

On Tue, Jul 3, 2018 at 10:23 PM David Mertz <mertz at gnosis.cx> wrote:

> Guido said he has mooted this discussion, so it's probably not reaching
> him.  It took one thousand fewer messages for him to stop following this
> than with PEP 572, for some reason :-).
>
> But before putting it on auto-archive, the BDFL said (1) NO GO on getting
> a new builtin; (2) NO OBJECTION to putting it in itertools.
>
> My problem with the second idea is that *I* find it very wrong to have
> something in itertools that does not return an iterator.  It wrecks the
> combinatorial algebra of the module.
>
> That said, it's easy to fix... and I believe independently useful.  Just
> make grouping() a generator function rather than a plain function.  This
> lets us get an incremental grouping of an iterable.  This can be useful if
> the iterable is slow or infinite, but the partial groupings are useful in
> themselves.
>
> Python 3.7.0 (default, Jun 28 2018, 07:39:16)
> [Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> >>> from grouping import grouping
> >>> grouped = grouping('AbBa', key=str.casefold)
> >>> for dct in grouped: print(dct)
> ...
> {'a': ['A']}
> {'a': ['A'], 'b': ['b']}
> {'a': ['A'], 'b': ['b', 'B']}
> {'a': ['A', 'a'], 'b': ['b', 'B']}
>
>
> This isn't so useful for the concrete sequence, but for this it would be
> great:
>
> for grouped in grouping(data_over_wire()):
>
>     process_partial_groups(grouped)
>
>
> The implementation need not and should not rely on "pre-grouping" with
> itertools.groupby:
>
> def grouping(iterable, key=None):
>     groups = {}
>     key = key or (lambda x: x)
>     for item in iterable:
>         groups.setdefault(key(item), []).append(item)
>         yield groups
>
>
>
> --
> Keeping medicines from the bloodstreams of the sick; food
> from the bellies of the hungry; books from the hands of the
> uneducated; technology from the underdeveloped; and putting
> advocates of freedom in prisons.  Intellectual property is
> to the 21st century what the slave trade was to the 16th.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
INADA Naoki  <songofacandy at gmail.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180704/8f219fef/attachment-0001.html>

From mertz at gnosis.cx  Wed Jul  4 09:34:19 2018
From: mertz at gnosis.cx (David Mertz)
Date: Wed, 4 Jul 2018 09:34:19 -0400
Subject: [Python-ideas] Fwd:  grouping / dict of lists
In-Reply-To: <CAEbHw4YafoZc_1cnB6a-psmWqW-SD2tw3E2Fw5hUHSJxqEsu-Q@mail.gmail.com>
References: <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAOmnVUBp9vJjTp6v17LHy2C=7GzR63fb4ARZW2Dco022uTDpfA@mail.gmail.com>
 <20180703153338.GG14437@ando.pearwood.info>
 <CALGmxEJBYFEoPOFo7aVSN=Xy=sgSMnrF0k8pnt=VhJ6cvhS_4Q@mail.gmail.com>
 <20180704013202.GK14437@ando.pearwood.info>
 <CAEbHw4YafoZc_1cnB6a-psmWqW-SD2tw3E2Fw5hUHSJxqEsu-Q@mail.gmail.com>
Message-ID: <CAEbHw4Yjue-V_CKQJe2k6f4E8ucNuG=ZNFt5O_njfrQoNKHeUw@mail.gmail.com>

Steven:

You've misunderstood part of the discussion. There are two different
signatures being discussed/proposed for a grouping() function.

The one you show we might call grouping_michael(). The alternate API we
might call grouping_chris(). These two calls will produce the same result
(the first output you show)

  grouping_michael(words, keyfunc=len)
  grouping_chris((len(word), word) for word in words)

I happen to prefer grouping_michael(), but recognize they each make
slightly different things obvious. Absolutely no one wants the behavior in
your second output.

On Tue, Jul 3, 2018, 9:32 PM Steven D'Aprano <steve at pearwood.info> wrote:

> Of course you can prepare the sequence any way you like, but these are
> not equivalent:
>
>     grouping(words, keyfunc=len)
>     grouping((len(word), word) for word in words)
>
> The first groups words by their length; the second groups pairs of
> (length, word) tuples by equality.
>
> py> grouping("a bb ccc d ee fff".split(), keyfunc=len)
> {1: ['a', 'd'], 2: ['bb', 'ee'], 3: ['ccc', 'fff']}
>
> py> grouping((len(w), w) for w in "a bb ccc d ee fff".split())
> {(3, 'ccc'): [(3, 'ccc')], (1, 'd'): [(1, 'd')], (2, 'ee'): [(2, 'ee')],
> (3, 'fff'): [(3, 'fff')], (1, 'a'): [(1, 'a')], (2, 'bb'): [(2, '
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180704/1956fa24/attachment.html>

From mike at selik.org  Wed Jul  4 14:00:18 2018
From: mike at selik.org (Michael Selik)
Date: Wed, 4 Jul 2018 11:00:18 -0700
Subject: [Python-ideas] Fwd: grouping / dict of lists
In-Reply-To: <CAEbHw4Yjue-V_CKQJe2k6f4E8ucNuG=ZNFt5O_njfrQoNKHeUw@mail.gmail.com>
References: <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAOmnVUBp9vJjTp6v17LHy2C=7GzR63fb4ARZW2Dco022uTDpfA@mail.gmail.com>
 <20180703153338.GG14437@ando.pearwood.info>
 <CALGmxEJBYFEoPOFo7aVSN=Xy=sgSMnrF0k8pnt=VhJ6cvhS_4Q@mail.gmail.com>
 <20180704013202.GK14437@ando.pearwood.info>
 <CAEbHw4YafoZc_1cnB6a-psmWqW-SD2tw3E2Fw5hUHSJxqEsu-Q@mail.gmail.com>
 <CAEbHw4Yjue-V_CKQJe2k6f4E8ucNuG=ZNFt5O_njfrQoNKHeUw@mail.gmail.com>
Message-ID: <CADqi794RqJH_a-YNZC4smFh+JvMaO5DRrdGdYgoVjL6F5kYnpA@mail.gmail.com>

There are some cases when that's the correct behavior. It mimics
pandas.DataFrame.groupby. For example, what if you have a sequence of (key,
v1, v2) triples? Group by key, then keep the triples intact is the right
choice sometimes.


On Wed, Jul 4, 2018, 6:39 AM David Mertz <mertz at gnosis.cx> wrote:

> Steven:
>
> You've misunderstood part of the discussion. There are two different
> signatures being discussed/proposed for a grouping() function.
>
> The one you show we might call grouping_michael(). The alternate API we
> might call grouping_chris(). These two calls will produce the same result
> (the first output you show)
>
>   grouping_michael(words, keyfunc=len)
>   grouping_chris((len(word), word) for word in words)
>
> I happen to prefer grouping_michael(), but recognize they each make
> slightly different things obvious. Absolutely no one wants the behavior in
> your second output.
>
> On Tue, Jul 3, 2018, 9:32 PM Steven D'Aprano <steve at pearwood.info> wrote:
>
>> Of course you can prepare the sequence any way you like, but these are
>> not equivalent:
>>
>>     grouping(words, keyfunc=len)
>>     grouping((len(word), word) for word in words)
>>
>> The first groups words by their length; the second groups pairs of
>> (length, word) tuples by equality.
>>
>> py> grouping("a bb ccc d ee fff".split(), keyfunc=len)
>> {1: ['a', 'd'], 2: ['bb', 'ee'], 3: ['ccc', 'fff']}
>>
>> py> grouping((len(w), w) for w in "a bb ccc d ee fff".split())
>> {(3, 'ccc'): [(3, 'ccc')], (1, 'd'): [(1, 'd')], (2, 'ee'): [(2, 'ee')],
>> (3, 'fff'): [(3, 'fff')], (1, 'a'): [(1, 'a')], (2, 'bb'): [(2, '
>>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180704/8d1ae31d/attachment.html>

From mike at selik.org  Wed Jul  4 14:03:30 2018
From: mike at selik.org (Michael Selik)
Date: Wed, 4 Jul 2018 11:03:30 -0700
Subject: [Python-ideas] Where should grouping() live (was: grouping /
 dict of lists)
In-Reply-To: <CAOMjWkmNCM-s-BQSb1Dp+KbufnTSBg=dB+6awmOAKj-roaijow@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CAP7+vJJ6vN9H8esj8tS+bSKsFELh8q8ywYHAW8Ti_upEJHf1vg@mail.gmail.com>
 <CADqi797HXzHcUnrNLp7r+pFEM5fcApiKHV4s=P5-D6mXHYYv-g@mail.gmail.com>
 <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
 <CAOMjWkmNCM-s-BQSb1Dp+KbufnTSBg=dB+6awmOAKj-roaijow@mail.gmail.com>
Message-ID: <CADqi794ss=kY6JP2pcv3GBsgSgh8M_dXL5FFNDaqyqazhLO2pQ@mail.gmail.com>

On Wed, Jul 4, 2018, 3:11 AM Ivan Levkivskyi <levkivskyi at gmail.com> wrote:

> Replying to the question in subject, I think it would be better in
> collections as a class.
> Having it just as a function doesn't  buy much, because one can do the
> same with three lines and a defaultdict.
>

Four lines. You'll need to convert from defaultdict back to a basic dict to
avoid mistaken inserts. For some use cases.


However, if this is a class it can support adding new elements, merge the
> groupeddicts, etc.
>
> --
> Ivan
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180704/8413abd1/attachment.html>

From alexander.belopolsky at gmail.com  Wed Jul  4 15:50:44 2018
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Wed, 4 Jul 2018 15:50:44 -0400
Subject: [Python-ideas] Add a __cite__ method for scientific packages
In-Reply-To: <CAEbHw4Y61oAT7a7rdRC+ShBzbNNsv0VamO1CGg+mTfaSSx+=mg@mail.gmail.com>
References: <CADit4boE-L+OOxKLpmyOzeS55F=VC+ifLgXxEpQNz1XgcFOu_g@mail.gmail.com>
 <CAPJVwBnUUS3Wz-NSsdUGxxp3Nr0gPhzoRbRaJq+1rxgp3JMwJQ@mail.gmail.com>
 <CAOMjWkm6e31d6c_5==t7rBEYu2jusNB3WAxJhknQRR4Mutkkbw@mail.gmail.com>
 <CAEbHw4Y61oAT7a7rdRC+ShBzbNNsv0VamO1CGg+mTfaSSx+=mg@mail.gmail.com>
Message-ID: <CAP7h-xb1tPatRYaYjpiTuokNBpHSFBBZ=8MGkw+_Ym5=JhD4Lw@mail.gmail.com>

On Sun, Jul 1, 2018 at 9:45 AM David Mertz <mertz at gnosis.cx> wrote:

> ..
> There's absolutely nothing in the idea that requires a change in Python,
> and Python developers or users are not, as such, the relevant experts.
>

This is not entirely true.  If some variant of __citation__ is endorsed by
the community, I would expect that pydoc would extract this information to
fill an appropriate section in the documentation page.  Note that pydoc
already treats a number of dunder variables
specially: '__author__', '__credits__', and '__version__' are a few that
come to mind, so I don't think the threshold for adding one more should be
too high.  On the other hand, maybe   '__author__', '__credits__', and
'__citation__' should be merged in one structured variable (a dict?) with
format designed with some extendability in mind.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180704/f3bef9f4/attachment.html>

From wes.turner at gmail.com  Wed Jul  4 21:05:33 2018
From: wes.turner at gmail.com (Wes Turner)
Date: Wed, 4 Jul 2018 21:05:33 -0400
Subject: [Python-ideas] Add a __cite__ method for scientific packages
In-Reply-To: <CAP7h-xb1tPatRYaYjpiTuokNBpHSFBBZ=8MGkw+_Ym5=JhD4Lw@mail.gmail.com>
References: <CADit4boE-L+OOxKLpmyOzeS55F=VC+ifLgXxEpQNz1XgcFOu_g@mail.gmail.com>
 <CAPJVwBnUUS3Wz-NSsdUGxxp3Nr0gPhzoRbRaJq+1rxgp3JMwJQ@mail.gmail.com>
 <CAOMjWkm6e31d6c_5==t7rBEYu2jusNB3WAxJhknQRR4Mutkkbw@mail.gmail.com>
 <CAEbHw4Y61oAT7a7rdRC+ShBzbNNsv0VamO1CGg+mTfaSSx+=mg@mail.gmail.com>
 <CAP7h-xb1tPatRYaYjpiTuokNBpHSFBBZ=8MGkw+_Ym5=JhD4Lw@mail.gmail.com>
Message-ID: <CACfEFw_MEe8TqgAT0TcMaDbur_NMwOdv+YsZpZUW8CcbY8nSjQ@mail.gmail.com>

https://schema.org/CreativeWork
  https://schema.org/Code
  https://schema.org/SoftwareApplication

CreativeWork has a https://schema.org/citation field with a range of
{CreativeWork, Text}

There's also a https://schema.org/funder attribute with a domain of
CreativeWork and a range of {Organization, Person}

- BibTeX is actually somewhat ill-specified, TBH.
- There is a repository of CSL styles at https://citationstyles.org .
- CSL is sponsored by both Zotero and Mendeley.
- A number of search engines support schema.org (and JSONLD)
- The schema.org RDFS vocabulary is designed to describe a graph of
resources (CreativeWork, Code, SoftwareApplication, ScholarlyArticle,
MedicalScholarlyArticle).

__citation__ = [{}, ]
__citation__ = {
  '@type': ['schema:ScholarlyArticle'],
  'schema:name': '',
  'schema:author': [{
      '@type': 'schema:Person',
      '...': '...'}]
}

JSONLD is ideal for describing a graph of resources with varied types.

If the overhead of __citation__ for every import is unjustified,
a lookup of methods with dotted names that finds entries for root modules
as well would be great:

>>> citations('json.loads')
>>> citations('list.sort')

A tracing debugger could lookup each and every package, module, function,
and method each ScholarlyArticle SoftwareApplication executes (from a
registry in e.g. a _citations_.py or a _citations_.jsonld.json).

It'd be a shame to need to manually format citations for a particular
Journal's CSL bibliographic  metadata template preference.

sphinxcontrib-bibtex is a Sphinx extension for BibTeX support (with a
bibliography directive and a cite role)
- Src: https://github.com/mcmtroffaes/sphinxcontrib-bibtex

Jupyter notebooks support document-level metadata (in JSON that's currently
only similar to schema.org JSONLD).

https://schema.org/ScholarlyArticle is search engine indexable.


On Wednesday, July 4, 2018, Alexander Belopolsky <
alexander.belopolsky at gmail.com> wrote:

>
>
> On Sun, Jul 1, 2018 at 9:45 AM David Mertz <mertz at gnosis.cx> wrote:
>
>> ..
>> There's absolutely nothing in the idea that requires a change in Python,
>> and Python developers or users are not, as such, the relevant experts.
>>
>
> This is not entirely true.  If some variant of __citation__ is endorsed by
> the community, I would expect that pydoc would extract this information to
> fill an appropriate section in the documentation page.  Note that pydoc
> already treats a number of dunder variables specially: '__author__', '__credits__',
> and '__version__' are a few that come to mind, so I don't think the
> threshold for adding one more should be too high.  On the other hand,
> maybe   '__author__', '__credits__', and '__citation__' should be merged
> in one structured variable (a dict?) with format designed with some
> extendability in mind.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180704/410dd07f/attachment-0001.html>

From wes.turner at gmail.com  Wed Jul  4 21:19:04 2018
From: wes.turner at gmail.com (Wes Turner)
Date: Wed, 4 Jul 2018 21:19:04 -0400
Subject: [Python-ideas] Add a __cite__ method for scientific packages
In-Reply-To: <CACfEFw_MEe8TqgAT0TcMaDbur_NMwOdv+YsZpZUW8CcbY8nSjQ@mail.gmail.com>
References: <CADit4boE-L+OOxKLpmyOzeS55F=VC+ifLgXxEpQNz1XgcFOu_g@mail.gmail.com>
 <CAPJVwBnUUS3Wz-NSsdUGxxp3Nr0gPhzoRbRaJq+1rxgp3JMwJQ@mail.gmail.com>
 <CAOMjWkm6e31d6c_5==t7rBEYu2jusNB3WAxJhknQRR4Mutkkbw@mail.gmail.com>
 <CAEbHw4Y61oAT7a7rdRC+ShBzbNNsv0VamO1CGg+mTfaSSx+=mg@mail.gmail.com>
 <CAP7h-xb1tPatRYaYjpiTuokNBpHSFBBZ=8MGkw+_Ym5=JhD4Lw@mail.gmail.com>
 <CACfEFw_MEe8TqgAT0TcMaDbur_NMwOdv+YsZpZUW8CcbY8nSjQ@mail.gmail.com>
Message-ID: <CACfEFw-xb-5Lz=T5dEy24hyTy4oNaxEejs6O2939ZQs5GuQ1ig@mail.gmail.com>

... a schema:Dataset may be part of a Creative work.

https://schema.org/Dataset
https://schema.org/isPartOf
https://schema.org/ScholarlyArticle

#LinkedReproducibility #nbmeta

On Wednesday, July 4, 2018, Wes Turner <wes.turner at gmail.com> wrote:

> https://schema.org/CreativeWork
>   https://schema.org/Code
>   https://schema.org/SoftwareApplication
>
> CreativeWork has a https://schema.org/citation field with a range of
> {CreativeWork, Text}
>
> There's also a https://schema.org/funder attribute with a domain of
> CreativeWork and a range of {Organization, Person}
>
> - BibTeX is actually somewhat ill-specified, TBH.
> - There is a repository of CSL styles at https://citationstyles.org .
> - CSL is sponsored by both Zotero and Mendeley.
> - A number of search engines support schema.org (and JSONLD)
> - The schema.org RDFS vocabulary is designed to describe a graph of
> resources (CreativeWork, Code, SoftwareApplication, ScholarlyArticle,
> MedicalScholarlyArticle).
>
> __citation__ = [{}, ]
> __citation__ = {
>   '@type': ['schema:ScholarlyArticle'],
>   'schema:name': '',
>   'schema:author': [{
>       '@type': 'schema:Person',
>       '...': '...'}]
> }
>
> JSONLD is ideal for describing a graph of resources with varied types.
>
> If the overhead of __citation__ for every import is unjustified,
> a lookup of methods with dotted names that finds entries for root modules
> as well would be great:
>
> >>> citations('json.loads')
> >>> citations('list.sort')
>
> A tracing debugger could lookup each and every package, module, function,
> and method each ScholarlyArticle SoftwareApplication executes (from a
> registry in e.g. a _citations_.py or a _citations_.jsonld.json).
>
> It'd be a shame to need to manually format citations for a particular
> Journal's CSL bibliographic  metadata template preference.
>
> sphinxcontrib-bibtex is a Sphinx extension for BibTeX support (with a
> bibliography directive and a cite role)
> - Src: https://github.com/mcmtroffaes/sphinxcontrib-bibtex
>
> Jupyter notebooks support document-level metadata (in JSON that's
> currently only similar to schema.org JSONLD).
>
> https://schema.org/ScholarlyArticle is search engine indexable.
>
>
> On Wednesday, July 4, 2018, Alexander Belopolsky <
> alexander.belopolsky at gmail.com> wrote:
>
>>
>>
>> On Sun, Jul 1, 2018 at 9:45 AM David Mertz <mertz at gnosis.cx> wrote:
>>
>>> ..
>>> There's absolutely nothing in the idea that requires a change in Python,
>>> and Python developers or users are not, as such, the relevant experts.
>>>
>>
>> This is not entirely true.  If some variant of __citation__ is endorsed
>> by the community, I would expect that pydoc would extract this information
>> to fill an appropriate section in the documentation page.  Note that pydoc
>> already treats a number of dunder variables specially: '__author__', '__credits__',
>> and '__version__' are a few that come to mind, so I don't think the
>> threshold for adding one more should be too high.  On the other hand,
>> maybe   '__author__', '__credits__', and '__citation__' should be merged
>> in one structured variable (a dict?) with format designed with some
>> extendability in mind.
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180704/49414dce/attachment.html>

From wes.turner at gmail.com  Wed Jul  4 22:10:21 2018
From: wes.turner at gmail.com (Wes Turner)
Date: Wed, 4 Jul 2018 22:10:21 -0400
Subject: [Python-ideas] Add a __cite__ method for scientific packages
In-Reply-To: <CACfEFw-xb-5Lz=T5dEy24hyTy4oNaxEejs6O2939ZQs5GuQ1ig@mail.gmail.com>
References: <CADit4boE-L+OOxKLpmyOzeS55F=VC+ifLgXxEpQNz1XgcFOu_g@mail.gmail.com>
 <CAPJVwBnUUS3Wz-NSsdUGxxp3Nr0gPhzoRbRaJq+1rxgp3JMwJQ@mail.gmail.com>
 <CAOMjWkm6e31d6c_5==t7rBEYu2jusNB3WAxJhknQRR4Mutkkbw@mail.gmail.com>
 <CAEbHw4Y61oAT7a7rdRC+ShBzbNNsv0VamO1CGg+mTfaSSx+=mg@mail.gmail.com>
 <CAP7h-xb1tPatRYaYjpiTuokNBpHSFBBZ=8MGkw+_Ym5=JhD4Lw@mail.gmail.com>
 <CACfEFw_MEe8TqgAT0TcMaDbur_NMwOdv+YsZpZUW8CcbY8nSjQ@mail.gmail.com>
 <CACfEFw-xb-5Lz=T5dEy24hyTy4oNaxEejs6O2939ZQs5GuQ1ig@mail.gmail.com>
Message-ID: <CACfEFw8uXQOoDLdMmMTOw79PoOB0CGGbQeTMV28Go0fpnFmsdQ@mail.gmail.com>

typeshed, dotted lookup, ScholarlyArticle semantic graphs with classes,
properties, and URIs

Would external metadata (similar to how typeshed is defined in a 'shadow
naming scheme' (?)) be advantageous
for dotted name lookup of citation metadata?

> Typeshed contains external type annotations for the Python standard
library and Python builtins, as well as third party packages.
>

> This data can e.g. be used for static analysis, type checking or type
inference.

https://github.com/python/typeshed
stdlib/{2, 2and3, 3, 3.5, 3.6, 3.7}
third_party/{2, 2and3, 3}/{jinja2,}

Ideally, a ScholarlyArticle can also be published as HTML with RDFa and/or
JSONLD (in addition to two column LaTeX/PDF which is lossy in regards to
structured data / linked data) with its own document-level metadata simply
as part of a graph of resources (such as schema:citation and
schema:Datasets) described using a search-indexed vocabulary such as the
Schema.org RDFS vocabulary.

An aside:
https://schema.org/unitCode has a range of {Text, URL} where the Text
should be a 3 character UN/CEFACT Common Code; but there's also QUDT for
unit URIs; fortunately, RDF allows repeated property values, so we can just
add both.

On Wednesday, July 4, 2018, Wes Turner <wes.turner at gmail.com> wrote:

> ... a schema:Dataset may be part of a Creative work.
>
> https://schema.org/Dataset
> https://schema.org/isPartOf
> https://schema.org/ScholarlyArticle
>
> #LinkedReproducibility #nbmeta
>
> On Wednesday, July 4, 2018, Wes Turner <wes.turner at gmail.com> wrote:
>
>> https://schema.org/CreativeWork
>>   https://schema.org/Code
>>   https://schema.org/SoftwareApplication
>>
>> CreativeWork has a https://schema.org/citation field with a range of
>> {CreativeWork, Text}
>>
>> There's also a https://schema.org/funder attribute with a domain of
>> CreativeWork and a range of {Organization, Person}
>>
>> - BibTeX is actually somewhat ill-specified, TBH.
>> - There is a repository of CSL styles at https://citationstyles.org .
>> - CSL is sponsored by both Zotero and Mendeley.
>> - A number of search engines support schema.org (and JSONLD)
>> - The schema.org RDFS vocabulary is designed to describe a graph of
>> resources (CreativeWork, Code, SoftwareApplication, ScholarlyArticle,
>> MedicalScholarlyArticle).
>>
>> __citation__ = [{}, ]
>> __citation__ = {
>>   '@type': ['schema:ScholarlyArticle'],
>>   'schema:name': '',
>>   'schema:author': [{
>>       '@type': 'schema:Person',
>>       '...': '...'}]
>> }
>>
>> JSONLD is ideal for describing a graph of resources with varied types.
>>
>> If the overhead of __citation__ for every import is unjustified,
>> a lookup of methods with dotted names that finds entries for root modules
>> as well would be great:
>>
>> >>> citations('json.loads')
>> >>> citations('list.sort')
>>
>> A tracing debugger could lookup each and every package, module, function,
>> and method each ScholarlyArticle SoftwareApplication executes (from a
>> registry in e.g. a _citations_.py or a _citations_.jsonld.json).
>>
>> It'd be a shame to need to manually format citations for a particular
>> Journal's CSL bibliographic  metadata template preference.
>>
>> sphinxcontrib-bibtex is a Sphinx extension for BibTeX support (with a
>> bibliography directive and a cite role)
>> - Src: https://github.com/mcmtroffaes/sphinxcontrib-bibtex
>>
>> Jupyter notebooks support document-level metadata (in JSON that's
>> currently only similar to schema.org JSONLD).
>>
>> https://schema.org/ScholarlyArticle is search engine indexable.
>>
>>
>> On Wednesday, July 4, 2018, Alexander Belopolsky <
>> alexander.belopolsky at gmail.com> wrote:
>>
>>>
>>>
>>> On Sun, Jul 1, 2018 at 9:45 AM David Mertz <mertz at gnosis.cx> wrote:
>>>
>>>> ..
>>>> There's absolutely nothing in the idea that requires a change in
>>>> Python, and Python developers or users are not, as such, the relevant
>>>> experts.
>>>>
>>>
>>> This is not entirely true.  If some variant of __citation__ is endorsed
>>> by the community, I would expect that pydoc would extract this information
>>> to fill an appropriate section in the documentation page.  Note that pydoc
>>> already treats a number of dunder variables specially: '__author__', '__credits__',
>>> and '__version__' are a few that come to mind, so I don't think the
>>> threshold for adding one more should be too high.  On the other hand,
>>> maybe   '__author__', '__credits__', and '__citation__' should be
>>> merged in one structured variable (a dict?) with format designed with some
>>> extendability in mind.
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180704/3c37d6d8/attachment-0001.html>

From chris.barker at noaa.gov  Thu Jul  5 01:14:04 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Wed, 4 Jul 2018 22:14:04 -0700
Subject: [Python-ideas] Where should grouping() live (was: grouping /
 dict of lists)
In-Reply-To: <CAEfz+TxsBJKQ6=JbiBnTbCgz=xpeOmLpLn-kFsFUdXvmvrsazw@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CAP7+vJJ6vN9H8esj8tS+bSKsFELh8q8ywYHAW8Ti_upEJHf1vg@mail.gmail.com>
 <CADqi797HXzHcUnrNLp7r+pFEM5fcApiKHV4s=P5-D6mXHYYv-g@mail.gmail.com>
 <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
 <CAEfz+TxsBJKQ6=JbiBnTbCgz=xpeOmLpLn-kFsFUdXvmvrsazw@mail.gmail.com>
Message-ID: <CALGmxEKTUKE8DdQtO-vOmMt6qOp5wtxkHdg4mkUKZckNgpT7QQ@mail.gmail.com>

On Wed, Jul 4, 2018 at 3:53 AM, INADA Naoki <songofacandy at gmail.com> wrote:

> But if it happens, I'm -1 on functools and collections.
> They are used very much.  Every Python tool import them regardless how
> much of their contents are used.
>

really? collections?  what for? I'm guessing namedtuple and maybe deque.

But collections already has 9 classes (well, things) in it so we'd be
adding a bit less than 10% more to it.

what is the concern? import time, memory?

In either case, it seems like the wrong driver for deciding where to put
new things.

> If you really want to add it in collections, I suggests
from collections.groupdict import GroupDict.

Perhaps the stdlib should have a deeper namespaces in general -- if that is
established as a policy, then this could be the first thing to follow that
policy. But I thought "flat is better than nested" -- sigh.

So maybe we need to bite the bullet and solve the problem at another level:

1) if, say, namedtuple has gotten very popular, maybe it should move to
builtins.

2) Whatever happened to the proposals to make it easier to lazy-load stuff
in modules? If that gets implemented, then we can speed up startup in
general, and not have to be too worried about adding "too much" to a module
because one thing in it is common use.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180704/5b85724f/attachment.html>

From chris.barker at noaa.gov  Thu Jul  5 01:15:15 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Wed, 4 Jul 2018 22:15:15 -0700
Subject: [Python-ideas] Where should grouping() live (was: grouping /
 dict of lists)
In-Reply-To: <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CAP7+vJJ6vN9H8esj8tS+bSKsFELh8q8ywYHAW8Ti_upEJHf1vg@mail.gmail.com>
 <CADqi797HXzHcUnrNLp7r+pFEM5fcApiKHV4s=P5-D6mXHYYv-g@mail.gmail.com>
 <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
Message-ID: <CALGmxELSOU6HEeQfDfiC++CuvEF3GBxEQDuvv7Kb=ZeLF-Jucw@mail.gmail.com>

On Tue, Jul 3, 2018 at 6:23 AM, David Mertz <mertz at gnosis.cx> wrote:

> Guido said he has mooted this discussion
>

...

But before putting it on auto-archive, the BDFL said (1) NO GO on getting a
new builtin; (2) NO OBJECTION to putting it in itertools.

I don't recall him offering an opinion on a class in collections, did he?

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180704/9386f297/attachment.html>

From chris.barker at noaa.gov  Thu Jul  5 01:23:38 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Wed, 4 Jul 2018 22:23:38 -0700
Subject: [Python-ideas] Fwd: grouping / dict of lists
In-Reply-To: <CAEbHw4Yjue-V_CKQJe2k6f4E8ucNuG=ZNFt5O_njfrQoNKHeUw@mail.gmail.com>
References: <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAOmnVUBp9vJjTp6v17LHy2C=7GzR63fb4ARZW2Dco022uTDpfA@mail.gmail.com>
 <20180703153338.GG14437@ando.pearwood.info>
 <CALGmxEJBYFEoPOFo7aVSN=Xy=sgSMnrF0k8pnt=VhJ6cvhS_4Q@mail.gmail.com>
 <20180704013202.GK14437@ando.pearwood.info>
 <CAEbHw4YafoZc_1cnB6a-psmWqW-SD2tw3E2Fw5hUHSJxqEsu-Q@mail.gmail.com>
 <CAEbHw4Yjue-V_CKQJe2k6f4E8ucNuG=ZNFt5O_njfrQoNKHeUw@mail.gmail.com>
Message-ID: <CALGmxE+m+1YXk1vRsdkAK6yQ12V=syi6WKZsSTMFmw1BoiKUYA@mail.gmail.com>

On Wed, Jul 4, 2018 at 6:34 AM, David Mertz <mertz at gnosis.cx> wrote:


> You've misunderstood part of the discussion. There are two different
> signatures being discussed/proposed for a grouping() function.
>
> The one you show we might call grouping_michael(). The alternate API we
> might call grouping_chris(). These two calls will produce the same result
> (the first output you show)
>
>   grouping_michael(words, keyfunc=len)
>   grouping_chris((len(word), word) for word in words)
>
> I happen to prefer grouping_michael(), but recognize they each make
> slightly different things obvious.
>

I starting thinking grouping_chris was the obvious and natural thing to do,
but his discussion has made it clear that grouping_michael is more natural
for some kinds of data.

and in some cases, it really comes down to taste, after all, who's to say
which of these is "better"

map(func, iterable)

or

(expression for item in iterable)

given that map existed in Python when comprehensions were added, I tend to
see the latter as more "Pythonic" but that's just me.


So I'm currently lobbying for both :-)

The default is iterable of (key. value) pairs, but the use can specify a
key function is they want to do it that way.

While a bit of a schizophrenic  API, it makes sens (to me), because
grouping_mikael isn't useful with a default key function anyway.

The other enhancement I suggest is that an (optional) value function be
added, as there are use cases where that would be really helpful.

-CHB

NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180704/45fdd987/attachment.html>

From mertz at gnosis.cx  Thu Jul  5 06:26:05 2018
From: mertz at gnosis.cx (David Mertz)
Date: Thu, 5 Jul 2018 06:26:05 -0400
Subject: [Python-ideas] Where should grouping() live (was: grouping /
 dict of lists)
In-Reply-To: <CALGmxELSOU6HEeQfDfiC++CuvEF3GBxEQDuvv7Kb=ZeLF-Jucw@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CAP7+vJJ6vN9H8esj8tS+bSKsFELh8q8ywYHAW8Ti_upEJHf1vg@mail.gmail.com>
 <CADqi797HXzHcUnrNLp7r+pFEM5fcApiKHV4s=P5-D6mXHYYv-g@mail.gmail.com>
 <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
 <CALGmxELSOU6HEeQfDfiC++CuvEF3GBxEQDuvv7Kb=ZeLF-Jucw@mail.gmail.com>
Message-ID: <CAEbHw4ZXqRoMWnOWjqgPQ0OjcGm0MFPW_VhvZdnDuVvWygDc+w@mail.gmail.com>

Yes, he said a definite no to a built-in. But he expressed a less specific
lack of enthusiasm for collections classes (including Counter, which exists
and which I personally use often).

On Thu, Jul 5, 2018, 1:16 AM Chris Barker <chris.barker at noaa.gov> wrote:

> On Tue, Jul 3, 2018 at 6:23 AM, David Mertz <mertz at gnosis.cx> wrote:
>
>> Guido said he has muted this discussion
>>
>
> ...
>
> But before putting it on auto-archive, the BDFL said (1) NO GO on getting
> a new builtin; (2) NO OBJECTION to putting it in itertools.
>
> I don't recall him offering an opinion on a class in collections, did he?
>
> -CHB
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180705/8a22510c/attachment-0001.html>

From paal.drange at gmail.com  Thu Jul  5 14:11:21 2018
From: paal.drange at gmail.com (=?UTF-8?B?UMOlbCBHcsO4bsOlcyBEcmFuZ2U=?=)
Date: Thu, 5 Jul 2018 20:11:21 +0200
Subject: [Python-ideas] Add `rc` to distutils.version.StrictVersion
Message-ID: <CAPL7A84_cUkG3DW270W8Y8NYELub5ZjszcgLuW4MEW=N+3xj7w@mail.gmail.com>

StrictVersion from distutils accepts version tags like
1.14.0
1.14.0a1
1.14.0b2

but not
1.14.0rc1 (nor 1.14.0c1).

My suggestion:  Add `rc` in the regexp and make it a `prerelease` (the
latter comes for free by the current implementation).

Most package maintainers have adopted the `rc` abbreviation for release
candidate versioning, e.g.
- numpy 1.14.0rc1
- scipy 1.1.0rc1
- plotly 3.0.0rc1
- pandas 0.23.0rc1
- matplotlib 2.2.0rc1
- dask 0.13.0rc1
- django 1.9rc1.
All of these are available on PyPI.

A natural way of sorting version numbers from pip is by simply using
    sorted(versions, key=distutils.version.StrictVersion),
however, due to StrictVersion only accepting `a` and `b` as
abbreviations, this does not work for the aforemention packages.

The very obvious cons are:
- touching 20 years old code [1]
- StrictVersion is preserved "for anal retentives and software
  idealists", and I don't know if they agree.

There might be more cons I fail to think of at this moment.

[1]
https://github.com/python/cpython/blob/master/Lib/distutils/version.py#L130

P?l Gr?n?s Drange
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180705/8af1c1b2/attachment.html>

From flavio.curella at gmail.com  Thu Jul  5 15:38:47 2018
From: flavio.curella at gmail.com (Flavio Curella)
Date: Thu, 5 Jul 2018 14:38:47 -0500
Subject: [Python-ideas] Add new `Symbol` type
Message-ID: <CAAjMAJWeXOofJtz=OsU8OWq3QTZVsxEKfbmBeDivQrD=-pNZTA@mail.gmail.com>

More than once I've found myself wanting to create a 'sentinel' value. The
most common use case is to differentiate between an argument that has not
been provided, and an argument provided with the value `None`.

This would be solvable by implementing something similar to what JavaScript
calls [`Symbol`](
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Symbol
).

This could be implemented as a 3rd-party library, but there won't be a way
to have ['Global' Symbols](
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Symbol/for
)

Furthermore, without a common implementation in the std library, various
Python libraries had to write their own implementations, which all differs
in functionality and behavior.

Is this something that the Python community is interested in? I'm willing
to write the PEP
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180705/7eae01e2/attachment.html>

From e+python-ideas at kellett.im  Thu Jul  5 15:44:46 2018
From: e+python-ideas at kellett.im (Ed Kellett)
Date: Thu, 5 Jul 2018 20:44:46 +0100
Subject: [Python-ideas] Add new `Symbol` type
In-Reply-To: <CAAjMAJWeXOofJtz=OsU8OWq3QTZVsxEKfbmBeDivQrD=-pNZTA@mail.gmail.com>
References: <CAAjMAJWeXOofJtz=OsU8OWq3QTZVsxEKfbmBeDivQrD=-pNZTA@mail.gmail.com>
Message-ID: <15c549d4-bdf1-69e0-2d11-58a38afe3d4b@kellett.im>

Hi!

On 2018-07-05 20:38, Flavio Curella wrote:
> More than once I've found myself wanting to create a 'sentinel' value. The
> most common use case is to differentiate between an argument that has not
> been provided, and an argument provided with the value `None`.

I generally do something like

    _nothing = object()

> Furthermore, without a common implementation in the std library, various
> Python libraries had to write their own implementations, which all differs
> in functionality and behavior.

What functionality does such a thing actually need?

From joejev at gmail.com  Thu Jul  5 16:22:07 2018
From: joejev at gmail.com (Joseph Jevnik)
Date: Thu, 5 Jul 2018 16:22:07 -0400
Subject: [Python-ideas] Add new `Symbol` type
In-Reply-To: <15c549d4-bdf1-69e0-2d11-58a38afe3d4b@kellett.im>
References: <CAAjMAJWeXOofJtz=OsU8OWq3QTZVsxEKfbmBeDivQrD=-pNZTA@mail.gmail.com>
 <15c549d4-bdf1-69e0-2d11-58a38afe3d4b@kellett.im>
Message-ID: <CAHGq92UbJ7y5vPotrge-LrEaO0YJ6r+sVnCseGzk+J=pW-bi2w@mail.gmail.com>

I have also wanted sentinel objects many times. These are often useful
for creating a "Not Specified" default value when explicitly passing
`None` has semantic meaning.

There are a few issues with the `sentinel = object()` code. One is
that they don't repr well so they make debugging harder. Another issue
is that they cannot be pickled or copied. You also cannot take a weak
reference to a sentinel which can break some caching code and makes
them harder to use. At work we have a small helper to create sentinels
with a name and optional doc string which is open sourced here:
https://github.com/quantopian/zipline/blob/master/zipline/utils/sentinel.py.

On Thu, Jul 5, 2018 at 3:44 PM, Ed Kellett <e+python-ideas at kellett.im> wrote:
> Hi!
>
> On 2018-07-05 20:38, Flavio Curella wrote:
>> More than once I've found myself wanting to create a 'sentinel' value. The
>> most common use case is to differentiate between an argument that has not
>> been provided, and an argument provided with the value `None`.
>
> I generally do something like
>
>     _nothing = object()
>
>> Furthermore, without a common implementation in the std library, various
>> Python libraries had to write their own implementations, which all differs
>> in functionality and behavior.
>
> What functionality does such a thing actually need?
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From flavio.curella at gmail.com  Thu Jul  5 16:25:18 2018
From: flavio.curella at gmail.com (Flavio Curella)
Date: Thu, 5 Jul 2018 15:25:18 -0500
Subject: [Python-ideas] Add new `Symbol` type
In-Reply-To: <CAAjMAJX1AcNNhZGSag-XPbZD3-d8jESPQw6Y71B4QqmOvXajxQ@mail.gmail.com>
References: <CAAjMAJWeXOofJtz=OsU8OWq3QTZVsxEKfbmBeDivQrD=-pNZTA@mail.gmail.com>
 <15c549d4-bdf1-69e0-2d11-58a38afe3d4b@kellett.im>
 <CAAjMAJVBTPtKbbajTWLeDYLCW24QNRq18VGGL=O28A=_sn0w1w@mail.gmail.com>
 <CAAjMAJX1AcNNhZGSag-XPbZD3-d8jESPQw6Y71B4QqmOvXajxQ@mail.gmail.com>
Message-ID: <CAAjMAJVCJsZgqFnOr_wZMyDfz6CGoh7ExRZyLQjq2OV6DB=GNA@mail.gmail.com>

> What functionality does such a thing actually need?

I think the requirements should be:
* The resulting symbol behave exactly like None. IE: the symbol should not
be an instance of object, but an instance of its own class
* A symbol can optionally be globally unique.
* Two symbols created by the same key must not be equal. IE: they have
equal key, but different value
   * if we're trying to create global symbols with the same key, an
exception is thrown

This is mostly based on the Javascript spec.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180705/df9a5ae6/attachment-0001.html>

From chris.barker at noaa.gov  Thu Jul  5 17:23:20 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Thu, 5 Jul 2018 14:23:20 -0700
Subject: [Python-ideas] Where should grouping() live (was: grouping /
 dict of lists)
In-Reply-To: <CAEbHw4ZXqRoMWnOWjqgPQ0OjcGm0MFPW_VhvZdnDuVvWygDc+w@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CAP7+vJJ6vN9H8esj8tS+bSKsFELh8q8ywYHAW8Ti_upEJHf1vg@mail.gmail.com>
 <CADqi797HXzHcUnrNLp7r+pFEM5fcApiKHV4s=P5-D6mXHYYv-g@mail.gmail.com>
 <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
 <CALGmxELSOU6HEeQfDfiC++CuvEF3GBxEQDuvv7Kb=ZeLF-Jucw@mail.gmail.com>
 <CAEbHw4ZXqRoMWnOWjqgPQ0OjcGm0MFPW_VhvZdnDuVvWygDc+w@mail.gmail.com>
Message-ID: <CALGmxEJJbRRmGBMKOUK6G4TUpxLGvceU3aqq6t0vrGMbFUwS_g@mail.gmail.com>

On Thu, Jul 5, 2018 at 3:26 AM, David Mertz <mertz at gnosis.cx> wrote:

> Yes, he said a definite no to a built-in. But he expressed a less specific
> lack of enthusiasm for collections classes (including Counter, which exists
> and which I personally use often).
>

And a Grouping class would do more than Counter, which I find trivial
enough that I generally don't bother to use it.

-CHB
-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180705/56836433/attachment.html>

From njs at pobox.com  Thu Jul  5 20:26:36 2018
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 5 Jul 2018 17:26:36 -0700
Subject: [Python-ideas] Add new `Symbol` type
In-Reply-To: <CAAjMAJVCJsZgqFnOr_wZMyDfz6CGoh7ExRZyLQjq2OV6DB=GNA@mail.gmail.com>
References: <CAAjMAJWeXOofJtz=OsU8OWq3QTZVsxEKfbmBeDivQrD=-pNZTA@mail.gmail.com>
 <15c549d4-bdf1-69e0-2d11-58a38afe3d4b@kellett.im>
 <CAAjMAJVBTPtKbbajTWLeDYLCW24QNRq18VGGL=O28A=_sn0w1w@mail.gmail.com>
 <CAAjMAJX1AcNNhZGSag-XPbZD3-d8jESPQw6Y71B4QqmOvXajxQ@mail.gmail.com>
 <CAAjMAJVCJsZgqFnOr_wZMyDfz6CGoh7ExRZyLQjq2OV6DB=GNA@mail.gmail.com>
Message-ID: <CAPJVwBnK-eLvTkOzuXEXu=xH_UJZuvWXh2HOopyiJPXMJk6ugQ@mail.gmail.com>

On Thu, Jul 5, 2018 at 1:25 PM, Flavio Curella <flavio.curella at gmail.com> wrote:
>
>> What functionality does such a thing actually need?
>
> I think the requirements should be:
> * The resulting symbol behave exactly like None. IE: the symbol should not
> be an instance of object, but an instance of its own class
> * A symbol can optionally be globally unique.
> * Two symbols created by the same key must not be equal. IE: they have equal
> key, but different value
>    * if we're trying to create global symbols with the same key, an
> exception is thrown
>
> This is mostly based on the Javascript spec.

I think the name "symbol" here is pretty confusing. It comes
originally from Lisp, where it's used to refer to an interned-string
data type. It's a common source of confusion even there. Then it
sounds like JS took that name, and it ended up drifting to mean
something that's almost exactly the opposite of a Lisp symbol. In
Lisp, symbols are always "global"; the whole point is that if two
different pieces of code use the same name for the same symbol then
they end up with the same object. So this is *super* confusing. I
think I see how JS ended up here [1], but the rationale really doesn't
translate to other languages.

The thing you're talking about is what Python devs call a "sentinel"
object. If your proposal is to add a sentinel type to the stdlib, then
your chance of success will be *much* higher if you use the word
"sentinel" instead of "symbol". People don't read mailing list threads
carefully, so if you keep calling it "symbol" then you'll likely spend
infinite time responding to people rushing in to critique your
proposal based on some misconception about what you're trying to do,
which is no fun at all. Honestly I'd probably start a new thread with
a new subject, ideally with an initial straw-man proposal for the
semantics of these objects.

-n

[1] What was JS thinking? Well, I'm not sure I have all the details
right, but AFAICT it's all very logical... JS objects, like Python
objects, have attributes, e.g. 'console.log' is the 'log' attribute of
the 'console' object. There's a table inside the 'console' object
mapping keys like 'log' to their corresponding values, much like a
Python object's __dict__. But a Python dict can use arbitrary objects
as keys. JS attribute tables are different: the keys are required to
be Lisp-style symbol objects: they're arbitrary strings (and only
strings), that are then interned for speed. This kind of table lookup
is exactly why Lisp invented symbols in the first place; a Lisp scope
is also a table mapping symbols to values. BUT THEN, they decided to
enhance JS to add the equivalent of special methods like Python's
__add__. Now how do you tell which attributes are ordinary attributes,
and which ones are supposed to be special? In Python of course we use
a naming convention, which is simple and works well. But in JS, by the
time they decided to do this, it was too late: people might already be
using names like "__add__" for regular attributes, and making them
special would break compatibility. In fact, *all* possible strings
were potentially already in use for ordinary attributes; there were no
names left for special attributes. SO, they decided, they needed to
expand the set of symbol objects (i.e., attribute names) to include
new values that were different from all possible strings. So now the
JS Symbol class is effectively the union of {strings, compared as
values} + {sentinels, compared by identity}. And for string
attributes, you can mostly ignore all this and pretend they're
ordinary strings and the JS interpreter will paper over the details.
So the main kind of symbol that JS devs actually have to *know* about
is the new sentinel values. And that's how the name "symbol" flipped
to mean the opposite of what it used to. See? I told you it was all
very logical.

-- 
Nathaniel J. Smith -- https://vorpus.org

From steve at pearwood.info  Fri Jul  6 02:29:30 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 6 Jul 2018 16:29:30 +1000
Subject: [Python-ideas] Add new `Symbol` type
In-Reply-To: <CAAjMAJWeXOofJtz=OsU8OWq3QTZVsxEKfbmBeDivQrD=-pNZTA@mail.gmail.com>
References: <CAAjMAJWeXOofJtz=OsU8OWq3QTZVsxEKfbmBeDivQrD=-pNZTA@mail.gmail.com>
Message-ID: <20180706062930.GF7318@ando.pearwood.info>

On Thu, Jul 05, 2018 at 02:38:47PM -0500, Flavio Curella wrote:
> More than once I've found myself wanting to create a 'sentinel' value. The
> most common use case is to differentiate between an argument that has not
> been provided, and an argument provided with the value `None`.
[...]
> Is this something that the Python community is interested in? I'm willing
> to write the PEP

I'm definitely interested, and coincidentally I started writing my own 
Symbol class a week or two ago (but put it aside unfinished due to other 
commitments) but honestly after four months of the assignment expression 
PEP I'm suffering from battle fatigue. It would be nice to have a break 
from controversial PEPs and catch my breath :-)


-- 
Steve

From fuzzyman at gmail.com  Fri Jul  6 04:36:39 2018
From: fuzzyman at gmail.com (Michael Foord)
Date: Fri, 6 Jul 2018 09:36:39 +0100
Subject: [Python-ideas] Add new `Symbol` type
In-Reply-To: <20180706062930.GF7318@ando.pearwood.info>
References: <CAAjMAJWeXOofJtz=OsU8OWq3QTZVsxEKfbmBeDivQrD=-pNZTA@mail.gmail.com>
 <20180706062930.GF7318@ando.pearwood.info>
Message-ID: <CAKCKLWw6MSmz2EVmVLipPDMymMWzJeGFDX0-Ar+D21U4MJjQAw@mail.gmail.com>

On Fri, 6 Jul 2018 at 07:30, Steven D'Aprano <steve at pearwood.info> wrote:

> On Thu, Jul 05, 2018 at 02:38:47PM -0500, Flavio Curella wrote:
> > More than once I've found myself wanting to create a 'sentinel' value.
> The
> > most common use case is to differentiate between an argument that has not
> > been provided, and an argument provided with the value `None`.
> [...]
> > Is this something that the Python community is interested in? I'm willing
> > to write the PEP
>
> I'm definitely interested, and coincidentally I started writing my own
> Symbol class a week or two ago (but put it aside unfinished due to other
> commitments) but honestly after four months of the assignment expression
> PEP I'm suffering from battle fatigue. It would be nice to have a break
> from controversial PEPs and catch my breath :-)
>

unittest.mock includes a sentinel object, with nice repr.

>>> from unittest.mock import sentinel
>>> sentinel.Foo
sentinel.Foo

Michael


>
> --
> Steve
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-- 

http://www.michaelfoord.co.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180706/28fb0e53/attachment-0001.html>

From ctaank at gmail.com  Fri Jul  6 04:49:37 2018
From: ctaank at gmail.com (Cammil Taank)
Date: Fri, 6 Jul 2018 09:49:37 +0100
Subject: [Python-ideas] Where should grouping() live (was: grouping /
 dict of lists)
In-Reply-To: <CALGmxEJJbRRmGBMKOUK6G4TUpxLGvceU3aqq6t0vrGMbFUwS_g@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CAP7+vJJ6vN9H8esj8tS+bSKsFELh8q8ywYHAW8Ti_upEJHf1vg@mail.gmail.com>
 <CADqi797HXzHcUnrNLp7r+pFEM5fcApiKHV4s=P5-D6mXHYYv-g@mail.gmail.com>
 <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CADiSq7c7F7wtPJH9R8Bn_rZMSuRQeUNdooTVV=rdG4Y_D6f8Hw@mail.gmail.com>
 <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
 <CALGmxELSOU6HEeQfDfiC++CuvEF3GBxEQDuvv7Kb=ZeLF-Jucw@mail.gmail.com>
 <CAEbHw4ZXqRoMWnOWjqgPQ0OjcGm0MFPW_VhvZdnDuVvWygDc+w@mail.gmail.com>
 <CALGmxEJJbRRmGBMKOUK6G4TUpxLGvceU3aqq6t0vrGMbFUwS_g@mail.gmail.com>
Message-ID: <CACc=A=2MmzE+=HLLCLMM=EY9HWTupfDZ7utaKPJQhUhP=eL0Gg@mail.gmail.com>

The way I see grouping is as an aggregation operation. As such, in my head,
grouping is similar to min/max. However, if builtins are a no-go, then I
feel I need to think a little outside the box:

Is there a possibility that there will be desired many more aggregate
functions in the near future? Is there a case for collecting aggregate
functions into another top level module? Also, I would consider statistics
to have similarities - median, mean etc are aggregate functions. Histograms
are also doing something similar to grouping.

Apologies I have not offered any concrete suggestions, but just thought I
should offer my thoughts.

On Thu, 5 Jul 2018, 22:24 Chris Barker via Python-ideas, <
python-ideas at python.org> wrote:

> On Thu, Jul 5, 2018 at 3:26 AM, David Mertz <mertz at gnosis.cx> wrote:
>
>> Yes, he said a definite no to a built-in. But he expressed a less
>> specific lack of enthusiasm for collections classes (including Counter,
>> which exists and which I personally use often).
>>
>
> And a Grouping class would do more than Counter, which I find trivial
> enough that I generally don't bother to use it.
>
> -CHB
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180706/af25f266/attachment.html>

From g.rodola at gmail.com  Fri Jul  6 05:07:34 2018
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Fri, 6 Jul 2018 11:07:34 +0200
Subject: [Python-ideas] Add new `Symbol` type
In-Reply-To: <CAAjMAJWeXOofJtz=OsU8OWq3QTZVsxEKfbmBeDivQrD=-pNZTA@mail.gmail.com>
References: <CAAjMAJWeXOofJtz=OsU8OWq3QTZVsxEKfbmBeDivQrD=-pNZTA@mail.gmail.com>
Message-ID: <CAFYqXL_DOyL5Rc76KZyjL4=J_ZdDZaHkktWfouC02M6=2BzSSw@mail.gmail.com>

Historically this has always been achieved by using:

_default = object()

def fun(arg=_default):
     if arg is not _default:
          ....

...which does its job just fine. If you need something like this you're
typically a medium/advanced Python user so you either already know about it
or you'll find the solution on Google pretty quickly. A dedicated
sentinel()/Sentinel/... thingy would just be more "official", but at the
end of the day it would achieve exactly the same thing by adding more
complexity to the language for no practical benefit (and I seriously think
Python is getting too big).

-1


On Thu, Jul 5, 2018 at 9:39 PM Flavio Curella <flavio.curella at gmail.com>
wrote:

> More than once I've found myself wanting to create a 'sentinel' value. The
> most common use case is to differentiate between an argument that has not
> been provided, and an argument provided with the value `None`.
>
> This would be solvable by implementing something similar to what
> JavaScript calls [`Symbol`](
> https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Symbol
> ).
>
> This could be implemented as a 3rd-party library, but there won't be a way
> to have ['Global' Symbols](
> https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Symbol/for
> )
>
> Furthermore, without a common implementation in the std library, various
> Python libraries had to write their own implementations, which all differs
> in functionality and behavior.
>
> Is this something that the Python community is interested in? I'm willing
> to write the PEP
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
Giampaolo - http://grodola.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180706/141858ec/attachment.html>

From steve at pearwood.info  Fri Jul  6 05:10:46 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 6 Jul 2018 19:10:46 +1000
Subject: [Python-ideas] Where should grouping() live (was: grouping /
 dict of lists)
In-Reply-To: <CACc=A=2MmzE+=HLLCLMM=EY9HWTupfDZ7utaKPJQhUhP=eL0Gg@mail.gmail.com>
References: <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
 <CALGmxELSOU6HEeQfDfiC++CuvEF3GBxEQDuvv7Kb=ZeLF-Jucw@mail.gmail.com>
 <CAEbHw4ZXqRoMWnOWjqgPQ0OjcGm0MFPW_VhvZdnDuVvWygDc+w@mail.gmail.com>
 <CALGmxEJJbRRmGBMKOUK6G4TUpxLGvceU3aqq6t0vrGMbFUwS_g@mail.gmail.com>
 <CACc=A=2MmzE+=HLLCLMM=EY9HWTupfDZ7utaKPJQhUhP=eL0Gg@mail.gmail.com>
Message-ID: <20180706091045.GH7318@ando.pearwood.info>

On Fri, Jul 06, 2018 at 09:49:37AM +0100, Cammil Taank wrote:

> I would consider statistics
> to have similarities - median, mean etc are aggregate functions. Histograms
> are also doing something similar to grouping.

I was thinking the same thing, but I don't think it is a good fit. 
Grouping records with arbitrary structure is very different from the 
numerically-focused statistics module. (Yes, a few statistics apply to 
nominal and ordinal data too, but the primary focus is on numbers.)



-- 
Steve

From flavio.curella at gmail.com  Fri Jul  6 11:20:26 2018
From: flavio.curella at gmail.com (Flavio Curella)
Date: Fri, 6 Jul 2018 10:20:26 -0500
Subject: [Python-ideas] Add new `Symbol` type
In-Reply-To: <CAFYqXL_DOyL5Rc76KZyjL4=J_ZdDZaHkktWfouC02M6=2BzSSw@mail.gmail.com>
References: <CAAjMAJWeXOofJtz=OsU8OWq3QTZVsxEKfbmBeDivQrD=-pNZTA@mail.gmail.com>
 <CAFYqXL_DOyL5Rc76KZyjL4=J_ZdDZaHkktWfouC02M6=2BzSSw@mail.gmail.com>
Message-ID: <CAAjMAJUzjfNr4dBQagVhKeF0TV8dPGtZqWbwhbAHM=xH10DCQg@mail.gmail.com>

@Nathaniel Smith:

> I think the name "symbol" here is pretty confusing. It comes originally
from Lisp
> The thing you're talking about is what Python devs call a "sentinel"
object.

 Thank you for clarifying. I don't know much about Lisp, and I definitely
appreciate the historical context that you provided :)

I will refer to the new proposed type as `sentinel` from now on.

@Michael Foord

> unittest.mock includes a sentinel object, with nice repr.

Thank you! I didn't think of looking in the mock library.

This is one step closer, but it still have a few considerations:
1. It doesn't follow the same behaviour as the JS spec. But honestly, we
don't have to.
2. It's kinda weird to have to import `unittest.<something>` in code that's
not tests. But I think it's just because I'm not used to see it.

@Giampaolo Rodola'
> Historically this has always been achieved by using [...] which does its
job just fine.

The main issue with that approach is that you won't get a nice repr

> adding more complexity to the language for no practical benefit

I'm not following. The Python language won't be modified. I'm proposing
adding the new type _purely_ for practical benefit.

I think this thread can be resolved as 'used unittest.mock.sentinel'. It
doesn't have 'global sentinels', but I'm not convinced they are actually
necessary, since `mock.sentinel` objects with the same name compare as
equal. Thanks to Nathaniel, I now understand that JS has global symbols for
historical reasons that we don't have, and I'm not convinced of their
usefulness.

Thank you everybody for you valuable feedback! I really appreciate your
time helping me thinking this through :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180706/46e783b8/attachment-0001.html>

From guido at python.org  Fri Jul  6 11:31:23 2018
From: guido at python.org (Guido van Rossum)
Date: Fri, 6 Jul 2018 08:31:23 -0700
Subject: [Python-ideas] Add new `Symbol` type
In-Reply-To: <CAAjMAJUzjfNr4dBQagVhKeF0TV8dPGtZqWbwhbAHM=xH10DCQg@mail.gmail.com>
References: <CAAjMAJWeXOofJtz=OsU8OWq3QTZVsxEKfbmBeDivQrD=-pNZTA@mail.gmail.com>
 <CAFYqXL_DOyL5Rc76KZyjL4=J_ZdDZaHkktWfouC02M6=2BzSSw@mail.gmail.com>
 <CAAjMAJUzjfNr4dBQagVhKeF0TV8dPGtZqWbwhbAHM=xH10DCQg@mail.gmail.com>
Message-ID: <CAP7+vJKPihPcWXJqB3V8EV3jAgXiL79YgDx_tpdY5txe+jYUQg@mail.gmail.com>

Thanks for an interesting discussion. I would also urge people to limit the
use of such sentinels for cases where it is *really* important to
distinguish between, say, f(None) and f(). In most cases using def
f(arg=None) is fine, and often it is even a virtue that passing None or
omitting an argument has exactly the same meaning. (I do know there a cases
where this doesn't apply -- I just think they ought to be fairly unusual.)

On Fri, Jul 6, 2018 at 8:21 AM Flavio Curella <flavio.curella at gmail.com>
wrote:

> @Nathaniel Smith:
>
> > I think the name "symbol" here is pretty confusing. It comes originally
> from Lisp
> > The thing you're talking about is what Python devs call a "sentinel"
> object.
>
>  Thank you for clarifying. I don't know much about Lisp, and I definitely
> appreciate the historical context that you provided :)
>
> I will refer to the new proposed type as `sentinel` from now on.
>
> @Michael Foord
>
> > unittest.mock includes a sentinel object, with nice repr.
>
> Thank you! I didn't think of looking in the mock library.
>
> This is one step closer, but it still have a few considerations:
> 1. It doesn't follow the same behaviour as the JS spec. But honestly, we
> don't have to.
> 2. It's kinda weird to have to import `unittest.<something>` in code
> that's not tests. But I think it's just because I'm not used to see it.
>
> @Giampaolo Rodola'
> > Historically this has always been achieved by using [...] which does
> its job just fine.
>
> The main issue with that approach is that you won't get a nice repr
>
> > adding more complexity to the language for no practical benefit
>
> I'm not following. The Python language won't be modified. I'm proposing
> adding the new type _purely_ for practical benefit.
>
> I think this thread can be resolved as 'used unittest.mock.sentinel'. It
> doesn't have 'global sentinels', but I'm not convinced they are actually
> necessary, since `mock.sentinel` objects with the same name compare as
> equal. Thanks to Nathaniel, I now understand that JS has global symbols for
> historical reasons that we don't have, and I'm not convinced of their
> usefulness.
>
> Thank you everybody for you valuable feedback! I really appreciate your
> time helping me thinking this through :)
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180706/ef37a43c/attachment.html>

From chris.barker at noaa.gov  Fri Jul  6 11:31:42 2018
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Fri, 6 Jul 2018 08:31:42 -0700
Subject: [Python-ideas] Where should grouping() live (was: grouping /
 dict of lists)
In-Reply-To: <20180706091045.GH7318@ando.pearwood.info>
References: <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
 <CALGmxELSOU6HEeQfDfiC++CuvEF3GBxEQDuvv7Kb=ZeLF-Jucw@mail.gmail.com>
 <CAEbHw4ZXqRoMWnOWjqgPQ0OjcGm0MFPW_VhvZdnDuVvWygDc+w@mail.gmail.com>
 <CALGmxEJJbRRmGBMKOUK6G4TUpxLGvceU3aqq6t0vrGMbFUwS_g@mail.gmail.com>
 <CACc=A=2MmzE+=HLLCLMM=EY9HWTupfDZ7utaKPJQhUhP=eL0Gg@mail.gmail.com>
 <20180706091045.GH7318@ando.pearwood.info>
Message-ID: <CALGmxEJCg-a22PB8KRWiqx8o6gNRXY7vYGQ4eSNSV-=8oMZE_w@mail.gmail.com>

On Jul 6, 2018, at 2:10 AM, Steven D'Aprano <steve at pearwood.info> wrote:


I would consider statistics

to have similarities - median, mean etc are aggregate functions.


Not really, more like reduce, actually -/ you get a single result.

Histograms

are also doing something similar to grouping.

.(Yes, a few statistics apply to
nominal and ordinal data too,


And for that, a generic grouping function could be used.

In fact, allowing Counter to be used as the accumulater was one suggestion
in this thread, and would build s histogram.

Now that I think about it, you could write a key function that built a
histogram for continuous data as well.

Though that might be a bit klunky.

But if someone thinks that?s a good idea, a PR for an example would be
accepted:

https://github.com/PythonCHB/grouper

-CHB






-- 
Steve
_______________________________________________
Python-ideas mailing list
Python-ideas at python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180706/b553110f/attachment.html>

From eric at trueblade.com  Fri Jul  6 12:23:32 2018
From: eric at trueblade.com (Eric V. Smith)
Date: Fri, 6 Jul 2018 12:23:32 -0400
Subject: [Python-ideas] Add new `Symbol` type
In-Reply-To: <CAAjMAJUzjfNr4dBQagVhKeF0TV8dPGtZqWbwhbAHM=xH10DCQg@mail.gmail.com>
References: <CAAjMAJWeXOofJtz=OsU8OWq3QTZVsxEKfbmBeDivQrD=-pNZTA@mail.gmail.com>
 <CAFYqXL_DOyL5Rc76KZyjL4=J_ZdDZaHkktWfouC02M6=2BzSSw@mail.gmail.com>
 <CAAjMAJUzjfNr4dBQagVhKeF0TV8dPGtZqWbwhbAHM=xH10DCQg@mail.gmail.com>
Message-ID: <d625da50-f7ea-05cd-89d5-2f314d8f2607@trueblade.com>

On 7/6/2018 11:20 AM, Flavio Curella wrote:
> I think this thread can be resolved as 'used unittest.mock.sentinel'. It 
> doesn't have 'global sentinels', but I'm not convinced they are actually 
> necessary, since `mock.sentinel` objects with the same name compare as 
> equal. Thanks to Nathaniel, I now understand that JS has global symbols 
> for historical reasons that we don't have, and I'm not convinced of 
> their usefulness.

Do all Python distributions ship with unittest.mock? I see to recall 
that Debian and/or Ubuntu strips out part of the normal distribution.

For example, dataclasses.py has a sentinel, and it includes some code to 
get a more helpful repr. It would make sense to re-use the 
unittest.mock.sentinel code, but not if that code isn't always 
guaranteed to be present.

Eric

From steve at pearwood.info  Fri Jul  6 13:31:24 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 7 Jul 2018 03:31:24 +1000
Subject: [Python-ideas] Add new `Symbol` type
In-Reply-To: <CAPJVwBnK-eLvTkOzuXEXu=xH_UJZuvWXh2HOopyiJPXMJk6ugQ@mail.gmail.com>
References: <CAAjMAJWeXOofJtz=OsU8OWq3QTZVsxEKfbmBeDivQrD=-pNZTA@mail.gmail.com>
 <15c549d4-bdf1-69e0-2d11-58a38afe3d4b@kellett.im>
 <CAAjMAJVBTPtKbbajTWLeDYLCW24QNRq18VGGL=O28A=_sn0w1w@mail.gmail.com>
 <CAAjMAJX1AcNNhZGSag-XPbZD3-d8jESPQw6Y71B4QqmOvXajxQ@mail.gmail.com>
 <CAAjMAJVCJsZgqFnOr_wZMyDfz6CGoh7ExRZyLQjq2OV6DB=GNA@mail.gmail.com>
 <CAPJVwBnK-eLvTkOzuXEXu=xH_UJZuvWXh2HOopyiJPXMJk6ugQ@mail.gmail.com>
Message-ID: <20180706173124.GI7318@ando.pearwood.info>

On Thu, Jul 05, 2018 at 05:26:36PM -0700, Nathaniel Smith wrote:

> I think the name "symbol" here is pretty confusing. It comes
> originally from Lisp, where it's used to refer to an interned-string
> data type.

Even if Lisp was the first programming language to use the term for a 
data type (which seems likely), that doesn't give Lisp ownership of the 
concept. A number of languages have a Symbol data type, without needing 
to ape Lisp semantics or implementation.

https://en.wikipedia.org/wiki/Symbol_%28programming%29

See for example:

https://planspace.org/20141129-ruby_symbols_are_not_lisp_symbols/

Since "symbol" is a plain English word that doesn't necessarily have 
anything to do with Lisp, we're free to use it provided we determine 
what it will mean in Python. That may not be identical to Lisp's 
semantics, or Javascript semantics, or Ruby semantics. But that's okay, 
they're semantics aren't identical with each other either.


> It's a common source of confusion even there. Then it
> sounds like JS took that name, and it ended up drifting to mean
> something that's almost exactly the opposite of a Lisp symbol. In
> Lisp, symbols are always "global";

I don't think that's right, at least not in Common Lisp. Symbols may or 
may not be accessible from a particular namespace:

http://www.lispworks.com/documentation/HyperSpec/Body/t_symbol.htm#symbol

In Ruby, symbols are global, the opposite of Lisp symbols:

http://chneukirchen.org/blog/archive/2005/12/ruby-symbols-are-not-lisp-symbols.html


> the whole point is that if two
> different pieces of code use the same name for the same symbol then
> they end up with the same object. So this is *super* confusing.

Well, it's certainly super confusing to me *wink* since I'm not sure 
whether your "whole point" comment is referring to how symbols work in 
Javascript, Lisp, how you think they ought to work, or something else, 
or who "this" is super confusing to. (Javascript programmers? Lisp 
programmers? You?)


[...]
> The thing you're talking about is what Python devs call a "sentinel"
> object.

"Sentinel" describes the *purpose* of the object, not the "kind of 
thing" it is. It is a special value used to indicate the end of data.

https://en.wikipedia.org/wiki/Sentinel_value

By extension, any placeholder value indicating missing data can be 
called a sentinel:

    def function(value=None):

Here, None is being used as a sentinel in the second sense.


> If your proposal is to add a sentinel type to the stdlib, then
> your chance of success will be *much* higher if you use the word
> "sentinel" instead of "symbol". 

Oh I really, really hope not. Symbols aren't merely sentinels. They're 
also a kind of enumerated value: C enums are a kind of symbol. None 
and NotImplemented and Ellipsis are kinds of symbols. Symbols can be 
used for all sorts of reasons, not just as a sentinel.

For example, NotImplemented is returned by operator dunder methods to 
indicate "give the other operand a chance to respond".

None is not a sentinel, but it can be used as a sentinel. But for that 
matter, so could -1 or "Aardvark" be used as sentinels, under the right 
circumstances.

Arguably, True and False could be considered symbols. Sometimes I need a 
three-state flag, and I often use True, False and None, but often what 
I really want is True, False and Maybe. Or True, False, Mu.

https://en.wikipedia.org/wiki/Mu_(negative)#In_popular_culture


If we had Symbols first, we possibly wouldn't need enum.Enum; arguably, 
since we do have enum.Enum, perhaps we don't need Symbols. Or maybe 
there's still use for them both -- any potential PEP would have to sort 
out that question.


> People don't read mailing list threads
> carefully, so if you keep calling it "symbol" then you'll likely spend
> infinite time responding to people rushing in to critique your
> proposal based on some misconception about what you're trying to do,
> which is no fun at all.

As opposed to them critiquing the proposal based on the misuse of the 
term "sentinel" for something which isn't (just) used as a sentinel.



-- 
Steve

From leewangzhong+python at gmail.com  Fri Jul  6 15:26:41 2018
From: leewangzhong+python at gmail.com (Franklin? Lee)
Date: Fri, 6 Jul 2018 15:26:41 -0400
Subject: [Python-ideas] Fwd: grouping / dict of lists
In-Reply-To: <CALGmxE+m+1YXk1vRsdkAK6yQ12V=syi6WKZsSTMFmw1BoiKUYA@mail.gmail.com>
References: <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAOmnVUBp9vJjTp6v17LHy2C=7GzR63fb4ARZW2Dco022uTDpfA@mail.gmail.com>
 <20180703153338.GG14437@ando.pearwood.info>
 <CALGmxEJBYFEoPOFo7aVSN=Xy=sgSMnrF0k8pnt=VhJ6cvhS_4Q@mail.gmail.com>
 <20180704013202.GK14437@ando.pearwood.info>
 <CAEbHw4YafoZc_1cnB6a-psmWqW-SD2tw3E2Fw5hUHSJxqEsu-Q@mail.gmail.com>
 <CAEbHw4Yjue-V_CKQJe2k6f4E8ucNuG=ZNFt5O_njfrQoNKHeUw@mail.gmail.com>
 <CALGmxE+m+1YXk1vRsdkAK6yQ12V=syi6WKZsSTMFmw1BoiKUYA@mail.gmail.com>
Message-ID: <CAB_e7izyUbkOXxpwivr0iyE4gXk58xn+SBfAZq5hcKAyDO-H_g@mail.gmail.com>

On Thu, Jul 5, 2018 at 1:23 AM, Chris Barker via Python-ideas
<python-ideas at python.org> wrote:
> On Wed, Jul 4, 2018 at 6:34 AM, David Mertz <mertz at gnosis.cx> wrote:
>
>>
>> You've misunderstood part of the discussion. There are two different
>> signatures being discussed/proposed for a grouping() function.
>>
>> The one you show we might call grouping_michael(). The alternate API we
>> might call grouping_chris(). These two calls will produce the same result
>> (the first output you show)
>>
>>   grouping_michael(words, keyfunc=len)
>>   grouping_chris((len(word), word) for word in words)
>>
>> I happen to prefer grouping_michael(), but recognize they each make
>> slightly different things obvious.
>
>
> I starting thinking grouping_chris was the obvious and natural thing to do,
> but his discussion has made it clear that grouping_michael is more natural
> for some kinds of data.
>
> and in some cases, it really comes down to taste, after all, who's to say
> which of these is "better"
>
> map(func, iterable)
>
> or
>
> (expression for item in iterable)
>
> given that map existed in Python when comprehensions were added, I tend to
> see the latter as more "Pythonic" but that's just me.
>
>
> So I'm currently lobbying for both :-)
>
> The default is iterable of (key. value) pairs, but the use can specify a key
> function is they want to do it that way.
>
> While a bit of a schizophrenic  API, it makes sens (to me), because
> grouping_mikael isn't useful with a default key function anyway.
>
> The other enhancement I suggest is that an (optional) value function be
> added, as there are use cases where that would be really helpful.

I use this kind of function in several different projects over the
years, and I rewrote it many times as needed.

I added several options, such as:
- key function
- value function
- "ignore": Skip values with these keys.
- "postprocess": Apply a function to each group after completion.
- Pass in the container to store in. For example, create an
OrderedDict and pass it in. It may already hold items.
- Specify the container for each group.
- Specify how to add to the container for each group.

Then I cut it down to two optional parameters:
- key function. If not provided, the iterable is considered to have
key-value pairs.
- The storage container.

Finally, I removed the key function, and only took pairs and an
optional container. However, I don't remember why I removed the key
function. It may be that I was writing throwaway lambdas, and I
decided I might as well just write the transformation into the
comprehension. I think a key function is worth having.

One thing I didn't do is create a class for this container. I used
defaultdict, then used a default dict but converted it to a plain
dict, and finally used to a plain dict.

Aside: I also wrote a lazy grouping function, which returned a lazy
container of generators. Asking for next(grouping[k]) will cause the
container to iterate through the original iterable until it had
something to add to group k. I have never used it, but it was fun to
try and make it.

    class IterBuckets(dict):
        def __init__(self, pairs):
            self.pairs = iter(pairs)
        def __missing__(self, key):
            return self.setdefault(key, IterBucket(self.advance))
        def advance(self):
            k, v = next(self.pairs)
            self[k].append(v)

    class IterBucket(collections.deque):
        def __init__(self, more):
            self.more = more
        def __iter__(self):
            return self
        def __next__(self):
            while not self:
                self.more()
            return self.popleft()

From donald at stufft.io  Fri Jul  6 16:02:20 2018
From: donald at stufft.io (Donald Stufft)
Date: Fri, 6 Jul 2018 16:02:20 -0400
Subject: [Python-ideas] Add `rc` to distutils.version.StrictVersion
In-Reply-To: <CAPL7A84_cUkG3DW270W8Y8NYELub5ZjszcgLuW4MEW=N+3xj7w@mail.gmail.com>
References: <CAPL7A84_cUkG3DW270W8Y8NYELub5ZjszcgLuW4MEW=N+3xj7w@mail.gmail.com>
Message-ID: <447B9D9A-F5CF-4D3C-A884-C7FC9A2AC5FD@stufft.io>

You should https://packaging.pypa.io/en/latest/version/ <https://packaging.pypa.io/en/latest/version/> instead of anything in distutils for handling version numbers.

> On Jul 5, 2018, at 2:11 PM, P?l Gr?n?s Drange <paal.drange at gmail.com> wrote:
> 
> StrictVersion from distutils accepts version tags like
> 1.14.0
> 1.14.0a1
> 1.14.0b2
> 
> but not
> 1.14.0rc1 (nor 1.14.0c1).
> 
> My suggestion:  Add `rc` in the regexp and make it a `prerelease` (the
> latter comes for free by the current implementation).
> 
> Most package maintainers have adopted the `rc` abbreviation for release
> candidate versioning, e.g.
> - numpy 1.14.0rc1
> - scipy 1.1.0rc1
> - plotly 3.0.0rc1
> - pandas 0.23.0rc1
> - matplotlib 2.2.0rc1
> - dask 0.13.0rc1
> - django 1.9rc1.
> All of these are available on PyPI.
> 
> A natural way of sorting version numbers from pip is by simply using
>     sorted(versions, key=distutils.version.StrictVersion),
> however, due to StrictVersion only accepting `a` and `b` as
> abbreviations, this does not work for the aforemention packages.
> 
> The very obvious cons are:
> - touching 20 years old code [1]
> - StrictVersion is preserved "for anal retentives and software
>   idealists", and I don't know if they agree.
> 
> There might be more cons I fail to think of at this moment.
> 
> [1] https://github.com/python/cpython/blob/master/Lib/distutils/version.py#L130 <https://github.com/python/cpython/blob/master/Lib/distutils/version.py#L130>
> 
> P?l Gr?n?s Drange
> 
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180706/1584f841/attachment-0001.html>

From mike at selik.org  Fri Jul  6 20:13:54 2018
From: mike at selik.org (Michael Selik)
Date: Fri, 6 Jul 2018 17:13:54 -0700
Subject: [Python-ideas] Where should grouping() live
In-Reply-To: <CALGmxEKHPcqY2rTJvz0T2nXv2PeL7Vu7URVkvz_QemyVDRDJDw@mail.gmail.com>
References: <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
 <20180703152413.GF14437@ando.pearwood.info>
 <5B3BFC41.8040207@canterbury.ac.nz>
 <20180704002102.GJ14437@ando.pearwood.info>
 <CALGmxEKHPcqY2rTJvz0T2nXv2PeL7Vu7URVkvz_QemyVDRDJDw@mail.gmail.com>
Message-ID: <CADqi797k215Lm-pk9kgao44bBHG4NUhj7o+cuQ0XmAO+cWhd4A@mail.gmail.com>

On Tue, Jul 3, 2018 at 10:11 PM Chris Barker via Python-ideas <
python-ideas at python.org> wrote:

> * There are types of data well suited to the key function approach, and
> other data not so well suited to it. If you want to support the not as well
> suited use cases, you should have a value function as well and/or take a
> (key, value) pair.
>
> * There are some nice advantages in flexibility to having a Grouping
> class, rather than simply a function.
>

The tri-grams example is interesting and shows some clever things you can
do. The bi-grams example I wrote in my draft PEP could be extended to
handle tri-grams with just a key-function, no value-function. However,
because this example is fun it may be distracting from the core value of
``grouped`` or ``grouping``.

I don't think we need a nicer API for complex grouping tasks. As the tasks
get increasingly sophisticated, any general-purpose API will be less nice
than something built for that specific task.

Instead, I want the easiest possible interface for making groups for
every-day use cases. The wide range of situations that ``sorted`` covers
with just a key-function suggests that ``grouped`` should follow the same
pattern.

I do think that the default, key=None, could be set to handle (key, value)
pairs. But I'm still reluctant to break the standard of sorted, min, max,
and groupby.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180706/ce0e6c85/attachment.html>

From agarciaillera at gmail.com  Sun Jul  8 04:01:32 2018
From: agarciaillera at gmail.com (Alberto Garcia)
Date: Sun, 8 Jul 2018 01:01:32 -0700
Subject: [Python-ideas] Calling python from C completely statically
Message-ID: <CA+fJkbn5r_PVwJtS_-aZ9qGp0mqY2BytFAmQkPLLkU+ZvryfyA@mail.gmail.com>

 Hi,

I've been working for a while on having the entire python interpreter with
all his modules statically linked in a binary that can be called with
arbitrary argument which will be passed to the python interpreter.

I've been able to statically compile Python in a single binary with no
dependencies for Python2.7 and using using this code to call the
interpreter with arbitrary code:

#define Py_NO_ENABLE_SHARED
#include <Python.h>

int main(int argc, char** argv)
{
Py_NoSiteFlag = 1;
Py_InitializeEx(0);
PyRun_SimpleString(argv[1]);
Py_Finalize();
}

This program compiles and does not rely in any dependency but the "non
frozen" (.py) modules are not loaded so when I do a simple `import random`
I get:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named random

I've created a bug issue (https://bugs.python.org/issue34057) speaking
about it and Nick Colghlan pointed:

" cx_freeze is an illustrative example to look at in that regard, as it
preconfigures the interpreter to be able to find the cx_freeze generated
zip archive that has the program's Python modules in it:
https://github.com/anthony-tuininga/cx_Freeze/blob/master/source/bases/Common.c

The technique that cx_freeze doesn't use yet is to combine the statically
linked Python binary and the generated zip archive into a single file
(similar to what zipapp does), and adjust the sys.path definition inside
the binary to refer back to the executable itself (since executable files
can have arbitrary content appended, while zip files can have arbitrary
content *pre*pended). "

As I understand, cx_freeze creates a zip file with the dependencies for a
specific python code and creates a zip with them. My question is, could I
create a zip file with every standard module (plus some extra that I may
install with pip) and using that zip file from C? If so, how can I do that?

I'm happy to work on this if I get indications.

BTW: I do not want to convert a python code snippet to exe or anything like
that but calling python from C in an absolutely standalone version.

Thank you


-- 
Alberto Garc?a Illera

GPG Public Key: https://goo.gl/twKUUv
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180708/9ce4d5cf/attachment.html>

From p.f.moore at gmail.com  Sun Jul  8 07:34:02 2018
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 8 Jul 2018 12:34:02 +0100
Subject: [Python-ideas] Calling python from C completely statically
In-Reply-To: <CA+fJkbn5r_PVwJtS_-aZ9qGp0mqY2BytFAmQkPLLkU+ZvryfyA@mail.gmail.com>
References: <CA+fJkbn5r_PVwJtS_-aZ9qGp0mqY2BytFAmQkPLLkU+ZvryfyA@mail.gmail.com>
Message-ID: <CACac1F-J+bctMVEa7CcTn46JOpWXY7RrxEH_Lzt0U0YZ8phzuw@mail.gmail.com>

On 8 July 2018 at 09:01, Alberto Garcia <agarciaillera at gmail.com> wrote:
> Hi,
>
> I've been working for a while on having the entire python interpreter with
> all his modules statically linked in a binary that can be called with
> arbitrary argument which will be passed to the python interpreter.
>
> I've been able to statically compile Python in a single binary with no
> dependencies for Python2.7 and using using this code to call the interpreter
> with arbitrary code:
>
> #define Py_NO_ENABLE_SHARED
> #include <Python.h>
>
> int main(int argc, char** argv)
> {
> Py_NoSiteFlag = 1;
> Py_InitializeEx(0);
> PyRun_SimpleString(argv[1]);
> Py_Finalize();
> }
>
> This program compiles and does not rely in any dependency but the "non
> frozen" (.py) modules are not loaded so when I do a simple `import random` I
> get:
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
> ImportError: No module named random
>
> I've created a bug issue (https://bugs.python.org/issue34057) speaking about
> it and Nick Colghlan pointed:
>
> " cx_freeze is an illustrative example to look at in that regard, as it
> preconfigures the interpreter to be able to find the cx_freeze generated zip
> archive that has the program's Python modules in it:
> https://github.com/anthony-tuininga/cx_Freeze/blob/master/source/bases/Common.c
>
> The technique that cx_freeze doesn't use yet is to combine the statically
> linked Python binary and the generated zip archive into a single file
> (similar to what zipapp does), and adjust the sys.path definition inside the
> binary to refer back to the executable itself (since executable files can
> have arbitrary content appended, while zip files can have arbitrary content
> *pre*pended). "
>
> As I understand, cx_freeze creates a zip file with the dependencies for a
> specific python code and creates a zip with them. My question is, could I
> create a zip file with every standard module (plus some extra that I may
> install with pip) and using that zip file from C? If so, how can I do that?
>
> I'm happy to work on this if I get indications.
>
> BTW: I do not want to convert a python code snippet to exe or anything like
> that but calling python from C in an absolutely standalone version.

This question is probably more appropriate for python-list, but yes,
you certainly can do this. The "Embedded" distributions of Python for
Windows essentially do this already. IIRC, they are only available for
Python 3.x, so you may find you have some hurdles to overcome if you
have to remain on Python 2.7, but in principle it's possible.

One further point you may need to consider - a proportion of the
standard library is in the form of shared C extensions (PYDs in
Windows - I don't know if you're using Windows or Unix from what you
say above). You can't (without significant extra work) load a binary
extension direct from a zip file, so you'll need to either do that
extra work (which is platform specific and fragile, I believe) or be
prepared to ship supporting DLLs alongside your application (this is
the approach the embedded distribution takes).

Hope this helps,
Paul

From ncoghlan at gmail.com  Sun Jul  8 10:07:34 2018
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 9 Jul 2018 00:07:34 +1000
Subject: [Python-ideas] Calling python from C completely statically
In-Reply-To: <CACac1F-J+bctMVEa7CcTn46JOpWXY7RrxEH_Lzt0U0YZ8phzuw@mail.gmail.com>
References: <CA+fJkbn5r_PVwJtS_-aZ9qGp0mqY2BytFAmQkPLLkU+ZvryfyA@mail.gmail.com>
 <CACac1F-J+bctMVEa7CcTn46JOpWXY7RrxEH_Lzt0U0YZ8phzuw@mail.gmail.com>
Message-ID: <CADiSq7cbrZJiLtgw4Y_2e_U4LjuJ9J67hiorA=JB-0rFMjSYEg@mail.gmail.com>

On 8 July 2018 at 21:34, Paul Moore <p.f.moore at gmail.com> wrote:
> This question is probably more appropriate for python-list, but yes,
> you certainly can do this. The "Embedded" distributions of Python for
> Windows essentially do this already. IIRC, they are only available for
> Python 3.x, so you may find you have some hurdles to overcome if you
> have to remain on Python 2.7, but in principle it's possible.
>
> One further point you may need to consider - a proportion of the
> standard library is in the form of shared C extensions (PYDs in
> Windows - I don't know if you're using Windows or Unix from what you
> say above). You can't (without significant extra work) load a binary
> extension direct from a zip file, so you'll need to either do that
> extra work (which is platform specific and fragile, I believe) or be
> prepared to ship supporting DLLs alongside your application (this is
> the approach the embedded distribution takes).

That's the part of the problem that Alberto's static linking solves -
all of the standard library's extension modules are top level ones (at
least as far as I am aware), so we support building them as statically
linked builtin modules instead (we just don't do it routinely, because
we don't have any great desire to make the main executable even larger
than it already is).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From agarciaillera at gmail.com  Sun Jul  8 13:10:38 2018
From: agarciaillera at gmail.com (Alberto Garcia)
Date: Sun, 8 Jul 2018 10:10:38 -0700
Subject: [Python-ideas] Calling python from C completely statically
In-Reply-To: <CADiSq7cbrZJiLtgw4Y_2e_U4LjuJ9J67hiorA=JB-0rFMjSYEg@mail.gmail.com>
References: <CA+fJkbn5r_PVwJtS_-aZ9qGp0mqY2BytFAmQkPLLkU+ZvryfyA@mail.gmail.com>
 <CACac1F-J+bctMVEa7CcTn46JOpWXY7RrxEH_Lzt0U0YZ8phzuw@mail.gmail.com>
 <CADiSq7cbrZJiLtgw4Y_2e_U4LjuJ9J67hiorA=JB-0rFMjSYEg@mail.gmail.com>
Message-ID: <CA+fJkbmC1A+R2ggCMa3h6y8eEg=i1-zy6yxBnecNNHqZdT5Z2w@mail.gmail.com>

Hey there,

Yes, the part of having the pyd modules built in in library is already
done. I followed the instructions in the README. What I would like to know
now is how to embed the non frozen python (py) modules. Can you guys please
point me in the right direction.

Thank you

On Sun, Jul 8, 2018 at 7:07 AM Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 8 July 2018 at 21:34, Paul Moore <p.f.moore at gmail.com> wrote:
> > This question is probably more appropriate for python-list, but yes,
> > you certainly can do this. The "Embedded" distributions of Python for
> > Windows essentially do this already. IIRC, they are only available for
> > Python 3.x, so you may find you have some hurdles to overcome if you
> > have to remain on Python 2.7, but in principle it's possible.
> >
> > One further point you may need to consider - a proportion of the
> > standard library is in the form of shared C extensions (PYDs in
> > Windows - I don't know if you're using Windows or Unix from what you
> > say above). You can't (without significant extra work) load a binary
> > extension direct from a zip file, so you'll need to either do that
> > extra work (which is platform specific and fragile, I believe) or be
> > prepared to ship supporting DLLs alongside your application (this is
> > the approach the embedded distribution takes).
>
> That's the part of the problem that Alberto's static linking solves -
> all of the standard library's extension modules are top level ones (at
> least as far as I am aware), so we support building them as statically
> linked builtin modules instead (we just don't do it routinely, because
> we don't have any great desire to make the main executable even larger
> than it already is).
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>


-- 
Alberto Garc?a Illera

GPG Public Key: https://goo.gl/twKUUv
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180708/360f4c29/attachment.html>

From davidfstr at gmail.com  Sun Jul  8 14:27:08 2018
From: davidfstr at gmail.com (David Foster)
Date: Sun, 8 Jul 2018 11:27:08 -0700
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
Message-ID: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>

In the past I have personally viewed Python as difficult to use for 
parallel applications, which need to do multiple things simultaneously 
for increased performance:

* The old Threads, Locks, & Shared State model is inefficient in Python 
due to the GIL, which limits CPU usage to only one thread at a time 
(ignoring certain functions implemented in C, such as I/O).

* The Actor model can be used with some effort via the ?multiprocessing? 
module, but it doesn?t seem that streamlined and forces there to be a 
separate OS process per line of execution, which is relatively expensive.

I was thinking it would be nice if there was a better way to implement 
the Actor model, with multiple lines of execution in the same process, 
yet avoiding contention from the GIL. This implies a separate GIL for 
each line of execution (to eliminate contention) and a controlled way to 
exchange data between different lines of execution.

So I was thinking of proposing a design for implementing such a system. 
Or at least get interested parties thinking about such a system.

With some additional research I notice that [PEP 554] (?Multiple 
subinterpeters in the stdlib?) appears to be putting forward a design 
similar to the one I described. I notice however it mentions that 
subinterpreters currently share the GIL, which would seem to make them 
unusable for parallel scenarios due to GIL contention.

I'd like to solicit some feedback on what might be the most efficient 
way to make forward progress on efficient parallelization in Python 
inside the same OS process. The most promising areas appear to be:

1. Make the current subinterpreter implementation in Python have more 
complete isolation, sharing almost no state between subinterpreters. In 
particular not sharing the GIL. The "Interpreter Isolation" section of 
PEP 554 enumerates areas that are currently shared, some of which 
probably shouldn't be.

2. Give up on making things work inside the same OS process and rather 
focus on implementing better abstractions on top of the existing 
multiprocessing API so that the actor model is easier to program 
against. For example, providing some notion of Channels to communicate 
between lines of execution, a way to monitor the number of Messages 
waiting in each channel for throughput profiling and diagnostics, 
Supervision, etc. In particular I could do this by using an existing 
library like Pykka or Thespian and extending it where necessary.

Thoughts?

[PEP 554]: https://www.python.org/dev/peps/pep-0554/

-- 
David Foster | Seattle, WA, USA

From ncoghlan at gmail.com  Mon Jul  9 10:16:11 2018
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 10 Jul 2018 00:16:11 +1000
Subject: [Python-ideas] Calling python from C completely statically
In-Reply-To: <CA+fJkbmC1A+R2ggCMa3h6y8eEg=i1-zy6yxBnecNNHqZdT5Z2w@mail.gmail.com>
References: <CA+fJkbn5r_PVwJtS_-aZ9qGp0mqY2BytFAmQkPLLkU+ZvryfyA@mail.gmail.com>
 <CACac1F-J+bctMVEa7CcTn46JOpWXY7RrxEH_Lzt0U0YZ8phzuw@mail.gmail.com>
 <CADiSq7cbrZJiLtgw4Y_2e_U4LjuJ9J67hiorA=JB-0rFMjSYEg@mail.gmail.com>
 <CA+fJkbmC1A+R2ggCMa3h6y8eEg=i1-zy6yxBnecNNHqZdT5Z2w@mail.gmail.com>
Message-ID: <CADiSq7dGY+F7XZreNd3_VDSBmv7+GQgoFbdTbBmMbKtUDXBTdw@mail.gmail.com>

On 9 July 2018 at 03:10, Alberto Garcia <agarciaillera at gmail.com> wrote:
> Hey there,
>
> Yes, the part of having the pyd modules built in in library is already done.
> I followed the instructions in the README. What I would like to know now is
> how to embed the non frozen python (py) modules. Can you guys please point
> me in the right direction.

The gist is to:

1. take the entire Lib directory and put it in a zip archive
2. use the approach demonstrated in cx_freeze to point sys.path in
your static executable at that zip archive
3. adjust your C code to point sys.path back at the executable itself,
and then combine your executable and the zip archive into a single
contiguous file (similar to what zipapp does with it's helper script
and app archive)

There are likely to still be rough edges when doing that, since this
isn't a well tested configuration. When all else fails, find the part
of the source code responsible for any error messages you're seeing,
and try to work out if there's a setting you can tweak to avoid
hitting that code path.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Mon Jul  9 10:31:29 2018
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 10 Jul 2018 00:31:29 +1000
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
Message-ID: <CADiSq7ftHJ7QY-cy5AQ+ccexdAXqr1TA-jsYbOXvB1WgV5Ou4Q@mail.gmail.com>

On 9 July 2018 at 04:27, David Foster <davidfstr at gmail.com> wrote:
> I'd like to solicit some feedback on what might be the most efficient way to
> make forward progress on efficient parallelization in Python inside the same
> OS process. The most promising areas appear to be:
>
> 1. Make the current subinterpreter implementation in Python have more
> complete isolation, sharing almost no state between subinterpreters. In
> particular not sharing the GIL. The "Interpreter Isolation" section of PEP
> 554 enumerates areas that are currently shared, some of which probably
> shouldn't be.
>
> 2. Give up on making things work inside the same OS process and rather focus
> on implementing better abstractions on top of the existing multiprocessing
> API so that the actor model is easier to program against. For example,
> providing some notion of Channels to communicate between lines of execution,
> a way to monitor the number of Messages waiting in each channel for
> throughput profiling and diagnostics, Supervision, etc. In particular I
> could do this by using an existing library like Pykka or Thespian and
> extending it where necessary.

Yep, that's basically the way Eric and I and a few others have been
thinking. Eric started off this year's language summit with a
presentation on the topic: https://lwn.net/Articles/754162/

The intent behind PEP 554 is to eventually get to a point where each
subinterpreter has its own dedicated eval loop lock, and the GIL
either disappears entirely (replaced by smaller purpose specific
locks) or becomes a read/write lock (where write access is only needed
to adjust certain state that is shared across subinterpreters).

On the multiprocessing front, it could be quite interesting to attempt
to adapt the channel API from PEP 554 to the
https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing.sharedctypes
data sharing capabilities in the modern multiprocessing module.

Also of relevance is Antoine Pitrou's work on a new version of the
pickle protocol that allows for out-of-band data sharing to avoid
redundant memory copies: https://www.python.org/dev/peps/pep-0574/

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From chris.barker at noaa.gov  Mon Jul  9 12:13:56 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Mon, 9 Jul 2018 09:13:56 -0700
Subject: [Python-ideas] Where should grouping() live
In-Reply-To: <CADqi797k215Lm-pk9kgao44bBHG4NUhj7o+cuQ0XmAO+cWhd4A@mail.gmail.com>
References: <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
 <20180703152413.GF14437@ando.pearwood.info>
 <5B3BFC41.8040207@canterbury.ac.nz>
 <20180704002102.GJ14437@ando.pearwood.info>
 <CALGmxEKHPcqY2rTJvz0T2nXv2PeL7Vu7URVkvz_QemyVDRDJDw@mail.gmail.com>
 <CADqi797k215Lm-pk9kgao44bBHG4NUhj7o+cuQ0XmAO+cWhd4A@mail.gmail.com>
Message-ID: <CALGmxEKq59Eow1jjeWFZjU+y+vdCfDcST4qD8W4aq8=4gpRa3A@mail.gmail.com>

On Fri, Jul 6, 2018 at 5:13 PM, Michael Selik <mike at selik.org> wrote:

> On Tue, Jul 3, 2018 at 10:11 PM Chris Barker via Python-ideas <
> python-ideas at python.org> wrote:
>
>> * There are types of data well suited to the key function approach, and
>> other data not so well suited to it. If you want to support the not as well
>> suited use cases, you should have a value function as well and/or take a
>> (key, value) pair.
>>
>> * There are some nice advantages in flexibility to having a Grouping
>> class, rather than simply a function.
>>
>
> The tri-grams example is interesting and shows some clever things you can
> do. The bi-grams example I wrote in my draft PEP could be extended to
> handle tri-grams with just a key-function, no value-function.
>

hmm, I'll take a look -- 'cause I found that I was really limited to only a
certain class of problems without a way to get "custom" values.

Do you mean the "foods" example?

>>> foods = [
...     ('fruit', 'apple'),
...     ('vegetable', 'broccoli'),
...     ('fruit', 'clementine'),
...     ('vegetable', 'daikon')
... ]
>>> groups = grouping(foods, key=lambda pair: pair[0])
>>> {k: [v for _, v in g] for k, g in groups.items()}
{'fruit': ['apple', 'clementine'], 'vegetable': ['broccoli', 'daikon']}


Because that one, I think, makes my point well. To get what you want, you
have to post-processthe Grouping with a (somewhat complex) comprehension.
If someone is that adept with comprehensions, and want to do it that way,
the grouping function isn't really buying them much at all, over
setdefault, or defaultdict, or roll your own.

Contrast this with:

groups = grouping(foods,
                  key=lambda pair: pair[0],
                  value=lambda pair: pair[1])

and you're done.

or:

groups = grouping(foods,
                  key=itemgetter(0),
                  value=itemgetter0))


Or even better:

groups = grouping(foods)

:-)

However, because this example is fun it may be distracting from the core
> value of ``grouped`` or ``grouping``.
>

Actually, I think it's the opposite -- it opens up the concept to be more
general purpose -- I guess I'm thinking of this a "dict with lists as the
values" that has many purposes beyond strictly "groupby". Maybe that's
because I'm a general python programmer, and not a database guy, but if
something is going to be added to the stdlib, why not add a more general
purpose class?


> I don't think we need a nicer API for complex grouping tasks. As the tasks
> get increasingly sophisticated, any general-purpose API will be less nice
> than something built for that specific task.
>

I guess this is where we disagree -- I think we've found an API that is
general purpose, and cleanly supports multiple tasks.

Instead, I want the easiest possible interface for making groups for
> every-day use cases. The wide range of situations that ``sorted`` covers
> with just a key-function suggests that ``grouped`` should follow the same
> pattern.
>

not at all -- sorted() is about, well, sorting -- which means rearranging
items. I certainly don't expect it to break up the items for me.

Again, this is a matter of perspective -- if you you start with "groupby"
as a concept, then I can see how you see the parallel with sorted -- you
are rearranging the items, but this time into groups.

But if you start with "a dict of lists", then you take a wider perspective:

- It can naturally an easily be used to group things
- It can do another nifty things
- And as a "dict of something", it's natural to think of keys AND values,
and to want a dict-like API -- i.e. pass in (key, value) pairs.

I do think that the default, key=None, could be set to handle (key, value)
> pairs.
>

OK, so for my part, if you provide the (key, value) pair API, then you
don't really need a value_func. But as the "pass in a function to process
the data" model IS well suited to some tasks, and some people simply like
the style, why not?

And it creates an asymetry: or you have a (key, the_item) problem, you can
use either the key function API or the (key, value) API -- but if you have
a (key, value) problem, you can only use the (key, value) API

But I'm still reluctant to break the standard of sorted, min, max, and
> groupby.
>

This is the power of Python's keyword parameters -- anyone coming to this
from a perspective of "I expect this to be like sorted, min, max, and
groupby" can simply ignore the value parameter :-)

One more argument :-)

There have been comments a bout how maybe some of the classes in
collections are maybe not needed -- Counter, in particular. I tend to
agree, but i think the reason Counter is not-that-useful is because it
doesn't do enough -- not that it isn't useful -- it's just such a thin
wrapper around a dict, that I hardly see the point.

Example:

In [12]: c = Counter()

In [13]: c['f'] += 1

In [14]: c['g'] = "some random thing"

In [15]: c
Out[15]: Counter({'f': 1, 'g': 'some random thing'})

Is that really that useful? I need to do the counting by hand, and can
easily use the regular dict interface to make a mess of it.

it has a handy constructor, but that's about it.

Anyway, I think we've got this nailed down to a handful of options /
decisions

1) a dict subclass vs a function that constructs a dict-of-lists

 - I think a dict subclass offers some real value -- but it comes down a
bit to goals: Do we want a general purpose special dict? or a function to
perform the "usual" groupby operation?

2) Do we have a value function keyword parameter?

  - I think this adds real value without taking anything away from the
convenience of the simpler key only API

3) Do we accept an iterable of (key, value) pairs if no key function is
provided?

  - I think yes, also because why not? a default of the identity function
for key and value is pretty useless.

So it comes down to what the community thinks.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180709/76f6ce94/attachment.html>

From trent at snakebite.org  Mon Jul  9 12:17:55 2018
From: trent at snakebite.org (Trent Nelson)
Date: Mon, 9 Jul 2018 12:17:55 -0400
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
Message-ID: <20180709161754.GA6167@trent.me>

On Sun, Jul 08, 2018 at 11:27:08AM -0700, David Foster wrote:

> I'd like to solicit some feedback on what might be the most
> efficient way to make forward progress on efficient parallelization
> in Python inside the same OS process. The most promising areas
> appear to be:

You might find PyParallel interesting, at least from a "here's what was
tried, it worked, but we're not doing it like that" perspective.

    http://pyparallel.org
    https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores

I still think it was a pretty successful proof-of-concept regarding
removing the GIL without having to actually remove it.  Performance was
pretty good too, as you can see in those graphs.

> -- 
> David Foster | Seattle, WA, USA

Regards,

    Trent.

--
https://trent.me

From chris.barker at noaa.gov  Mon Jul  9 12:22:03 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Mon, 9 Jul 2018 09:22:03 -0700
Subject: [Python-ideas] Fwd: grouping / dict of lists
In-Reply-To: <CAB_e7izyUbkOXxpwivr0iyE4gXk58xn+SBfAZq5hcKAyDO-H_g@mail.gmail.com>
References: <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAOmnVUBp9vJjTp6v17LHy2C=7GzR63fb4ARZW2Dco022uTDpfA@mail.gmail.com>
 <20180703153338.GG14437@ando.pearwood.info>
 <CALGmxEJBYFEoPOFo7aVSN=Xy=sgSMnrF0k8pnt=VhJ6cvhS_4Q@mail.gmail.com>
 <20180704013202.GK14437@ando.pearwood.info>
 <CAEbHw4YafoZc_1cnB6a-psmWqW-SD2tw3E2Fw5hUHSJxqEsu-Q@mail.gmail.com>
 <CAEbHw4Yjue-V_CKQJe2k6f4E8ucNuG=ZNFt5O_njfrQoNKHeUw@mail.gmail.com>
 <CALGmxE+m+1YXk1vRsdkAK6yQ12V=syi6WKZsSTMFmw1BoiKUYA@mail.gmail.com>
 <CAB_e7izyUbkOXxpwivr0iyE4gXk58xn+SBfAZq5hcKAyDO-H_g@mail.gmail.com>
Message-ID: <CALGmxE+OS-EDa7gNceEirnW1S0oryyinA7mMbUetXfZkJ67Zkw@mail.gmail.com>

On Fri, Jul 6, 2018 at 12:26 PM, Franklin? Lee I use this kind of function
in several different projects over the
>
> years, and I rewrote it many times as needed.
>


> I added several options, such as:
> - key function
> - value function
> - "ignore": Skip values with these keys.
> - "postprocess": Apply a function to each group after completion.
> - Pass in the container to store in. For example, create an
> OrderedDict and pass it in. It may already hold items.
> - Specify the container for each group.
> - Specify how to add to the container for each group.
>

interesting...


> Then I cut it down to two optional parameters:
> - key function. If not provided, the iterable is considered to have
> key-value pairs.
>

OK -- seems we're all converging on that :-)


> - The storage container.
>

so this means you'r passing in a full set of storage containers? I'm a vit
confused by that -- if they might be pre-populated, then they would need to
be instance,s an you'd need to have one for every key -- how would you know
in advance aht you needed???

I played around with passing in a optional storage object:

https://github.com/PythonCHB/grouper/commit/d986816905406ec402724beaed2b88c96df64469

but as we might want a list or a set, or a Counter, or ??? it got pretty
ugly, as lists and sets and Counters all have different APIs for adding
stuff. So I gave up and figured just saying "it's always a list) made the
most sense.


> Finally, I removed the key function, and only took pairs and an
> optional container. However, I don't remember why I removed the key
> function. It may be that I was writing throwaway lambdas, and I
> decided I might as well just write the transformation into the
> comprehension.


exactly -- but I suspect hat may be because you where writing a
comprehension anyway, as you needed to manipulate the values, also -- so if
there were a value function, you could use either API.


> I think a key function is worth having.
>

I think there's more or less consensus on that too.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180709/f060e7e2/attachment.html>

From agarciaillera at gmail.com  Mon Jul  9 12:48:33 2018
From: agarciaillera at gmail.com (Alberto Garcia)
Date: Mon, 9 Jul 2018 09:48:33 -0700
Subject: [Python-ideas] Calling python from C completely statically
In-Reply-To: <CADiSq7dGY+F7XZreNd3_VDSBmv7+GQgoFbdTbBmMbKtUDXBTdw@mail.gmail.com>
References: <CA+fJkbn5r_PVwJtS_-aZ9qGp0mqY2BytFAmQkPLLkU+ZvryfyA@mail.gmail.com>
 <CACac1F-J+bctMVEa7CcTn46JOpWXY7RrxEH_Lzt0U0YZ8phzuw@mail.gmail.com>
 <CADiSq7cbrZJiLtgw4Y_2e_U4LjuJ9J67hiorA=JB-0rFMjSYEg@mail.gmail.com>
 <CA+fJkbmC1A+R2ggCMa3h6y8eEg=i1-zy6yxBnecNNHqZdT5Z2w@mail.gmail.com>
 <CADiSq7dGY+F7XZreNd3_VDSBmv7+GQgoFbdTbBmMbKtUDXBTdw@mail.gmail.com>
Message-ID: <CA+fJkb=66-Do3BxtPU6OPgVi7SJF44s_hn8xYpAcYE=O+oqSwQ@mail.gmail.com>

Thank you for your response,

I was thinking on creating that zip file with the content of the Lib folder
and having my c code to download it over the network and have it in memory.
I guess that the zip file should have no compression at all right?

When you say that I need to use the cx_freeze approach what do you mean?
Can you point me to where they do that?

And why changing sys.path again to the executable again? Which part of the
executable?

I'll put my efforts in this.

Thank you

On Mon, Jul 9, 2018 at 7:16 AM Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 9 July 2018 at 03:10, Alberto Garcia <agarciaillera at gmail.com> wrote:
> > Hey there,
> >
> > Yes, the part of having the pyd modules built in in library is already
> done.
> > I followed the instructions in the README. What I would like to know now
> is
> > how to embed the non frozen python (py) modules. Can you guys please
> point
> > me in the right direction.
>
> The gist is to:
>
> 1. take the entire Lib directory and put it in a zip archive
> 2. use the approach demonstrated in cx_freeze to point sys.path in
> your static executable at that zip archive
> 3. adjust your C code to point sys.path back at the executable itself,
> and then combine your executable and the zip archive into a single
> contiguous file (similar to what zipapp does with it's helper script
> and app archive)
>
> There are likely to still be rough edges when doing that, since this
> isn't a well tested configuration. When all else fails, find the part
> of the source code responsible for any error messages you're seeing,
> and try to work out if there's a setting you can tweak to avoid
> hitting that code path.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>
-- 
Alberto Garc?a Illera

GPG Public Key: https://goo.gl/twKUUv
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180709/6e4bde28/attachment.html>

From agarciaillera at gmail.com  Mon Jul  9 12:59:43 2018
From: agarciaillera at gmail.com (Alberto Garcia)
Date: Mon, 9 Jul 2018 09:59:43 -0700
Subject: [Python-ideas] Calling python from C completely statically
In-Reply-To: <CA+fJkb=66-Do3BxtPU6OPgVi7SJF44s_hn8xYpAcYE=O+oqSwQ@mail.gmail.com>
References: <CA+fJkbn5r_PVwJtS_-aZ9qGp0mqY2BytFAmQkPLLkU+ZvryfyA@mail.gmail.com>
 <CACac1F-J+bctMVEa7CcTn46JOpWXY7RrxEH_Lzt0U0YZ8phzuw@mail.gmail.com>
 <CADiSq7cbrZJiLtgw4Y_2e_U4LjuJ9J67hiorA=JB-0rFMjSYEg@mail.gmail.com>
 <CA+fJkbmC1A+R2ggCMa3h6y8eEg=i1-zy6yxBnecNNHqZdT5Z2w@mail.gmail.com>
 <CADiSq7dGY+F7XZreNd3_VDSBmv7+GQgoFbdTbBmMbKtUDXBTdw@mail.gmail.com>
 <CA+fJkb=66-Do3BxtPU6OPgVi7SJF44s_hn8xYpAcYE=O+oqSwQ@mail.gmail.com>
Message-ID: <CA+fJkbkrRAmjvjOaUTsksDJawgv9O7zm2pkdKfiWu4eGFOXW_g@mail.gmail.com>

Ohhhhhhhh  I guess you mean this:
https://github.com/anthony-tuininga/cx_Freeze/blob/master/source/bases/Common.c

Right?

On Mon, Jul 9, 2018 at 9:48 AM Alberto Garcia <agarciaillera at gmail.com>
wrote:

> Thank you for your response,
>
> I was thinking on creating that zip file with the content of the Lib
> folder and having my c code to download it over the network and have it in
> memory.
> I guess that the zip file should have no compression at all right?
>
> When you say that I need to use the cx_freeze approach what do you mean?
> Can you point me to where they do that?
>
> And why changing sys.path again to the executable again? Which part of the
> executable?
>
> I'll put my efforts in this.
>
> Thank you
>
> On Mon, Jul 9, 2018 at 7:16 AM Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> On 9 July 2018 at 03:10, Alberto Garcia <agarciaillera at gmail.com> wrote:
>> > Hey there,
>> >
>> > Yes, the part of having the pyd modules built in in library is already
>> done.
>> > I followed the instructions in the README. What I would like to know
>> now is
>> > how to embed the non frozen python (py) modules. Can you guys please
>> point
>> > me in the right direction.
>>
>> The gist is to:
>>
>> 1. take the entire Lib directory and put it in a zip archive
>> 2. use the approach demonstrated in cx_freeze to point sys.path in
>> your static executable at that zip archive
>> 3. adjust your C code to point sys.path back at the executable itself,
>> and then combine your executable and the zip archive into a single
>> contiguous file (similar to what zipapp does with it's helper script
>> and app archive)
>>
>> There are likely to still be rough edges when doing that, since this
>> isn't a well tested configuration. When all else fails, find the part
>> of the source code responsible for any error messages you're seeing,
>> and try to work out if there's a setting you can tweak to avoid
>> hitting that code path.
>>
>> Cheers,
>> Nick.
>>
>> --
>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>>
> --
> Alberto Garc?a Illera
>
> GPG Public Key: https://goo.gl/twKUUv
>
-- 
Alberto Garc?a Illera

GPG Public Key: https://goo.gl/twKUUv
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180709/0971578b/attachment-0001.html>

From agarciaillera at gmail.com  Mon Jul  9 14:40:40 2018
From: agarciaillera at gmail.com (Alberto Garcia)
Date: Mon, 9 Jul 2018 11:40:40 -0700
Subject: [Python-ideas] Calling python from C completely statically
In-Reply-To: <CA+fJkbkrRAmjvjOaUTsksDJawgv9O7zm2pkdKfiWu4eGFOXW_g@mail.gmail.com>
References: <CA+fJkbn5r_PVwJtS_-aZ9qGp0mqY2BytFAmQkPLLkU+ZvryfyA@mail.gmail.com>
 <CACac1F-J+bctMVEa7CcTn46JOpWXY7RrxEH_Lzt0U0YZ8phzuw@mail.gmail.com>
 <CADiSq7cbrZJiLtgw4Y_2e_U4LjuJ9J67hiorA=JB-0rFMjSYEg@mail.gmail.com>
 <CA+fJkbmC1A+R2ggCMa3h6y8eEg=i1-zy6yxBnecNNHqZdT5Z2w@mail.gmail.com>
 <CADiSq7dGY+F7XZreNd3_VDSBmv7+GQgoFbdTbBmMbKtUDXBTdw@mail.gmail.com>
 <CA+fJkb=66-Do3BxtPU6OPgVi7SJF44s_hn8xYpAcYE=O+oqSwQ@mail.gmail.com>
 <CA+fJkbkrRAmjvjOaUTsksDJawgv9O7zm2pkdKfiWu4eGFOXW_g@mail.gmail.com>
Message-ID: <CA+fJkbkpbj=M+mm5DVHnj8KbjFA+yWUABKaB3=Uh32oo7VMGpw@mail.gmail.com>

Does the zip need to reside in disk to be loaded. Or can it be loaded from
memory? I don't want it to be loaded from disk but from Memory



On Mon, Jul 9, 2018 at 9:59 AM Alberto Garcia <agarciaillera at gmail.com>
wrote:

> Ohhhhhhhh  I guess you mean this:
>
> https://github.com/anthony-tuininga/cx_Freeze/blob/master/source/bases/Common.c
>
> Right?
>
> On Mon, Jul 9, 2018 at 9:48 AM Alberto Garcia <agarciaillera at gmail.com>
> wrote:
>
>> Thank you for your response,
>>
>> I was thinking on creating that zip file with the content of the Lib
>> folder and having my c code to download it over the network and have it in
>> memory.
>> I guess that the zip file should have no compression at all right?
>>
>> When you say that I need to use the cx_freeze approach what do you mean?
>> Can you point me to where they do that?
>>
>> And why changing sys.path again to the executable again? Which part of
>> the executable?
>>
>> I'll put my efforts in this.
>>
>> Thank you
>>
>> On Mon, Jul 9, 2018 at 7:16 AM Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>>> On 9 July 2018 at 03:10, Alberto Garcia <agarciaillera at gmail.com> wrote:
>>> > Hey there,
>>> >
>>> > Yes, the part of having the pyd modules built in in library is already
>>> done.
>>> > I followed the instructions in the README. What I would like to know
>>> now is
>>> > how to embed the non frozen python (py) modules. Can you guys please
>>> point
>>> > me in the right direction.
>>>
>>> The gist is to:
>>>
>>> 1. take the entire Lib directory and put it in a zip archive
>>> 2. use the approach demonstrated in cx_freeze to point sys.path in
>>> your static executable at that zip archive
>>> 3. adjust your C code to point sys.path back at the executable itself,
>>> and then combine your executable and the zip archive into a single
>>> contiguous file (similar to what zipapp does with it's helper script
>>> and app archive)
>>>
>>> There are likely to still be rough edges when doing that, since this
>>> isn't a well tested configuration. When all else fails, find the part
>>> of the source code responsible for any error messages you're seeing,
>>> and try to work out if there's a setting you can tweak to avoid
>>> hitting that code path.
>>>
>>> Cheers,
>>> Nick.
>>>
>>> --
>>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>>>
>> --
>> Alberto Garc?a Illera
>>
>> GPG Public Key: https://goo.gl/twKUUv
>>
> --
> Alberto Garc?a Illera
>
> GPG Public Key: https://goo.gl/twKUUv
>


-- 
Alberto Garc?a Illera

GPG Public Key: https://goo.gl/twKUUv
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180709/9af07a77/attachment.html>

From barry at barrys-emacs.org  Mon Jul  9 15:20:15 2018
From: barry at barrys-emacs.org (Barry Scott)
Date: Mon, 9 Jul 2018 20:20:15 +0100
Subject: [Python-ideas] Calling python from C completely statically
In-Reply-To: <CA+fJkbkpbj=M+mm5DVHnj8KbjFA+yWUABKaB3=Uh32oo7VMGpw@mail.gmail.com>
References: <CA+fJkbn5r_PVwJtS_-aZ9qGp0mqY2BytFAmQkPLLkU+ZvryfyA@mail.gmail.com>
 <CACac1F-J+bctMVEa7CcTn46JOpWXY7RrxEH_Lzt0U0YZ8phzuw@mail.gmail.com>
 <CADiSq7cbrZJiLtgw4Y_2e_U4LjuJ9J67hiorA=JB-0rFMjSYEg@mail.gmail.com>
 <CA+fJkbmC1A+R2ggCMa3h6y8eEg=i1-zy6yxBnecNNHqZdT5Z2w@mail.gmail.com>
 <CADiSq7dGY+F7XZreNd3_VDSBmv7+GQgoFbdTbBmMbKtUDXBTdw@mail.gmail.com>
 <CA+fJkb=66-Do3BxtPU6OPgVi7SJF44s_hn8xYpAcYE=O+oqSwQ@mail.gmail.com>
 <CA+fJkbkrRAmjvjOaUTsksDJawgv9O7zm2pkdKfiWu4eGFOXW_g@mail.gmail.com>
 <CA+fJkbkpbj=M+mm5DVHnj8KbjFA+yWUABKaB3=Uh32oo7VMGpw@mail.gmail.com>
Message-ID: <05AECDA7-4AB6-4153-84E3-14638898E8CB@barrys-emacs.org>

I think you might find Gordon McMillian's installer interesting to look at. It has a lot
f the tech that I think you are looking for.

Works up to python 2.7. I ended up taking over when Gordon stopped maintaining
it and kept it going up to python 2.7. In principle the same ideas could be made to
work in python 3 I believe.

	https://sourceforge.net/projects/meinc-installer/

The zip file is appended to the end of the .EXE or unix ELF fie.
The boot strap knows how to import form the ZIP at the end of
the binary.

It also has a way to split out the .SO/.DLL files from the ZIP and
allow them to be loaded. Single EXE mode.

There are docs that explain how it works in the sources.

Barry



> On 9 Jul 2018, at 19:40, Alberto Garcia <agarciaillera at gmail.com> wrote:
> 
> Does the zip need to reside in disk to be loaded. Or can it be loaded from memory? I don't want it to be loaded from disk but from Memory
> 
> 
> 
> On Mon, Jul 9, 2018 at 9:59 AM Alberto Garcia <agarciaillera at gmail.com <mailto:agarciaillera at gmail.com>> wrote:
> Ohhhhhhhh  I guess you mean this:
> https://github.com/anthony-tuininga/cx_Freeze/blob/master/source/bases/Common.c <https://github.com/anthony-tuininga/cx_Freeze/blob/master/source/bases/Common.c>
> Right?
> 
> On Mon, Jul 9, 2018 at 9:48 AM Alberto Garcia <agarciaillera at gmail.com <mailto:agarciaillera at gmail.com>> wrote:
> Thank you for your response,
> 
> I was thinking on creating that zip file with the content of the Lib folder and having my c code to download it over the network and have it in memory. 
> I guess that the zip file should have no compression at all right?
> 
> When you say that I need to use the cx_freeze approach what do you mean? Can you point me to where they do that? 
> 
> And why changing sys.path again to the executable again? Which part of the executable? 
> 
> I'll put my efforts in this.
> 
> Thank you 
> 
> On Mon, Jul 9, 2018 at 7:16 AM Nick Coghlan <ncoghlan at gmail.com <mailto:ncoghlan at gmail.com>> wrote:
> On 9 July 2018 at 03:10, Alberto Garcia <agarciaillera at gmail.com <mailto:agarciaillera at gmail.com>> wrote:
> > Hey there,
> >
> > Yes, the part of having the pyd modules built in in library is already done.
> > I followed the instructions in the README. What I would like to know now is
> > how to embed the non frozen python (py) modules. Can you guys please point
> > me in the right direction.
> 
> The gist is to:
> 
> 1. take the entire Lib directory and put it in a zip archive
> 2. use the approach demonstrated in cx_freeze to point sys.path in
> your static executable at that zip archive
> 3. adjust your C code to point sys.path back at the executable itself,
> and then combine your executable and the zip archive into a single
> contiguous file (similar to what zipapp does with it's helper script
> and app archive)
> 
> There are likely to still be rough edges when doing that, since this
> isn't a well tested configuration. When all else fails, find the part
> of the source code responsible for any error messages you're seeing,
> and try to work out if there's a setting you can tweak to avoid
> hitting that code path.
> 
> Cheers,
> Nick.
> 
> -- 
> Nick Coghlan   |   ncoghlan at gmail.com <mailto:ncoghlan at gmail.com>   |   Brisbane, Australia
> -- 
> Alberto Garc?a Illera
> 
> GPG Public Key: https://goo.gl/twKUUv <https://goo.gl/twKUUv>-- 
> Alberto Garc?a Illera
> 
> GPG Public Key: https://goo.gl/twKUUv <https://goo.gl/twKUUv>
> 
> -- 
> Alberto Garc?a Illera
> 
> GPG Public Key: https://goo.gl/twKUUv <https://goo.gl/twKUUv>_______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org <mailto:Python-ideas at python.org>
> https://mail.python.org/mailman/listinfo/python-ideas <https://mail.python.org/mailman/listinfo/python-ideas>
> Code of Conduct: http://python.org/psf/codeofconduct/ <http://python.org/psf/codeofconduct/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180709/079fd448/attachment-0001.html>

From agarciaillera at gmail.com  Mon Jul  9 16:11:57 2018
From: agarciaillera at gmail.com (Alberto Garcia)
Date: Mon, 9 Jul 2018 13:11:57 -0700
Subject: [Python-ideas] Calling python from C completely statically
In-Reply-To: <05AECDA7-4AB6-4153-84E3-14638898E8CB@barrys-emacs.org>
References: <CA+fJkbn5r_PVwJtS_-aZ9qGp0mqY2BytFAmQkPLLkU+ZvryfyA@mail.gmail.com>
 <CACac1F-J+bctMVEa7CcTn46JOpWXY7RrxEH_Lzt0U0YZ8phzuw@mail.gmail.com>
 <CADiSq7cbrZJiLtgw4Y_2e_U4LjuJ9J67hiorA=JB-0rFMjSYEg@mail.gmail.com>
 <CA+fJkbmC1A+R2ggCMa3h6y8eEg=i1-zy6yxBnecNNHqZdT5Z2w@mail.gmail.com>
 <CADiSq7dGY+F7XZreNd3_VDSBmv7+GQgoFbdTbBmMbKtUDXBTdw@mail.gmail.com>
 <CA+fJkb=66-Do3BxtPU6OPgVi7SJF44s_hn8xYpAcYE=O+oqSwQ@mail.gmail.com>
 <CA+fJkbkrRAmjvjOaUTsksDJawgv9O7zm2pkdKfiWu4eGFOXW_g@mail.gmail.com>
 <CA+fJkbkpbj=M+mm5DVHnj8KbjFA+yWUABKaB3=Uh32oo7VMGpw@mail.gmail.com>
 <05AECDA7-4AB6-4153-84E3-14638898E8CB@barrys-emacs.org>
Message-ID: <CA+fJkb=W=B=tiVDFOqRs3TLLYsaRRTPQDcCp9-UOcfKOzBVcEA@mail.gmail.com>

Hi,

thank you for your response. I've downloaded the sources but I couldn't
find any documentation. In addition I see that there is not a single C/C++
file. What I want to do is calling python from C.
Am I missing something?

Cheers


On Mon, Jul 9, 2018 at 12:20 PM Barry Scott <barry at barrys-emacs.org> wrote:

> I think you might find Gordon McMillian's installer interesting to look
> at. It has a lot
> f the tech that I think you are looking for.
>
> Works up to python 2.7. I ended up taking over when Gordon stopped
> maintaining
> it and kept it going up to python 2.7. In principle the same ideas could
> be made to
> work in python 3 I believe.
>
> https://sourceforge.net/projects/meinc-installer/
>
> The zip file is appended to the end of the .EXE or unix ELF fie.
> The boot strap knows how to import form the ZIP at the end of
> the binary.
>
> It also has a way to split out the .SO/.DLL files from the ZIP and
> allow them to be loaded. Single EXE mode.
>
> There are docs that explain how it works in the sources.
>
> Barry
>
>
>
> On 9 Jul 2018, at 19:40, Alberto Garcia <agarciaillera at gmail.com> wrote:
>
> Does the zip need to reside in disk to be loaded. Or can it be loaded from
> memory? I don't want it to be loaded from disk but from Memory
>
>
>
> On Mon, Jul 9, 2018 at 9:59 AM Alberto Garcia <agarciaillera at gmail.com>
> wrote:
>
>> Ohhhhhhhh  I guess you mean this:
>>
>> https://github.com/anthony-tuininga/cx_Freeze/blob/master/source/bases/Common.c
>>
>> Right?
>>
>> On Mon, Jul 9, 2018 at 9:48 AM Alberto Garcia <agarciaillera at gmail.com>
>> wrote:
>>
>>> Thank you for your response,
>>>
>>> I was thinking on creating that zip file with the content of the Lib
>>> folder and having my c code to download it over the network and have it in
>>> memory.
>>> I guess that the zip file should have no compression at all right?
>>>
>>> When you say that I need to use the cx_freeze approach what do you mean?
>>> Can you point me to where they do that?
>>>
>>> And why changing sys.path again to the executable again? Which part of
>>> the executable?
>>>
>>> I'll put my efforts in this.
>>>
>>> Thank you
>>>
>>> On Mon, Jul 9, 2018 at 7:16 AM Nick Coghlan <ncoghlan at gmail.com> wrote:
>>>
>>>> On 9 July 2018 at 03:10, Alberto Garcia <agarciaillera at gmail.com>
>>>> wrote:
>>>> > Hey there,
>>>> >
>>>> > Yes, the part of having the pyd modules built in in library is
>>>> already done.
>>>> > I followed the instructions in the README. What I would like to know
>>>> now is
>>>> > how to embed the non frozen python (py) modules. Can you guys please
>>>> point
>>>> > me in the right direction.
>>>>
>>>> The gist is to:
>>>>
>>>> 1. take the entire Lib directory and put it in a zip archive
>>>> 2. use the approach demonstrated in cx_freeze to point sys.path in
>>>> your static executable at that zip archive
>>>> 3. adjust your C code to point sys.path back at the executable itself,
>>>> and then combine your executable and the zip archive into a single
>>>> contiguous file (similar to what zipapp does with it's helper script
>>>> and app archive)
>>>>
>>>> There are likely to still be rough edges when doing that, since this
>>>> isn't a well tested configuration. When all else fails, find the part
>>>> of the source code responsible for any error messages you're seeing,
>>>> and try to work out if there's a setting you can tweak to avoid
>>>> hitting that code path.
>>>>
>>>> Cheers,
>>>> Nick.
>>>>
>>>> --
>>>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>>>>
>>> --
>>> Alberto Garc?a Illera
>>>
>>> GPG Public Key: https://goo.gl/twKUUv
>>>
>> --
>> Alberto Garc?a Illera
>>
>> GPG Public Key: https://goo.gl/twKUUv
>>
>
>
> --
> Alberto Garc?a Illera
>
> GPG Public Key: https://goo.gl/twKUUv
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>

-- 
Alberto Garc?a Illera

GPG Public Key: https://goo.gl/twKUUv
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180709/a3240650/attachment.html>

From mertz at gnosis.cx  Mon Jul  9 16:38:23 2018
From: mertz at gnosis.cx (David Mertz)
Date: Mon, 9 Jul 2018 16:38:23 -0400
Subject: [Python-ideas] Fwd: grouping / dict of lists
In-Reply-To: <CALGmxE+OS-EDa7gNceEirnW1S0oryyinA7mMbUetXfZkJ67Zkw@mail.gmail.com>
References: <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAOmnVUBp9vJjTp6v17LHy2C=7GzR63fb4ARZW2Dco022uTDpfA@mail.gmail.com>
 <20180703153338.GG14437@ando.pearwood.info>
 <CALGmxEJBYFEoPOFo7aVSN=Xy=sgSMnrF0k8pnt=VhJ6cvhS_4Q@mail.gmail.com>
 <20180704013202.GK14437@ando.pearwood.info>
 <CAEbHw4YafoZc_1cnB6a-psmWqW-SD2tw3E2Fw5hUHSJxqEsu-Q@mail.gmail.com>
 <CAEbHw4Yjue-V_CKQJe2k6f4E8ucNuG=ZNFt5O_njfrQoNKHeUw@mail.gmail.com>
 <CALGmxE+m+1YXk1vRsdkAK6yQ12V=syi6WKZsSTMFmw1BoiKUYA@mail.gmail.com>
 <CAB_e7izyUbkOXxpwivr0iyE4gXk58xn+SBfAZq5hcKAyDO-H_g@mail.gmail.com>
 <CALGmxE+OS-EDa7gNceEirnW1S0oryyinA7mMbUetXfZkJ67Zkw@mail.gmail.com>
Message-ID: <CAEbHw4Y7A4H3XZvRTgFb_z84CqAVWbe0KAM_xVgUAHCoYhw9pw@mail.gmail.com>

In my mind, I *rarely* (which is more than never) have my data in the form
of a sequence of key/value pairs.  The version of the API that assumes data
starts that way feels like either a niche case, or demands preprocessing
before it's ready to pass to grouping() or collections.Grouping().

That said, an identity key is rarely interesting either. So I think have
key=None mean "assume we get key/val pairs is harmless to the more common
case where we give an explicit key function.

The uncommon need for grouping on equality can be handled with 'key=lambda
x: x'.

On Mon, Jul 9, 2018, 12:22 PM Chris Barker <chris.barker at noaa.gov> wrote:

> On Fri, Jul 6, 2018 at 12:26 PM, Franklin? Lee I use this kind of function
> in several different projects over the
>>
>> years, and I rewrote it many times as needed.
>>
>
>
>> I added several options, such as:
>> - key function
>> - value function
>> - "ignore": Skip values with these keys.
>> - "postprocess": Apply a function to each group after completion.
>> - Pass in the container to store in. For example, create an
>> OrderedDict and pass it in. It may already hold items.
>> - Specify the container for each group.
>> - Specify how to add to the container for each group.
>>
>
> interesting...
>
>
>> Then I cut it down to two optional parameters:
>> - key function. If not provided, the iterable is considered to have
>> key-value pairs.
>>
>
> OK -- seems we're all converging on that :-)
>
>
>> - The storage container.
>>
>
> so this means you'r passing in a full set of storage containers? I'm a vit
> confused by that -- if they might be pre-populated, then they would need to
> be instance,s an you'd need to have one for every key -- how would you know
> in advance aht you needed???
>
> I played around with passing in a optional storage object:
>
>
> https://github.com/PythonCHB/grouper/commit/d986816905406ec402724beaed2b88c96df64469
>
> but as we might want a list or a set, or a Counter, or ??? it got pretty
> ugly, as lists and sets and Counters all have different APIs for adding
> stuff. So I gave up and figured just saying "it's always a list) made the
> most sense.
>
>
>> Finally, I removed the key function, and only took pairs and an
>> optional container. However, I don't remember why I removed the key
>> function. It may be that I was writing throwaway lambdas, and I
>> decided I might as well just write the transformation into the
>> comprehension.
>
>
> exactly -- but I suspect hat may be because you where writing a
> comprehension anyway, as you needed to manipulate the values, also -- so if
> there were a value function, you could use either API.
>
>
>> I think a key function is worth having.
>>
>
> I think there's more or less consensus on that too.
>
> -CHB
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180709/89009eea/attachment-0001.html>

From brett at python.org  Mon Jul  9 17:01:49 2018
From: brett at python.org (Brett Cannon)
Date: Mon, 9 Jul 2018 14:01:49 -0700
Subject: [Python-ideas] Add new `Symbol` type
In-Reply-To: <d625da50-f7ea-05cd-89d5-2f314d8f2607@trueblade.com>
References: <CAAjMAJWeXOofJtz=OsU8OWq3QTZVsxEKfbmBeDivQrD=-pNZTA@mail.gmail.com>
 <CAFYqXL_DOyL5Rc76KZyjL4=J_ZdDZaHkktWfouC02M6=2BzSSw@mail.gmail.com>
 <CAAjMAJUzjfNr4dBQagVhKeF0TV8dPGtZqWbwhbAHM=xH10DCQg@mail.gmail.com>
 <d625da50-f7ea-05cd-89d5-2f314d8f2607@trueblade.com>
Message-ID: <CAP1=2W7QgO8UvkBCj-wchDg119mLyK2c9ibKOM7Wp58wRvZrdw@mail.gmail.com>

On Fri, 6 Jul 2018 at 09:24 Eric V. Smith <eric at trueblade.com> wrote:

> On 7/6/2018 11:20 AM, Flavio Curella wrote:
> > I think this thread can be resolved as 'used unittest.mock.sentinel'. It
> > doesn't have 'global sentinels', but I'm not convinced they are actually
> > necessary, since `mock.sentinel` objects with the same name compare as
> > equal. Thanks to Nathaniel, I now understand that JS has global symbols
> > for historical reasons that we don't have, and I'm not convinced of
> > their usefulness.
>
> Do all Python distributions ship with unittest.mock? I see to recall
> that Debian and/or Ubuntu strips out part of the normal distribution.
>

It's usually tkinter and such, not unittest stuff from my understanding.


>
> For example, dataclasses.py has a sentinel, and it includes some code to
> get a more helpful repr. It would make sense to re-use the
> unittest.mock.sentinel code, but not if that code isn't always
> guaranteed to be present.
>

Would it make sense to abstract this out to the 'types'  to have a single
'types.sentinel' object for those rare cases that Guido pointed out?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180709/6e5f641a/attachment.html>

From eric at trueblade.com  Mon Jul  9 18:16:45 2018
From: eric at trueblade.com (Eric V. Smith)
Date: Mon, 9 Jul 2018 18:16:45 -0400
Subject: [Python-ideas] Add new `Symbol` type
In-Reply-To: <CAP1=2W7QgO8UvkBCj-wchDg119mLyK2c9ibKOM7Wp58wRvZrdw@mail.gmail.com>
References: <CAAjMAJWeXOofJtz=OsU8OWq3QTZVsxEKfbmBeDivQrD=-pNZTA@mail.gmail.com>
 <CAFYqXL_DOyL5Rc76KZyjL4=J_ZdDZaHkktWfouC02M6=2BzSSw@mail.gmail.com>
 <CAAjMAJUzjfNr4dBQagVhKeF0TV8dPGtZqWbwhbAHM=xH10DCQg@mail.gmail.com>
 <d625da50-f7ea-05cd-89d5-2f314d8f2607@trueblade.com>
 <CAP1=2W7QgO8UvkBCj-wchDg119mLyK2c9ibKOM7Wp58wRvZrdw@mail.gmail.com>
Message-ID: <e4c9ecf0-f97e-f3e4-2b40-2150ea46977f@trueblade.com>

On 7/9/2018 5:01 PM, Brett Cannon wrote:
> 
> 
> On Fri, 6 Jul 2018 at 09:24 Eric V. Smith <eric at trueblade.com 
> <mailto:eric at trueblade.com>> wrote:
> 
>     On 7/6/2018 11:20 AM, Flavio Curella wrote:
>      > I think this thread can be resolved as 'used
>     unittest.mock.sentinel'. It
>      > doesn't have 'global sentinels', but I'm not convinced they are
>     actually
>      > necessary, since `mock.sentinel` objects with the same name
>     compare as
>      > equal. Thanks to Nathaniel, I now understand that JS has global
>     symbols
>      > for historical reasons that we don't have, and I'm not convinced of
>      > their usefulness.
> 
>     Do all Python distributions ship with unittest.mock? I see to recall
>     that Debian and/or Ubuntu strips out part of the normal distribution.
> 
> 
> It's usually tkinter and such, not unittest stuff from my understanding.

Good to know. Thanks.

>     For example, dataclasses.py has a sentinel, and it includes some
>     code to
>     get a more helpful repr. It would make sense to re-use the
>     unittest.mock.sentinel code, but not if that code isn't always
>     guaranteed to be present.
> 
> 
> Would it make sense to abstract this out to the 'types'? to have a 
> single 'types.sentinel' object for those rare cases that Guido pointed out?

I think so. I'd hate to import unittest.mock just to get a sentinel 
object for dataclasses.

Eric


From leewangzhong+python at gmail.com  Mon Jul  9 18:55:19 2018
From: leewangzhong+python at gmail.com (Franklin? Lee)
Date: Mon, 9 Jul 2018 18:55:19 -0400
Subject: [Python-ideas] Fwd: grouping / dict of lists
In-Reply-To: <CALGmxE+OS-EDa7gNceEirnW1S0oryyinA7mMbUetXfZkJ67Zkw@mail.gmail.com>
References: <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAOmnVUBp9vJjTp6v17LHy2C=7GzR63fb4ARZW2Dco022uTDpfA@mail.gmail.com>
 <20180703153338.GG14437@ando.pearwood.info>
 <CALGmxEJBYFEoPOFo7aVSN=Xy=sgSMnrF0k8pnt=VhJ6cvhS_4Q@mail.gmail.com>
 <20180704013202.GK14437@ando.pearwood.info>
 <CAEbHw4YafoZc_1cnB6a-psmWqW-SD2tw3E2Fw5hUHSJxqEsu-Q@mail.gmail.com>
 <CAEbHw4Yjue-V_CKQJe2k6f4E8ucNuG=ZNFt5O_njfrQoNKHeUw@mail.gmail.com>
 <CALGmxE+m+1YXk1vRsdkAK6yQ12V=syi6WKZsSTMFmw1BoiKUYA@mail.gmail.com>
 <CAB_e7izyUbkOXxpwivr0iyE4gXk58xn+SBfAZq5hcKAyDO-H_g@mail.gmail.com>
 <CALGmxE+OS-EDa7gNceEirnW1S0oryyinA7mMbUetXfZkJ67Zkw@mail.gmail.com>
Message-ID: <CAB_e7iw=mCRtBGu2YJ_kV7yffSWh_Rp927Lpw_gwiyzbJtoqzA@mail.gmail.com>

On Mon, Jul 9, 2018 at 12:22 PM, Chris Barker <chris.barker at noaa.gov> wrote:
> On Fri, Jul 6, 2018 at 12:26 PM, Franklin? Lee I use this kind of function

>> I added several options, such as:
>> - key function
>> - value function
>> - "ignore": Skip values with these keys.
>> - "postprocess": Apply a function to each group after completion.
>> - Pass in the container to store in. For example, create an
>> OrderedDict and pass it in. It may already hold items.
>> - Specify the container for each group.
>> - Specify how to add to the container for each group.
>
>
> interesting...
>
>>
>> Then I cut it down to two optional parameters:
>> - key function. If not provided, the iterable is considered to have
>> key-value pairs.
>
>
> OK -- seems we're all converging on that :-)
>
>>
>> - The storage container.
>
>
> so this means you'r passing in a full set of storage containers? I'm a vit
> confused by that -- if they might be pre-populated, then they would need to
> be instance,s an you'd need to have one for every key -- how would you know
> in advance aht you needed???

No, I mean the mapping (outer) container. For example, I can pass in
an empty OrderedDict, or a dict that already contained some groups
from a previous call to the grouping function.

I took out the option for the per-group (inner) containers. I never
found it necessary to scrooge (scrooge on?) the memory, when I could
postprocessed the lists after grouping. A mapvalues function will make
postprocessing more convenient, and lend weight to a dicttools
suggestion.

    # Unfortunate double meaning of 'map' in the function signature:
    def mapvalues(f, mapping):
        try:
            items = mapping.items()
        except AttributeError:
            items = mapping
        return {k: f(v) for k, v in items}

> I played around with passing in a optional storage object:
>
> https://github.com/PythonCHB/grouper/commit/d986816905406ec402724beaed2b88c96df64469
>
> but as we might want a list or a set, or a Counter, or ??? it got pretty
> ugly, as lists and sets and Counters all have different APIs for adding
> stuff. So I gave up and figured just saying "it's always a list) made the
> most sense.

My solution at the time was to add another parameter to specify how to
add to the container.

In fuller generality, the option for the per-group container may
consist of specifying a monad (if I remember monads correctly). You
need to at least specify a per-group container constructor and a
binary function that adds to it. In the case of `Counter`, the
constructor is `int` and the binary function is `int.__add__`, and the
Counter constructor effectively runs concurrent `reduce`.

From leewangzhong+python at gmail.com  Mon Jul  9 23:30:25 2018
From: leewangzhong+python at gmail.com (Franklin? Lee)
Date: Mon, 9 Jul 2018 23:30:25 -0400
Subject: [Python-ideas] Where should grouping() live (was: grouping /
 dict of lists)
In-Reply-To: <CALGmxEJCg-a22PB8KRWiqx8o6gNRXY7vYGQ4eSNSV-=8oMZE_w@mail.gmail.com>
References: <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAEbHw4YG0oZ20CV9K3XEjjzkJ_ULW0fXqEqcatDxo9gWsuJQVg@mail.gmail.com>
 <CALGmxELSOU6HEeQfDfiC++CuvEF3GBxEQDuvv7Kb=ZeLF-Jucw@mail.gmail.com>
 <CAEbHw4ZXqRoMWnOWjqgPQ0OjcGm0MFPW_VhvZdnDuVvWygDc+w@mail.gmail.com>
 <CALGmxEJJbRRmGBMKOUK6G4TUpxLGvceU3aqq6t0vrGMbFUwS_g@mail.gmail.com>
 <CACc=A=2MmzE+=HLLCLMM=EY9HWTupfDZ7utaKPJQhUhP=eL0Gg@mail.gmail.com>
 <20180706091045.GH7318@ando.pearwood.info>
 <CALGmxEJCg-a22PB8KRWiqx8o6gNRXY7vYGQ4eSNSV-=8oMZE_w@mail.gmail.com>
Message-ID: <CAB_e7iyFvgVRHDG9nW2owOB78C0aszyGmQp4wBfPDGWXFvZFTA@mail.gmail.com>

(Fixing quote and attribution.)

On Fri, Jul 6, 2018, 11:32 Chris Barker - NOAA Federal via
Python-ideas <python-ideas at python.org> wrote:
>
> On Jul 6, 2018, at 2:10 AM, Steven D'Aprano <steve at pearwood.info> wrote:
>
> > On Fri, Jul 06, 2018 at 09:49:37AM +0100, Cammil Taank wrote:
> > > I would consider statistics
>
> > > to have similarities - median, mean etc are aggregate functions.
>
>
> Not really, more like reduce, actually -/ you get a single result.
>
> > > Histograms are also doing something similar to grouping.
>
> > .(Yes, a few statistics apply to nominal and ordinal data too,
>
>
> And for that, a generic grouping function could be used.
>
> In fact, allowing Counter to be used as the accumulater was one suggestion in this thread, and would build s histogram.
>
> Now that I think about it, you could write a key function that built a histogram for continuous data as well.
>
> Though that might be a bit klunky.
>
> But if someone thinks that?s a good idea, a PR for an example would be accepted:
>
> https://github.com/PythonCHB/grouper

+1 for `collections`, because it's where you look for something
similar to Counter.

-1 for `statistics`, because the need isn't specific to statistics.
It'd be like putting `commonprefix`, which is a general string
operation, into `os.path`. It's hacky to import a domain-specific
module to use one of its non-domain-specific helpers for a different
domain.

Someone can argue for functools, as that's the functional programming
module, containing `reduce`.

From pander at users.sourceforge.net  Tue Jul 10 03:57:02 2018
From: pander at users.sourceforge.net (Pander)
Date: Tue, 10 Jul 2018 09:57:02 +0200
Subject: [Python-ideas] Support localization of unicode descriptions
Message-ID: <e91cca70-d9b9-5224-0a45-537ed308dd4c@users.sourceforge.net>

Dear all,

Who might also be interested in setting up a project that supports localization for Unicode block description and character description.

Translations are available from https://github.com/unicode-table/unicode-table-data/tree/master/loc
<https://github.com/unicode-table/unicode-table-data/tree/master/loc> If possible, use a gettext approach similar to https://pypi.org/project/pycountry/ <https://pypi.org/project/pycountry/>

Implementing this feature will allow users to read Unicode descriptions in their own language, other than English.

For example, now is possible only in English:

    from unicodedata import name
    print(name('?'))
    LATIN SMALL LETTER SHARP S

So unicodedata could provide a way to translate LATIN SMALL LETTER SHARP S to e.g. German with:

    from unicodedata import name
    from unicodedata_l10n import LOCALED_DIR
    from gettext import translation
    german = translation('UnicodeData' LOCALED_DIR, languages=['de'])
    german.install()
    print(_(name('?')))
    LATEINISCHER KLEINBUCHSTABE SCHARFES S

and something similar for unicodedata.category

Best,

Pander

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180710/50f63873/attachment.html>

From p.f.moore at gmail.com  Tue Jul 10 04:34:18 2018
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 10 Jul 2018 09:34:18 +0100
Subject: [Python-ideas] Support localization of unicode descriptions
In-Reply-To: <e91cca70-d9b9-5224-0a45-537ed308dd4c@users.sourceforge.net>
References: <e91cca70-d9b9-5224-0a45-537ed308dd4c@users.sourceforge.net>
Message-ID: <CACac1F8-50F40d2jqiJfz085jjTc3xpNgYHeWcsU6oiuaQgH9A@mail.gmail.com>

On 10 July 2018 at 08:57, Pander <pander at users.sourceforge.net> wrote:
> Dear all,
>
> Who might also be interested in setting up a project that supports
> localization for Unicode block description and character description.
>
> Translations are available from
> https://github.com/unicode-table/unicode-table-data/tree/master/loc If
> possible, use a gettext approach similar to
> https://pypi.org/project/pycountry/
>
> Implementing this feature will allow users to read Unicode descriptions in
> their own language, other than English.

Is this a Unicode Consortium standard, or a 3rd party project? The
website wasn't completely clear on the matter, but there's nothing I
could find on the Unicode website about translations of the standard
name (there's also nothing that specifically explains the choice to
use English for the standard names...). If it's not part of the
standard, then there's an argument that the Python implementation of
this should also be a 3rd party package, rather than being in the
stdlib. Is this feature available on PyPI at the moment?

Also, would this not lead to non-English speakers expecting that the
localised names would work in "\N{...}" notation?

Paul

From pander at users.sourceforge.net  Tue Jul 10 04:45:06 2018
From: pander at users.sourceforge.net (Pander)
Date: Tue, 10 Jul 2018 10:45:06 +0200
Subject: [Python-ideas] Support localization of unicode descriptions
In-Reply-To: <CACac1F8-50F40d2jqiJfz085jjTc3xpNgYHeWcsU6oiuaQgH9A@mail.gmail.com>
References: <e91cca70-d9b9-5224-0a45-537ed308dd4c@users.sourceforge.net>
 <CACac1F8-50F40d2jqiJfz085jjTc3xpNgYHeWcsU6oiuaQgH9A@mail.gmail.com>
Message-ID: <f466ac5a-eec0-fde5-4800-87257558471a@users.sourceforge.net>



On 07/10/2018 10:34 AM, Paul Moore wrote:
> ...
> Is this a Unicode Consortium standard, or a 3rd party project? The
> website wasn't completely clear on the matter, but there's nothing I
> could find on the Unicode website about translations of the standard
> name (there's also nothing that specifically explains the choice to
> use English for the standard names...). If it's not part of the
> standard, then there's an argument that the Python implementation of
> this should also be a 3rd party package, rather than being in the
> stdlib. Is this feature available on PyPI at the moment?
This is a third party initiative. The translations are contributed by
volunteers. I have talked with Python core developers and they suggested
to post this here, as it is for them out of scope for Python std lib. At
the moment there is no implementation yet. That is what I would like to
discuss here.
>
> Also, would this not lead to non-English speakers expecting that the
> localised names would work in "\N{...}" notation?
Don't know. If an implementation has been made, it should be positioned
very carefully.

Pander
>
> Paul


From p.f.moore at gmail.com  Tue Jul 10 05:01:12 2018
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 10 Jul 2018 10:01:12 +0100
Subject: [Python-ideas] Support localization of unicode descriptions
In-Reply-To: <f466ac5a-eec0-fde5-4800-87257558471a@users.sourceforge.net>
References: <e91cca70-d9b9-5224-0a45-537ed308dd4c@users.sourceforge.net>
 <CACac1F8-50F40d2jqiJfz085jjTc3xpNgYHeWcsU6oiuaQgH9A@mail.gmail.com>
 <f466ac5a-eec0-fde5-4800-87257558471a@users.sourceforge.net>
Message-ID: <CACac1F_z3nxyLpf4c8ZKqzZ5++gNBevRNPED2EZ5r=zFagz=mg@mail.gmail.com>

On 10 July 2018 at 09:45, Pander <pander at users.sourceforge.net> wrote:
>
> On 07/10/2018 10:34 AM, Paul Moore wrote:
>> ...
>> Is this a Unicode Consortium standard, or a 3rd party project? The
>> website wasn't completely clear on the matter, but there's nothing I
>> could find on the Unicode website about translations of the standard
>> name (there's also nothing that specifically explains the choice to
>> use English for the standard names...). If it's not part of the
>> standard, then there's an argument that the Python implementation of
>> this should also be a 3rd party package, rather than being in the
>> stdlib. Is this feature available on PyPI at the moment?

> This is a third party initiative. The translations are contributed by
> volunteers. I have talked with Python core developers and they suggested
> to post this here, as it is for them out of scope for Python std lib. At
> the moment there is no implementation yet. That is what I would like to
> discuss here.

Thanks for the clarification. I'd say that in that case, this should
probably be created as a 3rd party project on PyPI in the first
instance. If it becomes popular and is useful to a sufficiently large
user base, it could then be included in the stdlib. Your example code
uses a separate unicodedata_l10n package, and it would be very easy to
publish that on PyPI and later move it unchanged to the stdlib if
needed.

>> Also, would this not lead to non-English speakers expecting that the
>> localised names would work in "\N{...}" notation?

> Don't know. If an implementation has been made, it should be positioned
> very carefully.

Having this as a 3rd party project would make it much less likely that
users would expect \N support, IMO.

Paul

From davidfstr at gmail.com  Tue Jul 10 10:31:49 2018
From: davidfstr at gmail.com (David Foster)
Date: Tue, 10 Jul 2018 07:31:49 -0700
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <20180709161754.GA6167@trent.me>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <20180709161754.GA6167@trent.me>
Message-ID: <c00bf3ef-ef42-9b43-247a-5adf9d4826d9@gmail.com>

I was not aware of PyParallel. The PyParellel "parallel thread" 
line-of-execution implementation is pretty interesting. Trent, big kudos 
to you on that effort.

Since you're speaking in the past tense and said "but we're not doing it 
like that", I infer that the notion of a parallel thread was turned down 
for integration into CPython, as that appears to have been the original 
goal.

However I am unable to locate a rationale for why that integration was 
turned down. Was it deemed to be too complex to execute, perhaps in the 
context of providing C extension compatibility? Was there a desire to 
see a similar implementation on Linux as well as Windows? Some other 
reason? Since I presume you were directly involved in the discussions, 
perhaps you have a link to the relevant thread handy?

The last update I see from you RE PyParallel on this list is:
https://mail.python.org/pipermail/python-ideas/2015-September/035725.html

David Foster | Seattle, WA, USA

On 7/9/18 9:17 AM, Trent Nelson wrote:
> On Sun, Jul 08, 2018 at 11:27:08AM -0700, David Foster wrote:
> 
>> I'd like to solicit some feedback on what might be the most
>> efficient way to make forward progress on efficient parallelization
>> in Python inside the same OS process. The most promising areas
>> appear to be:
> 
> You might find PyParallel interesting, at least from a "here's what was
> tried, it worked, but we're not doing it like that" perspective.
> 
>      http://pyparallel.org
>      https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores
> 
> I still think it was a pretty successful proof-of-concept regarding
> removing the GIL without having to actually remove it.  Performance was
> pretty good too, as you can see in those graphs.
> 
>> -- 
>> David Foster | Seattle, WA, USA
> 
> Regards,
> 
>      Trent.
> 
> --
> https://trent.me
> 

From barry at barrys-emacs.org  Tue Jul 10 13:25:40 2018
From: barry at barrys-emacs.org (Barry)
Date: Tue, 10 Jul 2018 18:25:40 +0100
Subject: [Python-ideas] Calling python from C completely statically
In-Reply-To: <CA+fJkb=W=B=tiVDFOqRs3TLLYsaRRTPQDcCp9-UOcfKOzBVcEA@mail.gmail.com>
References: <CA+fJkbn5r_PVwJtS_-aZ9qGp0mqY2BytFAmQkPLLkU+ZvryfyA@mail.gmail.com>
 <CACac1F-J+bctMVEa7CcTn46JOpWXY7RrxEH_Lzt0U0YZ8phzuw@mail.gmail.com>
 <CADiSq7cbrZJiLtgw4Y_2e_U4LjuJ9J67hiorA=JB-0rFMjSYEg@mail.gmail.com>
 <CA+fJkbmC1A+R2ggCMa3h6y8eEg=i1-zy6yxBnecNNHqZdT5Z2w@mail.gmail.com>
 <CADiSq7dGY+F7XZreNd3_VDSBmv7+GQgoFbdTbBmMbKtUDXBTdw@mail.gmail.com>
 <CA+fJkb=66-Do3BxtPU6OPgVi7SJF44s_hn8xYpAcYE=O+oqSwQ@mail.gmail.com>
 <CA+fJkbkrRAmjvjOaUTsksDJawgv9O7zm2pkdKfiWu4eGFOXW_g@mail.gmail.com>
 <CA+fJkbkpbj=M+mm5DVHnj8KbjFA+yWUABKaB3=Uh32oo7VMGpw@mail.gmail.com>
 <05AECDA7-4AB6-4153-84E3-14638898E8CB@barrys-emacs.org>
 <CA+fJkb=W=B=tiVDFOqRs3TLLYsaRRTPQDcCp9-UOcfKOzBVcEA@mail.gmail.com>
Message-ID: <B239329B-6D1B-4C06-AE77-B2CDB0590E7B@barrys-emacs.org>



> On 9 Jul 2018, at 21:11, Alberto Garcia <agarciaillera at gmail.com> wrote:
> 
> Hi,
> 
> thank you for your response. I've downloaded the sources but I couldn't find any documentation. In addition I see that there is not a single C/C++ file. What I want to do is calling python from C.
> Am I missing something?

You need to grab the sources with an svn checkout. I guess you grabbed the binary kit.

https://sourceforge.net/p/meinc-installer/code/HEAD/tree/trunk/MEINC_Installer/

Docs are in here.

https://sourceforge.net/p/meinc-installer/code/HEAD/tree/trunk/MEINC_Installer/Installer/doc/

Barry

> Cheers
> 
> 
>> On Mon, Jul 9, 2018 at 12:20 PM Barry Scott <barry at barrys-emacs.org> wrote:
>> I think you might find Gordon McMillian's installer interesting to look at. It has a lot
>> f the tech that I think you are looking for.
>> 
>> Works up to python 2.7. I ended up taking over when Gordon stopped maintaining
>> it and kept it going up to python 2.7. In principle the same ideas could be made to
>> work in python 3 I believe.
>> 
>> 	https://sourceforge.net/projects/meinc-installer/
>> 
>> The zip file is appended to the end of the .EXE or unix ELF fie.
>> The boot strap knows how to import form the ZIP at the end of
>> the binary.
>> 
>> It also has a way to split out the .SO/.DLL files from the ZIP and
>> allow them to be loaded. Single EXE mode.
>> 
>> There are docs that explain how it works in the sources.
>> 
>> Barry
>> 
>> 
>> 
>>> On 9 Jul 2018, at 19:40, Alberto Garcia <agarciaillera at gmail.com> wrote:
>>> 
>>> Does the zip need to reside in disk to be loaded. Or can it be loaded from memory? I don't want it to be loaded from disk but from Memory
>>> 
>>> 
>>> 
>>>> On Mon, Jul 9, 2018 at 9:59 AM Alberto Garcia <agarciaillera at gmail.com> wrote:
>>>> Ohhhhhhhh  I guess you mean this:
>>>> https://github.com/anthony-tuininga/cx_Freeze/blob/master/source/bases/Common.c
>>>> 
>>>> Right?
>>>> 
>>>>> On Mon, Jul 9, 2018 at 9:48 AM Alberto Garcia <agarciaillera at gmail.com> wrote:
>>>>> Thank you for your response,
>>>>> 
>>>>> I was thinking on creating that zip file with the content of the Lib folder and having my c code to download it over the network and have it in memory. 
>>>>> I guess that the zip file should have no compression at all right?
>>>>> 
>>>>> When you say that I need to use the cx_freeze approach what do you mean? Can you point me to where they do that? 
>>>>> 
>>>>> And why changing sys.path again to the executable again? Which part of the executable? 
>>>>> 
>>>>> I'll put my efforts in this.
>>>>> 
>>>>> Thank you 
>>>>> 
>>>>>> On Mon, Jul 9, 2018 at 7:16 AM Nick Coghlan <ncoghlan at gmail.com> wrote:
>>>>>> On 9 July 2018 at 03:10, Alberto Garcia <agarciaillera at gmail.com> wrote:
>>>>>> > Hey there,
>>>>>> >
>>>>>> > Yes, the part of having the pyd modules built in in library is already done.
>>>>>> > I followed the instructions in the README. What I would like to know now is
>>>>>> > how to embed the non frozen python (py) modules. Can you guys please point
>>>>>> > me in the right direction.
>>>>>> 
>>>>>> The gist is to:
>>>>>> 
>>>>>> 1. take the entire Lib directory and put it in a zip archive
>>>>>> 2. use the approach demonstrated in cx_freeze to point sys.path in
>>>>>> your static executable at that zip archive
>>>>>> 3. adjust your C code to point sys.path back at the executable itself,
>>>>>> and then combine your executable and the zip archive into a single
>>>>>> contiguous file (similar to what zipapp does with it's helper script
>>>>>> and app archive)
>>>>>> 
>>>>>> There are likely to still be rough edges when doing that, since this
>>>>>> isn't a well tested configuration. When all else fails, find the part
>>>>>> of the source code responsible for any error messages you're seeing,
>>>>>> and try to work out if there's a setting you can tweak to avoid
>>>>>> hitting that code path.
>>>>>> 
>>>>>> Cheers,
>>>>>> Nick.
>>>>>> 
>>>>>> -- 
>>>>>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>>>>> -- 
>>>>> Alberto Garc?a Illera
>>>>> 
>>>>> GPG Public Key: https://goo.gl/twKUUv
>>>> -- 
>>>> Alberto Garc?a Illera
>>>> 
>>>> GPG Public Key: https://goo.gl/twKUUv
>>> 
>>> 
>>> -- 
>>> Alberto Garc?a Illera
>>> 
>>> GPG Public Key: https://goo.gl/twKUUv
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>> 
> 
> 
> -- 
> Alberto Garc?a Illera
> 
> GPG Public Key: https://goo.gl/twKUUv
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180710/3a74f1e1/attachment.html>

From barry at barrys-emacs.org  Tue Jul 10 15:45:55 2018
From: barry at barrys-emacs.org (Barry)
Date: Tue, 10 Jul 2018 20:45:55 +0100
Subject: [Python-ideas] Calling python from C completely statically
In-Reply-To: <CA+fJkbmYvDA5kj+49k65OCNSqkxRYVtpBJc13MNPw7FW7bOEtg@mail.gmail.com>
References: <CA+fJkbn5r_PVwJtS_-aZ9qGp0mqY2BytFAmQkPLLkU+ZvryfyA@mail.gmail.com>
 <CACac1F-J+bctMVEa7CcTn46JOpWXY7RrxEH_Lzt0U0YZ8phzuw@mail.gmail.com>
 <CADiSq7cbrZJiLtgw4Y_2e_U4LjuJ9J67hiorA=JB-0rFMjSYEg@mail.gmail.com>
 <CA+fJkbmC1A+R2ggCMa3h6y8eEg=i1-zy6yxBnecNNHqZdT5Z2w@mail.gmail.com>
 <CADiSq7dGY+F7XZreNd3_VDSBmv7+GQgoFbdTbBmMbKtUDXBTdw@mail.gmail.com>
 <CA+fJkb=66-Do3BxtPU6OPgVi7SJF44s_hn8xYpAcYE=O+oqSwQ@mail.gmail.com>
 <CA+fJkbkrRAmjvjOaUTsksDJawgv9O7zm2pkdKfiWu4eGFOXW_g@mail.gmail.com>
 <CA+fJkbkpbj=M+mm5DVHnj8KbjFA+yWUABKaB3=Uh32oo7VMGpw@mail.gmail.com>
 <05AECDA7-4AB6-4153-84E3-14638898E8CB@barrys-emacs.org>
 <CA+fJkb=W=B=tiVDFOqRs3TLLYsaRRTPQDcCp9-UOcfKOzBVcEA@mail.gmail.com>
 <B239329B-6D1B-4C06-AE77-B2CDB0590E7B@barrys-emacs.org>
 <CA+fJkbmYvDA5kj+49k65OCNSqkxRYVtpBJc13MNPw7FW7bOEtg@mail.gmail.com>
Message-ID: <D37A260A-8A7C-4097-A9A3-A3E9A87FA8A8@barrys-emacs.org>



> On 10 Jul 2018, at 19:12, Alberto Garcia <agarciaillera at gmail.com> wrote:
> 
> Thank you Barry,
> 
> but I see that the the way yo do it is dropping files to disk. I want to prevent any file to touch the disk except for the actual binary that has everything embedded in it.

What file are you refering to? The zip is built onto the elf/exe.

BTW this is all off topic for python ideas. You might want to take this offline.


Barry

> 
> Cheers
> 
>> On Tue, Jul 10, 2018 at 10:25 AM Barry <barry at barrys-emacs.org> wrote:
>> 
>> 
>>> On 9 Jul 2018, at 21:11, Alberto Garcia <agarciaillera at gmail.com> wrote:
>>> 
>>> Hi,
>>> 
>>> thank you for your response. I've downloaded the sources but I couldn't find any documentation. In addition I see that there is not a single C/C++ file. What I want to do is calling python from C.
>>> Am I missing something?
>> 
>> You need to grab the sources with an svn checkout. I guess you grabbed the binary kit.
>> 
>> https://sourceforge.net/p/meinc-installer/code/HEAD/tree/trunk/MEINC_Installer/
>> 
>> Docs are in here.
>> 
>> https://sourceforge.net/p/meinc-installer/code/HEAD/tree/trunk/MEINC_Installer/Installer/doc/
>> 
>> Barry
>> 
>>> Cheers
>>> 
>>> 
>>>> On Mon, Jul 9, 2018 at 12:20 PM Barry Scott <barry at barrys-emacs.org> wrote:
>>>> I think you might find Gordon McMillian's installer interesting to look at. It has a lot
>>>> f the tech that I think you are looking for.
>>>> 
>>>> Works up to python 2.7. I ended up taking over when Gordon stopped maintaining
>>>> it and kept it going up to python 2.7. In principle the same ideas could be made to
>>>> work in python 3 I believe.
>>>> 
>>>> 	https://sourceforge.net/projects/meinc-installer/
>>>> 
>>>> The zip file is appended to the end of the .EXE or unix ELF fie.
>>>> The boot strap knows how to import form the ZIP at the end of
>>>> the binary.
>>>> 
>>>> It also has a way to split out the .SO/.DLL files from the ZIP and
>>>> allow them to be loaded. Single EXE mode.
>>>> 
>>>> There are docs that explain how it works in the sources.
>>>> 
>>>> Barry
>>>> 
>>>> 
>>>> 
>>>>> On 9 Jul 2018, at 19:40, Alberto Garcia <agarciaillera at gmail.com> wrote:
>>>>> 
>>>>> Does the zip need to reside in disk to be loaded. Or can it be loaded from memory? I don't want it to be loaded from disk but from Memory
>>>>> 
>>>>> 
>>>>> 
>>>>>> On Mon, Jul 9, 2018 at 9:59 AM Alberto Garcia <agarciaillera at gmail.com> wrote:
>>>>>> Ohhhhhhhh  I guess you mean this:
>>>>>> https://github.com/anthony-tuininga/cx_Freeze/blob/master/source/bases/Common.c
>>>>>> 
>>>>>> Right?
>>>>>> 
>>>>>>> On Mon, Jul 9, 2018 at 9:48 AM Alberto Garcia <agarciaillera at gmail.com> wrote:
>>>>>>> Thank you for your response,
>>>>>>> 
>>>>>>> I was thinking on creating that zip file with the content of the Lib folder and having my c code to download it over the network and have it in memory. 
>>>>>>> I guess that the zip file should have no compression at all right?
>>>>>>> 
>>>>>>> When you say that I need to use the cx_freeze approach what do you mean? Can you point me to where they do that? 
>>>>>>> 
>>>>>>> And why changing sys.path again to the executable again? Which part of the executable? 
>>>>>>> 
>>>>>>> I'll put my efforts in this.
>>>>>>> 
>>>>>>> Thank you 
>>>>>>> 
>>>>>>>> On Mon, Jul 9, 2018 at 7:16 AM Nick Coghlan <ncoghlan at gmail.com> wrote:
>>>>>>>> On 9 July 2018 at 03:10, Alberto Garcia <agarciaillera at gmail.com> wrote:
>>>>>>>> > Hey there,
>>>>>>>> >
>>>>>>>> > Yes, the part of having the pyd modules built in in library is already done.
>>>>>>>> > I followed the instructions in the README. What I would like to know now is
>>>>>>>> > how to embed the non frozen python (py) modules. Can you guys please point
>>>>>>>> > me in the right direction.
>>>>>>>> 
>>>>>>>> The gist is to:
>>>>>>>> 
>>>>>>>> 1. take the entire Lib directory and put it in a zip archive
>>>>>>>> 2. use the approach demonstrated in cx_freeze to point sys.path in
>>>>>>>> your static executable at that zip archive
>>>>>>>> 3. adjust your C code to point sys.path back at the executable itself,
>>>>>>>> and then combine your executable and the zip archive into a single
>>>>>>>> contiguous file (similar to what zipapp does with it's helper script
>>>>>>>> and app archive)
>>>>>>>> 
>>>>>>>> There are likely to still be rough edges when doing that, since this
>>>>>>>> isn't a well tested configuration. When all else fails, find the part
>>>>>>>> of the source code responsible for any error messages you're seeing,
>>>>>>>> and try to work out if there's a setting you can tweak to avoid
>>>>>>>> hitting that code path.
>>>>>>>> 
>>>>>>>> Cheers,
>>>>>>>> Nick.
>>>>>>>> 
>>>>>>>> -- 
>>>>>>>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>>>>>>> -- 
>>>>>>> Alberto Garc?a Illera
>>>>>>> 
>>>>>>> GPG Public Key: https://goo.gl/twKUUv
>>>>>> -- 
>>>>>> Alberto Garc?a Illera
>>>>>> 
>>>>>> GPG Public Key: https://goo.gl/twKUUv
>>>>> 
>>>>> 
>>>>> -- 
>>>>> Alberto Garc?a Illera
>>>>> 
>>>>> GPG Public Key: https://goo.gl/twKUUv
>>>>> _______________________________________________
>>>>> Python-ideas mailing list
>>>>> Python-ideas at python.org
>>>>> https://mail.python.org/mailman/listinfo/python-ideas
>>>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>> 
>>> 
>>> 
>>> -- 
>>> Alberto Garc?a Illera
>>> 
>>> GPG Public Key: https://goo.gl/twKUUv
> 
> 
> -- 
> Alberto Garc?a Illera
> 
> GPG Public Key: https://goo.gl/twKUUv
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180710/b125c2df/attachment-0001.html>

From barry at python.org  Tue Jul 10 16:53:51 2018
From: barry at python.org (Barry Warsaw)
Date: Tue, 10 Jul 2018 13:53:51 -0700
Subject: [Python-ideas] Add new `Symbol` type
In-Reply-To: <CAP7+vJKPihPcWXJqB3V8EV3jAgXiL79YgDx_tpdY5txe+jYUQg@mail.gmail.com>
References: <CAAjMAJWeXOofJtz=OsU8OWq3QTZVsxEKfbmBeDivQrD=-pNZTA@mail.gmail.com>
 <CAFYqXL_DOyL5Rc76KZyjL4=J_ZdDZaHkktWfouC02M6=2BzSSw@mail.gmail.com>
 <CAAjMAJUzjfNr4dBQagVhKeF0TV8dPGtZqWbwhbAHM=xH10DCQg@mail.gmail.com>
 <CAP7+vJKPihPcWXJqB3V8EV3jAgXiL79YgDx_tpdY5txe+jYUQg@mail.gmail.com>
Message-ID: <pi3697$pqi$1@blaine.gmane.org>

Guido van Rossum wrote on 7/6/18 08:31:
> Thanks for an interesting discussion. I would also urge people to limit the
> use of such sentinels for cases where it is *really* important to
> distinguish between, say, f(None) and f(). In most cases using def
> f(arg=None) is fine, and often it is even a virtue that passing None or
> omitting an argument has exactly the same meaning. (I do know there a cases
> where this doesn't apply -- I just think they ought to be fairly unusual.)

One of the most common places I use a non-None sentinel is when None is 
a valid value in a dictionary:

_missing = object()

if mydict.get('foo', _missing) is _missing:
      # it ain't there

I generally don't feel like a more complicated repr is valuable here, so 
I haven't really wanted a built-in sentinel in a long time.  My search 
fu is weak today, but I'm pretty sure I suggested such a thing (and was 
rightly persuaded against it) many years ago.

what-goes-around-comes-around-ly y'rs,
-Barry



From tjreedy at udel.edu  Tue Jul 10 17:10:32 2018
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 10 Jul 2018 17:10:32 -0400
Subject: [Python-ideas] Support localization of unicode descriptions
In-Reply-To: <f466ac5a-eec0-fde5-4800-87257558471a@users.sourceforge.net>
References: <e91cca70-d9b9-5224-0a45-537ed308dd4c@users.sourceforge.net>
 <CACac1F8-50F40d2jqiJfz085jjTc3xpNgYHeWcsU6oiuaQgH9A@mail.gmail.com>
 <f466ac5a-eec0-fde5-4800-87257558471a@users.sourceforge.net>
Message-ID: <pi3787$r6c$1@blaine.gmane.org>

On 7/10/2018 4:45 AM, Pander wrote:

> This is a third party initiative. The translations are contributed by
> volunteers. I have talked with Python core developers and they suggested
> to post this here, as it is for them out of scope for Python std lib.

Python-ideas list is for discussion of python and the stdlib library. 
This is not a place for prolonged discussion of pypi projects.
It *is* a place to discuss adding a hook that can be used to access 
translations.

There are both official doc translations, accessible from the official 
doc pages, and others that are independent.  The official ones, at 
least, are discussed on the doc-sig list
https://mail.python.org/mailman/listinfo/doc-sig
There are currently 7 languages and coordinators listed at
https://devguide.python.org/experts/#documentation-translations
4 have progressed far enough to be listed in the drop-down box on
https://docs.python.org/3/

I should think that these people should be asked if they want to be 
involved with unicode description translations.  They should certainly 
have some helpful advice.

The description vocabulary is rather restricted, so a word translation 
dictionary should be pretty easy.  For at least for some languages, it 
should be possible to generate the 200000 description translations from 
this. The main issues are word order and language-dependent 'word' 
units.  Hence, the three English words "LATIN SMALL LETTER" become two 
words in German, 'LATEINISCHER KLEINBUCHSTABE', versus three words in 
Spanish, but in reverse order, 'LETRA PEQUE?A LATINA'.  It is possible 
that the doc translators already uses translation software that deal 
with these issues.

-- 
Terry Jan Reedy



From rosuav at gmail.com  Tue Jul 10 17:18:01 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 11 Jul 2018 07:18:01 +1000
Subject: [Python-ideas] Add new `Symbol` type
In-Reply-To: <pi3697$pqi$1@blaine.gmane.org>
References: <CAAjMAJWeXOofJtz=OsU8OWq3QTZVsxEKfbmBeDivQrD=-pNZTA@mail.gmail.com>
 <CAFYqXL_DOyL5Rc76KZyjL4=J_ZdDZaHkktWfouC02M6=2BzSSw@mail.gmail.com>
 <CAAjMAJUzjfNr4dBQagVhKeF0TV8dPGtZqWbwhbAHM=xH10DCQg@mail.gmail.com>
 <CAP7+vJKPihPcWXJqB3V8EV3jAgXiL79YgDx_tpdY5txe+jYUQg@mail.gmail.com>
 <pi3697$pqi$1@blaine.gmane.org>
Message-ID: <CAPTjJmrFNV4XYtzxk-yz6irX_eyMg48paTvRgCK0o6FtjD3NuA@mail.gmail.com>

On Wed, Jul 11, 2018 at 6:53 AM, Barry Warsaw <barry at python.org> wrote:
> Guido van Rossum wrote on 7/6/18 08:31:
>>
>> Thanks for an interesting discussion. I would also urge people to limit
>> the
>> use of such sentinels for cases where it is *really* important to
>> distinguish between, say, f(None) and f(). In most cases using def
>> f(arg=None) is fine, and often it is even a virtue that passing None or
>> omitting an argument has exactly the same meaning. (I do know there a
>> cases
>> where this doesn't apply -- I just think they ought to be fairly unusual.)
>
>
> One of the most common places I use a non-None sentinel is when None is a
> valid value in a dictionary:
>
> _missing = object()
>
> if mydict.get('foo', _missing) is _missing:
>      # it ain't there
>
> I generally don't feel like a more complicated repr is valuable here, so I
> haven't really wanted a built-in sentinel in a long time.  My search fu is
> weak today, but I'm pretty sure I suggested such a thing (and was rightly
> persuaded against it) many years ago.

For that specific example, I would just use:

try: mydict['foo']
except KeyError:
    # it ain't there

ChrisA

From tjreedy at udel.edu  Tue Jul 10 17:20:23 2018
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 10 Jul 2018 17:20:23 -0400
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <c00bf3ef-ef42-9b43-247a-5adf9d4826d9@gmail.com>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <20180709161754.GA6167@trent.me>
 <c00bf3ef-ef42-9b43-247a-5adf9d4826d9@gmail.com>
Message-ID: <pi37qm$37p$1@blaine.gmane.org>

On 7/10/2018 10:31 AM, David Foster wrote:

> Since you're speaking in the past tense and said "but we're not doing it 
> like that", I infer that the notion of a parallel thread was turned down 
> for integration into CPython, as that appears to have been the original 
> goal.

A far as I remember, there was never a formal proposal (PEP).  And I 
just searched PEP 0 for 'parallel'.  Hence, no formal rejection, 
rationale, or thread.

> However I am unable to locate a rationale for why that integration was 
> turned down. Was it deemed to be too complex to execute, perhaps in the 
> context of providing C extension compatibility? Was there a desire to 
> see a similar implementation on Linux as well as Windows? Some other 
> reason? Since I presume you were directly involved in the discussions, 
> perhaps you have a link to the relevant thread handy?

As always, there may have been private, off-the-record, informal 
discussions.

-- 
Terry Jan Reedy


From mertz at gnosis.cx  Tue Jul 10 17:20:51 2018
From: mertz at gnosis.cx (David Mertz)
Date: Tue, 10 Jul 2018 17:20:51 -0400
Subject: [Python-ideas] Support localization of unicode descriptions
In-Reply-To: <pi3787$r6c$1@blaine.gmane.org>
References: <e91cca70-d9b9-5224-0a45-537ed308dd4c@users.sourceforge.net>
 <CACac1F8-50F40d2jqiJfz085jjTc3xpNgYHeWcsU6oiuaQgH9A@mail.gmail.com>
 <f466ac5a-eec0-fde5-4800-87257558471a@users.sourceforge.net>
 <pi3787$r6c$1@blaine.gmane.org>
Message-ID: <CAEbHw4ZmtiTMAQBJWun0bBKmHTaj6YfbGVi5gdsWrMWwDfO7Hw@mail.gmail.com>

The problem with non-canonical translations of the Unicode character names
is that there is not one unique possible rendering into language X.
Equally, I could find synonyms in general English for the names, but one
would be official, the others at best informally clarifying.

For informational purposes I think it's great to have a third party project
to find out "Unicode character named 'Something In English' is roughly
translated as <whatever> in your native language." But it's hard to see how
an unofficial loose cross-language dictionary should be party of the
standard library.

On Tue, Jul 10, 2018, 5:11 PM Terry Reedy <tjreedy at udel.edu> wrote:

> On 7/10/2018 4:45 AM, Pander wrote:
>
> > This is a third party initiative. The translations are contributed by
> > volunteers. I have talked with Python core developers and they suggested
> > to post this here, as it is for them out of scope for Python std lib.
>
> Python-ideas list is for discussion of python and the stdlib library.
> This is not a place for prolonged discussion of pypi projects.
> It *is* a place to discuss adding a hook that can be used to access
> translations.
>
> There are both official doc translations, accessible from the official
> doc pages, and others that are independent.  The official ones, at
> least, are discussed on the doc-sig list
> https://mail.python.org/mailman/listinfo/doc-sig
> There are currently 7 languages and coordinators listed at
> https://devguide.python.org/experts/#documentation-translations
> 4 have progressed far enough to be listed in the drop-down box on
> https://docs.python.org/3/
>
> I should think that these people should be asked if they want to be
> involved with unicode description translations.  They should certainly
> have some helpful advice.
>
> The description vocabulary is rather restricted, so a word translation
> dictionary should be pretty easy.  For at least for some languages, it
> should be possible to generate the 200000 description translations from
> this. The main issues are word order and language-dependent 'word'
> units.  Hence, the three English words "LATIN SMALL LETTER" become two
> words in German, 'LATEINISCHER KLEINBUCHSTABE', versus three words in
> Spanish, but in reverse order, 'LETRA PEQUE?A LATINA'.  It is possible
> that the doc translators already uses translation software that deal
> with these issues.
>
> --
> Terry Jan Reedy
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180710/e48c23c8/attachment-0001.html>

From tjreedy at udel.edu  Tue Jul 10 19:14:48 2018
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 10 Jul 2018 19:14:48 -0400
Subject: [Python-ideas] Support localization of unicode descriptions
In-Reply-To: <CAEbHw4ZmtiTMAQBJWun0bBKmHTaj6YfbGVi5gdsWrMWwDfO7Hw@mail.gmail.com>
References: <e91cca70-d9b9-5224-0a45-537ed308dd4c@users.sourceforge.net>
 <CACac1F8-50F40d2jqiJfz085jjTc3xpNgYHeWcsU6oiuaQgH9A@mail.gmail.com>
 <f466ac5a-eec0-fde5-4800-87257558471a@users.sourceforge.net>
 <pi3787$r6c$1@blaine.gmane.org>
 <CAEbHw4ZmtiTMAQBJWun0bBKmHTaj6YfbGVi5gdsWrMWwDfO7Hw@mail.gmail.com>
Message-ID: <pi3eh5$5ag$1@blaine.gmane.org>

On 7/10/2018 5:20 PM, David Mertz wrote:
> The problem with non-canonical translations of the Unicode character 
> names is that there is not one unique possible rendering into language 
> X. Equally, I could find synonyms in general English for the names, but 
> one would be official, the others at best informally clarifying.

If the Unicode consortium does not provide official translations, then 
we *might* 'bless' some other source.  The first place I would look 
would be the translators we already trust enough to display their work 
on the official doc page.

> For informational purposes I think it's great to have a third party 
> project to find out "Unicode character named 'Something In English' is 
> roughly translated as <whatever> in your native language." But it's hard 
> to see how an unofficial loose cross-language dictionary should be party 
> of the standard library.

The doc translations are intentionally not in the cpython repository and 
not in the cpython distribution and not considered part of the stdlib. 
In general, core devs have no particular expertise, interest, or time to 
vet translators and review translations.

A proposal to make turtle a package and put translations of turtle 
commands in a submodule got no traction.  "Put it on Pypi"

I have rejected proposals to put translations of IDLE's menus in an 
idlelib subdirectory, to be distributed with cpython as 'part' of the 
stdlib.  I am thinking about various ideas to allow users to customize 
the menu, either by editing a file or processing a download.  (For 
instance, the Japanese doc translation includes the IDLE chapter, which 
has a list of menu items and descriptions.)  But this is a different issue.

The repository for unicode description translations should also be other 
than the cpython repository.

> On Tue, Jul 10, 2018, 5:11 PM Terry Reedy 
> <tjreedy at udel.edu 
> <mailto:tjreedy at udel.edu>> wrote:
> 
>     On 7/10/2018 4:45 AM, Pander wrote:
> 
>      > This is a third party initiative. The translations are contributed by
>      > volunteers. I have talked with Python core developers and they
>     suggested
>      > to post this here, as it is for them out of scope for Python std lib.
> 
>     Python-ideas list is for discussion of python and the stdlib library.
>     This is not a place for prolonged discussion of pypi projects.
>     It *is* a place to discuss adding a hook that can be used to access
>     translations.
> 
>     There are both official doc translations, accessible from the official
>     doc pages, and others that are independent.? The official ones, at
>     least, are discussed on the doc-sig list
>     https://mail.python.org/mailman/listinfo/doc-sig
>     There are currently 7 languages and coordinators listed at
>     https://devguide.python.org/experts/#documentation-translations
>     4 have progressed far enough to be listed in the drop-down box on
>     https://docs.python.org/3/
> 
>     I should think that these people should be asked if they want to be
>     involved with unicode description translations.? They should certainly
>     have some helpful advice.
> 
>     The description vocabulary is rather restricted, so a word translation
>     dictionary should be pretty easy.? For at least for some languages, it
>     should be possible to generate the 200000 description translations from
>     this. The main issues are word order and language-dependent 'word'
>     units.? Hence, the three English words "LATIN SMALL LETTER" become two
>     words in German, 'LATEINISCHER KLEINBUCHSTABE', versus three words in
>     Spanish, but in reverse order, 'LETRA PEQUE?A LATINA'.? It is possible
>     that the doc translators already uses translation software that deal
>     with these issues.


-- 
Terry Jan Reedy



From greg at krypto.org  Wed Jul 11 19:25:56 2018
From: greg at krypto.org (Gregory P. Smith)
Date: Wed, 11 Jul 2018 16:25:56 -0700
Subject: [Python-ideas] anyone need a frozenset or bytearray literal?
Message-ID: <CAGE7PNJzr8TnuQi6qY7Nr4842EUj8Q68T9gCxFCZMxqtPg9F6Q@mail.gmail.com>

Completely obvious what it does, but it irritates my aesthetic
sensibilities every time I see:
  frozenset({spam, eggs})

Why? Because I assume under the hood that creates a set of spam and eggs
before calling frozenset to copy it into a new frozenset object before the
original set is garbage collected.  Wasteful.  This is in fact what happens
in CPython 3.7 today.

I'd love to avoid this.  I have no rational measured reason to believe it
even matters (thus seeding this on python-ideas and not elsewhere), even
though it would technically speed up frozenset creation.

(a) detecting frozenset({}) as syntax to encode a frozenset in the python
bytecode would be somewhat magical.  it could break the person unfortunate
enough to monkeypatch out the frozenset builtin (really? don't do that!).

(b) abusing the concept of letter prefixes as we already have for strings
on {} syntax would be possible but not at all obvious to a reader:

  f{} or c{} or r{} perhaps.  but then someone would want a frozendict.

(c) adding a .freeze() method to sets which would raise an exception if the
set's refcount were > 1 and would mutate the type of the set object into a
frozenset object in place.  refcount assertions are bad, not all VMs need
refcounts.  The concept of a method that can mutate the type of the
underlying object in place is... unpythonic.  even though technically
possible to implement within CPython.

I'm -1 on all of my ideas above.  But figured I'd toss them out there as
food for thought for the concept.

We lived for years without even having a set literal in the first place. So
this isn't a big deal.

frozenset is not the only base type that lacks a literals leading to
loading values into these types involving creation of an intermediate
throwaway object: bytearray.  bytearray(b'short lived bytes object')

I was going to suggest complex was in this boat as well, but upon
investigation we appear to do constant folding (or amazing parsingon that
so 0.3+0.6j does turn into a single LOAD_CONST instead of two consts and an
addition.  Nice!  Not that I expect practical code to use complex numbers.

-gps
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180711/129f6775/attachment.html>

From robertvandeneynde at hotmail.com  Wed Jul 11 19:34:10 2018
From: robertvandeneynde at hotmail.com (Robert Vanden Eynde)
Date: Wed, 11 Jul 2018 23:34:10 +0000
Subject: [Python-ideas] anyone need a frozenset or bytearray literal?
In-Reply-To: <CAGE7PNJzr8TnuQi6qY7Nr4842EUj8Q68T9gCxFCZMxqtPg9F6Q@mail.gmail.com>
References: <CAGE7PNJzr8TnuQi6qY7Nr4842EUj8Q68T9gCxFCZMxqtPg9F6Q@mail.gmail.com>
Message-ID: <VI1PR03MB2909428CEDF4231CB23E6020B65A0@VI1PR03MB2909.eurprd03.prod.outlook.com>

I completely get your pain, the Copy seems like a waste of ressource.
However I think making an optimisation on the C-Level is better than introducing the litteral, because Python is a general purpose langauge and most of the appplication don't need frozenset or bytearrays and that would clutter the base elements one must know and introduce questions on beginners.

So yes, I'm +1 on your (a) solution if it is completely a C-Level optimisation. Your (c) solution is an idea. Again, C-Level optimisation, because usuability-wise, what's wrong with a copy of a hash table ?

The problem with the f{} syntax is that the letters come from nowhere, frozenset{'hello', 'world'} would be a better syntax but it looks like "{} is a slicing operator like frozenset[''hello', 'world'] would call frozetset.__getitem__( ('hello', 'world') ).

Le jeu. 12 juil. 2018 ? 01:26, Gregory P. Smith <greg at krypto.org<mailto:greg at krypto.org>> a ?crit :
Completely obvious what it does, but it irritates my aesthetic sensibilities every time I see:
  frozenset({spam, eggs})

Why? Because I assume under the hood that creates a set of spam and eggs before calling frozenset to copy it into a new frozenset object before the original set is garbage collected.  Wasteful.  This is in fact what happens in CPython 3.7 today.

I'd love to avoid this.  I have no rational measured reason to believe it even matters (thus seeding this on python-ideas and not elsewhere), even though it would technically speed up frozenset creation.

(a) detecting frozenset({}) as syntax to encode a frozenset in the python bytecode would be somewhat magical.  it could break the person unfortunate enough to monkeypatch out the frozenset builtin (really? don't do that!).

(b) abusing the concept of letter prefixes as we already have for strings on {} syntax would be possible but not at all obvious to a reader:

  f{} or c{} or r{} perhaps.  but then someone would want a frozendict.

(c) adding a .freeze() method to sets which would raise an exception if the set's refcount were > 1 and would mutate the type of the set object into a frozenset object in place.  refcount assertions are bad, not all VMs need refcounts.  The concept of a method that can mutate the type of the underlying object in place is... unpythonic.  even though technically possible to implement within CPython.

I'm -1 on all of my ideas above.  But figured I'd toss them out there as food for thought for the concept.

We lived for years without even having a set literal in the first place. So this isn't a big deal.

frozenset is not the only base type that lacks a literals leading to loading values into these types involving creation of an intermediate throwaway object: bytearray.  bytearray(b'short lived bytes object')

I was going to suggest complex was in this boat as well, but upon investigation we appear to do constant folding (or amazing parsingon that so 0.3+0.6j does turn into a single LOAD_CONST instead of two consts and an addition.  Nice!  Not that I expect practical code to use complex numbers.

-gps
_______________________________________________
Python-ideas mailing list
Python-ideas at python.org<mailto:Python-ideas at python.org>
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180711/219f15d0/attachment.html>

From jelle.zijlstra at gmail.com  Wed Jul 11 19:45:29 2018
From: jelle.zijlstra at gmail.com (Jelle Zijlstra)
Date: Wed, 11 Jul 2018 16:45:29 -0700
Subject: [Python-ideas] anyone need a frozenset or bytearray literal?
In-Reply-To: <CAGE7PNJzr8TnuQi6qY7Nr4842EUj8Q68T9gCxFCZMxqtPg9F6Q@mail.gmail.com>
References: <CAGE7PNJzr8TnuQi6qY7Nr4842EUj8Q68T9gCxFCZMxqtPg9F6Q@mail.gmail.com>
Message-ID: <CAFp3-p9ZL=CpOLjOvfQzVex=EK7H6OMTaHswwcHnXwi2wKVjzw@mail.gmail.com>

2018-07-11 16:25 GMT-07:00 Gregory P. Smith <greg at krypto.org>:

> Completely obvious what it does, but it irritates my aesthetic
> sensibilities every time I see:
>   frozenset({spam, eggs})
>
> Why? Because I assume under the hood that creates a set of spam and eggs
> before calling frozenset to copy it into a new frozenset object before the
> original set is garbage collected.  Wasteful.  This is in fact what happens
> in CPython 3.7 today.
>
> I'd love to avoid this.  I have no rational measured reason to believe it
> even matters (thus seeding this on python-ideas and not elsewhere), even
> though it would technically speed up frozenset creation.
>
> (a) detecting frozenset({}) as syntax to encode a frozenset in the python
> bytecode would be somewhat magical.  it could break the person unfortunate
> enough to monkeypatch out the frozenset builtin (really? don't do that!).
>
> (b) abusing the concept of letter prefixes as we already have for strings
> on {} syntax would be possible but not at all obvious to a reader:
>
>   f{} or c{} or r{} perhaps.  but then someone would want a frozendict.
>
> (c) adding a .freeze() method to sets which would raise an exception if
> the set's refcount were > 1 and would mutate the type of the set object
> into a frozenset object in place.  refcount assertions are bad, not all VMs
> need refcounts.  The concept of a method that can mutate the type of the
> underlying object in place is... unpythonic.  even though technically
> possible to implement within CPython.
>
> This could be done safely and without too much craziness if .freeze() on a
set returned a new frozenset. The compiler could then safely optimize {a,
set, literal}.freeze() into a frozenset literal, because methods on builtin
types cannot be monkeypatched. There's been talk of a similar optimization
on calls to .format() on string literals (not sure whether it's been
implemented).

Whether implementing that is a good use of anyone's time is a different
question.


> I'm -1 on all of my ideas above.  But figured I'd toss them out there as
> food for thought for the concept.
>
> We lived for years without even having a set literal in the first place.
> So this isn't a big deal.
>
> frozenset is not the only base type that lacks a literals leading to
> loading values into these types involving creation of an intermediate
> throwaway object: bytearray.  bytearray(b'short lived bytes object')
>
> I was going to suggest complex was in this boat as well, but upon
> investigation we appear to do constant folding (or amazing parsingon that
> so 0.3+0.6j does turn into a single LOAD_CONST instead of two consts and an
> addition.  Nice!  Not that I expect practical code to use complex numbers.
>
> -gps
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180711/a07f0be3/attachment-0001.html>

From greg at krypto.org  Wed Jul 11 20:13:33 2018
From: greg at krypto.org (Gregory P. Smith)
Date: Wed, 11 Jul 2018 17:13:33 -0700
Subject: [Python-ideas] anyone need a frozenset or bytearray literal?
In-Reply-To: <CAFp3-p9ZL=CpOLjOvfQzVex=EK7H6OMTaHswwcHnXwi2wKVjzw@mail.gmail.com>
References: <CAGE7PNJzr8TnuQi6qY7Nr4842EUj8Q68T9gCxFCZMxqtPg9F6Q@mail.gmail.com>
 <CAFp3-p9ZL=CpOLjOvfQzVex=EK7H6OMTaHswwcHnXwi2wKVjzw@mail.gmail.com>
Message-ID: <CAGE7PN+yH2a+PaupCcbfOs5JWYh8TO-8geUJAd7ucoam5beUcA@mail.gmail.com>

On Wed, Jul 11, 2018 at 4:45 PM Jelle Zijlstra <jelle.zijlstra at gmail.com>
wrote:

> 2018-07-11 16:25 GMT-07:00 Gregory P. Smith <greg at krypto.org>:
>
>> Completely obvious what it does, but it irritates my aesthetic
>> sensibilities every time I see:
>>   frozenset({spam, eggs})
>>
>> Why? Because I assume under the hood that creates a set of spam and eggs
>> before calling frozenset to copy it into a new frozenset object before the
>> original set is garbage collected.  Wasteful.  This is in fact what happens
>> in CPython 3.7 today.
>>
>> I'd love to avoid this.  I have no rational measured reason to believe it
>> even matters (thus seeding this on python-ideas and not elsewhere), even
>> though it would technically speed up frozenset creation.
>>
>> (a) detecting frozenset({}) as syntax to encode a frozenset in the python
>> bytecode would be somewhat magical.  it could break the person unfortunate
>> enough to monkeypatch out the frozenset builtin (really? don't do that!).
>>
>> (b) abusing the concept of letter prefixes as we already have for strings
>> on {} syntax would be possible but not at all obvious to a reader:
>>
>>   f{} or c{} or r{} perhaps.  but then someone would want a frozendict.
>>
>> (c) adding a .freeze() method to sets which would raise an exception if
>> the set's refcount were > 1 and would mutate the type of the set object
>> into a frozenset object in place.  refcount assertions are bad, not all VMs
>> need refcounts.  The concept of a method that can mutate the type of the
>> underlying object in place is... unpythonic.  even though technically
>> possible to implement within CPython.
>>
>> This could be done safely and without too much craziness if .freeze() on
> a set returned a new frozenset. The compiler could then safely optimize {a,
> set, literal}.freeze() into a frozenset literal, because methods on builtin
> types cannot be monkeypatched. There's been talk of a similar optimization
> on calls to .format() on string literals (not sure whether it's been
> implemented).
>
> Whether implementing that is a good use of anyone's time is a different
> question.
>

Neat optimization.  I hadn't considered that.  We do know for sure it is a
builtin type at that point.

If that were implemented, bytes objects could gain a to_bytearray() (along
the lines of the int.to_bytes() API) method that could be optimized away in
literal circumstances.

-gps


>
>> I'm -1 on all of my ideas above.  But figured I'd toss them out there as
>> food for thought for the concept.
>>
>> We lived for years without even having a set literal in the first place.
>> So this isn't a big deal.
>>
>> frozenset is not the only base type that lacks a literals leading to
>> loading values into these types involving creation of an intermediate
>> throwaway object: bytearray.  bytearray(b'short lived bytes object')
>>
>> I was going to suggest complex was in this boat as well, but upon
>> investigation we appear to do constant folding (or amazing parsingon that
>> so 0.3+0.6j does turn into a single LOAD_CONST instead of two consts and an
>> addition.  Nice!  Not that I expect practical code to use complex numbers.
>>
>> -gps
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180711/030c901a/attachment.html>

From rosuav at gmail.com  Wed Jul 11 20:24:34 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 12 Jul 2018 10:24:34 +1000
Subject: [Python-ideas] anyone need a frozenset or bytearray literal?
In-Reply-To: <CAGE7PN+yH2a+PaupCcbfOs5JWYh8TO-8geUJAd7ucoam5beUcA@mail.gmail.com>
References: <CAGE7PNJzr8TnuQi6qY7Nr4842EUj8Q68T9gCxFCZMxqtPg9F6Q@mail.gmail.com>
 <CAFp3-p9ZL=CpOLjOvfQzVex=EK7H6OMTaHswwcHnXwi2wKVjzw@mail.gmail.com>
 <CAGE7PN+yH2a+PaupCcbfOs5JWYh8TO-8geUJAd7ucoam5beUcA@mail.gmail.com>
Message-ID: <CAPTjJmpf10xdrhtFMSH3V-0j904kMpginUgMABRfitqPvUMChA@mail.gmail.com>

On Thu, Jul 12, 2018 at 10:13 AM, Gregory P. Smith <greg at krypto.org> wrote:
>
> On Wed, Jul 11, 2018 at 4:45 PM Jelle Zijlstra <jelle.zijlstra at gmail.com>
> wrote:
>> This could be done safely and without too much craziness if .freeze() on a
>> set returned a new frozenset. The compiler could then safely optimize {a,
>> set, literal}.freeze() into a frozenset literal, because methods on builtin
>> types cannot be monkeypatched. There's been talk of a similar optimization
>> on calls to .format() on string literals (not sure whether it's been
>> implemented).
>>
>> Whether implementing that is a good use of anyone's time is a different
>> question.
>
>
> Neat optimization.  I hadn't considered that.  We do know for sure it is a
> builtin type at that point.
>
> If that were implemented, bytes objects could gain a to_bytearray() (along
> the lines of the int.to_bytes() API) method that could be optimized away in
> literal circumstances.

Be careful: a bytearray is mutable, so this isn't open to very many
optimizations. A .freeze() method on sets would allow a set display to
become a frozenset "literal", stored as a constant on the
corresponding function object, the way a tuple is; but that's safe
because the frozenset doesn't need to concern itself with identity,
only value. Example:

def f(x):
    a = (1, 2, 3) # can be optimized
    b = (x, 4, 5) # cannot
    c = [6, 7, 8] # cannot

Disassemble this or look at f.__code__.co_consts and you'll see (1, 2,
3) as a single constant; but the others have to be built.

+1 on set.freeze(); +0 on bytes.to_bytearray().

ChrisA

From robertvandeneynde at hotmail.com  Wed Jul 11 21:03:02 2018
From: robertvandeneynde at hotmail.com (Robert Vanden Eynde)
Date: Thu, 12 Jul 2018 01:03:02 +0000
Subject: [Python-ideas] anyone need a frozenset or bytearray literal?
In-Reply-To: <CAPTjJmpf10xdrhtFMSH3V-0j904kMpginUgMABRfitqPvUMChA@mail.gmail.com>
References: <CAGE7PNJzr8TnuQi6qY7Nr4842EUj8Q68T9gCxFCZMxqtPg9F6Q@mail.gmail.com>
 <CAFp3-p9ZL=CpOLjOvfQzVex=EK7H6OMTaHswwcHnXwi2wKVjzw@mail.gmail.com>
 <CAGE7PN+yH2a+PaupCcbfOs5JWYh8TO-8geUJAd7ucoam5beUcA@mail.gmail.com>
 <CAPTjJmpf10xdrhtFMSH3V-0j904kMpginUgMABRfitqPvUMChA@mail.gmail.com>
Message-ID: <VI1PR03MB29090EC17F2964936FACBA3EB6590@VI1PR03MB2909.eurprd03.prod.outlook.com>

{1,2,7}.freeze() or {1,2,7}.to_frozenset() seem very natural and if this can be optimized to avoid the copy, it's perfect.
For bytearray, one use case would be to optimise bytearray([1,2,7,2]) in something like [1,2,7,2].to_byterray().
About bytes, one could have (1,2,7,2).to_bytes() instead of bytes((1,2,7,2)) because b'\x01\x02\x07\x02' is long and boring.
What about variables in the values {1,2,x}.freeze() should work too ?
bytes((1,2,7,x)) is not writable as a b string and creates a copy.


Le jeu. 12 juil. 2018 ? 02:24, Chris Angelico <rosuav at gmail.com<mailto:rosuav at gmail.com>> a ?crit :
On Thu, Jul 12, 2018 at 10:13 AM, Gregory P. Smith <greg at krypto.org<mailto:greg at krypto.org>> wrote:
>
> On Wed, Jul 11, 2018 at 4:45 PM Jelle Zijlstra <jelle.zijlstra at gmail.com<mailto:jelle.zijlstra at gmail.com>>
> wrote:
>> This could be done safely and without too much craziness if .freeze() on a
>> set returned a new frozenset. The compiler could then safely optimize {a,
>> set, literal}.freeze() into a frozenset literal, because methods on builtin
>> types cannot be monkeypatched. There's been talk of a similar optimization
>> on calls to .format() on string literals (not sure whether it's been
>> implemented).
>>
>> Whether implementing that is a good use of anyone's time is a different
>> question.
>
>
> Neat optimization.  I hadn't considered that.  We do know for sure it is a
> builtin type at that point.
>
> If that were implemented, bytes objects could gain a to_bytearray() (along
> the lines of the int.to_bytes() API) method that could be optimized away in
> literal circumstances.

Be careful: a bytearray is mutable, so this isn't open to very many
optimizations. A .freeze() method on sets would allow a set display to
become a frozenset "literal", stored as a constant on the
corresponding function object, the way a tuple is; but that's safe
because the frozenset doesn't need to concern itself with identity,
only value. Example:

def f(x):
    a = (1, 2, 3) # can be optimized
    b = (x, 4, 5) # cannot
    c = [6, 7, 8] # cannot

Disassemble this or look at f.__code__.co_consts and you'll see (1, 2,
3) as a single constant; but the others have to be built.

+1 on set.freeze(); +0 on bytes.to_bytearray().

ChrisA
_______________________________________________
Python-ideas mailing list
Python-ideas at python.org<mailto:Python-ideas at python.org>
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180712/1114d47c/attachment-0001.html>

From guido at python.org  Wed Jul 11 21:12:46 2018
From: guido at python.org (Guido van Rossum)
Date: Wed, 11 Jul 2018 18:12:46 -0700
Subject: [Python-ideas] anyone need a frozenset or bytearray literal?
In-Reply-To: <VI1PR03MB29090EC17F2964936FACBA3EB6590@VI1PR03MB2909.eurprd03.prod.outlook.com>
References: <CAGE7PNJzr8TnuQi6qY7Nr4842EUj8Q68T9gCxFCZMxqtPg9F6Q@mail.gmail.com>
 <CAFp3-p9ZL=CpOLjOvfQzVex=EK7H6OMTaHswwcHnXwi2wKVjzw@mail.gmail.com>
 <CAGE7PN+yH2a+PaupCcbfOs5JWYh8TO-8geUJAd7ucoam5beUcA@mail.gmail.com>
 <CAPTjJmpf10xdrhtFMSH3V-0j904kMpginUgMABRfitqPvUMChA@mail.gmail.com>
 <VI1PR03MB29090EC17F2964936FACBA3EB6590@VI1PR03MB2909.eurprd03.prod.outlook.com>
Message-ID: <CAP7+vJKeRy21GfUee=XMSOyyXjdJpB=PUPaxormapyx7xjoBhw@mail.gmail.com>

I know of many use cases for frozenset({...}) and I think a hack along
those lines is fine -- but what's the common use for bytearray(b"...") or
bytearray((...))? Almost invariably a bytearray is created empty or as a
given number of zeros. I certainly wouldn't want to burden the tuple type
with a to_bytearray() method, the types are unrelated (unlike set and
frozenset).

On Wed, Jul 11, 2018 at 6:03 PM, Robert Vanden Eynde <
robertvandeneynde at hotmail.com> wrote:

> {1,2,7}.freeze() or {1,2,7}.to_frozenset() seem very natural and if this
> can be optimized to avoid the copy, it's perfect.
> For bytearray, one use case would be to optimise bytearray([1,2,7,2]) in
> something like [1,2,7,2].to_byterray().
> About bytes, one could have (1,2,7,2).to_bytes() instead of
> bytes((1,2,7,2)) because b'\x01\x02\x07\x02' is long and boring.
> What about variables in the values {1,2,x}.freeze() should work too ?
> bytes((1,2,7,x)) is not writable as a b string and creates a copy.
>
>
> Le jeu. 12 juil. 2018 ? 02:24, Chris Angelico <rosuav at gmail.com> a ?crit :
>
>> On Thu, Jul 12, 2018 at 10:13 AM, Gregory P. Smith <greg at krypto.org>
>> wrote:
>> >
>> > On Wed, Jul 11, 2018 at 4:45 PM Jelle Zijlstra <
>> jelle.zijlstra at gmail.com>
>> > wrote:
>> >> This could be done safely and without too much craziness if .freeze()
>> on a
>> >> set returned a new frozenset. The compiler could then safely optimize
>> {a,
>> >> set, literal}.freeze() into a frozenset literal, because methods on
>> builtin
>> >> types cannot be monkeypatched. There's been talk of a similar
>> optimization
>> >> on calls to .format() on string literals (not sure whether it's been
>> >> implemented).
>> >>
>> >> Whether implementing that is a good use of anyone's time is a different
>> >> question.
>> >
>> >
>> > Neat optimization.  I hadn't considered that.  We do know for sure it
>> is a
>> > builtin type at that point.
>> >
>> > If that were implemented, bytes objects could gain a to_bytearray()
>> (along
>> > the lines of the int.to_bytes() API) method that could be optimized
>> away in
>> > literal circumstances.
>>
>> Be careful: a bytearray is mutable, so this isn't open to very many
>> optimizations. A .freeze() method on sets would allow a set display to
>> become a frozenset "literal", stored as a constant on the
>> corresponding function object, the way a tuple is; but that's safe
>> because the frozenset doesn't need to concern itself with identity,
>> only value. Example:
>>
>> def f(x):
>>     a = (1, 2, 3) # can be optimized
>>     b = (x, 4, 5) # cannot
>>     c = [6, 7, 8] # cannot
>>
>> Disassemble this or look at f.__code__.co_consts and you'll see (1, 2,
>> 3) as a single constant; but the others have to be built.
>>
>> +1 on set.freeze(); +0 on bytes.to_bytearray().
>>
>> ChrisA
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180711/f8a04ec3/attachment.html>

From robertvandeneynde at hotmail.com  Wed Jul 11 22:03:07 2018
From: robertvandeneynde at hotmail.com (Robert Vanden Eynde)
Date: Thu, 12 Jul 2018 02:03:07 +0000
Subject: [Python-ideas] Unicode Name Aliases keyword argument abbreviation
 in unicodedata.name for missing names
Message-ID: <VI1PR03MB29093F08C0489886B5D758E2B6590@VI1PR03MB2909.eurprd03.prod.outlook.com>

unicodedata.name<http://unicodedata.name> raises KeyError for a few unicode characters like '\0' or '\n', altough the documentation is very clear on the implementation, this is often not what people want, ie. a string describing the character.

In Python 3.3, the name aliases became accepted in unicodedata.lookup('NULL') and '\N{NULL}' == '\N{NUL}'.

One could expect that lookup(name(x)) == x for all unicode character but this property doesn't hold because of the few characters that do not have a name (mainly control characters).

The use case where the KeyError is raised when a codepoint for a unused character or newest version of unicode is however still useful.

In the NameAliases https://www.unicode.org/Public/6.3.0/ucd/NameAliases.txt one can see that some characters have multiple aliases, so there are multiple ways to map a character to a name.

I propose adding a keyword argument, to unicodedata.name<http://unicodedata.name> that would implement one of some useful behavior when the value does not exist. In that case.

One simple behavior would be to chose the name in the "abbreviation" list. Currently all characters except three only have one and only one abbreviation so that would be a good pick, so I'd imagine name('\x00', abbreviation=True) == 'NUL'

The three characters in NameAlias.txt that have more than one abbreviation are :

'\n' with  ['LF', 'NL', 'EOL']
'\t' with ['HT', 'TAB']
'\ufeff' with ['BOM', 'ZWNBSP']

In case multiple abbreviations exist, one could take the first introduced to unicode (for backward compability with python versions). If this is a tie, one could take the first in the list. If it has no name and no abbreviation, unicodata.name<http://unicodata.name> raises an error or returns default as usual.

lookup(name(x)) == x for all x is natural isn't it ?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180712/170dbaa2/attachment.html>

From greg at krypto.org  Thu Jul 12 01:33:53 2018
From: greg at krypto.org (Gregory P. Smith)
Date: Wed, 11 Jul 2018 22:33:53 -0700
Subject: [Python-ideas] anyone need a frozenset or bytearray literal?
In-Reply-To: <CAP7+vJKeRy21GfUee=XMSOyyXjdJpB=PUPaxormapyx7xjoBhw@mail.gmail.com>
References: <CAGE7PNJzr8TnuQi6qY7Nr4842EUj8Q68T9gCxFCZMxqtPg9F6Q@mail.gmail.com>
 <CAFp3-p9ZL=CpOLjOvfQzVex=EK7H6OMTaHswwcHnXwi2wKVjzw@mail.gmail.com>
 <CAGE7PN+yH2a+PaupCcbfOs5JWYh8TO-8geUJAd7ucoam5beUcA@mail.gmail.com>
 <CAPTjJmpf10xdrhtFMSH3V-0j904kMpginUgMABRfitqPvUMChA@mail.gmail.com>
 <VI1PR03MB29090EC17F2964936FACBA3EB6590@VI1PR03MB2909.eurprd03.prod.outlook.com>
 <CAP7+vJKeRy21GfUee=XMSOyyXjdJpB=PUPaxormapyx7xjoBhw@mail.gmail.com>
Message-ID: <CAGE7PNLK5p4wx40k2f72nUAA-nzzCpVtpumLpOFKRe-0Xb+vOQ@mail.gmail.com>

On Wed, Jul 11, 2018 at 6:13 PM Guido van Rossum <guido at python.org> wrote:

> I know of many use cases for frozenset({...}) and I think a hack along
> those lines is fine -- but what's the common use for bytearray(b"...") or
> bytearray((...))? Almost invariably a bytearray is created empty or as a
> given number of zeros. I certainly wouldn't want to burden the tuple type
> with a to_bytearray() method, the types are unrelated (unlike set and
> frozenset).
>

Agreed, bytearray(b'...') should be way less common.  I don't immediately
have a use for that beyond merely than disliking the copy from temporary
bytes object and gc behind the scenes.  I could find a practical use for it
in micropython where ram is extremely limited, but that VM could already
implement such a compile time optimization on its own.  The concept of the
optimization that'd be required just seemed similar to that of frozenset to
me.

frozenset is the one that led me down this train of thought as I was
looking at code declaring a bunch on constants.

-gps


> On Wed, Jul 11, 2018 at 6:03 PM, Robert Vanden Eynde <
> robertvandeneynde at hotmail.com> wrote:
>
>> {1,2,7}.freeze() or {1,2,7}.to_frozenset() seem very natural and if this
>> can be optimized to avoid the copy, it's perfect.
>> For bytearray, one use case would be to optimise bytearray([1,2,7,2]) in
>> something like [1,2,7,2].to_byterray().
>> About bytes, one could have (1,2,7,2).to_bytes() instead of
>> bytes((1,2,7,2)) because b'\x01\x02\x07\x02' is long and boring.
>> What about variables in the values {1,2,x}.freeze() should work too ?
>> bytes((1,2,7,x)) is not writable as a b string and creates a copy.
>>
>>
>> Le jeu. 12 juil. 2018 ? 02:24, Chris Angelico <rosuav at gmail.com> a
>> ?crit :
>>
>>> On Thu, Jul 12, 2018 at 10:13 AM, Gregory P. Smith <greg at krypto.org>
>>> wrote:
>>> >
>>> > On Wed, Jul 11, 2018 at 4:45 PM Jelle Zijlstra <
>>> jelle.zijlstra at gmail.com>
>>> > wrote:
>>> >> This could be done safely and without too much craziness if .freeze()
>>> on a
>>> >> set returned a new frozenset. The compiler could then safely optimize
>>> {a,
>>> >> set, literal}.freeze() into a frozenset literal, because methods on
>>> builtin
>>> >> types cannot be monkeypatched. There's been talk of a similar
>>> optimization
>>> >> on calls to .format() on string literals (not sure whether it's been
>>> >> implemented).
>>> >>
>>> >> Whether implementing that is a good use of anyone's time is a
>>> different
>>> >> question.
>>> >
>>> >
>>> > Neat optimization.  I hadn't considered that.  We do know for sure it
>>> is a
>>> > builtin type at that point.
>>> >
>>> > If that were implemented, bytes objects could gain a to_bytearray()
>>> (along
>>> > the lines of the int.to_bytes() API) method that could be optimized
>>> away in
>>> > literal circumstances.
>>>
>>> Be careful: a bytearray is mutable, so this isn't open to very many
>>> optimizations. A .freeze() method on sets would allow a set display to
>>> become a frozenset "literal", stored as a constant on the
>>> corresponding function object, the way a tuple is; but that's safe
>>> because the frozenset doesn't need to concern itself with identity,
>>> only value. Example:
>>>
>>> def f(x):
>>>     a = (1, 2, 3) # can be optimized
>>>     b = (x, 4, 5) # cannot
>>>     c = [6, 7, 8] # cannot
>>>
>>> Disassemble this or look at f.__code__.co_consts and you'll see (1, 2,
>>> 3) as a single constant; but the others have to be built.
>>>
>>> +1 on set.freeze(); +0 on bytes.to_bytearray().
>>>
>>> ChrisA
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180711/16acdc34/attachment-0001.html>

From stephanh42 at gmail.com  Thu Jul 12 03:19:13 2018
From: stephanh42 at gmail.com (Stephan Houben)
Date: Thu, 12 Jul 2018 09:19:13 +0200
Subject: [Python-ideas] anyone need a frozenset or bytearray literal?
In-Reply-To: <CAGE7PNLK5p4wx40k2f72nUAA-nzzCpVtpumLpOFKRe-0Xb+vOQ@mail.gmail.com>
References: <CAGE7PNJzr8TnuQi6qY7Nr4842EUj8Q68T9gCxFCZMxqtPg9F6Q@mail.gmail.com>
 <CAFp3-p9ZL=CpOLjOvfQzVex=EK7H6OMTaHswwcHnXwi2wKVjzw@mail.gmail.com>
 <CAGE7PN+yH2a+PaupCcbfOs5JWYh8TO-8geUJAd7ucoam5beUcA@mail.gmail.com>
 <CAPTjJmpf10xdrhtFMSH3V-0j904kMpginUgMABRfitqPvUMChA@mail.gmail.com>
 <VI1PR03MB29090EC17F2964936FACBA3EB6590@VI1PR03MB2909.eurprd03.prod.outlook.com>
 <CAP7+vJKeRy21GfUee=XMSOyyXjdJpB=PUPaxormapyx7xjoBhw@mail.gmail.com>
 <CAGE7PNLK5p4wx40k2f72nUAA-nzzCpVtpumLpOFKRe-0Xb+vOQ@mail.gmail.com>
Message-ID: <CAOOa=pOwWqOvDYYKjxVFqLypL9c8WBRc3Ut6yevNRu4J1h-9FQ@mail.gmail.com>

Hi all,

While we are at it, could

{"a":1, "b":2}.freeze()

perhaps create a MappingProxyType?

I've a gist at
https://gist.github.com/stephanh42/d277170dd8a3a2f026c272a4fda15396
with a stand-alone freeze function which attempts to convert objects to a
read-only
version, and dict -> MappingProxyType is one of the transforms.

Stephan


2018-07-12 7:33 GMT+02:00 Gregory P. Smith <greg at krypto.org>:

>
>
> On Wed, Jul 11, 2018 at 6:13 PM Guido van Rossum <guido at python.org> wrote:
>
>> I know of many use cases for frozenset({...}) and I think a hack along
>> those lines is fine -- but what's the common use for bytearray(b"...") or
>> bytearray((...))? Almost invariably a bytearray is created empty or as a
>> given number of zeros. I certainly wouldn't want to burden the tuple type
>> with a to_bytearray() method, the types are unrelated (unlike set and
>> frozenset).
>>
>
> Agreed, bytearray(b'...') should be way less common.  I don't immediately
> have a use for that beyond merely than disliking the copy from temporary
> bytes object and gc behind the scenes.  I could find a practical use for it
> in micropython where ram is extremely limited, but that VM could already
> implement such a compile time optimization on its own.  The concept of the
> optimization that'd be required just seemed similar to that of frozenset to
> me.
>
> frozenset is the one that led me down this train of thought as I was
> looking at code declaring a bunch on constants.
>
> -gps
>
>
>> On Wed, Jul 11, 2018 at 6:03 PM, Robert Vanden Eynde <
>> robertvandeneynde at hotmail.com> wrote:
>>
>>> {1,2,7}.freeze() or {1,2,7}.to_frozenset() seem very natural and if this
>>> can be optimized to avoid the copy, it's perfect.
>>> For bytearray, one use case would be to optimise bytearray([1,2,7,2]) in
>>> something like [1,2,7,2].to_byterray().
>>> About bytes, one could have (1,2,7,2).to_bytes() instead of
>>> bytes((1,2,7,2)) because b'\x01\x02\x07\x02' is long and boring.
>>> What about variables in the values {1,2,x}.freeze() should work too ?
>>> bytes((1,2,7,x)) is not writable as a b string and creates a copy.
>>>
>>>
>>> Le jeu. 12 juil. 2018 ? 02:24, Chris Angelico <rosuav at gmail.com> a
>>> ?crit :
>>>
>>>> On Thu, Jul 12, 2018 at 10:13 AM, Gregory P. Smith <greg at krypto.org>
>>>> wrote:
>>>> >
>>>> > On Wed, Jul 11, 2018 at 4:45 PM Jelle Zijlstra <
>>>> jelle.zijlstra at gmail.com>
>>>> > wrote:
>>>> >> This could be done safely and without too much craziness if
>>>> .freeze() on a
>>>> >> set returned a new frozenset. The compiler could then safely
>>>> optimize {a,
>>>> >> set, literal}.freeze() into a frozenset literal, because methods on
>>>> builtin
>>>> >> types cannot be monkeypatched. There's been talk of a similar
>>>> optimization
>>>> >> on calls to .format() on string literals (not sure whether it's been
>>>> >> implemented).
>>>> >>
>>>> >> Whether implementing that is a good use of anyone's time is a
>>>> different
>>>> >> question.
>>>> >
>>>> >
>>>> > Neat optimization.  I hadn't considered that.  We do know for sure it
>>>> is a
>>>> > builtin type at that point.
>>>> >
>>>> > If that were implemented, bytes objects could gain a to_bytearray()
>>>> (along
>>>> > the lines of the int.to_bytes() API) method that could be optimized
>>>> away in
>>>> > literal circumstances.
>>>>
>>>> Be careful: a bytearray is mutable, so this isn't open to very many
>>>> optimizations. A .freeze() method on sets would allow a set display to
>>>> become a frozenset "literal", stored as a constant on the
>>>> corresponding function object, the way a tuple is; but that's safe
>>>> because the frozenset doesn't need to concern itself with identity,
>>>> only value. Example:
>>>>
>>>> def f(x):
>>>>     a = (1, 2, 3) # can be optimized
>>>>     b = (x, 4, 5) # cannot
>>>>     c = [6, 7, 8] # cannot
>>>>
>>>> Disassemble this or look at f.__code__.co_consts and you'll see (1, 2,
>>>> 3) as a single constant; but the others have to be built.
>>>>
>>>> +1 on set.freeze(); +0 on bytes.to_bytearray().
>>>>
>>>> ChrisA
>>>> _______________________________________________
>>>> Python-ideas mailing list
>>>> Python-ideas at python.org
>>>> https://mail.python.org/mailman/listinfo/python-ideas
>>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>>
>>>
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180712/7a3d6955/attachment.html>

From steve at pearwood.info  Thu Jul 12 03:17:26 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 12 Jul 2018 17:17:26 +1000
Subject: [Python-ideas] Unicode Name Aliases keyword argument
 abbreviation in unicodedata.name for missing names
In-Reply-To: <VI1PR03MB29093F08C0489886B5D758E2B6590@VI1PR03MB2909.eurprd03.prod.outlook.com>
References: <VI1PR03MB29093F08C0489886B5D758E2B6590@VI1PR03MB2909.eurprd03.prod.outlook.com>
Message-ID: <20180712071725.GR7318@ando.pearwood.info>

Replying to a few points out of order...

On Thu, Jul 12, 2018 at 02:03:07AM +0000, Robert Vanden Eynde wrote:

> lookup(name(x)) == x for all x is natural isn't it ?

The Unicode Consortium doesn't think so, or else they would mandate that 
all defined code points have a name.


> In the NameAliases 
> https://www.unicode.org/Public/6.3.0/ucd/NameAliases.txt
> one can see that some characters have multiple aliases, so there are 
> multiple ways to map a character to a name.

That's a pretty old version -- we're up to version 11 now.

https://www.unicode.org/Public/11.0.0/ucd/NameAliases.txt


> I propose adding a keyword argument, to 
> unicodedata.name<http://unicodedata.name> 

I don't think that's a real URL.


> that would implement one of some useful behavior when the value does 
> not exist.

I am cautious about overloading functions with keyword-only arguments to 
implement special behaviour. Guido has a strong preference for the "no 
constant flags" rule of thumb, (except I think we can extend it beyond 
just True/False to any N-state value) and I agree with that.

The rule of thumb says that if you have a function that takes an 
optional flag which chooses between two (or more) distinct behaviours, 
AND the function is usually called with that flag given as a constant, 
then we should usually prefer to split the function into two separately 
named functions.

For example, in the statistics module, I have stdev() and pstdev(), 
rather than stdev(population=False) and stdev(population=True).

(Its a rule of thumb, not a hard law of nature. There are exceptions.)

It sounds to me that your proposal would fit those conditions and so we 
should prefer a separate function, or a separate API, for doing more 
complex name look-ups.

*Especially* if there's a chance that we'll want to extend this some day 
to use more flags...

name(char, 
     abbreviation=False,
     correction=True,
     control=True,
     figment=True,
     alternate=False,
     )

which are all alias types defined by NameAliases.txt.


> One simple behavior would be to chose the name in the "abbreviation" 
> list. Currently all characters except three only have one and only one 
> abbreviation so that would be a good pick, so I'd imagine name('\x00', 
> abbreviation=True) == 'NUL'

To my mind, that calls out for a separate API to return character alias 
properties as a separate data type:

alias('\u0001')
=> UnicodeAlias(control='START OF HEADING', abbreviation='SOH')

alias('\u000B')
=> UnicodeAlias(control=('LINE TABULATION', 'VERTICAL TABULATION'),
                abbreviation='VT')

# alternatively, fields could be a single semi-colon delimited 
string rather than a tuple in the event of multiple aliases

alias('\u01A2')
=> UnicodeAlias(correction='LATIN CAPITAL LETTER GHA')

alias('\u0099')
=> UnicodeAlias(figment='SINGLE GRAPHIC CHARACTER INTRODUCER', 
                abbreviation='SGC')


Fields not shown return the empty string.

This avoids overloading the name() function, future-proofs against new 
alias types, and if UnicodeAlias is a mutable object, easily permits the 
caller to customise the records to suit their own application's needs:

def myalias(char):
    alias = unicodedata.alias(char)
    if char == '\U0001f346':
        alias.other = ('eggplant', 'purple vegetable')
        alias.slang = ('phallic', ... )
    return alias


-- 
Steve

From storchaka at gmail.com  Thu Jul 12 03:37:57 2018
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 12 Jul 2018 10:37:57 +0300
Subject: [Python-ideas] anyone need a frozenset or bytearray literal?
In-Reply-To: <CAGE7PNLK5p4wx40k2f72nUAA-nzzCpVtpumLpOFKRe-0Xb+vOQ@mail.gmail.com>
References: <CAGE7PNJzr8TnuQi6qY7Nr4842EUj8Q68T9gCxFCZMxqtPg9F6Q@mail.gmail.com>
 <CAFp3-p9ZL=CpOLjOvfQzVex=EK7H6OMTaHswwcHnXwi2wKVjzw@mail.gmail.com>
 <CAGE7PN+yH2a+PaupCcbfOs5JWYh8TO-8geUJAd7ucoam5beUcA@mail.gmail.com>
 <CAPTjJmpf10xdrhtFMSH3V-0j904kMpginUgMABRfitqPvUMChA@mail.gmail.com>
 <VI1PR03MB29090EC17F2964936FACBA3EB6590@VI1PR03MB2909.eurprd03.prod.outlook.com>
 <CAP7+vJKeRy21GfUee=XMSOyyXjdJpB=PUPaxormapyx7xjoBhw@mail.gmail.com>
 <CAGE7PNLK5p4wx40k2f72nUAA-nzzCpVtpumLpOFKRe-0Xb+vOQ@mail.gmail.com>
Message-ID: <pi70ci$jph$1@blaine.gmane.org>

12.07.18 08:33, Gregory P. Smith ????:
> Agreed, bytearray(b'...') should be way less common.? I don't 
> immediately have a use for that beyond merely than disliking the copy 
> from temporary bytes object and gc behind the scenes.

You can't avoid this since bytearray is mutable. The constant bytes 
argument can be shared, but the content of a new bytearray needs to be 
copied.

a = b'abc'
b = bytearray(b'abc')  # should make a copy
c = bytearray(b'abc')  # should make a copy
b[0] = 0
assert c[0] == 97

Although there is a possibility to apply in bytearray the same 
optimization as was made in BytesIO. The bytearray object can use an 
internal mutable bytes object for storing a content instead of a raw 
array. The constructor can save a reference to the passed bytes object, 
this is O(1) operation. bytes(bytearray) could just return a reference 
to that bytes object, it is O(1) too. Any mutating operation should 
check the refcount and make a copy if it is not 1. This will complicate 
the code, and I'm not sure if it is worth.


From storchaka at gmail.com  Thu Jul 12 03:41:48 2018
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 12 Jul 2018 10:41:48 +0300
Subject: [Python-ideas] anyone need a frozenset or bytearray literal?
In-Reply-To: <CAGE7PNJzr8TnuQi6qY7Nr4842EUj8Q68T9gCxFCZMxqtPg9F6Q@mail.gmail.com>
References: <CAGE7PNJzr8TnuQi6qY7Nr4842EUj8Q68T9gCxFCZMxqtPg9F6Q@mail.gmail.com>
Message-ID: <pi70jp$28h$1@blaine.gmane.org>

12.07.18 02:25, Gregory P. Smith ????:
> (c) adding a .freeze() method to sets which would raise an exception if 
> the set's refcount were > 1 and would mutate the type of the set object 
> into a frozenset object in place.? refcount assertions are bad, not all 
> VMs need refcounts.? The concept of a method that can mutate the type of 
> the underlying object in place is... unpythonic.? even though 
> technically possible to implement within CPython.

It is just an implementation detail that set and frozenset use the same 
internal representation, and just have different types. But it is 
possible to implement more efficient representation for frozensets. I 
have some ideas and could implement them if Raymond allows.


From steve at pearwood.info  Thu Jul 12 05:51:46 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 12 Jul 2018 19:51:46 +1000
Subject: [Python-ideas] anyone need a frozenset or bytearray literal?
In-Reply-To: <CAGE7PNJzr8TnuQi6qY7Nr4842EUj8Q68T9gCxFCZMxqtPg9F6Q@mail.gmail.com>
References: <CAGE7PNJzr8TnuQi6qY7Nr4842EUj8Q68T9gCxFCZMxqtPg9F6Q@mail.gmail.com>
Message-ID: <20180712095146.GU7318@ando.pearwood.info>

On Wed, Jul 11, 2018 at 04:25:56PM -0700, Gregory P. Smith wrote:
> Completely obvious what it does, but it irritates my aesthetic
> sensibilities every time I see:
>   frozenset({spam, eggs})

+1 to the idea of a new set method, freeze, returning a frozenset, and 
allowing the interpreter to optimize calls like:

   {a, b, c}.freeze() 

to skip making the temporary set.

I seem to have a vague recollection that there was already a CPython 
optimization in place that substitutes a frozenset for certain set 
displays... ah yes, here it is:

py> dis.dis("x in {2, 3}")
  1           0 LOAD_NAME                0 (x)
              3 LOAD_CONST               2 (frozenset({2, 3}))
              6 COMPARE_OP               6 (in)
              9 RETURN_VALUE

Nice!



-- 
Steve

From phd at phdru.name  Thu Jul 12 06:37:39 2018
From: phd at phdru.name (Oleg Broytman)
Date: Thu, 12 Jul 2018 12:37:39 +0200
Subject: [Python-ideas] Unicode Name Aliases keyword argument
 abbreviation in unicodedata.name for missing names
In-Reply-To: <20180712071725.GR7318@ando.pearwood.info>
References: <VI1PR03MB29093F08C0489886B5D758E2B6590@VI1PR03MB2909.eurprd03.prod.outlook.com>
 <20180712071725.GR7318@ando.pearwood.info>
Message-ID: <20180712103739.6hxsoyltxvjbsn4d@phdru.name>

On Thu, Jul 12, 2018 at 05:17:26PM +1000, Steven D'Aprano <steve at pearwood.info> wrote:
> > I propose adding a keyword argument, to 
> > unicodedata.name<http://unicodedata.name> 
> 
> I don't think that's a real URL.

   I'm sure it was a stupid autoreplacement by web mail (hotmail in this
case). As '.name' is a valid domain hotmail decided that
unicodedata.name is a host name. And "URLified" it, so to say.

> -- 
> Steve

Oleg.
-- 
    Oleg Broytman            https://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From storchaka at gmail.com  Thu Jul 12 07:55:58 2018
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 12 Jul 2018 14:55:58 +0300
Subject: [Python-ideas] Add the imath module
Message-ID: <pi7fga$flg$1@blaine.gmane.org>

What are your thoughts about adding a new imath module for integer 
mathematics? It could contain the following functions:

* factorial(n)

Is just moved from the math module, but non-integer types are rejected. 
Currently math.factorial() accepts also integer floats like 3.0. It 
looks to me, the rationale was that at the time when math.factorial() 
was added, all function in the math module worked with floats. But now 
we can revise this decision.

* gcd(n, m)

Is just moved from the math module.

* as_integer_ration(x)

Equivalents to:

def as_integer_ration(x):
     if hasattr(x, 'as_integer_ration'):
         return x.as_integer_ration()
     else:
         return (x.numerator, x.denominator)

* binom(n, k)

Returns factorial(n) // (factorial(k) * factorial(n-k)), but uses more 
efficient algorithm.

* sqrt(n)

Returns the largest integer r such that r**2 <= n and (r+1)**2 > n.

* isprime(n)

Tests if n is a prime number.

* primes()

Returns an iterator of prime numbers: 2, 3, 5, 7, 11, 13,...

Are there more ideas?


From J.Demeyer at UGent.be  Thu Jul 12 08:14:38 2018
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Thu, 12 Jul 2018 14:14:38 +0200
Subject: [Python-ideas] Add the imath module
In-Reply-To: <ebd5c102cd714c10b0efc8cd9086c3fe@xmail103.UGent.be>
References: <ebd5c102cd714c10b0efc8cd9086c3fe@xmail103.UGent.be>
Message-ID: <5B47462E.80509@UGent.be>

If you want inspiration:
https://github.com/sagemath/sage/blob/master/src/sage/arith/misc.py

From robertvandeneynde at hotmail.com  Thu Jul 12 08:15:50 2018
From: robertvandeneynde at hotmail.com (Robert Vanden Eynde)
Date: Thu, 12 Jul 2018 12:15:50 +0000
Subject: [Python-ideas] Unicode Name Aliases keyword argument
 abbreviation in unicodedata.name for missing names
In-Reply-To: <20180712103739.6hxsoyltxvjbsn4d@phdru.name>
References: <VI1PR03MB29093F08C0489886B5D758E2B6590@VI1PR03MB2909.eurprd03.prod.outlook.com>
 <20180712071725.GR7318@ando.pearwood.info>
 <20180712103739.6hxsoyltxvjbsn4d@phdru.name>
Message-ID: <VI1PR03MB2909445BDBF24848ECF8029AB6590@VI1PR03MB2909.eurprd03.prod.outlook.com>

Yes, my gmail client transformed unicodata . name to a url. I hope the mobile gmail client won't do it here.

Yes current version is 11. I noticed it after sending the mail, I've compared to the version 6 and all my arguments are still valid (they just added some characters in the "correction" set).

As I'm at, I mentionned the ffef character but we don't care about it because it already has a name, so that's mostly a control character issue.

Yes a new function name is also what I prefer but I thought it would clutter the unicodata namespace.

I like your alias(...) function, with that one, an application could code my function like try name(x) expect alias(x).abbreviations[0]. If the abbreviation list is sorted by AdditionToUnicodeDate.

However, having a standard canonical name for all character in the stdlib would help people choosing the same convention. A new function like "canonical_name" or a shorter name would be an idea.

Instead of name(char, abbreviation=True, correction=False) I would have Imagined a "default_behavior" ala csv.dialect such that name(char, default_bevior=unicodata.first_abbreviation) would use my algorithm. first_abbreviation would be a enum, or like in csv.dialect a class like : class first_abbreviation: abbreviation = True; correction = False; ...

But I guess that's too specific, abbreviation=True would mean "take the first abbreviation in the list".
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180712/b9d4154c/attachment-0001.html>

From mertz at gnosis.cx  Thu Jul 12 08:15:39 2018
From: mertz at gnosis.cx (David Mertz)
Date: Thu, 12 Jul 2018 08:15:39 -0400
Subject: [Python-ideas] Add the imath module
In-Reply-To: <pi7fga$flg$1@blaine.gmane.org>
References: <pi7fga$flg$1@blaine.gmane.org>
Message-ID: <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>

On Thu, Jul 12, 2018, 7:56 AM Serhiy Storchaka <storchaka at gmail.com> wrote:

> * isprime(n)
> Tests if n is a prime number.
>

How would you test this? The Miller-Rabin Primality test? For large
numbers, iterating through candidate prime divisors is pricey.

* primes()
> Returns an iterator of prime numbers: 2, 3, 5, 7, 11, 13,...
>

How would you implements this? Sieve of Eratosthenes is really fun to show
students as a Python generator function. But the cached primes DO grow
unboundedly as you utilize the generator. Wheel factorization as first
pass? Do you cached the first N primes so the each instance of iterator can
provide initial elements much faster? What is N?

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180712/7cca313f/attachment.html>

From dmoisset at machinalis.com  Thu Jul 12 08:19:10 2018
From: dmoisset at machinalis.com (Daniel Moisset)
Date: Thu, 12 Jul 2018 13:19:10 +0100
Subject: [Python-ideas] Add the imath module
In-Reply-To: <pi7fga$flg$1@blaine.gmane.org>
References: <pi7fga$flg$1@blaine.gmane.org>
Message-ID: <CALuYSZX42uSTC2fk-e-yeQPFAsVMrY3-c6=-_xorUNx9cz7Vuw@mail.gmail.com>

primefactors(n): an iterator on the primes that evenly divide n (repeating
such as the product is n

(I don't have a good name): something telling that an integer can be
represented exactly as a float

On 12 July 2018 at 12:55, Serhiy Storchaka <storchaka at gmail.com> wrote:

> What are your thoughts about adding a new imath module for integer
> mathematics? It could contain the following functions:
>
> * factorial(n)
>
> Is just moved from the math module, but non-integer types are rejected.
> Currently math.factorial() accepts also integer floats like 3.0. It looks
> to me, the rationale was that at the time when math.factorial() was added,
> all function in the math module worked with floats. But now we can revise
> this decision.
>
> * gcd(n, m)
>
> Is just moved from the math module.
>
> * as_integer_ration(x)
>
> Equivalents to:
>
> def as_integer_ration(x):
>     if hasattr(x, 'as_integer_ration'):
>         return x.as_integer_ration()
>     else:
>         return (x.numerator, x.denominator)
>
> * binom(n, k)
>
> Returns factorial(n) // (factorial(k) * factorial(n-k)), but uses more
> efficient algorithm.
>
> * sqrt(n)
>
> Returns the largest integer r such that r**2 <= n and (r+1)**2 > n.
>
> * isprime(n)
>
> Tests if n is a prime number.
>
> * primes()
>
> Returns an iterator of prime numbers: 2, 3, 5, 7, 11, 13,...
>
> Are there more ideas?
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
<https://www.machinalis.com/>
Daniel Moisset
Technical Leader

A:   1 Fore St, EC2Y 9DT London <https://goo.gl/maps/pH9BBLgE8dG2>
P:   +44 7398 827139 <+44+7398+827139>
M:   dmoisset at machinalis.com <dmoisset at machinalis.com>  |   S:   dmoisset
<http://www.linkedin.com/company/456525>
<http://www.twitter.com/machinalis>  <http://www.facebook.com/machinalis>
<https://www.instagram.com/machinalis.life/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180712/d827bf1a/attachment.html>

From robertvandeneynde at hotmail.com  Thu Jul 12 08:33:36 2018
From: robertvandeneynde at hotmail.com (Robert Vanden Eynde)
Date: Thu, 12 Jul 2018 12:33:36 +0000
Subject: [Python-ideas] Unicode Name Aliases keyword argument
 abbreviation in unicodedata.name for missing names
In-Reply-To: <VI1PR03MB2909445BDBF24848ECF8029AB6590@VI1PR03MB2909.eurprd03.prod.outlook.com>
References: <VI1PR03MB29093F08C0489886B5D758E2B6590@VI1PR03MB2909.eurprd03.prod.outlook.com>
 <20180712071725.GR7318@ando.pearwood.info>
 <20180712103739.6hxsoyltxvjbsn4d@phdru.name>
 <VI1PR03MB2909445BDBF24848ECF8029AB6590@VI1PR03MB2909.eurprd03.prod.outlook.com>
Message-ID: <VI1PR03MB29095F52C1CB4F80FC2E74E1B6590@VI1PR03MB2909.eurprd03.prod.outlook.com>


I like your alias(...) function, with that one, an application could code my function like try name(x) expect alias(x).abbreviations[0]. If the abbreviation list is sorted by AdditionToUnicodeDate.

Or

try:
    return name(x)
expect:
     if category(x) == 'Cc':
        return alias(x).abbreviations[0]
    else:
        raise

That would then raise only for unassigned codepoints.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180712/4106231f/attachment-0001.html>

From jeanpierreda at gmail.com  Thu Jul 12 08:39:30 2018
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Thu, 12 Jul 2018 05:39:30 -0700
Subject: [Python-ideas] Add the imath module
In-Reply-To: <CALuYSZX42uSTC2fk-e-yeQPFAsVMrY3-c6=-_xorUNx9cz7Vuw@mail.gmail.com>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CALuYSZX42uSTC2fk-e-yeQPFAsVMrY3-c6=-_xorUNx9cz7Vuw@mail.gmail.com>
Message-ID: <CABicbJKum0pv89asp-kP3x24Bqp27DG3353cbBVvX4k40diRDg@mail.gmail.com>

On Thu, Jul 12, 2018 at 5:20 AM Daniel Moisset <dmoisset at machinalis.com> wrote:
> (I don't have a good name): something telling that an integer can be represented exactly as a float

One might also ask that question of e.g. decimal.Decimal,
fractions.Fraction, so maybe it's better as a method or somewhere
else.

(Is this any different than float(x).as_integer_ratio() ==
(x.numerator, x.denominator) for ints/fractions?)

-- Devin

From storchaka at gmail.com  Thu Jul 12 09:11:33 2018
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 12 Jul 2018 16:11:33 +0300
Subject: [Python-ideas] Add the imath module
In-Reply-To: <pi7fga$flg$1@blaine.gmane.org>
References: <pi7fga$flg$1@blaine.gmane.org>
Message-ID: <pi7ju2$7ut$1@blaine.gmane.org>

12.07.18 14:55, Serhiy Storchaka ????:
> What are your thoughts about adding a new imath module for integer 
> mathematics? It could contain the following functions:

I forgot yet one useful function:

def divide_and_round(a, b):
     q, r = divmod(a, b)
     r *= 2
     greater_than_half = r > b if b > 0 else r < b
     if greater_than_half or r == b and q % 2 == 1:
         q += 1
     return q

(This is a copy of the implementation from datetime.py, there are yet 
few implementations in Python and C in other files.)


From robertve92 at gmail.com  Thu Jul 12 09:15:38 2018
From: robertve92 at gmail.com (Robert Vanden Eynde)
Date: Thu, 12 Jul 2018 15:15:38 +0200
Subject: [Python-ideas] Add the imath module
In-Reply-To: <pi7ju2$7ut$1@blaine.gmane.org>
References: <pi7fga$flg$1@blaine.gmane.org> <pi7ju2$7ut$1@blaine.gmane.org>
Message-ID: <CA+msPNk_wxKsHkv7nbWk5Kgn2HBoT+Mz9s6gvYeabaJqyGcxqA@mail.gmail.com>

About the name, why not intmath ?
And why not using sage ?
Or create a package on pypi ?

Le jeu. 12 juil. 2018 ? 15:11, Serhiy Storchaka <storchaka at gmail.com> a
?crit :

> 12.07.18 14:55, Serhiy Storchaka ????:
> > What are your thoughts about adding a new imath module for integer
> > mathematics? It could contain the following functions:
>
> I forgot yet one useful function:
>
> def divide_and_round(a, b):
>      q, r = divmod(a, b)
>      r *= 2
>      greater_than_half = r > b if b > 0 else r < b
>      if greater_than_half or r == b and q % 2 == 1:
>          q += 1
>      return q
>
> (This is a copy of the implementation from datetime.py, there are yet
> few implementations in Python and C in other files.)
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180712/a78758f0/attachment.html>

From storchaka at gmail.com  Thu Jul 12 09:30:33 2018
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 12 Jul 2018 16:30:33 +0300
Subject: [Python-ideas] Add the imath module
In-Reply-To: <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
Message-ID: <pi7l1m$lr5$1@blaine.gmane.org>

12.07.18 15:15, David Mertz ????:
> On Thu, Jul 12, 2018, 7:56 AM Serhiy Storchaka 
> <storchaka at gmail.com 
> <mailto:storchaka at gmail.com>> wrote:
> 
>     * isprime(n)
>     Tests if n is a prime number.
> 
> 
> How would you test this? The Miller-Rabin Primality test? For large 
> numbers, iterating through candidate prime divisors is pricey.
> 
>     * primes()
>     Returns an iterator of prime numbers: 2, 3, 5, 7, 11, 13,...
> 
> 
> How would you implements this? Sieve of Eratosthenes is really fun to 
> show students as a Python generator function. But the cached primes DO 
> grow unboundedly as you utilize the generator. Wheel factorization as 
> first pass? Do you cached the first N primes so the each instance of 
> iterator can provide initial elements much faster? What is N?

I didn't mean any concrete implementation. Sure there are enough 
efficient and simple. I sometimes need a sequence of prime numbers for 
solving toy problems, and quickly write something like:

def primes(n):
     return (i for i in range(2, n) if all(i % j for j in 
primes(int(sqrt(i)) + 1)))

Having such feature in the stdlib would be very convenient. Any 
reasonable implementation is enough efficient for my purposes.


From storchaka at gmail.com  Thu Jul 12 09:41:38 2018
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 12 Jul 2018 16:41:38 +0300
Subject: [Python-ideas] Add the imath module
In-Reply-To: <CA+msPNk_wxKsHkv7nbWk5Kgn2HBoT+Mz9s6gvYeabaJqyGcxqA@mail.gmail.com>
References: <pi7fga$flg$1@blaine.gmane.org> <pi7ju2$7ut$1@blaine.gmane.org>
 <CA+msPNk_wxKsHkv7nbWk5Kgn2HBoT+Mz9s6gvYeabaJqyGcxqA@mail.gmail.com>
Message-ID: <pi7lmf$2ea$1@blaine.gmane.org>

12.07.18 16:15, Robert Vanden Eynde ????:
> About the name, why not intmath ?

Because cmath. But if most core developers prefer intmath, I have no 
objections.

> And why not using sage ?
> Or create a package on pypi ?

Because some of these function could be useful in the stdlib.
Because the special purposed module is a better home for some math 
functions.
Because it is convenient to have these simple batteries included. Sage 
is too heavyweight.

The reasons are the same as for adding factorial() and gcd() in the math 
module. But after adding more such functions it is better to move them 
into the special module.


From p.f.moore at gmail.com  Thu Jul 12 10:30:39 2018
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 12 Jul 2018 15:30:39 +0100
Subject: [Python-ideas] Add the imath module
In-Reply-To: <pi7l1m$lr5$1@blaine.gmane.org>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
Message-ID: <CACac1F9L9ZTqCj0w69QH__pLLvbQ2ONmYb+17NNdyk18Vt5jJA@mail.gmail.com>

On 12 July 2018 at 14:30, Serhiy Storchaka <storchaka at gmail.com> wrote:
> 12.07.18 15:15, David Mertz ????:
>>
>> On Thu, Jul 12, 2018, 7:56 AM Serhiy Storchaka <storchaka at gmail.com
>> <mailto:storchaka at gmail.com>> wrote:
>>
>>     * isprime(n)
>>     Tests if n is a prime number.
>>
>>
>> How would you test this? The Miller-Rabin Primality test? For large
>> numbers, iterating through candidate prime divisors is pricey.
>>
>>     * primes()
>>     Returns an iterator of prime numbers: 2, 3, 5, 7, 11, 13,...
>>
>>
>> How would you implements this? Sieve of Eratosthenes is really fun to show
>> students as a Python generator function. But the cached primes DO grow
>> unboundedly as you utilize the generator. Wheel factorization as first pass?
>> Do you cached the first N primes so the each instance of iterator can
>> provide initial elements much faster? What is N?
>
>
> I didn't mean any concrete implementation. Sure there are enough efficient
> and simple. I sometimes need a sequence of prime numbers for solving toy
> problems, and quickly write something like:
>
> def primes(n):
>     return (i for i in range(2, n) if all(i % j for j in primes(int(sqrt(i))
> + 1)))
>
> Having such feature in the stdlib would be very convenient. Any reasonable
> implementation is enough efficient for my purposes.

Agreed, having these functions in the stdlib would be convenient, but
I think it's important that we have at least reasonably performing
implementations (they don't have to be suitable for specialist use,
but they should be performant enough for production, non-specialist
use). IMO, the statistics module got this balance right - good enough
for general use, but specialists will want something better.

I'm -1 on having inefficient versions like your primes(n)
implementation above in the stdlib. People will use them not realising
they may not be suitable for anything other than toy problems.

I'd be interested in the following (which is mostly a duplicate of your list)

* factorial
* gcd
* binom
* isprime (it's important to be clear if this is a guaranteed check,
or a probabilistic one like Miller Rabin)
* primes (an infinite generator of primes)
* factors (generates the prime factors of a number, in increasing
order, so list(factors(12)) = [2, 2, 3])

Your divide_and_round might be nice, but there are multiple common
ways of rounding 0.5 (up, down, towards 0, away from 0, to even, to
odd, ...) Your implementation does round to even, but that's not
necessarily the obvious one (schools often teach round up or down, so
use of your implementation in education could be confusing).

> Because cmath. But if most core developers prefer intmath, I have no objections.

I prefer imath as a parallel with cmath, as you say. But it's not a
big deal to me.

Paul

From turnbull.stephen.fw at u.tsukuba.ac.jp  Thu Jul 12 11:02:20 2018
From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull)
Date: Fri, 13 Jul 2018 00:02:20 +0900
Subject: [Python-ideas] Unicode Name Aliases keyword argument
 abbreviation in unicodedata.name for missing names
In-Reply-To: <VI1PR03MB2909445BDBF24848ECF8029AB6590@VI1PR03MB2909.eurprd03.prod.outlook.com>
References: <VI1PR03MB29093F08C0489886B5D758E2B6590@VI1PR03MB2909.eurprd03.prod.outlook.com>
 <20180712071725.GR7318@ando.pearwood.info>
 <20180712103739.6hxsoyltxvjbsn4d@phdru.name>
 <VI1PR03MB2909445BDBF24848ECF8029AB6590@VI1PR03MB2909.eurprd03.prod.outlook.com>
Message-ID: <23367.28028.242930.802545@turnbull.sk.tsukuba.ac.jp>

Robert Vanden Eynde writes:

 > As I'm at, I mentionned the ffef character but we don't care about
 > it because it already has a name, so that's mostly a control
 > character issue.

The problem with control characters is that from the point of view of
the Unicode Standard, the C0 and C1 registers are basically a space
reserved for private use (see ISO 6429 for the huge collection of
standardized control functions).  That is, unlike the rest of the
Unicode repertoire, the "characters" mapped there are neither unique
nor context-independent.  It's true that ISO 6429 recommends specific
C0 and C1 sets (but the recommended C1 set isn't even complete:
U+0080, U+0081, and U+0099 aren't assigned!)  However, Unicode only
suggests that those should be the default interpretations, because the
useful control functions are going to be dependent on context (eg,
input and output devices).

This is like the situation with Internet addresses and domain names.
The mapping is inherently many-many; round-tripping is not possible.

And in fact there are a number of graphic characters that have
multiple code points due to bugs in national character sets.  So for
graphic characters, it's possible to ensure name(code(x)) = x, but
it's not possible to ensure code(name(x)) = x, except under special
circumstances (which apply to the vast majority of characters, of
course).

 > I like your alias(...) function, with that one, an application
 > could code my function like try name(x) expect
 > alias(x).abbreviations[0]. If the abbreviation list is sorted by
 > AdditionToUnicodeDate.

I don't understand why that's particularly useful, especially in the
Han case (see below).

 > However, having a standard canonical name for all character in the
 > stdlib would help people choosing the same convention. A new
 > function like "canonical_name" or a shorter name would be an idea.

I don't understand what you're asking for.  The Unicode Standard
already provides canonical names.  Of course, the canonical name of
most Han ideographs (near and dear to my heart) are pretty useless
(they look like "CJK UNIFIED IDEOGRAPH-4E00").  (You probably don't
want to get the Japanese, Chinese---and there are a lot of different
kinds of Chinese---and Koreans started on what the "canonical" name
should be.  One Han Unification controversy is enough for this
geological epoch!)  This is closely related to the Unicode standard's
generic recommendation (Ch. 4.8):

    On the other hand, an API which returns a name for Unicode code
    points, but which is expected to provide useful, unique labels for
    unassigned, reserved code points and other special code point
    types, should return the value of the Unicode Name property for
    any code point for which it is non-null, but should otherwise con-
    struct a code point label to stand in for a character name.

(I suppose "should" here is used in the sense of RFC 2119.)  So, the
standard defines a canonical naming scheme, although many character
names are not terribly mnemonic even to native speakers.

On the other hand, if you want useful aliases for Han characters, for
many of them there could be scores of aliases, based on pronunciation,
semantics, and appearance, the first two of which of which vary
substantially within a single language, let alone across languages.
Worse, as far as I know there are no standard equivalent ways to
express these things in English, as when writing about these
characters in English you often adopt a romanized version of the
technical terms in the language you're studying.  And, it's a minor
point, but there are new Han characters discovered every day (I'm not
even sure that's an exaggeration), as scholars examine regional and
historical documents.

So for this to be most useful to me, I would want it developed OUTSIDE
of the stdlib, with releases even more frequent than pytz (that is an
exaggeration).  Not so much because I'll frequently need anything
outside of the main CJK block in Plane 0, but because the complexity
of character naming in East Asia suggests that improvements in
heuristics for assigning priority to aliases, language-specific
variations in heuristics, and so on will be rapid for the forseeable
future.  It would be a shame to shackle that to the current stdlib
release cycle even if it doesn't need to be as frenetic as pytz.  This
goes in spades for people who are waiting for their own scripts to be
standardized.

For the stdlib, I'm -1 on anything other than the canonical names plus
the primary aliases for characters which are well-defined in the code
charts of the Unicode Standard, such as those for the C0 and (most of)
the C1 control characters.  And even there I think a canonical name
based on block name + code point in hex is the best way to go.

I think this problem is a lot harder than many of the folk
participating in this discussion so far realize.

Steve


From robertvandeneynde at hotmail.com  Thu Jul 12 11:11:59 2018
From: robertvandeneynde at hotmail.com (Robert Vanden Eynde)
Date: Thu, 12 Jul 2018 15:11:59 +0000
Subject: [Python-ideas] Unicode Name Aliases keyword argument
 abbreviation in unicodedata.name for missing names
In-Reply-To: <23367.28028.242930.802545@turnbull.sk.tsukuba.ac.jp>
References: <VI1PR03MB29093F08C0489886B5D758E2B6590@VI1PR03MB2909.eurprd03.prod.outlook.com>
 <20180712071725.GR7318@ando.pearwood.info>
 <20180712103739.6hxsoyltxvjbsn4d@phdru.name>
 <VI1PR03MB2909445BDBF24848ECF8029AB6590@VI1PR03MB2909.eurprd03.prod.outlook.com>
 <23367.28028.242930.802545@turnbull.sk.tsukuba.ac.jp>
Message-ID: <VI1PR03MB2909E5A3D47263B6A9500646B6590@VI1PR03MB2909.eurprd03.prod.outlook.com>


 > I like your alias(...) function, with that one, an application
 > could code my function like try name(x) expect
 > alias(x).abbreviations[0]. If the abbreviation list is sorted by
 > AdditionToUnicodeDate.

I don't understand why that's particularly useful, especially in the
Han case (see below).

Since python 3.3 has the NameAliases.txt builtin in the distribution in order to full fil \N{} construct, I think it would be nice to have an api to access this files, to do like unicodedata.alias('\n').abbreviations[:3] == ['LF', 'NL', 'EOL']


I don't understand what you're asking for.  The Unicode Standard
already provides canonical names.

Not for control characters.

About the Han case, they all have a unicodedata.name<http://unicodedata.name> don't they ? (Sorry if I misread your message)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180712/1e3c6227/attachment.html>

From storchaka at gmail.com  Thu Jul 12 11:28:19 2018
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 12 Jul 2018 18:28:19 +0300
Subject: [Python-ideas] Add the imath module
In-Reply-To: <CACac1F9L9ZTqCj0w69QH__pLLvbQ2ONmYb+17NNdyk18Vt5jJA@mail.gmail.com>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CACac1F9L9ZTqCj0w69QH__pLLvbQ2ONmYb+17NNdyk18Vt5jJA@mail.gmail.com>
Message-ID: <pi7ruh$tn5$1@blaine.gmane.org>

12.07.18 17:30, Paul Moore ????:
> I'm -1 on having inefficient versions like your primes(n)
> implementation above in the stdlib. People will use them not realising
> they may not be suitable for anything other than toy problems.

This is just the shortest version that I write every time when I need 
primes. For the stdlib implementation we don't have such strong limit on 
the size.

> Your divide_and_round might be nice, but there are multiple common
> ways of rounding 0.5 (up, down, towards 0, away from 0, to even, to
> odd, ...) Your implementation does round to even, but that's not
> necessarily the obvious one (schools often teach round up or down, so
> use of your implementation in education could be confusing).

This is the one that is useful in many applications and can't be 
trivially written as expression in Python, but is useful in many 
applications (date and time, Fraction, Decimal).

Divide and rounding down: n//m.
Divide and rounding up: (n+m-1)//m or -((-n)//m).


From steve at pearwood.info  Thu Jul 12 12:05:29 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 13 Jul 2018 02:05:29 +1000
Subject: [Python-ideas] Add the imath module
In-Reply-To: <pi7fga$flg$1@blaine.gmane.org>
References: <pi7fga$flg$1@blaine.gmane.org>
Message-ID: <20180712160529.GV7318@ando.pearwood.info>

On Thu, Jul 12, 2018 at 02:55:58PM +0300, Serhiy Storchaka wrote:

> What are your thoughts about adding a new imath module for integer 
> mathematics? 

I'm not sure that we need a specific module for integer-valued maths. I 
think a more important change would be to allow math functions to be 
written in Python, not just C:

- move the current math library to a private _math library;

- add math.py, which calls "from _math import *".

I often miss having a good, fast, integer square root function in the 
stdlib. I have a slow implementation here:

http://code.activestate.com/recipes/577821-integer-square-root-function/

and as the commentary discusses, a lot of implementations on the 
internet are buggy.

As well as isqrt(n), there's also nsqrt(n), to return the integer 
nearest to the square root of n. They're equivalent to:

isqrt = floor sqrt
nsqrt = round sqrt

without overflowing for large n. (Presumably there's a ceiling sqrt 
function too, but I've never needed it, or seen anyone else use it.)



As far as your other suggestions:

* factorial(n)
* gcd(n, m)

Moving those seem like a fairly pedantic change. But if we did it, we 
could allow gcd to take an arbitrary number of values:

    gcd(*args)

and add a least common multiple function to match.


* as_integer_ration(x)

I think that's mispelled "ratio" :-)

There's currently a private function statistics._exact_ratio which 
does that.


* binom(n, k)

If we have number of combinations, I'd like to have number of 
permutations as well.


* isprime(n)
* primes()

Primality testing and generation of primes is a bigger job than it might 
seem, probably deserving of an entire library. (Not necessarily in the 
std lib.)

But if we did provide a "batteries included" solution, it should also 
include next prime and previous prime functions.

For isprime(), I have a pure-Python solution which is not too shabby, 
capable of giving exact results for all integers up to 2**64 and 
probabilitistic results above that. Despite being pure Python, it's 
reasonably fast. In the REPL, these are essentially instantaneous to the 
naked eye:

py> pyprimes.prev_prime(2**64)
18446744073709551557L
py> pyprimes.is_prime(18446744073709551557L)
True
py> pyprimes.next_prime(2**64)
pyprimes/__init__.py:278: MaybeComposite: 18446744073709551629 is only 
only probably prime
  warnings.warn(message, MaybeComposite)
18446744073709551629L


At the moment this is Python 2 only and frankly overkill for the stdlib. 
I have something like a dozen different implementations for generating 
or testing primes, some of which are (intentionally) just awful. One of 
my favourite so-awful-it's-good algorithms is this one for testing 
whether a number is prime:

def isprime(n):
    # Kids: don't try this at home!
    return not re.match(r'^1?$|^(11+?)\1+$', '1'*n)


But if there is interest in a pure-Python solution, I might be able to 
find time to clean it up to stdlib quality. (Not the regex solution, 
obviously.)



-- 
Steve

From steve at pearwood.info  Thu Jul 12 12:14:49 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 13 Jul 2018 02:14:49 +1000
Subject: [Python-ideas] Add the imath module
In-Reply-To: <pi7ruh$tn5$1@blaine.gmane.org>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CACac1F9L9ZTqCj0w69QH__pLLvbQ2ONmYb+17NNdyk18Vt5jJA@mail.gmail.com>
 <pi7ruh$tn5$1@blaine.gmane.org>
Message-ID: <20180712161447.GW7318@ando.pearwood.info>

On Thu, Jul 12, 2018 at 06:28:19PM +0300, Serhiy Storchaka wrote:

> This is the one that is useful in many applications and can't be 
> trivially written as expression in Python, but is useful in many 
> applications (date and time, Fraction, Decimal).
> 
> Divide and rounding down: n//m.
> Divide and rounding up: (n+m-1)//m or -((-n)//m).

Sorry, I don't understand why you say that divide and round up can't be 
trivially written in Python. Don't you give two implementations 
yourself?

I have this in my toolbox:

def ceildiv(a, b):
    """Return the ceiling of a/b."""
    return -(-a//b)

and if its buggy, I've never noticed it yet.



-- 
Steve

From storchaka at gmail.com  Thu Jul 12 12:50:32 2018
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 12 Jul 2018 19:50:32 +0300
Subject: [Python-ideas] Add the imath module
In-Reply-To: <20180712161447.GW7318@ando.pearwood.info>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CACac1F9L9ZTqCj0w69QH__pLLvbQ2ONmYb+17NNdyk18Vt5jJA@mail.gmail.com>
 <pi7ruh$tn5$1@blaine.gmane.org> <20180712161447.GW7318@ando.pearwood.info>
Message-ID: <pi80ok$1k8$1@blaine.gmane.org>

12.07.18 19:14, Steven D'Aprano ????:
> Sorry, I don't understand why you say that divide and round up can't be
> trivially written in Python. Don't you give two implementations
> yourself?

Sorry, perhaps I was unclear, but I said the opposite.


From python at mrabarnett.plus.com  Thu Jul 12 13:09:50 2018
From: python at mrabarnett.plus.com (MRAB)
Date: Thu, 12 Jul 2018 18:09:50 +0100
Subject: [Python-ideas] Unicode Name Aliases keyword argument
 abbreviation in unicodedata.name for missing names
In-Reply-To: <23367.28028.242930.802545@turnbull.sk.tsukuba.ac.jp>
References: <VI1PR03MB29093F08C0489886B5D758E2B6590@VI1PR03MB2909.eurprd03.prod.outlook.com>
 <20180712071725.GR7318@ando.pearwood.info>
 <20180712103739.6hxsoyltxvjbsn4d@phdru.name>
 <VI1PR03MB2909445BDBF24848ECF8029AB6590@VI1PR03MB2909.eurprd03.prod.outlook.com>
 <23367.28028.242930.802545@turnbull.sk.tsukuba.ac.jp>
Message-ID: <9c69142a-319c-b3dc-958c-b1cec464d08f@mrabarnett.plus.com>

On 2018-07-12 16:02, Stephen J. Turnbull wrote:
> Robert Vanden Eynde writes:
> 
>   > As I'm at, I mentionned the ffef character but we don't care about
>   > it because it already has a name, so that's mostly a control
>   > character issue.
> 
> The problem with control characters is that from the point of view of
> the Unicode Standard, the C0 and C1 registers are basically a space
> reserved for private use (see ISO 6429 for the huge collection of
> standardized control functions).  That is, unlike the rest of the
> Unicode repertoire, the "characters" mapped there are neither unique
> nor context-independent.  It's true that ISO 6429 recommends specific
> C0 and C1 sets (but the recommended C1 set isn't even complete:
> U+0080, U+0081, and U+0099 aren't assigned!)  However, Unicode only
> suggests that those should be the default interpretations, because the
> useful control functions are going to be dependent on context (eg,
> input and output devices).
> 
> This is like the situation with Internet addresses and domain names.
> The mapping is inherently many-many; round-tripping is not possible.
> 
> And in fact there are a number of graphic characters that have
> multiple code points due to bugs in national character sets.  So for
> graphic characters, it's possible to ensure name(code(x)) = x, but
> it's not possible to ensure code(name(x)) = x, except under special
> circumstances (which apply to the vast majority of characters, of
> course).
> 
>   > I like your alias(...) function, with that one, an application
>   > could code my function like try name(x) expect
>   > alias(x).abbreviations[0]. If the abbreviation list is sorted by
>   > AdditionToUnicodeDate.
> 
> I don't understand why that's particularly useful, especially in the
> Han case (see below).
> 
>   > However, having a standard canonical name for all character in the
>   > stdlib would help people choosing the same convention. A new
>   > function like "canonical_name" or a shorter name would be an idea.
> 
> I don't understand what you're asking for.  The Unicode Standard
> already provides canonical names.  Of course, the canonical name of
> most Han ideographs (near and dear to my heart) are pretty useless
> (they look like "CJK UNIFIED IDEOGRAPH-4E00").  (You probably don't
> want to get the Japanese, Chinese---and there are a lot of different
> kinds of Chinese---and Koreans started on what the "canonical" name
> should be.  One Han Unification controversy is enough for this
> geological epoch!)  This is closely related to the Unicode standard's
> generic recommendation (Ch. 4.8):
> 
>      On the other hand, an API which returns a name for Unicode code
>      points, but which is expected to provide useful, unique labels for
>      unassigned, reserved code points and other special code point
>      types, should return the value of the Unicode Name property for
>      any code point for which it is non-null, but should otherwise con-
>      struct a code point label to stand in for a character name.
> 
> (I suppose "should" here is used in the sense of RFC 2119.)  So, the
> standard defines a canonical naming scheme, although many character
> names are not terribly mnemonic even to native speakers.
> 
> On the other hand, if you want useful aliases for Han characters, for
> many of them there could be scores of aliases, based on pronunciation,
> semantics, and appearance, the first two of which of which vary
> substantially within a single language, let alone across languages.
> Worse, as far as I know there are no standard equivalent ways to
> express these things in English, as when writing about these
> characters in English you often adopt a romanized version of the
> technical terms in the language you're studying.  And, it's a minor
> point, but there are new Han characters discovered every day (I'm not
> even sure that's an exaggeration), as scholars examine regional and
> historical documents.
> 
> So for this to be most useful to me, I would want it developed OUTSIDE
> of the stdlib, with releases even more frequent than pytz (that is an
> exaggeration).  Not so much because I'll frequently need anything
> outside of the main CJK block in Plane 0, but because the complexity
> of character naming in East Asia suggests that improvements in
> heuristics for assigning priority to aliases, language-specific
> variations in heuristics, and so on will be rapid for the forseeable
> future.  It would be a shame to shackle that to the current stdlib
> release cycle even if it doesn't need to be as frenetic as pytz.  This
> goes in spades for people who are waiting for their own scripts to be
> standardized.
> 
> For the stdlib, I'm -1 on anything other than the canonical names plus
> the primary aliases for characters which are well-defined in the code
> charts of the Unicode Standard, such as those for the C0 and (most of)
> the C1 control characters.  And even there I think a canonical name
> based on block name + code point in hex is the best way to go.
> 
> I think this problem is a lot harder than many of the folk
> participating in this discussion so far realize.
> 
AFAIR, the last time codepoint names were talked about, I suggested that 
there could be a fallback to U+XXXX. unicodedata.name can accept a 
default, but unicodedata.lookup doesn't accept such a 'name'.

From steve at pearwood.info  Thu Jul 12 13:14:45 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 13 Jul 2018 03:14:45 +1000
Subject: [Python-ideas] Unicode Name Aliases keyword argument
 abbreviation in unicodedata.name for missing names
In-Reply-To: <23367.28028.242930.802545@turnbull.sk.tsukuba.ac.jp>
References: <VI1PR03MB29093F08C0489886B5D758E2B6590@VI1PR03MB2909.eurprd03.prod.outlook.com>
 <20180712071725.GR7318@ando.pearwood.info>
 <20180712103739.6hxsoyltxvjbsn4d@phdru.name>
 <VI1PR03MB2909445BDBF24848ECF8029AB6590@VI1PR03MB2909.eurprd03.prod.outlook.com>
 <23367.28028.242930.802545@turnbull.sk.tsukuba.ac.jp>
Message-ID: <20180712171444.GY7318@ando.pearwood.info>

On Fri, Jul 13, 2018 at 12:02:20AM +0900, Stephen J. Turnbull wrote:

>  > I like your alias(...) function, with that one, an application
>  > could code my function like try name(x) expect
>  > alias(x).abbreviations[0]. If the abbreviation list is sorted by
>  > AdditionToUnicodeDate.
> 
> I don't understand why that's particularly useful, especially in the
> Han case (see below).

Sorry, I'm not sure if you mean my proposed alias() function isn't 
useful, or Robert's try...except loop around it.

My alias() function is just an programmatic interface to information 
already available in the NameAliases.txt file. Don't you think that's 
useful enough as it stands?

What people do with it will depend on their application, of course.


[...]
> On the other hand, if you want useful aliases for Han characters, for
> many of them there could be scores of aliases, based on pronunciation,
> semantics, and appearance, the first two of which of which vary
> substantially within a single language, let alone across languages.

Indeed. That's also the case for emoji. That's why I suggested making 
alias() return a mutable record rather than an immutable tuple, so 
application writers can add their own records to suit their own needs.

I'll admit I haven't thought really deeply about what the most useful 
API would be -- this was only an initial post on Python-Ideas, not a 
fully-fledged PEP -- but I think the critical point here is that we 
shouldn't be privileging one alias type over the others. The 
NameAlias.txt file makes all that information available, but we can't 
access it (easily, or at all) from unicodedata.

[...]
> So for this to be most useful to me, I would want it developed OUTSIDE
> of the stdlib, with releases even more frequent than pytz (that is an
> exaggeration).

That seems fairly extreme. New Unicode versions don't come out that 
frequently. Surely we don't expect to track draft aliases, or characters 
outside of Unicode?

Application writers might choose to do so -- if somebody wants to 
support "Ay" as an alias for LATIN CAPITAL LETTER A they can be my 
guest, but the stdlib doesn't have to directly support it until it hits 
the NameAliases.txt file :-)

[...]
> For the stdlib, I'm -1 on anything other than the canonical names plus
> the primary aliases for characters which are well-defined in the code
> charts of the Unicode Standard, such as those for the C0 and (most of)
> the C1 control characters.

To clarify, do you mean the aliases defined in NameAliases.txt? Or a 
subset of them?


> And even there I think a canonical name
> based on block name + code point in hex is the best way to go.

I believe you might be thinking of the Unicode "code point label" 
concept. I have this implementation in my toolbox:


NONCHARACTERS = tuple(
    [unichr(n) for n in range(0xFDD0, 0xFDF0)] +
    [unichr(n*0x10000 + 0xFFFE +i) for n in range(17) for i in range(2)]
    )
assert len(NONCHARACTERS) == 66

def label(c):
    """Return the Code Point Label or character name of c.

    If c is a code point with a name, the name is used as the label;
    otherwise the Code Point Label is returned.

    >>> label(unichr(0x0394))  # u'?'
    'GREEK CAPITAL LETTER DELTA'
    >>> label(unichr(0x001F))
    '<control-001F>'

    """
    name = unicodedata.name(c, '')
    if name == '':
        # See section on Code Point Labels
        # http://www.unicode.org/versions/Unicode10.0.0/ch04.pdf
        number = ord(c)
        category = unicodedata.category(c)
        assert category in ('Cc', 'Cn', 'Co', 'Cs')
        if category == 'Cc':
            kind = 'control'
        elif category == 'Cn':
            if c in NONCHARACTERS:
                kind = 'noncharacter'
            else:
                kind = 'reserved'
        elif category == 'Co':
            kind = 'private-use'
        else:
            assert category == 'Cs'
            kind = 'surrogate'
        name = "<%s-%04X>" % (kind, number)
    return name



-- 
Steve

From mertz at gnosis.cx  Thu Jul 12 13:26:45 2018
From: mertz at gnosis.cx (David Mertz)
Date: Thu, 12 Jul 2018 13:26:45 -0400
Subject: [Python-ideas] Add the imath module
In-Reply-To: <pi7l1m$lr5$1@blaine.gmane.org>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
Message-ID: <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>

On Thu, Jul 12, 2018, 9:31 AM Serhiy Storchaka <storchaka at gmail.com> wrote:

> 12.07.18 15:15, David Mertz ????:
> > On Thu, Jul 12, 2018, 7:56 AM Serhiy Storchaka
> I didn't mean any concrete implementation. Sure there are enough
> efficient and simple. I sometimes need a sequence of prime numbers for
> solving toy problems, and quickly write something like:
>
> def primes(n):
>      return (i for i in range(2, n) if all(i % j for j in
> primes(int(sqrt(i)) + 1)))
>
> Having such feature in the stdlib would be very convenient. Any
> reasonable implementation is enough efficient for my purposes.
>

That's the point I was getting at. "For your purposes" isn't general enough
to include in the standard library. There are quite a few algorithms and
efficiency trade-offs to decide on. It's not clear to me that any choice is
sufficiently "one size fits all" to include.

I'm not saying there definitely IS NOT a reasonable compromise approach to
these things, but I'd have to judge concrete suggestions... Or better
still, people with better knowledge of number theory than mine should make
those judgements.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180712/9be89f4c/attachment.html>

From steve at pearwood.info  Thu Jul 12 13:27:04 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 13 Jul 2018 03:27:04 +1000
Subject: [Python-ideas] Unicode Name Aliases keyword argument
 abbreviation in unicodedata.name for missing names
In-Reply-To: <VI1PR03MB2909E5A3D47263B6A9500646B6590@VI1PR03MB2909.eurprd03.prod.outlook.com>
References: <VI1PR03MB29093F08C0489886B5D758E2B6590@VI1PR03MB2909.eurprd03.prod.outlook.com>
 <20180712071725.GR7318@ando.pearwood.info>
 <20180712103739.6hxsoyltxvjbsn4d@phdru.name>
 <VI1PR03MB2909445BDBF24848ECF8029AB6590@VI1PR03MB2909.eurprd03.prod.outlook.com>
 <23367.28028.242930.802545@turnbull.sk.tsukuba.ac.jp>
 <VI1PR03MB2909E5A3D47263B6A9500646B6590@VI1PR03MB2909.eurprd03.prod.outlook.com>
Message-ID: <20180712172704.GZ7318@ando.pearwood.info>

On Thu, Jul 12, 2018 at 03:11:59PM +0000, Robert Vanden Eynde wrote:

[Stephen] 
> I don't understand what you're asking for.  The Unicode Standard
> already provides canonical names.
> 
> Not for control characters.

That's because the Unicode Consortium considers that control characters 
have no canonical name. And I think that they are right.


> About the Han case, they all have a 
> unicodedata.name<http://unicodedata.name> don't they ? (Sorry if I 
> misread your message)

I think that the point Stephen is making is that the canonical name for 
most Han characters is terribly uninformative, even to native Han users. 
For Englishg speakers, the analogous situation would be if name("A") 
returned "LATIN CAPITAL LETTER 0041".

There are good reasons for that, but it does mean that if your intention 
is to report the name of the character to a non-technical end-user, in 
their own native language, using the Unicode name or even any of the 
aliases is probably not a great solution.

On the other hand, if you are in a lucky enough situation (unlike 
Stephen) of being able to say "Han characters? We'll fix that in the 
next version..." using the Unicode name is not a terrible solution.

At least, it's The Standard terrible solution *wink*


-- 
Steve

From steve at pearwood.info  Thu Jul 12 13:33:31 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 13 Jul 2018 03:33:31 +1000
Subject: [Python-ideas] Add the imath module
In-Reply-To: <pi80ok$1k8$1@blaine.gmane.org>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CACac1F9L9ZTqCj0w69QH__pLLvbQ2ONmYb+17NNdyk18Vt5jJA@mail.gmail.com>
 <pi7ruh$tn5$1@blaine.gmane.org> <20180712161447.GW7318@ando.pearwood.info>
 <pi80ok$1k8$1@blaine.gmane.org>
Message-ID: <20180712173331.GA7318@ando.pearwood.info>

On Thu, Jul 12, 2018 at 07:50:32PM +0300, Serhiy Storchaka wrote:
> 12.07.18 19:14, Steven D'Aprano ????:
> >Sorry, I don't understand why you say that divide and round up can't be
> >trivially written in Python. Don't you give two implementations
> >yourself?
> 
> Sorry, perhaps I was unclear, but I said the opposite.

Oh, sorry, you were talking about round to even! I misunderstood.


-- 
Steve

From steve at pearwood.info  Thu Jul 12 14:11:02 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 13 Jul 2018 04:11:02 +1000
Subject: [Python-ideas] Add the imath module
In-Reply-To: <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
Message-ID: <20180712181100.GB7318@ando.pearwood.info>

On Thu, Jul 12, 2018 at 01:26:45PM -0400, David Mertz wrote:

[talking about prime numbers]
 
> That's the point I was getting at. "For your purposes" isn't general enough
> to include in the standard library. There are quite a few algorithms and
> efficiency trade-offs to decide on. It's not clear to me that any choice is
> sufficiently "one size fits all" to include.

We're talking about "batteries included", not a nuclear reactor. (Thanks 
to Nick for that great analogy!)

This is a fairly narrow and deterministic set of requirements:

- test whether a number is prime;
- return some primes;

and maybe a couple of other related functions. Not a data structure, 
where we have to make serious tradeoffs in the type of values they 
support (hashable, orderable, or any arbitrary value, say) and what can 
be done with them. We already know what we have to support : ints, as 
large as will fit in memory.

While of course there are efficiency trade-offs, at the point you really 
care about them, you're in nuclear reactor territory and are probably 
using some library too specialised even for numpy *wink*

There is no reason why primality testing can't be deterministic up to 
2**64, and probabilistic with a ludicrously small chance of false 
positives beyond that. The implementation I use can be expected to fail 
on average once every 18 thousand years if you did nothing but test 
primes every millisecond of the day, 24 hours a day. That's good enough 
for most purposes :-)


> I'm not saying there definitely IS NOT a reasonable compromise approach to
> these things, but I'd have to judge concrete suggestions... Or better
> still, people with better knowledge of number theory than mine should make
> those judgements.

Miller-Rabin + trial division by the first few small primes is the 
work-horse of primality testing. There are "better, faster" algorithms, 
but they are more complex (more risk of bugs) and probably don't show 
any improvement until you're dealing with huge numbers. I mean *really* 
huge. My pure-Python version of Miller-Rabin takes:

about 4.7 seconds to test 2**800 + 1;
less than a tenth of a millisecond to test 2**900 + 1;
and about 8.6 seconds to test 2**1000 + 1.

(None of which are prime.) On a less old-and-tired computer, you can 
probably divide those numbers by, oh, five or ten.

You probably won't find any record-breaking Mersenne Primes using my 
version (certainly not on *my* computer), but I think it would be good 
enough for the standard library if we decided to add an isprime() 
function.

As far as generating primes, there's a lot more choices, but since they 
all do the same thing (generate prime numbers) we can easily swap one 
implementation for another and nobody will even know unless they're 
timing it.

*If* it is desirable to have this in the stdlib, we shouldn't be 
paralysed by implementation choice. We can start with something "good 
enough", and move to more complex implementations when and if somebody 
wants to do the work.



-- 
Steve

From andre.roberge at gmail.com  Thu Jul 12 14:20:17 2018
From: andre.roberge at gmail.com (Andre Roberge)
Date: Thu, 12 Jul 2018 15:20:17 -0300
Subject: [Python-ideas] Including the unparse module in the standard library
Message-ID: <CAGMu_=r8Jad-68Ho+57QkaY2r6oXaBQU-wuDN64NxRyb=vUmng@mail.gmail.com>

In the cPython repository, there is an unparse module in the Tools section.
https://github.com/python/cpython/blob/master/Tools/parser/unparse.py

However, as it is not part of the standard library, it cannot be easily
used; to do so, one needs to make a local copy in a place from where it can
be imported.

This module can be useful for people using the ast module to create and
parse trees, modify them ... and who want to convert the result back into
source code.  Since it is obviously maintained to be compatible with the
current Python version, would it be possible to include the unparse module
in the standard library?

Andr? Roberge
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180712/57cd2ead/attachment-0001.html>

From rosuav at gmail.com  Thu Jul 12 14:20:55 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 13 Jul 2018 04:20:55 +1000
Subject: [Python-ideas] Add the imath module
In-Reply-To: <20180712181100.GB7318@ando.pearwood.info>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <20180712181100.GB7318@ando.pearwood.info>
Message-ID: <CAPTjJmqS+2LSiM0XWy-gKcgcx1JHOhz3CVvCw=f9j23UdSNFqg@mail.gmail.com>

On Fri, Jul 13, 2018 at 4:11 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> There is no reason why primality testing can't be deterministic up to
> 2**64, and probabilistic with a ludicrously small chance of false
> positives beyond that. The implementation I use can be expected to fail
> on average once every 18 thousand years if you did nothing but test
> primes every millisecond of the day, 24 hours a day. That's good enough
> for most purposes :-)

What about false negatives? Guaranteed none? The failure mode of the
function should, IMO, be a defined and documented aspect of it.

ChrisA

From rymg19 at gmail.com  Thu Jul 12 14:54:50 2018
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Thu, 12 Jul 2018 13:54:50 -0500
Subject: [Python-ideas] Including the unparse module in the standard
 library
In-Reply-To: <CAGMu_=r8Jad-68Ho+57QkaY2r6oXaBQU-wuDN64NxRyb=vUmng@mail.gmail.com>
References: <CAGMu_=r8Jad-68Ho+57QkaY2r6oXaBQU-wuDN64NxRyb=vUmng@mail.gmail.com>
Message-ID: <1648fd88890.27a3.db5b03704c129196a4e9415e55413ce6@gmail.com>

If you want to get source code from an AST, you'd probably be better off 
with a more fully-featured library like Astor: 
https://github.com/berkerpeksag/astor


On July 12, 2018 1:21:23 PM Andre Roberge <andre.roberge at gmail.com> wrote:

> In the cPython repository, there is an unparse module in the Tools section.
> https://github.com/python/cpython/blob/master/Tools/parser/unparse.py
>
> However, as it is not part of the standard library, it cannot be easily
> used; to do so, one needs to make a local copy in a place from where it can
> be imported.
>
> This module can be useful for people using the ast module to create and
> parse trees, modify them ... and who want to convert the result back into
> source code.  Since it is obviously maintained to be compatible with the
> current Python version, would it be possible to include the unparse module
> in the standard library?
>
> Andr? Roberge
>
>
>
> ----------
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180712/dbc3598c/attachment.html>

From tim.peters at gmail.com  Thu Jul 12 14:57:00 2018
From: tim.peters at gmail.com (Tim Peters)
Date: Thu, 12 Jul 2018 13:57:00 -0500
Subject: [Python-ideas] Add the imath module
In-Reply-To: <pi7fga$flg$1@blaine.gmane.org>
References: <pi7fga$flg$1@blaine.gmane.org>
Message-ID: <CAExdVNmQjrK6KLgSzmEgk+cm832UKDvTWEukt0agfdigr0Wwiw@mail.gmail.com>

]Serhiy Storchaka <storchaka at gmail.com>]

> What are your thoughts about adding a new imath module for integer
> mathematics? It could contain the following functions:
>

I have mixed feelings :-(  There are a number of integer functions broadly
useful to people attracted to such things, and "it would be nice" to have
basic implementations ship with the language.

But the "basic" is important to me.  Algorithms for unbounded integers can
grow in complexity seemingly without bound too, and by the time there are
10 special cases 8 of which have been coded in C, it becomes an ongoing
maintenance headache - and the code has become so complex that users can't
easily look at the source to judge whether the implementation is _still_
too naive for their application.

For example, for generating primes, I'd balk at anything fancier than my
recoding in Python 3 of Will Ness's beautiful algorithm:

    https://stackoverflow.com/a/19391111/2705542

That generates primes in increasing order, with no need (or even
possibility) to specify an upper bound in advance.  But unlike nearly all
other reasonably efficient approaches to that, the memory burden grows
(very roughly) with the square root of the largest prime generated so far.
Most sieve algorithms require instead remembering all the ~P/log(P) primes
<= P, which grows much faster than sqrt(P).  That's a real practical
difference.

There are faster ways to generate the primes, and even that way can be sped
significantly by complicating the code significantly to incorporate a
multiple-small-prime wheel.

But I'd much rather leave significantly more complicated code to
_specialized_, non-core packages.

Anyway, in addition to your list, there are at least four others I view as
fundamental:

def iroot(n, k):  # the obvious generalization of integer square root
    """Return the floor of the k'th root of n."""

def egcd(a, b):   # extended gcd
    """Return (g, ga, gb) s.t. ga*a + gb*b == g == gcd(a, b)."""

def minv(p, q):
    """
    Return multiplicative modular inverse of p with respect to modulus q.

    (p * minv(p, q)) % q == 1
    """

def factor(n):
    """Return a list containing n's prime factors."""

State-of-the-art code for `factor()` alone could double the size of the
Python source distribution ;-)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180712/28e89099/attachment.html>

From mertz at gnosis.cx  Thu Jul 12 18:05:07 2018
From: mertz at gnosis.cx (David Mertz)
Date: Thu, 12 Jul 2018 18:05:07 -0400
Subject: [Python-ideas] Add the imath module
In-Reply-To: <CAPTjJmqS+2LSiM0XWy-gKcgcx1JHOhz3CVvCw=f9j23UdSNFqg@mail.gmail.com>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <20180712181100.GB7318@ando.pearwood.info>
 <CAPTjJmqS+2LSiM0XWy-gKcgcx1JHOhz3CVvCw=f9j23UdSNFqg@mail.gmail.com>
Message-ID: <CAEbHw4biUDjz9_uY1AbuPizYOqSGNa0nQfMTKKfOUJ2Mj-MDkw@mail.gmail.com>

On Thu, Jul 12, 2018 at 2:22 PM Chris Angelico <rosuav at gmail.com> wrote:

> On Fri, Jul 13, 2018 at 4:11 AM, Steven D'Aprano <steve at pearwood.info>
> wrote:
> > There is no reason why primality testing can't be deterministic up to
> > 2**64, and probabilistic with a ludicrously small chance of false
> > positives beyond that. The implementation I use can be expected to fail
> > on average once every 18 thousand years if you did nothing but test
> > primes every millisecond of the day, 24 hours a day. That's good enough
> > for most purposes :-)
>
> What about false negatives? Guaranteed none? The failure mode of the
> function should, IMO, be a defined and documented aspect of it.
>

Miller-Rabin or other pseudo-primality tests do not produce false negatives
IIUC.

I'm more concerned in the space/time tradeoff in the primes() generator. I
like this implementation for teaching purposes, but I'm well aware that it
grows in memory usage rather quickly (the one Tim Peters posted is probably
a better balance; but it depends on how many of these you create and how
long you run them).

from math import sqrt, ceil

def up_to(seq, lim):
    for n in seq:
        if n < lim:
            yield n
        else:
            break

def sieve_generator():
    "Pretty good Sieve; skip the even numbers, stop at sqrt(candidate)"
    yield 2
    candidate = 3
    found = []
    while True:
        lim = int(ceil(sqrt(candidate)))
        if all(candidate % prime != 0 for prime in up_to(found, lim)):
            yield candidate
            found.append(candidate)
        candidate += 2


So then how do you implement isprime().  One naive way is to compare it
against elements of sieve_generator() until we are equal or larger than the
test element.  But that's not super fast.   A pseudo-primality test is much
faster (except for in the first few hundred thousand primes, maybe).

-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180712/ebdd6874/attachment-0001.html>

From mertz at gnosis.cx  Thu Jul 12 18:24:08 2018
From: mertz at gnosis.cx (David Mertz)
Date: Thu, 12 Jul 2018 18:24:08 -0400
Subject: [Python-ideas] Add the imath module
In-Reply-To: <CAEbHw4biUDjz9_uY1AbuPizYOqSGNa0nQfMTKKfOUJ2Mj-MDkw@mail.gmail.com>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <20180712181100.GB7318@ando.pearwood.info>
 <CAPTjJmqS+2LSiM0XWy-gKcgcx1JHOhz3CVvCw=f9j23UdSNFqg@mail.gmail.com>
 <CAEbHw4biUDjz9_uY1AbuPizYOqSGNa0nQfMTKKfOUJ2Mj-MDkw@mail.gmail.com>
Message-ID: <CAEbHw4amrTfLXoKkTvH0Xb10YnofVJGqOGOt4dfWOPajC82Djw@mail.gmail.com>

I take it back... Tim's (really Will Ness's) version is *always* faster and
more memory friendly.  I'm still trying to understand it though :-).

On Thu, Jul 12, 2018 at 6:05 PM David Mertz <mertz at gnosis.cx> wrote:

>
> On Thu, Jul 12, 2018 at 2:22 PM Chris Angelico <rosuav at gmail.com> wrote:
>
>> On Fri, Jul 13, 2018 at 4:11 AM, Steven D'Aprano <steve at pearwood.info>
>> wrote:
>> > There is no reason why primality testing can't be deterministic up to
>> > 2**64, and probabilistic with a ludicrously small chance of false
>> > positives beyond that. The implementation I use can be expected to fail
>> > on average once every 18 thousand years if you did nothing but test
>> > primes every millisecond of the day, 24 hours a day. That's good enough
>> > for most purposes :-)
>>
>> What about false negatives? Guaranteed none? The failure mode of the
>> function should, IMO, be a defined and documented aspect of it.
>>
>
> Miller-Rabin or other pseudo-primality tests do not produce false
> negatives IIUC.
>
> I'm more concerned in the space/time tradeoff in the primes() generator. I
> like this implementation for teaching purposes, but I'm well aware that it
> grows in memory usage rather quickly (the one Tim Peters posted is probably
> a better balance; but it depends on how many of these you create and how
> long you run them).
>
> from math import sqrt, ceil
>
> def up_to(seq, lim):
>     for n in seq:
>         if n < lim:
>             yield n
>         else:
>             break
>
> def sieve_generator():
>     "Pretty good Sieve; skip the even numbers, stop at sqrt(candidate)"
>     yield 2
>     candidate = 3
>     found = []
>     while True:
>         lim = int(ceil(sqrt(candidate)))
>         if all(candidate % prime != 0 for prime in up_to(found, lim)):
>             yield candidate
>             found.append(candidate)
>         candidate += 2
>
>
> So then how do you implement isprime().  One naive way is to compare it
> against elements of sieve_generator() until we are equal or larger than the
> test element.  But that's not super fast.   A pseudo-primality test is much
> faster (except for in the first few hundred thousand primes, maybe).
>
> --
> Keeping medicines from the bloodstreams of the sick; food
> from the bellies of the hungry; books from the hands of the
> uneducated; technology from the underdeveloped; and putting
> advocates of freedom in prisons.  Intellectual property is
> to the 21st century what the slave trade was to the 16th.
>


-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180712/3466dc59/attachment.html>

From tim.peters at gmail.com  Thu Jul 12 18:35:54 2018
From: tim.peters at gmail.com (Tim Peters)
Date: Thu, 12 Jul 2018 17:35:54 -0500
Subject: [Python-ideas] Add the imath module
In-Reply-To: <CAEbHw4biUDjz9_uY1AbuPizYOqSGNa0nQfMTKKfOUJ2Mj-MDkw@mail.gmail.com>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <20180712181100.GB7318@ando.pearwood.info>
 <CAPTjJmqS+2LSiM0XWy-gKcgcx1JHOhz3CVvCw=f9j23UdSNFqg@mail.gmail.com>
 <CAEbHw4biUDjz9_uY1AbuPizYOqSGNa0nQfMTKKfOUJ2Mj-MDkw@mail.gmail.com>
Message-ID: <CAExdVN=0_z3x93u7aYT_6CyADPRaf7SxphKR7UV7+kToUwVGeA@mail.gmail.com>

[David Mertz]

> Miller-Rabin or other pseudo-primality tests do not produce false
> negatives IIUC.
>

That's true if they're properly implemented ;-)  If they say False, the
input is certainly composite (but there isn't a clue as to what a factor
may be); if the say True, the input is "probably" a prime.


> I'm more concerned in the space/time tradeoff in the primes() generator. I
> like this implementation for teaching purposes, but I'm well aware that it
> grows in memory usage rather quickly (the one Tim Peters posted is probably
> a better balance; but it depends on how many of these you create and how
> long you run them).
>

As you noted later:

I take it back... Tim's (really Will Ness's) version is *always* faster and
> more

memory friendly.


The original version of this came from an ActiveState recipe that many on
c.l.py contributed to (including me) years ago, and that's where all the
speed came from.  There is, for example, no division or square roots.
Will's contribution was the unexpected way to slash memory use, via the
generator calling itself recursively.  Elegant and effective!


> I'm still trying to understand it though :-).


Picture doing the Sieve of Eratosthenes by hand, crossing out multiples of
"the next" prime you find.  That's basically what it's doing, except doing
the "crossing out" part incrementally, using a dict to remember, for each
prime p, "the next" multiple of p you haven't yet gotten to.



> from math import sqrt, ceil
>
> def up_to(seq, lim):
>     for n in seq:
>         if n < lim:
>             yield n
>         else:
>             break
>
> def sieve_generator():
>     "Pretty good Sieve; skip the even numbers, stop at sqrt(candidate)"
>     yield 2
>     candidate = 3
>     found = []
>     while True:
>         lim = int(ceil(sqrt(candidate)))
>         if all(candidate % prime != 0 for prime in up_to(found, lim)):
>             yield candidate
>             found.append(candidate)
>         candidate += 2
>
>
>
Note that this generates squares of odd primes too (9, 25, ...).  `up_to`
needs to be more like `up_to_and_including`.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180712/a82c883c/attachment.html>

From mertz at gnosis.cx  Thu Jul 12 18:47:51 2018
From: mertz at gnosis.cx (David Mertz)
Date: Thu, 12 Jul 2018 18:47:51 -0400
Subject: [Python-ideas] Add the imath module
In-Reply-To: <CAExdVN=0_z3x93u7aYT_6CyADPRaf7SxphKR7UV7+kToUwVGeA@mail.gmail.com>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <20180712181100.GB7318@ando.pearwood.info>
 <CAPTjJmqS+2LSiM0XWy-gKcgcx1JHOhz3CVvCw=f9j23UdSNFqg@mail.gmail.com>
 <CAEbHw4biUDjz9_uY1AbuPizYOqSGNa0nQfMTKKfOUJ2Mj-MDkw@mail.gmail.com>
 <CAExdVN=0_z3x93u7aYT_6CyADPRaf7SxphKR7UV7+kToUwVGeA@mail.gmail.com>
Message-ID: <CAEbHw4YQ5VfSNn+jTncfu5mUz4LV0wnD0EnqK-v7r5noYW3_WQ@mail.gmail.com>

Oops... yeah.  I had fixed the "up to and including" previously, but
somehow I copied the wrong version to the thread.

On Thu, Jul 12, 2018 at 6:36 PM Tim Peters <tim.peters at gmail.com> wrote:

> [David Mertz]
>
>> Miller-Rabin or other pseudo-primality tests do not produce false
>> negatives IIUC.
>>
>
> That's true if they're properly implemented ;-)  If they say False, the
> input is certainly composite (but there isn't a clue as to what a factor
> may be); if the say True, the input is "probably" a prime.
>
>
>> I'm more concerned in the space/time tradeoff in the primes() generator.
>> I like this implementation for teaching purposes, but I'm well aware that
>> it grows in memory usage rather quickly (the one Tim Peters posted is
>> probably a better balance; but it depends on how many of these you create
>> and how long you run them).
>>
>
> As you noted later:
>
> I take it back... Tim's (really Will Ness's) version is *always* faster
>> and more
>
> memory friendly.
>
>
> The original version of this came from an ActiveState recipe that many on
> c.l.py contributed to (including me) years ago, and that's where all the
> speed came from.  There is, for example, no division or square roots.
> Will's contribution was the unexpected way to slash memory use, via the
> generator calling itself recursively.  Elegant and effective!
>
>
>> I'm still trying to understand it though :-).
>
>
> Picture doing the Sieve of Eratosthenes by hand, crossing out multiples of
> "the next" prime you find.  That's basically what it's doing, except doing
> the "crossing out" part incrementally, using a dict to remember, for each
> prime p, "the next" multiple of p you haven't yet gotten to.
>
>
>
>> from math import sqrt, ceil
>>
>> def up_to(seq, lim):
>>     for n in seq:
>>         if n < lim:
>>             yield n
>>         else:
>>             break
>>
>> def sieve_generator():
>>     "Pretty good Sieve; skip the even numbers, stop at sqrt(candidate)"
>>     yield 2
>>     candidate = 3
>>     found = []
>>     while True:
>>         lim = int(ceil(sqrt(candidate)))
>>         if all(candidate % prime != 0 for prime in up_to(found, lim)):
>>             yield candidate
>>             found.append(candidate)
>>         candidate += 2
>>
>>
>>
> Note that this generates squares of odd primes too (9, 25, ...).  `up_to`
> needs to be more like `up_to_and_including`.
>
>
>

-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180712/c6eb3550/attachment-0001.html>

From tim.peters at gmail.com  Thu Jul 12 19:52:55 2018
From: tim.peters at gmail.com (Tim Peters)
Date: Thu, 12 Jul 2018 18:52:55 -0500
Subject: [Python-ideas] Add the imath module
In-Reply-To: <CAEbHw4YQ5VfSNn+jTncfu5mUz4LV0wnD0EnqK-v7r5noYW3_WQ@mail.gmail.com>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <20180712181100.GB7318@ando.pearwood.info>
 <CAPTjJmqS+2LSiM0XWy-gKcgcx1JHOhz3CVvCw=f9j23UdSNFqg@mail.gmail.com>
 <CAEbHw4biUDjz9_uY1AbuPizYOqSGNa0nQfMTKKfOUJ2Mj-MDkw@mail.gmail.com>
 <CAExdVN=0_z3x93u7aYT_6CyADPRaf7SxphKR7UV7+kToUwVGeA@mail.gmail.com>
 <CAEbHw4YQ5VfSNn+jTncfu5mUz4LV0wnD0EnqK-v7r5noYW3_WQ@mail.gmail.com>
Message-ID: <CAExdVNkbma60os3A5TKz-Ptcbt+h+_A_adadcDwfFY3hpNzEYA@mail.gmail.com>

[David Mertz]

> Oops... yeah.  I had fixed the "up to and including" previously, but
> somehow I copied the wrong version to the thread.
>

Doesn't matter ;-)  The point here, to me, is that the prime generator I
pointed at is significantly faster, more memory-frugal, and even "more
correct", than even experienced Python programmers are likely to come up
with at first.

That gives it real value as a candidate for a standard library.  But
something much fancier than that?  I don't think so, for reasons already
given - the code is still understandable for non-experts if they give it
some effort.  Fancier stuff may not be.

For example, in my own code I use a fancier version of that incorporating a
wheel sieve too, and even the _setup_ function for that is harder to
understand all on its own:

def _setup_sieve(ps, show=True):
    from math import gcd

    assert ps[0] == 2
    prod = totient = 1
    for p in ps:
        assert pp(p)
        prod *= p
        totient *= p-1
    if show:
        print("ps", ps, "prod", prod)
    mod2i = [None] * prod
    mod2i[1] = 0
    deltas = []
    last = 1
    for i in range(3, prod, 2):
        if gcd(prod, i) == 1:
            deltas.append(i - last)
            mod2i[i] = len(deltas)
            last = i
    deltas.append(2)    # wrap around from prod-1 to 1
    assert len(deltas) == totient
    assert sum(deltas) == prod
    if show:
        print("deltas", deltas, len(deltas))
        print("mod2i", mod2i)
    return ps, deltas * 2, mod2i

_siv_glob = _setup_sieve([2, 3, 5, 7], show=False)

I don't want that in the standard library - and twice as much don't want it
if somebody else wrote  it ;-)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180712/8cbcc083/attachment.html>

From steve at pearwood.info  Thu Jul 12 20:40:48 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 13 Jul 2018 10:40:48 +1000
Subject: [Python-ideas] Add the imath module
In-Reply-To: <CAPTjJmqS+2LSiM0XWy-gKcgcx1JHOhz3CVvCw=f9j23UdSNFqg@mail.gmail.com>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <20180712181100.GB7318@ando.pearwood.info>
 <CAPTjJmqS+2LSiM0XWy-gKcgcx1JHOhz3CVvCw=f9j23UdSNFqg@mail.gmail.com>
Message-ID: <20180713004047.GD7318@ando.pearwood.info>

On Fri, Jul 13, 2018 at 04:20:55AM +1000, Chris Angelico wrote:
> On Fri, Jul 13, 2018 at 4:11 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> > There is no reason why primality testing can't be deterministic up to
> > 2**64, and probabilistic with a ludicrously small chance of false
> > positives beyond that. The implementation I use can be expected to fail
> > on average once every 18 thousand years if you did nothing but test
> > primes every millisecond of the day, 24 hours a day. That's good enough
> > for most purposes :-)
> 
> What about false negatives? Guaranteed none? The failure mode of the
> function should, IMO, be a defined and documented aspect of it.

If a non-deterministic primality tester such as (but not limited to) 
Miller-Rabin says a number is composite, it is definitely composite 
(ignoring bugs in the implementation, or hardware failures). If it says 
it is prime, in the worst case it is merely "probably prime", with an 
error rate proportional to 1/4**k where we can choose the k.

(The bigger the k the more work may be done in the worst case.)


-- 
Steve

From rosuav at gmail.com  Thu Jul 12 20:56:21 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 13 Jul 2018 10:56:21 +1000
Subject: [Python-ideas] Add the imath module
In-Reply-To: <20180713004047.GD7318@ando.pearwood.info>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <20180712181100.GB7318@ando.pearwood.info>
 <CAPTjJmqS+2LSiM0XWy-gKcgcx1JHOhz3CVvCw=f9j23UdSNFqg@mail.gmail.com>
 <20180713004047.GD7318@ando.pearwood.info>
Message-ID: <CAPTjJmrWDGOf9HPd=CAZ8wCgn4JV073nejO5C+JNtSe0s6jZGw@mail.gmail.com>

On Fri, Jul 13, 2018 at 10:40 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> On Fri, Jul 13, 2018 at 04:20:55AM +1000, Chris Angelico wrote:
>> On Fri, Jul 13, 2018 at 4:11 AM, Steven D'Aprano <steve at pearwood.info> wrote:
>> > There is no reason why primality testing can't be deterministic up to
>> > 2**64, and probabilistic with a ludicrously small chance of false
>> > positives beyond that. The implementation I use can be expected to fail
>> > on average once every 18 thousand years if you did nothing but test
>> > primes every millisecond of the day, 24 hours a day. That's good enough
>> > for most purposes :-)
>>
>> What about false negatives? Guaranteed none? The failure mode of the
>> function should, IMO, be a defined and documented aspect of it.
>
> If a non-deterministic primality tester such as (but not limited to)
> Miller-Rabin says a number is composite, it is definitely composite
> (ignoring bugs in the implementation, or hardware failures). If it says
> it is prime, in the worst case it is merely "probably prime", with an
> error rate proportional to 1/4**k where we can choose the k.
>
> (The bigger the k the more work may be done in the worst case.)

You can say that about algorithms easily enough. My point is that this
ought to be a constraint on the function - implementations may choose
other algorithms, but they MUST follow one pattern or the other,
meaning that a Python script can depend on it without knowing the
implementation. Like guaranteeing that list.sort() is stable without
stipulating the actual sort algo used.

ChrisA

From steve at pearwood.info  Thu Jul 12 21:42:48 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 13 Jul 2018 11:42:48 +1000
Subject: [Python-ideas] Add the imath module
In-Reply-To: <CAExdVN=0_z3x93u7aYT_6CyADPRaf7SxphKR7UV7+kToUwVGeA@mail.gmail.com>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <20180712181100.GB7318@ando.pearwood.info>
 <CAPTjJmqS+2LSiM0XWy-gKcgcx1JHOhz3CVvCw=f9j23UdSNFqg@mail.gmail.com>
 <CAEbHw4biUDjz9_uY1AbuPizYOqSGNa0nQfMTKKfOUJ2Mj-MDkw@mail.gmail.com>
 <CAExdVN=0_z3x93u7aYT_6CyADPRaf7SxphKR7UV7+kToUwVGeA@mail.gmail.com>
Message-ID: <20180713014248.GE7318@ando.pearwood.info>

On Thu, Jul 12, 2018 at 05:35:54PM -0500, Tim Peters wrote:
> [David Mertz]
> 
> > Miller-Rabin or other pseudo-primality tests do not produce false
> > negatives IIUC.
> >
> 
> That's true if they're properly implemented ;-)  If they say False, the
> input is certainly composite (but there isn't a clue as to what a factor
> may be); if the say True, the input is "probably" a prime.

I'm sure Tim knows this, but that's only sometimes true. Since I had no 
formal education on primality testing and had to pick this up in dribs 
and drabs over many years, I was under the impression for the longest 
time that Miller-Rabin was always probabilitistic, but that's not quite 
right.

Without going into the gory details, it turns out that for any N, you 
can do a brute-force primality test by checking against *every* 
candidate up to some maximum value. (Which is how trial division works, 
only Miller-Rabin tests are more expensive than a single division.) Such 
an exhaustive check is not practical, hence the need to fall back on 
randomly choosing candidates.

However, that's in the most general case. At least for small enough N, 
there are rather small sets of candidates which give a completely 
deterministic and conclusive result. For N <= 2**64, it never requires 
more than 12 Miller-Rabin tests to give a conclusive answer, and for 
some N, as few as a single test is enough.

To be clear, these are specially chosen tests, not random tests.

So in a good implementation, for N up to 2**64, there is never a need 
for a "probably prime" result. It either is, or isn't prime, and there's 
no question which.

In principle, there could be similar small sets of conclusive tests for 
larger N too, but (as far as I know) nobody has discovered them yet. 
Hence we fall back on choosing Miller-Rabin tests at random. The chances 
of an arbitrary composite number passing k such tests is on average 
1/4**k, so we can make the average probability of failure as small as we 
like, by doing more tests. With a dozen or twenty tests, the probability 
of a false positive (a composite number wrongly reported as prime) is of 
the same order of magnitude as that of a passing cosmic ray flipping a 
bit in memory and causing the wrong answer to appear.



-- 
Steve

From steve at pearwood.info  Thu Jul 12 21:48:43 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 13 Jul 2018 11:48:43 +1000
Subject: [Python-ideas] Add the imath module
In-Reply-To: <CAPTjJmrWDGOf9HPd=CAZ8wCgn4JV073nejO5C+JNtSe0s6jZGw@mail.gmail.com>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <20180712181100.GB7318@ando.pearwood.info>
 <CAPTjJmqS+2LSiM0XWy-gKcgcx1JHOhz3CVvCw=f9j23UdSNFqg@mail.gmail.com>
 <20180713004047.GD7318@ando.pearwood.info>
 <CAPTjJmrWDGOf9HPd=CAZ8wCgn4JV073nejO5C+JNtSe0s6jZGw@mail.gmail.com>
Message-ID: <20180713014842.GF7318@ando.pearwood.info>

On Fri, Jul 13, 2018 at 10:56:21AM +1000, Chris Angelico wrote:

> You can say that about algorithms easily enough. My point is that this
> ought to be a constraint on the function - implementations may choose
> other algorithms, but they MUST follow one pattern or the other,
> meaning that a Python script can depend on it without knowing the
> implementation. Like guaranteeing that list.sort() is stable without
> stipulating the actual sort algo used.

I cannot imagine an algorithm that wasn't totally brain-dead (like "flip 
a coin") which could wrongly report a prime number as composite. Maybe 
I'm not imaginative enough :-)

But yeah, if you want to formally specify that any such isprime test 
will never have false negatives (never report an actual prime as 
composite), sure thing. I think that's stating the bleeding obvious, 
like demanding that any algorithm we use for factorial(n) must not 
return the sqrt of n by mistake, but whatever :-)

I suppose that if you cared more about running time than correctness, 
one might simply report anything bigger than (let's say) 2**20 as 
"composite". But I call that "brain dead" :-)



-- 
Steve

From rosuav at gmail.com  Thu Jul 12 22:02:21 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 13 Jul 2018 12:02:21 +1000
Subject: [Python-ideas] Add the imath module
In-Reply-To: <20180713014842.GF7318@ando.pearwood.info>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <20180712181100.GB7318@ando.pearwood.info>
 <CAPTjJmqS+2LSiM0XWy-gKcgcx1JHOhz3CVvCw=f9j23UdSNFqg@mail.gmail.com>
 <20180713004047.GD7318@ando.pearwood.info>
 <CAPTjJmrWDGOf9HPd=CAZ8wCgn4JV073nejO5C+JNtSe0s6jZGw@mail.gmail.com>
 <20180713014842.GF7318@ando.pearwood.info>
Message-ID: <CAPTjJmrzrugNqK-+pUDAsqcLOihUwgVYo=k1DsdZU5HN6WTZJw@mail.gmail.com>

On Fri, Jul 13, 2018 at 11:48 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> On Fri, Jul 13, 2018 at 10:56:21AM +1000, Chris Angelico wrote:
>
>> You can say that about algorithms easily enough. My point is that this
>> ought to be a constraint on the function - implementations may choose
>> other algorithms, but they MUST follow one pattern or the other,
>> meaning that a Python script can depend on it without knowing the
>> implementation. Like guaranteeing that list.sort() is stable without
>> stipulating the actual sort algo used.
>
> I cannot imagine an algorithm that wasn't totally brain-dead (like "flip
> a coin") which could wrongly report a prime number as composite. Maybe
> I'm not imaginative enough :-)

Haha. Okay. I'm not familiar with every possible primality test, so I
had no idea that they ALL have the same failure mode :)

ChrisA

From tim.peters at gmail.com  Fri Jul 13 00:58:53 2018
From: tim.peters at gmail.com (Tim Peters)
Date: Thu, 12 Jul 2018 23:58:53 -0500
Subject: [Python-ideas] Add the imath module
In-Reply-To: <CAPTjJmrWDGOf9HPd=CAZ8wCgn4JV073nejO5C+JNtSe0s6jZGw@mail.gmail.com>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <20180712181100.GB7318@ando.pearwood.info>
 <CAPTjJmqS+2LSiM0XWy-gKcgcx1JHOhz3CVvCw=f9j23UdSNFqg@mail.gmail.com>
 <20180713004047.GD7318@ando.pearwood.info>
 <CAPTjJmrWDGOf9HPd=CAZ8wCgn4JV073nejO5C+JNtSe0s6jZGw@mail.gmail.com>
Message-ID: <CAExdVNmBmK=TtDVWK+djgmNKCG=qzbW1r556HVkrk7HBe3qCBg@mail.gmail.com>

[Chris Angelico, on "probable prime" testers]

> You can say that about algorithms easily enough. My point is that this
> ought to be a constraint on the function - implementations may choose
> other algorithms, but they MUST follow one pattern or the other,
> meaning that a Python script can depend on it without knowing the
> implementation. Like guaranteeing that list.sort() is stable without
> stipulating the actual sort algo used.
>

I agree!  Don't worry about it - if this module gets added, I'll make sure
of it.  My own "probable prime" function starts like so:

def pp(n, k=10):
    """
    Return True iff n is a probable prime, using k Miller-Rabin tests.

    When False, n is definitely composite.
    """

In the context of algorithmic number theory, "no false negatives" is
implied by that context, but it should be spelled out anyway.  A "probable
prime", by definition, is an integer that satisfies some property that
_all_ primes satisfy, but that some composites may satisfy too.  The
variation among probable prime algorithms is in which specific
property(ies) they test for.

For example:

def pp1(n):
    return True

meets the promise, but is useless ;-)

def pp2(n):
    return  n % 3 != 0 or n == 3

is slightly less useless.

But

def pp3(n):
    return n % 3 != 0

would be incorrect, because pp3(3) returns False - `n % 3 != 0` is not a
property that all primes satisfy.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180712/aa1a949e/attachment.html>

From rosuav at gmail.com  Fri Jul 13 01:00:41 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 13 Jul 2018 15:00:41 +1000
Subject: [Python-ideas] Add the imath module
In-Reply-To: <CAExdVNmBmK=TtDVWK+djgmNKCG=qzbW1r556HVkrk7HBe3qCBg@mail.gmail.com>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <20180712181100.GB7318@ando.pearwood.info>
 <CAPTjJmqS+2LSiM0XWy-gKcgcx1JHOhz3CVvCw=f9j23UdSNFqg@mail.gmail.com>
 <20180713004047.GD7318@ando.pearwood.info>
 <CAPTjJmrWDGOf9HPd=CAZ8wCgn4JV073nejO5C+JNtSe0s6jZGw@mail.gmail.com>
 <CAExdVNmBmK=TtDVWK+djgmNKCG=qzbW1r556HVkrk7HBe3qCBg@mail.gmail.com>
Message-ID: <CAPTjJmoYVibNSvD71xYRuRe_ig6oL60NQ5JxuaQA__KXJiZT+g@mail.gmail.com>

On Fri, Jul 13, 2018 at 2:58 PM, Tim Peters <tim.peters at gmail.com> wrote:
> [Chris Angelico, on "probable prime" testers]
>>
>> You can say that about algorithms easily enough. My point is that this
>> ought to be a constraint on the function - implementations may choose
>> other algorithms, but they MUST follow one pattern or the other,
>> meaning that a Python script can depend on it without knowing the
>> implementation. Like guaranteeing that list.sort() is stable without
>> stipulating the actual sort algo used.
>
>
> I agree!  Don't worry about it - if this module gets added, I'll make sure
> of it.  My own "probable prime" function starts like so:
>
> def pp(n, k=10):
>     """
>     Return True iff n is a probable prime, using k Miller-Rabin tests.
>
>     When False, n is definitely composite.
>     """
>
> In the context of algorithmic number theory, "no false negatives" is implied
> by that context, but it should be spelled out anyway.  A "probable prime",
> by definition, is an integer that satisfies some property that _all_ primes
> satisfy, but that some composites may satisfy too.  The variation among
> probable prime algorithms is in which specific property(ies) they test for.
>
> For example:
>
> def pp1(n):
>     return True
>
> meets the promise, but is useless ;-)
>
> def pp2(n):
>     return  n % 3 != 0 or n == 3
>
> is slightly less useless.
>
> But
>
> def pp3(n):
>     return n % 3 != 0
>
> would be incorrect, because pp3(3) returns False - `n % 3 != 0` is not a
> property that all primes satisfy.
>

Thank you. Great explanation. Much appreciated!

ChrisA

From chris.barker at noaa.gov  Fri Jul 13 01:08:47 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Fri, 13 Jul 2018 00:08:47 -0500
Subject: [Python-ideas] Fwd: grouping / dict of lists
In-Reply-To: <CAEbHw4Y7A4H3XZvRTgFb_z84CqAVWbe0KAM_xVgUAHCoYhw9pw@mail.gmail.com>
References: <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAOmnVUBp9vJjTp6v17LHy2C=7GzR63fb4ARZW2Dco022uTDpfA@mail.gmail.com>
 <20180703153338.GG14437@ando.pearwood.info>
 <CALGmxEJBYFEoPOFo7aVSN=Xy=sgSMnrF0k8pnt=VhJ6cvhS_4Q@mail.gmail.com>
 <20180704013202.GK14437@ando.pearwood.info>
 <CAEbHw4YafoZc_1cnB6a-psmWqW-SD2tw3E2Fw5hUHSJxqEsu-Q@mail.gmail.com>
 <CAEbHw4Yjue-V_CKQJe2k6f4E8ucNuG=ZNFt5O_njfrQoNKHeUw@mail.gmail.com>
 <CALGmxE+m+1YXk1vRsdkAK6yQ12V=syi6WKZsSTMFmw1BoiKUYA@mail.gmail.com>
 <CAB_e7izyUbkOXxpwivr0iyE4gXk58xn+SBfAZq5hcKAyDO-H_g@mail.gmail.com>
 <CALGmxE+OS-EDa7gNceEirnW1S0oryyinA7mMbUetXfZkJ67Zkw@mail.gmail.com>
 <CAEbHw4Y7A4H3XZvRTgFb_z84CqAVWbe0KAM_xVgUAHCoYhw9pw@mail.gmail.com>
Message-ID: <CALGmxEJRR+WhC88ZbV-DDcW1o5oz4S91oxG1apqm=diBHPhgDA@mail.gmail.com>

On Mon, Jul 9, 2018 at 3:38 PM, David Mertz <mertz at gnosis.cx> wrote:

> In my mind, I *rarely* (which is more than never) have my data in the form
> of a sequence of key/value pairs.  The version of the API that assumes data
> starts that way feels like either a niche case, or demands preprocessing
> before it's ready to pass to grouping() or collections.Grouping().
>

sure, but it makes it easy to use a different approach -- i.e. a
comprehension with expressions rather than a key (and maybe value)
function. AT least two people have expressed a preference for that.

That said, an identity key is rarely interesting either.
>

is it EVER interesting?? wouldn't it be essentially a Counter, without the
counting? :-)



> So I think have key=None mean "assume we get key/val pairs is harmless to
> the more common case where we give an explicit key function.
>

I'm not sure we'll know what's more common 'till it's loose in the wild --
any problem can be solved either way -- which one will people prefer?

Given that there Is a preference for comprehensions over map() -- I think
it will see a lot of use.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180713/008e669c/attachment-0001.html>

From chris.barker at noaa.gov  Fri Jul 13 01:11:46 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Fri, 13 Jul 2018 00:11:46 -0500
Subject: [Python-ideas] Fwd: grouping / dict of lists
In-Reply-To: <CAB_e7iw=mCRtBGu2YJ_kV7yffSWh_Rp927Lpw_gwiyzbJtoqzA@mail.gmail.com>
References: <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAOmnVUBp9vJjTp6v17LHy2C=7GzR63fb4ARZW2Dco022uTDpfA@mail.gmail.com>
 <20180703153338.GG14437@ando.pearwood.info>
 <CALGmxEJBYFEoPOFo7aVSN=Xy=sgSMnrF0k8pnt=VhJ6cvhS_4Q@mail.gmail.com>
 <20180704013202.GK14437@ando.pearwood.info>
 <CAEbHw4YafoZc_1cnB6a-psmWqW-SD2tw3E2Fw5hUHSJxqEsu-Q@mail.gmail.com>
 <CAEbHw4Yjue-V_CKQJe2k6f4E8ucNuG=ZNFt5O_njfrQoNKHeUw@mail.gmail.com>
 <CALGmxE+m+1YXk1vRsdkAK6yQ12V=syi6WKZsSTMFmw1BoiKUYA@mail.gmail.com>
 <CAB_e7izyUbkOXxpwivr0iyE4gXk58xn+SBfAZq5hcKAyDO-H_g@mail.gmail.com>
 <CALGmxE+OS-EDa7gNceEirnW1S0oryyinA7mMbUetXfZkJ67Zkw@mail.gmail.com>
 <CAB_e7iw=mCRtBGu2YJ_kV7yffSWh_Rp927Lpw_gwiyzbJtoqzA@mail.gmail.com>
Message-ID: <CALGmxE+tbVu52D5xDoCKdWOWjg_Q-hzGyX7fvLn74_Xb_mfQ_A@mail.gmail.com>

On Mon, Jul 9, 2018 at 5:55 PM, Franklin? Lee <leewangzhong+python at gmail.com
> wrote:

> >> - The storage container.
> >
> >
> > so this means you'r passing in a full set of storage containers? I'm a
> vit
> > confused by that -- if they might be pre-populated, then they would need
> to
> > be instance,s an you'd need to have one for every key -- how would you
> know
> > in advance aht you needed???
>
> No, I mean the mapping (outer) container. For example, I can pass in
> an empty OrderedDict, or a dict that already contained some groups
> from a previous call to the grouping function.
>

Sure -- that's what my prototype does if you pass a Mapping in (or use
.update() )

why not?

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180713/d13fdb7a/attachment.html>

From chris.barker at noaa.gov  Fri Jul 13 01:28:01 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Fri, 13 Jul 2018 00:28:01 -0500
Subject: [Python-ideas] Add the imath module
In-Reply-To: <20180712160529.GV7318@ando.pearwood.info>
References: <pi7fga$flg$1@blaine.gmane.org>
 <20180712160529.GV7318@ando.pearwood.info>
Message-ID: <CALGmxEKCMGp6OJ9dvFv42zu1GDQLm2X07TynZBYPAncfACXW_Q@mail.gmail.com>

On Thu, Jul 12, 2018 at 11:05 AM, Steven D'Aprano <steve at pearwood.info>
wrote:

> I'm not sure that we need a specific module for integer-valued maths. I
> think a more important change would be to allow math functions to be
> written in Python, not just C:
>
> - move the current math library to a private _math library;
>
> - add math.py, which calls "from _math import *".
>

FWIW, Guido rejected that idea when we added math.isclose() -- Victor had
even written it up already.

Not that people never change their mind...

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180713/426a9cd0/attachment.html>

From J.Demeyer at UGent.be  Fri Jul 13 04:10:00 2018
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Fri, 13 Jul 2018 10:10:00 +0200
Subject: [Python-ideas] Add the imath module
In-Reply-To: <51e71cc9801047e9b21c4dc01c64826e@xmail103.UGent.be>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <51e71cc9801047e9b21c4dc01c64826e@xmail103.UGent.be>
Message-ID: <5B485E58.1000503@UGent.be>

On 2018-07-12 20:11, Steven D'Aprano wrote:
> about 4.7 seconds to test 2**800 + 1;

In SageMath:

sage: n = 2**800+1; timeit('is_prime(n)')
625 loops, best of 3: 303 ?s per loop

That's 4 orders of magnitude faster...

From J.Demeyer at UGent.be  Fri Jul 13 04:21:51 2018
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Fri, 13 Jul 2018 10:21:51 +0200
Subject: [Python-ideas] Add the imath module
In-Reply-To: <54da9da43f254d2ab23ed4d91e78984d@xmail103.UGent.be>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <20180712181100.GB7318@ando.pearwood.info>
 <CAPTjJmqS+2LSiM0XWy-gKcgcx1JHOhz3CVvCw=f9j23UdSNFqg@mail.gmail.com>
 <20180713004047.GD7318@ando.pearwood.info>
 <CAPTjJmrWDGOf9HPd=CAZ8wCgn4JV073nejO5C+JNtSe0s6jZGw@mail.gmail.com>
 <20180713014842.GF7318@ando.pearwood.info>
 <54da9da43f254d2ab23ed4d91e78984d@xmail103.UGent.be>
Message-ID: <5B48611F.9040406@UGent.be>

On 2018-07-13 04:02, Chris Angelico wrote:
> Haha. Okay. I'm not familiar with every possible primality test, so I
> had no idea that they ALL have the same failure mode :)

There exist guaranteed primality tests, but those are much slower and 
more complicated. The state of the art is ECPP:

https://en.wikipedia.org/wiki/Elliptic_curve_primality

From j.van.dorp at deonet.nl  Fri Jul 13 04:43:12 2018
From: j.van.dorp at deonet.nl (Jacco van Dorp)
Date: Fri, 13 Jul 2018 10:43:12 +0200
Subject: [Python-ideas] Add the imath module
In-Reply-To: <5B48611F.9040406@UGent.be>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <20180712181100.GB7318@ando.pearwood.info>
 <CAPTjJmqS+2LSiM0XWy-gKcgcx1JHOhz3CVvCw=f9j23UdSNFqg@mail.gmail.com>
 <20180713004047.GD7318@ando.pearwood.info>
 <CAPTjJmrWDGOf9HPd=CAZ8wCgn4JV073nejO5C+JNtSe0s6jZGw@mail.gmail.com>
 <20180713014842.GF7318@ando.pearwood.info>
 <54da9da43f254d2ab23ed4d91e78984d@xmail103.UGent.be>
 <5B48611F.9040406@UGent.be>
Message-ID: <CAKnNRP=D5PAnN=cY2GXPQCY6GT1CQvR3ouxV991R5pzLyG_pHw@mail.gmail.com>

If there's going to be failure possible primality tests, there should
also exist certain ones. No matter how slow it can be to do it.

Python if often a language for beginners. I think it'd be confusing to
a lot of people that there's going to be tests that show "This one is
probably prime", instead of sure tests.

I'd call the functions is_prime() and is_likely_prime() (or
is_likely_not_prime(), depending on where it fails), to make it really
clear which you're picking and avoid confusion.

2018-07-13 10:21 GMT+02:00 Jeroen Demeyer <J.Demeyer at ugent.be>:
> On 2018-07-13 04:02, Chris Angelico wrote:
>>
>> Haha. Okay. I'm not familiar with every possible primality test, so I
>> had no idea that they ALL have the same failure mode :)
>
>
> There exist guaranteed primality tests, but those are much slower and more
> complicated. The state of the art is ECPP:
>
> https://en.wikipedia.org/wiki/Elliptic_curve_primality
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From J.Demeyer at UGent.be  Fri Jul 13 05:13:53 2018
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Fri, 13 Jul 2018 11:13:53 +0200
Subject: [Python-ideas] Add the imath module
In-Reply-To: <f9c10e21dbd940838a3e3b2b7be78e31@xmail103.UGent.be>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <20180712181100.GB7318@ando.pearwood.info>
 <CAPTjJmqS+2LSiM0XWy-gKcgcx1JHOhz3CVvCw=f9j23UdSNFqg@mail.gmail.com>
 <20180713004047.GD7318@ando.pearwood.info>
 <CAPTjJmrWDGOf9HPd=CAZ8wCgn4JV073nejO5C+JNtSe0s6jZGw@mail.gmail.com>
 <20180713014842.GF7318@ando.pearwood.info>
 <54da9da43f254d2ab23ed4d91e78984d@xmail103.UGent.be>
 <5B48611F.9040406@UGent.be>
 <f9c10e21dbd940838a3e3b2b7be78e31@xmail103.UGent.be>
Message-ID: <5B486D51.3050305@UGent.be>

On 2018-07-13 10:43, Jacco van Dorp wrote:
> If there's going to be failure possible primality tests, there should
> also exist certain ones. No matter how slow it can be to do it.

A certain primality test would be too specialized for the Python stdlib. 
If you really need that (which you typically don't), there are existing 
number theory packages.

From j.van.dorp at deonet.nl  Fri Jul 13 05:50:57 2018
From: j.van.dorp at deonet.nl (Jacco van Dorp)
Date: Fri, 13 Jul 2018 11:50:57 +0200
Subject: [Python-ideas] Add the imath module
In-Reply-To: <5B486D51.3050305@UGent.be>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <20180712181100.GB7318@ando.pearwood.info>
 <CAPTjJmqS+2LSiM0XWy-gKcgcx1JHOhz3CVvCw=f9j23UdSNFqg@mail.gmail.com>
 <20180713004047.GD7318@ando.pearwood.info>
 <CAPTjJmrWDGOf9HPd=CAZ8wCgn4JV073nejO5C+JNtSe0s6jZGw@mail.gmail.com>
 <20180713014842.GF7318@ando.pearwood.info>
 <54da9da43f254d2ab23ed4d91e78984d@xmail103.UGent.be>
 <5B48611F.9040406@UGent.be>
 <f9c10e21dbd940838a3e3b2b7be78e31@xmail103.UGent.be>
 <5B486D51.3050305@UGent.be>
Message-ID: <CAKnNRPmzJfwa1Z_Giiu-j3ACTpe1+gOwJGvXh4xf7Sdj9PYSGg@mail.gmail.com>

2018-07-13 11:13 GMT+02:00 Jeroen Demeyer <J.Demeyer at ugent.be>:
> On 2018-07-13 10:43, Jacco van Dorp wrote:
>>
>> If there's going to be failure possible primality tests, there should
>> also exist certain ones. No matter how slow it can be to do it.
>
>
> A certain primality test would be too specialized for the Python stdlib. If
> you really need that (which you typically don't), there are existing number
> theory packages.

You forgot to include my reasons for this:

>> Python if often a language for beginners. I think it'd be confusing to
>> a lot of people that there's going to be tests that show "This one is
>> probably prime", instead of sure tests.

Depending on function names, of course. It needn't be the fastest
available, but its what people would expect, especially those with
less experience. Also, there's existing number theory packages for
everything suggested here, so that's not an argument.

At the very least, use is_likely_prime() as function name. Otherwise,
the name of the function is just plain wrong. Don't claim accuracy you
don't have.

From J.Demeyer at UGent.be  Fri Jul 13 07:36:55 2018
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Fri, 13 Jul 2018 13:36:55 +0200
Subject: [Python-ideas] Add the imath module
In-Reply-To: <8f7a10f5c4234a70a4d217f3731d8b80@xmail103.UGent.be>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <20180712181100.GB7318@ando.pearwood.info>
 <CAPTjJmqS+2LSiM0XWy-gKcgcx1JHOhz3CVvCw=f9j23UdSNFqg@mail.gmail.com>
 <20180713004047.GD7318@ando.pearwood.info>
 <CAPTjJmrWDGOf9HPd=CAZ8wCgn4JV073nejO5C+JNtSe0s6jZGw@mail.gmail.com>
 <20180713014842.GF7318@ando.pearwood.info>
 <54da9da43f254d2ab23ed4d91e78984d@xmail103.UGent.be>
 <5B48611F.9040406@UGent.be>
 <f9c10e21dbd940838a3e3b2b7be78e31@xmail103.UGent.be>
 <5B486D51.3050305@UGent.be>
 <8f7a10f5c4234a70a4d217f3731d8b80@xmail103.UGent.be>
Message-ID: <5B488ED7.8060209@UGent.be>

On 2018-07-13 11:50, Jacco van Dorp wrote:
> At the very least, use is_likely_prime() as function name.

I agree with you here: the function name should reflect what it actually 
does. (Note that the technical term is "probable prime", so it should be 
is_probable_prime)

But I don't think that the Python stdlib should have a proven is_prime() 
function because that would be complicated, slow and have very limited 
benefit over is_probable_prime().

From steve at pearwood.info  Fri Jul 13 07:58:57 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 13 Jul 2018 21:58:57 +1000
Subject: [Python-ideas] Add the imath module
In-Reply-To: <CAKnNRP=D5PAnN=cY2GXPQCY6GT1CQvR3ouxV991R5pzLyG_pHw@mail.gmail.com>
References: <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <20180712181100.GB7318@ando.pearwood.info>
 <CAPTjJmqS+2LSiM0XWy-gKcgcx1JHOhz3CVvCw=f9j23UdSNFqg@mail.gmail.com>
 <20180713004047.GD7318@ando.pearwood.info>
 <CAPTjJmrWDGOf9HPd=CAZ8wCgn4JV073nejO5C+JNtSe0s6jZGw@mail.gmail.com>
 <20180713014842.GF7318@ando.pearwood.info>
 <54da9da43f254d2ab23ed4d91e78984d@xmail103.UGent.be>
 <5B48611F.9040406@UGent.be>
 <CAKnNRP=D5PAnN=cY2GXPQCY6GT1CQvR3ouxV991R5pzLyG_pHw@mail.gmail.com>
Message-ID: <20180713115857.GG7318@ando.pearwood.info>

On Fri, Jul 13, 2018 at 10:43:12AM +0200, Jacco van Dorp wrote:
> If there's going to be failure possible primality tests, there should
> also exist certain ones. No matter how slow it can be to do it.
>
> Python if often a language for beginners. I think it'd be confusing to
> a lot of people that there's going to be tests that show "This one is
> probably prime", instead of sure tests.

I think that it is precisely because this will be used by beginners that 
we shouldn't offer such a choice. Its our job to curate the APIs in the 
std lib, not to burden inexpert users with excess choices that will only 
worry them.

If this were a third-party module, I would say, "the more the merrier". 
Let it provide a dozen different implementations, that's cool. But the 
stdlib is batteries, not nuclear reactors. If you want a choice of 
algorithms, or the ability to configure space versus time versus 
guarantees, I think you ought to be using a 3rd party module, not the 
std library.


I'm not saying hide the fact that it is probabilistic, but that's an 
implementation detail that *in practice* makes no difference at all. 
Using Miller-Rabin and 30 randomly chosen witnesses, the probability of 
a wrong answer is < 0.000000000000000001. If we tested a million numbers 
a second, it would take an average of 18,000+ years to come across a 
single such error.

In comparison, your computer probably experiences something of the order 
of 1 cosmic-ray induced bit-flip causing data corruption per week.

https://stackoverflow.com/questions/2580933/cosmic-rays-what-is-the-probability-they-will-affect-a-program

Since using probabilistic methods is good enough for PGP and other 
high-security protocols, it's surely good enough for Tom Schoolboy 
testing whether 1234567890123456789 is prime.

(For the record: it isn't.)


-- 
Steve

From steve at pearwood.info  Fri Jul 13 08:11:11 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 13 Jul 2018 22:11:11 +1000
Subject: [Python-ideas] Add the imath module
In-Reply-To: <5B488ED7.8060209@UGent.be>
References: <CAPTjJmqS+2LSiM0XWy-gKcgcx1JHOhz3CVvCw=f9j23UdSNFqg@mail.gmail.com>
 <20180713004047.GD7318@ando.pearwood.info>
 <CAPTjJmrWDGOf9HPd=CAZ8wCgn4JV073nejO5C+JNtSe0s6jZGw@mail.gmail.com>
 <20180713014842.GF7318@ando.pearwood.info>
 <54da9da43f254d2ab23ed4d91e78984d@xmail103.UGent.be>
 <5B48611F.9040406@UGent.be>
 <f9c10e21dbd940838a3e3b2b7be78e31@xmail103.UGent.be>
 <5B486D51.3050305@UGent.be>
 <8f7a10f5c4234a70a4d217f3731d8b80@xmail103.UGent.be>
 <5B488ED7.8060209@UGent.be>
Message-ID: <20180713121110.GH7318@ando.pearwood.info>

On Fri, Jul 13, 2018 at 01:36:55PM +0200, Jeroen Demeyer wrote:
> On 2018-07-13 11:50, Jacco van Dorp wrote:
> >At the very least, use is_likely_prime() as function name.
> 
> I agree with you here: the function name should reflect what it actually 
> does.

What it *actually* does is:

is_almost_certainly_prime_except_for_a_ludicrously_microscopic_chance_of_error_thousands_of_times_less_likely_than_a_stray_cosmic_ray_flipping_a_bit_in_memory_and_causing_the_wrong_result_to_be_returned()

*At most*, I would agree that the isprime() function could raise a 
"Probably Prime" warning when necessary, with the warning disabled by 
default. Those that care about this sort of thing can enable warnings, 
those that don't shouldn't have to read scary warnings that have no 
practical meaning.

If your bank is satisfied with "mere probable prime number" to transfer 
billions of dollars around the world, then I'm sure that the users of 
Python's std library should be too.


-- 
Steve

From steve at pearwood.info  Fri Jul 13 08:26:42 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 13 Jul 2018 22:26:42 +1000
Subject: [Python-ideas] Add the imath module
In-Reply-To: <5B485E58.1000503@UGent.be>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <51e71cc9801047e9b21c4dc01c64826e@xmail103.UGent.be>
 <5B485E58.1000503@UGent.be>
Message-ID: <20180713122642.GI7318@ando.pearwood.info>

On Fri, Jul 13, 2018 at 10:10:00AM +0200, Jeroen Demeyer wrote:
> On 2018-07-12 20:11, Steven D'Aprano wrote:
> >about 4.7 seconds to test 2**800 + 1;
> 
> In SageMath:
> 
> sage: n = 2**800+1; timeit('is_prime(n)')
> 625 loops, best of 3: 303 ?s per loop
> 
> That's 4 orders of magnitude faster...

Go on, rub it in why don't you...

I'm pretty sure that Sage is using numpy, which would almost certainly 
have a C-based implementation, not pure Python like mine. And it's 
probably making far less conservative choices about the rate of false 
positives it is willing to accept.

And your computer is probably newer and faster than mine :-(

My point was that for casual use, even the bad cases are still fast 
enough for interactive use. If you think 5 seconds is ridiculously slow, 
you've never tried a naive trial division algorithm that took literally 
days to crawl through the millions of divisions needed.

*wink*



-- 
Steve

From J.Demeyer at UGent.be  Fri Jul 13 08:43:26 2018
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Fri, 13 Jul 2018 14:43:26 +0200
Subject: [Python-ideas] Add the imath module
In-Reply-To: <eb28e452293a43b2a4bf4c250525a8cb@xmail103.UGent.be>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <51e71cc9801047e9b21c4dc01c64826e@xmail103.UGent.be>
 <5B485E58.1000503@UGent.be>
 <eb28e452293a43b2a4bf4c250525a8cb@xmail103.UGent.be>
Message-ID: <5B489E6E.4010301@UGent.be>

On 2018-07-13 14:26, Steven D'Aprano wrote:
> I'm pretty sure that Sage is using numpy

This computation uses PARI/GP, a dedicated library for number theory 
(https://pari.math.u-bordeaux.fr/) with a Python interface 
(https://github.com/defeo/cypari2).

> And it's
> probably making far less conservative choices about the rate of false
> positives it is willing to accept.

It's an actual primality test, not a probable primality test. But for 
numbers which are not prime (like your examples), the difference doesn't 
actually matter.

From J.Demeyer at UGent.be  Fri Jul 13 08:44:55 2018
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Fri, 13 Jul 2018 14:44:55 +0200
Subject: [Python-ideas] Add the imath module
In-Reply-To: <d51b09179e2043869ea0b73e6b54ac9a@xmail103.UGent.be>
References: <CAPTjJmqS+2LSiM0XWy-gKcgcx1JHOhz3CVvCw=f9j23UdSNFqg@mail.gmail.com>
 <20180713004047.GD7318@ando.pearwood.info>
 <CAPTjJmrWDGOf9HPd=CAZ8wCgn4JV073nejO5C+JNtSe0s6jZGw@mail.gmail.com>
 <20180713014842.GF7318@ando.pearwood.info>
 <54da9da43f254d2ab23ed4d91e78984d@xmail103.UGent.be>
 <5B48611F.9040406@UGent.be>
 <f9c10e21dbd940838a3e3b2b7be78e31@xmail103.UGent.be>
 <5B486D51.3050305@UGent.be>
 <8f7a10f5c4234a70a4d217f3731d8b80@xmail103.UGent.be>
 <5B488ED7.8060209@UGent.be>
 <d51b09179e2043869ea0b73e6b54ac9a@xmail103.UGent.be>
Message-ID: <5B489EC7.5080604@UGent.be>

On 2018-07-13 14:11, Steven D'Aprano wrote:
> What it *actually* does is:
>
> is_almost_certainly_prime_except_for_a_ludicrously_microscopic_chance_of_error_thousands_of_times_less_likely_than_a_stray_cosmic_ray_flipping_a_bit_in_memory_and_causing_the_wrong_result_to_be_returned()

That's just a long variant of is_probable_prime()

> If your bank is satisfied with "mere probable prime number" to transfer
> billions of dollars around the world, then I'm sure that the users of
> Python's std library should be too.

That's besides the point. I agree that probable primes are good enough, 
just don't call them "prime".

From j.van.dorp at deonet.nl  Fri Jul 13 08:51:18 2018
From: j.van.dorp at deonet.nl (Jacco van Dorp)
Date: Fri, 13 Jul 2018 14:51:18 +0200
Subject: [Python-ideas] Add the imath module
In-Reply-To: <5B489EC7.5080604@UGent.be>
References: <CAPTjJmqS+2LSiM0XWy-gKcgcx1JHOhz3CVvCw=f9j23UdSNFqg@mail.gmail.com>
 <20180713004047.GD7318@ando.pearwood.info>
 <CAPTjJmrWDGOf9HPd=CAZ8wCgn4JV073nejO5C+JNtSe0s6jZGw@mail.gmail.com>
 <20180713014842.GF7318@ando.pearwood.info>
 <54da9da43f254d2ab23ed4d91e78984d@xmail103.UGent.be>
 <5B48611F.9040406@UGent.be>
 <f9c10e21dbd940838a3e3b2b7be78e31@xmail103.UGent.be>
 <5B486D51.3050305@UGent.be>
 <8f7a10f5c4234a70a4d217f3731d8b80@xmail103.UGent.be>
 <5B488ED7.8060209@UGent.be>
 <d51b09179e2043869ea0b73e6b54ac9a@xmail103.UGent.be>
 <5B489EC7.5080604@UGent.be>
Message-ID: <CAKnNRPm_eVGX=Dx4irF8a7ovPnDuSJT-0p82NkOgACTekEHW7w@mail.gmail.com>

I think the extra 9 characters of "_probable" are worth the extra accuracy.

Dont teach people to lie in their function names. It's a bad example.
and if you really want to lie, just

from imath import is_probably_prime as is_prime

2018-07-13 14:44 GMT+02:00 Jeroen Demeyer <J.Demeyer at ugent.be>:
> On 2018-07-13 14:11, Steven D'Aprano wrote:
>>
>> What it *actually* does is:
>>
>>
>> is_almost_certainly_prime_except_for_a_ludicrously_microscopic_chance_of_error_thousands_of_times_less_likely_than_a_stray_cosmic_ray_flipping_a_bit_in_memory_and_causing_the_wrong_result_to_be_returned()
>
>
> That's just a long variant of is_probable_prime()
>
>> If your bank is satisfied with "mere probable prime number" to transfer
>> billions of dollars around the world, then I'm sure that the users of
>> Python's std library should be too.
>
>
> That's besides the point. I agree that probable primes are good enough, just
> don't call them "prime".
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From stephanh42 at gmail.com  Fri Jul 13 08:51:17 2018
From: stephanh42 at gmail.com (Stephan Houben)
Date: Fri, 13 Jul 2018 14:51:17 +0200
Subject: [Python-ideas] Add the imath module
In-Reply-To: <5B489EC7.5080604@UGent.be>
References: <CAPTjJmqS+2LSiM0XWy-gKcgcx1JHOhz3CVvCw=f9j23UdSNFqg@mail.gmail.com>
 <20180713004047.GD7318@ando.pearwood.info>
 <CAPTjJmrWDGOf9HPd=CAZ8wCgn4JV073nejO5C+JNtSe0s6jZGw@mail.gmail.com>
 <20180713014842.GF7318@ando.pearwood.info>
 <54da9da43f254d2ab23ed4d91e78984d@xmail103.UGent.be>
 <5B48611F.9040406@UGent.be>
 <f9c10e21dbd940838a3e3b2b7be78e31@xmail103.UGent.be>
 <5B486D51.3050305@UGent.be>
 <8f7a10f5c4234a70a4d217f3731d8b80@xmail103.UGent.be>
 <5B488ED7.8060209@UGent.be>
 <d51b09179e2043869ea0b73e6b54ac9a@xmail103.UGent.be>
 <5B489EC7.5080604@UGent.be>
Message-ID: <CAOOa=pNT4uBLUWyBkxkZpPQCkaQzQCSYD-=y6-UmULp3H9Xjfw@mail.gmail.com>

Should we call sort then probable_sort, since the non-zero probability
exists of it going wrong due to a stray cosmic ray?

It's pointless to worry about failure modes which are order of magnitudes
unlikelier than hardware failure.

Stephan

Op vr 13 jul. 2018 14:45 schreef Jeroen Demeyer <J.Demeyer at ugent.be>:

> On 2018-07-13 14:11, Steven D'Aprano wrote:
> > What it *actually* does is:
> >
> >
> is_almost_certainly_prime_except_for_a_ludicrously_microscopic_chance_of_error_thousands_of_times_less_likely_than_a_stray_cosmic_ray_flipping_a_bit_in_memory_and_causing_the_wrong_result_to_be_returned()
>
> That's just a long variant of is_probable_prime()
>
> > If your bank is satisfied with "mere probable prime number" to transfer
> > billions of dollars around the world, then I'm sure that the users of
> > Python's std library should be too.
>
> That's besides the point. I agree that probable primes are good enough,
> just don't call them "prime".
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180713/2717fed6/attachment.html>

From mertz at gnosis.cx  Fri Jul 13 08:58:03 2018
From: mertz at gnosis.cx (David Mertz)
Date: Fri, 13 Jul 2018 08:58:03 -0400
Subject: [Python-ideas] Add the imath module
In-Reply-To: <CAKnNRP=D5PAnN=cY2GXPQCY6GT1CQvR3ouxV991R5pzLyG_pHw@mail.gmail.com>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <20180712181100.GB7318@ando.pearwood.info>
 <CAPTjJmqS+2LSiM0XWy-gKcgcx1JHOhz3CVvCw=f9j23UdSNFqg@mail.gmail.com>
 <20180713004047.GD7318@ando.pearwood.info>
 <CAPTjJmrWDGOf9HPd=CAZ8wCgn4JV073nejO5C+JNtSe0s6jZGw@mail.gmail.com>
 <20180713014842.GF7318@ando.pearwood.info>
 <54da9da43f254d2ab23ed4d91e78984d@xmail103.UGent.be>
 <5B48611F.9040406@UGent.be>
 <CAKnNRP=D5PAnN=cY2GXPQCY6GT1CQvR3ouxV991R5pzLyG_pHw@mail.gmail.com>
Message-ID: <CAEbHw4ZAsdcO9=mjP2NghueTkuZFt9CeD+49eitVOMaNfHjzkQ@mail.gmail.com>

There is a VERY slow certain test for primality: divide by every candidate
factor (up to sqrt(N)). This is certainly not what we want for very large
numbers.

But the uncertainty of the probabilistic tests is very small. Steven
likened the odds of the test being wrong to "less than the chance of a
cosmic ray hitting a memory cell and altering the result." In any case, an
amateur programmer could write a tight loop testing random large numbers
and leave it running for the rest of their life without ever encountering a
false positive. Number theorists have constructed these false positives...
But if you know enough to do that construction, this isn't going to confuse
you.

On Fri, Jul 13, 2018, 4:44 AM Jacco van Dorp <j.van.dorp at deonet.nl> wrote:

> If there's going to be failure possible primality tests, there should
> also exist certain ones. No matter how slow it can be to do it.
>
> Python if often a language for beginners. I think it'd be confusing to
> a lot of people that there's going to be tests that show "This one is
> probably prime", instead of sure tests.
>
> I'd call the functions is_prime() and is_likely_prime() (or
> is_likely_not_prime(), depending on where it fails), to make it really
> clear which you're picking and avoid confusion.
>
> 2018-07-13 10:21 GMT+02:00 Jeroen Demeyer <J.Demeyer at ugent.be>:
> > On 2018-07-13 04:02, Chris Angelico wrote:
> >>
> >> Haha. Okay. I'm not familiar with every possible primality test, so I
> >> had no idea that they ALL have the same failure mode :)
> >
> >
> > There exist guaranteed primality tests, but those are much slower and
> more
> > complicated. The state of the art is ECPP:
> >
> > https://en.wikipedia.org/wiki/Elliptic_curve_primality
> >
> > _______________________________________________
> > Python-ideas mailing list
> > Python-ideas at python.org
> > https://mail.python.org/mailman/listinfo/python-ideas
> > Code of Conduct: http://python.org/psf/codeofconduct/
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180713/226d4896/attachment-0001.html>

From nicolas.rolin at tiime.fr  Fri Jul 13 09:17:17 2018
From: nicolas.rolin at tiime.fr (Nicolas Rolin)
Date: Fri, 13 Jul 2018 15:17:17 +0200
Subject: [Python-ideas] Fwd: grouping / dict of lists
In-Reply-To: <CALGmxE+tbVu52D5xDoCKdWOWjg_Q-hzGyX7fvLn74_Xb_mfQ_A@mail.gmail.com>
References: <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAOmnVUBp9vJjTp6v17LHy2C=7GzR63fb4ARZW2Dco022uTDpfA@mail.gmail.com>
 <20180703153338.GG14437@ando.pearwood.info>
 <CALGmxEJBYFEoPOFo7aVSN=Xy=sgSMnrF0k8pnt=VhJ6cvhS_4Q@mail.gmail.com>
 <20180704013202.GK14437@ando.pearwood.info>
 <CAEbHw4YafoZc_1cnB6a-psmWqW-SD2tw3E2Fw5hUHSJxqEsu-Q@mail.gmail.com>
 <CAEbHw4Yjue-V_CKQJe2k6f4E8ucNuG=ZNFt5O_njfrQoNKHeUw@mail.gmail.com>
 <CALGmxE+m+1YXk1vRsdkAK6yQ12V=syi6WKZsSTMFmw1BoiKUYA@mail.gmail.com>
 <CAB_e7izyUbkOXxpwivr0iyE4gXk58xn+SBfAZq5hcKAyDO-H_g@mail.gmail.com>
 <CALGmxE+OS-EDa7gNceEirnW1S0oryyinA7mMbUetXfZkJ67Zkw@mail.gmail.com>
 <CAB_e7iw=mCRtBGu2YJ_kV7yffSWh_Rp927Lpw_gwiyzbJtoqzA@mail.gmail.com>
 <CALGmxE+tbVu52D5xDoCKdWOWjg_Q-hzGyX7fvLn74_Xb_mfQ_A@mail.gmail.com>
Message-ID: <CAOmnVUCN7BjzcJshDurPSWJe32OrPYjpV5SiVT_jWSCdKRcUDw@mail.gmail.com>

I noticed recently that *all* examples for collection.defaultdict (
https://docs.python.org/3.7/library/collections.html#collections.defaultdict)
are cases of grouping (for an int, a list and a set) from an iterator with
a key, value output.

I wondered how common those constructions were, and what are defaultdict
used for else. So I took a little dive into a few libs to see it (std lib,
pypy, pandas, tensorflow, ..), and I saw essentially :
A) basic cases of "grouping" with a simple for loop and a
default_dict[key].append(value). I saw many kind of default factory
utilized, with list, int, set, dict, and even defaultdict(list). ex :
https://frama.link/UtNqvpvb, https://frama.link/o3Hb3-4U,
https://frama.link/dw92yJ1q, https://frama.link/1Gqoa7WM,
https://frama.link/bWswbHsU, https://frama.link/SZh2q8pS
B) cases of grouping, but where the for loop used was alimenting more than
one "grouper". pretty annoying if we want to group something. ex:
https://frama.link/Db-Ny49a, https://frama.link/bZakUR33,
https://frama.link/MwJFqh5o,
C) classes attributes initialization (grouping is done by repeatably
calling a function, so any grouping constructor will be useless here). ex :
https://frama.link/GoGWuQwR, https://frama.link/BugcS8wU
D) Sometimes you just want to defautdict inside a defauldict inside a dict
and just have fun : https://frama.link/asBNLr1g,
https://frama.link/8j7gzfA5

>From what I saw, the most useful would be to add method to a defaultdict to
fill it from an iterable, and using a grouping method adapted to the
default_factor (so __add__ for list, int and str, add for set, update for
dict and proably __add__ for anything else)

A sample code would be :

from collections import defaultdict
class groupingdict(defaultdict):
    def group_by_iterator(self, iterator):
        empty_element = self.default_factory()
        if hasattr(empty_element, "__add__"):
            for key, element in iterator:
                self[key] += element
        elif hasattr(empty_element, "update"):
            for key, element in iterator:
                self[key].update(element)
        elif hasattr(empty_element, "add"):
            for key, element in iterator:
                self[key].add(element)
        else:
            raise TypeError('default factory does not support iteration')
        return self

So that for example :
>groupingdict(dict).group_by_iterator(
    (grouping_key, a_dict) for grouping_key, a_dict in [
        (1, {'a': 'c'}),
        (1, {'b': 'f'}),
        (1, {'a': 'e'}),
        (2, {'a': 'e'})
    ]
)
returns

>groupingdict(dict, {1: {'a': 'e', 'b': 'f'}, 2: {'a': 'e'}})


My implementation is garbage and There should be 2 method, one returning
the object and one modifing it, but I think it gives more leeway than just
a function returning a dict


2018-07-13 7:11 GMT+02:00 Chris Barker via Python-ideas <
python-ideas at python.org>:

> On Mon, Jul 9, 2018 at 5:55 PM, Franklin? Lee <
> leewangzhong+python at gmail.com> wrote:
>
>> >> - The storage container.
>> >
>> >
>> > so this means you'r passing in a full set of storage containers? I'm a
>> vit
>> > confused by that -- if they might be pre-populated, then they would
>> need to
>> > be instance,s an you'd need to have one for every key -- how would you
>> know
>> > in advance aht you needed???
>>
>> No, I mean the mapping (outer) container. For example, I can pass in
>> an empty OrderedDict, or a dict that already contained some groups
>> from a previous call to the grouping function.
>>
>
> Sure -- that's what my prototype does if you pass a Mapping in (or use
> .update() )
>
> why not?
>
> -CHB
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>


-- 

--
*Nicolas Rolin* | Data Scientist
+ 33 631992617 - nicolas.rolin at tiime.fr <prenom.nom at tiime.fr>


*15 rue Auber, **75009 Paris*
*www.tiime.fr <http://www.tiime.fr>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180713/e2f90ca7/attachment.html>

From steve at pearwood.info  Fri Jul 13 11:05:15 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 14 Jul 2018 01:05:15 +1000
Subject: [Python-ideas] Add the imath module
In-Reply-To: <CAKnNRPm_eVGX=Dx4irF8a7ovPnDuSJT-0p82NkOgACTekEHW7w@mail.gmail.com>
References: <20180713014842.GF7318@ando.pearwood.info>
 <54da9da43f254d2ab23ed4d91e78984d@xmail103.UGent.be>
 <5B48611F.9040406@UGent.be>
 <f9c10e21dbd940838a3e3b2b7be78e31@xmail103.UGent.be>
 <5B486D51.3050305@UGent.be>
 <8f7a10f5c4234a70a4d217f3731d8b80@xmail103.UGent.be>
 <5B488ED7.8060209@UGent.be>
 <d51b09179e2043869ea0b73e6b54ac9a@xmail103.UGent.be>
 <5B489EC7.5080604@UGent.be>
 <CAKnNRPm_eVGX=Dx4irF8a7ovPnDuSJT-0p82NkOgACTekEHW7w@mail.gmail.com>
Message-ID: <20180713150515.GJ7318@ando.pearwood.info>

On Fri, Jul 13, 2018 at 02:51:18PM +0200, Jacco van Dorp wrote:

> I think the extra 9 characters of "_probable" are worth the extra accuracy.
>
> Dont teach people to lie in their function names. It's a bad example.
> and if you really want to lie, just

In what way is 

    is_probable_prime(11)  # returns True

less true than

    isprime(11)  # returns True

If you want to *lie*, then claiming that 11 is a probable prime is 
certainly a lie.


-- 
Steve

From rosuav at gmail.com  Fri Jul 13 11:31:05 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 14 Jul 2018 01:31:05 +1000
Subject: [Python-ideas] Add the imath module
In-Reply-To: <20180713150515.GJ7318@ando.pearwood.info>
References: <20180713014842.GF7318@ando.pearwood.info>
 <54da9da43f254d2ab23ed4d91e78984d@xmail103.UGent.be>
 <5B48611F.9040406@UGent.be>
 <f9c10e21dbd940838a3e3b2b7be78e31@xmail103.UGent.be>
 <5B486D51.3050305@UGent.be>
 <8f7a10f5c4234a70a4d217f3731d8b80@xmail103.UGent.be>
 <5B488ED7.8060209@UGent.be>
 <d51b09179e2043869ea0b73e6b54ac9a@xmail103.UGent.be>
 <5B489EC7.5080604@UGent.be>
 <CAKnNRPm_eVGX=Dx4irF8a7ovPnDuSJT-0p82NkOgACTekEHW7w@mail.gmail.com>
 <20180713150515.GJ7318@ando.pearwood.info>
Message-ID: <CAPTjJmpbXwSg0s0ucZFs0KdiRi=k9hdGTubgNeUQGkoMiXmUxQ@mail.gmail.com>

On Sat, Jul 14, 2018 at 1:05 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> On Fri, Jul 13, 2018 at 02:51:18PM +0200, Jacco van Dorp wrote:
>
>> I think the extra 9 characters of "_probable" are worth the extra accuracy.
>>
>> Dont teach people to lie in their function names. It's a bad example.
>> and if you really want to lie, just
>
> In what way is
>
>     is_probable_prime(11)  # returns True
>
> less true than
>
>     isprime(11)  # returns True
>
> If you want to *lie*, then claiming that 11 is a probable prime is
> certainly a lie.
>

It's a probable lie?

ChrisA

From tim.peters at gmail.com  Fri Jul 13 11:52:38 2018
From: tim.peters at gmail.com (Tim Peters)
Date: Fri, 13 Jul 2018 10:52:38 -0500
Subject: [Python-ideas] Add the imath module
In-Reply-To: <5B485E58.1000503@UGent.be>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <51e71cc9801047e9b21c4dc01c64826e@xmail103.UGent.be>
 <5B485E58.1000503@UGent.be>
Message-ID: <CAExdVN=mmFfx9UsMB=KjMS-QpSM84Rn9V4SyytgMPRDMUEy43w@mail.gmail.com>

>
> [Steven D'Aprano]
 > about 4.7 seconds to test 2**800 + 1;

[Jeroen Demeyer]

> In SageMath:
>
> sage: n = 2**800+1; timeit('is_prime(n)')
> 625 loops, best of 3: 303 ?s per loop
>
> That's 4 orders of magnitude faster...
>

More like 6, yes?  My Python version is more like 3, but is way fast enough
for most things I do:

$ py -m timeit -s "from temp3 import pp; n=2**800+1" "pp(n)"
200 loops, best of 5: 1.71 msec per loop

$ py -m timeit -s "from temp3 import pp; n=2**900+1" "pp(n)"
100 loops, best of 5: 2.32 msec per loop

$ py -m timeit -s "from temp3 import pp; n=2**1000+1" "pp(n)"
100 loops, best of 5: 3.15 msec per loop

Steven's numbers are pretty baffling to me, since these are all composite
and so iterating Miller-Rabin "should get out" pretty fast:

about 4.7 seconds to test 2**800 + 1;
> less than a tenth of a millisecond to test 2**900 + 1;
> and about 8.6 seconds to test 2**1000 + 1.


Here's the Python code I used:

def pp(n, k=10):
    """
    Return True iff n is a probable prime, using k Miller-Rabin tests.

    When False, n is definitely composite.
    """
    from random import randrange

    if n < 4:
        return 2 <= n <= 3
    d = nm1 = n-1
    s = 0
    while d & 1 == 0:
        s += 1
        d >>= 1
    if s == 0:
        return False # n >= 4 and even
    ss = range(1, s)
    for _ in range(k):
        a = randrange(2, nm1)
        x = pow(a, d, n)
        if x == 1 or x == nm1:
            continue
        for _ in ss:
            x = x*x % n
            if x == nm1:
                break
            if x == 1:
                return False
        else:
            return False
    return True
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180713/76bdc1e1/attachment-0001.html>

From danilo.bellini at gmail.com  Fri Jul 13 12:15:41 2018
From: danilo.bellini at gmail.com (Danilo J. S. Bellini)
Date: Fri, 13 Jul 2018 13:15:41 -0300
Subject: [Python-ideas] Add the imath module
In-Reply-To: <CAExdVN=mmFfx9UsMB=KjMS-QpSM84Rn9V4SyytgMPRDMUEy43w@mail.gmail.com>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <51e71cc9801047e9b21c4dc01c64826e@xmail103.UGent.be>
 <5B485E58.1000503@UGent.be>
 <CAExdVN=mmFfx9UsMB=KjMS-QpSM84Rn9V4SyytgMPRDMUEy43w@mail.gmail.com>
Message-ID: <CA+g5Jwj0rXVXc7W+EBpujT0muunDUCbAOmQx91KmRiZF_oBM1Q@mail.gmail.com>

If we have a fast and exact primality test up to 2**64,
a signature like this would help:

def is_prime(n, probabilistic=False):

When probabilistic is False and the number is beyond 2**64,
it'll return OverflowError.
When probabilistic is True, it's unbounded, but probabilistic.
The docs should be explicit about that, though.

Instead of an OverflowError,
a number beyond 2**64 might simply use a slower strategy.
But to make it an explicit option, the signature could be:

def is_prime(n, exact=True, unbounded=False):

By tellling it should be both exact and unbounded,
the user is aware that it might be slow.

The alternatives are:

   - exact, unbounded: Slower test
   - not exact, unbounded: Probabilistic test
   - exact, bounded: Default, raises OverflowError beyond 2**64
   - not exact, bounded: Invalid, but it can just ignore exact input

-- 
Danilo J. S. Bellini
---------------
"*It is not our business to set up prohibitions, but to arrive at
conventions.*" (R. Carnap)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180713/030d1c41/attachment.html>

From tim.peters at gmail.com  Fri Jul 13 12:16:24 2018
From: tim.peters at gmail.com (Tim Peters)
Date: Fri, 13 Jul 2018 11:16:24 -0500
Subject: [Python-ideas] Add the imath module
In-Reply-To: <CAExdVN=mmFfx9UsMB=KjMS-QpSM84Rn9V4SyytgMPRDMUEy43w@mail.gmail.com>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <51e71cc9801047e9b21c4dc01c64826e@xmail103.UGent.be>
 <5B485E58.1000503@UGent.be>
 <CAExdVN=mmFfx9UsMB=KjMS-QpSM84Rn9V4SyytgMPRDMUEy43w@mail.gmail.com>
Message-ID: <CAExdVN=qHa1-kbekqMF2trJvnb+i3LZqzkJYQ6yLdYCN2h1iSg@mail.gmail.com>

>
> [Steven D'Aprano]
>  > about 4.7 seconds to test 2**800 + 1;
>
> [Jeroen Demeyer]
>
>> In SageMath:
>>
>> sage: n = 2**800+1; timeit('is_prime(n)')
>> 625 loops, best of 3: 303 ?s per loop
>>
>> That's 4 orders of magnitude faster...
>
>
[Tim]

> More like 6, yes?
>

Heh - sorry about that.  A speck of dirt on my monitor made me read '303"
as "3.03".


>   My Python version is more like 3, but is way fast enough for most things
> I do:
>

So my Python Miller-Rabin code (already shared) was about one order of
magnitude slower.
 ...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180713/1a3d0ef8/attachment.html>

From waksman at gmail.com  Fri Jul 13 13:06:42 2018
From: waksman at gmail.com (George Leslie-Waksman)
Date: Fri, 13 Jul 2018 10:06:42 -0700
Subject: [Python-ideas] Add the imath module
In-Reply-To: <CAExdVN=qHa1-kbekqMF2trJvnb+i3LZqzkJYQ6yLdYCN2h1iSg@mail.gmail.com>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <51e71cc9801047e9b21c4dc01c64826e@xmail103.UGent.be>
 <5B485E58.1000503@UGent.be>
 <CAExdVN=mmFfx9UsMB=KjMS-QpSM84Rn9V4SyytgMPRDMUEy43w@mail.gmail.com>
 <CAExdVN=qHa1-kbekqMF2trJvnb+i3LZqzkJYQ6yLdYCN2h1iSg@mail.gmail.com>
Message-ID: <CACef2akAjXtsrRjRAG68LOTJpvzN+rzDaPjuTFZVyCCzqzaU8Q@mail.gmail.com>

Instead of `imath`, how about `math.integer` for the module name?

On Fri, Jul 13, 2018 at 9:20 AM Tim Peters <tim.peters at gmail.com> wrote:

> [Steven D'Aprano]
>>  > about 4.7 seconds to test 2**800 + 1;
>>
>> [Jeroen Demeyer]
>>
>>> In SageMath:
>>>
>>> sage: n = 2**800+1; timeit('is_prime(n)')
>>> 625 loops, best of 3: 303 ?s per loop
>>>
>>> That's 4 orders of magnitude faster...
>>
>>
> [Tim]
>
>> More like 6, yes?
>>
>
> Heh - sorry about that.  A speck of dirt on my monitor made me read '303"
> as "3.03".
>
>
>>   My Python version is more like 3, but is way fast enough for most
>> things I do:
>>
>
> So my Python Miller-Rabin code (already shared) was about one order of
> magnitude slower.
>  ...
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180713/98c502b6/attachment.html>

From mike at selik.org  Fri Jul 13 13:38:44 2018
From: mike at selik.org (Michael Selik)
Date: Fri, 13 Jul 2018 10:38:44 -0700
Subject: [Python-ideas] Fwd: grouping / dict of lists
In-Reply-To: <CAOmnVUCN7BjzcJshDurPSWJe32OrPYjpV5SiVT_jWSCdKRcUDw@mail.gmail.com>
References: <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAOmnVUBp9vJjTp6v17LHy2C=7GzR63fb4ARZW2Dco022uTDpfA@mail.gmail.com>
 <20180703153338.GG14437@ando.pearwood.info>
 <CALGmxEJBYFEoPOFo7aVSN=Xy=sgSMnrF0k8pnt=VhJ6cvhS_4Q@mail.gmail.com>
 <20180704013202.GK14437@ando.pearwood.info>
 <CAEbHw4YafoZc_1cnB6a-psmWqW-SD2tw3E2Fw5hUHSJxqEsu-Q@mail.gmail.com>
 <CAEbHw4Yjue-V_CKQJe2k6f4E8ucNuG=ZNFt5O_njfrQoNKHeUw@mail.gmail.com>
 <CALGmxE+m+1YXk1vRsdkAK6yQ12V=syi6WKZsSTMFmw1BoiKUYA@mail.gmail.com>
 <CAB_e7izyUbkOXxpwivr0iyE4gXk58xn+SBfAZq5hcKAyDO-H_g@mail.gmail.com>
 <CALGmxE+OS-EDa7gNceEirnW1S0oryyinA7mMbUetXfZkJ67Zkw@mail.gmail.com>
 <CAB_e7iw=mCRtBGu2YJ_kV7yffSWh_Rp927Lpw_gwiyzbJtoqzA@mail.gmail.com>
 <CALGmxE+tbVu52D5xDoCKdWOWjg_Q-hzGyX7fvLn74_Xb_mfQ_A@mail.gmail.com>
 <CAOmnVUCN7BjzcJshDurPSWJe32OrPYjpV5SiVT_jWSCdKRcUDw@mail.gmail.com>
Message-ID: <CADqi795JMfzj4kCOmov-dhnYXTBCRJVLN0VNzG2c6QTAAvEr_A@mail.gmail.com>

Thanks for linking to these. I looked at many of them in my own research,
but for some reason didn't think to write down the links. I'll respond to
each one separately.

Throughout, I'm going to use my proposed ``grouped`` builtin to demonstrate
possible revisions. Note that I am *not* suggesting a replacement to
defaultdict. The proposal is to make a common task easier and more
reliable. It does not satisfy all uses of defaultdict.

https://github.com/selik/peps/blob/master/pep-9999.rst


On Fri, Jul 13, 2018 at 6:17 AM Nicolas Rolin <nicolas.rolin at tiime.fr>
wrote:

> I noticed recently that *all* examples for collection.defaultdict (
> https://docs.python.org/3.7/library/collections.html#collections.defaultdict)
> are cases of grouping (for an int, a list and a set) from an iterator with
> a key, value output.
>
> I wondered how common those constructions were, and what are defaultdict
> used for else. So I took a little dive into a few libs to see it (std lib,
> pypy, pandas, tensorflow, ..), and I saw essentially :
>


> A) basic cases of "grouping" with a simple for loop and a
> default_dict[key].append(value). I saw many kind of default factory
> utilized, with list, int, set, dict, and even defaultdict(list). ex :
> https://frama.link/UtNqvpvb,
>

    facesizes = collections.defaultdict(int)
    for face in cubofaces:
        facesizes[len(face)] += 1

This is more specifically a Counter. It might look better as:

    facesizes = collections.Counter(len(face) for face in cubofaces)



https://frama.link/o3Hb3-4U,
>

    accum = defaultdict(list)
    garbageitems = []

    for item in root:
        filename = findfile(opts.fileroot, item.attrib['classname'])
        accum[filename].append(float(item.attrib['time']))
        if filename is None:
            garbageitems.append(item)


This might be more clear if separated into two parts.

    def keyfunc(item):
        return findfile(opts.fileroot, item.attrib['classname'])
    groups = grouped(root, keyfunc)
    groups = {k: [float(v.attrib['time']) for v in g] for k, g in
groups.items()}
    garbage = groups.pop(None, [])


Interestingly, this code later makes a distinction between ``garbage`` and
``garbageitems`` even though they appear to be the same. The programmer
might not have made that error (?) if using the ``grouped`` function,
because the act of grouping becomes single-purpose.




> https://frama.link/dw92yJ1q
>

In this case, the grouping is being performed by Pandas, not by the
defaultdict.

    expected = defaultdict(dict)
    for n1, gp1 in data.groupby('A'):
        for n2, gp2 in gp1.groupby('B'):
            expected[n1][n2] = op(gp2.loc[:, ['C', 'D']])


However, this is still relevant to our discussion, because the defaultdict
had to be converted to a regular dict afterward. Presumably to ensure that
missing keys cause KeyErrors.

    expected = dict((k, DataFrame(v))
                    for k, v in compat.iteritems(expected))


If you felt like one-lining this, it might look like

    expected = {n1: DataFrame({n2: op(gp2.loc[:, ['C', 'D']]) for n2, gp2
in gp1}) for n1, gp1 in data.groupby('A')}

I'm not saying that's more clear, but the fact that's possible emphasizes
that the grouping is performed by Pandas. Grouping cannot be one-lined if
restricted to Python standard library.




> https://frama.link/1Gqoa7WM
>

    results = defaultdict(dict)
    for i, k1 in enumerate(arg1.columns):
        for j, k2 in enumerate(arg2.columns):
            if j < i and arg2 is arg1:
                # Symmetric case
                results[i][j] = results[j][i]
            else:
                results[i][j] = f(*_prep_binary(arg1.iloc[:, i],
                                                arg2.iloc[:, j]))


The nested loop looks like a cross-product. It might be clearer using
itertools, but I wouldn't use ``grouped`` here.

    cross = itertools.product(enumerate(arg1.columns),
enumerate(arg2.columns))




> https://frama.link/bWswbHsU,
>

    self.mapping = collections.defaultdict(set)
    for op in (op for op in graph.get_operations()):
      if op.name.startswith(common.SKIPPED_PREFIXES):
        continue
      for op_input in op.inputs:
        self.mapping[op_input].add(op)


This is a case of a single element being added to multiple groups, which is
your section B, below. The loop and filter could be better. It looks like
someone intended to convert if/continue to a comprehension, but stopped
partway through the revision.

    ops = (op for op in graph.get_operations() if not
op.name.startswith(common.SKIPPED_PREFIXES))


I'm not sure how I like it, but here's a revision to use ``grouped``:

    inputs = ((op_input, op) for op in ops for op_input in op.inputs)
    groups = grouped(inputs, key=itemgetter(0))
    groups = {k: set(v for _, v in g) for k, g in groups.items()}


https://frama.link/SZh2q8pS
>

    handler_results = collections.defaultdict(list)
    for handler in per_handler_ops.keys():
        for op in per_handler_ops[handler]:
            handler_results[handler].append(op_results[op])
    return handler_results


Here it looks like the results are already grouped by handler and this loop
is constructing a parallel dict of lists for the results instead of the ops
themselves.

    return {handler: [op_results[op] for op in group] for handler, group in
per_handler_ops.items()}


B) cases of grouping, but where the for loop used was alimenting more than
> one "grouper". pretty annoying if we want to group something. ex:
> https://frama.link/Db-Ny49a,
>

    def _get_headnode_dict(fixer_list):
        """ Accepts a list of fixers and returns a dictionary
            of head node type --> fixer list.  """
        head_nodes = collections.defaultdict(list)
        every = []
        for fixer in fixer_list:
            if fixer.pattern:
                try:
                    heads = _get_head_types(fixer.pattern)
                except _EveryNode:
                    every.append(fixer)
                else:
                    for node_type in heads:
                        head_nodes[node_type].append(fixer)
            else:
                if fixer._accept_type is not None:
                    head_nodes[fixer._accept_type].append(fixer)
                else:
                    every.append(fixer)
        for node_type in chain(pygram.python_grammar.symbol2number.values(),
                               pygram.python_grammar.tokens):
            head_nodes[node_type].extend(every)
        return dict(head_nodes)


Again this ends with a conversion from defaultdict to basic dict: ``return
dict(head_nodes)``. This is somewhat common in usage of defaultdict,
because after the creation of groups, the later usage wants missing keys to
raise KeyError, not insert a new, empty group.

The association of a single element to multiple groups makes this a little
awkward again.

    def _get_headnode_dict(fixer_list):
        """ Accepts a list of fixers and returns a dictionary
            of head node type --> fixer list.  """

        nodetypes = set(chain(pygram.python_grammar.symbol2number.values(),
                              pygram.python_grammar.tokens))

        heads = {}
        for fixer in fixer_list:
            if fixer.pattern:
                try:
                    heads[fixer] = _get_head_types(fixer.pattern)
                except _EveryNode:
                    heads[fixer] = nodetypes
            elif fixer._accept_type is not None:
                heads[fixer] = [fixer._accept_type]
            else:
                heads[fixer] = nodetypes

        pairs = ((head, fixer) for fixer, types in heads.items() for head
in types)
        groups = grouped(pairs, key=itemgetter(0))
        return {head: [fixer for _, fixer in g] for head, g in
groups.items()}



https://frama.link/bZakUR33,
>

    directories = defaultdict(set)
    cache_directories = defaultdict(set)
    groups = defaultdict(list)
    for source, target, group, disk_id, condition in files:
        target = PureWindowsPath(target)
        groups[group].append((source, target, disk_id, condition))

        if target.suffix.lower() in {".py", ".pyw"}:
            cache_directories[group].add(target.parent)

        for dirname in target.parents:
            parent = make_id(dirname.parent)
            if parent and parent != '.':
                directories[parent].add(dirname.name)


This one has 3 different groupings created simultaneously. We could
separate the 3 tasks, but they all rely on a converted ``target`` path, so
it'd really end up being 4 loops. At that point, perhaps it's best to do
just 1 loop as written.



> https://frama.link/MwJFqh5o
>

    paramdicts = {}
    testers = collections.defaultdict(list)
    for name, attr in cls.__dict__.items():
        if name.endswith('_params'):
            if not hasattr(attr, 'keys'):
                d = {}
                for x in attr:
                    if not hasattr(x, '__iter__'):
                        x = (x,)
                    n = '_'.join(str(v) for v in x).replace(' ', '_')
                    d[n] = x
                attr = d
            paramdicts[name[:-7] + '_as_'] = attr
        if '_as_' in name:
            testers[name.split('_as_')[0] + '_as_'].append(name)


This one isn't a multiple groups issue. We can separate the tasks of
paramdicts and testers.

    as_names = (name for name in cls.__dict__ if '_as_' in name)
    testers = grouped(as_names, key=lambda name: name.split('_as_')[0] +
'_as_')

The paramdicts isn't a dict of lists, so it's not really appropriate for
``grouped``.




>
> C) classes attributes initialization (grouping is done by repeatably
> calling a function, so any grouping constructor will be useless here). ex :
> https://frama.link/GoGWuQwR,
>

Instead of ``defaultdict(lambda: 0)`` it seems better to use ``Counter()``.
It's called "invocation_counts" after all.



> https://frama.link/BugcS8wU
>

This seems like one of the ideal uses for defaultdict. It even has bits of
code made to handle the possibly accidental insert of new keys by popping
them off afterwards.



D) Sometimes you just want to defautdict inside a defauldict inside a dict
> and just have fun : https://frama.link/asBNLr1g,
>

I only have a tenuous grasp of git, so the fact that this file isn't in the
master branch is stretching my mind. But again, this doesn't feel like a
grouping task to me. It probably could be rewritten as such, but not
trivially.



>  https://frama.link/8j7gzfA5
>

It's hard to tell at a glance if this should truly be a dict of dicts of
sets or a dict of sets where the keys are 2-tuples. I'll assume the former
is correct.

On a side note, it looks like they're doing some silly things to fit into
the 80 characters per line restriction, like looping over keys and looking
up values inside the loop instead of using ``.items()``. I understand that
the character limit is strict, but...

https://github.com/tensorflow/tensorflow/blob/b202db076ecdc8a0fc80a49252d4a8dd0c8015b3/tensorflow/python/debug/lib/debug_data.py#L1371




> From what I saw, the most useful would be to add method to a defaultdict
> to fill it from an iterable, and using a grouping method adapted to the
> default_factor (so __add__ for list, int and str, add for set, update for
> dict and proably __add__ for anything else)
>
> A sample code would be :
>
> from collections import defaultdict
> class groupingdict(defaultdict):
>     def group_by_iterator(self, iterator):
>         empty_element = self.default_factory()
>         if hasattr(empty_element, "__add__"):
>             for key, element in iterator:
>                 self[key] += element
>         elif hasattr(empty_element, "update"):
>             for key, element in iterator:
>                 self[key].update(element)
>         elif hasattr(empty_element, "add"):
>             for key, element in iterator:
>                 self[key].add(element)
>         else:
>             raise TypeError('default factory does not support iteration')
>         return self
>
> So that for example :
> >groupingdict(dict).group_by_iterator(
>     (grouping_key, a_dict) for grouping_key, a_dict in [
>         (1, {'a': 'c'}),
>         (1, {'b': 'f'}),
>         (1, {'a': 'e'}),
>         (2, {'a': 'e'})
>     ]
> )
> returns
>
> >groupingdict(dict, {1: {'a': 'e', 'b': 'f'}, 2: {'a': 'e'}})
>
>
> My implementation is garbage and There should be 2 method, one returning
> the object and one modifing it, but I think it gives more leeway than just
> a function returning a dict
>


Let's take a look at these two examples:

    grouped(contacts, key=itemgetter('city')
    grouped(os.listdir('.'), key=lambda filepath:
os.path.splitext(filepath)[1])

How would those be rewritten using your proposal?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180713/2787b536/attachment-0001.html>

From mike at selik.org  Fri Jul 13 13:45:50 2018
From: mike at selik.org (Michael Selik)
Date: Fri, 13 Jul 2018 10:45:50 -0700
Subject: [Python-ideas] grouping / dict of lists
In-Reply-To: <CALGmxELLvdpOniH8yohfZygx1Fb7nfT=ESeHrJF27++5dM9ohA@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CAP7+vJJ6vN9H8esj8tS+bSKsFELh8q8ywYHAW8Ti_upEJHf1vg@mail.gmail.com>
 <CADqi797HXzHcUnrNLp7r+pFEM5fcApiKHV4s=P5-D6mXHYYv-g@mail.gmail.com>
 <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CALGmxELLvdpOniH8yohfZygx1Fb7nfT=ESeHrJF27++5dM9ohA@mail.gmail.com>
Message-ID: <CADqi797gi6zgY+pTPZRjKUj2wCv=gbOVdGOjf71Qw4YLc5+kXw@mail.gmail.com>

On Mon, Jul 2, 2018 at 8:49 AM Chris Barker <chris.barker at noaa.gov> wrote:

> On Fri, Jun 29, 2018 at 11:25 PM, Guido van Rossum <guido at python.org>
> wrote:
>
>
>> Hm, this actually feels heavier to me. But then again I never liked or
>> understood the need for Counter --
>>
>
> actually, me neither -- and partly because it's too lightweight -- that
> is, it's still a regular dict, and you pretty much have to know that to use
> it. That it, it provides a nice counting constructor, but after that, it's
> just a key:integer dict :-)
>

Counter provides ``most_common`` which is often implemented inefficiently
if written from scratch. People mistakenly use ``sorted`` instead of
``heapq.nlargest``.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180713/26991190/attachment.html>

From chris.barker at noaa.gov  Fri Jul 13 15:16:08 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Fri, 13 Jul 2018 14:16:08 -0500
Subject: [Python-ideas] Fwd: grouping / dict of lists
In-Reply-To: <CADqi795JMfzj4kCOmov-dhnYXTBCRJVLN0VNzG2c6QTAAvEr_A@mail.gmail.com>
References: <CADqi797mzco4Qg-2Pq=5AH6JsSO=ui9YDDFJ-HMapZCWvT69FA@mail.gmail.com>
 <CAOmnVUC_0fM1EPESNZR_sajGVq+PW0MXRyNtZ975_1Fx1wgeOQ@mail.gmail.com>
 <CADqi795d+eWwmLnzTbT=fnVgAUOgR_AhmgeuR0s_MqJmJYkhFQ@mail.gmail.com>
 <CAOmnVUDQDbj9eiBc73atcW=D8jF6difDAg5JqmQfy3M6i1dp1g@mail.gmail.com>
 <CADqi796-OAWNgEzig1dj2TsdkBMM98uKwgXeNreqePJg2ZAt2w@mail.gmail.com>
 <CALGmxEJuSNg=F1hyuzQtR6bvE9W8QPO+3hvu9Q9xXQhN1jPMTg@mail.gmail.com>
 <CAEbHw4Y3OAfTFW5FhXJp0J6OuO64BOd7+6vFNx8MSyHVG0toSg@mail.gmail.com>
 <CAOmnVUBp9vJjTp6v17LHy2C=7GzR63fb4ARZW2Dco022uTDpfA@mail.gmail.com>
 <20180703153338.GG14437@ando.pearwood.info>
 <CALGmxEJBYFEoPOFo7aVSN=Xy=sgSMnrF0k8pnt=VhJ6cvhS_4Q@mail.gmail.com>
 <20180704013202.GK14437@ando.pearwood.info>
 <CAEbHw4YafoZc_1cnB6a-psmWqW-SD2tw3E2Fw5hUHSJxqEsu-Q@mail.gmail.com>
 <CAEbHw4Yjue-V_CKQJe2k6f4E8ucNuG=ZNFt5O_njfrQoNKHeUw@mail.gmail.com>
 <CALGmxE+m+1YXk1vRsdkAK6yQ12V=syi6WKZsSTMFmw1BoiKUYA@mail.gmail.com>
 <CAB_e7izyUbkOXxpwivr0iyE4gXk58xn+SBfAZq5hcKAyDO-H_g@mail.gmail.com>
 <CALGmxE+OS-EDa7gNceEirnW1S0oryyinA7mMbUetXfZkJ67Zkw@mail.gmail.com>
 <CAB_e7iw=mCRtBGu2YJ_kV7yffSWh_Rp927Lpw_gwiyzbJtoqzA@mail.gmail.com>
 <CALGmxE+tbVu52D5xDoCKdWOWjg_Q-hzGyX7fvLn74_Xb_mfQ_A@mail.gmail.com>
 <CAOmnVUCN7BjzcJshDurPSWJe32OrPYjpV5SiVT_jWSCdKRcUDw@mail.gmail.com>
 <CADqi795JMfzj4kCOmov-dhnYXTBCRJVLN0VNzG2c6QTAAvEr_A@mail.gmail.com>
Message-ID: <CALGmxEK53ZgWVzFCkoOHTYLR1gU7015rFPE=wFmOBixoNf9-sA@mail.gmail.com>

On Fri, Jul 13, 2018 at 12:38 PM, Michael Selik <mike at selik.org> wrote:

> Thanks for linking to these.
>

yup -- real use cases are really helpful.

Though the other paradigm for grouping is use of setdefault() rather than
defaultdict. So it would be nice to look for those, too.


> I looked at many of them in my own research, but for some reason didn't
> think to write down the links. I'll respond to each one separately.
>


> Throughout, I'm going to use my proposed ``grouped`` builtin to
> demonstrate possible revisions. Note that I am *not* suggesting a
> replacement to defaultdict. The proposal is to make a common task easier
> and more reliable. It does not satisfy all uses of defaultdict.
>

agreed -- and it shouldn't.

I"d like to see how some of these pan our with my proposed API:

either a Grouped class, or at least (key, value) iterables and/or a value
function.

I don't have time now to do them all, but for the moment:

I noticed recently that *all* examples for collection.defaultdict (
>> https://docs.python.org/3.7/library/collections.html#
>> collections.defaultdict) are cases of grouping (for an int, a list and a
>> set) from an iterator with a key, value output.
>>
>
and yet others on this thread think a (key, value) input would be rare -- I
guess it depends on whether you are thinking dict-like already....


>
>> https://frama.link/o3Hb3-4U,
>>
>
>     accum = defaultdict(list)
>     garbageitems = []
>
>     for item in root:
>         filename = findfile(opts.fileroot, item.attrib['classname'])
>         accum[filename].append(float(item.attrib['time']))
>         if filename is None:
>             garbageitems.append(item)
>
>
> This might be more clear if separated into two parts.
>
>     def keyfunc(item):
>         return findfile(opts.fileroot, item.attrib['classname'])
>     groups = grouped(root, keyfunc)
>     groups = {k: [float(v.attrib['time']) for v in g] for k, g in
> groups.items()}
>     garbage = groups.pop(None, [])
>

so this one is a prime case for a value function -- I think post-processing
the groups is a pretty common case -- why make people post-process it?

    def keyfunc(item):
        return findfile(opts.fileroot, item.attrib['classname'])
   def valuefunc(item):
        float(item.attrib['time'])
    groups = grouped(root, keyfunc, valuefunc)
    garbage = groups.pop(None, [])

And the post-processing is then mixing comprehension style with key
function style (what to call that -- "functional" style?), so why not use a
(key, value) iterable:

groups = grouped((findfile(opts.fileroot, item.attrib['classname']),
                  item.attrib['time'])
                  for item in root))

OK -- that's packing a bit too much into a line, so how about:

def keyfunc(item):
        return findfile(opts.fileroot, item.attrib['classname'])

groups = grouped( (keyfunc(item), item.attrib['time']) for item in root)

>
>     self.mapping = collections.defaultdict(set)
>     for op in (op for op in graph.get_operations()):
>       if op.name.startswith(common.SKIPPED_PREFIXES):
>         continue
>       for op_input in op.inputs:
>         self.mapping[op_input].add(op)
>
>
> This is a case of a single element being added to multiple groups, which
> is your section B, below. The loop and filter could be better. It looks
> like someone intended to convert if/continue to a comprehension, but
> stopped partway through the revision.
>

yeah, this is weird --

But it does make a case for having a class with the option f using a set to
collect (which I have in an older commit of my prototype:

    inputs = ((op_input, op) for op in ops for op_input in op.inputs)
    groups = Grouping(inputs, key=itemgetter(0), collection=set)

otherwise, you could have a method to do it:
    groups.map_on_groups(set)

(not sure I like that method name, but I hope you get the idea)

OK, back to work.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180713/4f77b60a/attachment-0001.html>

From greg.ewing at canterbury.ac.nz  Fri Jul 13 18:48:26 2018
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 14 Jul 2018 10:48:26 +1200
Subject: [Python-ideas] Add the imath module
In-Reply-To: <CAOOa=pNT4uBLUWyBkxkZpPQCkaQzQCSYD-=y6-UmULp3H9Xjfw@mail.gmail.com>
References: <CAPTjJmqS+2LSiM0XWy-gKcgcx1JHOhz3CVvCw=f9j23UdSNFqg@mail.gmail.com>
 <20180713004047.GD7318@ando.pearwood.info>
 <CAPTjJmrWDGOf9HPd=CAZ8wCgn4JV073nejO5C+JNtSe0s6jZGw@mail.gmail.com>
 <20180713014842.GF7318@ando.pearwood.info>
 <54da9da43f254d2ab23ed4d91e78984d@xmail103.UGent.be>
 <5B48611F.9040406@UGent.be>
 <f9c10e21dbd940838a3e3b2b7be78e31@xmail103.UGent.be>
 <5B486D51.3050305@UGent.be>
 <8f7a10f5c4234a70a4d217f3731d8b80@xmail103.UGent.be>
 <5B488ED7.8060209@UGent.be>
 <d51b09179e2043869ea0b73e6b54ac9a@xmail103.UGent.be>
 <5B489EC7.5080604@UGent.be>
 <CAOOa=pNT4uBLUWyBkxkZpPQCkaQzQCSYD-=y6-UmULp3H9Xjfw@mail.gmail.com>
Message-ID: <5B492C3A.1040803@canterbury.ac.nz>

Stephan Houben wrote:
> Should we call sort then probable_sort, since the non-zero probability 
> exists of it going wrong due to a stray cosmic ray?

And in the interests of full disclosure, we should call Python
a probable language, running on a probable computer.

-- 
Probable Greg

From greg.ewing at canterbury.ac.nz  Fri Jul 13 19:09:35 2018
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 14 Jul 2018 11:09:35 +1200
Subject: [Python-ideas] Add the imath module
In-Reply-To: <CA+g5Jwj0rXVXc7W+EBpujT0muunDUCbAOmQx91KmRiZF_oBM1Q@mail.gmail.com>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <51e71cc9801047e9b21c4dc01c64826e@xmail103.UGent.be>
 <5B485E58.1000503@UGent.be>
 <CAExdVN=mmFfx9UsMB=KjMS-QpSM84Rn9V4SyytgMPRDMUEy43w@mail.gmail.com>
 <CA+g5Jwj0rXVXc7W+EBpujT0muunDUCbAOmQx91KmRiZF_oBM1Q@mail.gmail.com>
Message-ID: <5B49312F.8020302@canterbury.ac.nz>

Danilo J. S. Bellini wrote:
> If we have a fast and exact primality test up to 2**64,
> a signature like this would help:
> 
> def is_prime(n, probabilistic=False):

Two separate functions would be better, by the "no constant
boolean parameters" rule.

-- 
Greg

From ericsnowcurrently at gmail.com  Fri Jul 13 20:22:24 2018
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Fri, 13 Jul 2018 18:22:24 -0600
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
Message-ID: <CALFfu7A3vciEdV0FmjuZ9svumzbMXVCAHM-cuFKwP0WD-xw1SA@mail.gmail.com>

On Sun, Jul 8, 2018 at 12:30 PM David Foster <davidfstr at gmail.com> wrote:
> In the past I have personally viewed Python as difficult to use for
> parallel applications, which need to do multiple things simultaneously
> for increased performance:
>
> * The old Threads, Locks, & Shared State model is inefficient in Python
> due to the GIL, which limits CPU usage to only one thread at a time
> (ignoring certain functions implemented in C, such as I/O).
>
> * The Actor model can be used with some effort via the ?multiprocessing?
> module, but it doesn?t seem that streamlined and forces there to be a
> separate OS process per line of execution, which is relatively expensive.

Yep, Python's multi-core story is a bit rough (Jython/IronPython
aside).  It's especially hard for folks used to
concurrency/parallelism in other languages.  I'm hopeful that we can
improve the situation.

> I was thinking it would be nice if there was a better way to implement
> the Actor model, with multiple lines of execution in the same process,

FWIW, at this point I'm a big fan of this concurrency model.  I find
it hurts my brain least. :)

> yet avoiding contention from the GIL. This implies a separate GIL for
> each line of execution (to eliminate contention) and a controlled way to
> exchange data between different lines of execution.
>
> So I was thinking of proposing a design for implementing such a system.
> Or at least get interested parties thinking about such a system.
>
> With some additional research I notice that [PEP 554] (?Multiple
> subinterpeters in the stdlib?) appears to be putting forward a design
> similar to the one I described. I notice however it mentions that
> subinterpreters currently share the GIL, which would seem to make them
> unusable for parallel scenarios due to GIL contention.

I'm glad you found PEP 554.  I wanted to keep the PEP focused on
exposing the existing subinterpreter support (and the basic,
CSP-inspired concurrency model), which is why it doesn't go into much
detail about changes to the CPython runtime that will allow GIL-free
multi-core parallelism.   As Nick mentioned, my talk at the language
summit covers my plans.

Improving Python's multi-core story has been the major focus of my
(sadly relatively small) contributions to CPython for several years
now.  I've made slow progress due to limited time, but things are
picking up, especially since I got a job in December at Microsoft that
allows me to work on CPython for part of each week.  On top of that,
several other people are directly helping now (including Emily
Morehouse) and I got a lot of positive feedback for the project at
PyCon this year.

> I'd like to solicit some feedback on what might be the most efficient
> way to make forward progress on efficient parallelization in Python
> inside the same OS process. The most promising areas appear to be:
>
> 1. Make the current subinterpreter implementation in Python have more
> complete isolation, sharing almost no state between subinterpreters. In
> particular not sharing the GIL. The "Interpreter Isolation" section of
> PEP 554 enumerates areas that are currently shared, some of which
> probably shouldn't be.

Right, this is the approach I'm driving.  At this point I have the
project broken down pretty well into manageable chunks.  You're
welcome to join in. :)  Regardless,  I'd be glad to discuss it with
you in more depth if you're interested.

> 2. Give up on making things work inside the same OS process and rather
> focus on implementing better abstractions on top of the existing
> multiprocessing API so that the actor model is easier to program
> against. For example, providing some notion of Channels to communicate
> between lines of execution, a way to monitor the number of Messages
> waiting in each channel for throughput profiling and diagnostics,
> Supervision, etc. In particular I could do this by using an existing
> library like Pykka or Thespian and extending it where necessary.

It may worth a shot.  You should ask Davin Potts (CC'ed) about this.
We discussed this a little at PyCon.  I'm sure he'd welcome help in
improving the multiprocessing module.

-eric

From ericsnowcurrently at gmail.com  Fri Jul 13 20:32:36 2018
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Fri, 13 Jul 2018 18:32:36 -0600
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <c00bf3ef-ef42-9b43-247a-5adf9d4826d9@gmail.com>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <20180709161754.GA6167@trent.me>
 <c00bf3ef-ef42-9b43-247a-5adf9d4826d9@gmail.com>
Message-ID: <CALFfu7AXSw24RCU-p6B6roG-UFMxsYow3HKYcKFmysFsCYEWSA@mail.gmail.com>

On Tue, Jul 10, 2018 at 8:32 AM David Foster <davidfstr at gmail.com> wrote:
>
> I was not aware of PyParallel. The PyParellel "parallel thread"
> line-of-execution implementation is pretty interesting. Trent, big kudos
> to you on that effort.

+1  It's a neat project.  Trent's pretty smart. :)

> Since you're speaking in the past tense and said "but we're not doing it
> like that", I infer that the notion of a parallel thread was turned down
> for integration into CPython, as that appears to have been the original
> goal.
>
> However I am unable to locate a rationale for why that integration was
> turned down. Was it deemed to be too complex to execute, perhaps in the
> context of providing C extension compatibility? Was there a desire to
> see a similar implementation on Linux as well as Windows? Some other
> reason?

Trent can correct me if I'm wrong, but I believe it boiled down to
challenges with the POSIX implementation (that email thread implies
this as well), likely coupled with limited time for Trent to work on
it.

-eric

From steve at pearwood.info  Sat Jul 14 00:54:18 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 14 Jul 2018 14:54:18 +1000
Subject: [Python-ideas] Add the imath module
In-Reply-To: <CAExdVN=mmFfx9UsMB=KjMS-QpSM84Rn9V4SyytgMPRDMUEy43w@mail.gmail.com>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <51e71cc9801047e9b21c4dc01c64826e@xmail103.UGent.be>
 <5B485E58.1000503@UGent.be>
 <CAExdVN=mmFfx9UsMB=KjMS-QpSM84Rn9V4SyytgMPRDMUEy43w@mail.gmail.com>
Message-ID: <20180714045417.GM7318@ando.pearwood.info>

On Fri, Jul 13, 2018 at 10:52:38AM -0500, Tim Peters wrote:

> Steven's numbers are pretty baffling to me, since these are all composite
> and so iterating Miller-Rabin "should get out" pretty fast:

That's because you haven't seen my code :-)

It's over-engineered, class-based, and written as a learning exercise. 
There's a compatibility layer so it will work back to Python 2.4 and an 
instrumentation layer which I inserted in a fit of enthusiasm to gather 
staticistics, which I have never once looked at since.

And *even so* it is still fast enough for casual use at the interactive 
interpreter, compared to more naive algorithms with worse Big O 
performance characteristics.

You might think 5 seconds is slow, but I'm serious when I say some of 
the other algorithms I played with would take *days* to generate, 
or check, largish primes.


-- 
Steve

From gadgetsteve at live.co.uk  Sat Jul 14 02:40:18 2018
From: gadgetsteve at live.co.uk (Steve Barnes)
Date: Sat, 14 Jul 2018 06:40:18 +0000
Subject: [Python-ideas] Add the imath module
In-Reply-To: <20180714045417.GM7318@ando.pearwood.info>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <51e71cc9801047e9b21c4dc01c64826e@xmail103.UGent.be>
 <5B485E58.1000503@UGent.be>
 <CAExdVN=mmFfx9UsMB=KjMS-QpSM84Rn9V4SyytgMPRDMUEy43w@mail.gmail.com>
 <20180714045417.GM7318@ando.pearwood.info>
Message-ID: <VI1P194MB007989DEB270351D32D7B7C09B5F0@VI1P194MB0079.EURP194.PROD.OUTLOOK.COM>

Looking for some information on how long it has taken to generate large 
primes I stumbled across 
https://arxiv.org/ftp/arxiv/papers/1709/1709.09963.pdf which makes 
interesting reading. It concentrates on giving no false negatives (never 
saying n is not a prime when it is) but giving an increasing probability 
that n is a prime as testing goes on.

I did find an article from 2016 that mentioned M77232917 (a 23,249,425 
digit Mersenne prime) and M74207281 (22,338,618 digit Mersenne prime) 
with the latter taking a month to check with the Lucas-Lehmer test.


On 14/07/2018 05:54, Steven D'Aprano wrote:
> On Fri, Jul 13, 2018 at 10:52:38AM -0500, Tim Peters wrote:
> 
>> Steven's numbers are pretty baffling to me, since these are all composite
>> and so iterating Miller-Rabin "should get out" pretty fast:
> 
> That's because you haven't seen my code :-)
> 
> It's over-engineered, class-based, and written as a learning exercise.
> There's a compatibility layer so it will work back to Python 2.4 and an
> instrumentation layer which I inserted in a fit of enthusiasm to gather
> staticistics, which I have never once looked at since.
> 
> And *even so* it is still fast enough for casual use at the interactive
> interpreter, compared to more naive algorithms with worse Big O
> performance characteristics.
> 
> You might think 5 seconds is slow, but I'm serious when I say some of
> the other algorithms I played with would take *days* to generate,
> or check, largish primes.
> 
> 

-- 
Steve (Gadget) Barnes
Any opinions in this message are my personal opinions and do not reflect 
those of my employer.

---
This email has been checked for viruses by AVG.
https://www.avg.com


From kenlhilton at gmail.com  Sat Jul 14 04:24:32 2018
From: kenlhilton at gmail.com (Ken Hilton)
Date: Sat, 14 Jul 2018 10:24:32 +0200
Subject: [Python-ideas] Make import an expression
Message-ID: <CADPFgprCWEZSG=o5puODzs_GDdQak69eb4ntwY_vgmw=+p7FEg@mail.gmail.com>

Hi all,

Just a curious idea I had. The subject says it all. "import" is currently a
statement, which means it cannot be used inside anything else. Currently,
the only way to import a module inside an expression is to use the
__import__ function, which is both cumbersome and takes up more space
(which is disadvantageous in things like code golf).

I propose making "import" an expression, thus allowing the syntactic sugar
of "import module" -> "__import__('module')" to be graduated to the
expression level. Because of operator precedence, I suspect that in most
cases one would need to surround the expression with parentheses.
An additional effect that is not seen with using __import__ is that the
module's name is bound to the current scope the way it is with a normal
import statement.

Examples of where this could be used:

Importing on the fly to generate one value:

    secret = (import random).randint(1, 100)

Quick use of itertools in generator expressions:

    (i * i for i in (import itertools).accumulate(generate_numbers()))

Re-using a function from a module after importing it in an expression:

    b = a * (import math).tan(math.radians(45)) #math.radians is valid
because (import math) binds "math" to the current scope

What are your thoughts?

Sharing,
Ken Hilton;
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180714/310d8e33/attachment.html>

From jfine2358 at gmail.com  Sat Jul 14 05:39:45 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Sat, 14 Jul 2018 10:39:45 +0100
Subject: [Python-ideas] Make import an expression
In-Reply-To: <CADPFgprCWEZSG=o5puODzs_GDdQak69eb4ntwY_vgmw=+p7FEg@mail.gmail.com>
References: <CADPFgprCWEZSG=o5puODzs_GDdQak69eb4ntwY_vgmw=+p7FEg@mail.gmail.com>
Message-ID: <CALD=Yf9fDn4pUeWwUNYLRAoj3jP4aOHqZ0ST-S0Y=05mW8vJaA@mail.gmail.com>

Hi Ken

Thank you for your clear subject line. As you probably already know, in
Python, assignments are not expressions, and so we can't write
   if (a=get()):
       # do something with a
One reason for this is, perhaps, that "Explicit is better than implicit"
(as in The Zen of Python, via "import this").

Your last example of the intended semantics is
    b = a * (import math).tan(math.radians(45))
which demonstrate that you intend the statement
    a = (import math)
to have the side effect of assigning a value to the identifier 'math'.

I'm concerned that your intended semantics are similar to "assignment as an
expression", and so wouldn't fit in well with the rest of Python. More
concretely, when reading a module I expect to see all the imports at the
top, much like a recipe starts with a list of ingredients.

It may be that the documentation regarding import and __import__ can be
improved, and that there's a practical problem behind your experience that
__import__ is cumbersome and takes space.

with best regards

Jonathan


On Sat, Jul 14, 2018 at 9:24 AM, Ken Hilton <kenlhilton at gmail.com> wrote:

> Hi all,
>
> Just a curious idea I had. The subject says it all. "import" is currently
> a statement, which means it cannot be used inside anything else. Currently,
> the only way to import a module inside an expression is to use the
> __import__ function, which is both cumbersome and takes up more space
> (which is disadvantageous in things like code golf).
>
> I propose making "import" an expression, thus allowing the syntactic sugar
> of "import module" -> "__import__('module')" to be graduated to the
> expression level. Because of operator precedence, I suspect that in most
> cases one would need to surround the expression with parentheses.
> An additional effect that is not seen with using __import__ is that the
> module's name is bound to the current scope the way it is with a normal
> import statement.
>
> Examples of where this could be used:
>
> Importing on the fly to generate one value:
>
>     secret = (import random).randint(1, 100)
>
> Quick use of itertools in generator expressions:
>
>     (i * i for i in (import itertools).accumulate(generate_numbers()))
>
> Re-using a function from a module after importing it in an expression:
>
>     b = a * (import math).tan(math.radians(45)) #math.radians is valid
> because (import math) binds "math" to the current scope
>
> What are your thoughts?
>
> Sharing,
> Ken Hilton;
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180714/1277f75a/attachment.html>

From solipsis at pitrou.net  Sat Jul 14 05:40:18 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 14 Jul 2018 11:40:18 +0200
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CALFfu7A3vciEdV0FmjuZ9svumzbMXVCAHM-cuFKwP0WD-xw1SA@mail.gmail.com>
Message-ID: <20180714114018.695a410a@fsol>

On Fri, 13 Jul 2018 18:22:24 -0600
Eric Snow <ericsnowcurrently at gmail.com>
wrote:
> > 2. Give up on making things work inside the same OS process and rather
> > focus on implementing better abstractions on top of the existing
> > multiprocessing API so that the actor model is easier to program
> > against. For example, providing some notion of Channels to communicate
> > between lines of execution, a way to monitor the number of Messages
> > waiting in each channel for throughput profiling and diagnostics,
> > Supervision, etc. In particular I could do this by using an existing
> > library like Pykka or Thespian and extending it where necessary.  
> 
> It may worth a shot.  You should ask Davin Potts (CC'ed) about this.
> We discussed this a little at PyCon.  I'm sure he'd welcome help in
> improving the multiprocessing module.

Davin has been mostly inactive.  I'm the de facto maintainer for
multiprocessing.

Regards

Antoine.



From rosuav at gmail.com  Sat Jul 14 06:05:48 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 14 Jul 2018 20:05:48 +1000
Subject: [Python-ideas] Make import an expression
In-Reply-To: <CALD=Yf9fDn4pUeWwUNYLRAoj3jP4aOHqZ0ST-S0Y=05mW8vJaA@mail.gmail.com>
References: <CADPFgprCWEZSG=o5puODzs_GDdQak69eb4ntwY_vgmw=+p7FEg@mail.gmail.com>
 <CALD=Yf9fDn4pUeWwUNYLRAoj3jP4aOHqZ0ST-S0Y=05mW8vJaA@mail.gmail.com>
Message-ID: <CAPTjJmp-u1=0rACWNenLwcupT5db8171R6fywPG+GSNOMnh59Q@mail.gmail.com>

On Sat, Jul 14, 2018 at 7:39 PM, Jonathan Fine <jfine2358 at gmail.com> wrote:
> Hi Ken
>
> Thank you for your clear subject line. As you probably already know, in
> Python, assignments are not expressions, and so we can't write
>    if (a=get()):
>        # do something with a
> One reason for this is, perhaps, that "Explicit is better than implicit" (as
> in The Zen of Python, via "import this").
>
> Your last example of the intended semantics is
>     b = a * (import math).tan(math.radians(45))
> which demonstrate that you intend the statement
>     a = (import math)
> to have the side effect of assigning a value to the identifier 'math'.
>
> I'm concerned that your intended semantics are similar to "assignment as an
> expression", and so wouldn't fit in well with the rest of Python. More
> concretely, when reading a module I expect to see all the imports at the
> top, much like a recipe starts with a list of ingredients.
>
> It may be that the documentation regarding import and __import__ can be
> improved, and that there's a practical problem behind your experience that
> __import__ is cumbersome and takes space.
>

>>> import importlib
>>> secret = importlib.import_module('random').randint(1, 100)
>>> secret
83

Looks like you CAN import as an expression :)

ChrisA

From Richard at Damon-family.org  Sat Jul 14 07:53:27 2018
From: Richard at Damon-family.org (Richard Damon)
Date: Sat, 14 Jul 2018 07:53:27 -0400
Subject: [Python-ideas] Make import an expression
In-Reply-To: <CADPFgprCWEZSG=o5puODzs_GDdQak69eb4ntwY_vgmw=+p7FEg@mail.gmail.com>
References: <CADPFgprCWEZSG=o5puODzs_GDdQak69eb4ntwY_vgmw=+p7FEg@mail.gmail.com>
Message-ID: <76DCBAED-1D2C-4961-9925-D0F5BB1D1D5B@Damon-family.org>



> On Jul 14, 2018, at 4:24 AM, Ken Hilton <kenlhilton at gmail.com> wrote:
> 
> Hi all,
> 
> Just a curious idea I had. The subject says it all. "import" is currently a statement, which means it cannot be used inside anything else. Currently, the only way to import a module inside an expression is to use the __import__ function, which is both cumbersome and takes up more space (which is disadvantageous in things like code golf).
> 
> I propose making "import" an expression, thus allowing the syntactic sugar of "import module" -> "__import__('module')" to be graduated to the expression level. Because of operator precedence, I suspect that in most cases one would need to surround the expression with parentheses.
> An additional effect that is not seen with using __import__ is that the module's name is bound to the current scope the way it is with a normal import statement.
> 
> Examples of where this could be used:
> 
> Importing on the fly to generate one value:
> 
>     secret = (import random).randint(1, 100)
> 
> Quick use of itertools in generator expressions:
> 
>     (i * i for i in (import itertools).accumulate(generate_numbers()))
> 
> Re-using a function from a module after importing it in an expression:
> 
>     b = a * (import math).tan(math.radians(45)) #math.radians is valid because (import math) binds "math" to the current scope
> 
> What are your thoughts?
> 
> Sharing,
> Ken Hilton;
> 

First thought is that ?code golf? is unlikely to be a big reason to add something like this, especially since there is already a way to do what you want, just spelled somewhat longer.

Your syntax is likely going to add complexity to the expression parser, and if as you indicate most of the time your going to need the parentheses, you might as well spell (import xx) as import(xx) which makes a lot less of a change to the parser.

This gives us two distinctions from what we currently have, the operation is spelt as import instead of __import__, and we have dropped the need for putting quotes around the module name.

I don?t see a major issue with the first point, (except for the Zen of having only 1 obvious way to do something) though it might make sense to look at the discussion when __import__ was added to see why it was spelt that way.

For the second, you are basically just removing the need to put the module name in quotes, but this then loses the ability to put a variable name there to allow making the module to use be changed at run-time.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180714/8b229bee/attachment.html>

From ericsnowcurrently at gmail.com  Sat Jul 14 13:19:08 2018
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Sat, 14 Jul 2018 11:19:08 -0600
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <20180714114018.695a410a@fsol>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CALFfu7A3vciEdV0FmjuZ9svumzbMXVCAHM-cuFKwP0WD-xw1SA@mail.gmail.com>
 <20180714114018.695a410a@fsol>
Message-ID: <CALFfu7BxSRq0KrmxW0Bdk5N-yTocBZ1voDQ6EmQt+8vaE_HxNw@mail.gmail.com>

On Sat, Jul 14, 2018 at 3:41 AM Antoine Pitrou <solipsis at pitrou.net> wrote:
> Davin has been mostly inactive.  I'm the de facto maintainer for
> multiprocessing.

Ah, that's great to know.  Sorry about the confusion.

-eric

From python at mrabarnett.plus.com  Sat Jul 14 14:30:48 2018
From: python at mrabarnett.plus.com (MRAB)
Date: Sat, 14 Jul 2018 19:30:48 +0100
Subject: [Python-ideas] Make import an expression
In-Reply-To: <CALD=Yf9fDn4pUeWwUNYLRAoj3jP4aOHqZ0ST-S0Y=05mW8vJaA@mail.gmail.com>
References: <CADPFgprCWEZSG=o5puODzs_GDdQak69eb4ntwY_vgmw=+p7FEg@mail.gmail.com>
 <CALD=Yf9fDn4pUeWwUNYLRAoj3jP4aOHqZ0ST-S0Y=05mW8vJaA@mail.gmail.com>
Message-ID: <0a823801-419f-9532-0207-e7464fb6c966@mrabarnett.plus.com>

On 2018-07-14 10:39, Jonathan Fine wrote:
> Hi Ken
> 
> Thank you for your clear subject line. As you probably already know, in 
> Python, assignments are not expressions, and so we can't write
>  ? ?if (a=get()):
>  ? ? ? ?# do something with a
> One reason for this is, perhaps, that "Explicit is better than implicit" 
> (as in The Zen of Python, via "import this").
> 
> Your last example of the intended semantics is
>  ? ? b = a * (import math).tan(math.radians(45))
> which demonstrate that you intend the statement
>  ? ? a = (import math)
> to have the side effect of assigning a value to the identifier 'math'.
> 
> I'm concerned that your intended semantics are similar to "assignment as 
> an expression", and so wouldn't fit in well with the rest of Python. 
> More concretely, when reading a module I expect to see all the imports 
> at the top, much like a recipe starts with a list of ingredients.
> 
> It may be that the documentation regarding import and __import__ can be 
> improved, and that there's a practical problem behind your experience 
> that __import__ is cumbersome and takes space.
> 
With PEP 572 you would be able to write:

     b = a * (math := __import__('math')).tan(math.radians(45))

but I don't think that would be an improvement over:

     import math
     b = a * math.tan(math.radians(45))

So -1.

[snip]
> 
> On Sat, Jul 14, 2018 at 9:24 AM, Ken Hilton <kenlhilton at gmail.com 
> <mailto:kenlhilton at gmail.com>> wrote:
> 
>     Hi all,
> 
>     Just a curious idea I had. The subject says it all. "import" is
>     currently a statement, which means it cannot be used inside anything
>     else. Currently, the only way to import a module inside an
>     expression is to use the __import__ function, which is both
>     cumbersome and takes up more space (which is disadvantageous in
>     things like code golf).
> 
>     I propose making "import" an expression, thus allowing the syntactic
>     sugar of "import module" -> "__import__('module')" to be graduated
>     to the expression level. Because of operator precedence, I suspect
>     that in most cases one would need to surround the expression with
>     parentheses.
>     An additional effect that is not seen with using __import__ is that
>     the module's name is bound to the current scope the way it is with a
>     normal import statement.
> 
>     Examples of where this could be used:
> 
>     Importing on the fly to generate one value:
> 
>      ? ? secret = (import random).randint(1, 100)
> 
>     Quick use of itertools in generator expressions:
> 
>      ? ? (i * i for i in (import itertools).accumulate(generate_numbers()))
> 
>     Re-using a function from a module after importing it in an expression:
> 
>      ? ? b = a * (import math).tan(math.radians(45)) #math.radians is
>     valid because (import math) binds "math" to the current scope
> 
>     What are your thoughts?
> 

From tjreedy at udel.edu  Sat Jul 14 15:25:04 2018
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 14 Jul 2018 15:25:04 -0400
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <20180714114018.695a410a@fsol>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CALFfu7A3vciEdV0FmjuZ9svumzbMXVCAHM-cuFKwP0WD-xw1SA@mail.gmail.com>
 <20180714114018.695a410a@fsol>
Message-ID: <pidiid$dvu$1@blaine.gmane.org>

On 7/14/2018 5:40 AM, Antoine Pitrou wrote:
> On Fri, 13 Jul 2018 18:22:24 -0600
> Eric Snow <ericsnowcurrently at gmail.com>
> wrote:
>>> 2. Give up on making things work inside the same OS process and rather
>>> focus on implementing better abstractions on top of the existing
>>> multiprocessing API so that the actor model is easier to program
>>> against. For example, providing some notion of Channels to communicate
>>> between lines of execution, a way to monitor the number of Messages
>>> waiting in each channel for throughput profiling and diagnostics,
>>> Supervision, etc. In particular I could do this by using an existing
>>> library like Pykka or Thespian and extending it where necessary.
>>
>> It may worth a shot.  You should ask Davin Potts (CC'ed) about this.
>> We discussed this a little at PyCon.  I'm sure he'd welcome help in
>> improving the multiprocessing module.
> 
> Davin has been mostly inactive.  I'm the de facto maintainer for
> multiprocessing.

It's good to know that there is an active coredev who can be added as 
nosy on multiprocessing issues.  The multiprocessing line in the 
Expert's Index,
https://devguide.python.org/experts/
has Davin *ed (assign issues to him) and you not (nosy only).  Perhaps 
Davin should be un-starred.

Some time ago, on pydev list, you suggested that the solution to IDLE's 
problems with subprocess and sockets might be to use multiprocessing and 
pipes.  I noticed then that there were numerous bug report and little 
activity and wondered then how usable multiprocessing was in practice.

Checking again, there are 52 open behavior and 6 open crash issues with 
'multiprocessing' in the title.  The must severe one for IDLE that I 
noticed is #33111: importing tkinter and running multiprocessing on 
MacOS does not seem to work.

This week I (re)read the main multiprocessing doc chapter.  The main 
issue I saw is 'Beware of replacing sys.stdin with a ?file like 
object?'.  I don't *think* that this is a showstopper, but a minimal 
failing example would help to be sure.

-- 
Terry Jan Reedy



From tim.peters at gmail.com  Sat Jul 14 16:39:25 2018
From: tim.peters at gmail.com (Tim Peters)
Date: Sat, 14 Jul 2018 15:39:25 -0500
Subject: [Python-ideas] Add the imath module
In-Reply-To: <20180714045417.GM7318@ando.pearwood.info>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <51e71cc9801047e9b21c4dc01c64826e@xmail103.UGent.be>
 <5B485E58.1000503@UGent.be>
 <CAExdVN=mmFfx9UsMB=KjMS-QpSM84Rn9V4SyytgMPRDMUEy43w@mail.gmail.com>
 <20180714045417.GM7318@ando.pearwood.info>
Message-ID: <CAExdVNkkG5mWbHvyCbmKGs3EfVxaE-Sfqf1wi1PDX+HhN0P5Ww@mail.gmail.com>

[Tim]

> > Steven's numbers are pretty baffling to me, since these are all composite
> > and so iterating Miller-Rabin "should get out" pretty fast:
>

[Steven]

> That's because you haven't seen my code :-)
>

I'd be baffled anyway ;-)


about 4.7 seconds to test 2**800 + 1;
>

I got 1.71 msec, over a thousand times faster.


> less than a tenth of a millisecond to test 2**900 + 1;
>

I got 2.32 msec, over 20 times _slower_(!).

and about 8.6 seconds to test 2**1000 + 1.


And I got 3.14 msec, again over a thousand times faster.

It's over-engineered, class-based, and written as a learning exercise.
> There's a compatibility layer so it will work back to Python 2.4 and an
> instrumentation layer which I inserted in a fit of enthusiasm to gather
> staticistics, which I have never once looked at since.
>

While my code had no fluff at all.  It's hard to believe you could add
enough cruft to slow it down by a factor of 1000, but pretty much
impossible to believe adding cruft could make it way faster in the middle
case above.

Or do you first try division by small primes?  2**900+1 could get out cheap
in that case, because it happens to be divisible by 17.



> And *even so* it is still fast enough for casual use at the interactive
> interpreter, compared to more naive algorithms with worse Big O
> performance characteristics.
>
> You might think 5 seconds is slow, but I'm serious when I say some of
> the other algorithms I played with would take *days* to generate,
> or check, largish primes.
>

No argument from me.  There's a reason I wrote my `pp()` to begin with too
-)  State-of-the-art code for these things requires planet-sized brains and
commitment, but even a little expert knowledge can yield simple code that's
an enormous improvement over any "dead obvious" approach.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180714/fa662926/attachment-0001.html>

From steve at pearwood.info  Sat Jul 14 20:59:42 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 15 Jul 2018 10:59:42 +1000
Subject: [Python-ideas] Add the imath module
In-Reply-To: <CAExdVNkkG5mWbHvyCbmKGs3EfVxaE-Sfqf1wi1PDX+HhN0P5Ww@mail.gmail.com>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <51e71cc9801047e9b21c4dc01c64826e@xmail103.UGent.be>
 <5B485E58.1000503@UGent.be>
 <CAExdVN=mmFfx9UsMB=KjMS-QpSM84Rn9V4SyytgMPRDMUEy43w@mail.gmail.com>
 <20180714045417.GM7318@ando.pearwood.info>
 <CAExdVNkkG5mWbHvyCbmKGs3EfVxaE-Sfqf1wi1PDX+HhN0P5Ww@mail.gmail.com>
Message-ID: <20180715005939.GN7318@ando.pearwood.info>

On Sat, Jul 14, 2018 at 03:39:25PM -0500, Tim Peters wrote:

> While my code had no fluff at all.  It's hard to believe you could add
> enough cruft to slow it down by a factor of 1000, 

There's a factor of 20 because my computer is older and slower than 
yours. (I ran your version, and it was ~20 times slower on my computer 
than the numbers you give.)

So the cruft factor is only 50 times :)


> but pretty much
> impossible to believe adding cruft could make it way faster in the middle
> case above.
> 
> Or do you first try division by small primes?  2**900+1 could get out cheap
> in that case, because it happens to be divisible by 17.

Yes indeed. My approach is:

1. trial division by small primes;
2. Miller-Rabin by a minimal set of provably deterministic 
   witnesses (if such a set exists, which it does for n < 2**64);
3. or by the first twelve primes if no such known set exists;
4. followed by 30 random witnesses.

12+30 = 42 Miller-Rabin tests may be overkill :-)


-- 
Steve

From ncoghlan at gmail.com  Sun Jul 15 00:20:05 2018
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 15 Jul 2018 14:20:05 +1000
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <c00bf3ef-ef42-9b43-247a-5adf9d4826d9@gmail.com>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <20180709161754.GA6167@trent.me>
 <c00bf3ef-ef42-9b43-247a-5adf9d4826d9@gmail.com>
Message-ID: <CADiSq7fEyM=bOhAr7nL4dR9hUR-QbB3pUuz=RO72GAt24k9oKw@mail.gmail.com>

On 11 July 2018 at 00:31, David Foster <davidfstr at gmail.com> wrote:
> I was not aware of PyParallel. The PyParellel "parallel thread"
> line-of-execution implementation is pretty interesting. Trent, big kudos to
> you on that effort.
>
> Since you're speaking in the past tense and said "but we're not doing it
> like that", I infer that the notion of a parallel thread was turned down for
> integration into CPython, as that appears to have been the original goal.
>
> However I am unable to locate a rationale for why that integration was
> turned down. Was it deemed to be too complex to execute, perhaps in the
> context of providing C extension compatibility? Was there a desire to see a
> similar implementation on Linux as well as Windows? Some other reason?

It was never extended beyond Windows, and a Windows-only solution
doesn't meet the needs of a lot of folks interested in more efficient
exploitation of multiple local CPU cores.

It's still an interesting design concept though, especially for
problems that can be deconstructed into a setup phase (read/write main
thread), and a parallel operation phase (ephemeral worker threads that
store all persistent state in memory mapped files, or otherwise
outside the current process).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sun Jul 15 00:49:43 2018
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 15 Jul 2018 14:49:43 +1000
Subject: [Python-ideas] Add the imath module
In-Reply-To: <pi7lmf$2ea$1@blaine.gmane.org>
References: <pi7fga$flg$1@blaine.gmane.org> <pi7ju2$7ut$1@blaine.gmane.org>
 <CA+msPNk_wxKsHkv7nbWk5Kgn2HBoT+Mz9s6gvYeabaJqyGcxqA@mail.gmail.com>
 <pi7lmf$2ea$1@blaine.gmane.org>
Message-ID: <CADiSq7fpj04xYJd0_2+1_Q48Kii==zNFodZfBaP-ApHxY2BoTQ@mail.gmail.com>

On 12 July 2018 at 23:41, Serhiy Storchaka <storchaka at gmail.com> wrote:
> 12.07.18 16:15, Robert Vanden Eynde ????:
>>
>> About the name, why not intmath ?
>
>
> Because cmath. But if most core developers prefer intmath, I have no
> objections.

My initial reaction from just the subject title was "But we already
have cmath, why would we need imath?", based on the fact that
mathematicians write complex numbers as "X + Yi", rather than the "X +
Yj" that Python borrowed from electrical engineering (where "i"
already had a different meaning as the symbol for AC current).

Calling the proposed module "intmath" instead would be clearer to me
(and I agree with the rationale that as the number of int-specific
functions increases, separating them out from the more float-centric
math module makes sense).

Beyond that, I think the analogy with the statistics module is a good one:

1. There are a number of integer-specific algorithms where naive
implementations are going to be slower than they need to be, subtly
incorrect in some cases, or both. With a standard library module, we
can provide a robust test suite, and pointers towards higher
performance alternatives for folks that need them.
2. For educational purposes, being able to introduce the use cases for
a capability before delving into the explanation of how that
capability works can be quite a powerful technique (the standard
implementations can also provide a cross-check on the accuracy of
student implementations).

And in the intmath case, there are likely to be more opportunities to
delete private implementations from other standard library modules in
favour of the public intmath functions.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From tim.peters at gmail.com  Sun Jul 15 02:27:40 2018
From: tim.peters at gmail.com (Tim Peters)
Date: Sun, 15 Jul 2018 01:27:40 -0500
Subject: [Python-ideas] Add the imath module
In-Reply-To: <VI1P194MB007989DEB270351D32D7B7C09B5F0@VI1P194MB0079.EURP194.PROD.OUTLOOK.COM>
References: <pi7fga$flg$1@blaine.gmane.org>
 <CAEbHw4bfQWRm0BB=NcW2AEWrwAhcudSHb7Bq17iQUDUTaxnXpQ@mail.gmail.com>
 <pi7l1m$lr5$1@blaine.gmane.org>
 <CAEbHw4aMSVSezVY=T=v2EwCG+hU=KYR3mn8FDHcj-jNEqzHRiA@mail.gmail.com>
 <51e71cc9801047e9b21c4dc01c64826e@xmail103.UGent.be>
 <5B485E58.1000503@UGent.be>
 <CAExdVN=mmFfx9UsMB=KjMS-QpSM84Rn9V4SyytgMPRDMUEy43w@mail.gmail.com>
 <20180714045417.GM7318@ando.pearwood.info>
 <VI1P194MB007989DEB270351D32D7B7C09B5F0@VI1P194MB0079.EURP194.PROD.OUTLOOK.COM>
Message-ID: <CAExdVNkSnw=mWsbk38OrYi8CY0G8Jo0oCBgpa2O_e3sZvbZ2bQ@mail.gmail.com>

[Steve Barnes]

> Looking for some information on how long it has taken to generate large
> primes I stumbled across
> https://arxiv.org/ftp/arxiv/papers/1709/1709.09963.pdf which makes
> interesting reading. It concentrates on giving no false negatives (never
> saying n is not a prime when it is) but giving an increasing probability
> that n is a prime as testing goes on.
>

That's a very nice, gentle introduction!  Thanks for sharing it.

The `pp()` function I posted before is a Python implementation of the
Miller-Rabin test they ended with.  The latter is very widely used, because
the code is relatively simple. short, and fast, and has no "bad cases".

For _generating_ primes, the article uses other tricks to weed out
sure-to-be-composite candidates before bothering with probable-prime
testing.  But the tricks they use in the paper are specific to a base 10
representation.  In binary, it's common to skip candidates such that

    gcd(candidate, 2 * 3 * 5 * ...) > 1

where the product-of-small-primes second argument is picked to be the
largest such that bigint division by it is exceptionally fast (usually one
"digit" in whatever internal power-of-2 base is used by the bigint
implementation).


I did find an article from 2016 that mentioned M77232917 (a 23,249,425
> digit Mersenne prime) and M74207281 (22,338,618 digit Mersenne prime)
> with the latter taking a month to check with the Lucas-Lehmer test.
>

More, it's a special version of Lucas-Lehmer (LL) that only works for
testing candidates of the form 2**p-1 where p is itself an odd prime.  In
that context, it's not probabilistic - it definitively answers whether
2**p-1 is prime.  Fast as it is, Miller-Rabin is impractical for guessing
about numbers of this size.  And as enormously faster as _it_ is, the
special version of LL used for Mersenne testing is largely hand-coded in
assembler to squeeze out every possible machine cycle.

I don't want any of that in Python's standard library either ;-)

Here!  You should enjoy this account of the even larger Mersenne prime
discovered this year:

https://www.mersenne.org/primes/press/M77232917.html

Note that the hyper-optimized testing code they use is so historically
error-prone (& so good at provoking HW errors on overheated machines!) that
they now verify each new Mersenne prime by 4 entirely different LL
implementations, on 4 different machines.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180715/5ad9aa4c/attachment.html>

From turnbull.stephen.fw at u.tsukuba.ac.jp  Sun Jul 15 14:39:29 2018
From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull)
Date: Mon, 16 Jul 2018 03:39:29 +0900
Subject: [Python-ideas] Unicode Name Aliases keyword argument
 abbreviation in unicodedata.name for missing names
In-Reply-To: <VI1PR03MB2909E5A3D47263B6A9500646B6590@VI1PR03MB2909.eurprd03.prod.outlook.com>
References: <VI1PR03MB29093F08C0489886B5D758E2B6590@VI1PR03MB2909.eurprd03.prod.outlook.com>
 <20180712071725.GR7318@ando.pearwood.info>
 <20180712103739.6hxsoyltxvjbsn4d@phdru.name>
 <VI1PR03MB2909445BDBF24848ECF8029AB6590@VI1PR03MB2909.eurprd03.prod.outlook.com>
 <23367.28028.242930.802545@turnbull.sk.tsukuba.ac.jp>
 <VI1PR03MB2909E5A3D47263B6A9500646B6590@VI1PR03MB2909.eurprd03.prod.outlook.com>
Message-ID: <23371.38113.323492.484537@turnbull.sk.tsukuba.ac.jp>

Robert Vanden Eynde writes:

 > Not for control characters.

There's a standard convention for "naming" control characters (U+0000,
U+0001, etc), which is recommended by the Unicode Standard (in
slightly generalized form) for characters that otherwise don't have
names, as "code point labels".  This has been suggested by MRAB in the
past.  Personally I would generalized Steven d'Aprano's function a
bit, and provide a "CONTROL-" prefix for these instead of "U+".

I don't see why even the C0 ASCII control function aliases should be
particularly privileged, especially since the main alias is the
spelled-out name, not the more commonly used 2- or 3-character
abbreviation (will people associate "alarm" with "BEL"? I don't).
Many are just meaningless (the 4 "device control" codes).  And some
are actively misleading: U+0018 (^X) "cancel" and U+001A (^Z)
"substitute", which are generally interpreted as "exit" (an
interactive program) and "end of file" (on Windows), or as "cut" and
"revert" in CUA UI.  I for one would find it more useful if they
aliased to "ctrl-c-prefix" and "zap-up-to-char".[1]

And nobody's ever heard of the C1 ISO 6249 control characters (not to
mention that three of them are literally figments of somebody's
imagination, and never standardized).

So I think using NameAliases.txt for this purpose is silly.  If we're
going to provide aliases based on the traditional control functions, I
would use only the NameAliases.txt aliases for the following: NUL,
BEL, BS, HT, LF, VT, FF, CR, ESC, SP, DEL, NEL, NBSP, and SHY.  (NEL
is included because it's recommended that it be treated as a newline
function in the Unicode standard.)  For the rest, I would use
CONTROL-<code>, which is more likely to make sense in most
contexts.[2]

 > About the Han case, they all have a
 > unicodedata.name<http://unicodedata.name> don't they ? (Sorry if I
 > misread your message)

Yes, they have names, constructed algorithmically from the code point:
"CJK UNIFIED IDEOGRAPH-4E00".  I know what that one is (the character
that denotes the number 1).  But that's the only one that I know
offhand.

I think Han (which are named daily, surely millions, if not billions,
of times) should be treated as well as controls (which even
programmers rarely bother to name, especially for those that don't
have standard escape sequences).  That's why I strongly advocate that
there be provision for extension, and that the databases at least be
provided by a module that can be updated far more frequently than the
stdlib is.

Footnotes: 
[1]  Those are the commands they are bound to in Emacs.

[2]  There are a few others that I personally would find useful and
unambiguous because they're used in multilingual ISO 2022 encodings,
but that's rather far into the weeds.  They're rarely seen in
practice; most of the time 7-bit codes with escape sequences are used,
or 8-bit codes without control sequences.

From santagada at gmail.com  Sun Jul 15 18:49:15 2018
From: santagada at gmail.com (Leonardo Santagada)
Date: Mon, 16 Jul 2018 00:49:15 +0200
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <CADiSq7fEyM=bOhAr7nL4dR9hUR-QbB3pUuz=RO72GAt24k9oKw@mail.gmail.com>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <20180709161754.GA6167@trent.me>
 <c00bf3ef-ef42-9b43-247a-5adf9d4826d9@gmail.com>
 <CADiSq7fEyM=bOhAr7nL4dR9hUR-QbB3pUuz=RO72GAt24k9oKw@mail.gmail.com>
Message-ID: <CA+HFc84TzOjUNzuwDVr3KYhzrCzNeW0h8qDMYQYqdwzStNkCeQ@mail.gmail.com>

No one talked about this, but modern cpus with multiple numa nodes are
atrocious to any shared memory (maybe threadripper is better, but
multiple socket xeon is slow) and more and more all cpus will move to
it on single chips, so a share nothing aproach can really make python
a good contender on modern hardware.

On Sun, Jul 15, 2018 at 6:20 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 11 July 2018 at 00:31, David Foster <davidfstr at gmail.com> wrote:
>> I was not aware of PyParallel. The PyParellel "parallel thread"
>> line-of-execution implementation is pretty interesting. Trent, big kudos to
>> you on that effort.
>>
>> Since you're speaking in the past tense and said "but we're not doing it
>> like that", I infer that the notion of a parallel thread was turned down for
>> integration into CPython, as that appears to have been the original goal.
>>
>> However I am unable to locate a rationale for why that integration was
>> turned down. Was it deemed to be too complex to execute, perhaps in the
>> context of providing C extension compatibility? Was there a desire to see a
>> similar implementation on Linux as well as Windows? Some other reason?
>
> It was never extended beyond Windows, and a Windows-only solution
> doesn't meet the needs of a lot of folks interested in more efficient
> exploitation of multiple local CPU cores.
>
> It's still an interesting design concept though, especially for
> problems that can be deconstructed into a setup phase (read/write main
> thread), and a parallel operation phase (ephemeral worker threads that
> store all persistent state in memory mapped files, or otherwise
> outside the current process).
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 

Leonardo Santagada

From njs at pobox.com  Sun Jul 15 20:31:17 2018
From: njs at pobox.com (Nathaniel Smith)
Date: Sun, 15 Jul 2018 17:31:17 -0700
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
Message-ID: <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>

On Sun, Jul 8, 2018 at 11:27 AM, David Foster <davidfstr at gmail.com> wrote:
> * The Actor model can be used with some effort via the ?multiprocessing?
> module, but it doesn?t seem that streamlined and forces there to be a
> separate OS process per line of execution, which is relatively expensive.

What do you mean by "the Actor model"? Just shared-nothing
concurrency? (My understanding is that in academia it means
shared-nothing + every thread/process/whatever gets an associated
queue + queues are globally addressable + queues have unbounded
buffering + every thread/process/whatever is implemented as a loop
that reads messages from its queue and responds to them, with no
internal concurrency. I don't know why this particular bundle of
features is considered special. Lots of people seem to use it in
looser sense though.)

> I'd like to solicit some feedback on what might be the most efficient way to
> make forward progress on efficient parallelization in Python inside the same
> OS process. The most promising areas appear to be:
>
> 1. Make the current subinterpreter implementation in Python have more
> complete isolation, sharing almost no state between subinterpreters. In
> particular not sharing the GIL. The "Interpreter Isolation" section of PEP
> 554 enumerates areas that are currently shared, some of which probably
> shouldn't be.
>
> 2. Give up on making things work inside the same OS process and rather focus
> on implementing better abstractions on top of the existing multiprocessing
> API so that the actor model is easier to program against. For example,
> providing some notion of Channels to communicate between lines of execution,
> a way to monitor the number of Messages waiting in each channel for
> throughput profiling and diagnostics, Supervision, etc. In particular I
> could do this by using an existing library like Pykka or Thespian and
> extending it where necessary.

I guess I would distinguish though between "multiple processes" and
"the multiprocessing module". The module might be at the point in its
lifecycle where starting over is at least worth considering, and one
thing I'm hoping to do with Trio is experiment with making worker
process patterns easier to work with.

But the nice thing about these two options is that subinterpreters are
basically a way to emulate multiple Python processes within a single
OS process, which means they're largely interchangeable. There are
trade-offs in terms of compatibility, how much work needs to be done,
probably speed, but if you come up with a great API based around one
model then you should be able to switch out the backend later without
affecting users. So if you want to start experimenting now, I'd use
multiple processes and plan to switch to subinterpreters later if it
turns out to make sense.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From rosuav at gmail.com  Sun Jul 15 21:00:34 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 16 Jul 2018 11:00:34 +1000
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>
Message-ID: <CAPTjJmqKGhNEcrAuzKSzcUW_kvdmPqL9_9CMyOUzcrBs=UdcEg@mail.gmail.com>

On Mon, Jul 16, 2018 at 10:31 AM, Nathaniel Smith <njs at pobox.com> wrote:
> On Sun, Jul 8, 2018 at 11:27 AM, David Foster <davidfstr at gmail.com> wrote:
>> * The Actor model can be used with some effort via the ?multiprocessing?
>> module, but it doesn?t seem that streamlined and forces there to be a
>> separate OS process per line of execution, which is relatively expensive.
>
> What do you mean by "the Actor model"? Just shared-nothing
> concurrency? (My understanding is that in academia it means
> shared-nothing + every thread/process/whatever gets an associated
> queue + queues are globally addressable + queues have unbounded
> buffering + every thread/process/whatever is implemented as a loop
> that reads messages from its queue and responds to them, with no
> internal concurrency. I don't know why this particular bundle of
> features is considered special. Lots of people seem to use it in
> looser sense though.)

Shared-nothing concurrency is, of course, the very easiest way to
parallelize. But let's suppose you're trying to create an online
multiplayer game. Since it's a popular genre at the moment, I'll go
for a battle royale game (think PUBG, H1Z1, Fortnite, etc). A hundred
people enter; one leaves. The game has to let those hundred people
interact, which means that all hundred people have to be connected to
the same server. And you have to process everyone's movements,
gunshots, projectiles, etc, etc, etc, fast enough to be able to run a
server "tick" enough times per second - I would say 32 ticks per
second is an absolute minimum, 64 is definitely better. So what
happens when the processing required takes more than one CPU core for
1/32 seconds? A shared-nothing model is either fundamentally
impossible, or a meaningless abstraction (if you interpret it to mean
"explicit queues/pipes for everything"). What would the "Actor" model
do here?

Ideally, I would like to be able to write my code as a set of
functions, then easily spin them off as separate threads, and have
them able to magically run across separate CPUs. Unicorns not being a
thing, I'm okay with warping my code a bit around the need for
parallelism, but I'm not sure how best to do that. Assume here that we
can't cheat by getting most of the processing work done with the GIL
released (eg in Numpy), and it actually does require Python-level
parallelism of CPU-heavy work.

ChrisA

From njs at pobox.com  Sun Jul 15 23:21:56 2018
From: njs at pobox.com (Nathaniel Smith)
Date: Sun, 15 Jul 2018 20:21:56 -0700
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <CAPTjJmqKGhNEcrAuzKSzcUW_kvdmPqL9_9CMyOUzcrBs=UdcEg@mail.gmail.com>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>
 <CAPTjJmqKGhNEcrAuzKSzcUW_kvdmPqL9_9CMyOUzcrBs=UdcEg@mail.gmail.com>
Message-ID: <CAPJVwBkpLw_qRirCFq4PK6n5zLVxc=t=_bf_SZRXLsSnrMVECQ@mail.gmail.com>

On Sun, Jul 15, 2018 at 6:00 PM, Chris Angelico <rosuav at gmail.com> wrote:
> On Mon, Jul 16, 2018 at 10:31 AM, Nathaniel Smith <njs at pobox.com> wrote:
>> On Sun, Jul 8, 2018 at 11:27 AM, David Foster <davidfstr at gmail.com> wrote:
>>> * The Actor model can be used with some effort via the ?multiprocessing?
>>> module, but it doesn?t seem that streamlined and forces there to be a
>>> separate OS process per line of execution, which is relatively expensive.
>>
>> What do you mean by "the Actor model"? Just shared-nothing
>> concurrency? (My understanding is that in academia it means
>> shared-nothing + every thread/process/whatever gets an associated
>> queue + queues are globally addressable + queues have unbounded
>> buffering + every thread/process/whatever is implemented as a loop
>> that reads messages from its queue and responds to them, with no
>> internal concurrency. I don't know why this particular bundle of
>> features is considered special. Lots of people seem to use it in
>> looser sense though.)
>
> Shared-nothing concurrency is, of course, the very easiest way to
> parallelize. But let's suppose you're trying to create an online
> multiplayer game. Since it's a popular genre at the moment, I'll go
> for a battle royale game (think PUBG, H1Z1, Fortnite, etc). A hundred
> people enter; one leaves. The game has to let those hundred people
> interact, which means that all hundred people have to be connected to
> the same server. And you have to process everyone's movements,
> gunshots, projectiles, etc, etc, etc, fast enough to be able to run a
> server "tick" enough times per second - I would say 32 ticks per
> second is an absolute minimum, 64 is definitely better. So what
> happens when the processing required takes more than one CPU core for
> 1/32 seconds? A shared-nothing model is either fundamentally
> impossible, or a meaningless abstraction (if you interpret it to mean
> "explicit queues/pipes for everything"). What would the "Actor" model
> do here?

"Shared-nothing" is a bit of jargon that means there's no *implicit*
sharing; your threads can still communicate, the communication just
has to be explicit. I don't know exactly what algorithms your
hypothetical game needs, but they might be totally fine in a
shared-nothing approach. It's not just for embarrassingly parallel
problems.

> Ideally, I would like to be able to write my code as a set of
> functions, then easily spin them off as separate threads, and have
> them able to magically run across separate CPUs. Unicorns not being a
> thing, I'm okay with warping my code a bit around the need for
> parallelism, but I'm not sure how best to do that. Assume here that we
> can't cheat by getting most of the processing work done with the GIL
> released (eg in Numpy), and it actually does require Python-level
> parallelism of CPU-heavy work.

If you need shared-memory threads, on multiple cores, for CPU-bound
logic, where the logic is implemented in Python, then yeah, you
basically need a free-threaded implementation of Python. Jython is
such an implementation. PyPy could be if anyone were interested in
funding it [1], but apparently no-one is. Probably removing the GIL
from CPython is impossible. (I'd be happy to be proven wrong.) Sorry I
don't have anything better to report.

The good news is that there are many, many situations where you don't
actually need "shared-memory threads, on multiple cores, for CPU-bound
logic, where the logic is implemented in Python". If you're in that
specific niche and don't have $100k to throw at PyPy, then I dunno, I
hear Rust is good at that sort of thing? It's frustrating for sure,
but there will always be niches where Python isn't the best choice.

-n

[1] https://morepypy.blogspot.com/2017/08/lets-remove-global-interpreter-lock.html

-- 
Nathaniel J. Smith -- https://vorpus.org

From rosuav at gmail.com  Mon Jul 16 00:24:49 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 16 Jul 2018 14:24:49 +1000
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <CAPJVwBkpLw_qRirCFq4PK6n5zLVxc=t=_bf_SZRXLsSnrMVECQ@mail.gmail.com>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>
 <CAPTjJmqKGhNEcrAuzKSzcUW_kvdmPqL9_9CMyOUzcrBs=UdcEg@mail.gmail.com>
 <CAPJVwBkpLw_qRirCFq4PK6n5zLVxc=t=_bf_SZRXLsSnrMVECQ@mail.gmail.com>
Message-ID: <CAPTjJmrt83c96JQ3PY2oZEm68ZMbG_JBwaJ1XpcyzqRnEcks9A@mail.gmail.com>

On Mon, Jul 16, 2018 at 1:21 PM, Nathaniel Smith <njs at pobox.com> wrote:
> On Sun, Jul 15, 2018 at 6:00 PM, Chris Angelico <rosuav at gmail.com> wrote:
>> On Mon, Jul 16, 2018 at 10:31 AM, Nathaniel Smith <njs at pobox.com> wrote:
>>> On Sun, Jul 8, 2018 at 11:27 AM, David Foster <davidfstr at gmail.com> wrote:
>>>> * The Actor model can be used with some effort via the ?multiprocessing?
>>>> module, but it doesn?t seem that streamlined and forces there to be a
>>>> separate OS process per line of execution, which is relatively expensive.
>>>
>>> What do you mean by "the Actor model"? Just shared-nothing
>>> concurrency? (My understanding is that in academia it means
>>> shared-nothing + every thread/process/whatever gets an associated
>>> queue + queues are globally addressable + queues have unbounded
>>> buffering + every thread/process/whatever is implemented as a loop
>>> that reads messages from its queue and responds to them, with no
>>> internal concurrency. I don't know why this particular bundle of
>>> features is considered special. Lots of people seem to use it in
>>> looser sense though.)
>>
>> Shared-nothing concurrency is, of course, the very easiest way to
>> parallelize. But let's suppose you're trying to create an online
>> multiplayer game. Since it's a popular genre at the moment, I'll go
>> for a battle royale game (think PUBG, H1Z1, Fortnite, etc). A hundred
>> people enter; one leaves. The game has to let those hundred people
>> interact, which means that all hundred people have to be connected to
>> the same server. And you have to process everyone's movements,
>> gunshots, projectiles, etc, etc, etc, fast enough to be able to run a
>> server "tick" enough times per second - I would say 32 ticks per
>> second is an absolute minimum, 64 is definitely better. So what
>> happens when the processing required takes more than one CPU core for
>> 1/32 seconds? A shared-nothing model is either fundamentally
>> impossible, or a meaningless abstraction (if you interpret it to mean
>> "explicit queues/pipes for everything"). What would the "Actor" model
>> do here?
>
> "Shared-nothing" is a bit of jargon that means there's no *implicit*
> sharing; your threads can still communicate, the communication just
> has to be explicit. I don't know exactly what algorithms your
> hypothetical game needs, but they might be totally fine in a
> shared-nothing approach. It's not just for embarrassingly parallel
> problems.

Right, so basically it's the exact model that Python *already* has for
multiprocessing - once you go to separate processes, nothing is
implicitly shared, and everything has to be done with queues.

>> Ideally, I would like to be able to write my code as a set of
>> functions, then easily spin them off as separate threads, and have
>> them able to magically run across separate CPUs. Unicorns not being a
>> thing, I'm okay with warping my code a bit around the need for
>> parallelism, but I'm not sure how best to do that. Assume here that we
>> can't cheat by getting most of the processing work done with the GIL
>> released (eg in Numpy), and it actually does require Python-level
>> parallelism of CPU-heavy work.
>
> If you need shared-memory threads, on multiple cores, for CPU-bound
> logic, where the logic is implemented in Python, then yeah, you
> basically need a free-threaded implementation of Python. Jython is
> such an implementation. PyPy could be if anyone were interested in
> funding it [1], but apparently no-one is. Probably removing the GIL
> from CPython is impossible. (I'd be happy to be proven wrong.) Sorry I
> don't have anything better to report.

(This was a purely hypothetical example.)

There could be some interesting results from using the GIL only for
truly global objects, and then having other objects guarded by arena
locks. The trouble is that, in CPython, as soon as you reference any
read-only object from the globals, you need to raise its refcount.
ISTR someone mentioned something along the lines of
sys.eternalize(obj) to flag something as "never GC this thing, it no
longer has a refcount", which would then allow global objects to be
referenced in a truly read-only way (eg to call a function). Sadly,
I'm not expert enough to actually look into implementing it, but it
does seem like a very cool concept. It also fits into the "warping my
code a bit" category (eg eternalizing a small handful of key objects,
and paying the price of "well, now they can never be garbage
collected"), with the potential to then parallelize more easily.

> The good news is that there are many, many situations where you don't
> actually need "shared-memory threads, on multiple cores, for CPU-bound
> logic, where the logic is implemented in Python".

Oh absolutely. MOST of my parallelism requirements involve regular
Python threads, because they spend most of their time blocked on
something. That one is easy. The hassle comes when something MIGHT
need parallelism and might not, based on (say) how much data it has to
work with; for those kinds of programs, I would like to be able to
code it the simple way with minimal code overhead, but still able to
split over cores. And yes, I'm aware that it's never going to be
perfect, but the closer the better.

ChrisA

From stephanh42 at gmail.com  Mon Jul 16 01:00:34 2018
From: stephanh42 at gmail.com (Stephan Houben)
Date: Mon, 16 Jul 2018 07:00:34 +0200
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <CAPTjJmrt83c96JQ3PY2oZEm68ZMbG_JBwaJ1XpcyzqRnEcks9A@mail.gmail.com>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>
 <CAPTjJmqKGhNEcrAuzKSzcUW_kvdmPqL9_9CMyOUzcrBs=UdcEg@mail.gmail.com>
 <CAPJVwBkpLw_qRirCFq4PK6n5zLVxc=t=_bf_SZRXLsSnrMVECQ@mail.gmail.com>
 <CAPTjJmrt83c96JQ3PY2oZEm68ZMbG_JBwaJ1XpcyzqRnEcks9A@mail.gmail.com>
Message-ID: <CAOOa=pPE=2j=6XFTKLPkWLqz=1nYiftU0XV4bVhehVzPd4NSzA@mail.gmail.com>

What about the following model: you have N Python interpreters, each with
their own GIL. Each *Python* object belongs to precisely one interpreter.

However, the interpreters share some common data storage: perhaps a shared
Numpy array, or a shared Sqlite in-memory db. Or some key-value store where
the key and values are binary data. The interpreters communicate through
that.

Stephan

Op ma 16 jul. 2018 06:25 schreef Chris Angelico <rosuav at gmail.com>:

> On Mon, Jul 16, 2018 at 1:21 PM, Nathaniel Smith <njs at pobox.com> wrote:
> > On Sun, Jul 15, 2018 at 6:00 PM, Chris Angelico <rosuav at gmail.com>
> wrote:
> >> On Mon, Jul 16, 2018 at 10:31 AM, Nathaniel Smith <njs at pobox.com>
> wrote:
> >>> On Sun, Jul 8, 2018 at 11:27 AM, David Foster <davidfstr at gmail.com>
> wrote:
> >>>> * The Actor model can be used with some effort via the
> ?multiprocessing?
> >>>> module, but it doesn?t seem that streamlined and forces there to be a
> >>>> separate OS process per line of execution, which is relatively
> expensive.
> >>>
> >>> What do you mean by "the Actor model"? Just shared-nothing
> >>> concurrency? (My understanding is that in academia it means
> >>> shared-nothing + every thread/process/whatever gets an associated
> >>> queue + queues are globally addressable + queues have unbounded
> >>> buffering + every thread/process/whatever is implemented as a loop
> >>> that reads messages from its queue and responds to them, with no
> >>> internal concurrency. I don't know why this particular bundle of
> >>> features is considered special. Lots of people seem to use it in
> >>> looser sense though.)
> >>
> >> Shared-nothing concurrency is, of course, the very easiest way to
> >> parallelize. But let's suppose you're trying to create an online
> >> multiplayer game. Since it's a popular genre at the moment, I'll go
> >> for a battle royale game (think PUBG, H1Z1, Fortnite, etc). A hundred
> >> people enter; one leaves. The game has to let those hundred people
> >> interact, which means that all hundred people have to be connected to
> >> the same server. And you have to process everyone's movements,
> >> gunshots, projectiles, etc, etc, etc, fast enough to be able to run a
> >> server "tick" enough times per second - I would say 32 ticks per
> >> second is an absolute minimum, 64 is definitely better. So what
> >> happens when the processing required takes more than one CPU core for
> >> 1/32 seconds? A shared-nothing model is either fundamentally
> >> impossible, or a meaningless abstraction (if you interpret it to mean
> >> "explicit queues/pipes for everything"). What would the "Actor" model
> >> do here?
> >
> > "Shared-nothing" is a bit of jargon that means there's no *implicit*
> > sharing; your threads can still communicate, the communication just
> > has to be explicit. I don't know exactly what algorithms your
> > hypothetical game needs, but they might be totally fine in a
> > shared-nothing approach. It's not just for embarrassingly parallel
> > problems.
>
> Right, so basically it's the exact model that Python *already* has for
> multiprocessing - once you go to separate processes, nothing is
> implicitly shared, and everything has to be done with queues.
>
> >> Ideally, I would like to be able to write my code as a set of
> >> functions, then easily spin them off as separate threads, and have
> >> them able to magically run across separate CPUs. Unicorns not being a
> >> thing, I'm okay with warping my code a bit around the need for
> >> parallelism, but I'm not sure how best to do that. Assume here that we
> >> can't cheat by getting most of the processing work done with the GIL
> >> released (eg in Numpy), and it actually does require Python-level
> >> parallelism of CPU-heavy work.
> >
> > If you need shared-memory threads, on multiple cores, for CPU-bound
> > logic, where the logic is implemented in Python, then yeah, you
> > basically need a free-threaded implementation of Python. Jython is
> > such an implementation. PyPy could be if anyone were interested in
> > funding it [1], but apparently no-one is. Probably removing the GIL
> > from CPython is impossible. (I'd be happy to be proven wrong.) Sorry I
> > don't have anything better to report.
>
> (This was a purely hypothetical example.)
>
> There could be some interesting results from using the GIL only for
> truly global objects, and then having other objects guarded by arena
> locks. The trouble is that, in CPython, as soon as you reference any
> read-only object from the globals, you need to raise its refcount.
> ISTR someone mentioned something along the lines of
> sys.eternalize(obj) to flag something as "never GC this thing, it no
> longer has a refcount", which would then allow global objects to be
> referenced in a truly read-only way (eg to call a function). Sadly,
> I'm not expert enough to actually look into implementing it, but it
> does seem like a very cool concept. It also fits into the "warping my
> code a bit" category (eg eternalizing a small handful of key objects,
> and paying the price of "well, now they can never be garbage
> collected"), with the potential to then parallelize more easily.
>
> > The good news is that there are many, many situations where you don't
> > actually need "shared-memory threads, on multiple cores, for CPU-bound
> > logic, where the logic is implemented in Python".
>
> Oh absolutely. MOST of my parallelism requirements involve regular
> Python threads, because they spend most of their time blocked on
> something. That one is easy. The hassle comes when something MIGHT
> need parallelism and might not, based on (say) how much data it has to
> work with; for those kinds of programs, I would like to be able to
> code it the simple way with minimal code overhead, but still able to
> split over cores. And yes, I'm aware that it's never going to be
> perfect, but the closer the better.
>
> ChrisA
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180716/493d9598/attachment-0001.html>

From rosuav at gmail.com  Mon Jul 16 01:07:05 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 16 Jul 2018 15:07:05 +1000
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <CAOOa=pPE=2j=6XFTKLPkWLqz=1nYiftU0XV4bVhehVzPd4NSzA@mail.gmail.com>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>
 <CAPTjJmqKGhNEcrAuzKSzcUW_kvdmPqL9_9CMyOUzcrBs=UdcEg@mail.gmail.com>
 <CAPJVwBkpLw_qRirCFq4PK6n5zLVxc=t=_bf_SZRXLsSnrMVECQ@mail.gmail.com>
 <CAPTjJmrt83c96JQ3PY2oZEm68ZMbG_JBwaJ1XpcyzqRnEcks9A@mail.gmail.com>
 <CAOOa=pPE=2j=6XFTKLPkWLqz=1nYiftU0XV4bVhehVzPd4NSzA@mail.gmail.com>
Message-ID: <CAPTjJmounkjTjzstCtd0RBGktXVMxxX5nwqp8dLQXg_ftk2m2Q@mail.gmail.com>

On Mon, Jul 16, 2018 at 3:00 PM, Stephan Houben <stephanh42 at gmail.com> wrote:
> What about the following model: you have N Python interpreters, each with
> their own GIL. Each *Python* object belongs to precisely one interpreter.
>
> However, the interpreters share some common data storage: perhaps a shared
> Numpy array, or a shared Sqlite in-memory db. Or some key-value store where
> the key and values are binary data. The interpreters communicate through
> that.

Interesting. The actual concrete idea that I had in mind was an image
comparison job, downloading umpteen billion separate images and trying
to find the one most similar to a template. Due to lack of easy
parallelism I was unable to then compare the top thousand against each
other quadratically, but it would be interesting to see if I could
have done something like that to share image comparison information.

ChrisA

From turnbull.stephen.fw at u.tsukuba.ac.jp  Mon Jul 16 01:27:57 2018
From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull)
Date: Mon, 16 Jul 2018 14:27:57 +0900
Subject: [Python-ideas] Unicode Name Aliases keyword argument
 abbreviation in unicodedata.name for missing names
In-Reply-To: <20180712171444.GY7318@ando.pearwood.info>
References: <VI1PR03MB29093F08C0489886B5D758E2B6590@VI1PR03MB2909.eurprd03.prod.outlook.com>
 <20180712071725.GR7318@ando.pearwood.info>
 <20180712103739.6hxsoyltxvjbsn4d@phdru.name>
 <VI1PR03MB2909445BDBF24848ECF8029AB6590@VI1PR03MB2909.eurprd03.prod.outlook.com>
 <23367.28028.242930.802545@turnbull.sk.tsukuba.ac.jp>
 <20180712171444.GY7318@ando.pearwood.info>
Message-ID: <23372.11485.773047.149869@turnbull.sk.tsukuba.ac.jp>

Steven D'Aprano writes:

 > Sorry, I'm not sure if you mean my proposed alias() function isn't
 > useful, or Robert's try...except loop around it.

I was questioning the utility of "If the abbreviation list is sorted
by AdditionToUnicodeDate."

But since you ask, neither function is useful TO ME, as I understand
them, because they're based on the UCD NameAliases.txt.  That doesn't
have any aliases I would actually use.  I've never needed aliases for
control characters, and for everything else the canonical name is
perfectly useful (including for Korean characters and Japanese kana,
which have phonetic names, as do Chinese bopomofo AIUI).  There's
nothing useful for Han characters yet, sadly.

 > My alias() function is just an programmatic interface to information 
 > already available in the NameAliases.txt file. Don't you think that's 
 > useful enough as it stands?

To be perfectly frank, if that's all it is, I don't know when I'd ever
use it.  Your label function is *much* more useful.

To be specific about the defects of NameAliases.txt: "DEVICE CONTROL
1" tells me a lot less about that control character than "U+0011"
does.  Other aliases in that file are just wrong: I don't believe I've
ever seen U+001A used as "SUBSTITUTE" for an unrepresentable coded
character entity.  That's the DOS "END OF FILE".  Certainly, the
aliases of category "correction" are useful, though not to me---I
don't read any of the relevant languages.  The "figment" category is
stupid; almost all the names of control characters are figments,
except for the half-dozen well-known whitespace characters, NUL, and
maybe DEL.  The 256 VSxxx "variation selectors" are somewhat useful,
but I would think that it would be even more useful to provide skin
color aliases for face emoji and X11 RGB.txt color aliases for hearts
and the like, which presumably are standardized across vendors.

If I were designing a feature for the stdlib, I would

0.  Allow the database to consist of multiple alias tables, and be
    extensible by adding tables via user configuration.
1.  Make the priority of the alias tables user-configurable.
2.  Provide default top-priority table more suited to likely Python
    usage than NameAliases.txt.
3.  Provide both a primary alias function, and a list of all aliases
    function.
4.  Provide a reverse lookup function.
5.  Perhaps provide a context-sensitive alias function.  The only
    context I can think of offhand is "position in file", ie, to
    distinguish between ZWNBSP and BOM, so perhaps that's not worth
    doing.  On the other hand, given that example, it's worth a few
    minutes thought to see if there are other context-sensitive naming
    practices that more than a few people would want to follow.

 > Indeed. [Multiple non-UCD aliases is] also the case for
 > emoji. That's why I suggested making alias() return a mutable
 > record rather than an immutable tuple, so application writers can
 > add their own records to suit their own needs.

Why should they add them to the tuple returned by the function, rather
than to the database the function consults?

 > fully-fledged PEP -- but I think the critical point here is that we 
 > shouldn't be privileging one alias type over the others.

I don't understand.  By providing stdlib support for NameAliases.txt
only, you are privileging those aliases.  If you mean privileging the
Name property over the aliases, well, that's what "canonical" means,
and yes, I think the Name property should be privileged (eg ZERO WIDTH
NO-BREAK SPACE over BYTE ORDER MARK).

 > That seems fairly extreme. New Unicode versions don't come out that
 > frequently. Surely we don't expect to track draft aliases, or
 > characters outside of Unicode?

Why not track draft aliases in a "draft alias" table?  More important,
why not track aliases of *Unicode* characters that could use aliases
(eg, translations), in separate tables?  For example, there are "shape
based names" for Han characters, which are standard enough so that
users would be able to construct them (Unicode 11 includes one such
system, see section 18.2).  And Japanese names for Han radicals often
vary from the UCD Name property, and are often more precise (many
describe the geometric relation of the radical to the rest of the
character).

It is not obvious to me that an alias() that only looks at
NameAliases.txt is so useful as to belong in the stdlib, but on the
other hand providing a module that can include rapidly accumulating
databases along the lines I've mentioned above definitely doesn't
belong in the stdlib (a la pytz).

On the other hand, the *access functions* might belong in the stdlib
---in the same way that timezone-sensitive datetime APIs do---but that
sort of requires knowing what databases and "schema" are out there,
and trying to set things up so that the same APIs can access a number
of databases.

 > To clarify, do you mean the aliases defined in NameAliases.txt? Or a 
 > subset of them?

I didn't understand your alias function correctly, which I think is
overengineered for the purpose of handling aliases.  I was thinking in
terms of returning a string, or at most general a list of strings.  If
you are going to define a class to represent metadata about a
character, why not make *all* metadata available?  Probably most of
the attributes would be properties, lazily accessing various
databases:

class Codepoint(object):
    def __init__(self, codepoint):
        self.codepoint = codepoint

    @property
    def name(self):
        # Access name database and cache result.

    @property
    def category(self):
        # Access category database and cache result.

    @property
    def alias(self):
        # Populates alias_list, and returns the first one.

    @property
    def alias_list(self):
        # Access alias database (not limited to NameAliases.txt) and
        # cache result.

    @property
    def label(self):
        # Populates and returns name, if available, otherwise a code
        # point label.

and so on.  But that's a new thread.

 > > And even there I think a canonical name based on block name +
 > > code point in hex is the best way to go.
 > 
 > I believe you might be thinking of the Unicode "code point label" 
 > concept.

Yes, as MRAB has suggested.  I would be a little more precise than he,
in that I would label the C0 and C1 control blocks with CONTROL-<code>
rather than just U+<code>.

Steve


From sebastian at realpath.org  Mon Jul 16 04:21:15 2018
From: sebastian at realpath.org (Sebastian Krause)
Date: Mon, 16 Jul 2018 10:21:15 +0200
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <CADiSq7fEyM=bOhAr7nL4dR9hUR-QbB3pUuz=RO72GAt24k9oKw@mail.gmail.com>
 (Nick Coghlan's message of "Sun, 15 Jul 2018 14:20:05 +1000")
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <20180709161754.GA6167@trent.me>
 <c00bf3ef-ef42-9b43-247a-5adf9d4826d9@gmail.com>
 <CADiSq7fEyM=bOhAr7nL4dR9hUR-QbB3pUuz=RO72GAt24k9oKw@mail.gmail.com>
Message-ID: <868t6by4jo.fsf@realpath.org>

Nick Coghlan <ncoghlan at gmail.com> wrote:
> It was never extended beyond Windows, and a Windows-only solution
> doesn't meet the needs of a lot of folks interested in more efficient
> exploitation of multiple local CPU cores.

On the other hand Windows has a higher need for a better multi-core
story. A reasonable Unix-only solution already exists with fork() if
you don't need a lot of shared memory, multi-processing on Windows
is just not on the same level.

From solipsis at pitrou.net  Mon Jul 16 05:16:01 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 16 Jul 2018 11:16:01 +0200
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>
 <CAPTjJmqKGhNEcrAuzKSzcUW_kvdmPqL9_9CMyOUzcrBs=UdcEg@mail.gmail.com>
 <CAPJVwBkpLw_qRirCFq4PK6n5zLVxc=t=_bf_SZRXLsSnrMVECQ@mail.gmail.com>
Message-ID: <20180716111601.100c560a@fsol>

On Sun, 15 Jul 2018 20:21:56 -0700
Nathaniel Smith <njs at pobox.com> wrote:
> 
> If you need shared-memory threads, on multiple cores, for CPU-bound
> logic, where the logic is implemented in Python, then yeah, you
> basically need a free-threaded implementation of Python. Jython is
> such an implementation. PyPy could be if anyone were interested in
> funding it [1], but apparently no-one is. Probably removing the GIL
> from CPython is impossible. (I'd be happy to be proven wrong.)

It's not that it's impossible, it's that everyone trying to remove it
ended up with a 30-40% slowdown in a single-threaded mode (*).  Perhaps
Larry manages to do better, though ;-)

(*) a figure which I assume is highly workload-dependent

Regards

Antoine.



From solipsis at pitrou.net  Mon Jul 16 05:16:44 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 16 Jul 2018 11:16:44 +0200
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>
 <CAPTjJmqKGhNEcrAuzKSzcUW_kvdmPqL9_9CMyOUzcrBs=UdcEg@mail.gmail.com>
 <CAPJVwBkpLw_qRirCFq4PK6n5zLVxc=t=_bf_SZRXLsSnrMVECQ@mail.gmail.com>
 <CAPTjJmrt83c96JQ3PY2oZEm68ZMbG_JBwaJ1XpcyzqRnEcks9A@mail.gmail.com>
 <CAOOa=pPE=2j=6XFTKLPkWLqz=1nYiftU0XV4bVhehVzPd4NSzA@mail.gmail.com>
Message-ID: <20180716111644.5bd2e87b@fsol>

On Mon, 16 Jul 2018 07:00:34 +0200
Stephan Houben <stephanh42 at gmail.com>
wrote:
> What about the following model: you have N Python interpreters, each with
> their own GIL. Each *Python* object belongs to precisely one interpreter.

This is roughly what Eric's subinterpreters approach tries to do.

Regards

Antoine.



From python at mrabarnett.plus.com  Mon Jul 16 13:00:37 2018
From: python at mrabarnett.plus.com (MRAB)
Date: Mon, 16 Jul 2018 18:00:37 +0100
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <CAPTjJmrt83c96JQ3PY2oZEm68ZMbG_JBwaJ1XpcyzqRnEcks9A@mail.gmail.com>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>
 <CAPTjJmqKGhNEcrAuzKSzcUW_kvdmPqL9_9CMyOUzcrBs=UdcEg@mail.gmail.com>
 <CAPJVwBkpLw_qRirCFq4PK6n5zLVxc=t=_bf_SZRXLsSnrMVECQ@mail.gmail.com>
 <CAPTjJmrt83c96JQ3PY2oZEm68ZMbG_JBwaJ1XpcyzqRnEcks9A@mail.gmail.com>
Message-ID: <6a5b228a-200a-b2c7-6067-df2b0022ae42@mrabarnett.plus.com>

On 2018-07-16 05:24, Chris Angelico wrote:
> On Mon, Jul 16, 2018 at 1:21 PM, Nathaniel Smith <njs at pobox.com> wrote:
>> On Sun, Jul 15, 2018 at 6:00 PM, Chris Angelico <rosuav at gmail.com> wrote:
>>> On Mon, Jul 16, 2018 at 10:31 AM, Nathaniel Smith <njs at pobox.com> wrote:
>>>> On Sun, Jul 8, 2018 at 11:27 AM, David Foster <davidfstr at gmail.com> wrote:
>>>>> * The Actor model can be used with some effort via the ?multiprocessing?
>>>>> module, but it doesn?t seem that streamlined and forces there to be a
>>>>> separate OS process per line of execution, which is relatively expensive.
>>>>
>>>> What do you mean by "the Actor model"? Just shared-nothing
>>>> concurrency? (My understanding is that in academia it means
>>>> shared-nothing + every thread/process/whatever gets an associated
>>>> queue + queues are globally addressable + queues have unbounded
>>>> buffering + every thread/process/whatever is implemented as a loop
>>>> that reads messages from its queue and responds to them, with no
>>>> internal concurrency. I don't know why this particular bundle of
>>>> features is considered special. Lots of people seem to use it in
>>>> looser sense though.)
>>>
>>> Shared-nothing concurrency is, of course, the very easiest way to
>>> parallelize. But let's suppose you're trying to create an online
>>> multiplayer game. Since it's a popular genre at the moment, I'll go
>>> for a battle royale game (think PUBG, H1Z1, Fortnite, etc). A hundred
>>> people enter; one leaves. The game has to let those hundred people
>>> interact, which means that all hundred people have to be connected to
>>> the same server. And you have to process everyone's movements,
>>> gunshots, projectiles, etc, etc, etc, fast enough to be able to run a
>>> server "tick" enough times per second - I would say 32 ticks per
>>> second is an absolute minimum, 64 is definitely better. So what
>>> happens when the processing required takes more than one CPU core for
>>> 1/32 seconds? A shared-nothing model is either fundamentally
>>> impossible, or a meaningless abstraction (if you interpret it to mean
>>> "explicit queues/pipes for everything"). What would the "Actor" model
>>> do here?
>>
>> "Shared-nothing" is a bit of jargon that means there's no *implicit*
>> sharing; your threads can still communicate, the communication just
>> has to be explicit. I don't know exactly what algorithms your
>> hypothetical game needs, but they might be totally fine in a
>> shared-nothing approach. It's not just for embarrassingly parallel
>> problems.
> 
> Right, so basically it's the exact model that Python *already* has for
> multiprocessing - once you go to separate processes, nothing is
> implicitly shared, and everything has to be done with queues.
> 
>>> Ideally, I would like to be able to write my code as a set of
>>> functions, then easily spin them off as separate threads, and have
>>> them able to magically run across separate CPUs. Unicorns not being a
>>> thing, I'm okay with warping my code a bit around the need for
>>> parallelism, but I'm not sure how best to do that. Assume here that we
>>> can't cheat by getting most of the processing work done with the GIL
>>> released (eg in Numpy), and it actually does require Python-level
>>> parallelism of CPU-heavy work.
>>
>> If you need shared-memory threads, on multiple cores, for CPU-bound
>> logic, where the logic is implemented in Python, then yeah, you
>> basically need a free-threaded implementation of Python. Jython is
>> such an implementation. PyPy could be if anyone were interested in
>> funding it [1], but apparently no-one is. Probably removing the GIL
>> from CPython is impossible. (I'd be happy to be proven wrong.) Sorry I
>> don't have anything better to report.
> 
> (This was a purely hypothetical example.)
> 
> There could be some interesting results from using the GIL only for
> truly global objects, and then having other objects guarded by arena
> locks. The trouble is that, in CPython, as soon as you reference any
> read-only object from the globals, you need to raise its refcount.
> ISTR someone mentioned something along the lines of
> sys.eternalize(obj) to flag something as "never GC this thing, it no
> longer has a refcount", which would then allow global objects to be
> referenced in a truly read-only way (eg to call a function). Sadly,
> I'm not expert enough to actually look into implementing it, but it
> does seem like a very cool concept. It also fits into the "warping my
> code a bit" category (eg eternalizing a small handful of key objects,
> and paying the price of "well, now they can never be garbage
> collected"), with the potential to then parallelize more easily.
> 
Could you explicitly share an object in a similar way to how you 
explicitly open a file?

The shared object's refcount would be incremented and the sharing 
function would return a proxy to the shared object.

Refcounting in the thread/process would be done on the proxy.

When the proxy is closed or garbage-collected, the shared object's 
refcount would be decremented.

The shared object could be garbage-collected when its refcount drops to 
zero.

>> The good news is that there are many, many situations where you don't
>> actually need "shared-memory threads, on multiple cores, for CPU-bound
>> logic, where the logic is implemented in Python".
> 
> Oh absolutely. MOST of my parallelism requirements involve regular
> Python threads, because they spend most of their time blocked on
> something. That one is easy. The hassle comes when something MIGHT
> need parallelism and might not, based on (say) how much data it has to
> work with; for those kinds of programs, I would like to be able to
> code it the simple way with minimal code overhead, but still able to
> split over cores. And yes, I'm aware that it's never going to be
> perfect, but the closer the better.
> 

From solipsis at pitrou.net  Mon Jul 16 13:07:35 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 16 Jul 2018 19:07:35 +0200
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>
 <CAPTjJmqKGhNEcrAuzKSzcUW_kvdmPqL9_9CMyOUzcrBs=UdcEg@mail.gmail.com>
 <CAPJVwBkpLw_qRirCFq4PK6n5zLVxc=t=_bf_SZRXLsSnrMVECQ@mail.gmail.com>
 <CAPTjJmrt83c96JQ3PY2oZEm68ZMbG_JBwaJ1XpcyzqRnEcks9A@mail.gmail.com>
 <6a5b228a-200a-b2c7-6067-df2b0022ae42@mrabarnett.plus.com>
Message-ID: <20180716190735.2908e86d@fsol>

On Mon, 16 Jul 2018 18:00:37 +0100
MRAB <python at mrabarnett.plus.com> wrote:
> Could you explicitly share an object in a similar way to how you 
> explicitly open a file?
> 
> The shared object's refcount would be incremented and the sharing 
> function would return a proxy to the shared object.
> 
> Refcounting in the thread/process would be done on the proxy.
> 
> When the proxy is closed or garbage-collected, the shared object's 
> refcount would be decremented.
> 
> The shared object could be garbage-collected when its refcount drops to 
> zero.

Yes, I'm assuming that would be how shareable buffers could be
implemented: a per-interpreter proxy (with a regular Python refcount)
mediating access to a shared object (which could have an atomic /
thread-safe refcount).

As for how shareable buffers could be useful, see my work on PEP 574:
https://www.python.org/dev/peps/pep-0574/

Regards

Antoine.



From mrbm74 at gmail.com  Mon Jul 16 13:56:34 2018
From: mrbm74 at gmail.com (Martin Bammer)
Date: Mon, 16 Jul 2018 19:56:34 +0200
Subject: [Python-ideas] Do not block threads when pickle/unpickle
Message-ID: <35b734bd-5b29-746f-404d-849de9c12e17@gmail.com>

Hi,

the old and slow python implementation of pickle didn't block background 
thread.

But the newer C-implementation blocks other threads while dump/load is 
running.

Wouldn't it be possible to allow other threads during this time?

Especially could load/loads release the GIL, because Python objects are 
not available to the Python code until these functions have finished?

Regards,

Martin



From joejev at gmail.com  Mon Jul 16 14:05:35 2018
From: joejev at gmail.com (Joseph Jevnik)
Date: Mon, 16 Jul 2018 14:05:35 -0400
Subject: [Python-ideas] Do not block threads when pickle/unpickle
In-Reply-To: <35b734bd-5b29-746f-404d-849de9c12e17@gmail.com>
References: <35b734bd-5b29-746f-404d-849de9c12e17@gmail.com>
Message-ID: <CAHGq92V3u5DX2r5HoEtAaERhKYQV-Qj4N713e7=DN=Arw1L6Og@mail.gmail.com>

The GIL must be held to allocate memory for Python objects and to
invoke the Python code to deserialize user defined picklable objects.
I don't think there is a long span of time where the code could leave
the GIL released. The Python implementation is just pausing to let
other Python threads run, but it is not actually able to parallelize.
The same would be true of the C implementation, is there a reason to
want to pause the thread to let another thread run?

On Mon, Jul 16, 2018 at 1:56 PM, Martin Bammer <mrbm74 at gmail.com> wrote:
> Hi,
>
> the old and slow python implementation of pickle didn't block background
> thread.
>
> But the newer C-implementation blocks other threads while dump/load is
> running.
>
> Wouldn't it be possible to allow other threads during this time?
>
> Especially could load/loads release the GIL, because Python objects are not
> available to the Python code until these functions have finished?
>
> Regards,
>
> Martin
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From solipsis at pitrou.net  Mon Jul 16 14:06:38 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 16 Jul 2018 20:06:38 +0200
Subject: [Python-ideas] Do not block threads when pickle/unpickle
References: <35b734bd-5b29-746f-404d-849de9c12e17@gmail.com>
Message-ID: <20180716200638.574b2542@fsol>


Hi,

On Mon, 16 Jul 2018 19:56:34 +0200
Martin Bammer <mrbm74 at gmail.com> wrote:
> Hi,
> 
> the old and slow python implementation of pickle didn't block background 
> thread.
> 
> But the newer C-implementation blocks other threads while dump/load is 
> running.

This is a fair comment.
Please open an issue on the bug tracker at https://bugs.python.org/

Regards

Antoine.



From mrbm74 at gmail.com  Mon Jul 16 14:47:25 2018
From: mrbm74 at gmail.com (Martin Bammer)
Date: Mon, 16 Jul 2018 20:47:25 +0200
Subject: [Python-ideas] Do not block threads when pickle/unpickle
In-Reply-To: <CAHGq92V3u5DX2r5HoEtAaERhKYQV-Qj4N713e7=DN=Arw1L6Og@mail.gmail.com>
References: <35b734bd-5b29-746f-404d-849de9c12e17@gmail.com>
 <CAHGq92V3u5DX2r5HoEtAaERhKYQV-Qj4N713e7=DN=Arw1L6Og@mail.gmail.com>
Message-ID: <a4ba10f5-2b64-8aec-cac9-137bed172f55@gmail.com>

If you've for example an application with a GUI. Every time bigger 
objects are pickled/unpickled the complete GUI blocks.

Or if you've a server application with multiple network connections and 
the application should have a guaranteed response time, then it is 
impossible if one single client could lead the server to block for a 
longer time if it sends a big object which takes long to unpickle.

Just 2 simple examples.


Am 2018-07-16 um 20:05 schrieb Joseph Jevnik:
> The GIL must be held to allocate memory for Python objects and to
> invoke the Python code to deserialize user defined picklable objects.
> I don't think there is a long span of time where the code could leave
> the GIL released. The Python implementation is just pausing to let
> other Python threads run, but it is not actually able to parallelize.
> The same would be true of the C implementation, is there a reason to
> want to pause the thread to let another thread run?
>
> On Mon, Jul 16, 2018 at 1:56 PM, Martin Bammer <mrbm74 at gmail.com> wrote:
>> Hi,
>>
>> the old and slow python implementation of pickle didn't block background
>> thread.
>>
>> But the newer C-implementation blocks other threads while dump/load is
>> running.
>>
>> Wouldn't it be possible to allow other threads during this time?
>>
>> Especially could load/loads release the GIL, because Python objects are not
>> available to the Python code until these functions have finished?
>>
>> Regards,
>>
>> Martin
>>
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/


From mrbm74 at gmail.com  Mon Jul 16 14:47:59 2018
From: mrbm74 at gmail.com (Martin Bammer)
Date: Mon, 16 Jul 2018 20:47:59 +0200
Subject: [Python-ideas] Do not block threads when pickle/unpickle
In-Reply-To: <CAHGq92V3u5DX2r5HoEtAaERhKYQV-Qj4N713e7=DN=Arw1L6Og@mail.gmail.com>
References: <35b734bd-5b29-746f-404d-849de9c12e17@gmail.com>
 <CAHGq92V3u5DX2r5HoEtAaERhKYQV-Qj4N713e7=DN=Arw1L6Og@mail.gmail.com>
Message-ID: <8b927fe7-e4ea-82fc-3e74-335900516bf5@gmail.com>

If you've for example an application with a GUI. Every time bigger 
objects are pickled/unpickled the complete GUI blocks.

Or if you've a server application with multiple network connections and 
the application should have a guaranteed response time, then it is 
impossible if one single client could lead the server to block for a 
longer time if it sends a big object which takes long to unpickle.

Just 2 simple examples.


Am 2018-07-16 um 20:05 schrieb Joseph Jevnik:
> The GIL must be held to allocate memory for Python objects and to
> invoke the Python code to deserialize user defined picklable objects.
> I don't think there is a long span of time where the code could leave
> the GIL released. The Python implementation is just pausing to let
> other Python threads run, but it is not actually able to parallelize.
> The same would be true of the C implementation, is there a reason to
> want to pause the thread to let another thread run?
>
> On Mon, Jul 16, 2018 at 1:56 PM, Martin Bammer <mrbm74 at gmail.com> wrote:
>> Hi,
>>
>> the old and slow python implementation of pickle didn't block background
>> thread.
>>
>> But the newer C-implementation blocks other threads while dump/load is
>> running.
>>
>> Wouldn't it be possible to allow other threads during this time?
>>
>> Especially could load/loads release the GIL, because Python objects are not
>> available to the Python code until these functions have finished?
>>
>> Regards,
>>
>> Martin
>>
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/


From brett at python.org  Mon Jul 16 15:24:43 2018
From: brett at python.org (Brett Cannon)
Date: Mon, 16 Jul 2018 12:24:43 -0700
Subject: [Python-ideas] Including the unparse module in the standard
 library
In-Reply-To: <CAGMu_=r8Jad-68Ho+57QkaY2r6oXaBQU-wuDN64NxRyb=vUmng@mail.gmail.com>
References: <CAGMu_=r8Jad-68Ho+57QkaY2r6oXaBQU-wuDN64NxRyb=vUmng@mail.gmail.com>
Message-ID: <CAP1=2W6gaEu7RE3fAycCu3J=gMbYZejsF6Xb0aei70s1obU9bw@mail.gmail.com>

On Thu, 12 Jul 2018 at 11:21 Andre Roberge <andre.roberge at gmail.com> wrote:

> In the cPython repository, there is an unparse module in the Tools section.
> https://github.com/python/cpython/blob/master/Tools/parser/unparse.py
>
> However, as it is not part of the standard library, it cannot be easily
> used; to do so, one needs to make a local copy in a place from where it can
> be imported.
>
> This module can be useful for people using the ast module to create and
> parse trees, modify them ... and who want to convert the result back into
> source code.  Since it is obviously maintained to be compatible with the
> current Python version,
>

Maintained doesn't mean it's check on very often, nor designed well or
updated. ;)


> would it be possible to include the unparse module in the standard library?
>

Since it isn't necessary for Python to function, I would say we probably
don''t want to pull it up. Then the maintenance burden grows much more.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180716/ef3cec4d/attachment.html>

From chris.barker at noaa.gov  Mon Jul 16 19:11:15 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Mon, 16 Jul 2018 16:11:15 -0700
Subject: [Python-ideas] Including the unparse module in the standard
 library
In-Reply-To: <CAP1=2W6gaEu7RE3fAycCu3J=gMbYZejsF6Xb0aei70s1obU9bw@mail.gmail.com>
References: <CAGMu_=r8Jad-68Ho+57QkaY2r6oXaBQU-wuDN64NxRyb=vUmng@mail.gmail.com>
 <CAP1=2W6gaEu7RE3fAycCu3J=gMbYZejsF6Xb0aei70s1obU9bw@mail.gmail.com>
Message-ID: <CALGmxE+ZaLsL+f4nQOfLZ5p+8gK2HozRJ0=3+z+01DwmVGVpSw@mail.gmail.com>

On Mon, Jul 16, 2018 at 12:24 PM, Brett Cannon <brett at python.org> wrote:

>
> Since it isn't necessary for Python to function, I would say we probably
> don''t want to pull it up. Then the maintenance burden grows much more.
>

might make sense to put it on pypi though, if someone want to take
responsibility for it.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180716/993485d2/attachment.html>

From ericsnowcurrently at gmail.com  Tue Jul 17 15:35:48 2018
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Tue, 17 Jul 2018 13:35:48 -0600
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <20180716190735.2908e86d@fsol>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>
 <CAPTjJmqKGhNEcrAuzKSzcUW_kvdmPqL9_9CMyOUzcrBs=UdcEg@mail.gmail.com>
 <CAPJVwBkpLw_qRirCFq4PK6n5zLVxc=t=_bf_SZRXLsSnrMVECQ@mail.gmail.com>
 <CAPTjJmrt83c96JQ3PY2oZEm68ZMbG_JBwaJ1XpcyzqRnEcks9A@mail.gmail.com>
 <6a5b228a-200a-b2c7-6067-df2b0022ae42@mrabarnett.plus.com>
 <20180716190735.2908e86d@fsol>
Message-ID: <CALFfu7BFLSSRf7_mVUP3Rh4tAg1ae1cawc9KFF66EiBc2Hfn7g@mail.gmail.com>

On Mon, Jul 16, 2018 at 11:08 AM Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Mon, 16 Jul 2018 18:00:37 +0100
> MRAB <python at mrabarnett.plus.com> wrote:
> > Could you explicitly share an object in a similar way to how you
> > explicitly open a file?
> >
> > The shared object's refcount would be incremented and the sharing
> > function would return a proxy to the shared object.
> >
> > Refcounting in the thread/process would be done on the proxy.
> >
> > When the proxy is closed or garbage-collected, the shared object's
> > refcount would be decremented.
> >
> > The shared object could be garbage-collected when its refcount drops to
> > zero.
>
> Yes, I'm assuming that would be how shareable buffers could be
> implemented: a per-interpreter proxy (with a regular Python refcount)
> mediating access to a shared object (which could have an atomic /
> thread-safe refcount).

Nice! That's exactly how I'm doing it. :)  The buffer protocol makes
it easier, but the idea could apply to arbitrary objects generally.
That's something I'll look into in a later phase of the project.

In both cases the tricky part is ensuring that the proxy does not
directly mutate the object (especially the refcount).  In fact, the
decref part above is the trickiest.  The trickiness is a consequence
of our goals.  In my multi-core project we're aiming for not sharing
the GIL between interpreters.  That means reaching and keeping proper
separation between interpreters.  Notably, without a GIL shared by
interpreters, refcount operations are not thread-safe.  Also, in the
decref case GC would happen under the wrong interpreter (which is
problematic for several reasons).

With this in mind, here's how I'm approaching the problem:

1. interp A "shares" an object with interp B (e.g. through a channel)
    * the object is incref'ed under A before it is sent to B
2. the object is wrapped in a proxy owned by B
    * the proxy may not make C-API calls that would mutate the object
or even cause an incref/decref
3. when the proxy is GC'd, the original object is decref'ed
    * the decref must happen in a thread in which A is running

In order to make all this work the missing piece is a mechanism by
which the decref (#3) happens under the original interpreter.  At the
moment Emily Morehouse and I are pursuing an approach that extends the
existing ceval "pending call" machinery currently used for handling
signals (see Py_AddPendingCall).  The new [*private*] API would work
the same way but on a per-interpreter basis rather than just the main
interpreter.  This would allow one interpreter to queue up a decref to
happen later under another interpreter.

FWIW, this ability to decref an object under a different interpreter
is a blocker right now for a number of things, including supporting
buffers in PEP 554 channels.

-eric

From ericsnowcurrently at gmail.com  Tue Jul 17 16:00:28 2018
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Tue, 17 Jul 2018 14:00:28 -0600
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <890D0D1D-7166-43B5-8BC0-8B2177B3BAE0@barrys-emacs.org>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>
 <CAPTjJmqKGhNEcrAuzKSzcUW_kvdmPqL9_9CMyOUzcrBs=UdcEg@mail.gmail.com>
 <CAPJVwBkpLw_qRirCFq4PK6n5zLVxc=t=_bf_SZRXLsSnrMVECQ@mail.gmail.com>
 <CAPTjJmrt83c96JQ3PY2oZEm68ZMbG_JBwaJ1XpcyzqRnEcks9A@mail.gmail.com>
 <6a5b228a-200a-b2c7-6067-df2b0022ae42@mrabarnett.plus.com>
 <20180716190735.2908e86d@fsol>
 <CALFfu7BFLSSRf7_mVUP3Rh4tAg1ae1cawc9KFF66EiBc2Hfn7g@mail.gmail.com>
 <890D0D1D-7166-43B5-8BC0-8B2177B3BAE0@barrys-emacs.org>
Message-ID: <CALFfu7CC=2ZYFoYxf_ijU_V43F_g90RAtJZr=+nT5vgRwP4Q5g@mail.gmail.com>

On Tue, Jul 17, 2018 at 1:44 PM Barry <barry at barrys-emacs.org> wrote:
> The decrement itself is not the problem, that can be made thread safe.

Yeah, by using the GIL. <wink>  Otherwise, please elaborate.  My
understanding is that if the decrement itself were not the problem
then we'd have gotten rid of the GIL already.

> Do you mean that once the ref reaches 0 you have to make the delete happen on the original interpreter?

Yep.  For one thing, GC can trigger __del__, which can do anything,
including modifying other objects from the original interpreter (incl.
decref'ing them).  __del__ should be run under the original
interpreter.  For another thing, during GC containers often decref
their items.  Also, separating the GIL between interpreters may mean
we'll need an allocator per interpreter.  In that case the
deallocation must happen relative to the interpreter where the object
was allocated.

-eric

From barry at barrys-emacs.org  Tue Jul 17 15:43:57 2018
From: barry at barrys-emacs.org (Barry)
Date: Tue, 17 Jul 2018 20:43:57 +0100
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <CALFfu7BFLSSRf7_mVUP3Rh4tAg1ae1cawc9KFF66EiBc2Hfn7g@mail.gmail.com>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>
 <CAPTjJmqKGhNEcrAuzKSzcUW_kvdmPqL9_9CMyOUzcrBs=UdcEg@mail.gmail.com>
 <CAPJVwBkpLw_qRirCFq4PK6n5zLVxc=t=_bf_SZRXLsSnrMVECQ@mail.gmail.com>
 <CAPTjJmrt83c96JQ3PY2oZEm68ZMbG_JBwaJ1XpcyzqRnEcks9A@mail.gmail.com>
 <6a5b228a-200a-b2c7-6067-df2b0022ae42@mrabarnett.plus.com>
 <20180716190735.2908e86d@fsol>
 <CALFfu7BFLSSRf7_mVUP3Rh4tAg1ae1cawc9KFF66EiBc2Hfn7g@mail.gmail.com>
Message-ID: <890D0D1D-7166-43B5-8BC0-8B2177B3BAE0@barrys-emacs.org>



> On 17 Jul 2018, at 20:35, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> 
>> On Mon, Jul 16, 2018 at 11:08 AM Antoine Pitrou <solipsis at pitrou.net> wrote:
>> On Mon, 16 Jul 2018 18:00:37 +0100
>> MRAB <python at mrabarnett.plus.com> wrote:
>>> Could you explicitly share an object in a similar way to how you
>>> explicitly open a file?
>>> 
>>> The shared object's refcount would be incremented and the sharing
>>> function would return a proxy to the shared object.
>>> 
>>> Refcounting in the thread/process would be done on the proxy.
>>> 
>>> When the proxy is closed or garbage-collected, the shared object's
>>> refcount would be decremented.
>>> 
>>> The shared object could be garbage-collected when its refcount drops to
>>> zero.
>> 
>> Yes, I'm assuming that would be how shareable buffers could be
>> implemented: a per-interpreter proxy (with a regular Python refcount)
>> mediating access to a shared object (which could have an atomic /
>> thread-safe refcount).
> 
> Nice! That's exactly how I'm doing it. :)  The buffer protocol makes
> it easier, but the idea could apply to arbitrary objects generally.
> That's something I'll look into in a later phase of the project.
> 
> In both cases the tricky part is ensuring that the proxy does not
> directly mutate the object (especially the refcount).  In fact, the
> decref part above is the trickiest.  The trickiness is a consequence
> of our goals.  In my multi-core project we're aiming for not sharing
> the GIL between interpreters.  That means reaching and keeping proper
> separation between interpreters.  Notably, without a GIL shared by
> interpreters, refcount operations are not thread-safe.  Also, in the
> decref case GC would happen under the wrong interpreter (which is
> problematic for several reasons).
> 
> With this in mind, here's how I'm approaching the problem:
> 
> 1. interp A "shares" an object with interp B (e.g. through a channel)
>    * the object is incref'ed under A before it is sent to B
> 2. the object is wrapped in a proxy owned by B
>    * the proxy may not make C-API calls that would mutate the object
> or even cause an incref/decref
> 3. when the proxy is GC'd, the original object is decref'ed
>    * the decref must happen in a thread in which A is running
> 
> In order to make all this work the missing piece is a mechanism by
> which the decref (#3) happens under the original interpreter.  At the
> moment Emily Morehouse and I are pursuing an approach that extends the
> existing ceval "pending call" machinery currently used for handling
> signals (see Py_AddPendingCall).  The new [*private*] API would work
> the same way but on a per-interpreter basis rather than just the main
> interpreter.  This would allow one interpreter to queue up a decref to
> happen later under another interpreter.

The decrement itself is not the problem, that can be made thread safe.

Do you mean that once the ref reaches 0 you have to make the delete happen on the original interpreter?

Barry

> 
> FWIW, this ability to decref an object under a different interpreter
> is a blocker right now for a number of things, including supporting
> buffers in PEP 554 channels.
> 
> -eric
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
> 


From greg.ewing at canterbury.ac.nz  Tue Jul 17 18:56:41 2018
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 18 Jul 2018 10:56:41 +1200
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <6a5b228a-200a-b2c7-6067-df2b0022ae42@mrabarnett.plus.com>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>
 <CAPTjJmqKGhNEcrAuzKSzcUW_kvdmPqL9_9CMyOUzcrBs=UdcEg@mail.gmail.com>
 <CAPJVwBkpLw_qRirCFq4PK6n5zLVxc=t=_bf_SZRXLsSnrMVECQ@mail.gmail.com>
 <CAPTjJmrt83c96JQ3PY2oZEm68ZMbG_JBwaJ1XpcyzqRnEcks9A@mail.gmail.com>
 <6a5b228a-200a-b2c7-6067-df2b0022ae42@mrabarnett.plus.com>
Message-ID: <5B4E7429.5080000@canterbury.ac.nz>

MRAB wrote:
> The shared object's refcount would be incremented and the sharing 
> function would return a proxy to the shared object.
> 
> Refcounting in the thread/process would be done on the proxy.
> 
> When the proxy is closed or garbage-collected, the shared object's 
> refcount would be decremented.

What about other objects accessed through the shared object?
They would need to get wrapped in proxies too.

Also, if the shared object is mutable, changes to it would
need to be protected by a lock of some kind.

Maybe all this could be taken care of by the proxy objects,
but it seems like it would be quite tricky to get right.

-- 
Greg

From trent at trent.me  Tue Jul 17 19:24:10 2018
From: trent at trent.me (Trent Nelson)
Date: Tue, 17 Jul 2018 19:24:10 -0400
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <c00bf3ef-ef42-9b43-247a-5adf9d4826d9@gmail.com>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <20180709161754.GA6167@trent.me>
 <c00bf3ef-ef42-9b43-247a-5adf9d4826d9@gmail.com>
Message-ID: <20180717232409.GA7352@trent.me>


    (Apologies for the slow reply, I'm in the middle of a relocation
     at the moment so e-mail access isn't consistent, and will be a
     lot worse over the next few weeks.)

On Tue, Jul 10, 2018 at 07:31:49AM -0700, David Foster wrote:
> I was not aware of PyParallel. The PyParellel "parallel thread"
> line-of-execution implementation is pretty interesting. Trent, big
> kudos to you on that effort.
> 
> Since you're speaking in the past tense and said "but we're not
> doing it like that", I infer that the notion of a parallel thread
> was turned down for integration into CPython, as that appears to
> have been the original goal.
> 
> However I am unable to locate a rationale for why that integration
> was turned down. Was it deemed to be too complex to execute, perhaps
> in the context of providing C extension compatibility? Was there a
> desire to see a similar implementation on Linux as well as Windows?
> Some other reason? Since I presume you were directly involved in the
> discussions, perhaps you have a link to the relevant thread handy?
> 
> The last update I see from you RE PyParallel on this list is:
> https://mail.python.org/pipermail/python-ideas/2015-September/035725.html


    PyParallel was... ambitious to say the least.  When I started it,
    I sort of *hand wavy* envisioned it would lead to something that
    I could formally pitch to python-dev at .  But there was a lot of
    blissful ignorance of the ensuing complexity in that initial
    sentiment, though.

    So, nothing was formally turned down by core developers, as I
    never really ended up pitching something formal that could be
    assessed for inclusion.  By the time I'd developed something
    that was at least an alpha-level proof-of-concept, I had to
    make 50+ pretty sizable implementation decisions that would
    have warranted their own PEP if the work ever made it into
    the mainline Python.

    I definitely think a PyParallel-esque approach (where we play
    it fast and loose with what's considered the GIL, how and when
    reference counting is done, etc.) is the only viable *performant*
    option we have for solving the problem -- i.e. I can't see how
    a "remove the GIL, introduce fine grained locking, use interlocked
    ops for ref counts"-type conventional approach will ever yield
    acceptable performance.

    But, yeah, I'm not optimistic we'll see a solution actually in
    the mainline Python any time soon.  I logged about 2500 hours
    of development time hacking PyParallel into it's initial alpha
    proof-of-concept state.  It only worked on one operating system,
    required intimate knowledge of Python innards (which I lacked at
    the start), and exposed a very brittle socket-server oriented
    interface to leverage the parallelism (there was no parallel
    compute/free-threading type support provided, really).

    I can't think of how we'll arrive at something production quality
    without it being a multi-year, many-developer (full time, ideally
    located in proximity to each other) project.  I think you'd really
    need a BDFL Guido/Linus/Cutler-type lead driving the whole effort
    too, as there will be a lot of tough, dividing decisions that need
    to be made.

    How would that be funded?!  It's almost a bit of a moon-shot type
    project.  Definitely high-risk.  There's no precedent for the PSF
    funding such projects, nor large corporate entities (i.e. Google,
    Amazon, Microsoft).  What's the ROI for those companies to take on
    so much cost and risk?  Perhaps if the end solution only ran on
    their cloud infrastructure (Azure, AWS, GCS) -- maybe at least
    initially.  That... that would be an interesting turn of events.

    Maybe we just wait 20 years 'til a NumPy/SciPy/Z3-stack does some
    cloud AI stuff to "solve" which parts of an existing program can
    be executed in parallel without any user/developer assistance :-)

> David Foster | Seattle, WA, USA

    Regards,

        Trent.
--
https://trent.me

From barry at barrys-emacs.org  Wed Jul 18 03:02:38 2018
From: barry at barrys-emacs.org (Barry)
Date: Wed, 18 Jul 2018 08:02:38 +0100
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <CALFfu7CC=2ZYFoYxf_ijU_V43F_g90RAtJZr=+nT5vgRwP4Q5g@mail.gmail.com>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>
 <CAPTjJmqKGhNEcrAuzKSzcUW_kvdmPqL9_9CMyOUzcrBs=UdcEg@mail.gmail.com>
 <CAPJVwBkpLw_qRirCFq4PK6n5zLVxc=t=_bf_SZRXLsSnrMVECQ@mail.gmail.com>
 <CAPTjJmrt83c96JQ3PY2oZEm68ZMbG_JBwaJ1XpcyzqRnEcks9A@mail.gmail.com>
 <6a5b228a-200a-b2c7-6067-df2b0022ae42@mrabarnett.plus.com>
 <20180716190735.2908e86d@fsol>
 <CALFfu7BFLSSRf7_mVUP3Rh4tAg1ae1cawc9KFF66EiBc2Hfn7g@mail.gmail.com>
 <890D0D1D-7166-43B5-8BC0-8B2177B3BAE0@barrys-emacs.org>
 <CALFfu7CC=2ZYFoYxf_ijU_V43F_g90RAtJZr=+nT5vgRwP4Q5g@mail.gmail.com>
Message-ID: <731EA367-F79E-45FD-939A-6DD8191C09F6@barrys-emacs.org>



>> On 17 Jul 2018, at 21:00, Eric Snow <ericsnowcurrently at gmail.com> wrote:
>> 
>> On Tue, Jul 17, 2018 at 1:44 PM Barry <barry at barrys-emacs.org> wrote:
>> The decrement itself is not the problem, that can be made thread safe.
> 
> Yeah, by using the GIL. <wink>  Otherwise, please elaborate.  My
> understanding is that if the decrement itself were not the problem
> then we'd have gotten rid of the GIL already.

All processors have thread safe ways to inc and dec and test, integers without holding a lock.

That is the mechanism that locks themselves are built out of. You can use that to avoid holding the GIL until the ref count reaches 0.

In c++ they built it into the language with std::atomic_int, you would have to find the way to do this C, i don?t have an answer at my finger tips for C.

Barry

> 
>> Do you mean that once the ref reaches 0 you have to make the delete happen on the original interpreter?
> 
> Yep.  For one thing, GC can trigger __del__, which can do anything,
> including modifying other objects from the original interpreter (incl.
> decref'ing them).  __del__ should be run under the original
> interpreter.  For another thing, during GC containers often decref
> their items.  Also, separating the GIL between interpreters may mean
> we'll need an allocator per interpreter.  In that case the
> deallocation must happen relative to the interpreter where the object
> was allocated.
> 
> -eric
> 


From ronaldoussoren at mac.com  Wed Jul 18 03:21:31 2018
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Wed, 18 Jul 2018 08:21:31 +0100
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <731EA367-F79E-45FD-939A-6DD8191C09F6@barrys-emacs.org>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>
 <CAPTjJmqKGhNEcrAuzKSzcUW_kvdmPqL9_9CMyOUzcrBs=UdcEg@mail.gmail.com>
 <CAPJVwBkpLw_qRirCFq4PK6n5zLVxc=t=_bf_SZRXLsSnrMVECQ@mail.gmail.com>
 <CAPTjJmrt83c96JQ3PY2oZEm68ZMbG_JBwaJ1XpcyzqRnEcks9A@mail.gmail.com>
 <6a5b228a-200a-b2c7-6067-df2b0022ae42@mrabarnett.plus.com>
 <20180716190735.2908e86d@fsol>
 <CALFfu7BFLSSRf7_mVUP3Rh4tAg1ae1cawc9KFF66EiBc2Hfn7g@mail.gmail.com>
 <890D0D1D-7166-43B5-8BC0-8B2177B3BAE0@barrys-emacs.org>
 <CALFfu7CC=2ZYFoYxf_ijU_V43F_g90RAtJZr=+nT5vgRwP4Q5g@mail.gmail.com>
 <731EA367-F79E-45FD-939A-6DD8191C09F6@barrys-emacs.org>
Message-ID: <E4C29EB7-B499-4739-82EA-737AAC0B275B@mac.com>

Op 18 jul. 2018 om 08:02 heeft Barry <barry at barrys-emacs.org> het volgende geschreven:

> 
> 
>>> On 17 Jul 2018, at 21:00, Eric Snow <ericsnowcurrently at gmail.com> wrote:
>>> 
>>> On Tue, Jul 17, 2018 at 1:44 PM Barry <barry at barrys-emacs.org> wrote:
>>> The decrement itself is not the problem, that can be made thread safe.
>> 
>> Yeah, by using the GIL. <wink>  Otherwise, please elaborate.  My
>> understanding is that if the decrement itself were not the problem
>> then we'd have gotten rid of the GIL already.
> 
> All processors have thread safe ways to inc and dec and test, integers without holding a lock.
> 
> That is the mechanism that locks themselves are built out of. You can use that to avoid holding the GIL until the ref count reaches 0.
> 
> In c++ they built it into the language with std::atomic_int, you would have to find the way to do this C, i don?t have an answer at my finger tips for C.
> 
 Some past attempts at getting rid of the GIL used atomic inc/dec, and that resulted in bad performance because these instructions  aren?t cheap. 

My gut feeling is that you?d have to get rid of refcounts to get high performance when getting rid of the GIL in a single interpreter, which would almost certainly result in breaking the C API.  

Ronald
> Barry
> 
>> 
>>> Do you mean that once the ref reaches 0 you have to make the delete happen on the original interpreter?
>> 
>> Yep.  For one thing, GC can trigger __del__, which can do anything,
>> including modifying other objects from the original interpreter (incl.
>> decref'ing them).  __del__ should be run under the original
>> interpreter.  For another thing, during GC containers often decref
>> their items.  Also, separating the GIL between interpreters may mean
>> we'll need an allocator per interpreter.  In that case the
>> deallocation must happen relative to the interpreter where the object
>> was allocated.
>> 
>> -eric
>> 
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From barry at barrys-emacs.org  Wed Jul 18 03:37:26 2018
From: barry at barrys-emacs.org (Barry Scott)
Date: Wed, 18 Jul 2018 08:37:26 +0100
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <CALFfu7CC=2ZYFoYxf_ijU_V43F_g90RAtJZr=+nT5vgRwP4Q5g@mail.gmail.com>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>
 <CAPTjJmqKGhNEcrAuzKSzcUW_kvdmPqL9_9CMyOUzcrBs=UdcEg@mail.gmail.com>
 <CAPJVwBkpLw_qRirCFq4PK6n5zLVxc=t=_bf_SZRXLsSnrMVECQ@mail.gmail.com>
 <CAPTjJmrt83c96JQ3PY2oZEm68ZMbG_JBwaJ1XpcyzqRnEcks9A@mail.gmail.com>
 <6a5b228a-200a-b2c7-6067-df2b0022ae42@mrabarnett.plus.com>
 <20180716190735.2908e86d@fsol>
 <CALFfu7BFLSSRf7_mVUP3Rh4tAg1ae1cawc9KFF66EiBc2Hfn7g@mail.gmail.com>
 <890D0D1D-7166-43B5-8BC0-8B2177B3BAE0@barrys-emacs.org>
 <CALFfu7CC=2ZYFoYxf_ijU_V43F_g90RAtJZr=+nT5vgRwP4Q5g@mail.gmail.com>
Message-ID: <9A7ACD82-CC6B-4C22-B3FC-B28963459046@barrys-emacs.org>



> On 17 Jul 2018, at 21:00, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> 
> On Tue, Jul 17, 2018 at 1:44 PM Barry <barry at barrys-emacs.org> wrote:
>> The decrement itself is not the problem, that can be made thread safe.
> 
> Yeah, by using the GIL. <wink>  Otherwise, please elaborate.  My
> understanding is that if the decrement itself were not the problem
> then we'd have gotten rid of the GIL already.

Let me try a longer answer. The inc+test and dec+test do not require a
lock if coded correctly. All OS and run times have solved this to provide
locks. All processors provide the instructions that are the building blocks
for lock primitives.

You cannot mutate a mutable python object that is not protected with the GIL as
the change of state involves multiple parts of the object changing.

If you know that an object is immutable then you could only do a check on the
ref count as you will never change the state of the object beyond its ref count.
To access the object you only have to ensure it will not be deleted, which the
ref count guarantees. The delete of the immutable object is then the only job
that the original interpreter must do.

> 
>> Do you mean that once the ref reaches 0 you have to make the delete happen on the original interpreter?
> 
> Yep.  For one thing, GC can trigger __del__, which can do anything,
> including modifying other objects from the original interpreter (incl.
> decref'ing them).  __del__ should be run under the original
> interpreter.  For another thing, during GC containers often decref
> their items.  Also, separating the GIL between interpreters may mean
> we'll need an allocator per interpreter.  In that case the
> deallocation must happen relative to the interpreter where the object
> was allocated.

Yep that I understand.

Barry

> 
> -eric
> 


From barry at barrys-emacs.org  Wed Jul 18 03:50:08 2018
From: barry at barrys-emacs.org (Barry Scott)
Date: Wed, 18 Jul 2018 08:50:08 +0100
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <E4C29EB7-B499-4739-82EA-737AAC0B275B@mac.com>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>
 <CAPTjJmqKGhNEcrAuzKSzcUW_kvdmPqL9_9CMyOUzcrBs=UdcEg@mail.gmail.com>
 <CAPJVwBkpLw_qRirCFq4PK6n5zLVxc=t=_bf_SZRXLsSnrMVECQ@mail.gmail.com>
 <CAPTjJmrt83c96JQ3PY2oZEm68ZMbG_JBwaJ1XpcyzqRnEcks9A@mail.gmail.com>
 <6a5b228a-200a-b2c7-6067-df2b0022ae42@mrabarnett.plus.com>
 <20180716190735.2908e86d@fsol>
 <CALFfu7BFLSSRf7_mVUP3Rh4tAg1ae1cawc9KFF66EiBc2Hfn7g@mail.gmail.com>
 <890D0D1D-7166-43B5-8BC0-8B2177B3BAE0@barrys-emacs.org>
 <CALFfu7CC=2ZYFoYxf_ijU_V43F_g90RAtJZr=+nT5vgRwP4Q5g@mail.gmail.com>
 <731EA367-F79E-45FD-939A-6DD8191C09F6@barrys-emacs.org>
 <E4C29EB7-B499-4739-82EA-737AAC0B275B@mac.com>
Message-ID: <A54F5EFE-543C-44D2-B7F9-F5F86E0F70E5@barrys-emacs.org>



> On 18 Jul 2018, at 08:21, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
> 
> Op 18 jul. 2018 om 08:02 heeft Barry <barry at barrys-emacs.org <mailto:barry at barrys-emacs.org>> het volgende geschreven:
> 
>> 
>> 
>>>> On 17 Jul 2018, at 21:00, Eric Snow <ericsnowcurrently at gmail.com> wrote:
>>>> 
>>>> On Tue, Jul 17, 2018 at 1:44 PM Barry <barry at barrys-emacs.org> wrote:
>>>> The decrement itself is not the problem, that can be made thread safe.
>>> 
>>> Yeah, by using the GIL. <wink>  Otherwise, please elaborate.  My
>>> understanding is that if the decrement itself were not the problem
>>> then we'd have gotten rid of the GIL already.
>> 
>> All processors have thread safe ways to inc and dec and test, integers without holding a lock.
>> 
>> That is the mechanism that locks themselves are built out of. You can use that to avoid holding the GIL until the ref count reaches 0.
>> 
>> In c++ they built it into the language with std::atomic_int, you would have to find the way to do this C, i don?t have an answer at my finger tips for C.
>> 
> Some past attempts at getting rid of the GIL used atomic inc/dec, and that resulted in bad performance because these instructions  aren?t cheap. 

Isn't this class of problem what leads to the per-processor caches and other optimisations in Linux kernel?
I wonder if kernel optimisations could be applied to this problem?

> 
> My gut feeling is that you?d have to get rid of refcounts to get high performance when getting rid of the GIL in a single interpreter, which would almost certainly result in breaking the C API.

Working on the ref count costs might be the enabling tech.

We already have the problem of unchanging objects being copied after a fork because of the ref counts being inside the object.
It was suggested that the ref count would have to move out of the object to help with this problem.

If there is a desirable solution to the parallel problem we can think about the C API migration problem.

Barry

>  
> 
> Ronald
>> Barry
>> 
>>> 
>>>> Do you mean that once the ref reaches 0 you have to make the delete happen on the original interpreter?
>>> 
>>> Yep.  For one thing, GC can trigger __del__, which can do anything,
>>> including modifying other objects from the original interpreter (incl.
>>> decref'ing them).  __del__ should be run under the original
>>> interpreter.  For another thing, during GC containers often decref
>>> their items.  Also, separating the GIL between interpreters may mean
>>> we'll need an allocator per interpreter.  In that case the
>>> deallocation must happen relative to the interpreter where the object
>>> was allocated.
>>> 
>>> -eric
>>> 
>> 
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org <mailto:Python-ideas at python.org>
>> https://mail.python.org/mailman/listinfo/python-ideas <https://mail.python.org/mailman/listinfo/python-ideas>
>> Code of Conduct: http://python.org/psf/codeofconduct/ <http://python.org/psf/codeofconduct/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180718/f7b350ef/attachment-0001.html>

From solipsis at pitrou.net  Wed Jul 18 05:30:56 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 18 Jul 2018 11:30:56 +0200
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>
 <CAPTjJmqKGhNEcrAuzKSzcUW_kvdmPqL9_9CMyOUzcrBs=UdcEg@mail.gmail.com>
 <CAPJVwBkpLw_qRirCFq4PK6n5zLVxc=t=_bf_SZRXLsSnrMVECQ@mail.gmail.com>
 <CAPTjJmrt83c96JQ3PY2oZEm68ZMbG_JBwaJ1XpcyzqRnEcks9A@mail.gmail.com>
 <6a5b228a-200a-b2c7-6067-df2b0022ae42@mrabarnett.plus.com>
 <20180716190735.2908e86d@fsol>
 <CALFfu7BFLSSRf7_mVUP3Rh4tAg1ae1cawc9KFF66EiBc2Hfn7g@mail.gmail.com>
 <890D0D1D-7166-43B5-8BC0-8B2177B3BAE0@barrys-emacs.org>
 <CALFfu7CC=2ZYFoYxf_ijU_V43F_g90RAtJZr=+nT5vgRwP4Q5g@mail.gmail.com>
 <731EA367-F79E-45FD-939A-6DD8191C09F6@barrys-emacs.org>
 <E4C29EB7-B499-4739-82EA-737AAC0B275B@mac.com>
Message-ID: <20180718113056.1845817f@fsol>

On Wed, 18 Jul 2018 08:21:31 +0100
Ronald Oussoren via Python-ideas
<python-ideas at python.org> wrote:
> Some past attempts at getting rid of the GIL used atomic inc/dec, and that resulted in bad performance because these instructions  aren?t cheap. 

Please read in context: we are not talking about making all refcounts
atomic, only a couple refcounts on shared objects (which probably
won't be Python objects, actually).

Regards

Antoine.



From desmoulinmichel at gmail.com  Wed Jul 18 09:35:54 2018
From: desmoulinmichel at gmail.com (Michel Desmoulin)
Date: Wed, 18 Jul 2018 15:35:54 +0200
Subject: [Python-ideas] grouping / dict of lists
In-Reply-To: <CADqi797gi6zgY+pTPZRjKUj2wCv=gbOVdGOjf71Qw4YLc5+kXw@mail.gmail.com>
References: <CAOmnVUA2Vc2jq63uWxms0OKd9hSNqVcqLcTOi-v=aCmB++DRyA@mail.gmail.com>
 <CADqi7940ZcW=j5MayJZYwmJct7aFYZAfV6zwVEgmxTAdVjCzpg@mail.gmail.com>
 <CADqi79521=Ny_seOpCB_wRXiJVX-N7kbeYv=1NZ_vZau=G0ywQ@mail.gmail.com>
 <CAP7+vJJ6vN9H8esj8tS+bSKsFELh8q8ywYHAW8Ti_upEJHf1vg@mail.gmail.com>
 <CADqi797HXzHcUnrNLp7r+pFEM5fcApiKHV4s=P5-D6mXHYYv-g@mail.gmail.com>
 <CAP7+vJ+3Hqk_cXyb2fynjpO0wiMs_EeayOMNYt9bf0y8dDXMSw@mail.gmail.com>
 <CALGmxELLvdpOniH8yohfZygx1Fb7nfT=ESeHrJF27++5dM9ohA@mail.gmail.com>
 <CADqi797gi6zgY+pTPZRjKUj2wCv=gbOVdGOjf71Qw4YLc5+kXw@mail.gmail.com>
Message-ID: <b7fffaf7-cc90-62a2-9eb5-bee6276dd3dd@gmail.com>

Counter also consider any missing key has the value "0".

With the constructor (accepting any iterable) and the most_common(n),
it's just a very set of features if you need to count anything.

Le 13/07/2018 ? 19:45, Michael Selik a ?crit?:
> On Mon, Jul 2, 2018 at 8:49 AM Chris Barker <chris.barker at noaa.gov
> <mailto:chris.barker at noaa.gov>> wrote:
> 
>     On Fri, Jun 29, 2018 at 11:25 PM, Guido van Rossum <guido at python.org
>     <mailto:guido at python.org>> wrote:
>     ?
> 
>         Hm, this actually feels heavier to me. But then again I never
>         liked or understood the need for Counter --
> 
> 
>     actually, me neither -- and partly because it's too lightweight --
>     that is, it's still a regular dict, and you pretty much have to know
>     that to use it. That it, it provides a nice counting constructor,
>     but after that, it's just a key:integer dict :-)
> 
> 
> Counter provides ``most_common`` which is often implemented
> inefficiently if written from scratch. People mistakenly use ``sorted``
> instead of ``heapq.nlargest``.
> 
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
> 

From ericsnowcurrently at gmail.com  Wed Jul 18 13:35:31 2018
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Wed, 18 Jul 2018 11:35:31 -0600
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <20180718113056.1845817f@fsol>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>
 <CAPTjJmqKGhNEcrAuzKSzcUW_kvdmPqL9_9CMyOUzcrBs=UdcEg@mail.gmail.com>
 <CAPJVwBkpLw_qRirCFq4PK6n5zLVxc=t=_bf_SZRXLsSnrMVECQ@mail.gmail.com>
 <CAPTjJmrt83c96JQ3PY2oZEm68ZMbG_JBwaJ1XpcyzqRnEcks9A@mail.gmail.com>
 <6a5b228a-200a-b2c7-6067-df2b0022ae42@mrabarnett.plus.com>
 <20180716190735.2908e86d@fsol>
 <CALFfu7BFLSSRf7_mVUP3Rh4tAg1ae1cawc9KFF66EiBc2Hfn7g@mail.gmail.com>
 <890D0D1D-7166-43B5-8BC0-8B2177B3BAE0@barrys-emacs.org>
 <CALFfu7CC=2ZYFoYxf_ijU_V43F_g90RAtJZr=+nT5vgRwP4Q5g@mail.gmail.com>
 <731EA367-F79E-45FD-939A-6DD8191C09F6@barrys-emacs.org>
 <E4C29EB7-B499-4739-82EA-737AAC0B275B@mac.com>
 <20180718113056.1845817f@fsol>
Message-ID: <CALFfu7Dmf9JwHngTTfPe1Jw7==YZ4JOV4Vza1TDNET-DfRKgaA@mail.gmail.com>

On Wed, Jul 18, 2018 at 3:31 AM Antoine Pitrou <solipsis at pitrou.net> wrote:
> Please read in context: we are not talking about making all refcounts
> atomic, only a couple refcounts on shared objects (which probably
> won't be Python objects, actually).

I have no plans to use refcounts for shared data (outside of Python objects).

-eric

From steve.dower at python.org  Wed Jul 18 13:43:36 2018
From: steve.dower at python.org (Steve Dower)
Date: Wed, 18 Jul 2018 10:43:36 -0700
Subject: [Python-ideas] PEP 505: None-aware operators
Message-ID: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>

Possibly this is exactly the wrong time to propose the next big syntax 
change, since we currently have nobody to declare on it, but since we're 
likely to argue for a while anyway it probably can't hurt (and maybe 
this will become the test PEP for whoever takes the reins?).

FWIW, Guido had previously indicated that he was generally favourable 
towards most of this proposal, provided we could figure out coherent 
semantics. Last time we tried, that didn't happen, so this time I've 
made the semantics much more precise, have implemented and verified 
them, and made much stronger statements about why we are proposing these.

Additional thanks to Mark Haase for writing most of the PEP. All the 
fair and balanced parts are his - all the overly strong opinions are mine.

Also thanks to Nick Coghlan for writing PEPs 531 and 532 last time we 
went through this - if you're unhappy with "None" being treated as a 
special kind of value, I recommend reading those before you start 
repeating them.

There is a formatted version of this PEP at 
https://www.python.org/dev/peps/pep-0505/

My current implementation is at 
https://github.com/zooba/cpython/tree/pep-505 (though I'm considering 
removing some of the new opcodes I added and just generating more 
complex code - in any case, let's get hung up on the proposal rather 
than the implementation :) )

Let the discussions begin!

---

PEP: 505
Title: None-aware operators
Version: $Revision$
Last-Modified: $Date$
Author: Mark E. Haase <mehaase at gmail.com>, Steve Dower 
<steve.dower at python.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 18-Sep-2015
Python-Version: 3.8

Abstract
========

Several modern programming languages have so-called "``null``-coalescing" or
"``null``- aware" operators, including C# [1]_, Dart [2]_, Perl, Swift, 
and PHP
(starting in version 7). These operators provide syntactic sugar for common
patterns involving null references.

* The "``null``-coalescing" operator is a binary operator that returns 
its left
   operand if it is not ``null``. Otherwise it returns its right operand.
* The "``null``-aware member access" operator accesses an instance 
member only
   if that instance is non-``null``. Otherwise it returns ``null``. 
(This is also
   called a "safe navigation" operator.)
* The "``null``-aware index access" operator accesses an element of a 
collection
   only if that collection is non-``null``. Otherwise it returns 
``null``. (This
   is another type of "safe navigation" operator.)

This PEP proposes three ``None``-aware operators for Python, based on the
definitions and other language's implementations of those above. 
Specifically:

* The "``None`` coalescing`` binary operator ``??`` returns the left 
hand side
   if it evaluates to a value that is not ``None``, or else it evaluates and
   returns the right hand side. A coalescing ``??=`` augmented assignment
   operator is included.
* The "``None``-aware attribute access" operator ``?.`` evaluates the 
complete
   expression if the left hand side evaluates to a value that is not 
``None``
* The "``None``-aware indexing" operator ``?[]`` evaluates the complete
   expression if the left hand site evaluates to a value that is not 
``None``

Syntax and Semantics
====================

Specialness of ``None``
-----------------------

The ``None`` object denotes the lack of a value. For the purposes of these
operators, the lack of a value indicates that the remainder of the 
expression
also lacks a value and should not be evaluated.

A rejected proposal was to treat any value that evaluates to false in a
Boolean context as not having a value. However, the purpose of these 
operators
is to propagate the "lack of value" state, rather that the "false" state.

Some argue that this makes ``None`` special. We contend that ``None`` is
already special, and that using it as both the test and the result of these
operators does not change the existing semantics in any way.

See the `Rejected Ideas`_ section for discussion on the rejected approaches.

Grammar changes
---------------

The following rules of the Python grammar are updated to read::

     augassign: ('+=' | '-=' | '*=' | '@=' | '/=' | '%=' | '&=' | '|=' | 
'^=' |
                 '<<=' | '>>=' | '**=' | '//=' | '??=')

     power: coalesce ['**' factor]
     coalesce: atom_expr ['??' factor]
     atom_expr: ['await'] atom trailer*
     trailer: ('(' [arglist] ')' |
               '[' subscriptlist ']' |
               '?[' subscriptlist ']' |
               '.' NAME |
               '?.' NAME)

Inserting the ``coalesce`` rule in this location ensures that expressions
resulting in ``None`` are natuarlly coalesced before they are used in
operations that would typically raise ``TypeError``. Like ``and`` and ``or``
the right-hand expression is not evaluated until the left-hand side is
determined to be ``None``. For example::

     a, b = None, None
     def c(): return None
     def ex(): raise Exception()

     (a ?? 2 ** b ?? 3) == a ?? (2 ** (b ?? 3))
     (a * b ?? c // d) == a * (b ?? c) // d
     (a ?? True and b ?? False) == (a ?? True) and (b ?? False)
     (c() ?? c() ?? True) == True
     (True ?? ex()) == True
     (c ?? ex)() == c()

Augmented coalescing assignment only rebinds the name if its current 
value is
``None``. If the target name already has a value, the right-hand side is not
evaluated. For example::

     a = None
     b = ''
     c = 0

     a ??= 'value'
     b ??= undefined_name
     c ??= shutil.rmtree('/')    # don't try this at home, kids

     assert a == 'value'
     assert b == ''
     assert c == '0' and any(os.scandir('/'))

Adding new trailers for the other ``None``-aware operators ensures that they
may be used in all valid locations for the existing equivalent operators,
including as part of an assignment target (more details below). As the 
existing
evaluation rules are not directly embedded in the grammar, we specify the
required changes here.

Assume that the ``atom`` is always successfully evaluated. Each 
``trailer`` is
then evaluated from left to right, applying its own parameter (either its
arguments, subscripts or attribute name) to produce the value for the next
``trailer``. Finally, if present, ``await`` is applied.

For example, ``await a.b(c).d[e]`` is currently parsed as
``['await', 'a', '.b', '(c)', '.d', '[e]']`` and evaluated::

     _v = a
     _v = _v.b
     _v = _v(c)
     _v = _v.d
     _v = _v[e]
     await _v

When a ``None``-aware operator is present, the left-to-right evaluation 
may be
short-circuited. For example, ``await a?.b(c).d?[e]`` is evaluated::

     _v = a
     if _v is not None:
         _v = _v.b
         _v = _v(c)
         _v = _v.d
         if _v is not None:
             _v = _v[e]
     await _v

.. note::
     ``await`` will almost certainly fail in this context, as it would in
     the case where code attempts ``await None``. We are not proposing 
to add a
     ``None``-aware ``await`` keyword here, and merely include it in this
     example for completeness of the specification, since the ``atom_expr``
     grammar rule includes the keyword. If it were in its own rule, we 
would have
     never mentioned it.

Parenthesised expressions are handled by the ``atom`` rule (not shown 
above),
which will implicitly terminate the short-circuiting behaviour of the above
transformation. For example, ``(a?.b ?? c).d?.e`` is evaluated as::

     # a?.b
     _v = a
     if _v is not None:
         _v = _v.b

     # ... ?? c
     if _v is None:
         _v = c

     # (...).d?.e
     _v = _v.d
     if _v is not None:
         _v = _v.e

When used as an assignment target, the ``None``-aware operations may only be
used in a "load" context. That is, ``a?.b = 1`` and ``a?[b] = 1`` will raise
``SyntaxError``. Use earlier in the expression (``a?.b.c = 1``) is 
permitted,
though unlikely to be useful unless combined with a coalescing operation::

     (a?.b ?? d).c = 1


Examples
========

This section presents some examples of common ``None`` patterns and 
shows what
conversion to use ``None``-aware operators may look like.

Standard Library
----------------

Using the ``find-pep505.py`` script[3]_ an analysis of the Python 3.7 
standard
library discovered up to 678 code snippets that could be replaced with 
use of
one of the ``None``-aware operators::

     $ find /usr/lib/python3.7 -name '*.py' | xargs python3.7 find-pep505.py
     <snip>
     Total None-coalescing `if` blocks: 449
     Total [possible] None-coalescing `or`: 120
     Total None-coalescing ternaries: 27
     Total Safe navigation `and`: 13
     Total Safe navigation `if` blocks: 61
     Total Safe navigation ternaries: 8

Some of these are shown below as examples before and after converting to 
use the
new operators.

 From ``bisect.py``::

     def insort_right(a, x, lo=0, hi=None):
         # ...
         if hi is None:
             hi = len(a)
         # ...

After updating to use the ``??=`` augmented assignment statement::

     def insort_right(a, x, lo=0, hi=None):
         # ...
         hi ??= len(a)
         # ...

 From ``calendar.py``::

     encoding = options.encoding
     if encoding is None:
         encoding = sys.getdefaultencoding()
     optdict = dict(encoding=encoding, css=options.css)

After updating to use the ``??`` operator::

     optdict = dict(encoding=encoding ?? sys.getdefaultencoding(),
                    css=options.css)

 From ``dis.py``::

     def _get_const_info(const_index, const_list):
         argval = const_index
         if const_list is not None:
             argval = const_list[const_index]
         return argval, repr(argval)

After updating to use the ``?[]`` and ``??`` operators::

     def _get_const_info(const_index, const_list):
         argval = const_list?[const_index] ?? const_index
         return argval, repr(argval)

 From ``inspect.py``::

     for base in object.__bases__:
         for name in getattr(base, "__abstractmethods__", ()):
             value = getattr(object, name, None)
             if getattr(value, "__isabstractmethod__", False):
                 return True

After updating to use the ``?.`` operator (and deliberately not 
converting to
use ``any()``)::

     for base in object.__bases__:
         for name in base?.__abstractmethods__ ?? ():
             if object?.name?.__isabstractmethod__:
                 return True

 From ``os.py``::

     if entry.is_dir():
         dirs.append(name)
         if entries is not None:
             entries.append(entry)
     else:
         nondirs.append(name)

After updating to use the ``?.`` operator::

     if entry.is_dir():
         dirs.append(name)
         entries?.append(entry)
     else:
         nondirs.append(name)


jsonify
-------

This example is from a Python web crawler that uses the Flask framework 
as its
front-end. This function retrieves information about a web site from a SQL
database and formats it as JSON to send to an HTTP client::

     class SiteView(FlaskView):
         @route('/site/<id_>', methods=['GET'])
         def get_site(self, id_):
             site = db.query('site_table').find(id_)

             return jsonify(
                 first_seen=site.first_seen.isoformat() if 
site.first_seen is not None else None,
                 id=site.id,
                 is_active=site.is_active,
                 last_seen=site.last_seen.isoformat() if site.last_seen 
is not None else None,
                 url=site.url.rstrip('/')
             )

Both ``first_seen`` and ``last_seen`` are allowed to be ``null`` in the
database, and they are also allowed to be ``null`` in the JSON response. 
JSON
does not have a native way to represent a ``datetime``, so the server's 
contract
states that any non-``null`` date is represented as an ISO-8601 string.

Without knowing the exact semantics of the ``first_seen`` and ``last_seen``
attributes, it is impossible to know whether the attribute can be safely or
performantly accessed multiple times.

One way to fix this code is to replace each conditional expression with an
explicit value assignment and a full ``if``/``else`` block::

     class SiteView(FlaskView):
         @route('/site/<id_>', methods=['GET'])
         def get_site(self, id_):
             site = db.query('site_table').find(id_)

             first_seen_dt = site.first_seen
             if first_seen_dt is None:
                 first_seen = None
             else:
                 first_seen = first_seen_dt.isoformat()

             last_seen_dt = site.last_seen
             if last_seen_dt is None:
                 last_seen = None
             else:
                 last_seen = last_seen_dt.isoformat()

             return jsonify(
                 first_seen=first_seen,
                 id=site.id,
                 is_active=site.is_active,
                 last_seen=last_seen,
                 url=site.url.rstrip('/')
             )

This adds ten lines of code and four new code paths to the function,
dramatically increasing the apparent complexity. Rewriting using the
``None``-aware attribute operator results in shorter code with more clear
intent::

     class SiteView(FlaskView):
         @route('/site/<id_>', methods=['GET'])
         def get_site(self, id_):
             site = db.query('site_table').find(id_)

             return jsonify(
                 first_seen=site.first_seen?.isoformat(),
                 id=site.id,
                 is_active=site.is_active,
                 last_seen=site.last_seen?.isoformat(),
                 url=site.url.rstrip('/')
             )

Grab
----

The next example is from a Python scraping library called `Grab
<https://github.com/lorien/grab/blob/4c95b18dcb0fa88eeca81f5643c0ebfb114bf728/gr
ab/upload.py>`_::

     class BaseUploadObject(object):
         def find_content_type(self, filename):
             ctype, encoding = mimetypes.guess_type(filename)
             if ctype is None:
                 return 'application/octet-stream'
             else:
                 return ctype

     class UploadContent(BaseUploadObject):
         def __init__(self, content, filename=None, content_type=None):
             self.content = content
             if filename is None:
                 self.filename = self.get_random_filename()
             else:
                 self.filename = filename
             if content_type is None:
                 self.content_type = self.find_content_type(self.filename)
             else:
                 self.content_type = content_type

     class UploadFile(BaseUploadObject):
         def __init__(self, path, filename=None, content_type=None):
             self.path = path
             if filename is None:
                 self.filename = os.path.split(path)[1]
             else:
                 self.filename = filename
             if content_type is None:
                 self.content_type = self.find_content_type(self.filename)
             else:
                 self.content_type = content_type

This example contains several good examples of needing to provide default
values. Rewriting to use conditional expressions reduces the overall 
lines of
code, but does not necessarily improve readability::

     class BaseUploadObject(object):
         def find_content_type(self, filename):
             ctype, encoding = mimetypes.guess_type(filename)
             return 'application/octet-stream' if ctype is None else ctype

     class UploadContent(BaseUploadObject):
         def __init__(self, content, filename=None, content_type=None):
             self.content = content
             self.filename = (self.get_random_filename() if filename
                 is None else filename)
             self.content_type = (self.find_content_type(self.filename)
                 if content_type is None else content_type)

     class UploadFile(BaseUploadObject):
         def __init__(self, path, filename=None, content_type=None):
             self.path = path
             self.filename = (os.path.split(path)[1] if filename is
                 None else filename)
             self.content_type = (self.find_content_type(self.filename)
                 if content_type is None else content_type)

The first ternary expression is tidy, but it reverses the intuitive order of
the operands: it should return ``ctype`` if it has a value and use the 
string
literal as fallback. The other ternary expressions are unintuitive and so
long that they must be wrapped. The overall readability is worsened, not
improved.

Rewriting using the ``None`` coalescing operator::

     class BaseUploadObject(object):
         def find_content_type(self, filename):
             ctype, encoding = mimetypes.guess_type(filename)
             return ctype ?? 'application/octet-stream'

     class UploadContent(BaseUploadObject):
         def __init__(self, content, filename=None, content_type=None):
             self.content = content
             self.filename = filename ?? self.get_random_filename()
             self.content_type = content_type ?? 
self.find_content_type(self.filename)

     class UploadFile(BaseUploadObject):
         def __init__(self, path, filename=None, content_type=None):
             self.path = path
             self.filename = filename ?? os.path.split(path)[1]
             self.content_type = content_type ?? 
self.find_content_type(self.filename)

This syntax has an intuitive ordering of the operands. In 
``find_content_type``,
for example, the preferred value ``ctype`` appears before the fallback 
value.
The terseness of the syntax also makes for fewer lines of code and less 
code to
visually parse, and reading from left-to-right and top-to-bottom more 
accurately
follows the execution flow.


Rejected Ideas
==============

The first three ideas in this section are oft-proposed alternatives to 
treating
``None`` as special. For further background on why these are rejected, 
see their
treatment in `PEP 531 <https://www.python.org/dev/peps/pep-0531/>`_ and
`PEP 532 <https://www.python.org/dev/peps/pep-0532/>`_ and the associated
discussions.

No-Value Protocol
-----------------

The operators could be generalised to user-defined types by defining a 
protocol
to indicate when a value represents "no value". Such a protocol may be a 
dunder
method ``__has_value__(self)` that returns ``True`` if the value should be
treated as having a value, and ``False`` if the value should be treated 
as no
value.

With this generalization, ``object`` would implement a dunder method 
equivalent
to this::

     def __has_value__(self):
         return True

``NoneType`` would implement a dunder method equivalent to this::

     def __has_value__(self):
         return False

In the specification section, all uses of ``x is None`` would be 
replaced with
``not x.__has_value__()``.

This generalization would allow for domain-specific "no-value" objects to be
coalesced just like ``None``. For example the ``pyasn1`` package has a type
called ``Null`` that represents an ASN.1 ``null``::

     >>> from pyasn1.type import univ
     >>> univ.Null() ?? univ.Integer(123)
     Integer(123)

Similarly, values such as ``math.nan`` and ``NotImplemented`` could be 
treated
as representing no value.

However, the "no-value" nature of these values is domain-specific, which 
means
they *should* be treated as a value by the language. For example,
``math.nan.imag`` is well defined (it's ``0.0``), and so short-circuiting
``math.nan?.imag`` to return ``math.nan`` would be incorrect.

As ``None`` is already defined by the language as being the value that
represents "no value", and the current specification would not preclude
switching to a protocol in the future (though changes to built-in 
objects would
not be compatible), this idea is rejected for now.

Boolean-aware operators
-----------------------

This suggestion is fundamentally the same as adding a no-value protocol, 
and so
the discussion above also applies.

Similar behavior to the ``??`` operator can be achieved with an ``or``
expression, however ``or`` checks whether its left operand is false-y 
and not
specifically ``None``. This approach is attractive, as it requires fewer 
changes
to the language, but ultimately does not solve the underlying problem 
correctly.

Assuming the check is for truthiness rather than ``None``, there is no 
longer a
need for the ``??`` operator. However, applying this check to the ``?.`` and
``?[]`` operators prevents perfectly valid operations applying

Consider the following example, where ``get_log_list()`` may return either a
list containing current log messages (potentially empty), or ``None`` if 
logging
is not enabled::

     lst = get_log_list()
     lst?.append('A log message')

If ``?.`` is checking for true values rather than specifically ``None`` 
and the
log has not been initialized with any items, no item will ever be 
appended. This
violates the obvious intent of the code, which is to append an item. The
``append`` method is available on an empty list, as are all other list 
methods,
and there is no reason to assume that these members should not be used 
because
the list is presently empty.

Further, there is no sensible result to use in place of the expression. A
normal ``lst.append`` returns ``None``, but under this idea 
``lst?.append`` may
result in either ``[]`` or ``None``, depending on the value of ``lst``. 
As with
the examples in the previous section, this makes no sense.

As checking for truthiness rather than ``None`` results in apparently valid
expressions no longer executing as intended, this idea is rejected.

Exception-aware operators
-------------------------

Arguably, the reason to short-circuit an expression when ``None`` is 
encountered
is to avoid the ``AttributeError`` or ``TypeError`` that would be raised 
under
normal circumstances. As an alternative to testing for ``None``, the 
``?.`` and
``?[]`` operators could instead handle ``AttributeError`` and ``TypeError``
raised by the operation and skip the remainder of the expression.

This produces a transformation for ``a?.b.c?.d.e`` similar to this::

     _v = a
     try:
         _v = _v.b
     except AttributeError:
         pass
     else:
         _v = _v.c
         try:
             _v = _v.d
         except AttributeError:
             pass
         else:
             _v = _v.e

One open question is which value should be returned as the expression 
when an
exception is handled. The above example simply leaves the partial 
result, but
this is not helpful for replacing with a default value. An alternative 
would be
to force the result to ``None``, which then raises the question as to why
``None`` is special enough to be the result but not special enough to be the
test.

Secondly, this approach masks errors within code executed implicitly as 
part of
the expression. For ``?.``, any ``AttributeError`` within a property or
``__getattr__`` implementation would be hidden, and similarly for 
``?[]`` and
``__getitem__`` implementations.

Similarly, simple typing errors such as ``{}?.ietms()`` could go unnoticed.

Existing conventions for handling these kinds of errors in the form of the
``getattr`` builtin and the ``.get(key, default)`` method pattern 
established by
``dict`` show that it is already possible to explicitly use this behaviour.

As this approach would hide errors in code, it is rejected.

``None``-aware Function Call
----------------------------

The ``None``-aware syntax applies to attribute and index access, so it seems
natural to ask if it should also apply to function invocation syntax. It 
might
be written as ``foo?()``, where ``foo`` is only called if it is not None.

This has been deferred on the basis of the proposed operators being intended
to aid traversal of partially populated hierarchical data structures, *not*
for traversal of arbitrary class hierarchies. This is reflected in the fact
that none of the other mainstream languages that already offer this syntax
have found it worthwhile to support a similar syntax for optional function
invocations.

A workaround similar to that used by C# would be to write
``maybe_none?.__call__(arguments)``. If the callable is ``None``, the
expression will not be evaluated. (The C# equivalent uses ``?.Invoke()`` 
on its
callable type.)

``?`` Unary Postfix Operator
----------------------------

To generalize the ``None``-aware behavior and limit the number of new 
operators
introduced, a unary, postfix operator spelled ``?`` was suggested. The 
idea is
that ``?`` might return a special object that could would override dunder
methods that return ``self``. For example, ``foo?`` would evaluate to 
``foo`` if
it is not ``None``, otherwise it would evaluate to an instance of
``NoneQuestion``::

     class NoneQuestion():
         def __call__(self, *args, **kwargs):
             return self

         def __getattr__(self, name):
             return self

         def __getitem__(self, key):
             return self


With this new operator and new type, an expression like ``foo?.bar[baz]``
evaluates to ``NoneQuestion`` if ``foo`` is None. This is a nifty
generalization, but it's difficult to use in practice since most 
existing code
won't know what ``NoneQuestion`` is.

Going back to one of the motivating examples above, consider the following::

     >>> import json
     >>> created = None
     >>> json.dumps({'created': created?.isoformat()})``

The JSON serializer does not know how to serialize ``NoneQuestion``, nor 
will
any other API. This proposal actually requires *lots of specialized logic*
throughout the standard library and any third party library.

At the same time, the ``?`` operator may also be **too general**, in the 
sense
that it can be combined with any other operator. What should the following
expressions mean?::

     >>> x? + 1
     >>> x? -= 1
     >>> x? == 1
     >>> ~x?

This degree of generalization is not useful. The operators actually proposed
herein are intentionally limited to a few operators that are expected to 
make it
easier to write common code patterns.

Built-in ``maybe``
------------------

Haskell has a concept called `Maybe <https://wiki.haskell.org/Maybe>`_ that
encapsulates the idea of an optional value without relying on any special
keyword (e.g. ``null``) or any special instance (e.g. ``None``). In 
Haskell, the
purpose of ``Maybe`` is to avoid separate handling of "something" and 
nothing".

A Python package called `pymaybe <https://pypi.org/p/pymaybe/>`_ provides a
rough approximation. The documentation shows the following example::

     >>> maybe('VALUE').lower()
     'value'

     >>> maybe(None).invalid().method().or_else('unknown')
     'unknown'

The function ``maybe()`` returns either a ``Something`` instance or a
``Nothing`` instance. Similar to the unary postfix operator described in the
previous section, ``Nothing`` overrides dunder methods in order to allow
chaining on a missing value.

Note that ``or_else()`` is eventually required to retrieve the 
underlying value
from ``pymaybe``'s wrappers. Furthermore, ``pymaybe`` does not short 
circuit any
evaluation. Although ``pymaybe`` has some strengths and may be useful in 
its own
right, it also demonstrates why a pure Python implementation of 
coalescing is
not nearly as powerful as support built into the language.

The idea of adding a builtin ``maybe`` type to enable this scenario is 
rejected.

Just use a conditional expression
---------------------------------

Another common way to initialize default values is to use the ternary 
operator.
Here is an excerpt from the popular `Requests package
<https://github.com/kennethreitz/requests/blob/14a555ac716866678bf17e43e23230d81
a8149f5/requests/models.py#L212>`_::

     data = [] if data is None else data
     files = [] if files is None else files
     headers = {} if headers is None else headers
     params = {} if params is None else params
     hooks = {} if hooks is None else hooks

This particular formulation has the undesirable effect of putting the 
operands
in an unintuitive order: the brain thinks, "use ``data`` if possible and use
``[]`` as a fallback," but the code puts the fallback *before* the preferred
value.

The author of this package could have written it like this instead::

     data = data if data is not None else []
     files = files if files is not None else []
     headers = headers if headers is not None else {}
     params = params if params is not None else {}
     hooks = hooks if hooks is not None else {}

This ordering of the operands is more intuitive, but it requires 4 extra
characters (for "not "). It also highlights the repetition of identifiers:
``data if data``, ``files if files``, etc.

When written using the ``None`` coalescing operator, the sample reads::

     data = data ?? []
     files = files ?? []
     headers = headers ?? {}
     params = params ?? {}
     hooks = hooks ?? {}


References
==========

.. [1] C# Reference: Operators
    (https://msdn.microsoft.com/en-us/library/6a71f45d.aspx)

.. [2] A Tour of the Dart Language: Operators
    (https://www.dartlang.org/docs/dart-up-and-running/ch02.html#operators)

.. [3] Associated scripts
    (https://github.com/python/peps/tree/master/pep-0505/)

Copyright
=========

This document has been placed in the public domain.

From ericsnowcurrently at gmail.com  Wed Jul 18 13:58:55 2018
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Wed, 18 Jul 2018 11:58:55 -0600
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <9A7ACD82-CC6B-4C22-B3FC-B28963459046@barrys-emacs.org>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>
 <CAPTjJmqKGhNEcrAuzKSzcUW_kvdmPqL9_9CMyOUzcrBs=UdcEg@mail.gmail.com>
 <CAPJVwBkpLw_qRirCFq4PK6n5zLVxc=t=_bf_SZRXLsSnrMVECQ@mail.gmail.com>
 <CAPTjJmrt83c96JQ3PY2oZEm68ZMbG_JBwaJ1XpcyzqRnEcks9A@mail.gmail.com>
 <6a5b228a-200a-b2c7-6067-df2b0022ae42@mrabarnett.plus.com>
 <20180716190735.2908e86d@fsol>
 <CALFfu7BFLSSRf7_mVUP3Rh4tAg1ae1cawc9KFF66EiBc2Hfn7g@mail.gmail.com>
 <890D0D1D-7166-43B5-8BC0-8B2177B3BAE0@barrys-emacs.org>
 <CALFfu7CC=2ZYFoYxf_ijU_V43F_g90RAtJZr=+nT5vgRwP4Q5g@mail.gmail.com>
 <9A7ACD82-CC6B-4C22-B3FC-B28963459046@barrys-emacs.org>
Message-ID: <CALFfu7BCH3Fmht5REXxk-NTW66ZMZhPoQQfCZDZGNyFn_+DLKQ@mail.gmail.com>

On Wed, Jul 18, 2018 at 1:37 AM Barry Scott <barry at barrys-emacs.org> wrote:
> Let me try a longer answer. The inc+test and dec+test do not require a
> lock if coded correctly. All OS and run times have solved this to provide
> locks. All processors provide the instructions that are the building blocks
> for lock primitives.
>
> You cannot mutate a mutable python object that is not protected with the GIL as
> the change of state involves multiple parts of the object changing.
>
> If you know that an object is immutable then you could only do a check on the
> ref count as you will never change the state of the object beyond its ref count.
> To access the object you only have to ensure it will not be deleted, which the
> ref count guarantees. The delete of the immutable object is then the only job
> that the original interpreter must do.

Perhaps we're agreeing?  Other than the single decref at when
"releasing" the object, it won't ever be directly modified (even the
refcount) in the other interpreter.  In effect that interpreter holds
a reference to the object which prevents GC in the "owning"
interpreter (the corresponding incref happened in that original
interpreter before the object was "shared").  The only issue is how to
"release" the object in the other interpreter so that the decref
happens in the "owning" interpreter.  As earlier noted, I'm planning
on taking advantage of the exiting ceval "pending calls" machinery.

So I'm not sure where an atomic int would factor in.  If you mean
switching the exiting refcount to an atomic int for the sake of the
cross-interpreter decref then that's not going to happen, as Ronald
suggested.  Larry could tell you about his Gilectomy experience. :)

Are you suggesting something like a second "cross-interpreter
refcount", which would be atomic, and add a check in Py_DECREF?  That
would imply an extra cross-interpreter-oriented C-API to parallel
Py_DECREF.  It would also mean either adding another field to PyObject
(yikes!) or keeping a separate table for tracking cross-interpreter
references.  I'm not sure any of that would be better than the
alternative I'm pursuing.  Then again, I've considered tracking which
interpreters hold a "reference" to an object, which isn't that
different.

-eric

From stephanh42 at gmail.com  Wed Jul 18 14:49:34 2018
From: stephanh42 at gmail.com (Stephan Houben)
Date: Wed, 18 Jul 2018 20:49:34 +0200
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <CALFfu7BCH3Fmht5REXxk-NTW66ZMZhPoQQfCZDZGNyFn_+DLKQ@mail.gmail.com>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>
 <CAPTjJmqKGhNEcrAuzKSzcUW_kvdmPqL9_9CMyOUzcrBs=UdcEg@mail.gmail.com>
 <CAPJVwBkpLw_qRirCFq4PK6n5zLVxc=t=_bf_SZRXLsSnrMVECQ@mail.gmail.com>
 <CAPTjJmrt83c96JQ3PY2oZEm68ZMbG_JBwaJ1XpcyzqRnEcks9A@mail.gmail.com>
 <6a5b228a-200a-b2c7-6067-df2b0022ae42@mrabarnett.plus.com>
 <20180716190735.2908e86d@fsol>
 <CALFfu7BFLSSRf7_mVUP3Rh4tAg1ae1cawc9KFF66EiBc2Hfn7g@mail.gmail.com>
 <890D0D1D-7166-43B5-8BC0-8B2177B3BAE0@barrys-emacs.org>
 <CALFfu7CC=2ZYFoYxf_ijU_V43F_g90RAtJZr=+nT5vgRwP4Q5g@mail.gmail.com>
 <9A7ACD82-CC6B-4C22-B3FC-B28963459046@barrys-emacs.org>
 <CALFfu7BCH3Fmht5REXxk-NTW66ZMZhPoQQfCZDZGNyFn_+DLKQ@mail.gmail.com>
Message-ID: <CAOOa=pNh_C=4Ysxf4sH6VS+fc+Wv4YO62QSU9xipPD4UV8ODyA@mail.gmail.com>

Hi Eric, Antoine, all

Antoine said that what I proposed earlier was very similar to what Eric
is trying to do, but from the direction the discussion has taken so far
that appears not to be the case.

I will therefore try to clarify my proposal.

Basically, what I am suggesting is a direct translation of Javascript's
Web Worker API (
https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API)
to Python.

The Web Worker API is generally considered a "share-nothing" approach,
although
as we will see some state can be shared.

The basic principle is that any object lives in a single Worker (Worker =
subinterpreter).
If a message is send from Worker A to Worker B, the message is not shared,
rather the so-called "structured clone" algorithm is used to create
recursively a NEW message
object in Worker B. This is roughly equivalent to pickling in A and then
unpickling in B,

Of course, this may become a bottleneck if large amounts of data need to be
communicated.
Therefore, there is a special object type designed to provide a view upon a
piece
of shared memory:  SharedArrayBuffer. Notable, this only provides a view
upon
raw "C"-style data (ints or floats or whatever), not on Javascript objects.

To translate this to the Python situation: each Python object is owned by a
single
subinterpreter, and may only be manipulated by a thread which holds the GIL
of that particular subinterpreter. Message sending between subinterpreters
will
require the message objects to be "structured cloned".

Certain C extension types may override what structured cloning means for
them.
In particular, some C extension types may have a two-layer structure where
the Py_Object contains a refcounted pointer to the actual data.
The structured cloning on such an object may create a second Py_Object which
references the same underlying object.
This secondary refcount will need to be properly atomic, since it may be
manipulated
from multiple subinterpreters.

In this way, interpreter-shared data structures can be implemented.
However, all the "normal" Python objects are not shared and can continue
to use the current, non-atomic refcounting implementation.

Hope this clarifies my proposal.

Stephan


2018-07-18 19:58 GMT+02:00 Eric Snow <ericsnowcurrently at gmail.com>:

> On Wed, Jul 18, 2018 at 1:37 AM Barry Scott <barry at barrys-emacs.org>
> wrote:
> > Let me try a longer answer. The inc+test and dec+test do not require a
> > lock if coded correctly. All OS and run times have solved this to provide
> > locks. All processors provide the instructions that are the building
> blocks
> > for lock primitives.
> >
> > You cannot mutate a mutable python object that is not protected with the
> GIL as
> > the change of state involves multiple parts of the object changing.
> >
> > If you know that an object is immutable then you could only do a check
> on the
> > ref count as you will never change the state of the object beyond its
> ref count.
> > To access the object you only have to ensure it will not be deleted,
> which the
> > ref count guarantees. The delete of the immutable object is then the
> only job
> > that the original interpreter must do.
>
> Perhaps we're agreeing?  Other than the single decref at when
> "releasing" the object, it won't ever be directly modified (even the
> refcount) in the other interpreter.  In effect that interpreter holds
> a reference to the object which prevents GC in the "owning"
> interpreter (the corresponding incref happened in that original
> interpreter before the object was "shared").  The only issue is how to
> "release" the object in the other interpreter so that the decref
> happens in the "owning" interpreter.  As earlier noted, I'm planning
> on taking advantage of the exiting ceval "pending calls" machinery.
>
> So I'm not sure where an atomic int would factor in.  If you mean
> switching the exiting refcount to an atomic int for the sake of the
> cross-interpreter decref then that's not going to happen, as Ronald
> suggested.  Larry could tell you about his Gilectomy experience. :)
>
> Are you suggesting something like a second "cross-interpreter
> refcount", which would be atomic, and add a check in Py_DECREF?  That
> would imply an extra cross-interpreter-oriented C-API to parallel
> Py_DECREF.  It would also mean either adding another field to PyObject
> (yikes!) or keeping a separate table for tracking cross-interpreter
> references.  I'm not sure any of that would be better than the
> alternative I'm pursuing.  Then again, I've considered tracking which
> interpreters hold a "reference" to an object, which isn't that
> different.
>
> -eric
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180718/4af66d10/attachment.html>

From jfine2358 at gmail.com  Wed Jul 18 14:52:47 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Wed, 18 Jul 2018 19:52:47 +0100
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <CAOOa=pNh_C=4Ysxf4sH6VS+fc+Wv4YO62QSU9xipPD4UV8ODyA@mail.gmail.com>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>
 <CAPTjJmqKGhNEcrAuzKSzcUW_kvdmPqL9_9CMyOUzcrBs=UdcEg@mail.gmail.com>
 <CAPJVwBkpLw_qRirCFq4PK6n5zLVxc=t=_bf_SZRXLsSnrMVECQ@mail.gmail.com>
 <CAPTjJmrt83c96JQ3PY2oZEm68ZMbG_JBwaJ1XpcyzqRnEcks9A@mail.gmail.com>
 <6a5b228a-200a-b2c7-6067-df2b0022ae42@mrabarnett.plus.com>
 <20180716190735.2908e86d@fsol>
 <CALFfu7BFLSSRf7_mVUP3Rh4tAg1ae1cawc9KFF66EiBc2Hfn7g@mail.gmail.com>
 <890D0D1D-7166-43B5-8BC0-8B2177B3BAE0@barrys-emacs.org>
 <CALFfu7CC=2ZYFoYxf_ijU_V43F_g90RAtJZr=+nT5vgRwP4Q5g@mail.gmail.com>
 <9A7ACD82-CC6B-4C22-B3FC-B28963459046@barrys-emacs.org>
 <CALFfu7BCH3Fmht5REXxk-NTW66ZMZhPoQQfCZDZGNyFn_+DLKQ@mail.gmail.com>
 <CAOOa=pNh_C=4Ysxf4sH6VS+fc+Wv4YO62QSU9xipPD4UV8ODyA@mail.gmail.com>
Message-ID: <CALD=Yf9LaHCMez4Ze37rQ0mUv2Qmdu0jXJZcTTXrkbhGmb76Fw@mail.gmail.com>

Hi

Python in the age of the multi-core processor is an important question. And
garbage collection is one of the many issues involved.

I've been thinking about the garbage collection problem, and lurking on
this list, for a while. I think it's now about time I showed myself, and
shared my thoughts. I intend to do this in a new thread, dealing only with
the problem of multi-core reference counting garbage collection. I hope you
don't mind my doing this. Expect the first instalment tomorrow.

with best regards

Jonathan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180718/f3af8aee/attachment-0001.html>

From ericsnowcurrently at gmail.com  Wed Jul 18 15:35:15 2018
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Wed, 18 Jul 2018 13:35:15 -0600
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <CAOOa=pNh_C=4Ysxf4sH6VS+fc+Wv4YO62QSU9xipPD4UV8ODyA@mail.gmail.com>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>
 <CAPTjJmqKGhNEcrAuzKSzcUW_kvdmPqL9_9CMyOUzcrBs=UdcEg@mail.gmail.com>
 <CAPJVwBkpLw_qRirCFq4PK6n5zLVxc=t=_bf_SZRXLsSnrMVECQ@mail.gmail.com>
 <CAPTjJmrt83c96JQ3PY2oZEm68ZMbG_JBwaJ1XpcyzqRnEcks9A@mail.gmail.com>
 <6a5b228a-200a-b2c7-6067-df2b0022ae42@mrabarnett.plus.com>
 <20180716190735.2908e86d@fsol>
 <CALFfu7BFLSSRf7_mVUP3Rh4tAg1ae1cawc9KFF66EiBc2Hfn7g@mail.gmail.com>
 <890D0D1D-7166-43B5-8BC0-8B2177B3BAE0@barrys-emacs.org>
 <CALFfu7CC=2ZYFoYxf_ijU_V43F_g90RAtJZr=+nT5vgRwP4Q5g@mail.gmail.com>
 <9A7ACD82-CC6B-4C22-B3FC-B28963459046@barrys-emacs.org>
 <CALFfu7BCH3Fmht5REXxk-NTW66ZMZhPoQQfCZDZGNyFn_+DLKQ@mail.gmail.com>
 <CAOOa=pNh_C=4Ysxf4sH6VS+fc+Wv4YO62QSU9xipPD4UV8ODyA@mail.gmail.com>
Message-ID: <CALFfu7BKtp4gkgRujL0uRZ1hwfxfNr+jTy_2qSP4YVvcLVOuZQ@mail.gmail.com>

On Wed, Jul 18, 2018 at 12:49 PM Stephan Houben <stephanh42 at gmail.com> wrote:
> Antoine said that what I proposed earlier was very similar to what Eric
> is trying to do, but from the direction the discussion has taken so far
> that appears not to be the case.

It looks like we are after the same thing actually. :)  Sorry for any confusion.


There are currently no provisions for actually sharing objects between
interpreters.  In fact, initially the plan is basically to support
sharing copies of basic builtin immuntable types.  The question of
refcounts comes in when we actually do share underlying data of
immutable objects (e.g. the buffer protocol).

> I will therefore try to clarify my proposal.
>
> Basically, what I am suggesting is a direct translation of Javascript's
> Web Worker API (https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API)
> to Python.
>
> The Web Worker API is generally considered a "share-nothing" approach, although
> as we will see some state can be shared.

Yes, there's a strong parallel to that model here.  In fact, I
mentioned web workers in my language summit talk at PyCon 2018.

> The basic principle is that any object lives in a single Worker (Worker = subinterpreter).
> If a message is send from Worker A to Worker B, the message is not shared,
> rather the so-called "structured clone" algorithm is used to create recursively a NEW message
> object in Worker B. This is roughly equivalent to pickling in A and then unpickling in B,

That is exactly what the channels in the PEP 554 implementation do,
though much more efficiently than pickling.  Initial support will be
for basic builtin immutable types.  We can later consider support for
other (even arbitrary?) types, but anything beyond copying (e.g.
pickle) is way off my radar.  Python's C-API is so closely tied to
refcounting that we simply cannot support safely sharing actual Python
objects between interpreters once we no longer share the GIL between
them.

> Of course, this may become a bottleneck if large amounts of data need to be communicated.
> Therefore, there is a special object type designed to provide a view upon a piece
> of shared memory:  SharedArrayBuffer. Notable, this only provides a view upon
> raw "C"-style data (ints or floats or whatever), not on Javascript objects.

Yep, that translates to buffers in Python, which is covered by PEP 554
(see SendChannel.send_buffer).

In this case, where some underlying data is actually shared, the
implementation has to deal with keeping a reference to the original
object and releasing it when done, which is what all the talk of
refcounts has been about.  However, the PEP does not talk about it
because it is an implementation detail that is not exposed in Python.

> To translate this to the Python situation: each Python object is owned by a single
> subinterpreter, and may only be manipulated by a thread which holds the GIL
> of that particular subinterpreter. Message sending between subinterpreters will
> require the message objects to be "structured cloned".

Correct.  That is what PEP 554 does.

As an aside, your phrasing "may only be manipulated by a thread which
holds the GIL of that particular subinterpreter" did spark something
I'll consider later:  perhaps interpreters can acquire each other's
GIL when (infrequently) necessary.  That could simplify a few things.

> Certain C extension types may override what structured cloning means for them.
> In particular, some C extension types may have a two-layer structure where
> the Py_Object contains a refcounted pointer to the actual data.
> The structured cloning on such an object may create a second Py_Object which
> references the same underlying object.
> This secondary refcount will need to be properly atomic, since it may be manipulated
> from multiple subinterpreters.

My implementation of PEP 554 supports this, though I have not made the
C-API for it public.  It's also not part of the PEP.  I was
considering adding it.

> In this way, interpreter-shared data structures can be implemented.
> However, all the "normal" Python objects are not shared and can continue
> to use the current, non-atomic refcounting implementation.

That is correct.  That entirely matches what I'm doing with PEP 554.
In fact, the isolation between interpreters is critical to my
multi-core Python project, of which PEP 554 is a part.  It's necessary
in order to stop sharing the GIL between interpreters.  So actual
objects will never be shared between interpreters.  They can't be.

> Hope this clarifies my proposal.

Yep.  Thanks!

-eric

From python at mrabarnett.plus.com  Wed Jul 18 16:18:29 2018
From: python at mrabarnett.plus.com (MRAB)
Date: Wed, 18 Jul 2018 21:18:29 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
Message-ID: <8767f4c3-34e9-2fff-c460-5bcae14994af@mrabarnett.plus.com>

On 2018-07-18 18:43, Steve Dower wrote:
> Possibly this is exactly the wrong time to propose the next big syntax
> change, since we currently have nobody to declare on it, but since we're
> likely to argue for a while anyway it probably can't hurt (and maybe
> this will become the test PEP for whoever takes the reins?).
> 
> FWIW, Guido had previously indicated that he was generally favourable
> towards most of this proposal, provided we could figure out coherent
> semantics. Last time we tried, that didn't happen, so this time I've
> made the semantics much more precise, have implemented and verified
> them, and made much stronger statements about why we are proposing these.
> 
> Additional thanks to Mark Haase for writing most of the PEP. All the
> fair and balanced parts are his - all the overly strong opinions are mine.
> 
> Also thanks to Nick Coghlan for writing PEPs 531 and 532 last time we
> went through this - if you're unhappy with "None" being treated as a
> special kind of value, I recommend reading those before you start
> repeating them.
> 
> There is a formatted version of this PEP at
> https://www.python.org/dev/peps/pep-0505/
> 
> My current implementation is at
> https://github.com/zooba/cpython/tree/pep-505 (though I'm considering
> removing some of the new opcodes I added and just generating more
> complex code - in any case, let's get hung up on the proposal rather
> than the implementation :) )
> 
> Let the discussions begin!
> 
[snip]
> 
> Grammar changes
> ---------------
> 
> The following rules of the Python grammar are updated to read::
> 
>       augassign: ('+=' | '-=' | '*=' | '@=' | '/=' | '%=' | '&=' | '|=' |
> '^=' |
>                   '<<=' | '>>=' | '**=' | '//=' | '??=')
> 
>       power: coalesce ['**' factor]
>       coalesce: atom_expr ['??' factor]
>       atom_expr: ['await'] atom trailer*
>       trailer: ('(' [arglist] ')' |
>                 '[' subscriptlist ']' |
>                 '?[' subscriptlist ']' |
>                 '.' NAME |
>                 '?.' NAME)
> 
The precedence is higher than I expected. I think of it more like 'or'. 
What is its precedence in the other languages?

> Inserting the ``coalesce`` rule in this location ensures that expressions
> resulting in ``None`` are natuarlly coalesced before they are used in

Typo "natuarlly".

> operations that would typically raise ``TypeError``. Like ``and`` and ``or``
> the right-hand expression is not evaluated until the left-hand side is
> determined to be ``None``. For example::
> 
>       a, b = None, None
>       def c(): return None
>       def ex(): raise Exception()
> 
>       (a ?? 2 ** b ?? 3) == a ?? (2 ** (b ?? 3))
>       (a * b ?? c // d) == a * (b ?? c) // d
>       (a ?? True and b ?? False) == (a ?? True) and (b ?? False)
>       (c() ?? c() ?? True) == True
>       (True ?? ex()) == True
>       (c ?? ex)() == c()
> 
> Augmented coalescing assignment only rebinds the name if its current
> value is
> ``None``. If the target name already has a value, the right-hand side is not
> evaluated. For example::
> 
>       a = None
>       b = ''
>       c = 0
> 
>       a ??= 'value'
>       b ??= undefined_name
>       c ??= shutil.rmtree('/')    # don't try this at home, kids
> 
>       assert a == 'value'
>       assert b == ''
>       assert c == '0' and any(os.scandir('/'))

Wouldn't the last assertion fail, because c == 0?

[snip]

From python at mrabarnett.plus.com  Wed Jul 18 16:37:51 2018
From: python at mrabarnett.plus.com (MRAB)
Date: Wed, 18 Jul 2018 21:37:51 +0100
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <CALFfu7BKtp4gkgRujL0uRZ1hwfxfNr+jTy_2qSP4YVvcLVOuZQ@mail.gmail.com>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>
 <CAPTjJmqKGhNEcrAuzKSzcUW_kvdmPqL9_9CMyOUzcrBs=UdcEg@mail.gmail.com>
 <CAPJVwBkpLw_qRirCFq4PK6n5zLVxc=t=_bf_SZRXLsSnrMVECQ@mail.gmail.com>
 <CAPTjJmrt83c96JQ3PY2oZEm68ZMbG_JBwaJ1XpcyzqRnEcks9A@mail.gmail.com>
 <6a5b228a-200a-b2c7-6067-df2b0022ae42@mrabarnett.plus.com>
 <20180716190735.2908e86d@fsol>
 <CALFfu7BFLSSRf7_mVUP3Rh4tAg1ae1cawc9KFF66EiBc2Hfn7g@mail.gmail.com>
 <890D0D1D-7166-43B5-8BC0-8B2177B3BAE0@barrys-emacs.org>
 <CALFfu7CC=2ZYFoYxf_ijU_V43F_g90RAtJZr=+nT5vgRwP4Q5g@mail.gmail.com>
 <9A7ACD82-CC6B-4C22-B3FC-B28963459046@barrys-emacs.org>
 <CALFfu7BCH3Fmht5REXxk-NTW66ZMZhPoQQfCZDZGNyFn_+DLKQ@mail.gmail.com>
 <CAOOa=pNh_C=4Ysxf4sH6VS+fc+Wv4YO62QSU9xipPD4UV8ODyA@mail.gmail.com>
 <CALFfu7BKtp4gkgRujL0uRZ1hwfxfNr+jTy_2qSP4YVvcLVOuZQ@mail.gmail.com>
Message-ID: <d68e6160-67ad-58cd-f50e-2bc24ef539e2@mrabarnett.plus.com>

On 2018-07-18 20:35, Eric Snow wrote:
> On Wed, Jul 18, 2018 at 12:49 PM Stephan Houben <stephanh42 at gmail.com> wrote:
>> Antoine said that what I proposed earlier was very similar to what Eric
>> is trying to do, but from the direction the discussion has taken so far
>> that appears not to be the case.
> 
> It looks like we are after the same thing actually. :)  Sorry for any confusion.
> 
> 
> There are currently no provisions for actually sharing objects between
> interpreters.  In fact, initially the plan is basically to support
> sharing copies of basic builtin immuntable types.  The question of
> refcounts comes in when we actually do share underlying data of
> immutable objects (e.g. the buffer protocol).
> 
What if an object is not going to be shared, but instead "moved" from 
one subinterpreter to another? The first subinterpreter would no longer 
have a reference to the object.

If the object's refcount is 1 and the object doesn't refer to any other 
object, then copying would not be necessary.

[snip]

From ericsnowcurrently at gmail.com  Wed Jul 18 16:51:38 2018
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Wed, 18 Jul 2018 14:51:38 -0600
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <d68e6160-67ad-58cd-f50e-2bc24ef539e2@mrabarnett.plus.com>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>
 <CAPTjJmqKGhNEcrAuzKSzcUW_kvdmPqL9_9CMyOUzcrBs=UdcEg@mail.gmail.com>
 <CAPJVwBkpLw_qRirCFq4PK6n5zLVxc=t=_bf_SZRXLsSnrMVECQ@mail.gmail.com>
 <CAPTjJmrt83c96JQ3PY2oZEm68ZMbG_JBwaJ1XpcyzqRnEcks9A@mail.gmail.com>
 <6a5b228a-200a-b2c7-6067-df2b0022ae42@mrabarnett.plus.com>
 <20180716190735.2908e86d@fsol>
 <CALFfu7BFLSSRf7_mVUP3Rh4tAg1ae1cawc9KFF66EiBc2Hfn7g@mail.gmail.com>
 <890D0D1D-7166-43B5-8BC0-8B2177B3BAE0@barrys-emacs.org>
 <CALFfu7CC=2ZYFoYxf_ijU_V43F_g90RAtJZr=+nT5vgRwP4Q5g@mail.gmail.com>
 <9A7ACD82-CC6B-4C22-B3FC-B28963459046@barrys-emacs.org>
 <CALFfu7BCH3Fmht5REXxk-NTW66ZMZhPoQQfCZDZGNyFn_+DLKQ@mail.gmail.com>
 <CAOOa=pNh_C=4Ysxf4sH6VS+fc+Wv4YO62QSU9xipPD4UV8ODyA@mail.gmail.com>
 <CALFfu7BKtp4gkgRujL0uRZ1hwfxfNr+jTy_2qSP4YVvcLVOuZQ@mail.gmail.com>
 <d68e6160-67ad-58cd-f50e-2bc24ef539e2@mrabarnett.plus.com>
Message-ID: <CALFfu7C6PLQz-UheKrcay4ESAzPBazLSO4XCa8ueP36ajekffQ@mail.gmail.com>

On Wed, Jul 18, 2018 at 2:38 PM MRAB <python at mrabarnett.plus.com> wrote:
> What if an object is not going to be shared, but instead "moved" from
> one subinterpreter to another? The first subinterpreter would no longer
> have a reference to the object.
>
> If the object's refcount is 1 and the object doesn't refer to any other
> object, then copying would not be necessary.

Yeah, that's something that I'm sure we'll investigate at some point,
but it's not part of the short-term plans.  This belongs to a whole
class of possibilities that we'll explore once we have the basic
functionality established. :)  FWIW, I don't think that "moving" an
object like this would be to hard to implement.

-eric

From steve.dower at python.org  Wed Jul 18 17:29:37 2018
From: steve.dower at python.org (Steve Dower)
Date: Wed, 18 Jul 2018 14:29:37 -0700
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <8767f4c3-34e9-2fff-c460-5bcae14994af@mrabarnett.plus.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <8767f4c3-34e9-2fff-c460-5bcae14994af@mrabarnett.plus.com>
Message-ID: <3d7941f4-9342-0ba0-dbd9-2a11d0e508a7@python.org>

Thanks! Bit of discussion below about precedence, but thanks for 
spotting the typos.

On 18Jul2018 1318, MRAB wrote:
> On 2018-07-18 18:43, Steve Dower wrote:
>> Grammar changes
>> ---------------
>>
>> The following rules of the Python grammar are updated to read::
>>
>> ????? augassign: ('+=' | '-=' | '*=' | '@=' | '/=' | '%=' | '&=' | '|=' |
>> '^=' |
>> ????????????????? '<<=' | '>>=' | '**=' | '//=' | '??=')
>>
>> ????? power: coalesce ['**' factor]
>> ????? coalesce: atom_expr ['??' factor]
>> ????? atom_expr: ['await'] atom trailer*
>> ????? trailer: ('(' [arglist] ')' |
>> ??????????????? '[' subscriptlist ']' |
>> ??????????????? '?[' subscriptlist ']' |
>> ??????????????? '.' NAME |
>> ??????????????? '?.' NAME)
>>
> The precedence is higher than I expected. I think of it more like 'or'. 
> What is its precedence in the other languages?

Yes, I expected this to be the contentious part. I may have to add a bit 
of discussion.

Mostly, I applied intuition rather than copying other languages on 
precedence (and if you could go through my non-git history, you'd see I 
tried four other places ;) ). The most "obvious" cases were these::

     a ?? 1 + b()

     b ** a() ?? 2

In the first case, both "(a ?? 1) + b()" and "a ?? (1 + b())" make 
sense, so it's really just my own personal preference that I think it 
looks like the first. If you flip the operands to get "b() + a ?? 1" 
then you end up with either "b() + (a ?? 1)" or "(b() + a) ?? 1", then 
it's more obvious that the latter doesn't make any sense (why would 
__add__ return None?), and so binding more tightly than "+" helps write 
sensible expressions with fewer parentheses.

Similarly, I feel like "b ** (a() ?? 2)" makes more sense than "(b ** 
a()) ?? 2", where for the latter we would have to assume a __pow__ 
implementation that returns None, or one that handles being passed None 
without raising a TypeError.

Contrasting this with "or", it is totally legitimate for arithmetic 
operators to return falsey values.

As I open the text file to correct the typos, I see this is what I tried 
to capture with:

> Inserting the ``coalesce`` rule in this location ensures that expressions
> resulting in ``None`` are naturally coalesced before they are used in
> operations that would typically raise ``TypeError``.

Take (2 ** a.b) ?? 0. The result of __pow__ is rarely going to be None, 
unless we train all the builtin types to do so (which, incidentally, I 
am not proposing and have no intention of proposing), whereas something 
like "2 ** coord?.exponent" attempting to call "2.__pow__(None)" seems 
comparatively likely. (Unfortunately, nobody writes code like this yet 
:) So there aren't any real-life examples. Originally I didn't include 
"??" in the proposal, but it became obvious in the examples that the 
presence of None-propagating operators ?. and ?[] just cause more pain 
without having the None-terminating operator ?? as well.)

>> Inserting the ``coalesce`` rule in this location ensures that expressions
>> resulting in ``None`` are natuarlly coalesced before they are used in
> 
> Typo "natuarlly".

Thanks.

>> ????? assert a == 'value'
>> ????? assert b == ''
>> ????? assert c == '0' and any(os.scandir('/'))
> 
> Wouldn't the last assertion fail, because c == 0?

Correct, another typo.

Cheers,
Steve

From njs at pobox.com  Wed Jul 18 19:33:20 2018
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 18 Jul 2018 16:33:20 -0700
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <CAOOa=pNh_C=4Ysxf4sH6VS+fc+Wv4YO62QSU9xipPD4UV8ODyA@mail.gmail.com>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>
 <CAPTjJmqKGhNEcrAuzKSzcUW_kvdmPqL9_9CMyOUzcrBs=UdcEg@mail.gmail.com>
 <CAPJVwBkpLw_qRirCFq4PK6n5zLVxc=t=_bf_SZRXLsSnrMVECQ@mail.gmail.com>
 <CAPTjJmrt83c96JQ3PY2oZEm68ZMbG_JBwaJ1XpcyzqRnEcks9A@mail.gmail.com>
 <6a5b228a-200a-b2c7-6067-df2b0022ae42@mrabarnett.plus.com>
 <20180716190735.2908e86d@fsol>
 <CALFfu7BFLSSRf7_mVUP3Rh4tAg1ae1cawc9KFF66EiBc2Hfn7g@mail.gmail.com>
 <890D0D1D-7166-43B5-8BC0-8B2177B3BAE0@barrys-emacs.org>
 <CALFfu7CC=2ZYFoYxf_ijU_V43F_g90RAtJZr=+nT5vgRwP4Q5g@mail.gmail.com>
 <9A7ACD82-CC6B-4C22-B3FC-B28963459046@barrys-emacs.org>
 <CALFfu7BCH3Fmht5REXxk-NTW66ZMZhPoQQfCZDZGNyFn_+DLKQ@mail.gmail.com>
 <CAOOa=pNh_C=4Ysxf4sH6VS+fc+Wv4YO62QSU9xipPD4UV8ODyA@mail.gmail.com>
Message-ID: <CAPJVwBk2RSF6=2cQtRHBF=XCPcaGDARNpXdAtw_rcvBendr+5w@mail.gmail.com>

On Wed, Jul 18, 2018 at 11:49 AM, Stephan Houben <stephanh42 at gmail.com> wrote:
> Basically, what I am suggesting is a direct translation of Javascript's
> Web Worker API
> (https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API)
> to Python.
>
> The Web Worker API is generally considered a "share-nothing" approach,
> although
> as we will see some state can be shared.
>
> The basic principle is that any object lives in a single Worker (Worker =
> subinterpreter).
> If a message is send from Worker A to Worker B, the message is not shared,
> rather the so-called "structured clone" algorithm is used to create
> recursively a NEW message
> object in Worker B. This is roughly equivalent to pickling in A and then
> unpickling in B,
>
> Of course, this may become a bottleneck if large amounts of data need to be
> communicated.
> Therefore, there is a special object type designed to provide a view upon a
> piece
> of shared memory:  SharedArrayBuffer. Notable, this only provides a view
> upon
> raw "C"-style data (ints or floats or whatever), not on Javascript objects.

Note that this everything you said here also exactly describes the
programming model for the existing 'multiprocessing' module:
"structured clone" is equivalent to how multiprocessing uses pickle to
transfer arbitrary objects, or you can use multiprocessing.Array to
get a shared view on raw "C"-style data.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From g.rodola at gmail.com  Wed Jul 18 19:55:19 2018
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Thu, 19 Jul 2018 01:55:19 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
Message-ID: <CAFYqXL8GsTyAHEkrQqSDpv_+bCcJAVv2j-0d_zwNuLLfK7Cuuw@mail.gmail.com>

On Wed, Jul 18, 2018 at 7:46 PM Steve Dower <steve.dower at python.org> wrote:
>
> Possibly this is exactly the wrong time to propose the next big syntax
> change, since we currently have nobody to declare on it, but since we're
> likely to argue for a while anyway it probably can't hurt (and maybe
> this will become the test PEP for whoever takes the reins?).
>
> FWIW, Guido had previously indicated that he was generally favourable
> towards most of this proposal, provided we could figure out coherent
> semantics. Last time we tried, that didn't happen, so this time I've
> made the semantics much more precise, have implemented and verified
> them, and made much stronger statements about why we are proposing these.
>
> Additional thanks to Mark Haase for writing most of the PEP. All the
> fair and balanced parts are his - all the overly strong opinions are mine.
>
> Also thanks to Nick Coghlan for writing PEPs 531 and 532 last time we
> went through this - if you're unhappy with "None" being treated as a
> special kind of value, I recommend reading those before you start
> repeating them.
>
> There is a formatted version of this PEP at
> https://www.python.org/dev/peps/pep-0505/
>
> My current implementation is at
> https://github.com/zooba/cpython/tree/pep-505 (though I'm considering
> removing some of the new opcodes I added and just generating more
> complex code - in any case, let's get hung up on the proposal rather
> than the implementation :) )
>
> Let the discussions begin!
>
> ---
>
> PEP: 505
> Title: None-aware operators
> Version: $Revision$
> Last-Modified: $Date$
> Author: Mark E. Haase <mehaase at gmail.com>, Steve Dower
> <steve.dower at python.org>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 18-Sep-2015
> Python-Version: 3.8
>
> Abstract
> ========
>
> Several modern programming languages have so-called "``null``-coalescing"
or
> "``null``- aware" operators, including C# [1]_, Dart [2]_, Perl, Swift,
> and PHP
> (starting in version 7). These operators provide syntactic sugar for
common
> patterns involving null references.
>
> * The "``null``-coalescing" operator is a binary operator that returns
> its left
>    operand if it is not ``null``. Otherwise it returns its right operand.
> * The "``null``-aware member access" operator accesses an instance
> member only
>    if that instance is non-``null``. Otherwise it returns ``null``.
> (This is also
>    called a "safe navigation" operator.)
> * The "``null``-aware index access" operator accesses an element of a
> collection
>    only if that collection is non-``null``. Otherwise it returns
> ``null``. (This
>    is another type of "safe navigation" operator.)
>
> This PEP proposes three ``None``-aware operators for Python, based on the
> definitions and other language's implementations of those above.
> Specifically:
>
> * The "``None`` coalescing`` binary operator ``??`` returns the left
> hand side
>    if it evaluates to a value that is not ``None``, or else it evaluates
and
>    returns the right hand side. A coalescing ``??=`` augmented assignment
>    operator is included.
> * The "``None``-aware attribute access" operator ``?.`` evaluates the
> complete
>    expression if the left hand side evaluates to a value that is not
> ``None``
> * The "``None``-aware indexing" operator ``?[]`` evaluates the complete
>    expression if the left hand site evaluates to a value that is not
> ``None``
>
> Syntax and Semantics
> ====================
>
> Specialness of ``None``
> -----------------------
>
> The ``None`` object denotes the lack of a value. For the purposes of these
> operators, the lack of a value indicates that the remainder of the
> expression
> also lacks a value and should not be evaluated.
>
> A rejected proposal was to treat any value that evaluates to false in a
> Boolean context as not having a value. However, the purpose of these
> operators
> is to propagate the "lack of value" state, rather that the "false" state.
>
> Some argue that this makes ``None`` special. We contend that ``None`` is
> already special, and that using it as both the test and the result of
these
> operators does not change the existing semantics in any way.
>
> See the `Rejected Ideas`_ section for discussion on the rejected
approaches.
>
> Grammar changes
> ---------------
>
> The following rules of the Python grammar are updated to read::
>
>      augassign: ('+=' | '-=' | '*=' | '@=' | '/=' | '%=' | '&=' | '|=' |
> '^=' |
>                  '<<=' | '>>=' | '**=' | '//=' | '??=')
>
>      power: coalesce ['**' factor]
>      coalesce: atom_expr ['??' factor]
>      atom_expr: ['await'] atom trailer*
>      trailer: ('(' [arglist] ')' |
>                '[' subscriptlist ']' |
>                '?[' subscriptlist ']' |
>                '.' NAME |
>                '?.' NAME)
>
> Inserting the ``coalesce`` rule in this location ensures that expressions
> resulting in ``None`` are natuarlly coalesced before they are used in
> operations that would typically raise ``TypeError``. Like ``and`` and
``or``
> the right-hand expression is not evaluated until the left-hand side is
> determined to be ``None``. For example::
>
>      a, b = None, None
>      def c(): return None
>      def ex(): raise Exception()
>
>      (a ?? 2 ** b ?? 3) == a ?? (2 ** (b ?? 3))
>      (a * b ?? c // d) == a * (b ?? c) // d
>      (a ?? True and b ?? False) == (a ?? True) and (b ?? False)
>      (c() ?? c() ?? True) == True
>      (True ?? ex()) == True
>      (c ?? ex)() == c()
>
> Augmented coalescing assignment only rebinds the name if its current
> value is
> ``None``. If the target name already has a value, the right-hand side is
not
> evaluated. For example::
>
>      a = None
>      b = ''
>      c = 0
>
>      a ??= 'value'
>      b ??= undefined_name
>      c ??= shutil.rmtree('/')    # don't try this at home, kids
>
>      assert a == 'value'
>      assert b == ''
>      assert c == '0' and any(os.scandir('/'))
>
> Adding new trailers for the other ``None``-aware operators ensures that
they
> may be used in all valid locations for the existing equivalent operators,
> including as part of an assignment target (more details below). As the
> existing
> evaluation rules are not directly embedded in the grammar, we specify the
> required changes here.
>
> Assume that the ``atom`` is always successfully evaluated. Each
> ``trailer`` is
> then evaluated from left to right, applying its own parameter (either its
> arguments, subscripts or attribute name) to produce the value for the next
> ``trailer``. Finally, if present, ``await`` is applied.
>
> For example, ``await a.b(c).d[e]`` is currently parsed as
> ``['await', 'a', '.b', '(c)', '.d', '[e]']`` and evaluated::
>
>      _v = a
>      _v = _v.b
>      _v = _v(c)
>      _v = _v.d
>      _v = _v[e]
>      await _v
>
> When a ``None``-aware operator is present, the left-to-right evaluation
> may be
> short-circuited. For example, ``await a?.b(c).d?[e]`` is evaluated::
>
>      _v = a
>      if _v is not None:
>          _v = _v.b
>          _v = _v(c)
>          _v = _v.d
>          if _v is not None:
>              _v = _v[e]
>      await _v
>
> .. note::
>      ``await`` will almost certainly fail in this context, as it would in
>      the case where code attempts ``await None``. We are not proposing
> to add a
>      ``None``-aware ``await`` keyword here, and merely include it in this
>      example for completeness of the specification, since the
``atom_expr``
>      grammar rule includes the keyword. If it were in its own rule, we
> would have
>      never mentioned it.
>
> Parenthesised expressions are handled by the ``atom`` rule (not shown
> above),
> which will implicitly terminate the short-circuiting behaviour of the
above
> transformation. For example, ``(a?.b ?? c).d?.e`` is evaluated as::
>
>      # a?.b
>      _v = a
>      if _v is not None:
>          _v = _v.b
>
>      # ... ?? c
>      if _v is None:
>          _v = c
>
>      # (...).d?.e
>      _v = _v.d
>      if _v is not None:
>          _v = _v.e
>
> When used as an assignment target, the ``None``-aware operations may only
be
> used in a "load" context. That is, ``a?.b = 1`` and ``a?[b] = 1`` will
raise
> ``SyntaxError``. Use earlier in the expression (``a?.b.c = 1``) is
> permitted,
> though unlikely to be useful unless combined with a coalescing operation::
>
>      (a?.b ?? d).c = 1
>
>
> Examples
> ========
>
> This section presents some examples of common ``None`` patterns and
> shows what
> conversion to use ``None``-aware operators may look like.
>
> Standard Library
> ----------------
>
> Using the ``find-pep505.py`` script[3]_ an analysis of the Python 3.7
> standard
> library discovered up to 678 code snippets that could be replaced with
> use of
> one of the ``None``-aware operators::
>
>      $ find /usr/lib/python3.7 -name '*.py' | xargs python3.7
find-pep505.py
>      <snip>
>      Total None-coalescing `if` blocks: 449
>      Total [possible] None-coalescing `or`: 120
>      Total None-coalescing ternaries: 27
>      Total Safe navigation `and`: 13
>      Total Safe navigation `if` blocks: 61
>      Total Safe navigation ternaries: 8
>
> Some of these are shown below as examples before and after converting to
> use the
> new operators.
>
>  From ``bisect.py``::
>
>      def insort_right(a, x, lo=0, hi=None):
>          # ...
>          if hi is None:
>              hi = len(a)
>          # ...
>
> After updating to use the ``??=`` augmented assignment statement::
>
>      def insort_right(a, x, lo=0, hi=None):
>          # ...
>          hi ??= len(a)
>          # ...
>
>  From ``calendar.py``::
>
>      encoding = options.encoding
>      if encoding is None:
>          encoding = sys.getdefaultencoding()
>      optdict = dict(encoding=encoding, css=options.css)
>
> After updating to use the ``??`` operator::
>
>      optdict = dict(encoding=encoding ?? sys.getdefaultencoding(),
>                     css=options.css)
>
>  From ``dis.py``::
>
>      def _get_const_info(const_index, const_list):
>          argval = const_index
>          if const_list is not None:
>              argval = const_list[const_index]
>          return argval, repr(argval)
>
> After updating to use the ``?[]`` and ``??`` operators::
>
>      def _get_const_info(const_index, const_list):
>          argval = const_list?[const_index] ?? const_index
>          return argval, repr(argval)
>
>  From ``inspect.py``::
>
>      for base in object.__bases__:
>          for name in getattr(base, "__abstractmethods__", ()):
>              value = getattr(object, name, None)
>              if getattr(value, "__isabstractmethod__", False):
>                  return True
>
> After updating to use the ``?.`` operator (and deliberately not
> converting to
> use ``any()``)::
>
>      for base in object.__bases__:
>          for name in base?.__abstractmethods__ ?? ():
>              if object?.name?.__isabstractmethod__:
>                  return True
>
>  From ``os.py``::
>
>      if entry.is_dir():
>          dirs.append(name)
>          if entries is not None:
>              entries.append(entry)
>      else:
>          nondirs.append(name)
>
> After updating to use the ``?.`` operator::
>
>      if entry.is_dir():
>          dirs.append(name)
>          entries?.append(entry)
>      else:
>          nondirs.append(name)
>
>
> jsonify
> -------
>
> This example is from a Python web crawler that uses the Flask framework
> as its
> front-end. This function retrieves information about a web site from a SQL
> database and formats it as JSON to send to an HTTP client::
>
>      class SiteView(FlaskView):
>          @route('/site/<id_>', methods=['GET'])
>          def get_site(self, id_):
>              site = db.query('site_table').find(id_)
>
>              return jsonify(
>                  first_seen=site.first_seen.isoformat() if
> site.first_seen is not None else None,
>                  id=site.id,
>                  is_active=site.is_active,
>                  last_seen=site.last_seen.isoformat() if site.last_seen
> is not None else None,
>                  url=site.url.rstrip('/')
>              )
>
> Both ``first_seen`` and ``last_seen`` are allowed to be ``null`` in the
> database, and they are also allowed to be ``null`` in the JSON response.
> JSON
> does not have a native way to represent a ``datetime``, so the server's
> contract
> states that any non-``null`` date is represented as an ISO-8601 string.
>
> Without knowing the exact semantics of the ``first_seen`` and
``last_seen``
> attributes, it is impossible to know whether the attribute can be safely
or
> performantly accessed multiple times.
>
> One way to fix this code is to replace each conditional expression with an
> explicit value assignment and a full ``if``/``else`` block::
>
>      class SiteView(FlaskView):
>          @route('/site/<id_>', methods=['GET'])
>          def get_site(self, id_):
>              site = db.query('site_table').find(id_)
>
>              first_seen_dt = site.first_seen
>              if first_seen_dt is None:
>                  first_seen = None
>              else:
>                  first_seen = first_seen_dt.isoformat()
>
>              last_seen_dt = site.last_seen
>              if last_seen_dt is None:
>                  last_seen = None
>              else:
>                  last_seen = last_seen_dt.isoformat()
>
>              return jsonify(
>                  first_seen=first_seen,
>                  id=site.id,
>                  is_active=site.is_active,
>                  last_seen=last_seen,
>                  url=site.url.rstrip('/')
>              )
>
> This adds ten lines of code and four new code paths to the function,
> dramatically increasing the apparent complexity. Rewriting using the
> ``None``-aware attribute operator results in shorter code with more clear
> intent::
>
>      class SiteView(FlaskView):
>          @route('/site/<id_>', methods=['GET'])
>          def get_site(self, id_):
>              site = db.query('site_table').find(id_)
>
>              return jsonify(
>                  first_seen=site.first_seen?.isoformat(),
>                  id=site.id,
>                  is_active=site.is_active,
>                  last_seen=site.last_seen?.isoformat(),
>                  url=site.url.rstrip('/')
>              )
>
> Grab
> ----
>
> The next example is from a Python scraping library called `Grab
> <
https://github.com/lorien/grab/blob/4c95b18dcb0fa88eeca81f5643c0ebfb114bf728/gr
> ab/upload.py>`_::
>
>      class BaseUploadObject(object):
>          def find_content_type(self, filename):
>              ctype, encoding = mimetypes.guess_type(filename)
>              if ctype is None:
>                  return 'application/octet-stream'
>              else:
>                  return ctype
>
>      class UploadContent(BaseUploadObject):
>          def __init__(self, content, filename=None, content_type=None):
>              self.content = content
>              if filename is None:
>                  self.filename = self.get_random_filename()
>              else:
>                  self.filename = filename
>              if content_type is None:
>                  self.content_type = self.find_content_type(self.filename)
>              else:
>                  self.content_type = content_type
>
>      class UploadFile(BaseUploadObject):
>          def __init__(self, path, filename=None, content_type=None):
>              self.path = path
>              if filename is None:
>                  self.filename = os.path.split(path)[1]
>              else:
>                  self.filename = filename
>              if content_type is None:
>                  self.content_type = self.find_content_type(self.filename)
>              else:
>                  self.content_type = content_type
>
> This example contains several good examples of needing to provide default
> values. Rewriting to use conditional expressions reduces the overall
> lines of
> code, but does not necessarily improve readability::
>
>      class BaseUploadObject(object):
>          def find_content_type(self, filename):
>              ctype, encoding = mimetypes.guess_type(filename)
>              return 'application/octet-stream' if ctype is None else ctype
>
>      class UploadContent(BaseUploadObject):
>          def __init__(self, content, filename=None, content_type=None):
>              self.content = content
>              self.filename = (self.get_random_filename() if filename
>                  is None else filename)
>              self.content_type = (self.find_content_type(self.filename)
>                  if content_type is None else content_type)
>
>      class UploadFile(BaseUploadObject):
>          def __init__(self, path, filename=None, content_type=None):
>              self.path = path
>              self.filename = (os.path.split(path)[1] if filename is
>                  None else filename)
>              self.content_type = (self.find_content_type(self.filename)
>                  if content_type is None else content_type)
>
> The first ternary expression is tidy, but it reverses the intuitive order
of
> the operands: it should return ``ctype`` if it has a value and use the
> string
> literal as fallback. The other ternary expressions are unintuitive and so
> long that they must be wrapped. The overall readability is worsened, not
> improved.
>
> Rewriting using the ``None`` coalescing operator::
>
>      class BaseUploadObject(object):
>          def find_content_type(self, filename):
>              ctype, encoding = mimetypes.guess_type(filename)
>              return ctype ?? 'application/octet-stream'
>
>      class UploadContent(BaseUploadObject):
>          def __init__(self, content, filename=None, content_type=None):
>              self.content = content
>              self.filename = filename ?? self.get_random_filename()
>              self.content_type = content_type ??
> self.find_content_type(self.filename)
>
>      class UploadFile(BaseUploadObject):
>          def __init__(self, path, filename=None, content_type=None):
>              self.path = path
>              self.filename = filename ?? os.path.split(path)[1]
>              self.content_type = content_type ??
> self.find_content_type(self.filename)
>
> This syntax has an intuitive ordering of the operands. In
> ``find_content_type``,
> for example, the preferred value ``ctype`` appears before the fallback
> value.
> The terseness of the syntax also makes for fewer lines of code and less
> code to
> visually parse, and reading from left-to-right and top-to-bottom more
> accurately
> follows the execution flow.
>
>
> Rejected Ideas
> ==============
>
> The first three ideas in this section are oft-proposed alternatives to
> treating
> ``None`` as special. For further background on why these are rejected,
> see their
> treatment in `PEP 531 <https://www.python.org/dev/peps/pep-0531/>`_ and
> `PEP 532 <https://www.python.org/dev/peps/pep-0532/>`_ and the associated
> discussions.
>
> No-Value Protocol
> -----------------
>
> The operators could be generalised to user-defined types by defining a
> protocol
> to indicate when a value represents "no value". Such a protocol may be a
> dunder
> method ``__has_value__(self)` that returns ``True`` if the value should be
> treated as having a value, and ``False`` if the value should be treated
> as no
> value.
>
> With this generalization, ``object`` would implement a dunder method
> equivalent
> to this::
>
>      def __has_value__(self):
>          return True
>
> ``NoneType`` would implement a dunder method equivalent to this::
>
>      def __has_value__(self):
>          return False
>
> In the specification section, all uses of ``x is None`` would be
> replaced with
> ``not x.__has_value__()``.
>
> This generalization would allow for domain-specific "no-value" objects to
be
> coalesced just like ``None``. For example the ``pyasn1`` package has a
type
> called ``Null`` that represents an ASN.1 ``null``::
>
>      >>> from pyasn1.type import univ
>      >>> univ.Null() ?? univ.Integer(123)
>      Integer(123)
>
> Similarly, values such as ``math.nan`` and ``NotImplemented`` could be
> treated
> as representing no value.
>
> However, the "no-value" nature of these values is domain-specific, which
> means
> they *should* be treated as a value by the language. For example,
> ``math.nan.imag`` is well defined (it's ``0.0``), and so short-circuiting
> ``math.nan?.imag`` to return ``math.nan`` would be incorrect.
>
> As ``None`` is already defined by the language as being the value that
> represents "no value", and the current specification would not preclude
> switching to a protocol in the future (though changes to built-in
> objects would
> not be compatible), this idea is rejected for now.
>
> Boolean-aware operators
> -----------------------
>
> This suggestion is fundamentally the same as adding a no-value protocol,
> and so
> the discussion above also applies.
>
> Similar behavior to the ``??`` operator can be achieved with an ``or``
> expression, however ``or`` checks whether its left operand is false-y
> and not
> specifically ``None``. This approach is attractive, as it requires fewer
> changes
> to the language, but ultimately does not solve the underlying problem
> correctly.
>
> Assuming the check is for truthiness rather than ``None``, there is no
> longer a
> need for the ``??`` operator. However, applying this check to the ``?.``
and
> ``?[]`` operators prevents perfectly valid operations applying
>
> Consider the following example, where ``get_log_list()`` may return
either a
> list containing current log messages (potentially empty), or ``None`` if
> logging
> is not enabled::
>
>      lst = get_log_list()
>      lst?.append('A log message')
>
> If ``?.`` is checking for true values rather than specifically ``None``
> and the
> log has not been initialized with any items, no item will ever be
> appended. This
> violates the obvious intent of the code, which is to append an item. The
> ``append`` method is available on an empty list, as are all other list
> methods,
> and there is no reason to assume that these members should not be used
> because
> the list is presently empty.
>
> Further, there is no sensible result to use in place of the expression. A
> normal ``lst.append`` returns ``None``, but under this idea
> ``lst?.append`` may
> result in either ``[]`` or ``None``, depending on the value of ``lst``.
> As with
> the examples in the previous section, this makes no sense.
>
> As checking for truthiness rather than ``None`` results in apparently
valid
> expressions no longer executing as intended, this idea is rejected.
>
> Exception-aware operators
> -------------------------
>
> Arguably, the reason to short-circuit an expression when ``None`` is
> encountered
> is to avoid the ``AttributeError`` or ``TypeError`` that would be raised
> under
> normal circumstances. As an alternative to testing for ``None``, the
> ``?.`` and
> ``?[]`` operators could instead handle ``AttributeError`` and
``TypeError``
> raised by the operation and skip the remainder of the expression.
>
> This produces a transformation for ``a?.b.c?.d.e`` similar to this::
>
>      _v = a
>      try:
>          _v = _v.b
>      except AttributeError:
>          pass
>      else:
>          _v = _v.c
>          try:
>              _v = _v.d
>          except AttributeError:
>              pass
>          else:
>              _v = _v.e
>
> One open question is which value should be returned as the expression
> when an
> exception is handled. The above example simply leaves the partial
> result, but
> this is not helpful for replacing with a default value. An alternative
> would be
> to force the result to ``None``, which then raises the question as to why
> ``None`` is special enough to be the result but not special enough to be
the
> test.
>
> Secondly, this approach masks errors within code executed implicitly as
> part of
> the expression. For ``?.``, any ``AttributeError`` within a property or
> ``__getattr__`` implementation would be hidden, and similarly for
> ``?[]`` and
> ``__getitem__`` implementations.
>
> Similarly, simple typing errors such as ``{}?.ietms()`` could go
unnoticed.
>
> Existing conventions for handling these kinds of errors in the form of the
> ``getattr`` builtin and the ``.get(key, default)`` method pattern
> established by
> ``dict`` show that it is already possible to explicitly use this
behaviour.
>
> As this approach would hide errors in code, it is rejected.
>
> ``None``-aware Function Call
> ----------------------------
>
> The ``None``-aware syntax applies to attribute and index access, so it
seems
> natural to ask if it should also apply to function invocation syntax. It
> might
> be written as ``foo?()``, where ``foo`` is only called if it is not None.
>
> This has been deferred on the basis of the proposed operators being
intended
> to aid traversal of partially populated hierarchical data structures,
*not*
> for traversal of arbitrary class hierarchies. This is reflected in the
fact
> that none of the other mainstream languages that already offer this syntax
> have found it worthwhile to support a similar syntax for optional function
> invocations.
>
> A workaround similar to that used by C# would be to write
> ``maybe_none?.__call__(arguments)``. If the callable is ``None``, the
> expression will not be evaluated. (The C# equivalent uses ``?.Invoke()``
> on its
> callable type.)
>
> ``?`` Unary Postfix Operator
> ----------------------------
>
> To generalize the ``None``-aware behavior and limit the number of new
> operators
> introduced, a unary, postfix operator spelled ``?`` was suggested. The
> idea is
> that ``?`` might return a special object that could would override dunder
> methods that return ``self``. For example, ``foo?`` would evaluate to
> ``foo`` if
> it is not ``None``, otherwise it would evaluate to an instance of
> ``NoneQuestion``::
>
>      class NoneQuestion():
>          def __call__(self, *args, **kwargs):
>              return self
>
>          def __getattr__(self, name):
>              return self
>
>          def __getitem__(self, key):
>              return self
>
>
> With this new operator and new type, an expression like ``foo?.bar[baz]``
> evaluates to ``NoneQuestion`` if ``foo`` is None. This is a nifty
> generalization, but it's difficult to use in practice since most
> existing code
> won't know what ``NoneQuestion`` is.
>
> Going back to one of the motivating examples above, consider the
following::
>
>      >>> import json
>      >>> created = None
>      >>> json.dumps({'created': created?.isoformat()})``
>
> The JSON serializer does not know how to serialize ``NoneQuestion``, nor
> will
> any other API. This proposal actually requires *lots of specialized logic*
> throughout the standard library and any third party library.
>
> At the same time, the ``?`` operator may also be **too general**, in the
> sense
> that it can be combined with any other operator. What should the following
> expressions mean?::
>
>      >>> x? + 1
>      >>> x? -= 1
>      >>> x? == 1
>      >>> ~x?
>
> This degree of generalization is not useful. The operators actually
proposed
> herein are intentionally limited to a few operators that are expected to
> make it
> easier to write common code patterns.
>
> Built-in ``maybe``
> ------------------
>
> Haskell has a concept called `Maybe <https://wiki.haskell.org/Maybe>`_
that
> encapsulates the idea of an optional value without relying on any special
> keyword (e.g. ``null``) or any special instance (e.g. ``None``). In
> Haskell, the
> purpose of ``Maybe`` is to avoid separate handling of "something" and
> nothing".
>
> A Python package called `pymaybe <https://pypi.org/p/pymaybe/>`_ provides
a
> rough approximation. The documentation shows the following example::
>
>      >>> maybe('VALUE').lower()
>      'value'
>
>      >>> maybe(None).invalid().method().or_else('unknown')
>      'unknown'
>
> The function ``maybe()`` returns either a ``Something`` instance or a
> ``Nothing`` instance. Similar to the unary postfix operator described in
the
> previous section, ``Nothing`` overrides dunder methods in order to allow
> chaining on a missing value.
>
> Note that ``or_else()`` is eventually required to retrieve the
> underlying value
> from ``pymaybe``'s wrappers. Furthermore, ``pymaybe`` does not short
> circuit any
> evaluation. Although ``pymaybe`` has some strengths and may be useful in
> its own
> right, it also demonstrates why a pure Python implementation of
> coalescing is
> not nearly as powerful as support built into the language.
>
> The idea of adding a builtin ``maybe`` type to enable this scenario is
> rejected.
>
> Just use a conditional expression
> ---------------------------------
>
> Another common way to initialize default values is to use the ternary
> operator.
> Here is an excerpt from the popular `Requests package
> <
https://github.com/kennethreitz/requests/blob/14a555ac716866678bf17e43e23230d81
> a8149f5/requests/models.py#L212>`_::
>
>      data = [] if data is None else data
>      files = [] if files is None else files
>      headers = {} if headers is None else headers
>      params = {} if params is None else params
>      hooks = {} if hooks is None else hooks
>
> This particular formulation has the undesirable effect of putting the
> operands
> in an unintuitive order: the brain thinks, "use ``data`` if possible and
use
> ``[]`` as a fallback," but the code puts the fallback *before* the
preferred
> value.
>
> The author of this package could have written it like this instead::
>
>      data = data if data is not None else []
>      files = files if files is not None else []
>      headers = headers if headers is not None else {}
>      params = params if params is not None else {}
>      hooks = hooks if hooks is not None else {}
>
> This ordering of the operands is more intuitive, but it requires 4 extra
> characters (for "not "). It also highlights the repetition of identifiers:
> ``data if data``, ``files if files``, etc.
>
> When written using the ``None`` coalescing operator, the sample reads::
>
>      data = data ?? []
>      files = files ?? []
>      headers = headers ?? {}
>      params = params ?? {}
>      hooks = hooks ?? {}
>
>
> References
> ==========
>
> .. [1] C# Reference: Operators
>     (https://msdn.microsoft.com/en-us/library/6a71f45d.aspx)
>
> .. [2] A Tour of the Dart Language: Operators
>     (https://www.dartlang.org/docs/dart-up-and-running/ch02.html#operators
)
>
> .. [3] Associated scripts
>     (https://github.com/python/peps/tree/master/pep-0505/)
>
> Copyright
> =========
>
> This document has been placed in the public domain.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

With all due respect (and I am sorry for being ?vocal? about a PEP once
again) I find this simply ugly. To me this basically doesn?t look like
python anymore, so a strong -1 from me.

On a more general note, I think Python is evolving too quickly. If I try to
envision a python code which uses all the syntax which was added in the
last 2-3 releases (say function annotations, assignment expressions, type
hints and now this) the picture I get is something which hardly resembles
the good old python I am used to. I don?t think we can have a clear sense
of how all these syntax changes put together will impact the future python
ecosystem when people will take them for granted and massively use them,
since as of now they are largely untested. With python 2 still being the
most used version I think we should go the opposite direction like we did
with u?? literals, possibly freeze the syntax and provide more palatable
?carrots? as an incentive for the migration (speed and stdlib improvements
come to mind). Keep adding new syntax will just keep the two versions even
more far apart, make the language harder to learn and harder to read
because the same thing can be expressed by using too many different styles.

My 2 cents

-- 
Giampaolo - http://grodola.blogspot.com
-- 
Giampaolo - http://grodola.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/3d101146/attachment-0001.html>

From rosuav at gmail.com  Wed Jul 18 20:05:36 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 19 Jul 2018 10:05:36 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAFYqXL8GsTyAHEkrQqSDpv_+bCcJAVv2j-0d_zwNuLLfK7Cuuw@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAFYqXL8GsTyAHEkrQqSDpv_+bCcJAVv2j-0d_zwNuLLfK7Cuuw@mail.gmail.com>
Message-ID: <CAPTjJmqPS_wiFTJaJ8YUKJfWsNcCX=h9AQ1EmWFv6Xf9Wes4tA@mail.gmail.com>

On Thu, Jul 19, 2018 at 9:55 AM, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
 [ please trim quotes, you just quoted the entire PEP in your post ]
> With all due respect (and I am sorry for being ?vocal? about a PEP once
> again) I find this simply ugly. To me this basically doesn?t look like
> python anymore, so a strong -1 from me.

I'd love to hear an explanation of WHY this doesn't look like Python
any more. For instance, is the + operator somehow wrong for Python,
and it should have been the word "add"? People complain about
proposals with words like these, but the best explanation I've ever
had is "well, Python uses words and this proposal uses punctuation".

Personally, I'm +0 on this. It'd be a few small wins here and there,
nothing huge, and I could easily live without it; but it's something
that I know some people will love.

ChrisA

From mertz at gnosis.cx  Wed Jul 18 20:05:56 2018
From: mertz at gnosis.cx (David Mertz)
Date: Wed, 18 Jul 2018 20:05:56 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
Message-ID: <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>

As before, I'm a strong -1 on this.

I recognize the semantic utility (albeit, I've needed it much less than
some folks testify). But the obscure characters are accumulating far too
fast with this.

Even with PEP 572, I would have preferred 'as' and restrictions on where
it's allowed, but ':=' is more familiar from numerous languages and
pseudo-code notations.  '??' and '?.' and ?[]' are really just marching
into APL and Perl territory. Yes, I know such operators exist in other
languages, but it feels very unpythonic.

On Wed, Jul 18, 2018, 1:46 PM Steve Dower <steve.dower at python.org> wrote:

> Possibly this is exactly the wrong time to propose the next big syntax
> change, since we currently have nobody to declare on it, but since we're
> likely to argue for a while anyway it probably can't hurt (and maybe
> this will become the test PEP for whoever takes the reins?).
>
> FWIW, Guido had previously indicated that he was generally favourable
> towards most of this proposal, provided we could figure out coherent
> semantics. Last time we tried, that didn't happen, so this time I've
> made the semantics much more precise, have implemented and verified
> them, and made much stronger statements about why we are proposing these.
>
> Additional thanks to Mark Haase for writing most of the PEP. All the
> fair and balanced parts are his - all the overly strong opinions are mine.
>
> Also thanks to Nick Coghlan for writing PEPs 531 and 532 last time we
> went through this - if you're unhappy with "None" being treated as a
> special kind of value, I recommend reading those before you start
> repeating them.
>
> There is a formatted version of this PEP at
> https://www.python.org/dev/peps/pep-0505/
>
> My current implementation is at
> https://github.com/zooba/cpython/tree/pep-505 (though I'm considering
> removing some of the new opcodes I added and just generating more
> complex code - in any case, let's get hung up on the proposal rather
> than the implementation :) )
>
> Let the discussions begin!
>
> ---
>
> PEP: 505
> Title: None-aware operators
> Version: $Revision$
> Last-Modified: $Date$
> Author: Mark E. Haase <mehaase at gmail.com>, Steve Dower
> <steve.dower at python.org>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 18-Sep-2015
> Python-Version: 3.8
>
> Abstract
> ========
>
> Several modern programming languages have so-called "``null``-coalescing"
> or
> "``null``- aware" operators, including C# [1]_, Dart [2]_, Perl, Swift,
> and PHP
> (starting in version 7). These operators provide syntactic sugar for common
> patterns involving null references.
>
> * The "``null``-coalescing" operator is a binary operator that returns
> its left
>    operand if it is not ``null``. Otherwise it returns its right operand.
> * The "``null``-aware member access" operator accesses an instance
> member only
>    if that instance is non-``null``. Otherwise it returns ``null``.
> (This is also
>    called a "safe navigation" operator.)
> * The "``null``-aware index access" operator accesses an element of a
> collection
>    only if that collection is non-``null``. Otherwise it returns
> ``null``. (This
>    is another type of "safe navigation" operator.)
>
> This PEP proposes three ``None``-aware operators for Python, based on the
> definitions and other language's implementations of those above.
> Specifically:
>
> * The "``None`` coalescing`` binary operator ``??`` returns the left
> hand side
>    if it evaluates to a value that is not ``None``, or else it evaluates
> and
>    returns the right hand side. A coalescing ``??=`` augmented assignment
>    operator is included.
> * The "``None``-aware attribute access" operator ``?.`` evaluates the
> complete
>    expression if the left hand side evaluates to a value that is not
> ``None``
> * The "``None``-aware indexing" operator ``?[]`` evaluates the complete
>    expression if the left hand site evaluates to a value that is not
> ``None``
>
> Syntax and Semantics
> ====================
>
> Specialness of ``None``
> -----------------------
>
> The ``None`` object denotes the lack of a value. For the purposes of these
> operators, the lack of a value indicates that the remainder of the
> expression
> also lacks a value and should not be evaluated.
>
> A rejected proposal was to treat any value that evaluates to false in a
> Boolean context as not having a value. However, the purpose of these
> operators
> is to propagate the "lack of value" state, rather that the "false" state.
>
> Some argue that this makes ``None`` special. We contend that ``None`` is
> already special, and that using it as both the test and the result of these
> operators does not change the existing semantics in any way.
>
> See the `Rejected Ideas`_ section for discussion on the rejected
> approaches.
>
> Grammar changes
> ---------------
>
> The following rules of the Python grammar are updated to read::
>
>      augassign: ('+=' | '-=' | '*=' | '@=' | '/=' | '%=' | '&=' | '|=' |
> '^=' |
>                  '<<=' | '>>=' | '**=' | '//=' | '??=')
>
>      power: coalesce ['**' factor]
>      coalesce: atom_expr ['??' factor]
>      atom_expr: ['await'] atom trailer*
>      trailer: ('(' [arglist] ')' |
>                '[' subscriptlist ']' |
>                '?[' subscriptlist ']' |
>                '.' NAME |
>                '?.' NAME)
>
> Inserting the ``coalesce`` rule in this location ensures that expressions
> resulting in ``None`` are natuarlly coalesced before they are used in
> operations that would typically raise ``TypeError``. Like ``and`` and
> ``or``
> the right-hand expression is not evaluated until the left-hand side is
> determined to be ``None``. For example::
>
>      a, b = None, None
>      def c(): return None
>      def ex(): raise Exception()
>
>      (a ?? 2 ** b ?? 3) == a ?? (2 ** (b ?? 3))
>      (a * b ?? c // d) == a * (b ?? c) // d
>      (a ?? True and b ?? False) == (a ?? True) and (b ?? False)
>      (c() ?? c() ?? True) == True
>      (True ?? ex()) == True
>      (c ?? ex)() == c()
>
> Augmented coalescing assignment only rebinds the name if its current
> value is
> ``None``. If the target name already has a value, the right-hand side is
> not
> evaluated. For example::
>
>      a = None
>      b = ''
>      c = 0
>
>      a ??= 'value'
>      b ??= undefined_name
>      c ??= shutil.rmtree('/')    # don't try this at home, kids
>
>      assert a == 'value'
>      assert b == ''
>      assert c == '0' and any(os.scandir('/'))
>
> Adding new trailers for the other ``None``-aware operators ensures that
> they
> may be used in all valid locations for the existing equivalent operators,
> including as part of an assignment target (more details below). As the
> existing
> evaluation rules are not directly embedded in the grammar, we specify the
> required changes here.
>
> Assume that the ``atom`` is always successfully evaluated. Each
> ``trailer`` is
> then evaluated from left to right, applying its own parameter (either its
> arguments, subscripts or attribute name) to produce the value for the next
> ``trailer``. Finally, if present, ``await`` is applied.
>
> For example, ``await a.b(c).d[e]`` is currently parsed as
> ``['await', 'a', '.b', '(c)', '.d', '[e]']`` and evaluated::
>
>      _v = a
>      _v = _v.b
>      _v = _v(c)
>      _v = _v.d
>      _v = _v[e]
>      await _v
>
> When a ``None``-aware operator is present, the left-to-right evaluation
> may be
> short-circuited. For example, ``await a?.b(c).d?[e]`` is evaluated::
>
>      _v = a
>      if _v is not None:
>          _v = _v.b
>          _v = _v(c)
>          _v = _v.d
>          if _v is not None:
>              _v = _v[e]
>      await _v
>
> .. note::
>      ``await`` will almost certainly fail in this context, as it would in
>      the case where code attempts ``await None``. We are not proposing
> to add a
>      ``None``-aware ``await`` keyword here, and merely include it in this
>      example for completeness of the specification, since the ``atom_expr``
>      grammar rule includes the keyword. If it were in its own rule, we
> would have
>      never mentioned it.
>
> Parenthesised expressions are handled by the ``atom`` rule (not shown
> above),
> which will implicitly terminate the short-circuiting behaviour of the above
> transformation. For example, ``(a?.b ?? c).d?.e`` is evaluated as::
>
>      # a?.b
>      _v = a
>      if _v is not None:
>          _v = _v.b
>
>      # ... ?? c
>      if _v is None:
>          _v = c
>
>      # (...).d?.e
>      _v = _v.d
>      if _v is not None:
>          _v = _v.e
>
> When used as an assignment target, the ``None``-aware operations may only
> be
> used in a "load" context. That is, ``a?.b = 1`` and ``a?[b] = 1`` will
> raise
> ``SyntaxError``. Use earlier in the expression (``a?.b.c = 1``) is
> permitted,
> though unlikely to be useful unless combined with a coalescing operation::
>
>      (a?.b ?? d).c = 1
>
>
> Examples
> ========
>
> This section presents some examples of common ``None`` patterns and
> shows what
> conversion to use ``None``-aware operators may look like.
>
> Standard Library
> ----------------
>
> Using the ``find-pep505.py`` script[3]_ an analysis of the Python 3.7
> standard
> library discovered up to 678 code snippets that could be replaced with
> use of
> one of the ``None``-aware operators::
>
>      $ find /usr/lib/python3.7 -name '*.py' | xargs python3.7
> find-pep505.py
>      <snip>
>      Total None-coalescing `if` blocks: 449
>      Total [possible] None-coalescing `or`: 120
>      Total None-coalescing ternaries: 27
>      Total Safe navigation `and`: 13
>      Total Safe navigation `if` blocks: 61
>      Total Safe navigation ternaries: 8
>
> Some of these are shown below as examples before and after converting to
> use the
> new operators.
>
>  From ``bisect.py``::
>
>      def insort_right(a, x, lo=0, hi=None):
>          # ...
>          if hi is None:
>              hi = len(a)
>          # ...
>
> After updating to use the ``??=`` augmented assignment statement::
>
>      def insort_right(a, x, lo=0, hi=None):
>          # ...
>          hi ??= len(a)
>          # ...
>
>  From ``calendar.py``::
>
>      encoding = options.encoding
>      if encoding is None:
>          encoding = sys.getdefaultencoding()
>      optdict = dict(encoding=encoding, css=options.css)
>
> After updating to use the ``??`` operator::
>
>      optdict = dict(encoding=encoding ?? sys.getdefaultencoding(),
>                     css=options.css)
>
>  From ``dis.py``::
>
>      def _get_const_info(const_index, const_list):
>          argval = const_index
>          if const_list is not None:
>              argval = const_list[const_index]
>          return argval, repr(argval)
>
> After updating to use the ``?[]`` and ``??`` operators::
>
>      def _get_const_info(const_index, const_list):
>          argval = const_list?[const_index] ?? const_index
>          return argval, repr(argval)
>
>  From ``inspect.py``::
>
>      for base in object.__bases__:
>          for name in getattr(base, "__abstractmethods__", ()):
>              value = getattr(object, name, None)
>              if getattr(value, "__isabstractmethod__", False):
>                  return True
>
> After updating to use the ``?.`` operator (and deliberately not
> converting to
> use ``any()``)::
>
>      for base in object.__bases__:
>          for name in base?.__abstractmethods__ ?? ():
>              if object?.name?.__isabstractmethod__:
>                  return True
>
>  From ``os.py``::
>
>      if entry.is_dir():
>          dirs.append(name)
>          if entries is not None:
>              entries.append(entry)
>      else:
>          nondirs.append(name)
>
> After updating to use the ``?.`` operator::
>
>      if entry.is_dir():
>          dirs.append(name)
>          entries?.append(entry)
>      else:
>          nondirs.append(name)
>
>
> jsonify
> -------
>
> This example is from a Python web crawler that uses the Flask framework
> as its
> front-end. This function retrieves information about a web site from a SQL
> database and formats it as JSON to send to an HTTP client::
>
>      class SiteView(FlaskView):
>          @route('/site/<id_>', methods=['GET'])
>          def get_site(self, id_):
>              site = db.query('site_table').find(id_)
>
>              return jsonify(
>                  first_seen=site.first_seen.isoformat() if
> site.first_seen is not None else None,
>                  id=site.id,
>                  is_active=site.is_active,
>                  last_seen=site.last_seen.isoformat() if site.last_seen
> is not None else None,
>                  url=site.url.rstrip('/')
>              )
>
> Both ``first_seen`` and ``last_seen`` are allowed to be ``null`` in the
> database, and they are also allowed to be ``null`` in the JSON response.
> JSON
> does not have a native way to represent a ``datetime``, so the server's
> contract
> states that any non-``null`` date is represented as an ISO-8601 string.
>
> Without knowing the exact semantics of the ``first_seen`` and ``last_seen``
> attributes, it is impossible to know whether the attribute can be safely or
> performantly accessed multiple times.
>
> One way to fix this code is to replace each conditional expression with an
> explicit value assignment and a full ``if``/``else`` block::
>
>      class SiteView(FlaskView):
>          @route('/site/<id_>', methods=['GET'])
>          def get_site(self, id_):
>              site = db.query('site_table').find(id_)
>
>              first_seen_dt = site.first_seen
>              if first_seen_dt is None:
>                  first_seen = None
>              else:
>                  first_seen = first_seen_dt.isoformat()
>
>              last_seen_dt = site.last_seen
>              if last_seen_dt is None:
>                  last_seen = None
>              else:
>                  last_seen = last_seen_dt.isoformat()
>
>              return jsonify(
>                  first_seen=first_seen,
>                  id=site.id,
>                  is_active=site.is_active,
>                  last_seen=last_seen,
>                  url=site.url.rstrip('/')
>              )
>
> This adds ten lines of code and four new code paths to the function,
> dramatically increasing the apparent complexity. Rewriting using the
> ``None``-aware attribute operator results in shorter code with more clear
> intent::
>
>      class SiteView(FlaskView):
>          @route('/site/<id_>', methods=['GET'])
>          def get_site(self, id_):
>              site = db.query('site_table').find(id_)
>
>              return jsonify(
>                  first_seen=site.first_seen?.isoformat(),
>                  id=site.id,
>                  is_active=site.is_active,
>                  last_seen=site.last_seen?.isoformat(),
>                  url=site.url.rstrip('/')
>              )
>
> Grab
> ----
>
> The next example is from a Python scraping library called `Grab
> <
> https://github.com/lorien/grab/blob/4c95b18dcb0fa88eeca81f5643c0ebfb114bf728/gr
> ab/upload.py
> <https://github.com/lorien/grab/blob/4c95b18dcb0fa88eeca81f5643c0ebfb114bf728/grab/upload.py>
> >`_::
>
>      class BaseUploadObject(object):
>          def find_content_type(self, filename):
>              ctype, encoding = mimetypes.guess_type(filename)
>              if ctype is None:
>                  return 'application/octet-stream'
>              else:
>                  return ctype
>
>      class UploadContent(BaseUploadObject):
>          def __init__(self, content, filename=None, content_type=None):
>              self.content = content
>              if filename is None:
>                  self.filename = self.get_random_filename()
>              else:
>                  self.filename = filename
>              if content_type is None:
>                  self.content_type = self.find_content_type(self.filename)
>              else:
>                  self.content_type = content_type
>
>      class UploadFile(BaseUploadObject):
>          def __init__(self, path, filename=None, content_type=None):
>              self.path = path
>              if filename is None:
>                  self.filename = os.path.split(path)[1]
>              else:
>                  self.filename = filename
>              if content_type is None:
>                  self.content_type = self.find_content_type(self.filename)
>              else:
>                  self.content_type = content_type
>
> This example contains several good examples of needing to provide default
> values. Rewriting to use conditional expressions reduces the overall
> lines of
> code, but does not necessarily improve readability::
>
>      class BaseUploadObject(object):
>          def find_content_type(self, filename):
>              ctype, encoding = mimetypes.guess_type(filename)
>              return 'application/octet-stream' if ctype is None else ctype
>
>      class UploadContent(BaseUploadObject):
>          def __init__(self, content, filename=None, content_type=None):
>              self.content = content
>              self.filename = (self.get_random_filename() if filename
>                  is None else filename)
>              self.content_type = (self.find_content_type(self.filename)
>                  if content_type is None else content_type)
>
>      class UploadFile(BaseUploadObject):
>          def __init__(self, path, filename=None, content_type=None):
>              self.path = path
>              self.filename = (os.path.split(path)[1] if filename is
>                  None else filename)
>              self.content_type = (self.find_content_type(self.filename)
>                  if content_type is None else content_type)
>
> The first ternary expression is tidy, but it reverses the intuitive order
> of
> the operands: it should return ``ctype`` if it has a value and use the
> string
> literal as fallback. The other ternary expressions are unintuitive and so
> long that they must be wrapped. The overall readability is worsened, not
> improved.
>
> Rewriting using the ``None`` coalescing operator::
>
>      class BaseUploadObject(object):
>          def find_content_type(self, filename):
>              ctype, encoding = mimetypes.guess_type(filename)
>              return ctype ?? 'application/octet-stream'
>
>      class UploadContent(BaseUploadObject):
>          def __init__(self, content, filename=None, content_type=None):
>              self.content = content
>              self.filename = filename ?? self.get_random_filename()
>              self.content_type = content_type ??
> self.find_content_type(self.filename)
>
>      class UploadFile(BaseUploadObject):
>          def __init__(self, path, filename=None, content_type=None):
>              self.path = path
>              self.filename = filename ?? os.path.split(path)[1]
>              self.content_type = content_type ??
> self.find_content_type(self.filename)
>
> This syntax has an intuitive ordering of the operands. In
> ``find_content_type``,
> for example, the preferred value ``ctype`` appears before the fallback
> value.
> The terseness of the syntax also makes for fewer lines of code and less
> code to
> visually parse, and reading from left-to-right and top-to-bottom more
> accurately
> follows the execution flow.
>
>
> Rejected Ideas
> ==============
>
> The first three ideas in this section are oft-proposed alternatives to
> treating
> ``None`` as special. For further background on why these are rejected,
> see their
> treatment in `PEP 531 <https://www.python.org/dev/peps/pep-0531/>`_ and
> `PEP 532 <https://www.python.org/dev/peps/pep-0532/>`_ and the associated
> discussions.
>
> No-Value Protocol
> -----------------
>
> The operators could be generalised to user-defined types by defining a
> protocol
> to indicate when a value represents "no value". Such a protocol may be a
> dunder
> method ``__has_value__(self)` that returns ``True`` if the value should be
> treated as having a value, and ``False`` if the value should be treated
> as no
> value.
>
> With this generalization, ``object`` would implement a dunder method
> equivalent
> to this::
>
>      def __has_value__(self):
>          return True
>
> ``NoneType`` would implement a dunder method equivalent to this::
>
>      def __has_value__(self):
>          return False
>
> In the specification section, all uses of ``x is None`` would be
> replaced with
> ``not x.__has_value__()``.
>
> This generalization would allow for domain-specific "no-value" objects to
> be
> coalesced just like ``None``. For example the ``pyasn1`` package has a type
> called ``Null`` that represents an ASN.1 ``null``::
>
>      >>> from pyasn1.type import univ
>      >>> univ.Null() ?? univ.Integer(123)
>      Integer(123)
>
> Similarly, values such as ``math.nan`` and ``NotImplemented`` could be
> treated
> as representing no value.
>
> However, the "no-value" nature of these values is domain-specific, which
> means
> they *should* be treated as a value by the language. For example,
> ``math.nan.imag`` is well defined (it's ``0.0``), and so short-circuiting
> ``math.nan?.imag`` to return ``math.nan`` would be incorrect.
>
> As ``None`` is already defined by the language as being the value that
> represents "no value", and the current specification would not preclude
> switching to a protocol in the future (though changes to built-in
> objects would
> not be compatible), this idea is rejected for now.
>
> Boolean-aware operators
> -----------------------
>
> This suggestion is fundamentally the same as adding a no-value protocol,
> and so
> the discussion above also applies.
>
> Similar behavior to the ``??`` operator can be achieved with an ``or``
> expression, however ``or`` checks whether its left operand is false-y
> and not
> specifically ``None``. This approach is attractive, as it requires fewer
> changes
> to the language, but ultimately does not solve the underlying problem
> correctly.
>
> Assuming the check is for truthiness rather than ``None``, there is no
> longer a
> need for the ``??`` operator. However, applying this check to the ``?.``
> and
> ``?[]`` operators prevents perfectly valid operations applying
>
> Consider the following example, where ``get_log_list()`` may return either
> a
> list containing current log messages (potentially empty), or ``None`` if
> logging
> is not enabled::
>
>      lst = get_log_list()
>      lst?.append('A log message')
>
> If ``?.`` is checking for true values rather than specifically ``None``
> and the
> log has not been initialized with any items, no item will ever be
> appended. This
> violates the obvious intent of the code, which is to append an item. The
> ``append`` method is available on an empty list, as are all other list
> methods,
> and there is no reason to assume that these members should not be used
> because
> the list is presently empty.
>
> Further, there is no sensible result to use in place of the expression. A
> normal ``lst.append`` returns ``None``, but under this idea
> ``lst?.append`` may
> result in either ``[]`` or ``None``, depending on the value of ``lst``.
> As with
> the examples in the previous section, this makes no sense.
>
> As checking for truthiness rather than ``None`` results in apparently valid
> expressions no longer executing as intended, this idea is rejected.
>
> Exception-aware operators
> -------------------------
>
> Arguably, the reason to short-circuit an expression when ``None`` is
> encountered
> is to avoid the ``AttributeError`` or ``TypeError`` that would be raised
> under
> normal circumstances. As an alternative to testing for ``None``, the
> ``?.`` and
> ``?[]`` operators could instead handle ``AttributeError`` and ``TypeError``
> raised by the operation and skip the remainder of the expression.
>
> This produces a transformation for ``a?.b.c?.d.e`` similar to this::
>
>      _v = a
>      try:
>          _v = _v.b
>      except AttributeError:
>          pass
>      else:
>          _v = _v.c
>          try:
>              _v = _v.d
>          except AttributeError:
>              pass
>          else:
>              _v = _v.e
>
> One open question is which value should be returned as the expression
> when an
> exception is handled. The above example simply leaves the partial
> result, but
> this is not helpful for replacing with a default value. An alternative
> would be
> to force the result to ``None``, which then raises the question as to why
> ``None`` is special enough to be the result but not special enough to be
> the
> test.
>
> Secondly, this approach masks errors within code executed implicitly as
> part of
> the expression. For ``?.``, any ``AttributeError`` within a property or
> ``__getattr__`` implementation would be hidden, and similarly for
> ``?[]`` and
> ``__getitem__`` implementations.
>
> Similarly, simple typing errors such as ``{}?.ietms()`` could go unnoticed.
>
> Existing conventions for handling these kinds of errors in the form of the
> ``getattr`` builtin and the ``.get(key, default)`` method pattern
> established by
> ``dict`` show that it is already possible to explicitly use this behaviour.
>
> As this approach would hide errors in code, it is rejected.
>
> ``None``-aware Function Call
> ----------------------------
>
> The ``None``-aware syntax applies to attribute and index access, so it
> seems
> natural to ask if it should also apply to function invocation syntax. It
> might
> be written as ``foo?()``, where ``foo`` is only called if it is not None.
>
> This has been deferred on the basis of the proposed operators being
> intended
> to aid traversal of partially populated hierarchical data structures, *not*
> for traversal of arbitrary class hierarchies. This is reflected in the fact
> that none of the other mainstream languages that already offer this syntax
> have found it worthwhile to support a similar syntax for optional function
> invocations.
>
> A workaround similar to that used by C# would be to write
> ``maybe_none?.__call__(arguments)``. If the callable is ``None``, the
> expression will not be evaluated. (The C# equivalent uses ``?.Invoke()``
> on its
> callable type.)
>
> ``?`` Unary Postfix Operator
> ----------------------------
>
> To generalize the ``None``-aware behavior and limit the number of new
> operators
> introduced, a unary, postfix operator spelled ``?`` was suggested. The
> idea is
> that ``?`` might return a special object that could would override dunder
> methods that return ``self``. For example, ``foo?`` would evaluate to
> ``foo`` if
> it is not ``None``, otherwise it would evaluate to an instance of
> ``NoneQuestion``::
>
>      class NoneQuestion():
>          def __call__(self, *args, **kwargs):
>              return self
>
>          def __getattr__(self, name):
>              return self
>
>          def __getitem__(self, key):
>              return self
>
>
> With this new operator and new type, an expression like ``foo?.bar[baz]``
> evaluates to ``NoneQuestion`` if ``foo`` is None. This is a nifty
> generalization, but it's difficult to use in practice since most
> existing code
> won't know what ``NoneQuestion`` is.
>
> Going back to one of the motivating examples above, consider the
> following::
>
>      >>> import json
>      >>> created = None
>      >>> json.dumps({'created': created?.isoformat()})``
>
> The JSON serializer does not know how to serialize ``NoneQuestion``, nor
> will
> any other API. This proposal actually requires *lots of specialized logic*
> throughout the standard library and any third party library.
>
> At the same time, the ``?`` operator may also be **too general**, in the
> sense
> that it can be combined with any other operator. What should the following
> expressions mean?::
>
>      >>> x? + 1
>      >>> x? -= 1
>      >>> x? == 1
>      >>> ~x?
>
> This degree of generalization is not useful. The operators actually
> proposed
> herein are intentionally limited to a few operators that are expected to
> make it
> easier to write common code patterns.
>
> Built-in ``maybe``
> ------------------
>
> Haskell has a concept called `Maybe <https://wiki.haskell.org/Maybe>`_
> that
> encapsulates the idea of an optional value without relying on any special
> keyword (e.g. ``null``) or any special instance (e.g. ``None``). In
> Haskell, the
> purpose of ``Maybe`` is to avoid separate handling of "something" and
> nothing".
>
> A Python package called `pymaybe <https://pypi.org/p/pymaybe/>`_ provides
> a
> rough approximation. The documentation shows the following example::
>
>      >>> maybe('VALUE').lower()
>      'value'
>
>      >>> maybe(None).invalid().method().or_else('unknown')
>      'unknown'
>
> The function ``maybe()`` returns either a ``Something`` instance or a
> ``Nothing`` instance. Similar to the unary postfix operator described in
> the
> previous section, ``Nothing`` overrides dunder methods in order to allow
> chaining on a missing value.
>
> Note that ``or_else()`` is eventually required to retrieve the
> underlying value
> from ``pymaybe``'s wrappers. Furthermore, ``pymaybe`` does not short
> circuit any
> evaluation. Although ``pymaybe`` has some strengths and may be useful in
> its own
> right, it also demonstrates why a pure Python implementation of
> coalescing is
> not nearly as powerful as support built into the language.
>
> The idea of adding a builtin ``maybe`` type to enable this scenario is
> rejected.
>
> Just use a conditional expression
> ---------------------------------
>
> Another common way to initialize default values is to use the ternary
> operator.
> Here is an excerpt from the popular `Requests package
> <
> https://github.com/kennethreitz/requests/blob/14a555ac716866678bf17e43e23230d81
> a8149f5/requests/models.py#L212
> <https://github.com/kennethreitz/requests/blob/14a555ac716866678bf17e43e23230d81a8149f5/requests/models.py#L212>
> >`_::
>
>      data = [] if data is None else data
>      files = [] if files is None else files
>      headers = {} if headers is None else headers
>      params = {} if params is None else params
>      hooks = {} if hooks is None else hooks
>
> This particular formulation has the undesirable effect of putting the
> operands
> in an unintuitive order: the brain thinks, "use ``data`` if possible and
> use
> ``[]`` as a fallback," but the code puts the fallback *before* the
> preferred
> value.
>
> The author of this package could have written it like this instead::
>
>      data = data if data is not None else []
>      files = files if files is not None else []
>      headers = headers if headers is not None else {}
>      params = params if params is not None else {}
>      hooks = hooks if hooks is not None else {}
>
> This ordering of the operands is more intuitive, but it requires 4 extra
> characters (for "not "). It also highlights the repetition of identifiers:
> ``data if data``, ``files if files``, etc.
>
> When written using the ``None`` coalescing operator, the sample reads::
>
>      data = data ?? []
>      files = files ?? []
>      headers = headers ?? {}
>      params = params ?? {}
>      hooks = hooks ?? {}
>
>
> References
> ==========
>
> .. [1] C# Reference: Operators
>     (https://msdn.microsoft.com/en-us/library/6a71f45d.aspx)
>
> .. [2] A Tour of the Dart Language: Operators
>     (https://www.dartlang.org/docs/dart-up-and-running/ch02.html#operators
> )
>
> .. [3] Associated scripts
>     (https://github.com/python/peps/tree/master/pep-0505/)
>
> Copyright
> =========
>
> This document has been placed in the public domain.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180718/13532e3b/attachment-0001.html>

From mertz at gnosis.cx  Wed Jul 18 20:16:16 2018
From: mertz at gnosis.cx (David Mertz)
Date: Wed, 18 Jul 2018 20:16:16 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
Message-ID: <CAEbHw4YgGJyKfRZZQGfPLe940o3MpEuhY6dQV_+BgTqerg-ADg@mail.gmail.com>

The examples in the PEP strengthen my opinion. I think every one of them
read much more clearly in the existing syntax. I can only understand any by
mentally inserting 'if Foo is None' everywhere... Basically mentally
replacing the "code golf" syntax with the actual intent that is spelled
like it is in current Python.

On Wed, Jul 18, 2018, 8:05 PM David Mertz <mertz at gnosis.cx> wrote:

> As before, I'm a strong -1 on this.
>
> I recognize the semantic utility (albeit, I've needed it much less than
> some folks testify). But the obscure characters are accumulating far too
> fast with this.
>
> Even with PEP 572, I would have preferred 'as' and restrictions on where
> it's allowed, but ':=' is more familiar from numerous languages and
> pseudo-code notations.  '??' and '?.' and ?[]' are really just marching
> into APL and Perl territory. Yes, I know such operators exist in other
> languages, but it feels very unpythonic.
>
> On Wed, Jul 18, 2018, 1:46 PM Steve Dower <steve.dower at python.org> wrote:
>
>> Possibly this is exactly the wrong time to propose the next big syntax
>> change, since we currently have nobody to declare on it, but since we're
>> likely to argue for a while anyway it probably can't hurt (and maybe
>> this will become the test PEP for whoever takes the reins?).
>>
>> FWIW, Guido had previously indicated that he was generally favourable
>> towards most of this proposal, provided we could figure out coherent
>> semantics. Last time we tried, that didn't happen, so this time I've
>> made the semantics much more precise, have implemented and verified
>> them, and made much stronger statements about why we are proposing these.
>>
>> Additional thanks to Mark Haase for writing most of the PEP. All the
>> fair and balanced parts are his - all the overly strong opinions are mine.
>>
>> Also thanks to Nick Coghlan for writing PEPs 531 and 532 last time we
>> went through this - if you're unhappy with "None" being treated as a
>> special kind of value, I recommend reading those before you start
>> repeating them.
>>
>> There is a formatted version of this PEP at
>> https://www.python.org/dev/peps/pep-0505/
>>
>> My current implementation is at
>> https://github.com/zooba/cpython/tree/pep-505 (though I'm considering
>> removing some of the new opcodes I added and just generating more
>> complex code - in any case, let's get hung up on the proposal rather
>> than the implementation :) )
>>
>> Let the discussions begin!
>>
>> ---
>>
>> PEP: 505
>> Title: None-aware operators
>> Version: $Revision$
>> Last-Modified: $Date$
>> Author: Mark E. Haase <mehaase at gmail.com>, Steve Dower
>> <steve.dower at python.org>
>> Status: Draft
>> Type: Standards Track
>> Content-Type: text/x-rst
>> Created: 18-Sep-2015
>> Python-Version: 3.8
>>
>> Abstract
>> ========
>>
>> Several modern programming languages have so-called "``null``-coalescing"
>> or
>> "``null``- aware" operators, including C# [1]_, Dart [2]_, Perl, Swift,
>> and PHP
>> (starting in version 7). These operators provide syntactic sugar for
>> common
>> patterns involving null references.
>>
>> * The "``null``-coalescing" operator is a binary operator that returns
>> its left
>>    operand if it is not ``null``. Otherwise it returns its right operand.
>> * The "``null``-aware member access" operator accesses an instance
>> member only
>>    if that instance is non-``null``. Otherwise it returns ``null``.
>> (This is also
>>    called a "safe navigation" operator.)
>> * The "``null``-aware index access" operator accesses an element of a
>> collection
>>    only if that collection is non-``null``. Otherwise it returns
>> ``null``. (This
>>    is another type of "safe navigation" operator.)
>>
>> This PEP proposes three ``None``-aware operators for Python, based on the
>> definitions and other language's implementations of those above.
>> Specifically:
>>
>> * The "``None`` coalescing`` binary operator ``??`` returns the left
>> hand side
>>    if it evaluates to a value that is not ``None``, or else it evaluates
>> and
>>    returns the right hand side. A coalescing ``??=`` augmented assignment
>>    operator is included.
>> * The "``None``-aware attribute access" operator ``?.`` evaluates the
>> complete
>>    expression if the left hand side evaluates to a value that is not
>> ``None``
>> * The "``None``-aware indexing" operator ``?[]`` evaluates the complete
>>    expression if the left hand site evaluates to a value that is not
>> ``None``
>>
>> Syntax and Semantics
>> ====================
>>
>> Specialness of ``None``
>> -----------------------
>>
>> The ``None`` object denotes the lack of a value. For the purposes of these
>> operators, the lack of a value indicates that the remainder of the
>> expression
>> also lacks a value and should not be evaluated.
>>
>> A rejected proposal was to treat any value that evaluates to false in a
>> Boolean context as not having a value. However, the purpose of these
>> operators
>> is to propagate the "lack of value" state, rather that the "false" state.
>>
>> Some argue that this makes ``None`` special. We contend that ``None`` is
>> already special, and that using it as both the test and the result of
>> these
>> operators does not change the existing semantics in any way.
>>
>> See the `Rejected Ideas`_ section for discussion on the rejected
>> approaches.
>>
>> Grammar changes
>> ---------------
>>
>> The following rules of the Python grammar are updated to read::
>>
>>      augassign: ('+=' | '-=' | '*=' | '@=' | '/=' | '%=' | '&=' | '|=' |
>> '^=' |
>>                  '<<=' | '>>=' | '**=' | '//=' | '??=')
>>
>>      power: coalesce ['**' factor]
>>      coalesce: atom_expr ['??' factor]
>>      atom_expr: ['await'] atom trailer*
>>      trailer: ('(' [arglist] ')' |
>>                '[' subscriptlist ']' |
>>                '?[' subscriptlist ']' |
>>                '.' NAME |
>>                '?.' NAME)
>>
>> Inserting the ``coalesce`` rule in this location ensures that expressions
>> resulting in ``None`` are natuarlly coalesced before they are used in
>> operations that would typically raise ``TypeError``. Like ``and`` and
>> ``or``
>> the right-hand expression is not evaluated until the left-hand side is
>> determined to be ``None``. For example::
>>
>>      a, b = None, None
>>      def c(): return None
>>      def ex(): raise Exception()
>>
>>      (a ?? 2 ** b ?? 3) == a ?? (2 ** (b ?? 3))
>>      (a * b ?? c // d) == a * (b ?? c) // d
>>      (a ?? True and b ?? False) == (a ?? True) and (b ?? False)
>>      (c() ?? c() ?? True) == True
>>      (True ?? ex()) == True
>>      (c ?? ex)() == c()
>>
>> Augmented coalescing assignment only rebinds the name if its current
>> value is
>> ``None``. If the target name already has a value, the right-hand side is
>> not
>> evaluated. For example::
>>
>>      a = None
>>      b = ''
>>      c = 0
>>
>>      a ??= 'value'
>>      b ??= undefined_name
>>      c ??= shutil.rmtree('/')    # don't try this at home, kids
>>
>>      assert a == 'value'
>>      assert b == ''
>>      assert c == '0' and any(os.scandir('/'))
>>
>> Adding new trailers for the other ``None``-aware operators ensures that
>> they
>> may be used in all valid locations for the existing equivalent operators,
>> including as part of an assignment target (more details below). As the
>> existing
>> evaluation rules are not directly embedded in the grammar, we specify the
>> required changes here.
>>
>> Assume that the ``atom`` is always successfully evaluated. Each
>> ``trailer`` is
>> then evaluated from left to right, applying its own parameter (either its
>> arguments, subscripts or attribute name) to produce the value for the next
>> ``trailer``. Finally, if present, ``await`` is applied.
>>
>> For example, ``await a.b(c).d[e]`` is currently parsed as
>> ``['await', 'a', '.b', '(c)', '.d', '[e]']`` and evaluated::
>>
>>      _v = a
>>      _v = _v.b
>>      _v = _v(c)
>>      _v = _v.d
>>      _v = _v[e]
>>      await _v
>>
>> When a ``None``-aware operator is present, the left-to-right evaluation
>> may be
>> short-circuited. For example, ``await a?.b(c).d?[e]`` is evaluated::
>>
>>      _v = a
>>      if _v is not None:
>>          _v = _v.b
>>          _v = _v(c)
>>          _v = _v.d
>>          if _v is not None:
>>              _v = _v[e]
>>      await _v
>>
>> .. note::
>>      ``await`` will almost certainly fail in this context, as it would in
>>      the case where code attempts ``await None``. We are not proposing
>> to add a
>>      ``None``-aware ``await`` keyword here, and merely include it in this
>>      example for completeness of the specification, since the
>> ``atom_expr``
>>      grammar rule includes the keyword. If it were in its own rule, we
>> would have
>>      never mentioned it.
>>
>> Parenthesised expressions are handled by the ``atom`` rule (not shown
>> above),
>> which will implicitly terminate the short-circuiting behaviour of the
>> above
>> transformation. For example, ``(a?.b ?? c).d?.e`` is evaluated as::
>>
>>      # a?.b
>>      _v = a
>>      if _v is not None:
>>          _v = _v.b
>>
>>      # ... ?? c
>>      if _v is None:
>>          _v = c
>>
>>      # (...).d?.e
>>      _v = _v.d
>>      if _v is not None:
>>          _v = _v.e
>>
>> When used as an assignment target, the ``None``-aware operations may only
>> be
>> used in a "load" context. That is, ``a?.b = 1`` and ``a?[b] = 1`` will
>> raise
>> ``SyntaxError``. Use earlier in the expression (``a?.b.c = 1``) is
>> permitted,
>> though unlikely to be useful unless combined with a coalescing operation::
>>
>>      (a?.b ?? d).c = 1
>>
>>
>> Examples
>> ========
>>
>> This section presents some examples of common ``None`` patterns and
>> shows what
>> conversion to use ``None``-aware operators may look like.
>>
>> Standard Library
>> ----------------
>>
>> Using the ``find-pep505.py`` script[3]_ an analysis of the Python 3.7
>> standard
>> library discovered up to 678 code snippets that could be replaced with
>> use of
>> one of the ``None``-aware operators::
>>
>>      $ find /usr/lib/python3.7 -name '*.py' | xargs python3.7
>> find-pep505.py
>>      <snip>
>>      Total None-coalescing `if` blocks: 449
>>      Total [possible] None-coalescing `or`: 120
>>      Total None-coalescing ternaries: 27
>>      Total Safe navigation `and`: 13
>>      Total Safe navigation `if` blocks: 61
>>      Total Safe navigation ternaries: 8
>>
>> Some of these are shown below as examples before and after converting to
>> use the
>> new operators.
>>
>>  From ``bisect.py``::
>>
>>      def insort_right(a, x, lo=0, hi=None):
>>          # ...
>>          if hi is None:
>>              hi = len(a)
>>          # ...
>>
>> After updating to use the ``??=`` augmented assignment statement::
>>
>>      def insort_right(a, x, lo=0, hi=None):
>>          # ...
>>          hi ??= len(a)
>>          # ...
>>
>>  From ``calendar.py``::
>>
>>      encoding = options.encoding
>>      if encoding is None:
>>          encoding = sys.getdefaultencoding()
>>      optdict = dict(encoding=encoding, css=options.css)
>>
>> After updating to use the ``??`` operator::
>>
>>      optdict = dict(encoding=encoding ?? sys.getdefaultencoding(),
>>                     css=options.css)
>>
>>  From ``dis.py``::
>>
>>      def _get_const_info(const_index, const_list):
>>          argval = const_index
>>          if const_list is not None:
>>              argval = const_list[const_index]
>>          return argval, repr(argval)
>>
>> After updating to use the ``?[]`` and ``??`` operators::
>>
>>      def _get_const_info(const_index, const_list):
>>          argval = const_list?[const_index] ?? const_index
>>          return argval, repr(argval)
>>
>>  From ``inspect.py``::
>>
>>      for base in object.__bases__:
>>          for name in getattr(base, "__abstractmethods__", ()):
>>              value = getattr(object, name, None)
>>              if getattr(value, "__isabstractmethod__", False):
>>                  return True
>>
>> After updating to use the ``?.`` operator (and deliberately not
>> converting to
>> use ``any()``)::
>>
>>      for base in object.__bases__:
>>          for name in base?.__abstractmethods__ ?? ():
>>              if object?.name?.__isabstractmethod__:
>>                  return True
>>
>>  From ``os.py``::
>>
>>      if entry.is_dir():
>>          dirs.append(name)
>>          if entries is not None:
>>              entries.append(entry)
>>      else:
>>          nondirs.append(name)
>>
>> After updating to use the ``?.`` operator::
>>
>>      if entry.is_dir():
>>          dirs.append(name)
>>          entries?.append(entry)
>>      else:
>>          nondirs.append(name)
>>
>>
>> jsonify
>> -------
>>
>> This example is from a Python web crawler that uses the Flask framework
>> as its
>> front-end. This function retrieves information about a web site from a SQL
>> database and formats it as JSON to send to an HTTP client::
>>
>>      class SiteView(FlaskView):
>>          @route('/site/<id_>', methods=['GET'])
>>          def get_site(self, id_):
>>              site = db.query('site_table').find(id_)
>>
>>              return jsonify(
>>                  first_seen=site.first_seen.isoformat() if
>> site.first_seen is not None else None,
>>                  id=site.id,
>>                  is_active=site.is_active,
>>                  last_seen=site.last_seen.isoformat() if site.last_seen
>> is not None else None,
>>                  url=site.url.rstrip('/')
>>              )
>>
>> Both ``first_seen`` and ``last_seen`` are allowed to be ``null`` in the
>> database, and they are also allowed to be ``null`` in the JSON response.
>> JSON
>> does not have a native way to represent a ``datetime``, so the server's
>> contract
>> states that any non-``null`` date is represented as an ISO-8601 string.
>>
>> Without knowing the exact semantics of the ``first_seen`` and
>> ``last_seen``
>> attributes, it is impossible to know whether the attribute can be safely
>> or
>> performantly accessed multiple times.
>>
>> One way to fix this code is to replace each conditional expression with an
>> explicit value assignment and a full ``if``/``else`` block::
>>
>>      class SiteView(FlaskView):
>>          @route('/site/<id_>', methods=['GET'])
>>          def get_site(self, id_):
>>              site = db.query('site_table').find(id_)
>>
>>              first_seen_dt = site.first_seen
>>              if first_seen_dt is None:
>>                  first_seen = None
>>              else:
>>                  first_seen = first_seen_dt.isoformat()
>>
>>              last_seen_dt = site.last_seen
>>              if last_seen_dt is None:
>>                  last_seen = None
>>              else:
>>                  last_seen = last_seen_dt.isoformat()
>>
>>              return jsonify(
>>                  first_seen=first_seen,
>>                  id=site.id,
>>                  is_active=site.is_active,
>>                  last_seen=last_seen,
>>                  url=site.url.rstrip('/')
>>              )
>>
>> This adds ten lines of code and four new code paths to the function,
>> dramatically increasing the apparent complexity. Rewriting using the
>> ``None``-aware attribute operator results in shorter code with more clear
>> intent::
>>
>>      class SiteView(FlaskView):
>>          @route('/site/<id_>', methods=['GET'])
>>          def get_site(self, id_):
>>              site = db.query('site_table').find(id_)
>>
>>              return jsonify(
>>                  first_seen=site.first_seen?.isoformat(),
>>                  id=site.id,
>>                  is_active=site.is_active,
>>                  last_seen=site.last_seen?.isoformat(),
>>                  url=site.url.rstrip('/')
>>              )
>>
>> Grab
>> ----
>>
>> The next example is from a Python scraping library called `Grab
>> <
>> https://github.com/lorien/grab/blob/4c95b18dcb0fa88eeca81f5643c0ebfb114bf728/gr
>> ab/upload.py
>> <https://github.com/lorien/grab/blob/4c95b18dcb0fa88eeca81f5643c0ebfb114bf728/grab/upload.py>
>> >`_::
>>
>>      class BaseUploadObject(object):
>>          def find_content_type(self, filename):
>>              ctype, encoding = mimetypes.guess_type(filename)
>>              if ctype is None:
>>                  return 'application/octet-stream'
>>              else:
>>                  return ctype
>>
>>      class UploadContent(BaseUploadObject):
>>          def __init__(self, content, filename=None, content_type=None):
>>              self.content = content
>>              if filename is None:
>>                  self.filename = self.get_random_filename()
>>              else:
>>                  self.filename = filename
>>              if content_type is None:
>>                  self.content_type = self.find_content_type(self.filename)
>>              else:
>>                  self.content_type = content_type
>>
>>      class UploadFile(BaseUploadObject):
>>          def __init__(self, path, filename=None, content_type=None):
>>              self.path = path
>>              if filename is None:
>>                  self.filename = os.path.split(path)[1]
>>              else:
>>                  self.filename = filename
>>              if content_type is None:
>>                  self.content_type = self.find_content_type(self.filename)
>>              else:
>>                  self.content_type = content_type
>>
>> This example contains several good examples of needing to provide default
>> values. Rewriting to use conditional expressions reduces the overall
>> lines of
>> code, but does not necessarily improve readability::
>>
>>      class BaseUploadObject(object):
>>          def find_content_type(self, filename):
>>              ctype, encoding = mimetypes.guess_type(filename)
>>              return 'application/octet-stream' if ctype is None else ctype
>>
>>      class UploadContent(BaseUploadObject):
>>          def __init__(self, content, filename=None, content_type=None):
>>              self.content = content
>>              self.filename = (self.get_random_filename() if filename
>>                  is None else filename)
>>              self.content_type = (self.find_content_type(self.filename)
>>                  if content_type is None else content_type)
>>
>>      class UploadFile(BaseUploadObject):
>>          def __init__(self, path, filename=None, content_type=None):
>>              self.path = path
>>              self.filename = (os.path.split(path)[1] if filename is
>>                  None else filename)
>>              self.content_type = (self.find_content_type(self.filename)
>>                  if content_type is None else content_type)
>>
>> The first ternary expression is tidy, but it reverses the intuitive order
>> of
>> the operands: it should return ``ctype`` if it has a value and use the
>> string
>> literal as fallback. The other ternary expressions are unintuitive and so
>> long that they must be wrapped. The overall readability is worsened, not
>> improved.
>>
>> Rewriting using the ``None`` coalescing operator::
>>
>>      class BaseUploadObject(object):
>>          def find_content_type(self, filename):
>>              ctype, encoding = mimetypes.guess_type(filename)
>>              return ctype ?? 'application/octet-stream'
>>
>>      class UploadContent(BaseUploadObject):
>>          def __init__(self, content, filename=None, content_type=None):
>>              self.content = content
>>              self.filename = filename ?? self.get_random_filename()
>>              self.content_type = content_type ??
>> self.find_content_type(self.filename)
>>
>>      class UploadFile(BaseUploadObject):
>>          def __init__(self, path, filename=None, content_type=None):
>>              self.path = path
>>              self.filename = filename ?? os.path.split(path)[1]
>>              self.content_type = content_type ??
>> self.find_content_type(self.filename)
>>
>> This syntax has an intuitive ordering of the operands. In
>> ``find_content_type``,
>> for example, the preferred value ``ctype`` appears before the fallback
>> value.
>> The terseness of the syntax also makes for fewer lines of code and less
>> code to
>> visually parse, and reading from left-to-right and top-to-bottom more
>> accurately
>> follows the execution flow.
>>
>>
>> Rejected Ideas
>> ==============
>>
>> The first three ideas in this section are oft-proposed alternatives to
>> treating
>> ``None`` as special. For further background on why these are rejected,
>> see their
>> treatment in `PEP 531 <https://www.python.org/dev/peps/pep-0531/>`_ and
>> `PEP 532 <https://www.python.org/dev/peps/pep-0532/>`_ and the associated
>> discussions.
>>
>> No-Value Protocol
>> -----------------
>>
>> The operators could be generalised to user-defined types by defining a
>> protocol
>> to indicate when a value represents "no value". Such a protocol may be a
>> dunder
>> method ``__has_value__(self)` that returns ``True`` if the value should be
>> treated as having a value, and ``False`` if the value should be treated
>> as no
>> value.
>>
>> With this generalization, ``object`` would implement a dunder method
>> equivalent
>> to this::
>>
>>      def __has_value__(self):
>>          return True
>>
>> ``NoneType`` would implement a dunder method equivalent to this::
>>
>>      def __has_value__(self):
>>          return False
>>
>> In the specification section, all uses of ``x is None`` would be
>> replaced with
>> ``not x.__has_value__()``.
>>
>> This generalization would allow for domain-specific "no-value" objects to
>> be
>> coalesced just like ``None``. For example the ``pyasn1`` package has a
>> type
>> called ``Null`` that represents an ASN.1 ``null``::
>>
>>      >>> from pyasn1.type import univ
>>      >>> univ.Null() ?? univ.Integer(123)
>>      Integer(123)
>>
>> Similarly, values such as ``math.nan`` and ``NotImplemented`` could be
>> treated
>> as representing no value.
>>
>> However, the "no-value" nature of these values is domain-specific, which
>> means
>> they *should* be treated as a value by the language. For example,
>> ``math.nan.imag`` is well defined (it's ``0.0``), and so short-circuiting
>> ``math.nan?.imag`` to return ``math.nan`` would be incorrect.
>>
>> As ``None`` is already defined by the language as being the value that
>> represents "no value", and the current specification would not preclude
>> switching to a protocol in the future (though changes to built-in
>> objects would
>> not be compatible), this idea is rejected for now.
>>
>> Boolean-aware operators
>> -----------------------
>>
>> This suggestion is fundamentally the same as adding a no-value protocol,
>> and so
>> the discussion above also applies.
>>
>> Similar behavior to the ``??`` operator can be achieved with an ``or``
>> expression, however ``or`` checks whether its left operand is false-y
>> and not
>> specifically ``None``. This approach is attractive, as it requires fewer
>> changes
>> to the language, but ultimately does not solve the underlying problem
>> correctly.
>>
>> Assuming the check is for truthiness rather than ``None``, there is no
>> longer a
>> need for the ``??`` operator. However, applying this check to the ``?.``
>> and
>> ``?[]`` operators prevents perfectly valid operations applying
>>
>> Consider the following example, where ``get_log_list()`` may return
>> either a
>> list containing current log messages (potentially empty), or ``None`` if
>> logging
>> is not enabled::
>>
>>      lst = get_log_list()
>>      lst?.append('A log message')
>>
>> If ``?.`` is checking for true values rather than specifically ``None``
>> and the
>> log has not been initialized with any items, no item will ever be
>> appended. This
>> violates the obvious intent of the code, which is to append an item. The
>> ``append`` method is available on an empty list, as are all other list
>> methods,
>> and there is no reason to assume that these members should not be used
>> because
>> the list is presently empty.
>>
>> Further, there is no sensible result to use in place of the expression. A
>> normal ``lst.append`` returns ``None``, but under this idea
>> ``lst?.append`` may
>> result in either ``[]`` or ``None``, depending on the value of ``lst``.
>> As with
>> the examples in the previous section, this makes no sense.
>>
>> As checking for truthiness rather than ``None`` results in apparently
>> valid
>> expressions no longer executing as intended, this idea is rejected.
>>
>> Exception-aware operators
>> -------------------------
>>
>> Arguably, the reason to short-circuit an expression when ``None`` is
>> encountered
>> is to avoid the ``AttributeError`` or ``TypeError`` that would be raised
>> under
>> normal circumstances. As an alternative to testing for ``None``, the
>> ``?.`` and
>> ``?[]`` operators could instead handle ``AttributeError`` and
>> ``TypeError``
>> raised by the operation and skip the remainder of the expression.
>>
>> This produces a transformation for ``a?.b.c?.d.e`` similar to this::
>>
>>      _v = a
>>      try:
>>          _v = _v.b
>>      except AttributeError:
>>          pass
>>      else:
>>          _v = _v.c
>>          try:
>>              _v = _v.d
>>          except AttributeError:
>>              pass
>>          else:
>>              _v = _v.e
>>
>> One open question is which value should be returned as the expression
>> when an
>> exception is handled. The above example simply leaves the partial
>> result, but
>> this is not helpful for replacing with a default value. An alternative
>> would be
>> to force the result to ``None``, which then raises the question as to why
>> ``None`` is special enough to be the result but not special enough to be
>> the
>> test.
>>
>> Secondly, this approach masks errors within code executed implicitly as
>> part of
>> the expression. For ``?.``, any ``AttributeError`` within a property or
>> ``__getattr__`` implementation would be hidden, and similarly for
>> ``?[]`` and
>> ``__getitem__`` implementations.
>>
>> Similarly, simple typing errors such as ``{}?.ietms()`` could go
>> unnoticed.
>>
>> Existing conventions for handling these kinds of errors in the form of the
>> ``getattr`` builtin and the ``.get(key, default)`` method pattern
>> established by
>> ``dict`` show that it is already possible to explicitly use this
>> behaviour.
>>
>> As this approach would hide errors in code, it is rejected.
>>
>> ``None``-aware Function Call
>> ----------------------------
>>
>> The ``None``-aware syntax applies to attribute and index access, so it
>> seems
>> natural to ask if it should also apply to function invocation syntax. It
>> might
>> be written as ``foo?()``, where ``foo`` is only called if it is not None.
>>
>> This has been deferred on the basis of the proposed operators being
>> intended
>> to aid traversal of partially populated hierarchical data structures,
>> *not*
>> for traversal of arbitrary class hierarchies. This is reflected in the
>> fact
>> that none of the other mainstream languages that already offer this syntax
>> have found it worthwhile to support a similar syntax for optional function
>> invocations.
>>
>> A workaround similar to that used by C# would be to write
>> ``maybe_none?.__call__(arguments)``. If the callable is ``None``, the
>> expression will not be evaluated. (The C# equivalent uses ``?.Invoke()``
>> on its
>> callable type.)
>>
>> ``?`` Unary Postfix Operator
>> ----------------------------
>>
>> To generalize the ``None``-aware behavior and limit the number of new
>> operators
>> introduced, a unary, postfix operator spelled ``?`` was suggested. The
>> idea is
>> that ``?`` might return a special object that could would override dunder
>> methods that return ``self``. For example, ``foo?`` would evaluate to
>> ``foo`` if
>> it is not ``None``, otherwise it would evaluate to an instance of
>> ``NoneQuestion``::
>>
>>      class NoneQuestion():
>>          def __call__(self, *args, **kwargs):
>>              return self
>>
>>          def __getattr__(self, name):
>>              return self
>>
>>          def __getitem__(self, key):
>>              return self
>>
>>
>> With this new operator and new type, an expression like ``foo?.bar[baz]``
>> evaluates to ``NoneQuestion`` if ``foo`` is None. This is a nifty
>> generalization, but it's difficult to use in practice since most
>> existing code
>> won't know what ``NoneQuestion`` is.
>>
>> Going back to one of the motivating examples above, consider the
>> following::
>>
>>      >>> import json
>>      >>> created = None
>>      >>> json.dumps({'created': created?.isoformat()})``
>>
>> The JSON serializer does not know how to serialize ``NoneQuestion``, nor
>> will
>> any other API. This proposal actually requires *lots of specialized logic*
>> throughout the standard library and any third party library.
>>
>> At the same time, the ``?`` operator may also be **too general**, in the
>> sense
>> that it can be combined with any other operator. What should the following
>> expressions mean?::
>>
>>      >>> x? + 1
>>      >>> x? -= 1
>>      >>> x? == 1
>>      >>> ~x?
>>
>> This degree of generalization is not useful. The operators actually
>> proposed
>> herein are intentionally limited to a few operators that are expected to
>> make it
>> easier to write common code patterns.
>>
>> Built-in ``maybe``
>> ------------------
>>
>> Haskell has a concept called `Maybe <https://wiki.haskell.org/Maybe>`_
>> that
>> encapsulates the idea of an optional value without relying on any special
>> keyword (e.g. ``null``) or any special instance (e.g. ``None``). In
>> Haskell, the
>> purpose of ``Maybe`` is to avoid separate handling of "something" and
>> nothing".
>>
>> A Python package called `pymaybe <https://pypi.org/p/pymaybe/>`_
>> provides a
>> rough approximation. The documentation shows the following example::
>>
>>      >>> maybe('VALUE').lower()
>>      'value'
>>
>>      >>> maybe(None).invalid().method().or_else('unknown')
>>      'unknown'
>>
>> The function ``maybe()`` returns either a ``Something`` instance or a
>> ``Nothing`` instance. Similar to the unary postfix operator described in
>> the
>> previous section, ``Nothing`` overrides dunder methods in order to allow
>> chaining on a missing value.
>>
>> Note that ``or_else()`` is eventually required to retrieve the
>> underlying value
>> from ``pymaybe``'s wrappers. Furthermore, ``pymaybe`` does not short
>> circuit any
>> evaluation. Although ``pymaybe`` has some strengths and may be useful in
>> its own
>> right, it also demonstrates why a pure Python implementation of
>> coalescing is
>> not nearly as powerful as support built into the language.
>>
>> The idea of adding a builtin ``maybe`` type to enable this scenario is
>> rejected.
>>
>> Just use a conditional expression
>> ---------------------------------
>>
>> Another common way to initialize default values is to use the ternary
>> operator.
>> Here is an excerpt from the popular `Requests package
>> <
>> https://github.com/kennethreitz/requests/blob/14a555ac716866678bf17e43e23230d81
>> a8149f5/requests/models.py#L212
>> <https://github.com/kennethreitz/requests/blob/14a555ac716866678bf17e43e23230d81a8149f5/requests/models.py#L212>
>> >`_::
>>
>>      data = [] if data is None else data
>>      files = [] if files is None else files
>>      headers = {} if headers is None else headers
>>      params = {} if params is None else params
>>      hooks = {} if hooks is None else hooks
>>
>> This particular formulation has the undesirable effect of putting the
>> operands
>> in an unintuitive order: the brain thinks, "use ``data`` if possible and
>> use
>> ``[]`` as a fallback," but the code puts the fallback *before* the
>> preferred
>> value.
>>
>> The author of this package could have written it like this instead::
>>
>>      data = data if data is not None else []
>>      files = files if files is not None else []
>>      headers = headers if headers is not None else {}
>>      params = params if params is not None else {}
>>      hooks = hooks if hooks is not None else {}
>>
>> This ordering of the operands is more intuitive, but it requires 4 extra
>> characters (for "not "). It also highlights the repetition of identifiers:
>> ``data if data``, ``files if files``, etc.
>>
>> When written using the ``None`` coalescing operator, the sample reads::
>>
>>      data = data ?? []
>>      files = files ?? []
>>      headers = headers ?? {}
>>      params = params ?? {}
>>      hooks = hooks ?? {}
>>
>>
>> References
>> ==========
>>
>> .. [1] C# Reference: Operators
>>     (https://msdn.microsoft.com/en-us/library/6a71f45d.aspx)
>>
>> .. [2] A Tour of the Dart Language: Operators
>>     (
>> https://www.dartlang.org/docs/dart-up-and-running/ch02.html#operators)
>>
>> .. [3] Associated scripts
>>     (https://github.com/python/peps/tree/master/pep-0505/)
>>
>> Copyright
>> =========
>>
>> This document has been placed in the public domain.
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180718/8ca21c09/attachment-0001.html>

From g.rodola at gmail.com  Wed Jul 18 21:05:00 2018
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Thu, 19 Jul 2018 03:05:00 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPTjJmqPS_wiFTJaJ8YUKJfWsNcCX=h9AQ1EmWFv6Xf9Wes4tA@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAFYqXL8GsTyAHEkrQqSDpv_+bCcJAVv2j-0d_zwNuLLfK7Cuuw@mail.gmail.com>
 <CAPTjJmqPS_wiFTJaJ8YUKJfWsNcCX=h9AQ1EmWFv6Xf9Wes4tA@mail.gmail.com>
Message-ID: <CAFYqXL9oi7L+Dg0-n2AoEYz+orio8Yz=DuaHjAEQq5gAiKj59Q@mail.gmail.com>

On Thu, Jul 19, 2018 at 2:06 AM Chris Angelico <rosuav at gmail.com> wrote:
> > With all due respect (and I am sorry for being ?vocal? about a PEP once
> > again) I find this simply ugly. To me this basically doesn?t look like
> > python anymore, so a strong -1 from me.
>
> I'd love to hear an explanation of WHY this doesn't look like Python
> any more.

Because it looks like Perl.

> For instance, is the + operator somehow wrong for Python,
> and it should have been the word "add"?

The meaning of "+" is obvious to anybody, including non programmers.
"?" is arbitrary so you cannot guess what it does or means, especially
when it can be spelled in so many different forms (?, ??, a?.b, ??=,
...), each form requiring a different mental replacement. Use this and
:= on the same line and the reader will practically be reading another
language.

>  [ please trim quotes, you just quoted the entire PEP in your post ]

Ouch! Sorry about that.

-- 
Giampaolo - http://grodola.blogspot.com

From graham.gott.software at gmail.com  Wed Jul 18 23:19:27 2018
From: graham.gott.software at gmail.com (Graham Gott)
Date: Wed, 18 Jul 2018 22:19:27 -0500
Subject: [Python-ideas] Revisiting str.rreplace()
Message-ID: <CAN83G44dWT=+t68DS=abk29eM-WOQm3ZjOTDFzBNOVLtgr8xGA@mail.gmail.com>

This was previously proposed here in 2014 <
https://mail.python.org/pipermail/python-ideas/2014-January/025091.html>,
but the discussion fizzled out. To me, str.rreplace() is an obvious and
necessary complement to str.replace(), just as str.rsplit() is a complement
to str.split(). It would make Python closer to the goal of the Zen of
Python: "There should be one-- and preferably only one --obvious way to do
it." To support its usefulness, this question has been asked on Stack
Overflow a number of times, (<https://stackoverflow.com/q/2556108>, <
https://stackoverflow.com/q/14496006>, <https://stackoverflow.com/q/9943504>)
with more than a trivial number of votes for the answers (>100 for the top
answer of the first two questions, and >50 for the third, not necessarily
mutually exclusive voters, but probably largely; even assuming the worst,
>100 is nothing to scoff at). While anonymous Stack Overflow votes are not
necessarily the best arbiter on what's a good idea, they at least show
interest.

My proposed behavior (though probably not the implementation details) would
be as follows (though the implementation would obviously be as a string
method, not a function, with 'self' replacing 's'):



def rreplace(s, old, new, count=-1):
    '''
    Return a copy with all occurrences of substring old replaced by new.

      count
        Maximum number of occurrences to replace.
        -1 (the default value) means replace all occurrences

    Substrings are replaced starting at the end of the string and working
to the front.
    '''
    return new.join(s.rsplit(old, count))



Addressing some (paraphrased) issues from the previous discussion, in the
form of an FAQ:

rreplace() looks weird.
>

This is maintaining consistency with the trend of 'r' prefixes for
'reverse' methods. Introducing 'reverse_' as a prefix would be
inconsistent, but worse, it would also encourage backwards incompatible
changes into existing methods. I think such a prefix naming change warrants
its own separate discussion.

There are already ways to do this. Why should we add another string method?
>

My main motivation is having one clear and efficient way to do this. I
explain this in greater detail in my introduction above.

The default of count=-1 has the same behavior as str.replace(), right?
>

Actually, it doesn't have the same behavior, as MRAB pointed out in the
previous discussion <
https://mail.python.org/pipermail/python-ideas/2014-January/025102.html>.
The default of -1 also keeps the syntax consistent with str.rsplit()'s
syntax.

If we're adding this, shouldn't we also add bytes.rreplace,
> bytearray.rreplace, bytearray.rremove, tuple.rindex, list.rindex,
> list.rremove?
>

Honestly, I don't know. I would prefer not to dilute this discussion too
much, or start making a slippery slope argument, but if it's directly
relevant, I think any thoughts are welcome.

Couldn't we just add a traverse_backwards parameter to str.replace()? In
> fact, why don't we just get rid of str.rfind() and str.rindex() entirely
> and just add new parameters to str.find() and str.index()?
>

I think Steven D'Aprano explained this well in the previous discussion
here: <
https://mail.python.org/pipermail/python-ideas/2014-January/025132.html>.
And addressing counterarguments here: <
https://mail.python.org/pipermail/python-ideas/2014-January/025135.html>.
Basically, different functions for different purposes make the purpose
clearer (no confusing/complicated parameter names and functions), and
str.rreplace() and str.replace() are usually going to be used in situations
where the traversal direction is known at edit time, so there's no need to
make the method determine the direction at runtime.



Thoughts? Support/oppose?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180718/acd0476f/attachment.html>

From tim.peters at gmail.com  Thu Jul 19 01:48:20 2018
From: tim.peters at gmail.com (Tim Peters)
Date: Thu, 19 Jul 2018 00:48:20 -0500
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
Message-ID: <CAExdVNkj1g00rdLjzyo5-Uhx2Yfgw0cS2CO9wa1Pvt07_eZx0g@mail.gmail.com>

[Steve Dower <steve.dower at python.org>]

> ...

* The "``None``-aware attribute access" operator ``?.`` evaluates the
> complete expression if the left hand side evaluates to a value that is not
> ``None``
>

And if the LHS does evaluate to `None` ...?  I'll assume the result is also
`None` then.


> ...
>  From ``inspect.py``::
>
>      for base in object.__bases__:
>          for name in getattr(base, "__abstractmethods__", ()):
>              value = getattr(object, name, None)
>              if getattr(value, "__isabstractmethod__", False):
>                  return True
>
> After updating to use the ``?.`` operator (and deliberately not
> converting to use ``any()``)::
>
>      for base in object.__bases__:
>          for name in base?.__abstractmethods__ ?? ():
>              if object?.name?.__isabstractmethod__:
>                  return True
>

I got lost on the `for` here.  The part following `in`:

    for name in getattr(base, "__abstractmethods__", ()):

looks in `base` (regardless of whether `base` is `None`) for an attribute
named "_abstractmethods__"..  If such an attribute exists, the value of the
attribute is returned (`None` or not).  Else an AttributeError is swallowed
and `()` is returned.  It's hard to see how

         for name in base?.__abstractmethods__ ?? ():

does the same.  If `base` itself is `None`, I guess it returns `()`, or
if  `base` has an "_abstractmethods__" attribute then the value of that
attribute is returned - unless its value is None, in which case `()` is
again returned.  But if `base` is not `None` and the attribute does not
exist, doesn't this raise AttributeError?  The later "Exception-aware
operators" section seemed to explicitly reject the idea that `?.` and `?[]`
would suppress AttributeError and/or TypeError.

In short, the original getattr() didn't care at all whether `base` was
`None`, or whether the value of its "__abstractmethods__" attribute was
`None`, but cared a whole lot about whether that attribute exists.  I just
can't see how the updated code matches that in any of those respects.

Ignoring that and pressing on, I suffer the same kind of confusions on the
`if` part.  What am I missing?  For example, do these operators swallow
exceptions after all?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/04b10ec2/attachment-0001.html>

From pfreixes at gmail.com  Thu Jul 19 02:32:22 2018
From: pfreixes at gmail.com (Pau Freixes)
Date: Thu, 19 Jul 2018 08:32:22 +0200
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <CAPJVwBk2RSF6=2cQtRHBF=XCPcaGDARNpXdAtw_rcvBendr+5w@mail.gmail.com>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>
 <CAPTjJmqKGhNEcrAuzKSzcUW_kvdmPqL9_9CMyOUzcrBs=UdcEg@mail.gmail.com>
 <CAPJVwBkpLw_qRirCFq4PK6n5zLVxc=t=_bf_SZRXLsSnrMVECQ@mail.gmail.com>
 <CAPTjJmrt83c96JQ3PY2oZEm68ZMbG_JBwaJ1XpcyzqRnEcks9A@mail.gmail.com>
 <6a5b228a-200a-b2c7-6067-df2b0022ae42@mrabarnett.plus.com>
 <20180716190735.2908e86d@fsol>
 <CALFfu7BFLSSRf7_mVUP3Rh4tAg1ae1cawc9KFF66EiBc2Hfn7g@mail.gmail.com>
 <890D0D1D-7166-43B5-8BC0-8B2177B3BAE0@barrys-emacs.org>
 <CALFfu7CC=2ZYFoYxf_ijU_V43F_g90RAtJZr=+nT5vgRwP4Q5g@mail.gmail.com>
 <9A7ACD82-CC6B-4C22-B3FC-B28963459046@barrys-emacs.org>
 <CALFfu7BCH3Fmht5REXxk-NTW66ZMZhPoQQfCZDZGNyFn_+DLKQ@mail.gmail.com>
 <CAOOa=pNh_C=4Ysxf4sH6VS+fc+Wv4YO62QSU9xipPD4UV8ODyA@mail.gmail.com>
 <CAPJVwBk2RSF6=2cQtRHBF=XCPcaGDARNpXdAtw_rcvBendr+5w@mail.gmail.com>
Message-ID: <CA+ULCcFL2FHfoAg9YZtfHKwQmYxqjMaMMSfsm6Hh0WtssoZsvA@mail.gmail.com>

Hi,

>
> Note that this everything you said here also exactly describes the
> programming model for the existing 'multiprocessing' module:
> "structured clone" is equivalent to how multiprocessing uses pickle to
> transfer arbitrary objects, or you can use multiprocessing.Array to
> get a shared view on raw "C"-style data.
>

That's good. If finally, CPython can provide a pattern plus an API
that has similitudes with the existing ones will make the adoption
less friction.

From jfine2358 at gmail.com  Thu Jul 19 02:33:13 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Thu, 19 Jul 2018 07:33:13 +0100
Subject: [Python-ideas] Multi-core reference count garbage collection
Message-ID: <CALD=Yf-Ht4uDMHkvXp8i+xesw=e9zw4i+nAuRJFMVppNSYrboA@mail.gmail.com>

Hi

Based on other people's work (including in particular talks by Larry
Hastings) and my own thinking, I've come up with a scheme for multi-core
reference count garbage collection. I think it works, and much or all of it
comes from others. It might even be in the literature. Of course, if it's
wrong then the debit goes to me.

I'll explain it in instalments. Please let me know what you think, as we go
along.

The basic ideas of reference counting garbage collection are:

1. Each allocated piece of memory is given an ID (which is often its
address in memory).
2. For each ID, the system keeps a count of how many references (to that
piece of memory).
3. The count starts at ONE.
4. Processes increment and decrement the count, to keep the count correct.
5. When the count reaches zero, the piece of memory is garbage collected.
6. The previous step might result in further reference count decrements.

The simplest possible garbage collection scheme is to do nothing (and let
the garbage build up). In other words, allocate on a stack, and never free
memory. This has very good performance, until memory is exhausted. It is
also conceptually useful. I'll call it the do-nothing garbage collection
scheme.

Suppose for definiteness we have a 6-core CPU, with 5 worker processes and
one garbage collection (GC) process. The worker processes will do as little
as possible, consistent with not running out of memory. To achieve this:

1. Each worker process with have two GC buffers, one for increment and the
other for decrement.
2. For increment, the process will append the ID to the increment buffer
(for the process). And similarly for decrement.
3. If the buffer to be appended to is full, the worker process will
(a) set a buffer-full flag
(b) pause until the buffer-full flag has been cleared
(c) and then do what it was previously blocked from doing

I call any such scheme BUFFERED multi-core reference count garbage
collection. The worker processes know nothing about how garbage collection
is managed. Instead, they pass over to the GC process sufficient
information to allow it to manage the garbage.

This is now a good place to pause. We've specified what it is that the
worker processes should do, regarding garbage collection. And we've passed
the problem on to the garbage collection process. If that process does
nothing, we have refactored the do-nothing garbage collection scheme.

Expect another instalment tomorrow.

with best regards

Jonathan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/bfbf2f7f/attachment.html>

From waksman at gmail.com  Thu Jul 19 03:38:58 2018
From: waksman at gmail.com (George Leslie-Waksman)
Date: Thu, 19 Jul 2018 00:38:58 -0700
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAExdVNkj1g00rdLjzyo5-Uhx2Yfgw0cS2CO9wa1Pvt07_eZx0g@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAExdVNkj1g00rdLjzyo5-Uhx2Yfgw0cS2CO9wa1Pvt07_eZx0g@mail.gmail.com>
Message-ID: <CACef2ak_UmF+NWAJ9KMzJLvBe51TKe544bGvNKjLjvAgc4SHMw@mail.gmail.com>

I am rather fond of the idea of null-coalescing, at the very least, for
mutable default values:

def foo(a=None):
    a ??= []
    ...

but I worry about the code messes we will run into with some of the other
options.

Woe be unto anyone forced to understand the behavior of:

thing?.attr?[key]?.subattr ?? 127

What if we added the Elvis operator "?:" for null coalescing and left the
rest for future consideration


On Wed, Jul 18, 2018 at 10:49 PM Tim Peters <tim.peters at gmail.com> wrote:

> [Steve Dower <steve.dower at python.org>]
>
>> ...
>
> * The "``None``-aware attribute access" operator ``?.`` evaluates the
>> complete expression if the left hand side evaluates to a value that is
>> not
>> ``None``
>>
>
> And if the LHS does evaluate to `None` ...?  I'll assume the result is
> also `None` then.
>
>
>> ...
>
>
>>  From ``inspect.py``::
>>
>>      for base in object.__bases__:
>>          for name in getattr(base, "__abstractmethods__", ()):
>>              value = getattr(object, name, None)
>>              if getattr(value, "__isabstractmethod__", False):
>>                  return True
>>
>> After updating to use the ``?.`` operator (and deliberately not
>> converting to use ``any()``)::
>>
>>      for base in object.__bases__:
>>          for name in base?.__abstractmethods__ ?? ():
>>              if object?.name?.__isabstractmethod__:
>>                  return True
>>
>
> I got lost on the `for` here.  The part following `in`:
>
>     for name in getattr(base, "__abstractmethods__", ()):
>
> looks in `base` (regardless of whether `base` is `None`) for an attribute
> named "_abstractmethods__"..  If such an attribute exists, the value of
> the attribute is returned (`None` or not).  Else an AttributeError is
> swallowed and `()` is returned.  It's hard to see how
>
>
>          for name in base?.__abstractmethods__ ?? ():
>
> does the same.  If `base` itself is `None`, I guess it returns `()`, or
> if  `base` has an "_abstractmethods__" attribute then the value of that
> attribute is returned - unless its value is None, in which case `()` is
> again returned.  But if `base` is not `None` and the attribute does not
> exist, doesn't this raise AttributeError?  The later "Exception-aware
> operators" section seemed to explicitly reject the idea that `?.` and `?[]`
> would suppress AttributeError and/or TypeError.
>
> In short, the original getattr() didn't care at all whether `base` was
> `None`, or whether the value of its "__abstractmethods__" attribute was
> `None`, but cared a whole lot about whether that attribute exists.  I just
> can't see how the updated code matches that in any of those respects.
>
> Ignoring that and pressing on, I suffer the same kind of confusions on the
> `if` part.  What am I missing?  For example, do these operators swallow
> exceptions after all?
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/82ec2c0a/attachment-0001.html>

From contact at brice.xyz  Thu Jul 19 03:45:05 2018
From: contact at brice.xyz (Brice Parent)
Date: Thu, 19 Jul 2018 09:45:05 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
Message-ID: <712febdd-b410-7f5e-7a96-211c6a6bd7bf@brice.xyz>

Hi,
I think this breaks one of the most important aspect which makes Python 
more and more popular every year: its readability. Until now, you didn't 
need to know the language very well to have some understanding about 
what a script did. Some parts of it could be a bit complicated (like 
comprehensions, which we only allow in unit tests and small utilities 
where I work, exactly for this reason) or non-obvious (like the new := 
operator, but even if you don't get exactly what it does, it's not 
really cryptic either. It's still some-kind-of-assignment).
We've been teaching for years that there is virtually no cost in adding 
a few simple lines of code from times to times, and that it has to be 
done to improve readability, while this proposal's goal seems to be the 
opposite (make the code more compact at the cost of readability), which 
complicates the understanding of the whole language to save a few `if x 
is None:`. To me, it goes the opposite of most of the first lines of the 
zen of Python.

Where I work, we update most of our projects to use new Python versions 
almost immediately when a new one is out and the libs we use are 
compatible. This would probably change that; we would likely wait for a 
few years, the time for the feature to be used widely in other projects, 
and for us to have some real-life feedback to know what to do about it, 
like: are there some cases it really is justified? Does it make it 
harder/simpler to read/write python code for experts and non-experts? 
Should we add a rule to enforce its use, or to prevent it?

The biggest drawback of this, is that (if I understand it well), it may 
be done quite easily without any change to the language:

def first_set(*elements):? # please don't mind the name of the function, 
it's not the purpose here
 ??? """ Will return the first element that is not None """
 ??? for element in elements:
 ??????? if element is not None:
 ??????????? return element

 ??? raise AllNoneException()

first_set(3, 5)? # -> 3
first_set(None, 5)? # -> 5
first_set(None, None, 8, 10)? # -> 8
first_set(None, Car(model="sport")).buy()? # calling 
Car(model="sport").buy()
first_set(None, ["a", "b", "c"])[1]? # -> "b"
first_set(None, None)? # -> exception is raised

(note that such function could even accept a "rejected_values" kwarg, 
like `rejected_values=(None, [], "")`, just by replacing the `if` clause 
by `if element not in rejected_values:`)

I might have missed some implications, or even not understood the PEP at 
all, though!

- Brice





From songofacandy at gmail.com  Thu Jul 19 04:25:53 2018
From: songofacandy at gmail.com (INADA Naoki)
Date: Thu, 19 Jul 2018 17:25:53 +0900
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
Message-ID: <CAEfz+TwwYJ8gGq5KRiGzuHusGeMfs=fP+Q61_jOY6_rZ8naXLg@mail.gmail.com>

Honestly speaking, I don't want make Python syntax more complex.

But when comparing to accepted PEP 572, I think this PEP is useful
often enough.  And PEP 505 doesn't break border between
expression and statement unlike PEP 572.

Especially, ?? and ??= seems useful very often.
And `x ?? default` seems much more readable than
`x if x is not None else default` or `default if x is None else x`.

On the other hand, `?.` and `?[]` seems useful less often and more confusing.

When looking `spam?.egg`, people can expect `getattr(spam, 'egg', None)`
rather than `spam.egg if spam is not None else None`.

Since `?.` and `?[]` can't avoid AttributeError, IndexError and KeyError,
I think it's confusing and not useful enough compared with it's ugliness.

So my current position is +1 for `??` and `??=`, but -1 for others.

Regards,
-- 
INADA Naoki  <songofacandy at gmail.com>

From turnbull.stephen.fw at u.tsukuba.ac.jp  Thu Jul 19 04:32:48 2018
From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull)
Date: Thu, 19 Jul 2018 17:32:48 +0900
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
Message-ID: <23376.19632.386221.927225@turnbull.sk.tsukuba.ac.jp>

Chris Angelico writes later in thread:
 > On Thu, Jul 19, 2018 at 9:55 AM, Giampaolo Rodola' <g.rodola at gmail.com> wrote:

 > Personally, I'm +0 on this. It'd be a few small wins here and there,
 > nothing huge, and I could easily live without it; but it's something
 > that I know some people will love.

I am 100% in sync with the reasoning, but -0 on the PEP (and only that
high because the advocates are so passionate).

To be honest, code transformations like this

 >      class BaseUploadObject(object):
 >          def find_content_type(self, filename):
 >              ctype, encoding = mimetypes.guess_type(filename)
 >              if ctype is None:
 >                  return 'application/octet-stream'
 >              else:
 >                  return ctype

to this

 >      class BaseUploadObject(object):
 >          def find_content_type(self, filename):
 >              ctype, encoding = mimetypes.guess_type(filename)
 >              return ctype ?? 'application/octet-stream'

make me cringe.  Exactly one of two things is true:

1.  mimetypes.guess_type guarantees that the only falsie it will ever
    return is None, or

2.  it doesn't.

In case 1, "ctype or 'application/octet-stream'" ALSO does the right
thing.  In case 2, ONLY "ctype or 'application/octet-stream'" does the
right thing, as few callers of BaseUploadObject.find_content_type will
be prepared for "", (), [], or any variety of 0 as the return value.
This example is extreme in that by the nature of application/octet-
stream, any caller that can handle it at all will do a very reasonable
thing if find_content_type barfs up a falsie that is not None. 

In many cases, of course, it would be better to (eventually) raise an
exception when a falsie escapes the expression.  But in this
particular example, it's hard to imagine that is true: if the user's
expectation is violated, they'll complain, and then you can go debug.
In the meantime, the application doesn't crash and work gets done.

The prevalence of these cringe-worthy examples in advocates' posts are
why I'm basically - on the idea.

So I would like to see various examples of code where

1.  in the original code, treating a falsie that is not None the same
    way that None is treated is a detectable bug; and

2a. letting such falsies escape to raise exceptions somewhere else is
    a good idea (and how you could know that); or

2b. catching spurious falsies and handling them here is a better idea;
    or

2c. some falsies that are not None are legitimate.

That's three kinds of examples.  I suspect the 2b examples are going
to be pretty "meh", because you're saving little reader cognition or
code.  More important are the 2a examples, because I suspect that most
examples will fall into the same category as find_content_type: "or"
is as good as "??", and that "??" is sometimes an accident waiting to
happen due to falsie escapes (eg, when receiving objects from somebody
else's code who might not understand the contract that the only falsie
they should ever provide is None).

Steve


From solipsis at pitrou.net  Thu Jul 19 04:33:21 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 19 Jul 2018 10:33:21 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
Message-ID: <20180719103321.7c193820@fsol>


This is adding a whole range of new operators without enough of a use
case.  It is also making code harder to read, as evaluation can stop at
any of the "?*" operators.  And it looks like noise (or like Perl 6,
which is the same).

There is a use case I sympathize with: the argument-is-None case.  For
that I would suggest a simpler form:  "A else B" which would evaluate
to A if A is not None, otherwise to B (parentheses may be mandatory).

So e.g. one of the examples would read:

     def insort_right(a, x, lo=0, hi=None):
         # ...
         hi = hi else len(a)
         # ...


Regards

Antoine.


On Wed, 18 Jul 2018 10:43:36 -0700
Steve Dower <steve.dower at python.org> wrote:
> Possibly this is exactly the wrong time to propose the next big syntax 
> change, since we currently have nobody to declare on it, but since we're 
> likely to argue for a while anyway it probably can't hurt (and maybe 
> this will become the test PEP for whoever takes the reins?).
> 
> FWIW, Guido had previously indicated that he was generally favourable 
> towards most of this proposal, provided we could figure out coherent 
> semantics. Last time we tried, that didn't happen, so this time I've 
> made the semantics much more precise, have implemented and verified 
> them, and made much stronger statements about why we are proposing these.
> 
> Additional thanks to Mark Haase for writing most of the PEP. All the 
> fair and balanced parts are his - all the overly strong opinions are mine.
> 
> Also thanks to Nick Coghlan for writing PEPs 531 and 532 last time we 
> went through this - if you're unhappy with "None" being treated as a 
> special kind of value, I recommend reading those before you start 
> repeating them.
> 
> There is a formatted version of this PEP at 
> https://www.python.org/dev/peps/pep-0505/
> 
> My current implementation is at 
> https://github.com/zooba/cpython/tree/pep-505 (though I'm considering 
> removing some of the new opcodes I added and just generating more 
> complex code - in any case, let's get hung up on the proposal rather 
> than the implementation :) )
> 
> Let the discussions begin!
> 
> ---
> 
> PEP: 505
> Title: None-aware operators
> Version: $Revision$
> Last-Modified: $Date$
> Author: Mark E. Haase <mehaase at gmail.com>, Steve Dower 
> <steve.dower at python.org>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 18-Sep-2015
> Python-Version: 3.8
> 
> Abstract
> ========
> 
> Several modern programming languages have so-called "``null``-coalescing" or
> "``null``- aware" operators, including C# [1]_, Dart [2]_, Perl, Swift, 
> and PHP
> (starting in version 7). These operators provide syntactic sugar for common
> patterns involving null references.
> 
> * The "``null``-coalescing" operator is a binary operator that returns 
> its left
>    operand if it is not ``null``. Otherwise it returns its right operand.
> * The "``null``-aware member access" operator accesses an instance 
> member only
>    if that instance is non-``null``. Otherwise it returns ``null``. 
> (This is also
>    called a "safe navigation" operator.)
> * The "``null``-aware index access" operator accesses an element of a 
> collection
>    only if that collection is non-``null``. Otherwise it returns 
> ``null``. (This
>    is another type of "safe navigation" operator.)
> 
> This PEP proposes three ``None``-aware operators for Python, based on the
> definitions and other language's implementations of those above. 
> Specifically:
> 
> * The "``None`` coalescing`` binary operator ``??`` returns the left 
> hand side
>    if it evaluates to a value that is not ``None``, or else it evaluates and
>    returns the right hand side. A coalescing ``??=`` augmented assignment
>    operator is included.
> * The "``None``-aware attribute access" operator ``?.`` evaluates the 
> complete
>    expression if the left hand side evaluates to a value that is not 
> ``None``
> * The "``None``-aware indexing" operator ``?[]`` evaluates the complete
>    expression if the left hand site evaluates to a value that is not 
> ``None``
> 
> Syntax and Semantics
> ====================
> 
> Specialness of ``None``
> -----------------------
> 
> The ``None`` object denotes the lack of a value. For the purposes of these
> operators, the lack of a value indicates that the remainder of the 
> expression
> also lacks a value and should not be evaluated.
> 
> A rejected proposal was to treat any value that evaluates to false in a
> Boolean context as not having a value. However, the purpose of these 
> operators
> is to propagate the "lack of value" state, rather that the "false" state.
> 
> Some argue that this makes ``None`` special. We contend that ``None`` is
> already special, and that using it as both the test and the result of these
> operators does not change the existing semantics in any way.
> 
> See the `Rejected Ideas`_ section for discussion on the rejected approaches.
> 
> Grammar changes
> ---------------
> 
> The following rules of the Python grammar are updated to read::
> 
>      augassign: ('+=' | '-=' | '*=' | '@=' | '/=' | '%=' | '&=' | '|=' | 
> '^=' |
>                  '<<=' | '>>=' | '**=' | '//=' | '??=')
> 
>      power: coalesce ['**' factor]
>      coalesce: atom_expr ['??' factor]
>      atom_expr: ['await'] atom trailer*
>      trailer: ('(' [arglist] ')' |
>                '[' subscriptlist ']' |
>                '?[' subscriptlist ']' |
>                '.' NAME |
>                '?.' NAME)
> 
> Inserting the ``coalesce`` rule in this location ensures that expressions
> resulting in ``None`` are natuarlly coalesced before they are used in
> operations that would typically raise ``TypeError``. Like ``and`` and ``or``
> the right-hand expression is not evaluated until the left-hand side is
> determined to be ``None``. For example::
> 
>      a, b = None, None
>      def c(): return None
>      def ex(): raise Exception()
> 
>      (a ?? 2 ** b ?? 3) == a ?? (2 ** (b ?? 3))
>      (a * b ?? c // d) == a * (b ?? c) // d
>      (a ?? True and b ?? False) == (a ?? True) and (b ?? False)
>      (c() ?? c() ?? True) == True
>      (True ?? ex()) == True
>      (c ?? ex)() == c()
> 
> Augmented coalescing assignment only rebinds the name if its current 
> value is
> ``None``. If the target name already has a value, the right-hand side is not
> evaluated. For example::
> 
>      a = None
>      b = ''
>      c = 0
> 
>      a ??= 'value'
>      b ??= undefined_name
>      c ??= shutil.rmtree('/')    # don't try this at home, kids
> 
>      assert a == 'value'
>      assert b == ''
>      assert c == '0' and any(os.scandir('/'))
> 
> Adding new trailers for the other ``None``-aware operators ensures that they
> may be used in all valid locations for the existing equivalent operators,
> including as part of an assignment target (more details below). As the 
> existing
> evaluation rules are not directly embedded in the grammar, we specify the
> required changes here.
> 
> Assume that the ``atom`` is always successfully evaluated. Each 
> ``trailer`` is
> then evaluated from left to right, applying its own parameter (either its
> arguments, subscripts or attribute name) to produce the value for the next
> ``trailer``. Finally, if present, ``await`` is applied.
> 
> For example, ``await a.b(c).d[e]`` is currently parsed as
> ``['await', 'a', '.b', '(c)', '.d', '[e]']`` and evaluated::
> 
>      _v = a
>      _v = _v.b
>      _v = _v(c)
>      _v = _v.d
>      _v = _v[e]
>      await _v
> 
> When a ``None``-aware operator is present, the left-to-right evaluation 
> may be
> short-circuited. For example, ``await a?.b(c).d?[e]`` is evaluated::
> 
>      _v = a
>      if _v is not None:
>          _v = _v.b
>          _v = _v(c)
>          _v = _v.d
>          if _v is not None:
>              _v = _v[e]
>      await _v
> 
> .. note::
>      ``await`` will almost certainly fail in this context, as it would in
>      the case where code attempts ``await None``. We are not proposing 
> to add a
>      ``None``-aware ``await`` keyword here, and merely include it in this
>      example for completeness of the specification, since the ``atom_expr``
>      grammar rule includes the keyword. If it were in its own rule, we 
> would have
>      never mentioned it.
> 
> Parenthesised expressions are handled by the ``atom`` rule (not shown 
> above),
> which will implicitly terminate the short-circuiting behaviour of the above
> transformation. For example, ``(a?.b ?? c).d?.e`` is evaluated as::
> 
>      # a?.b
>      _v = a
>      if _v is not None:
>          _v = _v.b
> 
>      # ... ?? c
>      if _v is None:
>          _v = c
> 
>      # (...).d?.e
>      _v = _v.d
>      if _v is not None:
>          _v = _v.e
> 
> When used as an assignment target, the ``None``-aware operations may only be
> used in a "load" context. That is, ``a?.b = 1`` and ``a?[b] = 1`` will raise
> ``SyntaxError``. Use earlier in the expression (``a?.b.c = 1``) is 
> permitted,
> though unlikely to be useful unless combined with a coalescing operation::
> 
>      (a?.b ?? d).c = 1
> 
> 
> Examples
> ========
> 
> This section presents some examples of common ``None`` patterns and 
> shows what
> conversion to use ``None``-aware operators may look like.
> 
> Standard Library
> ----------------
> 
> Using the ``find-pep505.py`` script[3]_ an analysis of the Python 3.7 
> standard
> library discovered up to 678 code snippets that could be replaced with 
> use of
> one of the ``None``-aware operators::
> 
>      $ find /usr/lib/python3.7 -name '*.py' | xargs python3.7 find-pep505.py
>      <snip>
>      Total None-coalescing `if` blocks: 449
>      Total [possible] None-coalescing `or`: 120
>      Total None-coalescing ternaries: 27
>      Total Safe navigation `and`: 13
>      Total Safe navigation `if` blocks: 61
>      Total Safe navigation ternaries: 8
> 
> Some of these are shown below as examples before and after converting to 
> use the
> new operators.
> 
>  From ``bisect.py``::
> 
>      def insort_right(a, x, lo=0, hi=None):
>          # ...
>          if hi is None:
>              hi = len(a)
>          # ...
> 
> After updating to use the ``??=`` augmented assignment statement::
> 
>      def insort_right(a, x, lo=0, hi=None):
>          # ...
>          hi ??= len(a)
>          # ...
> 
>  From ``calendar.py``::
> 
>      encoding = options.encoding
>      if encoding is None:
>          encoding = sys.getdefaultencoding()
>      optdict = dict(encoding=encoding, css=options.css)
> 
> After updating to use the ``??`` operator::
> 
>      optdict = dict(encoding=encoding ?? sys.getdefaultencoding(),
>                     css=options.css)
> 
>  From ``dis.py``::
> 
>      def _get_const_info(const_index, const_list):
>          argval = const_index
>          if const_list is not None:
>              argval = const_list[const_index]
>          return argval, repr(argval)
> 
> After updating to use the ``?[]`` and ``??`` operators::
> 
>      def _get_const_info(const_index, const_list):
>          argval = const_list?[const_index] ?? const_index
>          return argval, repr(argval)
> 
>  From ``inspect.py``::
> 
>      for base in object.__bases__:
>          for name in getattr(base, "__abstractmethods__", ()):
>              value = getattr(object, name, None)
>              if getattr(value, "__isabstractmethod__", False):
>                  return True
> 
> After updating to use the ``?.`` operator (and deliberately not 
> converting to
> use ``any()``)::
> 
>      for base in object.__bases__:
>          for name in base?.__abstractmethods__ ?? ():
>              if object?.name?.__isabstractmethod__:
>                  return True
> 
>  From ``os.py``::
> 
>      if entry.is_dir():
>          dirs.append(name)
>          if entries is not None:
>              entries.append(entry)
>      else:
>          nondirs.append(name)
> 
> After updating to use the ``?.`` operator::
> 
>      if entry.is_dir():
>          dirs.append(name)
>          entries?.append(entry)
>      else:
>          nondirs.append(name)
> 
> 
> jsonify
> -------
> 
> This example is from a Python web crawler that uses the Flask framework 
> as its
> front-end. This function retrieves information about a web site from a SQL
> database and formats it as JSON to send to an HTTP client::
> 
>      class SiteView(FlaskView):
>          @route('/site/<id_>', methods=['GET'])
>          def get_site(self, id_):
>              site = db.query('site_table').find(id_)
> 
>              return jsonify(
>                  first_seen=site.first_seen.isoformat() if 
> site.first_seen is not None else None,
>                  id=site.id,
>                  is_active=site.is_active,
>                  last_seen=site.last_seen.isoformat() if site.last_seen 
> is not None else None,
>                  url=site.url.rstrip('/')
>              )
> 
> Both ``first_seen`` and ``last_seen`` are allowed to be ``null`` in the
> database, and they are also allowed to be ``null`` in the JSON response. 
> JSON
> does not have a native way to represent a ``datetime``, so the server's 
> contract
> states that any non-``null`` date is represented as an ISO-8601 string.
> 
> Without knowing the exact semantics of the ``first_seen`` and ``last_seen``
> attributes, it is impossible to know whether the attribute can be safely or
> performantly accessed multiple times.
> 
> One way to fix this code is to replace each conditional expression with an
> explicit value assignment and a full ``if``/``else`` block::
> 
>      class SiteView(FlaskView):
>          @route('/site/<id_>', methods=['GET'])
>          def get_site(self, id_):
>              site = db.query('site_table').find(id_)
> 
>              first_seen_dt = site.first_seen
>              if first_seen_dt is None:
>                  first_seen = None
>              else:
>                  first_seen = first_seen_dt.isoformat()
> 
>              last_seen_dt = site.last_seen
>              if last_seen_dt is None:
>                  last_seen = None
>              else:
>                  last_seen = last_seen_dt.isoformat()
> 
>              return jsonify(
>                  first_seen=first_seen,
>                  id=site.id,
>                  is_active=site.is_active,
>                  last_seen=last_seen,
>                  url=site.url.rstrip('/')
>              )
> 
> This adds ten lines of code and four new code paths to the function,
> dramatically increasing the apparent complexity. Rewriting using the
> ``None``-aware attribute operator results in shorter code with more clear
> intent::
> 
>      class SiteView(FlaskView):
>          @route('/site/<id_>', methods=['GET'])
>          def get_site(self, id_):
>              site = db.query('site_table').find(id_)
> 
>              return jsonify(
>                  first_seen=site.first_seen?.isoformat(),
>                  id=site.id,
>                  is_active=site.is_active,
>                  last_seen=site.last_seen?.isoformat(),
>                  url=site.url.rstrip('/')
>              )
> 
> Grab
> ----
> 
> The next example is from a Python scraping library called `Grab
> <https://github.com/lorien/grab/blob/4c95b18dcb0fa88eeca81f5643c0ebfb114bf728/gr
> ab/upload.py>`_::  
> 
>      class BaseUploadObject(object):
>          def find_content_type(self, filename):
>              ctype, encoding = mimetypes.guess_type(filename)
>              if ctype is None:
>                  return 'application/octet-stream'
>              else:
>                  return ctype
> 
>      class UploadContent(BaseUploadObject):
>          def __init__(self, content, filename=None, content_type=None):
>              self.content = content
>              if filename is None:
>                  self.filename = self.get_random_filename()
>              else:
>                  self.filename = filename
>              if content_type is None:
>                  self.content_type = self.find_content_type(self.filename)
>              else:
>                  self.content_type = content_type
> 
>      class UploadFile(BaseUploadObject):
>          def __init__(self, path, filename=None, content_type=None):
>              self.path = path
>              if filename is None:
>                  self.filename = os.path.split(path)[1]
>              else:
>                  self.filename = filename
>              if content_type is None:
>                  self.content_type = self.find_content_type(self.filename)
>              else:
>                  self.content_type = content_type
> 
> This example contains several good examples of needing to provide default
> values. Rewriting to use conditional expressions reduces the overall 
> lines of
> code, but does not necessarily improve readability::
> 
>      class BaseUploadObject(object):
>          def find_content_type(self, filename):
>              ctype, encoding = mimetypes.guess_type(filename)
>              return 'application/octet-stream' if ctype is None else ctype
> 
>      class UploadContent(BaseUploadObject):
>          def __init__(self, content, filename=None, content_type=None):
>              self.content = content
>              self.filename = (self.get_random_filename() if filename
>                  is None else filename)
>              self.content_type = (self.find_content_type(self.filename)
>                  if content_type is None else content_type)
> 
>      class UploadFile(BaseUploadObject):
>          def __init__(self, path, filename=None, content_type=None):
>              self.path = path
>              self.filename = (os.path.split(path)[1] if filename is
>                  None else filename)
>              self.content_type = (self.find_content_type(self.filename)
>                  if content_type is None else content_type)
> 
> The first ternary expression is tidy, but it reverses the intuitive order of
> the operands: it should return ``ctype`` if it has a value and use the 
> string
> literal as fallback. The other ternary expressions are unintuitive and so
> long that they must be wrapped. The overall readability is worsened, not
> improved.
> 
> Rewriting using the ``None`` coalescing operator::
> 
>      class BaseUploadObject(object):
>          def find_content_type(self, filename):
>              ctype, encoding = mimetypes.guess_type(filename)
>              return ctype ?? 'application/octet-stream'
> 
>      class UploadContent(BaseUploadObject):
>          def __init__(self, content, filename=None, content_type=None):
>              self.content = content
>              self.filename = filename ?? self.get_random_filename()
>              self.content_type = content_type ?? 
> self.find_content_type(self.filename)
> 
>      class UploadFile(BaseUploadObject):
>          def __init__(self, path, filename=None, content_type=None):
>              self.path = path
>              self.filename = filename ?? os.path.split(path)[1]
>              self.content_type = content_type ?? 
> self.find_content_type(self.filename)
> 
> This syntax has an intuitive ordering of the operands. In 
> ``find_content_type``,
> for example, the preferred value ``ctype`` appears before the fallback 
> value.
> The terseness of the syntax also makes for fewer lines of code and less 
> code to
> visually parse, and reading from left-to-right and top-to-bottom more 
> accurately
> follows the execution flow.
> 
> 
> Rejected Ideas
> ==============
> 
> The first three ideas in this section are oft-proposed alternatives to 
> treating
> ``None`` as special. For further background on why these are rejected, 
> see their
> treatment in `PEP 531 <https://www.python.org/dev/peps/pep-0531/>`_ and
> `PEP 532 <https://www.python.org/dev/peps/pep-0532/>`_ and the associated
> discussions.
> 
> No-Value Protocol
> -----------------
> 
> The operators could be generalised to user-defined types by defining a 
> protocol
> to indicate when a value represents "no value". Such a protocol may be a 
> dunder
> method ``__has_value__(self)` that returns ``True`` if the value should be
> treated as having a value, and ``False`` if the value should be treated 
> as no
> value.
> 
> With this generalization, ``object`` would implement a dunder method 
> equivalent
> to this::
> 
>      def __has_value__(self):
>          return True
> 
> ``NoneType`` would implement a dunder method equivalent to this::
> 
>      def __has_value__(self):
>          return False
> 
> In the specification section, all uses of ``x is None`` would be 
> replaced with
> ``not x.__has_value__()``.
> 
> This generalization would allow for domain-specific "no-value" objects to be
> coalesced just like ``None``. For example the ``pyasn1`` package has a type
> called ``Null`` that represents an ASN.1 ``null``::
> 
>      >>> from pyasn1.type import univ
>      >>> univ.Null() ?? univ.Integer(123)  
>      Integer(123)
> 
> Similarly, values such as ``math.nan`` and ``NotImplemented`` could be 
> treated
> as representing no value.
> 
> However, the "no-value" nature of these values is domain-specific, which 
> means
> they *should* be treated as a value by the language. For example,
> ``math.nan.imag`` is well defined (it's ``0.0``), and so short-circuiting
> ``math.nan?.imag`` to return ``math.nan`` would be incorrect.
> 
> As ``None`` is already defined by the language as being the value that
> represents "no value", and the current specification would not preclude
> switching to a protocol in the future (though changes to built-in 
> objects would
> not be compatible), this idea is rejected for now.
> 
> Boolean-aware operators
> -----------------------
> 
> This suggestion is fundamentally the same as adding a no-value protocol, 
> and so
> the discussion above also applies.
> 
> Similar behavior to the ``??`` operator can be achieved with an ``or``
> expression, however ``or`` checks whether its left operand is false-y 
> and not
> specifically ``None``. This approach is attractive, as it requires fewer 
> changes
> to the language, but ultimately does not solve the underlying problem 
> correctly.
> 
> Assuming the check is for truthiness rather than ``None``, there is no 
> longer a
> need for the ``??`` operator. However, applying this check to the ``?.`` and
> ``?[]`` operators prevents perfectly valid operations applying
> 
> Consider the following example, where ``get_log_list()`` may return either a
> list containing current log messages (potentially empty), or ``None`` if 
> logging
> is not enabled::
> 
>      lst = get_log_list()
>      lst?.append('A log message')
> 
> If ``?.`` is checking for true values rather than specifically ``None`` 
> and the
> log has not been initialized with any items, no item will ever be 
> appended. This
> violates the obvious intent of the code, which is to append an item. The
> ``append`` method is available on an empty list, as are all other list 
> methods,
> and there is no reason to assume that these members should not be used 
> because
> the list is presently empty.
> 
> Further, there is no sensible result to use in place of the expression. A
> normal ``lst.append`` returns ``None``, but under this idea 
> ``lst?.append`` may
> result in either ``[]`` or ``None``, depending on the value of ``lst``. 
> As with
> the examples in the previous section, this makes no sense.
> 
> As checking for truthiness rather than ``None`` results in apparently valid
> expressions no longer executing as intended, this idea is rejected.
> 
> Exception-aware operators
> -------------------------
> 
> Arguably, the reason to short-circuit an expression when ``None`` is 
> encountered
> is to avoid the ``AttributeError`` or ``TypeError`` that would be raised 
> under
> normal circumstances. As an alternative to testing for ``None``, the 
> ``?.`` and
> ``?[]`` operators could instead handle ``AttributeError`` and ``TypeError``
> raised by the operation and skip the remainder of the expression.
> 
> This produces a transformation for ``a?.b.c?.d.e`` similar to this::
> 
>      _v = a
>      try:
>          _v = _v.b
>      except AttributeError:
>          pass
>      else:
>          _v = _v.c
>          try:
>              _v = _v.d
>          except AttributeError:
>              pass
>          else:
>              _v = _v.e
> 
> One open question is which value should be returned as the expression 
> when an
> exception is handled. The above example simply leaves the partial 
> result, but
> this is not helpful for replacing with a default value. An alternative 
> would be
> to force the result to ``None``, which then raises the question as to why
> ``None`` is special enough to be the result but not special enough to be the
> test.
> 
> Secondly, this approach masks errors within code executed implicitly as 
> part of
> the expression. For ``?.``, any ``AttributeError`` within a property or
> ``__getattr__`` implementation would be hidden, and similarly for 
> ``?[]`` and
> ``__getitem__`` implementations.
> 
> Similarly, simple typing errors such as ``{}?.ietms()`` could go unnoticed.
> 
> Existing conventions for handling these kinds of errors in the form of the
> ``getattr`` builtin and the ``.get(key, default)`` method pattern 
> established by
> ``dict`` show that it is already possible to explicitly use this behaviour.
> 
> As this approach would hide errors in code, it is rejected.
> 
> ``None``-aware Function Call
> ----------------------------
> 
> The ``None``-aware syntax applies to attribute and index access, so it seems
> natural to ask if it should also apply to function invocation syntax. It 
> might
> be written as ``foo?()``, where ``foo`` is only called if it is not None.
> 
> This has been deferred on the basis of the proposed operators being intended
> to aid traversal of partially populated hierarchical data structures, *not*
> for traversal of arbitrary class hierarchies. This is reflected in the fact
> that none of the other mainstream languages that already offer this syntax
> have found it worthwhile to support a similar syntax for optional function
> invocations.
> 
> A workaround similar to that used by C# would be to write
> ``maybe_none?.__call__(arguments)``. If the callable is ``None``, the
> expression will not be evaluated. (The C# equivalent uses ``?.Invoke()`` 
> on its
> callable type.)
> 
> ``?`` Unary Postfix Operator
> ----------------------------
> 
> To generalize the ``None``-aware behavior and limit the number of new 
> operators
> introduced, a unary, postfix operator spelled ``?`` was suggested. The 
> idea is
> that ``?`` might return a special object that could would override dunder
> methods that return ``self``. For example, ``foo?`` would evaluate to 
> ``foo`` if
> it is not ``None``, otherwise it would evaluate to an instance of
> ``NoneQuestion``::
> 
>      class NoneQuestion():
>          def __call__(self, *args, **kwargs):
>              return self
> 
>          def __getattr__(self, name):
>              return self
> 
>          def __getitem__(self, key):
>              return self
> 
> 
> With this new operator and new type, an expression like ``foo?.bar[baz]``
> evaluates to ``NoneQuestion`` if ``foo`` is None. This is a nifty
> generalization, but it's difficult to use in practice since most 
> existing code
> won't know what ``NoneQuestion`` is.
> 
> Going back to one of the motivating examples above, consider the following::
> 
>      >>> import json
>      >>> created = None
>      >>> json.dumps({'created': created?.isoformat()})``  
> 
> The JSON serializer does not know how to serialize ``NoneQuestion``, nor 
> will
> any other API. This proposal actually requires *lots of specialized logic*
> throughout the standard library and any third party library.
> 
> At the same time, the ``?`` operator may also be **too general**, in the 
> sense
> that it can be combined with any other operator. What should the following
> expressions mean?::
> 
>      >>> x? + 1
>      >>> x? -= 1
>      >>> x? == 1
>      >>> ~x?  
> 
> This degree of generalization is not useful. The operators actually proposed
> herein are intentionally limited to a few operators that are expected to 
> make it
> easier to write common code patterns.
> 
> Built-in ``maybe``
> ------------------
> 
> Haskell has a concept called `Maybe <https://wiki.haskell.org/Maybe>`_ that
> encapsulates the idea of an optional value without relying on any special
> keyword (e.g. ``null``) or any special instance (e.g. ``None``). In 
> Haskell, the
> purpose of ``Maybe`` is to avoid separate handling of "something" and 
> nothing".
> 
> A Python package called `pymaybe <https://pypi.org/p/pymaybe/>`_ provides a
> rough approximation. The documentation shows the following example::
> 
>      >>> maybe('VALUE').lower()  
>      'value'
> 
>      >>> maybe(None).invalid().method().or_else('unknown')  
>      'unknown'
> 
> The function ``maybe()`` returns either a ``Something`` instance or a
> ``Nothing`` instance. Similar to the unary postfix operator described in the
> previous section, ``Nothing`` overrides dunder methods in order to allow
> chaining on a missing value.
> 
> Note that ``or_else()`` is eventually required to retrieve the 
> underlying value
> from ``pymaybe``'s wrappers. Furthermore, ``pymaybe`` does not short 
> circuit any
> evaluation. Although ``pymaybe`` has some strengths and may be useful in 
> its own
> right, it also demonstrates why a pure Python implementation of 
> coalescing is
> not nearly as powerful as support built into the language.
> 
> The idea of adding a builtin ``maybe`` type to enable this scenario is 
> rejected.
> 
> Just use a conditional expression
> ---------------------------------
> 
> Another common way to initialize default values is to use the ternary 
> operator.
> Here is an excerpt from the popular `Requests package
> <https://github.com/kennethreitz/requests/blob/14a555ac716866678bf17e43e23230d81
> a8149f5/requests/models.py#L212>`_::  
> 
>      data = [] if data is None else data
>      files = [] if files is None else files
>      headers = {} if headers is None else headers
>      params = {} if params is None else params
>      hooks = {} if hooks is None else hooks
> 
> This particular formulation has the undesirable effect of putting the 
> operands
> in an unintuitive order: the brain thinks, "use ``data`` if possible and use
> ``[]`` as a fallback," but the code puts the fallback *before* the preferred
> value.
> 
> The author of this package could have written it like this instead::
> 
>      data = data if data is not None else []
>      files = files if files is not None else []
>      headers = headers if headers is not None else {}
>      params = params if params is not None else {}
>      hooks = hooks if hooks is not None else {}
> 
> This ordering of the operands is more intuitive, but it requires 4 extra
> characters (for "not "). It also highlights the repetition of identifiers:
> ``data if data``, ``files if files``, etc.
> 
> When written using the ``None`` coalescing operator, the sample reads::
> 
>      data = data ?? []
>      files = files ?? []
>      headers = headers ?? {}
>      params = params ?? {}
>      hooks = hooks ?? {}
> 
> 
> References
> ==========
> 
> .. [1] C# Reference: Operators
>     (https://msdn.microsoft.com/en-us/library/6a71f45d.aspx)
> 
> .. [2] A Tour of the Dart Language: Operators
>     (https://www.dartlang.org/docs/dart-up-and-running/ch02.html#operators)
> 
> .. [3] Associated scripts
>     (https://github.com/python/peps/tree/master/pep-0505/)
> 
> Copyright
> =========
> 
> This document has been placed in the public domain.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
> 




From rosuav at gmail.com  Thu Jul 19 04:36:33 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 19 Jul 2018 18:36:33 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <712febdd-b410-7f5e-7a96-211c6a6bd7bf@brice.xyz>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <712febdd-b410-7f5e-7a96-211c6a6bd7bf@brice.xyz>
Message-ID: <CAPTjJmrWbubncaD0c1f6cQHQg8athScXYSJV7c=mNxOq4eUJ9Q@mail.gmail.com>

On Thu, Jul 19, 2018 at 5:45 PM, Brice Parent <contact at brice.xyz> wrote:
> The biggest drawback of this, is that (if I understand it well), it may be
> done quite easily without any change to the language:
>
> def first_set(*elements):  # please don't mind the name of the function,
> it's not the purpose here
>     """ Will return the first element that is not None """
>     for element in elements:
>         if element is not None:
>             return element
>
>     raise AllNoneException()
>
> first_set(3, 5)  # -> 3
> first_set(None, 5)  # -> 5
> first_set(None, None, 8, 10)  # -> 8
> first_set(None, Car(model="sport")).buy()  # calling
> Car(model="sport").buy()
> first_set(None, ["a", "b", "c"])[1]  # -> "b"
> first_set(None, None)  # -> exception is raised
>
> (note that such function could even accept a "rejected_values" kwarg, like
> `rejected_values=(None, [], "")`, just by replacing the `if` clause by `if
> element not in rejected_values:`)

No it can't, for the same reason that the 'and' and 'or' operators
can't be implemented cleanly as functions: it short-circuits. The
right-hand operator _will not_ be evaluated unless the left is None.

ChrisA

From stephanh42 at gmail.com  Thu Jul 19 04:47:19 2018
From: stephanh42 at gmail.com (Stephan Houben)
Date: Thu, 19 Jul 2018 10:47:19 +0200
Subject: [Python-ideas] Multi-core reference count garbage collection
In-Reply-To: <CALD=Yf-Ht4uDMHkvXp8i+xesw=e9zw4i+nAuRJFMVppNSYrboA@mail.gmail.com>
References: <CALD=Yf-Ht4uDMHkvXp8i+xesw=e9zw4i+nAuRJFMVppNSYrboA@mail.gmail.com>
Message-ID: <CAOOa=pMR==+=5jDvaVeQsWyRf6m=2cZ0r4p6R=89-+=JnuwBJw@mail.gmail.com>

Hi Jonathan,

2018-07-19 8:33 GMT+02:00 Jonathan Fine <jfine2358 at gmail.com>:

> I call any such scheme BUFFERED multi-core reference count garbage
> collection. The worker processes know nothing about how garbage collection
> is managed. Instead, they pass over to the GC process sufficient
> information to allow it to manage the garbage.
>

This is actually a known idea (which is GOOD, unless you wanted to apply
for a patent ;-) ).

It is described, among other places, in
"The Garbage Collection Handbook: The Art of Automatic Memory Management",
by Richard Jones, Antony Hosking, Eliot Moss.

In fact, they also call it buffered reference counting.
Section 18.2: Buffered Reference Counting:
"...DeTreville[1990] had log mutators the old and new referents of each
pointer update to a buffer"

By the way, this book describes a ton of other ideas to speed up
reference counting in general and in the concurrent case, so
it may be worthwhile to get it.

I should warn you that many of these advanced refcounting ideas have been
proposed in the past already, although I am unaware of this particular
technique
having been implemented.

Stephan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/95204a17/attachment.html>

From contact at brice.xyz  Thu Jul 19 04:48:48 2018
From: contact at brice.xyz (Brice Parent)
Date: Thu, 19 Jul 2018 10:48:48 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPTjJmrWbubncaD0c1f6cQHQg8athScXYSJV7c=mNxOq4eUJ9Q@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <712febdd-b410-7f5e-7a96-211c6a6bd7bf@brice.xyz>
 <CAPTjJmrWbubncaD0c1f6cQHQg8athScXYSJV7c=mNxOq4eUJ9Q@mail.gmail.com>
Message-ID: <bb34218b-229a-73f2-2415-9021e8622062@brice.xyz>

Le 19/07/2018 ? 10:36, Chris Angelico a ?crit?:
> On Thu, Jul 19, 2018 at 5:45 PM, Brice Parent <contact at brice.xyz> wrote:
>> The biggest drawback of this, is that (if I understand it well), it may be
>> done quite easily without any change to the language:
>>
>> def first_set(*elements):  # please don't mind the name of the function,
>> it's not the purpose here
>>      """ Will return the first element that is not None """
>>      for element in elements:
>>          if element is not None:
>>              return element
>>
>>      raise AllNoneException()
>>
>> first_set(3, 5)  # -> 3
>> first_set(None, 5)  # -> 5
>> first_set(None, None, 8, 10)  # -> 8
>> first_set(None, Car(model="sport")).buy()  # calling
>> Car(model="sport").buy()
>> first_set(None, ["a", "b", "c"])[1]  # -> "b"
>> first_set(None, None)  # -> exception is raised
>>
>> (note that such function could even accept a "rejected_values" kwarg, like
>> `rejected_values=(None, [], "")`, just by replacing the `if` clause by `if
>> element not in rejected_values:`)
> No it can't, for the same reason that the 'and' and 'or' operators
> can't be implemented cleanly as functions: it short-circuits. The
> right-hand operator _will not_ be evaluated unless the left is None.
>
> ChrisA
Thanks for the clarification, I didn't think about that.

It doesn't make it beautiful or easily readable, but at least, I 
understand how it may be useful then!



From stephanh42 at gmail.com  Thu Jul 19 05:02:28 2018
From: stephanh42 at gmail.com (Stephan Houben)
Date: Thu, 19 Jul 2018 11:02:28 +0200
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <CAPJVwBk2RSF6=2cQtRHBF=XCPcaGDARNpXdAtw_rcvBendr+5w@mail.gmail.com>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>
 <CAPTjJmqKGhNEcrAuzKSzcUW_kvdmPqL9_9CMyOUzcrBs=UdcEg@mail.gmail.com>
 <CAPJVwBkpLw_qRirCFq4PK6n5zLVxc=t=_bf_SZRXLsSnrMVECQ@mail.gmail.com>
 <CAPTjJmrt83c96JQ3PY2oZEm68ZMbG_JBwaJ1XpcyzqRnEcks9A@mail.gmail.com>
 <6a5b228a-200a-b2c7-6067-df2b0022ae42@mrabarnett.plus.com>
 <20180716190735.2908e86d@fsol>
 <CALFfu7BFLSSRf7_mVUP3Rh4tAg1ae1cawc9KFF66EiBc2Hfn7g@mail.gmail.com>
 <890D0D1D-7166-43B5-8BC0-8B2177B3BAE0@barrys-emacs.org>
 <CALFfu7CC=2ZYFoYxf_ijU_V43F_g90RAtJZr=+nT5vgRwP4Q5g@mail.gmail.com>
 <9A7ACD82-CC6B-4C22-B3FC-B28963459046@barrys-emacs.org>
 <CALFfu7BCH3Fmht5REXxk-NTW66ZMZhPoQQfCZDZGNyFn_+DLKQ@mail.gmail.com>
 <CAOOa=pNh_C=4Ysxf4sH6VS+fc+Wv4YO62QSU9xipPD4UV8ODyA@mail.gmail.com>
 <CAPJVwBk2RSF6=2cQtRHBF=XCPcaGDARNpXdAtw_rcvBendr+5w@mail.gmail.com>
Message-ID: <CAOOa=pOKuqBDxMvWpMy_oGdVkvUZvAmU5AcpTPgX8x6qZwHceQ@mail.gmail.com>

Hi Nathaniel,

2018-07-19 1:33 GMT+02:00 Nathaniel Smith <njs at pobox.com>:

> Note that this everything you said here also exactly describes the
> programming model for the existing 'multiprocessing' module:
> "structured clone" is equivalent to how multiprocessing uses pickle to
> transfer arbitrary objects, or you can use multiprocessing.Array to
> get a shared view on raw "C"-style data.
>


This is true. In fact, I am a big fan of multiprocessing and I think it is
often
overlooked/underrated. Experience with multiprocessing is also
what has me convinced that share-nothing or share-explicit approach
to concurrency is a useful programming model.

The main limitation of multiprocessing comes when you need to go outside
Python, and you need to interact with C/C++ libraries or operating services
from multiple processes.
The support for this generally varies from "extremely weak" to "none at
all".

For example, things I would like to in parallel with a main thread/process:

* Upload data to the GPU using OpenGL or OpenCL
* Generate a picture in pyqt QImage, then hand over zero-copy to main thread
* interact with a complex scenegraph in C++ (shared with main thread)

This is impossible right now but would be possible if the interpreters were
all in-process.

In addition, there are things which are now hard with "multiprocessing" but
could be fixed.
For example, sharing a Numpy array is possible but very inconvenient.
You need to first allocate the raw data segment, communicate that, then
create in each process
an array which uses this data segment.

Ideally, this would rather work like this:

   ar = numpy.zeros((30, 30), shared=True)

and then "ar" would automatically be shared.

This is fixable but given the other limitations above the question is if it
is worthwhile
to fix it now. It would be a lot simpler to fix if we had the in-process
model.

But yeah, I am actually also very open to ideas on how multiprocessing
could be
made more convenient and powerful. Perhaps there are ways, and I am just
not seeing them.

Stephan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/3dac6996/attachment-0001.html>

From greg.ewing at canterbury.ac.nz  Thu Jul 19 02:06:56 2018
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 19 Jul 2018 18:06:56 +1200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPTjJmqPS_wiFTJaJ8YUKJfWsNcCX=h9AQ1EmWFv6Xf9Wes4tA@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAFYqXL8GsTyAHEkrQqSDpv_+bCcJAVv2j-0d_zwNuLLfK7Cuuw@mail.gmail.com>
 <CAPTjJmqPS_wiFTJaJ8YUKJfWsNcCX=h9AQ1EmWFv6Xf9Wes4tA@mail.gmail.com>
Message-ID: <5B502A80.3080406@canterbury.ac.nz>

Chris Angelico wrote:
> I'd love to hear an explanation of WHY this doesn't look like Python
> any more. For instance, is the + operator somehow wrong for Python,
> and it should have been the word "add"?

There's a very long tradition of using the symbol "+" to
represent addition, so it's something most people are
familiar with. There's no such tradition for the new
operators being proposed.

-- 
Greg

From rosuav at gmail.com  Thu Jul 19 05:11:33 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 19 Jul 2018 19:11:33 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <5B502A80.3080406@canterbury.ac.nz>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAFYqXL8GsTyAHEkrQqSDpv_+bCcJAVv2j-0d_zwNuLLfK7Cuuw@mail.gmail.com>
 <CAPTjJmqPS_wiFTJaJ8YUKJfWsNcCX=h9AQ1EmWFv6Xf9Wes4tA@mail.gmail.com>
 <5B502A80.3080406@canterbury.ac.nz>
Message-ID: <CAPTjJmquqsqX8XL5RGdvebwHw-vgx15sQU2Qbyg3SKz3WHso2A@mail.gmail.com>

On Thu, Jul 19, 2018 at 4:06 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Chris Angelico wrote:
>>
>> I'd love to hear an explanation of WHY this doesn't look like Python
>> any more. For instance, is the + operator somehow wrong for Python,
>> and it should have been the word "add"?
>
>
> There's a very long tradition of using the symbol "+" to
> represent addition, so it's something most people are
> familiar with. There's no such tradition for the new
> operators being proposed.

Okay. What about bitwise operators, then? They don't have centuries of
mathematical backing to support them, yet it isn't considered
"unpythonic" to have &|^~ peppering our code. Judging by the current
levels of backlash against symbolic operators, it would have been
better to use "more explicit" function calls for all bitwise
operations. Coalescing None to a value is _at least_ as common as
performing bit manipulations in integers.

ChrisA

From solipsis at pitrou.net  Thu Jul 19 05:21:19 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 19 Jul 2018 11:21:19 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAFYqXL8GsTyAHEkrQqSDpv_+bCcJAVv2j-0d_zwNuLLfK7Cuuw@mail.gmail.com>
 <CAPTjJmqPS_wiFTJaJ8YUKJfWsNcCX=h9AQ1EmWFv6Xf9Wes4tA@mail.gmail.com>
 <5B502A80.3080406@canterbury.ac.nz>
 <CAPTjJmquqsqX8XL5RGdvebwHw-vgx15sQU2Qbyg3SKz3WHso2A@mail.gmail.com>
Message-ID: <20180719112119.663c5732@fsol>

On Thu, 19 Jul 2018 19:11:33 +1000
Chris Angelico <rosuav at gmail.com> wrote:

> On Thu, Jul 19, 2018 at 4:06 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> > Chris Angelico wrote:  
> >>
> >> I'd love to hear an explanation of WHY this doesn't look like Python
> >> any more. For instance, is the + operator somehow wrong for Python,
> >> and it should have been the word "add"?  
> >
> >
> > There's a very long tradition of using the symbol "+" to
> > represent addition, so it's something most people are
> > familiar with. There's no such tradition for the new
> > operators being proposed.  
> 
> Okay. What about bitwise operators, then? They don't have centuries of
> mathematical backing to support them, yet it isn't considered
> "unpythonic" to have &|^~ peppering our code.

They have decades of widespread presence in other programming
languages, though.

> Coalescing None to a value is _at least_ as common as
> performing bit manipulations in integers.

Certainly, but spelling that as a "?*" operator is a syntactical
novelty.

Consider that for the ternary operator, Python chose "B if A else C"
over "A ? B : C", even though the latter had precedent in several
languages.

Regards

Antoine.



From jfine2358 at gmail.com  Thu Jul 19 06:53:28 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Thu, 19 Jul 2018 11:53:28 +0100
Subject: [Python-ideas] Multi-core reference count garbage collection
In-Reply-To: <CAOOa=pMR==+=5jDvaVeQsWyRf6m=2cZ0r4p6R=89-+=JnuwBJw@mail.gmail.com>
References: <CALD=Yf-Ht4uDMHkvXp8i+xesw=e9zw4i+nAuRJFMVppNSYrboA@mail.gmail.com>
 <CAOOa=pMR==+=5jDvaVeQsWyRf6m=2cZ0r4p6R=89-+=JnuwBJw@mail.gmail.com>
Message-ID: <CALD=Yf97vYr+6Bot2aV6Q6H_YBx3ApR2HaS-duo2x4-TbUfR3w@mail.gmail.com>

Hi Stephan

Thank you for the extract from the GC Handbook, which I think I may have
seen before. Yes, it is GOOD that it's an already known idea.

Searching for "buffered reference counting" I found
https://mail.python.org/pipermail/python-dev/2016-October/146696.html
in which Larry Hastings says that C-python "plays games with reference
counts" which makes implementing "buffered reference counting" harder. And
he gives examples.

Larry also writes [loc cit] about resurrecting objects. I don't know what
he means by this. It may be something to do with weak references. Larry's
post gives some details, in which difficulties may lie. In particular, he
writes
===
https://mail.python.org/pipermail/python-dev/2016-October/146604.html
It's my contention that this API [for weak references] is simply untenable
under the
Gilectomy, and that it needs to change to returning a new (strong)
reference.
===

However, I'm focused on the larger picture right now. And it good to know
something of where the hazards lie.

Once again, Stephan, thank you for your contribution to this thread.


On Thu, Jul 19, 2018 at 9:47 AM, Stephan Houben <stephanh42 at gmail.com>
wrote:

> Hi Jonathan,
>
> 2018-07-19 8:33 GMT+02:00 Jonathan Fine <jfine2358 at gmail.com>:
>
>> I call any such scheme BUFFERED multi-core reference count garbage
>> collection. The worker processes know nothing about how garbage collection
>> is managed. Instead, they pass over to the GC process sufficient
>> information to allow it to manage the garbage.
>>
>
> This is actually a known idea (which is GOOD, unless you wanted to apply
> for a patent ;-) ).
>
> It is described, among other places, in
> "The Garbage Collection Handbook: The Art of Automatic Memory Management",
> by Richard Jones, Antony Hosking, Eliot Moss.
>
> In fact, they also call it buffered reference counting.
> Section 18.2: Buffered Reference Counting:
> "...DeTreville[1990] had log mutators the old and new referents of each
> pointer update to a buffer"
>
> By the way, this book describes a ton of other ideas to speed up
> reference counting in general and in the concurrent case, so
> it may be worthwhile to get it.
>
> I should warn you that many of these advanced refcounting ideas have been
> proposed in the past already, although I am unaware of this particular
> technique
> having been implemented.
>
> Stephan
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/deb6b231/attachment.html>

From g.rodola at gmail.com  Thu Jul 19 07:34:31 2018
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Thu, 19 Jul 2018 13:34:31 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180719112119.663c5732@fsol>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAFYqXL8GsTyAHEkrQqSDpv_+bCcJAVv2j-0d_zwNuLLfK7Cuuw@mail.gmail.com>
 <CAPTjJmqPS_wiFTJaJ8YUKJfWsNcCX=h9AQ1EmWFv6Xf9Wes4tA@mail.gmail.com>
 <5B502A80.3080406@canterbury.ac.nz>
 <CAPTjJmquqsqX8XL5RGdvebwHw-vgx15sQU2Qbyg3SKz3WHso2A@mail.gmail.com>
 <20180719112119.663c5732@fsol>
Message-ID: <CAFYqXL8mRGfzSheKDyV5OD0y7SqiPC+d+v4da+oq_Fiyvj3TEg@mail.gmail.com>

On Thu, Jul 19, 2018 at 11:22 AM Antoine Pitrou <solipsis at pitrou.net> wrote:
>
> On Thu, 19 Jul 2018 19:11:33 +1000
> Chris Angelico <rosuav at gmail.com> wrote:
>
> > On Thu, Jul 19, 2018 at 4:06 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> > > Chris Angelico wrote:
> > >>
> > >> I'd love to hear an explanation of WHY this doesn't look like Python
> > >> any more. For instance, is the + operator somehow wrong for Python,
> > >> and it should have been the word "add"?
> > >
> > >
> > > There's a very long tradition of using the symbol "+" to
> > > represent addition, so it's something most people are
> > > familiar with. There's no such tradition for the new
> > > operators being proposed.
> >
> > Okay. What about bitwise operators, then? They don't have centuries of
> > mathematical backing to support them, yet it isn't considered
> > "unpythonic" to have &|^~ peppering our code.
>
> They have decades of widespread presence in other programming
> languages, though.
>
> > Coalescing None to a value is _at least_ as common as
> > performing bit manipulations in integers.
>
> Certainly, but spelling that as a "?*" operator is a syntactical
> novelty.
>
> Consider that for the ternary operator, Python chose "B if A else C"
> over "A ? B : C", even though the latter had precedent in several
> languages.
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

^
Agreed. To me even bitwise operators "feel" a bit weird when using
them but fortunately they are rare. On the other hand "?" has the
potential to be used (and abused) much more than bitwise operators.
Also I don't consider "since we have X then let's add Y" a valid
enough reasoning.

-- 
Giampaolo - http://grodola.blogspot.com

From jfine2358 at gmail.com  Thu Jul 19 07:57:01 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Thu, 19 Jul 2018 12:57:01 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
Message-ID: <CALD=Yf8j8eGxw5DBh7rgWES6OeA4MXK6nSuEDELvgm-TRmS_YQ@mail.gmail.com>

Hi

> Possibly this is exactly the wrong time to propose the next big syntax
> change, since we currently have nobody to declare on it, but since we're
> likely to argue for a while anyway it probably can't hurt [...]

Perhaps "argue" is not the right word here. It sounds too much for or
against. And it has implications of being heated and angry.

I think "discuss" might be a better word to use. This has associations
of a disinterested pursuit of truth, and an openness to new ideas.

The ability to see both sides of an issue is, I think, very important
for reaching the compromises that have helped Python succeed, both
technically and in creating a broad and harmonious community.

That said, there is a passion in "argue" that seems not to be present
in "discuss".

-- 
Jonathan

From mertz at gnosis.cx  Thu Jul 19 08:24:59 2018
From: mertz at gnosis.cx (David Mertz)
Date: Thu, 19 Jul 2018 08:24:59 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPTjJmquqsqX8XL5RGdvebwHw-vgx15sQU2Qbyg3SKz3WHso2A@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAFYqXL8GsTyAHEkrQqSDpv_+bCcJAVv2j-0d_zwNuLLfK7Cuuw@mail.gmail.com>
 <CAPTjJmqPS_wiFTJaJ8YUKJfWsNcCX=h9AQ1EmWFv6Xf9Wes4tA@mail.gmail.com>
 <5B502A80.3080406@canterbury.ac.nz>
 <CAPTjJmquqsqX8XL5RGdvebwHw-vgx15sQU2Qbyg3SKz3WHso2A@mail.gmail.com>
Message-ID: <CAEbHw4bQ4SC-jJCO5jApta5QxjwaGH=adFJYD7Pdr9RUVKJnxg@mail.gmail.com>

On Thu, Jul 19, 2018, 5:12 AM Chris Angelico <rosuav at gmail.com> wrote:

> On Thu, Jul 19, 2018 at 4:06 PM, Greg Ewing <greg.ewing at canterbury.ac.nz>
> wrote:
> > There's a very long tradition of using the symbol "+" to
> > represent addition, so it's something most people are
> > familiar with. There's no such tradition for the new
> > operators being proposed.
>
> Okay. What about bitwise operators, then? They don't have centuries of
> mathematical backing to support them, yet it isn't considered
> "unpythonic" to have &|^~ peppering our code.
>

I have quite literally NEVER seem Python code with much use of the bitwise
operators. I guess the closest I've come is in some NumPy and Pandas code
where filters cannot use plain 'and' and ' or' but every student would find
it more intuitive if they could. E.g.

myDf[(myDf.field1 > 4) & (myDf.field2 < 2)]

Everyone still gets tripped up by the need for those parentheses because of
operator precedence. But this already yells or "special domain" rather than
"plain Python".

Indeed, it not uncommon or unreasonable to recommend using np.bitwise_and()
to avoid those confusing operators.

In the case where bitwise masking is used in its C manner, the code screams
special domain even more loudly. It definitely feels strongly unPythonic,
but it's often a reasonable compromise for dealing with just a little bit
of binary data without having to write a C extension.

I would ALWAYS want the code that used bitwise operators wrapped in a
separate function that most users and developers didn't need to look at,
but rather they'd call a more Pythonic API for the overall operation.

A huge difference is that bitwise operators do something you simply cannot
do other ways in Python at all. None-aware operators *at best* allow you to
write something with an existing straightforward approach using a couple
fewer lines. I think the result is ALWAYS less clear to read. Code golf is
an anti-goal in Python.

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/b5ccef40/attachment.html>

From jfine2358 at gmail.com  Thu Jul 19 08:30:33 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Thu, 19 Jul 2018 13:30:33 +0100
Subject: [Python-ideas] PEP 505: None-aware operators: operators ?= and
 ?? and OR
Message-ID: <CALD=Yf-0dLkGKN+1gb11nv4s0J4yiySehszZ+rS2BZVciR+LHA@mail.gmail.com>

Hi

> There is a formatted version of this PEP at
> https://www.python.org/dev/peps/pep-0505/

I've taken a look at this, and have some comments on the first two
examples drawn from standard library code. (And a very grateful +10
for writing a script to find such examples.)

I've started a subthread, just to discuss the ?= and ?? operators. And
something newish, that I call OR.

FIRST EXAMPLE
The first example is
---
>From bisect.py:
def insort_right(a, x, lo=0, hi=None):
    # ...
    if hi is None:
        hi = len(a)
---

Here, None is a sentinel value. The simpler code
---
  hi = hi or len(a)
---
fails when hi is zero (or any other value that is False in the boolean context).

This can be fixed by introducing a new operator OR which is similar to
'or' but has the semantics this example requires. Thus, given OR we
can write
---
  hi = hi OR len(a)
---
where (A OR B) returns A if A is not None, otherwise it returns B.

(Recall that (A or B) returns A if bool(A), otherwise it returns B.)

SECOND EXAMPLE
The second example is
---
>From calendar.py:
encoding = options.encoding
if encoding is None:
    encoding = sys.getdefaultencoding()
optdict = dict(encoding=encoding, css=options.css)
---

Once we have OR we would write this as
---
encoding = encoding OR sys.getdefaultencoding()
optdict = dict(encoding=encoding, css=options.css)
---

And from here we can condense into a single (longish) line:
---
optdict = dict(encoding=encoding OR sys.getdefaultencoding(), css=options.css)
--

SUMMARY
Here, for reference, are the suggestions using ?= and ?? in PEP 505.
---
hi ??= len(a)
---
optdict = dict(encoding=encoding ?? sys.getdefaultencoding(), css=options.css)
---

Comments ?? suggestions. For example, would a None-aware AND operator be useful?

-- 
Jonathan

From rhodri at kynesim.co.uk  Thu Jul 19 08:39:27 2018
From: rhodri at kynesim.co.uk (Rhodri James)
Date: Thu, 19 Jul 2018 13:39:27 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
Message-ID: <d50cedd7-fa8d-0ec1-5356-1bebe51355ab@kynesim.co.uk>

On 18/07/18 18:43, Steve Dower wrote:
> Possibly this is exactly the wrong time to propose the next big syntax 
> change, since we currently have nobody to declare on it, but since we're 
> likely to argue for a while anyway it probably can't hurt (and maybe 
> this will become the test PEP for whoever takes the reins?).

Many thanks for your work on this, Steve.  Having just written a fair 
bit of code that would have benefited from None-coalescing, I'm +1 for 
"??" and "??=" and +0 for the rest.

> Grammar changes
> ---------------
> 
> The following rules of the Python grammar are updated to read::
> 
>  ??? augassign: ('+=' | '-=' | '*=' | '@=' | '/=' | '%=' | '&=' | '|=' | 
> '^=' |
>  ??????????????? '<<=' | '>>=' | '**=' | '//=' | '??=')
> 
>  ??? power: coalesce ['**' factor]
>  ??? coalesce: atom_expr ['??' factor]
>  ??? atom_expr: ['await'] atom trailer*
>  ??? trailer: ('(' [arglist] ')' |
>  ????????????? '[' subscriptlist ']' |
>  ????????????? '?[' subscriptlist ']' |
>  ????????????? '.' NAME |
>  ????????????? '?.' NAME)

I was a bit worried about the priority here, but your explanations 
elsewhere make sense.  Perhaps the PEP would benefit from being a bit 
more explicit about this?

>  From ``calendar.py``::
> 
>  ??? encoding = options.encoding
>  ??? if encoding is None:
>  ??????? encoding = sys.getdefaultencoding()
>  ??? optdict = dict(encoding=encoding, css=options.css)
> 
> After updating to use the ``??`` operator::
> 
>  ??? optdict = dict(encoding=encoding ?? sys.getdefaultencoding(),
>  ?????????????????? css=options.css)

ITYM
   optdict = dict(encoding=options.encoding ?? sys.getdefaultencoding(),
                  css=options.css)

>  From ``dis.py``::
> 
>  ??? def _get_const_info(const_index, const_list):
>  ??????? argval = const_index
>  ??????? if const_list is not None:
>  ??????????? argval = const_list[const_index]
>  ??????? return argval, repr(argval)
> 
> After updating to use the ``?[]`` and ``??`` operators::
> 
>  ??? def _get_const_info(const_index, const_list):
>  ??????? argval = const_list?[const_index] ?? const_index
>  ??????? return argval, repr(argval)

Here's where I start to part company.  To me, the updated version is 
markedly harder to read, even if it does (once deciphered) convey the 
intent of the function better than the original :-)  The "?." and "?[]" 
operators just aren't obvious enough not to trip my internal WTF filter; 
either that or I'll overlook the "?" part entirely, which is probably worse.

> Just use a conditional expression
> ---------------------------------
> 
> Another common way to initialize default values is to use the ternary 
> operator.
> Here is an excerpt from the popular `Requests package
> <https://github.com/kennethreitz/requests/blob/14a555ac716866678bf17e43e23230d81 
> 
> a8149f5/requests/models.py#L212>`_::
> 
>  ??? data = [] if data is None else data
>  ??? files = [] if files is None else files
>  ??? headers = {} if headers is None else headers
>  ??? params = {} if params is None else params
>  ??? hooks = {} if hooks is None else hooks
> 
> This particular formulation has the undesirable effect of putting the 
> operands
> in an unintuitive order: the brain thinks, "use ``data`` if possible and 
> use
> ``[]`` as a fallback," but the code puts the fallback *before* the 
> preferred
> value.
> 
> The author of this package could have written it like this instead::
> 
>  ??? data = data if data is not None else []
>  ??? files = files if files is not None else []
>  ??? headers = headers if headers is not None else {}
>  ??? params = params if params is not None else {}
>  ??? hooks = hooks if hooks is not None else {}
> 
> This ordering of the operands is more intuitive, but it requires 4 extra
> characters (for "not "). It also highlights the repetition of identifiers:
> ``data if data``, ``files if files``, etc.

While the four whole extra characters bothers me not one bit, the 
repetition is more of an issue.  It's much worse if what you have is 
more like:

item = spam.spam.spam.eggs if spam.spam.spam.eggs is not None else beans

-- 
Rhodri James *-* Kynesim Ltd

From rhodri at kynesim.co.uk  Thu Jul 19 08:46:38 2018
From: rhodri at kynesim.co.uk (Rhodri James)
Date: Thu, 19 Jul 2018 13:46:38 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180719103321.7c193820@fsol>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <20180719103321.7c193820@fsol>
Message-ID: <c49bd9b8-1242-fd07-29e8-9039bfaf9199@kynesim.co.uk>

On 19/07/18 09:33, Antoine Pitrou wrote:
> There is a use case I sympathize with: the argument-is-None case.  For
> that I would suggest a simpler form:  "A else B" which would evaluate
> to A if A is not None, otherwise to B (parentheses may be mandatory).
> 
> So e.g. one of the examples would read:
> 
>       def insort_right(a, x, lo=0, hi=None):
>           # ...
>           hi = hi else len(a)
>           # ...

Much as I would like a keyword, "else" is the wrong one.  It implies we 
are dealing with truthiness, which we aren't, and lays a subtle semantic 
trap as a consequence.

If anyone can think of a good word for "if it isn't None, otherwise", 
I'd be all for it :-)

-- 
Rhodri James *-* Kynesim Ltd

From rhodri at kynesim.co.uk  Thu Jul 19 08:48:29 2018
From: rhodri at kynesim.co.uk (Rhodri James)
Date: Thu, 19 Jul 2018 13:48:29 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <5B502A80.3080406@canterbury.ac.nz>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAFYqXL8GsTyAHEkrQqSDpv_+bCcJAVv2j-0d_zwNuLLfK7Cuuw@mail.gmail.com>
 <CAPTjJmqPS_wiFTJaJ8YUKJfWsNcCX=h9AQ1EmWFv6Xf9Wes4tA@mail.gmail.com>
 <5B502A80.3080406@canterbury.ac.nz>
Message-ID: <c5d3ec35-5dae-f6f2-affa-79303e7d0fe1@kynesim.co.uk>

On 19/07/18 07:06, Greg Ewing wrote:
> Chris Angelico wrote:
>> I'd love to hear an explanation of WHY this doesn't look like Python
>> any more. For instance, is the + operator somehow wrong for Python,
>> and it should have been the word "add"?
> 
> There's a very long tradition of using the symbol "+" to
> represent addition, so it's something most people are
> familiar with. There's no such tradition for the new
> operators being proposed.

There is, actually, it's just not a long one.  C# has had null-aware 
operators for a while, for example.  I'll admit the syntax sits better 
in a C-like language, though.

-- 
Rhodri James *-* Kynesim Ltd

From judah.j.levy at gmail.com  Thu Jul 19 08:59:42 2018
From: judah.j.levy at gmail.com (Judah Levy)
Date: Thu, 19 Jul 2018 08:59:42 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <c49bd9b8-1242-fd07-29e8-9039bfaf9199@kynesim.co.uk>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <20180719103321.7c193820@fsol>
 <c49bd9b8-1242-fd07-29e8-9039bfaf9199@kynesim.co.uk>
Message-ID: <CAFkYj5=Uy+QsUbGJNFNy6-bQezKo6BcvCa4EmXeJ6WQGPAhxBg@mail.gmail.com>

On Thu, Jul 19, 2018 at 8:47 AM Rhodri James <rhodri at kynesim.co.uk> wrote:

> On 19/07/18 09:33, Antoine Pitrou wrote:
> > There is a use case I sympathize with: the argument-is-None case.  For
> > that I would suggest a simpler form:  "A else B" which would evaluate
> > to A if A is not None, otherwise to B (parentheses may be mandatory).
> >
> > So e.g. one of the examples would read:
> >
> >       def insort_right(a, x, lo=0, hi=None):
> >           # ...
> >           hi = hi else len(a)
> >           # ...
>
> Much as I would like a keyword, "else" is the wrong one.  It implies we
> are dealing with truthiness, which we aren't, and lays a subtle semantic
> trap as a consequence.
>
> If anyone can think of a good word for "if it isn't None, otherwise",
> I'd be all for it :-)
>
> --
> Rhodri James *-* Kynesim Ltd
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>

I think that it may look better with the order switched and the word
unless, as in

    def insort_right(a, x, lo=0 hi=None):
        # ...
        hi = len(a) unless hi
        # ...

Unfortunately, this does maybe feel more like checking for truthiness than
non-Null value
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/45012e0b/attachment.html>

From p.f.moore at gmail.com  Thu Jul 19 09:01:29 2018
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 19 Jul 2018 14:01:29 +0100
Subject: [Python-ideas] PEP 505: None-aware operators: operators ?= and
 ?? and OR
In-Reply-To: <CALD=Yf-0dLkGKN+1gb11nv4s0J4yiySehszZ+rS2BZVciR+LHA@mail.gmail.com>
References: <CALD=Yf-0dLkGKN+1gb11nv4s0J4yiySehszZ+rS2BZVciR+LHA@mail.gmail.com>
Message-ID: <CACac1F8RYebzQRoEFLM1iticF-hMKEOWOAJmpF8tJ1FORkMNAg@mail.gmail.com>

On 19 July 2018 at 13:30, Jonathan Fine <jfine2358 at gmail.com> wrote:
> Hi
>
>> There is a formatted version of this PEP at
>> https://www.python.org/dev/peps/pep-0505/
>
> I've taken a look at this, and have some comments on the first two
> examples drawn from standard library code. (And a very grateful +10
> for writing a script to find such examples.)
>
> I've started a subthread, just to discuss the ?= and ?? operators. And
> something newish, that I call OR.
>
> FIRST EXAMPLE
> The first example is
> ---
> From bisect.py:
> def insort_right(a, x, lo=0, hi=None):
>     # ...
>     if hi is None:
>         hi = len(a)
> ---
>
> Here, None is a sentinel value. The simpler code
> ---
>   hi = hi or len(a)
> ---
> fails when hi is zero (or any other value that is False in the boolean context).
>
> This can be fixed by introducing a new operator OR which is similar to
> 'or' but has the semantics this example requires. Thus, given OR we
> can write
> ---
>   hi = hi OR len(a)
> ---
> where (A OR B) returns A if A is not None, otherwise it returns B.
>
> (Recall that (A or B) returns A if bool(A), otherwise it returns B.)

How does (A OR B) differ from the PEP's (A ?? B)?

I don't particularly like it in either form. But ?? at least has the
advantage of being used in other languages.

[...]
> Comments ?? suggestions. For example, would a None-aware AND operator be useful?

I see no value in a None-aware AND. Real use cases are the only
reasonable argument for such a thing, not arguments based on symmetry
(or generalisation). And I doubt I'd find real-world use cases
particularly compelling.

Paul

From paal.drange at gmail.com  Thu Jul 19 09:06:03 2018
From: paal.drange at gmail.com (=?UTF-8?B?UMOlbCBHcsO4bsOlcyBEcmFuZ2U=?=)
Date: Thu, 19 Jul 2018 15:06:03 +0200
Subject: [Python-ideas] PEP 505: None-aware operators: operators ?= and
 ?? and OR
In-Reply-To: <CALD=Yf-0dLkGKN+1gb11nv4s0J4yiySehszZ+rS2BZVciR+LHA@mail.gmail.com>
References: <CALD=Yf-0dLkGKN+1gb11nv4s0J4yiySehszZ+rS2BZVciR+LHA@mail.gmail.com>
Message-ID: <CAPL7A87J_jWnvTpvGrv0_zzZ9yr_xT9asy9+C2JfsgFB6zt0wA@mail.gmail.com>

> I've started a subthread, just to discuss the ?= and ?? operators. And
> something newish, that I call OR.

I would think `||` would be much better.

It could be a kind of "semantic or" which could use the aforementioned
dunder has_value.

-1, though, but to the general None-awareness.

P?l
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/bbba9b3a/attachment-0001.html>

From p.f.moore at gmail.com  Thu Jul 19 09:20:14 2018
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 19 Jul 2018 14:20:14 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <d50cedd7-fa8d-0ec1-5356-1bebe51355ab@kynesim.co.uk>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <d50cedd7-fa8d-0ec1-5356-1bebe51355ab@kynesim.co.uk>
Message-ID: <CACac1F_7ZpK3zir0JSVtZ5MTQHgVT2bSL47UoqZBGLZEmrr84w@mail.gmail.com>

On 19 July 2018 at 13:39, Rhodri James <rhodri at kynesim.co.uk> wrote:

>> After updating to use the ``?[]`` and ``??`` operators::
>>
>>      def _get_const_info(const_index, const_list):
>>          argval = const_list?[const_index] ?? const_index
>>          return argval, repr(argval)
>
>
> Here's where I start to part company.  To me, the updated version is
> markedly harder to read, even if it does (once deciphered) convey the intent
> of the function better than the original :-)  The "?." and "?[]" operators
> just aren't obvious enough not to trip my internal WTF filter; either that
> or I'll overlook the "?" part entirely, which is probably worse.

I completely agree. The semantics of ?. and ?[] is non-intuitive at
best - and particularly when chained. The Groovy language has the ?.
operator (see http://groovy-lang.org/operators.html#_safe_navigation_operator),
and I pretty much always read it as if it were the equivalent
expression without the ? signs, with an added proviso "it'll probably
do something useful if we hit None". But when it comes to what that
"something useful" is, I'm left hoping the original writer knew what
they were doing.

So I'd never write code using ?. or ?[] myself, and I'd be unable to
reasonably review or maintain code containing them. That's a pretty
serious condemnation of the syntax in my view.

Conversely, while I find ?= and ?? ugly, and would avoid them, the
semantics are relatively easy to assess when reading code. Of the two,
I dislike ?? more than ?=, but both are streets ahead of ?. and ?[].

Paul

From stephanh42 at gmail.com  Thu Jul 19 09:26:04 2018
From: stephanh42 at gmail.com (Stephan Houben)
Date: Thu, 19 Jul 2018 15:26:04 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAFkYj5=Uy+QsUbGJNFNy6-bQezKo6BcvCa4EmXeJ6WQGPAhxBg@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <20180719103321.7c193820@fsol>
 <c49bd9b8-1242-fd07-29e8-9039bfaf9199@kynesim.co.uk>
 <CAFkYj5=Uy+QsUbGJNFNy6-bQezKo6BcvCa4EmXeJ6WQGPAhxBg@mail.gmail.com>
Message-ID: <CAOOa=pN=EM1jsdBg5OvdP1WjDpC8pFr52FG8Asf2btNwq9BJTw@mail.gmail.com>

FWIW, I like ??

It is short and distinctive.
There is prior art in this spelling in c#.
It requires no new keyword, nor does it give new meaning to an existing one.

I understand why  ?[ needs to be spelled using only a single ?, but I am
afraid it will be used infrequently, and people will accidentally write
 a??[x]
which is legal but different.

I found the example code in the PEP using ?. and ?[ hard to read.
?? and ??= are compelling, though.

One more question: what does this do?

del x
x ??= 42

Stephan


Op do 19 jul. 2018 15:00 schreef Judah Levy <judah.j.levy at gmail.com>:

> On Thu, Jul 19, 2018 at 8:47 AM Rhodri James <rhodri at kynesim.co.uk> wrote:
>
>> On 19/07/18 09:33, Antoine Pitrou wrote:
>> > There is a use case I sympathize with: the argument-is-None case.  For
>> > that I would suggest a simpler form:  "A else B" which would evaluate
>> > to A if A is not None, otherwise to B (parentheses may be mandatory).
>> >
>> > So e.g. one of the examples would read:
>> >
>> >       def insort_right(a, x, lo=0, hi=None):
>> >           # ...
>> >           hi = hi else len(a)
>> >           # ...
>>
>> Much as I would like a keyword, "else" is the wrong one.  It implies we
>> are dealing with truthiness, which we aren't, and lays a subtle semantic
>> trap as a consequence.
>>
>> If anyone can think of a good word for "if it isn't None, otherwise",
>> I'd be all for it :-)
>>
>> --
>> Rhodri James *-* Kynesim Ltd
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
> I think that it may look better with the order switched and the word
> unless, as in
>
>     def insort_right(a, x, lo=0 hi=None):
>         # ...
>         hi = len(a) unless hi
>         # ...
>
> Unfortunately, this does maybe feel more like checking for truthiness than
> non-Null value
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/c1dd60c3/attachment.html>

From nathan.mugnier at wanadoo.fr  Thu Jul 19 09:20:27 2018
From: nathan.mugnier at wanadoo.fr (Nathan)
Date: Thu, 19 Jul 2018 15:20:27 +0200 (CEST)
Subject: [Python-ideas] PEP 505: None-aware operators: operators ?= and
 ?? and OR
In-Reply-To: <CAPL7A87J_jWnvTpvGrv0_zzZ9yr_xT9asy9+C2JfsgFB6zt0wA@mail.gmail.com>
References: <CALD=Yf-0dLkGKN+1gb11nv4s0J4yiySehszZ+rS2BZVciR+LHA@mail.gmail.com>
 <CAPL7A87J_jWnvTpvGrv0_zzZ9yr_xT9asy9+C2JfsgFB6zt0wA@mail.gmail.com>
Message-ID: <472563204.5357.1532006427790.JavaMail.www@wwinf1h10>

Hello.

Even if I'm -1 on the general idea of a None-aware, for a lot of reason already mentionned in this thread, the syntax "??" proposed (that I find much less readable than the current version) is still better IMHO.
How many programmer will read "||" or "OR" as a classic "or" operator? Programmers needs to learn the language, but making it easy for them to understand and learn is important in python.
More than that, if you use "||" or "OR" thinking you use "or", nothing will help you to understand what isn't working in your code.
So -1 on that idea even if PEP 505 were accepted.

Nathan Mugnier
?
?
?
?
> Message du 19/07/18 15:06
> De : "P?l Gr?n?s Drange" 

> A : "Jonathan Fine" 
> Copie ? : "Python-Ideas" 

> Objet : Re: [Python-ideas] PEP 505: None-aware operators: operators ?= and ?? and OR
> 
>
> I've started a subthread, just to discuss the ?= and ?? operators. And
> > something newish, that I call OR.
I would think `||` would be much better.
It could be a kind of "semantic or" which could use the aforementioned dunder has_value.
-1, though, but to the general None-awareness.
P?l



_______________________________________________
Python-ideas mailing list
Python-ideas at python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/ca5ef9a3/attachment.html>

From cspealma at redhat.com  Thu Jul 19 09:30:08 2018
From: cspealma at redhat.com (Calvin Spealman)
Date: Thu, 19 Jul 2018 09:30:08 -0400
Subject: [Python-ideas] PEP 505: None-aware operators: operators ?= and
 ?? and OR
In-Reply-To: <CAPL7A87J_jWnvTpvGrv0_zzZ9yr_xT9asy9+C2JfsgFB6zt0wA@mail.gmail.com>
References: <CALD=Yf-0dLkGKN+1gb11nv4s0J4yiySehszZ+rS2BZVciR+LHA@mail.gmail.com>
 <CAPL7A87J_jWnvTpvGrv0_zzZ9yr_xT9asy9+C2JfsgFB6zt0wA@mail.gmail.com>
Message-ID: <CACo5Rz5ohDPpOMtU-D-KZMdMj+AeP2MLfyH5sDr3uj8nQ1_BNw@mail.gmail.com>

Operators that only vary by case would be... confusing. I'm not super keen
on the other syntax (either the ?? or .? operators) but I do think they
read well in C# where they come from. Different things work in different
languages, some times.

What about a new keyword: default

So you'd write the above examples like this:

default hi = len(a)  # Only executes the assignment if the left-hand is None
default encoding = sys.getdefaultencoding()

On Thu, Jul 19, 2018 at 9:06 AM, P?l Gr?n?s Drange <paal.drange at gmail.com>
wrote:

> > I've started a subthread, just to discuss the ?= and ?? operators. And
> > something newish, that I call OR.
>
> I would think `||` would be much better.
>
> It could be a kind of "semantic or" which could use the aforementioned
> dunder has_value.
>
> -1, though, but to the general None-awareness.
>
> P?l
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/734d871a/attachment-0001.html>

From robertve92 at gmail.com  Thu Jul 19 09:35:17 2018
From: robertve92 at gmail.com (Robert Vanden Eynde)
Date: Thu, 19 Jul 2018 15:35:17 +0200
Subject: [Python-ideas] PEP 505: None-aware operators: operators ?= and
 ?? and OR
In-Reply-To: <CACo5Rz5ohDPpOMtU-D-KZMdMj+AeP2MLfyH5sDr3uj8nQ1_BNw@mail.gmail.com>
References: <CALD=Yf-0dLkGKN+1gb11nv4s0J4yiySehszZ+rS2BZVciR+LHA@mail.gmail.com>
 <CAPL7A87J_jWnvTpvGrv0_zzZ9yr_xT9asy9+C2JfsgFB6zt0wA@mail.gmail.com>
 <CACo5Rz5ohDPpOMtU-D-KZMdMj+AeP2MLfyH5sDr3uj8nQ1_BNw@mail.gmail.com>
Message-ID: <CA+msPNk0GrJZnQpXtjAM+uvHL9zyon11ApRUGMy4OVKWMEUOsQ@mail.gmail.com>

If we're about to use a new keyword, it could be infix too:

a = b ifnone c

Although the assignment version looks unusual:

b ifnone= c

Then with the "default b = c" would look like this:

ifnone b = c

Le jeu. 19 juil. 2018 ? 15:30, Calvin Spealman <cspealma at redhat.com> a
?crit :

> Operators that only vary by case would be... confusing. I'm not super keen
> on the other syntax (either the ?? or .? operators) but I do think they
> read well in C# where they come from. Different things work in different
> languages, some times.
>
> What about a new keyword: default
>
> So you'd write the above examples like this:
>
> default hi = len(a)  # Only executes the assignment if the left-hand is
> None
> default encoding = sys.getdefaultencoding()
>
> On Thu, Jul 19, 2018 at 9:06 AM, P?l Gr?n?s Drange <paal.drange at gmail.com>
> wrote:
>
>> > I've started a subthread, just to discuss the ?= and ?? operators. And
>> > something newish, that I call OR.
>>
>> I would think `||` would be much better.
>>
>> It could be a kind of "semantic or" which could use the aforementioned
>> dunder has_value.
>>
>> -1, though, but to the general None-awareness.
>>
>> P?l
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/751f9993/attachment.html>

From steve at pearwood.info  Thu Jul 19 09:38:21 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 19 Jul 2018 23:38:21 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
Message-ID: <20180719133820.GV7318@ando.pearwood.info>

On Wed, Jul 18, 2018 at 08:05:56PM -0400, David Mertz wrote:

> '??' and '?.' and ?[]' are really just marching
> into APL and Perl territory. Yes, I know such operators exist in other
> languages, but it feels very unpythonic.

If these features exist in "other languages", and *don't* exist in APL 
or Perl, how are they marching into APL and Perl territory?

Perl territory, like this perhaps?

print "hello world\n";
@days = ("Monday", "Tuesday", "Wednesday");
print $days[0]

Yes, I can see why we use "Perl syntax" as an insult *wink*

Okay, okay, my examples are a bit unfair. I deliberately chose examples 
where the syntax is almost identical to Python's. Aside from the array 
and scalar sigils @ and $ the above could be Python.

Tens of thousands of non-English speakers have had to learn the meaning 
of what might as well be meaningless, random sets of symbols (to them) 
like "class", "import", "while" and "True". If they can do so, perhaps 
we English-speakers should stop complaining about how hard it is to 
memorise the meaning of a couple of symbols like ??. Surely its no more 
difficult than learning the various meanings of ** and [] which we've 
already done.

*Its just spelling*. If it is a useful and well-defined feature, we'll 
get used to the spelling soon enough.

That's not to say that spelling is not important *at all*, or that we 
should never prefer words to symbols. But if the only objection we have 
is "this is useful but I don't like the spelling so -1" then that's 
usually a pretty weak argument against the feature.



-- 
Steve

From ericfahlgren at gmail.com  Thu Jul 19 09:47:06 2018
From: ericfahlgren at gmail.com (Eric Fahlgren)
Date: Thu, 19 Jul 2018 06:47:06 -0700
Subject: [Python-ideas] Revisiting str.rreplace()
In-Reply-To: <CAN83G44dWT=+t68DS=abk29eM-WOQm3ZjOTDFzBNOVLtgr8xGA@mail.gmail.com>
References: <CAN83G44dWT=+t68DS=abk29eM-WOQm3ZjOTDFzBNOVLtgr8xGA@mail.gmail.com>
Message-ID: <CAP2Qz+VPpRBTAfn1OdYUMCD3oUoV143av7juWGvNkkXO-BxrGQ@mail.gmail.com>

On Wed, Jul 18, 2018 at 8:20 PM Graham Gott <graham.gott.software at gmail.com>
wrote:

>
> Thoughts? Support/oppose?
>

?
+1, along with an overall rework of str methods to make them more
consistent.

The find, replace and split families should all gain the same
tuple-as-search-string that endswith and startswith use.  (Discussed in the
last year, I'm too lazy to look up references...)

?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/1398a74d/attachment.html>

From mertz at gnosis.cx  Thu Jul 19 09:50:04 2018
From: mertz at gnosis.cx (David Mertz)
Date: Thu, 19 Jul 2018 09:50:04 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180719133820.GV7318@ando.pearwood.info>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
Message-ID: <CAEbHw4Yt5nd-mxpQzGFH-ehbRZH=DFMjsg81_LNxRfg4c_rpoQ@mail.gmail.com>

"APL and Perl territory" means "use lots of punctuation characters in
somewhat cryptic ways, often combining several for a distinct semantics." I
did not mean "APL and Perl use those specific characters with the proposed
meaning."

On Thu, Jul 19, 2018, 9:39 AM Steven D'Aprano <steve at pearwood.info> wrote:

> On Wed, Jul 18, 2018 at 08:05:56PM -0400, David Mertz wrote:
>
> > '??' and '?.' and ?[]' are really just marching
> > into APL and Perl territory. Yes, I know such operators exist in other
> > languages, but it feels very unpythonic.
>
> If these features exist in "other languages", and *don't* exist in APL
> or Perl, how are they marching into APL and Perl territory?
>
> Perl territory, like this perhaps?
>
> print "hello world\n";
> @days = ("Monday", "Tuesday", "Wednesday");
> print $days[0]
>
> Yes, I can see why we use "Perl syntax" as an insult *wink*
>
> Okay, okay, my examples are a bit unfair. I deliberately chose examples
> where the syntax is almost identical to Python's. Aside from the array
> and scalar sigils @ and $ the above could be Python.
>
> Tens of thousands of non-English speakers have had to learn the meaning
> of what might as well be meaningless, random sets of symbols (to them)
> like "class", "import", "while" and "True". If they can do so, perhaps
> we English-speakers should stop complaining about how hard it is to
> memorise the meaning of a couple of symbols like ??. Surely its no more
> difficult than learning the various meanings of ** and [] which we've
> already done.
>
> *Its just spelling*. If it is a useful and well-defined feature, we'll
> get used to the spelling soon enough.
>
> That's not to say that spelling is not important *at all*, or that we
> should never prefer words to symbols. But if the only objection we have
> is "this is useful but I don't like the spelling so -1" then that's
> usually a pretty weak argument against the feature.
>
>
>
> --
> Steve
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/56bdb74b/attachment-0001.html>

From python-ideas at michaelhall.tech  Thu Jul 19 09:51:38 2018
From: python-ideas at michaelhall.tech (Michael Hall)
Date: Thu, 19 Jul 2018 09:51:38 -0400
Subject: [Python-ideas] Performance improvements via static typing
Message-ID: <CABypBWrqKatZBKy3VBdzpr+o6cOgxhjpnqt7tzKW1+sraeJMSA@mail.gmail.com>

While I am aware of projects like Cython and mypy, it seems to make sense
for CPython to allow optional enforcement of type hints, with compiler
optimizations related to it to be used. While this would not receive the
same level of performance benefits as using ctypes directly, there do
appear to be various gains available here.

My main concern with this as a thought was how to specify type hints as
optional, as for maximum benefit, this shouldn't prevent the ability to
type hint functions that you don't want to be treated in this manner. While
I don't have an answer for that yet, I thought I'd toss the idea out there
first. If it needs to be seen in action before deciding if it makes sense
to add, I can work on a potential implementation soon, but right now, this
is just an idea.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/b28beeef/attachment.html>

From cspealma at redhat.com  Thu Jul 19 10:01:14 2018
From: cspealma at redhat.com (Calvin Spealman)
Date: Thu, 19 Jul 2018 10:01:14 -0400
Subject: [Python-ideas] Revisiting str.rreplace()
In-Reply-To: <CAP2Qz+VPpRBTAfn1OdYUMCD3oUoV143av7juWGvNkkXO-BxrGQ@mail.gmail.com>
References: <CAN83G44dWT=+t68DS=abk29eM-WOQm3ZjOTDFzBNOVLtgr8xGA@mail.gmail.com>
 <CAP2Qz+VPpRBTAfn1OdYUMCD3oUoV143av7juWGvNkkXO-BxrGQ@mail.gmail.com>
Message-ID: <CACo5Rz4U2N0v2KPaWspCrtBgm1cLk011hD7XPQev-rYKoty9qw@mail.gmail.com>

As an alternative suggestion: What if the count parameter to str.replace()
counted from the right with negative values? That would be consistent with
other things like indexing and slicing.

On Thu, Jul 19, 2018 at 9:47 AM, Eric Fahlgren <ericfahlgren at gmail.com>
wrote:

> On Wed, Jul 18, 2018 at 8:20 PM Graham Gott <graham.gott.software at gmail.
> com> wrote:
>
>>
>> Thoughts? Support/oppose?
>>
>
> ?
> +1, along with an overall rework of str methods to make them more
> consistent.
>
> The find, replace and split families should all gain the same
> tuple-as-search-string that endswith and startswith use.  (Discussed in the
> last year, I'm too lazy to look up references...)
>
> ?
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/2391f745/attachment.html>

From ericfahlgren at gmail.com  Thu Jul 19 10:02:33 2018
From: ericfahlgren at gmail.com (Eric Fahlgren)
Date: Thu, 19 Jul 2018 07:02:33 -0700
Subject: [Python-ideas] Performance improvements via static typing
In-Reply-To: <CABypBWrqKatZBKy3VBdzpr+o6cOgxhjpnqt7tzKW1+sraeJMSA@mail.gmail.com>
References: <CABypBWrqKatZBKy3VBdzpr+o6cOgxhjpnqt7tzKW1+sraeJMSA@mail.gmail.com>
Message-ID: <CAP2Qz+UL-y3Fn+4mdmuYwU-np8qdcECG9OmQ8n7ro-QysYc05g@mail.gmail.com>

On Thu, Jul 19, 2018 at 6:52 AM Michael Hall <python-ideas at michaelhall.tech>
wrote:

> While I am aware of projects like Cython and mypy, it seems to make sense
> for CPython to allow optional enforcement of type hints, with compiler
> optimizations related to it to be used. While this would not receive the
> same level of performance benefits as using ctypes directly, there do
> appear to be various gains available here.
>

?Just to make sure I understand:  In other words, they would no longer be
"hints" but "guarantees".  This would allow an optimizer pass much greater
latitude in code generation, somehow or other.?

For purposes of illustration (this is not a proposal, just for
clarification):

@guaranteed_types
def my_sqrt(x:c_double) -> c_double:
    ...

would tell the compiler that it's now possible to replace the general
PyObject marshalling of this function with a pure-C one that only accepts
doubles and woe be unto those who use it otherwise.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/4cb777b7/attachment.html>

From ericfahlgren at gmail.com  Thu Jul 19 10:06:22 2018
From: ericfahlgren at gmail.com (Eric Fahlgren)
Date: Thu, 19 Jul 2018 07:06:22 -0700
Subject: [Python-ideas] Revisiting str.rreplace()
In-Reply-To: <CACo5Rz4U2N0v2KPaWspCrtBgm1cLk011hD7XPQev-rYKoty9qw@mail.gmail.com>
References: <CAN83G44dWT=+t68DS=abk29eM-WOQm3ZjOTDFzBNOVLtgr8xGA@mail.gmail.com>
 <CAP2Qz+VPpRBTAfn1OdYUMCD3oUoV143av7juWGvNkkXO-BxrGQ@mail.gmail.com>
 <CACo5Rz4U2N0v2KPaWspCrtBgm1cLk011hD7XPQev-rYKoty9qw@mail.gmail.com>
Message-ID: <CAP2Qz+Wx8i+Oh_JJcUnyne--q8hjQKvu6tiw6BQgwedzXfNZ+Q@mail.gmail.com>

On Thu, Jul 19, 2018 at 7:01 AM Calvin Spealman <cspealma at redhat.com> wrote:

> As an alternative suggestion: What if the count parameter to str.replace()
> counted from the right with negative values? That would be consistent with
> other things like indexing and slicing.
>

?That could certainly be made to work, but I think it fails the
discoverability test.  You'd have the [:]-syntax and replace working one
way, and split/rsplit, find/rfind and so on working another way.  Unless
you're proposing the negative index for the latter ones also??
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/195dda87/attachment.html>

From stephanh42 at gmail.com  Thu Jul 19 10:07:58 2018
From: stephanh42 at gmail.com (Stephan Houben)
Date: Thu, 19 Jul 2018 16:07:58 +0200
Subject: [Python-ideas] Performance improvements via static typing
In-Reply-To: <CAP2Qz+UL-y3Fn+4mdmuYwU-np8qdcECG9OmQ8n7ro-QysYc05g@mail.gmail.com>
References: <CABypBWrqKatZBKy3VBdzpr+o6cOgxhjpnqt7tzKW1+sraeJMSA@mail.gmail.com>
 <CAP2Qz+UL-y3Fn+4mdmuYwU-np8qdcECG9OmQ8n7ro-QysYc05g@mail.gmail.com>
Message-ID: <CAOOa=pMewmrDomQ=QyEYjUho965iGBQx94yfZGWXAhBpKBmJPg@mail.gmail.com>

You are aware of numba?

https://numba.pydata.org/

Stephan

Op do 19 jul. 2018 16:03 schreef Eric Fahlgren <ericfahlgren at gmail.com>:

> On Thu, Jul 19, 2018 at 6:52 AM Michael Hall <python-ideas at michaelhall.tech>
> wrote:
>
>> While I am aware of projects like Cython and mypy, it seems to make sense
>> for CPython to allow optional enforcement of type hints, with compiler
>> optimizations related to it to be used. While this would not receive the
>> same level of performance benefits as using ctypes directly, there do
>> appear to be various gains available here.
>>
>
> ?Just to make sure I understand:  In other words, they would no longer be
> "hints" but "guarantees".  This would allow an optimizer pass much greater
> latitude in code generation, somehow or other.?
>
> For purposes of illustration (this is not a proposal, just for
> clarification):
>
> @guaranteed_types
> def my_sqrt(x:c_double) -> c_double:
>     ...
>
> would tell the compiler that it's now possible to replace the general
> PyObject marshalling of this function with a pure-C one that only accepts
> doubles and woe be unto those who use it otherwise.
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/10bfb227/attachment-0001.html>

From cspealma at redhat.com  Thu Jul 19 10:09:41 2018
From: cspealma at redhat.com (Calvin Spealman)
Date: Thu, 19 Jul 2018 10:09:41 -0400
Subject: [Python-ideas] Revisiting str.rreplace()
In-Reply-To: <CAP2Qz+Wx8i+Oh_JJcUnyne--q8hjQKvu6tiw6BQgwedzXfNZ+Q@mail.gmail.com>
References: <CAN83G44dWT=+t68DS=abk29eM-WOQm3ZjOTDFzBNOVLtgr8xGA@mail.gmail.com>
 <CAP2Qz+VPpRBTAfn1OdYUMCD3oUoV143av7juWGvNkkXO-BxrGQ@mail.gmail.com>
 <CACo5Rz4U2N0v2KPaWspCrtBgm1cLk011hD7XPQev-rYKoty9qw@mail.gmail.com>
 <CAP2Qz+Wx8i+Oh_JJcUnyne--q8hjQKvu6tiw6BQgwedzXfNZ+Q@mail.gmail.com>
Message-ID: <CACo5Rz5eNYt0Bgs_L9UXr69JXTc7-58OBJntQaWgEfH1H8p++g@mail.gmail.com>

It would be consistent to apply it to other functions and I'd be in favour
of that, yes.

On Thu, Jul 19, 2018 at 10:06 AM, Eric Fahlgren <ericfahlgren at gmail.com>
wrote:

> On Thu, Jul 19, 2018 at 7:01 AM Calvin Spealman <cspealma at redhat.com>
> wrote:
>
>> As an alternative suggestion: What if the count parameter to
>> str.replace() counted from the right with negative values? That would be
>> consistent with other things like indexing and slicing.
>>
>
> ?That could certainly be made to work, but I think it fails the
> discoverability test.  You'd have the [:]-syntax and replace working one
> way, and split/rsplit, find/rfind and so on working another way.  Unless
> you're proposing the negative index for the latter ones also??
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/03ef0375/attachment.html>

From cspealma at redhat.com  Thu Jul 19 10:59:27 2018
From: cspealma at redhat.com (Calvin Spealman)
Date: Thu, 19 Jul 2018 10:59:27 -0400
Subject: [Python-ideas] PEP 505: None-aware operators: operators ?= and
 ?? and OR
In-Reply-To: <CA+msPNk0GrJZnQpXtjAM+uvHL9zyon11ApRUGMy4OVKWMEUOsQ@mail.gmail.com>
References: <CALD=Yf-0dLkGKN+1gb11nv4s0J4yiySehszZ+rS2BZVciR+LHA@mail.gmail.com>
 <CAPL7A87J_jWnvTpvGrv0_zzZ9yr_xT9asy9+C2JfsgFB6zt0wA@mail.gmail.com>
 <CACo5Rz5ohDPpOMtU-D-KZMdMj+AeP2MLfyH5sDr3uj8nQ1_BNw@mail.gmail.com>
 <CA+msPNk0GrJZnQpXtjAM+uvHL9zyon11ApRUGMy4OVKWMEUOsQ@mail.gmail.com>
Message-ID: <CACo5Rz7rZmNgw9OJSTndLSQ__+N7shhrfk-KCHCp6xpDajiy5w@mail.gmail.com>

The concatenated name reads a little weird. Alternatively, extend the
conditional expression syntax.

a = b if None c

equivalent to
a = b if b is None else c


On Thu, Jul 19, 2018 at 9:35 AM, Robert Vanden Eynde <robertve92 at gmail.com>
wrote:

> If we're about to use a new keyword, it could be infix too:
>
> a = b ifnone c
>
> Although the assignment version looks unusual:
>
> b ifnone= c
>
> Then with the "default b = c" would look like this:
>
> ifnone b = c
>
> Le jeu. 19 juil. 2018 ? 15:30, Calvin Spealman <cspealma at redhat.com> a
> ?crit :
>
>> Operators that only vary by case would be... confusing. I'm not super
>> keen on the other syntax (either the ?? or .? operators) but I do think
>> they read well in C# where they come from. Different things work in
>> different languages, some times.
>>
>> What about a new keyword: default
>>
>> So you'd write the above examples like this:
>>
>> default hi = len(a)  # Only executes the assignment if the left-hand is
>> None
>> default encoding = sys.getdefaultencoding()
>>
>> On Thu, Jul 19, 2018 at 9:06 AM, P?l Gr?n?s Drange <paal.drange at gmail.com
>> > wrote:
>>
>>> > I've started a subthread, just to discuss the ?= and ?? operators. And
>>> > something newish, that I call OR.
>>>
>>> I would think `||` would be much better.
>>>
>>> It could be a kind of "semantic or" which could use the aforementioned
>>> dunder has_value.
>>>
>>> -1, though, but to the general None-awareness.
>>>
>>> P?l
>>>
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/a6e33b1a/attachment.html>

From eric at trueblade.com  Thu Jul 19 11:17:13 2018
From: eric at trueblade.com (Eric V. Smith)
Date: Thu, 19 Jul 2018 11:17:13 -0400
Subject: [Python-ideas] Revisiting str.rreplace()
In-Reply-To: <CACo5Rz4U2N0v2KPaWspCrtBgm1cLk011hD7XPQev-rYKoty9qw@mail.gmail.com>
References: <CAN83G44dWT=+t68DS=abk29eM-WOQm3ZjOTDFzBNOVLtgr8xGA@mail.gmail.com>
 <CAP2Qz+VPpRBTAfn1OdYUMCD3oUoV143av7juWGvNkkXO-BxrGQ@mail.gmail.com>
 <CACo5Rz4U2N0v2KPaWspCrtBgm1cLk011hD7XPQev-rYKoty9qw@mail.gmail.com>
Message-ID: <900e9971-08d7-dad0-498d-8624ffa2d96f@trueblade.com>

On 7/19/2018 10:01 AM, Calvin Spealman wrote:
> As an alternative suggestion: What if the count parameter to 
> str.replace() counted from the right with negative values? That would be 
> consistent with other things like indexing and slicing.

We couldn't make this change because negative values already have a 
meaning: it's interpreted as a missing parameter:

 >>> 'abab'.replace('a', 'z')
'zbzb'
 >>> 'abab'.replace('a', 'z', 0)
'abab'
 >>> 'abab'.replace('a', 'z', 1)
'zbab'
 >>> 'abab'.replace('a', 'z', -1)
'zbzb'
 >>> 'abab'.replace('a', 'z', -2)
'zbzb'
 >>> 'abab'.replace('a', 'z', -100)
'zbzb'

I think .rreplace() is the better design.

Eric

From cspealma at redhat.com  Thu Jul 19 11:22:46 2018
From: cspealma at redhat.com (Calvin Spealman)
Date: Thu, 19 Jul 2018 11:22:46 -0400
Subject: [Python-ideas] Revisiting str.rreplace()
In-Reply-To: <900e9971-08d7-dad0-498d-8624ffa2d96f@trueblade.com>
References: <CAN83G44dWT=+t68DS=abk29eM-WOQm3ZjOTDFzBNOVLtgr8xGA@mail.gmail.com>
 <CAP2Qz+VPpRBTAfn1OdYUMCD3oUoV143av7juWGvNkkXO-BxrGQ@mail.gmail.com>
 <CACo5Rz4U2N0v2KPaWspCrtBgm1cLk011hD7XPQev-rYKoty9qw@mail.gmail.com>
 <900e9971-08d7-dad0-498d-8624ffa2d96f@trueblade.com>
Message-ID: <CACo5Rz78ovnUPszi6ROoyMH8Dj5V6EgbAmmsNDYDQpqcGj0iRA@mail.gmail.com>

If its treated as a missing parameter, and currently doesn't do anything,
then it wouldn't be used... right? and it could be safe to add behavior for
it... right?

On Thu, Jul 19, 2018 at 11:17 AM, Eric V. Smith <eric at trueblade.com> wrote:

> On 7/19/2018 10:01 AM, Calvin Spealman wrote:
>
>> As an alternative suggestion: What if the count parameter to
>> str.replace() counted from the right with negative values? That would be
>> consistent with other things like indexing and slicing.
>>
>
> We couldn't make this change because negative values already have a
> meaning: it's interpreted as a missing parameter:
>
> >>> 'abab'.replace('a', 'z')
> 'zbzb'
> >>> 'abab'.replace('a', 'z', 0)
> 'abab'
> >>> 'abab'.replace('a', 'z', 1)
> 'zbab'
> >>> 'abab'.replace('a', 'z', -1)
> 'zbzb'
> >>> 'abab'.replace('a', 'z', -2)
> 'zbzb'
> >>> 'abab'.replace('a', 'z', -100)
> 'zbzb'
>
> I think .rreplace() is the better design.
>
> Eric
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/ac6057ab/attachment-0001.html>

From python at mrabarnett.plus.com  Thu Jul 19 11:22:47 2018
From: python at mrabarnett.plus.com (MRAB)
Date: Thu, 19 Jul 2018 16:22:47 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAOOa=pN=EM1jsdBg5OvdP1WjDpC8pFr52FG8Asf2btNwq9BJTw@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <20180719103321.7c193820@fsol>
 <c49bd9b8-1242-fd07-29e8-9039bfaf9199@kynesim.co.uk>
 <CAFkYj5=Uy+QsUbGJNFNy6-bQezKo6BcvCa4EmXeJ6WQGPAhxBg@mail.gmail.com>
 <CAOOa=pN=EM1jsdBg5OvdP1WjDpC8pFr52FG8Asf2btNwq9BJTw@mail.gmail.com>
Message-ID: <58a4c3b6-028b-be25-b274-213ef00d2a93@mrabarnett.plus.com>

On 2018-07-19 14:26, Stephan Houben wrote:
> FWIW, I like ??
> 
> It is short and distinctive.
> There is prior art in this spelling in c#.
> It requires no new keyword, nor does it give new meaning to an existing one.
> 
> I understand why? ?[ needs to be spelled using only a single ?, but I am 
> afraid it will be used infrequently, and people will accidentally write
>  ?a??[x]
> which is legal but different.
> 
> I found the example code in the PEP using ?. and ?[ hard to read.
> ?? and ??= are compelling, though.
> 
> One more question: what does this do?
> 
> del x
> x ??= 42
> 
Well, what does this do?

del x
if x is None:
     x = 42

[snip]

From python at mrabarnett.plus.com  Thu Jul 19 11:33:27 2018
From: python at mrabarnett.plus.com (MRAB)
Date: Thu, 19 Jul 2018 16:33:27 +0100
Subject: [Python-ideas] Multi-core reference count garbage collection
In-Reply-To: <CALD=Yf97vYr+6Bot2aV6Q6H_YBx3ApR2HaS-duo2x4-TbUfR3w@mail.gmail.com>
References: <CALD=Yf-Ht4uDMHkvXp8i+xesw=e9zw4i+nAuRJFMVppNSYrboA@mail.gmail.com>
 <CAOOa=pMR==+=5jDvaVeQsWyRf6m=2cZ0r4p6R=89-+=JnuwBJw@mail.gmail.com>
 <CALD=Yf97vYr+6Bot2aV6Q6H_YBx3ApR2HaS-duo2x4-TbUfR3w@mail.gmail.com>
Message-ID: <902e400b-c7bb-82c5-6028-08d816717bc2@mrabarnett.plus.com>

On 2018-07-19 11:53, Jonathan Fine wrote:
> Hi Stephan
> 
> Thank you for the extract from the GC Handbook, which I think I may have 
> seen before. Yes, it is GOOD that it's an already known idea.
> 
> Searching for "buffered reference counting" I found
> https://mail.python.org/pipermail/python-dev/2016-October/146696.html
> in which Larry Hastings says that C-python "plays games with reference 
> counts" which makes implementing "buffered reference counting" harder. 
> And he gives examples.
> 
> Larry also writes [loc cit] about resurrecting objects. I don't know 
> what he means by this. It may be something to do with weak references. 
> Larry's post gives some details, in which difficulties may lie. In 
> particular, he writes
> ===
> https://mail.python.org/pipermail/python-dev/2016-October/146604.html
> It's my contention that this API [for weak references] is simply 
> untenable under the
> Gilectomy, and that it needs to change to returning a new (strong)
> reference.
> 
[snip]

When an object's refcount drops to 0, the object's __del__ method (if 
defined) is called, and then the object's memory can be reclaimed.

But what if the __del__ method creates a new reference to the object?

The object's refcount is no longer 0, so the object is no longer garbage.

That's called "resurrecting an object".

From python at mrabarnett.plus.com  Thu Jul 19 11:38:35 2018
From: python at mrabarnett.plus.com (MRAB)
Date: Thu, 19 Jul 2018 16:38:35 +0100
Subject: [Python-ideas] Revisiting str.rreplace()
In-Reply-To: <CACo5Rz78ovnUPszi6ROoyMH8Dj5V6EgbAmmsNDYDQpqcGj0iRA@mail.gmail.com>
References: <CAN83G44dWT=+t68DS=abk29eM-WOQm3ZjOTDFzBNOVLtgr8xGA@mail.gmail.com>
 <CAP2Qz+VPpRBTAfn1OdYUMCD3oUoV143av7juWGvNkkXO-BxrGQ@mail.gmail.com>
 <CACo5Rz4U2N0v2KPaWspCrtBgm1cLk011hD7XPQev-rYKoty9qw@mail.gmail.com>
 <900e9971-08d7-dad0-498d-8624ffa2d96f@trueblade.com>
 <CACo5Rz78ovnUPszi6ROoyMH8Dj5V6EgbAmmsNDYDQpqcGj0iRA@mail.gmail.com>
Message-ID: <6018db8e-5e27-94ed-9c98-583ff5045f3c@mrabarnett.plus.com>

On 2018-07-19 16:22, Calvin Spealman wrote:
> If its treated as a missing parameter, and currently doesn't do 
> anything, then it wouldn't be used... right? and it could be safe to add 
> behavior for it... right?
> 
Are you sure that it wouldn't break some existing code?

Plus, we already have .find/.rfind, .index/.rindex, etc, so 
.replace/.rreplace would be a better fit.

> On Thu, Jul 19, 2018 at 11:17 AM, Eric V. Smith <eric at trueblade.com 
> <mailto:eric at trueblade.com>> wrote:
> 
>     On 7/19/2018 10:01 AM, Calvin Spealman wrote:
> 
>         As an alternative suggestion: What if the count parameter to
>         str.replace() counted from the right with negative values? That
>         would be consistent with other things like indexing and slicing.
> 
> 
>     We couldn't make this change because negative values already have a
>     meaning: it's interpreted as a missing parameter:
> 
>      >>> 'abab'.replace('a', 'z')
>     'zbzb'
>      >>> 'abab'.replace('a', 'z', 0)
>     'abab'
>      >>> 'abab'.replace('a', 'z', 1)
>     'zbab'
>      >>> 'abab'.replace('a', 'z', -1)
>     'zbzb'
>      >>> 'abab'.replace('a', 'z', -2)
>     'zbzb'
>      >>> 'abab'.replace('a', 'z', -100)
>     'zbzb'
> 
>     I think .rreplace() is the better design.
> 

From eric at trueblade.com  Thu Jul 19 11:25:16 2018
From: eric at trueblade.com (Eric V. Smith)
Date: Thu, 19 Jul 2018 11:25:16 -0400
Subject: [Python-ideas] Revisiting str.rreplace()
In-Reply-To: <CACo5Rz78ovnUPszi6ROoyMH8Dj5V6EgbAmmsNDYDQpqcGj0iRA@mail.gmail.com>
References: <CAN83G44dWT=+t68DS=abk29eM-WOQm3ZjOTDFzBNOVLtgr8xGA@mail.gmail.com>
 <CAP2Qz+VPpRBTAfn1OdYUMCD3oUoV143av7juWGvNkkXO-BxrGQ@mail.gmail.com>
 <CACo5Rz4U2N0v2KPaWspCrtBgm1cLk011hD7XPQev-rYKoty9qw@mail.gmail.com>
 <900e9971-08d7-dad0-498d-8624ffa2d96f@trueblade.com>
 <CACo5Rz78ovnUPszi6ROoyMH8Dj5V6EgbAmmsNDYDQpqcGj0iRA@mail.gmail.com>
Message-ID: <11fe0bee-a200-70e0-3df5-f375bb33cddd@trueblade.com>

On 7/19/2018 11:22 AM, Calvin Spealman wrote:
> If its treated as a missing parameter, and currently doesn't do 
> anything, then it wouldn't be used... right? and it could be safe to add 
> behavior for it... right?

It currently does something: it replaces all instances, just as if you 
hadn't supplied a count (see my example below). You can't change its 
behavior.

Eric

> 
> On Thu, Jul 19, 2018 at 11:17 AM, Eric V. Smith <eric at trueblade.com 
> <mailto:eric at trueblade.com>> wrote:
> 
>     On 7/19/2018 10:01 AM, Calvin Spealman wrote:
> 
>         As an alternative suggestion: What if the count parameter to
>         str.replace() counted from the right with negative values? That
>         would be consistent with other things like indexing and slicing.
> 
> 
>     We couldn't make this change because negative values already have a
>     meaning: it's interpreted as a missing parameter:
> 
>      >>> 'abab'.replace('a', 'z')
>     'zbzb'
>      >>> 'abab'.replace('a', 'z', 0)
>     'abab'
>      >>> 'abab'.replace('a', 'z', 1)
>     'zbab'
>      >>> 'abab'.replace('a', 'z', -1)
>     'zbzb'
>      >>> 'abab'.replace('a', 'z', -2)
>     'zbzb'
>      >>> 'abab'.replace('a', 'z', -100)
>     'zbzb'
> 
>     I think .rreplace() is the better design.
> 
>     Eric
> 
> 

From p.f.moore at gmail.com  Thu Jul 19 12:29:17 2018
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 19 Jul 2018 17:29:17 +0100
Subject: [Python-ideas] Revisiting str.rreplace()
In-Reply-To: <11fe0bee-a200-70e0-3df5-f375bb33cddd@trueblade.com>
References: <CAN83G44dWT=+t68DS=abk29eM-WOQm3ZjOTDFzBNOVLtgr8xGA@mail.gmail.com>
 <CAP2Qz+VPpRBTAfn1OdYUMCD3oUoV143av7juWGvNkkXO-BxrGQ@mail.gmail.com>
 <CACo5Rz4U2N0v2KPaWspCrtBgm1cLk011hD7XPQev-rYKoty9qw@mail.gmail.com>
 <900e9971-08d7-dad0-498d-8624ffa2d96f@trueblade.com>
 <CACo5Rz78ovnUPszi6ROoyMH8Dj5V6EgbAmmsNDYDQpqcGj0iRA@mail.gmail.com>
 <11fe0bee-a200-70e0-3df5-f375bb33cddd@trueblade.com>
Message-ID: <CACac1F9sY0doY3TwvcVNfYWZudYMdD_zB3bLx4ytfvzr1iBXDw@mail.gmail.com>

On 19 July 2018 at 16:25, Eric V. Smith <eric at trueblade.com> wrote:
> It currently does something: it replaces all instances, just as if you
> hadn't supplied a count (see my example below). You can't change its
> behavior.

... without a deprecation cycle. Which is of course not worth it for
something which could much more easily be done by adding an rreplace
function - which is the real point of the comment.
Paul

From python-ideas at michaelhall.tech  Thu Jul 19 12:50:34 2018
From: python-ideas at michaelhall.tech (Michael Hall)
Date: Thu, 19 Jul 2018 12:50:34 -0400
Subject: [Python-ideas] Performance improvements via static typing
In-Reply-To: <CAP2Qz+UL-y3Fn+4mdmuYwU-np8qdcECG9OmQ8n7ro-QysYc05g@mail.gmail.com>
References: <CABypBWrqKatZBKy3VBdzpr+o6cOgxhjpnqt7tzKW1+sraeJMSA@mail.gmail.com>
 <CAP2Qz+UL-y3Fn+4mdmuYwU-np8qdcECG9OmQ8n7ro-QysYc05g@mail.gmail.com>
Message-ID: <CABypBWoUb5aBYGHSrcJKvxa0zxA_83-y+dkRh585FT9VVNpWhw@mail.gmail.com>

On Thu, Jul 19, 2018 at 10:02 AM, Eric Fahlgren <ericfahlgren at gmail.com>
wrote:

> On Thu, Jul 19, 2018 at 6:52 AM Michael Hall <python-ideas at michaelhall.tech>
> wrote:
>
>> While I am aware of projects like Cython and mypy, it seems to make sense
>> for CPython to allow optional enforcement of type hints, with compiler
>> optimizations related to it to be used. While this would not receive the
>> same level of performance benefits as using ctypes directly, there do
>> appear to be various gains available here.
>>
>
> ?Just to make sure I understand:  In other words, they would no longer be
> "hints" but "guarantees".  This would allow an optimizer pass much greater
> latitude in code generation, somehow or other.?
>
> For purposes of illustration (this is not a proposal, just for
> clarification):
>
> @guaranteed_types
> def my_sqrt(x:c_double) -> c_double:
>     ...
>
> would tell the compiler that it's now possible to replace the general
> PyObject marshalling of this function with a pure-C one that only accepts
> doubles and woe be unto those who use it otherwise.
>
>
Yes, Though this would likely require introducing new grammar of some sort
to have it treated by the compiler as a guarantee, rather than a hint.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/4f09278a/attachment.html>

From python-ideas at michaelhall.tech  Thu Jul 19 13:07:16 2018
From: python-ideas at michaelhall.tech (Michael Hall)
Date: Thu, 19 Jul 2018 13:07:16 -0400
Subject: [Python-ideas] Performance improvements via static typing
Message-ID: <CABypBWq8-1UaLo5jBr2tUdZFHiqkLGdB5Zc1=Ug_nbuT6NKw5A@mail.gmail.com>

On Thu, Jul 19, 2018 at 10:07 AM, Stephan Houben <stephanh42 at gmail.com>
wrote:

> You are aware of numba?
>
> https://numba.pydata.org/
>
> Stephan
>
> Op do 19 jul. 2018 16:03 schreef Eric Fahlgren <ericfahlgren at gmail.com>:
>
>> On Thu, Jul 19, 2018 at 6:52 AM Michael Hall
>> <python-ideas at michaelhall.tech> wrote:
>>
>>> While I am aware of projects like Cython and mypy, it seems to make
>>> sense for CPython to allow optional enforcement of type hints, with
>>> compiler optimizations related to it to be used. While this would not
>>> receive the same level of performance benefits as using ctypes directly,
>>> there do appear to be various gains available here.
>>>
>>
>> ?Just to make sure I understand:  In other words, they would no longer be
>> "hints" but "guarantees".  This would allow an optimizer pass much greater
>> latitude in code generation, somehow or other.?
>>
>> For purposes of illustration (this is not a proposal, just for
>> clarification):
>>
>> @guaranteed_types
>> def my_sqrt(x:c_double) -> c_double:
>>     ...
>>
>> would tell the compiler that it's now possible to replace the general
>> PyObject marshalling of this function with a pure-C one that only accepts
>> doubles and woe be unto those who use it otherwise.
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
Less so than I probably should have been given the idea I'm pitching. I've
given it a quick look, and at a glance, it seems to already be capable of
the desired behavior specifically centered around performance. I do still
think it *may* be beneficial to have in the CPython reference
implementation alongside a standard grammar, which could further enable the
various libraries and python implementations to make use of the additional
knowledge that the type hint is intended as a more than a hint.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/45132c23/attachment-0001.html>

From tjreedy at udel.edu  Thu Jul 19 14:19:04 2018
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 19 Jul 2018 14:19:04 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <23376.19632.386221.927225@turnbull.sk.tsukuba.ac.jp>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <23376.19632.386221.927225@turnbull.sk.tsukuba.ac.jp>
Message-ID: <piqkip$qqu$1@blaine.gmane.org>

On 7/19/2018 4:32 AM, Stephen J. Turnbull wrote:
> Chris Angelico writes later in thread:
>   > On Thu, Jul 19, 2018 at 9:55 AM, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
> 
>   > Personally, I'm +0 on this. It'd be a few small wins here and there,
>   > nothing huge, and I could easily live without it; but it's something
>   > that I know some people will love.
> 
> I am 100% in sync with the reasoning, but -0 on the PEP (and only that
> high because the advocates are so passionate).
> 
> To be honest, code transformations like this
> 
>   >      class BaseUploadObject(object):
>   >          def find_content_type(self, filename):
>   >              ctype, encoding = mimetypes.guess_type(filename)
>   >              if ctype is None:
>   >                  return 'application/octet-stream'
>   >              else:
>   >                  return ctype
> 
> to this
> 
>   >      class BaseUploadObject(object):
>   >          def find_content_type(self, filename):
>   >              ctype, encoding = mimetypes.guess_type(filename)
>   >              return ctype ?? 'application/octet-stream'
> 
> make me cringe.  Exactly one of two things is true:

It seems to me that the problem is returning None.  Guess_type should 
have default='application/octet-stream' (or whatever *is* the default) 
in its signature.

The premise of returning None as default is that users can follow with 
conditional code.  If we now think that this is too much of a burden, we 
should stop returning None, at least by default.

> 1.  mimetypes.guess_type guarantees that the only falsie it will ever
>      return is None, or
> 
> 2.  it doesn't.

or
3. it never returns anything other than a non-blank string unless the 
user so requests.


-- 
Terry Jan Reedy


From brenbarn at brenbarn.net  Thu Jul 19 14:36:14 2018
From: brenbarn at brenbarn.net (Brendan Barnwell)
Date: Thu, 19 Jul 2018 11:36:14 -0700
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPTjJmquqsqX8XL5RGdvebwHw-vgx15sQU2Qbyg3SKz3WHso2A@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAFYqXL8GsTyAHEkrQqSDpv_+bCcJAVv2j-0d_zwNuLLfK7Cuuw@mail.gmail.com>
 <CAPTjJmqPS_wiFTJaJ8YUKJfWsNcCX=h9AQ1EmWFv6Xf9Wes4tA@mail.gmail.com>
 <5B502A80.3080406@canterbury.ac.nz>
 <CAPTjJmquqsqX8XL5RGdvebwHw-vgx15sQU2Qbyg3SKz3WHso2A@mail.gmail.com>
Message-ID: <5B50DA1E.9030904@brenbarn.net>

On 2018-07-19 02:11, Chris Angelico wrote:
> Okay. What about bitwise operators, then? They don't have centuries of
> mathematical backing to support them, yet it isn't considered
> "unpythonic" to have &|^~ peppering our code. Judging by the current
> levels of backlash against symbolic operators, it would have been
> better to use "more explicit" function calls for all bitwise
> operations. Coalescing None to a value is_at least_  as common as
> performing bit manipulations in integers.

	The use of & to mean "and" does indeed have centuries of backing to 
support it.  The other operators, and the use of & to specifically mean 
bitwise-and, are newer, but still go back decades in other programming 
languages.  Moreover, the fact that these are bitwise versions of more 
general logical operations (conjunction, disjunction, etc.) means that 
they can be overridden for custom types in a way that is still 
intuitive.  For instance, sets override & to mean "intersection" because 
it means "everything that is in set A AND in set B", and similarly for | 
and ^.

	So basically I do not regard these operators as bitwise operators. 
They are symbols standing for logical operations.  Their default 
implementations on built-in numeric types do bitwise operations, but 
that is not what the symbols "mean".  Their meaning is more general and 
bitwise operations are one reasonable narrowing of that meaning for use 
in the context of builtin numeric types, but there are many other uses 
of these symbols that are equally consistent with their meaning.

	To me, this notion of symbols with an abstract meaning which can be 
narrowed for particular types by overriding magic methods is one of 
Python's greatest strengths.  + doesn't mean "add two numbers", it means 
"add", and that makes sense for many types, and they can override 
__add__ to define sensible behavior for their own purposes.  [] doesn't 
mean "get list item" it means "get", and types can override __getitem__ 
to define sensible behavior for their own purposes.

	As far as I can see, these null-coalescing operators would break that 
model.  The PEP doesn't seem to provide for a "real" magic method 
allowing users to override the actual behavior of the method.  (You can 
only override __has_value__ to hook into it, but not define by fiat what 
A ?? B does, as you can with other operators.)  And I think the reason 
for this is that the operator itself is too specific, much more specific 
in semantics than other operators.  (I had similar doubts about adding 
the matrix-multiplication operator @.)

	People keep saying that this null-coalescing behavior is so common and 
useful, etc., but that hasn't been my experience at all.  In my 
experience, the desire to shortcut this kind of logic is more often a 
sign of corner-cutting and insufficiently specified data formats, and is 
likely to cause bugs later on.  Eventually it has to actually matter 
whether something is None or not, and these operators just kick that can 
down the road.  In terms of their abstract meaning, they are not 
remotely close to as common or useful as operators like & and |.

-- 
Brendan Barnwell
"Do not follow where the path may lead.  Go, instead, where there is no 
path, and leave a trail."
    --author unknown

From brenbarn at brenbarn.net  Thu Jul 19 14:39:50 2018
From: brenbarn at brenbarn.net (Brendan Barnwell)
Date: Thu, 19 Jul 2018 11:39:50 -0700
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180719133820.GV7318@ando.pearwood.info>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
Message-ID: <5B50DAF6.7070406@brenbarn.net>

On 2018-07-19 06:38, Steven D'Aprano wrote:
> *Its just spelling*. If it is a useful and well-defined feature, we'll
> get used to the spelling soon enough.
>
> That's not to say that spelling is not important *at all*, or that we
> should never prefer words to symbols. But if the only objection we have
> is "this is useful but I don't like the spelling so -1" then that's
> usually a pretty weak argument against the feature.

	But we already have a spelling for the most common case.  It is:

x = a if a is not None else b

	That is the only use case of any of these operators that is actually 
common enough for me to care about --- but it's still not common enough 
to warrant the creation of a new operator, let alone multiple new operators.

-- 
Brendan Barnwell
"Do not follow where the path may lead.  Go, instead, where there is no 
path, and leave a trail."
    --author unknown

From tjreedy at udel.edu  Thu Jul 19 14:40:11 2018
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 19 Jul 2018 14:40:11 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180719103321.7c193820@fsol>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <20180719103321.7c193820@fsol>
Message-ID: <piqlqb$hlf$1@blaine.gmane.org>

On 7/19/2018 4:33 AM, Antoine Pitrou wrote:
> 
> This is adding a whole range of new operators without enough of a use
> case.  It is also making code harder to read, as evaluation can stop at
> any of the "?*" operators.  And it looks like noise (or like Perl 6,
> which is the same).
> 
> There is a use case I sympathize with: the argument-is-None case.  For
> that I would suggest a simpler form:  "A else B" which would evaluate
> to A if A is not None, otherwise to B (parentheses may be mandatory).
> 
> So e.g. one of the examples would read:
> 
>       def insort_right(a, x, lo=0, hi=None):
>           # ...
>           hi = hi else len(a)
>           # ...

I like this.  (A or B) and (A and B)  could now* be explained as an 
abbreviations of

A if A else B
A if not A else B

but with A only evaluated once, as in

tem = A; tem if tem else B
tem = A; tem if not A else B

(A if A else B) is equivalent to (A if bool(A) is not False else B)

(A else B) is then easily explained as an abbreviation of

A if A is not None else B

that only evaluates A once.


* In the future, tem if (tem := A) else B

-- 
Terry Jan Reedy


From elazarg at gmail.com  Thu Jul 19 14:43:29 2018
From: elazarg at gmail.com (Elazar)
Date: Thu, 19 Jul 2018 14:43:29 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <5B50DA1E.9030904@brenbarn.net>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAFYqXL8GsTyAHEkrQqSDpv_+bCcJAVv2j-0d_zwNuLLfK7Cuuw@mail.gmail.com>
 <CAPTjJmqPS_wiFTJaJ8YUKJfWsNcCX=h9AQ1EmWFv6Xf9Wes4tA@mail.gmail.com>
 <5B502A80.3080406@canterbury.ac.nz>
 <CAPTjJmquqsqX8XL5RGdvebwHw-vgx15sQU2Qbyg3SKz3WHso2A@mail.gmail.com>
 <5B50DA1E.9030904@brenbarn.net>
Message-ID: <CAPw6O2R6w+S5gUx63yWMNYchqJwjmZh7TSPhBjTt8Z4Spv48Bg@mail.gmail.com>

On Thu, Jul 19, 2018 at 11:37 AM Brendan Barnwell <brenbarn at brenbarn.net>
wrote:

>
>         As far as I can see, these null-coalescing operators would break
> that
> model.  The PEP doesn't seem to provide for a "real" magic method
> allowing users to override the actual behavior of the method.  (You can
> only override __has_value__ to hook into it, but not define by fiat what
> A ?? B does, as you can with other operators.)  And I think the reason
> for this is that the operator itself is too specific, much more specific
> in semantics than other operators.  (I had similar doubts about adding
> the matrix-multiplication operator @.)
>

It is just as specific as the `is` operator, and for the same reason.

Elazar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/705902cf/attachment-0001.html>

From stephanh42 at gmail.com  Thu Jul 19 14:45:01 2018
From: stephanh42 at gmail.com (Stephan Houben)
Date: Thu, 19 Jul 2018 20:45:01 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <5B50DA1E.9030904@brenbarn.net>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAFYqXL8GsTyAHEkrQqSDpv_+bCcJAVv2j-0d_zwNuLLfK7Cuuw@mail.gmail.com>
 <CAPTjJmqPS_wiFTJaJ8YUKJfWsNcCX=h9AQ1EmWFv6Xf9Wes4tA@mail.gmail.com>
 <5B502A80.3080406@canterbury.ac.nz>
 <CAPTjJmquqsqX8XL5RGdvebwHw-vgx15sQU2Qbyg3SKz3WHso2A@mail.gmail.com>
 <5B50DA1E.9030904@brenbarn.net>
Message-ID: <CAOOa=pPU2-KvWJotvO3D7sJ=q-uJGbZBw3khmHdReHZ-m2zuwA@mail.gmail.com>

Let me just address this point:

2018-07-19 20:36 GMT+02:00 Brendan Barnwell <brenbarn at brenbarn.net>:

>         As far as I can see, these null-coalescing operators would break
> that model.  The PEP doesn't seem to provide for a "real" magic method
> allowing users to override the actual behavior of the method.  (You can
> only override __has_value__ to hook into it, but not define by fiat what A
> ?? B does, as you can with other operators.)  And I think the reason for
> this is that the operator itself is too specific, much more specific in
> semantics than other operators.  (I had similar doubts about adding the
> matrix-multiplication operator @.)
>

I think the actual reason is that it is a short-cutting operator, and none
of
the shortcutting operators (and, or,  if/else) have an associated method.
They cannot have, since they induce a non-standard evaluation order,
hence their effect cannot be emulated with a method invocation.

Stephan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/bb8da553/attachment.html>

From tjreedy at udel.edu  Thu Jul 19 14:47:11 2018
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 19 Jul 2018 14:47:11 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <c49bd9b8-1242-fd07-29e8-9039bfaf9199@kynesim.co.uk>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <20180719103321.7c193820@fsol>
 <c49bd9b8-1242-fd07-29e8-9039bfaf9199@kynesim.co.uk>
Message-ID: <piqm7f$dpt$1@blaine.gmane.org>

On 7/19/2018 8:46 AM, Rhodri James wrote:
> On 19/07/18 09:33, Antoine Pitrou wrote:
>> There is a use case I sympathize with: the argument-is-None case.? For
>> that I would suggest a simpler form:? "A else B" which would evaluate
>> to A if A is not None, otherwise to B (parentheses may be mandatory).
>>
>> So e.g. one of the examples would read:
>>
>> ????? def insort_right(a, x, lo=0, hi=None):
>> ????????? # ...
>> ????????? hi = hi else len(a)
>> ????????? # ...
> 
> Much as I would like a keyword, "else" is the wrong one.? It implies we 
> are dealing with truthiness, which we aren't,

'A else B' is about the truthiness of 'A is not None', just as
'A or B' is about the truthiness of 'bool(a) is not False'.
  See my response to Antoine.

A else B' would abbreviate 'tem = A; tem if tem is not None else B'

> and lays a subtle semantic trap as a consequence.

I think most anyone should be able to get that 'A else B' is similar to 
'A or B' but must be different.

> If anyone can think of a good word for "if it isn't None, otherwise", 
> I'd be all for it :-)
-- 
Terry Jan Reedy



From brenbarn at brenbarn.net  Thu Jul 19 14:48:16 2018
From: brenbarn at brenbarn.net (Brendan Barnwell)
Date: Thu, 19 Jul 2018 11:48:16 -0700
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPw6O2R6w+S5gUx63yWMNYchqJwjmZh7TSPhBjTt8Z4Spv48Bg@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAFYqXL8GsTyAHEkrQqSDpv_+bCcJAVv2j-0d_zwNuLLfK7Cuuw@mail.gmail.com>
 <CAPTjJmqPS_wiFTJaJ8YUKJfWsNcCX=h9AQ1EmWFv6Xf9Wes4tA@mail.gmail.com>
 <5B502A80.3080406@canterbury.ac.nz>
 <CAPTjJmquqsqX8XL5RGdvebwHw-vgx15sQU2Qbyg3SKz3WHso2A@mail.gmail.com>
 <5B50DA1E.9030904@brenbarn.net>
 <CAPw6O2R6w+S5gUx63yWMNYchqJwjmZh7TSPhBjTt8Z4Spv48Bg@mail.gmail.com>
Message-ID: <5B50DCF0.7040707@brenbarn.net>

On 2018-07-19 11:43, Elazar wrote:
>
> On Thu, Jul 19, 2018 at 11:37 AM Brendan Barnwell <brenbarn at brenbarn.net
> <mailto:brenbarn at brenbarn.net>> wrote:
>
>
>              As far as I can see, these null-coalescing operators would
>     break that
>     model.  The PEP doesn't seem to provide for a "real" magic method
>     allowing users to override the actual behavior of the method.  (You can
>     only override __has_value__ to hook into it, but not define by fiat
>     what
>     A ?? B does, as you can with other operators.)  And I think the reason
>     for this is that the operator itself is too specific, much more
>     specific
>     in semantics than other operators.  (I had similar doubts about adding
>     the matrix-multiplication operator @.)
>
>
> It is just as specific as the `is` operator, and for the same reason.

	What reason is that?

	The "is" operator is the ONLY operator that is un-overridable in this 
way, and that's because by its nature it is concerned with object 
identity, which we don't want people to be able to redefine.  (Okay, the 
logical operators like "and" and "or" are also un-overridable, but 
that's not because we don't want people to be able to override them, 
it's because it's hard to figure out a way to do it practically without 
giving up short-circuiting.)

	From my perspective "A if A is None else B" is just not remotely close 
to being as special as "A is B".  If we didn't have the "is" operator 
there would be no way to do what it does.  Everything that these new 
operators do can already be done.

-- 
Brendan Barnwell
"Do not follow where the path may lead.  Go, instead, where there is no 
path, and leave a trail."
    --author unknown

From rosuav at gmail.com  Thu Jul 19 14:51:47 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 20 Jul 2018 04:51:47 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAOOa=pPU2-KvWJotvO3D7sJ=q-uJGbZBw3khmHdReHZ-m2zuwA@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAFYqXL8GsTyAHEkrQqSDpv_+bCcJAVv2j-0d_zwNuLLfK7Cuuw@mail.gmail.com>
 <CAPTjJmqPS_wiFTJaJ8YUKJfWsNcCX=h9AQ1EmWFv6Xf9Wes4tA@mail.gmail.com>
 <5B502A80.3080406@canterbury.ac.nz>
 <CAPTjJmquqsqX8XL5RGdvebwHw-vgx15sQU2Qbyg3SKz3WHso2A@mail.gmail.com>
 <5B50DA1E.9030904@brenbarn.net>
 <CAOOa=pPU2-KvWJotvO3D7sJ=q-uJGbZBw3khmHdReHZ-m2zuwA@mail.gmail.com>
Message-ID: <CAPTjJmo_7H8ppV6JnZzQu5B6DdYxympenws8YcaWyYSCqAvhLQ@mail.gmail.com>

On Fri, Jul 20, 2018 at 4:45 AM, Stephan Houben <stephanh42 at gmail.com> wrote:
> Let me just address this point:
>
> 2018-07-19 20:36 GMT+02:00 Brendan Barnwell <brenbarn at brenbarn.net>:
>>
>>         As far as I can see, these null-coalescing operators would break
>> that model.  The PEP doesn't seem to provide for a "real" magic method
>> allowing users to override the actual behavior of the method.  (You can only
>> override __has_value__ to hook into it, but not define by fiat what A ?? B
>> does, as you can with other operators.)  And I think the reason for this is
>> that the operator itself is too specific, much more specific in semantics
>> than other operators.  (I had similar doubts about adding the
>> matrix-multiplication operator @.)
>
>
> I think the actual reason is that it is a short-cutting operator, and none
> of
> the shortcutting operators (and, or,  if/else) have an associated method.
> They cannot have, since they induce a non-standard evaluation order,
> hence their effect cannot be emulated with a method invocation.

Also for the same reason that the 'is' operator doesn't have a
corresponding dunder. You can't ask an object if it's the same object
as another; it either is or is not, intrinsically. It's the same here;
it either is None, or is not None, intrinsically. These new operators
have very clearly defined semantics involving the special object None,
and they short-circuit; two good reasons NOT to have them overridable.

ChrisA

From tjreedy at udel.edu  Thu Jul 19 15:23:01 2018
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 19 Jul 2018 15:23:01 -0400
Subject: [Python-ideas] PEP 505: None-aware operators: operators ?= and
 ?? and OR
In-Reply-To: <CALD=Yf-0dLkGKN+1gb11nv4s0J4yiySehszZ+rS2BZVciR+LHA@mail.gmail.com>
References: <CALD=Yf-0dLkGKN+1gb11nv4s0J4yiySehszZ+rS2BZVciR+LHA@mail.gmail.com>
Message-ID: <piqoai$49e$1@blaine.gmane.org>

On 7/19/2018 8:30 AM, Jonathan Fine wrote:
> Hi
> 
>> There is a formatted version of this PEP at
>> https://www.python.org/dev/peps/pep-0505/
> 
> I've taken a look at this, and have some comments on the first two
> examples drawn from standard library code. (And a very grateful +10
> for writing a script to find such examples.)
> 
> I've started a subthread, just to discuss the ?= and ?? operators. And
> something newish, that I call OR.
> 
> FIRST EXAMPLE
> The first example is
> ---
>  From bisect.py:
> def insort_right(a, x, lo=0, hi=None):
>      # ...
>      if hi is None:
>          hi = len(a)
> ---
> 
> Here, None is a sentinel value. The simpler code
> ---
>    hi = hi or len(a)
> ---
> fails when hi is zero (or any other value that is False in the boolean context).
> 
> This can be fixed by introducing a new operator OR which is similar to
> 'or' but has the semantics this example requires. Thus, given OR we
> can write
> ---
>    hi = hi OR len(a)

Antoine Pitrou proposed the same thing, using the existing 'else' as the 
spelling.  'hi else len(s)' abbreviates the existing
'hi if hi is not None else len(s)'.
with the boilerplate 'if x is not None' removed.

> ---
> where (A OR B) returns A if A is not None, otherwise it returns B.
> 
> (Recall that (A or B) returns A if bool(A), otherwise it returns B.)

To make the parallel more obvious, (A or B) also abbreviates
tem = A; tem if bool(tem) is not False else B

-- 
Terry Jan Reedy


From python at mrabarnett.plus.com  Thu Jul 19 15:41:41 2018
From: python at mrabarnett.plus.com (MRAB)
Date: Thu, 19 Jul 2018 20:41:41 +0100
Subject: [Python-ideas] PEP 505: None-aware operators: operators ?= and
 ?? and OR
In-Reply-To: <piqoai$49e$1@blaine.gmane.org>
References: <CALD=Yf-0dLkGKN+1gb11nv4s0J4yiySehszZ+rS2BZVciR+LHA@mail.gmail.com>
 <piqoai$49e$1@blaine.gmane.org>
Message-ID: <f4c91961-1b48-f6b0-6e74-09f405b64fe2@mrabarnett.plus.com>

On 2018-07-19 20:23, Terry Reedy wrote:
> On 7/19/2018 8:30 AM, Jonathan Fine wrote:
>> Hi
>> 
>>> There is a formatted version of this PEP at
>>> https://www.python.org/dev/peps/pep-0505/
>> 
>> I've taken a look at this, and have some comments on the first two
>> examples drawn from standard library code. (And a very grateful +10
>> for writing a script to find such examples.)
>> 
>> I've started a subthread, just to discuss the ?= and ?? operators. And
>> something newish, that I call OR.
>> 
>> FIRST EXAMPLE
>> The first example is
>> ---
>>  From bisect.py:
>> def insort_right(a, x, lo=0, hi=None):
>>      # ...
>>      if hi is None:
>>          hi = len(a)
>> ---
>> 
>> Here, None is a sentinel value. The simpler code
>> ---
>>    hi = hi or len(a)
>> ---
>> fails when hi is zero (or any other value that is False in the boolean context).
>> 
>> This can be fixed by introducing a new operator OR which is similar to
>> 'or' but has the semantics this example requires. Thus, given OR we
>> can write
>> ---
>>    hi = hi OR len(a)
> 
> Antoine Pitrou proposed the same thing, using the existing 'else' as the
> spelling.  'hi else len(s)' abbreviates the existing
> 'hi if hi is not None else len(s)'.
> with the boilerplate 'if x is not None' removed.
> 
>> ---
>> where (A OR B) returns A if A is not None, otherwise it returns B.
>> 
>> (Recall that (A or B) returns A if bool(A), otherwise it returns B.)
> 
> To make the parallel more obvious, (A or B) also abbreviates
> tem = A; tem if bool(tem) is not False else B
> 
If you want a not-too-long keyword, how about "anders"? :-)

From steve.dower at python.org  Thu Jul 19 17:28:47 2018
From: steve.dower at python.org (Steve Dower)
Date: Thu, 19 Jul 2018 14:28:47 -0700
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPTjJmo_7H8ppV6JnZzQu5B6DdYxympenws8YcaWyYSCqAvhLQ@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAFYqXL8GsTyAHEkrQqSDpv_+bCcJAVv2j-0d_zwNuLLfK7Cuuw@mail.gmail.com>
 <CAPTjJmqPS_wiFTJaJ8YUKJfWsNcCX=h9AQ1EmWFv6Xf9Wes4tA@mail.gmail.com>
 <5B502A80.3080406@canterbury.ac.nz>
 <CAPTjJmquqsqX8XL5RGdvebwHw-vgx15sQU2Qbyg3SKz3WHso2A@mail.gmail.com>
 <5B50DA1E.9030904@brenbarn.net>
 <CAOOa=pPU2-KvWJotvO3D7sJ=q-uJGbZBw3khmHdReHZ-m2zuwA@mail.gmail.com>
 <CAPTjJmo_7H8ppV6JnZzQu5B6DdYxympenws8YcaWyYSCqAvhLQ@mail.gmail.com>
Message-ID: <dc623fec-26be-4b74-f13b-63d74eb3824a@python.org>

Thanks everyone for the feedback and discussion so far. I want to 
address some of the themes, so apologies for not quoting individuals and 
for doing this in one post instead of twenty.

------


* "It looks like line noise"

Thanks for the feedback. There's nothing constructive for me to take 
from this.

* "I've never needed this"

Also not very actionable, but as background I'll say that this was 
exactly my argument against adding them to C#. But my coding style has 
adapted to suit (for example, I'm more likely to use "null" as a default 
value and have a single function implementation than two 
mostly-duplicated overloads).

* "It makes it more complex"
* "It's harder to follow the flow"

Depends on your measure of complexity. For me, I prioritise "area under 
the indentation" as my preferred complexity metric (more lines*indents 
== more complex), as well as left-to-right reading of each line (more 
random access == more complex). By these measures, ?. significantly 
reduces the complexity over any of the current or future alternatives::

def f(a=None):
     name = 'default'
     if a is not None:
         user = a.get_user()
         if user is not None:
             name = user.name
     print(name)

def f(a=None):
     if a is not None:
         user = a.get_user()
         name = user.name if user is not None else 'default'
         print(name)
     else
         print('default')

def f(a=None):
     user = a.get_user() if a is not None else None
     name = user.name if user is not None else 'default'
     print(name)

def f(a=None):
     print(user.name
           if (user := a.get_user() if a is not None else None) is not None
           else 'default')

def f(a=None):
     print(a?.get_user()?.name ?? 'none')

* "We have 'or', we don't need '??'"

Nearly-agreed, but I think the tighter binding on ?? makes it more 
valuable and tighter test make it valuable in place of 'or'.

For example, compare:

     a ** b() or 2 # actual:   (a ** b()) or 2
     a ** b() ?? 2 # proposed:  a ** (b() ?? 2)

In the first, the presence of 'or' implies that either b() or __pow__(a, 
b()) could return a non-True value. This is correct (it could return 0 
if a == 0). And the current precedence results in the result of __pow__ 
being used for the check.

In the second one, the presence of the '??' implies that either b() or 
__pow__(a, b()) could return None. The latter should never happen, and 
so the choices are to make the built-in types propagate Nones when 
passed None (uhh... no) or to make '??' bind to the closer part of the 
expression.

(If you don't think it's likely enough that a function could return 
[float, None], then assume 'a ** b?.c ?? 2' instead.)

* "We could have '||', we don't need '??'"

Perhaps, though this is basically just choosing the bikeshed colour. In 
the absence of a stronger argument, matching existing languages 
equivalent operators instead of operators that do different things in 
those languages should win.

* "We could have 'else', we don't need '??'"

This is the "a else 'default'" rather than "a ?? 'default'" proposal, 
which I do like the look of, but I think it will simultaneously mess 
with operator precedence and also force me to search for the 'if' that 
we actually need to be comparing "(a else 'default')" vs. "a ?? 'default'"::

     x = a if b else c else d
     x = a if (b else c) else d
     x = a if b else (c else d)

* "It's not clear whether it's 'is not None' or 'hasattr' checks"

I'm totally sympathetic to this. Ultimately, like everything else, this 
is a concept that has to be taught/learned rather than known intrinsically.

The main reasons for not having 'a?.b' be directly equivalent to 
getattr(a, 'b', ???) is that you lose the easy ability to find typos, 
and we also already have the getattr() approach.

(Aside: in this context, why should the result be 'None' if an attribute 
is missing? For None, the None value propagates (getattr(a, 'b', a)), 
while for falsies you could argue the same thing applies. But for a 
silently handled AttributeError? You still have to make the case that 
None is special here, just special as a return value vs. special as a test.)

* "The semantics of this example changed from getattr() with ?."

Yes, this was a poor example. On re-reading, all of the checks are 
indeed looking for optional attributes, rather than looking them up on 
optional targets. I'll find a better one (I've certainly seen and/or 
written code like this that was intended to avoid crashing on None, but 
I stopped my search of the stdlib too soon after finding this example).

* "Bitwise operators"

Uh... yeah. Have fun over there :)

* "Assumes the only falsie ever returned [in some context] is None"

I argue that it assumes the only falsie you want to replace with a 
different value is None. In many cases, I'd expect the None to be 
replaced with a falsie of the intended type:

     x = maybe_get_int() ?? 0
     y = maybe_get_list() ?? []

Particularly for the second case, if you are about to mutate the list, 
then it could be very important that you don't replace the provided 
reference with your own, just because it happens to be empty right now 
(this is the logging example in the PEP). Supporting a default value in 
the get() calls is obviously a better way to do this though, provided 
you have the ability to modify the API.

But for me, the ?? operator makes its strongest case when you are 
getting attributes from potentially None values:

     x = maybe_get()?.get_int() ?? 0
     y = maybe_get()?.get_list() ?? []

In this case, using 'or' may replace an intentionally falsie value with 
your own, while using a default parameter is still going to leave you 
with None if the first maybe_get() returned nothing.

"?." without "??" feels like a trap, while "??" without "?." feels 
largely unnecessary. But both together lets you turn many lines of code 
into a much shorter snippet that reads left-to-right and lets you assume 
success until you reach the "??". That, for me, is peak readability.


From steve at pearwood.info  Thu Jul 19 19:32:57 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 20 Jul 2018 09:32:57 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <5B50DAF6.7070406@brenbarn.net>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info> <5B50DAF6.7070406@brenbarn.net>
Message-ID: <20180719233256.GW7318@ando.pearwood.info>

On Thu, Jul 19, 2018 at 11:39:50AM -0700, Brendan Barnwell wrote:
> On 2018-07-19 06:38, Steven D'Aprano wrote:
> >*Its just spelling*. If it is a useful and well-defined feature, we'll
> >get used to the spelling soon enough.
> >
> >That's not to say that spelling is not important *at all*, or that we
> >should never prefer words to symbols. But if the only objection we have
> >is "this is useful but I don't like the spelling so -1" then that's
> >usually a pretty weak argument against the feature.
> 
> 	But we already have a spelling for the most common case.  It is:
> 
> x = a if a is not None else b
> 
> 	That is the only use case of any of these operators that is actually 
> common enough for me to care about --- but it's still not common enough 
> to warrant the creation of a new operator, let alone multiple new operators.


That's a reasonable argument: "there's no need for this operator 
because...", albeit it is a subjective argument. (There's no objective 
rule about how common an operation should be before allowing it to be an 
operator.)

What's not a reasonable argument is "I see that there could be a need 
for this operator, but I don't like the spelling so -1 on the entire 
proposal", which was my point. (One of my points.)


-- 
Steve

From mike at selik.org  Thu Jul 19 19:42:18 2018
From: mike at selik.org (Michael Selik)
Date: Thu, 19 Jul 2018 19:42:18 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <piqkip$qqu$1@blaine.gmane.org>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <23376.19632.386221.927225@turnbull.sk.tsukuba.ac.jp>
 <piqkip$qqu$1@blaine.gmane.org>
Message-ID: <CADqi797C2tL7AVc3d3QJH_Mv=UeMM6VWKzhmWEfQEckr4xX67Q@mail.gmail.com>

On Thu, Jul 19, 2018 at 2:19 PM Terry Reedy <tjreedy at udel.edu> wrote:

> It seems to me that the problem is returning None.


If functions raised errors instead of returning None, we wouldn't have so
much trouble. Much of the benefit of PEP-572 was handling Nones and similar
sentinel values.

Instead of providing better tools for handling Nones, I'd rather encourage
folks to raise exceptions with a nice tool like PEP 463, exception-catching
expressions. Re-reading Guido's explanation for rejecting PEP 463, I now
understand the acceptance of PEP 572 better, as he also disagrees with the
preference for EAFP over LBYL.

For what it's worth, I agree with the folks that think ?. and ?[ create too
much visual clutter and are too easy to overlook or misunderstand. The ??=
and ?? operators will have spaces around them, so they're easier to read.
However, their benefit is so minor, if any, that it's better to stick with
the status quo.

I'd rather write one of these patterns:

    # when x was assigned elsewhere
    if x is None:
        x = value

    # when x is typically None
    value if x is None else x

    # when x is typically not None
    x if x is not None else value

    # when I'm playing code golf
    x or value


In many of these cases I'd rather write 4 lines of code. It reads how I'd
speak the logic to another person, though out-loud I'd probably add a
"then" and say "otherwise" instead of "else":

    if x is None:
        y = a
    else:
        y = b
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/7b7dde12/attachment.html>

From mike at selik.org  Thu Jul 19 19:57:16 2018
From: mike at selik.org (Michael Selik)
Date: Thu, 19 Jul 2018 19:57:16 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <dc623fec-26be-4b74-f13b-63d74eb3824a@python.org>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAFYqXL8GsTyAHEkrQqSDpv_+bCcJAVv2j-0d_zwNuLLfK7Cuuw@mail.gmail.com>
 <CAPTjJmqPS_wiFTJaJ8YUKJfWsNcCX=h9AQ1EmWFv6Xf9Wes4tA@mail.gmail.com>
 <5B502A80.3080406@canterbury.ac.nz>
 <CAPTjJmquqsqX8XL5RGdvebwHw-vgx15sQU2Qbyg3SKz3WHso2A@mail.gmail.com>
 <5B50DA1E.9030904@brenbarn.net>
 <CAOOa=pPU2-KvWJotvO3D7sJ=q-uJGbZBw3khmHdReHZ-m2zuwA@mail.gmail.com>
 <CAPTjJmo_7H8ppV6JnZzQu5B6DdYxympenws8YcaWyYSCqAvhLQ@mail.gmail.com>
 <dc623fec-26be-4b74-f13b-63d74eb3824a@python.org>
Message-ID: <CADqi797OmydggTO9qfkU1T0aLGsXX2msTpq1SxWdrDfxBs8XFQ@mail.gmail.com>

On Thu, Jul 19, 2018 at 5:29 PM Steve Dower <steve.dower at python.org> wrote:

> * "It makes it more complex"
> * "It's harder to follow the flow"
>
> Depends on your measure of complexity. For me, I prioritise "area under
> the indentation" as my preferred complexity metric (more lines*indents
> == more complex), as well as left-to-right reading of each line (more
> random access == more complex). By these measures, ?. significantly
> reduces the complexity over any of the current or future alternatives::
>
> def f(a=None):
>      name = 'default'
>      if a is not None:
>          user = a.get_user()
>          if user is not None:
>              name = user.name
>      print(name)
>


You left one out, I think, that looks decent. I converted the print to a
return, because I think it's more common to return than print.

    def f(a=None):
        try:
            return a.get_user().name
        except AttributeError:
            return 'default'


def f(a=None):
>      print(a?.get_user()?.name ?? 'none')
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/1aba7c33/attachment-0001.html>

From greg.ewing at canterbury.ac.nz  Thu Jul 19 19:57:31 2018
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 20 Jul 2018 11:57:31 +1200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CALD=Yf8j8eGxw5DBh7rgWES6OeA4MXK6nSuEDELvgm-TRmS_YQ@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CALD=Yf8j8eGxw5DBh7rgWES6OeA4MXK6nSuEDELvgm-TRmS_YQ@mail.gmail.com>
Message-ID: <5B51256B.4080801@canterbury.ac.nz>

Jonathan Fine wrote:

> Perhaps "argue" is not the right word here. It sounds too much for or
> against. And it has implications of being heated and angry.

At least arguing about the nature of argument is a very
Pythonic thing to do.

"That's not arguing, it's just contradiction!"
"No, it isn't!"

--
Greg

From greg.ewing at canterbury.ac.nz  Thu Jul 19 20:03:47 2018
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 20 Jul 2018 12:03:47 +1200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <c49bd9b8-1242-fd07-29e8-9039bfaf9199@kynesim.co.uk>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <20180719103321.7c193820@fsol>
 <c49bd9b8-1242-fd07-29e8-9039bfaf9199@kynesim.co.uk>
Message-ID: <5B5126E3.4090708@canterbury.ac.nz>

Rhodri James wrote:
> If anyone can think of a good word for "if it isn't None, otherwise", 
> I'd be all for it :-)

I don't think there's any single Engish word that captures
all of that, so we'd have to invent one.

Some suggestions:

inno (If Not None, Otherwise)

oft  (Or, Failing That)

-- 
Greg

From rosuav at gmail.com  Thu Jul 19 20:06:40 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 20 Jul 2018 10:06:40 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <5B5126E3.4090708@canterbury.ac.nz>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <20180719103321.7c193820@fsol>
 <c49bd9b8-1242-fd07-29e8-9039bfaf9199@kynesim.co.uk>
 <5B5126E3.4090708@canterbury.ac.nz>
Message-ID: <CAPTjJmpjRoVzH2dkWArHz0Jo0h_51Ufp=euY-t5_9HbuTqPd-g@mail.gmail.com>

On Fri, Jul 20, 2018 at 10:03 AM, Greg Ewing
<greg.ewing at canterbury.ac.nz> wrote:
> Rhodri James wrote:
>>
>> If anyone can think of a good word for "if it isn't None, otherwise", I'd
>> be all for it :-)
>
>
> I don't think there's any single Engish word that captures
> all of that, so we'd have to invent one.
>
> Some suggestions:
>
> inno (If Not None, Otherwise)
>
> oft  (Or, Failing That)

Iunno, that'd oft be confusing.

Kappa

ChrisA

From greg.ewing at canterbury.ac.nz  Thu Jul 19 20:30:09 2018
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 20 Jul 2018 12:30:09 +1200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <c5d3ec35-5dae-f6f2-affa-79303e7d0fe1@kynesim.co.uk>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAFYqXL8GsTyAHEkrQqSDpv_+bCcJAVv2j-0d_zwNuLLfK7Cuuw@mail.gmail.com>
 <CAPTjJmqPS_wiFTJaJ8YUKJfWsNcCX=h9AQ1EmWFv6Xf9Wes4tA@mail.gmail.com>
 <5B502A80.3080406@canterbury.ac.nz>
 <c5d3ec35-5dae-f6f2-affa-79303e7d0fe1@kynesim.co.uk>
Message-ID: <5B512D11.30701@canterbury.ac.nz>

Rhodri James wrote:
> On 19/07/18 07:06, Greg Ewing wrote:
> 
>> There's no such tradition for the new
>> operators being proposed.
> 
> There is, actually, it's just not a long one.  C# has had null-aware 
> operators for a while, for example.

THere's a precedent, yes, but I wouldn't call it a tradition.
A substantial period of time is part of the definition of the
word.

Wikipedia:

"A tradition is a belief or behavior passed down within a group
or society with symbolic meaning or special significance with
origins in the past."

Merriam-Webster:

"the handing down of information, beliefs, or customs from one
generation to another."

I don't think C# has been around long enought to span multiple
generations of programmers.

-- 
Greg

From toddrjen at gmail.com  Thu Jul 19 20:32:57 2018
From: toddrjen at gmail.com (Todd)
Date: Thu, 19 Jul 2018 20:32:57 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <c49bd9b8-1242-fd07-29e8-9039bfaf9199@kynesim.co.uk>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <20180719103321.7c193820@fsol>
 <c49bd9b8-1242-fd07-29e8-9039bfaf9199@kynesim.co.uk>
Message-ID: <CAFpSVpLRacGN7SYfEWU3P1-n4UG4A8V=dbLtyU+Rca-V_ALqdw@mail.gmail.com>

On Thu, Jul 19, 2018, 08:47 Rhodri James <rhodri at kynesim.co.uk> wrote:

> On 19/07/18 09:33, Antoine Pitrou wrote:
> > There is a use case I sympathize with: the argument-is-None case.  For
> > that I would suggest a simpler form:  "A else B" which would evaluate
> > to A if A is not None, otherwise to B (parentheses may be mandatory).
> >
> > So e.g. one of the examples would read:
> >
> >       def insort_right(a, x, lo=0, hi=None):
> >           # ...
> >           hi = hi else len(a)
> >           # ...
>
> Much as I would like a keyword, "else" is the wrong one.  It implies we
> are dealing with truthiness, which we aren't, and lays a subtle semantic
> trap as a consequence.
>
> If anyone can think of a good word for "if it isn't None, otherwise",
> I'd be all for it :-)
>

What about "nelse"?

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/37b336ed/attachment.html>

From greg.ewing at canterbury.ac.nz  Thu Jul 19 20:37:35 2018
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 20 Jul 2018 12:37:35 +1200
Subject: [Python-ideas] PEP 505: None-aware operators: operators ?= and
 ?? and OR
In-Reply-To: <472563204.5357.1532006427790.JavaMail.www@wwinf1h10>
References: <CALD=Yf-0dLkGKN+1gb11nv4s0J4yiySehszZ+rS2BZVciR+LHA@mail.gmail.com>
 <CAPL7A87J_jWnvTpvGrv0_zzZ9yr_xT9asy9+C2JfsgFB6zt0wA@mail.gmail.com>
 <472563204.5357.1532006427790.JavaMail.www@wwinf1h10>
Message-ID: <5B512ECF.9060401@canterbury.ac.nz>

Nathan wrote:

> How many programmer will read "||" or "OR" as a classic "or" operator?

It will certainly confuse someone who has any C reflexes, since
"||" in C is just a boolean "or".

-- 
Greg


From michael.lee.0x2a at gmail.com  Thu Jul 19 20:40:23 2018
From: michael.lee.0x2a at gmail.com (Michael Lee)
Date: Thu, 19 Jul 2018 17:40:23 -0700
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPTjJmpjRoVzH2dkWArHz0Jo0h_51Ufp=euY-t5_9HbuTqPd-g@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <20180719103321.7c193820@fsol>
 <c49bd9b8-1242-fd07-29e8-9039bfaf9199@kynesim.co.uk>
 <5B5126E3.4090708@canterbury.ac.nz>
 <CAPTjJmpjRoVzH2dkWArHz0Jo0h_51Ufp=euY-t5_9HbuTqPd-g@mail.gmail.com>
Message-ID: <CAMhfa1WU+itm+5z8+OWhj8HC1HU4m1vqX-twYata7J2_kj8GtQ@mail.gmail.com>

Hi -- I'm a new voice here, though I've been lurking for a while now.

How do people feel about removing "??=" and "foo?[bar]" from the PEP and
sticking with just "foo?.bar" and "foo ?? bar"?

Reasons for not implementing "??=":

   1. I think the other operators are useful in that they can be chained
   together to shrink many different lines of code. In contrast, ??= can
   shrink at most one line of code: we get to remove a single 'if foo is
   None'. If the rest of the PEP is implemented, we can also just express
   this by doing "foo = foo ?? some_expr" which I think is still relatively
   concise.

   2. None of the other short-circuiting operators have an augmented
   assignment form -- e.g. we can do "foo = foo or bar", but there's no
   such thing as "foo or= bar". I don't really see why ?? in particular
   deserves to have an augmented assignment form.

   3. We already have two different kinds of assignment operators with the
   inclusion of PEP 572 -- in the interests of keeping Python as simple as
   possible, I think it'd be a good idea not to add a third one.


Reasons for not implementing "foo?[bar]":

   1. It seems like "foo?[bar]" could be easily confused with "foo??[bar]".
   I don't think it's immediately obvious to a newcomer which spelling
   corresponds to which usage, and getting the two mixed up seems like the
   sort of thing that could cause all sorts of subtle bugs.

   2. We can already easily get the same functionality using standard
   Python. E.g., instead of doing foo?["bar"]?[0]?["baz"], we could do
lookup(foo,
   "bar", 0, "baz") where lookup is a function that looks roughly like this:

       def lookup(item, *parts):
           for part in parts:
               if item is None:
                   return None
               item = item[parts]
           return item

   Of course, we could probably create the same sort of function to replace
   foo?.bar (albeit less ergonomically), but unlike foo?[bar], there's no
   possibility for confusion: doing foo??.bar will never be a valid Python
   expression.

   3. One counter-argument against removing foo?[bar] is that it would make
   expression that need both safe index and attribute lookups look weird --
   you'd sometimes be using the "lookup" function described above (or
   something similar) and sometimes using ".?". However, I'd argue that these
   sorts of scenarios are relatively rare in practice, and that if you really
   need to deal with a bunch of code that requires you to use both forms of
   safe navigation, your code is likely already becoming pretty unreadable and
   should be rewritten -- maybe split up into multiple lines or something, or
   maybe just redesigned from scratch so you don't need to constantly manage a
   mess of Nones.


More broadly, I think I agree with the sentiment some other people have
that Python has acquired a lot of new features in a relatively short period
of time, and that it would be nice to have some cooldown to let tooling and
other implementations catch up. In that regard, I'd personally be happy if
we didn't implement this PEP or just deferred it again. But if we *are*
moving forward with it, I think it's worth trying to simplify it as much as
possible/try and make it orthogonal with existing Python features.

(But of course, I'm just some random person on the internet, so IDK if my
opinion counts for much.)

Regards,
-- Michael


On Thu, Jul 19, 2018 at 5:06 PM, Chris Angelico <rosuav at gmail.com> wrote:

> On Fri, Jul 20, 2018 at 10:03 AM, Greg Ewing
> <greg.ewing at canterbury.ac.nz> wrote:
> > Rhodri James wrote:
> >>
> >> If anyone can think of a good word for "if it isn't None, otherwise",
> I'd
> >> be all for it :-)
> >
> >
> > I don't think there's any single Engish word that captures
> > all of that, so we'd have to invent one.
> >
> > Some suggestions:
> >
> > inno (If Not None, Otherwise)
> >
> > oft  (Or, Failing That)
>
> Iunno, that'd oft be confusing.
>
> Kappa
>
> ChrisA
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/4e57d936/attachment-0001.html>

From greg.ewing at canterbury.ac.nz  Thu Jul 19 20:42:44 2018
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 20 Jul 2018 12:42:44 +1200
Subject: [Python-ideas] PEP 505: None-aware operators: operators ?= and
 ?? and OR
In-Reply-To: <CACo5Rz5ohDPpOMtU-D-KZMdMj+AeP2MLfyH5sDr3uj8nQ1_BNw@mail.gmail.com>
References: <CALD=Yf-0dLkGKN+1gb11nv4s0J4yiySehszZ+rS2BZVciR+LHA@mail.gmail.com>
 <CAPL7A87J_jWnvTpvGrv0_zzZ9yr_xT9asy9+C2JfsgFB6zt0wA@mail.gmail.com>
 <CACo5Rz5ohDPpOMtU-D-KZMdMj+AeP2MLfyH5sDr3uj8nQ1_BNw@mail.gmail.com>
Message-ID: <5B513004.5000904@canterbury.ac.nz>

Calvin Spealman wrote:
> Operators that only vary by case would be... confusing.

Also I'd prefer to see AND and OR (or maybe And and Or) reserved for
possible use as overridable, non-short-circuiting forms of "and"
and "or" for use by DSLs.

-- 
Greg

From asweigart at gmail.com  Thu Jul 19 20:45:20 2018
From: asweigart at gmail.com (Al Sweigart)
Date: Thu, 19 Jul 2018 17:45:20 -0700
Subject: [Python-ideas] Adding Python interpreter info to "pip install"
Message-ID: <CAPyZGSkyh+6RSq4trtLSQ8Tkkn7um8rhworuFcf=ztK-QO=DMA@mail.gmail.com>

 The goal of this idea is to make it easier to find out when someone has
installed packages for the wrong python installation. I'm coming across
quite a few StackOverflow posts and emails where beginners are using pip to
install a package, but then finding they can't import it because they have
multiple python installations and used the wrong pip.

For example, this guy has this problem:
https://stackoverflow.com/questions/37662012/which-pip-is-with-which-python

I'd propose adding a simple line to the output of "pip install" that
changes this:

user at user:~$ pip3 install pyperclip
Collecting pyperclip
Installing collected packages: pyperclip
Successfully installed pyperclip-1.6.2

...to something like this:

user at user:~$ pip3 install pyperclip
Running pip for /usr/bin/python3
Collecting pyperclip
Installing collected packages: pyperclip
Successfully installed pyperclip-1.6.2

This way, when they copy/paste their output to StackOverflow, it'll be
somewhat more obvious to their helper that they used pip for the wrong
python installation.

This info would also be useful for the output of "pip info", but that would
break scripts that reads that output.

Any thoughts?

-Al
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/ce67e745/attachment.html>

From rosuav at gmail.com  Thu Jul 19 20:50:07 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 20 Jul 2018 10:50:07 +1000
Subject: [Python-ideas] Adding Python interpreter info to "pip install"
In-Reply-To: <CAPyZGSkyh+6RSq4trtLSQ8Tkkn7um8rhworuFcf=ztK-QO=DMA@mail.gmail.com>
References: <CAPyZGSkyh+6RSq4trtLSQ8Tkkn7um8rhworuFcf=ztK-QO=DMA@mail.gmail.com>
Message-ID: <CAPTjJmo7JFj15M_Y-+uBnCBmqL7KYi80NoihG5=2b+nS5uvQoA@mail.gmail.com>

On Fri, Jul 20, 2018 at 10:45 AM, Al Sweigart <asweigart at gmail.com> wrote:
> The goal of this idea is to make it easier to find out when someone has
> installed packages for the wrong python installation. I'm coming across
> quite a few StackOverflow posts and emails where beginners are using pip to
> install a package, but then finding they can't import it because they have
> multiple python installations and used the wrong pip.
>
> For example, this guy has this problem:
> https://stackoverflow.com/questions/37662012/which-pip-is-with-which-python
>
> I'd propose adding a simple line to the output of "pip install" that changes
> this:
>
> user at user:~$ pip3 install pyperclip
> Collecting pyperclip
> Installing collected packages: pyperclip
> Successfully installed pyperclip-1.6.2
>
> ...to something like this:
>
> user at user:~$ pip3 install pyperclip
> Running pip for /usr/bin/python3
> Collecting pyperclip
> Installing collected packages: pyperclip
> Successfully installed pyperclip-1.6.2
>
> This way, when they copy/paste their output to StackOverflow, it'll be
> somewhat more obvious to their helper that they used pip for the wrong
> python installation.
>
> This info would also be useful for the output of "pip info", but that would
> break scripts that reads that output.
>
> Any thoughts?

You can get some very useful information from "pip3 --version". As
well as pip's own version, it tells you the version of Python that
it's running under, AND what directory it's being run from. If you
want to request that similar info be added to other commands, I would
strongly recommend lifting the exact format of --version and using
that.

(I'm not sure what "pip info" is, btw. My pip doesn't seem to have that.)

ChrisA

From greg.ewing at canterbury.ac.nz  Thu Jul 19 20:51:23 2018
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 20 Jul 2018 12:51:23 +1200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4Yt5nd-mxpQzGFH-ehbRZH=DFMjsg81_LNxRfg4c_rpoQ@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAEbHw4Yt5nd-mxpQzGFH-ehbRZH=DFMjsg81_LNxRfg4c_rpoQ@mail.gmail.com>
Message-ID: <5B51320B.40106@canterbury.ac.nz>

David Mertz wrote:
> "APL and Perl territory" means "use lots of punctuation characters in 
> somewhat cryptic ways, often combining several for a distinct 
> semantics."

And APL takes it a step further by freely making up new
characters when it runs out of existing ones. At least
nobody has suggested doing that in Python yet...

-- 
Greg

From mike at selik.org  Thu Jul 19 20:57:50 2018
From: mike at selik.org (Michael Selik)
Date: Thu, 19 Jul 2018 20:57:50 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAMhfa1WU+itm+5z8+OWhj8HC1HU4m1vqX-twYata7J2_kj8GtQ@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <20180719103321.7c193820@fsol>
 <c49bd9b8-1242-fd07-29e8-9039bfaf9199@kynesim.co.uk>
 <5B5126E3.4090708@canterbury.ac.nz>
 <CAPTjJmpjRoVzH2dkWArHz0Jo0h_51Ufp=euY-t5_9HbuTqPd-g@mail.gmail.com>
 <CAMhfa1WU+itm+5z8+OWhj8HC1HU4m1vqX-twYata7J2_kj8GtQ@mail.gmail.com>
Message-ID: <CADqi797pajvdV1FAO0Pnb2mnC=Q3O2B4-gYg4Ec_YqtK7KPYew@mail.gmail.com>

On Thu, Jul 19, 2018 at 8:40 PM Michael Lee <michael.lee.0x2a at gmail.com>
wrote:

>
>    1. ...
>
>    2. We can already easily get the same functionality using standard
>    Python. E.g., instead of doing foo?["bar"]?[0]?["baz"], we could do lookup(foo,
>    "bar", 0, "baz") where lookup is a function that looks roughly like
>    this:
>
>        def lookup(item, *parts):
>            for part in parts:
>                if item is None:
>                    return None
>                item = item[parts]
>            return item
>
>
Try/except also looks decent.

    try:
        x = foo['bar'][0]
    except TypeError:
        x = 'default'
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/92841f15/attachment.html>

From mertz at gnosis.cx  Thu Jul 19 22:13:50 2018
From: mertz at gnosis.cx (David Mertz)
Date: Thu, 19 Jul 2018 22:13:50 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180719133820.GV7318@ando.pearwood.info>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
Message-ID: <CAEbHw4Z+2LEto9c5bKHiMoCV2BZ+=G2pynGt8_RqmCsJQaQwkw@mail.gmail.com>

On Thu, Jul 19, 2018, 9:39 AM Steven D'Aprano <steve at pearwood.info> wrote:

> *Its just spelling*. If it is a useful and well-defined feature, we'll
> get used to the spelling soon enough.
>
> That's not to say that spelling is not important *at all*, or that we
> should never prefer words to symbols. But if the only objection we have is
> "this is useful but I don't like the spelling so -1" then that's
> usually a pretty weak argument against the feature.
>

I could not disagree more. Spelling is extremely important to readability,
and that only goes away partially with familiarity. As a teaching language,
moreover, there will always be a late share of Python users who haven't
become familiar, and for whom "executable pseudo-code" is a big advantage.

I don't believe Python should be ONLY for those who do not already know it,
but neither should it be even half so arcane as this PEP would make it.

Here's are well defined features of the J programming language, for
comparison... very compact too:

For example, in the following contrived expression the exclamation point
<https://en.m.wikipedia.org/wiki/Exclamation_point> !refers to three
distinct functions:

2!!7!4

The function below can be used to list all of the prime numbers between 1
and R with:

2_&{&/x!/:2_!x}'!R
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/432cd9a1/attachment-0001.html>

From asweigart at gmail.com  Thu Jul 19 23:10:47 2018
From: asweigart at gmail.com (Al Sweigart)
Date: Thu, 19 Jul 2018 20:10:47 -0700
Subject: [Python-ideas] Adding Python interpreter info to "pip install"
In-Reply-To: <CAPTjJmo7JFj15M_Y-+uBnCBmqL7KYi80NoihG5=2b+nS5uvQoA@mail.gmail.com>
References: <CAPyZGSkyh+6RSq4trtLSQ8Tkkn7um8rhworuFcf=ztK-QO=DMA@mail.gmail.com>
 <CAPTjJmo7JFj15M_Y-+uBnCBmqL7KYi80NoihG5=2b+nS5uvQoA@mail.gmail.com>
Message-ID: <CAPyZGSmsDhZJDAEzbn4Wf4izJXzWmNf+a0Woh39K2W6F=nmbkQ@mail.gmail.com>

 Sorry, I meant "pip list", rather than "pip info".

I thought about the fact that "pip --version" provides this info, but 1) it
provides the location of pip, not the python interpreter it installs
packages for and 2) it would be an additional step for the question-asker
to go through after posting the output of "pip install".

It would be nice to display output that the question-asker can compare
directly with the output of "which python". And I'd like to shorten the
potential amount of back-and-forth before the helper can get the
information they need.

Additionally, while they could always just run "python -m pip install spam"
but most tutorials tell them to run pip directly, so I still see the need
for this.


On Thu, Jul 19, 2018 at 5:50 PM, Chris Angelico <rosuav at gmail.com> wrote:

> On Fri, Jul 20, 2018 at 10:45 AM, Al Sweigart <asweigart at gmail.com> wrote:
> > The goal of this idea is to make it easier to find out when someone has
> > installed packages for the wrong python installation. I'm coming across
> > quite a few StackOverflow posts and emails where beginners are using pip
> to
> > install a package, but then finding they can't import it because they
> have
> > multiple python installations and used the wrong pip.
> >
> > For example, this guy has this problem:
> > https://stackoverflow.com/questions/37662012/which-pip-is-
> with-which-python
> >
> > I'd propose adding a simple line to the output of "pip install" that
> changes
> > this:
> >
> > user at user:~$ pip3 install pyperclip
> > Collecting pyperclip
> > Installing collected packages: pyperclip
> > Successfully installed pyperclip-1.6.2
> >
> > ...to something like this:
> >
> > user at user:~$ pip3 install pyperclip
> > Running pip for /usr/bin/python3
> > Collecting pyperclip
> > Installing collected packages: pyperclip
> > Successfully installed pyperclip-1.6.2
> >
> > This way, when they copy/paste their output to StackOverflow, it'll be
> > somewhat more obvious to their helper that they used pip for the wrong
> > python installation.
> >
> > This info would also be useful for the output of "pip info", but that
> would
> > break scripts that reads that output.
> >
> > Any thoughts?
>
> You can get some very useful information from "pip3 --version". As
> well as pip's own version, it tells you the version of Python that
> it's running under, AND what directory it's being run from. If you
> want to request that similar info be added to other commands, I would
> strongly recommend lifting the exact format of --version and using
> that.
>
> (I'm not sure what "pip info" is, btw. My pip doesn't seem to have that.)
>
> ChrisA
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180719/08b4b560/attachment.html>

From turnbull.stephen.fw at u.tsukuba.ac.jp  Thu Jul 19 23:26:51 2018
From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull)
Date: Fri, 20 Jul 2018 12:26:51 +0900
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <dc623fec-26be-4b74-f13b-63d74eb3824a@python.org>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAFYqXL8GsTyAHEkrQqSDpv_+bCcJAVv2j-0d_zwNuLLfK7Cuuw@mail.gmail.com>
 <CAPTjJmqPS_wiFTJaJ8YUKJfWsNcCX=h9AQ1EmWFv6Xf9Wes4tA@mail.gmail.com>
 <5B502A80.3080406@canterbury.ac.nz>
 <CAPTjJmquqsqX8XL5RGdvebwHw-vgx15sQU2Qbyg3SKz3WHso2A@mail.gmail.com>
 <5B50DA1E.9030904@brenbarn.net>
 <CAOOa=pPU2-KvWJotvO3D7sJ=q-uJGbZBw3khmHdReHZ-m2zuwA@mail.gmail.com>
 <CAPTjJmo_7H8ppV6JnZzQu5B6DdYxympenws8YcaWyYSCqAvhLQ@mail.gmail.com>
 <dc623fec-26be-4b74-f13b-63d74eb3824a@python.org>
Message-ID: <23377.22139.121236.641756@turnbull.sk.tsukuba.ac.jp>

Steve Dower writes:

 > But for me, the ?? operator makes its strongest case when you are 
 > getting attributes from potentially None values:
 > 
 >      x = maybe_get()?.get_int() ?? 0
 >      y = maybe_get()?.get_list() ?? []
 > 
 > In this case, using 'or' may replace an intentionally falsie value
 > with your own,

I wish the PEP's examples had such cases marked and explained.  I
didn't understand the context for most examples (ie, *why* the left
operand could be None, and why that needed to be distinguished from
"intentionally falsie values").

In the "find_content_type" example, None is just a way of saying
"oops, you handle it" that is less "cute" than "".  "or" is far more
resilient in that context than "??" would be.  I am a mail geek, and
*much* more likely to encounter that code than the others.  I would be
distressed if the email module or Mailman grew a lot of "??".  Email
is a domain where resilience really matters because you can receive
arbitrary data over SMTP, there's no provision for rejecting garbage,
and all too often you do get garbage. :-/

 > while using a default parameter is still going to leave you 
 > with None if the first maybe_get() returned nothing.
 > 
 > "?." without "??" feels like a trap, while "??" without "?." feels 
 > largely unnecessary. But both together lets you turn many lines of code 
 > into a much shorter snippet that reads left-to-right and lets you assume 
 > success until you reach the "??".

I find that pretty persuasive.  Perhaps that should be added to PEP 8.

I would advocate write those 

    x = maybe_get() ?. get_int() ?? 0
    y = maybe_get() ?. get_list() ?? []

to emphasize that this is an operator that "does something" to the
operand, rather than "just" extracting an attribute.  (No, I'm not
going to try to make that more precise, so feel free to ignore.)  I'm
not sure about the ?[ operator:

    x = a?[b] ?? c
    x = a ?[b] ?? c
    x = a ?[ b] ?? c
    x = a ?[ b ] ?? c

I guess I like the second line best (and

    x = a?[b]??c

is right out, of course).

I do think Terry Reedy's criticism that in many cases having
maybe_get() return None rather than an appropriate default object is
bad API design may be important.  I suspect that the "a ?. b ?? c" 
pattern will discourage refactoring of that code in favor of writing
short workarounds.


From turnbull.stephen.fw at u.tsukuba.ac.jp  Thu Jul 19 23:30:39 2018
From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull)
Date: Fri, 20 Jul 2018 12:30:39 +0900
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <piqkip$qqu$1@blaine.gmane.org>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <23376.19632.386221.927225@turnbull.sk.tsukuba.ac.jp>
 <piqkip$qqu$1@blaine.gmane.org>
Message-ID: <23377.22367.812408.500559@turnbull.sk.tsukuba.ac.jp>

Terry Reedy writes:
 > On 7/19/2018 4:32 AM, Stephen J. Turnbull wrote:

 > > make me cringe.  Exactly one of two things is true:
 > 
 > It seems to me that the problem is returning None.

Exactly, but this seems to indicate that the problem I'm talking about
is the code.  It's not; my issue is with the use of THIS code as an
example justifying "??".

 > The premise of returning None as default is that users can follow
 > with conditional code.  If we now think that this is too much of a
 > burden, we should stop returning None, at least by default.

Note that in the case of an MUA, code might look like something like
this:

    class BaseSaveObject(object):        # NOTE NAME CHANGE!
        def find_content_type(self, filename):
            ctype, encoding = mimetypes.guess_type(filename)
            while ctype is None:
                ctype = input('Couldn't recognize MIME type. Ideas?")
                ctype = self.validate_mimetype(ctype)  # returns None on bad
            return ctype

So returning "application/octet-stream" is probably inappropriate if
parsing the message header for Content-Type fails.

I'm not sure why you say "burdensome".  Using "x() or y()" *because*
it short-circuits is idiomatic (although some deprecate it).  The main
point of "??" is that "or" is not good enough because it does the same
thing with other falsies that it does with None.  There are other
tweaks (such as very high precedence), but that's the main thing.

What I'm looking for is a set of examples where "or" won't do, and
explanations of *why* that is true and why programmers *won't* abuse
"??" to create less resilient code, to balance against the avalanche
of examples where "??" leads to less resilient code, such as this
case.

Note that if you want to use "??" rather than "or", there's at least
one other falsie that you want to *avoid* a default for, and then in
cases like this one, the question is "why aren't you checking for it
to prevent exceptions in a module written long long ago in a galaxy
far far away?"

 > > 1.  mimetypes.guess_type guarantees that the only falsie it will ever
 > >      return is None, or
 > > 
 > > 2.  it doesn't.
 > 
 > or
 > 3. it never returns anything other than a non-blank string unless the 
 > user so requests.

That would be nice, but that's off-topic.  The point here is that
Steve ripped a page from Tim's book and transformed a pile of real
code.  (This deserves a standing ovation IMO!)  So he has to take the
API as it is, warts and all.

Steve


From arj.python at gmail.com  Fri Jul 20 00:15:36 2018
From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer)
Date: Fri, 20 Jul 2018 08:15:36 +0400
Subject: [Python-ideas] Adding Python interpreter info to "pip install"
In-Reply-To: <CAPyZGSkyh+6RSq4trtLSQ8Tkkn7um8rhworuFcf=ztK-QO=DMA@mail.gmail.com>
References: <CAPyZGSkyh+6RSq4trtLSQ8Tkkn7um8rhworuFcf=ztK-QO=DMA@mail.gmail.com>
Message-ID: <CADrxXX=aB9U69YMR8LJhpdk-6irLjstakpZizrApWBnCZpmUTQ@mail.gmail.com>

as it requires only a log, i think it is worth it as pip acts covertly
meaning you don't see which pip is being used in a multi-py environment,
yes you can have version pip info

Abdur-Rahmaan Janhangeer
https://github.com/Abdur-rahmaanJ
Mauritius


>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180720/68477f3a/attachment-0001.html>

From njs at pobox.com  Fri Jul 20 00:21:47 2018
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 19 Jul 2018 21:21:47 -0700
Subject: [Python-ideas] Adding Python interpreter info to "pip install"
In-Reply-To: <CAPyZGSkyh+6RSq4trtLSQ8Tkkn7um8rhworuFcf=ztK-QO=DMA@mail.gmail.com>
References: <CAPyZGSkyh+6RSq4trtLSQ8Tkkn7um8rhworuFcf=ztK-QO=DMA@mail.gmail.com>
Message-ID: <CAPJVwB=RfpDQPQADWT7V5FiJj-ohXXHWLdtkob5T7DP258SH8A@mail.gmail.com>

On Thu, Jul 19, 2018 at 5:45 PM, Al Sweigart <asweigart at gmail.com> wrote:
> The goal of this idea is to make it easier to find out when someone has
> installed packages for the wrong python installation. I'm coming across
> quite a few StackOverflow posts and emails where beginners are using pip to
> install a package, but then finding they can't import it because they have
> multiple python installations and used the wrong pip.

This sounds like a great idea to me, but pip is developer separately
from python itself, and I don't think the pip maintainers monitor
python-ideas. I'd suggest filing a feature request on the pip tracker:

https://github.com/pypa/pip/issues/new?template=feature-request.md

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From wes.turner at gmail.com  Fri Jul 20 01:26:56 2018
From: wes.turner at gmail.com (Wes Turner)
Date: Fri, 20 Jul 2018 01:26:56 -0400
Subject: [Python-ideas] Adding Python interpreter info to "pip install"
In-Reply-To: <CAPJVwB=RfpDQPQADWT7V5FiJj-ohXXHWLdtkob5T7DP258SH8A@mail.gmail.com>
References: <CAPyZGSkyh+6RSq4trtLSQ8Tkkn7um8rhworuFcf=ztK-QO=DMA@mail.gmail.com>
 <CAPJVwB=RfpDQPQADWT7V5FiJj-ohXXHWLdtkob5T7DP258SH8A@mail.gmail.com>
Message-ID: <CACfEFw81WTYrzYAfbuODuWN1dm9r2po_QRFj=LAGOcpTZb1=xQ@mail.gmail.com>

`python -m site` and its docs may also be useful for troubleshooting
end-user installations.

A diagnostic script with e.g. these commands could also be helpful:

which python
which pip
python -m site  # sys.path, USER_SITE (pip --user)
python -m pip --version

>>> print((os.name, sys.platform, platform.system()))


https://docs.python.org/3/library/site.html

"""
The site module also provides a way to get the user directories from the
command line:

$ python3 -m site --user-site
/home/user/.local/lib/python3.3/site-packages

If it is called without arguments, it will print the contents of sys.path
on the standard output, followed by the value of USER_BASE and whether the
directory exists, then the same thing for USER_SITE, and finally the value
of ENABLE_USER_SITE.

--user-base

Print the path to the user base directory.

--user-site

Print the path to the user site-packages directory.

If both options are given, user base and user site will be printed (always
in this order), separated by os.pathsep.

If any option is given, the script will exit with one of these values: O if
the user site-packages directory is enabled, 1 if it was disabled by the
user, 2 if it is disabled for security reasons or by an administrator, and
a value greater than 2 if there is an error.
"""


On Friday, July 20, 2018, Nathaniel Smith <njs at pobox.com> wrote:

> On Thu, Jul 19, 2018 at 5:45 PM, Al Sweigart <asweigart at gmail.com> wrote:
> > The goal of this idea is to make it easier to find out when someone has
> > installed packages for the wrong python installation. I'm coming across
> > quite a few StackOverflow posts and emails where beginners are using pip
> to
> > install a package, but then finding they can't import it because they
> have
> > multiple python installations and used the wrong pip.
>
> This sounds like a great idea to me, but pip is developer separately
> from python itself, and I don't think the pip maintainers monitor
> python-ideas. I'd suggest filing a feature request on the pip tracker:
>
> https://github.com/pypa/pip/issues/new?template=feature-request.md
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180720/2fb187f8/attachment.html>

From steve at pearwood.info  Fri Jul 20 01:22:35 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 20 Jul 2018 15:22:35 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <23376.19632.386221.927225@turnbull.sk.tsukuba.ac.jp>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <23376.19632.386221.927225@turnbull.sk.tsukuba.ac.jp>
Message-ID: <20180720052234.GX7318@ando.pearwood.info>

On Thu, Jul 19, 2018 at 05:32:48PM +0900, Stephen J. Turnbull wrote:

> To be honest, code transformations like this
> 
>  >      class BaseUploadObject(object):
>  >          def find_content_type(self, filename):
>  >              ctype, encoding = mimetypes.guess_type(filename)
>  >              if ctype is None:
>  >                  return 'application/octet-stream'
>  >              else:
>  >                  return ctype
> 
> to this
> 
>  >      class BaseUploadObject(object):
>  >          def find_content_type(self, filename):
>  >              ctype, encoding = mimetypes.guess_type(filename)
>  >              return ctype ?? 'application/octet-stream'
> 
> make me cringe. 

I don't think that's a good use of ?? and that method should 
either remain in the original form, or at most, be re-written using

return ctype if ctypes is not None else 'application/octet-stream'

(although I prefer the original form). There doesn't seem to be any 
advantage to ?? in this example except to transform a four-line 
imperative-style return statement to a one-line return statement, 
trading off vertical space for horizontal. But there's no improvement in 
expressiveness that I can see.


> Exactly one of two things is true:
> 
> 1.  mimetypes.guess_type guarantees that the only falsie it will ever
>     return is None, or
> 
> 2.  it doesn't.
> 
> In case 1, "ctype or 'application/octet-stream'" ALSO does the right
> thing. 

It does the right thing, but it doesn't look like it does the right 
thing.

The ?? operator promises to only transform None, and nothing but None. 
To the reader, the `or` operator promises to transform any falsey value. 
Is `or` too broad and do too much? There's no way of knowing.

Instead of treating `guess_type` as a black-box, the reader now has to 
dig-deep into its documentation and/or implementation to find out which 
falsey values it might return and whether or not they ought to be 
treated the same as None.


> In case 2, ONLY "ctype or 'application/octet-stream'" does the
> right thing, as few callers of BaseUploadObject.find_content_type will
> be prepared for "", (), [], or any variety of 0 as the return value.

I doubt that guess_type (and hence find_content_type) will return 0 or 
[] etc as part of their documented interface (as opposed to a bug), but 
if they did, surely we ought to assume that the caller is prepared to 
handle its output.

More likely is that it might return the empty string "". At face value I 
would expect that a better interface for guess_type would be to return 
the empty string rather than None.

But perhaps there is a legitimate reason to treat the empty string as 
distinct from None, it should *not* be transformed.

On perhaps the function is avoiding the "failure looks like success" 
anti-pattern. Who hasn't been bitten by writing something like this?

    chunk = text[text.find('spam')+4:]

It looks okay at a glance: return the chunk of text immediately 
following "spam". But if "spam" isn't present at all, it returns the 
entire text minus the first three characters, which is probably not what 
you wanted.

So there is a good argument to be made that guess_type ought to promise:

- return None if it cannot guess the type;
- otherwise return a non-empty string.

(or that the empty string is itself a legitimate output that ought to be 
considered distinct from None and should *not* be transformed).

In that case, `or` reads wrongly, because it either wrongly transforms 
"" into the default when it should not, or it suggests to the reader 
that it might.

Either way, an explicit test for None -- whether we use "if ... is None" 
or the proposed ?? operator -- is better than a test for falsey.



> So I would like to see various examples of code where
> 
> 1.  in the original code, treating a falsie that is not None the same
>     way that None is treated is a detectable bug; and

I don't see that this is relevant. We already had this discussion, when 
the ternary if was introduced, and a million times when discussing why 
it is better to write "if spam is None" rather than "if not spam".

In other words, we ought to be comparing the expressiveness of 

    process(spam ?? something)

versus:

   process(something if spam is None else spam)

(and similar variations), not against 

   process(spam if spam else something)
   process(spam or something)

both of which do something different from the ?? operator.



-- 
Steve

From pradyunsg at gmail.com  Fri Jul 20 01:32:58 2018
From: pradyunsg at gmail.com (Pradyun Gedam)
Date: Fri, 20 Jul 2018 11:02:58 +0530
Subject: [Python-ideas] Adding Python interpreter info to "pip install"
In-Reply-To: <CAPJVwB=RfpDQPQADWT7V5FiJj-ohXXHWLdtkob5T7DP258SH8A@mail.gmail.com>
References: <CAPyZGSkyh+6RSq4trtLSQ8Tkkn7um8rhworuFcf=ztK-QO=DMA@mail.gmail.com>
 <CAPJVwB=RfpDQPQADWT7V5FiJj-ohXXHWLdtkob5T7DP258SH8A@mail.gmail.com>
Message-ID: <CAPA-EquuAjK-VSuM1Szmv4kpr71yK+pwvBxtsR__DCbGd0WsKA@mail.gmail.com>

On Fri, 20 Jul 2018, 09:52 Nathaniel Smith, <njs at pobox.com> wrote:

> On Thu, Jul 19, 2018 at 5:45 PM, Al Sweigart <asweigart at gmail.com> wrote:
> > The goal of this idea is to make it easier to find out when someone has
> > installed packages for the wrong python installation. I'm coming across
> > quite a few StackOverflow posts and emails where beginners are using pip
> to
> > install a package, but then finding they can't import it because they
> have
> > multiple python installations and used the wrong pip.
>
> This sounds like a great idea to me, but pip is developer separately
> from python itself, and I don't think the pip maintainers monitor
> python-ideas.


At least one (me) does lurk here. ;)

I'd suggest filing a feature request on the pip tracker:
>
> https://github.com/pypa/pip/issues/new?template=feature-request.md


I second this suggestion to file a feature request for this on the tracker.

Additionally, you should also take a look to see if this has been filed
already since I do remember having a discussion where this came. :)

--
Pradyun


> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180720/4cf6db5c/attachment-0001.html>

From a.eletsky at gmail.com  Fri Jul 20 02:52:14 2018
From: a.eletsky at gmail.com (Aliaksei Yaletski)
Date: Fri, 20 Jul 2018 09:52:14 +0300
Subject: [Python-ideas] PEP 505: None-aware operators: operators ?= and
 ?? and OR
In-Reply-To: <CALD=Yf-0dLkGKN+1gb11nv4s0J4yiySehszZ+rS2BZVciR+LHA@mail.gmail.com>
References: <CALD=Yf-0dLkGKN+1gb11nv4s0J4yiySehszZ+rS2BZVciR+LHA@mail.gmail.com>
Message-ID: <CA+w-FZ8GgTRGm+duzmT=CgVeX_=mpWgUPNzOHH=D-a=W8J7-AA@mail.gmail.com>

Hi

I'm new to that list, hope, that my idea will be helpfull.

  hi = hi OR len(a)
> ---
> where (A OR B) returns A if A is not None, otherwise it returns B.
>

Probably, we can introduce syntactic sugar for "if not equal" comparison?
Not only for "is not None" check, but for comparison with any expression.

Something like:

x = foo if 'value': bar

equal to :

x = foo if foo != 'value' else bar

so, expression:

x = foo if foo is not None else bar

become:

x = foo if None: bar

That syntax does not use new keywords or operators, looks pythonic and
simplify most common "is not None" expression.

-- 
Regards,

Aliaksei Yaletski.

Homepage: http://tiendil.org
Skype: Tiendil
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180720/0e227da0/attachment.html>

From gregory.lielens at gmail.com  Fri Jul 20 03:10:36 2018
From: gregory.lielens at gmail.com (=?UTF-8?Q?Gr=C3=A9gory_Lielens?=)
Date: Fri, 20 Jul 2018 00:10:36 -0700 (PDT)
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <5B50DA1E.9030904@brenbarn.net>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAFYqXL8GsTyAHEkrQqSDpv_+bCcJAVv2j-0d_zwNuLLfK7Cuuw@mail.gmail.com>
 <CAPTjJmqPS_wiFTJaJ8YUKJfWsNcCX=h9AQ1EmWFv6Xf9Wes4tA@mail.gmail.com>
 <5B502A80.3080406@canterbury.ac.nz>
 <CAPTjJmquqsqX8XL5RGdvebwHw-vgx15sQU2Qbyg3SKz3WHso2A@mail.gmail.com>
 <5B50DA1E.9030904@brenbarn.net>
Message-ID: <8856e53b-14c5-4f50-82c8-a48260a63ba2@googlegroups.com>



On 2018-07-19 02:11, Chris Angelico wrote: 

 

> -snip-
>         As far as I can see, these null-coalescing operators would break 
> that 
> model.  The PEP doesn't seem to provide for a "real" magic method 
> allowing users to override the actual behavior of the method.  (You can 
> only override __has_value__ to hook into it, but not define by fiat what 
> A ?? B does, as you can with other operators.)  And I think the reason 
> for this is that the operator itself is too specific, much more specific 
> in semantics than other operators.  (I had similar doubts about adding 
> the matrix-multiplication operator @.) 
>
>         People keep saying that this null-coalescing behavior is so common 
> and 
> useful, etc., but that hasn't been my experience at all.  In my 
> experience, the desire to shortcut this kind of logic is more often a 
> sign of corner-cutting and insufficiently specified data formats, and is 
> likely to cause bugs later on.  Eventually it has to actually matter 
> whether something is None or not, and these operators just kick that can 
> down the road.  In terms of their abstract meaning, they are not 
> remotely close to as common or useful as operators like & and |. 
>

Fully agree with you on this, so -1 on those new operators. I like 
operators (I was behind one of the early proposals for matmul operator 
(even proposed a bunch of others, for orthogonality purpose: 
element-wise/object-as-a-whole is a dichotomy that can happen for other 
things that multiplication), but they need to benefit from operators 
specifics: precendence order, tradition meaning direct translation of 
formula/recipies already written with a similar notation, and be generic 
enough to allow productive use of overloading (instead of confusing use).

Matmul have this (with possible exception for overloading, but they are 
quite a few other mathematical entities where you want to have more than 1 
multiplication, beside matrices).

Those new operators, not so much, the only benefit is more compact notation 
of something that is already quite compact (A if A is not None else B) and 
very explicit (it reads like pseudocode, which is a huge strengh of python).
?? and associates is much more cryptic, force you to think much harder 
about precedence, just to be slightly more compact. Granted, compact 
notation can be valuable, but usually if you want to couple multiple such 
things (like you often do with matmul), and precedence rules allows you to 
do it elegantly combine multiple operator with infix notation (again matmul 
is present in formulas that have scalar mul, + and -) . 
Do we do this with None-coalescence? I don't think so, not often anyway...
Do we want to do this more with None-coalescence? Not me, certainly not: 
When I None coalesce, I want it to be as clear and atomic as possible 
because it's a corner case.

so strong -1...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180720/de397f0d/attachment.html>

From tjreedy at udel.edu  Fri Jul 20 03:37:26 2018
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 20 Jul 2018 03:37:26 -0400
Subject: [Python-ideas] Adding Python interpreter info to "pip install"
In-Reply-To: <CAPJVwB=RfpDQPQADWT7V5FiJj-ohXXHWLdtkob5T7DP258SH8A@mail.gmail.com>
References: <CAPyZGSkyh+6RSq4trtLSQ8Tkkn7um8rhworuFcf=ztK-QO=DMA@mail.gmail.com>
 <CAPJVwB=RfpDQPQADWT7V5FiJj-ohXXHWLdtkob5T7DP258SH8A@mail.gmail.com>
Message-ID: <pis3bj$9qv$1@blaine.gmane.org>

On 7/20/2018 12:21 AM, Nathaniel Smith wrote:
> On Thu, Jul 19, 2018 at 5:45 PM, Al Sweigart <asweigart at gmail.com> wrote:
>> The goal of this idea is to make it easier to find out when someone has
>> installed packages for the wrong python installation. I'm coming across
>> quite a few StackOverflow posts and emails where beginners are using pip to
>> install a package, but then finding they can't import it because they have
>> multiple python installations and used the wrong pip.

So have I.  The solution is to run pip the way the core developers 
recommend, which is to explicitly run the version of python you want pip 
to install to.  On Windows,

py -x.y -m pip install somemod

I have given that answer on SO and just gave it on python-list, thereby 
solving the users 2.7 versus 3.7 problem.

> This sounds like a great idea to me, but pip is developer separately
> from python itself, and I don't think the pip maintainers monitor
> python-ideas. I'd suggest filing a feature request on the pip tracker:
> 
> https://github.com/pypa/pip/issues/new?template=feature-request.md

Definitely.


-- 
Terry Jan Reedy


From tjol at tjol.eu  Fri Jul 20 03:56:08 2018
From: tjol at tjol.eu (Thomas Jollans)
Date: Fri, 20 Jul 2018 09:56:08 +0200
Subject: [Python-ideas] Adding Python interpreter info to "pip install"
In-Reply-To: <CAPyZGSmsDhZJDAEzbn4Wf4izJXzWmNf+a0Woh39K2W6F=nmbkQ@mail.gmail.com>
References: <CAPyZGSkyh+6RSq4trtLSQ8Tkkn7um8rhworuFcf=ztK-QO=DMA@mail.gmail.com>
 <CAPTjJmo7JFj15M_Y-+uBnCBmqL7KYi80NoihG5=2b+nS5uvQoA@mail.gmail.com>
 <CAPyZGSmsDhZJDAEzbn4Wf4izJXzWmNf+a0Woh39K2W6F=nmbkQ@mail.gmail.com>
Message-ID: <e54cfc98-19bc-883b-f13c-4783a60a004b@tjol.eu>

On 20/07/18 05:10, Al Sweigart wrote:
> Sorry, I meant "pip list", rather than "pip info".
> 
> I thought about the fact that "pip --version" provides this info, but 1)
> it provides the location of pip, not the python interpreter it installs
> packages for and 2) it would be an additional step for the
> question-asker to go through after posting the output of "pip install".

If the extra line of output was something like

This is $(pip --version)

that would be more consistent, and provide enough information.

> 
> It would be nice to display output that the question-asker can compare
> directly with the output of "which python".?And I'd like to shorten the
> potential amount of back-and-forth before the helper can get the
> information they need.
> 
> Additionally, while they could always just run "python -m pip install
> spam" but most tutorials tell them to run pip directly, so I still see
> the need for this.
> 
> 
> On Thu, Jul 19, 2018 at 5:50 PM, Chris Angelico <rosuav at gmail.com
> <mailto:rosuav at gmail.com>> wrote:
> 
>     On Fri, Jul 20, 2018 at 10:45 AM, Al Sweigart <asweigart at gmail.com
>     <mailto:asweigart at gmail.com>> wrote:
>     > The goal of this idea is to make it easier to find out when
>     someone has
>     > installed packages for the wrong python installation. I'm coming
>     across
>     > quite a few StackOverflow posts and emails where beginners are
>     using pip to
>     > install a package, but then finding they can't import it because
>     they have
>     > multiple python installations and used the wrong pip.
>     >
>     > For example, this guy has this problem:
>     >
>     https://stackoverflow.com/questions/37662012/which-pip-is-with-which-python
>     <https://stackoverflow.com/questions/37662012/which-pip-is-with-which-python>
>     >
>     > I'd propose adding a simple line to the output of "pip install"
>     that changes
>     > this:
>     >
>     > user at user:~$ pip3 install pyperclip
>     > Collecting pyperclip
>     > Installing collected packages: pyperclip
>     > Successfully installed pyperclip-1.6.2
>     >
>     > ...to something like this:
>     >
>     > user at user:~$ pip3 install pyperclip
>     > Running pip for /usr/bin/python3
>     > Collecting pyperclip
>     > Installing collected packages: pyperclip
>     > Successfully installed pyperclip-1.6.2
>     >
>     > This way, when they copy/paste their output to StackOverflow, it'll be
>     > somewhat more obvious to their helper that they used pip for the wrong
>     > python installation.
>     >
>     > This info would also be useful for the output of "pip info", but
>     that would
>     > break scripts that reads that output.
>     >
>     > Any thoughts?
> 
>     You can get some very useful information from "pip3 --version". As
>     well as pip's own version, it tells you the version of Python that
>     it's running under, AND what directory it's being run from. If you
>     want to request that similar info be added to other commands, I would
>     strongly recommend lifting the exact format of --version and using
>     that.
> 
>     (I'm not sure what "pip info" is, btw. My pip doesn't seem to have
>     that.)
> 
>     ChrisA
>     _______________________________________________
>     Python-ideas mailing list
>     Python-ideas at python.org <mailto:Python-ideas at python.org>
>     https://mail.python.org/mailman/listinfo/python-ideas
>     <https://mail.python.org/mailman/listinfo/python-ideas>
>     Code of Conduct: http://python.org/psf/codeofconduct/
>     <http://python.org/psf/codeofconduct/>
> 
> 
> 
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
> 


From peter.ed.oconnor at gmail.com  Fri Jul 20 05:03:12 2018
From: peter.ed.oconnor at gmail.com (Peter O'Connor)
Date: Fri, 20 Jul 2018 11:03:12 +0200
Subject: [Python-ideas] Idea: Deferred Default Arguments?
Message-ID: <CACCbJKrUq_vUYKO8ZzF0PCU8Ss0m_XG8MG3bLKkZjH3OucsV5Q@mail.gmail.com>

Often when programming I run into a situation where it would be nice to
have "deferred defaults".  Here is an example of what I mean:

    def subfunction_1(a=2, b=3, c=4):
        return a+b*c

    def subfunction_2(d=5, e=6, f=7):
        return d*e+f

    def main_function(a=2, b=3, c=4, d=5, e=6, f=7):
        return subfunction_1(a=a, b=b, c=c) + subfunction_2(d=d, e=e, f=f)

Here you can see that I had to redefine the defaults in the main_function.
In larger codebases, I find bugs often arise because defaults are defined
in multiple places, and somebody changes them in a lower-level function but
fails to realize that they are still defined differently in a higher
function.

The only way I currently see to achieve this is not very nice at all, and
completely obfuscates the signature of the function:

    def main_function(**kwargs):
        return subfunction_1(**{k: v for k, v in kwargs.items() if k in
['a', 'b', 'c']})
               + subfunction_2(**{k: v for k, v in kwargs.items() if k in
['d', 'e', 'f']})

What I was thinking was a "deferred" builtin that would just allow a lower
function to define the value (and raise an exception if anyone tried to use
it before it was defined)

    def main_function(a=deferred, b=deferred, c=deferred, d=deferred,
e=deferred, f=deferred):
        return subfunction_1(a=a, b=b, c=c) + subfunction_2(d=d, e=e, f=f)

I assume this has been discussed before somewhere, but couldn't find
anything on it, so please feel free to point me towards any previous
discussion on the topic.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180720/2e168bc2/attachment.html>

From rosuav at gmail.com  Fri Jul 20 05:06:43 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 20 Jul 2018 19:06:43 +1000
Subject: [Python-ideas] Adding Python interpreter info to "pip install"
In-Reply-To: <e54cfc98-19bc-883b-f13c-4783a60a004b@tjol.eu>
References: <CAPyZGSkyh+6RSq4trtLSQ8Tkkn7um8rhworuFcf=ztK-QO=DMA@mail.gmail.com>
 <CAPTjJmo7JFj15M_Y-+uBnCBmqL7KYi80NoihG5=2b+nS5uvQoA@mail.gmail.com>
 <CAPyZGSmsDhZJDAEzbn4Wf4izJXzWmNf+a0Woh39K2W6F=nmbkQ@mail.gmail.com>
 <e54cfc98-19bc-883b-f13c-4783a60a004b@tjol.eu>
Message-ID: <CAPTjJmo2xEVm_ccbvL7Zi96yZT8jZesRQVz9cZLvgcV8vicpPw@mail.gmail.com>

On Fri, Jul 20, 2018 at 5:56 PM, Thomas Jollans <tjol at tjol.eu> wrote:
> On 20/07/18 05:10, Al Sweigart wrote:
>> Sorry, I meant "pip list", rather than "pip info".
>>
>> I thought about the fact that "pip --version" provides this info, but 1)
>> it provides the location of pip, not the python interpreter it installs
>> packages for and 2) it would be an additional step for the
>> question-asker to go through after posting the output of "pip install".
>
> If the extra line of output was something like
>
> This is $(pip --version)
>
> that would be more consistent, and provide enough information.

Yes, that's what I was saying. It would remove the extra step, at the
price of one line of spam every 'pip install'. Is that worth it? Not
sure, but at least it's not a whole lot of complexity.

ChrisA

From rosuav at gmail.com  Fri Jul 20 05:07:52 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 20 Jul 2018 19:07:52 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <8856e53b-14c5-4f50-82c8-a48260a63ba2@googlegroups.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAFYqXL8GsTyAHEkrQqSDpv_+bCcJAVv2j-0d_zwNuLLfK7Cuuw@mail.gmail.com>
 <CAPTjJmqPS_wiFTJaJ8YUKJfWsNcCX=h9AQ1EmWFv6Xf9Wes4tA@mail.gmail.com>
 <5B502A80.3080406@canterbury.ac.nz>
 <CAPTjJmquqsqX8XL5RGdvebwHw-vgx15sQU2Qbyg3SKz3WHso2A@mail.gmail.com>
 <5B50DA1E.9030904@brenbarn.net>
 <8856e53b-14c5-4f50-82c8-a48260a63ba2@googlegroups.com>
Message-ID: <CAPTjJmrDM9=QaGjD4H4WoTCcED_r_sAa2o7GFBDZH2ZOrnc0Zg@mail.gmail.com>

On Fri, Jul 20, 2018 at 5:10 PM, Gr?gory Lielens
<gregory.lielens at gmail.com> wrote:
>
>
>> On 2018-07-19 02:11, Chris Angelico wrote:
>
>
>>
>> -snip-
>>         As far as I can see, these null-coalescing operators would break
>> that
>> model.  The PEP doesn't seem to provide for a "real" magic method
>> allowing users to override the actual behavior of the method.  (You can
>> only override __has_value__ to hook into it, but not define by fiat what
>> A ?? B does, as you can with other operators.)  And I think the reason
>> for this is that the operator itself is too specific, much more specific
>> in semantics than other operators.  (I had similar doubts about adding
>> the matrix-multiplication operator @.)
>>
>>         People keep saying that this null-coalescing behavior is so common
>> and
>> useful, etc., but that hasn't been my experience at all.  In my
>> experience, the desire to shortcut this kind of logic is more often a
>> sign of corner-cutting and insufficiently specified data formats, and is
>> likely to cause bugs later on.  Eventually it has to actually matter
>> whether something is None or not, and these operators just kick that can
>> down the road.  In terms of their abstract meaning, they are not
>> remotely close to as common or useful as operators like & and |.
>
>
> Fully agree with you on this

Excuse me, that wasn't my words you quoted. Watch your citations please :)

ChrisA

From jfine2358 at gmail.com  Fri Jul 20 05:16:39 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Fri, 20 Jul 2018 10:16:39 +0100
Subject: [Python-ideas] Idea: Deferred Default Arguments?
In-Reply-To: <CACCbJKrUq_vUYKO8ZzF0PCU8Ss0m_XG8MG3bLKkZjH3OucsV5Q@mail.gmail.com>
References: <CACCbJKrUq_vUYKO8ZzF0PCU8Ss0m_XG8MG3bLKkZjH3OucsV5Q@mail.gmail.com>
Message-ID: <CALD=Yf8OD90H9Qqwv8gibqxfvy99a7Dxnm_j7ngO3YmEiDVQrg@mail.gmail.com>

Hi Peter

Interesting problem. We can already get something like your proposed
solution by using None instead of deferred.

You have to start with
    def subfunction_1(a=None, b=None, c=None):
        if a is None: a = 2
        # similarly for b and c.
        return a+b*c

You will loose the default values being shown when you do help(subfunction).

There's another thread on PEP 505, active right now, that would allow
you to write something like
    a = a OR 2
to provide the default values.
https://mail.python.org/pipermail/python-ideas/2018-July/052071.html

I hope this helps. Please let us know if that might work for you.

-- 
Jonathan

On Fri, Jul 20, 2018 at 10:03 AM, Peter O'Connor
<peter.ed.oconnor at gmail.com> wrote:
> Often when programming I run into a situation where it would be nice to have
> "deferred defaults".  Here is an example of what I mean:
>
>     def subfunction_1(a=2, b=3, c=4):
>         return a+b*c
>
>     def subfunction_2(d=5, e=6, f=7):
>         return d*e+f
>
>     def main_function(a=2, b=3, c=4, d=5, e=6, f=7):
>         return subfunction_1(a=a, b=b, c=c) + subfunction_2(d=d, e=e, f=f)
>
> Here you can see that I had to redefine the defaults in the main_function.
> In larger codebases, I find bugs often arise because defaults are defined in
> multiple places, and somebody changes them in a lower-level function but
> fails to realize that they are still defined differently in a higher
> function.
>
> The only way I currently see to achieve this is not very nice at all, and
> completely obfuscates the signature of the function:
>
>     def main_function(**kwargs):
>         return subfunction_1(**{k: v for k, v in kwargs.items() if k in
> ['a', 'b', 'c']})
>                + subfunction_2(**{k: v for k, v in kwargs.items() if k in
> ['d', 'e', 'f']})
>
> What I was thinking was a "deferred" builtin that would just allow a lower
> function to define the value (and raise an exception if anyone tried to use
> it before it was defined)
>
>     def main_function(a=deferred, b=deferred, c=deferred, d=deferred,
> e=deferred, f=deferred):
>         return subfunction_1(a=a, b=b, c=c) + subfunction_2(d=d, e=e, f=f)
>
> I assume this has been discussed before somewhere, but couldn't find
> anything on it, so please feel free to point me towards any previous
> discussion on the topic.
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>

From gregory.lielens at gmail.com  Fri Jul 20 05:22:12 2018
From: gregory.lielens at gmail.com (=?UTF-8?Q?Gr=C3=A9gory_Lielens?=)
Date: Fri, 20 Jul 2018 02:22:12 -0700 (PDT)
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPTjJmrDM9=QaGjD4H4WoTCcED_r_sAa2o7GFBDZH2ZOrnc0Zg@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAFYqXL8GsTyAHEkrQqSDpv_+bCcJAVv2j-0d_zwNuLLfK7Cuuw@mail.gmail.com>
 <CAPTjJmqPS_wiFTJaJ8YUKJfWsNcCX=h9AQ1EmWFv6Xf9Wes4tA@mail.gmail.com>
 <5B502A80.3080406@canterbury.ac.nz>
 <CAPTjJmquqsqX8XL5RGdvebwHw-vgx15sQU2Qbyg3SKz3WHso2A@mail.gmail.com>
 <5B50DA1E.9030904@brenbarn.net>
 <8856e53b-14c5-4f50-82c8-a48260a63ba2@googlegroups.com>
 <CAPTjJmrDM9=QaGjD4H4WoTCcED_r_sAa2o7GFBDZH2ZOrnc0Zg@mail.gmail.com>
Message-ID: <d44595d8-c3e6-4b82-a76b-bb0a78cd3635@googlegroups.com>



On Friday, July 20, 2018 at 11:12:26 AM UTC+2, Chris Angelico wrote:
>
> On Fri, Jul 20, 2018 at 5:10 PM, Gr?gory Lielens
> <gregory... at gmail.com <javascript:>> wrote:
>
> Excuse me, that wasn't my words you quoted. Watch your citations please :)
>
> ChrisA
>

Yes, sorry, it was Brendan Barnwell. And I can not edit my previous 
post....:-/ 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180720/2a1d1203/attachment-0001.html>

From rosuav at gmail.com  Fri Jul 20 05:27:29 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 20 Jul 2018 19:27:29 +1000
Subject: [Python-ideas] Idea: Deferred Default Arguments?
In-Reply-To: <CACCbJKrUq_vUYKO8ZzF0PCU8Ss0m_XG8MG3bLKkZjH3OucsV5Q@mail.gmail.com>
References: <CACCbJKrUq_vUYKO8ZzF0PCU8Ss0m_XG8MG3bLKkZjH3OucsV5Q@mail.gmail.com>
Message-ID: <CAPTjJmrBduXbahSP9mp7sfa95NuUT-9vtadq77ZtAu5W9RzkAw@mail.gmail.com>

On Fri, Jul 20, 2018 at 7:03 PM, Peter O'Connor
<peter.ed.oconnor at gmail.com> wrote:
> Often when programming I run into a situation where it would be nice to have
> "deferred defaults".  Here is an example of what I mean:
>
>     def subfunction_1(a=2, b=3, c=4):
>         return a+b*c
>
>     def subfunction_2(d=5, e=6, f=7):
>         return d*e+f
>
>     def main_function(a=2, b=3, c=4, d=5, e=6, f=7):
>         return subfunction_1(a=a, b=b, c=c) + subfunction_2(d=d, e=e, f=f)
>
> Here you can see that I had to redefine the defaults in the main_function.
> In larger codebases, I find bugs often arise because defaults are defined in
> multiple places, and somebody changes them in a lower-level function but
> fails to realize that they are still defined differently in a higher
> function.
>
> The only way I currently see to achieve this is not very nice at all, and
> completely obfuscates the signature of the function:
>
>     def main_function(**kwargs):
>         return subfunction_1(**{k: v for k, v in kwargs.items() if k in
> ['a', 'b', 'c']})
>                + subfunction_2(**{k: v for k, v in kwargs.items() if k in
> ['d', 'e', 'f']})

Hmm. This might be something where a bit of a helper could, well,
help. Is this a pattern that you make use of a lot? Suppose you write
your main function like this:

@combines(subfunction_1, subfunction_2)
def main_function(sf1, sf2):
    return subfunction_1(**sf1) + subfunction_2(**sf2)

or even like this:

@precombine
def main_function(subfunction_1, subfunction_2):
    return subfunction_1 + subfunction_2

The decorator could then generate a new function, with a proper
signature (available to help()), that hides the messy details of
filtering kwargs - and in the case of precombine, actually calls the
function as well. (Obviously you can't use this if you might want to
call a subfunction conditionally.)

Another option would be to have all your subfunctions absorb and
ignore all unexpected arguments.

    def subfunction_1(a=2, b=3, c=4, **_):
        return a+b*c

    def subfunction_2(d=5, e=6, f=7, **_):
        return d*e+f

    def main_function(**kw):
        return subfunction_1(**kw) + subfunction_2(**kw)

This doesn't help with the signature, but it's an easy thing to do.
You'd have to document your parameters separately from your functions;
appropriate for something like the subprocess module (where a bunch of
functions basically pipe their args straight into Popen), not
appropriate for a lot of other places.

On the plus side, both of these ideas work with existing Python
versions, so you wouldn't have to wait for 3.8 or 3.9 :)

ChrisA

From gregory.lielens at gmail.com  Fri Jul 20 05:31:14 2018
From: gregory.lielens at gmail.com (=?UTF-8?Q?Gr=C3=A9gory_Lielens?=)
Date: Fri, 20 Jul 2018 02:31:14 -0700 (PDT)
Subject: [Python-ideas] Idea: Deferred Default Arguments?
In-Reply-To: <CACCbJKrUq_vUYKO8ZzF0PCU8Ss0m_XG8MG3bLKkZjH3OucsV5Q@mail.gmail.com>
References: <CACCbJKrUq_vUYKO8ZzF0PCU8Ss0m_XG8MG3bLKkZjH3OucsV5Q@mail.gmail.com>
Message-ID: <048cd1da-b163-4b40-9343-a57f319cb8bf@googlegroups.com>

Excellent point! I am not fully convinced by the syntax yet, but having 
proposed something is already very valuable and  I do not have a better 
proposal. As I had to defer defaults countless times, and each times 
grumped about it, I hope something will come out of this...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180720/2ff0650e/attachment.html>

From stefan_ml at behnel.de  Fri Jul 20 05:32:04 2018
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 20 Jul 2018 11:32:04 +0200
Subject: [Python-ideas] Performance improvements via static typing
In-Reply-To: <CABypBWrqKatZBKy3VBdzpr+o6cOgxhjpnqt7tzKW1+sraeJMSA@mail.gmail.com>
References: <CABypBWrqKatZBKy3VBdzpr+o6cOgxhjpnqt7tzKW1+sraeJMSA@mail.gmail.com>
Message-ID: <pisa2h$8rj$1@blaine.gmane.org>

Michael Hall schrieb am 19.07.2018 um 15:51:
> While I am aware of projects like Cython and mypy, it seems to make sense
> for CPython to allow optional enforcement of type hints, with compiler
> optimizations related to it to be used. While this would not receive the
> same level of performance benefits as using ctypes directly, there do
> appear to be various gains available here.

Well, first of all, a C level type check at runtime is quite fast compared
to a byte code dispatch with stack argument handling, and can be done
regardless of any type hints. There are various code patterns that would
suggest a certain type ("x.append()" probably appends to a list, "x.get()"
will likely be a dict lookup, etc.), and that can be optimised for without
static type declarations and even without any type hints.

Then, the mere fact that user code says "a: List" does not help in any way.
Even "a: list" would not help, because any behaviour of that "list" might
have been overridden by the list subtype that the function eventually receives.

The same applies to "x: float". Here, in order to gain speed, the compiler
would have to generate two separate code paths through the entire function,
one for C double computations, and one for Python float subtypes. And then
exponentiate that by the number of other typed arguments that may or may
not contain subtypes. Quite some overhead.

It's unclear if the gain would have any reasonable relation to the effort
in the end. Sure, type hints could be used as a bare trigger for certain
optimistic optimisations, but then, it's cheap enough to always apply these
optimisations regardless, via a C type check.

Stefan


From contact at brice.xyz  Fri Jul 20 06:02:10 2018
From: contact at brice.xyz (Brice Parent)
Date: Fri, 20 Jul 2018 12:02:10 +0200
Subject: [Python-ideas] Idea: Deferred Default Arguments?
In-Reply-To: <CACCbJKrUq_vUYKO8ZzF0PCU8Ss0m_XG8MG3bLKkZjH3OucsV5Q@mail.gmail.com>
References: <CACCbJKrUq_vUYKO8ZzF0PCU8Ss0m_XG8MG3bLKkZjH3OucsV5Q@mail.gmail.com>
Message-ID: <0ba1c66c-b752-2565-3932-64fd0f966fad@brice.xyz>

That's a problem I've also encountered countless times.

But the solution you propose only covers the use cases where we always 
pass the arguments. If your main function looks like the following, you 
wouldn't be able to set them as deferred:

def main_function(action, a=2, b=3):
 ??? if action== "product":
 ??????? return a * b

 ??? if action== "complicated_action":
 ? ?? ?? return complicated_action(a=a, b=b)

 ??? ...

Maybe we could make deferred a keywork (only in this context), like this:

def main_function(action, deferred a=2, deferred b=3):
 ??? if action== "product":
 ??????? return a * b? # will return 6

 ??? if action== "complicated_action":
 ? ?? ?? return complicated_action(a=a, b=b)# will return the same as 
complicated_action()

 ??? ...


deferred variables would just be classic variables some 'deferred' flag, 
which when they are passed to a function with a default value, are 
redefined to that value.
And probably, when we use them (read them or assign a new value to 
them), they would lose this deferred flag.


Le 20/07/2018 ? 11:03, Peter O'Connor a ?crit?:
> Often when programming I run into a situation where it would be nice 
> to have "deferred defaults".? Here is an example of what I mean:
> ? ? def subfunction_1(a=2, b=3, c=4):
> ? ? ? ? return a+b*c
>
> ? ? def subfunction_2(d=5, e=6, f=7):
> ? ? ? ? return d*e+f
>
> ? ? def main_function(a=2, b=3, c=4, d=5, e=6, f=7):
> ? ? ? ? return subfunction_1(a=a, b=b, c=c) + subfunction_2(d=d, e=e, f=f)
>
> Here you can see that I had to redefine the defaults in the 
> main_function.? In larger codebases, I find bugs often arise because 
> defaults are defined in multiple places, and somebody changes them in 
> a lower-level function but fails to realize that they are still 
> defined differently in a higher function.
>
> The only way I currently see to achieve this is not very nice at all, 
> and completely obfuscates the signature of the function:
>
> ? ? def main_function(**kwargs):
> ? ? ? ? return subfunction_1(**{k: v for k, v in kwargs.items() if k 
> in ['a', 'b', 'c']})
> ? ? ? ? ? ? ? ?+ subfunction_2(**{k: v for k, v in kwargs.items() if k 
> in ['d', 'e', 'f']})
>
> What I was thinking was a "deferred" builtin that would just allow a 
> lower function to define the value (and raise an exception if anyone 
> tried to use it before it was defined)
>
> ? ? def main_function(a=deferred, b=deferred, c=deferred, d=deferred, 
> e=deferred, f=deferred):
> ? ? ? ? return subfunction_1(a=a, b=b, c=c) + subfunction_2(d=d, e=e, f=f)
>
> I assume this has been discussed before somewhere, but couldn't find 
> anything on it, so please feel free to point me towards any previous 
> discussion on the topic.
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180720/fe58d1ac/attachment-0001.html>

From jfine2358 at gmail.com  Fri Jul 20 06:50:00 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Fri, 20 Jul 2018 11:50:00 +0100
Subject: [Python-ideas] Multi-core reference count garbage collection
In-Reply-To: <CALD=Yf-Ht4uDMHkvXp8i+xesw=e9zw4i+nAuRJFMVppNSYrboA@mail.gmail.com>
References: <CALD=Yf-Ht4uDMHkvXp8i+xesw=e9zw4i+nAuRJFMVppNSYrboA@mail.gmail.com>
Message-ID: <CALD=Yf9jbZoG6W0Qn-jDAs0iqruGocc0Fuu_yC=U6LNhrCk8Cg@mail.gmail.com>

In an earlier post, I defined BUFFERED multi-core reference count
garbage collection. The basic idea is that each worker process store
in buffers the need to do an INCR and DECR on the reference count for
a piece of memory with a given ID. There is then a garbage collection
process that does the right thing.
https://mail.python.org/pipermail/python-ideas/2018-July/052054.html

An important aside. I'm very grateful for all the comments, which are
very helpful, made to the first part. It's really nice to know that
mine is not a new idea, and about some of the features and quirks (or
optimisations if you like) in the CPython GC system.

Here, I describe the easy part of doing the right thing. And make a
start on the hard part.

EASY PART
The GC process is responsible for keeping the running totals of the
reference counts. It does this by processing the INCR and DECR buffers
that are passed to it. (Once these buffers are processed they can, of
course, be passed back to the system for reuse.)

Where to store the totals of the reference counts is a design
decision. The main constraint is that ONLY the GC process should be
changing these counts. The buffering is roughly equivalent to the
changes made by the worker processes being queued for the GC process
to perform. But buffering also allows the 'queue' to be reordered.
This can substantially reduce the number of system calls required to
coordinate the various processes. And the amount of time a process is
locked out from running.

We could, following CPython, store the reference counts as a system
field in the object being stored. This is probably most economical,
regarding use of memory. However, if the GC process has associated to
it some high-speed memory of its own, it will be quicker if possible
to store the reference count totals in that high-speed memory. And so
storing the counts separately will probably make better use of
process-specific high speed memory.

Either way the algorithm is easy.

HARD PART
The do-nothing garbage collection avoids the hard part, which is to
know when a piece of memory can be reclaimed. With single-core
reference counting, this would be when the count becomes zero. But is
the same true for multi-core garbage collection?

The answer is both YES and NO.

First for the NO. They buffered scheme described here is asychronous,
and in particular reordering. So there might be INCR operations
pending, that would take the global reference count from zero to one.
(And it can go the other way. The object might be inaccessible, but
not all of its DECR operations are processed yet.)

So, and this is important, we cannot rely on the reference count
totals owned by the GC process to tell is if there is or is not a
reference to the object in the worker threads.

Now for the YES. The operations for the processes can be put in a
definite temporal order (i.e. serialized). When this is done then
(assuming every process is behaving as it should) the following
statement is true:

Suppose at some point T in time the quantity
* the GC process reference count for the object with some ID
* plus any pending INCR operations in worker process buffers
* minus any similarly pending DECR operations
is zero. Call this the adjusted reference count.

If that is true at T, then after time T the object with the given ID
is inaccessible to the worker processes. And so the piece of memory
can be reclaimed.

We can, of course, compute the adjusted reference count by pausing all
working threads. In other words, provide and use a global interpreter
lock.

The remainder of the hard part is to get hold of adjusted reference
counts, without having to lock the worker threads.

As before, remarks and comments very welcome. The previous ones were
much appreciated.

-- 
Jonathan

From gregory.lielens at gmail.com  Fri Jul 20 07:03:57 2018
From: gregory.lielens at gmail.com (=?UTF-8?Q?Gr=C3=A9gory_Lielens?=)
Date: Fri, 20 Jul 2018 04:03:57 -0700 (PDT)
Subject: [Python-ideas] Idea: Deferred Default Arguments?
In-Reply-To: <0ba1c66c-b752-2565-3932-64fd0f966fad@brice.xyz>
References: <CACCbJKrUq_vUYKO8ZzF0PCU8Ss0m_XG8MG3bLKkZjH3OucsV5Q@mail.gmail.com>
 <0ba1c66c-b752-2565-3932-64fd0f966fad@brice.xyz>
Message-ID: <7561290a-7baa-490b-9afd-5f6f0dbb04f1@googlegroups.com>

A crazy idea, didn't think much about it yet:

def subfunc(c,a=0,b=1):
  Blabla

def function(c,d=3, from args(subfunc) import a,b):
  Blabla
  return anotherfunc(a+b+c+d,subfunc(c,a,b))
  


From jfine2358 at gmail.com  Fri Jul 20 07:30:26 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Fri, 20 Jul 2018 12:30:26 +0100
Subject: [Python-ideas] Idea: Deferred Default Arguments?
In-Reply-To: <0ba1c66c-b752-2565-3932-64fd0f966fad@brice.xyz>
References: <CACCbJKrUq_vUYKO8ZzF0PCU8Ss0m_XG8MG3bLKkZjH3OucsV5Q@mail.gmail.com>
 <0ba1c66c-b752-2565-3932-64fd0f966fad@brice.xyz>
Message-ID: <CALD=Yf8zZV8bXqeXJt0jnonjS7h_k0ceMP9d=xYXoKHMWP5B3w@mail.gmail.com>

Excellent contributions. I'm going to try to (partially) consolidate
what we've got.

REVIEW
=======
I'll start by reviewing the situation regarding default arguments.
There are two basic patterns for default arguments.

The first is
---
def fn(a=EXP):
    # body of function
---

The second is
---
def fn(a=None):
    if a is None:
        a = EXP
    # body of function
---

Here, EXP is any Python expression. A fairly gotcha is to use a list,
or some other mutable object, as EXP. This happens when you write
---
def fn(a=[]):
   # body of function
---
because then EXP = '[]' which will be evaluated just once, and every
call fn() will be using the same list! To avoid this you should use
the second pattern. I think there may be an example of this in the
standard Python tutorial.

(An aside. You probably need the second pattern if you EXP is, say,
([], []). Although technically immutable, this value has mutable
members. And can't be hashed, or use as a dictionary key or element of
a set.)

WHEN TO USE None
=================

If you want something mutable as the 'default argument' you have to
use the second pattern. If your default argument is immutable then you
can if you wish use the second pattern.

But you don't have to use the second pattern. When you use the second
pattern, the expression f(None) means 'create for me the default
argument' (and raise an exception if there isn't one).

Think about it. For immutable EXP, fn() is the same, whether fn is
coded using the first pattern or the second. But the value of fn(None)
depends very much on which pattern is used to code fn().

So here's the big conclusion (drum roll):
===
fn should be coded using the second pattern if we wish to pass None as
a sentinel argument to fn.
===

SUMMARY
=========
My suggestion was to use the second pattern to solve Peter O'Connor's
original problem. It can be done now, but is a bit verbose, and looses
useful information in help(fn).

Brice Parent's suggestion was to introduce a keyword deferred, like so
---
def fn(deferred a=EXP):
    # body of function
---
which I like to think of as a syntactic shorthand for the second pattern.

I sort of think we've now got a reasonable answer for Peter's problem.
What do you think, Peter? And Brice, are you happy with my
interpretation of your deferred keyword?

---
Jonathan

From rhodri at kynesim.co.uk  Fri Jul 20 08:14:33 2018
From: rhodri at kynesim.co.uk (Rhodri James)
Date: Fri, 20 Jul 2018 13:14:33 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <5B512D11.30701@canterbury.ac.nz>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAFYqXL8GsTyAHEkrQqSDpv_+bCcJAVv2j-0d_zwNuLLfK7Cuuw@mail.gmail.com>
 <CAPTjJmqPS_wiFTJaJ8YUKJfWsNcCX=h9AQ1EmWFv6Xf9Wes4tA@mail.gmail.com>
 <5B502A80.3080406@canterbury.ac.nz>
 <c5d3ec35-5dae-f6f2-affa-79303e7d0fe1@kynesim.co.uk>
 <5B512D11.30701@canterbury.ac.nz>
Message-ID: <7c6f0141-62be-48a3-822e-53350d66318e@kynesim.co.uk>

On 20/07/18 01:30, Greg Ewing wrote:
> Rhodri James wrote:
>> On 19/07/18 07:06, Greg Ewing wrote:
>>
>>> There's no such tradition for the new
>>> operators being proposed.
>>
>> There is, actually, it's just not a long one.? C# has had null-aware 
>> operators for a while, for example.
> 
> THere's a precedent, yes, but I wouldn't call it a tradition.
> A substantial period of time is part of the definition of the
> word.
> 
> Wikipedia:
> 
> "A tradition is a belief or behavior passed down within a group
> or society with symbolic meaning or special significance with
> origins in the past."
> 
> Merriam-Webster:
> 
> "the handing down of information, beliefs, or customs from one
> generation to another."
> 
> I don't think C# has been around long enought to span multiple
> generations of programmers.

I go with SF fandom's traditional :-) definition: "somebody did it 
once."  If it's been done more than once, it's an honoured tradition.

-- 
Rhodri James *-* Kynesim Ltd

From rosuav at gmail.com  Fri Jul 20 08:16:20 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 20 Jul 2018 22:16:20 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <7c6f0141-62be-48a3-822e-53350d66318e@kynesim.co.uk>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAFYqXL8GsTyAHEkrQqSDpv_+bCcJAVv2j-0d_zwNuLLfK7Cuuw@mail.gmail.com>
 <CAPTjJmqPS_wiFTJaJ8YUKJfWsNcCX=h9AQ1EmWFv6Xf9Wes4tA@mail.gmail.com>
 <5B502A80.3080406@canterbury.ac.nz>
 <c5d3ec35-5dae-f6f2-affa-79303e7d0fe1@kynesim.co.uk>
 <5B512D11.30701@canterbury.ac.nz>
 <7c6f0141-62be-48a3-822e-53350d66318e@kynesim.co.uk>
Message-ID: <CAPTjJmrYdoD8KjJr41HLpWon--Y9PQ23aMAamp0TPUXdMxU4zw@mail.gmail.com>

On Fri, Jul 20, 2018 at 10:14 PM, Rhodri James <rhodri at kynesim.co.uk> wrote:
> I go with SF fandom's traditional :-) definition: "somebody did it once."
> If it's been done more than once, it's an honoured tradition.

But if Shakespeare did it, it's just the way the language is.

I think Fortran is the programming world's Shakespeare.

ChrisA

From p.f.moore at gmail.com  Fri Jul 20 08:24:14 2018
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 20 Jul 2018 13:24:14 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPTjJmrYdoD8KjJr41HLpWon--Y9PQ23aMAamp0TPUXdMxU4zw@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAFYqXL8GsTyAHEkrQqSDpv_+bCcJAVv2j-0d_zwNuLLfK7Cuuw@mail.gmail.com>
 <CAPTjJmqPS_wiFTJaJ8YUKJfWsNcCX=h9AQ1EmWFv6Xf9Wes4tA@mail.gmail.com>
 <5B502A80.3080406@canterbury.ac.nz>
 <c5d3ec35-5dae-f6f2-affa-79303e7d0fe1@kynesim.co.uk>
 <5B512D11.30701@canterbury.ac.nz>
 <7c6f0141-62be-48a3-822e-53350d66318e@kynesim.co.uk>
 <CAPTjJmrYdoD8KjJr41HLpWon--Y9PQ23aMAamp0TPUXdMxU4zw@mail.gmail.com>
Message-ID: <CACac1F-gj6PEriFTbbBfZ+uity7GA2N5F-fYO4wZpKvJH2+QXA@mail.gmail.com>

On 20 July 2018 at 13:16, Chris Angelico <rosuav at gmail.com> wrote:
> On Fri, Jul 20, 2018 at 10:14 PM, Rhodri James <rhodri at kynesim.co.uk> wrote:
>> I go with SF fandom's traditional :-) definition: "somebody did it once."
>> If it's been done more than once, it's an honoured tradition.
>
> But if Shakespeare did it, it's just the way the language is.
>
> I think Fortran is the programming world's Shakespeare.

Or maybe COBOL:

"And lo from yonder file a record doth appear"...

Paul

From rosuav at gmail.com  Fri Jul 20 08:30:40 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 20 Jul 2018 22:30:40 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CACac1F-gj6PEriFTbbBfZ+uity7GA2N5F-fYO4wZpKvJH2+QXA@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAFYqXL8GsTyAHEkrQqSDpv_+bCcJAVv2j-0d_zwNuLLfK7Cuuw@mail.gmail.com>
 <CAPTjJmqPS_wiFTJaJ8YUKJfWsNcCX=h9AQ1EmWFv6Xf9Wes4tA@mail.gmail.com>
 <5B502A80.3080406@canterbury.ac.nz>
 <c5d3ec35-5dae-f6f2-affa-79303e7d0fe1@kynesim.co.uk>
 <5B512D11.30701@canterbury.ac.nz>
 <7c6f0141-62be-48a3-822e-53350d66318e@kynesim.co.uk>
 <CAPTjJmrYdoD8KjJr41HLpWon--Y9PQ23aMAamp0TPUXdMxU4zw@mail.gmail.com>
 <CACac1F-gj6PEriFTbbBfZ+uity7GA2N5F-fYO4wZpKvJH2+QXA@mail.gmail.com>
Message-ID: <CAPTjJmpcHQkhR6C2hjQp+a=qmmg87pAXMkm_cD3_Cv9DAWrERA@mail.gmail.com>

On Fri, Jul 20, 2018 at 10:24 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 20 July 2018 at 13:16, Chris Angelico <rosuav at gmail.com> wrote:
>> On Fri, Jul 20, 2018 at 10:14 PM, Rhodri James <rhodri at kynesim.co.uk> wrote:
>>> I go with SF fandom's traditional :-) definition: "somebody did it once."
>>> If it's been done more than once, it's an honoured tradition.
>>
>> But if Shakespeare did it, it's just the way the language is.
>>
>> I think Fortran is the programming world's Shakespeare.
>
> Or maybe COBOL:
>
> "And lo from yonder file a record doth appear"...

Hah! But I was thinking of all those uber-obvious features like "a +
b" meaning addition. Those symbols, and the infix style, aren't
"traditions" - they're the baseline that we measure everything else
against. Also, the use of decimal digits to represent literals; if you
*don't* use decimal, you're unusual. (Which makes the Shakespeare
Programming Language [1] an ironic example, since it doesn't use
decimal digits for numeric literals.)

ChrisA
[1] http://shakespearelang.sourceforge.net/report/shakespeare/

From peter.ed.oconnor at gmail.com  Fri Jul 20 08:39:13 2018
From: peter.ed.oconnor at gmail.com (Peter O'Connor)
Date: Fri, 20 Jul 2018 14:39:13 +0200
Subject: [Python-ideas] Idea: Deferred Default Arguments?
In-Reply-To: <CALD=Yf8zZV8bXqeXJt0jnonjS7h_k0ceMP9d=xYXoKHMWP5B3w@mail.gmail.com>
References: <CACCbJKrUq_vUYKO8ZzF0PCU8Ss0m_XG8MG3bLKkZjH3OucsV5Q@mail.gmail.com>
 <0ba1c66c-b752-2565-3932-64fd0f966fad@brice.xyz>
 <CALD=Yf8zZV8bXqeXJt0jnonjS7h_k0ceMP9d=xYXoKHMWP5B3w@mail.gmail.com>
Message-ID: <CACCbJKrZZG0T_Avn0Y_YKgN3OWdX-GEtC97BX9JEg7JZuEfZgQ@mail.gmail.com>

On Fri, Jul 20, 2018 at 1:30 PM, Jonathan Fine <jfine2358 at gmail.com> wrote:
>
>
> I sort of think we've now got a reasonable answer for Peter's problem.
> What do you think, Peter? And Brice, are you happy with my
> interpretation of your deferred keyword?


I think the problem with the "None" approach (second pattern) is that it
forces the writer of the subfunction to write their defaults in a more
awkward way in anticipation of other functions which defer their defaults
to it.  Translated to the original example, it would become:

    def subfunction_1(a=None, b=None, c=None):
        if a is None: a=1
        if b is None: b=2
        if c is None: c=3
        return a+b*c

    def subfunction_2(d=None, e=None, f=None):
        if d is None: d=5
        if e is None: e=6
        if f is None: f=7
        return d*e+f

    def main_function(a=None, b=None, c=None, d=None, e=None, f=None):
        return subfunction_1(a=a, b=b, c=c) + subfunction_2(d=d, e=e, f=f)

subfunction_1 may be written by someone totally different from the author
of main_function, and may even be in a different codebase.  For the author
of subfunction_1, it makes no sense to use the "None" approach instead of
python's normal default mechanism (since all arguments here are
immutables).



On Fri, Jul 20, 2018 at 1:30 PM, Jonathan Fine <jfine2358 at gmail.com> wrote:

> Excellent contributions. I'm going to try to (partially) consolidate
> what we've got.
>
> REVIEW
> =======
> I'll start by reviewing the situation regarding default arguments.
> There are two basic patterns for default arguments.
>
> The first is
> ---
> def fn(a=EXP):
>     # body of function
> ---
>
> The second is
> ---
> def fn(a=None):
>     if a is None:
>         a = EXP
>     # body of function
> ---
>
> Here, EXP is any Python expression. A fairly gotcha is to use a list,
> or some other mutable object, as EXP. This happens when you write
> ---
> def fn(a=[]):
>    # body of function
> ---
> because then EXP = '[]' which will be evaluated just once, and every
> call fn() will be using the same list! To avoid this you should use
> the second pattern. I think there may be an example of this in the
> standard Python tutorial.
>
> (An aside. You probably need the second pattern if you EXP is, say,
> ([], []). Although technically immutable, this value has mutable
> members. And can't be hashed, or use as a dictionary key or element of
> a set.)
>
> WHEN TO USE None
> =================
>
> If you want something mutable as the 'default argument' you have to
> use the second pattern. If your default argument is immutable then you
> can if you wish use the second pattern.
>
> But you don't have to use the second pattern. When you use the second
> pattern, the expression f(None) means 'create for me the default
> argument' (and raise an exception if there isn't one).
>
> Think about it. For immutable EXP, fn() is the same, whether fn is
> coded using the first pattern or the second. But the value of fn(None)
> depends very much on which pattern is used to code fn().
>
> So here's the big conclusion (drum roll):
> ===
> fn should be coded using the second pattern if we wish to pass None as
> a sentinel argument to fn.
> ===
>
> SUMMARY
> =========
> My suggestion was to use the second pattern to solve Peter O'Connor's
> original problem. It can be done now, but is a bit verbose, and looses
> useful information in help(fn).
>
> Brice Parent's suggestion was to introduce a keyword deferred, like so
> ---
> def fn(deferred a=EXP):
>     # body of function
> ---
> which I like to think of as a syntactic shorthand for the second pattern.
>
> I sort of think we've now got a reasonable answer for Peter's problem.
> What do you think, Peter? And Brice, are you happy with my
> interpretation of your deferred keyword?
>
> ---
> Jonathan
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180720/375ac625/attachment.html>

From judah.j.levy at gmail.com  Fri Jul 20 08:47:04 2018
From: judah.j.levy at gmail.com (Judah Levy)
Date: Fri, 20 Jul 2018 08:47:04 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPTjJmpcHQkhR6C2hjQp+a=qmmg87pAXMkm_cD3_Cv9DAWrERA@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAFYqXL8GsTyAHEkrQqSDpv_+bCcJAVv2j-0d_zwNuLLfK7Cuuw@mail.gmail.com>
 <CAPTjJmqPS_wiFTJaJ8YUKJfWsNcCX=h9AQ1EmWFv6Xf9Wes4tA@mail.gmail.com>
 <5B502A80.3080406@canterbury.ac.nz>
 <c5d3ec35-5dae-f6f2-affa-79303e7d0fe1@kynesim.co.uk>
 <5B512D11.30701@canterbury.ac.nz>
 <7c6f0141-62be-48a3-822e-53350d66318e@kynesim.co.uk>
 <CAPTjJmrYdoD8KjJr41HLpWon--Y9PQ23aMAamp0TPUXdMxU4zw@mail.gmail.com>
 <CACac1F-gj6PEriFTbbBfZ+uity7GA2N5F-fYO4wZpKvJH2+QXA@mail.gmail.com>
 <CAPTjJmpcHQkhR6C2hjQp+a=qmmg87pAXMkm_cD3_Cv9DAWrERA@mail.gmail.com>
Message-ID: <CAFkYj5=Pf45T8T_pqP6q-K7W2VQ8OF4kUUiKrj0tHUocGVJGKg@mail.gmail.com>

One of the big problems with most of the proposed spellings is that
question marks to a learner will feel like truthiness tests and not None
tests. I would propose adding a new default statement for the most useful
part of this PEP, the ?= assignment

currently:
if x is None:
     x = blah

with `?=`:
x ?= blah

with defalult:
default x: blah

This a) makes it obvious even to an amateur python-reader that x will only
be changed if it does not yet exist, and b) has no similarity to the
truthiness checks involved with the standard meaning of `?` as the ternary
operator in many other languages.

On Fri, Jul 20, 2018 at 8:30 AM Chris Angelico <rosuav at gmail.com> wrote:

> On Fri, Jul 20, 2018 at 10:24 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> > On 20 July 2018 at 13:16, Chris Angelico <rosuav at gmail.com> wrote:
> >> On Fri, Jul 20, 2018 at 10:14 PM, Rhodri James <rhodri at kynesim.co.uk>
> wrote:
> >>> I go with SF fandom's traditional :-) definition: "somebody did it
> once."
> >>> If it's been done more than once, it's an honoured tradition.
> >>
> >> But if Shakespeare did it, it's just the way the language is.
> >>
> >> I think Fortran is the programming world's Shakespeare.
> >
> > Or maybe COBOL:
> >
> > "And lo from yonder file a record doth appear"...
>
> Hah! But I was thinking of all those uber-obvious features like "a +
> b" meaning addition. Those symbols, and the infix style, aren't
> "traditions" - they're the baseline that we measure everything else
> against. Also, the use of decimal digits to represent literals; if you
> *don't* use decimal, you're unusual. (Which makes the Shakespeare
> Programming Language [1] an ironic example, since it doesn't use
> decimal digits for numeric literals.)
>
> ChrisA
> [1] http://shakespearelang.sourceforge.net/report/shakespeare/
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180720/7310f491/attachment-0001.html>

From steve at pearwood.info  Fri Jul 20 08:45:16 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 20 Jul 2018 22:45:16 +1000
Subject: [Python-ideas] Idea: Deferred Default Arguments?
In-Reply-To: <CACCbJKrUq_vUYKO8ZzF0PCU8Ss0m_XG8MG3bLKkZjH3OucsV5Q@mail.gmail.com>
References: <CACCbJKrUq_vUYKO8ZzF0PCU8Ss0m_XG8MG3bLKkZjH3OucsV5Q@mail.gmail.com>
Message-ID: <20180720124516.GA8744@ando.pearwood.info>

On Fri, Jul 20, 2018 at 11:03:12AM +0200, Peter O'Connor wrote:
> Often when programming I run into a situation where it would be nice to
> have "deferred defaults".  Here is an example of what I mean:
> 
>     def subfunction_1(a=2, b=3, c=4):
>         return a+b*c
> 
>     def subfunction_2(d=5, e=6, f=7):
>         return d*e+f
> 
>     def main_function(a=2, b=3, c=4, d=5, e=6, f=7):
>         return subfunction_1(a=a, b=b, c=c) + subfunction_2(d=d, e=e, f=f)
> 
> Here you can see that I had to redefine the defaults in the main_function.

Perhaps you mean duplicate, or repeat, or copy. But surely they're not 
redefined -- then they would have different values. Being able to 
redefine the defaults in a wrapper function is a feature.

Putting aside the terminology, I think this is a minor annoyance: DRY 
violations when setting default values.


> In larger codebases, I find bugs often arise because defaults are defined
> in multiple places, and somebody changes them in a lower-level function but
> fails to realize that they are still defined differently in a higher
> function.

Changing function defaults in production code without a period of 
deprecation and warnings is a no-no. Default values are a part of the 
function API, and changing the public API of a function without warning 
is asking for trouble.

But during development, there are no such constraints (your only users 
are your development team) and it is a real nuisance keeping defaults in 
sync across multiple functions and/or classes.

I've raised this issue in the past:

https://mail.python.org/pipermail/python-list/2016-September/714546.html

and I still don't have a great solution for it.


> The only way I currently see to achieve this is not very nice at all, and
> completely obfuscates the signature of the function:
[...]

There are lots of other ways to solve this. None of which are great.


> What I was thinking was a "deferred" builtin that would just allow a lower
> function to define the value (and raise an exception if anyone tried to use
> it before it was defined)
[...]
> I assume this has been discussed before somewhere, but couldn't find
> anything on it, so please feel free to point me towards any previous
> discussion on the topic.

https://mail.python.org/pipermail/python-ideas/2011-July/010678.html


-- 
Steve

From desmoulinmichel at gmail.com  Fri Jul 20 09:06:54 2018
From: desmoulinmichel at gmail.com (Michel Desmoulin)
Date: Fri, 20 Jul 2018 15:06:54 +0200
Subject: [Python-ideas] PEP 505: None-aware operators: operators ?= and
 ?? and OR
In-Reply-To: <CA+w-FZ8GgTRGm+duzmT=CgVeX_=mpWgUPNzOHH=D-a=W8J7-AA@mail.gmail.com>
References: <CALD=Yf-0dLkGKN+1gb11nv4s0J4yiySehszZ+rS2BZVciR+LHA@mail.gmail.com>
 <CA+w-FZ8GgTRGm+duzmT=CgVeX_=mpWgUPNzOHH=D-a=W8J7-AA@mail.gmail.com>
Message-ID: <6ac81150-57a8-732e-d7c8-f2b3f31bbf1a@gmail.com>

Proposal a __very__ dumbed down "try" expression
====================================================

    variable = try Expression as not sentinel [else default]

The semantic is:

1 - try the expression. If any call (from a normal method, or a
__dunder__ one) returns the sentinel, we shortcircuit.

2 - if we shortcircuit, the variable will be set do the default value.
If no default is provided, the variable is set to the sentinel.

3 - if we don't shortcircuit, the returned value is the one from the
expression.


Example
==========


    value = try foo.bar[0] as not None

If anything returns None, we shortcircuit and value will end up being
None. Otherwise value contains the result of foo.bar[0].

    value = try foo.bar[0] as not True

If anything returns True, we shortcircuit and value will end being True.
Otherwise value contains the result of foo.bar[0].

    value = try foo.bar[0] as not None else "foo"

If anything returns None, we shortcircuit and value will end up being
'foo'. Otherwise value contains the result of foo.bar[0].


    value = try foo.bar[0] as not True else "foo"

If anything returns True, we shortcircuit and value will end up being
'foo'. Otherwise value contains the result of foo.bar[0].


This as several benefits:
=========================

- Does not introduce new keywords or operators

- Read pretty much as ordinary Python

- Allow for inline try, as suggested before several times on
python-idea, but restrict it to a very narrow subset of what it can do.
This will end the debate about inline try/except, while sill providing a
sane subsets of use cases it would have provided.

- more flexible than ? as it allows to choose what to choose the
sentinel and the default value instead of just None.

- the expression is untouched. It's surrounded by the rest, so it still
reads as the expression would without try / as.

- avoding "except", exception names and ":" make sure we don't confuse
this try expression with the regular try/except statements. try would
then be a very restricted version of it, like lambda is for def.


If you don't like these keywords
===============================

We can opt for something different:

value = try Expression as not sentinel [or default]
value = try Expression if not sentinel [else default]
value = try Expression with not sentinel [else default]

The proposal is more about the structure try (stuff), guard against a
sentinel, and optionally return a default value.

Besides, I don't know what the grammar would allow.

From jfine2358 at gmail.com  Fri Jul 20 09:39:25 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Fri, 20 Jul 2018 14:39:25 +0100
Subject: [Python-ideas] Idea: Deferred Default Arguments?
In-Reply-To: <CACCbJKrZZG0T_Avn0Y_YKgN3OWdX-GEtC97BX9JEg7JZuEfZgQ@mail.gmail.com>
References: <CACCbJKrUq_vUYKO8ZzF0PCU8Ss0m_XG8MG3bLKkZjH3OucsV5Q@mail.gmail.com>
 <0ba1c66c-b752-2565-3932-64fd0f966fad@brice.xyz>
 <CALD=Yf8zZV8bXqeXJt0jnonjS7h_k0ceMP9d=xYXoKHMWP5B3w@mail.gmail.com>
 <CACCbJKrZZG0T_Avn0Y_YKgN3OWdX-GEtC97BX9JEg7JZuEfZgQ@mail.gmail.com>
Message-ID: <CALD=Yf8A6tg8S80V6xVnRiMhHZZRF7EBB5M=fc9qmXa+oMw5GA@mail.gmail.com>

Hi Peter

You make the very good point, that

> subfunction_1 may be written by someone totally different from the author of
> main_function, and may even be in a different codebase.  For the author of
> subfunction_1, it makes no sense to use the "None" approach instead of
> python's normal default mechanism (since all arguments here are immutables).

Good point. To rephrase, what should we do if we want to use a third
party or legacy function, which begins
===
def fn(a=1, b=2, c=3):
  # function body
===

We can solve this by defining a function decorator. Suppose we have a
function fix_it, whose argument and return value are both functions.
The basic specification of fix_it is that
---
fixed_fn = fix_it(fn)
---
is in practice equivalent to
---
def fixed_fn(a=None, b=None, c=None):
    if a is None: a = 1
    if b is None: b = 2
    if c is None: c = 3
    # function body for fn
    # or if you prefer
    return fn(a, b, c)
---

An aside. We can code fix_it by using
https://docs.python.org/3/library/inspect.html
===
>>> import inspect
>>> def fn(a=1, b=2, c=3): pass
...
>>> str(inspect.signature(fn))
'(a=1, b=2, c=3)'
===

You could also use with new code, like so:
---
@fix_it
def fn(a=1, b=2, c=3):
  # function body
---

I think this helps solve your problem. Is there, Peter, anything else
that would be left to do (except, of course, write the fix_it
function).

Thank you again for your problem and comments.

-- 
Jonathan

From gregory.lielens at gmail.com  Fri Jul 20 09:44:16 2018
From: gregory.lielens at gmail.com (=?UTF-8?Q?Gr=C3=A9gory_Lielens?=)
Date: Fri, 20 Jul 2018 06:44:16 -0700 (PDT)
Subject: [Python-ideas] PEP 505: None-aware operators: operators ?= and
 ?? and OR
In-Reply-To: <6ac81150-57a8-732e-d7c8-f2b3f31bbf1a@gmail.com>
References: <CALD=Yf-0dLkGKN+1gb11nv4s0J4yiySehszZ+rS2BZVciR+LHA@mail.gmail.com>
 <CA+w-FZ8GgTRGm+duzmT=CgVeX_=mpWgUPNzOHH=D-a=W8J7-AA@mail.gmail.com>
 <6ac81150-57a8-732e-d7c8-f2b3f31bbf1a@gmail.com>
Message-ID: <d59fd192-cc54-4c62-a579-8cf9ef3cb9d1@googlegroups.com>

I like it much better than the operator version, if I understand correctly, it's like a watcher for special value, that monitor the stack.
Danger is that it monitor the whole stack, so that final result is dependent on how much you decompose your algorithm into nested fiction calls: all return values will be monitored, and only return values...
But if you refactor a big function into nested functions, and happen to return the special value in the new subfunctions, it will have side effects, that can not be known within the context: it expose internals that used to be hidden, deliberately so...

From rhodri at kynesim.co.uk  Fri Jul 20 10:35:26 2018
From: rhodri at kynesim.co.uk (Rhodri James)
Date: Fri, 20 Jul 2018 15:35:26 +0100
Subject: [Python-ideas] PEP 505: None-aware operators: operators ?= and
 ?? and OR
In-Reply-To: <6ac81150-57a8-732e-d7c8-f2b3f31bbf1a@gmail.com>
References: <CALD=Yf-0dLkGKN+1gb11nv4s0J4yiySehszZ+rS2BZVciR+LHA@mail.gmail.com>
 <CA+w-FZ8GgTRGm+duzmT=CgVeX_=mpWgUPNzOHH=D-a=W8J7-AA@mail.gmail.com>
 <6ac81150-57a8-732e-d7c8-f2b3f31bbf1a@gmail.com>
Message-ID: <f9c45c3a-c4f3-9eec-16cf-14df9194ec6f@kynesim.co.uk>

On 20/07/18 14:06, Michel Desmoulin wrote:
> Proposal a __very__ dumbed down "try" expression
> ====================================================
> 
>      variable = try Expression as not sentinel [else default]

I like this approach.  I think it should play nice with the ternary if, 
though I haven't actually checked the grammar.

My only doubt is how much value there is to having a sentinel other than 
"None".

-- 
Rhodri James *-* Kynesim Ltd

From contact at brice.xyz  Fri Jul 20 10:41:02 2018
From: contact at brice.xyz (Brice Parent)
Date: Fri, 20 Jul 2018 16:41:02 +0200
Subject: [Python-ideas] Idea: Deferred Default Arguments?
In-Reply-To: <CACCbJKrUq_vUYKO8ZzF0PCU8Ss0m_XG8MG3bLKkZjH3OucsV5Q@mail.gmail.com>
References: <CACCbJKrUq_vUYKO8ZzF0PCU8Ss0m_XG8MG3bLKkZjH3OucsV5Q@mail.gmail.com>
Message-ID: <47145352-9562-414c-a071-dca047e00641@brice.xyz>

It might be stupid, but how about solving this problem using the following:

from . import other_func, SomeClass

def my_func(a=other_func.defaults.a, b=other_func.defaults.b, 
c=SomeClass.some_method.defaults.c):
 ??? ...

or
def my_func(a=None, b=None, c=None):? # or use some sentinel value 
instead of None
 ??? if a is None:
 ??????? a = other_func.defaults.a
 ??? if b is None:
 ??????? b = other_func.defaults.b
 ??? if c is None:
 ??????? c = SomeClass.some_method.defaults.c
 ??? ...

or even
def my_func(a=None, b=None, c=None):
 ??? if a is None:
 ??????? a = default(other_func, "a")
 ??? if b is None:
 ??????? b = default(other_func, "b")
 ??? if c is None:
 ??????? c = default(SomeClass.some_method, "c")
 ???? ...

I used *.defaults.* but it might be something else, as well as the 
function I named 'default' which might be anything else.

I prefer the first, as it's both short and easy to read, but I'm not 
sure about the implications about such a thing. And it probably has 
already been proposed for other use cases.
The second and third versions are more verbose, but probably easier to 
implement, specially the third which should already be doable using 
something like

import inspect

def default(function, argument):
 ??? return inspect.signature(function).parameters[argument].default.

From peter.ed.oconnor at gmail.com  Fri Jul 20 10:43:56 2018
From: peter.ed.oconnor at gmail.com (Peter O'Connor)
Date: Fri, 20 Jul 2018 16:43:56 +0200
Subject: [Python-ideas] Idea: Deferred Default Arguments?
In-Reply-To: <CALD=Yf8A6tg8S80V6xVnRiMhHZZRF7EBB5M=fc9qmXa+oMw5GA@mail.gmail.com>
References: <CACCbJKrUq_vUYKO8ZzF0PCU8Ss0m_XG8MG3bLKkZjH3OucsV5Q@mail.gmail.com>
 <0ba1c66c-b752-2565-3932-64fd0f966fad@brice.xyz>
 <CALD=Yf8zZV8bXqeXJt0jnonjS7h_k0ceMP9d=xYXoKHMWP5B3w@mail.gmail.com>
 <CACCbJKrZZG0T_Avn0Y_YKgN3OWdX-GEtC97BX9JEg7JZuEfZgQ@mail.gmail.com>
 <CALD=Yf8A6tg8S80V6xVnRiMhHZZRF7EBB5M=fc9qmXa+oMw5GA@mail.gmail.com>
Message-ID: <CACCbJKpWhfubhFJ6CdOzg1s95K+Eoti2CE7nSqb-JOa7b5pUmw@mail.gmail.com>

Ah, right, the fix_it(fcn) is a nice idea.  It might also be a good idea,
if we're making an external library anyway, to have a "deferred" object to
avoid overloading "None" (which may mean something else than "differ
argument").  I implemented the decorator here
<https://github.com/petered/peters_example_code/blob/master/peters_example_code/deferral.py>,
and it can be used as:

    from deferral import deferrable_args, deferred

    @deferrable_args
    def subfunction_1(a=2, b=3, c=4):
        return a+b*c

    @deferrable_args
    def subfunction_2(d=5, e=6, f=7):
        return d*e+f

    def main_function(a=deferred, b=deferred, c=deferred, d=deferred,
e=deferred, f=deferred):
        return subfunction_1(a=a, b=b, c=c) + subfunction_2(d=d, e=e, f=f)

    assert main_function() == (2+3*4)+(5*6+7)
    assert main_function(a=8) == (8+3*4)+(5*6+7)

I still think it would be nice to have this as a built-in python feature,
for a few reasons:
- When using non-differable functions (say in other codebases), we have to
do a bunch of "func = deferrable_args(func)" at the top of the module (or
we can just do them at runtime, but then we're doing inspection every time,
which is probably slow).
- It adds a layer to the call stack for every deferrable function you're in.
- To avoid annoying errors where you've defined an arg as deferred but
forgot to wrap the function in question.


On Fri, Jul 20, 2018 at 3:39 PM, Jonathan Fine <jfine2358 at gmail.com> wrote:

> Hi Peter
>
> You make the very good point, that
>
> > subfunction_1 may be written by someone totally different from the
> author of
> > main_function, and may even be in a different codebase.  For the author
> of
> > subfunction_1, it makes no sense to use the "None" approach instead of
> > python's normal default mechanism (since all arguments here are
> immutables).
>
> Good point. To rephrase, what should we do if we want to use a third
> party or legacy function, which begins
> ===
> def fn(a=1, b=2, c=3):
>   # function body
> ===
>
> We can solve this by defining a function decorator. Suppose we have a
> function fix_it, whose argument and return value are both functions.
> The basic specification of fix_it is that
> ---
> fixed_fn = fix_it(fn)
> ---
> is in practice equivalent to
> ---
> def fixed_fn(a=None, b=None, c=None):
>     if a is None: a = 1
>     if b is None: b = 2
>     if c is None: c = 3
>     # function body for fn
>     # or if you prefer
>     return fn(a, b, c)
> ---
>
> An aside. We can code fix_it by using
> https://docs.python.org/3/library/inspect.html
> ===
> >>> import inspect
> >>> def fn(a=1, b=2, c=3): pass
> ...
> >>> str(inspect.signature(fn))
> '(a=1, b=2, c=3)'
> ===
>
> You could also use with new code, like so:
> ---
> @fix_it
> def fn(a=1, b=2, c=3):
>   # function body
> ---
>
> I think this helps solve your problem. Is there, Peter, anything else
> that would be left to do (except, of course, write the fix_it
> function).
>
> Thank you again for your problem and comments.
>
> --
> Jonathan
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180720/34c86253/attachment.html>

From jfine2358 at gmail.com  Fri Jul 20 10:57:00 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Fri, 20 Jul 2018 15:57:00 +0100
Subject: [Python-ideas] Idea: Deferred Default Arguments?
In-Reply-To: <CACCbJKpWhfubhFJ6CdOzg1s95K+Eoti2CE7nSqb-JOa7b5pUmw@mail.gmail.com>
References: <CACCbJKrUq_vUYKO8ZzF0PCU8Ss0m_XG8MG3bLKkZjH3OucsV5Q@mail.gmail.com>
 <0ba1c66c-b752-2565-3932-64fd0f966fad@brice.xyz>
 <CALD=Yf8zZV8bXqeXJt0jnonjS7h_k0ceMP9d=xYXoKHMWP5B3w@mail.gmail.com>
 <CACCbJKrZZG0T_Avn0Y_YKgN3OWdX-GEtC97BX9JEg7JZuEfZgQ@mail.gmail.com>
 <CALD=Yf8A6tg8S80V6xVnRiMhHZZRF7EBB5M=fc9qmXa+oMw5GA@mail.gmail.com>
 <CACCbJKpWhfubhFJ6CdOzg1s95K+Eoti2CE7nSqb-JOa7b5pUmw@mail.gmail.com>
Message-ID: <CALD=Yf9zDReH43VDrvsRgN3RV5QHPJJf5gNj0VedEfTBj_eHow@mail.gmail.com>

Hi Peter

You wrote:
On Fri, Jul 20, 2018 at 3:43 PM, Peter O'Connor
<peter.ed.oconnor at gmail.com> wrote:
> Ah, right, the fix_it(fcn) is a nice idea.  It might also be a good idea, if
> we're making an external library anyway, to have a "deferred" object to
> avoid overloading "None" (which may mean something else than "differ
> argument").  I implemented the decorator here
https://github.com/petered/peters_example_code/blob/master/peters_example_code/deferral.py

Oh, well done Peter. Thank you. About 15 lines of code, and the same
again for comments. And all for the good of the community (and
scratching your own itch).

I think we've now got pretty much the right basic ideas for solving
your original problem. And your original problem is fairly widespread.

So what, do you think, are the next steps?

-- 
Jonathan

From steve at pearwood.info  Fri Jul 20 11:41:54 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 21 Jul 2018 01:41:54 +1000
Subject: [Python-ideas] Idea: Deferred Default Arguments?
In-Reply-To: <CACCbJKpWhfubhFJ6CdOzg1s95K+Eoti2CE7nSqb-JOa7b5pUmw@mail.gmail.com>
References: <CACCbJKrUq_vUYKO8ZzF0PCU8Ss0m_XG8MG3bLKkZjH3OucsV5Q@mail.gmail.com>
 <0ba1c66c-b752-2565-3932-64fd0f966fad@brice.xyz>
 <CALD=Yf8zZV8bXqeXJt0jnonjS7h_k0ceMP9d=xYXoKHMWP5B3w@mail.gmail.com>
 <CACCbJKrZZG0T_Avn0Y_YKgN3OWdX-GEtC97BX9JEg7JZuEfZgQ@mail.gmail.com>
 <CALD=Yf8A6tg8S80V6xVnRiMhHZZRF7EBB5M=fc9qmXa+oMw5GA@mail.gmail.com>
 <CACCbJKpWhfubhFJ6CdOzg1s95K+Eoti2CE7nSqb-JOa7b5pUmw@mail.gmail.com>
Message-ID: <20180720154154.GB8744@ando.pearwood.info>

On Fri, Jul 20, 2018 at 04:43:56PM +0200, Peter O'Connor wrote:

> I still think it would be nice to have this as a built-in python feature,
> for a few reasons:
> - When using non-differable functions (say in other codebases), we have to
> do a bunch of "func = deferrable_args(func)" at the top of the module (or
> we can just do them at runtime, but then we're doing inspection every time,
> which is probably slow).
> - It adds a layer to the call stack for every deferrable function you're in.
> - To avoid annoying errors where you've defined an arg as deferred but
> forgot to wrap the function in question.

What makes you think that a built-in deferred feature won't have exactly 
the same issues? Do you have an implementation that doesn't need to do 
intraspection?


-- 
Steve

From ncoghlan at gmail.com  Fri Jul 20 11:53:39 2018
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 21 Jul 2018 01:53:39 +1000
Subject: [Python-ideas] Idea: Deferred Default Arguments?
In-Reply-To: <20180720124516.GA8744@ando.pearwood.info>
References: <CACCbJKrUq_vUYKO8ZzF0PCU8Ss0m_XG8MG3bLKkZjH3OucsV5Q@mail.gmail.com>
 <20180720124516.GA8744@ando.pearwood.info>
Message-ID: <CADiSq7dFmqJTbtKG7YjStsbXvjR7NVhsdt5JZtMXuaP4kmga1A@mail.gmail.com>

On 20 July 2018 at 22:45, Steven D'Aprano <steve at pearwood.info> wrote:
> Perhaps you mean duplicate, or repeat, or copy. But surely they're not
> redefined -- then they would have different values. Being able to
> redefine the defaults in a wrapper function is a feature.
>
> Putting aside the terminology, I think this is a minor annoyance: DRY
> violations when setting default values.

FWIW, I tend to handle this problem the same way I handle other DRY
problems with magic constants: give the default value a name and
either export it directly, or export an API for retrieving it.

If that results in name sprawl ("But now I have 15 defaults to
export!"), then I take it as a hint that I may not be modeling my data
correctly, and am missing a class definition or two somewhere.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From peter.ed.oconnor at gmail.com  Fri Jul 20 11:54:44 2018
From: peter.ed.oconnor at gmail.com (Peter O'Connor)
Date: Fri, 20 Jul 2018 17:54:44 +0200
Subject: [Python-ideas] Idea: Deferred Default Arguments?
In-Reply-To: <20180720154154.GB8744@ando.pearwood.info>
References: <CACCbJKrUq_vUYKO8ZzF0PCU8Ss0m_XG8MG3bLKkZjH3OucsV5Q@mail.gmail.com>
 <0ba1c66c-b752-2565-3932-64fd0f966fad@brice.xyz>
 <CALD=Yf8zZV8bXqeXJt0jnonjS7h_k0ceMP9d=xYXoKHMWP5B3w@mail.gmail.com>
 <CACCbJKrZZG0T_Avn0Y_YKgN3OWdX-GEtC97BX9JEg7JZuEfZgQ@mail.gmail.com>
 <CALD=Yf8A6tg8S80V6xVnRiMhHZZRF7EBB5M=fc9qmXa+oMw5GA@mail.gmail.com>
 <CACCbJKpWhfubhFJ6CdOzg1s95K+Eoti2CE7nSqb-JOa7b5pUmw@mail.gmail.com>
 <20180720154154.GB8744@ando.pearwood.info>
Message-ID: <CACCbJKo9uyhBuhG=a5=D2+s8iiY4gMibKQEgYmT9hGs3yL_nPg@mail.gmail.com>

On Fri, Jul 20, 2018 at 5:41 PM, Steven D'Aprano <steve at pearwood.info>
wrote:
>
>
> What makes you think that a built-in deferred feature won't have exactly
> the same issues? Do you have an implementation that doesn't need to do
> intraspection?


I don't know about these low level things, but I assume it'd be implemented
in C and wouldn't have the same cost as entering a new function in Python.
 I imagine it just being a small modification of the mechanism that Python
already uses to assign values to arguments when a function is called.  Is
that not the case?



On Fri, Jul 20, 2018 at 5:41 PM, Steven D'Aprano <steve at pearwood.info>
wrote:

> On Fri, Jul 20, 2018 at 04:43:56PM +0200, Peter O'Connor wrote:
>
> > I still think it would be nice to have this as a built-in python feature,
> > for a few reasons:
> > - When using non-differable functions (say in other codebases), we have
> to
> > do a bunch of "func = deferrable_args(func)" at the top of the module (or
> > we can just do them at runtime, but then we're doing inspection every
> time,
> > which is probably slow).
> > - It adds a layer to the call stack for every deferrable function you're
> in.
> > - To avoid annoying errors where you've defined an arg as deferred but
> > forgot to wrap the function in question.
>
> What makes you think that a built-in deferred feature won't have exactly
> the same issues? Do you have an implementation that doesn't need to do
> intraspection?
>
>
> --
> Steve
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180720/e11f07b4/attachment.html>

From ncoghlan at gmail.com  Fri Jul 20 12:09:25 2018
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 21 Jul 2018 02:09:25 +1000
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <CALFfu7BFLSSRf7_mVUP3Rh4tAg1ae1cawc9KFF66EiBc2Hfn7g@mail.gmail.com>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>
 <CAPTjJmqKGhNEcrAuzKSzcUW_kvdmPqL9_9CMyOUzcrBs=UdcEg@mail.gmail.com>
 <CAPJVwBkpLw_qRirCFq4PK6n5zLVxc=t=_bf_SZRXLsSnrMVECQ@mail.gmail.com>
 <CAPTjJmrt83c96JQ3PY2oZEm68ZMbG_JBwaJ1XpcyzqRnEcks9A@mail.gmail.com>
 <6a5b228a-200a-b2c7-6067-df2b0022ae42@mrabarnett.plus.com>
 <20180716190735.2908e86d@fsol>
 <CALFfu7BFLSSRf7_mVUP3Rh4tAg1ae1cawc9KFF66EiBc2Hfn7g@mail.gmail.com>
Message-ID: <CADiSq7fu4=f-aS7Ew8bQjuQozxibpvnLJgrP_9ZenzSmuMQjrg@mail.gmail.com>

On 18 July 2018 at 05:35, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> In order to make all this work the missing piece is a mechanism by
> which the decref (#3) happens under the original interpreter.  At the
> moment Emily Morehouse and I are pursuing an approach that extends the
> existing ceval "pending call" machinery currently used for handling
> signals (see Py_AddPendingCall).  The new [*private*] API would work
> the same way but on a per-interpreter basis rather than just the main
> interpreter.  This would allow one interpreter to queue up a decref to
> happen later under another interpreter.
>
> FWIW, this ability to decref an object under a different interpreter
> is a blocker right now for a number of things, including supporting
> buffers in PEP 554 channels.

Aw, I guess the original idea of just doing an active interpreter
context switch in the current thread around the shared object decref
operation didn't work out? That's a shame.

I'd be curious as to the technical details of what actually failed in
that approach, as I would have expected it to at least work, even if
the performance might not have been wonderful. (Although thinking
about it further now given a per-interpreter locking model, I suspect
there could be some wonderful opportunities for cross-interpreter
deadlocks that we didn't consider in our initial design sketch...)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From steve at pearwood.info  Fri Jul 20 13:01:29 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 21 Jul 2018 03:01:29 +1000
Subject: [Python-ideas] Multi-core reference count garbage collection
In-Reply-To: <CALD=Yf-Ht4uDMHkvXp8i+xesw=e9zw4i+nAuRJFMVppNSYrboA@mail.gmail.com>
References: <CALD=Yf-Ht4uDMHkvXp8i+xesw=e9zw4i+nAuRJFMVppNSYrboA@mail.gmail.com>
Message-ID: <20180720170129.GE8744@ando.pearwood.info>

On Thu, Jul 19, 2018 at 07:33:13AM +0100, Jonathan Fine wrote:
> Hi
> 
> Based on other people's work (including in particular talks by Larry
> Hastings) and my own thinking, I've come up with a scheme for multi-core
> reference count garbage collection.

Sorry Jonathan, can you please explain:

- what problem you are solving with this?

- and what's wrong with the existing garbage collector?



-- 
Steve

From steve at pearwood.info  Fri Jul 20 13:04:39 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 21 Jul 2018 03:04:39 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CALD=Yf8j8eGxw5DBh7rgWES6OeA4MXK6nSuEDELvgm-TRmS_YQ@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CALD=Yf8j8eGxw5DBh7rgWES6OeA4MXK6nSuEDELvgm-TRmS_YQ@mail.gmail.com>
Message-ID: <20180720170439.GF8744@ando.pearwood.info>

On Thu, Jul 19, 2018 at 12:57:01PM +0100, Jonathan Fine wrote:

> > Possibly this is exactly the wrong time to propose the next big syntax
> > change, since we currently have nobody to declare on it, but since we're
> > likely to argue for a while anyway it probably can't hurt [...]
> 
> Perhaps "argue" is not the right word here. It sounds too much for or
> against. And it has implications of being heated and angry.

You're new here, aren't you?

*wink*


-- 
Steve

From steve.dower at python.org  Fri Jul 20 13:06:27 2018
From: steve.dower at python.org (Steve Dower)
Date: Fri, 20 Jul 2018 10:06:27 -0700
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180720052234.GX7318@ando.pearwood.info>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <23376.19632.386221.927225@turnbull.sk.tsukuba.ac.jp>
 <20180720052234.GX7318@ando.pearwood.info>
Message-ID: <ffc42af6-e202-33d6-d727-0be4a172d1aa@python.org>

Just for fun, I decided to go through some recently written code by some genuine Python experts (without their permission...) to see what changes would be worth taking. So I went to the sources of our github bots.

Honestly, I only found three places that were worth changing (though I'm now kind of leaning towards ?[] eating LookupError, since that seems much more useful when traversing the result of json.loads()...). I'm also not holding up the third one as the strongest example :)


>From https://github.com/python/miss-islington/blob/master/miss_islington/status_change.py:

async def check_status(event, gh, *args, **kwargs):
    if (
        event.data["commit"].get("committer")
        and event.data["commit"]["committer"]["login"] == "miss-islington"
    ):
        sha = event.data["sha"]
        await check_ci_status_and_approval(gh, sha, leave_comment=True)

After:

async def check_status(event, gh, *args, **kwargs):
    if event.data["commit"].get("committer")?["login"] == "miss-islington":
        sha = event.data["sha"]
        await check_ci_status_and_approval(gh, sha, leave_comment=True)


>From https://github.com/python/bedevere/blob/master/bedevere/__main__.py:

        try:
            print('GH requests remaining:', gh.rate_limit.remaining)
        except AttributeError:
            pass

Assuming you want to continue hiding the message when no value is available:
        if (remaining := gh.rate_limit?.remaining) is not None:
            print('GH requests remaining:', remaining)

Assuming you want the message printed anyway:
        print(f'GH requests remaining: {gh.rate_limit?.remaining ?? "N/A"}')


>From https://github.com/python/bedevere/blob/master/bedevere/news.py (this is the one I'm including for completeness, not because it's the most compelling example I've ever seen):

async def check_news(gh, pull_request, filenames=None):
    if not filenames:
        filenames = await util.filenames_for_PR(gh, pull_request)

After:
async def check_news(gh, pull_request, filenames=None):
    filenames ??= await util.filenames_for_PR(gh, pull_request)


On 19Jul2018 2222, Steven D'Aprano wrote:
> In other words, we ought to be comparing the expressiveness of
> 
>      process(spam ?? something)
> 
> versus:
> 
>     process(something if spam is None else spam)

Agreed, though to make it a more favourable comparison I'd replace "spam" with "spam()?.eggs" and put it in a class/module definition where you don't want temporary names leaking ;)

Cheers,
Steve

From mehaase at gmail.com  Fri Jul 20 13:16:13 2018
From: mehaase at gmail.com (Mark E. Haase)
Date: Fri, 20 Jul 2018 13:16:13 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <5B50DA1E.9030904@brenbarn.net>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAFYqXL8GsTyAHEkrQqSDpv_+bCcJAVv2j-0d_zwNuLLfK7Cuuw@mail.gmail.com>
 <CAPTjJmqPS_wiFTJaJ8YUKJfWsNcCX=h9AQ1EmWFv6Xf9Wes4tA@mail.gmail.com>
 <5B502A80.3080406@canterbury.ac.nz>
 <CAPTjJmquqsqX8XL5RGdvebwHw-vgx15sQU2Qbyg3SKz3WHso2A@mail.gmail.com>
 <5B50DA1E.9030904@brenbarn.net>
Message-ID: <CALb0Rk6ZRpqL-eBurVtDeFAz=CUJ16pAGfZcV_qYrMrHQWGuyQ@mail.gmail.com>

On Thu, Jul 19, 2018 at 2:36 PM Brendan Barnwell <brenbarn at brenbarn.net>
wrote:

>         People keep saying that this null-coalescing behavior is so common
> and
> useful, etc., but that hasn't been my experience at all.  In my
> experience, the desire to shortcut this kind of logic is more often a
> sign of corner-cutting and insufficiently specified data formats, and is
> likely to cause bugs later on.  Eventually it has to actually matter
> whether something is None or not, and these operators just kick that can
> down the road.  In terms of their abstract meaning, they are not
> remotely close to as common or useful as operators like & and |.
>
>
Brendan,

I am sure you didn't intend any offense, but the phrase "corner-cutting" is
pejorative, especially when stated as a generalization and not as a
critique of a specific example. I have used these operators in professional
projects in other languages (i.e. Dart), and I used them because they
seemed like the best tool for the job at hand, not because I was shirking
effort or short on time.

There are accepted PEPs that I don't find useful, e.g. PEP-465 (infix
matrix multiplication operator). It's not because it's a bad PEP; it's just
aimed at a type of programming that I don't do. That PEP had to address
some of the same criticisms that arise from PEP-505: there's no precedent
for that spelling, it's a small niche, and we can already to that in pure
python.[1] But I trust the Python numeric computing community in their
assertion that the @ syntax is valuable to their work.

In the same way that PEP-465 is valuable to a certain type of programming,
None-coalescing (as an abstract concept, not the concrete proposal in
PEP-505) is valuable to another type of programming. Python is often used a
glue language.[2] In my own experience, it's very common to request data
from one system, do some processing on it, and send it to another system.
For example, I might run a SQL query, convert the results to JSON, and
return it in an HTTP response. As a developer, I may not get to choose the
database schema. I may also not get to choose the JSON schema. My job is to
marshal data from one system to another. Consider the following example:

def get_user_json(user_id):
  user = user_table.find_by_id(user_id)
  user_dict = {
    # Username is always non-null in database.
    'username': user['username'],

    # Creation datetime is always non-null in database; must be ISO-8601
string in JSON.
    'created_at': user['created'].isoformat(),

    # Signature can be null in database and null in JSON.
    'signature': user['signature'],

    # Background color can be null in database but must not be null in JSON.
    'background_color': user.get('background_color', 'blue')

    # Login datetime can be null in database if user has never logged in.
Must either
    # be null or an ISO-8601 string in JSON.
    'last_logged_in_at': user['last_logged_in_at'].isoformat() if
user['last_login'] is not None else None,

    # Remaining fields omitted for brevity
  }
  return json.dumps(user_dict)

Python makes a lot of the desired behavior concise to write. For example,
the DBAPI (PEP-249) states that null and None are interchanged when reading
and writing from the database. The stdlib json module also converts between
None and null. This makes a lot of the mappings trivial to express in
Python. But it gets tricky for cases like "last_logged_in_at", where a null
is permitted by the business rules I've been given, but if it's non-null
then I need to call a method on it. As written above, it is over 100
characters long. With safe-navigation operators, it could be made to fit
the line length without loss of clarity:

    'last_logged_in_at': user['last_logged_in_at'] ?. isoformat(),

Many people in this thread prefer to write it as a block:

  if user['last_logged_in_at'] is None:
    user_dict['last_logged_in_at'] = None
  else:
    user_dict['last_logged_in_at'] = user['last_logged_in_at'].isoformat()

I agree that the block is readable, but only because my made-up function is
very short. In real-world glue code, you may have dozens of mappings, and
multiplying by 4 leads to functions that have so many lines of code that
readability is significantly worse, and that highly repetitive code
inevitably leads to typos.

It's not just databases and web services. An early draft of PEP-505
contains additional examples of where None arises in glue code.[4] And it's
not even just glue code: the standard library itself contains around 500
examples of none-coalescing or safe-navigation![5] I certainly don't think
anybody here will claim that stdlib authors have cut hundreds of corners.



[1] https://www.python.org/dev/peps/pep-0465/
[2] https://www.python.org/doc/essays/omg-darpa-mcc-position/
[3] https://www.python.org/dev/peps/pep-0249/
[4]
https://github.com/python/peps/blob/c5270848fe4481947ee951c2a415824b4dcd8a4f/pep-0505.txt#L63
[5] https://www.python.org/dev/peps/pep-0505/#examples
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180720/5b13a24f/attachment-0001.html>

From steve at pearwood.info  Fri Jul 20 13:16:37 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 21 Jul 2018 03:16:37 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CADqi797pajvdV1FAO0Pnb2mnC=Q3O2B4-gYg4Ec_YqtK7KPYew@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <20180719103321.7c193820@fsol>
 <c49bd9b8-1242-fd07-29e8-9039bfaf9199@kynesim.co.uk>
 <5B5126E3.4090708@canterbury.ac.nz>
 <CAPTjJmpjRoVzH2dkWArHz0Jo0h_51Ufp=euY-t5_9HbuTqPd-g@mail.gmail.com>
 <CAMhfa1WU+itm+5z8+OWhj8HC1HU4m1vqX-twYata7J2_kj8GtQ@mail.gmail.com>
 <CADqi797pajvdV1FAO0Pnb2mnC=Q3O2B4-gYg4Ec_YqtK7KPYew@mail.gmail.com>
Message-ID: <20180720171637.GG8744@ando.pearwood.info>

On Thu, Jul 19, 2018 at 08:57:50PM -0400, Michael Selik wrote:

> Try/except also looks decent.
> 
>     try:
>         x = foo['bar'][0]
>     except TypeError:
>         x = 'default'

Consider the case that foo['bar'] is supposed to return either a 
collection (something that can be indexed) or None. But due to a bug, it 
returns a float 1.234. Now the try...except will *wrongly* catch the 
exception from 1.234[0] and return the default.

The problem here is that the try...except block is NOT equivalent to the 
None-aware pattern:

obj = foo['bar']  # avoid time-of-check-to-time-of-use bugs
if obj is None:
    x = 'default'
else:
   x = obj[0]

except in the special case that we can guarantee that foo['bar'] can 
only ever return a collection or None and will never, ever be buggy.



-- 
Steve

From steve at pearwood.info  Fri Jul 20 13:24:41 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 21 Jul 2018 03:24:41 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180719103321.7c193820@fsol>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <20180719103321.7c193820@fsol>
Message-ID: <20180720172441.GH8744@ando.pearwood.info>

On Thu, Jul 19, 2018 at 10:33:21AM +0200, Antoine Pitrou wrote:

>      def insort_right(a, x, lo=0, hi=None):
>          # ...
>          hi = hi else len(a)

I read that as "hi, if it is truthy, else len(a)".

The advantage of ?? as None-aware operator is that it cannot possibly be 
misread as applying to arbitrary falsey objects:

- for those familiar with the equivalent from other languages, 
  the use of ?? to check for nil/null/None is familiar and obvious;

- for those who aren't, the ?? syntax avoids leading them to guess
  the wrong behaviour, as "else" would do.


P.S. I just had to delete 40+ screenfuls of irrelevant quoted text. Come 
on folks, please trim your posts! Don't blame your tools.



-- 
Steve

From steve at pearwood.info  Fri Jul 20 13:31:48 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 21 Jul 2018 03:31:48 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <5B5126E3.4090708@canterbury.ac.nz>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <20180719103321.7c193820@fsol>
 <c49bd9b8-1242-fd07-29e8-9039bfaf9199@kynesim.co.uk>
 <5B5126E3.4090708@canterbury.ac.nz>
Message-ID: <20180720173148.GI8744@ando.pearwood.info>

On Fri, Jul 20, 2018 at 12:03:47PM +1200, Greg Ewing wrote:
> Rhodri James wrote:
> >If anyone can think of a good word for "if it isn't None, otherwise", 
> >I'd be all for it :-)
> 
> I don't think there's any single Engish word that captures
> all of that, so we'd have to invent one.
> 
> Some suggestions:
> 
> inno (If Not None, Otherwise)
> 
> oft  (Or, Failing That)

How about pttpciseano?

("pity that the python community is so edgy about new operators")

*wink*

Remember that adding a new keyword risks breaking code that uses that 
keyword as a variable. Adding a new operator does not.


-- 
Steve

From jfine2358 at gmail.com  Fri Jul 20 13:37:49 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Fri, 20 Jul 2018 18:37:49 +0100
Subject: [Python-ideas] Multi-core reference count garbage collection
In-Reply-To: <20180720170129.GE8744@ando.pearwood.info>
References: <CALD=Yf-Ht4uDMHkvXp8i+xesw=e9zw4i+nAuRJFMVppNSYrboA@mail.gmail.com>
 <20180720170129.GE8744@ando.pearwood.info>
Message-ID: <CALD=Yf8E3hV6LWwxh4sj8Hs6eyLiT9Sk87NNoAoGjxi1P1ScXA@mail.gmail.com>

Hi Steve

You wrote
> Sorry Jonathan, can you please explain:
>
> - what problem you are solving with this?
>
> - and what's wrong with the existing garbage collector?

The problem I'm trying to solve doing multi-core reference counting
garbage collection without a GIL. As far as I know, we're not yet able
to do efficient multi-core garbage collection using Python. If anyone
can tell me that's already been solved, I'd be delighted.

Stephan Houben's post tells us that buffered reference counting
appears in the major reference work
"The Garbage Collection Handbook: The Art of Automatic Memory Management",
by Richard Jones, Antony Hosking, Eliot Moss

So it might be a good idea.

Please let me know if you'd like further information.
-- 
Jonathan

From steve at pearwood.info  Fri Jul 20 13:52:34 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 21 Jul 2018 03:52:34 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <piqm7f$dpt$1@blaine.gmane.org>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <20180719103321.7c193820@fsol>
 <c49bd9b8-1242-fd07-29e8-9039bfaf9199@kynesim.co.uk>
 <piqm7f$dpt$1@blaine.gmane.org>
Message-ID: <20180720175234.GJ8744@ando.pearwood.info>

On Thu, Jul 19, 2018 at 02:47:11PM -0400, Terry Reedy wrote:

> I think most anyone should be able to get that 'A else B' is similar to 
> 'A or B' but must be different.

Knowing that `else` is different to `or` is not the problem. As you say, 
that's obvious. 

The problem is that `A else B` looks like it ought to be the same as 
"else B" in if...else statements and the ternary if operator. That is, 
"if the condition is false", and in this case there is nothing that 
even hints that the condition is "A is None" rather than just A.


    A else B if condition else C else D

Oh my aching head. Things which are different ought to look different, 
not look like what they aren't.

    A ?? B if condition ?? C else D


The precedence is still unclear and will probably benefit from some 
parentheses, but at least here the None-aware operator and the ternary 
if operator look different and can't be confused.


-- 
Steve

From steve at pearwood.info  Fri Jul 20 14:02:06 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 21 Jul 2018 04:02:06 +1000
Subject: [Python-ideas] Idea: Deferred Default Arguments?
In-Reply-To: <CACCbJKo9uyhBuhG=a5=D2+s8iiY4gMibKQEgYmT9hGs3yL_nPg@mail.gmail.com>
References: <CACCbJKrUq_vUYKO8ZzF0PCU8Ss0m_XG8MG3bLKkZjH3OucsV5Q@mail.gmail.com>
 <0ba1c66c-b752-2565-3932-64fd0f966fad@brice.xyz>
 <CALD=Yf8zZV8bXqeXJt0jnonjS7h_k0ceMP9d=xYXoKHMWP5B3w@mail.gmail.com>
 <CACCbJKrZZG0T_Avn0Y_YKgN3OWdX-GEtC97BX9JEg7JZuEfZgQ@mail.gmail.com>
 <CALD=Yf8A6tg8S80V6xVnRiMhHZZRF7EBB5M=fc9qmXa+oMw5GA@mail.gmail.com>
 <CACCbJKpWhfubhFJ6CdOzg1s95K+Eoti2CE7nSqb-JOa7b5pUmw@mail.gmail.com>
 <20180720154154.GB8744@ando.pearwood.info>
 <CACCbJKo9uyhBuhG=a5=D2+s8iiY4gMibKQEgYmT9hGs3yL_nPg@mail.gmail.com>
Message-ID: <20180720180206.GK8744@ando.pearwood.info>

On Fri, Jul 20, 2018 at 05:54:44PM +0200, Peter O'Connor wrote:
> On Fri, Jul 20, 2018 at 5:41 PM, Steven D'Aprano <steve at pearwood.info>
> wrote:
> >
> >
> > What makes you think that a built-in deferred feature won't have exactly
> > the same issues? Do you have an implementation that doesn't need to do
> > intraspection?
> 
> 
> I don't know about these low level things, but I assume it'd be implemented
> in C and wouldn't have the same cost as entering a new function in Python.
>  I imagine it just being a small modification of the mechanism that Python
> already uses to assign values to arguments when a function is called.  Is
> that not the case?

I don't know. That's why I'm asking why you think it wouldn't.


-- 
Steve

From brenbarn at brenbarn.net  Fri Jul 20 14:19:16 2018
From: brenbarn at brenbarn.net (Brendan Barnwell)
Date: Fri, 20 Jul 2018 11:19:16 -0700
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180720175234.GJ8744@ando.pearwood.info>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <20180719103321.7c193820@fsol>
 <c49bd9b8-1242-fd07-29e8-9039bfaf9199@kynesim.co.uk>
 <piqm7f$dpt$1@blaine.gmane.org> <20180720175234.GJ8744@ando.pearwood.info>
Message-ID: <5B5227A4.5020508@brenbarn.net>

On 2018-07-20 10:52, Steven D'Aprano wrote:
> The problem is that `A else B` looks like it ought to be the same as
> "else B" in if...else statements and the ternary if operator. That is,
> "if the condition is false", and in this case there is nothing that
> even hints that the condition is "A is None" rather than just A.

	Personally I consider this a problem with the whole null-coalescing 
idea.  Although it's true that the proposed null-coalescing operators 
don't look like existing syntax, it's also true that there's nothing 
about them that suggests to the uninitiated that they have anything to 
do with comparing to None.  In this situation I lean toward "explicit is 
better than implicit" --- if you want to compare against None, you 
should do so explicitly --- and "special cases aren't special enough to 
break the rules" --- that is, None is not special enough to warrant the 
creation of multiple new operators solely to compare things against this 
specific value.

-- 
Brendan Barnwell
"Do not follow where the path may lead.  Go, instead, where there is no 
path, and leave a trail."
    --author unknown

From steve at pearwood.info  Fri Jul 20 14:20:38 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 21 Jul 2018 04:20:38 +1000
Subject: [Python-ideas] Multi-core reference count garbage collection
In-Reply-To: <CALD=Yf8E3hV6LWwxh4sj8Hs6eyLiT9Sk87NNoAoGjxi1P1ScXA@mail.gmail.com>
References: <CALD=Yf-Ht4uDMHkvXp8i+xesw=e9zw4i+nAuRJFMVppNSYrboA@mail.gmail.com>
 <20180720170129.GE8744@ando.pearwood.info>
 <CALD=Yf8E3hV6LWwxh4sj8Hs6eyLiT9Sk87NNoAoGjxi1P1ScXA@mail.gmail.com>
Message-ID: <20180720182037.GL8744@ando.pearwood.info>

On Fri, Jul 20, 2018 at 06:37:49PM +0100, Jonathan Fine wrote:
> Hi Steve
> 
> You wrote
> > Sorry Jonathan, can you please explain:
> >
> > - what problem you are solving with this?
> >
> > - and what's wrong with the existing garbage collector?
> 
> The problem I'm trying to solve doing multi-core reference counting
> garbage collection without a GIL. 

Perhaps I'm being a bit slow tonight, but that sounds like you're just 
repeating the solution: "multi-core reference counting garbage 
collection without a GIL". What problem does that solve?

If the quote-unquote "problem" is "get rid of the GIL", there are 
already at least two implementations of Python without a GIL (IronPython 
and Jython). Do they solve your problem?

If there's some other problem you are trying to solve, you should 
explain what it is.


> As far as I know, we're not yet able
> to do efficient multi-core garbage collection using Python.

Why would that be an advantage?

My understanding is that reference counting is both deterministic and 
immediate. Shifting the reference counting into another thread so that 
it becomes non-deterministic and potentially delayed doesn't sound like 
an advantage to my naive understanding.

But then I've never been one to think that threads are the only, or even 
the best, form of concurrency. I've never understood why so many people 
care about the GIL.

(Perhaps I'm jaundiced from dealing with too many people who honestly 
believe that the GIL is why their single-threaded code runs slowly. My 
favourite was the fellow I knew who was so convinced that Python was 
slow because of the GIL that he moved to Ruby, which also has a GIL and 
is slower than Python.)


-- 
Steve

From steve.dower at python.org  Fri Jul 20 15:05:54 2018
From: steve.dower at python.org (Steve Dower)
Date: Fri, 20 Jul 2018 12:05:54 -0700
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <5B5227A4.5020508@brenbarn.net>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <20180719103321.7c193820@fsol>
 <c49bd9b8-1242-fd07-29e8-9039bfaf9199@kynesim.co.uk>
 <piqm7f$dpt$1@blaine.gmane.org> <20180720175234.GJ8744@ando.pearwood.info>
 <5B5227A4.5020508@brenbarn.net>
Message-ID: <89d39a36-d678-4099-f2fd-7ec003cadf37@python.org>

On 20Jul2018 1119, Brendan Barnwell wrote:
> In this situation I lean toward "explicit is 
> better than implicit" --- if you want to compare against None, you 
> should do so explicitly --- and "special cases aren't special enough to 
> break the rules" --- that is, None is not special enough to warrant the 
> creation of multiple new operators solely to compare things against this 
> specific value.

"The rules" declare that None is special - it's the one and only value 
that represents "no value". So is giving it special meaning here 
breaking the rules or following them? (See also the ~50% of the PEP 
dedicated to this subject, and also consider proposing a non-special 
result for "??? if has_no_value(value) else value" in the 'True' case.)

Cheers,
Steve

From jfine2358 at gmail.com  Fri Jul 20 16:44:39 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Fri, 20 Jul 2018 21:44:39 +0100
Subject: [Python-ideas] Multi-core reference count garbage collection
In-Reply-To: <20180720182037.GL8744@ando.pearwood.info>
References: <CALD=Yf-Ht4uDMHkvXp8i+xesw=e9zw4i+nAuRJFMVppNSYrboA@mail.gmail.com>
 <20180720170129.GE8744@ando.pearwood.info>
 <CALD=Yf8E3hV6LWwxh4sj8Hs6eyLiT9Sk87NNoAoGjxi1P1ScXA@mail.gmail.com>
 <20180720182037.GL8744@ando.pearwood.info>
Message-ID: <CALD=Yf8S2THNqFNvOyRfTjX=pUBY09orSYFXGExKrVnJKSV3Aw@mail.gmail.com>

Hi Steve

You wrote:
> My understanding is that reference counting is both deterministic and
> immediate. Shifting the reference counting into another thread so that
> it becomes non-deterministic and potentially delayed doesn't sound like
> an advantage to my naive understanding.

The choice is not as simple as that. Here's an example.

Suppose you and your friend want to sort a deck of cards. Here's an algorithm.

* Divide the deck into two
* Each person sorts their half deck
* One of you does a merge of the two half decks

This algorithm is non-deterministic. But for a large deck it's quicker
than any deterministic algorithm.

I hope this helps.

-- 
Jonathan

From rosuav at gmail.com  Fri Jul 20 16:47:32 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 21 Jul 2018 06:47:32 +1000
Subject: [Python-ideas] Multi-core reference count garbage collection
In-Reply-To: <CALD=Yf8S2THNqFNvOyRfTjX=pUBY09orSYFXGExKrVnJKSV3Aw@mail.gmail.com>
References: <CALD=Yf-Ht4uDMHkvXp8i+xesw=e9zw4i+nAuRJFMVppNSYrboA@mail.gmail.com>
 <20180720170129.GE8744@ando.pearwood.info>
 <CALD=Yf8E3hV6LWwxh4sj8Hs6eyLiT9Sk87NNoAoGjxi1P1ScXA@mail.gmail.com>
 <20180720182037.GL8744@ando.pearwood.info>
 <CALD=Yf8S2THNqFNvOyRfTjX=pUBY09orSYFXGExKrVnJKSV3Aw@mail.gmail.com>
Message-ID: <CAPTjJmpFPLMHAMOiR=1iGoSz0RZ9PYPhKj_d+ZjKMKfoZP3Z_A@mail.gmail.com>

On Sat, Jul 21, 2018 at 6:44 AM, Jonathan Fine <jfine2358 at gmail.com> wrote:
> Hi Steve
>
> You wrote:
>> My understanding is that reference counting is both deterministic and
>> immediate. Shifting the reference counting into another thread so that
>> it becomes non-deterministic and potentially delayed doesn't sound like
>> an advantage to my naive understanding.
>
> The choice is not as simple as that. Here's an example.
>
> Suppose you and your friend want to sort a deck of cards. Here's an algorithm.
>
> * Divide the deck into two
> * Each person sorts their half deck
> * One of you does a merge of the two half decks
>
> This algorithm is non-deterministic. But for a large deck it's quicker
> than any deterministic algorithm.

Divide the deck into "first half" and "second half". Then sort the
half decks using a stable algorithm. Finally, merge the decks,
favouring cards from the first half. Voila! You now have a fully
deterministic and stable sort.

ChrisA

From brett at python.org  Fri Jul 20 20:29:37 2018
From: brett at python.org (Brett Cannon)
Date: Fri, 20 Jul 2018 17:29:37 -0700
Subject: [Python-ideas] if we were to ever shorten `A if A else B` (was: PEP
 505: None-aware operators)
In-Reply-To: <dc623fec-26be-4b74-f13b-63d74eb3824a@python.org>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAFYqXL8GsTyAHEkrQqSDpv_+bCcJAVv2j-0d_zwNuLLfK7Cuuw@mail.gmail.com>
 <CAPTjJmqPS_wiFTJaJ8YUKJfWsNcCX=h9AQ1EmWFv6Xf9Wes4tA@mail.gmail.com>
 <5B502A80.3080406@canterbury.ac.nz>
 <CAPTjJmquqsqX8XL5RGdvebwHw-vgx15sQU2Qbyg3SKz3WHso2A@mail.gmail.com>
 <5B50DA1E.9030904@brenbarn.net>
 <CAOOa=pPU2-KvWJotvO3D7sJ=q-uJGbZBw3khmHdReHZ-m2zuwA@mail.gmail.com>
 <CAPTjJmo_7H8ppV6JnZzQu5B6DdYxympenws8YcaWyYSCqAvhLQ@mail.gmail.com>
 <dc623fec-26be-4b74-f13b-63d74eb3824a@python.org>
Message-ID: <CAP1=2W5YtcBHtScMXi8oL_OWUmzqFhbMGp_94kmqb4OKYaDZYw@mail.gmail.com>

On Thu, 19 Jul 2018 at 14:29 Steve Dower <steve.dower at python.org> wrote:

> [SNIP]
>
> * "We could have 'else', we don't need '??'"
>
> This is the "a else 'default'" rather than "a ?? 'default'" proposal,
> which I do like the look of, but I think it will simultaneously mess
> with operator precedence and also force me to search for the 'if' that
> we actually need to be comparing "(a else 'default')" vs. "a ??
> 'default'"::
>
>      x = a if b else c else d
>      x = a if (b else c) else d
>      x = a if b else (c else d)
>

The searching for the 'if' also throws me with this proposal. I think
someone else proposed `A unless B` but I always prefer knowing upfront what
I'm working with, not later by having to read farther along.

Personally, if the `A?? B` is the part people like but hate the syntax then
my vote would go to `A otherwise B` since it's unambiguous, the case you
care about the state  of comes first, and it doesn't trip your brain up
looking for 'if'. :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180720/389381f0/attachment.html>

From steve at pearwood.info  Fri Jul 20 21:23:26 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 21 Jul 2018 11:23:26 +1000
Subject: [Python-ideas] Multi-core reference count garbage collection
In-Reply-To: <CALD=Yf8S2THNqFNvOyRfTjX=pUBY09orSYFXGExKrVnJKSV3Aw@mail.gmail.com>
References: <CALD=Yf-Ht4uDMHkvXp8i+xesw=e9zw4i+nAuRJFMVppNSYrboA@mail.gmail.com>
 <20180720170129.GE8744@ando.pearwood.info>
 <CALD=Yf8E3hV6LWwxh4sj8Hs6eyLiT9Sk87NNoAoGjxi1P1ScXA@mail.gmail.com>
 <20180720182037.GL8744@ando.pearwood.info>
 <CALD=Yf8S2THNqFNvOyRfTjX=pUBY09orSYFXGExKrVnJKSV3Aw@mail.gmail.com>
Message-ID: <20180721012324.GM8744@ando.pearwood.info>

On Fri, Jul 20, 2018 at 09:44:39PM +0100, Jonathan Fine wrote:
> Hi Steve
> 
> You wrote:
> > My understanding is that reference counting is both deterministic and
> > immediate. Shifting the reference counting into another thread so that
> > it becomes non-deterministic and potentially delayed doesn't sound like
> > an advantage to my naive understanding.
> 
> The choice is not as simple as that. Here's an example.
> 
> Suppose you and your friend want to sort a deck of cards. Here's an algorithm.

Jonathan, you keep avoiding my direct questions. What problem are you 
trying to solve?

Its okay if there is no immediate problem, that you're just exploring 
alternative garbage collection strategies. Or if you're trying to remove 
the GIL. Or you think that this will make print faster. (Probably not 
the last one *wink*)

And please don't patronise me -- I might not be an expert on garbage 
collection but I know that parallelising certain tasks can make them run 
faster. Your sorting example is irrelevant to understanding this 
reference counting proposal.

Help us understand where you are coming from in this discussion.


> * Divide the deck into two
> * Each person sorts their half deck
> * One of you does a merge of the two half decks
> 
> This algorithm is non-deterministic.

No in any important way it isn't. Although the sub-sorting steps may run 
in any order, the final merge doesn't occur until they are both 
completed. There are no race conditions where the merge happens before 
the sort, for example (unless your implementation is *really* bad). 
Which means from the perspective of the outsider the overall sort is 
deterministic: you start with an unsorted list, call sort, and when the 
sort() function returns, you have a sorted list. The non-determinism is 
no more important than the random partition in Quicksort.

But in the case of reference counting, I understand that multithreaded 
reference counting can suffer from race conditions when incrementing or 
decrementing:

http://flyingfrogblog.blogspot.com/2013/09/how-do-reference-counting-and-tracing.html

That would be a bad thing.

But as I said, my view of multi-threaded reference counting is very 
naive, and I'm no expert. I'm less interested in the technical details 
of how and whether it works as to *why* you are interested in trying it.

Hence my repeated requests for you to explain what problem(s) (if any) 
you are hoping to solve.


-- 
Steve

From jfine2358 at gmail.com  Sat Jul 21 03:02:38 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Sat, 21 Jul 2018 08:02:38 +0100
Subject: [Python-ideas] Multi-core reference count garbage collection
In-Reply-To: <CAPTjJmpFPLMHAMOiR=1iGoSz0RZ9PYPhKj_d+ZjKMKfoZP3Z_A@mail.gmail.com>
References: <CALD=Yf-Ht4uDMHkvXp8i+xesw=e9zw4i+nAuRJFMVppNSYrboA@mail.gmail.com>
 <20180720170129.GE8744@ando.pearwood.info>
 <CALD=Yf8E3hV6LWwxh4sj8Hs6eyLiT9Sk87NNoAoGjxi1P1ScXA@mail.gmail.com>
 <20180720182037.GL8744@ando.pearwood.info>
 <CALD=Yf8S2THNqFNvOyRfTjX=pUBY09orSYFXGExKrVnJKSV3Aw@mail.gmail.com>
 <CAPTjJmpFPLMHAMOiR=1iGoSz0RZ9PYPhKj_d+ZjKMKfoZP3Z_A@mail.gmail.com>
Message-ID: <CALD=Yf91_CrWyNm5LJzJeKcPtTcMTVQRos4D083wsuUfa_k3Rw@mail.gmail.com>

Hi Chris

Thank you for your message about two processes together sorting a deck of cards.

My example was in response to a comment from Steve D'Aprano. He
understood in the way it was intended, which was the algorithm is in
execution non-deterministic, but that the outcome is deterministic.
Steve, in a message after yours, explains this point. And so I won't
repeat what would be essentially the same explanation.

Once again, I thank you (and Steve) for their contributions.

-- 
Jonathan

From jfine2358 at gmail.com  Sat Jul 21 03:54:35 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Sat, 21 Jul 2018 08:54:35 +0100
Subject: [Python-ideas] Multi-core reference count garbage collection
In-Reply-To: <20180721012324.GM8744@ando.pearwood.info>
References: <CALD=Yf-Ht4uDMHkvXp8i+xesw=e9zw4i+nAuRJFMVppNSYrboA@mail.gmail.com>
 <20180720170129.GE8744@ando.pearwood.info>
 <CALD=Yf8E3hV6LWwxh4sj8Hs6eyLiT9Sk87NNoAoGjxi1P1ScXA@mail.gmail.com>
 <20180720182037.GL8744@ando.pearwood.info>
 <CALD=Yf8S2THNqFNvOyRfTjX=pUBY09orSYFXGExKrVnJKSV3Aw@mail.gmail.com>
 <20180721012324.GM8744@ando.pearwood.info>
Message-ID: <CALD=Yf_bbd0b=tSqD+EJxV7iHjoSHGrJatmKG9tPPVV0O5UamQ@mail.gmail.com>

Hi Steve

Thank you for your message. I think my response below allows us to go move
forward.

WHAT'S THE PROBLEM
You asked:
> What problem are you trying to solve?
> Its okay if there is no immediate problem, that you're just exploring
> alternative garbage collection strategies. Or if you're trying to remove
> the GIL.

I'm exploring. I hope we'll find something that helps CPython run faster on
multi-core machines, at least for some users. I hope this helps you
understand where I'm coming from.

Please, in this thread, can we confine ourselves to getting a better
understanding of multi-core reference count garbage collection. That is for
me already hard enough, in isolation, without mixing in other issues.

RACE CONDITION
You wrote
> multithreaded reference counting can suffer from race conditions
> when incrementing or decrementing
In support you referenced (the very useful)
>
http://flyingfrogblog.blogspot.com/2013/09/how-do-reference-counting-and-tracing.html
The relevant statement on that page is
> multithreaded reference counting is non-deterministic because increments
and decrements race.

According to https://en.wikipedia.org/wiki/Race_condition
---
A race condition or race hazard is the behavior of an electronics,
software, or other system where the output is dependent on the sequence or
timing of other uncontrollable events. It becomes a bug when events do not
happen in the order the programmer intended.
---

I like this, but I would change the last sentence to read
> It becomes a bug when the outputs are incorrect, for example are in a
wrong order.
Sometimes, eg iterating over a set, there is no intended order.

DEALING WITH THE RACE CONDITION
There is a race condition (or hazard) in buffered reference counting
garbage collection. You wrote that this "would be a bad thing". I hope
that, at the expense of some complexity in the code, the potential for a
bug arising from race hazard can be avoided. Certainly, a bug in garbage
collection is a bad thing.

I was aware of this race hazard when I started this discussion thread. In
my first post of yesterday I made "a start on the hard part" of "doing the
right thing". I also then said
> The remainder of the hard part is to get hold of adjusted reference
> counts, without having to lock the worker threads.

GOING FORWARD
I hope in the next day or two to make a start on the remainder of the hard
part. Your link to flyingfrog is valuable. I hope you'll contribute again.

Finally, you might help you to know that I'm trying in my exploration to
use top-down or stepwise design.

-- 
Jonathan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180721/4b5a2bf6/attachment.html>

From barry at barrys-emacs.org  Sat Jul 21 05:03:32 2018
From: barry at barrys-emacs.org (Barry Scott)
Date: Sat, 21 Jul 2018 10:03:32 +0100
Subject: [Python-ideas] Multi-core reference count garbage collection
In-Reply-To: <CALD=Yf_bbd0b=tSqD+EJxV7iHjoSHGrJatmKG9tPPVV0O5UamQ@mail.gmail.com>
References: <CALD=Yf-Ht4uDMHkvXp8i+xesw=e9zw4i+nAuRJFMVppNSYrboA@mail.gmail.com>
 <20180720170129.GE8744@ando.pearwood.info>
 <CALD=Yf8E3hV6LWwxh4sj8Hs6eyLiT9Sk87NNoAoGjxi1P1ScXA@mail.gmail.com>
 <20180720182037.GL8744@ando.pearwood.info>
 <CALD=Yf8S2THNqFNvOyRfTjX=pUBY09orSYFXGExKrVnJKSV3Aw@mail.gmail.com>
 <20180721012324.GM8744@ando.pearwood.info>
 <CALD=Yf_bbd0b=tSqD+EJxV7iHjoSHGrJatmKG9tPPVV0O5UamQ@mail.gmail.com>
Message-ID: <E25755DA-489B-4AE9-9A83-0114215F0354@barrys-emacs.org>


> On 21 Jul 2018, at 08:54, Jonathan Fine <jfine2358 at gmail.com> wrote:
> 
> Hi Steve
> 
> Thank you for your message. I think my response below allows us to go move forward.
> 
> WHAT'S THE PROBLEM
> You asked:
> > What problem are you trying to solve?
> > Its okay if there is no immediate problem, that you're just exploring
> > alternative garbage collection strategies. Or if you're trying to remove
> > the GIL.
> 
> I'm exploring. I hope we'll find something that helps CPython run faster on multi-core machines, at least for some users. I hope this helps you understand where I'm coming from.
> 

I think that the problem that needs a solution is providing each thread with knowledge
that is safe to update state.

The questions that any solution needs to answer is then:
1) How does the solution provide each thread with the knowledge that it is safe to mutate state?
2) How does the solution manage the life time of the objects?

For 1 today:

In python all state is shared between all threads. One lock, the GIL, provides the code with
the knowledge that it is safe to mutate state.

For 2 today:

The reference counts track the number of uses of an object.
In the simple case when the ref count reaches 0 run the __del__ protocol.

In the cases where circular reference patterns prevents detecting that an object
is no longer needed via reference counts the exist python garbage collector
figures out that such objects can be deleted.

I'd note that the reference counts are just one piece of state of many pieces of state in an object.

As an example of a idea that is looking directly at these questions is the work on using a sub-interpreter for
each thread is interesting. It is aiming to solve (1) by using a per/interpreter lock, which will allow
each thread to run at full speed. I'm following with interest how the life time management will work.

In multi processing (1) is solved by no longer sharing object state.

> Please, in this thread, can we confine ourselves to getting a better understanding of multi-core reference count garbage collection. That is for me already hard enough, in isolation, without mixing in other issues.

I'd argue that the ref counts are not interesting at all, only a side effect of one possible solution to the object life time problem.

Barry




From barry at barrys-emacs.org  Sat Jul 21 05:38:44 2018
From: barry at barrys-emacs.org (Barry Scott)
Date: Sat, 21 Jul 2018 10:38:44 +0100
Subject: [Python-ideas] The future of Python parallelism. The GIL.
 Subinterpreters. Actors.
In-Reply-To: <CALFfu7BFLSSRf7_mVUP3Rh4tAg1ae1cawc9KFF66EiBc2Hfn7g@mail.gmail.com>
References: <1534ef29-f3c5-e629-afe2-9fe6601aa5cc@gmail.com>
 <CAPJVwBkAZWM=aKO852sHXvL7jyBB+LaT=DdBV4M0d11agszGcg@mail.gmail.com>
 <CAPTjJmqKGhNEcrAuzKSzcUW_kvdmPqL9_9CMyOUzcrBs=UdcEg@mail.gmail.com>
 <CAPJVwBkpLw_qRirCFq4PK6n5zLVxc=t=_bf_SZRXLsSnrMVECQ@mail.gmail.com>
 <CAPTjJmrt83c96JQ3PY2oZEm68ZMbG_JBwaJ1XpcyzqRnEcks9A@mail.gmail.com>
 <6a5b228a-200a-b2c7-6067-df2b0022ae42@mrabarnett.plus.com>
 <20180716190735.2908e86d@fsol>
 <CALFfu7BFLSSRf7_mVUP3Rh4tAg1ae1cawc9KFF66EiBc2Hfn7g@mail.gmail.com>
Message-ID: <6A3CFE9A-8FCF-40E3-8A00-074C80F6424B@barrys-emacs.org>

Eric,

> On 17 Jul 2018, at 20:35, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> 
> With this in mind, here's how I'm approaching the problem:
> 
> 1. interp A "shares" an object with interp B (e.g. through a channel)
>    * the object is incref'ed under A before it is sent to B
> 2. the object is wrapped in a proxy owned by B
>    * the proxy may not make C-API calls that would mutate the object
> or even cause an incref/decref
> 3. when the proxy is GC'd, the original object is decref'ed
>    * the decref must happen in a thread in which A is running

How does the proxy at the same time make the object accessible and prevent mutation?

Would it help if there was an explicit owning thread for each object?

I'm thinking that you can do a fast check that the object belongs to the current thread and can use that knowledge to avoid locking.
If the object is owned by another thread acquire the GIL in the traditional way and mutating the state will be safe.

The "sharing" process can ensure that until an explicit "unsharing" the object remains safe to test in all threads that share an object,
avoiding the need for the special processor instructions.

Barry

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180721/2e92e6d2/attachment-0001.html>

From jfine2358 at gmail.com  Sat Jul 21 06:38:51 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Sat, 21 Jul 2018 11:38:51 +0100
Subject: [Python-ideas] Multi-core reference count garbage collection
In-Reply-To: <E25755DA-489B-4AE9-9A83-0114215F0354@barrys-emacs.org>
References: <CALD=Yf-Ht4uDMHkvXp8i+xesw=e9zw4i+nAuRJFMVppNSYrboA@mail.gmail.com>
 <20180720170129.GE8744@ando.pearwood.info>
 <CALD=Yf8E3hV6LWwxh4sj8Hs6eyLiT9Sk87NNoAoGjxi1P1ScXA@mail.gmail.com>
 <20180720182037.GL8744@ando.pearwood.info>
 <CALD=Yf8S2THNqFNvOyRfTjX=pUBY09orSYFXGExKrVnJKSV3Aw@mail.gmail.com>
 <20180721012324.GM8744@ando.pearwood.info>
 <CALD=Yf_bbd0b=tSqD+EJxV7iHjoSHGrJatmKG9tPPVV0O5UamQ@mail.gmail.com>
 <E25755DA-489B-4AE9-9A83-0114215F0354@barrys-emacs.org>
Message-ID: <CALD=Yf_B73DeWPy_7qv3jwT2iJkmz=Vju0qDKFpdsHpZ0rtiiA@mail.gmail.com>

Hi Barry

We've met before. Nice to meet you again, this time electronically.

You suggested that is a different problem that needs a solution. To help
maintain focus, I won't respond to that now.

You wrote:
> I'd argue that the ref counts are not interesting at all, only a
> side effect of one possible solution to the object life time problem.

I'm happy for you to regard multi-core reference counting (MCRC) as a toy
problem, which won't become part of useful software.

We can learn a lot by understanding a good toy. Einstein was fascinated by
the compass, and the spinning top is surprisingly deep.

https://www.johnshepler.com/articles/einstein.html
https://www.einsteinscompass.com/
https://en.wikipedia.org/wiki/Gyrocompass

Of course, I hope you won't mind if MCRC turns out to useful.

-- 
Jonathan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180721/0bb09efe/attachment.html>

From steve at pearwood.info  Sat Jul 21 09:03:36 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 21 Jul 2018 23:03:36 +1000
Subject: [Python-ideas] Multi-core reference count garbage collection
In-Reply-To: <CALuYSZXFx+nu8DxC8s8q-kddSzEsMmmd9tMRBDL9-AVb=9wu5A@mail.gmail.com>
References: <CALD=Yf-Ht4uDMHkvXp8i+xesw=e9zw4i+nAuRJFMVppNSYrboA@mail.gmail.com>
 <20180720170129.GE8744@ando.pearwood.info>
 <CALD=Yf8E3hV6LWwxh4sj8Hs6eyLiT9Sk87NNoAoGjxi1P1ScXA@mail.gmail.com>
 <20180720182037.GL8744@ando.pearwood.info>
 <CALD=Yf8S2THNqFNvOyRfTjX=pUBY09orSYFXGExKrVnJKSV3Aw@mail.gmail.com>
 <20180721012324.GM8744@ando.pearwood.info>
 <CALuYSZXFx+nu8DxC8s8q-kddSzEsMmmd9tMRBDL9-AVb=9wu5A@mail.gmail.com>
Message-ID: <20180721130335.GN8744@ando.pearwood.info>

On Sat, Jul 21, 2018 at 11:05:43AM +0100, Daniel Moisset wrote:

[snip interesting and informative discussion, thank you]

> @Steven D'Aprano: you mentioned soemthign about race conditions but I don't
> think this algorithm has any (the article you linked just said that doing
> refcounting in the traditional way and without synchronization in a multi
> core environment has race conditions, which is not what is being discussed
> here). Could you expand on this?

Certainly not. I already said that my view of this was very naive.

Now that I have a better understanding that Jonathan doesn't have a 
specific issue to solve, other than "improved performance through better 
ref counting", I'll most likely just sit back and lurk.



-- 
Steve

From levkivskyi at gmail.com  Sat Jul 21 13:25:06 2018
From: levkivskyi at gmail.com (Ivan Levkivskyi)
Date: Sat, 21 Jul 2018 18:25:06 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAMhfa1WU+itm+5z8+OWhj8HC1HU4m1vqX-twYata7J2_kj8GtQ@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <20180719103321.7c193820@fsol>
 <c49bd9b8-1242-fd07-29e8-9039bfaf9199@kynesim.co.uk>
 <5B5126E3.4090708@canterbury.ac.nz>
 <CAPTjJmpjRoVzH2dkWArHz0Jo0h_51Ufp=euY-t5_9HbuTqPd-g@mail.gmail.com>
 <CAMhfa1WU+itm+5z8+OWhj8HC1HU4m1vqX-twYata7J2_kj8GtQ@mail.gmail.com>
Message-ID: <CAOMjWk=-Gcy=37-9V8TtSafBm1XQwM_=q9HRVjNUf3Rud=rhLw@mail.gmail.com>

I think I am with Michael here. I like the parallel between `??` and `or`,
we don't have `or=`, so `??=` is also not needed.

Although I understand a parallel between `obj.attr` and `obj['attr']`, I
think there is an additional point (in addition to two valid points by
Michael) why I don't like `?[`: in many situations
in my experience optional container attributes were results of suboptimal
APIs, so I would not encourage this pattern.

FWIW, I am rather -0 on adding all proposed operators, but I would be +1 on
adding just the two essential ones: ?? and ?.

--
Ivan



On 20 July 2018 at 01:40, Michael Lee <michael.lee.0x2a at gmail.com> wrote:

> Hi -- I'm a new voice here, though I've been lurking for a while now.
>
> How do people feel about removing "??=" and "foo?[bar]" from the PEP and
> sticking with just "foo?.bar" and "foo ?? bar"?
>
> Reasons for not implementing "??=":
>
>    1. I think the other operators are useful in that they can be chained
>    together to shrink many different lines of code. In contrast, ??= can
>    shrink at most one line of code: we get to remove a single 'if foo is
>    None'. If the rest of the PEP is implemented, we can also just express
>    this by doing "foo = foo ?? some_expr" which I think is still
>    relatively concise.
>
>    2. None of the other short-circuiting operators have an augmented
>    assignment form -- e.g. we can do "foo = foo or bar", but there's no
>    such thing as "foo or= bar". I don't really see why ?? in particular
>    deserves to have an augmented assignment form.
>
>    3. We already have two different kinds of assignment operators with
>    the inclusion of PEP 572 -- in the interests of keeping Python as simple as
>    possible, I think it'd be a good idea not to add a third one.
>
>
> Reasons for not implementing "foo?[bar]":
>
>    1. It seems like "foo?[bar]" could be easily confused with "foo??[bar]".
>    I don't think it's immediately obvious to a newcomer which spelling
>    corresponds to which usage, and getting the two mixed up seems like the
>    sort of thing that could cause all sorts of subtle bugs.
>
>    2. We can already easily get the same functionality using standard
>    Python. E.g., instead of doing foo?["bar"]?[0]?["baz"], we could do lookup(foo,
>    "bar", 0, "baz") where lookup is a function that looks roughly like
>    this:
>
>        def lookup(item, *parts):
>            for part in parts:
>                if item is None:
>                    return None
>                item = item[parts]
>            return item
>
>    Of course, we could probably create the same sort of function to
>    replace foo?.bar (albeit less ergonomically), but unlike foo?[bar],
>    there's no possibility for confusion: doing foo??.bar will never be a
>    valid Python expression.
>
>    3. One counter-argument against removing foo?[bar] is that it would
>    make expression that need both safe index and attribute lookups look weird
>    -- you'd sometimes be using the "lookup" function described above (or
>    something similar) and sometimes using ".?". However, I'd argue that these
>    sorts of scenarios are relatively rare in practice, and that if you really
>    need to deal with a bunch of code that requires you to use both forms of
>    safe navigation, your code is likely already becoming pretty unreadable and
>    should be rewritten -- maybe split up into multiple lines or something, or
>    maybe just redesigned from scratch so you don't need to constantly manage a
>    mess of Nones.
>
>
> More broadly, I think I agree with the sentiment some other people have
> that Python has acquired a lot of new features in a relatively short period
> of time, and that it would be nice to have some cooldown to let tooling and
> other implementations catch up. In that regard, I'd personally be happy if
> we didn't implement this PEP or just deferred it again. But if we *are*
> moving forward with it, I think it's worth trying to simplify it as much as
> possible/try and make it orthogonal with existing Python features.
>
> (But of course, I'm just some random person on the internet, so IDK if my
> opinion counts for much.)
>
> Regards,
> -- Michael
>
>
> On Thu, Jul 19, 2018 at 5:06 PM, Chris Angelico <rosuav at gmail.com> wrote:
>
>> On Fri, Jul 20, 2018 at 10:03 AM, Greg Ewing
>> <greg.ewing at canterbury.ac.nz> wrote:
>> > Rhodri James wrote:
>> >>
>> >> If anyone can think of a good word for "if it isn't None, otherwise",
>> I'd
>> >> be all for it :-)
>> >
>> >
>> > I don't think there's any single Engish word that captures
>> > all of that, so we'd have to invent one.
>> >
>> > Some suggestions:
>> >
>> > inno (If Not None, Otherwise)
>> >
>> > oft  (Or, Failing That)
>>
>> Iunno, that'd oft be confusing.
>>
>> Kappa
>>
>> ChrisA
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180721/385673b0/attachment.html>

From chris.barker at noaa.gov  Sat Jul 21 13:38:07 2018
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Sat, 21 Jul 2018 13:38:07 -0400
Subject: [Python-ideas] if we were to ever shorten `A if A else B` (was:
 PEP 505: None-aware operators)
In-Reply-To: <CAP1=2W5YtcBHtScMXi8oL_OWUmzqFhbMGp_94kmqb4OKYaDZYw@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAFYqXL8GsTyAHEkrQqSDpv_+bCcJAVv2j-0d_zwNuLLfK7Cuuw@mail.gmail.com>
 <CAPTjJmqPS_wiFTJaJ8YUKJfWsNcCX=h9AQ1EmWFv6Xf9Wes4tA@mail.gmail.com>
 <5B502A80.3080406@canterbury.ac.nz>
 <CAPTjJmquqsqX8XL5RGdvebwHw-vgx15sQU2Qbyg3SKz3WHso2A@mail.gmail.com>
 <5B50DA1E.9030904@brenbarn.net>
 <CAOOa=pPU2-KvWJotvO3D7sJ=q-uJGbZBw3khmHdReHZ-m2zuwA@mail.gmail.com>
 <CAPTjJmo_7H8ppV6JnZzQu5B6DdYxympenws8YcaWyYSCqAvhLQ@mail.gmail.com>
 <dc623fec-26be-4b74-f13b-63d74eb3824a@python.org>
 <CAP1=2W5YtcBHtScMXi8oL_OWUmzqFhbMGp_94kmqb4OKYaDZYw@mail.gmail.com>
Message-ID: <CALGmxE+oKc=ZK1eyfHuEiiXgqR-ekj9v7VLwwT6iDJY6aCKyQw@mail.gmail.com>

> my vote would go to `A otherwise B` since it's unambiguous, the case you care about the state  of comes first, and it doesn't trip your brain up looking for 'if'. :)

And I?d hope ?otherwise? is a rare variable name :-)

- CHB

>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From chris.barker at noaa.gov  Sat Jul 21 13:50:55 2018
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Sat, 21 Jul 2018 13:50:55 -0400
Subject: [Python-ideas] Multi-core reference count garbage collection
In-Reply-To: <CALD=Yf_B73DeWPy_7qv3jwT2iJkmz=Vju0qDKFpdsHpZ0rtiiA@mail.gmail.com>
References: <CALD=Yf-Ht4uDMHkvXp8i+xesw=e9zw4i+nAuRJFMVppNSYrboA@mail.gmail.com>
 <20180720170129.GE8744@ando.pearwood.info>
 <CALD=Yf8E3hV6LWwxh4sj8Hs6eyLiT9Sk87NNoAoGjxi1P1ScXA@mail.gmail.com>
 <20180720182037.GL8744@ando.pearwood.info>
 <CALD=Yf8S2THNqFNvOyRfTjX=pUBY09orSYFXGExKrVnJKSV3Aw@mail.gmail.com>
 <20180721012324.GM8744@ando.pearwood.info>
 <CALD=Yf_bbd0b=tSqD+EJxV7iHjoSHGrJatmKG9tPPVV0O5UamQ@mail.gmail.com>
 <E25755DA-489B-4AE9-9A83-0114215F0354@barrys-emacs.org>
 <CALD=Yf_B73DeWPy_7qv3jwT2iJkmz=Vju0qDKFpdsHpZ0rtiiA@mail.gmail.com>
Message-ID: <CALGmxEJdGHznMB9E551wgP_Lv0HNnYL0FLJxoJ6nLZgAhxXk3Q@mail.gmail.com>

> You wrote:
> > I'd argue that the ref counts are not interesting at all, only a
> > side effect of one possible solution to the object life time problem.
>
> I'm happy for you to regard multi-core reference counting (MCRC) as a toy problem, which won't become part of useful software.

Perhaps the point was that reference counting is only one of the
issues (race conditions?) the GIL solves.

So a GIL - free reference counting scheme may not prove to be helpful.

If that were it, we could (optionally) turn off reference counting in
multi-threaded code, and run a garbage collector once in a while
instead.

After all, cPython?s (mostly) deterministic garbage collection is not
guaranteed by the language standard.

-CHB

From chris.barker at noaa.gov  Sat Jul 21 14:09:38 2018
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Sat, 21 Jul 2018 14:09:38 -0400
Subject: [Python-ideas] Performance improvements via static typing
In-Reply-To: <pisa2h$8rj$1@blaine.gmane.org>
References: <CABypBWrqKatZBKy3VBdzpr+o6cOgxhjpnqt7tzKW1+sraeJMSA@mail.gmail.com>
 <pisa2h$8rj$1@blaine.gmane.org>
Message-ID: <CALGmxEKKA6+Cny6HAVOAT-TH2kgZ+MYBtL3Yp0=H8QaNu_eN=w@mail.gmail.com>

A note here:

Earlier in the conversation about standardizing type hinting, I (among
others) was interested in applying it to C-level static typing (e.g.
Cython).

Guido made it very clear that that was NOT a goal of type hints ?
rather, they were to be used for dynamic, python style types ? so a
?list? is guaranteed to act like a list in python code, but not to
have the same underlying binary representation.

We could still use the same syntax for things like Cython, but they
would have a different meaning.

And a JIT compiler may not benefit from the hints at all, as it would
have to check the actual type at run-time anyway.

-CHB


Sent from my iPhone

> On Jul 20, 2018, at 2:32 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
> Michael Hall schrieb am 19.07.2018 um 15:51:
>> While I am aware of projects like Cython and mypy, it seems to make sense
>> for CPython to allow optional enforcement of type hints, with compiler
>> optimizations related to it to be used. While this would not receive the
>> same level of performance benefits as using ctypes directly, there do
>> appear to be various gains available here.
>
> Well, first of all, a C level type check at runtime is quite fast compared
> to a byte code dispatch with stack argument handling, and can be done
> regardless of any type hints. There are various code patterns that would
> suggest a certain type ("x.append()" probably appends to a list, "x.get()"
> will likely be a dict lookup, etc.), and that can be optimised for without
> static type declarations and even without any type hints.
>
> Then, the mere fact that user code says "a: List" does not help in any way.
> Even "a: list" would not help, because any behaviour of that "list" might
> have been overridden by the list subtype that the function eventually receives.
>
> The same applies to "x: float". Here, in order to gain speed, the compiler
> would have to generate two separate code paths through the entire function,
> one for C double computations, and one for Python float subtypes. And then
> exponentiate that by the number of other typed arguments that may or may
> not contain subtypes. Quite some overhead.
>
> It's unclear if the gain would have any reasonable relation to the effort
> in the end. Sure, type hints could be used as a bare trigger for certain
> optimistic optimisations, but then, it's cheap enough to always apply these
> optimisations regardless, via a C type check.
>
> Stefan
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From brett at python.org  Sat Jul 21 14:43:21 2018
From: brett at python.org (Brett Cannon)
Date: Sat, 21 Jul 2018 11:43:21 -0700
Subject: [Python-ideas] Performance improvements via static typing
In-Reply-To: <pisa2h$8rj$1@blaine.gmane.org>
References: <CABypBWrqKatZBKy3VBdzpr+o6cOgxhjpnqt7tzKW1+sraeJMSA@mail.gmail.com>
 <pisa2h$8rj$1@blaine.gmane.org>
Message-ID: <CAP1=2W5Ap_M_6ap5tMMmYxK0eGT1aj2zbAy7gJsgd32scy1qKg@mail.gmail.com>

On Fri, Jul 20, 2018, 02:33 Stefan Behnel, <stefan_ml at behnel.de> wrote:

> Michael Hall schrieb am 19.07.2018 um 15:51:
> > While I am aware of projects like Cython and mypy, it seems to make sense
> > for CPython to allow optional enforcement of type hints, with compiler
> > optimizations related to it to be used. While this would not receive the
> > same level of performance benefits as using ctypes directly, there do
> > appear to be various gains available here.
>
> Well, first of all, a C level type check at runtime is quite fast compared
> to a byte code dispatch with stack argument handling, and can be done
> regardless of any type hints. There are various code patterns that would
> suggest a certain type ("x.append()" probably appends to a list, "x.get()"
> will likely be a dict lookup, etc.), and that can be optimised for without
> static type declarations and even without any type hints.
>
> Then, the mere fact that user code says "a: List" does not help in any way.
> Even "a: list" would not help, because any behaviour of that "list" might
> have been overridden by the list subtype that the function eventually
> receives.
>
> The same applies to "x: float". Here, in order to gain speed, the compiler
> would have to generate two separate code paths through the entire function,
> one for C double computations, and one for Python float subtypes. And then
> exponentiate that by the number of other typed arguments that may or may
> not contain subtypes. Quite some overhead.
>
> It's unclear if the gain would have any reasonable relation to the effort
> in the end. Sure, type hints could be used as a bare trigger for certain
> optimistic optimisations, but then, it's cheap enough to always apply these
> optimisations regardless, via a C type check.
>

I'll also mention that my masters thesis disproved the benefit of
type-specific opcodes for CPython:
https://scholar.google.com/scholar?cluster=2007053175643839734&hl=en&as_sdt=0,5&sciodt=0,5

What this means is someone will have to demonstrate actual performance
benefits before we talk about semantic changes or modifying the
interpreter. IOW don't prematurely optimize for an optimization until it's
actually been proven to be an optimization. ?


> Stefan
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180721/9bb3ae44/attachment.html>

From gregory.lielens at gmail.com  Sat Jul 21 15:29:33 2018
From: gregory.lielens at gmail.com (=?UTF-8?Q?Gr=C3=A9gory_Lielens?=)
Date: Sat, 21 Jul 2018 12:29:33 -0700 (PDT)
Subject: [Python-ideas] if we were to ever shorten `A if A else B` (was:
 PEP 505: None-aware operators)
In-Reply-To: <CALGmxE+oKc=ZK1eyfHuEiiXgqR-ekj9v7VLwwT6iDJY6aCKyQw@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAFYqXL8GsTyAHEkrQqSDpv_+bCcJAVv2j-0d_zwNuLLfK7Cuuw@mail.gmail.com>
 <CAPTjJmqPS_wiFTJaJ8YUKJfWsNcCX=h9AQ1EmWFv6Xf9Wes4tA@mail.gmail.com>
 <5B502A80.3080406@canterbury.ac.nz>
 <CAPTjJmquqsqX8XL5RGdvebwHw-vgx15sQU2Qbyg3SKz3WHso2A@mail.gmail.com>
 <5B50DA1E.9030904@brenbarn.net>
 <CAOOa=pPU2-KvWJotvO3D7sJ=q-uJGbZBw3khmHdReHZ-m2zuwA@mail.gmail.com>
 <CAPTjJmo_7H8ppV6JnZzQu5B6DdYxympenws8YcaWyYSCqAvhLQ@mail.gmail.com>
 <dc623fec-26be-4b74-f13b-63d74eb3824a@python.org>
 <CAP1=2W5YtcBHtScMXi8oL_OWUmzqFhbMGp_94kmqb4OKYaDZYw@mail.gmail.com>
 <CALGmxE+oKc=ZK1eyfHuEiiXgqR-ekj9v7VLwwT6iDJY6aCKyQw@mail.gmail.com>
Message-ID: <613de488-623d-4b10-bad4-3b7288aa0d3d@googlegroups.com>

Coming from the @ side (I was strong +1 on this), I have troubles seeing the real benefits from ?? (And even more from associates): did we really have long and complex expressions where the compactness of an operator would help? 

Operators are inherently obscure (except for those that are learnt in elementary school), but they help when you combine multiple operators and benefit from their precedence rules. There you can benefit from them, even if explicit it's better than implicit...
It was the case for @, and even with proofs of long formulas, and a history of matrix algebra from hundreds of years before computing science, the resistance was strong: a majority of non-numpy users resisted it, saying a function matmul(A,B) was good enough and A at B would bring nothing.
It was eventually accepted, 7 or so years after the initial proposal, through another PEP, when relative weight of numpy community was probably larger.
So i'd like to see examples of long expressions that would really benefit groin using an very specific operator. At this point, the explicitness of "a if a is not None else []" wins, by a long shot...

From g.rodola at gmail.com  Sat Jul 21 19:56:35 2018
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Sun, 22 Jul 2018 01:56:35 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180719133820.GV7318@ando.pearwood.info>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
Message-ID: <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>

On Thu, Jul 19, 2018 at 3:39 PM Steven D'Aprano <steve at pearwood.info> wrote:
> Tens of thousands of non-English speakers have had to learn the meaning
> of what might as well be meaningless, random sets of symbols (to them)
> like "class", "import", "while" and "True". If they can do so, perhaps
> we English-speakers should stop complaining about how hard it is to
> memorise the meaning of a couple of symbols like ??.

"class", "import", "while" and "True" are keywords, not symbols. A
symbol is more cryptic than a keyword hence it comes at a higher cost
in terms of readability. Which is the reason why conditionals use
keywords instead symbols:

- we say "and" instead of "&&"
- we say "or" instead of "||"
- we say "not" instead of "!"
- we say "is" instead of "==="

AFAICT "?" would be the first one breaking this rule for conditionals
[1], and not only for them since it can also arbitrarily appear in
non-conditionals (x.?y).

[1] I know, we say "==" instead of "equal" and "!=" instead of
"different" but the equal sign implies an equality. "?" does not imply
"None-ness": it's arbitrary. Furthermore "==" and "!=" are much more
common than testing for "None", hence they have more reason to exist.

> Surely its no more
> difficult than learning the various meanings of ** and [] which we've
> already done.

It is more difficult if you consider that ? has different spellings
(?, ??, a?.b, ??=, ...), each spelling with a different meaning.

> *Its just spelling*. If it is a useful and well-defined feature, we'll
> get used to the spelling soon enough.

Such an argument may apply to other languages but not Python. The
philosophy of the language is all about readability and beauty since
day 1, and the main reason why Python got so successful despite being
slow. That's why we have mandatory indentation, to say one. We don't
*have to* get used to idioms which can decrease readability. When we
do there must be a good reason and the spelling should matter (I
remember the debates about decorators' syntax).

--
Giampaolo - http://grodola.blogspot.com

From steve at pearwood.info  Sat Jul 21 21:54:19 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 22 Jul 2018 11:54:19 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
Message-ID: <20180722015418.GO8744@ando.pearwood.info>

On Sun, Jul 22, 2018 at 01:56:35AM +0200, Giampaolo Rodola' wrote:
> On Thu, Jul 19, 2018 at 3:39 PM Steven D'Aprano <steve at pearwood.info> wrote:
> > Tens of thousands of non-English speakers have had to learn the meaning
> > of what might as well be meaningless, random sets of symbols (to them)
> > like "class", "import", "while" and "True". If they can do so, perhaps
> > we English-speakers should stop complaining about how hard it is to
> > memorise the meaning of a couple of symbols like ??.
> 
> "class", "import", "while" and "True" are keywords, not symbols.

They are only key WORDS if you are an English speaker. If your language 
doesn't use the Latin script, they don't even look like words. They look 
like gibberish: ?????


> A symbol is more cryptic than a keyword

I was using "symbol" in its most general sense, "a visible sign or 
representation of an idea". Words, letters, icons, emblems etc are all 
symbols. Sorry for being unclear.


> hence it comes at a higher cost in terms of readability. 

It is not clear to me that this is always true even when it comes to 
non-word symbols. I don't think that "+" is harder to read than 
"standard_mathematics_operators_numeric_addition" for example.

It is possible to be too verbose as well as too terse. This is why we 
aren't copying COBOL.


> which is the reason why conditionals use
> keywords instead symbols:
> 
> - we say "and" instead of "&&"
[...]

Indeed we do. But we also say:

- we say "+" instead of "add"
- we say "//" instead of "floor division"
- we say "**" instead of "exponentiation"
- we say "&" instead of "bitwise AND"
- we say "f( ... )" instead of "call f with arguments ..."

etc. Python has no shortage of non-word symbols:

  == != ~ - + * ** / // @ % ^ & | << >> < <= > >= [] ()


[...]
> > *Its just spelling*. If it is a useful and well-defined feature, we'll
> > get used to the spelling soon enough.
> 
> Such an argument may apply to other languages but not Python.

I disagree. I think it applies equally to Python. Python functions and 
keywords are usually English words, but not always:

  def elif

have to be memorised even by English speakers. If we gained a function 
or even a keyword from Italian, let's say "ripetere", would that really 
change the nature of Python? I don't think so. English speakers are 
adaptable, we don't so much borrow words from other languages as chase 
them down the alley and mug them for new vocabulary.

The same applies to non-word symbols. Look at how quickly and easily 
people adapted to @ (the commercial "at" sign, a variation of 
multiplication) as a separator in emails, not to mention #hashtags.

I'll admit that the number and variety of new operators gives me some 
reason to pause, but for the simplest and most obvious case, the 
proposed ?? operator, I think that the fears about readability are 
grossly exaggerated.


> The philosophy of the language is all about readability and beauty since
> day 1, and the main reason why Python got so successful despite being
> slow. 

Let's talk about Python's readability: the first time I saw Python code, 
I had *no idea* how to read it. This was Python 1.5, long before 
comprehensions, @ decorator syntax, etc. I could read Pascal, Forth, 
Hypertalk, BASIC and Fortran, and I looked at Python and couldn't make 
heads or tails of it. It was full of cryptic idioms like:

    for i in range(len(sequence))

which is meaningless until you learn what range does and learn to read 
it as a for loop:

    for i = 0 to len(sequence) - 1

And as for slice notation:

    suffix = word[n:]

that might as well have been advanced quantum mechanics transliterated 
into Korean by a native Navaho speaker for all I could read it. I knew 
round brackets were used for function calls f() but the only experience 
I had with square and curly brackets was in mathematics, where they are 
used as second and third levels of brackets:

    x = {[a (b+1)][b (a - 1)] - 1}/2

Now *that* I understood. What on earth was this mysterious {'a': 1} 
syntax that I keep seeing in Python code?

My initial experience with this bizarre, cryptic language Python was so 
off-putting that I put it aside for literally three or four years before 
coming back to it.

Of course I believe that Python is, generally speaking, a beautiful and 
elegant language, extremely readable. But readability doesn't exist in a 
vacuum. We still need to learn the symbols of the language, not just 
operators like ** % and // but words as well. To appreciate the elegance 
of the language, we need to know the common idioms, as well as the 
semantics of keywords and functions.

Unfortunately, ?? does have one disadvantage: even though it is used as 
an operator in a number of languages, Google doesn't seem to have 
indexed it. Goggling for "??" is unhelpful. But then googling for "@ 
python" is similarly unhelpful. Clearly decorators was a mistake. /s


> That's why we have mandatory indentation, to say one. We don't
> *have to* get used to idioms which can decrease readability. When we
> do there must be a good reason and the spelling should matter (I
> remember the debates about decorators' syntax).

Exactly. Years later, do we still think that @decorator syntax is 
unreadable and unnecessary?


In my opinion, writing

    expression if expression is None else default

is the *opposite* of Pythonic, it is verbose and the DRY violation is 
inelegant (as well as inefficient). I'd much rather use:

    expression ?? default

although with PEP 572 approved, there is an alternative:

    temp := expression if temp is None else default

which avoids the DRY violation but is more verbose than even the first 
version.


-- 
Steve

From rosuav at gmail.com  Sat Jul 21 21:58:29 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 22 Jul 2018 11:58:29 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180722015418.GO8744@ando.pearwood.info>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
Message-ID: <CAPTjJmqcSE2mn-M9K_FWNzs-OkCNHCc7ihzAWG5R8nJyS=QnYQ@mail.gmail.com>

On Sun, Jul 22, 2018 at 11:54 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> If we gained a function
> or even a keyword from Italian, let's say "ripetere", would that really
> change the nature of Python? I don't think so. English speakers are
> adaptable, we don't so much borrow words from other languages as chase
> them down the alley and mug them for new vocabulary.

"lambda" would like to have a word with you. Or maybe a letter. I'm
not really sure.

ChrisA

From chris.barker at noaa.gov  Sun Jul 22 00:36:14 2018
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Sat, 21 Jul 2018 21:36:14 -0700
Subject: [Python-ideas] Revisiting str.rreplace()
In-Reply-To: <CACac1F9sY0doY3TwvcVNfYWZudYMdD_zB3bLx4ytfvzr1iBXDw@mail.gmail.com>
References: <CAN83G44dWT=+t68DS=abk29eM-WOQm3ZjOTDFzBNOVLtgr8xGA@mail.gmail.com>
 <CAP2Qz+VPpRBTAfn1OdYUMCD3oUoV143av7juWGvNkkXO-BxrGQ@mail.gmail.com>
 <CACo5Rz4U2N0v2KPaWspCrtBgm1cLk011hD7XPQev-rYKoty9qw@mail.gmail.com>
 <900e9971-08d7-dad0-498d-8624ffa2d96f@trueblade.com>
 <CACo5Rz78ovnUPszi6ROoyMH8Dj5V6EgbAmmsNDYDQpqcGj0iRA@mail.gmail.com>
 <11fe0bee-a200-70e0-3df5-f375bb33cddd@trueblade.com>
 <CACac1F9sY0doY3TwvcVNfYWZudYMdD_zB3bLx4ytfvzr1iBXDw@mail.gmail.com>
Message-ID: <CALGmxEJmh-5L=x_KxhZ-1mkkEHpjKJfo5=mPnZraRkX=Xv_giw@mail.gmail.com>

I starting g reading this thread in the middle, on a phone.

But was very confused for a while because I didn?t notice that there
were two ?r?s at the beginning of .rreplace

Just sayin?

-CHB

Sent from my iPhone

> On Jul 19, 2018, at 9:29 AM, Paul Moore <p.f.moore at gmail.com> wrote:
>
>> On 19 July 2018 at 16:25, Eric V. Smith <eric at trueblade.com> wrote:
>> It currently does something: it replaces all instances, just as if you
>> hadn't supplied a count (see my example below). You can't change its
>> behavior.
>
> ... without a deprecation cycle. Which is of course not worth it for
> something which could much more easily be done by adding an rreplace
> function - which is the real point of the comment.
> Paul
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From p.f.moore at gmail.com  Sun Jul 22 03:32:03 2018
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 22 Jul 2018 08:32:03 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180722015418.GO8744@ando.pearwood.info>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
Message-ID: <CACac1F_n3L4cLfgxN74ZamG9OgNEhY-8ZpBDzjv6BM8CtN7PvA@mail.gmail.com>

On 22 July 2018 at 02:54, Steven D'Aprano <steve at pearwood.info> wrote:

> I'll admit that the number and variety of new operators gives me some
> reason to pause, but for the simplest and most obvious case, the
> proposed ?? operator, I think that the fears about readability are
> grossly exaggerated.

Certainly *my* concerns about readability are around the other
proposed operators (?[ and ?. in particular).

> In my opinion, writing
>
>     expression if expression is None else default
>
> is the *opposite* of Pythonic, it is verbose and the DRY violation is
> inelegant (as well as inefficient). I'd much rather use:
>
>     expression ?? default

Agreed. But the PEP proposes three other operators, and it's not at
all clear to me that those are such clear wins.

Paul

From gregory.lielens at gmail.com  Sun Jul 22 04:51:31 2018
From: gregory.lielens at gmail.com (=?UTF-8?Q?Gr=C3=A9gory_Lielens?=)
Date: Sun, 22 Jul 2018 01:51:31 -0700 (PDT)
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CACac1F_n3L4cLfgxN74ZamG9OgNEhY-8ZpBDzjv6BM8CtN7PvA@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
 <CACac1F_n3L4cLfgxN74ZamG9OgNEhY-8ZpBDzjv6BM8CtN7PvA@mail.gmail.com>
Message-ID: <86da19bd-394f-40e5-bda5-c293c3513365@googlegroups.com>

To get rid of the two other ( ?. And ?[] ), we could also define getitem and getattr for None to always return None...;-)

I'm joking, although such an "absorbing" None may have been a good choice when None was introduced, and maybe a way to do an absorbing-None per-statement maybe nice...Nice enough to add more subtleties to python? I don't think so, but it would be readable...

From g.rodola at gmail.com  Sun Jul 22 06:13:04 2018
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Sun, 22 Jul 2018 12:13:04 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180722015418.GO8744@ando.pearwood.info>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
Message-ID: <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>

On Sun, Jul 22, 2018 at 3:55 AM Steven D'Aprano <steve at pearwood.info> wrote:

> Indeed we do. But we also say:
>
> - we say "+" instead of "add"
> - we say "//" instead of "floor division"
> - we say "**" instead of "exponentiation"
> - we say "&" instead of "bitwise AND"
> - we say "f( ... )" instead of "call f with arguments ..."

[...]

I don't think that "+" is harder to read than

"standard_mathematics_operators_numeric_addition"


Please let's drop the argument that + - * / = and ? are the same. They
clearly are not. Anybody learned those symbols at elementary schools, all
programming languages have them and using math in programming is common
enough to justify a symbol over a keyword. "a + b" is literally just an
addition and nothing else. The "?" variants have multiple meanings,
spellings and implications:

- "a ?? b" means "b is chosen over a if a is None"

- "a ??= b" means "a is set to b if a is None"

- "a?.b" means "a.b is executed but only if a is not None"

- "a?[2] ?? 3" means "index 2 of list a is picked up if a is not None, else
use 3"

"a?.b"and "a?[2]" in particular go way beyond the mere "it's not pretty"
argument which, I concur, can be subjective, as you don't know where
evaluation stops. Even "a ??= b" goes beyond that as it introduces yet
another assignment operator (the third, as we now have = and :=). So again,
I don't think it's fair to dismiss the whole thing as "it's just another
symbol" or "it's like a + b".

As for bitwise operators: they are kinda obscure and low-levelish and when
I bump into them I still have to pause to reason what's going on. The
difference with ? though is that you basically have no other way to do the
same thing. Also they are much more rare and also are present in many other
languages since... forever.

-- 
Giampaolo - http://grodola.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180722/5e0d629e/attachment.html>

From p.f.moore at gmail.com  Sun Jul 22 06:26:22 2018
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 22 Jul 2018 11:26:22 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
 <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>
Message-ID: <CACac1F_kA-=uKNdmxfG_TYQ1A6W-S+VF9orvpG8aoLLg+Pashw@mail.gmail.com>

On 22 July 2018 at 11:13, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
> - "a?[2] ?? 3" means "index 2 of list a is picked up if a is not None, else
> use 3"

Actually, doesn't it mean

if a is not None, pick up index 2 of the list.
If a is None, OR IF a[2] IS NONE, then use 3.
If a is None but a[2] is not None, use a[2].

?

Which is subtly different, and probably at least as prone to
"accidental" errors as some of the constructs the None-aware operators
are intended to replace.

Paul

From g.rodola at gmail.com  Sun Jul 22 06:46:35 2018
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Sun, 22 Jul 2018 12:46:35 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CACac1F_kA-=uKNdmxfG_TYQ1A6W-S+VF9orvpG8aoLLg+Pashw@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
 <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>
 <CACac1F_kA-=uKNdmxfG_TYQ1A6W-S+VF9orvpG8aoLLg+Pashw@mail.gmail.com>
Message-ID: <CAFYqXL_0CRqRsrQiqm7L3rkC7V99=Gfo7=9q5jbE4MzDr9ig5Q@mail.gmail.com>

On Sun, Jul 22, 2018 at 12:26 PM Paul Moore <p.f.moore at gmail.com> wrote:

> On 22 July 2018 at 11:13, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
> > - "a?[2] ?? 3" means "index 2 of list a is picked up if a is not None,
> else
> > use 3"
>
> Actually, doesn't it mean
>
> if a is not None, pick up index 2 of the list.
> If a is None, OR IF a[2] IS NONE, then use 3.
> If a is None but a[2] is not None, use a[2].
>
> ?
>
> Which is subtly different, and probably at least as prone to
> "accidental" errors as some of the constructs the None-aware operators
> are intended to replace.
>
> Paul
>

Yes, I think you're right.

-- 
Giampaolo - http://grodola.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180722/095d2367/attachment.html>

From gregory.lielens at gmail.com  Sun Jul 22 06:54:51 2018
From: gregory.lielens at gmail.com (=?UTF-8?Q?Gr=C3=A9gory_Lielens?=)
Date: Sun, 22 Jul 2018 03:54:51 -0700 (PDT)
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAFYqXL_0CRqRsrQiqm7L3rkC7V99=Gfo7=9q5jbE4MzDr9ig5Q@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
 <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>
 <CACac1F_kA-=uKNdmxfG_TYQ1A6W-S+VF9orvpG8aoLLg+Pashw@mail.gmail.com>
 <CAFYqXL_0CRqRsrQiqm7L3rkC7V99=Gfo7=9q5jbE4MzDr9ig5Q@mail.gmail.com>
Message-ID: <3dea3539-5413-4b7e-a450-d6adb5ab71de@googlegroups.com>

Except that the third possibility is not possible...if a is None, a[2] will throw an exception...
For now at least ;-)

From p.f.moore at gmail.com  Sun Jul 22 07:09:40 2018
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 22 Jul 2018 12:09:40 +0100
Subject: [Python-ideas] Fwd:  PEP 505: None-aware operators
In-Reply-To: <CACac1F87diQ-qs5-P9oFrm1R10+A+CK608P53pLtfs0JO5EnWA@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
 <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>
 <CACac1F_kA-=uKNdmxfG_TYQ1A6W-S+VF9orvpG8aoLLg+Pashw@mail.gmail.com>
 <CAFYqXL_0CRqRsrQiqm7L3rkC7V99=Gfo7=9q5jbE4MzDr9ig5Q@mail.gmail.com>
 <3dea3539-5413-4b7e-a450-d6adb5ab71de@googlegroups.com>
 <CACac1F87diQ-qs5-P9oFrm1R10+A+CK608P53pLtfs0JO5EnWA@mail.gmail.com>
Message-ID: <CACac1F8LWbLZq5vXs6L09wAh9+53F_c9t5ySNsqqubnPG9cv-Q@mail.gmail.com>

Aargh, I hate Google Groups with a vengeance. If people *have* to post
from there, can they please change reply-to so that replies don't get
messed up. Or is that not possible, and yet another way that GG is
just broken?
Paul


---------- Forwarded message ----------
From: Paul Moore <p.f.moore at gmail.com>
Date: 22 July 2018 at 12:05
Subject: Re: [Python-ideas] PEP 505: None-aware operators
To: Gr?gory Lielens <gregory.lielens at gmail.com>
Cc: python-ideas <python-ideas at googlegroups.com>


On 22 July 2018 at 11:54, Gr?gory Lielens <gregory.lielens at gmail.com> wrote:
> Except that the third possibility is not possible...if a is None, a[2] will throw an exception...
> For now at least ;-)

Doh. True, I should have said "If a is not None and a[2] is not None, use a[2]".

But my point about unintended behaviour if a[2] is None stands. And
the wider point that these operators are hard to reason correctly
about is probably emphasised by my mistake.

Paul

From python-ideas-2018 at tutnicht.de  Sun Jul 22 07:03:40 2018
From: python-ideas-2018 at tutnicht.de (=?iso-8859-1?Q?J=F6rn?= Heissler)
Date: Sun, 22 Jul 2018 13:03:40 +0200
Subject: [Python-ideas] Add function readbyte to asyncio.StreamReader
Message-ID: <20180722110340.GA23320@carrot.tutnicht.de>

Hello,

I'm implementing a protocol where I need to read individual bytes until
a condition is met (value & 0x80 == 0).

My current approach is: value = (await reader.readexactly(1))[0]

To speed this up, I propose that a new function is added to
asyncio.StreamReader: value = await reader.readbyte()

I duplicated readexactly and stripped out some parts. Below code appears
to work:

    async def readbyte(self):
        if self._exception is not None:
            raise self._exception

        while not self._buffer:
            if self._eof:
                raise EOFError()
            await self._wait_for_data('readbyte')

        data = self._buffer[0]
        del self._buffer[0]
        self._maybe_resume_transport()
        return data

For comparing the speed, I'm receiving a 50 MiB file byte-by-byte.

cpython-3.7.0:
    readexactly: 42.43 seconds
    readbyte   : 22.05 seconds
    speedup    : 92.4%

pypy3-v6.0.0:
    readexactly: 3.21 seconds
    readbyte   : 2.76 seconds
    speedup    : 16.3%

Thanks

From steve at pearwood.info  Sun Jul 22 08:10:06 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 22 Jul 2018 22:10:06 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
 <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>
Message-ID: <20180722121006.GQ8744@ando.pearwood.info>

On Sun, Jul 22, 2018 at 12:13:04PM +0200, Giampaolo Rodola' wrote:
> On Sun, Jul 22, 2018 at 3:55 AM Steven D'Aprano <steve at pearwood.info> wrote:
[...]
> > I don't think that "+" is harder to read than
> > "standard_mathematics_operators_numeric_addition"
> 
> 
> Please let's drop the argument that + - * / = and ? are the same.

In context, the argument was that non-word symbols are not always worse 
than words, not that all symbols are "the same".

Obviously symbols that we use regularly will be more familiar and easier 
to recognise than symbols we use rarely. I still have to make a 
conscious action to recall which of ? and ? is set union and 
intersection, and don't ask me what set symmetric difference is. And 
even after 20 years of Python I still occasionally write ^ for 
exponentiation instead of **.

But if we insist that every symbol we use is instantly recognisable and 
intuitively obvious to every programmer, we're putting the bar for 
acceptance impossibly high.


> They clearly are not. Anybody learned those symbols at elementary 
> schools, all programming languages have them and using math in 
> programming is common enough to justify a symbol over a keyword. "a + 
> b" is literally just an addition and nothing else.

That's not quite correct: '+' in Python is used for both addition and 
sequence concatenation. And with operator overloading, it can be 
anything at all. But it is *usually* addition, or concatenation.

I don't know what it is like in your country, but here in Australia, I 
don't know any school that teaches * for multiplication except in 
programming classes, which is not a core subject. The usual symbol we 
have for multiplication is ? and sometimes ? (dot operator).

This is a good point: after learning * for multiplication, it becomes so 
familiar that most of us forget that we haven't been using it forever. 
It becomes second-nature. In the same way that @ for decorators has 
become second nature, or slice notation. Both of which are terribly 
mysterious to people just starting out.

We shouldn't judge proposals on how mysterious they are the first time 
we see them, because everything is mysterious the first time. We should 
try to look forward to when we've seen them ten or twenty times. How 
will the "usefulness versus surprise" trade-off appear when, let's say, 
50% of the surprise has been worn away with experience?

As a community, we're risk-adverse. I understand why we should be 
conservative in what we add to the language (once added, it cannot 
easily be removed if it turns out to be a mistake) but on Python-Ideas 
we regularly demand levels of obviousness and "readability" that 
existing syntax does not reach.

(For example, the dot operator for attribute access fails the "syntax 
should not look like grit on Tim's monitor" test.)

I believe that it is fine to prefer that new syntax is no harder to 
learn or use than (for example) // or ** (neither of which is taught in 
maths class), or slice notation. But I don't think it is fair, or 
desirable, to demand levels of readability greater than what we already 
have in the less common corners of the language.

All the obvious operators are already in use. Anything we add now is 
going to be a little bit niche, a little bit unusual. It's not like 
we're going to suddenly realise we forgot to include a "subtract" 
operator. So it is perfectly natural that any new operators won't be as 
familiar as + or - operators. But it might become as familiar as ** or 
<< operators, or some of the less common regex patterns, and that's 
okay. Not everything needs to be as recognisable as the plus sign.


> The "?" variants have multiple meanings, spellings and implications:
[...]

Indeed. And I think we ought to think carefully about the benefits and 
costs of all of those variants separately.

To me, the ?? operator seems like a clear and obvious win. The other 
variants are more complex and the benefit is not as obvious to me, so I 
haven't decided where I stand on them.


-- 
Steve

From rosuav at gmail.com  Sun Jul 22 08:16:28 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 22 Jul 2018 22:16:28 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180722121006.GQ8744@ando.pearwood.info>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
 <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>
 <20180722121006.GQ8744@ando.pearwood.info>
Message-ID: <CAPTjJmpnqc4-X7G-R6PtsoGVL_C3OEOYYASFCVGJGW=sXfZHJQ@mail.gmail.com>

On Sun, Jul 22, 2018 at 10:10 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> As a community, we're risk-adverse. I understand why we should be
> conservative in what we add to the language (once added, it cannot
> easily be removed if it turns out to be a mistake) but on Python-Ideas
> we regularly demand levels of obviousness and "readability" that
> existing syntax does not reach.
>
> (For example, the dot operator for attribute access fails the "syntax
> should not look like grit on Tim's monitor" test.)

My understanding of that test is, more or less: "syntax should not be
such that grit on Tim's monitor can make it ambiguous". Which would
mean that attribute access does pass, since there's no logical meaning
for "list sort()" or "random randint" with just a space between them.
But otherwise, yes, I absolutely agree.

ChrisA

From mertz at gnosis.cx  Sun Jul 22 09:01:58 2018
From: mertz at gnosis.cx (David Mertz)
Date: Sun, 22 Jul 2018 09:01:58 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180722121006.GQ8744@ando.pearwood.info>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
 <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>
 <20180722121006.GQ8744@ando.pearwood.info>
Message-ID: <CAEbHw4b7hSgu28RwfwMav=cR-uxrErnGBDpeDVuirOevFvLo3g@mail.gmail.com>

On Sun, Jul 22, 2018, 8:11 AM Steven D'Aprano <steve at pearwood.info> wrote:

> On Sun, Jul 22, 2018 at 12:13:04PM +0200, Giampaolo Rodola' wrote:
> > Please let's drop the argument that + - * / = and ? are the same.
> We shouldn't judge proposals on how mysterious they are the first time we
> see them, because everything is mysterious the first time.
> All the obvious operators are already in use. Anything we add now is going
> to be a little bit niche, a little bit unusual. It's not like
> we're going to suddenly realise we forgot to include a "subtract"
> operator. So it is perfectly natural that any new operators won't be as
> familiar as + or - operators. But it might become as familiar as ** or <<
> operators, or some of the less common regex patterns,
>

I'm glad you've moved to better acknowledging the relative familiarity of
symbols. This simple can't be brushed away as a non-concern, as you did in
prior posts.

We're not promoting or reaching Python to Martians with entirely different
systems of writing. Nor, for that matter, even to humans of the 14th
entirety CE, before some basic math symbology became universal.

+, -,  /, <, >, <=, >= are taught to small children throughout the world,
over the last 5 or more generations (the compound inequalities drawn in
combined glyphs, but iconically very close). Yes they are characters with
no inherent meaning, but they are as well known as the sounds of Latin
letters, or probably more. '*' is a bit more computer specific. 'x' or some
nicer typographic variation is taught to children. But '*' is close to
universal in computer languages. '**' is funny. It's used in many computer
languages, but ^ is also a commonplace and probably more iconic for
"superscript", the actual middle school notation on exponentiation.

Actually, I have a ten Pythin line command-line calculator that accepts 'x'
and '^' in their "obvious" meanings.

The various question mark operators are only reasonable to compare to e.g.
@ for either __matmul__ or introducing decorators. Or to bitwise operators
like |, ~, ^, <<. But in both of those areas, the programs are ones dealing
with specialized domains where "experts" are necessary to understand the
semantics, not only the iconography.

Here the question mark operators are very different. They are proposed as
general constructs with no particularly special area of use. Programmers or
hobbyists who come from different or no programming language would
encounter them. Even as a scientific Python export, my linear algebra is
weak... if I see code with a bunch of '@' marks I know I'm not really going
to understand it, but before I look I'm warned about the domain
difficulties by the comments, function names, and so on.

The question marks are basic?though obscure and hard to reason about?flow
control.

To me, the ?? operator seems like a clear and obvious win. The other
> variants are more complex and the benefit is not as obvious to me, so I
> haven't decided where I stand on them.
>

Yes, that is the only one I'm merely -0 on. The others I'm -1000. Even
experienced developers in this thread keep tripping over the actual
semantics of those others, and the PEP proposer is unsure about what's best
for some edge behaviors that are actually really important were they to be
adopted.

The most any of these can possibly do is save a few characters in a ternary
line, or perhaps a few lines of if/elif that make the intention far more
obvious.

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180722/8b1c293a/attachment-0001.html>

From gregory.lielens at gmail.com  Sun Jul 22 09:22:59 2018
From: gregory.lielens at gmail.com (=?UTF-8?Q?Gr=C3=A9gory_Lielens?=)
Date: Sun, 22 Jul 2018 06:22:59 -0700 (PDT)
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4b7hSgu28RwfwMav=cR-uxrErnGBDpeDVuirOevFvLo3g@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
 <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>
 <20180722121006.GQ8744@ando.pearwood.info>
 <CAEbHw4b7hSgu28RwfwMav=cR-uxrErnGBDpeDVuirOevFvLo3g@mail.gmail.com>
Message-ID: <6562164f-9be1-40ec-890d-508fd5c81014@googlegroups.com>

The ?? operator is probably the less scary one regarding legibility, and in guessing (or remembering) what it exactly does...
Well, at least I think I understand what it does exactly, but if I'm not wrong there, what it does is also quite simple and minimal.

A function returning it's first non-None argument (or None, if all args are None) will provide the same functionality, with not much typing. You have parenthesis for the call, but you will probably need them anyway to group things, for correcting precedence, or helping the reader to parse your expression even if precedence was right.
You have an extra call, so ?? may be more efficient...maybe. 

Is that a reason enough, together with a few letters saved typing, to introduce ?? ? Not for me...

From rosuav at gmail.com  Sun Jul 22 09:26:15 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 22 Jul 2018 23:26:15 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <6562164f-9be1-40ec-890d-508fd5c81014@googlegroups.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
 <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>
 <20180722121006.GQ8744@ando.pearwood.info>
 <CAEbHw4b7hSgu28RwfwMav=cR-uxrErnGBDpeDVuirOevFvLo3g@mail.gmail.com>
 <6562164f-9be1-40ec-890d-508fd5c81014@googlegroups.com>
Message-ID: <CAPTjJmr8cMiW_K1jZwa4WJ_Tmy406+MxrXF_XR+OLWc0W47r3w@mail.gmail.com>

On Sun, Jul 22, 2018 at 11:22 PM, Gr?gory Lielens
<gregory.lielens at gmail.com> wrote:
> The ?? operator is probably the less scary one regarding legibility, and in guessing (or remembering) what it exactly does...
> Well, at least I think I understand what it does exactly, but if I'm not wrong there, what it does is also quite simple and minimal.
>
> A function returning it's first non-None argument (or None, if all args are None) will provide the same functionality, with not much typing. You have parenthesis for the call, but you will probably need them anyway to group things, for correcting precedence, or helping the reader to parse your expression even if precedence was right.
> You have an extra call, so ?? may be more efficient...maybe.
>
> Is that a reason enough, together with a few letters saved typing, to introduce ?? ? Not for me...

You forget that the operator will *short-circuit*. It will not
evaluate the second argument if the first argument is None. You cannot
do this with a function, other than with a hack like a lambda
function.

THAT is reason enough for an operator.

ChrisA

From g.rodola at gmail.com  Sun Jul 22 09:35:02 2018
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Sun, 22 Jul 2018 15:35:02 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180722121006.GQ8744@ando.pearwood.info>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
 <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>
 <20180722121006.GQ8744@ando.pearwood.info>
Message-ID: <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>

On Sun, Jul 22, 2018 at 2:10 PM Steven D'Aprano <steve at pearwood.info> wrote:
>
> On Sun, Jul 22, 2018 at 12:13:04PM +0200, Giampaolo Rodola' wrote:
> > On Sun, Jul 22, 2018 at 3:55 AM Steven D'Aprano <steve at pearwood.info> wrote:
> [...]
> > > I don't think that "+" is harder to read than
> > > "standard_mathematics_operators_numeric_addition"
> >
> >
> > Please let's drop the argument that + - * / = and ? are the same.
> [...]
> But if we insist that every symbol we use is instantly recognisable and
> intuitively obvious to every programmer, we're putting the bar for
> acceptance impossibly high.

I personally don't find "a ?? b" too bad (let's say I'm -0 about it)
but idioms such as "a?.b", "a ??= b" and "a?[3] ?? 4" look too
Perl-ish to me, non pythonic and overall not explicit, no matter what
the chosen symbol is gonna be. It looks like they want to do too much
for the sole reason of allowing people to write more compact code and
save a few lines. Compact code is not necessarily a good thing,
especially when it comes at the expense of readability and
explicitness, as I think is this case.

> All the obvious operators are already in use. Anything we add now is
> going to be a little bit niche, a little bit unusual.

That's basically my point. And I know I'll sound very conservative
here but to me that is a valid enough reason to not take action or be
extremely careful at the very least. Not to state the obvious but it's
not that we *have to* use the remaining unused symbols just because
they're there.

-- 
Giampaolo - http://grodola.blogspot.com

From rosuav at gmail.com  Sun Jul 22 09:38:18 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 22 Jul 2018 23:38:18 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
 <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>
 <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
Message-ID: <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>

On Sun, Jul 22, 2018 at 11:35 PM, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
> On Sun, Jul 22, 2018 at 2:10 PM Steven D'Aprano <steve at pearwood.info> wrote:
>>
>> On Sun, Jul 22, 2018 at 12:13:04PM +0200, Giampaolo Rodola' wrote:
>> > On Sun, Jul 22, 2018 at 3:55 AM Steven D'Aprano <steve at pearwood.info> wrote:
>> [...]
>> > > I don't think that "+" is harder to read than
>> > > "standard_mathematics_operators_numeric_addition"
>> >
>> >
>> > Please let's drop the argument that + - * / = and ? are the same.
>> [...]
>> But if we insist that every symbol we use is instantly recognisable and
>> intuitively obvious to every programmer, we're putting the bar for
>> acceptance impossibly high.
>
> I personally don't find "a ?? b" too bad (let's say I'm -0 about it)
> but idioms such as "a?.b", "a ??= b" and "a?[3] ?? 4" look too
> Perl-ish to me, non pythonic and overall not explicit, no matter what
> the chosen symbol is gonna be.

Please explain what is not explicit about it. "a?.b" is very simple
and perfectly explicit: it means "None if a is None else a.b". What
does "not explicit" mean, other than "I don't like this code"?

ChrisA

From gregory.lielens at gmail.com  Sun Jul 22 10:02:26 2018
From: gregory.lielens at gmail.com (=?UTF-8?Q?Gr=C3=A9gory_Lielens?=)
Date: Sun, 22 Jul 2018 07:02:26 -0700 (PDT)
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPTjJmr8cMiW_K1jZwa4WJ_Tmy406+MxrXF_XR+OLWc0W47r3w@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
 <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>
 <20180722121006.GQ8744@ando.pearwood.info>
 <CAEbHw4b7hSgu28RwfwMav=cR-uxrErnGBDpeDVuirOevFvLo3g@mail.gmail.com>
 <6562164f-9be1-40ec-890d-508fd5c81014@googlegroups.com>
 <CAPTjJmr8cMiW_K1jZwa4WJ_Tmy406+MxrXF_XR+OLWc0W47r3w@mail.gmail.com>
Message-ID: <3f036cb4-e11e-4766-90bd-b03db7dcf5a3@googlegroups.com>

Short circuit if the first argument is NOT None, I guess? ;-)

Yes, so a short circuit is sometimes good. Not often imho, for a default triggered by None, but sometimes...
In the case it is, do you want it to be hidden in an expression? Usually it would be better to draw attention, when the default is either costly to compute, or worse, it's evaluation have side effect. I would use an old fashioned if in those cases...

From turnbull.stephen.fw at u.tsukuba.ac.jp  Sun Jul 22 10:17:55 2018
From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull)
Date: Sun, 22 Jul 2018 23:17:55 +0900
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180722015418.GO8744@ando.pearwood.info>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
Message-ID: <23380.37395.816074.899031@turnbull.sk.tsukuba.ac.jp>

Steven D'Aprano writes:

 > In my opinion, writing
 > 
 >     expression if expression is None else default
 > 
 > is the *opposite* of Pythonic, it is verbose and the DRY violation is 
 > inelegant (as well as inefficient). I'd much rather use:
 > 
 >     expression ?? default

Sure, if "expression or default" won't do.  But in my code, I can't
recall encountering a case where it wouldn't.

That is, in the vast majority of my code, the point of using None as
the "oops, try again" sentinel is not that it's a different *value*
from a falsie of the expected type, it's that it's a different *type*.
It has very few attributes, and so will most likely eventually raise
if I forget the "or default" clause and a falsie escapes from the
enclosing code, rather than silently doing an erroneous computation.

Of course "in my code" is very anecdotal, and we do have testimony
from many people that these cases are important to them.  I'd still
like to know, not how much code looks better with "expr ?? default"
vs. spelling it out as a conditional statement or expression, but
rather how much code *must* be expressed as "expr ??  default" (or
"expr ?. attr ?? default", as Steve Dower reminds us) because the
falsie of expected type is a useful value of expr (or expr.attr).

 > although with PEP 572 approved, there is an alternative:
 > 
 >     temp := expression if temp is None else default
 > 
 > which avoids the DRY violation but is more verbose than even the first 
 > version.

No, it somewhat obfuscates the DRY violation, but you've still used
"temp" twice.  (Consider "var if var is None else default".)  I also
don't agree that it's more verbose than the first version if
"expression" is more complex than a variable reference, not to mention
the possibility of side effects in expression.

Steve

From g.rodola at gmail.com  Sun Jul 22 11:09:39 2018
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Sun, 22 Jul 2018 17:09:39 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
 <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>
 <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
 <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
Message-ID: <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>

On Sun, Jul 22, 2018 at 3:38 PM Chris Angelico <rosuav at gmail.com> wrote:
>
> On Sun, Jul 22, 2018 at 11:35 PM, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
> > On Sun, Jul 22, 2018 at 2:10 PM Steven D'Aprano <steve at pearwood.info> wrote:
> >>
> >> On Sun, Jul 22, 2018 at 12:13:04PM +0200, Giampaolo Rodola' wrote:
> >> > On Sun, Jul 22, 2018 at 3:55 AM Steven D'Aprano <steve at pearwood.info> wrote:
> >> [...]
> >> > > I don't think that "+" is harder to read than
> >> > > "standard_mathematics_operators_numeric_addition"
> >> >
> >> >
> >> > Please let's drop the argument that + - * / = and ? are the same.
> >> [...]
> >> But if we insist that every symbol we use is instantly recognisable and
> >> intuitively obvious to every programmer, we're putting the bar for
> >> acceptance impossibly high.
> >
> > I personally don't find "a ?? b" too bad (let's say I'm -0 about it)
> > but idioms such as "a?.b", "a ??= b" and "a?[3] ?? 4" look too
> > Perl-ish to me, non pythonic and overall not explicit, no matter what
> > the chosen symbol is gonna be.
>
> Please explain what is not explicit about it. "a?.b" is very simple
> and perfectly explicit: it means "None if a is None else a.b". What
> does "not explicit" mean, other than "I don't like this code"?

I find it less explicit mainly because it does 3 things at once: check
if attribute is None, use it if it's not None and continue the
evaluation from left to right. I find that logic to be more explicit
when living on different lines or is clearly delimited by keywords and
spaces. ? has no spaces, it's literally "variable names interrupted by
question marks" and evaluation can stop at any time while scanning the
line from left to right. Multiple "?" can live on the same line so
that's incentive to write one-liners, really, and to me one-liners are
always less explicit than the same logic split on multiple lines.

-- 
Giampaolo - http://grodola.blogspot.com

From hjp-python at hjp.at  Sun Jul 22 12:21:35 2018
From: hjp-python at hjp.at (Peter J. Holzer)
Date: Sun, 22 Jul 2018 18:21:35 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4b7hSgu28RwfwMav=cR-uxrErnGBDpeDVuirOevFvLo3g@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
 <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>
 <20180722121006.GQ8744@ando.pearwood.info>
 <CAEbHw4b7hSgu28RwfwMav=cR-uxrErnGBDpeDVuirOevFvLo3g@mail.gmail.com>
Message-ID: <20180722162135.pnp3kjmrm4hz4b7r@hjp.at>

On 2018-07-22 09:01:58 -0400, David Mertz wrote:
> On Sun, Jul 22, 2018, 8:11 AM Steven D'Aprano <steve at pearwood.info> wrote:
>     To me, the ?? operator seems like a clear and obvious win. The other
>     variants are more complex and the benefit is not as obvious to me, so I
>     haven't decided where I stand on them.
> 
> 
> Yes, that is the only one I'm merely -0 on. The others I'm -1000.

For me it's the opposite. ?? is +0: While I use the equivalent // in
Perl quite frequently, the benefit isn't all that great.

But ?. and ?[] are really useful.

Sequences like request.context.user.email occur quite frequently in
code, and I find 

    request?.context?.user?.email

much more readable than

    email = None
    context = request.context
    if context is not None:
        user = context.user
        if user is not None:
            email = user.email

Note that 

    request and request.context and request.context.user and request.context.user.email

is not equivalent even if you assume that None is the only possible
falsey value in this context. It evaluates request 4 times,
request.context 3 times, and request.context.user 2 times.

        hp


-- 
   _  | Peter J. Holzer    | we build much bigger, better disasters now
|_|_) |                    | because we have much more sophisticated
| |   | hjp at hjp.at         | management tools.
__/   | http://www.hjp.at/ | -- Ross Anderson <https://www.edge.org/>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180722/8ff0b301/attachment.sig>

From mike at selik.org  Sun Jul 22 13:33:23 2018
From: mike at selik.org (Michael Selik)
Date: Sun, 22 Jul 2018 10:33:23 -0700
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180722015418.GO8744@ando.pearwood.info>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
Message-ID: <CADqi796s3UpRe6RSvsw+2S0JpqBg-ALRsLga20=4WgbE5rDC7Q@mail.gmail.com>

On Sat, Jul 21, 2018, 6:55 PM Steven D'Aprano <steve at pearwood.info> wrote:

> On Sun, Jul 22, 2018 at 01:56:35AM +0200, Giampaolo Rodola' wrote:
> > On Thu, Jul 19, 2018 at 3:39 PM Steven D'Aprano <steve at pearwood.info>
> wrote:
> > > Tens of thousands of non-English speakers have had to learn the meaning
> > > of what might as well be meaningless, random sets of symbols (to them)
> > > like "class", "import", "while" and "True". If they can do so, perhaps
> > > we English-speakers should stop complaining about how hard it is to
> > > memorise the meaning of a couple of symbols like ??.
> >
> > "class", "import", "while" and "True" are keywords, not symbols.
>
> They are only key WORDS if you are an English speaker. If your language
> doesn't use the Latin script, they don't even look like words. They look
> like gibberish: ?????
>

Are you familiar with how people who don't speak English code? I'm curious
how they teach and use Python.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180722/3542bd5d/attachment.html>

From toddrjen at gmail.com  Sun Jul 22 15:03:42 2018
From: toddrjen at gmail.com (Todd)
Date: Sun, 22 Jul 2018 15:03:42 -0400
Subject: [Python-ideas] slice[] to get more complex slices
Message-ID: <CAFpSVpKnE8b=QrjxLy6anAF3ammq2QbB4_K1Ey02Vk3n8EVwyQ@mail.gmail.com>

For basic slices, the normal "slice(start, stop, step)" syntax works well.
But it becomes much more verbose to create more complicated slices that you
want to re-use for multiple multidimensional data structures, like numpy,
pandas, xarray, etc.

One idea I had was to allow creating slices by using indexing on the slice
class.  So for example:

    x = slice[5:1:-1, 10:20:2, 5:end]

Would be equivalent to:

    x = (slice(5, 1, -1), slice(10, 20, 2), slice(5, None))

Note that this wouldn't be done on a slice instance, it would be done on
the slice class.  The basic idea is that it would simply return whatever is
given to __getitem__.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180722/5b463d21/attachment.html>

From stefan_ml at behnel.de  Sun Jul 22 15:21:40 2018
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 22 Jul 2018 21:21:40 +0200
Subject: [Python-ideas] slice[] to get more complex slices
In-Reply-To: <CAFpSVpKnE8b=QrjxLy6anAF3ammq2QbB4_K1Ey02Vk3n8EVwyQ@mail.gmail.com>
References: <CAFpSVpKnE8b=QrjxLy6anAF3ammq2QbB4_K1Ey02Vk3n8EVwyQ@mail.gmail.com>
Message-ID: <pj2lc1$n0c$1@blaine.gmane.org>

Todd schrieb am 22.07.2018 um 21:03:
> For basic slices, the normal "slice(start, stop, step)" syntax works well.
> But it becomes much more verbose to create more complicated slices that you
> want to re-use for multiple multidimensional data structures, like numpy,
> pandas, xarray, etc.
> 
> One idea I had was to allow creating slices by using indexing on the slice
> class.  So for example:
> 
>     x = slice[5:1:-1, 10:20:2, 5:end]
> 
> Would be equivalent to:
> 
>     x = (slice(5, 1, -1), slice(10, 20, 2), slice(5, None))
> 
> Note that this wouldn't be done on a slice instance, it would be done on
> the slice class.  The basic idea is that it would simply return whatever is
> given to __getitem__.

Personally, I always likes that idea, but it would be worth looking up the
previous discussions in the list archive to find out if they lead to any
conclusion. AFAICT, "slice.literal" was the latest such proposal that was
discussed.

Stefan


From storchaka at gmail.com  Sun Jul 22 15:24:07 2018
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sun, 22 Jul 2018 22:24:07 +0300
Subject: [Python-ideas] slice[] to get more complex slices
In-Reply-To: <CAFpSVpKnE8b=QrjxLy6anAF3ammq2QbB4_K1Ey02Vk3n8EVwyQ@mail.gmail.com>
References: <CAFpSVpKnE8b=QrjxLy6anAF3ammq2QbB4_K1Ey02Vk3n8EVwyQ@mail.gmail.com>
Message-ID: <pj2lgk$c6c$1@blaine.gmane.org>

22.07.18 22:03, Todd ????:
> For basic slices, the normal "slice(start, stop, step)" syntax works 
> well.? But it becomes much more verbose to create more complicated 
> slices that you want to re-use for multiple multidimensional data 
> structures, like numpy, pandas, xarray, etc.
> 
> One idea I had was to allow creating slices by using indexing on the 
> slice class.? So for example:
> 
>  ??? x = slice[5:1:-1, 10:20:2, 5:end]
> 
> Would be equivalent to:
> 
>  ??? x = (slice(5, 1, -1), slice(10, 20, 2), slice(5, None))
> 
> Note that this wouldn't be done on a slice instance, it would be done on 
> the slice class.? The basic idea is that it would simply return whatever 
> is given to __getitem__.

See https://bugs.python.org/issue24379 .


From cpitclaudel at gmail.com  Sun Jul 22 15:58:34 2018
From: cpitclaudel at gmail.com (=?UTF-8?Q?Cl=c3=a9ment_Pit-Claudel?=)
Date: Sun, 22 Jul 2018 15:58:34 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180722121006.GQ8744@ando.pearwood.info>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
 <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>
 <20180722121006.GQ8744@ando.pearwood.info>
Message-ID: <f447925a-1ff0-0513-7e59-cd13293af462@gmail.com>

On 2018-07-22 08:10, Steven D'Aprano wrote:
> Indeed. And I think we ought to think carefully about the benefits and 
> costs of all of those variants separately.
> 
> To me, the ?? operator seems like a clear and obvious win. The other 
> variants are more complex and the benefit is not as obvious to me, so I 
> haven't decided where I stand on them.

I like the ?? operator too; I like the short circuiting behavior a lot, and the semantics are simple.

I guess I'd use the other operators fairly often, too, mostly in quick-n-dirty scripts.  The one place where I miss them is when browsing through dictionaries that I get by querying a remote server and deserializing the resulting JSON.  I foten have situation where the value I'm interested in is e.g. either in response[0]["addresses"]["workplace"]["email"], or in response["records"][0]["contactInfo"]["emails"][0], and any of these subrecords may be missing.

Rewriting these using the ?[?] and ?. operators, I guess I would write something like this:

tmp = response?.get("records")
try:
    tmp = tmp?[0]
except IndexError:
    tmp = None
tmp = tmp?.get("contactInfo")?.get("emails")
try:
    tmp = tmp?[0]
except IndexError:
    tmp = None

Is there a shorter way to write these with the "?[?]" and "?." operators?  I guess the difficulty is that I need to swallow index and key errors, not just the type errors that come from indexing into None.

For cases like the one above, I usually use something like nget(response, ["records"], [0], ["contactInfo"], ["emails"], [0]), where nget is defined as shown below (in this use case, the lack of short-circuiting isn't an issue):

def nget(obj, *fields, default=None):
    for field in fields:
        if obj is None:
            return default
        if isinstance(field, str):
            obj = getattr(obj, field, None)
        elif isinstance(field, list):
            try:
                obj = obj.__getitem__(field[0])
            except (TypeError, KeyError, IndexError):
                obj = None
    return obj

class Test():
    def __init__(self):
        self.x = [{"y": 42, "z": ["aBc", "def"]}, [1]]

a = Test()
print(nget(a, "x", [0], ["z"], [0], [1]))              # B
print(nget(a, "x", [0], ["y"]))                        # 42
print(nget(a, "z", [0], ["y"], default="not found"))   # not found
print(nget(a, "z", [57], ["y"], default="not found"))  # not found

It would probably not be hard to wrap this into a special object, to be able to write something like wrap(response)["records"][0]["contactInfo"]["emails"][0].unwrap(). "wrap" would change its argument into a proxy returning a special indexable variant of None on key errors, and that dictionary would also call "wrap" on the results of __getitem__.  Something like this:

class wrap():
    SENTINEL = object()

    def __init__(self, obj):
        self.obj = obj

    def unwrap(self):
        return self.obj

    def __getitem__(self, key):
        try:
            return wrap(self.obj.__getitem__(key))
        except (TypeError, AttributeError, KeyError):
            return wrap(None)

a = [{"y": 42, "z": ["aBc", "def"]}, [1]]
print(wrap(a)[0]["z"][0][1].unwrap())

I think that's more or less what pymaybe does, in fact.

Cheers,
Cl?ment.

From rosuav at gmail.com  Sun Jul 22 16:01:14 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 23 Jul 2018 06:01:14 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
 <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>
 <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
 <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
Message-ID: <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>

On Mon, Jul 23, 2018 at 1:09 AM, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
> On Sun, Jul 22, 2018 at 3:38 PM Chris Angelico <rosuav at gmail.com> wrote:
>>
>> On Sun, Jul 22, 2018 at 11:35 PM, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
>> > On Sun, Jul 22, 2018 at 2:10 PM Steven D'Aprano <steve at pearwood.info> wrote:
>> >>
>> >> On Sun, Jul 22, 2018 at 12:13:04PM +0200, Giampaolo Rodola' wrote:
>> >> > On Sun, Jul 22, 2018 at 3:55 AM Steven D'Aprano <steve at pearwood.info> wrote:
>> >> [...]
>> >> > > I don't think that "+" is harder to read than
>> >> > > "standard_mathematics_operators_numeric_addition"
>> >> >
>> >> >
>> >> > Please let's drop the argument that + - * / = and ? are the same.
>> >> [...]
>> >> But if we insist that every symbol we use is instantly recognisable and
>> >> intuitively obvious to every programmer, we're putting the bar for
>> >> acceptance impossibly high.
>> >
>> > I personally don't find "a ?? b" too bad (let's say I'm -0 about it)
>> > but idioms such as "a?.b", "a ??= b" and "a?[3] ?? 4" look too
>> > Perl-ish to me, non pythonic and overall not explicit, no matter what
>> > the chosen symbol is gonna be.
>>
>> Please explain what is not explicit about it. "a?.b" is very simple
>> and perfectly explicit: it means "None if a is None else a.b". What
>> does "not explicit" mean, other than "I don't like this code"?
>
> I find it less explicit mainly because it does 3 things at once: check
> if attribute is None, use it if it's not None and continue the
> evaluation from left to right. I find that logic to be more explicit
> when living on different lines or is clearly delimited by keywords and
> spaces. ? has no spaces, it's literally "variable names interrupted by
> question marks" and evaluation can stop at any time while scanning the
> line from left to right. Multiple "?" can live on the same line so
> that's incentive to write one-liners, really, and to me one-liners are
> always less explicit than the same logic split on multiple lines.

Ah, I see what you mean. Well, think about what actually happens when
you write "lst.sort()". In terms of "hidden behaviour", there is far
FAR more of it in existing syntax than in the new proposals. Which is
back to what Steven said: people demand such a high bar for new syntax
that few existing pieces of syntax would pass it.

ChrisA

From g.rodola at gmail.com  Sun Jul 22 16:43:15 2018
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Sun, 22 Jul 2018 22:43:15 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
 <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>
 <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
 <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
Message-ID: <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>

On Sun, Jul 22, 2018 at 10:01 PM Chris Angelico <rosuav at gmail.com> wrote:
>
> On Mon, Jul 23, 2018 at 1:09 AM, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
> > On Sun, Jul 22, 2018 at 3:38 PM Chris Angelico <rosuav at gmail.com> wrote:
> > I find it less explicit mainly because it does 3 things at once: check
> > if attribute is None, use it if it's not None and continue the
> > evaluation from left to right. I find that logic to be more explicit
> > when living on different lines or is clearly delimited by keywords and
> > spaces. ? has no spaces, it's literally "variable names interrupted by
> > question marks" and evaluation can stop at any time while scanning the
> > line from left to right. Multiple "?" can live on the same line so
> > that's incentive to write one-liners, really, and to me one-liners are
> > always less explicit than the same logic split on multiple lines.
>
> Ah, I see what you mean. Well, think about what actually happens when
> you write "lst.sort()". In terms of "hidden behaviour", there is far
> FAR more of it in existing syntax than in the new proposals.

I am not sure I'm following you (what does lst.sort() have to do with "?"?).

> Which is back to what Steven said: people demand such a high
> bar for new syntax that few existing pieces of syntax would pass it.

Probably. That's what happens when a language is mature. Personally I
don't think that's a bad thing.

-- 
Giampaolo - http://grodola.blogspot.com

From solipsis at pitrou.net  Sun Jul 22 16:49:32 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 22 Jul 2018 22:49:32 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
 <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>
 <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
 <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
Message-ID: <20180722224932.502e0cdb@fsol>

On Sun, 22 Jul 2018 22:43:15 +0200
"Giampaolo Rodola'" <g.rodola at gmail.com>
wrote:

> On Sun, Jul 22, 2018 at 10:01 PM Chris Angelico <rosuav at gmail.com> wrote:
> >
> > On Mon, Jul 23, 2018 at 1:09 AM, Giampaolo Rodola' <g.rodola at gmail.com> wrote:  
> > > On Sun, Jul 22, 2018 at 3:38 PM Chris Angelico <rosuav at gmail.com> wrote:
> > > I find it less explicit mainly because it does 3 things at once: check
> > > if attribute is None, use it if it's not None and continue the
> > > evaluation from left to right. I find that logic to be more explicit
> > > when living on different lines or is clearly delimited by keywords and
> > > spaces. ? has no spaces, it's literally "variable names interrupted by
> > > question marks" and evaluation can stop at any time while scanning the
> > > line from left to right. Multiple "?" can live on the same line so
> > > that's incentive to write one-liners, really, and to me one-liners are
> > > always less explicit than the same logic split on multiple lines.  
> >
> > Ah, I see what you mean. Well, think about what actually happens when
> > you write "lst.sort()". In terms of "hidden behaviour", there is far
> > FAR more of it in existing syntax than in the new proposals.  
> 
> I am not sure I'm following you (what does lst.sort() have to do with "?"?).
> 
> > Which is back to what Steven said: people demand such a high
> > bar for new syntax that few existing pieces of syntax would pass it.  
> 
> Probably. That's what happens when a language is mature. Personally I
> don't think that's a bad thing.

Agreed with Giampaolo.  The opportunities for syntax additions should
become rarer and rarer.

Regards

Antoine.



From rosuav at gmail.com  Sun Jul 22 16:53:53 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 23 Jul 2018 06:53:53 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
 <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>
 <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
 <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
Message-ID: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>

On Mon, Jul 23, 2018 at 6:43 AM, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
> On Sun, Jul 22, 2018 at 10:01 PM Chris Angelico <rosuav at gmail.com> wrote:
>>
>> On Mon, Jul 23, 2018 at 1:09 AM, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
>> > On Sun, Jul 22, 2018 at 3:38 PM Chris Angelico <rosuav at gmail.com> wrote:
>> > I find it less explicit mainly because it does 3 things at once: check
>> > if attribute is None, use it if it's not None and continue the
>> > evaluation from left to right. I find that logic to be more explicit
>> > when living on different lines or is clearly delimited by keywords and
>> > spaces. ? has no spaces, it's literally "variable names interrupted by
>> > question marks" and evaluation can stop at any time while scanning the
>> > line from left to right. Multiple "?" can live on the same line so
>> > that's incentive to write one-liners, really, and to me one-liners are
>> > always less explicit than the same logic split on multiple lines.
>>
>> Ah, I see what you mean. Well, think about what actually happens when
>> you write "lst.sort()". In terms of "hidden behaviour", there is far
>> FAR more of it in existing syntax than in the new proposals.
>
> I am not sure I'm following you (what does lst.sort() have to do with "?"?).

The "." in "lst.sort" is an operator. How much hidden behaviour is
there in that? Do you actually even know every possible thing that can
happen? Don't feel bad if you don't - it's not an indictment of your
quality as a programmer, but an acknowledgement that Python's
attribute access is incredibly complicated.

>> Which is back to what Steven said: people demand such a high
>> bar for new syntax that few existing pieces of syntax would pass it.
>
> Probably. That's what happens when a language is mature. Personally I
> don't think that's a bad thing.

I do. It means people place crazily high demands on new proposals.
Imagine if we were talking about people, rather than features in a
language; imagine if, to join the Warriors Guild, you had to first
slay a red dragon with nothing but a rusty dagger, despite none of the
existing members having done so. Is that reasonable to ask? Can you
say "well, the guild is mature now, so yeah, it's a good thing"?

ChrisA

From hjp-python at hjp.at  Sun Jul 22 16:54:36 2018
From: hjp-python at hjp.at (Peter J. Holzer)
Date: Sun, 22 Jul 2018 22:54:36 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CADqi796s3UpRe6RSvsw+2S0JpqBg-ALRsLga20=4WgbE5rDC7Q@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
 <CADqi796s3UpRe6RSvsw+2S0JpqBg-ALRsLga20=4WgbE5rDC7Q@mail.gmail.com>
Message-ID: <20180722205436.u2aovg3ul7rvpk7s@hjp.at>

On 2018-07-22 10:33:23 -0700, Michael Selik wrote:
> On Sat, Jul 21, 2018, 6:55 PM Steven D'Aprano <steve at pearwood.info> wrote:
>     On Sun, Jul 22, 2018 at 01:56:35AM +0200, Giampaolo Rodola' wrote:
>     > On Thu, Jul 19, 2018 at 3:39 PM Steven D'Aprano <steve at pearwood.info>
>     > wrote:
>     > > Tens of thousands of non-English speakers have had to learn the meaning
>     > > of what might as well be meaningless, random sets of symbols (to them)
>     > > like "class", "import", "while" and "True". If they can do so, perhaps
>     > > we English-speakers should stop complaining about how hard it is to
>     > > memorise the meaning of a couple of symbols like ??.
>     >
>     > "class", "import", "while" and "True" are keywords, not symbols.
> 
>     They are only key WORDS if you are an English speaker.

They are also words if you are not an English speaker. I don't speak
Chinese, but "P?tonghu?" is certainly a word for me (although I wouldn't
recognize ??? as that word).

>     If your language
>     doesn't use the Latin script, they don't even look like words. They look
>     like gibberish: ?????
> 
> 
> Are you familiar with how people who don't speak English code? I'm curious how
> they teach and use Python.

I know a few people who don't know enough English to read English
documentation. But AFAIK they didn't have a problem memorizing a few
dozen keywords. Learning the semantics of a programming language is a
much larger task than learning a few words, and having familiar keywords
probably doesn't really help much (they still don't mean what they mean
in English).

Of course we do use the Latin alphabet, I don't know how somebody who
had to learn the Latin alphabet specifically for programming would cope.
It's probably like programming in APL.

I guess I could learn PerlYuYan[1] to find out ;-).

        hp

[1] https://metacpan.org/pod/Lingua::Sinica::PerlYuYan


-- 
   _  | Peter J. Holzer    | we build much bigger, better disasters now
|_|_) |                    | because we have much more sophisticated
| |   | hjp at hjp.at         | management tools.
__/   | http://www.hjp.at/ | -- Ross Anderson <https://www.edge.org/>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180722/0f139844/attachment-0001.sig>

From solipsis at pitrou.net  Sun Jul 22 17:10:02 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 22 Jul 2018 23:10:02 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
 <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>
 <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
 <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
Message-ID: <20180722231002.0c8f0812@fsol>

On Mon, 23 Jul 2018 06:53:53 +1000
Chris Angelico <rosuav at gmail.com> wrote:
> 
> >> Which is back to what Steven said: people demand such a high
> >> bar for new syntax that few existing pieces of syntax would pass it.  
> >
> > Probably. That's what happens when a language is mature. Personally I
> > don't think that's a bad thing.  
> 
> I do. It means people place crazily high demands on new proposals.

"Crazy" is just a personal judgement.  I do think high (and even very
high) demands are entirely justified by the language's maturity.

> Imagine if we were talking about people, rather than features in a
> language; imagine if, to join the Warriors Guild, you had to first
> slay a red dragon with nothing but a rusty dagger, despite none of the
> existing members having done so.

This is the silliest analogy I have seen on this list for a very long
time.

If someone thinks getting their PEP accepted is like belonging to a
? Warriors Guild ? (whatever that is in the real world), I'd question
their motivations for contributing.

Regards

Antoine.



From mertz at gnosis.cx  Sun Jul 22 17:38:52 2018
From: mertz at gnosis.cx (David Mertz)
Date: Sun, 22 Jul 2018 17:38:52 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
 <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>
 <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
 <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
Message-ID: <CAEbHw4ZK0TF75JPaXM_b7sFGpiwqTmA9SVst2tNjPM3xU7ajww@mail.gmail.com>

On Sun, Jul 22, 2018, 4:56 PM Chris Angelico <rosuav at gmail.com> wrote:

> It means people place crazily high demands on new proposals.
>

I think the bar has been much too low for introducing new features over the
last 5 years or so. Internal changes like the new dictionary implementation
are fine, but user-facing changes should be exceedingly rare in the base
language. This proposal doesn't come remotely close to such a good standard.

I was consistently +0 on the 572 idea, as long as its worst excesses were
trimmed, as in the final PEP. But after reading this discussion, I almost
reconsider that opinion since its social effect seems to be a move towards
accepting wild and unnecessary changes that "might be useful" for a few
unusual programming patterns.

Honestly, if you want Perl, and as many different ways to approach each
problem as there are programmers (each with their own syntax niche), that
language continues to be fully working. I'm not even writing that to be
dismissive... There are actually some pretty and interesting ideas over
there. But I very much want Python not to be like that, and to do most of
my work in a readable language with as few special characters/signils as
feasible.

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180722/63c92374/attachment.html>

From g.rodola at gmail.com  Sun Jul 22 17:51:07 2018
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Sun, 22 Jul 2018 23:51:07 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
 <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>
 <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
 <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
Message-ID: <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>

On Sun, Jul 22, 2018 at 10:55 PM Chris Angelico <rosuav at gmail.com> wrote:
>
> On Mon, Jul 23, 2018 at 6:43 AM, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
> > On Sun, Jul 22, 2018 at 10:01 PM Chris Angelico <rosuav at gmail.com> wrote:
> >>
> >> On Mon, Jul 23, 2018 at 1:09 AM, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
> >> > On Sun, Jul 22, 2018 at 3:38 PM Chris Angelico <rosuav at gmail.com> wrote:
> >> > I find it less explicit mainly because it does 3 things at once: check
> >> > if attribute is None, use it if it's not None and continue the
> >> > evaluation from left to right. I find that logic to be more explicit
> >> > when living on different lines or is clearly delimited by keywords and
> >> > spaces. ? has no spaces, it's literally "variable names interrupted by
> >> > question marks" and evaluation can stop at any time while scanning the
> >> > line from left to right. Multiple "?" can live on the same line so
> >> > that's incentive to write one-liners, really, and to me one-liners are
> >> > always less explicit than the same logic split on multiple lines.
> >>
> >> Ah, I see what you mean. Well, think about what actually happens when
> >> you write "lst.sort()". In terms of "hidden behaviour", there is far
> >> FAR more of it in existing syntax than in the new proposals.
> >
> > I am not sure I'm following you (what does lst.sort() have to do with "?"?).
>
> The "." in "lst.sort" is an operator. How much hidden behaviour is
> there in that? Do you actually even know every possible thing that can
> happen? Don't feel bad if you don't - it's not an indictment of your
> quality as a programmer, but an acknowledgement that Python's
> attribute access is incredibly complicated.

I'm going to engage into a discussion about the analogy between "?"
and "." because simply there is none. It doesn't prove anything except
that you're not really interested in having a serious discussion about
the pros and cons of this PEP: you just want it to happen no matter
what.

> Imagine if we were talking about people, rather than features in a
> language; imagine if, to join the Warriors Guild, you had to first
> slay a red dragon with nothing but a rusty dagger, despite none of the
> existing members having done so. Is that reasonable to ask? Can you
> say "well, the guild is mature now, so yeah, it's a good thing"?

Ditto.

-- 
Giampaolo - http://grodola.blogspot.com

From g.rodola at gmail.com  Sun Jul 22 17:52:49 2018
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Sun, 22 Jul 2018 23:52:49 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
 <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>
 <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
 <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
Message-ID: <CAFYqXL9z2SLUuPJZkS62z_sEVJhB7BVitArs=8qBjDvxSfssqQ@mail.gmail.com>

On Sun, Jul 22, 2018 at 11:51 PM Giampaolo Rodola' <g.rodola at gmail.com> wrote:
>
> On Sun, Jul 22, 2018 at 10:55 PM Chris Angelico <rosuav at gmail.com> wrote:
> >
> > On Mon, Jul 23, 2018 at 6:43 AM, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
> > > On Sun, Jul 22, 2018 at 10:01 PM Chris Angelico <rosuav at gmail.com> wrote:
> > >>
> > >> On Mon, Jul 23, 2018 at 1:09 AM, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
> > >> > On Sun, Jul 22, 2018 at 3:38 PM Chris Angelico <rosuav at gmail.com> wrote:
> > >> > I find it less explicit mainly because it does 3 things at once: check
> > >> > if attribute is None, use it if it's not None and continue the
> > >> > evaluation from left to right. I find that logic to be more explicit
> > >> > when living on different lines or is clearly delimited by keywords and
> > >> > spaces. ? has no spaces, it's literally "variable names interrupted by
> > >> > question marks" and evaluation can stop at any time while scanning the
> > >> > line from left to right. Multiple "?" can live on the same line so
> > >> > that's incentive to write one-liners, really, and to me one-liners are
> > >> > always less explicit than the same logic split on multiple lines.
> > >>
> > >> Ah, I see what you mean. Well, think about what actually happens when
> > >> you write "lst.sort()". In terms of "hidden behaviour", there is far
> > >> FAR more of it in existing syntax than in the new proposals.
> > >
> > > I am not sure I'm following you (what does lst.sort() have to do with "?"?).
> >
> > The "." in "lst.sort" is an operator. How much hidden behaviour is
> > there in that? Do you actually even know every possible thing that can
> > happen? Don't feel bad if you don't - it's not an indictment of your
> > quality as a programmer, but an acknowledgement that Python's
> > attribute access is incredibly complicated.
>
> I'm going to engage into a discussion [...]

s/I'm going/I'm not going

-- 
Giampaolo - http://grodola.blogspot.com

From rosuav at gmail.com  Sun Jul 22 18:07:36 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 23 Jul 2018 08:07:36 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
 <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>
 <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
 <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
Message-ID: <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>

On Mon, Jul 23, 2018 at 7:51 AM, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
> On Sun, Jul 22, 2018 at 10:55 PM Chris Angelico <rosuav at gmail.com> wrote:
>>
>> On Mon, Jul 23, 2018 at 6:43 AM, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
>> > On Sun, Jul 22, 2018 at 10:01 PM Chris Angelico <rosuav at gmail.com> wrote:
>> >>
>> >> On Mon, Jul 23, 2018 at 1:09 AM, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
>> >> > On Sun, Jul 22, 2018 at 3:38 PM Chris Angelico <rosuav at gmail.com> wrote:
>> >> > I find it less explicit mainly because it does 3 things at once: check
>> >> > if attribute is None, use it if it's not None and continue the
>> >> > evaluation from left to right. I find that logic to be more explicit
>> >> > when living on different lines or is clearly delimited by keywords and
>> >> > spaces. ? has no spaces, it's literally "variable names interrupted by
>> >> > question marks" and evaluation can stop at any time while scanning the
>> >> > line from left to right. Multiple "?" can live on the same line so
>> >> > that's incentive to write one-liners, really, and to me one-liners are
>> >> > always less explicit than the same logic split on multiple lines.
>> >>
>> >> Ah, I see what you mean. Well, think about what actually happens when
>> >> you write "lst.sort()". In terms of "hidden behaviour", there is far
>> >> FAR more of it in existing syntax than in the new proposals.
>> >
>> > I am not sure I'm following you (what does lst.sort() have to do with "?"?).
>>
>> The "." in "lst.sort" is an operator. How much hidden behaviour is
>> there in that? Do you actually even know every possible thing that can
>> happen? Don't feel bad if you don't - it's not an indictment of your
>> quality as a programmer, but an acknowledgement that Python's
>> attribute access is incredibly complicated.
>
> I'm [not] going to engage into a discussion about the analogy between "?"
> and "." because simply there is none. It doesn't prove anything except
> that you're not really interested in having a serious discussion about
> the pros and cons of this PEP: you just want it to happen no matter
> what.

That's because the dot already exists in the language, and you have
become so accustomed to it that you don't see it any more. You've just
proven my point.

ChrisA

From g.rodola at gmail.com  Sun Jul 22 18:59:20 2018
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Mon, 23 Jul 2018 00:59:20 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
 <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>
 <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
 <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
Message-ID: <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>

On Mon, Jul 23, 2018 at 12:08 AM Chris Angelico <rosuav at gmail.com> wrote:
>
> On Mon, Jul 23, 2018 at 7:51 AM, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
> > On Sun, Jul 22, 2018 at 10:55 PM Chris Angelico <rosuav at gmail.com> wrote:
> >>
> >> On Mon, Jul 23, 2018 at 6:43 AM, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
> >> > On Sun, Jul 22, 2018 at 10:01 PM Chris Angelico <rosuav at gmail.com> wrote:
> >> >>
> >> >> On Mon, Jul 23, 2018 at 1:09 AM, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
> >> >> > On Sun, Jul 22, 2018 at 3:38 PM Chris Angelico <rosuav at gmail.com> wrote:
> >> >> > I find it less explicit mainly because it does 3 things at once: check
> >> >> > if attribute is None, use it if it's not None and continue the
> >> >> > evaluation from left to right. I find that logic to be more explicit
> >> >> > when living on different lines or is clearly delimited by keywords and
> >> >> > spaces. ? has no spaces, it's literally "variable names interrupted by
> >> >> > question marks" and evaluation can stop at any time while scanning the
> >> >> > line from left to right. Multiple "?" can live on the same line so
> >> >> > that's incentive to write one-liners, really, and to me one-liners are
> >> >> > always less explicit than the same logic split on multiple lines.
> >> >>
> >> >> Ah, I see what you mean. Well, think about what actually happens when
> >> >> you write "lst.sort()". In terms of "hidden behaviour", there is far
> >> >> FAR more of it in existing syntax than in the new proposals.
> >> >
> >> > I am not sure I'm following you (what does lst.sort() have to do with "?"?).
> >>
> >> The "." in "lst.sort" is an operator. How much hidden behaviour is
> >> there in that? Do you actually even know every possible thing that can
> >> happen? Don't feel bad if you don't - it's not an indictment of your
> >> quality as a programmer, but an acknowledgement that Python's
> >> attribute access is incredibly complicated.
> >
> > I'm [not] going to engage into a discussion about the analogy between "?"
> > and "." because simply there is none. It doesn't prove anything except
> > that you're not really interested in having a serious discussion about
> > the pros and cons of this PEP: you just want it to happen no matter
> > what.
>
> That's because the dot already exists in the language, and you have
> become so accustomed to it that you don't see it any more. You've just
> proven my point.

You're back at "since we have X that justifies the addition of Y" [1]
and AFAICT that's the only argument you have provided so far in a 100+
messages discussion.

[1] https://mail.python.org/pipermail/python-ideas/2018-July/052068.html

-- 
Giampaolo - http://grodola.blogspot.com

From steve at pearwood.info  Sun Jul 22 20:51:15 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 23 Jul 2018 10:51:15 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPTjJmr8cMiW_K1jZwa4WJ_Tmy406+MxrXF_XR+OLWc0W47r3w@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
 <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>
 <20180722121006.GQ8744@ando.pearwood.info>
 <CAEbHw4b7hSgu28RwfwMav=cR-uxrErnGBDpeDVuirOevFvLo3g@mail.gmail.com>
 <6562164f-9be1-40ec-890d-508fd5c81014@googlegroups.com>
 <CAPTjJmr8cMiW_K1jZwa4WJ_Tmy406+MxrXF_XR+OLWc0W47r3w@mail.gmail.com>
Message-ID: <20180723005114.GT8744@ando.pearwood.info>

On Sun, Jul 22, 2018 at 11:26:15PM +1000, Chris Angelico wrote:

> You forget that the operator will *short-circuit*. It will not
> evaluate the second argument if the first argument is None. You cannot
> do this with a function, other than with a hack like a lambda
> function.

We keep running up to this issue. What if there was a language 
supported, non-hackish way to officially delay evaluation of 
expressions until explicitly requested?

That would allow us to write a function:

func(arg, default = delayed-expression)

and avoid new punctuation. Then the only bike-shedding will be where the 
function should live and what it is called and how many arguments it 
ought to take...


-- 
Steve

From steve at pearwood.info  Sun Jul 22 21:11:22 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 23 Jul 2018 11:11:22 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
 <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>
 <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
 <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
Message-ID: <20180723011121.GU8744@ando.pearwood.info>

On Sun, Jul 22, 2018 at 05:09:39PM +0200, Giampaolo Rodola' wrote:

> > > I personally don't find "a ?? b" too bad (let's say I'm -0 about it)
> > > but idioms such as "a?.b", "a ??= b" and "a?[3] ?? 4" look too
> > > Perl-ish to me, non pythonic and overall not explicit, no matter what
> > > the chosen symbol is gonna be.
> >
> > Please explain what is not explicit about it. "a?.b" is very simple
> > and perfectly explicit: it means "None if a is None else a.b". What
> > does "not explicit" mean, other than "I don't like this code"?
> 
> I find it less explicit mainly because it does 3 things at once: check
> if attribute is None, use it if it's not None and continue the
> evaluation from left to right. I find that logic to be more explicit
> when living on different lines or is clearly delimited by keywords and
> spaces.

Does this mean that instead of writing:

    result = obj.attr + 1

you prefer to be "explicit" and split it over multiple lines?

    # named functions are always better than punctuation
    from operator import add  
    # explicit is better than punctuation
    value = getattr(obj, "attr")  
    result = add(value, 1)


> ? has no spaces, it's literally "variable names interrupted by
> question marks" and evaluation can stop at any time while scanning the
> line from left to right.

Just like ordinary attribute access.

This is the point I was making earlier: you accept existing punctuation 
doing these things:

    try:
        obj.spam.egsg.tomato.cheese  # oops a typo
    except AttributeError:
        # evaluation can stop at any time
        ...

while demanding a higher standard for new punctuation.

All of your criticisms of ? punctuation applies to . as well. Do you 
think that Python is a worse language than it should have been because 
we use . for attribute access?


> Multiple "?" can live on the same line so
> that's incentive to write one-liners, really, and to me one-liners are
> always less explicit than the same logic split on multiple lines.

Explicit is not always better.

    import this

is much better than:


    for location in sys.path:
        try:
            for file in os.listdir(location):
                if os.splitext(file) in ('.pyc', '.py', '.so'):
                    ...


etc. There is a HUGE amount of implicit complexity and work done behind 
the scenes in every import, and we're happy to keep it that way.


-- 
Steve

From steve at pearwood.info  Sun Jul 22 21:36:14 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 23 Jul 2018 11:36:14 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
References: <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
 <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
Message-ID: <20180723013614.GV8744@ando.pearwood.info>

On Mon, Jul 23, 2018 at 12:59:20AM +0200, Giampaolo Rodola' wrote:

> You're back at "since we have X that justifies the addition of Y" [1]
> and AFAICT that's the only argument you have provided so far in a 100+
> messages discussion.

The PEP itself justifies the addition of Y.

Chris' argument, and mine, is countering *your* arguments in opposition. 
It is not a positive argument for Y, since the PEP does an admirable job 
at that. It is a response to the FUD (Fear, Uncertainty, Doubt) that 
Python is becoming "Perl-like", or even more ludicrously, like APL and 
J. (That's David Mertz' position.)

To my eyes, your opposition basically comes down to "it is new, and I 
don't like it because it is new". It looks to me like pure resistance to 
change simply due to dislike of change. See also David's response that 
he is against the changes made to Python over the last five years.

The concrete arguments you are making against this change apply equally 
to existing features. If your (and others') arguments are valid now, 
they would have been equally valid back in Python 1.5. If punctuation is 
unreadable and Perlish, so is "." and ":" punctuation. If ?? is bad 
because it is "implicit", then so is import or sorted.

Arguments by slogan ("explicit is better than implicit") are rarely good 
arguments -- especially when nobody seems to be able to define implicit 
and explicit explicitly.

As for the argument that Python is "mature" and so we should resist 
change, we could have said the same thing going all the way back to 
Python 1.5 and probably beyond. Have you tried using Python 1.5 
recently? Or even Python 2.4. I have. I wonder how I managed to get 
anything useful done.

Some of us think that Python 3.6 or 3.7 is fantastic and so good that 
every addition to the language can only make it worse. I suggest that 
when we have Python 4.5 or 4.6, we will wonder how on earth we managed 
to get any useful work done with Python 3.7.

The Python community has always been conservative and resistant to 
change, but the level of conservativeness is now pushing towards fear 
of change rather than justifiable caution about adding new features that 
cannot easily be reverted.


-- 
Steve

From nicholas.cole at gmail.com  Mon Jul 23 02:23:31 2018
From: nicholas.cole at gmail.com (Nicholas Cole)
Date: Mon, 23 Jul 2018 07:23:31 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180723013614.GV8744@ando.pearwood.info>
References: <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
 <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
Message-ID: <CAAu18hc35N11jVQj+0691aH8Oe2+4Cqfn0yFpwWtjqou161ATg@mail.gmail.com>

On Mon, Jul 23, 2018 at 2:38 AM Steven D'Aprano <steve at pearwood.info> wrote:
>
> On Mon, Jul 23, 2018 at 12:59:20AM +0200, Giampaolo Rodola' wrote:
>
> > You're back at "since we have X that justifies the addition of Y" [1]
> > and AFAICT that's the only argument you have provided so far in a 100+
> > messages discussion.
>
> The PEP itself justifies the addition of Y.

This thread seems to be unnecessarily heated.

Other languages have these operators, and so they aren't a wild idea.
That said, from what I've seen, Swift optionals are very different
things, and Python really has nothing like them.

In Python, is None really special enough to need an operator like this?

One issue for me is that the trivial case is already a one-liner:

if a is None: a = 10

And it works for other things too:

if a is -1: a = 10

if not a: a = 10

Again, is None special enough in Python to need this operator? I don't know.

And that leads to a simple question: how many times does this actually
occur in real-world by python code? -- i.e. how many times do I want
to check the value of an existing label, and, finding it is None (and
None specifically), then assign it a value?

What does a scan through the existing core library say?

I'm +0 on this proposal.

Best wishes,

N

From gregory.lielens at gmail.com  Mon Jul 23 03:08:40 2018
From: gregory.lielens at gmail.com (=?UTF-8?Q?Gr=C3=A9gory_Lielens?=)
Date: Mon, 23 Jul 2018 00:08:40 -0700 (PDT)
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAAu18hc35N11jVQj+0691aH8Oe2+4Cqfn0yFpwWtjqou161ATg@mail.gmail.com>
References: <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
 <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAAu18hc35N11jVQj+0691aH8Oe2+4Cqfn0yFpwWtjqou161ATg@mail.gmail.com>
Message-ID: <1898d063-8179-423d-988a-01afa1b75075@googlegroups.com>



On Monday, July 23, 2018 at 8:24:45 AM UTC+2, Nicholas Cole wrote:

And that leads to a simple question: how many times does this actually 
> occur in real-world by python code? -- i.e. how many times do I want 
> to check the value of an existing label, and, finding it is None (and 
> None specifically), then assign it a value? 
>

The PEP present a few examples.

I think the compactness and clarity is really gained when doing a descent 
into an "attribute tree", where any (or selected) members can be None, like 
getting back to the  example by Steven (spell-corrected, and expanded):

 meal = obj?.spam?.eggs.tomato?.cheese ?? "Mozzarella"

Where, to put the proposal in even better light, I have assumed that eggs 
instances always have a tomato attributes (i.e. are never None).

This is indeed much more compact that the current version, and arguably 
more readable (it's easier to parse once you know the syntax...but you have 
to know the special syntax that will be used for this kind of stuff only). 
The more you deepen the attribute access, the more you gain, but of course 
deep attribute access is not so common. The new operators may make deep 
attribute hierarchies (or deeply nested dicts/lists) slightly more common, 
and there also I do not think it's a good thing.

For me the issue is that you gain little, each operator is very limited in 
scope, but at the same it's a operator, and those are introduced very 
carefully in Python (the resistance to line noise is high, it's one of the 
defining aspect of Python since forever).

Each of them do not do enough IMHO, it's too special case and do not unlock 
especially powerfull/elegant/generalization reuse of other part of Python 
syntax. If you add something, it should play nice with the existing feature 
and solve nagging issues that broaden its appeal, or be powerful/general 
enough to justify it's existence on its own, or be so self-evident for a 
python user that it feels it's solving a bug.

It's not the case for ?? and friends imho, far from it.

Also, what happen if you want to modify your deep nested tree, not only 
access it?

obj?.spam?.eggs.tomato?.cheese = "Mozzarella"

will not work, and the proposal will not really help there (Or will it? I 
am not the proposer, I am strongly against it, so I could easily miss 
capabilities).
If it's not possible, you lose a kind of symmetry, and that I do not like 
(orthogonality and symmetries are really nice in in a computer language, it 
helps quickly remembering the syntax as you are not cluttered by special 
cases and exceptions)

Steven wrote:


The Python community has always been conservative and resistant to 
change, but the level of conservativeness is now pushing towards fear 
of change rather than justifiable caution about adding new features that 
cannot easily be reverted. 

That's probably because you like this proposal. Really, compared to the 2.0 
days, it seems that Python has became much less resistant to change. You 
have resistance even when proposing changes in standard library, or even 
the C interpreter internal, unseen in the Python layer, and imho it's often 
justified (what can be discussed is how arbitrary the actual selection of 
change is, but a general resistance is not a bad thing)
It is a little bit optimistic to expect ?? and friends will be a walk in 
the park, especially after := ...


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/bb745a48/attachment.html>

From gregory.lielens at gmail.com  Mon Jul 23 04:41:03 2018
From: gregory.lielens at gmail.com (=?UTF-8?Q?Gr=C3=A9gory_Lielens?=)
Date: Mon, 23 Jul 2018 01:41:03 -0700 (PDT)
Subject: [Python-ideas] slice[] to get more complex slices
In-Reply-To: <pj2lgk$c6c$1@blaine.gmane.org>
References: <CAFpSVpKnE8b=QrjxLy6anAF3ammq2QbB4_K1Ey02Vk3n8EVwyQ@mail.gmail.com>
 <pj2lgk$c6c$1@blaine.gmane.org>
Message-ID: <09cb9784-6a97-49b9-b5e7-1c9a9d8c8391@googlegroups.com>

+1 on this. Seems easy a limited in scope, and it reduce the need to 
remember the details of slice constructor, just reuse the same syntax for 
getitem slices.

Seems that the initial proposal was skipped just for timing reasons, and 
have a large support among numpy users. More mixed outside numpy, but still 
I think it was put under the rug a little too quickly...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/2ed311e1/attachment.html>

From jfine2358 at gmail.com  Mon Jul 23 05:03:10 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Mon, 23 Jul 2018 10:03:10 +0100
Subject: [Python-ideas] Python docs page: In what ways is None special
Message-ID: <CALD=Yf_PRXZYiNdC9FGE5pd1cjrS8gyr8q5iPxzrMKXzX027YA@mail.gmail.com>

This arises out of PEP 505 - None-aware operators.

I thought, a page on how None is special would be nice.
I've not found such a page on the web. We do have
===
https://docs.python.org/3/library/constants.html
None
The sole value of the type NoneType. None is
frequently used to represent the absence of a
value, as when default arguments are not passed
to a function. Assignments to None are illegal
and raise a SyntaxError.
===

So decided to start writing such a page, perhaps to be
added to the docs.  All code examples in Python3.4.

Here's my first attempt. I'm sure I've missed some
important examples. Please help, if you can.

None is a keyword
==============
>>> None = 0
SyntaxError: can't assign to keyword

>>> import keyword
>>> keyword.kwlist
['False', 'None', 'True', 'and', 'as', 'assert', 'break', 'class',
'continue', 'def', 'del', 'elif', 'else', 'except', 'finally', 'for',
'from', 'global', 'if', 'import', 'in', 'is', 'lambda', 'nonlocal',
'not', 'or', 'pass', 'raise', 'return', 'try', 'while', 'with', 'yield']

The Command Line Interpreter hides None
=================================
>>> None
>>>

None is false in a boolean context
==========================
>>> bool(None)
False

Procedures return None
==================
>>> a = [3,1,2]
>>> b = a.sort()
>>> a, b
([1, 2, 3], None)

Dictionary get returns None if not found
==============================
>>> {}.get('dne') is None
True

None is default return value
=====================
>>> def fn(): pass
...
>>> fn() # No response!
>>> print(fn()) # Here's why.
None

None is used as a sentinel default value
==============================
Particularly useful when default value must be determined
in body of function.
---
def insort_right(a, x, lo=0, hi=None):
    # ...
    if hi is None:
        hi = len(a)
---

Thank you for your attention. What have I missed?

-- 
Jonathan

From steve.dower at python.org  Mon Jul 23 05:16:37 2018
From: steve.dower at python.org (Steve Dower)
Date: Mon, 23 Jul 2018 10:16:37 +0100
Subject: [Python-ideas] Python docs page: In what ways is None special
In-Reply-To: <CALD=Yf_PRXZYiNdC9FGE5pd1cjrS8gyr8q5iPxzrMKXzX027YA@mail.gmail.com>
References: <CALD=Yf_PRXZYiNdC9FGE5pd1cjrS8gyr8q5iPxzrMKXzX027YA@mail.gmail.com>
Message-ID: <d7651538-1831-b7c7-f050-252826591ac2@python.org>

On 23Jul2018 1003, Jonathan Fine wrote:
> This arises out of PEP 505 - None-aware operators.
> 
> I thought, a page on how None is special would be nice.
> I've not found such a page on the web. We do have
> ===
> https://docs.python.org/3/library/constants.html
> None
> The sole value of the type NoneType. None is
> frequently used to represent the absence of a
> value, as when default arguments are not passed
> to a function. Assignments to None are illegal
> and raise a SyntaxError.
> ===
> 
> So decided to start writing such a page, perhaps to be
> added to the docs.  All code examples in Python3.4.

There's also 
https://docs.python.org/3/c-api/none.html?highlight=py_none#c.Py_None

"The Python None object, denoting lack of value. This object has no 
methods. It needs to be treated just like any other object with respect 
to reference counts."

I don't know that documenting the behaviours of None are that 
interesting (e.g. not displaying anything at the interactive prompt), 
though it'd be perfect for a blog and/or conference talk. But if there 
appear to be behaviours that are not consistent or cannot be easily 
inferred from the existing documentation, then we should think about why 
that is and how we could enhance the documentation to ensure it 
accurately describes what None is supposed to be.

That said, your examples are good :)

Cheers,
Steve


From jfine2358 at gmail.com  Mon Jul 23 05:29:42 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Mon, 23 Jul 2018 10:29:42 +0100
Subject: [Python-ideas] slice[] to get more complex slices
In-Reply-To: <09cb9784-6a97-49b9-b5e7-1c9a9d8c8391@googlegroups.com>
References: <CAFpSVpKnE8b=QrjxLy6anAF3ammq2QbB4_K1Ey02Vk3n8EVwyQ@mail.gmail.com>
 <pj2lgk$c6c$1@blaine.gmane.org>
 <09cb9784-6a97-49b9-b5e7-1c9a9d8c8391@googlegroups.com>
Message-ID: <CALD=Yf-MNLW7z8s=ruc_C13t7eWp8KC6OWvqf9durZgpN=8=sw@mail.gmail.com>

Hi Gr?gory

You wrote:
> Seems that the initial proposal was skipped just for timing reasons

The discussion is in https://bugs.python.org/issue24379. It was first
skipped for that reason. Second time around our Then and now Former
BDFL wrote
===
https://bugs.python.org/msg280721
Actually I'm with Raymond -- I'm also not sure or mildly against (-0).
Given how easy it is to do without and how cumbersome using the operator
module is I don't see a great benefit. (IMO the operator module should be
the measure of *last* resort -- it is not to become part of anybody's
toolkit for common operators. But I seem to be a minority opinion here.)
===
and also
===
https://bugs.python.org/msg319695
Let's close it. Just because someone spent a lot of effort on a patch
we don't have to accept it.
===

That said, I see merits in providing standard tools for creating
slices. These tools, as discussed, to take advantage of the power of
the [...] syntax. I'd start here:
https://bugs.python.org/msg244801

-- 
Jonathan

From steve.dower at python.org  Mon Jul 23 05:48:23 2018
From: steve.dower at python.org (Steve Dower)
Date: Mon, 23 Jul 2018 10:48:23 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180723005114.GT8744@ando.pearwood.info>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
 <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>
 <20180722121006.GQ8744@ando.pearwood.info>
 <CAEbHw4b7hSgu28RwfwMav=cR-uxrErnGBDpeDVuirOevFvLo3g@mail.gmail.com>
 <6562164f-9be1-40ec-890d-508fd5c81014@googlegroups.com>
 <CAPTjJmr8cMiW_K1jZwa4WJ_Tmy406+MxrXF_XR+OLWc0W47r3w@mail.gmail.com>
 <20180723005114.GT8744@ando.pearwood.info>
Message-ID: <1b7f56aa-b851-8604-5c94-96aeed77efa6@python.org>

On 23Jul2018 0151, Steven D'Aprano wrote:
> What if there was a language
> supported, non-hackish way to officially delay evaluation of
> expressions until explicitly requested?

The current spelling for this is "lambda: delayed-expression" and the 
way to request the value is "()". :)

(I'm not even being that facetious here. People ask for delayed 
expressions all the time, and it's only 7 characters, provided the 
callee knows they're getting it, and the semantics are already well 
defined and likely match what you want.)

Cheers,
Steve

From steve.dower at python.org  Mon Jul 23 05:51:31 2018
From: steve.dower at python.org (Steve Dower)
Date: Mon, 23 Jul 2018 10:51:31 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAAu18hc35N11jVQj+0691aH8Oe2+4Cqfn0yFpwWtjqou161ATg@mail.gmail.com>
References: <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
 <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAAu18hc35N11jVQj+0691aH8Oe2+4Cqfn0yFpwWtjqou161ATg@mail.gmail.com>
Message-ID: <0ffb89a1-3f63-a49d-e5a9-e98b25f60616@python.org>

Responding to a few more ideas that have come up here. Again, apologies 
for not directing them to the original authors, but I want to focus on 
the ideas that are leading towards a more informed decision, and not 
getting distracted by providing customised examples for people or 
getting into side debates.

I'm also going to try and update the PEP text today (or this week at 
least) to better clarify some of the questions that have come up (and 
fix that embarrassingly broken example :( )

Cheers,
Steve

False: '?.' should be surrounded by spaces
------------------------------------------

It's basically the same as '.'. Spell it 'a?.b', not 'a ?. b' (like 
'a.b' rather than 'a + b').

It's an enhancement to attribute access, not a new type of binary 
operator. The right-hand side cannot be evaluated in isolation.

In my opinion, it can also be read aloud the same as '.' as well (see 
the next point).

False: 'a?.b' is totally different from 'a.b'
---------------------------------------------

The expression 'a.b' either results in 'a.b' or AttributeError (assuming 
no descriptors are involved).

The expression 'a?.b' either results in 'a.b' or None (again, assuming 
no descriptors).

This isn't a crazy new idea, it really just short-circuits a specific 
error that can only be precisely avoided with "if None" checks (catching 
AttributeError is not the same).

The trivial case is already a one-liner
---------------------------------------

That may be the case if you have a single character variable, but this 
proposal is not intended to try and further simplify already simple 
cases. It is for complex cases, particularly where you do not want to 
reevaluate the arguments or potentially leak temporary names into a 
module or class namespace.

(Brief aside: 'a if (a := expr) is not None else None' is going to be 
the best workaround. The suggested 'a := expr if a is not None else 
None' is incorrect because the condition is evaluated first and so has 
to contain the assignment.)

False: ??= is a new form of assignment
--------------------------------------

No, it's just augmented assignment for a binary operator. "a ??= b" is 
identical to "a = a ?? b", just like "+=" and friends.

It has no relationship to assignment expressions. '??=' can only be used 
as a statement, and is not strictly necessary, but if we add a new 
binary operator '??' and it does not have an equivalent augmented 
assignment statement, people will justifiably wonder about the 
inconsistency.

The PEP author is unsure about how it works
-------------------------------------------

I wish this statement had come with some context, because the only thing 
I'm unsure about is what I'm supposed to be unsure about.

That said, I'm willing to make changes to the PEP based on the feedback 
and discussion. I haven't come into this with a "my way is 100% right 
and it will never change" mindset, so if this is a misinterpretation of 
my willingness to listen to feedback then I'm sorry I wasn't more clear. 
I *do* care about your opinions (when presented fairly and constructively).

Which is the most important operator?
-------------------------------------

Personally, I think '?.' is the most valuable. The value of '??' arises 
because (unless changing the semantics from None-aware to False-aware) 
it provides a way of setting the default that is consistent with how we 
got to the no-value value (e.g. `None?.a ?? b` and `""?.a ?? b` are 
different, whereas `None?.a or b` and `""?.a or b` are equivalent).

I'm borderline on ?[] right now. Honestly, I think it works best if it 
also silently handles LookupError (e.g. for traversing a loaded JSON 
dict), but then it's inconsistent with ?. which I think works best if it 
handles None but allows AttributeError. Either way, both have the 
ability to directly handle the exception. For example, (assuming e1, e2 
are expressions and not values):

     v = e1?[e2]

Could be handled as this example (for None-aware):

     _temp1 = (e1)
     v = _temp1[e2] if _temp1 is not None else None

Or for silent exception handling of the lookup only:

     _temp1 = (e1)
     _temp2 = (e2)
     try:
         v = _temp1[_temp2] if _temp1 is not None else None
     except LookupError:
         v = None

Note that this second example is _not_ how most people protect against 
invalid lookups (most people use `.get` when it's available, or they 
accept that LookupErrors raised from e1 or e2 should also be silently 
handled). So there would be value in ?[] being able to more precisely 
handle the exception.

However, with ?. being available, and _most_ lookups being on dicts that 
have .get(), you can also traverse JSON values fairly easily like this:

     d = json.load(f)
     name = d.get('user')?.get('details')?.get('name') ?? '<no name>'

With ?[] doing the safe lookup as well, this could be:

     d = json.load(f)
     name = d?['user']?['details']?['name'] ?? '<no name>'

Now, my *least* favourite part of this is that (as someone pointed out), 
it looks very similar to using '??' with a list as the default value. 
And because of that, I'm okay with removing this part of the proposal if 
it is unpopular.

From gregory.lielens at gmail.com  Mon Jul 23 05:58:48 2018
From: gregory.lielens at gmail.com (=?UTF-8?Q?Gr=C3=A9gory_Lielens?=)
Date: Mon, 23 Jul 2018 02:58:48 -0700 (PDT)
Subject: [Python-ideas] slice[] to get more complex slices
In-Reply-To: <CALD=Yf-MNLW7z8s=ruc_C13t7eWp8KC6OWvqf9durZgpN=8=sw@mail.gmail.com>
References: <CAFpSVpKnE8b=QrjxLy6anAF3ammq2QbB4_K1Ey02Vk3n8EVwyQ@mail.gmail.com>
 <pj2lgk$c6c$1@blaine.gmane.org>
 <09cb9784-6a97-49b9-b5e7-1c9a9d8c8391@googlegroups.com>
 <CALD=Yf-MNLW7z8s=ruc_C13t7eWp8KC6OWvqf9durZgpN=8=sw@mail.gmail.com>
Message-ID: <f2acd012-78f4-4477-b787-a6e604d8396d@googlegroups.com>

Oh yes , I see. Seems like almost everybody think using the [...] syntax to 
define reusable slices is a good idea, the discussion is where this could 
be implemented, knowing that numpy.s_ already exists.

Discussion is not easy to follow, with the patch almost accepted then 
delayed then rejected, in a very short amount of time.

In particular, it's not clear why direct indexing of slice type (not 
instance) was not chosen. It was the first proposal, and here again Todd's 
choice. it was silently skipped/changed to slice.literal then moved to 
operators, at least that's the impression I got reading  
https://bugs.python.org/issue24379 ...
Not sure slice[1::3] can be done, but this has my preference too: it's the 
most direct exposure I can think of...


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/159ddf0f/attachment.html>

From solipsis at pitrou.net  Mon Jul 23 06:11:56 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 23 Jul 2018 12:11:56 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
References: <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
 <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAAu18hc35N11jVQj+0691aH8Oe2+4Cqfn0yFpwWtjqou161ATg@mail.gmail.com>
 <0ffb89a1-3f63-a49d-e5a9-e98b25f60616@python.org>
Message-ID: <20180723121156.4337a316@fsol>

On Mon, 23 Jul 2018 10:51:31 +0100
Steve Dower <steve.dower at python.org> wrote:
> 
> Which is the most important operator?
> -------------------------------------
> 
> Personally, I think '?.' is the most valuable.

For me, it's the most contentious.  The fact that a single '?' added to
a regular line of Python code can short-circuit execution silently is a
net detriment to readability, IMHO.

In a code review, this means I must be careful about '?' sigils lest I
miss important bug magnets.

Regards

Antoine.



From J.Demeyer at UGent.be  Mon Jul 23 06:24:09 2018
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Mon, 23 Jul 2018 12:24:09 +0200
Subject: [Python-ideas] slice[] to get more complex slices
In-Reply-To: <399ff7d70c204f679bdb5a79f9bc0460@xmail103.UGent.be>
References: <CAFpSVpKnE8b=QrjxLy6anAF3ammq2QbB4_K1Ey02Vk3n8EVwyQ@mail.gmail.com>
 <pj2lgk$c6c$1@blaine.gmane.org>
 <09cb9784-6a97-49b9-b5e7-1c9a9d8c8391@googlegroups.com>
 <CALD=Yf-MNLW7z8s=ruc_C13t7eWp8KC6OWvqf9durZgpN=8=sw@mail.gmail.com>
 <399ff7d70c204f679bdb5a79f9bc0460@xmail103.UGent.be>
Message-ID: <5B55ACC9.8070303@UGent.be>

On 2018-07-23 11:58, Gr?gory Lielens wrote:
> Not sure slice[1::3] can be done

It can be done. Since "slice" is a class, it would require a metaclass 
though.

Another solution that nobody has mentioned (as far as I know) is to add 
additional syntax to the language for that. For example, one could say 
that (1:3) could be used to construct slice(1, 3) directly. The 
parentheses are required to avoid confusion with type hints. I'm not a 
Python language expert, but I don't think that type hints can occur 
inside parentheses like that.


Jeroen.

From steve.dower at python.org  Mon Jul 23 06:25:53 2018
From: steve.dower at python.org (Steve Dower)
Date: Mon, 23 Jul 2018 11:25:53 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180723121156.4337a316@fsol>
References: <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
 <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAAu18hc35N11jVQj+0691aH8Oe2+4Cqfn0yFpwWtjqou161ATg@mail.gmail.com>
 <0ffb89a1-3f63-a49d-e5a9-e98b25f60616@python.org>
 <20180723121156.4337a316@fsol>
Message-ID: <beac393d-08db-62ae-ac17-7745525683a8@python.org>

On 23Jul2018 1111, Antoine Pitrou wrote:
> On Mon, 23 Jul 2018 10:51:31 +0100
> Steve Dower <steve.dower at python.org> wrote:
>>
>> Which is the most important operator?
>> -------------------------------------
>>
>> Personally, I think '?.' is the most valuable.
> 
> For me, it's the most contentious.  The fact that a single '?' added to
> a regular line of Python code can short-circuit execution silently is a
> net detriment to readability, IMHO.

The only time it would short-circuit is when it would otherwise raise 
AttributeError for trying to access an attribute from None, which is 
also going to short-circuit. The difference is that it short-circuits 
the expression only, and not all statements up until the next except 
handler.

Cheers,
Steve

From jfine2358 at gmail.com  Mon Jul 23 06:28:04 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Mon, 23 Jul 2018 11:28:04 +0100
Subject: [Python-ideas] slice[] to get more complex slices
In-Reply-To: <f2acd012-78f4-4477-b787-a6e604d8396d@googlegroups.com>
References: <CAFpSVpKnE8b=QrjxLy6anAF3ammq2QbB4_K1Ey02Vk3n8EVwyQ@mail.gmail.com>
 <pj2lgk$c6c$1@blaine.gmane.org>
 <09cb9784-6a97-49b9-b5e7-1c9a9d8c8391@googlegroups.com>
 <CALD=Yf-MNLW7z8s=ruc_C13t7eWp8KC6OWvqf9durZgpN=8=sw@mail.gmail.com>
 <f2acd012-78f4-4477-b787-a6e604d8396d@googlegroups.com>
Message-ID: <CALD=Yf-z5L1zJ7ESP0CFK+wgx6ajMX=njks54hG_UX3F81oz8g@mail.gmail.com>

Hi Gr?gory

You wrote:
> Oh yes , I see. Seems like almost everybody think using the [...] syntax to
> define reusable slices is a good idea,

> Not sure slice[1::3] can be done, but this has my preference too: it's the
> most direct exposure I can think of...

The slice class already has all the core functionality it needs. This
thread is looking for something extra. My preference is to create a
new module, slicetools, that contains the functionality the people on
this thread want to have.

What I like about this approach is that the code can be written and
deployed without getting permission from anyone. And when it's done
and has a body of practice, the PEP would simply be "Add slicetools to
the standard library".

Anyone here up for working on that sort of approach?

-- 
Jonathan

From antoine at python.org  Mon Jul 23 06:29:33 2018
From: antoine at python.org (Antoine Pitrou)
Date: Mon, 23 Jul 2018 12:29:33 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <beac393d-08db-62ae-ac17-7745525683a8@python.org>
References: <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
 <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAAu18hc35N11jVQj+0691aH8Oe2+4Cqfn0yFpwWtjqou161ATg@mail.gmail.com>
 <0ffb89a1-3f63-a49d-e5a9-e98b25f60616@python.org>
 <20180723121156.4337a316@fsol>
 <beac393d-08db-62ae-ac17-7745525683a8@python.org>
Message-ID: <b50a44e4-b3c8-3df1-ab1c-f0a0d802244e@python.org>


Le 23/07/2018 ? 12:25, Steve Dower a ?crit?:
> On 23Jul2018 1111, Antoine Pitrou wrote:
>> On Mon, 23 Jul 2018 10:51:31 +0100
>> Steve Dower <steve.dower at python.org> wrote:
>>>
>>> Which is the most important operator?
>>> -------------------------------------
>>>
>>> Personally, I think '?.' is the most valuable.
>>
>> For me, it's the most contentious.  The fact that a single '?' added to
>> a regular line of Python code can short-circuit execution silently is a
>> net detriment to readability, IMHO.
> 
> The only time it would short-circuit is when it would otherwise raise 
> AttributeError for trying to access an attribute from None, which is 
> also going to short-circuit.

But AttributeError is going to bubble up as soon as it's raised, unless
it's explicitly handled by an except block.  Simply returning None may
have silent undesired effects (perhaps even security flaws).

This whole thing reminds of PHP's malicious "@" operator.

Regards

Antoine.

From J.Demeyer at UGent.be  Mon Jul 23 06:31:06 2018
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Mon, 23 Jul 2018 12:31:06 +0200
Subject: [Python-ideas] slice[] to get more complex slices
In-Reply-To: <4243953379d74d2aa5393be2434a99a7@xmail103.UGent.be>
References: <CAFpSVpKnE8b=QrjxLy6anAF3ammq2QbB4_K1Ey02Vk3n8EVwyQ@mail.gmail.com>
 <pj2lgk$c6c$1@blaine.gmane.org>
 <09cb9784-6a97-49b9-b5e7-1c9a9d8c8391@googlegroups.com>
 <CALD=Yf-MNLW7z8s=ruc_C13t7eWp8KC6OWvqf9durZgpN=8=sw@mail.gmail.com>
 <399ff7d70c204f679bdb5a79f9bc0460@xmail103.UGent.be>
 <4243953379d74d2aa5393be2434a99a7@xmail103.UGent.be>
Message-ID: <5B55AE6A.7010305@UGent.be>

On 2018-07-23 12:24, Jeroen Demeyer wrote:
> Another solution that nobody has mentioned (as far as I know) is to add
> additional syntax to the language for that. For example, one could say
> that (1:3) could be used to construct slice(1, 3) directly. The
> parentheses are required to avoid confusion with type hints. I'm not a
> Python language expert, but I don't think that type hints can occur
> inside parentheses like that.

And this could be extended to tuples (1:3, 2:4) and lists [1:3, 2:4] of 
slices too.

From steve.dower at python.org  Mon Jul 23 06:38:11 2018
From: steve.dower at python.org (Steve Dower)
Date: Mon, 23 Jul 2018 11:38:11 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <b50a44e4-b3c8-3df1-ab1c-f0a0d802244e@python.org>
References: <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
 <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAAu18hc35N11jVQj+0691aH8Oe2+4Cqfn0yFpwWtjqou161ATg@mail.gmail.com>
 <0ffb89a1-3f63-a49d-e5a9-e98b25f60616@python.org>
 <20180723121156.4337a316@fsol>
 <beac393d-08db-62ae-ac17-7745525683a8@python.org>
 <b50a44e4-b3c8-3df1-ab1c-f0a0d802244e@python.org>
Message-ID: <a46fc434-1bbb-f17c-42ef-840e3d121572@python.org>

On 23Jul2018 1129, Antoine Pitrou wrote:
> 
> Le 23/07/2018 ? 12:25, Steve Dower a ?crit?:
>> On 23Jul2018 1111, Antoine Pitrou wrote:
>>> On Mon, 23 Jul 2018 10:51:31 +0100
>>> Steve Dower <steve.dower at python.org> wrote:
>>>>
>>>> Which is the most important operator?
>>>> -------------------------------------
>>>>
>>>> Personally, I think '?.' is the most valuable.
>>>
>>> For me, it's the most contentious.  The fact that a single '?' added to
>>> a regular line of Python code can short-circuit execution silently is a
>>> net detriment to readability, IMHO.
>>
>> The only time it would short-circuit is when it would otherwise raise
>> AttributeError for trying to access an attribute from None, which is
>> also going to short-circuit.
> 
> But AttributeError is going to bubble up as soon as it's raised, unless
> it's explicitly handled by an except block.  Simply returning None may
> have silent undesired effects (perhaps even security flaws).

You're right that the silent/undesired effects would be bad, which is 
why I'm not proposing silent changes to existing code (such as 
None.__getattr__ always returning None).

This is a substitute for explicitly checking None before the attribute 
access, or explicitly handling AttributeError for this case (and 
unintentionally handling others as well). And "?." may be very small 
compared to the extra 3+ lines required to do exactly the same thing, 
but it is still an explicit change that can be reviewed and evaluated as 
"is None a valid but not-useful value here? or is it an indication of 
another error and should we fail immediately instead".

Cheers,
Steve

> This whole thing reminds of PHP's malicious "@" operator.

General comment to everyone (not just Antoine): these arguments have 
zero value to me. Feel free to keep making them, but I am uninterested. 
Perhaps whoever gets to decide on the PEP will be swayed by them?

From antoine at python.org  Mon Jul 23 06:45:22 2018
From: antoine at python.org (Antoine Pitrou)
Date: Mon, 23 Jul 2018 12:45:22 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <a46fc434-1bbb-f17c-42ef-840e3d121572@python.org>
References: <20180722121006.GQ8744@ando.pearwood.info>
 <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAAu18hc35N11jVQj+0691aH8Oe2+4Cqfn0yFpwWtjqou161ATg@mail.gmail.com>
 <0ffb89a1-3f63-a49d-e5a9-e98b25f60616@python.org>
 <20180723121156.4337a316@fsol>
 <beac393d-08db-62ae-ac17-7745525683a8@python.org>
 <b50a44e4-b3c8-3df1-ab1c-f0a0d802244e@python.org>
 <a46fc434-1bbb-f17c-42ef-840e3d121572@python.org>
Message-ID: <f7363e81-9a77-4178-2f22-7281b0d5e917@python.org>


Le 23/07/2018 ? 12:38, Steve Dower a ?crit?:
> 
> General comment to everyone (not just Antoine): these arguments have 
> zero value to me. Feel free to keep making them, but I am uninterested. 

So you're uninterested in learning from past mistakes?

You sound like a child who thinks their demands should be satisfied
because they are the center of the world.

Regards

Antoine.

From steve.dower at python.org  Mon Jul 23 07:09:00 2018
From: steve.dower at python.org (Steve Dower)
Date: Mon, 23 Jul 2018 12:09:00 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <f7363e81-9a77-4178-2f22-7281b0d5e917@python.org>
References: <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAAu18hc35N11jVQj+0691aH8Oe2+4Cqfn0yFpwWtjqou161ATg@mail.gmail.com>
 <0ffb89a1-3f63-a49d-e5a9-e98b25f60616@python.org>
 <20180723121156.4337a316@fsol>
 <beac393d-08db-62ae-ac17-7745525683a8@python.org>
 <b50a44e4-b3c8-3df1-ab1c-f0a0d802244e@python.org>
 <a46fc434-1bbb-f17c-42ef-840e3d121572@python.org>
 <f7363e81-9a77-4178-2f22-7281b0d5e917@python.org>
Message-ID: <13681cf0-1571-2b74-e451-bb234f1952b7@python.org>

On 23Jul2018 1145, Antoine Pitrou wrote:
> 
> Le 23/07/2018 ? 12:38, Steve Dower a ?crit?:
>>
>> General comment to everyone (not just Antoine): these arguments have
>> zero value to me. Feel free to keep making them, but I am uninterested.
> 
> So you're uninterested in learning from past mistakes?
> 
> You sound like a child who thinks their demands should be satisfied
> because they are the center of the world.

Sorry if it came across like that, it wasn't the intention. A bit of 
context on why you think it's a mistake would have helped, but if it's a 
purely subjective "I don't like the look of it" (as most similar 
arguments have turned out) then it doesn't add anything to enhancing the 
PEP. As a result, I do not see any reason to engage with this class of 
argument.

I hope you'll also notice that I've been making very few demands in this 
thread, and have indicated a number of times that I'm very open to 
adjusting the proposal in the face of honest and useful feedback.

Cheers,
Steve

From p.f.moore at gmail.com  Mon Jul 23 07:15:14 2018
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 23 Jul 2018 12:15:14 +0100
Subject: [Python-ideas] Python docs page: In what ways is None special
In-Reply-To: <d7651538-1831-b7c7-f050-252826591ac2@python.org>
References: <CALD=Yf_PRXZYiNdC9FGE5pd1cjrS8gyr8q5iPxzrMKXzX027YA@mail.gmail.com>
 <d7651538-1831-b7c7-f050-252826591ac2@python.org>
Message-ID: <CACac1F9qgTdD3Bac9EmKc+Af8EQRFH=KfVWzpOied0Se7qc0EQ@mail.gmail.com>

On 23 July 2018 at 10:16, Steve Dower <steve.dower at python.org> wrote:
> On 23Jul2018 1003, Jonathan Fine wrote:
>>
>> This arises out of PEP 505 - None-aware operators.
>>
>> I thought, a page on how None is special would be nice.
>> I've not found such a page on the web. We do have
>> ===
>> https://docs.python.org/3/library/constants.html
>> None
>> The sole value of the type NoneType. None is
>> frequently used to represent the absence of a
>> value, as when default arguments are not passed
>> to a function. Assignments to None are illegal
>> and raise a SyntaxError.
>> ===
>>
>> So decided to start writing such a page, perhaps to be
>> added to the docs.  All code examples in Python3.4.
>
>
> There's also
> https://docs.python.org/3/c-api/none.html?highlight=py_none#c.Py_None
>
> "The Python None object, denoting lack of value. This object has no methods.
> It needs to be treated just like any other object with respect to reference
> counts."
>
> I don't know that documenting the behaviours of None are that interesting
> (e.g. not displaying anything at the interactive prompt), though it'd be
> perfect for a blog and/or conference talk. But if there appear to be
> behaviours that are not consistent or cannot be easily inferred from the
> existing documentation, then we should think about why that is and how we
> could enhance the documentation to ensure it accurately describes what None
> is supposed to be.
>
> That said, your examples are good :)

The examples are interesting, agreed. One thing they highlight to me
is that most uses of None are effectively convention. The only two
behaviours that are unique to None and implemented in the interpreter
are:

1. Functions that don't return an explicit value return None.
2. The interactive interpreter does not display the value of a typed
expression when that value is None.

(the fact that None is a keyword is not unique - True and False are
also keywords - although it is a distinguishing feature).

One of the key points about this proposal is that it adds a number of
additional ways in which the interpreter treats None specially (i.e.,
four dedicated operators). That elevates None from the position of
being a commonly used sentinel value to one that has language support
for being a sentinel. That's not necessarily a bad thing, but it is a
point that should be considered when reviewing this PEP (people who
prefer to use object() for their sentinels are probably less happy
about making None "special", for example - Pandas uses NaN as the
"missing data" sentinel rather than None, although Pandas' focus on
bulk processing means that dedicated operators probably wouldn't help
anyway).

Overall, my personal feeling is that I do use None specially (and a
lot of code I see does too), so having None-aware operators has some
merit. But I don't find None to be special enough to warrant quite
this *many* operators, nor do I find that the patterns that operators
like ?. and ?[ support are ones I use a lot. Actually, the ?. and ?[
operators seem like they'd be much more useful if I did more JSON
processing - but I don't, so I have little feel for whether language
support is needed here or whether better library support (stdlib
and/or 3rd party) for optional data is sufficient.

Paul

From jfine2358 at gmail.com  Mon Jul 23 07:24:40 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Mon, 23 Jul 2018 12:24:40 +0100
Subject: [Python-ideas] Python docs page: In what ways is None special
In-Reply-To: <CACac1F9qgTdD3Bac9EmKc+Af8EQRFH=KfVWzpOied0Se7qc0EQ@mail.gmail.com>
References: <CALD=Yf_PRXZYiNdC9FGE5pd1cjrS8gyr8q5iPxzrMKXzX027YA@mail.gmail.com>
 <d7651538-1831-b7c7-f050-252826591ac2@python.org>
 <CACac1F9qgTdD3Bac9EmKc+Af8EQRFH=KfVWzpOied0Se7qc0EQ@mail.gmail.com>
Message-ID: <CALD=Yf9_VO1Njn3hEW+2VGs1nS76Hujqo7DpfxK-koCYsuAYKA@mail.gmail.com>

Hi Paul

Thank you for your contribution

> The examples are interesting, agreed. One thing they highlight to me
> is that most uses of None are effectively convention. The only two
> behaviours that are unique to None and implemented in the interpreter
> are:
>
> 1. Functions that don't return an explicit value return None.
> 2. The interactive interpreter does not display the value of a typed
> expression when that value is None.

I broadly agree with what you say, particularly the compliment.

You also wrote:
> One of the key points about this proposal [PEP 505]

My goal in this thread is to document clearly and develop a clear
shared understanding of the ways in which None is special in Python.
Please, please, please can we not discuss PEP 505 in this thread.
There's already another thread for that.

I want this thread to produce a web page that is useful for beginners
and intermediate Python programmers (which perhaps some experts might
also benefit from reading).

-- 
Jonathan

From p.f.moore at gmail.com  Mon Jul 23 07:24:40 2018
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 23 Jul 2018 12:24:40 +0100
Subject: [Python-ideas] slice[] to get more complex slices
In-Reply-To: <5B55AE6A.7010305@UGent.be>
References: <CAFpSVpKnE8b=QrjxLy6anAF3ammq2QbB4_K1Ey02Vk3n8EVwyQ@mail.gmail.com>
 <pj2lgk$c6c$1@blaine.gmane.org>
 <09cb9784-6a97-49b9-b5e7-1c9a9d8c8391@googlegroups.com>
 <CALD=Yf-MNLW7z8s=ruc_C13t7eWp8KC6OWvqf9durZgpN=8=sw@mail.gmail.com>
 <399ff7d70c204f679bdb5a79f9bc0460@xmail103.UGent.be>
 <4243953379d74d2aa5393be2434a99a7@xmail103.UGent.be>
 <5B55AE6A.7010305@UGent.be>
Message-ID: <CACac1F_XOi+1gmEMgZdbmW_3UXs661JS5BF=OvWLrF47H0fJmg@mail.gmail.com>

On 23 July 2018 at 11:31, Jeroen Demeyer <J.Demeyer at ugent.be> wrote:
> On 2018-07-23 12:24, Jeroen Demeyer wrote:
>>
>> Another solution that nobody has mentioned (as far as I know) is to add
>> additional syntax to the language for that. For example, one could say
>> that (1:3) could be used to construct slice(1, 3) directly. The
>> parentheses are required to avoid confusion with type hints. I'm not a
>> Python language expert, but I don't think that type hints can occur
>> inside parentheses like that.
>
>
> And this could be extended to tuples (1:3, 2:4) and lists [1:3, 2:4] of
> slices too.

I thought the reason the proposal got nowhere was because it's pretty
simple to define it yourself:

>>> class SliceHelper:
...     def __getitem__(self, slice):
...         return slice
...
>>> SH = SliceHelper()
>>> SH[1::3]
slice(1, None, 3)

Did I miss something significant about why this wasn't sufficient?

Paul

From gregory.lielens at gmail.com  Mon Jul 23 07:26:56 2018
From: gregory.lielens at gmail.com (=?UTF-8?Q?Gr=C3=A9gory_Lielens?=)
Date: Mon, 23 Jul 2018 04:26:56 -0700 (PDT)
Subject: [Python-ideas] slice[] to get more complex slices
In-Reply-To: <5B55AE6A.7010305@UGent.be>
References: <CAFpSVpKnE8b=QrjxLy6anAF3ammq2QbB4_K1Ey02Vk3n8EVwyQ@mail.gmail.com>
 <pj2lgk$c6c$1@blaine.gmane.org>
 <09cb9784-6a97-49b9-b5e7-1c9a9d8c8391@googlegroups.com>
 <CALD=Yf-MNLW7z8s=ruc_C13t7eWp8KC6OWvqf9durZgpN=8=sw@mail.gmail.com>
 <399ff7d70c204f679bdb5a79f9bc0460@xmail103.UGent.be>
 <4243953379d74d2aa5393be2434a99a7@xmail103.UGent.be>
 <5B55AE6A.7010305@UGent.be>
Message-ID: <6be5cf78-6f6b-47a3-baee-109554b670d4@googlegroups.com>

That would be even more direct....but it require syntax support, usefull 
mainly for people doing multdim complex slicing (i.e numpy users). I may be 
wrong, but I do not see it gain much support outside numpy...

slice[....] is probably much more easy to swallow for standard python 
users, and almost as good imho.
It reuse getitem, so it imply the produced slice will behaves exactly like 
it would if it was not stored/reused, and almost garantee it will be the 
case indeed (even if the slice syntax is extended)

Getting this to work including a new module would be nice. Eventually, 
having it standard is positive too, it means slice manipulation will become 
more standardised.

On Monday, July 23, 2018 at 12:32:04 PM UTC+2, Jeroen Demeyer wrote:
>
> On 2018-07-23 12:24, Jeroen Demeyer wrote: 
> > Another solution that nobody has mentioned (as far as I know) is to add 
> > additional syntax to the language for that. For example, one could say 
> > that (1:3) could be used to construct slice(1, 3) directly. The 
> > parentheses are required to avoid confusion with type hints. I'm not a 
> > Python language expert, but I don't think that type hints can occur 
> > inside parentheses like that. 
>
> And this could be extended to tuples (1:3, 2:4) and lists [1:3, 2:4] of 
> slices too. 
> _______________________________________________ 
> Python-ideas mailing list 
> Python... at python.org <javascript:> 
> https://mail.python.org/mailman/listinfo/python-ideas 
> Code of Conduct: http://python.org/psf/codeofconduct/ 
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/1d63a288/attachment-0001.html>

From nicholas.cole at gmail.com  Mon Jul 23 07:38:17 2018
From: nicholas.cole at gmail.com (Nicholas Cole)
Date: Mon, 23 Jul 2018 12:38:17 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <1898d063-8179-423d-988a-01afa1b75075@googlegroups.com>
References: <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
 <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAAu18hc35N11jVQj+0691aH8Oe2+4Cqfn0yFpwWtjqou161ATg@mail.gmail.com>
 <1898d063-8179-423d-988a-01afa1b75075@googlegroups.com>
Message-ID: <CAAu18hcaVtxfG=gyUR5mdBEjyMA_hL-a++0Uj8i9DyHR_emUDg@mail.gmail.com>

On Mon, Jul 23, 2018 at 8:08 AM Gr?gory Lielens
<gregory.lielens at gmail.com> wrote:
>
>
>
> On Monday, July 23, 2018 at 8:24:45 AM UTC+2, Nicholas Cole wrote:
>
>> And that leads to a simple question: how many times does this actually
>> occur in real-world by python code? -- i.e. how many times do I want
>> to check the value of an existing label, and, finding it is None (and
>> None specifically), then assign it a value?
>
>
> The PEP present a few examples.
>
> I think the compactness and clarity is really gained when doing a descent into an "attribute tree", where any (or selected) members can be None, like getting back to the  example by Steven (spell-corrected, and expanded):
>
>  meal = obj?.spam?.eggs.tomato?.cheese ?? "Mozzarella"
>
> Where, to put the proposal in even better light, I have assumed that eggs instances always have a tomato attributes (i.e. are never None).
>
> This is indeed much more compact that the current version, and arguably more readable (it's easier to parse once you know the syntax...but you have to know the special syntax that will be used for this kind of stuff only). The more you deepen the attribute access, the more you gain, but of course deep attribute access is not so common. The new operators may make deep attribute hierarchies (or deeply nested dicts/lists) slightly more common, and there also I do not think it's a good thing.
>

That above example looks terrible to read (to me).  Yes, that's a
subjective statement.  This example from the PEP is even worse:

Example:

if entry.is_dir():
    dirs.append(name)
    if entries is not None:
        entries.append(entry)
else:
    nondirs.append(name)

After updating to use the ?. operator:

if entry.is_dir():
    dirs.append(name)
    entries?.append(entry)
else:
    nondirs.append(name)


In the first, it's totally clear to me when entries would be appended
to and when it wouldn't.  The second I had to read several times
before I could see what was going on.  The fact that it is already
embedded in an if...else statement perhaps made it harder to
understand.

I guess people will say that we will just get used to the new syntax,
but I don't see the benefit of making that particular "if" statement
more compact, and leaving all the others in place.  Or to put it
another way, I'm just not convinced that "None" is sufficiently
special.

N.

From gregory.lielens at gmail.com  Mon Jul 23 07:39:09 2018
From: gregory.lielens at gmail.com (=?UTF-8?Q?Gr=C3=A9gory_Lielens?=)
Date: Mon, 23 Jul 2018 04:39:09 -0700 (PDT)
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <13681cf0-1571-2b74-e451-bb234f1952b7@python.org>
References: <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAAu18hc35N11jVQj+0691aH8Oe2+4Cqfn0yFpwWtjqou161ATg@mail.gmail.com>
 <0ffb89a1-3f63-a49d-e5a9-e98b25f60616@python.org>
 <20180723121156.4337a316@fsol>
 <beac393d-08db-62ae-ac17-7745525683a8@python.org>
 <b50a44e4-b3c8-3df1-ab1c-f0a0d802244e@python.org>
 <a46fc434-1bbb-f17c-42ef-840e3d121572@python.org>
 <f7363e81-9a77-4178-2f22-7281b0d5e917@python.org>
 <13681cf0-1571-2b74-e451-bb234f1952b7@python.org>
Message-ID: <a79f0fbc-6277-4a01-a7d7-2255192aa4ec@googlegroups.com>

Maybe it would help if you mention in which context you will benefit the 
most? If the python sub-community related to this context agree "?? and 
friends" is a good idea, then it will add weight to the proposal. Else, 
probably better to forget it.

It seems related to JSON, but as I have never used it, it's a wild guess.

Anyway, it's a special pattern with deep attribute hierarchies, whose 
traversal is shortcutted when one attribute happen to be None.
This is common, with 2 exception:
 -it's rare it's really deep, or then it is arbitrarily deep and you need a 
procedural descent, not a fixed expression.
 -None break the descend, but then so does missing attributes.
You address the first with special syntax. not the second, so I suspect in 
your case the second does not happen. Or rarely happen.

Hope this will help finding why some finds the ? operators add little, 
while some others think they add enough to overcome python traditional 
operator-averse nature.


On Monday, July 23, 2018 at 1:10:03 PM UTC+2, Steve Dower wrote:
>
> On 23Jul2018 1145, Antoine Pitrou wrote:
> > 
> > Le 23/07/2018 ? 12:38, Steve Dower a ?crit :
> >>
> >> General comment to everyone (not just Antoine): these arguments have
> >> zero value to me. Feel free to keep making them, but I am uninterested.
> > 
> > So you're uninterested in learning from past mistakes?
> > 
> > You sound like a child who thinks their demands should be satisfied
> > because they are the center of the world.
>
> Sorry if it came across like that, it wasn't the intention. A bit of 
> context on why you think it's a mistake would have helped, but if it's a 
> purely subjective "I don't like the look of it" (as most similar 
> arguments have turned out) then it doesn't add anything to enhancing the 
> PEP. As a result, I do not see any reason to engage with this class of 
> argument.
>
> I hope you'll also notice that I've been making very few demands in this 
> thread, and have indicated a number of times that I'm very open to 
> adjusting the proposal in the face of honest and useful feedback.
>
> Cheers,
> Steve
> _______________________________________________
> Python-ideas mailing list
> Python... at python.org <javascript:>
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/b6f1f907/attachment.html>

From antoine at python.org  Mon Jul 23 07:39:15 2018
From: antoine at python.org (Antoine Pitrou)
Date: Mon, 23 Jul 2018 13:39:15 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <13681cf0-1571-2b74-e451-bb234f1952b7@python.org>
References: <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAAu18hc35N11jVQj+0691aH8Oe2+4Cqfn0yFpwWtjqou161ATg@mail.gmail.com>
 <0ffb89a1-3f63-a49d-e5a9-e98b25f60616@python.org>
 <20180723121156.4337a316@fsol>
 <beac393d-08db-62ae-ac17-7745525683a8@python.org>
 <b50a44e4-b3c8-3df1-ab1c-f0a0d802244e@python.org>
 <a46fc434-1bbb-f17c-42ef-840e3d121572@python.org>
 <f7363e81-9a77-4178-2f22-7281b0d5e917@python.org>
 <13681cf0-1571-2b74-e451-bb234f1952b7@python.org>
Message-ID: <e5325b45-8d10-633a-e437-77e1ea7eeadf@python.org>


Le 23/07/2018 ? 13:09, Steve Dower a ?crit?:
> On 23Jul2018 1145, Antoine Pitrou wrote:
>>
>> Le 23/07/2018 ? 12:38, Steve Dower a ?crit?:
>>>
>>> General comment to everyone (not just Antoine): these arguments have
>>> zero value to me. Feel free to keep making them, but I am uninterested.
>>
>> So you're uninterested in learning from past mistakes?
>>
>> You sound like a child who thinks their demands should be satisfied
>> because they are the center of the world.
> 
> Sorry if it came across like that, it wasn't the intention.

Thanks for your apologies, and I apology for being a bit abrasive too.

Regards

Antoine.

From p.f.moore at gmail.com  Mon Jul 23 07:43:28 2018
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 23 Jul 2018 12:43:28 +0100
Subject: [Python-ideas] Python docs page: In what ways is None special
In-Reply-To: <CALD=Yf9_VO1Njn3hEW+2VGs1nS76Hujqo7DpfxK-koCYsuAYKA@mail.gmail.com>
References: <CALD=Yf_PRXZYiNdC9FGE5pd1cjrS8gyr8q5iPxzrMKXzX027YA@mail.gmail.com>
 <d7651538-1831-b7c7-f050-252826591ac2@python.org>
 <CACac1F9qgTdD3Bac9EmKc+Af8EQRFH=KfVWzpOied0Se7qc0EQ@mail.gmail.com>
 <CALD=Yf9_VO1Njn3hEW+2VGs1nS76Hujqo7DpfxK-koCYsuAYKA@mail.gmail.com>
Message-ID: <CACac1F_Gy_H=qc04yLKGa-dDEbGAEt2TicTrn637gVHjM0rkaA@mail.gmail.com>

On 23 July 2018 at 12:24, Jonathan Fine <jfine2358 at gmail.com> wrote:
> You also wrote:
>> One of the key points about this proposal [PEP 505]
>
> My goal in this thread is to document clearly and develop a clear
> shared understanding of the ways in which None is special in Python.
> Please, please, please can we not discuss PEP 505 in this thread.
> There's already another thread for that.

OK, apologies for my confusion between the threads.

> I want this thread to produce a web page that is useful for beginners
> and intermediate Python programmers (which perhaps some experts might
> also benefit from reading).

My view would be basically "None isn't special, it's just another
value. By the time you understand enough to want to treat None
specially, you probably already know enough about how the language
does". That's very over-simplified, sure, but I really don't see None
as *that* special except in the context of non-beginner ideas like
"it's similar to SQL NULL", or "it's like a null pointer in C". In my
experience, I've found I tend more to be explaining to beginners why
None *isn't* special than why it is.

Which is why I found your list of examples interesting, because many
of them were about convention rather than language support.

Paul

From p.f.moore at gmail.com  Mon Jul 23 07:59:35 2018
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 23 Jul 2018 12:59:35 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <a79f0fbc-6277-4a01-a7d7-2255192aa4ec@googlegroups.com>
References: <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAAu18hc35N11jVQj+0691aH8Oe2+4Cqfn0yFpwWtjqou161ATg@mail.gmail.com>
 <0ffb89a1-3f63-a49d-e5a9-e98b25f60616@python.org>
 <20180723121156.4337a316@fsol>
 <beac393d-08db-62ae-ac17-7745525683a8@python.org>
 <b50a44e4-b3c8-3df1-ab1c-f0a0d802244e@python.org>
 <a46fc434-1bbb-f17c-42ef-840e3d121572@python.org>
 <f7363e81-9a77-4178-2f22-7281b0d5e917@python.org>
 <13681cf0-1571-2b74-e451-bb234f1952b7@python.org>
 <a79f0fbc-6277-4a01-a7d7-2255192aa4ec@googlegroups.com>
Message-ID: <CACac1F_jKbotHsokdB1vcPBbJFSyDqQpv6T4h8VunT-qhpkjLg@mail.gmail.com>

On 23 July 2018 at 12:39, Gr?gory Lielens <gregory.lielens at gmail.com> wrote:
> Maybe it would help if you mention in which context you will benefit the
> most? If the python sub-community related to this context agree "?? and
> friends" is a good idea, then it will add weight to the proposal. Else,
> probably better to forget it.
>
> It seems related to JSON, but as I have never used it, it's a wild guess.

This is my impression, as well. It seems like something that's helpful
in dealing with unstructured object hierarchies with lots of optional
attributes - which is where JSON tends to be used.

But given that, I'm really much more interested in seeing the new
operators compared against a well-written "JSON object hierarchy
traversal" library than against raw Python code. I'll happily agree
that traversing JSON-style data in current Python is pretty
unpleasant. But I don't honestly think that anyone has explored how
far a well-written library can go in making it easy to handle such
data (well, I certainly haven't, and I haven't found any particularly
good examples on PyPI). And until that's been tried, I think it's
premature to propose a syntax change (if it *has* been tried, adding
references to the PEP would be useful).

Again, this is more about ?. and ?[. I can see general uses for ??
(and its augmented assignment form ??=), but the None-aware attribute
and item access operators seem to me to be the most domain-specific
aspects of the PEP (as well as being the ugliest IMO ;-)). So
comparing against domain-specific libraries rather than against "write
your own" raw Python code seems reasonable to me.

Paul

From g.rodola at gmail.com  Mon Jul 23 08:04:17 2018
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Mon, 23 Jul 2018 14:04:17 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180723013614.GV8744@ando.pearwood.info>
References: <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
 <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
Message-ID: <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>

On Mon, Jul 23, 2018 at 3:12 AM Steven D'Aprano <steve at pearwood.info> wrote:
> > ? has no spaces, it's literally "variable names interrupted by
> > question marks" and evaluation can stop at any time while scanning the
> > line from left to right.
>
> Just like ordinary attribute access.
>
> This is the point I was making earlier: you accept existing punctuation
> doing these things:
>
>     try:
>         obj.spam.egsg.tomato.cheese  # oops a typo
>     except AttributeError:
>         # evaluation can stop at any time
>         ...
>
> while demanding a higher standard for new punctuation.
>
> All of your criticisms of ? punctuation applies to . as well.

I don't think they do. For once, "a.b" does one and one thing only,
"a?.b" does two and that's a fundamental difference (explicitness). It
does so by introducing a brand new operator ("?") which can be spelled
in two forms ("a?.b" and "a?[b]") by using two adjacent symbols not
interrupted by any space, which is an absolute first in the Python
syntax and that's the second and fundamental difference. I cannot move
the same criticism to the "a.b" form: it's simpler, it does one thing
and it uses one symbol.

Your argument is basically a revisitation of "it's just another
symbol" and "it's like a + b" which you have being pulling different
times in this thread already. You want to imply that since symbols are
already used in the grammar (and "." in particular) then it's
perfectly fine to also have "?" and its spell variants (which are 4 in
total). I don't think that's how changes of such importance should be
discussed.

> > Multiple "?" can live on the same line so
> > that's incentive to write one-liners, really, and to me one-liners are
> > always less explicit than the same logic split on multiple lines.
>
> Explicit is not always better.
>
>     import this
>
> is much better than:
>
>     for location in sys.path:
>         try:
>             for file in os.listdir(location):
>                 if os.splitext(file) in ('.pyc', '.py', '.so'):
>                     ...

I honestly don't see how this example is related with anything discussed so far.

> If punctuation is unreadable and Perlish, so is "." and ":" punctuation.
> If ?? is bad because it is "implicit", then so is import or sorted.

I'm not even sure how to reply to this.

-- 
Giampaolo - http://grodola.blogspot.com

From steve at pearwood.info  Mon Jul 23 08:13:51 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 23 Jul 2018 22:13:51 +1000
Subject: [Python-ideas] Python docs page: In what ways is None special
In-Reply-To: <CACac1F9qgTdD3Bac9EmKc+Af8EQRFH=KfVWzpOied0Se7qc0EQ@mail.gmail.com>
References: <CALD=Yf_PRXZYiNdC9FGE5pd1cjrS8gyr8q5iPxzrMKXzX027YA@mail.gmail.com>
 <d7651538-1831-b7c7-f050-252826591ac2@python.org>
 <CACac1F9qgTdD3Bac9EmKc+Af8EQRFH=KfVWzpOied0Se7qc0EQ@mail.gmail.com>
Message-ID: <20180723121351.GW8744@ando.pearwood.info>

On Mon, Jul 23, 2018 at 12:15:14PM +0100, Paul Moore wrote:

> The examples are interesting, agreed. One thing they highlight to me
> is that most uses of None are effectively convention. The only two
> behaviours that are unique to None and implemented in the interpreter
> are:
> 
> 1. Functions that don't return an explicit value return None.

That is built into the language.

> 2. The interactive interpreter does not display the value of a typed
> expression when that value is None.

But that is not built-in, it is merely the default behaviour. It is 
controlled by sys.displayhook, which can be customized:

https://docs.python.org/3/library/sys.html#sys.displayhook

We can have many interactive interpreters, including iPython, bpython, 
IDLE, etc and they don't necessarily suppress the printing of None.

 
> One of the key points about this proposal 

That would be the None-aware PEP, not Jonathon's proposal to add 
documentation about None.


> is that it adds a number of
> additional ways in which the interpreter treats None specially (i.e.,
> four dedicated operators). That elevates None from the position of
> being a commonly used sentinel value to one that has language support
> for being a sentinel.

None is already a special value. It is so special, it is one of only 
three built-in values given keyword status. All the other built-ins, 
including such things that are used implicitly by the interpreter (such 
as NotImplemented) are merely ordinary names.


> That's not necessarily a bad thing, but it is a
> point that should be considered when reviewing this PEP (people who
> prefer to use object() for their sentinels are probably less happy
> about making None "special",

You make it sound like its a mere aesthetic choice. None is the default, 
recommended, most convenient sentinel.

I use object() as a sentinel, when None is an otherwise ordinary value. 
I am also happy for None to be special, because it *is* special.



-- 
Steve

From toddrjen at gmail.com  Mon Jul 23 08:13:48 2018
From: toddrjen at gmail.com (Todd)
Date: Mon, 23 Jul 2018 08:13:48 -0400
Subject: [Python-ideas] slice[] to get more complex slices
In-Reply-To: <5B55ACC9.8070303@UGent.be>
References: <CAFpSVpKnE8b=QrjxLy6anAF3ammq2QbB4_K1Ey02Vk3n8EVwyQ@mail.gmail.com>
 <pj2lgk$c6c$1@blaine.gmane.org>
 <09cb9784-6a97-49b9-b5e7-1c9a9d8c8391@googlegroups.com>
 <CALD=Yf-MNLW7z8s=ruc_C13t7eWp8KC6OWvqf9durZgpN=8=sw@mail.gmail.com>
 <399ff7d70c204f679bdb5a79f9bc0460@xmail103.UGent.be>
 <5B55ACC9.8070303@UGent.be>
Message-ID: <CAFpSVp+xL8q73sogt8wCZ+dh-QEKxVHGxP-1OVXXQpb5jE8yvQ@mail.gmail.com>

On Mon, Jul 23, 2018 at 6:24 AM, Jeroen Demeyer <J.Demeyer at ugent.be> wrote:

> On 2018-07-23 11:58, Gr?gory Lielens wrote:
>
>> Not sure slice[1::3] can be done
>>
>
> It can be done. Since "slice" is a class, it would require a metaclass
> though.
>
> Another solution that nobody has mentioned (as far as I know) is to add
> additional syntax to the language for that. For example, one could say that
> (1:3) could be used to construct slice(1, 3) directly. The parentheses are
> required to avoid confusion with type hints. I'm not a Python language
> expert, but I don't think that type hints can occur inside parentheses like
> that.
>
>
If I remember correctly Guido explicitly rejected requests for such
syntax.  This proposal was intended as a compromise.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/4ff08766/attachment.html>

From toddrjen at gmail.com  Mon Jul 23 08:17:13 2018
From: toddrjen at gmail.com (Todd)
Date: Mon, 23 Jul 2018 08:17:13 -0400
Subject: [Python-ideas] slice[] to get more complex slices
In-Reply-To: <CALD=Yf-z5L1zJ7ESP0CFK+wgx6ajMX=njks54hG_UX3F81oz8g@mail.gmail.com>
References: <CAFpSVpKnE8b=QrjxLy6anAF3ammq2QbB4_K1Ey02Vk3n8EVwyQ@mail.gmail.com>
 <pj2lgk$c6c$1@blaine.gmane.org>
 <09cb9784-6a97-49b9-b5e7-1c9a9d8c8391@googlegroups.com>
 <CALD=Yf-MNLW7z8s=ruc_C13t7eWp8KC6OWvqf9durZgpN=8=sw@mail.gmail.com>
 <f2acd012-78f4-4477-b787-a6e604d8396d@googlegroups.com>
 <CALD=Yf-z5L1zJ7ESP0CFK+wgx6ajMX=njks54hG_UX3F81oz8g@mail.gmail.com>
Message-ID: <CAFpSVpLvruVf2v1qTn5Erqak_xsryk8Ezaa-JrZf8zjZTnWmfw@mail.gmail.com>

On Mon, Jul 23, 2018 at 6:28 AM, Jonathan Fine <jfine2358 at gmail.com> wrote:

> Hi Gr?gory
>
> You wrote:
> > Oh yes , I see. Seems like almost everybody think using the [...] syntax
> to
> > define reusable slices is a good idea,
>
> > Not sure slice[1::3] can be done, but this has my preference too: it's
> the
> > most direct exposure I can think of...
>
> The slice class already has all the core functionality it needs. This
> thread is looking for something extra. My preference is to create a
> new module, slicetools, that contains the functionality the people on
> this thread want to have.
>
> What I like about this approach is that the code can be written and
> deployed without getting permission from anyone. And when it's done
> and has a body of practice, the PEP would simply be "Add slicetools to
> the standard library".
>
> Anyone here up for working on that sort of approach?
>
>
I only proposed on very limited thing here.  My thought was that if it
wasn't accepted in the slice class the next best place would be a dedicated
class in boltons or toolz or some package like that.

Besides the __getitem__ --> slice classmethod, what other functionality do
you think is needed in such a module?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/400212a3/attachment-0001.html>

From toddrjen at gmail.com  Mon Jul 23 08:19:35 2018
From: toddrjen at gmail.com (Todd)
Date: Mon, 23 Jul 2018 08:19:35 -0400
Subject: [Python-ideas] slice[] to get more complex slices
In-Reply-To: <CACac1F_XOi+1gmEMgZdbmW_3UXs661JS5BF=OvWLrF47H0fJmg@mail.gmail.com>
References: <CAFpSVpKnE8b=QrjxLy6anAF3ammq2QbB4_K1Ey02Vk3n8EVwyQ@mail.gmail.com>
 <pj2lgk$c6c$1@blaine.gmane.org>
 <09cb9784-6a97-49b9-b5e7-1c9a9d8c8391@googlegroups.com>
 <CALD=Yf-MNLW7z8s=ruc_C13t7eWp8KC6OWvqf9durZgpN=8=sw@mail.gmail.com>
 <399ff7d70c204f679bdb5a79f9bc0460@xmail103.UGent.be>
 <4243953379d74d2aa5393be2434a99a7@xmail103.UGent.be>
 <5B55AE6A.7010305@UGent.be>
 <CACac1F_XOi+1gmEMgZdbmW_3UXs661JS5BF=OvWLrF47H0fJmg@mail.gmail.com>
Message-ID: <CAFpSVpKWUbWBDLxY0XBBs7ddWH2rvKKETrkCeysBnx2X=7AfGA@mail.gmail.com>

On Mon, Jul 23, 2018 at 7:24 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 23 July 2018 at 11:31, Jeroen Demeyer <J.Demeyer at ugent.be> wrote:
> > On 2018-07-23 12:24, Jeroen Demeyer wrote:
> >>
> >> Another solution that nobody has mentioned (as far as I know) is to add
> >> additional syntax to the language for that. For example, one could say
> >> that (1:3) could be used to construct slice(1, 3) directly. The
> >> parentheses are required to avoid confusion with type hints. I'm not a
> >> Python language expert, but I don't think that type hints can occur
> >> inside parentheses like that.
> >
> >
> > And this could be extended to tuples (1:3, 2:4) and lists [1:3, 2:4] of
> > slices too.
>
> I thought the reason the proposal got nowhere was because it's pretty
> simple to define it yourself:
>
> >>> class SliceHelper:
> ...     def __getitem__(self, slice):
> ...         return slice
> ...
> >>> SH = SliceHelper()
> >>> SH[1::3]
> slice(1, None, 3)
>
> Did I miss something significant about why this wasn't sufficient?
>
> Paul


That involves initializing an instance, which doesn't serve any purpose in
this case and I was hoping to avoid.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/8f25e21d/attachment.html>

From steve at pearwood.info  Mon Jul 23 08:22:04 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 23 Jul 2018 22:22:04 +1000
Subject: [Python-ideas] Python docs page: In what ways is None special
In-Reply-To: <CALD=Yf_PRXZYiNdC9FGE5pd1cjrS8gyr8q5iPxzrMKXzX027YA@mail.gmail.com>
References: <CALD=Yf_PRXZYiNdC9FGE5pd1cjrS8gyr8q5iPxzrMKXzX027YA@mail.gmail.com>
Message-ID: <20180723122204.GX8744@ando.pearwood.info>

On Mon, Jul 23, 2018 at 10:03:10AM +0100, Jonathan Fine wrote:

> I thought, a page on how None is special would be nice.

I think your document would be a great blog post. I don't think it is 
very helpful as part of the standard documention, as it is more a 
collection of arbitrary facts about None than a coherent "big picture" 
document.

We could come up with such arbitrary lists for many other standard 
objects too:

"Did you know... that operator dunder methods should return 
NotImplemented to signal to the interpreter that the other operand's 
method should be called?"

"Did you know that Ellipsis is the sole instance of class ellipsis, 
reversing the usual convention that classes use InitialCaps and 
instances lowercase?"



-- 
Steve

From mhroncok at redhat.com  Mon Jul 23 08:16:14 2018
From: mhroncok at redhat.com (=?UTF-8?Q?Miro_Hron=c4=8dok?=)
Date: Mon, 23 Jul 2018 14:16:14 +0200
Subject: [Python-ideas] Idea: msgfmt.py and pygettext..py should be -m
 executable modules
Message-ID: <f047b0cb-fc29-d3b5-b68a-18bb2cef1fc8@redhat.com>

Currently, the documentation (as well as plenty of other places on the 
Internet, such as Stack Overflow answers) mentions msgfmt.py and 
pygettext.py.

See https://docs.python.org/3/library/gettext.html

 > (Python also includes pure-Python versions of these programs, called
 > pygettext.py and msgfmt.py; some Python distributions will install
 > them for you. pygettext.py is similar to xgettext, but only
 > understands Python source code and cannot handle other programming
 > languages such as C or C++. pygettext.py supports a command-line
 > interface similar to xgettext; for details on its use, run
 > pygettext.py --help. msgfmt.py is binary compatible with GNU msgfmt.
 > With these two programs, you may not need the GNU gettext package to
 > internationalize your Python applications.)

The problematic part is: "some Python distributions will install them 
for you"

As a Python distributor, I may ask myself a question: Where do I install 
those?

As a Python user, I may ask: Where did the distributors put them, and 
did they?

So I suggest we make those modules instead and we document how to use them.

It might be:

   $ python3 -m gettext

And:

   $ python3 -m gettext.msgfmt

And (provided as a shortcut):

   $ python3 -m msgfmt

I feel that using -m is the general direction Python is going, 
considering this message:

   $ pyvenv ...
   WARNING: the pyenv script is deprecated in favour of `python3.7 -m venv`

It also gives the user a choice for Python version on a system with 
multiple Python versions installed as the user could do:

   $ python3.6 -m gettext

Or

   $ python3.7 -m gettext

While with e.g. /usr/bin/pygettext.py this adds unnecessary burden on 
the distributor to package /usr/bin/pygettext-3.7.py etc...

This idea originates at https://bugzilla.redhat.com/show_bug.cgi?id=1571474

-- 
Miro Hron?ok
--
Phone: +420777974800
IRC: mhroncok

From jfine2358 at gmail.com  Mon Jul 23 08:26:34 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Mon, 23 Jul 2018 13:26:34 +0100
Subject: [Python-ideas] Python docs page: In what ways is None special
In-Reply-To: <20180723121351.GW8744@ando.pearwood.info>
References: <CALD=Yf_PRXZYiNdC9FGE5pd1cjrS8gyr8q5iPxzrMKXzX027YA@mail.gmail.com>
 <d7651538-1831-b7c7-f050-252826591ac2@python.org>
 <CACac1F9qgTdD3Bac9EmKc+Af8EQRFH=KfVWzpOied0Se7qc0EQ@mail.gmail.com>
 <20180723121351.GW8744@ando.pearwood.info>
Message-ID: <CALD=Yf9uEe6yKNSSj5JARCZWruSaziw53+eFnxBk9RpYqR1-6A@mail.gmail.com>

Hi Steve

In this thread you wrote, replying to Paul Moore's comments on PEP 505

> None is already a special value. It is so special, it is one of only
> three built-in values given keyword status. All the other built-ins,
> including such things that are used implicitly by the interpreter (such
> as NotImplemented) are merely ordinary names.

Half an hour earlier, Paul wrote (in reply to me)

>> My goal in this thread is to document clearly and develop a clear
>> shared understanding of the ways in which None is special in Python.
>> Please, please, please can we not discuss PEP 505 in this thread.
>> There's already another thread for that.
>
> OK, apologies for my confusion between the threads.

Paul has implicitly removed from this thread his comments on PEP 505.
If you wish to respond to Paul's comments, please do so elsewhere.
There's already a thread for PEP 505.

-- 
Jonathan

From jfine2358 at gmail.com  Mon Jul 23 08:38:32 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Mon, 23 Jul 2018 13:38:32 +0100
Subject: [Python-ideas] Python docs page: In what ways is None special
In-Reply-To: <20180723122204.GX8744@ando.pearwood.info>
References: <CALD=Yf_PRXZYiNdC9FGE5pd1cjrS8gyr8q5iPxzrMKXzX027YA@mail.gmail.com>
 <20180723122204.GX8744@ando.pearwood.info>
Message-ID: <CALD=Yf_jToxOguVsTdP1sW_5BXmK69iX0N4dnSmG-ECb9ZCb_w@mail.gmail.com>

Hi Steve

You wrote

> I think your document would be a great blog post. I don't think it is
> very helpful as part of the standard documention, as it is more a
> collection of arbitrary facts about None than a coherent "big picture"
> document.

Thank you. That's nice. However, I think there is, or should be, more
coherence than you're seeing.

Here's something I forgot to say. For standard documentation I was
thinking The Python Tutorial.  For example, the Brief Tour.
https://docs.python.org/3/tutorial/stdlib.html
https://docs.python.org/3/tutorial/stdlib2.html

That said, a blog post would be useful stepping stone, and the
standard documentation later. (Know where your heading, take one step
at a time.)

Once again, thank you for your comment.

-- 
Jonathan

From p.f.moore at gmail.com  Mon Jul 23 08:41:37 2018
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 23 Jul 2018 13:41:37 +0100
Subject: [Python-ideas] slice[] to get more complex slices
In-Reply-To: <CAFpSVpKWUbWBDLxY0XBBs7ddWH2rvKKETrkCeysBnx2X=7AfGA@mail.gmail.com>
References: <CAFpSVpKnE8b=QrjxLy6anAF3ammq2QbB4_K1Ey02Vk3n8EVwyQ@mail.gmail.com>
 <pj2lgk$c6c$1@blaine.gmane.org>
 <09cb9784-6a97-49b9-b5e7-1c9a9d8c8391@googlegroups.com>
 <CALD=Yf-MNLW7z8s=ruc_C13t7eWp8KC6OWvqf9durZgpN=8=sw@mail.gmail.com>
 <399ff7d70c204f679bdb5a79f9bc0460@xmail103.UGent.be>
 <4243953379d74d2aa5393be2434a99a7@xmail103.UGent.be>
 <5B55AE6A.7010305@UGent.be>
 <CACac1F_XOi+1gmEMgZdbmW_3UXs661JS5BF=OvWLrF47H0fJmg@mail.gmail.com>
 <CAFpSVpKWUbWBDLxY0XBBs7ddWH2rvKKETrkCeysBnx2X=7AfGA@mail.gmail.com>
Message-ID: <CACac1F-8GM6KqNQAz0tSrZjVOQ_KUKC1JcNjyvzxM+nLcX02PQ@mail.gmail.com>

On 23 July 2018 at 13:19, Todd <toddrjen at gmail.com> wrote:
> On Mon, Jul 23, 2018 at 7:24 AM, Paul Moore <p.f.moore at gmail.com> wrote:
>>
>> On 23 July 2018 at 11:31, Jeroen Demeyer <J.Demeyer at ugent.be> wrote:
>> > On 2018-07-23 12:24, Jeroen Demeyer wrote:
>> >>
>> >> Another solution that nobody has mentioned (as far as I know) is to add
>> >> additional syntax to the language for that. For example, one could say
>> >> that (1:3) could be used to construct slice(1, 3) directly. The
>> >> parentheses are required to avoid confusion with type hints. I'm not a
>> >> Python language expert, but I don't think that type hints can occur
>> >> inside parentheses like that.
>> >
>> >
>> > And this could be extended to tuples (1:3, 2:4) and lists [1:3, 2:4] of
>> > slices too.
>>
>> I thought the reason the proposal got nowhere was because it's pretty
>> simple to define it yourself:
>>
>> >>> class SliceHelper:
>> ...     def __getitem__(self, slice):
>> ...         return slice
>> ...
>> >>> SH = SliceHelper()
>> >>> SH[1::3]
>> slice(1, None, 3)
>>
>> Did I miss something significant about why this wasn't sufficient?
>>
>> Paul
>
>
> That involves initializing an instance, which doesn't serve any purpose in
> this case and I was hoping to avoid.

Well it serves the purpose that you can do it already in current
Python, rather than needing a core interpreter change and limiting
your code to Python 3.8+ only ;-)

Paul

From gregory.lielens at gmail.com  Mon Jul 23 08:46:31 2018
From: gregory.lielens at gmail.com (=?UTF-8?Q?Gr=C3=A9gory_Lielens?=)
Date: Mon, 23 Jul 2018 05:46:31 -0700 (PDT)
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CACac1F_jKbotHsokdB1vcPBbJFSyDqQpv6T4h8VunT-qhpkjLg@mail.gmail.com>
References: <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAAu18hc35N11jVQj+0691aH8Oe2+4Cqfn0yFpwWtjqou161ATg@mail.gmail.com>
 <0ffb89a1-3f63-a49d-e5a9-e98b25f60616@python.org>
 <20180723121156.4337a316@fsol>
 <beac393d-08db-62ae-ac17-7745525683a8@python.org>
 <b50a44e4-b3c8-3df1-ab1c-f0a0d802244e@python.org>
 <a46fc434-1bbb-f17c-42ef-840e3d121572@python.org>
 <f7363e81-9a77-4178-2f22-7281b0d5e917@python.org>
 <13681cf0-1571-2b74-e451-bb234f1952b7@python.org>
 <a79f0fbc-6277-4a01-a7d7-2255192aa4ec@googlegroups.com>
 <CACac1F_jKbotHsokdB1vcPBbJFSyDqQpv6T4h8VunT-qhpkjLg@mail.gmail.com>
Message-ID: <ec40dae6-01f5-4ae5-a8f4-6baf116934fc@googlegroups.com>

 Paul Moore wrote:
>
> This is my impression, as well. It seems like something that's helpful
> in dealing with unstructured object hierarchies with lots of optional
> attributes - which is where JSON tends to be used.
>
> But given that, I'm really much more interested in seeing the new
> operators compared against a well-written "JSON object hierarchy
> traversal" library than against raw Python code.
>

Good point, I did not think about that when suggesting to give some 
context, but indeed if it's linked to a library in particular, there is 
always the possibility to choose another object than None as the "nothing 
here" marker. 

One that will absorb getitems and  getattr accesses so that ?.  and ?[] 
behavior is reproduced by plain . and []. Guard/NoValues should be chosen 
so that typical lib use is easier. 


Anyway, addressing partially-populated nodes would need lib support: 
None-coalescence will not help when you traverse a dict/attribute hierarchy 
where some nodes implement some attributes/keys but not others. It help 
only when a node is either regular, or a guard object without any 
attributes...So an irregular tree, but not too irregular ;-)... 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/0366b983/attachment.html>

From toddrjen at gmail.com  Mon Jul 23 09:00:12 2018
From: toddrjen at gmail.com (Todd)
Date: Mon, 23 Jul 2018 09:00:12 -0400
Subject: [Python-ideas] slice[] to get more complex slices
In-Reply-To: <CACac1F-8GM6KqNQAz0tSrZjVOQ_KUKC1JcNjyvzxM+nLcX02PQ@mail.gmail.com>
References: <CAFpSVpKnE8b=QrjxLy6anAF3ammq2QbB4_K1Ey02Vk3n8EVwyQ@mail.gmail.com>
 <pj2lgk$c6c$1@blaine.gmane.org>
 <09cb9784-6a97-49b9-b5e7-1c9a9d8c8391@googlegroups.com>
 <CALD=Yf-MNLW7z8s=ruc_C13t7eWp8KC6OWvqf9durZgpN=8=sw@mail.gmail.com>
 <399ff7d70c204f679bdb5a79f9bc0460@xmail103.UGent.be>
 <4243953379d74d2aa5393be2434a99a7@xmail103.UGent.be>
 <5B55AE6A.7010305@UGent.be>
 <CACac1F_XOi+1gmEMgZdbmW_3UXs661JS5BF=OvWLrF47H0fJmg@mail.gmail.com>
 <CAFpSVpKWUbWBDLxY0XBBs7ddWH2rvKKETrkCeysBnx2X=7AfGA@mail.gmail.com>
 <CACac1F-8GM6KqNQAz0tSrZjVOQ_KUKC1JcNjyvzxM+nLcX02PQ@mail.gmail.com>
Message-ID: <CAFpSVpK8h_VHv7URkV40fYO2FD+WCC_Ooi9A2DTea=wP--GUmw@mail.gmail.com>

On Mon, Jul 23, 2018 at 8:41 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 23 July 2018 at 13:19, Todd <toddrjen at gmail.com> wrote:
> > On Mon, Jul 23, 2018 at 7:24 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> >>
> >> On 23 July 2018 at 11:31, Jeroen Demeyer <J.Demeyer at ugent.be> wrote:
> >> > On 2018-07-23 12:24, Jeroen Demeyer wrote:
> >> >>
> >> >> Another solution that nobody has mentioned (as far as I know) is to
> add
> >> >> additional syntax to the language for that. For example, one could
> say
> >> >> that (1:3) could be used to construct slice(1, 3) directly. The
> >> >> parentheses are required to avoid confusion with type hints. I'm not
> a
> >> >> Python language expert, but I don't think that type hints can occur
> >> >> inside parentheses like that.
> >> >
> >> >
> >> > And this could be extended to tuples (1:3, 2:4) and lists [1:3, 2:4]
> of
> >> > slices too.
> >>
> >> I thought the reason the proposal got nowhere was because it's pretty
> >> simple to define it yourself:
> >>
> >> >>> class SliceHelper:
> >> ...     def __getitem__(self, slice):
> >> ...         return slice
> >> ...
> >> >>> SH = SliceHelper()
> >> >>> SH[1::3]
> >> slice(1, None, 3)
> >>
> >> Did I miss something significant about why this wasn't sufficient?
> >>
> >> Paul
> >
> >
> > That involves initializing an instance, which doesn't serve any purpose
> in
> > this case and I was hoping to avoid.
>
> Well it serves the purpose that you can do it already in current
> Python, rather than needing a core interpreter change and limiting
> your code to Python 3.8+ only ;-)
>
> Paul
>

Making it a classmethod would only require a metaclass, not a core
interpreter change.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/1aeecbba/attachment.html>

From mehaase at gmail.com  Mon Jul 23 09:37:44 2018
From: mehaase at gmail.com (Mark E. Haase)
Date: Mon, 23 Jul 2018 09:37:44 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAAu18hc35N11jVQj+0691aH8Oe2+4Cqfn0yFpwWtjqou161ATg@mail.gmail.com>
References: <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
 <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAAu18hc35N11jVQj+0691aH8Oe2+4Cqfn0yFpwWtjqou161ATg@mail.gmail.com>
Message-ID: <CALb0Rk4ph4LXEShiT=rsZEmd7D1ACdxZ_w6AS2WcEaqPnOWFNg@mail.gmail.com>

On Mon, Jul 23, 2018 at 2:23 AM Nicholas Cole <nicholas.cole at gmail.com>
wrote:

> One issue for me is that the trivial case is already a one-liner:
>
> if a is None: a = 10
>

Yes, if you have no indentation and a 1-character name, then it fits on a
single line. If you have a longer expression and/or side effects, then it's
not a one-liner anymore.


> And that leads to a simple question: how many times does this actually
> occur in real-world by python code? -- i.e. how many times do I want
> to check the value of an existing label, and, finding it is None (and
> None specifically), then assign it a value?
>
> What does a scan through the existing core library say?


Please read the PEP before you shoot it down. It answers this _exact_
question.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/53f97bab/attachment-0001.html>

From nicholas.cole at gmail.com  Mon Jul 23 10:02:31 2018
From: nicholas.cole at gmail.com (Nicholas Cole)
Date: Mon, 23 Jul 2018 15:02:31 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CALb0Rk4ph4LXEShiT=rsZEmd7D1ACdxZ_w6AS2WcEaqPnOWFNg@mail.gmail.com>
References: <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
 <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAAu18hc35N11jVQj+0691aH8Oe2+4Cqfn0yFpwWtjqou161ATg@mail.gmail.com>
 <CALb0Rk4ph4LXEShiT=rsZEmd7D1ACdxZ_w6AS2WcEaqPnOWFNg@mail.gmail.com>
Message-ID: <CAAu18hfrHfo27kLCqx1B6sY4FU3niweGKawFvXhQ0QDOuk0F6g@mail.gmail.com>

On Mon, Jul 23, 2018 at 2:37 PM Mark E. Haase <mehaase at gmail.com> wrote:

>> What does a scan through the existing core library say?
>
>
> Please read the PEP before you shoot it down. It answers this _exact_ question.

My apologies.  I'd missed the line where it says the examples were
taken from the core library. So then a couple of points. Given the
size of the core library, 678 seems like a very low number to me for a
proposal like this -- but then I don't know quite what the threshold
number would be for something like this.  I guess I had expected that
if None really were used in this way that there would be thousands of
places where it might be used in the core libraries.   Secondly (and I
mean this with no animus at all), in some of the examples given I
really struggle to see that the new version is more readable /
maintainable / obvious than the old version.  I can see the point in a
couple of others.  You'll say this is subjective as a test, and it is,
I suppose.

N.

From steve at pearwood.info  Mon Jul 23 10:03:25 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 24 Jul 2018 00:03:25 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <1b7f56aa-b851-8604-5c94-96aeed77efa6@python.org>
References: <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
 <CAFYqXL_3GZpB2+3SSvjzJtkrjyRoC+=iU9yx2fDAs5tAE_DQ1Q@mail.gmail.com>
 <20180722121006.GQ8744@ando.pearwood.info>
 <CAEbHw4b7hSgu28RwfwMav=cR-uxrErnGBDpeDVuirOevFvLo3g@mail.gmail.com>
 <6562164f-9be1-40ec-890d-508fd5c81014@googlegroups.com>
 <CAPTjJmr8cMiW_K1jZwa4WJ_Tmy406+MxrXF_XR+OLWc0W47r3w@mail.gmail.com>
 <20180723005114.GT8744@ando.pearwood.info>
 <1b7f56aa-b851-8604-5c94-96aeed77efa6@python.org>
Message-ID: <20180723140324.GZ8744@ando.pearwood.info>

On Mon, Jul 23, 2018 at 10:48:23AM +0100, Steve Dower wrote:
> On 23Jul2018 0151, Steven D'Aprano wrote:
> >What if there was a language
> >supported, non-hackish way to officially delay evaluation of
> >expressions until explicitly requested?
> 
> The current spelling for this is "lambda: delayed-expression" and the 
> way to request the value is "()". :)
> 
> (I'm not even being that facetious here. People ask for delayed 
> expressions all the time, and it's only 7 characters, provided the 
> callee knows they're getting it, and the semantics are already well 
> defined and likely match what you want.)

I know you not being facetious, and delaying computation through a 
function call is not an awful solution. But its not a great solution 
either. Contrast the elegance of syntax with delayed evaluation:

    1/x if x != 0 else func(y)

versus the lambda solution:

    if_else(lambda: 1/x, x != 0, lambda: func(y))()


For clarity, or perhaps the opposite *wink* I've kept the same order of 
arguments. It's not just the extra seven characters (plus spaces) per 
delayed expression, or the extra parentheses at the end to force 
evaluation, but the ease at which we can forget and write this:

    if_else(1/x, x != 0, func(y))


Anyway, it was just an idle thought, not in a state to compete with this 
PEP. And even if it were, I'd still prefer to see at least ?? as a 
dedicated operator rather than a function.



-- 
Steve

From andre.roberge at gmail.com  Mon Jul 23 10:03:24 2018
From: andre.roberge at gmail.com (Andre Roberge)
Date: Mon, 23 Jul 2018 11:03:24 -0300
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <0ffb89a1-3f63-a49d-e5a9-e98b25f60616@python.org>
References: <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
 <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAAu18hc35N11jVQj+0691aH8Oe2+4Cqfn0yFpwWtjqou161ATg@mail.gmail.com>
 <0ffb89a1-3f63-a49d-e5a9-e98b25f60616@python.org>
Message-ID: <CAGMu_=qB5+yi2mLVzneZKpRYj_qJ8bn989qGZ1z=_ZyFBjDb9Q@mail.gmail.com>

On Mon, Jul 23, 2018 at 6:52 AM Steve Dower <steve.dower at python.org> wrote:

> Responding to a few more ideas that have come up here.
>


?Thank you for the clarifications.?

?I'm trying to wrap my head around the various facets of None aware
operators proposal after reading the whole discussion - as well as having
read the PEP a few times.  Below is a
summary of what I gathered from the discussion, with a few added other
points that I have not seen addressed.

1. It is an operator present in other languages (e.g. C#, Dart), that uses
the same notation

a) This demonstrates that such an operator has been found to be useful and
is
definitely worth considering.

b) The fact that it uses the same notation is a plus for people that know
these other languages but it does not mean that the notation is necessarily
the best choice for Python. To wit, Python does not use the combination of
? and
: for ternary operator; instead it reuses existing keywords.

c) Some people use languages (e.g. Ruby) that allow ? to be part of an
identifier,
which means that one could, in principle, write something like

a = b?.c

in Ruby with a completely different meaning than what it would mean in C#
or Dart,
or as proposed for Python.Thus, I don't think that that language X uses
this operator written this way is,
**on its own**, a valid justification for using the exact same syntax for
Python.

d) Dart is given as an example. Reading from the link mentioned in the PEP,
I was lead to https://www.dartlang.org/guides/language/language-tour#classes
where the only mention of ?. was the following:

===
// If p is non-null, set its y value to 4.
p?.y = 4;
===

However, PEP 505 mentions that something like the above would raise a
SyntaxError.
Given that Dart's operator has a different semantics than that proposed for
Python,
I do not think that mentioning Dart in this context without any qualifier
is a valid supportive justification for this proposal.

2. On the specific choices of ??, ??=, ?., ?[]

a) Trying to find what Csharp ?? means by doing a quick internet search is
not exactly
productive.  By comparison, if one were to use a new keyword (say ifnone
instead
of ??) or keywords, it would make it easier to find their meaning. The
problem with new
keywords is that they may introduce some backwards incompatibility.

b) Admitedly, there are very few remaining symbols in Python that can be
used for defining new operators. Introducing operators using ? does not
cause problems with existing code.

c) While using a new keyword for ?? and ??= might work, it is less clear,
at least to me, how this could extend to ?. and ?[]

d) ? and ?? are already used by IPython with a totally different meaning.
I almost never use IPython I have no idea what problems this might cause
for IPython users.

3. On some examples given

PEP 505 gives these examples from Request:

data = [] if data is None else data
files = [] if files is None else files
headers = {} if headers is None else headers
params = {} if params is None else params
hooks = {} if hooks is None else hooks

It then argues that this is undesirable and that it could have been written
instead
as

data = data if data is not None else []
files = files if files is not None else []
headers = headers if headers is not None else {}
params = params if params is not None else {}
hooks = hooks if hooks is not None else {}

which, as written in PEP 505, is deemed to be "more intuitive" - but longer.

Having looked at the code in the Request module, I would argue that it
could have been written instead as

if data is None: data = []
if files is None: files = []
if headers is None: headers = {}
if params is None: params = {}
if hooks is None: hooks = {}

which gives the same result and is shorter than the original - but
admittedly longer than
the proposed notation. I do not think that this specific example as
currently written
in PEP 505 gives a fair picture of what is currently possible with Python.

4) On comparisons with other "domain specific operators", like @ and
the bitwise operators.

I do not remember the exact words that were used in the discussions, but
I seem to recall people saying that operators like ??, ?., etc. are no
more "mysterious" than @ or the bitwise operators might be: users should
not be surprised to have to learn new operators.

Anyone that does numerical work knows that matrix multiplications do not
obey the same rules as multiplications of numbers. If you don't do numerical
work, you are almost certain not to encounter @ as an operator (but only
as a symbol for decorator), so there won't be any mystery to solve.
A similar situation exists for bitwise operators.
Having dedicated operators to represent special methods on these types of
objects makes the code easier to read for people working in these fields.
I also note that these operators have corresponding dunder methods.

By contrast, code like

if a is None:
   a = []

can occur in pretty much any type of program and is, arguably, already very
readable.
Saying that it could/should be written instead as

a ??= []

where ??= would be an operator without corresponding dunder method is not
the
same. Unless I am mistaken, it would be the first time that operators
are introduced in Python without having corresponding dunder methods.
If I am correct in this, I believe that the PEP should at the very least
address this situation and provide an explanation as to why new operators
should be introduced that break this non-written rule in Python.

= = =

**Personal biases**: I do admit that I prefer code written in indented
blocks
to show the logic, over trying to cram everything in as few lines as
possible.  (The only exception to this is for list comprehensions,
but this might be because I am very familiar with mathematical notation
used for sets.)

Thus, I would likely always prefer to write

    if a is None:
        a = []

over

    if a is None: a = []


but I do admit that, after the initial shock, I do find

    a ??= []

to be fairly readable.  One thing I do find helpful is that it is
possible (and desirable) to have spaces around the operator.
Still, if such an operator was introduced, I would prefer a new
keyword over the proposed symbols, to be consistent with choices made
when the ternary operator construct was introduced.

However, even after reading it over many times, I do find code using
?. and ?[] to be much harder to read then using the current notation.
Python is often described as executable pseudo-code, which I consider
one of its strengths, and I find these proposed operators to look like
anything but executable pseudo-code.

Unlike the recently approved new operator, :=, whose meaning can be
at least guessed based on the presence of "=", there is nothing
intuitive about the choices of operators built from ?. The fact that
they would not be easily found by doing an internet search (other
than specifically looking for "python operators") compared with a search
for keywords based operators (e.g. "python ifnone") is not desirable imo.?

?Andr? Roberge?




> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/5f23e939/attachment-0001.html>

From mertz at gnosis.cx  Mon Jul 23 10:30:09 2018
From: mertz at gnosis.cx (David Mertz)
Date: Mon, 23 Jul 2018 10:30:09 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <0ffb89a1-3f63-a49d-e5a9-e98b25f60616@python.org>
References: <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
 <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAAu18hc35N11jVQj+0691aH8Oe2+4Cqfn0yFpwWtjqou161ATg@mail.gmail.com>
 <0ffb89a1-3f63-a49d-e5a9-e98b25f60616@python.org>
Message-ID: <CAEbHw4ZFnXr-zcm+wy-U4hs3nLwz-CPKXQxEJsyxgZ49a1mSjA@mail.gmail.com>

On Mon, Jul 23, 2018 at 5:52 AM Steve Dower <steve.dower at python.org> wrote:

> The PEP author is unsure about how it works
> -------------------------------------------
> I wish this statement had come with some context, because the only thing
> I'm unsure about is what I'm supposed to be unsure about.
>

In general?as I haven't been shy of saying?I find the entire idea awful.  I
recognize you have done sincere and high quality work in arguing for it; it
just feels like a very wrong direction for Python.

But bracketing that, I'm probably the person you have in mind in that
comment.  And it's funny that you write you are unsure what you are
supposed to be unsure about, but the very next section is exactly what I
had in mind.  Of course I don't mean that if implemented the semantics
would be ambiguous... rather, the proper "swallowing" of different kinds of
exceptions is not intuitively obvious, not even to you, Steve.  And if some
decision was reached and documented, it would remain unclear to new (or
even experienced) users of the feature.

I'm borderline on ?[] right now. Honestly, I think it works best if it
> also silently handles LookupError (e.g. for traversing a loaded JSON
> dict), but then it's inconsistent with ?. which I think works best if it
> handles None but allows AttributeError.
>

Moreover, at a couple points in your clarification, you say "ignoring
descriptors." But ultimately, the language cannot do that.  When a
programmer writes `obj?.descriptor?.other_descriptor` SOMETHING has to
happen (no matter what actually happens within the code of the property).

This can certainly be specified in some unambiguous way, but I believe that
any decision made will be strongly counter-intuitive for certain concrete
code.

-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/44675656/attachment.html>

From g.rodola at gmail.com  Mon Jul 23 11:11:55 2018
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Mon, 23 Jul 2018 17:11:55 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <0ffb89a1-3f63-a49d-e5a9-e98b25f60616@python.org>
References: <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
 <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAAu18hc35N11jVQj+0691aH8Oe2+4Cqfn0yFpwWtjqou161ATg@mail.gmail.com>
 <0ffb89a1-3f63-a49d-e5a9-e98b25f60616@python.org>
Message-ID: <CAFYqXL_uEv3d++e9rRgpTM2kawOeTFm7DJ6UYR8ePpDyaskxVw@mail.gmail.com>

On Mon, Jul 23, 2018 at 11:52 AM Steve Dower <steve.dower at python.org> wrote:

> I'm borderline on ?[] right now. Honestly, I think it works best if it
> also silently handles LookupError (e.g. for traversing a loaded JSON
> dict), but then it's inconsistent with ?. which I think works best if it
> handles None but allows AttributeError.

That would easily make typos pass unnoticed:

    request.context.user.usernme  # raises AttributeError
    request?.context?.user?.usernme  # return None

Same goes for LookupError: if a key or index is missing on 'a?[b]' I
do want an exception. If I don't, which should be the exception rather
than the rule, I will simply take the risk myself and do:

    default = '<no name>'
    try:
        name = d['user']['details']['name'] or default
    except KeyError:
        name = default

But certainly there should be no native syntax encouraging me to do
any of that. Talking about arbitrarily swallowing exceptions is the
worst direction this proposal can take as it breaks yet another
fundamental Python Zen: "errors should never pass silently". IMO this
shows how fundamentally detached from the base philosophy of the
language this whole idea is.

--
Giampaolo - http://grodola.blogspot.com

From mertz at gnosis.cx  Mon Jul 23 11:12:45 2018
From: mertz at gnosis.cx (David Mertz)
Date: Mon, 23 Jul 2018 11:12:45 -0400
Subject: [Python-ideas] A better (simpler) approach to PEP 505
Message-ID: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>

The need addressed by PEP 505 is real; it's also MUCH more niche and
uncommon than something that would merit new syntax.  Moreover, the actual
legitimate purpose served by the PEP 505 syntax is easily served by
existing Python simply by using a wrapper class.

Here is a way of solving the "deep attribute access to messy data" problem
that is:

(1) Much more explicit
(2) Requires no change in syntax
(3) Will not be a bug magnet
(4) Inasmuch as there are semantic traps, they are announced by the use of
a class whose documentation would be pointed to for readers

The API that could be useful might be something like this:

In [1]: from none_aware import NoneAware
In [2]: from types import SimpleNamespace
In [3]: foo = SimpleNamespace()
In [4]: foo.bar = SimpleNamespace()
In [5]: foo.bar.baz = SimpleNamespace()
In [6]: foo.bar.baz.blat = 42
In [7]: NoneAware(foo).bar.blim
Out[7]: <none_aware.NoneAware at 0x11156a748>
In [8]: NoneAware(foo).bar.blim.unbox()
In [9]: NoneAware(foo).bar.baz.blat.unbox()
Out[9]: 42
In [10]: NoneAware(foo).bar.baz.blat
Out[10]: <none_aware.NoneAware at 0x11157d908>
In [11]: NoneAware(foo).bar.baz.flam.unbox()
In [12]: NoneAware(foo).bar.baz.flam
Out[12]: <none_aware.NoneAware at 0x1115832b0>


The particular names I use are nothing special, and better ones might be
found.  I just called the class NoneAware and the "escape" method
`.unbox()` because that seemed intuitive at first brush.

I don't disagree that needing to call .unbox() at the end of the chained
attribute access is a little bit ugly.  But it's a lot less ugly than large
family of new operators.  And honestly, it's a nice way of being explicit
about the fact that we're entering then leaving a special world where
attribute accesses don't fail.

I haven't implemented the equivalent dictionary lookups in the below.  That
would be straightforward, and I'm sure my 5 minute throwaway code could be
improved in other ways also.  But something better than this in the
standard library would address ALL the actual needs described in PEP 505.
Even the pattern Steve Dower is especially fond of like:

favorite = cfg?.user?.profile?.food ?? "Spam"


(i.e. a configuration may be incomplete at any level, if levels are missing
default favorite food is Spam).  We could simply spell that:

favorite = NoneAware(cfg, "Spam").user.profile.food.unbox()


I think that's 14 characters more in this example, but still compact.  We
could get that down to 2 characters if we used one-letter names for the
class and method.  I suppose down to zero characters if .unbox() was a
property.

So completely toy implementation:

class NoneAware(object):
    def __init__(self, thing, sentinal=None):
        self.thing = thing
        self.sentinal = sentinal

    def __getattr__(self, attr):
        try:
            return NoneAware(getattr(self.thing, attr))
        except AttributeError:
            return NoneAware(self.sentinal)

    def unbox(self):
        return self.thing

-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/bc2a74ab/attachment-0001.html>

From guido at python.org  Mon Jul 23 11:16:33 2018
From: guido at python.org (Guido van Rossum)
Date: Mon, 23 Jul 2018 08:16:33 -0700
Subject: [Python-ideas] slice[] to get more complex slices
In-Reply-To: <5B55ACC9.8070303@UGent.be>
References: <CAFpSVpKnE8b=QrjxLy6anAF3ammq2QbB4_K1Ey02Vk3n8EVwyQ@mail.gmail.com>
 <pj2lgk$c6c$1@blaine.gmane.org>
 <09cb9784-6a97-49b9-b5e7-1c9a9d8c8391@googlegroups.com>
 <CALD=Yf-MNLW7z8s=ruc_C13t7eWp8KC6OWvqf9durZgpN=8=sw@mail.gmail.com>
 <399ff7d70c204f679bdb5a79f9bc0460@xmail103.UGent.be>
 <5B55ACC9.8070303@UGent.be>
Message-ID: <CAP7+vJLyM6HG6YShSN2mB=Hpek3Rxp2ypR9ZLn1fRTDZwOgZpg@mail.gmail.com>

On Mon, Jul 23, 2018 at 3:24 AM, Jeroen Demeyer <J.Demeyer at ugent.be> wrote:

> On 2018-07-23 11:58, Gr?gory Lielens wrote:
>
>> Not sure slice[1::3] can be done
>>
>
> It can be done. Since "slice" is a class, it would require a metaclass
> though.
>

Since PEP 560 it won't need a metaclass -- it can be implemented as
__class_getitem__.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/9fddab25/attachment.html>

From p.f.moore at gmail.com  Mon Jul 23 11:26:51 2018
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 23 Jul 2018 16:26:51 +0100
Subject: [Python-ideas] A better (simpler) approach to PEP 505
In-Reply-To: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
References: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
Message-ID: <CACac1F_f67oA22vFkuHH+d6xP5nFhNdj4oEF4vwGYU+uk-e9Jw@mail.gmail.com>

On 23 July 2018 at 16:12, David Mertz <mertz at gnosis.cx> wrote:
> The need addressed by PEP 505 is real; it's also MUCH more niche and
> uncommon than something that would merit new syntax.  Moreover, the actual
> legitimate purpose served by the PEP 505 syntax is easily served by existing
> Python simply by using a wrapper class.
>
> Here is a way of solving the "deep attribute access to messy data" problem
> that is:
>
> (1) Much more explicit
> (2) Requires no change in syntax
> (3) Will not be a bug magnet
> (4) Inasmuch as there are semantic traps, they are announced by the use of a
> class whose documentation would be pointed to for readers

[...]

> I haven't implemented the equivalent dictionary lookups in the below.  That
> would be straightforward, and I'm sure my 5 minute throwaway code could be
> improved in other ways also.  But something better than this in the standard
> library would address ALL the actual needs described in PEP 505.  Even the
> pattern Steve Dower is especially fond of like:
>
> favorite = cfg?.user?.profile?.food ?? "Spam"
>
>
> (i.e. a configuration may be incomplete at any level, if levels are missing
> default favorite food is Spam).  We could simply spell that:
>
> favorite = NoneAware(cfg, "Spam").user.profile.food.unbox()

Thank you. That's the sort of "what would this look like if
implemented as a library rather than language syntax" solution that I
had in mind with my earlier post.

I would be very interested to hear discussion of the pros and cons of
adding new syntax to the language as per PEP 505 in comparison to a
solution like this (ultimately more fleshed out and "production
quality") rather than comparisons PEP 505 to raw "roll your own"
Python code.

For me:

* Library solution works with all versions of Python
* The need for unbox is a little ugly, but arguably less so than ?.
(conceded that's a subjective view)
* Mixing ?. and . is terser than unboxing and reboxing - but are there
any real examples where that's needed?
* Having the default at the beginning rather than at the end doesn't
follow natural reading order (but again that's pretty subjective)

There's little here that isn't subjective, so I expect a lot of heated
debate that ultimately convinces no-one. But the "you can use this
solution now" aspect of the library solution seems pretty compelling
to me - at least in terms of "let's publish the library solution and
then based on experience with it, review PEP 505".

Paul

From robertve92 at gmail.com  Mon Jul 23 11:32:47 2018
From: robertve92 at gmail.com (Robert Vanden Eynde)
Date: Mon, 23 Jul 2018 17:32:47 +0200
Subject: [Python-ideas] A better (simpler) approach to PEP 505
In-Reply-To: <CACac1F_f67oA22vFkuHH+d6xP5nFhNdj4oEF4vwGYU+uk-e9Jw@mail.gmail.com>
References: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
 <CACac1F_f67oA22vFkuHH+d6xP5nFhNdj4oEF4vwGYU+uk-e9Jw@mail.gmail.com>
Message-ID: <CA+msPNmjFjJvTxrR28brMMCjPDVoW+t4g6L4qiVpJBmETqOZLg@mail.gmail.com>

The default could be at the end with an argument to unboxing :

favorite = NoneAware(cfg).user.profile.food.unbox("Spam")


Le lun. 23 juil. 2018 ? 17:26, Paul Moore <p.f.moore at gmail.com> a ?crit :

> On 23 July 2018 at 16:12, David Mertz <mertz at gnosis.cx> wrote:
> > The need addressed by PEP 505 is real; it's also MUCH more niche and
> > uncommon than something that would merit new syntax.  Moreover, the
> actual
> > legitimate purpose served by the PEP 505 syntax is easily served by
> existing
> > Python simply by using a wrapper class.
> >
> > Here is a way of solving the "deep attribute access to messy data"
> problem
> > that is:
> >
> > (1) Much more explicit
> > (2) Requires no change in syntax
> > (3) Will not be a bug magnet
> > (4) Inasmuch as there are semantic traps, they are announced by the use
> of a
> > class whose documentation would be pointed to for readers
>
> [...]
>
> > I haven't implemented the equivalent dictionary lookups in the below.
> That
> > would be straightforward, and I'm sure my 5 minute throwaway code could
> be
> > improved in other ways also.  But something better than this in the
> standard
> > library would address ALL the actual needs described in PEP 505.  Even
> the
> > pattern Steve Dower is especially fond of like:
> >
> > favorite = cfg?.user?.profile?.food ?? "Spam"
> >
> >
> > (i.e. a configuration may be incomplete at any level, if levels are
> missing
> > default favorite food is Spam).  We could simply spell that:
> >
> > favorite = NoneAware(cfg, "Spam").user.profile.food.unbox()
>
> Thank you. That's the sort of "what would this look like if
> implemented as a library rather than language syntax" solution that I
> had in mind with my earlier post.
>
> I would be very interested to hear discussion of the pros and cons of
> adding new syntax to the language as per PEP 505 in comparison to a
> solution like this (ultimately more fleshed out and "production
> quality") rather than comparisons PEP 505 to raw "roll your own"
> Python code.
>
> For me:
>
> * Library solution works with all versions of Python
> * The need for unbox is a little ugly, but arguably less so than ?.
> (conceded that's a subjective view)
> * Mixing ?. and . is terser than unboxing and reboxing - but are there
> any real examples where that's needed?
> * Having the default at the beginning rather than at the end doesn't
> follow natural reading order (but again that's pretty subjective)
>
> There's little here that isn't subjective, so I expect a lot of heated
> debate that ultimately convinces no-one. But the "you can use this
> solution now" aspect of the library solution seems pretty compelling
> to me - at least in terms of "let's publish the library solution and
> then based on experience with it, review PEP 505".
>
> Paul
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/eb204805/attachment.html>

From jfine2358 at gmail.com  Mon Jul 23 11:35:21 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Mon, 23 Jul 2018 16:35:21 +0100
Subject: [Python-ideas] A better (simpler) approach to PEP 505
In-Reply-To: <CACac1F_f67oA22vFkuHH+d6xP5nFhNdj4oEF4vwGYU+uk-e9Jw@mail.gmail.com>
References: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
 <CACac1F_f67oA22vFkuHH+d6xP5nFhNdj4oEF4vwGYU+uk-e9Jw@mail.gmail.com>
Message-ID: <CALD=Yf9p02vLuBfdzBM_njmf6zajG-2izOgUqj3KVqFo1mR58g@mail.gmail.com>

Hi David and Paul

Thank you both for your contributions, and for starting a new thread.

David, you wrote
> The need addressed by PEP 505 is real;

I completely agree. My view is that with PEP 505 as it it, the problem
is better than the solution. But all the same, something can be done
to improve matters.

> Moreover, the actual
> legitimate purpose served by the PEP 505 syntax is easily served by existing
> Python simply by using a wrapper class.

I'm with you here also. First explore solving the problem using
existing syntax. If nothing else, it helps us understand the problem.

There's one area where I find attraction in a new syntax. The ??
operator is well described, I think, as a None-aware 'or'. But not in
the name '??'. We all understand
    val1 = EXP1 or EXP2
So how about
    val2 = EXP1 or-if-None EXP2
with 'or-if-None' being a new language keyword!

> (1) Much more explicit
> (2) Requires no change in syntax
> (3) Will not be a bug magnet
> (4) [Fewer, clear] semantic traps

These are all good goals. And (2) makes exploration much easier.

> The API that could be useful might be something like this:

I have some different ideas, which I'll post later (to this thread, if
you don't mind).

Paul, you wrote

> I would be very interested to hear discussion of the pros and cons of
> adding new syntax to the language as per PEP 505 in comparison to a
> solution like this (ultimately more fleshed out and "production
> quality") rather than comparisons PEP 505 to raw "roll your own"
> Python code.

I like that approach. I find exploration using Python modules to be
more open (democratic?) than syntax changes that require building a
new executable.

Once again, thank you both for starting at the problem and pointing in
a different direction.

-- 
Jonathan

From mertz at gnosis.cx  Mon Jul 23 12:00:31 2018
From: mertz at gnosis.cx (David Mertz)
Date: Mon, 23 Jul 2018 12:00:31 -0400
Subject: [Python-ideas] A better (simpler) approach to PEP 505
In-Reply-To: <CACac1F_f67oA22vFkuHH+d6xP5nFhNdj4oEF4vwGYU+uk-e9Jw@mail.gmail.com>
References: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
 <CACac1F_f67oA22vFkuHH+d6xP5nFhNdj4oEF4vwGYU+uk-e9Jw@mail.gmail.com>
Message-ID: <CAEbHw4Y9DkCfmGKZtVd8On5hJhJPjah1e+QzaRn9oYHdeFxYRA@mail.gmail.com>

On Mon, Jul 23, 2018 at 11:26 AM Paul Moore <p.f.moore at gmail.com> wrote:

> * Library solution works with all versions of Python
> * The need for unbox is a little ugly, but arguably less so than ?.
> (conceded that's a subjective view)
> * Mixing ?. and . is terser than unboxing and reboxing - but are there
> any real examples where that's needed?
> * Having the default at the beginning rather than at the end doesn't
> follow natural reading order (but again that's pretty subjective)
>

On the last point, it would be easy enough to change the API to make it
`NoneAware(obj).a.b.c.unbox(sentinel)` if that was thought a better API
than `NoneAware(obj, sentinel).a.b.c.unbox()`.

Oh... and the production code should DEFINITELY spell 'sentinel' correctly
if that's a part of the API. :-)

-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/90bfcd99/attachment.html>

From steve at pearwood.info  Mon Jul 23 12:04:44 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 24 Jul 2018 02:04:44 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <13681cf0-1571-2b74-e451-bb234f1952b7@python.org>
References: <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAAu18hc35N11jVQj+0691aH8Oe2+4Cqfn0yFpwWtjqou161ATg@mail.gmail.com>
 <0ffb89a1-3f63-a49d-e5a9-e98b25f60616@python.org>
 <20180723121156.4337a316@fsol>
 <beac393d-08db-62ae-ac17-7745525683a8@python.org>
 <b50a44e4-b3c8-3df1-ab1c-f0a0d802244e@python.org>
 <a46fc434-1bbb-f17c-42ef-840e3d121572@python.org>
 <f7363e81-9a77-4178-2f22-7281b0d5e917@python.org>
 <13681cf0-1571-2b74-e451-bb234f1952b7@python.org>
Message-ID: <20180723160443.GA8744@ando.pearwood.info>

On Mon, Jul 23, 2018 at 12:09:00PM +0100, Steve Dower wrote:
> On 23Jul2018 1145, Antoine Pitrou wrote:
> >
> >Le 23/07/2018 ? 12:38, Steve Dower a ?crit?:
> >>
> >>General comment to everyone (not just Antoine): these arguments have
> >>zero value to me. Feel free to keep making them, but I am uninterested.
> >
> >So you're uninterested in learning from past mistakes?
> >
> >You sound like a child who thinks their demands should be satisfied
> >because they are the center of the world.
> 
> Sorry if it came across like that, it wasn't the intention. 

Steve, I don't think you should apologise to somebody who has just 
quoted you out of context and insulted you for no good reason.

In context, your comment about "these arguments" was a perfectly 
reasonable thing to say. You didn't say you were uninterested in *all* 
arguments, only certain kinds of low-value arguments that keep getting 
made over and over again. Antoine snipped the context, unfairly accused 
you on zero evidence of being "uninterested in learning from past 
mistakes", and described you as a self-centred child.

I think Antoine should be apologising to you, not the other way around.


-- 
Steve

From c at anthonyrisinger.com  Mon Jul 23 12:19:20 2018
From: c at anthonyrisinger.com (C Anthony Risinger)
Date: Mon, 23 Jul 2018 11:19:20 -0500
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <ec40dae6-01f5-4ae5-a8f4-6baf116934fc@googlegroups.com>
References: <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAAu18hc35N11jVQj+0691aH8Oe2+4Cqfn0yFpwWtjqou161ATg@mail.gmail.com>
 <0ffb89a1-3f63-a49d-e5a9-e98b25f60616@python.org>
 <20180723121156.4337a316@fsol>
 <beac393d-08db-62ae-ac17-7745525683a8@python.org>
 <b50a44e4-b3c8-3df1-ab1c-f0a0d802244e@python.org>
 <a46fc434-1bbb-f17c-42ef-840e3d121572@python.org>
 <f7363e81-9a77-4178-2f22-7281b0d5e917@python.org>
 <13681cf0-1571-2b74-e451-bb234f1952b7@python.org>
 <a79f0fbc-6277-4a01-a7d7-2255192aa4ec@googlegroups.com>
 <CACac1F_jKbotHsokdB1vcPBbJFSyDqQpv6T4h8VunT-qhpkjLg@mail.gmail.com>
 <ec40dae6-01f5-4ae5-a8f4-6baf116934fc@googlegroups.com>
Message-ID: <CAGAVQTH4VYJq28n_NUNDEg38SzT2n+WvpEtuyo9T9w3Yqph1HQ@mail.gmail.com>

On Mon, Jul 23, 2018 at 7:46 AM, Gr?gory Lielens <gregory.lielens at gmail.com>
wrote:

>  Paul Moore wrote:
>>
>> This is my impression, as well. It seems like something that's helpful
>> in dealing with unstructured object hierarchies with lots of optional
>> attributes - which is where JSON tends to be used.
>>
>> But given that, I'm really much more interested in seeing the new
>> operators compared against a well-written "JSON object hierarchy
>> traversal" library than against raw Python code.
>>
>
> Good point, I did not think about that when suggesting to give some
> context, but indeed if it's linked to a library in particular, there is
> always the possibility to choose another object than None as the "nothing
> here" marker.
>
>From what I gather, disallowing reassignment all but cements None as the
"nothing here" marker (the originally intent?), _especially_ when "nothing
here" is candidate for replacement by a more appropriate, type- or domain-
or context-specific "nothing here" marker. Imbuing None with any other
meaning brings nothing but headache. It's an attractive nuisance. None is
None is None, there can be only one! You don't know what it is, only that
it's not anything else. What possible meaning can such an object usefully
have in an application built on disparate libraries with their own ideas on
the matter?

Every API I've used (apologies for coming up blank on a concrete example!)
granting None meaning is awkward to consume. `.get()` interfaces are less
useful (must carry your own internal sentinels) or more try/except blocks
are required to achieve the same end (not a bad thing per se, but
a diminished experience to be sure). Early APIs I wrote in this "well it's
None, but but but, with this convenient meaning, see?! SEE??"-style were
later regarded -- quite thoroughly -- as a bad idea by myself, my peers,
and downstream consumers.

I'd personally use these new operators both frequently and judiciously.
They align very well with a "set 'em up and knock 'em down"-style I use:
normalized, progressively conditioned input values fast-followed by
aggressive short-circuits and clear early returns. IME this pattern
generates clean, readable, and performant code. Honestly the
short-circuiting capability alone is enough to sell me :-) This PEP would
find legitimate use by me every day. I'm not 100% sold on `?[` (then again,
attributes are pulled from an object namespace via `?.` and namespaces are
containers by definition) but `?.` and `??` deliver immense value.

Not sure if useful, but this discussion reminds me of a pattern prevalent
in the Elixir community. They use `?` and `!` in function definitions to
denote variants differing only on return behavior (not function clauses!
This is by convention only, they're entirely new functions with a symbol in
their name). It looks something like this:

# Default function.
# Return a tuple {interesting-or-nil, error-or-nil}.
def open(path) do ... end

# Maybe variant.
# Return a boolean, or less often, interesting-or-nil (replaces `is_` or
`can_` methods in Python).
def open?(path) do ... end

# Forceful variant.
# Return interesting or die trying (inverse of `.get()` methods in Python;
raising is not the default expectation in Elixir).
def open!(path) do ... end

The `?.`-operator reminds me of this. It offers to perform an extremely
common operation (simple attribute access) while short-circuiting on the
most frequently triggered guard condition (AttributeError).

I don't think the utility here is restricted to deeply nested JSON
`loads()` or one-off scripts. It better aligns the community on semantics,
encouraging more idiomatic -- and uniform! -- interactions with None.

-- 

C Anthony
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/68436deb/attachment-0001.html>

From steve.dower at python.org  Mon Jul 23 12:24:08 2018
From: steve.dower at python.org (Steve Dower)
Date: Mon, 23 Jul 2018 17:24:08 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4ZFnXr-zcm+wy-U4hs3nLwz-CPKXQxEJsyxgZ49a1mSjA@mail.gmail.com>
References: <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
 <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAAu18hc35N11jVQj+0691aH8Oe2+4Cqfn0yFpwWtjqou161ATg@mail.gmail.com>
 <0ffb89a1-3f63-a49d-e5a9-e98b25f60616@python.org>
 <CAEbHw4ZFnXr-zcm+wy-U4hs3nLwz-CPKXQxEJsyxgZ49a1mSjA@mail.gmail.com>
Message-ID: <d1d4587a-6366-16d4-43c9-b5acd347e80b@python.org>

On 23Jul2018 1530, David Mertz wrote:
> Of course I don't mean that if implemented the semantics 
> would be ambiguous... rather, the proper "swallowing" of different kinds 
> of exceptions is not intuitively obvious, not even to you, Steve.? And 
> if some decision was reached and documented, it would remain unclear to 
> new (or even experienced) users of the feature.

As written in the PEP, no exceptions are ever swallowed. The translation 
into existing syntax is very clearly and unambiguously shown, and there 
is no exception handling at all. All the exception handling discussion 
in the PEP is under the heading of "rejected ideas".

This email discussion includes some hypotheticals, since that's the 
point - I want thoughts and counter-proposals for semantics and 
discussion. I am 100% committed to an unambiguous PEP, and I believe the 
current proposal is most defensible. However, I don't want to have a 
"discussion" where I simply assume that I'm right, everyone else is 
wrong, and I refuse to discuss or consider alternatives.

So sorry for letting you all think that everything I write is actually 
the PEP. I had assumed that because my emails are not the PEP that 
people would realise that they are not the PEP. I'm going to duck out of 
the discussions here now, since they are not as productive as I'd hoped, 
and once we have a BDFL-replacement I'll reawaken it and see what is 
required at that point.

Cheers,
Steve

From contact at brice.xyz  Mon Jul 23 12:28:49 2018
From: contact at brice.xyz (Brice Parent)
Date: Mon, 23 Jul 2018 18:28:49 +0200
Subject: [Python-ideas] A better (simpler) approach to PEP 505
In-Reply-To: <CAEbHw4Y9DkCfmGKZtVd8On5hJhJPjah1e+QzaRn9oYHdeFxYRA@mail.gmail.com>
References: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
 <CACac1F_f67oA22vFkuHH+d6xP5nFhNdj4oEF4vwGYU+uk-e9Jw@mail.gmail.com>
 <CAEbHw4Y9DkCfmGKZtVd8On5hJhJPjah1e+QzaRn9oYHdeFxYRA@mail.gmail.com>
Message-ID: <5450925c-7d24-c127-306e-b520dfff13a8@brice.xyz>

Le 23/07/2018 ? 18:00, David Mertz a ?crit?:
> On Mon, Jul 23, 2018 at 11:26 AM Paul Moore <p.f.moore at gmail.com 
> <mailto:p.f.moore at gmail.com>> wrote:
>
>     * Library solution works with all versions of Python
>     * The need for unbox is a little ugly, but arguably less so than ?.
>     (conceded that's a subjective view)
>     * Mixing ?. and . is terser than unboxing and reboxing - but are there
>     any real examples where that's needed?
>     * Having the default at the beginning rather than at the end doesn't
>     follow natural reading order (but again that's pretty subjective)
>
>
> On the last point, it would be easy enough to change the API to make 
> it `NoneAware(obj).a.b.c.unbox(sentinel)` if that was thought a better 
> API than `NoneAware(obj, sentinel).a.b.c.unbox()`.
> Oh... and the production code should DEFINITELY spell 'sentinel' 
> correctly if that's a part of the API. :-)
>
One of the good things about this proposal, is that it may be expanded 
to other sentinels:

ret = NoneAware(obj, sentinel=0).a.unbox(42)

# would be equivalent to
ret = obj.a if obj.a is not 0 else 42

# or, if tmp_a had side effects
tmp_a = obj.a
ret = tmp_a if tmp_a is not 0 else 42

One thing that this proposal doesn't cover easily (although I'd refactor 
the code before using this syntax, as excepted in some very specific 
cases like when using ORMs, I don't like chaining methods and properties 
like this):

favorite = cfg?.user.profile?.food ?? "Spam"? # every user should have a 
profile

favorite = NoneAware(cfg).user.profile.food.unbox("Spam") # We're 
silencing an exception that I didn't plan to silence

Another thing is that both parts are executed. If instead of "Spam" we 
had any variable with a side effect, it would have been executed even if 
cfg.user.profile.food existed. We're missing the lazy evaluation of this 
part here.

That being said, I deeply believe the use case for that are not spread 
enough to justify such a change. And I can see from here code where 
every '.' is prepended by a '?' just in case (the same way we encounter 
frequently exceptions catching for Exception).

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/e0e56db7/attachment.html>

From mikhailwas at gmail.com  Mon Jul 23 12:35:37 2018
From: mikhailwas at gmail.com (Mikhail V)
Date: Mon, 23 Jul 2018 19:35:37 +0300
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAGAVQTH4VYJq28n_NUNDEg38SzT2n+WvpEtuyo9T9w3Yqph1HQ@mail.gmail.com>
References: <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAAu18hc35N11jVQj+0691aH8Oe2+4Cqfn0yFpwWtjqou161ATg@mail.gmail.com>
 <0ffb89a1-3f63-a49d-e5a9-e98b25f60616@python.org>
 <20180723121156.4337a316@fsol>
 <beac393d-08db-62ae-ac17-7745525683a8@python.org>
 <b50a44e4-b3c8-3df1-ab1c-f0a0d802244e@python.org>
 <a46fc434-1bbb-f17c-42ef-840e3d121572@python.org>
 <f7363e81-9a77-4178-2f22-7281b0d5e917@python.org>
 <13681cf0-1571-2b74-e451-bb234f1952b7@python.org>
 <a79f0fbc-6277-4a01-a7d7-2255192aa4ec@googlegroups.com>
 <CACac1F_jKbotHsokdB1vcPBbJFSyDqQpv6T4h8VunT-qhpkjLg@mail.gmail.com>
 <ec40dae6-01f5-4ae5-a8f4-6baf116934fc@googlegroups.com>
 <CAGAVQTH4VYJq28n_NUNDEg38SzT2n+WvpEtuyo9T9w3Yqph1HQ@mail.gmail.com>
Message-ID: <CAE6NSRBtU5ScN_BDpiMZVBh1RjtfNLTB9rhu_RHbuyrGN3jh8g@mail.gmail.com>

I personally do almost only classical programming, and I am somewhat
opposed to OOP in general. So here my somewhat outlandish view,
(and I am biased becase I will probably never need this feature).

First thoughts after reading the PEP: what is so super-special and fundamental
about None value?
Is it more special and required to treat more often than "true" or
"false" values in
similar manner?
Is there statistics for occurence of code checking 'None' versus 'not None'?

So:

if x is None:
   x = 10

converts to:

x ??= 10

But what if one need to do something else than just return a value?
E.g. I want to insert `print(x)` in the first example?
Or check against other value than 'None'?
Supposed to write it so then rewrite it so, then if I need it other way -
again rewrite it so.

So from the PEP I understand only two things:
1) de-facto None became special in some contexts.
2) one may need to short-circuit an expression if None pops up.

So it is syntax good only for special kind of applications I suppose.
It seems motivating examples come mainly from situations where
one has a Python script serving some database framework or something alike.
So it smells like specialized usage and design-specific pattern.
It looks (from a passer-by POV) like the proposed syntax tries to
treat some sticking out
parts of something that may not be necessarily present in an application at all.
Of course that maybe argued easily because many concepts that
does not directly belong to programming are sometimes appearing in form of
dedicated syntax. Though such cases not so common in Python.

I don't know whether the feature adressed by '?.' is often in those contexts,
but first thing that comes to mind - it regards only experts and NOT
intended for reading the code that I am not familiar with.

E.g. this "improved" code in examples:

def find_module(self, fullname, path):
    return getattr(self, 'find_spec', None)?.__call__(fullname, path)?.loader

I could not decipher this and I have tried for a while.

I could probably understand the motivation better if it were some
more or less fundamental or general pattern, but is it? IDK


-----

More important that the syntax part is really worrying.
Does everybody realize that there are only 2 or 3 ASCII characters left free?
IIRC those are question mark "?", exclamation mark "!" and dollar sign "$".

Imagine that in some time someone comes up with a good general syntax feature
that would require new symbol - and there is nothing left. That's not
funny actually.
This makes decisions about symbols into agony and heats up discussions
to extreme
because some PEP authors maybe hoping for a better-looking symbol for
their 'child' while other proposals and possible future proposals may
be more important.

At this time point I think one should first consider exploring the
possibility to
add non-ASCII characters to syntax. I think it is resonable because Unicode
became standard.
It should at least help to relax the atmosphere around syntax proposals.

'Niche' features IMO should be better added as functions, or, if non-ASCII
symbols could be added, as some '2nd class' symbols (i.e. not so nice looking).
And nicely looking symbols should be reserved for future proposals for
general syntax features which are potentially useful for wider user groups.

From gregory.lielens at gmail.com  Mon Jul 23 12:37:36 2018
From: gregory.lielens at gmail.com (=?UTF-8?Q?Gr=C3=A9gory_Lielens?=)
Date: Mon, 23 Jul 2018 09:37:36 -0700 (PDT)
Subject: [Python-ideas] A better (simpler) approach to PEP 505
In-Reply-To: <CAEbHw4Y9DkCfmGKZtVd8On5hJhJPjah1e+QzaRn9oYHdeFxYRA@mail.gmail.com>
References: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
 <CACac1F_f67oA22vFkuHH+d6xP5nFhNdj4oEF4vwGYU+uk-e9Jw@mail.gmail.com>
 <CAEbHw4Y9DkCfmGKZtVd8On5hJhJPjah1e+QzaRn9oYHdeFxYRA@mail.gmail.com>
Message-ID: <d9a80f01-a2b6-4b15-8aba-1bebb7e9cf1d@googlegroups.com>

Your wrapper class can also, I think:
 Swallow exceptions, especially AttributeError
 Specify the sentinel, if it is not None
 Work for getitem 

Mixing is indeed more difficult, and there is probably performance impact.
Still, it's general and does not need to touch the implementation of the class you want to descend...

I like it better than the operators

Another idea, may be stupid, maybe not: what if None fails with a subinstance of AttributeError, and a subinstance of KeyError, or IndexError. Then a try/except catching only those exceptions would also work quite nicely...

From solipsis at pitrou.net  Mon Jul 23 12:46:07 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 23 Jul 2018 18:46:07 +0200
Subject: [Python-ideas] A better (simpler) approach to PEP 505
References: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
Message-ID: <20180723184607.3a402793@fsol>

On Mon, 23 Jul 2018 11:12:45 -0400
David Mertz <mertz at gnosis.cx> wrote:
> 
> The particular names I use are nothing special, and better ones might be
> found.  I just called the class NoneAware and the "escape" method
> `.unbox()` because that seemed intuitive at first brush.
> 
> I don't disagree that needing to call .unbox() at the end of the chained
> attribute access is a little bit ugly.  But it's a lot less ugly than large
> family of new operators.  And honestly, it's a nice way of being explicit
> about the fact that we're entering then leaving a special world where
> attribute accesses don't fail.
> 
> I haven't implemented the equivalent dictionary lookups in the below.  That
> would be straightforward, and I'm sure my 5 minute throwaway code could be
> improved in other ways also.  But something better than this in the
> standard library would address ALL the actual needs described in PEP 505.
> Even the pattern Steve Dower is especially fond of like:
> 
> favorite = cfg?.user?.profile?.food ?? "Spam"
> 
> 
> (i.e. a configuration may be incomplete at any level, if levels are missing
> default favorite food is Spam).  We could simply spell that:
> 
> favorite = NoneAware(cfg, "Spam").user.profile.food.unbox()
> 
> 
> I think that's 14 characters more in this example, but still compact.  We
> could get that down to 2 characters if we used one-letter names for the
> class and method.  I suppose down to zero characters if .unbox() was a
> property.

You could use .__call__() instead of .unbox().  Also you can make
unboxing unnecessary in most cases by having your class proxy most
operations, like weakref.proxy does.  The "wrapt" library may help with
that:
http://wrapt.readthedocs.io/en/latest/wrappers.html

Regards

Antoine.



From steve at pearwood.info  Mon Jul 23 12:52:56 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 24 Jul 2018 02:52:56 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
References: <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
Message-ID: <20180723165256.GB8744@ando.pearwood.info>

On Mon, Jul 23, 2018 at 02:04:17PM +0200, Giampaolo Rodola' wrote:

[I wrote this]
> > This is the point I was making earlier: you accept existing punctuation
> > doing these things:
> >
> >     try:
> >         obj.spam.egsg.tomato.cheese  # oops a typo
> >     except AttributeError:
> >         # evaluation can stop at any time
> >         ...
> >
> > while demanding a higher standard for new punctuation.
> >
> > All of your criticisms of ? punctuation applies to . as well.

[Giampaolo] 
> I don't think they do. For once, "a.b" does one and one thing only,

Attribute lookup is a bit more complex than just "one thing only", but 
okay, I'll accept that for a sufficiently complex "thing", dot access 
does one thing.


> "a?.b" does two and that's a fundamental difference (explicitness).

How is "two things" less explicit than "one thing"?

Comments like the above is why I think that "explicit" and "implicit" 
are used to mean "I like it" and "I don't like it" rather than being 
objective arguments, or indeed having anything to do with explicitness 
or implicitness.

If this PEP is approved, then *by definition* the ?? operator will mean 

    return the first operand if it isn't None
    otherwise evaluate and return the second

making it just as explicit as:

   +  # add or concatenate the two operands

   ==  # return True if the two operands are equal otherwise False

   sorted(x)  # make a list copy of x and sort it

etc. Just because the spelling is short doesn't make it implicit.


> It
> does so by introducing a brand new operator ("?") which can be spelled
> in two forms ("a?.b" and "a?[b]") by using two adjacent symbols not
> interrupted by any space, which is an absolute first in the Python
> syntax 

It isn't a first. Many existing operators use two adjacent symbols not 
interrupted by a space:

e.g.  ==  <=  >=  !=  **  //  << >> +=  -=  *= etc.



> and that's the second and fundamental difference. I cannot move
> the same criticism to the "a.b" form: it's simpler, it does one thing
> and it uses one symbol.

You criticised ?. because it can interupt left-to-right execution:

    a?.b?.c?.d

True. But so can a single dot:

    a.b.c.d

is no more guaranteed to execute all the way to the right.

Likewise the logical operators "or" and "and" are designed to 
short-circuit. If ?? and friends are a mistake because they 
short-circuit, why aren't "or" and "and" mistakes?

I'm not asking this as a rhetorical question. If you think there is a 
reason why it is okay for or/and to short-circuit, but it is bad for ?? 
and friends to short-circuit, then please explain why they are 
different. I will be very happy to listen to your arguments.


> > > Multiple "?" can live on the same line so
> > > that's incentive to write one-liners, really, and to me one-liners are
> > > always less explicit than the same logic split on multiple lines.
> >
> > Explicit is not always better.
> >
> >     import this
> >
> > is much better than:
> >
> >     for location in sys.path:
> >         try:
> >             for file in os.listdir(location):
> >                 if os.splitext(file) in ('.pyc', '.py', '.so'):
> >                     ...
> 
> I honestly don't see how this example is related with anything discussed so far.

You keep saying that the proposed ?? etc operators aren't explicit and 
you criticise them for doing "two things". The import statement does 
at least ten things:

- searches the cache of modules;
- if not found, traverse the search path looking for not one kind of
  file, but multiple kinds of files that the user has no control over;
- if a matching file is found, check for a pre-compiled version;
- or compile it;
- save the compiled byte-code in a new file;
- load the compiled byte-code into a module object;
- execute that code;
- add the module object to the cache;
- create a new name in the local namespace;
- bind the module object to the name.

If doing "two things" is bad, then doing "ten things" is five times 
worse. If ?? is too implicit, then what is importing? Why is it okay for 
importing to be "implicit" but ?? needs to be written out over two 
lines?

If behaviour that is acceptable, even desirable in existing features is 
harmful in these new ? operators, then please tell us how and why.



-- 
Steve

From python at mrabarnett.plus.com  Mon Jul 23 12:58:06 2018
From: python at mrabarnett.plus.com (MRAB)
Date: Mon, 23 Jul 2018 17:58:06 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <0ffb89a1-3f63-a49d-e5a9-e98b25f60616@python.org>
References: <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
 <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAAu18hc35N11jVQj+0691aH8Oe2+4Cqfn0yFpwWtjqou161ATg@mail.gmail.com>
 <0ffb89a1-3f63-a49d-e5a9-e98b25f60616@python.org>
Message-ID: <a7a0ae82-c326-9e6a-b306-99799834ac25@mrabarnett.plus.com>

On 2018-07-23 10:51, Steve Dower wrote:
[snip]

> False: 'a?.b' is totally different from 'a.b'
> ---------------------------------------------
> 
> The expression 'a.b' either results in 'a.b' or AttributeError (assuming
> no descriptors are involved).
> 
> The expression 'a?.b' either results in 'a.b' or None (again, assuming
> no descriptors).

It could still raise AttributeError.
> 
> This isn't a crazy new idea, it really just short-circuits a specific
> error that can only be precisely avoided with "if None" checks (catching
> AttributeError is not the same).
> 
[snip]

From python at mrabarnett.plus.com  Mon Jul 23 12:58:23 2018
From: python at mrabarnett.plus.com (MRAB)
Date: Mon, 23 Jul 2018 17:58:23 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
References: <20180722121006.GQ8744@ando.pearwood.info>
 <CAFYqXL--vGPphd6rtUNTj2DeSNWMKH+1w21VtwmZrAdaESrtnw@mail.gmail.com>
 <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
Message-ID: <729132de-833a-f837-be1d-0c1cffc0a5a5@mrabarnett.plus.com>

On 2018-07-23 13:04, Giampaolo Rodola' wrote:
> On Mon, Jul 23, 2018 at 3:12 AM Steven D'Aprano <steve at pearwood.info> wrote:
>> > ? has no spaces, it's literally "variable names interrupted by
>> > question marks" and evaluation can stop at any time while scanning the
>> > line from left to right.
>>
>> Just like ordinary attribute access.
>>
>> This is the point I was making earlier: you accept existing punctuation
>> doing these things:
>>
>>     try:
>>         obj.spam.egsg.tomato.cheese  # oops a typo
>>     except AttributeError:
>>         # evaluation can stop at any time
>>         ...
>>
>> while demanding a higher standard for new punctuation.
>>
>> All of your criticisms of ? punctuation applies to . as well.
> 
> I don't think they do. For once, "a.b" does one and one thing only,
> "a?.b" does two and that's a fundamental difference (explicitness). It
> does so by introducing a brand new operator ("?") which can be spelled
> in two forms ("a?.b" and "a?[b]") by using two adjacent symbols not
> interrupted by any space, which is an absolute first in the Python
> syntax and that's the second and fundamental difference. I cannot move
> the same criticism to the "a.b" form: it's simpler, it does one thing
> and it uses one symbol.
> 
[snip]
I think you're misunderstanding something: we're not talking about a 
special operator "?" that somehow combines with existing operators, 
we're talking about completely new and separate operators "?.", "?[", 
etc., which resemble the existing ".", "[", etc.

From shoyer at gmail.com  Mon Jul 23 13:01:13 2018
From: shoyer at gmail.com (Stephan Hoyer)
Date: Mon, 23 Jul 2018 10:01:13 -0700
Subject: [Python-ideas] slice[] to get more complex slices
In-Reply-To: <CACac1F_XOi+1gmEMgZdbmW_3UXs661JS5BF=OvWLrF47H0fJmg@mail.gmail.com>
References: <CAFpSVpKnE8b=QrjxLy6anAF3ammq2QbB4_K1Ey02Vk3n8EVwyQ@mail.gmail.com>
 <pj2lgk$c6c$1@blaine.gmane.org>
 <09cb9784-6a97-49b9-b5e7-1c9a9d8c8391@googlegroups.com>
 <CALD=Yf-MNLW7z8s=ruc_C13t7eWp8KC6OWvqf9durZgpN=8=sw@mail.gmail.com>
 <399ff7d70c204f679bdb5a79f9bc0460@xmail103.UGent.be>
 <4243953379d74d2aa5393be2434a99a7@xmail103.UGent.be>
 <5B55AE6A.7010305@UGent.be>
 <CACac1F_XOi+1gmEMgZdbmW_3UXs661JS5BF=OvWLrF47H0fJmg@mail.gmail.com>
Message-ID: <CAEQ_Tvf2Y-up_uWoUdutyoreHeM6=X52bZzZcath-r1btYeqow@mail.gmail.com>

On Mon, Jul 23, 2018 at 4:24 AM Paul Moore <p.f.moore at gmail.com> wrote:

> I thought the reason the proposal got nowhere was because it's pretty
> simple to define it yourself:
>
> >>> class SliceHelper:
> ...     def __getitem__(self, slice):
> ...         return slice
> ...
> >>> SH = SliceHelper()
> >>> SH[1::3]
> slice(1, None, 3)
>
> Did I miss something significant about why this wasn't sufficient?


I think a SliceHelper class like this is a reasonable solution, but there
would be a lot of value having it a standard place somewhere in the
standard library (e.g., operator.subscript).

Both pandas and NumPy include this helper object under different names
(pandas.IndexSlice and numpy.index_exp / numpy.s_), but it would be
surprising/unexpected for pandas/numpy specific helpers to show up when not
using one of those libraries. I do the exact same sorts of indexing
manipulations with xarray, dask and TensorFlow.

Given that this is basically a simple feature to make it easier to work
with Python syntax (so there's no danger it will change in the future), I
think there is a lot to be said for putting it in the standard library in
one place so it's obvious what to use and users don't have to relearn that
name for this object and/or reimplement it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/4ac17cd2/attachment-0001.html>

From steve at pearwood.info  Mon Jul 23 13:01:30 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 24 Jul 2018 03:01:30 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180722015418.GO8744@ando.pearwood.info>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <CAEbHw4ZVua=e86DQN4e-kUv2fO1tgB2QxMaOYFY5REKYvJ4t8g@mail.gmail.com>
 <20180719133820.GV7318@ando.pearwood.info>
 <CAFYqXL_2enUA+9mv8PSgoUAFfiQnKcWQFQw0U2_5DZb+21rh_A@mail.gmail.com>
 <20180722015418.GO8744@ando.pearwood.info>
Message-ID: <20180723170129.GC8744@ando.pearwood.info>

On Sun, Jul 22, 2018 at 11:54:19AM +1000, Steven D'Aprano wrote:

> In my opinion, writing
> 
>     expression if expression is None else default
> 
> is the *opposite* of Pythonic, it is verbose and the DRY violation is 
> inelegant (as well as inefficient). I'd much rather use:
> 
>     expression ?? default
> 
> although with PEP 572 approved, there is an alternative:
> 
>     temp := expression if temp is None else default


I was mistaken. That would need to be written as:

    temp if (temp := expression) is None else default

which is obscure enough for me to prefer the ?? syntax again.



-- 
Steve

From mertz at gnosis.cx  Mon Jul 23 13:03:15 2018
From: mertz at gnosis.cx (David Mertz)
Date: Mon, 23 Jul 2018 13:03:15 -0400
Subject: [Python-ideas] A better (simpler) approach to PEP 505
In-Reply-To: <20180723184607.3a402793@fsol>
References: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
 <20180723184607.3a402793@fsol>
Message-ID: <CAEbHw4YgjcuqtCtgButJHwgEKn+hnEJ89xaoC2hgnLMz2EiFVQ@mail.gmail.com>

On Mon, Jul 23, 2018 at 12:47 PM Antoine Pitrou <solipsis at pitrou.net> wrote:

> > favorite = cfg?.user?.profile?.food ?? "Spam"
> > favorite = NoneAware(cfg, "Spam").user.profile.food.unbox()
>


> You could use .__call__() instead of .unbox().  Also you can make
> unboxing unnecessary in most cases by having your class proxy most
> operations, like weakref.proxy does.  The "wrapt" library may help with
> that: http://wrapt.readthedocs.io/en/latest/wrappers.html


I'm not sure I entirely understand how that class proxy could be made
transparent.  Maybe you can make a toy implementation to show me?  How can
we make this work?

    favorite = NoneAware(cfg, "Spam").user.profile.food + "and more spam"

I just noticed in my scratch directory that I had implemented almost the
same thing a year ago when this discussion last came up.  It's simple
enough that the code was almost the same back then.  I had included
`.__getitem__()` in that version, but had not considered sentinels.

However, one thing I *did* implement was `.__call__()`, but that reminds me
of why your idea cannot work.  I had it either call or fall back to more
boxing.  The end point of the chained attributes (or dict lookups) can
perfectly well be a callable.  E.g.

    favorite = NoneAware(cfg, "Spam").user.profile.get_food()

Well, that's not going to work in my current toy code either since it would
need to be:

    favorite = NoneAware(cfg, "Spam").user.profile.get_food.unbox()()

But I could implement special stuff for "if the tail is a callable, unbox
it automatically", I suppose.

However, it would be fundamentally ambiguous, I think, whether those final
parens should mean "unbox" or "call".  Or I guess if calling *always* meant
"unbox" we could have:

    favorite = NoneAware(cfg, "Spam").user.profile.get_food()()

That feels weird to me though.

-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/50a05f8c/attachment.html>

From mertz at gnosis.cx  Mon Jul 23 13:21:53 2018
From: mertz at gnosis.cx (David Mertz)
Date: Mon, 23 Jul 2018 13:21:53 -0400
Subject: [Python-ideas] A better (simpler) approach to PEP 505
In-Reply-To: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
References: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
Message-ID: <CAEbHw4Zhdpta6o02axv0SC0WtQBuxoU+Ye-4k7TCVtp3UD08tA@mail.gmail.com>

Btw. I *do* realize that the semantics of my suggested NoneAware class is
different from the coalescing syntax.  I just look at whether attribute
access succeeds or fails in that code, not whether the starting value is
specifically None (nor any particular sentinel).

I believe that that behavior better fits the ACTUAL need underlying these
ideas (i.e. potentially deeply nested but messy data; from JSON or similar
sources).  If we wanted to match the semantics of PEP 505 more exactly, it
would be easy enough to compare a current level of the hierarchy to None
specifically rather than check more generically "is this a thing that has
that attribute?"

However, if the PEP 505 semantics really are the ones most desirable (i.e.
including raising AttributeError on things that are
not-None-but-don't-have-attribute), that's perfectly straightforward to
implement using existing syntax and an API very close to what I suggest.

On Mon, Jul 23, 2018 at 11:12 AM David Mertz <mertz at gnosis.cx> wrote:

> The need addressed by PEP 505 is real; it's also MUCH more niche and
> uncommon than something that would merit new syntax.  Moreover, the actual
> legitimate purpose served by the PEP 505 syntax is easily served by
> existing Python simply by using a wrapper class.
>
> Here is a way of solving the "deep attribute access to messy data" problem
> that is:
>
> (1) Much more explicit
> (2) Requires no change in syntax
> (3) Will not be a bug magnet
> (4) Inasmuch as there are semantic traps, they are announced by the use of
> a class whose documentation would be pointed to for readers
>
> The API that could be useful might be something like this:
>
> In [1]: from none_aware import NoneAware
> In [2]: from types import SimpleNamespace
> In [3]: foo = SimpleNamespace()
> In [4]: foo.bar = SimpleNamespace()
> In [5]: foo.bar.baz = SimpleNamespace()
> In [6]: foo.bar.baz.blat = 42
> In [7]: NoneAware(foo).bar.blim
> Out[7]: <none_aware.NoneAware at 0x11156a748>
> In [8]: NoneAware(foo).bar.blim.unbox()
> In [9]: NoneAware(foo).bar.baz.blat.unbox()
> Out[9]: 42
> In [10]: NoneAware(foo).bar.baz.blat
> Out[10]: <none_aware.NoneAware at 0x11157d908>
> In [11]: NoneAware(foo).bar.baz.flam.unbox()
> In [12]: NoneAware(foo).bar.baz.flam
> Out[12]: <none_aware.NoneAware at 0x1115832b0>
>
>
> The particular names I use are nothing special, and better ones might be
> found.  I just called the class NoneAware and the "escape" method
> `.unbox()` because that seemed intuitive at first brush.
>
> I don't disagree that needing to call .unbox() at the end of the chained
> attribute access is a little bit ugly.  But it's a lot less ugly than large
> family of new operators.  And honestly, it's a nice way of being explicit
> about the fact that we're entering then leaving a special world where
> attribute accesses don't fail.
>
> I haven't implemented the equivalent dictionary lookups in the below.
> That would be straightforward, and I'm sure my 5 minute throwaway code
> could be improved in other ways also.  But something better than this in
> the standard library would address ALL the actual needs described in PEP
> 505.  Even the pattern Steve Dower is especially fond of like:
>
> favorite = cfg?.user?.profile?.food ?? "Spam"
>
>
> (i.e. a configuration may be incomplete at any level, if levels are
> missing default favorite food is Spam).  We could simply spell that:
>
> favorite = NoneAware(cfg, "Spam").user.profile.food.unbox()
>
>
> I think that's 14 characters more in this example, but still compact.  We
> could get that down to 2 characters if we used one-letter names for the
> class and method.  I suppose down to zero characters if .unbox() was a
> property.
>
> So completely toy implementation:
>
> class NoneAware(object):
>     def __init__(self, thing, sentinel=None):
>         self.thing = thing
>         self.sentinel = sentinal
>
>     def __getattr__(self, attr):
>         try:
>             return NoneAware(getattr(self.thing, attr))
>         except AttributeError:
>             return NoneAware(self.sentinel)
>
>     def unbox(self):
>         return self.thing
>
> --
> Keeping medicines from the bloodstreams of the sick; food
> from the bellies of the hungry; books from the hands of the
> uneducated; technology from the underdeveloped; and putting
> advocates of freedom in prisons.  Intellectual property is
> to the 21st century what the slave trade was to the 16th.
>


-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/e185d86d/attachment-0001.html>

From mehaase at gmail.com  Mon Jul 23 13:23:40 2018
From: mehaase at gmail.com (Mark E. Haase)
Date: Mon, 23 Jul 2018 13:23:40 -0400
Subject: [Python-ideas] A better (simpler) approach to PEP 505
In-Reply-To: <CA+msPNmjFjJvTxrR28brMMCjPDVoW+t4g6L4qiVpJBmETqOZLg@mail.gmail.com>
References: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
 <CACac1F_f67oA22vFkuHH+d6xP5nFhNdj4oEF4vwGYU+uk-e9Jw@mail.gmail.com>
 <CA+msPNmjFjJvTxrR28brMMCjPDVoW+t4g6L4qiVpJBmETqOZLg@mail.gmail.com>
Message-ID: <CALb0Rk5iLvA4Mnm3-cEROAnVs6VjLxqr_S72s4tw4cro+YduSg@mail.gmail.com>

On Mon, Jul 23, 2018 at 11:34 AM Robert Vanden Eynde <robertve92 at gmail.com>
wrote:

> The default could be at the end with an argument to unboxing :
>
> favorite = NoneAware(cfg).user.profile.food.unbox("Spam")
>

That's what PyMaybe does:

https://pymaybe.readthedocs.io/en/latest/readme.html#examples-use-cases

PyMaybe is a great implementation of the idea, but it's obviously limited
in what it can do. It doesn't have short-circuiting (but it uses lambda to
approximate it). It swallows a multitude of attribute and look up errors.
It doesn't support coalescing. And it can't be used with any library that
isn't PyMaybe-aware.

For people who want to explore deeply nested objects/dictionaries (perhaps
from unserialized JSON), there's also a pure Python library for that:

http://objectpath.org/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/c9f1a296/attachment.html>

From steve at pearwood.info  Mon Jul 23 13:27:44 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 24 Jul 2018 03:27:44 +1000
Subject: [Python-ideas] A better (simpler) approach to PEP 505
In-Reply-To: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
References: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
Message-ID: <20180723172744.GD8744@ando.pearwood.info>

On Mon, Jul 23, 2018 at 11:12:45AM -0400, David Mertz wrote:

> Here is a way of solving the "deep attribute access to messy data" problem
> that is:
> 
> (1) Much more explicit
> (2) Requires no change in syntax
> (3) Will not be a bug magnet

That's one opinion.


> (4) Inasmuch as there are semantic traps, they are announced by the use of
> a class whose documentation would be pointed to for readers

Operators have documentation too. If you think people won't read the 
documentation for the operator, what makes you think they're read the 
documentation for the class?


> The API that could be useful might be something like this:
> 
> In [1]: from none_aware import NoneAware
> In [2]: from types import SimpleNamespace
> In [3]: foo = SimpleNamespace()
> In [4]: foo.bar = SimpleNamespace()
> In [5]: foo.bar.baz = SimpleNamespace()
> In [6]: foo.bar.baz.blat = 42
> In [7]: NoneAware(foo).bar.blim
> Out[7]: <none_aware.NoneAware at 0x11156a748>

There's the first bug right there. foo.bar.blim ought to raise 
AttributeError, since there is no blim attribute defined on foo.bar.

Reminder: the proposal is for a null-coalescing operator, not an 
AttributeError suppressing operator.

The PEP is explicit that catching AttributeError is rejected:

https://www.python.org/dev/peps/pep-0505/#id21


> In [8]: NoneAware(foo).bar.blim.unbox()
> In [9]: NoneAware(foo).bar.baz.blat.unbox()
> Out[9]: 42

How many bugs will be caused by people forgetting to unbox when they're 
done?


> In [10]: NoneAware(foo).bar.baz.blat
> Out[10]: <none_aware.NoneAware at 0x11157d908>
> In [11]: NoneAware(foo).bar.baz.flam.unbox()

That ought to be AttributeError again, since there is no "flam" 
attribute defined on foo.bar.baz.


> In [12]: NoneAware(foo).bar.baz.flam
> Out[12]: <none_aware.NoneAware at 0x1115832b0>
> 
> 
> The particular names I use are nothing special, and better ones might be
> found.  I just called the class NoneAware and the "escape" method
> `.unbox()` because that seemed intuitive at first brush.
> 
> I don't disagree that needing to call .unbox() at the end of the chained
> attribute access is a little bit ugly.  But it's a lot less ugly than large
> family of new operators.  And honestly, it's a nice way of being explicit
> about the fact that we're entering then leaving a special world where
> attribute accesses don't fail.

But we aren't entering such a world, at least not in PEP 505. Attribute 
access can fail.

spam.eggs = 42

spam?.eggs?.upper

is still going to raise AttributeError, because eggs is not None, it is 
an int, and ints don't have an attribute "upper".


> I haven't implemented the equivalent dictionary lookups in the below.  That
> would be straightforward, and I'm sure my 5 minute throwaway code could be
> improved in other ways also.  But something better than this in the
> standard library would address ALL the actual needs described in PEP 505.

How does your class implement short-circuit behaviour?


> Even the pattern Steve Dower is especially fond of like:
> 
> favorite = cfg?.user?.profile?.food ?? "Spam"
> 
> 
> (i.e. a configuration may be incomplete at any level, if levels are missing
> default favorite food is Spam).  We could simply spell that:
> 
> favorite = NoneAware(cfg, "Spam").user.profile.food.unbox()
> 
> 
> I think that's 14 characters more in this example, but still compact.  We
> could get that down to 2 characters if we used one-letter names for the
> class and method.  I suppose down to zero characters if .unbox() was a
> property.

So we don't like operators like ?? because its too cryptic, but you're 
happy to have one-character class and property names.

favorite = N(cfg, "Spam").user.profile.food.u


What happens if you have an attribute that happens to be 
called "unbox" in your attribute look-up chain?

result = NoneAware(something, "default").spam.foo.unbox.eggs.unbox()



-- 
Steve

From gregory.lielens at gmail.com  Mon Jul 23 13:53:11 2018
From: gregory.lielens at gmail.com (=?UTF-8?Q?Gr=C3=A9gory_Lielens?=)
Date: Mon, 23 Jul 2018 10:53:11 -0700 (PDT)
Subject: [Python-ideas] A better (simpler) approach to PEP 505
In-Reply-To: <20180723172744.GD8744@ando.pearwood.info>
References: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
 <20180723172744.GD8744@ando.pearwood.info>
Message-ID: <76b7e1c1-73bb-4f15-9e43-9b9329d51f07@googlegroups.com>

The proto here swallow and short circuit on attribute error. Changing to do it on Noneness is easy, and you can choose between the two behavior: it's a strength compared to the operator approach.

It's working in current python, another strength.

I think short-circuit behavior is similar: once stopping condition has been met, the gard object cascade till the end of the nested attribute lookup. Like None do for the operator.

More complex stuff when None-coalescing attribute access is mixed with normal access is better done with operators.
Maybe complex operations fine of the middle of the access chain would also be easier with the operators.

How many times would you encounter that in real cases? I bet on very few.
In those few case, would a one line ?. chain expression be better than a multiple line logic with temporaries? I think no.

That's my feeling, but in the absence of concrete examples, feelings are ok.

From mertz at gnosis.cx  Mon Jul 23 13:55:48 2018
From: mertz at gnosis.cx (David Mertz)
Date: Mon, 23 Jul 2018 13:55:48 -0400
Subject: [Python-ideas] A better (simpler) approach to PEP 505
In-Reply-To: <20180723172744.GD8744@ando.pearwood.info>
References: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
 <20180723172744.GD8744@ando.pearwood.info>
Message-ID: <CAEbHw4bYdksOML6kU2p3_GSe4QkirKJMHRrVMCupa4kCxZhpSA@mail.gmail.com>

On Mon, Jul 23, 2018 at 1:28 PM Steven D'Aprano <steve at pearwood.info> wrote:

> There's the first bug right there. foo.bar.blim ought to raise
> AttributeError, since there is no blim attribute defined on foo.bar.
>

Note my observation that this is deliberately different semantics.  I want
to solve the underlying need, not simply emulate the less useful semantics
of PEP 505.  But those same semantics COULD be perfectly well emulated with
a similar class.


> > In [8]: NoneAware(foo).bar.blim.unbox()
> > In [9]: NoneAware(foo).bar.baz.blat.unbox()
> > Out[9]: 42
>
> How many bugs will be caused by people forgetting to unbox when they're
> done?
>

Far fewer and far more easily caught.  When the object you get wherever the
real failure happens is ` <none_aware.NoneAware at 0x11156a748>` or the
like, the source of the problem become dead obvious.  Having `None`
somewhere unexpected is much more mysterious and difficult to diagnose.

That said, NoneAware is the wrong name for the class I wrote.  Maybe
GreedyAccess would be a better name.  Perhaps I'll make slightly less toy
implementations of an actual NoneCoalesce and GreedyAccess class and post
in this thread and/or on PyPI.

It does look like PyMaybe does much of what I'm thinking of.  I didn't
think my toy was the first or best implementation.


> But we aren't entering such a world, at least not in PEP 505. Attribute
> access can fail.



> spam.eggs = 42

spam?.eggs?.upper



> is still going to raise AttributeError, because eggs is not None, it is an
> int, and ints don't have an attribute "upper".


True enough.  But it's extremely difficult to imagine a real-world case
where those particular semantics are actually particularly useful.  I guess
my revulsion at the syntax blinded me to the fact that the semantics are
actually pretty bad on their own.


> How does your class implement short-circuit behaviour?
>

You mean if we want something like this?

    favorite = GreedyAccess(cfg,
sentinel=ExpensiveLookup()).user.profile.food.unbox()

It doesn't.  Something else (like PyMaybe) could defer the computation
using a lambda or other means.  My toy code doesn't do that.  We could also
avoid the visible lambda by doing an 'iscallable(sentinel)' and only
calling it if it turns out to be needed, but that might be too magical.
E.g.:

    # If we fail, favorite = ExpensiveLookup(), by magic
    favorite = GreedyAccess(cfg,
sentinel=ExpensiveLookup).user.profile.food.unbox()

So we don't like operators like ?? because its too cryptic, but you're
> happy to have one-character class and property names.
>

No, I'm not happy with that.  I was just pointing out that code golf is
possible for someone who really wants it.  I'm happy to spend a few
characters for readability.  But one COULD write:

    from nested import GreedyAccess as G
    G._ = G.unbox


> What happens if you have an attribute that happens to be
> called "unbox" in your attribute look-up chain?
>

Don't use this class and/or modify the API slightly? This is getting
trite.  My 5 minute code isn't a final proposal, just a way of seeing a
napkin-sketch of a better approach.

-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/0df2cec7/attachment-0001.html>

From jelle.zijlstra at gmail.com  Mon Jul 23 14:01:58 2018
From: jelle.zijlstra at gmail.com (Jelle Zijlstra)
Date: Mon, 23 Jul 2018 11:01:58 -0700
Subject: [Python-ideas] slice[] to get more complex slices
In-Reply-To: <CAP7+vJLyM6HG6YShSN2mB=Hpek3Rxp2ypR9ZLn1fRTDZwOgZpg@mail.gmail.com>
References: <CAFpSVpKnE8b=QrjxLy6anAF3ammq2QbB4_K1Ey02Vk3n8EVwyQ@mail.gmail.com>
 <pj2lgk$c6c$1@blaine.gmane.org>
 <09cb9784-6a97-49b9-b5e7-1c9a9d8c8391@googlegroups.com>
 <CALD=Yf-MNLW7z8s=ruc_C13t7eWp8KC6OWvqf9durZgpN=8=sw@mail.gmail.com>
 <399ff7d70c204f679bdb5a79f9bc0460@xmail103.UGent.be>
 <5B55ACC9.8070303@UGent.be>
 <CAP7+vJLyM6HG6YShSN2mB=Hpek3Rxp2ypR9ZLn1fRTDZwOgZpg@mail.gmail.com>
Message-ID: <CAFp3-p_GEJ8z-YiojP=x6OruF+Jn_SwVL1WUS0UjmZJ1Ms8bjg@mail.gmail.com>

2018-07-23 8:16 GMT-07:00 Guido van Rossum <guido at python.org>:

> On Mon, Jul 23, 2018 at 3:24 AM, Jeroen Demeyer <J.Demeyer at ugent.be>
> wrote:
>
>> On 2018-07-23 11:58, Gr?gory Lielens wrote:
>>
>>> Not sure slice[1::3] can be done
>>>
>>
>> It can be done. Since "slice" is a class, it would require a metaclass
>> though.
>>
>
> Since PEP 560 it won't need a metaclass -- it can be implemented as
> __class_getitem__.
>
> But that brings up the related problem that `slice[]` for creating slice
objects would now conflict with typing syntax. And we might reasonably make
`slice` generic in the future: https://github.com/python/typing/issues/159.

> --
> --Guido van Rossum (python.org/~guido)
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/6b88f868/attachment.html>

From mertz at gnosis.cx  Mon Jul 23 14:09:00 2018
From: mertz at gnosis.cx (David Mertz)
Date: Mon, 23 Jul 2018 14:09:00 -0400
Subject: [Python-ideas] Python docs page: In what ways is None special
In-Reply-To: <CALD=Yf_jToxOguVsTdP1sW_5BXmK69iX0N4dnSmG-ECb9ZCb_w@mail.gmail.com>
References: <CALD=Yf_PRXZYiNdC9FGE5pd1cjrS8gyr8q5iPxzrMKXzX027YA@mail.gmail.com>
 <20180723122204.GX8744@ando.pearwood.info>
 <CALD=Yf_jToxOguVsTdP1sW_5BXmK69iX0N4dnSmG-ECb9ZCb_w@mail.gmail.com>
Message-ID: <CAEbHw4b5tLh4-wAKveQkMwt+MOh8SB5KUk=taALC5a2JSbD-uA@mail.gmail.com>

I think your description of the uses of None is really great.  There's
definitely no reason it cannot be a blog post immediately, but perhaps at
some later point included in The Python Tutorial.

On Mon, Jul 23, 2018 at 8:39 AM Jonathan Fine <jfine2358 at gmail.com> wrote:

> Hi Steve
>
> You wrote
>
> > I think your document would be a great blog post. I don't think it is
> > very helpful as part of the standard documention, as it is more a
> > collection of arbitrary facts about None than a coherent "big picture"
> > document.
>
> Thank you. That's nice. However, I think there is, or should be, more
> coherence than you're seeing.
>
> Here's something I forgot to say. For standard documentation I was
> thinking The Python Tutorial.  For example, the Brief Tour.
> https://docs.python.org/3/tutorial/stdlib.html
> https://docs.python.org/3/tutorial/stdlib2.html
>
> That said, a blog post would be useful stepping stone, and the
> standard documentation later. (Know where your heading, take one step
> at a time.)
>
> Once again, thank you for your comment.
>
> --
> Jonathan
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/97825a66/attachment.html>

From rhodri at kynesim.co.uk  Mon Jul 23 14:11:39 2018
From: rhodri at kynesim.co.uk (Rhodri James)
Date: Mon, 23 Jul 2018 19:11:39 +0100
Subject: [Python-ideas] A better (simpler) approach to PEP 505
In-Reply-To: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
References: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
Message-ID: <46c5439d-a806-d7e2-233f-91dfd77498ad@kynesim.co.uk>

On 23/07/18 16:12, David Mertz wrote:
> Here is a way of solving the "deep attribute access to messy data" problem

Before we go too far, and /pace/ Steve's work, how real is this problem? 
  I get that there is a use in accessing JSON trees, but this feels 
awfully like a rather specific issue about avoiding cleaning data before 
using it.  As such, should be really be encouraging it?

(For the avoidance of doubt, this refers to the proposed ?. and ?[] 
operators.

> that is:
> 
> (1) Much more explicit
> (2) Requires no change in syntax
> (3) Will not be a bug magnet
> (4) Inasmuch as there are semantic traps, they are announced by the use of
> a class whose documentation would be pointed to for readers

Worthy goals.

> The API that could be useful might be something like this:
> 
> In [1]: from none_aware import NoneAware
> In [2]: from types import SimpleNamespace
> In [3]: foo = SimpleNamespace()
> In [4]: foo.bar = SimpleNamespace()
> In [5]: foo.bar.baz = SimpleNamespace()
> In [6]: foo.bar.baz.blat = 42
> In [7]: NoneAware(foo).bar.blim
> Out[7]: <none_aware.NoneAware at 0x11156a748>

...and here's the first semantic trap.  I was expecting to get None back 
there, and even your pre-announcement in goal (4) didn't stop me 
puzzling over it for a couple of minutes.  The only reason I wasn't 
expecting an AttributeError was that I was confident you wouldn't 
propose this without doing something magic to effectively override the 
meaning of "."  A naive user might be very surprised.

> In [8]: NoneAware(foo).bar.blim.unbox()
> In [9]: NoneAware(foo).bar.baz.blat.unbox()
> Out[9]: 42

[snip]

> I don't disagree that needing to call .unbox() at the end of the chained
> attribute access is a little bit ugly.  But it's a lot less ugly than large
> family of new operators.  And honestly, it's a nice way of being explicit
> about the fact that we're entering then leaving a special world where
> attribute accesses don't fail.

You and I have very different definitions of "nice" ;-)  I have to say 
it's questionable which of "?." and "NoneAware()...unbox()" are uglier. 
Then again, I go back to my original comment: is this really a problem 
we want to solve?

How are you supposed to do method calling, the equivalent of 
"foo?.bar()" ?  "NoneAware(foo).bar.unbox()()" looks downright weird. 
Is there more magic in NoneAware to cover this case?  (Not that I think 
we should be encouraging people to do this, but...)

-- 
Rhodri James *-* Kynesim Ltd

From mertz at gnosis.cx  Mon Jul 23 14:21:37 2018
From: mertz at gnosis.cx (David Mertz)
Date: Mon, 23 Jul 2018 14:21:37 -0400
Subject: [Python-ideas] A better (simpler) approach to PEP 505
In-Reply-To: <46c5439d-a806-d7e2-233f-91dfd77498ad@kynesim.co.uk>
References: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
 <46c5439d-a806-d7e2-233f-91dfd77498ad@kynesim.co.uk>
Message-ID: <CAEbHw4bMYVANcvOsgSuhomKHeKMb=vAHb3hZV_=pMfCanTVP7A@mail.gmail.com>

On Mon, Jul 23, 2018 at 2:12 PM Rhodri James <rhodri at kynesim.co.uk> wrote:

> How are you supposed to do method calling, the equivalent of
> "foo?.bar()" ?  "NoneAware(foo).bar.unbox()()" looks downright weird.
> Is there more magic in NoneAware to cover this case?  (Not that I think
> we should be encouraging people to do this, but...)
>

Is there more magic? I don't know, and I don't really care that much.
That's the point.  This is just a plain old Python class that will work
back to probably Python 1.4 or so.  If you want to write a version that has
just the right amount of magic, you are free to.

That said, I *DID* give it the wrong name in my first post of this thread.
GreedyAccess is more accurate, and a NoneCoalesce class would behave
somewhat differently.

In my opinion, the need at issue is worthwhile, but niche.  Using a special
class to deal with such a case is absolutely the right level of
abstraction.  Syntax is not! So sure, figure out how to tweak the API to be
most useful, find a better name for the class, etc.

I wouldn't think it terrible if a class like this (but better) found a home
in the standard library, but it doesn't deserve more prominence than that.
Not even builtins, and *definitely* not syntax.

-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/5d3cef51/attachment-0001.html>

From python-ideas-2018 at tutnicht.de  Mon Jul 23 14:30:06 2018
From: python-ideas-2018 at tutnicht.de (=?iso-8859-1?Q?J=F6rn?= Heissler)
Date: Mon, 23 Jul 2018 20:30:06 +0200
Subject: [Python-ideas] Python docs page: In what ways is None special
In-Reply-To: <CALD=Yf_PRXZYiNdC9FGE5pd1cjrS8gyr8q5iPxzrMKXzX027YA@mail.gmail.com>
References: <CALD=Yf_PRXZYiNdC9FGE5pd1cjrS8gyr8q5iPxzrMKXzX027YA@mail.gmail.com>
Message-ID: <20180723183006.GA25280@carrot.tutnicht.de>

On Mon, Jul 23, 2018 at 10:03:10 +0100, Jonathan Fine wrote:
> I thought, a page on how None is special would be nice.
> I've not found such a page on the web. We do have
> ===
> https://docs.python.org/3/library/constants.html

Hi,

there's also
https://docs.python.org/3/reference/datamodel.html#the-standard-type-hierarchy

None

    This type has a single value. There is a single object with this
    value. This object is accessed through the built-in name None. It is
    used to signify the absence of a value in many situations, e.g., it
    is returned from functions that don?t explicitly return anything.
    Its truth value is false.


Cheers
J?rn Heissler

From storchaka at gmail.com  Mon Jul 23 14:38:40 2018
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 23 Jul 2018 21:38:40 +0300
Subject: [Python-ideas] slice[] to get more complex slices
In-Reply-To: <CAP7+vJLyM6HG6YShSN2mB=Hpek3Rxp2ypR9ZLn1fRTDZwOgZpg@mail.gmail.com>
References: <CAFpSVpKnE8b=QrjxLy6anAF3ammq2QbB4_K1Ey02Vk3n8EVwyQ@mail.gmail.com>
 <pj2lgk$c6c$1@blaine.gmane.org>
 <09cb9784-6a97-49b9-b5e7-1c9a9d8c8391@googlegroups.com>
 <CALD=Yf-MNLW7z8s=ruc_C13t7eWp8KC6OWvqf9durZgpN=8=sw@mail.gmail.com>
 <399ff7d70c204f679bdb5a79f9bc0460@xmail103.UGent.be>
 <5B55ACC9.8070303@UGent.be>
 <CAP7+vJLyM6HG6YShSN2mB=Hpek3Rxp2ypR9ZLn1fRTDZwOgZpg@mail.gmail.com>
Message-ID: <pj577d$vef$1@blaine.gmane.org>

23.07.18 18:16, Guido van Rossum ????:
> On Mon, Jul 23, 2018 at 3:24 AM, Jeroen Demeyer 
> <J.Demeyer at ugent.be 
> <mailto:J.Demeyer at ugent.be>> wrote:
> 
>     On 2018-07-23 11:58, Gr?gory Lielens wrote:
> 
>         Not sure slice[1::3] can be done
> 
> 
>     It can be done. Since "slice" is a class, it would require a
>     metaclass though.
> 
> 
> Since PEP 560 it won't need a metaclass -- it can be implemented as 
> __class_getitem__.

Do you bless using __class_getitem__ for something other than typing?


From mehaase at gmail.com  Mon Jul 23 14:43:03 2018
From: mehaase at gmail.com (Mark E. Haase)
Date: Mon, 23 Jul 2018 14:43:03 -0400
Subject: [Python-ideas] A better (simpler) approach to PEP 505
In-Reply-To: <46c5439d-a806-d7e2-233f-91dfd77498ad@kynesim.co.uk>
References: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
 <46c5439d-a806-d7e2-233f-91dfd77498ad@kynesim.co.uk>
Message-ID: <CALb0Rk6by4_UedEzstqbR6kBdD4drQeNKr28Df+7An0mMqzyGg@mail.gmail.com>

On Mon, Jul 23, 2018 at 2:12 PM Rhodri James <rhodri at kynesim.co.uk> wrote:

>
> How are you supposed to do method calling, the equivalent of
> "foo?.bar()" ?  "NoneAware(foo).bar.unbox()()" looks downright weird.
> Is there more magic in NoneAware to cover this case?  (Not that I think
> we should be encouraging people to do this, but...)
>
>
In PyMaybe, you would do this:

maybe(foo).bar().or_else(None)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/099978c0/attachment.html>

From jfine2358 at gmail.com  Mon Jul 23 14:50:13 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Mon, 23 Jul 2018 19:50:13 +0100
Subject: [Python-ideas] slice[] to get more complex slices
In-Reply-To: <pj577d$vef$1@blaine.gmane.org>
References: <CAFpSVpKnE8b=QrjxLy6anAF3ammq2QbB4_K1Ey02Vk3n8EVwyQ@mail.gmail.com>
 <pj2lgk$c6c$1@blaine.gmane.org>
 <09cb9784-6a97-49b9-b5e7-1c9a9d8c8391@googlegroups.com>
 <CALD=Yf-MNLW7z8s=ruc_C13t7eWp8KC6OWvqf9durZgpN=8=sw@mail.gmail.com>
 <399ff7d70c204f679bdb5a79f9bc0460@xmail103.UGent.be>
 <5B55ACC9.8070303@UGent.be>
 <CAP7+vJLyM6HG6YShSN2mB=Hpek3Rxp2ypR9ZLn1fRTDZwOgZpg@mail.gmail.com>
 <pj577d$vef$1@blaine.gmane.org>
Message-ID: <CALD=Yf-vZoC8Gf4_BtcWDUPhF8Xu9J2CL_M6gS3RzTMO2XD-Fg@mail.gmail.com>

Humour warning.

Serhiy wrote:

> Do you bless using __class_getitem__ for something other than typing?

As in https://perldoc.perl.org/functions/bless.html, perhaps?

-- 
Jonathan

From joejev at gmail.com  Mon Jul 23 14:59:59 2018
From: joejev at gmail.com (Joseph Jevnik)
Date: Mon, 23 Jul 2018 14:59:59 -0400
Subject: [Python-ideas] slice[] to get more complex slices
In-Reply-To: <pj577d$vef$1@blaine.gmane.org>
References: <CAFpSVpKnE8b=QrjxLy6anAF3ammq2QbB4_K1Ey02Vk3n8EVwyQ@mail.gmail.com>
 <pj2lgk$c6c$1@blaine.gmane.org>
 <09cb9784-6a97-49b9-b5e7-1c9a9d8c8391@googlegroups.com>
 <CALD=Yf-MNLW7z8s=ruc_C13t7eWp8KC6OWvqf9durZgpN=8=sw@mail.gmail.com>
 <399ff7d70c204f679bdb5a79f9bc0460@xmail103.UGent.be>
 <5B55ACC9.8070303@UGent.be>
 <CAP7+vJLyM6HG6YShSN2mB=Hpek3Rxp2ypR9ZLn1fRTDZwOgZpg@mail.gmail.com>
 <pj577d$vef$1@blaine.gmane.org>
Message-ID: <CAHGq92Wv5Owbb+_pk8rMqNLX=abCX4EUQMf1JhDRzL0RgwptPg@mail.gmail.com>

I still think that 'operator.subscript' would be valuable to me for
all of the same reasons discussed in the previous threads and issues.
I don't understand why it was reverted without any serious discussion
given that it was already accepted and many people find this useful.

On Mon, Jul 23, 2018 at 2:38 PM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> 23.07.18 18:16, Guido van Rossum ????:
>>
>> On Mon, Jul 23, 2018 at 3:24 AM, Jeroen Demeyer <J.Demeyer at ugent.be
>> <mailto:J.Demeyer at ugent.be>> wrote:
>>
>>     On 2018-07-23 11:58, Gr?gory Lielens wrote:
>>
>>         Not sure slice[1::3] can be done
>>
>>
>>     It can be done. Since "slice" is a class, it would require a
>>     metaclass though.
>>
>>
>> Since PEP 560 it won't need a metaclass -- it can be implemented as
>> __class_getitem__.
>
>
> Do you bless using __class_getitem__ for something other than typing?
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From rhodri at kynesim.co.uk  Mon Jul 23 14:54:20 2018
From: rhodri at kynesim.co.uk (Rhodri James)
Date: Mon, 23 Jul 2018 19:54:20 +0100
Subject: [Python-ideas] A better (simpler) approach to PEP 505
In-Reply-To: <CAEbHw4bMYVANcvOsgSuhomKHeKMb=vAHb3hZV_=pMfCanTVP7A@mail.gmail.com>
References: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
 <46c5439d-a806-d7e2-233f-91dfd77498ad@kynesim.co.uk>
 <CAEbHw4bMYVANcvOsgSuhomKHeKMb=vAHb3hZV_=pMfCanTVP7A@mail.gmail.com>
Message-ID: <bfb2ab68-51e2-d5eb-667d-2d4a09e951a7@kynesim.co.uk>

On 23/07/18 19:21, David Mertz wrote:
> On Mon, Jul 23, 2018 at 2:12 PM Rhodri James <rhodri at kynesim.co.uk> wrote:
> 
>> How are you supposed to do method calling, the equivalent of
>> "foo?.bar()" ?  "NoneAware(foo).bar.unbox()()" looks downright weird.
>> Is there more magic in NoneAware to cover this case?  (Not that I think
>> we should be encouraging people to do this, but...)
>>
> 
> Is there more magic? I don't know, and I don't really care that much.
> That's the point.  This is just a plain old Python class that will work
> back to probably Python 1.4 or so.  If you want to write a version that has
> just the right amount of magic, you are free to.

I care only in as much as you were proposing an incomplete solution (in 
my eyes at least).  I'm pretty convinced by now I would never use it.

> In my opinion, the need at issue is worthwhile, but niche.  Using a special
> class to deal with such a case is absolutely the right level of
> abstraction.  Syntax is not! So sure, figure out how to tweak the API to be
> most useful, find a better name for the class, etc.

I think part of my point was that a special class is not as readable or 
trap-free as writing the conditions out explicitly, given the unexpected 
"boxed" results, which makes me question whether the issue *is* worthwhile.

> I wouldn't think it terrible if a class like this (but better) found a home
> in the standard library, but it doesn't deserve more prominence than that.
> Not even builtins, and *definitely* not syntax.

I think I might find it terrible.

-- 
Rhodri James *-* Kynesim Ltd

From desmoulinmichel at gmail.com  Mon Jul 23 15:17:52 2018
From: desmoulinmichel at gmail.com (Michel Desmoulin)
Date: Mon, 23 Jul 2018 21:17:52 +0200
Subject: [Python-ideas] A better (simpler) approach to PEP 505
In-Reply-To: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
References: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
Message-ID: <2bc835d4-163d-dcf1-6c38-1dd87a725011@gmail.com>




Le 23/07/2018 ? 17:12, David Mertz a ?crit :
> The need addressed by PEP 505 is real; it's also MUCH more niche and
> uncommon than something that would merit new syntax.  Moreover, the
> actual legitimate purpose served by the PEP 505 syntax is easily served
> by existing Python simply by using a wrapper class.
>
> Here is a way of solving the "deep attribute access to messy data"
> problem that is:
>
> (1) Much more explicit
> (2) Requires no change in syntax
> (3) Will not be a bug magnet
> (4) Inasmuch as there are semantic traps, they are announced by the use
> of a class whose documentation would be pointed to for readers
>
> The API that could be useful might be something like this:
>
>     In [1]: from none_aware import NoneAware
>     In [2]: from types import SimpleNamespace
>     In [3]: foo = SimpleNamespace()
>     In [4]: foo.bar = SimpleNamespace()
>     In [5]: foo.bar.baz = SimpleNamespace()
>     In [6]: foo.bar.baz.blat = 42
>     In [7]: NoneAware(foo).bar.blim
>     Out[7]: <none_aware.NoneAware at 0x11156a748>
>     In [8]: NoneAware(foo).bar.blim.unbox()
>     In [9]: NoneAware(foo).bar.baz.blat.unbox()
>     Out[9]: 42
>     In [10]: NoneAware(foo).bar.baz.blat
>     Out[10]: <none_aware.NoneAware at 0x11157d908>
>     In [11]: NoneAware(foo).bar.baz.flam.unbox()
>     In [12]: NoneAware(foo).bar.baz.flam
>     Out[12]: <none_aware.NoneAware at 0x1115832b0>
This has existed as libs for a while:

https://github.com/ekampf/pymaybe


It's interest is much more limited than in Haskell because in Python,
calls are not lazy, objects are dynamic and function calls are expensive.


Take the following:

foo().bar[0]

Doing it safely would be:

    val = foo()

    try:
        val = val.bar
    except AttributeError:
        val = None
    else:
        try:
            val = val[0]
        except KeyError:
            val = None

Clearly we see that the operation goes up to the end if everything goes
right. Otherwise, it jumps to the default value.

With the operator proposal:

    val = foo()?.bar?[0]

The "?" operator is like "or" / "and", and it will shotcircuit as well.

With the try / else proposal:

    val = try foo().bar[0] else None

This is supposed to transform this ast to into the first try/except
form. So it shortcircuit as well.

But with your proposal, it becomes:

    val = maybe(foo()).bar[0].or_else(None)

Which means

    foo() # call

    maybe(foo()) # call

    maybe(foo()).__getattr__('bar') # call

    maybe(foo()).__getattr__('bar').__getitem__(0) # call

    maybe(foo()).__getattr__('bar').__getitem__(0).or_else(None) # call

There is no shortcircuit, you get 5 calls in all cases, plus the maybe
object proxing the call to the underlying value every time. So that's 7
calls, added to the try/excepts you still have behind the scene.

Plus, you don't get the same calls depending of the values, but the
switch is implicit and behind the maybe object: no linter can save you
from a typo.

So yes, it works right now. But it's a suboptimal solution.

From gregory.lielens at gmail.com  Mon Jul 23 16:42:48 2018
From: gregory.lielens at gmail.com (=?UTF-8?Q?Gr=C3=A9gory_Lielens?=)
Date: Mon, 23 Jul 2018 13:42:48 -0700 (PDT)
Subject: [Python-ideas] A better (simpler) approach to PEP 505
In-Reply-To: <2bc835d4-163d-dcf1-6c38-1dd87a725011@gmail.com>
References: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
 <2bc835d4-163d-dcf1-6c38-1dd87a725011@gmail.com>
Message-ID: <720ea42c-a662-430e-9ee7-9a302a866ce3@googlegroups.com>

Short circuit is indeed the main difference. 
Makes me re-think about the None trigger subclasses of invalid operations exceptions.
Like None.a trigger a NoneAttributeError(AttributeError), and so on...
I think it solves many issues without much problems nor syntax changes, but probably I miss something...

Sure, it's a big change, but maybe less so than 3 non-classic operators working only on None...

From waksman at gmail.com  Mon Jul 23 17:15:59 2018
From: waksman at gmail.com (George Leslie-Waksman)
Date: Mon, 23 Jul 2018 14:15:59 -0700
Subject: [Python-ideas] slice[] to get more complex slices
In-Reply-To: <CAHGq92Wv5Owbb+_pk8rMqNLX=abCX4EUQMf1JhDRzL0RgwptPg@mail.gmail.com>
References: <CAFpSVpKnE8b=QrjxLy6anAF3ammq2QbB4_K1Ey02Vk3n8EVwyQ@mail.gmail.com>
 <pj2lgk$c6c$1@blaine.gmane.org>
 <09cb9784-6a97-49b9-b5e7-1c9a9d8c8391@googlegroups.com>
 <CALD=Yf-MNLW7z8s=ruc_C13t7eWp8KC6OWvqf9durZgpN=8=sw@mail.gmail.com>
 <399ff7d70c204f679bdb5a79f9bc0460@xmail103.UGent.be>
 <5B55ACC9.8070303@UGent.be>
 <CAP7+vJLyM6HG6YShSN2mB=Hpek3Rxp2ypR9ZLn1fRTDZwOgZpg@mail.gmail.com>
 <pj577d$vef$1@blaine.gmane.org>
 <CAHGq92Wv5Owbb+_pk8rMqNLX=abCX4EUQMf1JhDRzL0RgwptPg@mail.gmail.com>
Message-ID: <CACef2a=q-V4zboqv===GTA3AxfOmfnAcPaMOEO8ZMVJ7FwipmQ@mail.gmail.com>

May I propose `slice.L` as in "slice literal":

```
class slice:
    ...
    class L:
    """Slice literal.

    slice.L[1:2:3, 1] -> (slice(1, 2, 3), slice(1))
    """

    @classmethod
    def __class_getitem__(cls, item):
        return item
```

On Mon, Jul 23, 2018 at 12:02 PM Joseph Jevnik <joejev at gmail.com> wrote:

> I still think that 'operator.subscript' would be valuable to me for
> all of the same reasons discussed in the previous threads and issues.
> I don't understand why it was reverted without any serious discussion
> given that it was already accepted and many people find this useful.
>
> On Mon, Jul 23, 2018 at 2:38 PM, Serhiy Storchaka <storchaka at gmail.com>
> wrote:
> > 23.07.18 18:16, Guido van Rossum ????:
> >>
> >> On Mon, Jul 23, 2018 at 3:24 AM, Jeroen Demeyer <J.Demeyer at ugent.be
> >> <mailto:J.Demeyer at ugent.be>> wrote:
> >>
> >>     On 2018-07-23 11:58, Gr?gory Lielens wrote:
> >>
> >>         Not sure slice[1::3] can be done
> >>
> >>
> >>     It can be done. Since "slice" is a class, it would require a
> >>     metaclass though.
> >>
> >>
> >> Since PEP 560 it won't need a metaclass -- it can be implemented as
> >> __class_getitem__.
> >
> >
> > Do you bless using __class_getitem__ for something other than typing?
> >
> >
> > _______________________________________________
> > Python-ideas mailing list
> > Python-ideas at python.org
> > https://mail.python.org/mailman/listinfo/python-ideas
> > Code of Conduct: http://python.org/psf/codeofconduct/
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/927e88c0/attachment.html>

From gjcarneiro at gmail.com  Mon Jul 23 17:25:14 2018
From: gjcarneiro at gmail.com (Gustavo Carneiro)
Date: Mon, 23 Jul 2018 22:25:14 +0100
Subject: [Python-ideas] Add function readbyte to asyncio.StreamReader
In-Reply-To: <20180722110340.GA23320@carrot.tutnicht.de>
References: <20180722110340.GA23320@carrot.tutnicht.de>
Message-ID: <CAO-CpEKfFwNo__dyL0260dRbYZyuiUEC9tYT3KJxD8V6jHYQ1g@mail.gmail.com>

Well, even if it is worth, i.e. your use case is not rare enough, I would
suggest at least making it private, readexactly can call this specialised
function if nbytes==1:

def _readbyte(self):
   ....

def readexactly(self, num):
   if num == 1:
      return self._readbyte()
  ... the rest stays the same..


But to be honest, you are probably better off managing the buffer yourself:
Just call, e.g., stream.read(4096), it will return a buffer of up to 4k
length, then you can iterate over the buffer byte by byte until the
condition is met, repeat until the end of stream, or whatever.


On Sun, 22 Jul 2018 at 12:11, J?rn Heissler <python-ideas-2018 at tutnicht.de>
wrote:

> Hello,
>
> I'm implementing a protocol where I need to read individual bytes until
> a condition is met (value & 0x80 == 0).
>
> My current approach is: value = (await reader.readexactly(1))[0]
>
> To speed this up, I propose that a new function is added to
> asyncio.StreamReader: value = await reader.readbyte()
>
> I duplicated readexactly and stripped out some parts. Below code appears
> to work:
>
>     async def readbyte(self):
>         if self._exception is not None:
>             raise self._exception
>
>         while not self._buffer:
>             if self._eof:
>                 raise EOFError()
>             await self._wait_for_data('readbyte')
>
>         data = self._buffer[0]
>         del self._buffer[0]
>         self._maybe_resume_transport()
>         return data
>
> For comparing the speed, I'm receiving a 50 MiB file byte-by-byte.
>
> cpython-3.7.0:
>     readexactly: 42.43 seconds
>     readbyte   : 22.05 seconds
>     speedup    : 92.4%
>
> pypy3-v6.0.0:
>     readexactly: 3.21 seconds
>     readbyte   : 2.76 seconds
>     speedup    : 16.3%
>
> Thanks
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
Gustavo J. A. M. Carneiro
Gambit Research
"The universe is always one step beyond logic." -- Frank Herbert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/2bcc1d3e/attachment-0001.html>

From g.rodola at gmail.com  Mon Jul 23 18:05:14 2018
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Tue, 24 Jul 2018 00:05:14 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180723165256.GB8744@ando.pearwood.info>
References: <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
Message-ID: <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>

On Mon, Jul 23, 2018 at 6:53 PM Steven D'Aprano <steve at pearwood.info> wrote:
>
> On Mon, Jul 23, 2018 at 02:04:17PM +0200, Giampaolo Rodola' wrote:
> > "a?.b" does two and that's a fundamental difference (explicitness).
>
> How is "two things" less explicit than "one thing"?
> Comments like the above is why I think that "explicit" and "implicit"
> are used to mean "I like it" and "I don't like it" rather than being
> objective arguments, or indeed having anything to do with explicitness
> or implicitness.

This:

    v = a?.b

...*implicitly* checks if value is not None [and continues execution]. This:

    v = a
    if a.b is not None:
        v = a.b

...*explicitly* checks if value is not None and continues execution.
If for some reason '?'[ is also going to swallow LookupError then
*that* would further decrease explicitness, because LookupError would
be nowhere in sight, the same way "if", "is", "not", "None", ":", "new
line" are nowhere in sight in the 'a?.b' example. Some argued "A ?? B"
is less explicit than "A if A is not None else B" for the same reason.
One may argue that silently returning None instead of raising
AttributeError is also less explicit.
This - and this only - is my argument about explicitness. It doesn't
have to do with how many things are hidden behind an import statement
or what happens on sorted() (that's comparing apples and oranges).
I hope it's clear now.

> > It
> > does so by introducing a brand new operator ("?") which can be spelled
> > in two forms ("a?.b" and "a?[b]") by using two adjacent symbols not
> > interrupted by any space, which is an absolute first in the Python
> > syntax
>
> It isn't a first. Many existing operators use two adjacent symbols not
> interrupted by a space:
>
> e.g.  ==  <=  >=  !=  **  //  << >> +=  -=  *= etc.

You say 'a == b'. You can't say 'a ?. b' (not that it matters, it
would be less intuitive anyway). You can't because '.?' is the only
couple of contiguous symbols requiring "something" before and after
with no spaces in between, and that's a first in the language. The
argument about this is that it's ugly and less readable. My additional
argument at the beginning of this thread was that if you add PEP-572
to the mix you dangerously enter into Perl territory:

    foo(x=(x := a?.b?[c] ?? d))

> > and that's the second and fundamental difference. I cannot move
> > the same criticism to the "a.b" form: it's simpler, it does one thing
> > and it uses one symbol.
>
> You criticised ?. because it can interupt left-to-right execution:
>
>     a?.b?.c?.d
>
> True. But so can a single dot:
>
>     a.b.c.d
>
> is no more guaranteed to execute all the way to the right.

The difference is that 'a.b.c.d' will result in AttributeError as soon
as something is None while 'a?.b?.c?.d' will return None instead.

> Likewise the logical operators "or" and "and" are designed to
> short-circuit. If ?? and friends are a mistake because they
> short-circuit, why aren't "or" and "and" mistakes?
> I'm not asking this as a rhetorical question. If you think there is a
> reason why it is okay for or/and to short-circuit, but it is bad for ??
> and friends to short-circuit, then please explain why they are
> different. I will be very happy to listen to your arguments.

The argument about this is that '?.' short-circuits execution
*silently*. Instead of AttributeError you get None. You may chain ?.
in order to lazily traverse a long tree, inadvertently assign None to
a variable, continue code execution and fail later rather than sooner:

     email = request?.context?.user?.email  # None
     ...
     sendmail(subject, body, email)

Some (Antoine) rightly argued this may even have security implications
(replace 'email' with 'password').

-- 
Giampaolo - http://grodola.blogspot.com

From rosuav at gmail.com  Mon Jul 23 18:23:48 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 24 Jul 2018 08:23:48 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
References: <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
Message-ID: <CAPTjJmqKvC0PbuQxFXs55tT=Zz=ZWh9P-Ct1H-e23DXVz3Lqdg@mail.gmail.com>

On Tue, Jul 24, 2018 at 8:05 AM, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
> The argument about this is that '?.' short-circuits execution
> *silently*. Instead of AttributeError you get None. You may chain ?.
> in order to lazily traverse a long tree, inadvertently assign None to
> a variable, continue code execution and fail later rather than sooner:
>
>      email = request?.context?.user?.email  # None
>      ...
>      sendmail(subject, body, email)
>
> Some (Antoine) rightly argued this may even have security implications
> (replace 'email' with 'password').
>

This thread has been long and rambling. Can you elaborate on the
security implications? Normally, I would expect that a password of
None would always fail to validate.

ChrisA

From tjol at tjol.eu  Mon Jul 23 18:22:05 2018
From: tjol at tjol.eu (Thomas Jollans)
Date: Tue, 24 Jul 2018 00:22:05 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
Message-ID: <7805016a-dc57-39bc-be07-c3e3e26ee5d4@tjol.eu>

On 18/07/18 19:43, Steve Dower wrote:
> When a ``None``-aware operator is present, the left-to-right evaluation
> may be
> short-circuited. For example, ``await a?.b(c).d?[e]`` is evaluated::
> 
> ??? _v = a
> ??? if _v is not None:
> ??????? _v = _v.b
> ??????? _v = _v(c)
> ??????? _v = _v.d
> ??????? if _v is not None:
> ??????????? _v = _v[e]
> ??? await _v

## NB I only skimmed most of this thread after reading the PEP, so I ##
##    apologize if this has been discussed before and I missed it.   ##

I quite like the general idea, but I'm nervous about a rather
fundamental aspect: This adds a special case in which you can't add
parentheses to an expression involving a chain of operators to make
precedence and evaluation order clear.

To use your example,
a ?? 2 ** b ?? 3   ===  (a ?? 2) ** (b ?? 3) # fine

In the present day,
a or 2 + 3 * c()   ===   a or (2 + (3 * (c())))

a.b(c).d[e]   ===   (((a.b)(c)).d)[e] # silly, but true.

Short-circuiting doesn't break this. With and and or, the expression
that's short-circuited away is a self-contained expression in imagined
(or actual) parentheses.

With this ?. operator, the chain a?.b(c).d?[e] can no longer be broken
into sub-expressions, but becomes one single, long, atomic expression,
just like a comparison chain. If I try:

(a?.b)(c).d?[e] # TypeError: 'NoneType' object is not callable
a?.(b(c).d?[e]) # SyntaxError, and illogical

Also, where does this end? if a is None, is (a?.b,c()) equal to None or
(None, c())? Presumably the latter because of operator precedence, but
still.

Is introducing the idea of an "attribute reference, call and
subscription" chain worth it?

This could be fixed by adding a maybe-call ?( operator, which would allow

a?.b?(C())?.d?[E()]
   ===
	((((a?.b)
                ?( C() ))
                         ?.d)
                           ?[ E() ])

_v = a
_v = _v.b if _v is not None else None
_v = _v(C()) if _v is not None else None
_v = _v.d if _v is not None else None
_v = _v[E()] if _v is not None else None

with None falling all the way through, and the calls to C() and E()
being short-circuited out.

Of course you need either attribute-call-subscription chains or ??()?
maybe-calls for ??.? to be worthwhile at all, since otherwise you can't
write None-aware method calls.

Aside:  other languages cited
  From a quick look at the C# docs linked in the PEP [1], I'm guessing
  that C# allows A?.B?.C?.Do(E), but does not allow A?.B?.C.Do(E). Does
  anybody know? Obviously method calls work differently in C# than in
  Python.

  Dart? I dunno. The docs aren't as thorough.


Am I missing something?


Cheers
Thomas


[1]
https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/operators/null-conditional-operators


From rosuav at gmail.com  Mon Jul 23 18:39:34 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 24 Jul 2018 08:39:34 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <7805016a-dc57-39bc-be07-c3e3e26ee5d4@tjol.eu>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <7805016a-dc57-39bc-be07-c3e3e26ee5d4@tjol.eu>
Message-ID: <CAPTjJmqfHhSGmT4R-kovk-ufLpcwFzgaP=qeLgygEpywTFoLaQ@mail.gmail.com>

On Tue, Jul 24, 2018 at 8:22 AM, Thomas Jollans <tjol at tjol.eu> wrote:
> On 18/07/18 19:43, Steve Dower wrote:
>> When a ``None``-aware operator is present, the left-to-right evaluation
>> may be
>> short-circuited. For example, ``await a?.b(c).d?[e]`` is evaluated::
>>
>>     _v = a
>>     if _v is not None:
>>         _v = _v.b
>>         _v = _v(c)
>>         _v = _v.d
>>         if _v is not None:
>>             _v = _v[e]
>>     await _v
>
> ## NB I only skimmed most of this thread after reading the PEP, so I ##
> ##    apologize if this has been discussed before and I missed it.   ##
>
> I quite like the general idea, but I'm nervous about a rather
> fundamental aspect: This adds a special case in which you can't add
> parentheses to an expression involving a chain of operators to make
> precedence and evaluation order clear.
>
> To use your example,
> a ?? 2 ** b ?? 3   ===  (a ?? 2) ** (b ?? 3) # fine
>
> In the present day,
> a or 2 + 3 * c()   ===   a or (2 + (3 * (c())))
>
> a.b(c).d[e]   ===   (((a.b)(c)).d)[e] # silly, but true.
>
> Short-circuiting doesn't break this. With and and or, the expression
> that's short-circuited away is a self-contained expression in imagined
> (or actual) parentheses.

What about:

5 < x < 10

Can you add parentheses to that to "make precedence and evaluation order clear"?

ChrisA

From tjol at tjol.eu  Mon Jul 23 18:54:25 2018
From: tjol at tjol.eu (Thomas Jollans)
Date: Tue, 24 Jul 2018 00:54:25 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPTjJmqfHhSGmT4R-kovk-ufLpcwFzgaP=qeLgygEpywTFoLaQ@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <7805016a-dc57-39bc-be07-c3e3e26ee5d4@tjol.eu>
 <CAPTjJmqfHhSGmT4R-kovk-ufLpcwFzgaP=qeLgygEpywTFoLaQ@mail.gmail.com>
Message-ID: <63bf2add-661f-8f31-0c48-9d0f8efdc25a@tjol.eu>

On 24/07/18 00:39, Chris Angelico wrote:
> On Tue, Jul 24, 2018 at 8:22 AM, Thomas Jollans <tjol at tjol.eu> wrote:
>> On 18/07/18 19:43, Steve Dower wrote:
>>> When a ``None``-aware operator is present, the left-to-right evaluation
>>> may be
>>> short-circuited. For example, ``await a?.b(c).d?[e]`` is evaluated::
>>>
>>>     _v = a
>>>     if _v is not None:
>>>         _v = _v.b
>>>         _v = _v(c)
>>>         _v = _v.d
>>>         if _v is not None:
>>>             _v = _v[e]
>>>     await _v
>>
>> ## NB I only skimmed most of this thread after reading the PEP, so I ##
>> ##    apologize if this has been discussed before and I missed it.   ##
>>
>> I quite like the general idea, but I'm nervous about a rather
>> fundamental aspect: This adds a special case in which you can't add
>> parentheses to an expression involving a chain of operators to make
>> precedence and evaluation order clear.
>>
>> To use your example,
>> a ?? 2 ** b ?? 3   ===  (a ?? 2) ** (b ?? 3) # fine
>>
>> In the present day,
>> a or 2 + 3 * c()   ===   a or (2 + (3 * (c())))
>>
>> a.b(c).d[e]   ===   (((a.b)(c)).d)[e] # silly, but true.
>>
>> Short-circuiting doesn't break this. With and and or, the expression
>> that's short-circuited away is a self-contained expression in imagined
>> (or actual) parentheses.
> 
> What about:
> 
> 5 < x < 10
> 
> Can you add parentheses to that to "make precedence and evaluation order clear"?


Well, no.

TJ wrote:
> [...] becomes one single, long, atomic expression,
> just like a comparison chain. [...]
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This does leave the question,

> Is introducing the idea of an "attribute reference, call and
> subscription" chain worth it?




From stefan_ml at behnel.de  Mon Jul 23 19:19:06 2018
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 24 Jul 2018 00:19:06 +0100
Subject: [Python-ideas] slice[] to get more complex slices
In-Reply-To: <CAEQ_Tvf2Y-up_uWoUdutyoreHeM6=X52bZzZcath-r1btYeqow@mail.gmail.com>
References: <CAFpSVpKnE8b=QrjxLy6anAF3ammq2QbB4_K1Ey02Vk3n8EVwyQ@mail.gmail.com>
 <pj2lgk$c6c$1@blaine.gmane.org>
 <09cb9784-6a97-49b9-b5e7-1c9a9d8c8391@googlegroups.com>
 <CALD=Yf-MNLW7z8s=ruc_C13t7eWp8KC6OWvqf9durZgpN=8=sw@mail.gmail.com>
 <399ff7d70c204f679bdb5a79f9bc0460@xmail103.UGent.be>
 <4243953379d74d2aa5393be2434a99a7@xmail103.UGent.be>
 <5B55AE6A.7010305@UGent.be>
 <CACac1F_XOi+1gmEMgZdbmW_3UXs661JS5BF=OvWLrF47H0fJmg@mail.gmail.com>
 <CAEQ_Tvf2Y-up_uWoUdutyoreHeM6=X52bZzZcath-r1btYeqow@mail.gmail.com>
Message-ID: <pj5nl6$p6b$1@blaine.gmane.org>

Stephan Hoyer schrieb am 23.07.2018 um 18:01:
> On Mon, Jul 23, 2018 at 4:24 AM Paul Moore wrote:
> 
>> I thought the reason the proposal got nowhere was because it's pretty
>> simple to define it yourself:
>>
>> >>> class SliceHelper:
>> ...     def __getitem__(self, slice):
>> ...         return slice
>> ...
>> >>> SH = SliceHelper()
>> >>> SH[1::3]
>> slice(1, None, 3)
>>
>> Did I miss something significant about why this wasn't sufficient?
> 
> 
> I think a SliceHelper class like this is a reasonable solution, but there
> would be a lot of value having it a standard place somewhere in the
> standard library (e.g., operator.subscript).
> 
> Both pandas and NumPy include this helper object under different names
> (pandas.IndexSlice and numpy.index_exp / numpy.s_), but it would be
> surprising/unexpected for pandas/numpy specific helpers to show up when not
> using one of those libraries. I do the exact same sorts of indexing
> manipulations with xarray, dask and TensorFlow.
> 
> Given that this is basically a simple feature to make it easier to work
> with Python syntax (so there's no danger it will change in the future), I
> think there is a lot to be said for putting it in the standard library in
> one place so it's obvious what to use and users don't have to relearn that
> name for this object and/or reimplement it.

Please copy that comment into the ticket and ask for it to be reopened.

https://bugs.python.org/issue24379

Stefan


From klahnakoski at mozilla.com  Mon Jul 23 19:23:31 2018
From: klahnakoski at mozilla.com (Kyle Lahnakoski)
Date: Mon, 23 Jul 2018 19:23:31 -0400
Subject: [Python-ideas] A better (simpler) approach to PEP 505
In-Reply-To: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
References: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
Message-ID: <1269368e-ab73-db53-c006-4f66d6fffd1e@mozilla.com>


I agree a class can provide a very good alternative to PEP505.? I built
one a while ago, and still use it https://github.com/klahnakoski/mo-dots
for most my data transformation code.

The library only boxes dicts and lists, which removes the need for
.unbox() in many of my use cases. My point is, if a class is chosen
instead of PEP505, I doubt the unboxing will be a big issue.


On 2018-07-23 11:12, David Mertz wrote:
> Here is a way of solving the "deep attribute access to messy data"
> problem that is:

From mertz at gnosis.cx  Mon Jul 23 19:37:08 2018
From: mertz at gnosis.cx (David Mertz)
Date: Mon, 23 Jul 2018 19:37:08 -0400
Subject: [Python-ideas] slice[] to get more complex slices
In-Reply-To: <pj5nl6$p6b$1@blaine.gmane.org>
References: <CAFpSVpKnE8b=QrjxLy6anAF3ammq2QbB4_K1Ey02Vk3n8EVwyQ@mail.gmail.com>
 <pj2lgk$c6c$1@blaine.gmane.org>
 <09cb9784-6a97-49b9-b5e7-1c9a9d8c8391@googlegroups.com>
 <CALD=Yf-MNLW7z8s=ruc_C13t7eWp8KC6OWvqf9durZgpN=8=sw@mail.gmail.com>
 <399ff7d70c204f679bdb5a79f9bc0460@xmail103.UGent.be>
 <4243953379d74d2aa5393be2434a99a7@xmail103.UGent.be>
 <5B55AE6A.7010305@UGent.be>
 <CACac1F_XOi+1gmEMgZdbmW_3UXs661JS5BF=OvWLrF47H0fJmg@mail.gmail.com>
 <CAEQ_Tvf2Y-up_uWoUdutyoreHeM6=X52bZzZcath-r1btYeqow@mail.gmail.com>
 <pj5nl6$p6b$1@blaine.gmane.org>
Message-ID: <CAEbHw4YuS99=cVbhV6Riuwij4H_xNdcxp1Df6tMQfD-__CaO_w@mail.gmail.com>

I find pandas.IndexSlice makes a lot of operations easier to spell. As
simple as it is, it's a valuable capability. Rather than every library?or a
number of them anyway?creating the same 4 lines of code with a different
name, it would be much nicer to have it as a class __getitem__ method, e.g.
slice[...], or as an attribute of the slice object, such as
slice.literal[...].

On Mon, Jul 23, 2018, 7:20 PM Stefan Behnel <stefan_ml at behnel.de> wrote:

> Stephan Hoyer schrieb am 23.07.2018 um 18:01:
> > On Mon, Jul 23, 2018 at 4:24 AM Paul Moore wrote:
> >
> >> I thought the reason the proposal got nowhere was because it's pretty
> >> simple to define it yourself:
> >>
> >> >>> class SliceHelper:
> >> ...     def __getitem__(self, slice):
> >> ...         return slice
> >> ...
> >> >>> SH = SliceHelper()
> >> >>> SH[1::3]
> >> slice(1, None, 3)
> >>
> >> Did I miss something significant about why this wasn't sufficient?
> >
> >
> > I think a SliceHelper class like this is a reasonable solution, but there
> > would be a lot of value having it a standard place somewhere in the
> > standard library (e.g., operator.subscript).
> >
> > Both pandas and NumPy include this helper object under different names
> > (pandas.IndexSlice and numpy.index_exp / numpy.s_), but it would be
> > surprising/unexpected for pandas/numpy specific helpers to show up when
> not
> > using one of those libraries. I do the exact same sorts of indexing
> > manipulations with xarray, dask and TensorFlow.
> >
> > Given that this is basically a simple feature to make it easier to work
> > with Python syntax (so there's no danger it will change in the future), I
> > think there is a lot to be said for putting it in the standard library in
> > one place so it's obvious what to use and users don't have to relearn
> that
> > name for this object and/or reimplement it.
>
> Please copy that comment into the ticket and ask for it to be reopened.
>
> https://bugs.python.org/issue24379
>
> Stefan
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/ceff5294/attachment.html>

From stefan_ml at behnel.de  Mon Jul 23 19:57:05 2018
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 24 Jul 2018 00:57:05 +0100
Subject: [Python-ideas] A better (simpler) approach to PEP 505
In-Reply-To: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
References: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
Message-ID: <pj5psd$tl9$1@blaine.gmane.org>

David Mertz schrieb am 23.07.2018 um 16:12:
> The need addressed by PEP 505 is real; it's also MUCH more niche and
> uncommon than something that would merit new syntax.  Moreover, the actual
> legitimate purpose served by the PEP 505 syntax is easily served by
> existing Python simply by using a wrapper class.

The discussion so far made it clear to me that

a) there is a use case for this feature, although I never needed it myself
b) throwing new syntax at it is not the right solution

Especially since there seem to be slightly diverging ideas about the exact
details in behaviour. Since this can be done in form of a library, people
should just drop it into a couple of competing libraries and let users
choose what they like better in their specific situation. And since we
already have a PEP now, let's continue to use it as a basis for discussion
about how these libraries should best behave in general and what mistakes
they should avoid.

Stefan


From shoyer at gmail.com  Mon Jul 23 20:00:06 2018
From: shoyer at gmail.com (Stephan Hoyer)
Date: Mon, 23 Jul 2018 17:00:06 -0700
Subject: [Python-ideas] slice[] to get more complex slices
In-Reply-To: <pj5nl6$p6b$1@blaine.gmane.org>
References: <CAFpSVpKnE8b=QrjxLy6anAF3ammq2QbB4_K1Ey02Vk3n8EVwyQ@mail.gmail.com>
 <pj2lgk$c6c$1@blaine.gmane.org>
 <09cb9784-6a97-49b9-b5e7-1c9a9d8c8391@googlegroups.com>
 <CALD=Yf-MNLW7z8s=ruc_C13t7eWp8KC6OWvqf9durZgpN=8=sw@mail.gmail.com>
 <399ff7d70c204f679bdb5a79f9bc0460@xmail103.UGent.be>
 <4243953379d74d2aa5393be2434a99a7@xmail103.UGent.be>
 <5B55AE6A.7010305@UGent.be>
 <CACac1F_XOi+1gmEMgZdbmW_3UXs661JS5BF=OvWLrF47H0fJmg@mail.gmail.com>
 <CAEQ_Tvf2Y-up_uWoUdutyoreHeM6=X52bZzZcath-r1btYeqow@mail.gmail.com>
 <pj5nl6$p6b$1@blaine.gmane.org>
Message-ID: <CAEQ_TvdtjzK1wsQfbQWHn8P5T_h7oQrFVH2wKoUyJ71Xr3aXAA@mail.gmail.com>

On Mon, Jul 23, 2018 at 4:19 PM Stefan Behnel <stefan_ml at behnel.de> wrote:

> Stephan Hoyer schrieb am 23.07.2018 um 18:01:
> > I think a SliceHelper class like this is a reasonable solution, but there
> > would be a lot of value having it a standard place somewhere in the
> > standard library (e.g., operator.subscript).
> >
> > Both pandas and NumPy include this helper object under different names
> > (pandas.IndexSlice and numpy.index_exp / numpy.s_), but it would be
> > surprising/unexpected for pandas/numpy specific helpers to show up when
> not
> > using one of those libraries. I do the exact same sorts of indexing
> > manipulations with xarray, dask and TensorFlow.
> >
> > Given that this is basically a simple feature to make it easier to work
> > with Python syntax (so there's no danger it will change in the future), I
> > think there is a lot to be said for putting it in the standard library in
> > one place so it's obvious what to use and users don't have to relearn
> that
> > name for this object and/or reimplement it.
>
> Please copy that comment into the ticket and ask for it to be reopened.
>
> https://bugs.python.org/issue24379
>
> Stefan
>

I basically did exactly that last week! See
https://bugs.python.org/issue24379#msg321966

I was told, "You may get more traction on python-ideas" :)

Cheers,
Stephan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/0dbcdc56/attachment-0001.html>

From python at mrabarnett.plus.com  Mon Jul 23 20:21:27 2018
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 24 Jul 2018 01:21:27 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
References: <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
Message-ID: <4899548e-b068-5ce5-b917-f1588da565c6@mrabarnett.plus.com>

On 2018-07-23 23:05, Giampaolo Rodola' wrote:
> On Mon, Jul 23, 2018 at 6:53 PM Steven D'Aprano <steve at pearwood.info> wrote:
>>
>> On Mon, Jul 23, 2018 at 02:04:17PM +0200, Giampaolo Rodola' wrote:
>> > "a?.b" does two and that's a fundamental difference (explicitness).
>>
>> How is "two things" less explicit than "one thing"?
>> Comments like the above is why I think that "explicit" and "implicit"
>> are used to mean "I like it" and "I don't like it" rather than being
>> objective arguments, or indeed having anything to do with explicitness
>> or implicitness.
> 
> This:
> 
>      v = a?.b
> 
> ...*implicitly* checks if value is not None [and continues execution].

It's no more implicit than 'or' checking for falseness.

> This:
> 
>      v = a
>      if a.b is not None:
>          v = a.b
> 
> ...*explicitly* checks if value is not None and continues execution.
> If for some reason '?'[ is also going to swallow LookupError then
> *that* would further decrease explicitness, because LookupError would
> be nowhere in sight, the same way "if", "is", "not", "None", ":", "new
> line" are nowhere in sight in the 'a?.b' example. Some argued "A ?? B"
> is less explicit than "A if A is not None else B" for the same reason.
> One may argue that silently returning None instead of raising
> AttributeError is also less explicit.
> This - and this only - is my argument about explicitness. It doesn't
> have to do with how many things are hidden behind an import statement
> or what happens on sorted() (that's comparing apples and oranges).
> I hope it's clear now.
> 
>> > It
>> > does so by introducing a brand new operator ("?") which can be spelled
>> > in two forms ("a?.b" and "a?[b]") by using two adjacent symbols not
>> > interrupted by any space, which is an absolute first in the Python
>> > syntax
>>
>> It isn't a first. Many existing operators use two adjacent symbols not
>> interrupted by a space:
>>
>> e.g.  ==  <=  >=  !=  **  //  << >> +=  -=  *= etc.
> 
> You say 'a == b'. You can't say 'a ?. b' (not that it matters, it
> would be less intuitive anyway). You can't because '.?' is the only
> couple of contiguous symbols requiring "something" before and after
> with no spaces in between, and that's a first in the language.

You _can_ say 'a ?. b', just as you _can_ say 'a . b'.

Also, it's not a couple of contiguous symbols, it's a single symbol, 
just as '<=' is a single symbol (and you can't put a space in the middle 
of that either).
[snip]

From steve at pearwood.info  Mon Jul 23 20:38:11 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 24 Jul 2018 10:38:11 +1000
Subject: [Python-ideas] A better (simpler) approach to PEP 505
In-Reply-To: <76b7e1c1-73bb-4f15-9e43-9b9329d51f07@googlegroups.com>
References: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
 <20180723172744.GD8744@ando.pearwood.info>
 <76b7e1c1-73bb-4f15-9e43-9b9329d51f07@googlegroups.com>
Message-ID: <20180724003811.GE8744@ando.pearwood.info>

On Mon, Jul 23, 2018 at 10:53:11AM -0700, Gr?gory Lielens wrote:

> The proto here swallow and short circuit on attribute error. Changing 
> to do it on Noneness is easy, and you can choose between the two 
> behavior: it's a strength compared to the operator approach.

Being able to choose between one desirable behaviour and one 
undesirable, rejected behaviour is *not* a strength.

PEP 505 has a section explaining why catching AttributeError 
is undesirable. I find the reasons it gives are compelling.

Can you explain why you reject the PEP's arguments against catching 
AttributeError?



-- 
Steve

From steve at pearwood.info  Mon Jul 23 21:08:37 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 24 Jul 2018 11:08:37 +1000
Subject: [Python-ideas] A better (simpler) approach to PEP 505
In-Reply-To: <CAEbHw4bYdksOML6kU2p3_GSe4QkirKJMHRrVMCupa4kCxZhpSA@mail.gmail.com>
References: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
 <20180723172744.GD8744@ando.pearwood.info>
 <CAEbHw4bYdksOML6kU2p3_GSe4QkirKJMHRrVMCupa4kCxZhpSA@mail.gmail.com>
Message-ID: <20180724010836.GF8744@ando.pearwood.info>

On Mon, Jul 23, 2018 at 01:55:48PM -0400, David Mertz wrote:
> On Mon, Jul 23, 2018 at 1:28 PM Steven D'Aprano <steve at pearwood.info> wrote:
> 
> > There's the first bug right there. foo.bar.blim ought to raise
> > AttributeError, since there is no blim attribute defined on foo.bar.
> >
> 
> Note my observation that this is deliberately different semantics.  I want
> to solve the underlying need, not simply emulate the less useful semantics
> of PEP 505.

I know that "laziness and hubris" are virtues in a programmer, but the 
PEP authors describe use-cases where *testing for None* is precisely the 
semantics wanted. What are your use-cases for greedily swallowing 
AttributeError?

Before telling the PEP authors that they have misunderstood their own 
uses-cases and that their need isn't what they thought (coalescing None) 
but something radically different which they have already explicitly 
rejected (suppressing AttributeError), you better have some damn 
compelling use-cases.

"Laziness and hubris" are supposed to be virtues in programmers, but 
when you insist that people have their own use-cases wrong, and that the 
designers of C#, Dart and Swift made a serious mistake introducing this 
feature, you ought to back it up with more than just an assertion.


[...]
> It does look like PyMaybe does much of what I'm thinking of.  I didn't
> think my toy was the first or best implementation.

Are you aware that the PEP includes pymaybe as a rejected idea?


> > But we aren't entering such a world, at least not in PEP 505. Attribute
> > access can fail.
> >     spam.eggs = 42
> >     spam?.eggs?.upper
> > is still going to raise AttributeError, because eggs is not None, it is an
> > int, and ints don't have an attribute "upper".
> 
> True enough.  But it's extremely difficult to imagine a real-world case
> where those particular semantics are actually particularly useful.

You can't imagine why it would be useful to NOT suppress the exception 
from a coding error like trying to uppercase an int.



-- 
Steve

From chris.barker at noaa.gov  Mon Jul 23 21:32:10 2018
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Mon, 23 Jul 2018 18:32:10 -0700
Subject: [Python-ideas] Python docs page: In what ways is None special
In-Reply-To: <CACac1F9qgTdD3Bac9EmKc+Af8EQRFH=KfVWzpOied0Se7qc0EQ@mail.gmail.com>
References: <CALD=Yf_PRXZYiNdC9FGE5pd1cjrS8gyr8q5iPxzrMKXzX027YA@mail.gmail.com>
 <d7651538-1831-b7c7-f050-252826591ac2@python.org>
 <CACac1F9qgTdD3Bac9EmKc+Af8EQRFH=KfVWzpOied0Se7qc0EQ@mail.gmail.com>
Message-ID: <CALGmxEJYV6XwwSMOxfo+JPC6_U9Y7ic7Yzf0ceiAxGcnKCkGFw@mail.gmail.com>

>lot. Actually, the ?. and ?[
> operators seem like they'd be much more useful if I did more JSON
> processing -

This has been mentioned a lot in this discussion?

Maybe what we need is a smarter JSON processing package, rather than
new operators.

Granted, these operators would help the authors of suck a package(s),
but if the bulk of users didn?t need them, then no point in adding
them to the language.

-CHB

From chris.barker at noaa.gov  Mon Jul 23 21:43:29 2018
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Mon, 23 Jul 2018 18:43:29 -0700
Subject: [Python-ideas] Python docs page: In what ways is None special
In-Reply-To: <CALD=Yf_PRXZYiNdC9FGE5pd1cjrS8gyr8q5iPxzrMKXzX027YA@mail.gmail.com>
References: <CALD=Yf_PRXZYiNdC9FGE5pd1cjrS8gyr8q5iPxzrMKXzX027YA@mail.gmail.com>
Message-ID: <CALGmxE+3kWQRq547oP==GedQzOkk53O-r_SrszQNM+Okyke_-Q@mail.gmail.com>

I agree that some more docs on the specialness of None (and, to a
lessor extent, True and False).

A few comments:

> None is a keyword
> ==============
>>>> None = 0
> SyntaxError: can't assign to keyword

One of the implications of this is that ?None? will always be the
Singleton None object ? so you can (and should) use:

Something is None

To test for None.

> The Command Line Interpreter hides None
> =================================
>>>> None

That?s a good one to highlight!

>
> None is false in a boolean context
> ==========================
>>>> bool(None)
> False

Maybe this belongs more in a discussion of ?Falseyness?

> Procedures return None
> ==================
>>>> a = [3,1,2]
>>>> b = a.sort()
>>>> a, b
> ([1, 2, 3], None)

This is less about None than about the convention that mutating
methods return None. Maybe that discussion belongs elsewhere.

> Dictionary get returns None if not found
> ==============================
>>>> {}.get('dne') is None
> True

Belongs with dict docs really, and not really true ? dict.get()
returns the default value, which is None be default.

> None is default return value
> =====================
>>>> def fn(): pass
> ...
>>>> fn() # No response!
>>>> print(fn()) # Here's why.
> None

Yup.

> None is used as a sentinel default value
> ==============================
> Particularly useful when default value must be determined
> in body of function.
> ---
> def insort_right(a, x, lo=0, hi=None):
>    # ...
>    if hi is None:
>        hi = len(a)
> ---

This is also a convention ? and primarily applies to mutable defaults,
which you hardly ever want to assign directly.

So a good example of None being used as a sentinel, but nog really
anything special about None.

-CHB
> /

From mertz at gnosis.cx  Mon Jul 23 23:24:25 2018
From: mertz at gnosis.cx (David Mertz)
Date: Mon, 23 Jul 2018 23:24:25 -0400
Subject: [Python-ideas] A better PEP 505 (test package: coalescing)
Message-ID: <CAEbHw4akdWcOtsMB=ejUpLUQrfgNxF3DOnAMgarT=YSHKi_b0A@mail.gmail.com>

I proposed a very toy example of a coalescing class that would solve the
problem solved by brand new syntax in PEP 505.  In my 5 minute version,
there were several faults or limitations.  For one, the class I wrote was
misnamed, since it wasn't really about None-coalescing, but rather about
exception-free nested access.

I've written a more robust version at https://pypi.org/project/coalescing/.
I encourage issues or contributions at
https://github.com/DavidMertz/coalesce.  Mind you, this is still a
few-hour-of-effort version, not rigorously tested.

I want to thank Antoine Pitrou for proposing using Graham Dumpleton's
warpt.ObjectProxy (which is truly magic).  In almost no lines, I think I
have something perhaps even better than PyMaybe.  In particular, most of
the time, there is no need to do an explicit `.unbox()` call... but at the
suggestion of several people, I changed my first API to optionally pass a
default value other than None within that method, if desired.  I even made
laziness in getting that value the default (but switchable) behavior.

Dumpleton's magic means that boxed values can do pretty much anything the
raw values can, so can generally ignore the .unbox() call at the end.
Perhaps most magic of all, you can even ASSIGN into proxied values and that
gets written to the original value (not shown in the doctests below).  This
adds something powerful that PEP 505 is completely unable to do.

I believe that I have captured ALL the semantics of PEP 505 with no changes
to Python syntax... and in a way that reads FAR better than all those new
operators do.

Access messy nested data structures

GreedyAccess will keep marching down trees even if failure occurred earlier:

    >>> from coalesce import GreedyAccess, make_test
    >>> cfg = make_test()
    >>> GreedyAccess(cfg).user.profile.song
    <GreedyAccess proxy for 'Nightclubbing'>
    >>> GreedyAccess(cfg).user.profile.song + ' and spam'
    'Nightclubbing and spam'
    >>> GreedyAccess(cfg).user.profile.food
    <GreedyAccess proxy for None>
    >>> print(GreedyAccess(cfg).user.profile.food.unbox())
    None
    >>> GreedyAccess(cfg).user.profile.food.unbox('spam')
    'spam'
    >>> GreedyAccess(cfg).user.profile.food.unbox('spam') + ' and spam'
    'spam and spam'

NoneCoalesce only descends until a None is encountered.  Accessing
attributes
or keys of None will still fail:

    >>> from coalesce import NoneCoalesce
    >>> NoneCoalesce(cfg).user.profile.song
    <NoneCoalesce proxy for 'Nightclubbing'>
    >>> NoneCoalesce(cfg).user.profile.song.unbox()
    'Nightclubbing'
    >>> NoneCoalesce(cfg).user.profile.food
    Traceback (most recent call last):
        ...
    AttributeError: 'types.SimpleNamespace' object has no attribute 'food'
    >>> NoneCoalesce(cfg).user.profile
    <NoneCoalesce proxy for namespace(arms=2, song='Nightclubbing')>

    >>> val = None
    >>> print(NoneCoalesce(val).attr)
    None

We provide for returning values other than None if some other default is
more
useful for your use case (a zero-argument lambda function would often be
useful here):

    >>> def say_spam():
    ...     return "spam"
    ...
    >>> GreedyAccess(cfg).user.profile.food.unbox(say_spam)
    'spam'
    >>> GreedyAccess(cfg).user.profile.food.unbox(say_spam, lazy=False)
#doctest: +ELLIPSIS
    <function say_spam ...>


-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/d6e724d8/attachment.html>

From c at anthonyrisinger.com  Tue Jul 24 00:51:51 2018
From: c at anthonyrisinger.com (C Anthony Risinger)
Date: Mon, 23 Jul 2018 23:51:51 -0500
Subject: [Python-ideas] Python docs page: In what ways is None special
In-Reply-To: <CALGmxE+3kWQRq547oP==GedQzOkk53O-r_SrszQNM+Okyke_-Q@mail.gmail.com>
References: <CALD=Yf_PRXZYiNdC9FGE5pd1cjrS8gyr8q5iPxzrMKXzX027YA@mail.gmail.com>
 <CALGmxE+3kWQRq547oP==GedQzOkk53O-r_SrszQNM+Okyke_-Q@mail.gmail.com>
Message-ID: <CAGAVQTHWFJtNYYsPcmVZ74ncY1jk9Bmd0pLOT+gxKjBkVnd+Pw@mail.gmail.com>

On Jul 23, 2018 8:43 PM, "Chris Barker - NOAA Federal via Python-ideas" <
python-ideas at python.org> wrote:


> Procedures return None
> ==================
>>>> a = [3,1,2]
>>>> b = a.sort()
>>>> a, b
> ([1, 2, 3], None)

This is less about None than about the convention that mutating
methods return None. Maybe that discussion belongs elsewhere.


> None is default return value
> =====================
>>>> def fn(): pass
> ...
>>>> fn() # No response!
>>>> print(fn()) # Here's why.
> None

Yup.


I believe these two are related and an artifact of how code/function
objects always leave *something* on TOS/top-of-stack.

IIRC even module objects have a discarded "return value" (in CPython at
least). This internal, unseen, and very-much-special-syntax-worthy value,
is None other.

-- 

C Anthony
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180723/76bd2061/attachment.html>

From contact at brice.xyz  Tue Jul 24 03:26:59 2018
From: contact at brice.xyz (Brice Parent)
Date: Tue, 24 Jul 2018 09:26:59 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPTjJmqfHhSGmT4R-kovk-ufLpcwFzgaP=qeLgygEpywTFoLaQ@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <7805016a-dc57-39bc-be07-c3e3e26ee5d4@tjol.eu>
 <CAPTjJmqfHhSGmT4R-kovk-ufLpcwFzgaP=qeLgygEpywTFoLaQ@mail.gmail.com>
Message-ID: <3278b515-cd53-885d-ed25-2b1bcc312a94@brice.xyz>

Le 24/07/2018 ? 00:39, Chris Angelico a ?crit?:
> On Tue, Jul 24, 2018 at 8:22 AM, Thomas Jollans <tjol at tjol.eu> wrote:
...
> What about:
>
> 5 < x < 10
>
> Can you add parentheses to that to "make precedence and evaluation order clear"?
Correct me if I'm wrong, but to my knowledge, this is just a shorthand 
to `5 < x and x < 10`.

Making the precedence and evaluation order clear doesn't necessarily 
rely on adding parenthesis, it may rely on expanding the expression (you 
may add parenthesis after that, if you think it's needed).

From gregory.lielens at gmail.com  Tue Jul 24 03:28:23 2018
From: gregory.lielens at gmail.com (=?UTF-8?Q?Gr=C3=A9gory_Lielens?=)
Date: Tue, 24 Jul 2018 00:28:23 -0700 (PDT)
Subject: [Python-ideas] A better (simpler) approach to PEP 505
In-Reply-To: <20180724003811.GE8744@ando.pearwood.info>
References: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
 <20180723172744.GD8744@ando.pearwood.info>
 <76b7e1c1-73bb-4f15-9e43-9b9329d51f07@googlegroups.com>
 <20180724003811.GE8744@ando.pearwood.info>
Message-ID: <b74cd8aa-7671-43e9-b073-f7307fe3addb@googlegroups.com>



On Tuesday, July 24, 2018 at 2:39:19 AM UTC+2, Steven D'Aprano wrote:

> PEP 505 has a section explaining why catching AttributeError 
> is undesirable. I find the reasons it gives are compelling. 
>
> Can you explain why you reject the PEP's arguments against catching 
> AttributeError? 
>   
>

Yes, and people have done it multiple time already. I will try to make it 
as explicit as possible, with a terminology that some may find useful...
when u have a object hierarchy linked through attributes, which -seems- the 
use pattern of ?. , the hierarchy is either
- 1) fully regular (each attribute contains the same type of objects (same 
type=with the same attributes), then normal . access is what you want
- 2) regular-or-None (contain same type of objects, or None). That's what 
?. is for, and ?. is only for that.
- 3) its partially regular (does not contain the same type of objects, some 
objects have more attributes than other, e.g. partial objects).
- 4) its irregular (objects may be of completely different type (int, 
str,... and do not specailly share attributes).

?. is intended for 2).
2) and 3) can be dealed with swallowing AttributeError
4) can also be traversed swallowing AttributeError, but it's probably a 
very bad idea in almost all cases.

I've encountered all cases, from time to time, but none is really typical 
of most of my code. Which is more common? not sure, I would say 3). 
What is sure is that 2) do not stand as hugely more common than the others, 
so introducing a special syntax for 2) do not  please me, so +0... 
Now as I find ?. unpleasant to parse (personal opinion), and having grown 
to share the general operaror / line noise allergy mainstream in the python 
community, i am a firm -1 on this: too small a niche, do not bring much 
compared to current approach, and visual line noise.
firm -1 on ?? too, it's less polemic, less visually displeasing, but even 
less usefull I think: it does not gain that much compared to a more general 
ternary.

As the pymaybe/this thread cover 2) and 3), and do it without any change to 
python, I find it better. No change > lib change > backward compatible 
object change  - like dict becoming ordered, or None triggering 
NoneAttributeError(AttributeError) > syntax change (expecially a bunch of 
linenoisy operators).

Now I will stop for a while here, arguments have become circular, 
proponents and opponents have exposed their view, but mostly proponents 
have provided very few additional information for a while, especially about 
real use cases, despite a few requests. 
We still do not know what motivated the proposal..We had to guess (JSON). 
Even the PEP do not say, it provide a set of real world example from the 
standard library, which is admirable but different. And personaly, while am 
very thankful for providing those concrete examples, I find them 
unconvicing: the code before is usually fine and improved little by the new 
operators. Sometimes it's worse: only slightly shorter and less clear.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180724/0c528d15/attachment-0001.html>

From rosuav at gmail.com  Tue Jul 24 03:30:18 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 24 Jul 2018 17:30:18 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <3278b515-cd53-885d-ed25-2b1bcc312a94@brice.xyz>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <7805016a-dc57-39bc-be07-c3e3e26ee5d4@tjol.eu>
 <CAPTjJmqfHhSGmT4R-kovk-ufLpcwFzgaP=qeLgygEpywTFoLaQ@mail.gmail.com>
 <3278b515-cd53-885d-ed25-2b1bcc312a94@brice.xyz>
Message-ID: <CAPTjJmp0vyHQ2YV0v5f4jS2M2T9KPLRmRgaQdudp68+9Jznaww@mail.gmail.com>

On Tue, Jul 24, 2018 at 5:26 PM, Brice Parent <contact at brice.xyz> wrote:
> Le 24/07/2018 ? 00:39, Chris Angelico a ?crit :
>>
>> On Tue, Jul 24, 2018 at 8:22 AM, Thomas Jollans <tjol at tjol.eu> wrote:
>
> ...
>>
>> What about:
>>
>> 5 < x < 10
>>
>> Can you add parentheses to that to "make precedence and evaluation order
>> clear"?
>
> Correct me if I'm wrong, but to my knowledge, this is just a shorthand to `5
> < x and x < 10`.

Not quite; the chained form will evaluate 'x' only once. It's broadly
the same, but not identical.

ChrisA

From gregory.lielens at gmail.com  Tue Jul 24 03:38:06 2018
From: gregory.lielens at gmail.com (=?UTF-8?Q?Gr=C3=A9gory_Lielens?=)
Date: Tue, 24 Jul 2018 00:38:06 -0700 (PDT)
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <3278b515-cd53-885d-ed25-2b1bcc312a94@brice.xyz>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <7805016a-dc57-39bc-be07-c3e3e26ee5d4@tjol.eu>
 <CAPTjJmqfHhSGmT4R-kovk-ufLpcwFzgaP=qeLgygEpywTFoLaQ@mail.gmail.com>
 <3278b515-cd53-885d-ed25-2b1bcc312a94@brice.xyz>
Message-ID: <bc270bf0-ef81-4623-9513-39f3ebb93354@googlegroups.com>



On Tuesday, July 24, 2018 at 9:28:02 AM UTC+2, Brice Parent wrote:
>
> Le 24/07/2018 ? 00:39, Chris Angelico a ?crit :
> > On Tue, Jul 24, 2018 at 8:22 AM, Thomas Jollans <tj... at tjol.eu 
> <javascript:>> wrote:
> ...
> > What about:
> >
> > 5 < x < 10
> >
> > Can you add parentheses to that to "make precedence and evaluation order 
> clear"?
> Correct me if I'm wrong, but to my knowledge, this is just a shorthand 
> to `5 < x and x < 10`.


I learned something here:
>>>   -5<-2<-1<=-1>-3
True

I wonder how this works, especially as < and friends have magical __ 
methods... How is it expanded in tne AST?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180724/4db0fbe6/attachment.html>

From p.f.moore at gmail.com  Tue Jul 24 03:44:50 2018
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 24 Jul 2018 08:44:50 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <bc270bf0-ef81-4623-9513-39f3ebb93354@googlegroups.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <7805016a-dc57-39bc-be07-c3e3e26ee5d4@tjol.eu>
 <CAPTjJmqfHhSGmT4R-kovk-ufLpcwFzgaP=qeLgygEpywTFoLaQ@mail.gmail.com>
 <3278b515-cd53-885d-ed25-2b1bcc312a94@brice.xyz>
 <bc270bf0-ef81-4623-9513-39f3ebb93354@googlegroups.com>
Message-ID: <CACac1F9ayRQ9YwBhUV3mesVOf1fwzLBhVV3ApyTOjGpXEk5CKg@mail.gmail.com>

On 24 July 2018 at 08:38, Gr?gory Lielens <gregory.lielens at gmail.com> wrote:
>
>
> On Tuesday, July 24, 2018 at 9:28:02 AM UTC+2, Brice Parent wrote:
>>
>> Le 24/07/2018 ? 00:39, Chris Angelico a ?crit :
>> > On Tue, Jul 24, 2018 at 8:22 AM, Thomas Jollans <tj... at tjol.eu> wrote:
>> ...
>> > What about:
>> >
>> > 5 < x < 10
>> >
>> > Can you add parentheses to that to "make precedence and evaluation order
>> > clear"?
>> Correct me if I'm wrong, but to my knowledge, this is just a shorthand
>> to `5 < x and x < 10`.
>
>
> I learned something here:
>>>>   -5<-2<-1<=-1>-3
> True
>
> I wonder how this works, especially as < and friends have magical __
> methods... How is it expanded in tne AST?

>>> import ast
>>> m = ast.parse('-5<-2<-1<=-1>-3')
>>> ast.dump(m)
'Module(body=[Expr(value=Compare(left=UnaryOp(op=USub(),
operand=Num(n=5)), ops=[Lt(), Lt(), LtE(), Gt()],
comparators=[UnaryOp(op=USub(), operand=Num(n=2)), UnaryOp(op=USub(),
operand=Num(n=1)), UnaryOp(op=USub(), operand=Num(n=1)),
UnaryOp(op=USub(), operand=Num(n=3))]))])'

Looks like there's a Compare node that takes a list of operators - and
indeed the docs say (at
https://docs.python.org/3.7/library/ast.html#abstract-grammar)

    Compare(expr left, cmpop* ops, expr* comparators)

Paul

From g.rodola at gmail.com  Tue Jul 24 05:05:24 2018
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Tue, 24 Jul 2018 11:05:24 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <4899548e-b068-5ce5-b917-f1588da565c6@mrabarnett.plus.com>
References: <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <4899548e-b068-5ce5-b917-f1588da565c6@mrabarnett.plus.com>
Message-ID: <CAFYqXL--NjzKEz7HAEy=scAtwFVy63TcPv2cb2qJh7uh_4v5HQ@mail.gmail.com>

On Tue, Jul 24, 2018 at 2:22 AM MRAB <python at mrabarnett.plus.com> wrote:

> >> > It
> >> > does so by introducing a brand new operator ("?") which can be spelled
> >> > in two forms ("a?.b" and "a?[b]") by using two adjacent symbols not
> >> > interrupted by any space, which is an absolute first in the Python
> >> > syntax
> >>
> >> It isn't a first. Many existing operators use two adjacent symbols not
> >> interrupted by a space:
> >>
> >> e.g.  ==  <=  >=  !=  **  //  << >> +=  -=  *= etc.
> >
> > You say 'a == b'. You can't say 'a ?. b' (not that it matters, it
> > would be less intuitive anyway). You can't because '.?' is the only
> > couple of contiguous symbols requiring "something" before and after
> > with no spaces in between, and that's a first in the language.
>
> You _can_ say 'a ?. b', just as you _can_ say 'a . b'.

You're right. It's so uncommon I forgot this style was valid. Anyway,
as I said 'a ?. b' would be even worse the same way 'a . b' is worse
than 'a.b'. The recommended and broadly used spelling would be 'a?.b'.

-- 
Giampaolo - http://grodola.blogspot.com

From steve at pearwood.info  Tue Jul 24 05:49:18 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 24 Jul 2018 19:49:18 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
References: <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
Message-ID: <20180724094917.GH8744@ando.pearwood.info>

On Tue, Jul 24, 2018 at 12:05:14AM +0200, Giampaolo Rodola' wrote:

> This:
> 
>     v = a?.b
> 
> ...*implicitly* checks if value is not None [and continues execution]. 

Do you agree that:

    obj.attribute
    x + 1
    func(arg)

explicitly looks up an attribute on obj, explicitly adds 1 to x, and 
explicitly calls func with a single argument? I don't think that we have 
to write COBOL-esque code to be explicit:

    GET ATTRIBUTE "attribute" FROM obj
    ADD 1 TO x
    CALL FUNCTION func WITH ARGUMENT arg

I don't accept that the use of punctuation makes something implicit. But 
if you want to argue that anything with punctuation is "implicit", then 
okay, Python has lots of implicit punctuation.

By definition, ?. checks for None before doing the attribute lookup. 
That is completely explicit, regardless of how it is spelled:

    obj?.attribute

    NULL-AWARE GET ATTRIBUTE "attribute" FROM obj

    null_aware_getattr(obj, "attribute")

    getattr_if_not_none(obj, "attribute")

But what certainly *is* implicity is David Mertz' suggestion for a 
magical None-aware proxy:

    x.attribute

The only way to tell whether that was an ordinary attribute lookup or a 
none-aware lookup would be to carefully inspect x and find out whether 
it was an instance of the None-aware proxy class or not.


> This
> 
>     v = a
>     if a.b is not None:
>         v = a.b
> 
> ...*explicitly* checks if value is not None and continues execution.

If you are trying to match the behaviour of a?.b above, it is also 
completely buggy and doesn't do what is intended.

    # Equivalent of a?.b
    v = a
    if v is not None:
        v = v.b


> If for some reason '?'[ is also going to swallow LookupError 

What makes you think that ?[...] will swallow LookupError?

Please don't argue against misfeatures that the PEP doesn't propose. 
Nothing in PEP 505 swallows any exceptions. Swallowing exceptions is 
explicitly rejected, and swallowing LookupError isn't part of the 
proposal.


[...]
> One may argue that silently returning None instead of raising
> AttributeError is also less explicit.

And again, you are arguing against a misfeature which PEP 505 does not 
propose. The ?. operator will not suppress AttributeErrors.

    # Wrong! No! This is not what the PEP proposes!
    obj = 1.234
    assert obj?.hexx is None


[...]
> > It isn't a first. Many existing operators use two adjacent symbols not
> > interrupted by a space:
> >
> > e.g.  ==  <=  >=  !=  **  //  << >> +=  -=  *= etc.
> 
> You say 'a == b'. You can't say 'a ?. b' (not that it matters, it
> would be less intuitive anyway). You can't because '.?' is the only
> couple of contiguous symbols requiring "something" before and after
> with no spaces in between, and that's a first in the language.

Why do you think spaces aren't allowed? The PEP explicitly says that 
the new operators can be used wherever the regular operators can be 
used:

    "The maybe-dot and maybe-subscript operators are added as
    trailers for atoms, so that they may be used in all the same
    locations as the regular operators"

and explicitly shows the grammar changes required:

    trailer: ('(' [arglist] ')' |
              '[' subscriptlist ']' |
              '?[' subscriptlist ']' |
              '.' NAME |
              '?.' NAME)


That tells me that ?. will be legal anywhere . is legal, so if x . y is 
legal (and it is) so will x ?. y be legal.


[...]
> The difference is that 'a.b.c.d' will result in AttributeError as soon
> as something is None while 'a?.b?.c?.d' will return None instead.

Correct. Because sometimes you want an AttributeError, and sometimes you 
want None.

You are criticising the operator for doing what it is designed and 
intended to do. You might as well criticise getattr(obj, 'spam', None) 
for returning None.

If you want an AttributeError, then don't use ?. and use ordinary . 
instead.


> > Likewise the logical operators "or" and "and" are designed to
> > short-circuit. If ?? and friends are a mistake because they
> > short-circuit, why aren't "or" and "and" mistakes?
> > I'm not asking this as a rhetorical question. If you think there is a
> > reason why it is okay for or/and to short-circuit, but it is bad for ??
> > and friends to short-circuit, then please explain why they are
> > different. I will be very happy to listen to your arguments.
> 
> The argument about this is that '?.' short-circuits execution
> *silently*.

Other short-circuit operators also short-circuit execution silently. 
That's what they are designed to do.

    # This isn't what actually happens.
    py> x = 0
    py> result = x and 1/x
    __main__:1: UserWarning:
    Short-cut operation occurred, the right hand operand was not 
    evaluated!!! Do not panic, this is the expected behaviour!!!
    py> print(result)
    0


> Instead of AttributeError you get None. You may chain ?.
> in order to lazily traverse a long tree, 

Correct, that is what it is designed to do.


> inadvertently assign None to
> a variable, continue code execution and fail later rather than sooner:
> 
>      email = request?.context?.user?.email  # None
>      ...
>      sendmail(subject, body, email)
> 
> Some (Antoine) rightly argued this may even have security implications
> (replace 'email' with 'password').

I'll have to remember that. Whenever there's any proposal for a feature 
I don't like, just claim it "may even have security implications". Who 
needs evidence when we have Fear, Uncertainty, Doubt?



-- 
Steve

From g.rodola at gmail.com  Tue Jul 24 06:38:19 2018
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Tue, 24 Jul 2018 12:38:19 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180724094917.GH8744@ando.pearwood.info>
References: <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
Message-ID: <CAFYqXL-JQ-7bkH7YL-ZS74X5Rq12yp3-wHVb5mj52Znp9mkgnQ@mail.gmail.com>

On Tue, Jul 24, 2018 at 11:50 AM Steven D'Aprano <steve at pearwood.info> wrote:
>
> On Tue, Jul 24, 2018 at 12:05:14AM +0200, Giampaolo Rodola' wrote:
>
> > This:
> >
> >     v = a?.b
> >
> > ...*implicitly* checks if value is not None [and continues execution].
>
> Do you agree that:
>
>     obj.attribute
>     x + 1
>     func(arg)
>
> explicitly looks up an attribute on obj, explicitly adds 1 to x, and
> explicitly calls func with a single argument? I don't think that we have
> to write COBOL-esque code to be explicit:
>
>     GET ATTRIBUTE "attribute" FROM obj
>     ADD 1 TO x
>     CALL FUNCTION func WITH ARGUMENT arg
>
> I don't accept that the use of punctuation makes something implicit. But
> if you want to argue that anything with punctuation is "implicit", then
> okay, Python has lots of implicit punctuation.
> By definition, ?. checks for None before doing the attribute lookup.
> That is completely explicit, regardless of how it is spelled:
>
>     obj?.attribute

You totally missed my point about explicitness. Nevermind.

> > This
> >
> >     v = a
> >     if a.b is not None:
> >         v = a.b
> >
> > ...*explicitly* checks if value is not None and continues execution.
>
> If you are trying to match the behaviour of a?.b above, it is also
> completely buggy and doesn't do what is intended.
>
>     # Equivalent of a?.b
>     v = a
>     if v is not None:
>         v = v.b
>
>
> > If for some reason '?'[ is also going to swallow LookupError
>
> What makes you think that ?[...] will swallow LookupError?
>
> Please don't argue against misfeatures that the PEP doesn't propose.
> Nothing in PEP 505 swallows any exceptions. Swallowing exceptions is
> explicitly rejected, and swallowing LookupError isn't part of the
> proposal.

I know it's not in the PEP. I merely mentioned that as PEP author was
questioning that possibility in previous messages. It was an example
(for you) on how *that* would make things even less explicit.

> [...]
> > One may argue that silently returning None instead of raising
> > AttributeError is also less explicit.
>
> And again, you are arguing against a misfeature which PEP 505 does not
> propose. The ?. operator will not suppress AttributeErrors.
>
>     # Wrong! No! This is not what the PEP proposes!
>     obj = 1.234
>     assert obj?.hexx is None

That is not what I meant at all!
I seriously question whether you really don't understand or you're
just pretending.
What I meat in here was 'a?.b?.c?' returning None in case 'b' is None.

> [...]
> > > It isn't a first. Many existing operators use two adjacent symbols not
> > > interrupted by a space:
> > >
> > > e.g.  ==  <=  >=  !=  **  //  << >> +=  -=  *= etc.
> >
> > You say 'a == b'. You can't say 'a ?. b' (not that it matters, it
> > would be less intuitive anyway). You can't because '.?' is the only
> > couple of contiguous symbols requiring "something" before and after
> > with no spaces in between, and that's a first in the language.
>
> Why do you think spaces aren't allowed? The PEP explicitly says that
> the new operators can be used wherever the regular operators can be
> used.
> [...]
> That tells me that ?. will be legal anywhere . is legal, so if x . y is
> legal (and it is) so will x ?. y be legal.

Yes, I forgot 'a . b' was legal - my bad.

> [...]
> > The difference is that 'a.b.c.d' will result in AttributeError as soon
> > as something is None while 'a?.b?.c?.d' will return None instead.
>
> Correct. Because sometimes you want an AttributeError, and sometimes you
> want None.
>
> You are criticising the operator for doing what it is designed and
> intended to do. You might as well criticise getattr(obj, 'spam', None)
> for returning None.
>
> If you want an AttributeError, then don't use ?. and use ordinary .
> instead.

Again, you missed my point.

> > > Likewise the logical operators "or" and "and" are designed to
> > > short-circuit. If ?? and friends are a mistake because they
> > > short-circuit, why aren't "or" and "and" mistakes?
> > > I'm not asking this as a rhetorical question. If you think there is a
> > > reason why it is okay for or/and to short-circuit, but it is bad for ??
> > > and friends to short-circuit, then please explain why they are
> > > different. I will be very happy to listen to your arguments.
> >
> > The argument about this is that '?.' short-circuits execution
> > *silently*.
>
> Other short-circuit operators also short-circuit execution silently.
> That's what they are designed to do.
>
>     # This isn't what actually happens.
>     py> x = 0
>     py> result = x and 1/x
>     __main__:1: UserWarning:
>     Short-cut operation occurred, the right hand operand was not
>     evaluated!!! Do not panic, this is the expected behaviour!!!
>     py> print(result)
>     0
>
>
> > Instead of AttributeError you get None. You may chain ?.
> > in order to lazily traverse a long tree,
>
> Correct, that is what it is designed to do.
>
>
> > inadvertently assign None to
> > a variable, continue code execution and fail later rather than sooner:
> >
> >      email = request?.context?.user?.email  # None
> >      ...
> >      sendmail(subject, body, email)
> >
> > Some (Antoine) rightly argued this may even have security implications
> > (replace 'email' with 'password').
>
> I'll have to remember that. Whenever there's any proposal for a feature
> I don't like, just claim it "may even have security implications". Who
> needs evidence when we have Fear, Uncertainty, Doubt?

OK, I'm done replying to you. I just wish I did it earlier (my bad).

--
Giampaolo - http://grodola.blogspot.com

From mertz at gnosis.cx  Tue Jul 24 07:02:54 2018
From: mertz at gnosis.cx (David Mertz)
Date: Tue, 24 Jul 2018 07:02:54 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180724094917.GH8744@ando.pearwood.info>
References: <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
Message-ID: <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>

On Tue, Jul 24, 2018, 5:50 AM Steven D'Aprano <steve at pearwood.info> wrote:

> But what certainly *is* implicity is David Mertz' suggestion for a
> magical None-aware proxy:
>
>     x.attribute
>
> The only way to tell whether that was an ordinary attribute lookup or a
> none-aware lookup would be to carefully inspect x and find out whether it
> was an instance of the None-aware proxy class or not.
>

Every use I've suggested for the magic proxy is similar to:

  NullCoalesce(cfg).user.profile.food

Yes, the class is magic. That much more so in the library I published last
night that utilizes wrapt.ObjectProxy. But it's also pretty explicit in
that an actual *word* announces that funny stuff is going to happen on the
same line.

Of course the this could be abused with:

  cfg = NoneCoalesce(cfg)
  ... 1000 lines ...
  do_something(cfg)

But then, I could also write a property that actually started a computation
of the millionth digit of pi while launching a DDoS attack on python.org
when a user accessed 'x.attribute'.

NoneCoalesce or GreedyAccess are magic, but in their intended use, they are
as little magical as possible to deal with messy nested data.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180724/89c72a3d/attachment.html>

From mertz at gnosis.cx  Tue Jul 24 07:07:48 2018
From: mertz at gnosis.cx (David Mertz)
Date: Tue, 24 Jul 2018 07:07:48 -0400
Subject: [Python-ideas] Fwd:  A better (simpler) approach to PEP 505
In-Reply-To: <CAEbHw4YstVcs-=fxhJ9oSSeupmM6X9L1H7OJCgVkjbe5nm0Opg@mail.gmail.com>
References: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
 <20180723172744.GD8744@ando.pearwood.info>
 <76b7e1c1-73bb-4f15-9e43-9b9329d51f07@googlegroups.com>
 <20180724003811.GE8744@ando.pearwood.info>
 <b74cd8aa-7671-43e9-b073-f7307fe3addb@googlegroups.com>
 <CAEbHw4YstVcs-=fxhJ9oSSeupmM6X9L1H7OJCgVkjbe5nm0Opg@mail.gmail.com>
Message-ID: <CAEbHw4ZDVMFKXEuGb8-w1hfuo4OELo7T8GdqA89xZik5oBdBXg@mail.gmail.com>

I started a new thread where I show a small library I made yesterday that
ALSO matches 505 semantics, as well as more general AttributeError
swallowing. It has two classes with sightly different behavior. So we don't
need syntax for either semantics.

On Tue, Jul 24, 2018, 3:29 AM Gr?gory Lielens <gregory.lielens at gmail.com>
wrote:

>
>
> On Tuesday, July 24, 2018 at 2:39:19 AM UTC+2, Steven D'Aprano wrote:
>
>> PEP 505 has a section explaining why catching AttributeError
>> is undesirable. I find the reasons it gives are compelling.
>>
>> Can you explain why you reject the PEP's arguments against catching
>> AttributeError?
>>
>>
>
> Yes, and people have done it multiple time already. I will try to make it
> as explicit as possible, with a terminology that some may find useful...
> when u have a object hierarchy linked through attributes, which -seems-
> the use pattern of ?. , the hierarchy is either
> - 1) fully regular (each attribute contains the same type of objects (same
> type=with the same attributes), then normal . access is what you want
> - 2) regular-or-None (contain same type of objects, or None). That's what
> ?. is for, and ?. is only for that.
> - 3) its partially regular (does not contain the same type of objects,
> some objects have more attributes than other, e.g. partial objects).
> - 4) its irregular (objects may be of completely different type (int,
> str,... and do not specailly share attributes).
>
> ?. is intended for 2).
> 2) and 3) can be dealed with swallowing AttributeError
> 4) can also be traversed swallowing AttributeError, but it's probably a
> very bad idea in almost all cases.
>
> I've encountered all cases, from time to time, but none is really typical
> of most of my code. Which is more common? not sure, I would say 3).
> What is sure is that 2) do not stand as hugely more common than the
> others, so introducing a special syntax for 2) do not  please me, so +0...
> Now as I find ?. unpleasant to parse (personal opinion), and having grown
> to share the general operaror / line noise allergy mainstream in the python
> community, i am a firm -1 on this: too small a niche, do not bring much
> compared to current approach, and visual line noise.
> firm -1 on ?? too, it's less polemic, less visually displeasing, but even
> less usefull I think: it does not gain that much compared to a more general
> ternary.
>
> As the pymaybe/this thread cover 2) and 3), and do it without any change
> to python, I find it better. No change > lib change > backward compatible
> object change  - like dict becoming ordered, or None triggering
> NoneAttributeError(AttributeError) > syntax change (expecially a bunch of
> linenoisy operators).
>
> Now I will stop for a while here, arguments have become circular,
> proponents and opponents have exposed their view, but mostly proponents
> have provided very few additional information for a while, especially about
> real use cases, despite a few requests.
> We still do not know what motivated the proposal..We had to guess (JSON).
> Even the PEP do not say, it provide a set of real world example from the
> standard library, which is admirable but different. And personaly, while am
> very thankful for providing those concrete examples, I find them
> unconvicing: the code before is usually fine and improved little by the new
> operators. Sometimes it's worse: only slightly shorter and less clear.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180724/3cb309e0/attachment-0001.html>

From rhodri at kynesim.co.uk  Tue Jul 24 07:37:35 2018
From: rhodri at kynesim.co.uk (Rhodri James)
Date: Tue, 24 Jul 2018 12:37:35 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
References: <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
Message-ID: <a039f890-a40e-20af-c591-5df7d5cc4855@kynesim.co.uk>

On 24/07/18 12:02, David Mertz wrote:
> Every use I've suggested for the magic proxy is similar to:
> 
>    NullCoalesce(cfg).user.profile.food
> 
> Yes, the class is magic. That much more so in the library I published last
> night that utilizes wrapt.ObjectProxy. But it's also pretty explicit in
> that an actual*word*  announces that funny stuff is going to happen on the
> same line.

      Foo(cfg).user.profile.food

Is that explicit that funny stuff is going to happen on the same line? 
I wouldn't generally assume so, I'd just assume the coder created a 
throwaway object to get at an attribute.  You have to know that 
"NullCoalesce" does magic before it is at all explicit that funny stuff 
will happen.  Thinking about it, NullCoalesce() may be *less* explicit 
than ?. because at least that doesn't look like ordinary attribute 
reference.

I'm still of the opinion that both approaches are trying to solve a 
problem that's too niche to merit them, BTW.

-- 
Rhodri James *-* Kynesim Ltd

From gregory.lielens at gmail.com  Tue Jul 24 07:56:41 2018
From: gregory.lielens at gmail.com (=?UTF-8?Q?Gr=C3=A9gory_Lielens?=)
Date: Tue, 24 Jul 2018 04:56:41 -0700 (PDT)
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <a039f890-a40e-20af-c591-5df7d5cc4855@kynesim.co.uk>
References: <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <a039f890-a40e-20af-c591-5df7d5cc4855@kynesim.co.uk>
Message-ID: <0c8a90cf-db9d-411d-b76a-9b4573ce3ec4@googlegroups.com>



On Tuesday, July 24, 2018 at 1:38:42 PM UTC+2, Rhodri James wrote:
>
> -snip-
> I'm still of the opinion that both approaches are trying to solve a 
> problem that's too niche to merit them, BTW. 


That's also my impression. Hence the second approach: it does not require 
any change to python, it's just a tool for that niche, so imho it's the 
right approach. Such a wrapper can even be provided by the lib that 
produced such nested attributes in the first place. The operator are a tool 
which seems designed to the same niche issue, but is exposed as a core 
language feature. It may be interesting if it provides a lot of side 
benefits, so the tool is so braodly useful it outgrowned it's niche origin. 
At this point, I do not think it does.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180724/4d595867/attachment.html>

From rhodri at kynesim.co.uk  Tue Jul 24 08:00:19 2018
From: rhodri at kynesim.co.uk (Rhodri James)
Date: Tue, 24 Jul 2018 13:00:19 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <0c8a90cf-db9d-411d-b76a-9b4573ce3ec4@googlegroups.com>
References: <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <a039f890-a40e-20af-c591-5df7d5cc4855@kynesim.co.uk>
 <0c8a90cf-db9d-411d-b76a-9b4573ce3ec4@googlegroups.com>
Message-ID: <22b52674-2e6b-9ff8-867f-e3a976cab7c3@kynesim.co.uk>

On 24/07/18 12:56, Gr?gory Lielens wrote:
> 
> On Tuesday, July 24, 2018 at 1:38:42 PM UTC+2, Rhodri James wrote:
>> -snip-
>> I'm still of the opinion that both approaches are trying to solve a
>> problem that's too niche to merit them, BTW.
> 
> That's also my impression. Hence the second approach: it does not require
> any change to python, it's just a tool for that niche, so imho it's the
> right approach.

OK, I'm confused.  Either you agree with me, and think the second 
approach is also wrong, or you think the second approach is right and 
disagree with me.  Which is it?

-- 
Rhodri James *-* Kynesim Ltd

From Richard at Damon-family.org  Tue Jul 24 08:07:36 2018
From: Richard at Damon-family.org (Richard Damon)
Date: Tue, 24 Jul 2018 08:07:36 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <a039f890-a40e-20af-c591-5df7d5cc4855@kynesim.co.uk>
References: <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <a039f890-a40e-20af-c591-5df7d5cc4855@kynesim.co.uk>
Message-ID: <FAFE9409-E86A-483E-A83A-BC334EBBCCAF@Damon-family.org>



> On Jul 24, 2018, at 7:37 AM, Rhodri James <rhodri at kynesim.co.uk> wrote:
> 
>> On 24/07/18 12:02, David Mertz wrote:
>> Every use I've suggested for the magic proxy is similar to:
>>   NullCoalesce(cfg).user.profile.food
>> Yes, the class is magic. That much more so in the library I published last
>> night that utilizes wrapt.ObjectProxy. But it's also pretty explicit in
>> that an actual*word*  announces that funny stuff is going to happen on the
>> same line.
> 
>     Foo(cfg).user.profile.food
> 
> Is that explicit that funny stuff is going to happen on the same line? I wouldn't generally assume so, I'd just assume the coder created a throwaway object to get at an attribute.  You have to know that "NullCoalesce" does magic before it is at all explicit that funny stuff will happen.  Thinking about it, NullCoalesce() may be *less* explicit than ?. because at least that doesn't look like ordinary attribute reference.
> 
> I'm still of the opinion that both approaches are trying to solve a problem that's too niche to merit them, BTW.
> 
> -- 
> Rhodri James *-* Kynesim Ltd

The fact that you changed NullCoalesce into Foo to show lack of explicitness seems a straw-man. Words are FULL of meaning, while symbols are less so. The biggest issue I see with the use of ? here is that ? does have some meaning, it says we are going to be (or have) asked a question, it doesn?t tell us what the question is. Most of the other symbols used have a long history of meaning (yes = has the problem that historically it has had two possible meanings). To me, ? gives no indication that it is going to ask about Nullness.

?. has some indication that we are doing an attribute access that is in some way conditional, but a?.b could mean that we are conditional on a not being null, or it could be asking to suppress any and all error in getting b, even if a is an int and thus doesn?t have a b. The words carry a lot more meaning.

From gregory.lielens at gmail.com  Tue Jul 24 08:11:00 2018
From: gregory.lielens at gmail.com (=?UTF-8?Q?Gr=C3=A9gory_Lielens?=)
Date: Tue, 24 Jul 2018 05:11:00 -0700 (PDT)
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <22b52674-2e6b-9ff8-867f-e3a976cab7c3@kynesim.co.uk>
References: <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <a039f890-a40e-20af-c591-5df7d5cc4855@kynesim.co.uk>
 <0c8a90cf-db9d-411d-b76a-9b4573ce3ec4@googlegroups.com>
 <22b52674-2e6b-9ff8-867f-e3a976cab7c3@kynesim.co.uk>
Message-ID: <158c3842-5992-4238-aa81-86ae788e01b6@googlegroups.com>

Both approaches should not be exposed as core language, but as facilitating tools to be used if, in your application, you have to traverse deep semi-regular attributes hierarchies.
The operator approach, by definition, is part of the core, so -1
The wrapper does not need to be included in python, so the discussion is meaningless: people define the tools they want and use them as they please.
Personally, I think it would be nice to have that in the standard distrib, especially wrapt or something as powerful. But I don't really care in fact, what good is that now I know about it so I can use it...

From klahnakoski at mozilla.com  Tue Jul 24 08:26:09 2018
From: klahnakoski at mozilla.com (Kyle Lahnakoski)
Date: Tue, 24 Jul 2018 08:26:09 -0400
Subject: [Python-ideas] Idea: Deferred Default Arguments?
In-Reply-To: <CACCbJKrUq_vUYKO8ZzF0PCU8Ss0m_XG8MG3bLKkZjH3OucsV5Q@mail.gmail.com>
References: <CACCbJKrUq_vUYKO8ZzF0PCU8Ss0m_XG8MG3bLKkZjH3OucsV5Q@mail.gmail.com>
Message-ID: <4c19c673-930f-b3b6-58cf-eb212b77c1e6@mozilla.com>


I agree this is a problem, which I have seen solved by removing the
method signature, which is unfortunate:

> def flexible_method(**kwargs):
> ? ? # Read the code to find out the expected parameters ??

I have an @override decorator to handle this type of pattern. It will
perform the null-coalescing with properties found in a special "kwargs"
parameter. "kwargs" is assigned a dict that has a copy of the method
arguments. The value of a callee's argument is, in order,

* a not None value provided by the caller or
* a not None value found in the kwargs dict or
* the default value provided by the method declaration or
* None

I was not clear on where you wanted to define your defaults.? Either
like this:

>???? @override
> ? ? def subfunction_1(a=None, b=None, c=None, kwargs=None):
> ? ? ? ? return a+b*c
>
>???? @override
> ? ? def subfunction_2(d=None, e=None, f=None, kwargs=None):
> ? ? ? ? return d*e+f
>
>???? @orverride
> ? ? def main_function(a=2, b=3, c=4, d=5, e=6, f=7, kwargs=None):
> ? ? ? ? return subfunction_1(a, b, c) + subfunction_2(d, e, f)
> ? ? ? ? return subfunction_1(kwargs) + subfunction_2(kwargs)? # IF YOU
WANT TO BE LAZY

or like this:

>???? @override
> ? ? def subfunction_1(a=2, b=3, c=4, kwargs=None):
> ? ? ? ? return a+b*c
>
>???? @override
> ? ? def subfunction_2(d=5, e=6, f=7, kwargs=None):
> ? ? ? ? return d*e+f
>
>???? @orverride
> ? ? def main_function(a=None, b=None, c=None, d=None, e=None, f=None,
kwargs=None):
> ? ? ? ? return subfunction_1(a, b, c) + subfunction_2(d, e, f)
> ? ? ? ? return subfunction_1(kwargs) + subfunction_2(kwargs)? # IF YOU
WANT TO BE LAZY

both are identical except for where you declare the default values.


https://github.com/klahnakoski/mo-kwargs


From g.rodola at gmail.com  Tue Jul 24 08:33:34 2018
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Tue, 24 Jul 2018 14:33:34 +0200
Subject: [Python-ideas] A better (simpler) approach to PEP 505
In-Reply-To: <pj5psd$tl9$1@blaine.gmane.org>
References: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
 <pj5psd$tl9$1@blaine.gmane.org>
Message-ID: <CAFYqXL80ZhEh8aO-PThetxeyAdMuMDV8ru_mh8thUv6RcPMRng@mail.gmail.com>

On Tue, Jul 24, 2018 at 1:57 AM Stefan Behnel <stefan_ml at behnel.de> wrote:
>
> David Mertz schrieb am 23.07.2018 um 16:12:
> > The need addressed by PEP 505 is real; it's also MUCH more niche and
> > uncommon than something that would merit new syntax.  Moreover, the actual
> > legitimate purpose served by the PEP 505 syntax is easily served by
> > existing Python simply by using a wrapper class.
>
> The discussion so far made it clear to me that
>
> a) there is a use case for this feature, although I never needed it myself
> b) throwing new syntax at it is not the right solution
>
> Especially since there seem to be slightly diverging ideas about the exact
> details in behaviour. Since this can be done in form of a library, people
> should just drop it into a couple of competing libraries and let users
> choose what they like better in their specific situation. And since we
> already have a PEP now, let's continue to use it as a basis for discussion
> about how these libraries should best behave in general and what mistakes
> they should avoid.
>
> Stefan

+1
There is still no proof that such a programming pattern would be
widely used or is desirable in practice.
A library on PYPI could help clarifying that (and the PEP should mention it).

-- 
Giampaolo - http://grodola.blogspot.com

From rhodri at kynesim.co.uk  Tue Jul 24 08:59:29 2018
From: rhodri at kynesim.co.uk (Rhodri James)
Date: Tue, 24 Jul 2018 13:59:29 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <FAFE9409-E86A-483E-A83A-BC334EBBCCAF@Damon-family.org>
References: <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <a039f890-a40e-20af-c591-5df7d5cc4855@kynesim.co.uk>
 <FAFE9409-E86A-483E-A83A-BC334EBBCCAF@Damon-family.org>
Message-ID: <2f24c991-28e9-69ef-c6f0-98c38be98295@kynesim.co.uk>

On 24/07/18 13:07, Richard Damon wrote:
> The fact that you changed NullCoalesce into Foo to show lack of explicitness seems a straw-man. Words are FULL of meaning, while symbols are less so. The biggest issue I see with the use of ? here is that ? does have some meaning, it says we are going to be (or have) asked a question, it doesn?t tell us what the question is. Most of the other symbols used have a long history of meaning (yes = has the problem that historically it has had two possible meanings). To me, ? gives no indication that it is going to ask about Nullness.

Oh, I don't disagree with you about ? not giving any much indication 
that it's about Nullness except for the (relatively short) history of 
using it to mean exactly that in C# etc.  However I don't think that a 
class of whatever name doing something magic is any better.

-- 
Rhodri James *-* Kynesim Ltd

From mertz at gnosis.cx  Tue Jul 24 09:02:27 2018
From: mertz at gnosis.cx (David Mertz)
Date: Tue, 24 Jul 2018 09:02:27 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <a039f890-a40e-20af-c591-5df7d5cc4855@kynesim.co.uk>
References: <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <a039f890-a40e-20af-c591-5df7d5cc4855@kynesim.co.uk>
Message-ID: <CAEbHw4YZ8H461RoBw6CC2gtMKfQohaYsTZF-4tyw_9Vn_aoRRw@mail.gmail.com>

On Tue, Jul 24, 2018, 7:38 AM Rhodri James <rhodri at kynesim.co.uk> wrote:

> On 24/07/18 12:02, David Mertz wrote:
> > Every use I've suggested for the magic proxy is similar to:
> >
> >    NullCoalesce(cfg).user.profile.food
> >
> > Yes, the class is magic. That much more so in the library I published
> last
> > night that utilizes wrapt.ObjectProxy. But it's also pretty explicit in
> > that an actual*word*  announces that funny stuff is going to happen on
> the
> > same line.
>
>       Foo(cfg).user.profile.food
>
> Is that explicit that funny stuff is going to happen on the same line? I
> wouldn't generally assume so, I'd just assume the coder created a throwaway
> object to get at an attribute.


Foo isn't a very indicative name. NoneCoalesce (or NullCoalesce, or
GreedyAccess) is. But if I had any doubt, I could read the docstring for
Foo to find out how magical it was, and in what way. I definitely know
*something* is happening by creating that new instance in the line.

I'm still of the opinion that both approaches are trying to solve a
> problem that's too niche to merit them, BTW.
>

That doesn't make sense to me. You think my little library shouldn't be
allowed on PyPI? I don't force the couple classes on anyone, but if they
happen to help someone (or PyMaybe, or some other library, does) they don't
change anything about Python itself. Syntax is a very different matter.

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180724/f6fe0893/attachment.html>

From rhodri at kynesim.co.uk  Tue Jul 24 09:08:06 2018
From: rhodri at kynesim.co.uk (Rhodri James)
Date: Tue, 24 Jul 2018 14:08:06 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4YZ8H461RoBw6CC2gtMKfQohaYsTZF-4tyw_9Vn_aoRRw@mail.gmail.com>
References: <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <a039f890-a40e-20af-c591-5df7d5cc4855@kynesim.co.uk>
 <CAEbHw4YZ8H461RoBw6CC2gtMKfQohaYsTZF-4tyw_9Vn_aoRRw@mail.gmail.com>
Message-ID: <6b05f6e8-c431-70de-b2b5-5e8bd322dc0b@kynesim.co.uk>

On 24/07/18 14:02, David Mertz wrote:
> On Tue, Jul 24, 2018, 7:38 AM Rhodri James <rhodri at kynesim.co.uk> wrote:
>> I'm still of the opinion that both approaches are trying to solve a
>> problem that's too niche to merit them, BTW.
>>
> 
> That doesn't make sense to me. You think my little library shouldn't be
> allowed on PyPI? I don't force the couple classes on anyone, but if they
> happen to help someone (or PyMaybe, or some other library, does) they don't
> change anything about Python itself. Syntax is a very different matter.

I have no objection to anyone putting anything on PyPI.  Putting it (or 
an equivalent) in the standard library is much more problematical, and 
you were talking about that.  I think your little library is a much 
richer source of bugs than you think it is, and dealing with messy data 
access isn't a good enough reason to put that much temptation in front 
of naive users.

-- 
Rhodri James *-* Kynesim Ltd

From rosuav at gmail.com  Tue Jul 24 09:08:10 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 24 Jul 2018 23:08:10 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4YZ8H461RoBw6CC2gtMKfQohaYsTZF-4tyw_9Vn_aoRRw@mail.gmail.com>
References: <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <a039f890-a40e-20af-c591-5df7d5cc4855@kynesim.co.uk>
 <CAEbHw4YZ8H461RoBw6CC2gtMKfQohaYsTZF-4tyw_9Vn_aoRRw@mail.gmail.com>
Message-ID: <CAPTjJmoWNV9yz5hZq9tCHL3ojMtaZg9uQGRv3RV9Cq+skciNcw@mail.gmail.com>

On Tue, Jul 24, 2018 at 11:02 PM, David Mertz <mertz at gnosis.cx> wrote:
> On Tue, Jul 24, 2018, 7:38 AM Rhodri James <rhodri at kynesim.co.uk> wrote:
>>
>> On 24/07/18 12:02, David Mertz wrote:
>> > Every use I've suggested for the magic proxy is similar to:
>> >
>> >    NullCoalesce(cfg).user.profile.food
>> >
>> > Yes, the class is magic. That much more so in the library I published
>> > last
>> > night that utilizes wrapt.ObjectProxy. But it's also pretty explicit in
>> > that an actual*word*  announces that funny stuff is going to happen on
>> > the
>> > same line.
>>
>>       Foo(cfg).user.profile.food
>>
>> Is that explicit that funny stuff is going to happen on the same line? I
>> wouldn't generally assume so, I'd just assume the coder created a throwaway
>> object to get at an attribute.
>
>
> Foo isn't a very indicative name. NoneCoalesce (or NullCoalesce, or
> GreedyAccess) is. But if I had any doubt, I could read the docstring for Foo
> to find out how magical it was, and in what way. I definitely know
> *something* is happening by creating that new instance in the line.
>

Okay. Check this out, then:

>>> x = Foo(cfg).user.profile
>>> x.food

Remember, these lines could be a very long way apart. You could pass
'x' to a function unrelated to the line of code that created it. And
'x.food' still has to have the magic. You can't mandate that the call
to Coalesce be on the same line as the attribute access - Python
doesn't work that way.

So your perfectly ordinary dot operator now does magic in addition to
normal attribute access. See why it's a dangerous thing?

ChrisA

From mertz at gnosis.cx  Tue Jul 24 09:20:58 2018
From: mertz at gnosis.cx (David Mertz)
Date: Tue, 24 Jul 2018 09:20:58 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <6b05f6e8-c431-70de-b2b5-5e8bd322dc0b@kynesim.co.uk>
References: <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <a039f890-a40e-20af-c591-5df7d5cc4855@kynesim.co.uk>
 <CAEbHw4YZ8H461RoBw6CC2gtMKfQohaYsTZF-4tyw_9Vn_aoRRw@mail.gmail.com>
 <6b05f6e8-c431-70de-b2b5-5e8bd322dc0b@kynesim.co.uk>
Message-ID: <CAEbHw4b5kw48VCMw2jgru2m-w_oV6M1oZWckvzC3+z+9_rTryg@mail.gmail.com>

I *definitely* don't think a little tool I wrote in a couple hours last
night belongs in the standard library (with most of the heavy lifting
actually done by wrapt?which is really well designed, and is also not in
the standard library). I also don't think PyMaybe belongs there, even
though it's a couple years old.

Now, perhaps, if 'coalescing' is widely used for a year or two. And bugs
are fixed. And the API is tweaked based on experience with it's use. And so
on... At that point, *maybe* something derived from it might be appropriate.

On Tue, Jul 24, 2018, 9:08 AM Rhodri James <rhodri at kynesim.co.uk> wrote:

> On 24/07/18 14:02, David Mertz wrote:
> > On Tue, Jul 24, 2018, 7:38 AM Rhodri James <rhodri at kynesim.co.uk> wrote:
> >> I'm still of the opinion that both approaches are trying to solve a
> >> problem that's too niche to merit them, BTW.
> >>
> >
> > That doesn't make sense to me. You think my little library shouldn't be
> > allowed on PyPI? I don't force the couple classes on anyone, but if they
> > happen to help someone (or PyMaybe, or some other library, does) they
> don't
> > change anything about Python itself. Syntax is a very different matter.
>
> I have no objection to anyone putting anything on PyPI.  Putting it (or
> an equivalent) in the standard library is much more problematical, and
> you were talking about that.  I think your little library is a much
> richer source of bugs than you think it is, and dealing with messy data
> access isn't a good enough reason to put that much temptation in front
> of naive users.
>
> --
> Rhodri James *-* Kynesim Ltd
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180724/36de665c/attachment.html>

From mertz at gnosis.cx  Tue Jul 24 09:29:51 2018
From: mertz at gnosis.cx (David Mertz)
Date: Tue, 24 Jul 2018 09:29:51 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPTjJmoWNV9yz5hZq9tCHL3ojMtaZg9uQGRv3RV9Cq+skciNcw@mail.gmail.com>
References: <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <a039f890-a40e-20af-c591-5df7d5cc4855@kynesim.co.uk>
 <CAEbHw4YZ8H461RoBw6CC2gtMKfQohaYsTZF-4tyw_9Vn_aoRRw@mail.gmail.com>
 <CAPTjJmoWNV9yz5hZq9tCHL3ojMtaZg9uQGRv3RV9Cq+skciNcw@mail.gmail.com>
Message-ID: <CAEbHw4a1aHCOf_ujY7RecOV+GsGytSZnwd2tgqAm_7OhEOeZ6w@mail.gmail.com>

On Tue, Jul 24, 2018, 9:09 AM Chris Angelico <rosuav at gmail.com> wrote:

> >>> x = Foo(cfg).user.profile
> >>> x.food
>
> Remember, these lines could be a very long way apart. You could pass
> 'x' to a function unrelated to the line of code that created it. And
> 'x.food' still has to have the magic. You can't mandate that the call
> to Coalesce be on the same line as the attribute access - Python
> doesn't work that way.
>
> So your perfectly ordinary dot operator now does magic in addition to
> normal attribute access. See why it's a dangerous thing?
>

Yes, of course. That's why I would recommend best practice is to unbox or
otherwise use the conditional value as close to the magic code as feasible.

Likewise, as I noted a little while ago, 'x.food' could equally well be a
property that executed arbitrarily slow, magical, obscure, or even
malicious operations. Equally, 'x + y' could do absolutely anything if we
define .__add__() or .__radd__() methods.

Everything in Python is magical in that sense, but we should deliberately
keep the magic constrained to the amount needed.

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180724/97585e1c/attachment.html>

From jfine2358 at gmail.com  Tue Jul 24 09:49:09 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Tue, 24 Jul 2018 14:49:09 +0100
Subject: [Python-ideas] Python docs page: In what ways is None special
In-Reply-To: <CAGAVQTHWFJtNYYsPcmVZ74ncY1jk9Bmd0pLOT+gxKjBkVnd+Pw@mail.gmail.com>
References: <CALD=Yf_PRXZYiNdC9FGE5pd1cjrS8gyr8q5iPxzrMKXzX027YA@mail.gmail.com>
 <CALGmxE+3kWQRq547oP==GedQzOkk53O-r_SrszQNM+Okyke_-Q@mail.gmail.com>
 <CAGAVQTHWFJtNYYsPcmVZ74ncY1jk9Bmd0pLOT+gxKjBkVnd+Pw@mail.gmail.com>
Message-ID: <CALD=Yf_d+xW7bqxU81TB+epzYXh6hLmB2jKPsViGRXYmc2VviA@mail.gmail.com>

Here's another way that None is special.
===
>>> NoneType = type(None)
>>>
>>> class myNoneType(NoneType): pass
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: type 'NoneType' is not an acceptable base type
===

For more information see
https://stackoverflow.com/questions/10061752/which-classes-cannot-be-subclassed

Related are
https://stackoverflow.com/questions/2172189/why-i-cant-extend-bool-in-python/2172204#2172204
https://stackoverflow.com/questions/2825364/final-classes-in-python-3-x-something-guido-isnt-telling-me

-- 
Jonathan

From gregory.lielens at gmail.com  Wed Jul 25 00:48:50 2018
From: gregory.lielens at gmail.com (=?UTF-8?Q?Gr=C3=A9gory_Lielens?=)
Date: Tue, 24 Jul 2018 21:48:50 -0700 (PDT)
Subject: [Python-ideas] A better (simpler) approach to PEP 505
In-Reply-To: <CAP7+vJJ_NDnviAwbt6s-7mGuPx9Aar=DBiJh-9DkTuRbBA7F5g@mail.gmail.com>
References: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
 <20180723172744.GD8744@ando.pearwood.info>
 <76b7e1c1-73bb-4f15-9e43-9b9329d51f07@googlegroups.com>
 <20180724003811.GE8744@ando.pearwood.info>
 <b74cd8aa-7671-43e9-b073-f7307fe3addb@googlegroups.com>
 <CAP7+vJJ_NDnviAwbt6s-7mGuPx9Aar=DBiJh-9DkTuRbBA7F5g@mail.gmail.com>
Message-ID: <1b3ef558-6717-4039-9b6e-99904961eaf5@googlegroups.com>

Interesting, looking into, let's say, 100 of those is [not] None, randomly chosen, should give an very good idea of typical use for your code base.

I am curious about why you don't like solutions involving exceptions?
Performance?
Catching too much, like typing errors?
Less easy to distinguish between None and attr missing?
Less easy to combine None-aware and classic attr lookup on the same expression?
Something else?

From gregory.lielens at gmail.com  Wed Jul 25 02:49:05 2018
From: gregory.lielens at gmail.com (=?UTF-8?Q?Gr=C3=A9gory_Lielens?=)
Date: Tue, 24 Jul 2018 23:49:05 -0700 (PDT)
Subject: [Python-ideas] A better (simpler) approach to PEP 505
In-Reply-To: <1b3ef558-6717-4039-9b6e-99904961eaf5@googlegroups.com>
References: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
 <20180723172744.GD8744@ando.pearwood.info>
 <76b7e1c1-73bb-4f15-9e43-9b9329d51f07@googlegroups.com>
 <20180724003811.GE8744@ando.pearwood.info>
 <b74cd8aa-7671-43e9-b073-f7307fe3addb@googlegroups.com>
 <CAP7+vJJ_NDnviAwbt6s-7mGuPx9Aar=DBiJh-9DkTuRbBA7F5g@mail.gmail.com>
 <1b3ef558-6717-4039-9b6e-99904961eaf5@googlegroups.com>
Message-ID: <aff8b61e-3446-4a97-ba8a-f2d856b5dc21@googlegroups.com>

BTW, I did (very quickly, so it's rough, my regexps are not really catching 
everything) the same or our code base:

"is None"+ "is not None": 5600
AttributeError: 160
3 args getattr: 60
hasattr: 1800

So very similar to your pattern....except for hasattr, which is much more 
common in my case...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180724/31018b10/attachment.html>

From contact at brice.xyz  Wed Jul 25 04:32:33 2018
From: contact at brice.xyz (Brice Parent)
Date: Wed, 25 Jul 2018 10:32:33 +0200
Subject: [Python-ideas] A better (simpler) approach to PEP 505
In-Reply-To: <aff8b61e-3446-4a97-ba8a-f2d856b5dc21@googlegroups.com>
References: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
 <20180723172744.GD8744@ando.pearwood.info>
 <76b7e1c1-73bb-4f15-9e43-9b9329d51f07@googlegroups.com>
 <20180724003811.GE8744@ando.pearwood.info>
 <b74cd8aa-7671-43e9-b073-f7307fe3addb@googlegroups.com>
 <CAP7+vJJ_NDnviAwbt6s-7mGuPx9Aar=DBiJh-9DkTuRbBA7F5g@mail.gmail.com>
 <1b3ef558-6717-4039-9b6e-99904961eaf5@googlegroups.com>
 <aff8b61e-3446-4a97-ba8a-f2d856b5dc21@googlegroups.com>
Message-ID: <0f314f4d-6d8f-2c58-f825-afc7458128de@brice.xyz>

Le 25/07/2018 ? 08:49, Gr?gory Lielens a ?crit?:
> BTW, I did (very quickly, so it's rough, my regexps are not really 
> catching everything) the same or our code base:
>
> "is None"+ "is not None": 5600
> AttributeError: 160
> 3 args getattr: 60
> hasattr: 1800
>
> So very similar to your pattern....except for hasattr, which is much 
> more common in my case...
(I'm not going to speak about how nice or ugly the syntax is, just about 
the use cases and statistics)

I think the use case here is not really the simple 'is? None' + 'is not 
None'.

if the case is just to replace one None value by something else, it's 
not really an improvement as we just save one short line, and once the 
new syntax is accepted and we're used to it, both solution are quite 
explicit with what they do, so I'd prefer the status quo over a second 
way of doing the same thing.

It gets interesting in json-like cases, when we traverse a deep tree in 
which we might encounter None at multiple levels. To me, the new 
functionality is better by than the status quo as it shortens the code 
drastically and makes it more readable (you may see the entire traversal 
at once).

So for the statistics, what's getting interesting is knowing when we are 
in that second case (but of course, it's harder to get them using simple 
regexes). Maybe we could find when there are more than one "is None" or 
"is not None" within the small block of code, and then manually check a 
few random dozens of those cases to see how many of them would be improved?
Another thing that would be interesting in those statistics, but 
probably even harder to get automatically, is the proportion of cases 
where there is a side effect involved (so the cases when we also need a 
temporary variable not to have the side effect executed twice). Those 
cases benefit more of the new syntax than from the old one, until 
there's a good solution with ':=' or a lazy evaluation syntax, or some 
function return caching from the caller side.
Also, it would be nice to know if those use cases come from specific 
libraries/functionalities (like json tree traversal or use of ORM) or 
from many different situations. In the first case, the improvement would 
probably belong inside the libraries themselves, or helpers could be 
created for this very purpose, and in the second case, if the problem is 
quite common, it gets interesting finding a solution inside Python 
itself, like what's described in this PEP.

I didn't automate those search in my codebase, I just searched for 'is 
None' and looked if it was complicated enough to justify a new syntax, 
and over 937 'is None', 43 were close to another 'is None', so I looked 
them up manually, and 13 could have benefited from the proposal, in the 
sense I described sooner, so more than 1 'is None' removed in a single 
line of code, and/or there was a side effect (I needed a temp variable 
not to have the side effect repeated). But I didn't check for side 
effects in the other 894 'is None', so this number should probably be a 
bit bigger.
Side note: we're not big JSON users, and when we do use some, we catch 
exceptions as it's never (here) normal not have well formed json 
strings, so the code never continues and we never have to use default 
values given by the backend.


From gregory.lielens at gmail.com  Wed Jul 25 05:02:01 2018
From: gregory.lielens at gmail.com (=?UTF-8?Q?Gr=C3=A9gory_Lielens?=)
Date: Wed, 25 Jul 2018 02:02:01 -0700 (PDT)
Subject: [Python-ideas] A better (simpler) approach to PEP 505
In-Reply-To: <0f314f4d-6d8f-2c58-f825-afc7458128de@brice.xyz>
References: <CAEbHw4ZxtXwpV4h0Dng_yApuKw_tWpZPW1jcuw8QJwHN9L1NLA@mail.gmail.com>
 <20180723172744.GD8744@ando.pearwood.info>
 <76b7e1c1-73bb-4f15-9e43-9b9329d51f07@googlegroups.com>
 <20180724003811.GE8744@ando.pearwood.info>
 <b74cd8aa-7671-43e9-b073-f7307fe3addb@googlegroups.com>
 <CAP7+vJJ_NDnviAwbt6s-7mGuPx9Aar=DBiJh-9DkTuRbBA7F5g@mail.gmail.com>
 <1b3ef558-6717-4039-9b6e-99904961eaf5@googlegroups.com>
 <aff8b61e-3446-4a97-ba8a-f2d856b5dc21@googlegroups.com>
 <0f314f4d-6d8f-2c58-f825-afc7458128de@brice.xyz>
Message-ID: <be3fa903-f823-409a-bccd-edcebf70d208@googlegroups.com>



On Wednesday, July 25, 2018 at 10:33:37 AM UTC+2, Brice Parent wrote:
>
> I think the use case here is not really the simple 'is  None' + 'is not 
> None'.
>

Sure, that's why I also proposed to manually check a non-too-small samples 
of the None-testing occurences found by Guido .
You did it on your sample, and your findings findings are what I roughly 
expected.
I didn't do it. Well, not true, I sort of did it ;-), but I was lazy so I 
did not look enough to be representative, it was only a quick look at a 
very few instances:

I did not encounter deep hierarchy descent (I think the few we have use 
hasattr), but I found a couple where the proposed syntax would help but 
just a little: it's short and would use single ?? or ??=
By far, the most common case was using None as a marker for "we need a 
default sensible in the context", most of the time for a default argument. 
?? and ??= indeed makes things slightly shorter and usually slightly 
clearer, but the improvement is very small as  things are short and clear 
already. That's also why I was interested in the default argument 
delegating: In quite a few cases, the None default was because the real 
default was in a subfunction, and a way to delegate would be much more 
useful there.

if the case is just to replace one None value by something else, it's 
> not really an improvement as we just save one short line, and once the 
> new syntax is accepted and we're used to it, both solution are quite 
> explicit with what they do, so I'd prefer the status quo over a second 
> way of doing the same thing.
>

Yep, ditto. That's why I am -1 on ?? and ??= : They are useful, but not 
enough imho

It gets interesting in json-like cases, when we traverse a deep tree in 
> which we might encounter None at multiple levels. 
>

I hoped the PEP writers would submit such reallife examples, I think that's 
the motivation behind the PEP. But either I missed it, or they didn't show 
it yet. 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180725/8558860f/attachment-0001.html>

From jfine2358 at gmail.com  Wed Jul 25 06:26:54 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Wed, 25 Jul 2018 11:26:54 +0100
Subject: [Python-ideas] Multi-core reference count garbage collection
In-Reply-To: <CALD=Yf9jbZoG6W0Qn-jDAs0iqruGocc0Fuu_yC=U6LNhrCk8Cg@mail.gmail.com>
References: <CALD=Yf-Ht4uDMHkvXp8i+xesw=e9zw4i+nAuRJFMVppNSYrboA@mail.gmail.com>
 <CALD=Yf9jbZoG6W0Qn-jDAs0iqruGocc0Fuu_yC=U6LNhrCk8Cg@mail.gmail.com>
Message-ID: <CALD=Yf9zFNJVih5yWUrOO5=XU=rvkMckjH8DgkdCNvLCeasqCw@mail.gmail.com>

Hi All

INTRODUCTION

This is the third and concluding post, where I describe a scheme for
multi-core reference counting garbage collection. The first two posts
are
https://mail.python.org/pipermail/python-ideas/2018-July/052054.html
https://mail.python.org/pipermail/python-ideas/2018-July/052151.html

This subject is quite technical, with pitfalls and a race hazard. If
you don't understand what I've written, or think it wrong, it might be
that I screwed up and got it wrong. I personally find it a hard
subject. Or it might be that you've missed something in one of the two
previous posts. Or we might both be wrong. Or both right!

For the race hazard, see
https://mail.python.org/pipermail/python-ideas/2018-July/052190.html

There have been many valuable contributions to this discussion thread,
which I will acknowledge later, in another post. However, I do repeat
that this scheme is based on other people's work (particularly talks
by Larry Hastings), as well as my own thinking.

And it is actually a known and published idea (which is reassuring).
https://mail.python.org/pipermail/python-ideas/2018-July/052061.html

THE STORY SO FAR

We have 5 worker processes and one garbage collection (GC) process.
Each worker process has an INCR buffer in which it logs an ID, every
time the process increases the refcount of the object with that ID.
And similarly for DECR.

When the worker INCR and DECR buffers are full, they are passed over
to the GC process. The GC process keeps, for each object ID, a running
total of how many references. In this way it consumes the information
in the INCR and DECR buffers.

The problem, as always in garbage collection, is to reclaim objects
that will no longer be used. With reference counting, objects whose
reference count is zero can be reclaimed. This is often done, in
single process systems, immediately after the reference count becomes
zero. (An important aside: For multi-core machines this is an
unhelpful constraint. Relaxing it will allow the worker processes to
run more freely.)

At each point in time, there are for each object ID two reference
counts. The first is the running total, updated as full INCR and DECR
buffers are applied to it. I will call this the RAW TOTAL. The second
is the raw total, adjusted by all the INCR and DECR buffers in the
worker processes. I will call this the REAL TOTAL.

Once the real total is zero, it stays zero. This is an immediate
consequence of the worker threads having no references to the object.
Having no references, they cannot do an INCR or DECR. When the real
total is zero, the object can be reclaimed.

The problem is to refine what we have, so that all objects whose real
count is zero are in good time reclaimed. We want to avoid locking all
the worker threads, for example to compute real counts for all IDs.

BEING TRICKY

Think of the worker threads as an adversary, who are trying to delay
or fool the garbage collector.

Here's one thing that can be done. Have a worker thread
1. Acquire (references to) some objects (say from other worker processes).
2. Delete those object references, without filling either the INCR or
DECR buffer.
3. Spin its wheels endlessly.

This will defeat the GC process, unless the system from time to time
sends the INCR and DECR buffers to the GC process, whether or not they
are full. So we'll add this to the system requirements.

Here's something else, which is part of normal use. We have one worker
thread do a calculation for another. The return value is passed from
one thread to another. It could happen that, for some ID, an INCR in
one thread is followed by a DECR in another. Now suppose that the GC
process gets
* First, the process buffer containing the DECR.
* And then the process buffer that contains the INCR.
(This is a race hazard. The order of operations has been reversed.)

Thus it can happen that the raw total for an ID can go down to zero
and then increase up to one. This can't happen for the true total, of
course. (With more work, the raw counts can go negative!)

DEFEATING TRICKINESS

The problem is for the GC process to discover IDs for which the real
total is zero. And in good time. Here's how we'll do it. Suppose an ID
has raw total zero. That makes it a candidate ID.

Now do a global INCR update. By this we mean that we have the GC
thread get INCR buffers from all the worker threads, and applies them
to the raw totals.

If the raw total for the candidate ID (drum roll) is still zero, then
the real total is also zero.

That's it. The rest is implementation.

Let's see why this is. First, note that we don't need to know exactly
when the real count became zero. We just need to know that it is now
zero. (And once zero, it stays zero.)

Now note that the real count can never be negative. Now suppose at
time T the raw count (for an ID) is zero, but that the real count is
one or more. That means that some worker process has a reference, and
so the ID appears in some INCR buffer.

Well, we just did a global INCR update, and found that the candidate
ID was still zero. So the ID wasn't in an INCR buffer. We can reclaim
the memory.

A MATHEMATICAL APPROACH

The previous argument was perhaps a computer science approach. Here's
a more mathematical approach.

We are using a special property of global INCR update. Suppose that at
time T the raw count is N. Now do a global INCR update, to get raw
count M. Let R be the real count at time T.

We have
* N <= M, because processing only INCR buffers.
* R <= M, because
** we skipped the DECR buffers
** and at most enlarged the INCR buffers

We always have 0 <= R, and after the global INCR update we have R <=
M. That was at our time T, when we started the global INCR update. But
once zero, R stays zero.

So M == 0 is our condition for garbage collection, when M is the value
after a global INCR update.

WHERE NOW

For me, this thread has been an exploration. I'm now convinced,
although you may not be, that buffered multi-core reference counting
garbage collections is something that can be done. And that it will
allow the worker processes to run freely, at least when it comes to
garbage collection.

This discussion thread contributes nothing to important problem of
multiple processes mutating (writing to) the same object. (Except that
buffering provides a solution in the special case where order doesn't
matter, such 1 + 2 + 3 = 2 + 3 + 1.)

Well done for getting to the end of this long and difficult material.
I've written it with extra special care, but expect that it can still
be made clearer, and might have errors.

I'd be delighted if this discussion thread helps us make CPython run
faster on multi-core machines, at least for some users. If you've got
some ideas about that, I'd love to hear them.

Questions and comments are, of course, welcome. Especially corrections
and improvements.

-- 
Jonathan

From nicholas.chammas at gmail.com  Wed Jul 25 12:12:40 2018
From: nicholas.chammas at gmail.com (Nicholas Chammas)
Date: Wed, 25 Jul 2018 12:12:40 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
References: <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
Message-ID: <CAOhmDzfbzNgBJa3CZbnQyWCmLZ7aQLWzakwkX1-M1oysj=B5mw@mail.gmail.com>

On Mon, Jul 23, 2018 at 6:05 PM Giampaolo Rodola' <g.rodola at gmail.com>
wrote:

> This:
>
>     v = a?.b
>
> ...*implicitly* checks if value is not None [and continues execution].
> This:
>
>     v = a
>     if a.b is not None:
>         v = a.b
>
> ...*explicitly* checks if value is not None and continues execution.
>

I think both of those are equally explicit. It's just that one notation is
more concise than the other. Explicitness and conciseness are related but
different things.

When something is "explicit", as I understand it, that means it does what
it says on the cover. There is no unstated behavior. The plain meaning of
`v = a?.b` is that it expands to the longer form (`v = a; if a.b ...`), and
it is just as explicit.

This reminds me of something I read about once called Stroustrup's Rule
<https://thefeedbackloop.xyz/stroustrups-rule-and-layering-over-time/> [1]:

> For new features, people insist on LOUD explicit syntax.
> For established features, people want terse notation.

I think the "explicit vs. implicit" part of this discussion is probably
better expressed as a discussion about "loud vs. terse" syntax. None of the
operators in PEP 505 have implicit behavior, to the best of my
understanding. It's just that the operators are new and have terse
spellings.

As a point of comparison, I think a good example of implicit behavior is
type coercion. When you ask Python to add an int to a float

    a = 3 + 4.5

all that you've explicitly asked for is the addition. However, Python
implicitly converts the 3 from an int to a float as part of the operation.
The type conversion isn't anywhere "on the cover" of the + operator. It's
implicit behavior.

[1] Bjarne Stroustrup makes the observation in this talk
<https://channel9.msdn.com/Events/Lang-NEXT/Lang-NEXT-2014/Keynote> at
23:00.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180725/c198b5bd/attachment.html>

From nicholas.chammas at gmail.com  Wed Jul 25 12:16:08 2018
From: nicholas.chammas at gmail.com (Nicholas Chammas)
Date: Wed, 25 Jul 2018 12:16:08 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAOhmDzfbzNgBJa3CZbnQyWCmLZ7aQLWzakwkX1-M1oysj=B5mw@mail.gmail.com>
References: <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <CAOhmDzfbzNgBJa3CZbnQyWCmLZ7aQLWzakwkX1-M1oysj=B5mw@mail.gmail.com>
Message-ID: <CAOhmDzf07Vnr+Uj+J4ebLJ9LhMdGep6So3v2nPss2_zcsgcyfg@mail.gmail.com>

On Wed, Jul 25, 2018 at 12:12 PM Nicholas Chammas <
nicholas.chammas at gmail.com> wrote:

> On Mon, Jul 23, 2018 at 6:05 PM Giampaolo Rodola' <g.rodola at gmail.com>
> wrote:
>
>> This:
>>
>>     v = a?.b
>>
>> ...*implicitly* checks if value is not None [and continues execution].
>> This:
>>
>>     v = a
>>     if a.b is not None:
>>         v = a.b
>>
>> ...*explicitly* checks if value is not None and continues execution.
>>
>
> I think both of those are equally explicit. It's just that one notation is
> more concise than the other. Explicitness and conciseness are related but
> different things.
>
> <snipped>
>

It looks like others already discussed this point later in the thread.
Apologies for rehashing the argument.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180725/46444442/attachment.html>

From abedillon at gmail.com  Wed Jul 25 18:11:08 2018
From: abedillon at gmail.com (Abe Dillon)
Date: Wed, 25 Jul 2018 17:11:08 -0500
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAOhmDzfbzNgBJa3CZbnQyWCmLZ7aQLWzakwkX1-M1oysj=B5mw@mail.gmail.com>
References: <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <CAOhmDzfbzNgBJa3CZbnQyWCmLZ7aQLWzakwkX1-M1oysj=B5mw@mail.gmail.com>
Message-ID: <CAKr=oZvBO1sNd0RdmUJ3xocp45etLpRXAo5dZuaa1j2qsKisDw@mail.gmail.com>

The problem here is not whether it's explicit. It's about Readability and
conciseness. Using symbols in place of words almost always harms
readability in favor of conciseness.

value = person.name if person.name else person

almost reads like english (aside from being a weird and totally uncommon
use case)

value = person?.name

Is a huge step towards the concise illegible soup of symbols that Perl is
famous for. It's a huge No from me.



On Wed, Jul 25, 2018 at 11:12 AM, Nicholas Chammas <
nicholas.chammas at gmail.com> wrote:

> On Mon, Jul 23, 2018 at 6:05 PM Giampaolo Rodola' <g.rodola at gmail.com>
> wrote:
>
>> This:
>>
>>     v = a?.b
>>
>> ...*implicitly* checks if value is not None [and continues execution].
>> This:
>>
>>     v = a
>>     if a.b is not None:
>>         v = a.b
>>
>> ...*explicitly* checks if value is not None and continues execution.
>>
>
> I think both of those are equally explicit. It's just that one notation is
> more concise than the other. Explicitness and conciseness are related but
> different things.
>
> When something is "explicit", as I understand it, that means it does what
> it says on the cover. There is no unstated behavior. The plain meaning of
> `v = a?.b` is that it expands to the longer form (`v = a; if a.b ...`), and
> it is just as explicit.
>
> This reminds me of something I read about once called Stroustrup's Rule
> <https://thefeedbackloop.xyz/stroustrups-rule-and-layering-over-time/>
>  [1]:
>
> > For new features, people insist on LOUD explicit syntax.
> > For established features, people want terse notation.
>
> I think the "explicit vs. implicit" part of this discussion is probably
> better expressed as a discussion about "loud vs. terse" syntax. None of the
> operators in PEP 505 have implicit behavior, to the best of my
> understanding. It's just that the operators are new and have terse
> spellings.
>
> As a point of comparison, I think a good example of implicit behavior is
> type coercion. When you ask Python to add an int to a float
>
>     a = 3 + 4.5
>
> all that you've explicitly asked for is the addition. However, Python
> implicitly converts the 3 from an int to a float as part of the operation.
> The type conversion isn't anywhere "on the cover" of the + operator. It's
> implicit behavior.
>
> [1] Bjarne Stroustrup makes the observation in this talk
> <https://channel9.msdn.com/Events/Lang-NEXT/Lang-NEXT-2014/Keynote> at
> 23:00.
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180725/a0b22f81/attachment-0001.html>

From jamtlu at gmail.com  Wed Jul 25 13:07:50 2018
From: jamtlu at gmail.com (James Lu)
Date: Wed, 25 Jul 2018 13:07:50 -0400
Subject: [Python-ideas] As-do statements/anonymous blocks in python
Message-ID: <CANQHGB2aCT3KOGnDP-qxA4PfV=34oiPGRO+rrbMBJG51K_4T-g@mail.gmail.com>

I'm open to any changes or criticism.

```
import atexit
as atexit.register:
    # ...do various cleanup tasks...
    print('Goodbye')

# is approximately equivalent to =>
import atexit
def _():
    # ...do various cleanup tasks...
    print('Goodbye')
atexit.register(_)

# flask example
@app.route("/")
def hello():
    return "Hello World!"
# is approximately equivalent to =>
as app.route('/'):
    return "Hello World!"

@app.route('/user/<username>')
def show_user_profile(username):
    # show the user profile for that user
    return 'User %s' % username

as app.route('/user/<username>') do username:
    return "Hello World!"

def print_sorted(iterable, block):
    sorted(iterable, )

l = [1, 2, 3, 4, 'spam']

as l.sort(key=%) do obj:
    return str(obj)

# multiple arguments
as spam do a, b:
    ...
```
## `%`  function call syntax
Calling a function with a single percent in place of an argument creates a
new function.

```
lumberjack(15, %)
# is equivalent to the expression
lambda x: lumberjack(15, %)
```
Using `*` instead of `%` could also be possible.

```
import threading, time
def interval(seconds, block):
    def target():
        while True:
            time.sleep(seconds)
            if block():  # stop looping if block returns True
                break
    threading.Thread(target=target).start()

as interval(5, %):
   print("chirp")
# => chirp every 5 seconds on a seperate thread
as threading.Timer(5, %):
   print("hi")
# => say "hi" in 5 seconds
```

## `^` currying function definition syntax?
I'm not sure this is necessary or a good idea.
```
def interval(seconds, ^block):
    def target():
        while True:
            time.sleep(seconds)
            if block():  # stop looping if block returns True
                break
    threading.Thread(target=target).start()
# is aprroximately equivalent to
def interval(seconds, block=None):
    def inner(block):
        def target():
            while True:
                time.sleep(seconds)
                if block():
                    break
        threading.Thread(target=target).start()
    if block == None:
        def outer(block):
            return inner(block)
    else:
        return inner(block)

as interval(5):
    print('chirp')
# equivalent to
interval(5)(lambda: print('chirp'))
```
### Lazy evaluation of chained `%` calls?
This would allow things like:
```
start_on_new_thread = threading.Thread(target=%).start()
def bong():
    while True:
        time.sleep(6*60)
        print('BONG')
start_on_new_thread(bong)
# alternatively
as start_on_new_thread:
    while True:
        time.sleep(6*60)
        print('BONG')
```

## As-do statements in classes
```
class M():
     def __init__(self):
         self.time = 5
     as interval(self.time, %):
         print('chirp')
```

I'm not sure if this should be valid, and I'd like the community's input on
when as-do statements should be bound when as-do.

## Proposed Type Hinting Syntax
```
as app.route('/user/<username>') do (username: str):
    return "Hello World!"
```

Alternatives:
```
as app.route('/user/<username>') do username: str%
    return "Hello World!"
# like objective-c
as app.route('/user/<username>') do username: str^
    return "Hello World!"
# like coffeescript
as app.route('/user/<username>') do username: str ->
    return "Hello World!"
```

I?m not totally sure of practical uses, but I?m imagining it would make
passing a function to another function much more convenient. In React, you
pass an argument called `render` to a `FlatList`, and `render` renders an
item of the list. `FlatList` is a scrollable list that handles unloading
off-screen items and loading items that will appear in the scroll box soon.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180725/d35883cb/attachment-0001.html>

From mertz at gnosis.cx  Wed Jul 25 18:36:24 2018
From: mertz at gnosis.cx (David Mertz)
Date: Wed, 25 Jul 2018 18:36:24 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAOhmDzfbzNgBJa3CZbnQyWCmLZ7aQLWzakwkX1-M1oysj=B5mw@mail.gmail.com>
References: <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <CAOhmDzfbzNgBJa3CZbnQyWCmLZ7aQLWzakwkX1-M1oysj=B5mw@mail.gmail.com>
Message-ID: <CAEbHw4a2ZNuyUC1Qxq26+kqF0v5Q1uWU3nrvhFuzi52K+uY7Bg@mail.gmail.com>

The fact that a while bunch have people have commented on this subthread
while not recognizing that the semantics of the '?.' and the if blocks are
entirely different suggests the operators are but magnets.

On Wed, Jul 25, 2018, 5:17 PM Nicholas Chammas <nicholas.chammas at gmail.com>
wrote:

> On Mon, Jul 23, 2018 at 6:05 PM Giampaolo Rodola' <g.rodola at gmail.com>
> wrote:
>
>> This:
>>
>>     v = a?.b
>>
>> ...*implicitly* checks if value is not None [and continues execution].
>> This:
>>
>>     v = a
>>     if a.b is not None:
>>         v = a.b
>>
>> ...*explicitly* checks if value is not None and continues execution.
>>
>
> I think both of those are equally explicit. It's just that one notation is
> more concise than the other. Explicitness and conciseness are related but
> different things.
>
> When something is "explicit", as I understand it, that means it does what
> it says on the cover. There is no unstated behavior. The plain meaning of
> `v = a?.b` is that it expands to the longer form (`v = a; if a.b ...`), and
> it is just as explicit.
>
> This reminds me of something I read about once called Stroustrup's Rule
> <https://thefeedbackloop.xyz/stroustrups-rule-and-layering-over-time/>
>  [1]:
>
> > For new features, people insist on LOUD explicit syntax.
> > For established features, people want terse notation.
>
> I think the "explicit vs. implicit" part of this discussion is probably
> better expressed as a discussion about "loud vs. terse" syntax. None of the
> operators in PEP 505 have implicit behavior, to the best of my
> understanding. It's just that the operators are new and have terse
> spellings.
>
> As a point of comparison, I think a good example of implicit behavior is
> type coercion. When you ask Python to add an int to a float
>
>     a = 3 + 4.5
>
> all that you've explicitly asked for is the addition. However, Python
> implicitly converts the 3 from an int to a float as part of the operation.
> The type conversion isn't anywhere "on the cover" of the + operator. It's
> implicit behavior.
>
> [1] Bjarne Stroustrup makes the observation in this talk
> <https://channel9.msdn.com/Events/Lang-NEXT/Lang-NEXT-2014/Keynote> at
> 23:00.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180725/2a62ddc1/attachment.html>

From nicholas.chammas at gmail.com  Wed Jul 25 18:44:12 2018
From: nicholas.chammas at gmail.com (Nicholas Chammas)
Date: Wed, 25 Jul 2018 18:44:12 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4a2ZNuyUC1Qxq26+kqF0v5Q1uWU3nrvhFuzi52K+uY7Bg@mail.gmail.com>
References: <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <CAOhmDzfbzNgBJa3CZbnQyWCmLZ7aQLWzakwkX1-M1oysj=B5mw@mail.gmail.com>
 <CAEbHw4a2ZNuyUC1Qxq26+kqF0v5Q1uWU3nrvhFuzi52K+uY7Bg@mail.gmail.com>
Message-ID: <CAOhmDzdr+A_+quk0pL7ttUex2oYQugKGkVWcSeOtNN-3X7wTkw@mail.gmail.com>

On Wed, Jul 25, 2018 at 6:36 PM David Mertz <mertz at gnosis.cx> wrote:

> The fact that a while bunch have people have commented on this subthread
> while not recognizing that the semantics of the '?.' and the if blocks are
> entirely different suggests the operators are but magnets.
>
> On Wed, Jul 25, 2018, 5:17 PM Nicholas Chammas <nicholas.chammas at gmail.com>
> wrote:
>
>> On Mon, Jul 23, 2018 at 6:05 PM Giampaolo Rodola' <g.rodola at gmail.com>
>> wrote:
>>
>>> This:
>>>
>>>     v = a?.b
>>>
>>> ...*implicitly* checks if value is not None [and continues execution].
>>> This:
>>>
>>>     v = a
>>>     if a.b is not None:
>>>         v = a.b
>>>
>>> ...*explicitly* checks if value is not None and continues execution.
>>>
>>
Sorry, lazy reading on my part. I skimmed the expanded form assuming it was
correct. I think it should instead read `if a is not None: ...`.

Is that what you're getting at?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180725/29f5ec34/attachment.html>

From abedillon at gmail.com  Wed Jul 25 19:01:22 2018
From: abedillon at gmail.com (Abe Dillon)
Date: Wed, 25 Jul 2018 18:01:22 -0500
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4a2ZNuyUC1Qxq26+kqF0v5Q1uWU3nrvhFuzi52K+uY7Bg@mail.gmail.com>
References: <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <CAOhmDzfbzNgBJa3CZbnQyWCmLZ7aQLWzakwkX1-M1oysj=B5mw@mail.gmail.com>
 <CAEbHw4a2ZNuyUC1Qxq26+kqF0v5Q1uWU3nrvhFuzi52K+uY7Bg@mail.gmail.com>
Message-ID: <CAKr=oZtZrhNzY+kUbbAsOGp6w198BuSizNNdJObwjrCpb3tLcg@mail.gmail.com>

>
> The fact that a while bunch have people have commented on this subthread
> while not recognizing that the semantics of the '?.' and the if blocks are
> entirely different suggests the operators are but magnets.
>

Can you explain? What do you mean by "the operators are but magnets"?

The "None coalescing" operator seems so similar to the short circuit
behavior of "or" that it has pretty much no merit. It's compared to ternary
statements in the last section of the PEP (which is suspiciously lacking
the "or" pattern).

I would agree that Python could use some more support for EAFP
<https://en.wikipedia.org/wiki/EAFP> style coding. An express-ionized
version of try/catch might help there, but I'm pretty sure the search for
an elegant solution to that has been relatively fruitless. The attribute
access and indexing are just unreadable in my view. Maybe if the question
mark came at the end of the expression it would be more readable and just
mean, "if the preceding expression raises an attribute exception on a
none-type object, ignore it and evaluate to None otherwise return the
result of the evaluation" Then just use parentheses to capture the scope:

initial = (person.name[0])? # handles if person is None or person.name is
None

but that still seems like a good way to end up with very ugly code.
Haskel's Maybe seems like a much better and more readable approach.



On Wed, Jul 25, 2018 at 5:36 PM, David Mertz <mertz at gnosis.cx> wrote:

> The fact that a while bunch have people have commented on this subthread
> while not recognizing that the semantics of the '?.' and the if blocks are
> entirely different suggests the operators are but magnets.
>
> On Wed, Jul 25, 2018, 5:17 PM Nicholas Chammas <nicholas.chammas at gmail.com>
> wrote:
>
>> On Mon, Jul 23, 2018 at 6:05 PM Giampaolo Rodola' <g.rodola at gmail.com>
>> wrote:
>>
>>> This:
>>>
>>>     v = a?.b
>>>
>>> ...*implicitly* checks if value is not None [and continues execution].
>>> This:
>>>
>>>     v = a
>>>     if a.b is not None:
>>>         v = a.b
>>>
>>> ...*explicitly* checks if value is not None and continues execution.
>>>
>>
>> I think both of those are equally explicit. It's just that one notation
>> is more concise than the other. Explicitness and conciseness are related
>> but different things.
>>
>> When something is "explicit", as I understand it, that means it does what
>> it says on the cover. There is no unstated behavior. The plain meaning of
>> `v = a?.b` is that it expands to the longer form (`v = a; if a.b ...`), and
>> it is just as explicit.
>>
>> This reminds me of something I read about once called Stroustrup's Rule
>> <https://thefeedbackloop.xyz/stroustrups-rule-and-layering-over-time/>
>>  [1]:
>>
>> > For new features, people insist on LOUD explicit syntax.
>> > For established features, people want terse notation.
>>
>> I think the "explicit vs. implicit" part of this discussion is probably
>> better expressed as a discussion about "loud vs. terse" syntax. None of the
>> operators in PEP 505 have implicit behavior, to the best of my
>> understanding. It's just that the operators are new and have terse
>> spellings.
>>
>> As a point of comparison, I think a good example of implicit behavior is
>> type coercion. When you ask Python to add an int to a float
>>
>>     a = 3 + 4.5
>>
>> all that you've explicitly asked for is the addition. However, Python
>> implicitly converts the 3 from an int to a float as part of the operation.
>> The type conversion isn't anywhere "on the cover" of the + operator. It's
>> implicit behavior.
>>
>> [1] Bjarne Stroustrup makes the observation in this talk
>> <https://channel9.msdn.com/Events/Lang-NEXT/Lang-NEXT-2014/Keynote> at
>> 23:00.
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180725/fff69ae7/attachment-0001.html>

From elazarg at gmail.com  Wed Jul 25 19:01:49 2018
From: elazarg at gmail.com (Elazar)
Date: Wed, 25 Jul 2018 19:01:49 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAKr=oZvBO1sNd0RdmUJ3xocp45etLpRXAo5dZuaa1j2qsKisDw@mail.gmail.com>
References: <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <CAOhmDzfbzNgBJa3CZbnQyWCmLZ7aQLWzakwkX1-M1oysj=B5mw@mail.gmail.com>
 <CAKr=oZvBO1sNd0RdmUJ3xocp45etLpRXAo5dZuaa1j2qsKisDw@mail.gmail.com>
Message-ID: <CAPw6O2Sg+Kt5d-vccamfQqyf+sJPX2zp8=jMKt2DYLqz6EjBJQ@mail.gmail.com>

On Wed, Jul 25, 2018 at 3:11 PM Abe Dillon <abedillon at gmail.com> wrote:

> The problem here is not whether it's explicit. It's about Readability and
> conciseness. Using symbols in place of words almost always harms
> readability in favor of conciseness.
>
> value = person.name if person.name else person
>
> almost reads like english (aside from being a weird and totally uncommon
> use case)
>
> value = person?.name
>
> Is a huge step towards the concise illegible soup of symbols that Perl is
> famous for. It's a huge No from me.
>
>
> Similarly,

Value is name of person

almost reads like english.

value = person.name

Starts to look like pearl (but does not avoid repetition; only hurts
english-like-readability) - or perhaps some other programming languages
that use similar operators, such as C#, Swift, Dart, F#, Kotlin and others.
As far as I know it is not generally considered a bad addition in any of
these languages, all of which put emphasis on readability.

Elazar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180725/58624ea7/attachment.html>

From mertz at gnosis.cx  Wed Jul 25 19:06:35 2018
From: mertz at gnosis.cx (David Mertz)
Date: Wed, 25 Jul 2018 19:06:35 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAKr=oZtZrhNzY+kUbbAsOGp6w198BuSizNNdJObwjrCpb3tLcg@mail.gmail.com>
References: <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <CAOhmDzfbzNgBJa3CZbnQyWCmLZ7aQLWzakwkX1-M1oysj=B5mw@mail.gmail.com>
 <CAEbHw4a2ZNuyUC1Qxq26+kqF0v5Q1uWU3nrvhFuzi52K+uY7Bg@mail.gmail.com>
 <CAKr=oZtZrhNzY+kUbbAsOGp6w198BuSizNNdJObwjrCpb3tLcg@mail.gmail.com>
Message-ID: <CAEbHw4Z9=VPWwpgFTx896M8tyjoR3Jq_RBx4r+FTQzJdegY5HQ@mail.gmail.com>

Sorry. From my tablet. "Bug magnets" (it really, really wants to
autocorrect that)

And yes, the problem is that the equivalent is actually:

    v = a
    if v is not None: v=a.b

The semantics are simply not the ones that are intuitive to most people
reading 'v = a?.b'

On Wed, Jul 25, 2018, 7:01 PM Abe Dillon <abedillon at gmail.com> wrote:

> The fact that a while bunch have people have commented on this subthread
>> while not recognizing that the semantics of the '?.' and the if blocks are
>> entirely different suggests the operators are but magnets.
>>
>
> Can you explain? What do you mean by "the operators are but magnets"?
>
> The "None coalescing" operator seems so similar to the short circuit
> behavior of "or" that it has pretty much no merit. It's compared to ternary
> statements in the last section of the PEP (which is suspiciously lacking
> the "or" pattern).
>
> I would agree that Python could use some more support for EAFP
> <https://en.wikipedia.org/wiki/EAFP> style coding. An express-ionized
> version of try/catch might help there, but I'm pretty sure the search for
> an elegant solution to that has been relatively fruitless. The attribute
> access and indexing are just unreadable in my view. Maybe if the question
> mark came at the end of the expression it would be more readable and just
> mean, "if the preceding expression raises an attribute exception on a
> none-type object, ignore it and evaluate to None otherwise return the
> result of the evaluation" Then just use parentheses to capture the scope:
>
> initial = (person.name[0])? # handles if person is None or person.name is
> None
>
> but that still seems like a good way to end up with very ugly code.
> Haskel's Maybe seems like a much better and more readable approach.
>
>
>
> On Wed, Jul 25, 2018 at 5:36 PM, David Mertz <mertz at gnosis.cx> wrote:
>
>> The fact that a while bunch have people have commented on this subthread
>> while not recognizing that the semantics of the '?.' and the if blocks are
>> entirely different suggests the operators are but magnets.
>>
>> On Wed, Jul 25, 2018, 5:17 PM Nicholas Chammas <
>> nicholas.chammas at gmail.com> wrote:
>>
>>> On Mon, Jul 23, 2018 at 6:05 PM Giampaolo Rodola' <g.rodola at gmail.com>
>>> wrote:
>>>
>>>> This:
>>>>
>>>>     v = a?.b
>>>>
>>>> ...*implicitly* checks if value is not None [and continues execution].
>>>> This:
>>>>
>>>>     v = a
>>>>     if a.b is not None:
>>>>         v = a.b
>>>>
>>>> ...*explicitly* checks if value is not None and continues execution.
>>>>
>>>
>>> I think both of those are equally explicit. It's just that one notation
>>> is more concise than the other. Explicitness and conciseness are related
>>> but different things.
>>>
>>> When something is "explicit", as I understand it, that means it does
>>> what it says on the cover. There is no unstated behavior. The plain meaning
>>> of `v = a?.b` is that it expands to the longer form (`v = a; if a.b ...`),
>>> and it is just as explicit.
>>>
>>> This reminds me of something I read about once called Stroustrup's Rule
>>> <https://thefeedbackloop.xyz/stroustrups-rule-and-layering-over-time/>
>>>  [1]:
>>>
>>> > For new features, people insist on LOUD explicit syntax.
>>> > For established features, people want terse notation.
>>>
>>> I think the "explicit vs. implicit" part of this discussion is probably
>>> better expressed as a discussion about "loud vs. terse" syntax. None of the
>>> operators in PEP 505 have implicit behavior, to the best of my
>>> understanding. It's just that the operators are new and have terse
>>> spellings.
>>>
>>> As a point of comparison, I think a good example of implicit behavior is
>>> type coercion. When you ask Python to add an int to a float
>>>
>>>     a = 3 + 4.5
>>>
>>> all that you've explicitly asked for is the addition. However, Python
>>> implicitly converts the 3 from an int to a float as part of the operation.
>>> The type conversion isn't anywhere "on the cover" of the + operator. It's
>>> implicit behavior.
>>>
>>> [1] Bjarne Stroustrup makes the observation in this talk
>>> <https://channel9.msdn.com/Events/Lang-NEXT/Lang-NEXT-2014/Keynote> at
>>> 23:00.
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180725/b6c42ee5/attachment.html>

From nicholas.chammas at gmail.com  Wed Jul 25 19:06:44 2018
From: nicholas.chammas at gmail.com (Nicholas Chammas)
Date: Wed, 25 Jul 2018 19:06:44 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAKr=oZvBO1sNd0RdmUJ3xocp45etLpRXAo5dZuaa1j2qsKisDw@mail.gmail.com>
References: <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <CAOhmDzfbzNgBJa3CZbnQyWCmLZ7aQLWzakwkX1-M1oysj=B5mw@mail.gmail.com>
 <CAKr=oZvBO1sNd0RdmUJ3xocp45etLpRXAo5dZuaa1j2qsKisDw@mail.gmail.com>
Message-ID: <CAOhmDzfKuO7WJmqpeb1bfbRU51=JDV1PoAFK98G8omDJGRK4pg@mail.gmail.com>

On Wed, Jul 25, 2018 at 6:11 PM Abe Dillon <abedillon at gmail.com> wrote:

> The problem here is not whether it's explicit. It's about Readability and
> conciseness. Using symbols in place of words almost always harms
> readability in favor of conciseness.
>
> value = person.name if person.name else person
>
> almost reads like english (aside from being a weird and totally uncommon
> use case)
>
> value = person?.name
>
> Is a huge step towards the concise illegible soup of symbols that Perl is
> famous for. It's a huge No from me.
>

The two statements you wrote are not the same. The first statement will
error out if person is None. The proposed None-aware operators are
specifically designed to handle variables that may be None.

The first statement should instead read:

    value = person.name if person is not None else person

That's what `value = person?.name` means.

As others have pointed out, I suppose the fact that multiple people have
messed up the meaning of the proposed operators is concerning. Perhaps the
PEP could be improved by adding some dead simple examples of each operator
and an equivalent statement that doesn't use the operator, to better
illustrate their meaning. But I gather that will do little in the way of
addressing some of the stronger objections raised here.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180725/6617730d/attachment-0001.html>

From abedillon at gmail.com  Wed Jul 25 19:15:25 2018
From: abedillon at gmail.com (Abe Dillon)
Date: Wed, 25 Jul 2018 18:15:25 -0500
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAOhmDzfKuO7WJmqpeb1bfbRU51=JDV1PoAFK98G8omDJGRK4pg@mail.gmail.com>
References: <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <CAOhmDzfbzNgBJa3CZbnQyWCmLZ7aQLWzakwkX1-M1oysj=B5mw@mail.gmail.com>
 <CAKr=oZvBO1sNd0RdmUJ3xocp45etLpRXAo5dZuaa1j2qsKisDw@mail.gmail.com>
 <CAOhmDzfKuO7WJmqpeb1bfbRU51=JDV1PoAFK98G8omDJGRK4pg@mail.gmail.com>
Message-ID: <CAKr=oZsZYxO6xV0iHWA=i-Yij5B+MBv8+zuyseSAO=E__NSHSg@mail.gmail.com>

>
> The two statements you wrote are not the same. The first statement will
> error out if person is None.

That's my bad. I was copying off of an erroneous example. Thanks for
correcting me.

The proposed None-aware operators are specifically designed to handle
> variables that may be None.
>
Yes, I think the syntax that you've landed on is confusing enough that it
opens the door to more errors than it closes. Just reading "(a?.b ?? c).d?.e"
and "await a?.b(c).d?[e]"


On Wed, Jul 25, 2018 at 6:06 PM, Nicholas Chammas <
nicholas.chammas at gmail.com> wrote:

> On Wed, Jul 25, 2018 at 6:11 PM Abe Dillon <abedillon at gmail.com> wrote:
>
>> The problem here is not whether it's explicit. It's about Readability and
>> conciseness. Using symbols in place of words almost always harms
>> readability in favor of conciseness.
>>
>> value = person.name if person.name else person
>>
>> almost reads like english (aside from being a weird and totally uncommon
>> use case)
>>
>> value = person?.name
>>
>> Is a huge step towards the concise illegible soup of symbols that Perl is
>> famous for. It's a huge No from me.
>>
>
> The two statements you wrote are not the same. The first statement will
> error out if person is None. The proposed None-aware operators are
> specifically designed to handle variables that may be None.
>
> The first statement should instead read:
>
>     value = person.name if person is not None else person
>
> That's what `value = person?.name` means.
>
> As others have pointed out, I suppose the fact that multiple people have
> messed up the meaning of the proposed operators is concerning. Perhaps the
> PEP could be improved by adding some dead simple examples of each operator
> and an equivalent statement that doesn't use the operator, to better
> illustrate their meaning. But I gather that will do little in the way of
> addressing some of the stronger objections raised here.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180725/b7aa17ec/attachment.html>

From abedillon at gmail.com  Wed Jul 25 19:32:11 2018
From: abedillon at gmail.com (Abe Dillon)
Date: Wed, 25 Jul 2018 18:32:11 -0500
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAKr=oZsZYxO6xV0iHWA=i-Yij5B+MBv8+zuyseSAO=E__NSHSg@mail.gmail.com>
References: <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <CAOhmDzfbzNgBJa3CZbnQyWCmLZ7aQLWzakwkX1-M1oysj=B5mw@mail.gmail.com>
 <CAKr=oZvBO1sNd0RdmUJ3xocp45etLpRXAo5dZuaa1j2qsKisDw@mail.gmail.com>
 <CAOhmDzfKuO7WJmqpeb1bfbRU51=JDV1PoAFK98G8omDJGRK4pg@mail.gmail.com>
 <CAKr=oZsZYxO6xV0iHWA=i-Yij5B+MBv8+zuyseSAO=E__NSHSg@mail.gmail.com>
Message-ID: <CAKr=oZup=FJx84nZNT_U13=s632ctyUFKJMwzO0UB8G_R=KTFw@mail.gmail.com>

*sorry, cat hit "send"...

Just reading those examples made me want to cry and go hide in a dark dark
cave and never come out. I'm sure using actual variable names would help a
bit, but not much.

As for the non-english nature of:
value = person.name

I highly disagree with the argument that since that is a step removed from
natural language readability that:
value = person?.name

Should be considered fair game. Nor do I buy the "other languages do it"
argument. Some of the syntax of Python is based on familiar patterns in
other languages (like '.' access) some of it is based on common math (e.g.
"=", "+", etc.) which is also taught in grade school. Some of the patterns
borrowed from other languages were a mistake and considered cruft. Some of
that cruft was scraped off in the 2to3 migration. Maybe "." should have
been apostrophe "s" all along. Maybe lambda should have been
'make_function'. That's not what we're here to discuss. A lot of languages
have a ternary operator (x ? y : z). Python wisely used words instead of
symbols and now many students don't even have to crack a book to decipher
Python's ternary operator. Adding crap to a language is easy. Removing it
is damn near impossible. You have to have extremely good reasons to add new
syntax and I don't have to defend any of Python's warts to justify
rejecting yours.

On Wed, Jul 25, 2018 at 6:15 PM, Abe Dillon <abedillon at gmail.com> wrote:

> The two statements you wrote are not the same. The first statement will
>> error out if person is None.
>
> That's my bad. I was copying off of an erroneous example. Thanks for
> correcting me.
>
> The proposed None-aware operators are specifically designed to handle
>> variables that may be None.
>>
> Yes, I think the syntax that you've landed on is confusing enough that it
> opens the door to more errors than it closes. Just reading "(a?.b ?? c).d?.e"
> and "await a?.b(c).d?[e]"
>
>
> On Wed, Jul 25, 2018 at 6:06 PM, Nicholas Chammas <
> nicholas.chammas at gmail.com> wrote:
>
>> On Wed, Jul 25, 2018 at 6:11 PM Abe Dillon <abedillon at gmail.com> wrote:
>>
>>> The problem here is not whether it's explicit. It's about Readability
>>> and conciseness. Using symbols in place of words almost always harms
>>> readability in favor of conciseness.
>>>
>>> value = person.name if person.name else person
>>>
>>> almost reads like english (aside from being a weird and totally uncommon
>>> use case)
>>>
>>> value = person?.name
>>>
>>> Is a huge step towards the concise illegible soup of symbols that Perl
>>> is famous for. It's a huge No from me.
>>>
>>
>> The two statements you wrote are not the same. The first statement will
>> error out if person is None. The proposed None-aware operators are
>> specifically designed to handle variables that may be None.
>>
>> The first statement should instead read:
>>
>>     value = person.name if person is not None else person
>>
>> That's what `value = person?.name` means.
>>
>> As others have pointed out, I suppose the fact that multiple people have
>> messed up the meaning of the proposed operators is concerning. Perhaps the
>> PEP could be improved by adding some dead simple examples of each operator
>> and an equivalent statement that doesn't use the operator, to better
>> illustrate their meaning. But I gather that will do little in the way of
>> addressing some of the stronger objections raised here.
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180725/1bf0a786/attachment-0001.html>

From greg.ewing at canterbury.ac.nz  Wed Jul 25 19:35:34 2018
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 26 Jul 2018 11:35:34 +1200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4Z9=VPWwpgFTx896M8tyjoR3Jq_RBx4r+FTQzJdegY5HQ@mail.gmail.com>
References: <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <CAOhmDzfbzNgBJa3CZbnQyWCmLZ7aQLWzakwkX1-M1oysj=B5mw@mail.gmail.com>
 <CAEbHw4a2ZNuyUC1Qxq26+kqF0v5Q1uWU3nrvhFuzi52K+uY7Bg@mail.gmail.com>
 <CAKr=oZtZrhNzY+kUbbAsOGp6w198BuSizNNdJObwjrCpb3tLcg@mail.gmail.com>
 <CAEbHw4Z9=VPWwpgFTx896M8tyjoR3Jq_RBx4r+FTQzJdegY5HQ@mail.gmail.com>
Message-ID: <5B590946.3030402@canterbury.ac.nz>

David Mertz wrote:
> Sorry. From my tablet. "Bug magnets" (it really, really wants to 
> autocorrect that)

At least it didn't correct it to "butt magnets". :-)

-- 
Greg

From steve at pearwood.info  Wed Jul 25 13:19:07 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 26 Jul 2018 03:19:07 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAOhmDzfbzNgBJa3CZbnQyWCmLZ7aQLWzakwkX1-M1oysj=B5mw@mail.gmail.com>
References: <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <CAOhmDzfbzNgBJa3CZbnQyWCmLZ7aQLWzakwkX1-M1oysj=B5mw@mail.gmail.com>
Message-ID: <20180725171907.GJ8744@ando.pearwood.info>

On Wed, Jul 25, 2018 at 12:12:40PM -0400, Nicholas Chammas wrote:

> When something is "explicit", as I understand it, that means it does what
> it says on the cover. There is no unstated behavior. The plain meaning of
> `v = a?.b` is that it expands to the longer form (`v = a; if a.b ...`), and
> it is just as explicit.

Right. We don't insist on writing

    mydict.look_up_key_in_self_and_raise_keyerror_if_the_key_is_not_found(key="spam")

instead of 

    mydict["spam"]

out of a mistaken idea that explicitness requires verbosity and that 
punctuation is always implicit.

Symbols (whether made of text or punctuation) have meaning, and that 
meaning has to be taken into account. You *can't* spell everything out 
in full. We always have to take the meaning of something as given, and 
that can be punctuation just as easily as words.


> This reminds me of something I read about once called Stroustrup's Rule
> <https://thefeedbackloop.xyz/stroustrups-rule-and-layering-over-time/> [1]:
> 
> > For new features, people insist on LOUD explicit syntax.
> > For established features, people want terse notation.
> 
> I think the "explicit vs. implicit" part of this discussion is probably
> better expressed as a discussion about "loud vs. terse" syntax. None of the
> operators in PEP 505 have implicit behavior, to the best of my
> understanding. It's just that the operators are new and have terse
> spellings.

That is a great observation! Thanks.


-- 
Steve

From steve at pearwood.info  Wed Jul 25 09:20:18 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 25 Jul 2018 23:20:18 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <FAFE9409-E86A-483E-A83A-BC334EBBCCAF@Damon-family.org>
References: <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <a039f890-a40e-20af-c591-5df7d5cc4855@kynesim.co.uk>
 <FAFE9409-E86A-483E-A83A-BC334EBBCCAF@Damon-family.org>
Message-ID: <20180725132017.GI8744@ando.pearwood.info>

On Tue, Jul 24, 2018 at 08:07:36AM -0400, Richard Damon wrote:

> The fact that you changed NullCoalesce into Foo to show lack of 
> explicitness seems a straw-man.

I understood Rhodri as making the point that if you don't know what 
NullCoalesce means or does, it might as well be called Foo. There's no 
hint in the syntax that something magical is happening:


    MyClass(obj).spam.eggs  # ordinary attribute lookup

    YourClass(obj).spam.eggs  # still ordinary attribute lookup

    Mxyzptlk(obj).spam.eggs  # looks like ordinary attribute lookup

    NullCoalesce(obj).spam.eggs  # IT'S A TRAP!



> Words are FULL of meaning, while symbols are less so.

It isn't a competition to squeeze as much meaning as possible into a 
single word or symbol. Precision is more important than fullness.

A case in point: the word "add" has six meanings listed by WordNet, and 
the Mobi Thesaurus gives 102 synonyms for it. A function or method called "add" 
could do anything:

- add users to a database;
- add pages to a chapter;
- add elements to a set;
- add items to a queue;

etc. In contrast, the + symbol in Python has two standard meanings:

- numeric addition;
- sequence concatenation;

and while it is true that with operator overloading a class could make 
the + operator do anything, that is usually taken as an argument to 
avoid operator overloading, or at least use it cautiously.

Surely you don't think that the + operator is a mistake because the word 
"add" is more full of meaning than the symbol? If not, what point were 
you trying to make?


> The biggest issue I see with the use of ? here is 
> that ? does have some meaning, it says we are going to be (or have) 
> asked a question, it doesn?t tell us what the question is.

Sometimes you just have to learn the meanings of words or symbols.

What does NullCoalesce mean? Until this PEP was proposed, I had no idea 
this was even a thing, and in truth I actually had to look up "coalesce" 
in a dictionary to be sure I understood it correctly, because the 
context doesn't seem quite right.

(It still doesn't -- it might be the standard comp sci term for this 
feature, but I don't think it quite matches the ordinary usage of the 
word.)

Ask a dozen programming beginners what "NullCoalesce" does, and I expect 
every one of them will say "No idea".

Strangely enough, nobody complains about having to learn what "import" 
does, or slicing syntax x[:], or string backslash escapes "\n", or 
"property". Because we're used to them, we just accept that you have to 
learn the meaning of things the first time you see them. We aren't born 
knowing what "class" does, or the difference between x[a] and x(a).

Every single one of us, without exception, had to learn what . means at 
some point or another.

And yet, its wailing and gnashing of teeth and panic in the streets over 
the idea that people might have to learn what ?. means in the same way 
they learned what **kwargs or mylist[1:] or obj.attr means.

    "It's a question, but what is the question? How will I ever
    find out? If only there was a website where I might look up
    Python operators and learn what they do!!!"


> ?. has some indication that we are doing an attribute access that is 
> in some way conditional, but a?.b could mean that we are conditional 
> on a not being null, or it could be asking to suppress any and all 
> error in getting b, even if a is an int and thus doesn?t have a b. The 
> words carry a lot more meaning.

Yeah, because words are always obvious.

import ast  # Who can remember all these damned TLAs?
import bdm  # Births Deaths Marriages?
import csv  # Court Services Victoria?
import mailcap  # what the postman wears on his head?
import pickle  # something to do with condiments?
import turtle  # somebody is taking the micky
import curses  # good thing I'm not superstitious


    "But Steve, you don't understand. Having to learn the meaning
     of words you haven't learned before is right and proper and
     unavoidable. But all new punctuation symbols must be intuitively
     obvious to newborn babies, or else they are the Devil's Mark,
     or Perl, not sure which is worse."

(Its called sarcasm, not "strawman". Just sayin'.)


-- 
Steve

From steve at pearwood.info  Wed Jul 25 13:40:41 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 26 Jul 2018 03:40:41 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
Message-ID: <20180725174041.GK8744@ando.pearwood.info>

On Tue, Jul 24, 2018 at 07:02:54AM -0400, David Mertz wrote:
> On Tue, Jul 24, 2018, 5:50 AM Steven D'Aprano <steve at pearwood.info> wrote:
> 
> > But what certainly *is* implicity is David Mertz' suggestion for a
> > magical None-aware proxy:
> >
> >     x.attribute
> >
> > The only way to tell whether that was an ordinary attribute lookup or a
> > none-aware lookup would be to carefully inspect x and find out whether it
> > was an instance of the None-aware proxy class or not.
> >
> 
> Every use I've suggested for the magic proxy is similar to:
> 
>   NullCoalesce(cfg).user.profile.food

I trust that you don't expect everyone to religiously follow the 
exact letter of your examples. Being able to extract out subexpressions 
into variables is an important programming technique, and we ought to be 
able to do things like this:

    config = NullCoalesce(cfg)
    process(config)

    # later...
    user = config.user
    user_records[user.id] = user
    food = get_preferred_food(user)

Now there are at least four potentially magic proxy objects floating 
around: config, user, the user.id key, and food.

It is naive (if not a little disingenuous) to suggest that this 
NullCoalesce proxy object will always be used in a single expression and 
never passed as argument to a function, inserted in a list or dict, or 
otherwise processed any distance away from the call to NullCoalesce.

Consequently, the moment we see 

    from module import NullCoalesce

in a module, *every dot attribute access* becomes questionable until 
we've satisfied ourselves that the object in question isn't magic.


    spam.attribute
    eggs.attribute


One of those is ordinary attribute access that can raise AttributeError, 
and the other is magic attribute access that silently suppresses 
AttributeErrors. Which is which?


    spam?.attribute
    eggs.attribute


One of those is ordinary attribute access that can raise AttributeError 
if the object is None, and the other is None-aware "maybe dot" attribute 
access that returns None if the object is None.

I wish there was a way to tell which is which... if only they used 
different syntax or something... *wink*



> Yes, the class is magic. That much more so in the library I published last
> night that utilizes wrapt.ObjectProxy. But it's also pretty explicit in
> that an actual *word* announces that funny stuff is going to happen on the
> same line.
> 
> Of course the this could be abused with:
> 
>   cfg = NoneCoalesce(cfg)
>   ... 1000 lines ...
>   do_something(cfg)

Why does Python have named variables if it is "abuse" to use them?



-- 
Steve

From mertz at gnosis.cx  Wed Jul 25 21:02:33 2018
From: mertz at gnosis.cx (David Mertz)
Date: Wed, 25 Jul 2018 21:02:33 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180725174041.GK8744@ando.pearwood.info>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
Message-ID: <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>

On Wed, Jul 25, 2018 at 8:00 PM Steven D'Aprano <steve at pearwood.info> wrote:

> > Every use I've suggested for the magic proxy is similar to:
> >   NullCoalesce(cfg).user.profile.food
>

Thanks Steven!  Several things you suggest urge some minor improvements to
my slightly-more-than-toy library `coalescing` (
https://pypi.org/project/coalescing/).


> I trust that you don't expect everyone to religiously follow the
> exact letter of your examples. Being able to extract out subexpressions
> into variables is an important programming technique, and we ought to be
> able to do things like this:
>
>     config = NullCoalesce(cfg)
>     process(config)
>
>     # later...
>     user = config.user
>     user_records[user.id] = user
>     food = get_preferred_food(user)
>

Yes, absolutely! In many cases this will JUST WORK because of the magic in
wrapt.ObjectProxy.  But it also makes me realize that I should provide a
plain old function `unbox()` as well as the method.  In particular,
`unbox(obj)` can and will figure out whether the object is really a proxy
at all, and if not simply return `obj` itself.  Then you do no harm by
scattering unbox() calls in places where they may or may not be needed.

I also think that even though it was a typo at first, the name NullCoalesce
is better than NoneCoalesce.  I'll provide an alias.


>     spam?.attribute
>     eggs.attribute
> One of those is ordinary attribute access that can raise AttributeError
> if the object is None, and the other is None-aware "maybe dot" attribute
> access that returns None if the object is None.
>

That is disingenuous, I think.  Can this raise an AttributeError?

    spam?.eggs?.bacon

Of course it can! And this is exactly the pattern used in many examples in
the PEP and the discussion. So the PEP would create a situation where code
will raise AttributeError in a slightly?and subtly?different set of
circumstances than plain attribute access will.

The fact that half the readers of this thread won't immediately see just
when that AttributeError might occur (based on lots of prior
misunderstandings by smart posters) makes the operator that much more
magical.

-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180725/415a806e/attachment-0001.html>

From jamtlu at gmail.com  Wed Jul 25 21:12:14 2018
From: jamtlu at gmail.com (James Lu)
Date: Wed, 25 Jul 2018 21:12:14 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <mailman.288.1532561533.7656.python-ideas@python.org>
References: <mailman.288.1532561533.7656.python-ideas@python.org>
Message-ID: <B88AE8D9-97E6-4B07-9C5B-59D90DB4C34F@gmail.com>

What if we used ? after the statement beginning?

name ?= person.name
custom_query ?= entity.get_query(context)
# Becomes None if entity is None. Raise an exception if entity is not None and get_query is None or undefined.
custom_query ??= entity.get_query(context)
# If entity, entity.get_query, entity.get_query(context) evaluate to null, the operation short-circuits and custom_query becomes None
await? foo
with? bar as baz:
    # this only runs if bar is not None
    pass


?= only short circuits into None when the first evaluation is None (the weak operator)
??= short circuits into None whenever any evaluation is None or raises an AttributeError. (the strong operator)

I?m imagining the strong operator would be useful especially for duck typing. No more hasattr checks, no more isinstance checks. 

I would like to see some real world use cases for none aware operators that couldn?t be covered by these none aware assignment operators.

Previous code:
# original code
a ?= b.c
# another, second, programmer comes along and changes it to
a ?= b.c.d

The interpreter would raise an exception, if ?d? was None or not defined on ?c?. If this is intended behavior, this forces the second programmer to explicitly mark that all attribute access is coalescing with the strong operator, instead of the interpreter swallowing the exception now and emitting an exception later. The previous code assumes ?b? is None, which in reality may represent the state of a network socket or a file transfer. The program may display that the operation was complete, leading to a bug in the output. 

I believe this is more readable as well.
> On Jul 25, 2018, at 7:32 PM, python-ideas-request at python.org wrote:
> 
> PEP 505: None-aware operators

From rosuav at gmail.com  Wed Jul 25 21:20:26 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 26 Jul 2018 11:20:26 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
Message-ID: <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>

On Thu, Jul 26, 2018 at 11:02 AM, David Mertz <mertz at gnosis.cx> wrote:
> That is disingenuous, I think.  Can this raise an AttributeError?
>
>     spam?.eggs?.bacon
>
> Of course it can! And this is exactly the pattern used in many examples in
> the PEP and the discussion. So the PEP would create a situation where code
> will raise AttributeError in a slightly?and subtly?different set of
> circumstances than plain attribute access will.

I don't understand. If it were to raise AttributeError, it would be
because spam (or spam.eggs) isn't None, but doesn't have an attribute
eggs (or bacon). Exactly the same as regular attribute access. How is
it slightly different? Have I missed something?

ChrisA

From nicholas.chammas at gmail.com  Wed Jul 25 21:45:37 2018
From: nicholas.chammas at gmail.com (Nicholas Chammas)
Date: Wed, 25 Jul 2018 21:45:37 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
Message-ID: <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>

On Wed, Jul 25, 2018 at 9:20 PM Chris Angelico <rosuav at gmail.com> wrote:

> On Thu, Jul 26, 2018 at 11:02 AM, David Mertz <mertz at gnosis.cx> wrote:
> > That is disingenuous, I think.  Can this raise an AttributeError?
> >
> >     spam?.eggs?.bacon
> >
> > Of course it can! And this is exactly the pattern used in many examples
> in
> > the PEP and the discussion. So the PEP would create a situation where
> code
> > will raise AttributeError in a slightly?and subtly?different set of
> > circumstances than plain attribute access will.
>
> I don't understand. If it were to raise AttributeError, it would be
> because spam (or spam.eggs) isn't None, but doesn't have an attribute
> eggs (or bacon). Exactly the same as regular attribute access. How is
> it slightly different? Have I missed something?
>

That was my reaction, too.

    food = spam?.eggs?.bacon

Can be rewritten as:

    food = spam
    if spam is not None and spam.eggs is not None:
        food = spam.eggs.bacon

They both behave identically, no? Maybe I missed the point David was trying
to make.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180725/56dd7b91/attachment.html>

From rosuav at gmail.com  Wed Jul 25 21:56:39 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 26 Jul 2018 11:56:39 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
Message-ID: <CAPTjJmrYhCtvLPGMb6==QKsr3ODXeaERiENnCdgeB+1ABNGp5A@mail.gmail.com>

On Thu, Jul 26, 2018 at 11:45 AM, Nicholas Chammas
<nicholas.chammas at gmail.com> wrote:
> On Wed, Jul 25, 2018 at 9:20 PM Chris Angelico <rosuav at gmail.com> wrote:
>>
>> On Thu, Jul 26, 2018 at 11:02 AM, David Mertz <mertz at gnosis.cx> wrote:
>> > That is disingenuous, I think.  Can this raise an AttributeError?
>> >
>> >     spam?.eggs?.bacon
>> >
>> > Of course it can! And this is exactly the pattern used in many examples
>> > in
>> > the PEP and the discussion. So the PEP would create a situation where
>> > code
>> > will raise AttributeError in a slightly?and subtly?different set of
>> > circumstances than plain attribute access will.
>>
>> I don't understand. If it were to raise AttributeError, it would be
>> because spam (or spam.eggs) isn't None, but doesn't have an attribute
>> eggs (or bacon). Exactly the same as regular attribute access. How is
>> it slightly different? Have I missed something?
>
>
> That was my reaction, too.
>
>     food = spam?.eggs?.bacon
>
> Can be rewritten as:
>
>     food = spam
>     if spam is not None and spam.eggs is not None:
>         food = spam.eggs.bacon
>
> They both behave identically, no? Maybe I missed the point David was trying
> to make.

Aside from questions of repeated evaluation/assignment, yes. The broad
semantics should be the same.

(If you want to get technical, "spam" gets evaluated exactly once,
"spam.eggs" a maximum of once, and "food" gets assigned exactly once.
Your equivalent may evaluate and assign multiple times.)

ChrisA

From steve at pearwood.info  Wed Jul 25 22:10:47 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 26 Jul 2018 12:10:47 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAKr=oZvBO1sNd0RdmUJ3xocp45etLpRXAo5dZuaa1j2qsKisDw@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <CAOhmDzfbzNgBJa3CZbnQyWCmLZ7aQLWzakwkX1-M1oysj=B5mw@mail.gmail.com>
 <CAKr=oZvBO1sNd0RdmUJ3xocp45etLpRXAo5dZuaa1j2qsKisDw@mail.gmail.com>
Message-ID: <20180726021047.GM8744@ando.pearwood.info>

On Wed, Jul 25, 2018 at 05:11:08PM -0500, Abe Dillon wrote:
> The problem here is not whether it's explicit. It's about Readability and
> conciseness. Using symbols in place of words almost always harms
> readability in favor of conciseness.

And that is why we prefer COBOL over unreadable Perl-like languages that 
use unreadable symbols like:

    arithmetic operators  + - * / // % **
    bitwise operators  & | ^ >> << ~
    comparison operators  == != < <= >= >
    function call symbols  func(arg)
    argument packing and unpacking  *args, **kwargs
    sequence slicing  seq[::]
    assignment  =
    comments  #
    decorator syntax @

What a horrible, unreadable language that would be.

REMARK this is much better than Perl-like x = func(a, b, c)
PUT THE RESULT OF CALLING FUNCTION func WITH ARGUMENTS a AND b AND c INTO x

\s


Why do you claim that symbols and conciseness is "almost always" worse 
for readability? Your assertion doesn't survive even a cursory 
consideration.

Readability doesn't occur in a vacuum. You have to understand the 
symbols, and it doesn't matter whether they are punctuation or words, if 
you don't understand them, they're "unreadable":

    result = page_count + 1  # readable
    jwrtsv = xpwvabs_ue + 1  # what?
    result obqxayrhs page_count wmsna 1 # what?

The first time I encounted Python 1.5, I found it unreadable: it was 
full of mysterious keywords "class", "def", strange punctuation like 
[1:] and x.y, and idioms I couldn't guess like

    for i in range(len(seq))

With no context, it might as well have been Ancient Sumarian. 

Do you know what helps readability? *Learning to read*. Once you have 
learned to read ?. and friends, they will be as readable as . and 
slicing is now.



-- 
Steve

From mertz at gnosis.cx  Wed Jul 25 22:12:29 2018
From: mertz at gnosis.cx (David Mertz)
Date: Wed, 25 Jul 2018 22:12:29 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
Message-ID: <CAEbHw4Yzt1Wfr1iBw+mmTYWhh8b2SwyFLX1AHOL-4YP6H+_ChA@mail.gmail.com>

On Wed, Jul 25, 2018 at 9:47 PM Nicholas Chammas <nicholas.chammas at gmail.com>
wrote:

> > That is disingenuous, I think.  Can this raise an AttributeError?
>> >     spam?.eggs?.bacon
>> > Of course it can! And this is exactly the pattern used in many examples
>> in
>> > the PEP and the discussion. So the PEP would create a situation where
>> code
>> > will raise AttributeError in a slightly?and subtly?different set of
>> > circumstances than plain attribute access will.
>>
>

>     food = spam?.eggs?.bacon
> Can be rewritten as:
>     food = spam
>     if spam is not None and spam.eggs is not None:
>         food = spam.eggs.bacon
> They both behave identically, no? Maybe I missed the point David was
> trying to make.
>

No, you illustrate it perfectly! I had to stare at your translation for a
while to decide if it was really identical to the proposed
`spam?.eggs?.bacon`.  The fact I have to think so hard makes the syntax
feel non-obvious.

Plus, there's the fact that your best effort at translating the proposed
syntax is WRONG.  Even a strong proponent cannot explain the behavior on a
first try.  And indeed, it behaves subtly different from plain attribute
access in where it raises AttributeError.

>>> spam = SimpleNamespace()
>>> spam.eggs = None
>>> spam.eggs.bacon
AttributeError: 'NoneType' object has no attribute 'bacon'

>>> # spam?.eggs?.bacon
>>> # Should be: None

>>> "Translation" does something different
>>> food = spam
>>> if spam is not None and spam.eggs is not None:
...     food = spam.eggs.bacon
>>> food
namespace(eggs=None)


-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180725/71560565/attachment.html>

From mertz at gnosis.cx  Wed Jul 25 22:14:07 2018
From: mertz at gnosis.cx (David Mertz)
Date: Wed, 25 Jul 2018 22:14:07 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPTjJmrYhCtvLPGMb6==QKsr3ODXeaERiENnCdgeB+1ABNGp5A@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAPTjJmrYhCtvLPGMb6==QKsr3ODXeaERiENnCdgeB+1ABNGp5A@mail.gmail.com>
Message-ID: <CAEbHw4Zc7Q63ZC1wXXPxWdytQDbdZBt0cPK_LCxTpskzyW8_Lg@mail.gmail.com>

So now at least TWO proponents of 505 cannot successfully translate a very
simple example taken almost directly from the PEP!

Is that REALLY a good argument for it being helpful, and not being a bug
magnet?!

On Wed, Jul 25, 2018 at 9:57 PM Chris Angelico <rosuav at gmail.com> wrote:

> On Thu, Jul 26, 2018 at 11:45 AM, Nicholas Chammas
> <nicholas.chammas at gmail.com> wrote:
> > On Wed, Jul 25, 2018 at 9:20 PM Chris Angelico <rosuav at gmail.com> wrote:
> >>
> >> On Thu, Jul 26, 2018 at 11:02 AM, David Mertz <mertz at gnosis.cx> wrote:
> >> > That is disingenuous, I think.  Can this raise an AttributeError?
> >> >
> >> >     spam?.eggs?.bacon
> >> >
> >> > Of course it can! And this is exactly the pattern used in many
> examples
> >> > in
> >> > the PEP and the discussion. So the PEP would create a situation where
> >> > code
> >> > will raise AttributeError in a slightly?and subtly?different set of
> >> > circumstances than plain attribute access will.
> >>
> >> I don't understand. If it were to raise AttributeError, it would be
> >> because spam (or spam.eggs) isn't None, but doesn't have an attribute
> >> eggs (or bacon). Exactly the same as regular attribute access. How is
> >> it slightly different? Have I missed something?
> >
> >
> > That was my reaction, too.
> >
> >     food = spam?.eggs?.bacon
> >
> > Can be rewritten as:
> >
> >     food = spam
> >     if spam is not None and spam.eggs is not None:
> >         food = spam.eggs.bacon
> >
> > They both behave identically, no? Maybe I missed the point David was
> trying
> > to make.
>
> Aside from questions of repeated evaluation/assignment, yes. The broad
> semantics should be the same.
>
> (If you want to get technical, "spam" gets evaluated exactly once,
> "spam.eggs" a maximum of once, and "food" gets assigned exactly once.
> Your equivalent may evaluate and assign multiple times.)
>
> ChrisA
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180725/5ff2bb3c/attachment.html>

From rosuav at gmail.com  Wed Jul 25 22:28:08 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 26 Jul 2018 12:28:08 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4Zc7Q63ZC1wXXPxWdytQDbdZBt0cPK_LCxTpskzyW8_Lg@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAPTjJmrYhCtvLPGMb6==QKsr3ODXeaERiENnCdgeB+1ABNGp5A@mail.gmail.com>
 <CAEbHw4Zc7Q63ZC1wXXPxWdytQDbdZBt0cPK_LCxTpskzyW8_Lg@mail.gmail.com>
Message-ID: <CAPTjJmoxJze7nfLLaxWfse3O=+gkmYXRDma2Y2m8eZ2G29UO0Q@mail.gmail.com>

On Thu, Jul 26, 2018 at 12:14 PM, David Mertz <mertz at gnosis.cx> wrote:
> So now at least TWO proponents of 505 cannot successfully translate a very
> simple example taken almost directly from the PEP!
>
> Is that REALLY a good argument for it being helpful, and not being a bug
> magnet?!
>

Okay. I'll give you the ACTUAL transformation.

food = spam?.eggs?.bacon

can be rewritten as

_tmp = spam
if _tmp is not None:
    _tmp = _tmp.eggs
    if _tmp is not None:
        _tmp = _tmp.bacon
food = _tmp

Except for the fact that _tmp doesn't exist. Do you see why we accept
"mostly-correct" transformations? It is *actually impossible* to
perfectly represent short-circuiting semantics in Python! And before
you go "well that proves my point, this suggestion is bad", let's
apply the same test to a few other pieces of syntax. Rewrite the
following statements without using the syntactic feature named in the
comment:

# 1) Decorators
@deco
def func():
    ...

# 2) "yield from"
def chain(*iters):
    for iter in iters:
        yield from iter

# 3) and the big one: generator expressions
# yes, I'm deliberately using x multiple ways here
def f(x): return x*x
x = range(10)
x = (f(x) for x in x if x % 2)

I can pretty much guarantee you that you'll get these at least
slightly wrong. Even experts will get genexps wrong. The most pedantic
will put caveats on their equivalencies (like where I said "_tmp
doesn't exist"), but chances are you won't even notice the
discrepancies.

So if that makes ?. bad, it makes decorators, yield from, and genexps
far FAR worse. We'd better go back to Python 2.3, before they all
existed.

ChrisA

From mertz at gnosis.cx  Wed Jul 25 22:30:13 2018
From: mertz at gnosis.cx (David Mertz)
Date: Wed, 25 Jul 2018 22:30:13 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4Zc7Q63ZC1wXXPxWdytQDbdZBt0cPK_LCxTpskzyW8_Lg@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAPTjJmrYhCtvLPGMb6==QKsr3ODXeaERiENnCdgeB+1ABNGp5A@mail.gmail.com>
 <CAEbHw4Zc7Q63ZC1wXXPxWdytQDbdZBt0cPK_LCxTpskzyW8_Lg@mail.gmail.com>
Message-ID: <CAEbHw4agWFkhHrySoJHMVJHrGMxKN-dm=CAbCmKMgf0Dwr41cA@mail.gmail.com>

Btw. Here's a way of spelling the proposed syntax that gets the semantics
right:

>>> # pip install coalescing
>>> NullCoalesce(spam).eggs.bacon

On Wed, Jul 25, 2018 at 10:14 PM David Mertz <mertz at gnosis.cx> wrote:

> So now at least TWO proponents of 505 cannot successfully translate a very
> simple example taken almost directly from the PEP!
>
> Is that REALLY a good argument for it being helpful, and not being a bug
> magnet?!
>
> On Wed, Jul 25, 2018 at 9:57 PM Chris Angelico <rosuav at gmail.com> wrote:
>
>> On Thu, Jul 26, 2018 at 11:45 AM, Nicholas Chammas
>> <nicholas.chammas at gmail.com> wrote:
>> > On Wed, Jul 25, 2018 at 9:20 PM Chris Angelico <rosuav at gmail.com>
>> wrote:
>> >>
>> >> On Thu, Jul 26, 2018 at 11:02 AM, David Mertz <mertz at gnosis.cx> wrote:
>> >> > That is disingenuous, I think.  Can this raise an AttributeError?
>> >> >
>> >> >     spam?.eggs?.bacon
>> >> >
>> >> > Of course it can! And this is exactly the pattern used in many
>> examples
>> >> > in
>> >> > the PEP and the discussion. So the PEP would create a situation where
>> >> > code
>> >> > will raise AttributeError in a slightly?and subtly?different set of
>> >> > circumstances than plain attribute access will.
>> >>
>> >> I don't understand. If it were to raise AttributeError, it would be
>> >> because spam (or spam.eggs) isn't None, but doesn't have an attribute
>> >> eggs (or bacon). Exactly the same as regular attribute access. How is
>> >> it slightly different? Have I missed something?
>> >
>> >
>> > That was my reaction, too.
>> >
>> >     food = spam?.eggs?.bacon
>> >
>> > Can be rewritten as:
>> >
>> >     food = spam
>> >     if spam is not None and spam.eggs is not None:
>> >         food = spam.eggs.bacon
>> >
>> > They both behave identically, no? Maybe I missed the point David was
>> trying
>> > to make.
>>
>> Aside from questions of repeated evaluation/assignment, yes. The broad
>> semantics should be the same.
>>
>> (If you want to get technical, "spam" gets evaluated exactly once,
>> "spam.eggs" a maximum of once, and "food" gets assigned exactly once.
>> Your equivalent may evaluate and assign multiple times.)
>>
>> ChrisA
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> --
> Keeping medicines from the bloodstreams of the sick; food
> from the bellies of the hungry; books from the hands of the
> uneducated; technology from the underdeveloped; and putting
> advocates of freedom in prisons.  Intellectual property is
> to the 21st century what the slave trade was to the 16th.
>


-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180725/65c45eea/attachment.html>

From rosuav at gmail.com  Wed Jul 25 22:40:30 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 26 Jul 2018 12:40:30 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4agWFkhHrySoJHMVJHrGMxKN-dm=CAbCmKMgf0Dwr41cA@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAPTjJmrYhCtvLPGMb6==QKsr3ODXeaERiENnCdgeB+1ABNGp5A@mail.gmail.com>
 <CAEbHw4Zc7Q63ZC1wXXPxWdytQDbdZBt0cPK_LCxTpskzyW8_Lg@mail.gmail.com>
 <CAEbHw4agWFkhHrySoJHMVJHrGMxKN-dm=CAbCmKMgf0Dwr41cA@mail.gmail.com>
Message-ID: <CAPTjJmp25T_wVA_E_7zNwEFxcfrrLOMB8t0as3k7cenV+gZGTA@mail.gmail.com>

On Thu, Jul 26, 2018 at 12:30 PM, David Mertz <mertz at gnosis.cx> wrote:
> Btw. Here's a way of spelling the proposed syntax that gets the semantics
> right:
>
>>>> # pip install coalescing
>>>> NullCoalesce(spam).eggs.bacon

Let's try it.

rosuav at sikorsky:~$ sudo python3 -m pip install coalescing
Collecting coalescing
  Downloading https://files.pythonhosted.org/packages/f3/f4/120f04cc59f9fa8c55c711b67f1c9c34d8a59c34cd69249e6ff61b098987/coalescing-0.1.1.tar.gz
Installing collected packages: coalescing
  Running setup.py install for coalescing ... done
Successfully installed coalescing-0.1.1

rosuav at sikorsky:~$ python3
Python 3.8.0a0 (heads/literal_eval-exception:ddcb2eb331, Feb 21 2018, 04:32:23)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from coalescing import NullCoalesce
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'coalescing'
>>> from coalesce import NullCoalesce
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/site-packages/coalesce.py", line 56,
in <module>
    import wrapt
ModuleNotFoundError: No module named 'wrapt'

A bit problematic. But after (a) figuring out that your module is
named "coalesce" even though I installed "coalescing" AND (b) going
and separately installing wrapt, and finally (c) doing the import that
you didn't mention, we still have this fundamental problem:

rosuav at sikorsky:~$ python3
Python 3.8.0a0 (heads/literal_eval-exception:ddcb2eb331, Feb 21 2018, 04:32:23)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from coalesce import NullCoalesce
>>> from types import SimpleNamespace
>>> spam, spam.eggs, spam.eggs.bacon = SimpleNamespace(), SimpleNamespace(), 42
>>> NullCoalesce(spam).eggs.bacon
<NullCoalesce proxy for 42>

That isn't 42. That's a thing that, forever afterwards, will be a
proxy. And look at this:

>>> spam.nil = None
>>> print(NullCoalesce(spam).nil)
<NullCoalesce proxy for None>
>>> print(NullCoalesce(spam).nil.nil)
None
>>> print(NullCoalesce(spam).nil.nil.nil)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'nil'
>>>

Whoooooops.

So, no, this is most definitely NOT equivalent to the proposed semantics.

ChrisA

From mertz at gnosis.cx  Wed Jul 25 22:45:04 2018
From: mertz at gnosis.cx (David Mertz)
Date: Wed, 25 Jul 2018 22:45:04 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPTjJmoxJze7nfLLaxWfse3O=+gkmYXRDma2Y2m8eZ2G29UO0Q@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAPTjJmrYhCtvLPGMb6==QKsr3ODXeaERiENnCdgeB+1ABNGp5A@mail.gmail.com>
 <CAEbHw4Zc7Q63ZC1wXXPxWdytQDbdZBt0cPK_LCxTpskzyW8_Lg@mail.gmail.com>
 <CAPTjJmoxJze7nfLLaxWfse3O=+gkmYXRDma2Y2m8eZ2G29UO0Q@mail.gmail.com>
Message-ID: <CAEbHw4a+Nt0pd1RHbuVRQO=GktEb9cY=PJXMoz-7OA4P2YeJoA@mail.gmail.com>

On Wed, Jul 25, 2018 at 10:29 PM Chris Angelico <rosuav at gmail.com> wrote:

> food = spam?.eggs?.bacon
>
> can be rewritten as
>
> _tmp = spam
> if _tmp is not None:
>     _tmp = _tmp.eggs
>     if _tmp is not None:
>         _tmp = _tmp.bacon
> food = _tmp
>

Yes, that looks right.  Well, you need a `del _tmp` at the end; but it's
almost right.  My point was that both you and Nicholas Chammas failed to
recognize that the other translation was wrong... I recognize it does
something "kinda similar."  But the semantics of the operators are just
plain hard to grok, even by their strongest advocates.

I can write lots of things that are "mostly correct" already in Python.
Most easily, I can write:

try:
   food = spam.eggs.bacon
except:
  food = None


That does what is actually needed about 95% of the time.  It's also clear
and easy to understand.

It is *actually impossible* to
> perfectly represent short-circuiting semantics in Python!


It's INCREDIBLY EASY to represent short-circuiting semantics in Python!
What on earth are you talking about?  That's what the if/elif/else blocks
do.

And before you go "well that proves my point, this suggestion is bad", let's
> apply the same test to a few other pieces of syntax. Rewrite the
> following statements without using the syntactic feature named in the
> comment:
>

This is childishly simple:


> # 1) Decorators
> @deco
> def func():
>     ...
>

def func():
   ...
func = deco(func)


OK, this one is harder.  The "mostly correct" version is easy.  But the
actual full version is nuanced (see
https://www.python.org/dev/peps/pep-0380/ for details).

# 2) "yield from"
> def chain(*iters):
>     for iter in iters:
>         yield from iter
>

# The simple approximation:
for iter in iters:

    for _ in iter:

        yield iter



> # 3) and the big one: generator expressions
> # yes, I'm deliberately using x multiple ways here
> def f(x): return x*x
> x = range(10)
> x = (f(x) for x in x if x % 2)
>

I'm not going to bother with that.  I'd fire anyone who wrote it, after
code review.  Minus the abuse of names, it's just:

def gen(xs):

    for x in xs:

        if x % 2:

            yield f(x)

x = gen(xs)



-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180725/098989b4/attachment-0001.html>

From nicholas.chammas at gmail.com  Wed Jul 25 22:49:42 2018
From: nicholas.chammas at gmail.com (Nicholas Chammas)
Date: Wed, 25 Jul 2018 22:49:42 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4Yzt1Wfr1iBw+mmTYWhh8b2SwyFLX1AHOL-4YP6H+_ChA@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAEbHw4Yzt1Wfr1iBw+mmTYWhh8b2SwyFLX1AHOL-4YP6H+_ChA@mail.gmail.com>
Message-ID: <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>

On Wed, Jul 25, 2018 at 10:12 PM David Mertz <mertz at gnosis.cx> wrote:

> On Wed, Jul 25, 2018 at 9:47 PM Nicholas Chammas <
> nicholas.chammas at gmail.com> wrote:
>
>> > That is disingenuous, I think.  Can this raise an AttributeError?
>>> >     spam?.eggs?.bacon
>>> > Of course it can! And this is exactly the pattern used in many
>>> examples in
>>> > the PEP and the discussion. So the PEP would create a situation where
>>> code
>>> > will raise AttributeError in a slightly?and subtly?different set of
>>> > circumstances than plain attribute access will.
>>>
>>
>
>>     food = spam?.eggs?.bacon
>> Can be rewritten as:
>>     food = spam
>>     if spam is not None and spam.eggs is not None:
>>         food = spam.eggs.bacon
>> They both behave identically, no? Maybe I missed the point David was
>> trying to make.
>>
>
> No, you illustrate it perfectly! I had to stare at your translation for a
> while to decide if it was really identical to the proposed
> `spam?.eggs?.bacon`.  The fact I have to think so hard makes the syntax
> feel non-obvious.
>
> Plus, there's the fact that your best effort at translating the proposed
> syntax is WRONG.  Even a strong proponent cannot explain the behavior on a
> first try.  And indeed, it behaves subtly different from plain attribute
> access in where it raises AttributeError.
>
> >>> spam = SimpleNamespace()
> >>> spam.eggs = None
> >>> spam.eggs.bacon
> AttributeError: 'NoneType' object has no attribute 'bacon'
>
> >>> # spam?.eggs?.bacon
> >>> # Should be: None
>
> >>> "Translation" does something different
> >>> food = spam
> >>> if spam is not None and spam.eggs is not None:
> ...     food = spam.eggs.bacon
> >>> food
> namespace(eggs=None)
>

Indeed. Thanks for the counter-example. I think the correct translation is
as follows:

    food = spam?.eggs?.bacon

Becomes:

    food = None
    if spam is not None and spam.eggs is not None:
        food = spam.eggs.bacon

Did I get it right now? :)

What misled me was this example from the PEP
<https://www.python.org/dev/peps/pep-0505/#the-maybe-dot-and-maybe-subscript-operators>
showing
how atoms are evaluated. The breakdown begins with `_v = a`, so I copied
that pattern incorrectly when trying to explain how an example assignment
would work, instead of translating directly from what I understood the PEP
505 variant should do.

So, shame on me. I think this particular mistake reflects more on me than
on PEP 505, but I see how this kind of mistake reflects badly on the folks
advocating for the PEP (or at least, playing devil's advocate).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180725/c5590029/attachment.html>

From mertz at gnosis.cx  Wed Jul 25 22:56:57 2018
From: mertz at gnosis.cx (David Mertz)
Date: Wed, 25 Jul 2018 22:56:57 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPTjJmp25T_wVA_E_7zNwEFxcfrrLOMB8t0as3k7cenV+gZGTA@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAPTjJmrYhCtvLPGMb6==QKsr3ODXeaERiENnCdgeB+1ABNGp5A@mail.gmail.com>
 <CAEbHw4Zc7Q63ZC1wXXPxWdytQDbdZBt0cPK_LCxTpskzyW8_Lg@mail.gmail.com>
 <CAEbHw4agWFkhHrySoJHMVJHrGMxKN-dm=CAbCmKMgf0Dwr41cA@mail.gmail.com>
 <CAPTjJmp25T_wVA_E_7zNwEFxcfrrLOMB8t0as3k7cenV+gZGTA@mail.gmail.com>
Message-ID: <CAEbHw4a+5hK1p4ZEsEXkyKSm91ZoVVPO_JwThv+j3T65-gvvJg@mail.gmail.com>

On Wed, Jul 25, 2018 at 10:41 PM Chris Angelico <rosuav at gmail.com> wrote:

> A bit problematic. But after (a) figuring out that your module is
> named "coalesce" even though I installed "coalescing" AND (b) going
> and separately installing wrapt, and finally (c) doing the import that
> you didn't mention, we still have this fundamental problem:
>

Yeah, yeah.  I know it's alpha software I wrote two nights ago, and
slightly patched 5 minutes before that post.  You fixed those concerns;
I'll happily take PRs on fixing them better.


> >>> from coalesce import NullCoalesce
> >>> from types import SimpleNamespace
> >>> spam, spam.eggs, spam.eggs.bacon = SimpleNamespace(),
> SimpleNamespace(), 42
> >>> NullCoalesce(spam).eggs.bacon
> <NullCoalesce proxy for 42>
>
> That isn't 42. That's a thing that, forever afterwards, will be a
> proxy.


Yeah.  That's a thing it does.  It's less of an issue than you think since,
e.g.:

>>> from coalesce import NullCoalesce
>>> from types import SimpleNamespace
>>> spam, spam.eggs, spam.eggs.bacon = SimpleNamespace(),
SimpleNamespace(), 42
>>> NullCoalesce(spam).eggs.bacon + 1
43
>>> NullCoalesce(spam).eggs.bacon * 1
42


Most things you actually do with the proxy wind up getting the value back
once it is used.  However, this seems to be a bug that I inherit from
wrapt.ObjectProxy:

>>> NullCoalesce(spam).eggs.bacon + 0
ValueError: wrapper has not been initialized


If you do an operation that combines the proxy value with "Falsey" values,
it doesn't do the implicit unboxing.  Same with e.g. `proxyval + ""` for
strings, unfortunately.  I'm not sure how to fix that.

>>> spam.nil = None
> >>> print(NullCoalesce(spam).nil.nil)
> None
> >>> print(NullCoalesce(spam).nil.nil.nil)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> AttributeError: 'NoneType' object has no attribute 'nil'
>

Why is this wrong? This is EXACTLY the same behavior as the `?.` operator
would have in the last case.  Do you not recognize the behavior you are
advocating in PEP 505?

I recognize that the proxy value not always "auto-unboxing" is a limitation
that I don't like.

-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180725/e25b4026/attachment.html>

From rosuav at gmail.com  Wed Jul 25 23:04:13 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 26 Jul 2018 13:04:13 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4a+Nt0pd1RHbuVRQO=GktEb9cY=PJXMoz-7OA4P2YeJoA@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAPTjJmrYhCtvLPGMb6==QKsr3ODXeaERiENnCdgeB+1ABNGp5A@mail.gmail.com>
 <CAEbHw4Zc7Q63ZC1wXXPxWdytQDbdZBt0cPK_LCxTpskzyW8_Lg@mail.gmail.com>
 <CAPTjJmoxJze7nfLLaxWfse3O=+gkmYXRDma2Y2m8eZ2G29UO0Q@mail.gmail.com>
 <CAEbHw4a+Nt0pd1RHbuVRQO=GktEb9cY=PJXMoz-7OA4P2YeJoA@mail.gmail.com>
Message-ID: <CAPTjJmpHGm325VsMAaJj+u3jzsLPmf_2kmNmZv7dQ5vDkPWUiw@mail.gmail.com>

On Thu, Jul 26, 2018 at 12:45 PM, David Mertz <mertz at gnosis.cx> wrote:
> On Wed, Jul 25, 2018 at 10:29 PM Chris Angelico <rosuav at gmail.com> wrote:
>> It is *actually impossible* to
>> perfectly represent short-circuiting semantics in Python!
>
>
> It's INCREDIBLY EASY to represent short-circuiting semantics in Python! What
> on earth are you talking about?  That's what the if/elif/else blocks do.

Except for the aforementioned "single lookup, single assignment"
semantics. You can't achieve that with an if/else block.

>> And before you go "well that proves my point, this suggestion is bad",
>> let's
>> apply the same test to a few other pieces of syntax. Rewrite the
>> following statements without using the syntactic feature named in the
>> comment:
>
>
> This is childishly simple:
>
>>
>> # 1) Decorators
>> @deco
>> def func():
>>     ...
>
>
> def func():
>    ...
> func = deco(func)

Okay. Try this then:

def deco(f):
    print(globals()[f.__name__])
    return f
func = "funky"
@deco
def func(): pass

Childishly simple? Or does it have its own subtleties? You might think
this is trivial and pointless, but consider the case of a writable
property:

class X:
    @property
    def spam(self): return 42
    @spam.setter
    def spam(self, val): print("Setting spam to", val)

This version won't work:

class X:
    def spam(self): return 42
    spam = property(spam)
    def spam(self, val): print("Setting spam to", val)
    spam = spam.setter(spam)

See how easy it is to create an imperfect representation?

>> # 3) and the big one: generator expressions
>> # yes, I'm deliberately using x multiple ways here
>> def f(x): return x*x
>> x = range(10)
>> x = (f(x) for x in x if x % 2)
>
>
> I'm not going to bother with that.  I'd fire anyone who wrote it, after code
> review.  Minus the abuse of names, it's just:
>
> def gen(xs):
>
>     for x in xs:
>
>         if x % 2:
>
>             yield f(x)
>
> x = gen(xs)

Modulo a TON of subtleties about exactly what gets evaluated when.
Again, you've made an imperfect equivalence. So if you're going to
complain about people making ?. equivalences that aren't perfect, make
sure you're just as pedantic about existing syntax. You scored one out
of three, and only by punting to the existing PEP.

ChrisA

From rosuav at gmail.com  Wed Jul 25 23:07:12 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 26 Jul 2018 13:07:12 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4a+5hK1p4ZEsEXkyKSm91ZoVVPO_JwThv+j3T65-gvvJg@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAPTjJmrYhCtvLPGMb6==QKsr3ODXeaERiENnCdgeB+1ABNGp5A@mail.gmail.com>
 <CAEbHw4Zc7Q63ZC1wXXPxWdytQDbdZBt0cPK_LCxTpskzyW8_Lg@mail.gmail.com>
 <CAEbHw4agWFkhHrySoJHMVJHrGMxKN-dm=CAbCmKMgf0Dwr41cA@mail.gmail.com>
 <CAPTjJmp25T_wVA_E_7zNwEFxcfrrLOMB8t0as3k7cenV+gZGTA@mail.gmail.com>
 <CAEbHw4a+5hK1p4ZEsEXkyKSm91ZoVVPO_JwThv+j3T65-gvvJg@mail.gmail.com>
Message-ID: <CAPTjJmq70XuWPND3J4YEAhioBJ1oZYiSR=tZM3Ypu59wYjA9HA@mail.gmail.com>

On Thu, Jul 26, 2018 at 12:56 PM, David Mertz <mertz at gnosis.cx> wrote:
> On Wed, Jul 25, 2018 at 10:41 PM Chris Angelico <rosuav at gmail.com> wrote:
>>
>> A bit problematic. But after (a) figuring out that your module is
>> named "coalesce" even though I installed "coalescing" AND (b) going
>> and separately installing wrapt, and finally (c) doing the import that
>> you didn't mention, we still have this fundamental problem:
>
>
> Yeah, yeah.  I know it's alpha software I wrote two nights ago, and slightly
> patched 5 minutes before that post.  You fixed those concerns; I'll happily
> take PRs on fixing them better.

PRs? Nope. I don't think it's possible to do this with correct
semantics without language support.

>> >>> spam.nil = None
>> >>> print(NullCoalesce(spam).nil.nil)
>> None
>> >>> print(NullCoalesce(spam).nil.nil.nil)
>> Traceback (most recent call last):
>>   File "<stdin>", line 1, in <module>
>> AttributeError: 'NoneType' object has no attribute 'nil'
>
>
> Why is this wrong? This is EXACTLY the same behavior as the `?.` operator
> would have in the last case.  Do you not recognize the behavior you are
> advocating in PEP 505?

With PEP 505, I could write this:

>>> spam = SimpleNamespace()
>>> spam.nil = None
>>> print(spam?.nil)
None
>>> print(spam?.nil?.nil)
None
>>> print(spam?.nil?.nil?.nil)
None
>>> print(spam?.nil?.nil?.nil?.nil)
None

Can you do that with your proxy?

ChrisA

From mertz at gnosis.cx  Wed Jul 25 23:09:35 2018
From: mertz at gnosis.cx (David Mertz)
Date: Wed, 25 Jul 2018 23:09:35 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAEbHw4Yzt1Wfr1iBw+mmTYWhh8b2SwyFLX1AHOL-4YP6H+_ChA@mail.gmail.com>
 <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>
Message-ID: <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>

On Wed, Jul 25, 2018 at 10:50 PM Nicholas Chammas <
nicholas.chammas at gmail.com> wrote:

> Indeed. Thanks for the counter-example. I think the correct translation is
> as follows:
>     food = spam?.eggs?.bacon
> Becomes:
>     food = None
>     if spam is not None and spam.eggs is not None:
>         food = spam.eggs.bacon
>
Did I get it right now? :)
>

Nope, still not right, I'm afraid!

Chris Angelica provided a more accurate translation.  Do you not see that
the fact that your *second* try at understanding the actual behavior is
still wrong suggest that this operator is a HUGE bug magnet?!


> So, shame on me. I think this particular mistake reflects more on me than
> on PEP 505, but I see how this kind of mistake reflects badly on the folks
> advocating for the PEP (or at least, playing devil's advocate).
>

I really, really don't.  I think you see an intuitive behavior that would
be nice and useful in a certain area.  That behavior just isn't what the
PEP proposes though... it's kinda-sorta close enough to be lured into
thinking it's a good idea.

Honestly, I think the behavior of GreedyAccess in my little library I wrote
over the last couple nights is FAR more often what programmers ACTUALLY
want than NullCoalesce is.  Even Steve Dower?in the PEP and in this
discussion?acknowledges the appeal and utility of the GreedyAccess
behavior.  It's in the "Rejected Ideas" section, which is fair enough.

But in a library like mine... or indeed, in a much better library that you
or someone else writes... it's perfectly easy to have both classes, and
choose which behavior is more useful for your case.  A new syntax feature
can't let user decide which behavior (or maybe some other behavior
altogether) is most useful for their specific case.  A library does that
easily[*].

[*] In version 0.1.1 of coalescing?changed from 0.1?I added the option to
use a sentinel other than None if you want.  I'm not sure how useful that
is, but that idea was in some old PEPs, and I think in the Rejected Ideas
of 505.  With a library, I have a parameter that need not be used to switch
that[**].  E.g.:

NullCoalesce(foo, sentinel=float('nan')).bar.baz.blam

[**] Yes, I even handle NaN's in a special way because they are non-equal
even to themselves.  You could use empty string, or 0, or my_null =
object(), or whatever.
-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180725/928356fa/attachment.html>

From mertz at gnosis.cx  Wed Jul 25 23:19:58 2018
From: mertz at gnosis.cx (David Mertz)
Date: Wed, 25 Jul 2018 23:19:58 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPTjJmq70XuWPND3J4YEAhioBJ1oZYiSR=tZM3Ypu59wYjA9HA@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAPTjJmrYhCtvLPGMb6==QKsr3ODXeaERiENnCdgeB+1ABNGp5A@mail.gmail.com>
 <CAEbHw4Zc7Q63ZC1wXXPxWdytQDbdZBt0cPK_LCxTpskzyW8_Lg@mail.gmail.com>
 <CAEbHw4agWFkhHrySoJHMVJHrGMxKN-dm=CAbCmKMgf0Dwr41cA@mail.gmail.com>
 <CAPTjJmp25T_wVA_E_7zNwEFxcfrrLOMB8t0as3k7cenV+gZGTA@mail.gmail.com>
 <CAEbHw4a+5hK1p4ZEsEXkyKSm91ZoVVPO_JwThv+j3T65-gvvJg@mail.gmail.com>
 <CAPTjJmq70XuWPND3J4YEAhioBJ1oZYiSR=tZM3Ypu59wYjA9HA@mail.gmail.com>
Message-ID: <CAEbHw4Z3AVp6oP_UDhyM4kEPBKWD-F8bWwWs0gjHB9uGP0ERrg@mail.gmail.com>

On Wed, Jul 25, 2018 at 11:08 PM Chris Angelico <rosuav at gmail.com> wrote:

> > Yeah, yeah.  I know it's alpha software I wrote two nights ago, and
> slightly
> > patched 5 minutes before that post.  You fixed those concerns; I'll
> happily
> > take PRs on fixing them better.
>
> PRs? Nope. I don't think it's possible to do this with correct
> semantics without language support.
>
> With PEP 505, I could write this:
>
> >>> print(spam?.nil?.nil?.nil?.nil)
> None
>
> Can you do that with your proxy?
>

Oh yeah.  You are right! Thank you.

That's a bug in my proxy too.  I'll figure out how to fix it in 0.1.2
soon.  This is early alpha, and the things you're noting are valuable bug
reports.  But none of this is fundamentally unfixable in a library, nor
even especially difficult.

I doubt I'll ever even use my own software.  It's just a proof-of-concept
that we can achieve the ACTUAL purpose of PEP 505 with no language
changes.  I don't very often have a need to solve the problem PEP 505
does... even though I very often work in the very domain it is intended to
address (semi-structured nested data).  Even if the PEP could be a little
bit more elegant for a very few circumstances, it's just not anywhere close
to deserving syntax... especially not syntax that even proponents tend to
misunderstand the semantics of.

-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180725/8892fbaa/attachment-0001.html>

From rosuav at gmail.com  Wed Jul 25 23:24:50 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 26 Jul 2018 13:24:50 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4Z3AVp6oP_UDhyM4kEPBKWD-F8bWwWs0gjHB9uGP0ERrg@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAPTjJmrYhCtvLPGMb6==QKsr3ODXeaERiENnCdgeB+1ABNGp5A@mail.gmail.com>
 <CAEbHw4Zc7Q63ZC1wXXPxWdytQDbdZBt0cPK_LCxTpskzyW8_Lg@mail.gmail.com>
 <CAEbHw4agWFkhHrySoJHMVJHrGMxKN-dm=CAbCmKMgf0Dwr41cA@mail.gmail.com>
 <CAPTjJmp25T_wVA_E_7zNwEFxcfrrLOMB8t0as3k7cenV+gZGTA@mail.gmail.com>
 <CAEbHw4a+5hK1p4ZEsEXkyKSm91ZoVVPO_JwThv+j3T65-gvvJg@mail.gmail.com>
 <CAPTjJmq70XuWPND3J4YEAhioBJ1oZYiSR=tZM3Ypu59wYjA9HA@mail.gmail.com>
 <CAEbHw4Z3AVp6oP_UDhyM4kEPBKWD-F8bWwWs0gjHB9uGP0ERrg@mail.gmail.com>
Message-ID: <CAPTjJmqQXTtEUjJQkaFBP=q+94OLw8v0xGtKVnO0Xam773W1Mw@mail.gmail.com>

On Thu, Jul 26, 2018 at 1:19 PM, David Mertz <mertz at gnosis.cx> wrote:
> On Wed, Jul 25, 2018 at 11:08 PM Chris Angelico <rosuav at gmail.com> wrote:
>>
>> > Yeah, yeah.  I know it's alpha software I wrote two nights ago, and
>> > slightly
>> > patched 5 minutes before that post.  You fixed those concerns; I'll
>> > happily
>> > take PRs on fixing them better.
>>
>> PRs? Nope. I don't think it's possible to do this with correct
>> semantics without language support.
>>
>> With PEP 505, I could write this:
>>
>> >>> print(spam?.nil?.nil?.nil?.nil)
>> None
>>
>> Can you do that with your proxy?
>
>
> Oh yeah.  You are right! Thank you.
>
> That's a bug in my proxy too.  I'll figure out how to fix it in 0.1.2 soon.
> This is early alpha, and the things you're noting are valuable bug reports.
> But none of this is fundamentally unfixable in a library, nor even
> especially difficult.

If you're going to make that work, then by definition you would be
wrapping up the None, right? Which would mean that every one of the
prints would say "<NullCoalesce proxy for None>". Which, in turn,
means that you cannot do this:

>>> NullCoalesce(spam).nil is None

or rather, it will always be False. With PEP 505, you most certainly
*can* do this check, because it would really truly be None.

This IS fundamentally unfixable in a library.

ChrisA

From david.mertz at gmail.com  Wed Jul 25 23:53:59 2018
From: david.mertz at gmail.com (David Mertz)
Date: Wed, 25 Jul 2018 23:53:59 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPTjJmqQXTtEUjJQkaFBP=q+94OLw8v0xGtKVnO0Xam773W1Mw@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAPTjJmrYhCtvLPGMb6==QKsr3ODXeaERiENnCdgeB+1ABNGp5A@mail.gmail.com>
 <CAEbHw4Zc7Q63ZC1wXXPxWdytQDbdZBt0cPK_LCxTpskzyW8_Lg@mail.gmail.com>
 <CAEbHw4agWFkhHrySoJHMVJHrGMxKN-dm=CAbCmKMgf0Dwr41cA@mail.gmail.com>
 <CAPTjJmp25T_wVA_E_7zNwEFxcfrrLOMB8t0as3k7cenV+gZGTA@mail.gmail.com>
 <CAEbHw4a+5hK1p4ZEsEXkyKSm91ZoVVPO_JwThv+j3T65-gvvJg@mail.gmail.com>
 <CAPTjJmq70XuWPND3J4YEAhioBJ1oZYiSR=tZM3Ypu59wYjA9HA@mail.gmail.com>
 <CAEbHw4Z3AVp6oP_UDhyM4kEPBKWD-F8bWwWs0gjHB9uGP0ERrg@mail.gmail.com>
 <CAPTjJmqQXTtEUjJQkaFBP=q+94OLw8v0xGtKVnO0Xam773W1Mw@mail.gmail.com>
Message-ID: <CAEbHw4ZvoPcuEyzMaaj2ggELi7BBAaQcaGDPCteYQnc5yoe5YA@mail.gmail.com>

On Wed, Jul 25, 2018, 11:27 PM Chris Angelico <rosuav at gmail.com> wrote:

> > That's a bug in my proxy too.  I'll figure out how to fix it in 0.1.2
> soon.
> > This is early alpha, and the things you're noting are valuable bug
> reports.
> > But none of this is fundamentally unfixable in a library, nor even
> > especially difficult.
>
> If you're going to make that work, then by definition you would be
> wrapping up the None, right? Which would mean that every one of the
> prints would say "<NullCoalesce proxy for None>". Which, in turn,
> means that you cannot do this:
>
> >>> NullCoalesce(spam).nil is None
>
> or rather, it will always be False. With PEP 505, you most certainly
> *can* do this check, because it would really truly be None.
>
> This IS fundamentally unfixable in a library.
>

If your meaning of "fix" is simply "add new syntax", of course that's true.
If it's "solve the actual problem that motivates the PEP" I can definitely
fix it.

Right now, you can still always call .unbox() at the end to get the
underlying value. I agree that's a little ugly, hence why I added the wrapt
proxy stuff. Most operations trigger unboxing, but indeed not simply
echoing to the shell.

I think I'll add a property spelled '_' to make it less busy (but more
cryptic, I know). E.g.

NullCoalesce(spam).nil.nil.nil._ is None'

I also added an unbox() function that will pass through non-proxy objects
untouched. So it's safe to unbox anything if you are unsure what it is.

Of course I'm not claiming any library is without a need to work with it's
quirks and limits. But they are very few even in this 4-hours-of-work alpha.

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180725/cc9427e6/attachment.html>

From nicholas.chammas at gmail.com  Thu Jul 26 00:00:01 2018
From: nicholas.chammas at gmail.com (Nicholas Chammas)
Date: Thu, 26 Jul 2018 00:00:01 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAEbHw4Yzt1Wfr1iBw+mmTYWhh8b2SwyFLX1AHOL-4YP6H+_ChA@mail.gmail.com>
 <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>
 <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
Message-ID: <CAOhmDzf_noqLS-_TvZtzhuREU_g-FOjTyDMiOGpN4p8VksbOfg@mail.gmail.com>

On Wed, Jul 25, 2018 at 11:09 PM David Mertz <mertz at gnosis.cx> wrote:

> On Wed, Jul 25, 2018 at 10:50 PM Nicholas Chammas <
> nicholas.chammas at gmail.com> wrote:
>
>> Indeed. Thanks for the counter-example. I think the correct translation
>> is as follows:
>>     food = spam?.eggs?.bacon
>> Becomes:
>>     food = None
>>     if spam is not None and spam.eggs is not None:
>>         food = spam.eggs.bacon
>>
> Did I get it right now? :)
>>
>
> Nope, still not right, I'm afraid!
>
> Chris Angelica provided a more accurate translation.
>

Forgive me for being slow. I'm missing what's different in semantics
between the translation above and Chris's translation below:

_tmp = spam
> if _tmp is not None:
>     _tmp = _tmp.eggs
>     if _tmp is not None:
>         _tmp = _tmp.bacon
> food = _tmp
>

What's a case where they would do something different?

* If spam is None, they work the same -> None
* If spam is not None, but spam.eggs exists and is None, they work the same
-> None
* If spam is not None, but spam.eggs doesn't exist, they work the same ->
AttributeError
* If spam is not None, and spam.eggs is not None, but spam.eggs.bacon is
None, they work the same -> None
* If spam is not None, and spam.eggs is not None, but spam.eggs.bacon
doesn't exist, they work the same -> AttributeError
* If spam is not None, and spam.eggs is not None, and spam.eggs.bacon is
not None, they work the same -> bacon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180726/da0310aa/attachment-0001.html>

From mertz at gnosis.cx  Thu Jul 26 00:03:15 2018
From: mertz at gnosis.cx (David Mertz)
Date: Thu, 26 Jul 2018 00:03:15 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4ZvoPcuEyzMaaj2ggELi7BBAaQcaGDPCteYQnc5yoe5YA@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAPTjJmrYhCtvLPGMb6==QKsr3ODXeaERiENnCdgeB+1ABNGp5A@mail.gmail.com>
 <CAEbHw4Zc7Q63ZC1wXXPxWdytQDbdZBt0cPK_LCxTpskzyW8_Lg@mail.gmail.com>
 <CAEbHw4agWFkhHrySoJHMVJHrGMxKN-dm=CAbCmKMgf0Dwr41cA@mail.gmail.com>
 <CAPTjJmp25T_wVA_E_7zNwEFxcfrrLOMB8t0as3k7cenV+gZGTA@mail.gmail.com>
 <CAEbHw4a+5hK1p4ZEsEXkyKSm91ZoVVPO_JwThv+j3T65-gvvJg@mail.gmail.com>
 <CAPTjJmq70XuWPND3J4YEAhioBJ1oZYiSR=tZM3Ypu59wYjA9HA@mail.gmail.com>
 <CAEbHw4Z3AVp6oP_UDhyM4kEPBKWD-F8bWwWs0gjHB9uGP0ERrg@mail.gmail.com>
 <CAPTjJmqQXTtEUjJQkaFBP=q+94OLw8v0xGtKVnO0Xam773W1Mw@mail.gmail.com>
 <CAEbHw4ZvoPcuEyzMaaj2ggELi7BBAaQcaGDPCteYQnc5yoe5YA@mail.gmail.com>
Message-ID: <CAEbHw4bHGHKHz+XsvqPf97hFd06U-fP85zL_Z03XD7pVWKu5kw@mail.gmail.com>

BTW, even for the "compare to None" issue, the situation in 0.1.1 isn't as
bad as you might think.  Definitely a "<NullCoalesce proxy for None>"
cannot be compared as "is None".  But even there, this works:

>>> NullCoalesce(spam).bacon
<NullCoalesce proxy for None>
>>> NullCoalesce(spam).bacon == None
True
>>> NullCoalesce(spam).bacon == 0
False
>>> NullCoalesce(spam).bacon is None
False


So yeah, the docs should say "If you are using the `coalescing` library,
use `==` rather than `is` to compare values with None".

Yes, that's contrary to best style in general Python... but it's a pretty
small change to make in that specialized code that needs to deal with
"hierarchical semi-structured data that uses None (rather than missing
attributes or other sentinels) to mark unreachable branches."


On Wed, Jul 25, 2018 at 11:53 PM David Mertz <david.mertz at gmail.com> wrote:

> On Wed, Jul 25, 2018, 11:27 PM Chris Angelico <rosuav at gmail.com> wrote:
>
>> > That's a bug in my proxy too.  I'll figure out how to fix it in 0.1.2
>> soon.
>> > This is early alpha, and the things you're noting are valuable bug
>> reports.
>> > But none of this is fundamentally unfixable in a library, nor even
>> > especially difficult.
>>
>> If you're going to make that work, then by definition you would be
>> wrapping up the None, right? Which would mean that every one of the
>> prints would say "<NullCoalesce proxy for None>". Which, in turn,
>> means that you cannot do this:
>>
>> >>> NullCoalesce(spam).nil is None
>>
>> or rather, it will always be False. With PEP 505, you most certainly
>> *can* do this check, because it would really truly be None.
>>
>> This IS fundamentally unfixable in a library.
>>
>
> If your meaning of "fix" is simply "add new syntax", of course that's
> true. If it's "solve the actual problem that motivates the PEP" I can
> definitely fix it.
>
> Right now, you can still always call .unbox() at the end to get the
> underlying value. I agree that's a little ugly, hence why I added the wrapt
> proxy stuff. Most operations trigger unboxing, but indeed not simply
> echoing to the shell.
>
> I think I'll add a property spelled '_' to make it less busy (but more
> cryptic, I know). E.g.
>
> NullCoalesce(spam).nil.nil.nil._ is None'
>
> I also added an unbox() function that will pass through non-proxy objects
> untouched. So it's safe to unbox anything if you are unsure what it is.
>
> Of course I'm not claiming any library is without a need to work with it's
> quirks and limits. But they are very few even in this 4-hours-of-work alpha.
>
>>

-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180726/94c2ddd1/attachment.html>

From mertz at gnosis.cx  Thu Jul 26 00:16:55 2018
From: mertz at gnosis.cx (David Mertz)
Date: Thu, 26 Jul 2018 00:16:55 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAOhmDzf_noqLS-_TvZtzhuREU_g-FOjTyDMiOGpN4p8VksbOfg@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAEbHw4Yzt1Wfr1iBw+mmTYWhh8b2SwyFLX1AHOL-4YP6H+_ChA@mail.gmail.com>
 <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>
 <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
 <CAOhmDzf_noqLS-_TvZtzhuREU_g-FOjTyDMiOGpN4p8VksbOfg@mail.gmail.com>
Message-ID: <CAEbHw4ZyeVBbhxLD9OeOrXfNq_GVfYo+r533h4P5B_TtV2JKZA@mail.gmail.com>

On Thu, Jul 26, 2018 at 12:00 AM Nicholas Chammas <
nicholas.chammas at gmail.com> wrote:

> Forgive me for being slow. I'm missing what's different in semantics
> between the translation above and Chris's translation below:
>

You are VERY close now.  You have more SPAM, so yours is better:

In [1]: class Spam:
   ...:     @property
   ...:     def eggs(self):
   ...:         print("SPAM SPAM SPAM")
   ...:         return "eggs"
   ...:
In [2]: spam = Spam()
In [3]: _tmp = spam
In [4]: if _tmp is not None:
   ...:      _tmp = _tmp.eggs
   ...:      if _tmp is not None:
   ...:          _tmp = _tmp.bacon
   ...: food = _tmp
   ...: del _tmp
   ...: food
   ...:
SPAM SPAM SPAM
---------------------------------------------------------------------
AttributeError                      Traceback (most recent call last)
<ipython-input-4-9a2f963239f8> in <module>()
      2      _tmp = _tmp.eggs
      3      if _tmp is not None:
----> 4          _tmp = _tmp.bacon
      5 food = _tmp
      6 del _tmp

AttributeError: 'str' object has no attribute 'bacon'
In [5]: if spam is not None and spam.eggs is not None:
   ...:     food = spam.eggs.bacon
   ...:
SPAM SPAM SPAM
SPAM SPAM SPAM
---------------------------------------------------------------------
AttributeError                      Traceback (most recent call last)
<ipython-input-5-a24834cdb92b> in <module>()
      1 if spam is not None and spam.eggs is not None:
----> 2     food = spam.eggs.bacon
      3

AttributeError: 'str' object has no attribute 'bacon'



-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180726/a221ebd4/attachment.html>

From raymond.hettinger at gmail.com  Thu Jul 26 00:25:44 2018
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Wed, 25 Jul 2018 21:25:44 -0700
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
Message-ID: <3F4C2CB7-ED88-4670-AD7C-C4C88125E895@gmail.com>


> On Jul 18, 2018, at 10:43 AM, Steve Dower <steve.dower at python.org> wrote:
> 
> Possibly this is exactly the wrong time to propose the next big syntax change, since we currently have nobody to declare on it, but since we're likely to argue for a while anyway it probably can't hurt (and maybe this will become the test PEP for whoever takes the reins?).

It probably is the wrong time and probably can hurt (by introducing divisiveness when we most need for be focusing on coming together).

This PEP also shares some traits with PEP 572 in that it solves a somewhat minor problem with   new syntax and grammar changes that affect the look and feel of the language in a way that at least some of us (me for example) find to be repulsive.

This PEP is one step further away from Python reading like executable pseudo-code.  That trait is currently a major draw to the language and I don't think it should get tossed away just to mitigate a minor irritant.

We should also consider a moratorium on language changes for while.  There is more going on than just a transition to a post-bdfl world.  The other implementations of Python are having a hard time keeping up with our recent, ferocious rate of change.  Even among the core developers, most people are not fully up to date learning all the new features that have already been added (how many of you are competent with typing, data classes, generalized unpacking, concurrent futures, async, the scoping rules for exceptions and comprehensions, the hundreds of niggling changes in the past few releases, __init_subclass__, __set_name__, details of import logic, issues with SSL certificates, new collections ABCs, etc.?)  We've been putting major changes in faster than anyone can keep up with them.  We really need to take a breath.


Raymond





From nicholas.chammas at gmail.com  Thu Jul 26 00:30:34 2018
From: nicholas.chammas at gmail.com (Nicholas Chammas)
Date: Thu, 26 Jul 2018 00:30:34 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4ZyeVBbhxLD9OeOrXfNq_GVfYo+r533h4P5B_TtV2JKZA@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAEbHw4Yzt1Wfr1iBw+mmTYWhh8b2SwyFLX1AHOL-4YP6H+_ChA@mail.gmail.com>
 <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>
 <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
 <CAOhmDzf_noqLS-_TvZtzhuREU_g-FOjTyDMiOGpN4p8VksbOfg@mail.gmail.com>
 <CAEbHw4ZyeVBbhxLD9OeOrXfNq_GVfYo+r533h4P5B_TtV2JKZA@mail.gmail.com>
Message-ID: <CAOhmDzeQN6igsuEuOUWasDdjnLFivorNUStHV+2A2njJTi8rTA@mail.gmail.com>

On Thu, Jul 26, 2018 at 12:17 AM David Mertz <mertz at gnosis.cx> wrote:

> On Thu, Jul 26, 2018 at 12:00 AM Nicholas Chammas <
> nicholas.chammas at gmail.com> wrote:
>
>> Forgive me for being slow. I'm missing what's different in semantics
>> between the translation above and Chris's translation below:
>>
>
> You are VERY close now.  You have more SPAM, so yours is better:
>

<snipped>

Understood. Thanks for clarifying. I think Chris was referring to this in
an earlier message (when I made my first attempt at translating) when he
said:

> Aside from questions of repeated evaluation/assignment, yes. The broad
> semantics should be the same.

So the meaning of my translation matches the "intention" of the original
PEP 505 expression, but I see the danger in offering up such loose
translations. Getting the meaning right but missing important details (like
repeated evaluation) does not inspire confidence.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180726/d2366c2d/attachment.html>

From amit.mixie at gmail.com  Thu Jul 26 01:02:47 2018
From: amit.mixie at gmail.com (Amit Green)
Date: Thu, 26 Jul 2018 01:02:47 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <3F4C2CB7-ED88-4670-AD7C-C4C88125E895@gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <3F4C2CB7-ED88-4670-AD7C-C4C88125E895@gmail.com>
Message-ID: <CAKHFnb+ta+V0THPj1WB14Uk+A9V=wMV7YfqCQ7SFpky1Czai2A@mail.gmail.com>

 On Thu, Jul 26, 2018 at 12:25 AM, Raymond Hettinger <
raymond.hettinger at gmail.com> wrote:

> It probably is the wrong time and probably can hurt (by introducing
> divisiveness when we most need for be focusing on coming together).
>
> This PEP also shares some traits with PEP 572 in that it solves a somewhat
> minor problem with   new syntax and grammar changes that affect the look
> and feel of the language in a way that at least some of us (me for example)
> find to be repulsive.
>
> This PEP is one step further away from Python reading like executable
> pseudo-code.  That trait is currently a major draw to the language and I
> don't think it should get tossed away just to mitigate a minor irritant.
>

+1.

Also this whole none-aware problem is really really complicated, so I'd
like to add a few thoughts:

1.  I spent a few days on it last year, and came to the following
conclusions:

2.  It is a *really* useful feature -- that I want in quite a lot of code
that I write.

3.  The problem is way deeper than simply adding '?.' and other operators.
For real use cases, you also need to say "how far" the an operator can
"spread" -- and this is real hard to solve.

4.  Coming up with a readable syntax that doesn't look like line noise is
really hard; and very subjective.

Based on all that, I have to agree -- now is not the time to try to resolve
these issues, there are more important issues to resolve -- I'll write more
on that tomorrow.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180726/66656d5f/attachment.html>

From steve at pearwood.info  Thu Jul 26 01:04:51 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 26 Jul 2018 15:04:51 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
References: <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAEbHw4Yzt1Wfr1iBw+mmTYWhh8b2SwyFLX1AHOL-4YP6H+_ChA@mail.gmail.com>
 <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>
 <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
Message-ID: <20180726050450.GN8744@ando.pearwood.info>

On Wed, Jul 25, 2018 at 11:09:35PM -0400, David Mertz wrote:

> Chris Angelica provided a more accurate translation.  Do you not see that
> the fact that your *second* try at understanding the actual behavior is
> still wrong suggest that this operator is a HUGE bug magnet?!

Oh what a load of FUD.

The whole point of having an operator do this means that we don't have 
to duplicate the semantics of the operator in plain Python.

Very few people get the semantics of x + y right. If (generic) you think 
that it is:

    return x.__add__(y)

you're wrong. If you think it is:

    try:
        return x.__add__(y)
    except AttributeError:
        return y.__radd__(x)

you're still wrong. If you think it is something like this:

    try:
        adder = type(x).__add__
        operands = (x, y)
    except AttributeError:
        try:
            adder = type(y).__radd__
            operands = (y, x)
        except AttributeError:
            raise TypeError from None
    return adder(*operands)

you're still wrong. But *nobody cares* because outside of some 
incredibly rare and specialised uses, nobody needs to emulate the 
semantics of + in pure Python. They just call the + operator, or at 
worst call the operator.add function.

We have the syntax "yield from" because it is *really complicated* for 
one generator to delegate to another generator. Emulating the yield from 
call isn't necessary, because we have yield from.

(If you think that "yield from" is equivalent to "for x in subgenerator: 
yield x", you're wrong too.)

And likewise, if we have these new operators, that means that those who 
need them won't have to emulate them in pure Python. They can just use 
the operators.

The idea that using the operators will be "a bug magnet" because the 
implementation is (allegedly) complex is ludicrous.


-- 
Steve

From steve at pearwood.info  Thu Jul 26 01:09:36 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 26 Jul 2018 15:09:36 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAKHFnb+ta+V0THPj1WB14Uk+A9V=wMV7YfqCQ7SFpky1Czai2A@mail.gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <3F4C2CB7-ED88-4670-AD7C-C4C88125E895@gmail.com>
 <CAKHFnb+ta+V0THPj1WB14Uk+A9V=wMV7YfqCQ7SFpky1Czai2A@mail.gmail.com>
Message-ID: <20180726050935.GO8744@ando.pearwood.info>

On Thu, Jul 26, 2018 at 01:02:47AM -0400, Amit Green wrote:

> 3.  The problem is way deeper than simply adding '?.' and other operators.
> For real use cases, you also need to say "how far" the an operator can
> "spread" -- and this is real hard to solve.

Why do you think that is the case? No other operators "spread". Why 
should these?


> 4.  Coming up with a readable syntax that doesn't look like line noise is
> really hard; and very subjective.

Define "line noise".

Is a.b.c syntax line noise?

Is a**b**c syntax line noise?

Is a == b == c line noise?



-- 
Steve

From amit.mixie at gmail.com  Thu Jul 26 01:20:55 2018
From: amit.mixie at gmail.com (Amit Green)
Date: Thu, 26 Jul 2018 01:20:55 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180726050935.GO8744@ando.pearwood.info>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <3F4C2CB7-ED88-4670-AD7C-C4C88125E895@gmail.com>
 <CAKHFnb+ta+V0THPj1WB14Uk+A9V=wMV7YfqCQ7SFpky1Czai2A@mail.gmail.com>
 <20180726050935.GO8744@ando.pearwood.info>
Message-ID: <CAKHFnbLhfF9wJ237fbar-rXAoJrddaUDfrYi_gOri+dsuV+Lmg@mail.gmail.com>

On Thu, Jul 26, 2018 at 1:09 AM, Steven D'Aprano <steve at pearwood.info>
wrote:

> On Thu, Jul 26, 2018 at 01:02:47AM -0400, Amit Green wrote:
>
> > 3.  The problem is way deeper than simply adding '?.' and other
> operators.
> > For real use cases, you also need to say "how far" the an operator can
> > "spread" -- and this is real hard to solve.
>
> Why do you think that is the case? No other operators "spread". Why
> should these?
>

If you take 'a?.b.c' then if the first '?.' yields none, logically you
don't want to evaluate the second '.'; otherwise you have to write:

a?.b?.c -- And this starts to look like line noise.

If you also implemented a concept such as use non-aware operators in
function calls: such as:

f(a, b?, c) -- which would mean collapse the whole thing out of
non-existance if 'b' yields none (i.e.: don't call function f) -- how far
does this spread?  Outside of f, if you then write:

f(a, b?, c).d.e

Does this also mean don't evaluate the '.d' & '.e' ?

And both of these examples are from *REAL* uses cases in my code I would
want to use none-aware operators.


>
> > 4.  Coming up with a readable syntax that doesn't look like line noise is
> > really hard; and very subjective.
>
> Define "line noise".
>
> Is a.b.c syntax line noise?
>
> Is a**b**c syntax line noise?
>
> Is a == b == c line noise?
>
>
>
These examples, from python, all look great & are very readable.  Which is
why python is such a great language :)

Simple non-aware operators are also very readable, such as: a?.b

This starts to become unreadable & line noise: a?.b.c.d ?? "hi".

Again, though it is subjective -- as is clearly evident by the discussions
in this group.

My simple point above is -- I really do want this feature -- but having
tried for days, I can't come up with a really clean syntax to capture all
of my *OWN* use cases ... let alone others...

Anyway, as I said, I think we have more important issues to address right
now, than this very difficult issue & I'll write up more tomorrow.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180726/bcd50fa3/attachment-0001.html>

From steve at pearwood.info  Thu Jul 26 01:42:26 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 26 Jul 2018 15:42:26 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4Z9=VPWwpgFTx896M8tyjoR3Jq_RBx4r+FTQzJdegY5HQ@mail.gmail.com>
References: <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <CAOhmDzfbzNgBJa3CZbnQyWCmLZ7aQLWzakwkX1-M1oysj=B5mw@mail.gmail.com>
 <CAEbHw4a2ZNuyUC1Qxq26+kqF0v5Q1uWU3nrvhFuzi52K+uY7Bg@mail.gmail.com>
 <CAKr=oZtZrhNzY+kUbbAsOGp6w198BuSizNNdJObwjrCpb3tLcg@mail.gmail.com>
 <CAEbHw4Z9=VPWwpgFTx896M8tyjoR3Jq_RBx4r+FTQzJdegY5HQ@mail.gmail.com>
Message-ID: <20180726054225.GP8744@ando.pearwood.info>

On Wed, Jul 25, 2018 at 07:06:35PM -0400, David Mertz wrote:

> And yes, the problem is that the equivalent is actually:
> 
>     v = a
>     if v is not None: v=a.b
> 
> The semantics are simply not the ones that are intuitive to most people
> reading 'v = a?.b'

Tell us more about these "intuitive to most people" semantics. What are 
they? Did you do a survey? How many people did you ask?

Did you do a comparison to the "intuitive to most people" semantics of 
slicing syntax?

Intuitive does not mean "familiar".

If you're going to dismiss a proposal because it's not "intuitive"[1] 
then to avoid accusations of intellectual hypocracy you need to do one 
of two things:

- demonstrate that the syntactic features we know and love and use 
  frequently (e.g. slicing, decorators, comprehensions) are intuitively
  obvious and don't need to be learned;

- or say that *those features were terrible errors* that Python still
  has not recovered from.


You can't have it both ways: its okay that people have to learn slicing, 
decorators, comprehensions, never mind that they aren't intuitive, for 
*those* features "intuitiveness" isn't that important; but for this 
proposal, "intuitiveness" is all the matters: usefulness, conciseness 
and the increase in expressivity don't matter one whit.

Its one thing to say that slicing, dot attribute access, decorators, 
augmented assignment, comprehensions etc are mistakes we should not 
repeat. But this double-standard of demanding higher standards for new 
features than we accept in existing, beloved features is one that we 
should reject with extreme prejudice.





[1] To whom? Babies? First-time programmers? People with a Ph.D. in comp 
sci and thirty years experience in half a dozen languages?


-- 
Steve

From jfine2358 at gmail.com  Thu Jul 26 02:03:18 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Thu, 26 Jul 2018 07:03:18 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <3F4C2CB7-ED88-4670-AD7C-C4C88125E895@gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <3F4C2CB7-ED88-4670-AD7C-C4C88125E895@gmail.com>
Message-ID: <CALD=Yf-QgVDGF85eujREh4QrYSJ2LeXOTxoeRWq9ROsx4-jZ2w@mail.gmail.com>

Hi All

To start this thread, on 18 July, Steve Dower wrote:
> Possibly this is exactly the wrong time to propose the next big syntax
> change, since we currently have nobody to declare on it, but since we're
> likely to argue for a while anyway it probably can't hurt (and maybe this
> will become the test PEP for whoever takes the reins?).

On 26 July, Raymond Hettinger wrote:
> It probably is the wrong time and probably can hurt (by introducing
> divisiveness when we most need for be focusing on coming together). [...]

+10

Here's some good news for Raymond. On 23 July, Steve Dower wrote:
---
> [...] I'm going to duck out of the discussions
> here now, since they are not as productive as I'd hoped, and once we have a
> BDFL-replacement I'll reawaken it and see what is required at that point.
---
and this is Steve's last post (so far) to the discussion.

+10

If you've not seen this message from Steve before, you're forgiven.
This topic has about 200 messages. Who's got time to read them all
them all?

If you want to have a go, they're listed by date at
https://mail.python.org/pipermail/python-ideas/2018-July/thread.html#52036

Regarding discussion of PEP 505, I'll follow the advice and example of
Raymond and Steve. Which is to pick it up again later, once we've got
proper cover for the BDFL's vacation. Have a nice holiday, Guido.

-- 
Jonathan

From elazarg at gmail.com  Thu Jul 26 02:08:17 2018
From: elazarg at gmail.com (Elazar)
Date: Wed, 25 Jul 2018 23:08:17 -0700
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180726054225.GP8744@ando.pearwood.info>
References: <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <CAOhmDzfbzNgBJa3CZbnQyWCmLZ7aQLWzakwkX1-M1oysj=B5mw@mail.gmail.com>
 <CAEbHw4a2ZNuyUC1Qxq26+kqF0v5Q1uWU3nrvhFuzi52K+uY7Bg@mail.gmail.com>
 <CAKr=oZtZrhNzY+kUbbAsOGp6w198BuSizNNdJObwjrCpb3tLcg@mail.gmail.com>
 <CAEbHw4Z9=VPWwpgFTx896M8tyjoR3Jq_RBx4r+FTQzJdegY5HQ@mail.gmail.com>
 <20180726054225.GP8744@ando.pearwood.info>
Message-ID: <CAPw6O2TgXE-jF1LP1aK-Pqgrd2gOUY6z5v80NZqsq3fYB6QRug@mail.gmail.com>

Just wanted to note another possibility (most likely to be added to the
"rejected solutions" section).

Add a prefix "!" operator, which will work similarly to the
iterator-unpacking operator "*", but for None.
The expression "[!x]" is equivalent to the expression "[] if x is None else
[x]".

This can be combined with the "or" operator, now working as expected since
it's about lists, and a simple "first" or "single" library function.

Example:

    optdict = dict(encoding=single([!encoding] or
sys.getdefaultencoding()), css=options.css)

It is more verbose than the proposed "??" operator (which I personally
like), but maybe it's less line noise.
One can use it with simple parens and get a tuple, "(!x) or y". note that
"(!x or y)" is a syntax error in this case.


Another, related idea: instead of "??" use a double-token operator "! or":

    value = ham! or spam! or eggs

    optdict = dict(encoding=encoding! or sys.getdefaultencoding(),
css=options.css)

Elazar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180725/c6077c7d/attachment.html>

From nicholas.cole at gmail.com  Thu Jul 26 02:47:53 2018
From: nicholas.cole at gmail.com (Nicholas Cole)
Date: Thu, 26 Jul 2018 07:47:53 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPw6O2TgXE-jF1LP1aK-Pqgrd2gOUY6z5v80NZqsq3fYB6QRug@mail.gmail.com>
References: <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <CAOhmDzfbzNgBJa3CZbnQyWCmLZ7aQLWzakwkX1-M1oysj=B5mw@mail.gmail.com>
 <CAEbHw4a2ZNuyUC1Qxq26+kqF0v5Q1uWU3nrvhFuzi52K+uY7Bg@mail.gmail.com>
 <CAKr=oZtZrhNzY+kUbbAsOGp6w198BuSizNNdJObwjrCpb3tLcg@mail.gmail.com>
 <CAEbHw4Z9=VPWwpgFTx896M8tyjoR3Jq_RBx4r+FTQzJdegY5HQ@mail.gmail.com>
 <20180726054225.GP8744@ando.pearwood.info>
 <CAPw6O2TgXE-jF1LP1aK-Pqgrd2gOUY6z5v80NZqsq3fYB6QRug@mail.gmail.com>
Message-ID: <CAAu18hdX1bMumJ=LTtaQztskMfJXRkj__evCtga7sqv3jDvNvg@mail.gmail.com>

The curious thing about PEP 505 as far as I can see is that it
introduces a new piece of syntax -- and for many people (to judge from
the reactions here) a controversial piece of syntax -- to solve what
seems to be a rather specific problem.  The use-case that seems most
discussed is unpacking information from nested JSON.

One of the things that makes me really twitch about the examples like

food = ham?.eggs?.spam?.chips

is that I don't usually deal with deeply nested structures like that
where it wouldn't matter to me which level of the nest returned a
value.

But let's leave that aside, because it seems to be an issue people
face. For the specific case mentioned, what would be wrong with a
utility function that looked something like:

# Warning - typed on gmail before I've had my morning coffee and so not
# at all tested, but I hope you can see what I mean.

def nested_conditional_find(search_this, *args):
     this = search_this
     for term in args:
          new_this = getattr(search_this, term)
          if new_this is None:
               return this
          else:
               this = new_this

then you could use this function as:

food = nested_conditional_find(ham, 'eggs', 'spam', 'chips')

One could even imagine adding extra tests along the way to a function
like that, raising attribute errors if the wrong kind of data was
encountered etc.

What problem does PEP 505 add that can't be solved very nearly as
elegantly by a utility function?  PEP 505 doesn't seem to be about
speed, only convenience.

For that matter, for specific use cases, why couldn't objects be
created where the attribute access functioned in just this kind of
way?  (not example code I'm willing to attempt before coffee ;-) )

Just a thought.

I've carefully avoided words like 'pythonic', 'unpythonic',
'explicit', 'implicit', 'compact', 'bug-magnet' or 'perl'.

Best wishes,

N

P.S. As an aside, in examples like food = spam?.eggs?.bacon -- if
bacon is None then food is None, even if there were in fact eggs and
spam.  That thought tells me it must be breakfast time.

From abedillon at gmail.com  Thu Jul 26 03:32:17 2018
From: abedillon at gmail.com (Abe Dillon)
Date: Thu, 26 Jul 2018 02:32:17 -0500
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPTjJmp25T_wVA_E_7zNwEFxcfrrLOMB8t0as3k7cenV+gZGTA@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAPTjJmrYhCtvLPGMb6==QKsr3ODXeaERiENnCdgeB+1ABNGp5A@mail.gmail.com>
 <CAEbHw4Zc7Q63ZC1wXXPxWdytQDbdZBt0cPK_LCxTpskzyW8_Lg@mail.gmail.com>
 <CAEbHw4agWFkhHrySoJHMVJHrGMxKN-dm=CAbCmKMgf0Dwr41cA@mail.gmail.com>
 <CAPTjJmp25T_wVA_E_7zNwEFxcfrrLOMB8t0as3k7cenV+gZGTA@mail.gmail.com>
Message-ID: <CAKr=oZtEvMiq0e3rNMq3W2xoZX_fjZdVcU33Qm+yPSoJhcJJRw@mail.gmail.com>

>
> Do you know what helps readability? *Learning to read*

Do you know what helps? *leveraging intuition*
If you're going to throw a question mark in there, which by the way, is far
more visually intrusive than a '.', then it makes more sense to at least
try to use it in a familiar way:

v = (a.b)?

## equivalent to

v = None
try:
    v = a.b
except AttributeError as e:
    if not e.args[0].startswith("'NonezType'"):
        raise e

Once you have learned to read ?. and friends, they will be as readable as .
> and
> slicing is now.


That's an assumption that's difficult to prove. If ?. gets added to Python
and proves to be a readability nightmare (as I believe it will be), it's
too late to go back. It's a gamble. The question is: is the potential
benefit worth the risk? and is there, perhaps, a better solution? I,
personally am not convinced of either. I think the 'maybe' solution seems
like a better alternative.

On Wed, Jul 25, 2018 at 9:40 PM, Chris Angelico <rosuav at gmail.com> wrote:

> On Thu, Jul 26, 2018 at 12:30 PM, David Mertz <mertz at gnosis.cx> wrote:
> > Btw. Here's a way of spelling the proposed syntax that gets the semantics
> > right:
> >
> >>>> # pip install coalescing
> >>>> NullCoalesce(spam).eggs.bacon
>
> Let's try it.
>
> rosuav at sikorsky:~$ sudo python3 -m pip install coalescing
> Collecting coalescing
>   Downloading https://files.pythonhosted.org/packages/f3/f4/
> 120f04cc59f9fa8c55c711b67f1c9c34d8a59c34cd69249e6ff61b098987
> /coalescing-0.1.1.tar.gz
> Installing collected packages: coalescing
>   Running setup.py install for coalescing ... done
> Successfully installed coalescing-0.1.1
>
> rosuav at sikorsky:~$ python3
> Python 3.8.0a0 (heads/literal_eval-exception:ddcb2eb331, Feb 21 2018,
> 04:32:23)
> [GCC 6.3.0 20170516] on linux
> Type "help", "copyright", "credits" or "license" for more information.
> >>> from coalescing import NullCoalesce
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> ModuleNotFoundError: No module named 'coalescing'
> >>> from coalesce import NullCoalesce
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/usr/local/lib/python3.8/site-packages/coalesce.py", line 56,
> in <module>
>     import wrapt
> ModuleNotFoundError: No module named 'wrapt'
>
> A bit problematic. But after (a) figuring out that your module is
> named "coalesce" even though I installed "coalescing" AND (b) going
> and separately installing wrapt, and finally (c) doing the import that
> you didn't mention, we still have this fundamental problem:
>
> rosuav at sikorsky:~$ python3
> Python 3.8.0a0 (heads/literal_eval-exception:ddcb2eb331, Feb 21 2018,
> 04:32:23)
> [GCC 6.3.0 20170516] on linux
> Type "help", "copyright", "credits" or "license" for more information.
> >>> from coalesce import NullCoalesce
> >>> from types import SimpleNamespace
> >>> spam, spam.eggs, spam.eggs.bacon = SimpleNamespace(),
> SimpleNamespace(), 42
> >>> NullCoalesce(spam).eggs.bacon
> <NullCoalesce proxy for 42>
>
> That isn't 42. That's a thing that, forever afterwards, will be a
> proxy. And look at this:
>
> >>> spam.nil = None
> >>> print(NullCoalesce(spam).nil)
> <NullCoalesce proxy for None>
> >>> print(NullCoalesce(spam).nil.nil)
> None
> >>> print(NullCoalesce(spam).nil.nil.nil)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> AttributeError: 'NoneType' object has no attribute 'nil'
> >>>
>
> Whoooooops.
>
> So, no, this is most definitely NOT equivalent to the proposed semantics.
>
> ChrisA
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180726/35798d45/attachment-0001.html>

From mhroncok at redhat.com  Thu Jul 26 04:12:01 2018
From: mhroncok at redhat.com (=?UTF-8?Q?Miro_Hron=c4=8dok?=)
Date: Thu, 26 Jul 2018 10:12:01 +0200
Subject: [Python-ideas] Change repr of collections.OrderedDict to be more
 dict-like
Message-ID: <378a7b7d-eb60-0a90-1f35-802913b6b501@redhat.com>

Hi,

now when dicts are sorted, can we change the repr of 
collections.OrderedDict to look like dict?

I mean from:

     OrderedDict([('a', '1'), ('b', '2')])

To:

     OrderedDict({'a': '1', 'b': '2'})

I assume the current list-of-tuples repr is there so copy pasting the 
repr to the interpreter results in the same OrderedDict. When dicts 
preserve insertion order, this now works with dict.

I consider the dict-based repr much easier to read.

Thoughts?
-- 
Miro Hron?ok
--
Phone: +420777974800
IRC: mhroncok

From gregory.lielens at gmail.com  Thu Jul 26 04:24:04 2018
From: gregory.lielens at gmail.com (=?UTF-8?Q?Gr=C3=A9gory_Lielens?=)
Date: Thu, 26 Jul 2018 01:24:04 -0700 (PDT)
Subject: [Python-ideas] Change repr of collections.OrderedDict to be
 more dict-like
In-Reply-To: <378a7b7d-eb60-0a90-1f35-802913b6b501@redhat.com>
References: <378a7b7d-eb60-0a90-1f35-802913b6b501@redhat.com>
Message-ID: <d0929e3d-2a48-4ff6-a981-6400b6659d3a@googlegroups.com>

I would wait a little bit. OrderedDict will probable become a pure synonym 
for dict,
in a few releases, and the repr issue will solve itself, both giving the 
even simpler/nicer {'a': '1', 'b': '2'}

Your proposal expose that dicts are ordered (which they are, but it's 
hidden). Once exposed, OrderedDict are obsolete and should be in the 
namespace only for backward compatibitlity.

On Thursday, July 26, 2018 at 10:13:12 AM UTC+2, Miro Hron?ok wrote:
>
> Hi,
>
> now when dicts are sorted, can we change the repr of 
> collections.OrderedDict to look like dict?
>
> I mean from:
>
>      OrderedDict([('a', '1'), ('b', '2')])
>
> To:
>
>      OrderedDict({'a': '1', 'b': '2'})
>
> I assume the current list-of-tuples repr is there so copy pasting the 
> repr to the interpreter results in the same OrderedDict. When dicts 
> preserve insertion order, this now works with dict.
>
> I consider the dict-based repr much easier to read.
>
> Thoughts?
> -- 
> Miro Hron?ok
> --
> Phone: +420777974800
> IRC: mhroncok
> _______________________________________________
> Python-ideas mailing list
> Python... at python.org <javascript:>
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180726/721419a9/attachment.html>

From mhroncok at redhat.com  Thu Jul 26 04:27:50 2018
From: mhroncok at redhat.com (=?UTF-8?Q?Miro_Hron=c4=8dok?=)
Date: Thu, 26 Jul 2018 10:27:50 +0200
Subject: [Python-ideas] Change repr of collections.OrderedDict to be
 more dict-like
In-Reply-To: <d0929e3d-2a48-4ff6-a981-6400b6659d3a@googlegroups.com>
References: <378a7b7d-eb60-0a90-1f35-802913b6b501@redhat.com>
 <d0929e3d-2a48-4ff6-a981-6400b6659d3a@googlegroups.com>
Message-ID: <91c6abf5-3ccc-fa59-754a-446fdff4b1dc@redhat.com>

On 26.7.2018 10:24, Gr?gory Lielens wrote:
> I would wait a little bit. OrderedDict will probable become a pure 
> synonym for dict,
> in a few releases, and the repr issue will solve itself, both giving the 
> even simpler/nicer {'a': '1', 'b': '2'}
> 
> Your proposal expose that dicts are ordered (which they are, but it's 
> hidden). Once exposed, OrderedDict are obsolete and should be in the 
> namespace only for backward compatibitlity.

I don't agree. See this thread:

https://mail.python.org/pipermail/python-dev/2018-July/154765.html

Namely:

On 26.7.2018 05:23, INADA Naoki wrote:
 > No.  There are some major difference.
 >
 > * d1 == d2 ignores order / od1 == od2 compares order
 > * OrderedDict has move_to_end() method.
 > * OrderedDict.pop() takes `last=True` keyword.

-- 
Miro Hron?ok
--
Phone: +420777974800
IRC: mhroncok

From gregory.lielens at gmail.com  Thu Jul 26 04:55:41 2018
From: gregory.lielens at gmail.com (=?UTF-8?Q?Gr=C3=A9gory_Lielens?=)
Date: Thu, 26 Jul 2018 01:55:41 -0700 (PDT)
Subject: [Python-ideas] Change repr of collections.OrderedDict to be
 more dict-like
In-Reply-To: <91c6abf5-3ccc-fa59-754a-446fdff4b1dc@redhat.com>
References: <378a7b7d-eb60-0a90-1f35-802913b6b501@redhat.com>
 <d0929e3d-2a48-4ff6-a981-6400b6659d3a@googlegroups.com>
 <91c6abf5-3ccc-fa59-754a-446fdff4b1dc@redhat.com>
Message-ID: <0798edf6-9352-43e5-8daa-f64c5259b297@googlegroups.com>

True, I was not aware of that...
But the thread also predict  OrderedDict will become very unfrequent, and 
that's also my impression: we are currently removing our OrderedDict.
I expect OrderedDict will be used only in place where the sequential order 
is a primary characteristic, with key access the secondary use pattern. 
dict will be used in the opposite context, key are the preffered access and 
order is only a secondary charateristic (that's our use case, order is 
needed only to make iteration order deterministic).
In this sense, the current repr is imho better: it  emphasize the order 
aspect (through the list) while your proposal emphasize the dict aspect.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180726/b61d29e6/attachment.html>

From rhodri at kynesim.co.uk  Thu Jul 26 07:43:44 2018
From: rhodri at kynesim.co.uk (Rhodri James)
Date: Thu, 26 Jul 2018 12:43:44 +0100
Subject: [Python-ideas] As-do statements/anonymous blocks in python
In-Reply-To: <CANQHGB2aCT3KOGnDP-qxA4PfV=34oiPGRO+rrbMBJG51K_4T-g@mail.gmail.com>
References: <CANQHGB2aCT3KOGnDP-qxA4PfV=34oiPGRO+rrbMBJG51K_4T-g@mail.gmail.com>
Message-ID: <abfc5a5a-32fa-c22e-529b-83d444dfa28e@kynesim.co.uk>

On 25/07/18 18:07, James Lu wrote:
> I'm open to any changes or criticism.
> 
> ```
> import atexit
> as atexit.register:
>      # ...do various cleanup tasks...
>      print('Goodbye')
> 
> # is approximately equivalent to =>
> import atexit
> def _():
>      # ...do various cleanup tasks...
>      print('Goodbye')
> atexit.register(_)
> 
> # flask example
> @app.route("/")
> def hello():
>      return "Hello World!"
> # is approximately equivalent to =>
> as app.route('/'):
>      return "Hello World!"

Could you explain what you expect "as" to do *in words* please?  These 
two examples give me conflicting ideas of what you mean.

-- 
Rhodri James *-* Kynesim Ltd

From rhodri at kynesim.co.uk  Thu Jul 26 07:48:40 2018
From: rhodri at kynesim.co.uk (Rhodri James)
Date: Thu, 26 Jul 2018 12:48:40 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <3F4C2CB7-ED88-4670-AD7C-C4C88125E895@gmail.com>
References: <4606c1a0-ba16-fa67-8ce8-e015fc23efcd@python.org>
 <3F4C2CB7-ED88-4670-AD7C-C4C88125E895@gmail.com>
Message-ID: <470bf1cc-7891-85e9-2297-ff9a4fe470bb@kynesim.co.uk>

On 26/07/18 05:25, Raymond Hettinger wrote:
> This PEP is one step further away from Python reading like executable pseudo-code.

+1000

-- 
Rhodri James *-* Kynesim Ltd

From desmoulinmichel at gmail.com  Thu Jul 26 08:07:07 2018
From: desmoulinmichel at gmail.com (Michel Desmoulin)
Date: Thu, 26 Jul 2018 14:07:07 +0200
Subject: [Python-ideas] As-do statements/anonymous blocks in python
In-Reply-To: <CANQHGB2aCT3KOGnDP-qxA4PfV=34oiPGRO+rrbMBJG51K_4T-g@mail.gmail.com>
References: <CANQHGB2aCT3KOGnDP-qxA4PfV=34oiPGRO+rrbMBJG51K_4T-g@mail.gmail.com>
Message-ID: <6c2ab9a5-c02d-40b3-ec8a-8f05b6295948@gmail.com>

I like the concept, and I several times wished I could have had a
reference to the block of code in a `with` clause.

However, it is unlikely to happen:

1 - Guido restricted lambda to one expression, and this would just be a
way around that

2 - This will be tempting to use for callbacks and chaining things,
leading to the nesting hell we tried very carefully to avoid fom other
languages. The example for sorted is also kinda twisted.

3 - Most of the example (and you show it yourself with flask) are
already possible with a more verbose syntaxe based on decorators. Example:


    @atexit.register
    def _():
        print('Goodbye')

4 - introducing a new keyword is the hardest thing you can ever ask on
this list.



Le 25/07/2018 ? 19:07, James Lu a ?crit?:
> I'm open to any changes or criticism.
> 
> ```
> import atexit
> as atexit.register:
> ? ? # ...do various cleanup tasks...
> ? ? print('Goodbye')
> 
> # is approximately equivalent to =>?
> import atexit
> def _():
> ? ??# ...do various cleanup tasks...
> ? ? print('Goodbye')
> atexit.register(_)
> 
> # flask example
> @app.route("/")
> def hello():
> ? ? return "Hello World!"
> # is approximately?equivalent to =>
> as app.route('/'):
> ? ? return "Hello World!"
> 
> @app.route('/user/<username>')
> def show_user_profile(username):
> ? ? # show the user profile for that user
> ? ? return 'User %s' % username
> 
> as app.route('/user/<username>') do username:
> ? ? return "Hello World!"
> 
> def print_sorted(iterable, block):
> ? ? sorted(iterable, )
> 
> l = [1, 2, 3, 4, 'spam']
> 
> as l.sort(key=%) do obj:
> ? ? return str(obj)
> 
> # multiple arguments
> as spam do a, b:
> ? ? ...
> ```
> ## `%`? function call syntax
> Calling a function with a single percent in place of an argument creates
> a new function.?
> 
> ```
> lumberjack(15, %)
> # is equivalent to the expression
> lambda x: lumberjack(15, %)
> ```
> Using `*` instead of `%` could also be possible.
> 
> ```
> import threading, time
> def interval(seconds, block):
> ? ? def target():
> ? ? ? ? while True:
> ? ? ? ? ? ? time.sleep(seconds)
> ? ? ? ? ? ? if block():? # stop looping if block returns True
> ? ? ? ? ? ? ? ? break
> ? ? threading.Thread(target=target).start()
> 
> as interval(5, %):
> ? ?print("chirp")
> # => chirp every 5 seconds on a seperate thread
> as threading.Timer(5, %):
> ? ?print("hi")
> # => say "hi" in 5 seconds
> ```
> 
> ## `^` currying function definition syntax?
> I'm not sure this is necessary?or a good idea.
> ```
> def interval(seconds, ^block):
> ? ? def target():
> ? ? ? ? while True:
> ? ? ? ? ? ? time.sleep(seconds)
> ? ? ? ? ? ? if block():? # stop looping if block returns True
> ? ? ? ? ? ? ? ? break
> ? ? threading.Thread(target=target).start()
> # is aprroximately equivalent to
> def interval(seconds, block=None):
> ? ? def inner(block):
> ? ? ? ? def target():
> ? ? ? ? ? ? while True:
> ? ? ? ? ? ? ? ? time.sleep(seconds)
> ? ? ? ? ? ? ? ? if block():
> ? ? ? ? ? ? ? ? ? ? break
> ? ? ? ? threading.Thread(target=target).start()
> ? ? if block == None:
> ? ? ? ? def outer(block):
> ? ? ? ? ? ? return inner(block)
> ? ? else:
> ? ? ? ? return inner(block)
> 
> as interval(5):
> ? ? print('chirp')
> # equivalent to?
> interval(5)(lambda: print('chirp'))
> ```
> ### Lazy evaluation of chained `%` calls?
> This would allow things like:
> ```
> start_on_new_thread = threading.Thread(target=%).start()
> def bong():
> ? ? while True:
> ? ? ? ? time.sleep(6*60)
> ? ? ? ? print('BONG')
> start_on_new_thread(bong)
> # alternatively
> as start_on_new_thread:
> ? ? while True:
> ? ? ? ? time.sleep(6*60)
> ? ? ? ? print('BONG')
> ```
> 
> ## As-do statements in classes
> ```
> class M():
> ? ? ?def __init__(self):
> ? ? ? ? ?self.time = 5
> ? ? ?as interval(self.time, %):
> ? ? ? ? ?print('chirp')
> ```
> 
> I'm not sure if this should be valid, and I'd like the community's input
> on when as-do statements should be bound when as-do.
> 
> ## Proposed Type Hinting Syntax
> ```
> as app.route('/user/<username>') do (username: str):?
> ? ? return "Hello World!"
> ```
> 
> Alternatives:
> ```
> as app.route('/user/<username>') do username: str%
> ? ? return "Hello World!"
> # like objective-c
> as app.route('/user/<username>') do username: str^
> ? ? return "Hello World!"
> # like coffeescript
> as app.route('/user/<username>') do username: str ->
> ? ? return "Hello World!"
> ```
> 
> I?m not totally sure of practical uses, but I?m imagining it would make
> passing a function to another function much more convenient. In React,
> you pass an argument called `render` to a `FlatList`, and `render`
> renders an item of the list. `FlatList` is a scrollable list that
> handles unloading off-screen items and loading items that will appear in
> the scroll box soon.
> 
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
> 

From rhodri at kynesim.co.uk  Thu Jul 26 08:13:44 2018
From: rhodri at kynesim.co.uk (Rhodri James)
Date: Thu, 26 Jul 2018 13:13:44 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4a2ZNuyUC1Qxq26+kqF0v5Q1uWU3nrvhFuzi52K+uY7Bg@mail.gmail.com>
References: <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <CAOhmDzfbzNgBJa3CZbnQyWCmLZ7aQLWzakwkX1-M1oysj=B5mw@mail.gmail.com>
 <CAEbHw4a2ZNuyUC1Qxq26+kqF0v5Q1uWU3nrvhFuzi52K+uY7Bg@mail.gmail.com>
Message-ID: <141028ff-a54a-39e6-afa7-4ac1dd0deb87@kynesim.co.uk>

On 25/07/18 23:36, David Mertz wrote:
> The fact that a while bunch have people have commented on this subthread
> while not recognizing that the semantics of the '?.' and the if blocks are
> entirely different suggests the operators are but magnets.

*Entirely* different?  Rubbish.  It's more like the difference between 
"+" and attempting to render "+" as a function using dunder methods.

I'm not keen on "?." and "?[]", but they are nothing like as bad as you 
are implying, and certainly not as bad as introducing a whole new 
concept like boxed types.

-- 
Rhodri James *-* Kynesim Ltd

From robertve92 at gmail.com  Thu Jul 26 08:20:16 2018
From: robertve92 at gmail.com (Robert Vanden Eynde)
Date: Thu, 26 Jul 2018 14:20:16 +0200
Subject: [Python-ideas] As-do statements/anonymous blocks in python
In-Reply-To: <6c2ab9a5-c02d-40b3-ec8a-8f05b6295948@gmail.com>
References: <CANQHGB2aCT3KOGnDP-qxA4PfV=34oiPGRO+rrbMBJG51K_4T-g@mail.gmail.com>
 <6c2ab9a5-c02d-40b3-ec8a-8f05b6295948@gmail.com>
Message-ID: <CA+msPN=+t8==+J=BYVh+agPS9yO8+U+FYE+j8LJ_o4b=_9Y6zQ@mail.gmail.com>

> lumberjack(15, %)
> # is equivalent to the expression
> lambda x: lumberjack(15, %)

You mean lambda x: lumberjack(15, x) ?

So you'd want a better syntax for functools.partial, here your example is
partial(lumberjack, 15).

However this syntax you allow to write lumberjack(%, 15) which is only
possible with partial using keyword arguments, like lumberjack (y=15)
(given that the second argument is called "y") and in that case I know at
least one library on pip doing the job with Ellipsis :

from funcoperators import elipartial
elipartial(lumberjack, ..., 15)
# gives lambda x: lumberjack (x, 15)

elipartial (lumberjack, ..., 15, ..., 12, ...)
# gives lambda x, y, z: lumberjack(x, 15, y, 12, z)

And the lib even provide a decorator to have that kind of syntax using "[" :

partially(lumberjack)[15]
would be the same as partial(lumberjack, 15)

So, that functionality does really need new syntax.


Le jeu. 26 juil. 2018 ? 14:07, Michel Desmoulin <desmoulinmichel at gmail.com>
a ?crit :

> I like the concept, and I several times wished I could have had a
> reference to the block of code in a `with` clause.
>
> However, it is unlikely to happen:
>
> 1 - Guido restricted lambda to one expression, and this would just be a
> way around that
>
> 2 - This will be tempting to use for callbacks and chaining things,
> leading to the nesting hell we tried very carefully to avoid fom other
> languages. The example for sorted is also kinda twisted.
>
> 3 - Most of the example (and you show it yourself with flask) are
> already possible with a more verbose syntaxe based on decorators. Example:
>
>
>     @atexit.register
>     def _():
>         print('Goodbye')
>
> 4 - introducing a new keyword is the hardest thing you can ever ask on
> this list.
>
>
>
> Le 25/07/2018 ? 19:07, James Lu a ?crit :
> > I'm open to any changes or criticism.
> >
> > ```
> > import atexit
> > as atexit.register:
> >     # ...do various cleanup tasks...
> >     print('Goodbye')
> >
> > # is approximately equivalent to =>
> > import atexit
> > def _():
> >     # ...do various cleanup tasks...
> >     print('Goodbye')
> > atexit.register(_)
> >
> > # flask example
> > @app.route("/")
> > def hello():
> >     return "Hello World!"
> > # is approximately equivalent to =>
> > as app.route('/'):
> >     return "Hello World!"
> >
> > @app.route('/user/<username>')
> > def show_user_profile(username):
> >     # show the user profile for that user
> >     return 'User %s' % username
> >
> > as app.route('/user/<username>') do username:
> >     return "Hello World!"
> >
> > def print_sorted(iterable, block):
> >     sorted(iterable, )
> >
> > l = [1, 2, 3, 4, 'spam']
> >
> > as l.sort(key=%) do obj:
> >     return str(obj)
> >
> > # multiple arguments
> > as spam do a, b:
> >     ...
> > ```
> > ## `%`  function call syntax
> > Calling a function with a single percent in place of an argument creates
> > a new function.
> >
> > ```
> > lumberjack(15, %)
> > # is equivalent to the expression
> > lambda x: lumberjack(15, %)
> > ```
> > Using `*` instead of `%` could also be possible.
> >
> > ```
> > import threading, time
> > def interval(seconds, block):
> >     def target():
> >         while True:
> >             time.sleep(seconds)
> >             if block():  # stop looping if block returns True
> >                 break
> >     threading.Thread(target=target).start()
> >
> > as interval(5, %):
> >    print("chirp")
> > # => chirp every 5 seconds on a seperate thread
> > as threading.Timer(5, %):
> >    print("hi")
> > # => say "hi" in 5 seconds
> > ```
> >
> > ## `^` currying function definition syntax?
> > I'm not sure this is necessary or a good idea.
> > ```
> > def interval(seconds, ^block):
> >     def target():
> >         while True:
> >             time.sleep(seconds)
> >             if block():  # stop looping if block returns True
> >                 break
> >     threading.Thread(target=target).start()
> > # is aprroximately equivalent to
> > def interval(seconds, block=None):
> >     def inner(block):
> >         def target():
> >             while True:
> >                 time.sleep(seconds)
> >                 if block():
> >                     break
> >         threading.Thread(target=target).start()
> >     if block == None:
> >         def outer(block):
> >             return inner(block)
> >     else:
> >         return inner(block)
> >
> > as interval(5):
> >     print('chirp')
> > # equivalent to
> > interval(5)(lambda: print('chirp'))
> > ```
> > ### Lazy evaluation of chained `%` calls?
> > This would allow things like:
> > ```
> > start_on_new_thread = threading.Thread(target=%).start()
> > def bong():
> >     while True:
> >         time.sleep(6*60)
> >         print('BONG')
> > start_on_new_thread(bong)
> > # alternatively
> > as start_on_new_thread:
> >     while True:
> >         time.sleep(6*60)
> >         print('BONG')
> > ```
> >
> > ## As-do statements in classes
> > ```
> > class M():
> >      def __init__(self):
> >          self.time = 5
> >      as interval(self.time, %):
> >          print('chirp')
> > ```
> >
> > I'm not sure if this should be valid, and I'd like the community's input
> > on when as-do statements should be bound when as-do.
> >
> > ## Proposed Type Hinting Syntax
> > ```
> > as app.route('/user/<username>') do (username: str):
> >     return "Hello World!"
> > ```
> >
> > Alternatives:
> > ```
> > as app.route('/user/<username>') do username: str%
> >     return "Hello World!"
> > # like objective-c
> > as app.route('/user/<username>') do username: str^
> >     return "Hello World!"
> > # like coffeescript
> > as app.route('/user/<username>') do username: str ->
> >     return "Hello World!"
> > ```
> >
> > I?m not totally sure of practical uses, but I?m imagining it would make
> > passing a function to another function much more convenient. In React,
> > you pass an argument called `render` to a `FlatList`, and `render`
> > renders an item of the list. `FlatList` is a scrollable list that
> > handles unloading off-screen items and loading items that will appear in
> > the scroll box soon.
> >
> >
> > _______________________________________________
> > Python-ideas mailing list
> > Python-ideas at python.org
> > https://mail.python.org/mailman/listinfo/python-ideas
> > Code of Conduct: http://python.org/psf/codeofconduct/
> >
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180726/8ab86059/attachment-0001.html>

From rhodri at kynesim.co.uk  Thu Jul 26 08:27:47 2018
From: rhodri at kynesim.co.uk (Rhodri James)
Date: Thu, 26 Jul 2018 13:27:47 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAKr=oZvBO1sNd0RdmUJ3xocp45etLpRXAo5dZuaa1j2qsKisDw@mail.gmail.com>
References: <CAPTjJmoJ8cKZLUn+KeXYyCDERR4FNB=w0WxqEuF4JtVz8mzodw@mail.gmail.com>
 <CAFYqXL9HQgZdLTgxpj7o3XLA--6UzaiixfQVL_FTvO3PTxh9sg@mail.gmail.com>
 <CAPTjJmoxRfAuR2K9MZguf_gg-uQG3EVxkFjBv+ftubUv+-TPMA@mail.gmail.com>
 <CAFYqXL_kr+k2upZHDt-CsdXGw5W8dmk=77uyFRvQjZnPSLXF+Q@mail.gmail.com>
 <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <CAOhmDzfbzNgBJa3CZbnQyWCmLZ7aQLWzakwkX1-M1oysj=B5mw@mail.gmail.com>
 <CAKr=oZvBO1sNd0RdmUJ3xocp45etLpRXAo5dZuaa1j2qsKisDw@mail.gmail.com>
Message-ID: <d55416ce-6166-1cff-3d56-539a6c37c267@kynesim.co.uk>

On 25/07/18 23:11, Abe Dillon wrote:
> The problem here is not whether it's explicit. It's about Readability and
> conciseness. Using symbols in place of words almost always harms
> readability in favor of conciseness.
> 
> value = person.name if person.name else person

ITYM
   value = person.name if person.name is not None else None

> almost reads like english (aside from being a weird and totally uncommon
> use case)

I think Nicholas (?) was right to observe that it's uncommon to not want 
to know where a deeply nested attribute access failed.  Exactly what you 
want to do when an underlying library (presumably) has "helpfully" 
turned non-existent attributes into Nones is going to vary tremendously 
from application to application.

On the other hand,

   value = parameter ?? default

encapsulates a very common usage, and doesn't read badly compared with:

   value = parameter if parameter is not None else default

which is a tad wordy.  That example isn't too bad, but replace 
"parameter" with "spam.spam.spam.eggs" and DRY suddenly becomes a rather 
more important principle :-)

-- 
Rhodri James *-* Kynesim Ltd

From steve at pearwood.info  Thu Jul 26 09:24:26 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 26 Jul 2018 23:24:26 +1000
Subject: [Python-ideas] As-do statements/anonymous blocks in python
In-Reply-To: <6c2ab9a5-c02d-40b3-ec8a-8f05b6295948@gmail.com>
References: <CANQHGB2aCT3KOGnDP-qxA4PfV=34oiPGRO+rrbMBJG51K_4T-g@mail.gmail.com>
 <6c2ab9a5-c02d-40b3-ec8a-8f05b6295948@gmail.com>
Message-ID: <20180726132422.GS8744@ando.pearwood.info>

On Thu, Jul 26, 2018 at 02:07:07PM +0200, Michel Desmoulin wrote:

> 4 - introducing a new keyword is the hardest thing you can ever ask on
> this list.

Introducing a new keyword is like falling off a log compared to 
introducing a new operator.

-- 
Steve

From jamtlu at gmail.com  Thu Jul 26 09:41:40 2018
From: jamtlu at gmail.com (James Lu)
Date: Thu, 26 Jul 2018 09:41:40 -0400
Subject: [Python-ideas] As-do statements in Python
In-Reply-To: <mailman.353.1532579149.7656.python-ideas@python.org>
References: <mailman.353.1532579149.7656.python-ideas@python.org>
Message-ID: <E6D03D22-AAB2-4B09-9373-EBDE951CA9D1@gmail.com>

> 4 - introducing a new keyword is the hardest thing you can ever ask on
> this list.

As is already a keyword from with-as and except-as. Perhaps for compatibility ?as? is allowed but discouraged as a variable name, and it?s only interpreted as a keyword when followed by an expression and a colon.

> 3 - Most of the example (and you show it yourself with flask) are
> already possible with a more verbose syntaxe based on decorators

Yup, a more terse syntax puts the programmer?s focus on the right things.
> 2 - This will be tempting to use for callbacks and chaining things,
> leading to the nesting hell we tried very carefully to avoid fom other
> languages. The example for sorted is also kinda twisted.

I do agree as-do could be easily used for callbacks. Could you please clarify what you mean by nesting hell? 

Is it better to use as-do where you?d normally pass your own instantiation of a class you create that implements a protocol? It?s always possible to simply pass a bound method of a class you create yourself if you want to keep state.

From jamtlu at gmail.com  Thu Jul 26 09:45:35 2018
From: jamtlu at gmail.com (James Lu)
Date: Thu, 26 Jul 2018 09:45:35 -0400
Subject: [Python-ideas] Python-ideas Digest, Vol 140, Issue 169
In-Reply-To: <mailman.481.1532607631.7656.python-ideas@python.org>
References: <mailman.481.1532607631.7656.python-ideas@python.org>
Message-ID: <07135BEF-F0AA-4C0B-A560-B254AA5596E7@gmail.com>

> You mean lambda x: lumberjack(15, x) ?
Yes, that was my mistake.

>  However this syntax you allow to write lumberjack(%, 15) which is only
> possible with partial using keyword arguments, like lumberjack (y=15)
> (given that the second argument is called "y") and in that case I know at
> least one library on pip doing the job with Ellipsis :
> 
> from funcoperators import elipartial
> elipartial(lumberjack, ..., 15)
> # gives lambda x: lumberjack (x, 15)
> 
> elipartial (lumberjack, ..., 15, ..., 12, ...)
> # gives lambda x, y, z: lumberjack(x, 15, y, 12, z)
> 
> And the lib even provide a decorator to have that kind of syntax using "[" :
> 
> partially(lumberjack)[15]
> would be the same as partial(lumberjack, 15)
> 
> So, that functionality does really need new syntax.

Hm, yes. Perhaps elipartial could be aliased and/or made a global built in. Let?s discuss the merits of the statement itself then.

From jamtlu at gmail.com  Thu Jul 26 10:02:40 2018
From: jamtlu at gmail.com (James Lu)
Date: Thu, 26 Jul 2018 10:02:40 -0400
Subject: [Python-ideas] Python-ideas Digest, Vol 140, Issue 169
In-Reply-To: <mailman.481.1532607631.7656.python-ideas@python.org>
References: <mailman.481.1532607631.7656.python-ideas@python.org>
Message-ID: <2B2FA2B4-EC75-43AD-ADE2-F3990CEB65BF@gmail.com>

> I several times wished I could have had a reference to the block of code in a `with` clause.

Could you abstractly describe a real-world example of this? I?d like to hear your use case. 

> The example for sorted is also kinda twisted.

It is, it?s probably a counterexample on how to use it.

I forgot to mention, could it be possible to only treat do as a keyword when it?s used in an as-do expression? So, do remains a legal variable name, and do is only interpreted as a keyword when it?s in an as-do expression. I believe this is possible using the Python grammar. 

From mikhailwas at gmail.com  Thu Jul 26 10:56:29 2018
From: mikhailwas at gmail.com (Mikhail V)
Date: Thu, 26 Jul 2018 17:56:29 +0300
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180726021047.GM8744@ando.pearwood.info>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <CAOhmDzfbzNgBJa3CZbnQyWCmLZ7aQLWzakwkX1-M1oysj=B5mw@mail.gmail.com>
 <CAKr=oZvBO1sNd0RdmUJ3xocp45etLpRXAo5dZuaa1j2qsKisDw@mail.gmail.com>
 <20180726021047.GM8744@ando.pearwood.info>
Message-ID: <CAE6NSRCZhE7=QKz2gcHt2hbKHZ9C7WXzz6ypBOtAQ1QDFnVdOA@mail.gmail.com>

On Thu, Jul 26, 2018 at 5:10 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> On Wed, Jul 25, 2018 at 05:11:08PM -0500, Abe Dillon wrote:
>> The problem here is not whether it's explicit. It's about Readability and
>> conciseness. Using symbols in place of words almost always harms
>> readability in favor of conciseness.
>
[...]

> Do you know what helps readability? *Learning to read*. Once you have
> learned to read ?. and friends, they will be as readable as . and
> slicing is now.

No it is not like that.
E.g. slicing is intuitive and straightforward concept even for a non-programmer.

And no, it is not (only) about learning.
For example, no matter how experienced a programmer is,
complex comprehensions will be less readable than the
same construct written in loops and ifs.

Or say this:

    user?.profile?.food

will be less readable than say:

    user ? profile ? food

No matter how long you learn it, the former remains ugly and
obfuscating.
With that said, I don't say this syntax deserves something better,
I just say that I can't see how your "learn more" argument applies
here.



Mikhail

From klahnakoski at mozilla.com  Thu Jul 26 10:58:53 2018
From: klahnakoski at mozilla.com (Kyle Lahnakoski)
Date: Thu, 26 Jul 2018 10:58:53 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4ZvoPcuEyzMaaj2ggELi7BBAaQcaGDPCteYQnc5yoe5YA@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAPTjJmrYhCtvLPGMb6==QKsr3ODXeaERiENnCdgeB+1ABNGp5A@mail.gmail.com>
 <CAEbHw4Zc7Q63ZC1wXXPxWdytQDbdZBt0cPK_LCxTpskzyW8_Lg@mail.gmail.com>
 <CAEbHw4agWFkhHrySoJHMVJHrGMxKN-dm=CAbCmKMgf0Dwr41cA@mail.gmail.com>
 <CAPTjJmp25T_wVA_E_7zNwEFxcfrrLOMB8t0as3k7cenV+gZGTA@mail.gmail.com>
 <CAEbHw4a+5hK1p4ZEsEXkyKSm91ZoVVPO_JwThv+j3T65-gvvJg@mail.gmail.com>
 <CAPTjJmq70XuWPND3J4YEAhioBJ1oZYiSR=tZM3Ypu59wYjA9HA@mail.gmail.com>
 <CAEbHw4Z3AVp6oP_UDhyM4kEPBKWD-F8bWwWs0gjHB9uGP0ERrg@mail.gmail.com>
 <CAPTjJmqQXTtEUjJQkaFBP=q+94OLw8v0xGtKVnO0Xam773W1Mw@mail.gmail.com>
 <CAEbHw4ZvoPcuEyzMaaj2ggELi7BBAaQcaGDPCteYQnc5yoe5YA@mail.gmail.com>
Message-ID: <1b348298-fe78-26c2-aa78-6b2e830d6a64@mozilla.com>



On 2018-07-25 23:53, David Mertz wrote:
> On Wed, Jul 25, 2018, 11:27 PM Chris Angelico <rosuav at gmail.com
> <mailto:rosuav at gmail.com>> wrote:
>
>
>     means that you cannot do this:
>
>     >>> NullCoalesce(spam).nil is None
>
>     This IS fundamentally unfixable in a library.
>
>
> Right now, you can still always call .unbox() at the end to get the
> underlying value. I agree that's a little ugly, hence why I added the
> wrapt proxy stuff. Most operations trigger unboxing, but indeed not
> simply echoing to the shell.
>

Chris is correct to point out this problem with comparing to None. I
have that problem with my own version of the proxy library, similar to
what David is building:? I use the proxy? heavily; to the point where
almost any line may be touching a proxy rather than a real value. To
avoid bugs, I disallow "is None" comparisons, mandating "== None" instead.?

Using unbox() is an alternative,? but it is uglier than than swapping
"as" for "==".


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180726/2b5e686e/attachment.html>

From shoyer at gmail.com  Thu Jul 26 13:17:04 2018
From: shoyer at gmail.com (Stephan Hoyer)
Date: Thu, 26 Jul 2018 10:17:04 -0700
Subject: [Python-ideas] slice[] to get more complex slices
In-Reply-To: <CAEbHw4YuS99=cVbhV6Riuwij4H_xNdcxp1Df6tMQfD-__CaO_w@mail.gmail.com>
References: <CAFpSVpKnE8b=QrjxLy6anAF3ammq2QbB4_K1Ey02Vk3n8EVwyQ@mail.gmail.com>
 <pj2lgk$c6c$1@blaine.gmane.org>
 <09cb9784-6a97-49b9-b5e7-1c9a9d8c8391@googlegroups.com>
 <CALD=Yf-MNLW7z8s=ruc_C13t7eWp8KC6OWvqf9durZgpN=8=sw@mail.gmail.com>
 <399ff7d70c204f679bdb5a79f9bc0460@xmail103.UGent.be>
 <4243953379d74d2aa5393be2434a99a7@xmail103.UGent.be>
 <5B55AE6A.7010305@UGent.be>
 <CACac1F_XOi+1gmEMgZdbmW_3UXs661JS5BF=OvWLrF47H0fJmg@mail.gmail.com>
 <CAEQ_Tvf2Y-up_uWoUdutyoreHeM6=X52bZzZcath-r1btYeqow@mail.gmail.com>
 <pj5nl6$p6b$1@blaine.gmane.org>
 <CAEbHw4YuS99=cVbhV6Riuwij4H_xNdcxp1Df6tMQfD-__CaO_w@mail.gmail.com>
Message-ID: <CAEQ_Tvc3RzZrew324XZ-sauegHGssWC2sBk520X0T_QQyaeG0A@mail.gmail.com>

On Mon, Jul 23, 2018 at 4:37 PM David Mertz <mertz at gnosis.cx> wrote:

> I find pandas.IndexSlice makes a lot of operations easier to spell. As
> simple as it is, it's a valuable capability. Rather than every library?or a
> number of them anyway?creating the same 4 lines of code with a different
> name, it would be much nicer to have it as a class __getitem__ method, e.g.
> slice[...], or as an attribute of the slice object, such as
> slice.literal[...].
>

I'd really like to move this proposal forward in some form.

There have been three basic proposals for where to put the __getitem__
based constructor:
1. slice: slice[...] potentially conflicts with typing syntax.
2. slice.literal: This is potentially confusing, because in many cases it
does not actually create a single slice object.
3. operator.subscript: This is the last proposal standing, and was also the
outcome of the previous discussion on python-ideas:
https://mail.python.org/pipermail/python-ideas/2015-June/034086.html

I think the repeated interest in this topic demonstrates that there are
real use cases for operator.subscript, even though it would be slightly
less accessible than putting it directly on slice or slice.literal.

As a next step, would it be helpful to summarize the use cases and
reasoning in a PEP?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180726/ba009a85/attachment.html>

From tjreedy at udel.edu  Thu Jul 26 13:21:23 2018
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 26 Jul 2018 13:21:23 -0400
Subject: [Python-ideas] Change repr of collections.OrderedDict to be
 more dict-like
In-Reply-To: <378a7b7d-eb60-0a90-1f35-802913b6b501@redhat.com>
References: <378a7b7d-eb60-0a90-1f35-802913b6b501@redhat.com>
Message-ID: <pjcvqh$t6e$1@blaine.gmane.org>

On 7/26/2018 4:12 AM, Miro Hron?ok wrote:
> Hi,
> 
> now when dicts are sorted, can we change the repr of 
> collections.OrderedDict to look like dict?
> 
> I mean from:
> 
>  ??? OrderedDict([('a', '1'), ('b', '2')])
> 
> To:
> 
>  ??? OrderedDict({'a': '1', 'b': '2'})
> 
> I assume the current list-of-tuples repr is there so copy pasting the 
> repr to the interpreter results in the same OrderedDict. When dicts 
> preserve insertion order, this now works with dict.
> 
> I consider the dict-based repr much easier to read.
> 
> Thoughts?

On python-idea,  Miro Hron?ok asked today whether we can change the 
OrderedDict repr from, for instance,

OrderedDict([('a', '1'), ('b', '2')]) # to
OrderedDict({'a': '1', 'b': '2'})

I am not sure what our repr change policy is, as there is a 
back-compatibility issue but I remember there being changes.

-- 
Terry Jan Reedy



From mertz at gnosis.cx  Thu Jul 26 13:28:12 2018
From: mertz at gnosis.cx (David Mertz)
Date: Thu, 26 Jul 2018 13:28:12 -0400
Subject: [Python-ideas] Change repr of collections.OrderedDict to be
 more dict-like
In-Reply-To: <pjcvqh$t6e$1@blaine.gmane.org>
References: <378a7b7d-eb60-0a90-1f35-802913b6b501@redhat.com>
 <pjcvqh$t6e$1@blaine.gmane.org>
Message-ID: <CAEbHw4ZB3MSvjv9uOqAOd6pOWO0FBxchyjDE5ZSveAmcToq+0A@mail.gmail.com>

Repr changes are likely to break doctests, at least. That's not necessarily
an absolute "no", but it does raise the bar.

On Thu, Jul 26, 2018, 1:22 PM Terry Reedy <tjreedy at udel.edu> wrote:

> On 7/26/2018 4:12 AM, Miro Hron?ok wrote:
> > Hi,
> >
> > now when dicts are sorted, can we change the repr of
> > collections.OrderedDict to look like dict?
> >
> > I mean from:
> >
> >      OrderedDict([('a', '1'), ('b', '2')])
> >
> > To:
> >
> >      OrderedDict({'a': '1', 'b': '2'})
> >
> > I assume the current list-of-tuples repr is there so copy pasting the
> > repr to the interpreter results in the same OrderedDict. When dicts
> > preserve insertion order, this now works with dict.
> >
> > I consider the dict-based repr much easier to read.
> >
> > Thoughts?
>
> On python-idea,  Miro Hron?ok asked today whether we can change the
> OrderedDict repr from, for instance,
>
> OrderedDict([('a', '1'), ('b', '2')]) # to
> OrderedDict({'a': '1', 'b': '2'})
>
> I am not sure what our repr change policy is, as there is a
> back-compatibility issue but I remember there being changes.
>
> --
> Terry Jan Reedy
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180726/2b6b4ff5/attachment.html>

From tjreedy at udel.edu  Thu Jul 26 13:23:42 2018
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 26 Jul 2018 13:23:42 -0400
Subject: [Python-ideas] Change repr of collections.OrderedDict to be
 more dict-like
In-Reply-To: <pjcvqh$t6e$1@blaine.gmane.org>
References: <378a7b7d-eb60-0a90-1f35-802913b6b501@redhat.com>
 <pjcvqh$t6e$1@blaine.gmane.org>
Message-ID: <pjcvus$t6e$3@blaine.gmane.org>

On 7/26/2018 1:21 PM, Terry Reedy wrote:
> On python-idea,? Miro Hron?ok asked today whether we can change the 
> OrderedDict repr from, for instance,
> 
> OrderedDict([('a', '1'), ('b', '2')]) # to
> OrderedDict({'a': '1', 'b': '2'})
> 
> I am not sure what our repr change policy is, as there is a 
> back-compatibility issue but I remember there being changes.

Whoops, I meant to send this query elsewhere, so I could answer properly 
here.

-- 
Terry Jan Reedy



From abedillon at gmail.com  Thu Jul 26 17:28:44 2018
From: abedillon at gmail.com (Abe Dillon)
Date: Thu, 26 Jul 2018 16:28:44 -0500
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <CAFYqXL_Qohp3ndCbmjVRkBdoFeTo6pmr84HKwuT6Ee460K6XuA@mail.gmail.com>
 <CAPTjJmp-czp=f=8AcJNK2h+DMvSP+XzZAQuKgJh8EqD8XSBRQg@mail.gmail.com>
 <CAFYqXL9gr=2CqVOd+EwkWazVK6g6PvQ6t3O7CiPZoVdXX75WcQ@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAEbHw4Yzt1Wfr1iBw+mmTYWhh8b2SwyFLX1AHOL-4YP6H+_ChA@mail.gmail.com>
 <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>
 <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
Message-ID: <CAKr=oZtfYM2_fxYnde76o+jvuc_Y3AH9R93BCY6BoNdriuKqKw@mail.gmail.com>

The readability issue isn't just a matter of a new set of symbols to learn.
It isn't even that the symbols are used in a non-intuitive way (though that
doesn't help). It's not even that the question mark, unlike the dot, is
very visually obtrusive (which also doesn't help). It also screws with the
grammar in an unexpected way. I would be fine with something like:

def func(data=None):
    data ?= []
    ...

If there were some corresponding dunder method to implement that behavior,
but it's not clear to me that such a thing is possible. If it isn't, that
would make `?=` (or '??=' though I don't see the need for double '?'s) a
special statement, a special case that someone who's familiar with how
other in-place operators work, would be thrown for a loop. The other use
cases look like they break down in a different manner than they actually do:

person?.name

looks like it would break down similar to

person().name

but it doesn't because '?' by itself is not an operator.
similarly:

person?[0]

looks like it would break down similar to

person()[0]

but again, the operator is '?[...]' not '?'

This also raises the question of other operators that could be made null
aware. Why not:

func?()

?-num

?~num

num ?+ other

Why not other reflective operators?

num ?+= other

I think this could be helped by simply making '?' an operator with a very
high order of operations, so that pretty-much any expression on the LHS is
guarded for 'NoneType' AttributeErrors:

val = person[0]?
val = person.name?

That has the double benefit of making multiple '?.' or '?[]' operations
unnecessary. I mean the following expression would almost never make sense
under the current proposal:

instead of:

val = person?.name?[0]?.lower()

you could write:

val = person.name[0].lower()?

That also aligns a little better with common usage of '?' in natural
language.

But again; there doesn't seem to be a reasonable way to implement a
corresponding dunder method for those operators because they're special.
I also disagree with the idea that None is special enough to warrant such
special behavior. There are other special None-like objects (as David Mertz
pointed out) like NaN and custom place-holders for cases where None is a
valid argument.

Yet another possibility is to make '?' signify a parameter who's default
value is an expression that should be evaluated when necessary:

def func(data: List = []?):
    ...

That gets closer to another often requested feature: delayed expression
evaluation (as Steve D'Aprano pointed out).
So there are other, possibly better uses for the '?' symbol. It's something
that needs to be carefully considered.


On Wed, Jul 25, 2018 at 10:09 PM, David Mertz <mertz at gnosis.cx> wrote:

> On Wed, Jul 25, 2018 at 10:50 PM Nicholas Chammas <
> nicholas.chammas at gmail.com> wrote:
>
>> Indeed. Thanks for the counter-example. I think the correct translation
>> is as follows:
>>     food = spam?.eggs?.bacon
>> Becomes:
>>     food = None
>>     if spam is not None and spam.eggs is not None:
>>         food = spam.eggs.bacon
>>
> Did I get it right now? :)
>>
>
> Nope, still not right, I'm afraid!
>
> Chris Angelica provided a more accurate translation.  Do you not see that
> the fact that your *second* try at understanding the actual behavior is
> still wrong suggest that this operator is a HUGE bug magnet?!
>
>
>> So, shame on me. I think this particular mistake reflects more on me than
>> on PEP 505, but I see how this kind of mistake reflects badly on the folks
>> advocating for the PEP (or at least, playing devil's advocate).
>>
>
> I really, really don't.  I think you see an intuitive behavior that would
> be nice and useful in a certain area.  That behavior just isn't what the
> PEP proposes though... it's kinda-sorta close enough to be lured into
> thinking it's a good idea.
>
> Honestly, I think the behavior of GreedyAccess in my little library I
> wrote over the last couple nights is FAR more often what programmers
> ACTUALLY want than NullCoalesce is.  Even Steve Dower?in the PEP and in
> this discussion?acknowledges the appeal and utility of the GreedyAccess
> behavior.  It's in the "Rejected Ideas" section, which is fair enough.
>
> But in a library like mine... or indeed, in a much better library that you
> or someone else writes... it's perfectly easy to have both classes, and
> choose which behavior is more useful for your case.  A new syntax feature
> can't let user decide which behavior (or maybe some other behavior
> altogether) is most useful for their specific case.  A library does that
> easily[*].
>
> [*] In version 0.1.1 of coalescing?changed from 0.1?I added the option to
> use a sentinel other than None if you want.  I'm not sure how useful that
> is, but that idea was in some old PEPs, and I think in the Rejected Ideas
> of 505.  With a library, I have a parameter that need not be used to switch
> that[**].  E.g.:
>
> NullCoalesce(foo, sentinel=float('nan')).bar.baz.blam
>
> [**] Yes, I even handle NaN's in a special way because they are non-equal
> even to themselves.  You could use empty string, or 0, or my_null =
> object(), or whatever.
> --
> Keeping medicines from the bloodstreams of the sick; food
> from the bellies of the hungry; books from the hands of the
> uneducated; technology from the underdeveloped; and putting
> advocates of freedom in prisons.  Intellectual property is
> to the 21st century what the slave trade was to the 16th.
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180726/64309d5e/attachment-0001.html>

From mike at selik.org  Thu Jul 26 18:41:12 2018
From: mike at selik.org (Michael Selik)
Date: Thu, 26 Jul 2018 15:41:12 -0700
Subject: [Python-ideas] Change repr of collections.OrderedDict to be
 more dict-like
In-Reply-To: <CAEbHw4ZB3MSvjv9uOqAOd6pOWO0FBxchyjDE5ZSveAmcToq+0A@mail.gmail.com>
References: <378a7b7d-eb60-0a90-1f35-802913b6b501@redhat.com>
 <pjcvqh$t6e$1@blaine.gmane.org>
 <CAEbHw4ZB3MSvjv9uOqAOd6pOWO0FBxchyjDE5ZSveAmcToq+0A@mail.gmail.com>
Message-ID: <CADqi795grCcoBr7SEOyEmi-QKfkp0a4JVdG=CfjX8kH-sOZ=Rw@mail.gmail.com>

Sounds good to me. It'll make OrderedDict look much nicer at the REPL.


On Thu, Jul 26, 2018 at 10:21 AM Terry Reedy <tjreedy at udel.edu> wrote:

> I am not sure what our repr change policy is, as there is a
> back-compatibility issue but I remember there being changes.


Given the way different Python prompts feel free to change reprs, I always
understood the repr to have no compatibility guarantees.


On Thu, Jul 26, 2018 at 10:31 AM David Mertz <mertz at gnosis.cx> wrote:

> Repr changes are likely to break doctests, at least. That's not
> necessarily an absolute "no", but it does raise the bar.
>

Breaking doctests is somewhat annoying, but I understand that to be more
about verifying the accuracy of documentation than verifying the
correctness of code. It's handy for following TDD, but that's during the
initial development and a changed repr won't interfere.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180726/2c1a1663/attachment.html>

From mertz at gnosis.cx  Thu Jul 26 18:50:20 2018
From: mertz at gnosis.cx (David Mertz)
Date: Thu, 26 Jul 2018 18:50:20 -0400
Subject: [Python-ideas] Change repr of collections.OrderedDict to be
 more dict-like
In-Reply-To: <CADqi795grCcoBr7SEOyEmi-QKfkp0a4JVdG=CfjX8kH-sOZ=Rw@mail.gmail.com>
References: <378a7b7d-eb60-0a90-1f35-802913b6b501@redhat.com>
 <pjcvqh$t6e$1@blaine.gmane.org>
 <CAEbHw4ZB3MSvjv9uOqAOd6pOWO0FBxchyjDE5ZSveAmcToq+0A@mail.gmail.com>
 <CADqi795grCcoBr7SEOyEmi-QKfkp0a4JVdG=CfjX8kH-sOZ=Rw@mail.gmail.com>
Message-ID: <CAEbHw4ZRpb8CJGv-_B8QeCJ2zU7fWK7bMTXDqrGuuTtv7-iDJw@mail.gmail.com>

I often use doctests to verify APIs and behaviors when I update code. I
know I'm in a minority and most developers slightly disparage doctests.

On Thu, Jul 26, 2018, 6:41 PM Michael Selik <mike at selik.org> wrote:

> Sounds good to me. It'll make OrderedDict look much nicer at the REPL.
>
>
> On Thu, Jul 26, 2018 at 10:21 AM Terry Reedy <tjreedy at udel.edu> wrote:
>
>> I am not sure what our repr change policy is, as there is a
>> back-compatibility issue but I remember there being changes.
>
>
> Given the way different Python prompts feel free to change reprs, I always
> understood the repr to have no compatibility guarantees.
>
>
> On Thu, Jul 26, 2018 at 10:31 AM David Mertz <mertz at gnosis.cx> wrote:
>
>> Repr changes are likely to break doctests, at least. That's not
>> necessarily an absolute "no", but it does raise the bar.
>>
>
> Breaking doctests is somewhat annoying, but I understand that to be more
> about verifying the accuracy of documentation than verifying the
> correctness of code. It's handy for following TDD, but that's during the
> initial development and a changed repr won't interfere.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180726/88e36245/attachment.html>

From J.Demeyer at UGent.be  Thu Jul 26 19:01:30 2018
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Fri, 27 Jul 2018 01:01:30 +0200
Subject: [Python-ideas] Change repr of collections.OrderedDict to be
 more dict-like
In-Reply-To: <78c8f2d255e1458fa61fd5a451920ed2@xmail101.UGent.be>
References: <378a7b7d-eb60-0a90-1f35-802913b6b501@redhat.com>
 <pjcvqh$t6e$1@blaine.gmane.org>
 <CAEbHw4ZB3MSvjv9uOqAOd6pOWO0FBxchyjDE5ZSveAmcToq+0A@mail.gmail.com>
 <CADqi795grCcoBr7SEOyEmi-QKfkp0a4JVdG=CfjX8kH-sOZ=Rw@mail.gmail.com>
 <78c8f2d255e1458fa61fd5a451920ed2@xmail101.UGent.be>
Message-ID: <5B5A52CA.1070800@UGent.be>

On 2018-07-27 00:50, David Mertz wrote:
> I often use doctests to verify APIs and behaviors when I update code. I
> know I'm in a minority and most developers slightly disparage doctests.

If it makes you feel better, SageMath development uses only doctests for 
testing. We have over 300000 doctests and it works great!

We do use a customized doctesting framework though.

From mike at selik.org  Thu Jul 26 19:20:23 2018
From: mike at selik.org (Michael Selik)
Date: Thu, 26 Jul 2018 16:20:23 -0700
Subject: [Python-ideas] Python docs page: In what ways is None special
In-Reply-To: <CALD=Yf_PRXZYiNdC9FGE5pd1cjrS8gyr8q5iPxzrMKXzX027YA@mail.gmail.com>
References: <CALD=Yf_PRXZYiNdC9FGE5pd1cjrS8gyr8q5iPxzrMKXzX027YA@mail.gmail.com>
Message-ID: <CADqi797wz1kUdV0nyG5YdGGxrt3sTkbFTyevEYm_nGDVfnLYyQ@mail.gmail.com>

On Mon, Jul 23, 2018 at 2:03 AM Jonathan Fine <jfine2358 at gmail.com> wrote:

> Thank you for your attention. What have I missed?
>

None and a few other things are special-cased by CPython. The compiler
won't bother to write bytecode instructions when an if-statement obviously
evaluates false. That might surprise some folks.

    In [1]: import dis

    In [2]: def foo():
       ...:     if None:
       ...:         print(1)
       ...:     if 0:
       ...:         print(2)
       ...:     if 'a' == 'b':
       ...:         print(3)
       ...:

    In [3]: dis.dis(foo)
      6           0 LOAD_CONST               1 ('a')
                  2 LOAD_CONST               2 ('b')
                  4 COMPARE_OP               2 (==)
                  6 POP_JUMP_IF_FALSE       16

      7           8 LOAD_GLOBAL              0 (print)
                 10 LOAD_CONST               3 (3)
                 12 CALL_FUNCTION            1
                 14 POP_TOP
            >>   16 LOAD_CONST               0 (None)
                 18 RETURN_VALUE
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180726/408faa93/attachment.html>

From rosuav at gmail.com  Thu Jul 26 19:27:33 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 27 Jul 2018 09:27:33 +1000
Subject: [Python-ideas] Python docs page: In what ways is None special
In-Reply-To: <CADqi797wz1kUdV0nyG5YdGGxrt3sTkbFTyevEYm_nGDVfnLYyQ@mail.gmail.com>
References: <CALD=Yf_PRXZYiNdC9FGE5pd1cjrS8gyr8q5iPxzrMKXzX027YA@mail.gmail.com>
 <CADqi797wz1kUdV0nyG5YdGGxrt3sTkbFTyevEYm_nGDVfnLYyQ@mail.gmail.com>
Message-ID: <CAPTjJmrp41srr4GOFxhdfMEL9KgVVf0uDXx=0Y5CzQh_RYmSTg@mail.gmail.com>

On Fri, Jul 27, 2018 at 9:20 AM, Michael Selik <mike at selik.org> wrote:
> On Mon, Jul 23, 2018 at 2:03 AM Jonathan Fine <jfine2358 at gmail.com> wrote:
>>
>> Thank you for your attention. What have I missed?
>
>
> None and a few other things are special-cased by CPython. The compiler won't
> bother to write bytecode instructions when an if-statement obviously
> evaluates false. That might surprise some folks.
>
>     In [1]: import dis
>
>     In [2]: def foo():
>        ...:     if None:
>        ...:         print(1)
>        ...:     if 0:
>        ...:         print(2)
>        ...:     if 'a' == 'b':
>        ...:         print(3)
>        ...:
>
>     In [3]: dis.dis(foo)
>       6           0 LOAD_CONST               1 ('a')
>                   2 LOAD_CONST               2 ('b')
>                   4 COMPARE_OP               2 (==)
>                   6 POP_JUMP_IF_FALSE       16
>
>       7           8 LOAD_GLOBAL              0 (print)
>                  10 LOAD_CONST               3 (3)
>                  12 CALL_FUNCTION            1
>                  14 POP_TOP
>             >>   16 LOAD_CONST               0 (None)
>                  18 RETURN_VALUE
>

That's true of ANY constant, though. And a future version of Python
could well constant-fold the comparison, and thus optimize out the
third condition too. There's nothing special about None here.

ChrisA

From steve at pearwood.info  Fri Jul 27 01:22:47 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 27 Jul 2018 15:22:47 +1000
Subject: [Python-ideas] Change repr of collections.OrderedDict to be
 more dict-like
In-Reply-To: <CAEbHw4ZRpb8CJGv-_B8QeCJ2zU7fWK7bMTXDqrGuuTtv7-iDJw@mail.gmail.com>
References: <378a7b7d-eb60-0a90-1f35-802913b6b501@redhat.com>
 <pjcvqh$t6e$1@blaine.gmane.org>
 <CAEbHw4ZB3MSvjv9uOqAOd6pOWO0FBxchyjDE5ZSveAmcToq+0A@mail.gmail.com>
 <CADqi795grCcoBr7SEOyEmi-QKfkp0a4JVdG=CfjX8kH-sOZ=Rw@mail.gmail.com>
 <CAEbHw4ZRpb8CJGv-_B8QeCJ2zU7fWK7bMTXDqrGuuTtv7-iDJw@mail.gmail.com>
Message-ID: <20180727052246.GA22554@ando.pearwood.info>

On Thu, Jul 26, 2018 at 06:50:20PM -0400, David Mertz wrote:
> I often use doctests to verify APIs and behaviors when I update code. I
> know I'm in a minority and most developers slightly disparage doctests.

I don't know why you think that "most developers" disparage doctests. 
Perhaps they're the same ones who think that "read the source" is all 
the documentation anyone needs.

I know I don't. I love them. Tim Peters loves them (he should, he wrote 
the module) and if its good enough for Uncle Timmy then anyone who 
disparages them better have a damn good reason.

Doc tests aren't perfect, but they fill a very important niche which is 
otherwise badly neglected by developers.

I think we'd need to ask on Python-Dev to be sure, but my understanding 
is that backwards compatibility guarantees for reprs are weaker than 
those for regular APIs but stronger than those for error messages.

Earlier, Michael Selik <mike at selik.org> wrote:

> > Given the way different Python prompts feel free to change reprs, I always
> > understood the repr to have no compatibility guarantees.

That doesn't follow. The Python language makes no guarantees about 
how applications can display objects to the user, and a REPL is just 
another application. I suppose there might be an implied expectation 
that the official REPL that comes with the standard interpreter probably 
shouldn't make too many changes to the way things are displayed, but 
that's as far as it goes.

[Michael (I think)]
> > Breaking doctests is somewhat annoying, but I understand that to be more
> > about verifying the accuracy of documentation than verifying the
> > correctness of code.

Breaking doctests is more than "somewhat annoying", and you seem to have 
missed the point that the way we verify the accuracy of documentation is 
by verifying the correctness of code.

> > It's handy for following TDD, but that's during the
> > initial development and a changed repr won't interfere.

I don't understand the reasoning here. Surely doctests are *less* 
valuable during initial development, when function APIs are in rapid 
change. (On the other hand, I use doctests to nail down what API I 
want, before I have an implementation.)

Unless you delete all your documentation or stop running doctests when 
the project reaches the first stable version, I don't understand why you 
think that changing reprs won't interfere.



-- 
Steve

From robertvandeneynde at hotmail.com  Fri Jul 27 01:30:35 2018
From: robertvandeneynde at hotmail.com (Robert Vanden Eynde)
Date: Fri, 27 Jul 2018 05:30:35 +0000
Subject: [Python-ideas] Change repr of collections.OrderedDict to be
 more dict-like
In-Reply-To: <20180727052246.GA22554@ando.pearwood.info>
References: <378a7b7d-eb60-0a90-1f35-802913b6b501@redhat.com>
 <pjcvqh$t6e$1@blaine.gmane.org>
 <CAEbHw4ZB3MSvjv9uOqAOd6pOWO0FBxchyjDE5ZSveAmcToq+0A@mail.gmail.com>
 <CADqi795grCcoBr7SEOyEmi-QKfkp0a4JVdG=CfjX8kH-sOZ=Rw@mail.gmail.com>
 <CAEbHw4ZRpb8CJGv-_B8QeCJ2zU7fWK7bMTXDqrGuuTtv7-iDJw@mail.gmail.com>
 <20180727052246.GA22554@ando.pearwood.info>
Message-ID: <VI1PR03MB2909BBCE8C9ACC0C81DF3050B62A0@VI1PR03MB2909.eurprd03.prod.outlook.com>

Currently, what's the best way to implement a function f(OrderedDict([(1,2),(3,4)])) == '{1: 2, 3: 4}', works for all possible types, and also availaible for pprint with nice indent?
If I could set a parameter in ipython or python repl that would print that, that would already be very useful.

Le ven. 27 juil. 2018 ? 07:23, Steven D'Aprano <steve at pearwood.info<mailto:steve at pearwood.info>> a ?crit :
On Thu, Jul 26, 2018 at 06:50:20PM -0400, David Mertz wrote:
> I often use doctests to verify APIs and behaviors when I update code. I
> know I'm in a minority and most developers slightly disparage doctests.

I don't know why you think that "most developers" disparage doctests.
Perhaps they're the same ones who think that "read the source" is all
the documentation anyone needs.

I know I don't. I love them. Tim Peters loves them (he should, he wrote
the module) and if its good enough for Uncle Timmy then anyone who
disparages them better have a damn good reason.

Doc tests aren't perfect, but they fill a very important niche which is
otherwise badly neglected by developers.

I think we'd need to ask on Python-Dev to be sure, but my understanding
is that backwards compatibility guarantees for reprs are weaker than
those for regular APIs but stronger than those for error messages.

Earlier, Michael Selik <mike at selik.org<mailto:mike at selik.org>> wrote:

> > Given the way different Python prompts feel free to change reprs, I always
> > understood the repr to have no compatibility guarantees.

That doesn't follow. The Python language makes no guarantees about
how applications can display objects to the user, and a REPL is just
another application. I suppose there might be an implied expectation
that the official REPL that comes with the standard interpreter probably
shouldn't make too many changes to the way things are displayed, but
that's as far as it goes.

[Michael (I think)]
> > Breaking doctests is somewhat annoying, but I understand that to be more
> > about verifying the accuracy of documentation than verifying the
> > correctness of code.

Breaking doctests is more than "somewhat annoying", and you seem to have
missed the point that the way we verify the accuracy of documentation is
by verifying the correctness of code.

> > It's handy for following TDD, but that's during the
> > initial development and a changed repr won't interfere.

I don't understand the reasoning here. Surely doctests are *less*
valuable during initial development, when function APIs are in rapid
change. (On the other hand, I use doctests to nail down what API I
want, before I have an implementation.)

Unless you delete all your documentation or stop running doctests when
the project reaches the first stable version, I don't understand why you
think that changing reprs won't interfere.



--
Steve
_______________________________________________
Python-ideas mailing list
Python-ideas at python.org<mailto:Python-ideas at python.org>
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180727/f6ff2ffb/attachment.html>

From steve at pearwood.info  Fri Jul 27 01:58:40 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 27 Jul 2018 15:58:40 +1000
Subject: [Python-ideas] Change repr of collections.OrderedDict to be
 more dict-like
In-Reply-To: <VI1PR03MB2909BBCE8C9ACC0C81DF3050B62A0@VI1PR03MB2909.eurprd03.prod.outlook.com>
References: <378a7b7d-eb60-0a90-1f35-802913b6b501@redhat.com>
 <pjcvqh$t6e$1@blaine.gmane.org>
 <CAEbHw4ZB3MSvjv9uOqAOd6pOWO0FBxchyjDE5ZSveAmcToq+0A@mail.gmail.com>
 <CADqi795grCcoBr7SEOyEmi-QKfkp0a4JVdG=CfjX8kH-sOZ=Rw@mail.gmail.com>
 <CAEbHw4ZRpb8CJGv-_B8QeCJ2zU7fWK7bMTXDqrGuuTtv7-iDJw@mail.gmail.com>
 <20180727052246.GA22554@ando.pearwood.info>
 <VI1PR03MB2909BBCE8C9ACC0C81DF3050B62A0@VI1PR03MB2909.eurprd03.prod.outlook.com>
Message-ID: <20180727055840.GC22554@ando.pearwood.info>

On Fri, Jul 27, 2018 at 05:30:35AM +0000, Robert Vanden Eynde wrote:

> Currently, what's the best way to implement a function 
> f(OrderedDict([(1,2),(3,4)])) == '{1: 2, 3: 4}', works for all 
> possible types, and also availaible for pprint with nice indent?

I don't understand the question, and I especially don't understand the 
"all possible types" part. Do you actually mean *all* possible types, 
like int, float, str, MyClassThatDoesSomethingWeird?

But guessing (possibly incorrectly) what you want:

py> from collections import OrderedDict
py> od = OrderedDict([(1,2),(3,4)])
py> od == {1:2, 3: 4}
True

If efficiency is no concern, then the simplest way to get something that 
has a dict repr from an OrderedDict is to use a dict:

py> dict(od)
{1: 2, 3: 4}

This also works as OrderedDict is a subclass of dict, and should avoid 
the cost of building an entire dict:

py> dict.__repr__(od)
'{1: 2, 3: 4}'


> If I 
> could set a parameter in ipython or python repl that would print that, 
> that would already be very useful.

See sys.displayhook.



-- 
Steve

From robertvandeneynde at hotmail.com  Fri Jul 27 02:06:57 2018
From: robertvandeneynde at hotmail.com (Robert Vanden Eynde)
Date: Fri, 27 Jul 2018 06:06:57 +0000
Subject: [Python-ideas] Change repr of collections.OrderedDict to be
 more dict-like
In-Reply-To: <20180727055840.GC22554@ando.pearwood.info>
References: <378a7b7d-eb60-0a90-1f35-802913b6b501@redhat.com>
 <pjcvqh$t6e$1@blaine.gmane.org>
 <CAEbHw4ZB3MSvjv9uOqAOd6pOWO0FBxchyjDE5ZSveAmcToq+0A@mail.gmail.com>
 <CADqi795grCcoBr7SEOyEmi-QKfkp0a4JVdG=CfjX8kH-sOZ=Rw@mail.gmail.com>
 <CAEbHw4ZRpb8CJGv-_B8QeCJ2zU7fWK7bMTXDqrGuuTtv7-iDJw@mail.gmail.com>
 <20180727052246.GA22554@ando.pearwood.info>
 <VI1PR03MB2909BBCE8C9ACC0C81DF3050B62A0@VI1PR03MB2909.eurprd03.prod.outlook.com>
 <20180727055840.GC22554@ando.pearwood.info>
Message-ID: <VI1PR03MB2909BA6A32DF1D7927708BF8B62A0@VI1PR03MB2909.eurprd03.prod.outlook.com>

Thanks for your response, I want to print/repr an OrderedDict() without relying on the fact that "dict are ordered" ie. I want a solution < python 3.7.
Currently, if I do repr( OrderedDict([(1,2),(3,4)]) ), I get the string "OrderedDict([(1,2),(3,4)])", I'd like a function that would return the string "{1: 2, 3: 4}" in the correct order.
If I do repr(dict( OrderedDict([(1,2),(3,4)]) )) I get "{1: 2, 3: 4}" because dict are ordered since python 3.7.
And for pprint, currently pformat( OrderedDict([(1,2),(3,4)]) ) gives the string 'OrderedDict([(1, 2), (3, 4)])' (and adds \n for bigger dict).
I could do pprint(dict( OrderedDict([(1,2),(3,4)]) )) but again that relies on python 3.7 behavior.
I'm wondering if there exists an easy way to code this "order preserving repr and pprint/pformat".

Le ven. 27 juil. 2018 ? 07:58, Steven D'Aprano <steve at pearwood.info<mailto:steve at pearwood.info>> a ?crit :
On Fri, Jul 27, 2018 at 05:30:35AM +0000, Robert Vanden Eynde wrote:

> Currently, what's the best way to implement a function
> f(OrderedDict([(1,2),(3,4)])) == '{1: 2, 3: 4}', works for all
> possible types, and also availaible for pprint with nice indent?

I don't understand the question, and I especially don't understand the
"all possible types" part. Do you actually mean *all* possible types,
like int, float, str, MyClassThatDoesSomethingWeird?

But guessing (possibly incorrectly) what you want:

py> from collections import OrderedDict
py> od = OrderedDict([(1,2),(3,4)])
py> od == {1:2, 3: 4}
True

If efficiency is no concern, then the simplest way to get something that
has a dict repr from an OrderedDict is to use a dict:

py> dict(od)
{1: 2, 3: 4}

This also works as OrderedDict is a subclass of dict, and should avoid
the cost of building an entire dict:

py> dict.__repr__(od)
'{1: 2, 3: 4}'


> If I
> could set a parameter in ipython or python repl that would print that,
> that would already be very useful.

See sys.displayhook.



--
Steve
_______________________________________________
Python-ideas mailing list
Python-ideas at python.org<mailto:Python-ideas at python.org>
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180727/c84a7d89/attachment.html>

From steve at pearwood.info  Fri Jul 27 02:35:42 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 27 Jul 2018 16:35:42 +1000
Subject: [Python-ideas] As-do statements/anonymous blocks in python
In-Reply-To: <CANQHGB2aCT3KOGnDP-qxA4PfV=34oiPGRO+rrbMBJG51K_4T-g@mail.gmail.com>
References: <CANQHGB2aCT3KOGnDP-qxA4PfV=34oiPGRO+rrbMBJG51K_4T-g@mail.gmail.com>
Message-ID: <20180727063542.GD22554@ando.pearwood.info>

I *think* you are trying to find a syntax for "code blocks" like Ruby
has, but I'm not sure. Your examples are a little confusing to me 
(possibly because you have at least three separate suggestions here, 
"as" blocks, a mysterious "do" keyword I don't understand, and partial 
application using % as argument placeholder).

Many people have tried to find a syntax for code block expressions, none 
have succeeded. Please note that being an *expression* is critical. We 
already have syntax for code block statements, it is the "def" keyword.

You probably ought to spend a few days searching the mailing list 
archives for previous discussions before you get your hopes up.

Before I spend any more time on this proposal, I'll skip right to the
end of your post:


On Wed, Jul 25, 2018 at 01:07:50PM -0400, James Lu wrote:
 
> I?m not totally sure of practical uses,

If you don't have practical uses, this proposal has ZERO chance of being
accepted.

Sometimes we can't get proposals accepted even with practical uses.


> but I?m imagining it would make
> passing a function to another function much more convenient.

I don't see how it can be more convenient than:

function(another_function)



-- 
Steve

From python-ideas-2018 at tutnicht.de  Fri Jul 27 03:48:51 2018
From: python-ideas-2018 at tutnicht.de (=?iso-8859-1?Q?J=F6rn?= Heissler)
Date: Fri, 27 Jul 2018 09:48:51 +0200
Subject: [Python-ideas] Add function readbyte to asyncio.StreamReader
In-Reply-To: <CAO-CpEKfFwNo__dyL0260dRbYZyuiUEC9tYT3KJxD8V6jHYQ1g@mail.gmail.com>
References: <20180722110340.GA23320@carrot.tutnicht.de>
 <CAO-CpEKfFwNo__dyL0260dRbYZyuiUEC9tYT3KJxD8V6jHYQ1g@mail.gmail.com>
Message-ID: <20180727074851.GB25280@carrot.tutnicht.de>

On Mon, Jul 23, 2018 at 22:25:14 +0100, Gustavo Carneiro wrote:
> Well, even if it is worth, i.e. your use case is not rare enough,

Reading a single byte is certainly a special case, but I think it's generic
enough to warrant its own function.
I believe there should be many binary protocols out there that would benefit
from such a function.

For example the first socks5 (rfc1928) message could be parsed like this:

async def read_methods(reader):
    if (version := await reader.readbyte()) != 5:
        raise Exception(f'Bad version: {version}')
    if (nmethods := await reader.readbyte()) == 0:
        raise Exception('Need at least one method')
    return await reader.readexactly(nmethods)

> I would
> suggest at least making it private, readexactly can call this specialised
> function if nbytes==1:
> 
> def _readbyte(self):
>    ....
> 
> def readexactly(self, num):
>    if num == 1:
>       return self._readbyte()
>   ... the rest stays the same..

Maybe I wasn't clear in my intent: readbyte would not return a bytes object but
an integer, i.e. the byte value.

My current approach is this:

value = (await reader.readexactly(1))[0]

I'd like to make it more readable (and faster at the same time):

value = await reader.readbyte()

> But to be honest, you are probably better off managing the buffer yourself:
> Just call, e.g., stream.read(4096), it will return a buffer of up to 4k
> length, then you can iterate over the buffer byte by byte until the
> condition is met, repeat until the end of stream, or whatever.

StreamReader already manages a buffer. Managing a second buffer would
mean I'd need to copy all my data from one buffer to another.

But let's assume I went this way and iterated over my own buffer:

* I receive some bytes. Maybe it's exactly the amount I need, then I can parse
  it and discard the buffer.
* Or it's less than I need. I'd have to wait for more data and either restart my
  parser or remember the state from before.
* Or it's more than I need. I'd have to remove the parsed bytes from the buffer.
  Alternatively I could push back the unparsed bytes to my buffer.

This adds lots of code complexity. And the code is likely slower than calling
readbyte() couple of times; for my current use case, calling it once is usually
sufficient.

I like my current approach way better than managing my own buffer and thus
reinventing StreamReader.

Adding the new function as proposed would improve the code in both readability
and speed.

From tjol at tjol.eu  Fri Jul 27 05:45:03 2018
From: tjol at tjol.eu (Thomas Jollans)
Date: Fri, 27 Jul 2018 11:45:03 +0200
Subject: [Python-ideas] Change repr of collections.OrderedDict to be
 more dict-like
In-Reply-To: <VI1PR03MB2909BA6A32DF1D7927708BF8B62A0@VI1PR03MB2909.eurprd03.prod.outlook.com>
References: <378a7b7d-eb60-0a90-1f35-802913b6b501@redhat.com>
 <pjcvqh$t6e$1@blaine.gmane.org>
 <CAEbHw4ZB3MSvjv9uOqAOd6pOWO0FBxchyjDE5ZSveAmcToq+0A@mail.gmail.com>
 <CADqi795grCcoBr7SEOyEmi-QKfkp0a4JVdG=CfjX8kH-sOZ=Rw@mail.gmail.com>
 <CAEbHw4ZRpb8CJGv-_B8QeCJ2zU7fWK7bMTXDqrGuuTtv7-iDJw@mail.gmail.com>
 <20180727052246.GA22554@ando.pearwood.info>
 <VI1PR03MB2909BBCE8C9ACC0C81DF3050B62A0@VI1PR03MB2909.eurprd03.prod.outlook.com>
 <20180727055840.GC22554@ando.pearwood.info>
 <VI1PR03MB2909BA6A32DF1D7927708BF8B62A0@VI1PR03MB2909.eurprd03.prod.outlook.com>
Message-ID: <fd2de256-331e-8cd1-0561-f80f26268cf1@tjol.eu>

On 27/07/18 08:06, Robert Vanden Eynde wrote:
> Thanks for your response, I want to print/repr an OrderedDict() 
> without relying on the fact that "dict are ordered" ie. I want a 
> solution < python 3.7.
> Currently, if I do repr( OrderedDict([(1,2),(3,4)]) ), I get the 
> string "OrderedDict([(1,2),(3,4)])", I'd like a function that would 
> return the string "{1: 2, 3: 4}" in the correct order.
> If I do repr(dict( OrderedDict([(1,2),(3,4)]) )) I get "{1: 2, 3: 4}" 
> because dict are ordered since python 3.7.
> And for pprint, currently pformat( OrderedDict([(1,2),(3,4)]) ) gives 
> the string 'OrderedDict([(1, 2), (3, 4)])' (and adds \n for bigger dict).
> I could do pprint(dict( OrderedDict([(1,2),(3,4)]) )) but again that 
> relies on python 3.7 behavior.
> I'm wondering if there exists an easy way to code this "order 
> preserving repr and pprint/pformat".

It's a fairly non-standard thing to do as you're not representing the 
ordered dict itself, but it's easy enough...

 >>> od = OrderedDict([('a', 1), ('b', 2)])
 >>> '{%s}' % ', '.join('{!r}: {!r}'.format(k, v) for (k, v) in? od.items())
"{'a': 1, 'b': 2}"
 >>>

It's a bit more work if you want pretty-printing.

And there's always json.

>
> Le?ven. 27 juil. 2018 ??07:58, Steven D'Aprano <steve at pearwood.info 
> <mailto:steve at pearwood.info>> a ?crit?:
>
>     On Fri, Jul 27, 2018 at 05:30:35AM +0000, Robert Vanden Eynde wrote:
>
>     > Currently, what's the best way to implement a function
>     > f(OrderedDict([(1,2),(3,4)])) == '{1: 2, 3: 4}', works for all
>     > possible types, and also availaible for pprint with nice indent?
>
>     I don't understand the question, and I especially don't understand
>     the
>     "all possible types" part. Do you actually mean *all* possible types,
>     like int, float, str, MyClassThatDoesSomethingWeird?
>
>     But guessing (possibly incorrectly) what you want:
>
>     py> from collections import OrderedDict
>     py> od = OrderedDict([(1,2),(3,4)])
>     py> od == {1:2, 3: 4}
>     True
>
>     If efficiency is no concern, then the simplest way to get
>     something that
>     has a dict repr from an OrderedDict is to use a dict:
>
>     py> dict(od)
>     {1: 2, 3: 4}
>
>     This also works as OrderedDict is a subclass of dict, and should
>     avoid
>     the cost of building an entire dict:
>
>     py> dict.__repr__(od)
>     '{1: 2, 3: 4}'
>
>
>     > If I
>     > could set a parameter in ipython or python repl that would print
>     that,
>     > that would already be very useful.
>
>     See sys.displayhook.
>
>
>
>     -- 
>     Steve
>     _______________________________________________
>     Python-ideas mailing list
>     Python-ideas at python.org <mailto:Python-ideas at python.org>
>     https://mail.python.org/mailman/listinfo/python-ideas
>     Code of Conduct: http://python.org/psf/codeofconduct/
>     <http://python.org/psf/codeofconduct/>
>
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180727/5e814402/attachment-0001.html>

From rosuav at gmail.com  Fri Jul 27 05:53:50 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 27 Jul 2018 19:53:50 +1000
Subject: [Python-ideas] Change repr of collections.OrderedDict to be
 more dict-like
In-Reply-To: <fd2de256-331e-8cd1-0561-f80f26268cf1@tjol.eu>
References: <378a7b7d-eb60-0a90-1f35-802913b6b501@redhat.com>
 <pjcvqh$t6e$1@blaine.gmane.org>
 <CAEbHw4ZB3MSvjv9uOqAOd6pOWO0FBxchyjDE5ZSveAmcToq+0A@mail.gmail.com>
 <CADqi795grCcoBr7SEOyEmi-QKfkp0a4JVdG=CfjX8kH-sOZ=Rw@mail.gmail.com>
 <CAEbHw4ZRpb8CJGv-_B8QeCJ2zU7fWK7bMTXDqrGuuTtv7-iDJw@mail.gmail.com>
 <20180727052246.GA22554@ando.pearwood.info>
 <VI1PR03MB2909BBCE8C9ACC0C81DF3050B62A0@VI1PR03MB2909.eurprd03.prod.outlook.com>
 <20180727055840.GC22554@ando.pearwood.info>
 <VI1PR03MB2909BA6A32DF1D7927708BF8B62A0@VI1PR03MB2909.eurprd03.prod.outlook.com>
 <fd2de256-331e-8cd1-0561-f80f26268cf1@tjol.eu>
Message-ID: <CAPTjJmrzEttTF6ieW8apyUr0Dph_MX7m3mZyamTVY38roSbt0g@mail.gmail.com>

On Fri, Jul 27, 2018 at 7:45 PM, Thomas Jollans <tjol at tjol.eu> wrote:
> On 27/07/18 08:06, Robert Vanden Eynde wrote:
>
> Thanks for your response, I want to print/repr an OrderedDict() without
> relying on the fact that "dict are ordered" ie. I want a solution < python
> 3.7.
> Currently, if I do repr( OrderedDict([(1,2),(3,4)]) ), I get the string
> "OrderedDict([(1,2),(3,4)])", I'd like a function that would return the
> string "{1: 2, 3: 4}" in the correct order.
> If I do repr(dict( OrderedDict([(1,2),(3,4)]) )) I get "{1: 2, 3: 4}"
> because dict are ordered since python 3.7.
> And for pprint, currently pformat( OrderedDict([(1,2),(3,4)]) ) gives the
> string 'OrderedDict([(1, 2), (3, 4)])' (and adds \n for bigger dict).
> I could do pprint(dict( OrderedDict([(1,2),(3,4)]) )) but again that relies
> on python 3.7 behavior.
> I'm wondering if there exists an easy way to code this "order preserving
> repr and pprint/pformat".
>
>
> It's a fairly non-standard thing to do as you're not representing the
> ordered dict itself, but it's easy enough...
>
>>>> od = OrderedDict([('a', 1), ('b', 2)])
>>>> '{%s}' % ', '.join('{!r}: {!r}'.format(k, v) for (k, v) in  od.items())
> "{'a': 1, 'b': 2}"
>>>>
>
> It's a bit more work if you want pretty-printing.
>
> And there's always json.

>>> from collections import OrderedDict
>>> od = OrderedDict([('a', 1), ('b', 2)])
>>> dict.__repr__(od)
"{'a': 1, 'b': 2}"

:)

ChrisA

From tjreedy at udel.edu  Fri Jul 27 07:02:23 2018
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 27 Jul 2018 07:02:23 -0400
Subject: [Python-ideas] Change repr of collections.OrderedDict to be
 more dict-like
In-Reply-To: <pjcvus$t6e$3@blaine.gmane.org>
References: <378a7b7d-eb60-0a90-1f35-802913b6b501@redhat.com>
 <pjcvqh$t6e$1@blaine.gmane.org> <pjcvus$t6e$3@blaine.gmane.org>
Message-ID: <pjetvt$6o7$1@blaine.gmane.org>

On 7/26/2018 1:23 PM, Terry Reedy wrote:
> On 7/26/2018 1:21 PM, Terry Reedy wrote:
>> On python-idea,? Miro Hron?ok asked today whether we can change the 
>> OrderedDict repr from, for instance,
>>
>> OrderedDict([('a', '1'), ('b', '2')]) # to
>> OrderedDict({'a': '1', 'b': '2'})
>>
>> I am not sure what our repr change policy is, as there is a 
>> back-compatibility issue but I remember there being changes.
> 
> Whoops, I meant to send this query elsewhere, so I could answer properly 
> here.

I posted on pydev thread
[issue34221] Any plans to combine collections.OrderedDict with dict
and Raymond Hettinger, collections author and maintainer, replied:

We are allowed to change the repr in future versions of the language. 
Doing so does come at a cost though. There is a small performance 
penalty (see the timings below).  Some doctests will break.  And Python 
3.8 printed output in books and blog posts would get shuffled if typed 
in to Python 3.5 -- this is problematic because one of the few remaining 
use cases for OrderedDict is to write code that is compatible with older 
Pythons.

The proposed repr does look pretty but probably isn't worth the disruption.

$ python3.7 -m timeit -r 7 'from collections import OrderedDict' 
"OrderedDict([('a', '1'), ('b', '2')])"
200000 loops, best of 7: 1.12 usec per loop
$ python3.7 -m timeit -r 7 'from collections import OrderedDict' 
"OrderedDict({'a': '1', 'b': '2'})"
200000 loops, best of 7: 1.22 usec per loop
$ python3.7 -m timeit -r 7 'from collections import OrderedDict' 
"OrderedDict([('a', '1'), ('b', '2')])"
200000 loops, best of 7: 1.13 usec per loop
$ python3.7 -m timeit -r 7 'from collections import OrderedDict' 
"OrderedDict({'a': '1', 'b': '2'})"
200000 loops, best of 7: 1.2 usec per loop
$ python3.7 -m timeit -r 7 'from collections import OrderedDict' 
"OrderedDict([('a', '1'), ('b', '2')])"
200000 loops, best of 7: 1.12 usec per loop
$ python3.7 -m timeit -r 7 'from collections import OrderedDict' 
"OrderedDict({'a': '1', 'b': '2'})"
200000 loops, best of 7: 1.2 usec per loop

-- 
Terry Jan Reedy



From robertve92 at gmail.com  Fri Jul 27 07:24:22 2018
From: robertve92 at gmail.com (Robert Vanden Eynde)
Date: Fri, 27 Jul 2018 13:24:22 +0200
Subject: [Python-ideas] Change repr of collections.OrderedDict to be
 more dict-like
In-Reply-To: <CAPTjJmrzEttTF6ieW8apyUr0Dph_MX7m3mZyamTVY38roSbt0g@mail.gmail.com>
References: <378a7b7d-eb60-0a90-1f35-802913b6b501@redhat.com>
 <pjcvqh$t6e$1@blaine.gmane.org>
 <CAEbHw4ZB3MSvjv9uOqAOd6pOWO0FBxchyjDE5ZSveAmcToq+0A@mail.gmail.com>
 <CADqi795grCcoBr7SEOyEmi-QKfkp0a4JVdG=CfjX8kH-sOZ=Rw@mail.gmail.com>
 <CAEbHw4ZRpb8CJGv-_B8QeCJ2zU7fWK7bMTXDqrGuuTtv7-iDJw@mail.gmail.com>
 <20180727052246.GA22554@ando.pearwood.info>
 <VI1PR03MB2909BBCE8C9ACC0C81DF3050B62A0@VI1PR03MB2909.eurprd03.prod.outlook.com>
 <20180727055840.GC22554@ando.pearwood.info>
 <VI1PR03MB2909BA6A32DF1D7927708BF8B62A0@VI1PR03MB2909.eurprd03.prod.outlook.com>
 <fd2de256-331e-8cd1-0561-f80f26268cf1@tjol.eu>
 <CAPTjJmrzEttTF6ieW8apyUr0Dph_MX7m3mZyamTVY38roSbt0g@mail.gmail.com>
Message-ID: <CA+msPNnOaYiWcv3GxL9Dqi2kc0B2Vfp_DOWFiMxzOsiX-UK+8g@mail.gmail.com>

Sorry Steven I didn't see your clever dict.__repr__ solution that Chris
pointed out too. That's what I was looking for ^^

And now for pprint, apparently that doesn't seem difficult to implement.

About the part where "the representation looses the type information",
pformat(od) could be "OrderedDict{1:2, 3,4}" or "OrderedDict({1:2, 3:4})"
but I don't ask for repr or pprint to change, just curious about how to
implement that.

</my_parenthesis>

Le ven. 27 juil. 2018 ? 11:53, Chris Angelico <rosuav at gmail.com> a ?crit :

> On Fri, Jul 27, 2018 at 7:45 PM, Thomas Jollans <tjol at tjol.eu> wrote:
> > On 27/07/18 08:06, Robert Vanden Eynde wrote:
> >
> > Thanks for your response, I want to print/repr an OrderedDict() without
> > relying on the fact that "dict are ordered" ie. I want a solution <
> python
> > 3.7.
> > Currently, if I do repr( OrderedDict([(1,2),(3,4)]) ), I get the string
> > "OrderedDict([(1,2),(3,4)])", I'd like a function that would return the
> > string "{1: 2, 3: 4}" in the correct order.
> > If I do repr(dict( OrderedDict([(1,2),(3,4)]) )) I get "{1: 2, 3: 4}"
> > because dict are ordered since python 3.7.
> > And for pprint, currently pformat( OrderedDict([(1,2),(3,4)]) ) gives the
> > string 'OrderedDict([(1, 2), (3, 4)])' (and adds \n for bigger dict).
> > I could do pprint(dict( OrderedDict([(1,2),(3,4)]) )) but again that
> relies
> > on python 3.7 behavior.
> > I'm wondering if there exists an easy way to code this "order preserving
> > repr and pprint/pformat".
> >
> >
> > It's a fairly non-standard thing to do as you're not representing the
> > ordered dict itself, but it's easy enough...
> >
> >>>> od = OrderedDict([('a', 1), ('b', 2)])
> >>>> '{%s}' % ', '.join('{!r}: {!r}'.format(k, v) for (k, v) in
> od.items())
> > "{'a': 1, 'b': 2}"
> >>>>
> >
> > It's a bit more work if you want pretty-printing.
> >
> > And there's always json.
>
> >>> from collections import OrderedDict
> >>> od = OrderedDict([('a', 1), ('b', 2)])
> >>> dict.__repr__(od)
> "{'a': 1, 'b': 2}"
>
> :)
>
> ChrisA
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180727/8432389b/attachment.html>

From storchaka at gmail.com  Fri Jul 27 07:24:29 2018
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Fri, 27 Jul 2018 14:24:29 +0300
Subject: [Python-ideas] Change repr of collections.OrderedDict to be
 more dict-like
In-Reply-To: <CAPTjJmrzEttTF6ieW8apyUr0Dph_MX7m3mZyamTVY38roSbt0g@mail.gmail.com>
References: <378a7b7d-eb60-0a90-1f35-802913b6b501@redhat.com>
 <pjcvqh$t6e$1@blaine.gmane.org>
 <CAEbHw4ZB3MSvjv9uOqAOd6pOWO0FBxchyjDE5ZSveAmcToq+0A@mail.gmail.com>
 <CADqi795grCcoBr7SEOyEmi-QKfkp0a4JVdG=CfjX8kH-sOZ=Rw@mail.gmail.com>
 <CAEbHw4ZRpb8CJGv-_B8QeCJ2zU7fWK7bMTXDqrGuuTtv7-iDJw@mail.gmail.com>
 <20180727052246.GA22554@ando.pearwood.info>
 <VI1PR03MB2909BBCE8C9ACC0C81DF3050B62A0@VI1PR03MB2909.eurprd03.prod.outlook.com>
 <20180727055840.GC22554@ando.pearwood.info>
 <VI1PR03MB2909BA6A32DF1D7927708BF8B62A0@VI1PR03MB2909.eurprd03.prod.outlook.com>
 <fd2de256-331e-8cd1-0561-f80f26268cf1@tjol.eu>
 <CAPTjJmrzEttTF6ieW8apyUr0Dph_MX7m3mZyamTVY38roSbt0g@mail.gmail.com>
Message-ID: <pjev9a$fpm$1@blaine.gmane.org>

27.07.18 12:53, Chris Angelico ????:
>>>> from collections import OrderedDict
>>>> od = OrderedDict([('a', 1), ('b', 2)])
>>>> dict.__repr__(od)
> "{'a': 1, 'b': 2}"

dict.__repr__() can output items in wrong order.

 >>> from collections import OrderedDict
 >>> od = OrderedDict([('a', 1), ('b', 2)])
 >>> od.move_to_end('a')
 >>> print(repr(od))
OrderedDict([('b', 2), ('a', 1)])
 >>> print(dict.__repr__(od))
{'a': 1, 'b': 2}


From mertz at gnosis.cx  Fri Jul 27 07:46:46 2018
From: mertz at gnosis.cx (David Mertz)
Date: Fri, 27 Jul 2018 07:46:46 -0400
Subject: [Python-ideas] Change repr of collections.OrderedDict to be
 more dict-like
In-Reply-To: <20180727052246.GA22554@ando.pearwood.info>
References: <378a7b7d-eb60-0a90-1f35-802913b6b501@redhat.com>
 <pjcvqh$t6e$1@blaine.gmane.org>
 <CAEbHw4ZB3MSvjv9uOqAOd6pOWO0FBxchyjDE5ZSveAmcToq+0A@mail.gmail.com>
 <CADqi795grCcoBr7SEOyEmi-QKfkp0a4JVdG=CfjX8kH-sOZ=Rw@mail.gmail.com>
 <CAEbHw4ZRpb8CJGv-_B8QeCJ2zU7fWK7bMTXDqrGuuTtv7-iDJw@mail.gmail.com>
 <20180727052246.GA22554@ando.pearwood.info>
Message-ID: <CAEbHw4aoY6=BC=4uzQuZYCk=K4biNAu1dWYNOhwA0Nzz2ZOAsg@mail.gmail.com>

On Fri, Jul 27, 2018, 1:23 AM Steven D'Aprano <steve at pearwood.info> wrote:

> On Thu, Jul 26, 2018 at 06:50:20PM -0400, David Mertz wrote:
> > I often use doctests to verify APIs and behaviors when I update code. I
> > know I'm in a minority and most developers slightly disparage doctests.
>
> I don't know why you think that "most developers" disparage doctests.
> Perhaps they're the same ones who think that "read the source" is all the
> documentation anyone needs.
>

I'm not talking about folks saying not to use tests at all. I've just read
or heard a fair number of programmers say that doctests are not best
practice, and using unittest or nose or something similar is more
professional.

It was an offhand content not a scientific poll. I'm delighted that you
also use doctests, Steven. And if Timmy also does, I feel fully vindicated
:-).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180727/cdc4522e/attachment-0001.html>

From rhodri at kynesim.co.uk  Fri Jul 27 08:00:09 2018
From: rhodri at kynesim.co.uk (Rhodri James)
Date: Fri, 27 Jul 2018 13:00:09 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAKr=oZtfYM2_fxYnde76o+jvuc_Y3AH9R93BCY6BoNdriuKqKw@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAEbHw4Yzt1Wfr1iBw+mmTYWhh8b2SwyFLX1AHOL-4YP6H+_ChA@mail.gmail.com>
 <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>
 <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
 <CAKr=oZtfYM2_fxYnde76o+jvuc_Y3AH9R93BCY6BoNdriuKqKw@mail.gmail.com>
Message-ID: <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>

On 26/07/18 22:28, Abe Dillon wrote:
> The readability issue isn't just a matter of a new set of symbols to learn.
> It isn't even that the symbols are used in a non-intuitive way (though that
> doesn't help).

We could argue about how intuitive or not these operators are, but they 
are used in other languages, so they clearly aren't completely unintuitive.

> It's not even that the question mark, unlike the dot, is
> very visually obtrusive (which also doesn't help).

Being visually obtrusive would help a great deal.  Unfortunately I don't 
think it is visually obtrusive in the ?. and ?[] operators.

> It also screws with the
> grammar in an unexpected way. I would be fine with something like:
> 
> def func(data=None):
>      data ?= []
>      ...
> 
> If there were some corresponding dunder method to implement that behavior,
> but it's not clear to me that such a thing is possible.

We could conceivably have a dunder method that None implemented as 
returning the other argument and object implemented as returning itself. 
  I'm not really convinced that it's worth it, though.

> If it isn't, that
> would make `?=` (or '??=' though I don't see the need for double '?'s)

Because it's the in-place version of "??", obviously.

> a
> special statement, a special case that someone who's familiar with how
> other in-place operators work, would be thrown for a loop.

Really?  Quick straw poll: how often do people care how in-place 
operators are implemented?

> The other use
> cases look like they break down in a different manner than they actually do:
> 
> person?.name
> 
> looks like it would break down similar to
> 
> person().name
> 
> but it doesn't because '?' by itself is not an operator.
> similarly:
> 
> person?[0]
> 
> looks like it would break down similar to
> 
> person()[0]
> 
> but again, the operator is '?[...]' not '?'

Sorry, not buying this one.  As you said, "?" is not an operator, so 
"a?.b" clearly can't break down into "a? .b".  This all rather feels 
like you are trying to find justifications for not liking these 
operators rather just saying "I don't like them," and frankly I'd 
respect you more if you just said "I don't like them."

(For the record, I don't like most of them.  There are clear use cases 
for "??" and "??=", but the use cases I've seen for "?." and "?[]" 
aren't convincing to me.)

> This also raises the question of other operators that could be made null
> aware. Why not:
> 
> func?()
> 
> ?-num
> 
> ?~num
> 
> num ?+ other
> 
> Why not other reflective operators?
> 
> num ?+= other

Why not indeed.  I haven't seen any convincing use cases (and nor has 
the PEP, though I doubt Steve was looking for them), but they aren't 
inherently any more weird than "?." and "?[]".

> I think this could be helped by simply making '?' an operator with a very
> high order of operations, so that pretty-much any expression on the LHS is
> guarded for 'NoneType' AttributeErrors:
> 
> val = person[0]?
> val = person.name?

Um.  How is that supposed to parse?  "person[0]" has already been 
evaluated by the time the "?" operator gets anywhere near it.  Unless 
you are proposing some major surgery to the parser (which I'm pretty 
sure isn't going to happen), this has no chance of getting through. 
Good thing too, because it's a lot more surprising than the alternatives.

> I also disagree with the idea that None is special enough to warrant such
> special behavior. There are other special None-like objects (as David Mertz
> pointed out) like NaN and custom place-holders for cases where None is a
> valid argument.

NaN isn't used like None, though, and custom sentinels aren't that common.

> Yet another possibility is to make '?' signify a parameter who's default
> value is an expression that should be evaluated when necessary:
> 
> def func(data: List = []?):
>      ...
> 
> That gets closer to another often requested feature: delayed expression
> evaluation (as Steve D'Aprano pointed out).
> So there are other, possibly better uses for the '?' symbol. It's something
> that needs to be carefully considered.

While my immediate reaction is "yuk", your use of "?" does not conflict 
with any use in the PEP, so this is a red herring.

-- 
Rhodri James *-* Kynesim Ltd

From rosuav at gmail.com  Fri Jul 27 09:29:52 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 27 Jul 2018 23:29:52 +1000
Subject: [Python-ideas] Change repr of collections.OrderedDict to be
 more dict-like
In-Reply-To: <pjev9a$fpm$1@blaine.gmane.org>
References: <378a7b7d-eb60-0a90-1f35-802913b6b501@redhat.com>
 <pjcvqh$t6e$1@blaine.gmane.org>
 <CAEbHw4ZB3MSvjv9uOqAOd6pOWO0FBxchyjDE5ZSveAmcToq+0A@mail.gmail.com>
 <CADqi795grCcoBr7SEOyEmi-QKfkp0a4JVdG=CfjX8kH-sOZ=Rw@mail.gmail.com>
 <CAEbHw4ZRpb8CJGv-_B8QeCJ2zU7fWK7bMTXDqrGuuTtv7-iDJw@mail.gmail.com>
 <20180727052246.GA22554@ando.pearwood.info>
 <VI1PR03MB2909BBCE8C9ACC0C81DF3050B62A0@VI1PR03MB2909.eurprd03.prod.outlook.com>
 <20180727055840.GC22554@ando.pearwood.info>
 <VI1PR03MB2909BA6A32DF1D7927708BF8B62A0@VI1PR03MB2909.eurprd03.prod.outlook.com>
 <fd2de256-331e-8cd1-0561-f80f26268cf1@tjol.eu>
 <CAPTjJmrzEttTF6ieW8apyUr0Dph_MX7m3mZyamTVY38roSbt0g@mail.gmail.com>
 <pjev9a$fpm$1@blaine.gmane.org>
Message-ID: <CAPTjJmo+jdDZvRCS94k9pfoTiOzwrag751wA6SQtqq-1y+dZPw@mail.gmail.com>

On Fri, Jul 27, 2018 at 9:24 PM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> 27.07.18 12:53, Chris Angelico ????:
>>>>>
>>>>> from collections import OrderedDict
>>>>> od = OrderedDict([('a', 1), ('b', 2)])
>>>>> dict.__repr__(od)
>>
>> "{'a': 1, 'b': 2}"
>
>
> dict.__repr__() can output items in wrong order.
>
>>>> from collections import OrderedDict
>>>> od = OrderedDict([('a', 1), ('b', 2)])
>>>> od.move_to_end('a')
>>>> print(repr(od))
> OrderedDict([('b', 2), ('a', 1)])
>>>> print(dict.__repr__(od))
> {'a': 1, 'b': 2}
>

Ah, fair point. Interestingly, the same problem hits repr(dict(od)),
which I would have thought a reliable solution here. The simplest way
that I've found is:

>>> dict(od.items())
{'b': 2, 'a': 1}

That seems very odd. Iterating over the OD produces its keys in the
correct order (b, a), but constructing a dict from it ignores
iteration order and just goes "oh hey, this is a dict, we can snag
that". Is that correct?

ChrisA

From peter.ed.oconnor at gmail.com  Fri Jul 27 13:02:37 2018
From: peter.ed.oconnor at gmail.com (Peter O'Connor)
Date: Fri, 27 Jul 2018 19:02:37 +0200
Subject: [Python-ideas] Can we add "zip and assert equal length" to the
 standard library?
Message-ID: <CACCbJKqb07XLG_bGOJzpDgi7C4of_ikbydqtWam5fa6Cz8JfnQ@mail.gmail.com>

I find that about 90% of the time I want want to zip iterators together, I
expect them to be the same length and want to throw an exception if they
aren't.  Yet there is not currently a solution for this in the standard
library for this, and as a result I always have to take this function
everywhere I go:


    def zip_equal(*iterables):
        """
        Zip and raise exception if lengths are not equal.

        Taken from solution by Martijn Pieters, here:

http://stackoverflow.com/questions/32954486/zip-iterators-asserting-for-equal-length-in-python

        :param iterables: Iterable objects
        :return: A new iterator outputting tuples where one element comes
from each iterable
        """
        sentinel = object()
        for combo in zip_longest(*iterables, fillvalue=sentinel):
            if any(sentinel is c for c in combo):
                raise ValueError('Iterables have different lengths.
Iterable(s) #{} (of 0..{}) ran out first.'.format([i for i, c in
enumerate(combo) if c is sentinel], len(combo)-1))
            yield combo

Would anybody object to adding this to the standard library for Python 3.8?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180727/d96f238f/attachment.html>

From tir.karthi at gmail.com  Fri Jul 27 14:08:09 2018
From: tir.karthi at gmail.com (Karthikeyan)
Date: Fri, 27 Jul 2018 11:08:09 -0700 (PDT)
Subject: [Python-ideas] Type annotations in TypeError messages - Improved
 error reporting
Message-ID: <cc1e92f0-f1cf-4e99-b211-3e1e7528c1a7@googlegroups.com>

Python 3 has support for type annotations. Normally when we call a function 
with too few arguments then it returns a TypeError message with required 
arguments name but it gives no information about the type of the variables 
when there are type annotations for the function. I have added a little 
patch to return type annotations for the parameters and return type when 
there are type errors. This doesn't affect the existing code but improves 
on the code that has type annotations. My C skills are low and there is a 
lot of room for improvement. Feedback welcome.

The code is available at 
https://github.com/tirkarthi/cpython/tree/error-message-annotations .  
Gist : https://gist.github.com/tirkarthi/8e6e01fde74398cac7a4eb7246169956

Sorry for the double post if you have received any since my previous post 
was rejected as I have not subscribed to the group.

A sample run is as below : 

# foo_signature.py

from typing import List, Dict
import sys

def get_profile_a(user_id: int, likes: Dict[str, int]) -> Dict[str, int]:
    return {'user_id': user_id, 'likes': len(likes)}

def get_profile_b(user_id, likes):
    return {'user_id': user_id, 'likes': len(likes.keys())}

if len(sys.argv) > 1:
    get_profile_a()
else:
    get_profile_b()

# Without type hinting

cpython git:(error-message-annotations) ./python foo_signature.py
Traceback (most recent call last):
  File "../backups/foo_signature.py", line 13, in <module>
    get_profile_b()
TypeError: get_profile_b() missing 2 required positional arguments: 
'user_id' and 'likes'

# With type hinting patch

cpython git:(error-message-annotations) ./python foo_signature.py 1
Traceback (most recent call last):
  File "../backups/foo_signature.py", line 11, in <module>
    get_profile_a()
TypeError: get_profile_a() missing 2 required positional arguments: 
'user_id' and 'likes'
annotation: (user_id: <class 'int'>, likes: typing.Dict[str, int], return: 
typing.Dict[str, int])
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180727/0298feea/attachment-0001.html>

From jfine2358 at gmail.com  Fri Jul 27 14:54:14 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Fri, 27 Jul 2018 19:54:14 +0100
Subject: [Python-ideas] Can we add "zip and assert equal length" to the
 standard library?
In-Reply-To: <CACCbJKqb07XLG_bGOJzpDgi7C4of_ikbydqtWam5fa6Cz8JfnQ@mail.gmail.com>
References: <CACCbJKqb07XLG_bGOJzpDgi7C4of_ikbydqtWam5fa6Cz8JfnQ@mail.gmail.com>
Message-ID: <CALD=Yf_XzjvxSi-xFSaNHOMR6nXKXrkEPqe9FPDCYYBRug3q9g@mail.gmail.com>

Hi Peter

Thank you for your deferred default values idea, which we're now
working on together.
https://github.com/petered/peters_example_code/blob/master/peters_example_code/deferral.py

You wrote:
> I find that about 90% of the time I want want to zip iterators together, I
> expect them to be the same length and want to throw an exception if they
> aren't.  Yet there is not currently a solution for this in the standard
> library for this, and as a result I always have to take this function
> everywhere I go
[function definition snipped]

You need this function wherever you go. You can solve that problem
right now by putting it into PyPI. And if you put it on
github, perhaps someone will fork it and contribute. (Note to self. I
should do the same for my own useful little tools.)

You asked about putting in the standard library. There's a process for that.
https://www.python.org/dev/peps/pep-0001/

You might also want to look at the PEP index, to find a successful PEP
that's similar to yours.
https://www.python.org/dev/peps/

Here's one I've found that looks similar. It proposes the inclusion of
a third-party module, pathlib, in the standard library.
https://www.python.org/dev/peps/pep-0428/

Putting your code into PyPI into PyPI solves your immediate problem.
It also also gives you a reference implementation.
https://www.python.org/dev/peps/pep-0001/#what-belongs-in-a-successful-pep

I hope your next post will be URL for code on github. I'd be
interested in making a fork.

Best wishes

Jonathan

From steve at pearwood.info  Fri Jul 27 19:27:32 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 28 Jul 2018 09:27:32 +1000
Subject: [Python-ideas] Can we add "zip and assert equal length" to the
 standard library?
In-Reply-To: <CACCbJKqb07XLG_bGOJzpDgi7C4of_ikbydqtWam5fa6Cz8JfnQ@mail.gmail.com>
References: <CACCbJKqb07XLG_bGOJzpDgi7C4of_ikbydqtWam5fa6Cz8JfnQ@mail.gmail.com>
Message-ID: <20180727232731.GE22554@ando.pearwood.info>

On Fri, Jul 27, 2018 at 07:02:37PM +0200, Peter O'Connor wrote:

> I find that about 90% of the time I want want to zip iterators together, I
> expect them to be the same length and want to throw an exception if they
> aren't.  Yet there is not currently a solution for this in the standard
> library for this, and as a result I always have to take this function
> everywhere I go:
[snip]

Indeed. The need is real, and the question has come up many times on 
Python-List as well.


> Would anybody object to adding this to the standard library for Python 3.8?

Given that the obvious place to put it is itertools, the person you need 
to convince is Raymond Hettinger.


-- 
Steve

From mertz at gnosis.cx  Fri Jul 27 19:54:05 2018
From: mertz at gnosis.cx (David Mertz)
Date: Fri, 27 Jul 2018 19:54:05 -0400
Subject: [Python-ideas] Can we add "zip and assert equal length" to the
 standard library?
In-Reply-To: <CACCbJKqb07XLG_bGOJzpDgi7C4of_ikbydqtWam5fa6Cz8JfnQ@mail.gmail.com>
References: <CACCbJKqb07XLG_bGOJzpDgi7C4of_ikbydqtWam5fa6Cz8JfnQ@mail.gmail.com>
Message-ID: <CAEbHw4bs9Qpv5sWxCwdWOdwtrCmd3Drf5HM+yNkWr3170zmo0Q@mail.gmail.com>

I have never wanted this behavior myself.  So that's 0% of the time for
me.  I happily believe that Peter O'Connor wants it 90% of the time...
although I guess it suggests he probably organizes his algorithms
differently than I do at some broader level.  For my needs, zip() is
common, and itertools.zip_longest() is somewhat uncommon, but not unheard
of.

I do note that https://more-itertools.readthedocs.io also does not have
this zip_equal() functionality. At least I don't see it (I could have
missed it under a different name, but I looked around). This is pretty
clearly the most widely used "extra stuff that might go in itertools"
library.

I think part of the problem is that raising an exception when something is
exhausted (other than StopIteration) is contrary to the spirit of itertools
(or more_itertools).  I think part of the reason those iterator functions
do not do that is that an exception often indicates "something went wrong,"
but if this problem occurs after an indefinitely long iteration that is
later than you'd like to know.

A more common pattern, in my experience would be to put an exception in the
higher-up block where an iterator is consumed.  Various "things went wrong"
conditions can be checked, not exclusively that one iterator ran out before
the other.

On Fri, Jul 27, 2018 at 1:03 PM Peter O'Connor <peter.ed.oconnor at gmail.com>
wrote:

> I find that about 90% of the time I want want to zip iterators together, I
> expect them to be the same length and want to throw an exception if they
> aren't.  Yet there is not currently a solution for this in the standard
> library for this, and as a result I always have to take this function
> everywhere I go:
>
>
>     def zip_equal(*iterables):
>         """
>         Zip and raise exception if lengths are not equal.
>
>         Taken from solution by Martijn Pieters, here:
>
> http://stackoverflow.com/questions/32954486/zip-iterators-asserting-for-equal-length-in-python
>
>         :param iterables: Iterable objects
>         :return: A new iterator outputting tuples where one element comes
> from each iterable
>         """
>         sentinel = object()
>         for combo in zip_longest(*iterables, fillvalue=sentinel):
>             if any(sentinel is c for c in combo):
>                 raise ValueError('Iterables have different lengths.
> Iterable(s) #{} (of 0..{}) ran out first.'.format([i for i, c in
> enumerate(combo) if c is sentinel], len(combo)-1))
>             yield combo
>
> Would anybody object to adding this to the standard library for Python 3.8?
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180727/ce2fd3d6/attachment.html>

From robertve92 at gmail.com  Fri Jul 27 21:03:53 2018
From: robertve92 at gmail.com (Robert Vanden Eynde)
Date: Sat, 28 Jul 2018 03:03:53 +0200
Subject: [Python-ideas] Can we add "zip and assert equal length" to the
 standard library?
In-Reply-To: <CAEbHw4bs9Qpv5sWxCwdWOdwtrCmd3Drf5HM+yNkWr3170zmo0Q@mail.gmail.com>
References: <CACCbJKqb07XLG_bGOJzpDgi7C4of_ikbydqtWam5fa6Cz8JfnQ@mail.gmail.com>
 <CAEbHw4bs9Qpv5sWxCwdWOdwtrCmd3Drf5HM+yNkWr3170zmo0Q@mail.gmail.com>
Message-ID: <CA+msPNmGWJUJ1725GPSsQ-2t9qWjW3Kkcx_Fs2WfvQZ8GXcxHA@mail.gmail.com>

This is a functionality I sometimes need.

Maybe you can do a pull request to more-itertools and that would be the end
of it? I don't know if that's general enough for being added to the
standard library, more-itertools seems the way to go for me. Go find out if
raising a ValueError suits their Api (at least one of their function,
"first" does it).

The David's comment makes senses, what happens when you pass an infinite
iterable, will it be consumed half? That seems logical.

from itertools import count
it = count(0)
try:
   L = zip_equal(range(5), it)
except ValueError:
   print(next(it)) # prints 6 but I'd expect 5

However, maybe that's irrelevant because most of the time you don't care
about the iterable when the exception is raised.

Le sam. 28 juil. 2018 ? 01:54, David Mertz <mertz at gnosis.cx> a ?crit :

> I have never wanted this behavior myself.  So that's 0% of the time for
> me.  I happily believe that Peter O'Connor wants it 90% of the time...
> although I guess it suggests he probably organizes his algorithms
> differently than I do at some broader level.  For my needs, zip() is
> common, and itertools.zip_longest() is somewhat uncommon, but not unheard
> of.
>
> I do note that https://more-itertools.readthedocs.io also does not have
> this zip_equal() functionality. At least I don't see it (I could have
> missed it under a different name, but I looked around). This is pretty
> clearly the most widely used "extra stuff that might go in itertools"
> library.
>
> I think part of the problem is that raising an exception when something is
> exhausted (other than StopIteration) is contrary to the spirit of itertools
> (or more_itertools).  I think part of the reason those iterator functions
> do not do that is that an exception often indicates "something went wrong,"
> but if this problem occurs after an indefinitely long iteration that is
> later than you'd like to know.
>
> A more common pattern, in my experience would be to put an exception in
> the higher-up block where an iterator is consumed.  Various "things went
> wrong" conditions can be checked, not exclusively that one iterator ran out
> before the other.
>
> On Fri, Jul 27, 2018 at 1:03 PM Peter O'Connor <peter.ed.oconnor at gmail.com>
> wrote:
>
>> I find that about 90% of the time I want want to zip iterators together,
>> I expect them to be the same length and want to throw an exception if they
>> aren't.  Yet there is not currently a solution for this in the standard
>> library for this, and as a result I always have to take this function
>> everywhere I go:
>>
>>
>>     def zip_equal(*iterables):
>>         """
>>         Zip and raise exception if lengths are not equal.
>>
>>         Taken from solution by Martijn Pieters, here:
>>
>> http://stackoverflow.com/questions/32954486/zip-iterators-asserting-for-equal-length-in-python
>>
>>         :param iterables: Iterable objects
>>         :return: A new iterator outputting tuples where one element comes
>> from each iterable
>>         """
>>         sentinel = object()
>>         for combo in zip_longest(*iterables, fillvalue=sentinel):
>>             if any(sentinel is c for c in combo):
>>                 raise ValueError('Iterables have different lengths.
>> Iterable(s) #{} (of 0..{}) ran out first.'.format([i for i, c in
>> enumerate(combo) if c is sentinel], len(combo)-1))
>>             yield combo
>>
>> Would anybody object to adding this to the standard library for Python
>> 3.8?
>>
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> --
> Keeping medicines from the bloodstreams of the sick; food
> from the bellies of the hungry; books from the hands of the
> uneducated; technology from the underdeveloped; and putting
> advocates of freedom in prisons.  Intellectual property is
> to the 21st century what the slave trade was to the 16th.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180728/19f152a3/attachment-0001.html>

From robertve92 at gmail.com  Fri Jul 27 21:13:42 2018
From: robertve92 at gmail.com (Robert Vanden Eynde)
Date: Sat, 28 Jul 2018 03:13:42 +0200
Subject: [Python-ideas] Idea: Deferred Default Arguments?
In-Reply-To: <4c19c673-930f-b3b6-58cf-eb212b77c1e6@mozilla.com>
References: <CACCbJKrUq_vUYKO8ZzF0PCU8Ss0m_XG8MG3bLKkZjH3OucsV5Q@mail.gmail.com>
 <4c19c673-930f-b3b6-58cf-eb212b77c1e6@mozilla.com>
Message-ID: <CA+msPNnDnbZ90RtSFRufKrQ6x14p9yKk1jsd++HRX77VegDH6g@mail.gmail.com>

> Someone wrote :
> Thank you for your deferred default values idea, which we're now
working on together.
>
https://github.com/petered/peters_example_code/blob/master/peters_example_code/deferral.py

Allowing to write:

from deferral import deferrable_args, deferred
@deferrable_args
def f(x, y=2, z=3):
    return (x,y,z)

f(5, deferred, 7) == (5,2,7)

(I'd rename "deferrable_args" to simply "deferrable")

The api chosen in deferall.py is a deferall.deferred, one could also use
None or Ellipsis ? That looks nice :

from deferral import elideferrable
@elideferrable
def f(x, y=2, z=3):
    return (x,y,z)

f(5, ..., 7) == (5, 2, 7)

from deferral import nonedeferrable
@nonedeferrable
def f(x, y=2, z=3):
    return (x,y,z)

f(5, None, 7) == (5, 2, 7)

Le mar. 24 juil. 2018 ? 14:26, Kyle Lahnakoski <klahnakoski at mozilla.com> a
?crit :

>
> I agree this is a problem, which I have seen solved by removing the
> method signature, which is unfortunate:
>
> > def flexible_method(**kwargs):
> >     # Read the code to find out the expected parameters
>
> I have an @override decorator to handle this type of pattern. It will
> perform the null-coalescing with properties found in a special "kwargs"
> parameter. "kwargs" is assigned a dict that has a copy of the method
> arguments. The value of a callee's argument is, in order,
>
> * a not None value provided by the caller or
> * a not None value found in the kwargs dict or
> * the default value provided by the method declaration or
> * None
>
> I was not clear on where you wanted to define your defaults.  Either
> like this:
>
> >     @override
> >     def subfunction_1(a=None, b=None, c=None, kwargs=None):
> >         return a+b*c
> >
> >     @override
> >     def subfunction_2(d=None, e=None, f=None, kwargs=None):
> >         return d*e+f
> >
> >     @orverride
> >     def main_function(a=2, b=3, c=4, d=5, e=6, f=7, kwargs=None):
> >         return subfunction_1(a, b, c) + subfunction_2(d, e, f)
> >         return subfunction_1(kwargs) + subfunction_2(kwargs)  # IF YOU
> WANT TO BE LAZY
>
> or like this:
>
> >     @override
> >     def subfunction_1(a=2, b=3, c=4, kwargs=None):
> >         return a+b*c
> >
> >     @override
> >     def subfunction_2(d=5, e=6, f=7, kwargs=None):
> >         return d*e+f
> >
> >     @orverride
> >     def main_function(a=None, b=None, c=None, d=None, e=None, f=None,
> kwargs=None):
> >         return subfunction_1(a, b, c) + subfunction_2(d, e, f)
> >         return subfunction_1(kwargs) + subfunction_2(kwargs)  # IF YOU
> WANT TO BE LAZY
>
> both are identical except for where you declare the default values.
>
>
> https://github.com/klahnakoski/mo-kwargs
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180728/e8988633/attachment.html>

From tim.peters at gmail.com  Sat Jul 28 01:27:20 2018
From: tim.peters at gmail.com (Tim Peters)
Date: Sat, 28 Jul 2018 00:27:20 -0500
Subject: [Python-ideas] Change repr of collections.OrderedDict to be
 more dict-like
In-Reply-To: <20180727052246.GA22554@ando.pearwood.info>
References: <378a7b7d-eb60-0a90-1f35-802913b6b501@redhat.com>
 <pjcvqh$t6e$1@blaine.gmane.org>
 <CAEbHw4ZB3MSvjv9uOqAOd6pOWO0FBxchyjDE5ZSveAmcToq+0A@mail.gmail.com>
 <CADqi795grCcoBr7SEOyEmi-QKfkp0a4JVdG=CfjX8kH-sOZ=Rw@mail.gmail.com>
 <CAEbHw4ZRpb8CJGv-_B8QeCJ2zU7fWK7bMTXDqrGuuTtv7-iDJw@mail.gmail.com>
 <20180727052246.GA22554@ando.pearwood.info>
Message-ID: <CAExdVNmRcsZAaZrKnpnUS5vfUDqJE0xfgA7kcmuCEWdSYpyQfQ@mail.gmail.com>

[David Mertz]
 > I often use doctests to verify APIs and behaviors when I update code. I

> > know I'm in a minority and most developers slightly disparage doctests.

[Steven D'Aprano]

> I don't know why you think that "most developers" disparage doctests.
> Perhaps they're the same ones who think that "read the source" is all
> the documentation anyone needs.
>

Or ones who aren't happy unless they can demonstrate their mastery of a
convoluted class hierarchy with hundreds of methods to memorize.  Or unless
they're Guido!  He was never fond of doctest either.


I know I don't. I love them. Tim Peters loves them (he should, he wrote
> the module) and if its good enough for Uncle Timmy then anyone who
> disparages them better have a damn good reason.
>

Nope!  They don't need any reason at all.  I don't mind :-)

It was a sneaky, although minor, purpose of doctest to raise the bar for
making gratuitous changes to Python's output.  Developers have
approximately no appreciation for how off-putting it can be for users to
struggle with docs showing output they don't actually see.  Is the
installation buggy?  Did they type something wrong?  Was the doc author
just making stuff up?  Etc.  "Something's wrong here, but I don't know
what" is not a happy experience :-(

So I always count it as a Good Thing when someone points out "but doctests
will break".  _That_ developers can understand.  It's a proxy for the real
costs of making gratuitous changes, which are pretty much invisible to
developers.

Not that it should drive a decision - but it should be counted as _a_ cost.

...

I think we'd need to ask on Python-Dev to be sure, but my understanding
> is that backwards compatibility guarantees for reprs are weaker than
> those for regular APIs but stronger than those for error messages.
>

That sounds about right to me.  The docs for doctest already point out that
relying on reprs is dubious in various cases, so if someone is relying on
the repr of OrderedDict in their doctests they presumably care about the
repr more than the semantics.  Or they were just confused ;-)

In any case, since OrderedDict seems to be viewed as a legacy feature now,
making any change to it is an odd idea on the face of it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180728/5d9e2616/attachment.html>

From jfine2358 at gmail.com  Sat Jul 28 04:58:05 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Sat, 28 Jul 2018 09:58:05 +0100
Subject: [Python-ideas] Can we add "zip and assert equal length" to the
 standard library?
In-Reply-To: <CA+msPNmGWJUJ1725GPSsQ-2t9qWjW3Kkcx_Fs2WfvQZ8GXcxHA@mail.gmail.com>
References: <CACCbJKqb07XLG_bGOJzpDgi7C4of_ikbydqtWam5fa6Cz8JfnQ@mail.gmail.com>
 <CAEbHw4bs9Qpv5sWxCwdWOdwtrCmd3Drf5HM+yNkWr3170zmo0Q@mail.gmail.com>
 <CA+msPNmGWJUJ1725GPSsQ-2t9qWjW3Kkcx_Fs2WfvQZ8GXcxHA@mail.gmail.com>
Message-ID: <CALD=Yf8y01NC5VObX5FMBut9mqt5S24Zsff_mrr9RdfrygvAkQ@mail.gmail.com>

Hi All

I've followed my own advice, earlier in this thread. The newly created
repository https://github.com/jfine2358/py-jfine2358 contains personal
Python code that others might find useful.

And you'll find in it:
===
https://github.com/jfine2358/py-jfine2358/blob/master/jfine2358/itertools.py

def zipclose(*argv, close):
    '''As zip(*argv), but call close before raising StopIteration.

   [snip]'''
===

Given zipclose, we can solve Peter's problem by writing a suitable
close function. And some other problems can be solved, by using a
different close function. Comment on my code at
https://github.com/jfine2358/py-jfine2358/issues (preferred), or on
this thread.

Finally, thank you Peter for another good problem.

-- 
Jonathan

From mhroncok at redhat.com  Sat Jul 28 07:02:39 2018
From: mhroncok at redhat.com (=?UTF-8?Q?Miro_Hron=c4=8dok?=)
Date: Sat, 28 Jul 2018 13:02:39 +0200
Subject: [Python-ideas] Change repr of collections.OrderedDict to be
 more dict-like
In-Reply-To: <CAExdVNmRcsZAaZrKnpnUS5vfUDqJE0xfgA7kcmuCEWdSYpyQfQ@mail.gmail.com>
References: <378a7b7d-eb60-0a90-1f35-802913b6b501@redhat.com>
 <pjcvqh$t6e$1@blaine.gmane.org>
 <CAEbHw4ZB3MSvjv9uOqAOd6pOWO0FBxchyjDE5ZSveAmcToq+0A@mail.gmail.com>
 <CADqi795grCcoBr7SEOyEmi-QKfkp0a4JVdG=CfjX8kH-sOZ=Rw@mail.gmail.com>
 <CAEbHw4ZRpb8CJGv-_B8QeCJ2zU7fWK7bMTXDqrGuuTtv7-iDJw@mail.gmail.com>
 <20180727052246.GA22554@ando.pearwood.info>
 <CAExdVNmRcsZAaZrKnpnUS5vfUDqJE0xfgA7kcmuCEWdSYpyQfQ@mail.gmail.com>
Message-ID: <2ce60172-e93b-0348-c089-e46e6f7e9068@redhat.com>

On 28.7.2018 07:27, Tim Peters wrote:
> [David Mertz]
>  ?> I often use doctests to verify APIs and behaviors when I update code. I
> 
>  > know I'm in a minority and most developers slightly disparage doctests.
> 
> [Steven D'Aprano]
> 
>     I don't know why you think that "most developers" disparage doctests.
>     Perhaps they're the same ones who think that "read the source" is all
>     the documentation anyone needs.
> 
> 
> Or ones who aren't happy unless they can demonstrate their mastery of a 
> convoluted class hierarchy with hundreds of methods to memorize.? Or 
> unless they're Guido!? He was never fond of doctest either.
> 
> 
>     I know I don't. I love them. Tim Peters loves them (he should, he wrote
>     the module) and if its good enough for Uncle Timmy then anyone who
>     disparages them better have a damn good reason.
> 
> 
> Nope!? They don't need any reason at all.? I don't mind :-)
> 
> It was a sneaky, although minor, purpose of doctest to raise the bar for 
> making gratuitous changes to Python's output.? Developers have 
> approximately no appreciation for how off-putting it can be for users to 
> struggle with docs showing output they don't actually see.? Is the 
> installation buggy?? Did they type something wrong?? Was the doc author 
> just making stuff up?? Etc.? "Something's wrong here, but I don't know 
> what" is not a happy experience :-(
> 
> So I always count it as a Good Thing when someone points out "but 
> doctests will break".? _That_ developers can understand.? It's a proxy 
> for the real costs of making gratuitous changes, which are pretty much 
> invisible to developers.
> 
> Not that it should drive a decision - but it should be counted as _a_ cost.
> 
>     ... 
> 
>     I think we'd need to ask on Python-Dev to be sure, but my understanding
>     is that backwards compatibility guarantees for reprs are weaker than
>     those for regular APIs but stronger than those for error messages.
> 
> 
> That sounds about right to me.? The docs for doctest already point out 
> that relying on reprs is dubious in various cases, so if someone is 
> relying on the repr of OrderedDict in their doctests they presumably 
> care about the repr more than the semantics.? Or they were just confused ;-)

Note that there were coupl repr changes in 3.7 as well:

repr for BaseException has changed to not include the trailing comma. 
Most exceptions are affected by this change. (Contributed by Serhiy 
Storchaka in bpo-30399.)

repr for datetime.timedelta has changed to include the keyword arguments 
in the output. (Contributed by Utkarsh Upadhyay in bpo-30302.)

https://docs.python.org/3.7/whatsnew/3.7.html#changes-in-the-python-api

> In any case, since OrderedDict seems to be viewed as a legacy feature 
> now, making any change to it is an odd idea on the face of it.

As far as I understood the discussions on python-dev, it is not viewed 
as legacy feature by everybody. At least when dicts with different order 
(otherwise the same) remain equal. I don't see that changing (it would 
be very backwards incompatible), hence I think OrderedDict still has 
it's place (far from legacy).

https://mail.python.org/pipermail/python-dev/2018-July/154766.html

-- 
Miro Hron?ok
--
Phone: +420777974800
IRC: mhroncok

From jamtlu at gmail.com  Fri Jul 27 23:51:55 2018
From: jamtlu at gmail.com (James Lu)
Date: Fri, 27 Jul 2018 23:51:55 -0400
Subject: [Python-ideas] As-do statements/anonymous blocks in    python
In-Reply-To: <mailman.702.1532684857.7656.python-ideas@python.org>
References: <mailman.702.1532684857.7656.python-ideas@python.org>
Message-ID: <6C2DBB7E-2FA6-4ECB-92CF-CD3DA862E271@gmail.com>

By passing a function to another function I meant passing a code block as an inline function to a function call.

The do statement is simply the arguments the function is called with

Brackets = optional
as expr [do comma-separated-expressions]:
    block

means evaluate expr, then call the result of the expression. If do is present, call it with the argument list after do. Otherwise, call it with no arguments. 

> If you don't have practical uses, this proposal has ZERO chance of being
> accepted.

I have already shown a Flask use case and another poster has mentioned they would find it useful in their own code. (Though they had reservations on whether it should be included in Python.)

I simply said I wasn?t totally aware of all practical use cases, especially practical abuse cases. 

From jfine2358 at gmail.com  Sat Jul 28 10:54:31 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Sat, 28 Jul 2018 15:54:31 +0100
Subject: [Python-ideas] As-do statements/anonymous blocks in python
In-Reply-To: <6C2DBB7E-2FA6-4ECB-92CF-CD3DA862E271@gmail.com>
References: <mailman.702.1532684857.7656.python-ideas@python.org>
 <6C2DBB7E-2FA6-4ECB-92CF-CD3DA862E271@gmail.com>
Message-ID: <CALD=Yf-cJp5XS8k1fu5fhfsYEpM3DejRfxq_Z6Av0qgEYjM-zQ@mail.gmail.com>

Hi James

This is an attempt to respond to the issues that be lying behind your
request for code blocks in Python. It's my two cents worth
(https://en.wikipedia.org/wiki/My_two_cents) of opinion.

I really do wish we could have language that had all of Ruby's
strengths, and also all of Python's. That would be really nice. Quite
something indeed.

But most strengths, in another situation, can be a weakness. Language
design is often a compromise between conciseness and readability, ease
of use and performance, good for beginners and good for experts, and
many other factors. Such as innovation and stability.

Guido's decisions, as BDFL, have shaped Python and its community into
what it is now. It is one set of compromises. Other languages, such as
Ruby, have made different compromises. Now that the BDFL is on
vacation, the challenge is to maintain the essence of Python while
continuing to innovate.

Python's compromises and implementation give the language a large sweet spot.
===
https://en.wikipedia.org/wiki/Sweet_spot_(sports)
The sweet spot is a place where a combination of factors results in a
maximum response for a given amount of effort.
===

Languages do influence each other. Ruby is good at internal Domain
Specific Languages (DSLs). And there's Perrotta's influential book on
Ruby Metaprogramming. That's something I think Python could learn
from.

But I don't see any need (or even benefit) in adding new language
features to Python, so it can do better at DSLs. It's fairly easy to
write a decorator that turns a class into a dictionary of 'code
blocks'. I think that might be what you really want.

I'd be pleased to hear what you have to say about this.

with best regards

Jonathan

From jfine2358 at gmail.com  Sat Jul 28 12:44:29 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Sat, 28 Jul 2018 17:44:29 +0100
Subject: [Python-ideas] slice[] to get more complex slices
In-Reply-To: <CAEQ_Tvc3RzZrew324XZ-sauegHGssWC2sBk520X0T_QQyaeG0A@mail.gmail.com>
References: <CAFpSVpKnE8b=QrjxLy6anAF3ammq2QbB4_K1Ey02Vk3n8EVwyQ@mail.gmail.com>
 <pj2lgk$c6c$1@blaine.gmane.org>
 <09cb9784-6a97-49b9-b5e7-1c9a9d8c8391@googlegroups.com>
 <CALD=Yf-MNLW7z8s=ruc_C13t7eWp8KC6OWvqf9durZgpN=8=sw@mail.gmail.com>
 <399ff7d70c204f679bdb5a79f9bc0460@xmail103.UGent.be>
 <4243953379d74d2aa5393be2434a99a7@xmail103.UGent.be>
 <5B55AE6A.7010305@UGent.be>
 <CACac1F_XOi+1gmEMgZdbmW_3UXs661JS5BF=OvWLrF47H0fJmg@mail.gmail.com>
 <CAEQ_Tvf2Y-up_uWoUdutyoreHeM6=X52bZzZcath-r1btYeqow@mail.gmail.com>
 <pj5nl6$p6b$1@blaine.gmane.org>
 <CAEbHw4YuS99=cVbhV6Riuwij4H_xNdcxp1Df6tMQfD-__CaO_w@mail.gmail.com>
 <CAEQ_Tvc3RzZrew324XZ-sauegHGssWC2sBk520X0T_QQyaeG0A@mail.gmail.com>
Message-ID: <CALD=Yf-E0mPkKEzbZkP-agRVTqfGJPppDN5zR1+taP2chSCdYA@mail.gmail.com>

Hi Stephan

I took a look at your personal web site. Your PhD sounds interesting.
Quantum coherence and photosynthesis. But off-topic here. (Sadly,
personal web sites are a bit of mess.)

You wrote:
> I'd really like to move this proposal forward in some form.

+10

> As a next step, would it be helpful to summarize the use cases and reasoning
> in a PEP?

0

Here's my two cents worth of opinion.

By all means start a PEP (on github) if you find it helps you. I think
it's too early.

If this problem has a pure Python solution, then I think it best to
develop it as a third party module. I suggest the name slicetools.
When slicetools is established, has users and is stable. That would be
the time to submit the PEP. I think it would give a smoother and
easier path.

For example, PEP 428 proposes the inclusion of a third-party module,
pathlib, in the standard library.
https://mail.python.org/pipermail/python-ideas/2018-July/052255.html
https://www.python.org/dev/peps/pep-0428/

For more on my general approach to building a third-party library, see my posts.
https://mail.python.org/pipermail/python-ideas/2018-July/052464.html
https://mail.python.org/pipermail/python-ideas/2018-July/052470.html

That's the end of my two cents.

Oh, and those who have time and energy to contribute. They get the biggest vote.

best regards

From eric at trueblade.com  Sat Jul 28 13:46:15 2018
From: eric at trueblade.com (Eric V. Smith)
Date: Sat, 28 Jul 2018 13:46:15 -0400
Subject: [Python-ideas] Change repr of collections.OrderedDict to be
 more dict-like
In-Reply-To: <2ce60172-e93b-0348-c089-e46e6f7e9068@redhat.com>
References: <378a7b7d-eb60-0a90-1f35-802913b6b501@redhat.com>
 <pjcvqh$t6e$1@blaine.gmane.org>
 <CAEbHw4ZB3MSvjv9uOqAOd6pOWO0FBxchyjDE5ZSveAmcToq+0A@mail.gmail.com>
 <CADqi795grCcoBr7SEOyEmi-QKfkp0a4JVdG=CfjX8kH-sOZ=Rw@mail.gmail.com>
 <CAEbHw4ZRpb8CJGv-_B8QeCJ2zU7fWK7bMTXDqrGuuTtv7-iDJw@mail.gmail.com>
 <20180727052246.GA22554@ando.pearwood.info>
 <CAExdVNmRcsZAaZrKnpnUS5vfUDqJE0xfgA7kcmuCEWdSYpyQfQ@mail.gmail.com>
 <2ce60172-e93b-0348-c089-e46e6f7e9068@redhat.com>
Message-ID: <addfaaac-4254-8009-b967-b71c34b86b57@trueblade.com>

On 7/28/2018 7:02 AM, Miro Hron?ok wrote:
> On 28.7.2018 07:27, Tim Peters wrote:
>> In any case, since OrderedDict seems to be viewed as a legacy feature 
>> now, making any change to it is an odd idea on the face of it.
> 
> As far as I understood the discussions on python-dev, it is not viewed 
> as legacy feature by everybody. At least when dicts with different order 
> (otherwise the same) remain equal. I don't see that changing (it would 
> be very backwards incompatible), hence I think OrderedDict still has 
> it's place (far from legacy).

I think it's legacy in the sense that for new code, very few people are 
now going to reach for an OrderedDict.

Eric


From abedillon at gmail.com  Sat Jul 28 16:07:40 2018
From: abedillon at gmail.com (Abe Dillon)
Date: Sat, 28 Jul 2018 15:07:40 -0500
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAEbHw4Yzt1Wfr1iBw+mmTYWhh8b2SwyFLX1AHOL-4YP6H+_ChA@mail.gmail.com>
 <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>
 <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
 <CAKr=oZtfYM2_fxYnde76o+jvuc_Y3AH9R93BCY6BoNdriuKqKw@mail.gmail.com>
 <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
Message-ID: <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>

>
> We could argue about how intuitive or not these operators are, but they
> are used in other languages, so they clearly aren't completely unintuitive.


Other languages are other languages. Other languages use the "<condition> ?
<expr_a> : <expr_b>" form of the the ternary operator. That doesn't mean we
should adopt that into Python.

Being visually obtrusive would help a great deal.  Unfortunately I don't
> think it is visually obtrusive in the ?. and ?[] operators.


My argument about visual obtrusiveness is in the context of me claiming
that python attempts to mimic natural language to achieve readability,
others countering that `person.name` is not how periods are used in natural
languages, so using other symbols in unintuitive ways is perfectly fine. My
point is that '.' is unobtrusive, so it's not *that* different from `person
name' which isn't *that* different from `person's name`. The '?' symbol is
most definitely more visually obtrusive than a simple '.' because it takes
up more space.

We could conceivably have a dunder method that None implemented as
> returning the other argument and object implemented as returning itself.
> I'm not really convinced that it's worth it, though.


Yes, I thought of that and came to the same conclusion. It's my
understanding that None may not be an actual object, but a special memory
location. I'm not sure though and didn't look it up.

Really?  Quick straw poll: how often do people care how in-place operators
> are implemented?


The point I was trying to make is that it doesn't behave or compose like
other expressions.

As you said, "?" is not an operator, so "a?.b" clearly can't break down
> into "a? .b".


The problem is that '.' IS a stand-alone operator, so it's natural to
visually parse `<expr>.b` as `<expr>  .b`, but adding '?.' causes double
takes, more mental load, general interruption of the flow of reading. It
also sets up the later discussion of other possible uses of the '?' symbol
that may or may not have more merit.

Um.  How is that supposed to parse?  "person[0]" has already been evaluated
> by the time the "?" operator gets anywhere near it.


It parses by placing the '?' operator high in the order of operations; like
I said. Do I really need to explain how order of operations works?

Why not indeed.  I haven't seen any convincing use cases (and nor has the
> PEP, though I doubt Steve was looking for them), but they aren't inherently
> any more weird than "?." and "?[]".


It seems like a natural extension. It also seems like a lot to add for
something as trivial as None handling.

While my immediate reaction is "yuk"


Is that more legit than my reaction to '?.' and '?[]'? At least in this
case, the '?' comes at the end of a clause...

your use of "?" does not conflict with any use in the PEP, so this is a red
> herring.


I suppose it doesn't *technically* conflict, but it would lead to some
beautiful code, like:

lambda data=a?.b?[0]?: data?.d

On Fri, Jul 27, 2018 at 7:00 AM, Rhodri James <rhodri at kynesim.co.uk> wrote:

> On 26/07/18 22:28, Abe Dillon wrote:
>
>> The readability issue isn't just a matter of a new set of symbols to
>> learn.
>> It isn't even that the symbols are used in a non-intuitive way (though
>> that
>> doesn't help).
>>
>
> We could argue about how intuitive or not these operators are, but they
> are used in other languages, so they clearly aren't completely unintuitive.
>
> It's not even that the question mark, unlike the dot, is
>> very visually obtrusive (which also doesn't help).
>>
>
> Being visually obtrusive would help a great deal.  Unfortunately I don't
> think it is visually obtrusive in the ?. and ?[] operators.
>
> It also screws with the
>> grammar in an unexpected way. I would be fine with something like:
>>
>> def func(data=None):
>>      data ?= []
>>      ...
>>
>> If there were some corresponding dunder method to implement that behavior,
>> but it's not clear to me that such a thing is possible.
>>
>
> We could conceivably have a dunder method that None implemented as
> returning the other argument and object implemented as returning itself.
> I'm not really convinced that it's worth it, though.
>
> If it isn't, that
>> would make `?=` (or '??=' though I don't see the need for double '?'s)
>>
>
> Because it's the in-place version of "??", obviously.
>
> a
>> special statement, a special case that someone who's familiar with how
>> other in-place operators work, would be thrown for a loop.
>>
>
> Really?  Quick straw poll: how often do people care how in-place operators
> are implemented?
>
> The other use
>> cases look like they break down in a different manner than they actually
>> do:
>>
>> person?.name
>>
>> looks like it would break down similar to
>>
>> person().name
>>
>> but it doesn't because '?' by itself is not an operator.
>> similarly:
>>
>> person?[0]
>>
>> looks like it would break down similar to
>>
>> person()[0]
>>
>> but again, the operator is '?[...]' not '?'
>>
>
> Sorry, not buying this one.  As you said, "?" is not an operator, so
> "a?.b" clearly can't break down into "a? .b".  This all rather feels like
> you are trying to find justifications for not liking these operators rather
> just saying "I don't like them," and frankly I'd respect you more if you
> just said "I don't like them."
>
> (For the record, I don't like most of them.  There are clear use cases for
> "??" and "??=", but the use cases I've seen for "?." and "?[]" aren't
> convincing to me.)
>
> This also raises the question of other operators that could be made null
>> aware. Why not:
>>
>> func?()
>>
>> ?-num
>>
>> ?~num
>>
>> num ?+ other
>>
>> Why not other reflective operators?
>>
>> num ?+= other
>>
>
> Why not indeed.  I haven't seen any convincing use cases (and nor has the
> PEP, though I doubt Steve was looking for them), but they aren't inherently
> any more weird than "?." and "?[]".
>
> I think this could be helped by simply making '?' an operator with a very
>> high order of operations, so that pretty-much any expression on the LHS is
>> guarded for 'NoneType' AttributeErrors:
>>
>> val = person[0]?
>> val = person.name?
>>
>
> Um.  How is that supposed to parse?  "person[0]" has already been
> evaluated by the time the "?" operator gets anywhere near it.  Unless you
> are proposing some major surgery to the parser (which I'm pretty sure isn't
> going to happen), this has no chance of getting through. Good thing too,
> because it's a lot more surprising than the alternatives.
>
> I also disagree with the idea that None is special enough to warrant such
>> special behavior. There are other special None-like objects (as David
>> Mertz
>> pointed out) like NaN and custom place-holders for cases where None is a
>> valid argument.
>>
>
> NaN isn't used like None, though, and custom sentinels aren't that common.
>
> Yet another possibility is to make '?' signify a parameter who's default
>> value is an expression that should be evaluated when necessary:
>>
>> def func(data: List = []?):
>>      ...
>>
>> That gets closer to another often requested feature: delayed expression
>> evaluation (as Steve D'Aprano pointed out).
>> So there are other, possibly better uses for the '?' symbol. It's
>> something
>> that needs to be carefully considered.
>>
>
> While my immediate reaction is "yuk", your use of "?" does not conflict
> with any use in the PEP, so this is a red herring.
>
> --
> Rhodri James *-* Kynesim Ltd
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180728/793060b0/attachment-0001.html>

From rosuav at gmail.com  Sat Jul 28 16:14:47 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 29 Jul 2018 06:14:47 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAEbHw4Yzt1Wfr1iBw+mmTYWhh8b2SwyFLX1AHOL-4YP6H+_ChA@mail.gmail.com>
 <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>
 <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
 <CAKr=oZtfYM2_fxYnde76o+jvuc_Y3AH9R93BCY6BoNdriuKqKw@mail.gmail.com>
 <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
Message-ID: <CAPTjJmp2tN1TBz96EfnLRDsdO==QJ0xtYbCVkTBJDQ9+qOp6nA@mail.gmail.com>

On Sun, Jul 29, 2018 at 6:07 AM, Abe Dillon <abedillon at gmail.com> wrote:
> Yes, I thought of that and came to the same conclusion. It's my
> understanding that None may not be an actual object, but a special memory
> location. I'm not sure though and didn't look it up.

Python does not have memory locations. None *is* an actual object. It
has attributes, it has standard behaviours, it fits in the same object
model as everything else in Python does.

>> As you said, "?" is not an operator, so "a?.b" clearly can't break down
>> into "a? .b".
>
>
> The problem is that '.' IS a stand-alone operator, so it's natural to
> visually parse `<expr>.b` as `<expr>  .b`, but adding '?.' causes double
> takes, more mental load, general interruption of the flow of reading. It
> also sets up the later discussion of other possible uses of the '?' symbol
> that may or may not have more merit.

This is utter nonsense on par with trying to claim that "x <= y"
should be parsed as if it's a modified form of assignment since "x =
y" would be assignment.

Do I really need to explain how two-character operators work?

ChrisA

From greg.ewing at canterbury.ac.nz  Sat Jul 28 20:14:55 2018
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 29 Jul 2018 12:14:55 +1200
Subject: [Python-ideas] As-do statements/anonymous blocks in    python
In-Reply-To: <6C2DBB7E-2FA6-4ECB-92CF-CD3DA862E271@gmail.com>
References: <mailman.702.1532684857.7656.python-ideas@python.org>
 <6C2DBB7E-2FA6-4ECB-92CF-CD3DA862E271@gmail.com>
Message-ID: <5B5D06FF.8050607@canterbury.ac.nz>

James Lu wrote:
> as expr [do comma-separated-expressions]:
 >    block
> 
> means evaluate expr, then call the result of the expression. If do is
> present, call it with the argument list after do.

This kind of thing has been proposed before. Although it seems
like a straightforward idea, there are some thorny edge cases
concerning how things like 'return' and 'break' in the code
block should be handled. For example, what should this do:

    def f():
       as g:
          return

Should that return from g or from f?

Or this:

    while some_codition:
       as g:
          if some_other_condition:
             break

If you want to flesh out your proposal you will have to address
these. Also you should seek out previous discussions about code
blocks to make sure you're not just rehashing old ground.

-- 
Greg




  Otherwise, call it with no
> arguments.
> 
> 
>> If you don't have practical uses, this proposal has ZERO chance of being 
>> accepted.
> 
> 
> I have already shown a Flask use case and another poster has mentioned they
> would find it useful in their own code. (Though they had reservations on
> whether it should be included in Python.)
> 
> I simply said I wasn?t totally aware of all practical use cases, especially
> practical abuse cases. _______________________________________________ 
> Python-ideas mailing list Python-ideas at python.org 
> https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct:
> http://python.org/psf/codeofconduct/


From abedillon at gmail.com  Sat Jul 28 23:56:13 2018
From: abedillon at gmail.com (Abe Dillon)
Date: Sat, 28 Jul 2018 22:56:13 -0500
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPTjJmp2tN1TBz96EfnLRDsdO==QJ0xtYbCVkTBJDQ9+qOp6nA@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAEbHw4Yzt1Wfr1iBw+mmTYWhh8b2SwyFLX1AHOL-4YP6H+_ChA@mail.gmail.com>
 <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>
 <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
 <CAKr=oZtfYM2_fxYnde76o+jvuc_Y3AH9R93BCY6BoNdriuKqKw@mail.gmail.com>
 <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
 <CAPTjJmp2tN1TBz96EfnLRDsdO==QJ0xtYbCVkTBJDQ9+qOp6nA@mail.gmail.com>
Message-ID: <CAKr=oZsTsR8Mv7_i386eOrEzQWTjiwbONt+7C=6D9V94f56kaQ@mail.gmail.com>

>
> Python does not have memory locations.
>

CPython does, form the documentation on the `id` function:

*CPython implementation detail:* This is the address of the object in
> memory.


I'm not sure what performance implications there would be for adding a
__none_check__ or whatever method to None.

None *is* an actual object. It
> has attributes, it has standard behaviours, it fits in the same object
> model as everything else in Python does.


 Cool. Thanks for the info. I wasn't sure because it's considered un-kosher
(according to PyCharm) to use "if x == None:" instead of "if x is None".

This is utter nonsense on par with trying to claim that "x <= y"
> should be parsed as if it's a modified form of assignment since "x =
> y" would be assignment.


This is getting tedious. It's only comparable to '<=' if you disregard
every other point I've made.
Python is supposed to emphasize readability and learnability. To that end,
expressions should strive to resemble natural language.
Otherwise they should try to resemble common mathematical notation. Failing
that, they should try to resemble common programming idioms.
There's no easy way to do subscripts, so we use square brackets for
indexing. That's a reasonable and pragmatic compromise.
`<=` is a fine compromise too because there's no easy way to type the ?
symbol on most keyboards and it's a perfectly familiar symbol from
grade-school math.
Sure, someone could confuse '<=' for an assignment operator, I suppose; but
there's enough going for '<=' that it's easy to justify its existence for
pragmatism's sake.
The same can not be said for '?.'. It's not even remotely close to the same
thing.

"?." doesn't resemble anything in natural language. It doesn't resemble
anything in common mathematics. It isn't a well established pattern in
programming that has decades of precedent (like square bracket indexing).
It's not hard to check for None with a ternary statement. Adding a '?'
operator that goes at the end of an expression could do the same thing more
elegantly and resemble natural language at the same time. For all of those
reasons: I'm against this proposal.

Do I really need to explain how two-character operators work?


Point taken, I'll watch my tone. Rhodri James, I'm sorry for that flippant
remark about order of operations. It was uncalled for.

On Sat, Jul 28, 2018 at 3:14 PM, Chris Angelico <rosuav at gmail.com> wrote:

> On Sun, Jul 29, 2018 at 6:07 AM, Abe Dillon <abedillon at gmail.com> wrote:
> > Yes, I thought of that and came to the same conclusion. It's my
> > understanding that None may not be an actual object, but a special memory
> > location. I'm not sure though and didn't look it up.
>
> Python does not have memory locations. None *is* an actual object. It
> has attributes, it has standard behaviours, it fits in the same object
> model as everything else in Python does.
>
> >> As you said, "?" is not an operator, so "a?.b" clearly can't break down
> >> into "a? .b".
> >
> >
> > The problem is that '.' IS a stand-alone operator, so it's natural to
> > visually parse `<expr>.b` as `<expr>  .b`, but adding '?.' causes double
> > takes, more mental load, general interruption of the flow of reading. It
> > also sets up the later discussion of other possible uses of the '?'
> symbol
> > that may or may not have more merit.
>
> This is utter nonsense on par with trying to claim that "x <= y"
> should be parsed as if it's a modified form of assignment since "x =
> y" would be assignment.
>
> Do I really need to explain how two-character operators work?
>
> ChrisA
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180728/660d0844/attachment-0001.html>

From rosuav at gmail.com  Sun Jul 29 00:27:27 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 29 Jul 2018 14:27:27 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAKr=oZsTsR8Mv7_i386eOrEzQWTjiwbONt+7C=6D9V94f56kaQ@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <20180723013614.GV8744@ando.pearwood.info>
 <CAFYqXL9NxzUXn9e31NcxKhhjuqDwEeWsFKZUODX=a4S6rFdCEQ@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAEbHw4Yzt1Wfr1iBw+mmTYWhh8b2SwyFLX1AHOL-4YP6H+_ChA@mail.gmail.com>
 <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>
 <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
 <CAKr=oZtfYM2_fxYnde76o+jvuc_Y3AH9R93BCY6BoNdriuKqKw@mail.gmail.com>
 <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
 <CAPTjJmp2tN1TBz96EfnLRDsdO==QJ0xtYbCVkTBJDQ9+qOp6nA@mail.gmail.com>
 <CAKr=oZsTsR8Mv7_i386eOrEzQWTjiwbONt+7C=6D9V94f56kaQ@mail.gmail.com>
Message-ID: <CAPTjJmrf-ERm3ikbw8KfwmJTCXgp=MVEefnpEv233WpS=diezw@mail.gmail.com>

On Sun, Jul 29, 2018 at 1:56 PM, Abe Dillon <abedillon at gmail.com> wrote:
>> Python does not have memory locations.
>
>
> CPython does, form the documentation on the `id` function:
>
>> CPython implementation detail: This is the address of the object in
>> memory.

Right, which is an important distinction; CPython, being a concrete
implementation, does involve memory. For instance, you can ask "how
many bytes of memory does this object take up?" and CPython can answer
that (sys.getsizeof). You can't ask that in Python generally, because
the abstract language doesn't have memory or any such concept.

The "None" object exists in the abstract sense. It will be present in
ANY Python implementation. In theory, a Python could use a null
pointer to represent None, just as long as you can't tell the
difference between that behaviour and any other.

> I'm not sure what performance implications there would be for adding a
> __none_check__ or whatever method to None.

Considering that None is a singleton, there's no need to create a
protocol for asking "are you None?". The obvious way to say "are you
None?" is to inquire if some object and None are the same object,
which is spelled "x is None".

ChrisA

From greg.ewing at canterbury.ac.nz  Sat Jul 28 20:49:13 2018
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 29 Jul 2018 12:49:13 +1200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAEbHw4Yzt1Wfr1iBw+mmTYWhh8b2SwyFLX1AHOL-4YP6H+_ChA@mail.gmail.com>
 <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>
 <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
 <CAKr=oZtfYM2_fxYnde76o+jvuc_Y3AH9R93BCY6BoNdriuKqKw@mail.gmail.com>
 <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
Message-ID: <5B5D0F09.5080209@canterbury.ac.nz>

Abe Dillon wrote:
> others countering that `person.name <http://person.name>` is not how 
> periods are used in natural languages, so using other symbols in 
> unintuitive ways is perfectly fine.

Dots have been used for attribute access in so many languages
for so long that it has become the normal and expected syntax
to use. ?. is much more recent. Maybe in another 30 years, if
it has stood the test of time, it could be argued for on the
basis of familiarity, but not now.

-- 
Greg

From steve at pearwood.info  Sun Jul 29 01:54:42 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 29 Jul 2018 15:54:42 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAKr=oZsTsR8Mv7_i386eOrEzQWTjiwbONt+7C=6D9V94f56kaQ@mail.gmail.com>
References: <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAEbHw4Yzt1Wfr1iBw+mmTYWhh8b2SwyFLX1AHOL-4YP6H+_ChA@mail.gmail.com>
 <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>
 <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
 <CAKr=oZtfYM2_fxYnde76o+jvuc_Y3AH9R93BCY6BoNdriuKqKw@mail.gmail.com>
 <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
 <CAPTjJmp2tN1TBz96EfnLRDsdO==QJ0xtYbCVkTBJDQ9+qOp6nA@mail.gmail.com>
 <CAKr=oZsTsR8Mv7_i386eOrEzQWTjiwbONt+7C=6D9V94f56kaQ@mail.gmail.com>
Message-ID: <20180729055442.GH22554@ando.pearwood.info>

On Sat, Jul 28, 2018 at 10:56:13PM -0500, Abe Dillon wrote:
> >
> > Python does not have memory locations.
> >
> 
> CPython does, form the documentation on the `id` function:

No, that is the same id() function as the id() provided by Jython, and 
IronPython, and Stackless. Like all Python implementations, it returns 
an opaque integer ID number.

That's why it is called "id" rather than "memory_location" or "address".


> *CPython implementation detail:* This is the address of the object in
> > memory.

An irrelevant implementation detail. You can't use the id() as a memory 
address, and even if you could, that wouldn't be portable Python code, 
it would be an implementation-dependent hack that is unsafe or 
unreliable to use.

Being an implementation detail, CPython is free to change it at any 
time, without warning or notice, even in a bug-fix release. If CPython 
ever gets a memory manager that can move objects around, as they can 
move in Jython and IronPython, CPython will also have to change id().

id() may, sometimes, *use* the memory address, but the semantics are 
not that of a memory address.


> I'm not sure what performance implications there would be for adding a
> __none_check__ or whatever method to None.

The point of testing for None is because you want None, and absolutely 
no other object but None. A __none_check__ method would allow other 
objects to lie about being None-like. The point of using "is" is to 
avoid that.

If you want any arbitrary object which happens to be a bit None-like, 
then call "x == None" and let x decide for itself. But be prepared to 
have experienced Python programmers correct your "mistake" unless you 
carefully document why you are doing this.


> Python is supposed to emphasize readability and learnability.

You're thinking of Scratch and other languages aimed at school kids.

Python emphasizes many things. Its not merely a kiddies' language, or a 
beginners' language, or an educational language. It is a language used 
by professionals, and frankly I'm getting sick to the back teeth of 
people saying "Mustn't do that useful thing because it will make Python 
harder for beginners to learn".

Fine. So it takes them an extra day to learn one more operator. Big 
deal. It is commonly believed to take ten years to master a field or 
language. Amortize that one day over ten years and its virtually 
nothing.


> To that end, expressions should strive to resemble natural language.

If you want a programming language that looks like natural language, you 
want something like Hypertalk:

  put the second word of field "Address" after line 3 of field "Record"

or Inform7. Or the granddaddy of "natural language" programming, COBOL.

Python is very little like natural language. Your earlier idea that 
attribute access spam.eggs is like natural language (in English) because 
its only a few characters different from "spam's eggs" really doesn't 
hold up. Apart from the word order, there's no similarity.


> Otherwise they should try to resemble common mathematical notation. Failing
> that, they should try to resemble common programming idioms.

Yes, common programming idioms like null-coalescing and null-aware 
operators, which are used by a good half dozen popular, modern 
languages.


> "?." doesn't resemble anything in natural language.

That's okay. Neither does spam.eggs, or spam[eggs], or anything to do 
with async, or function call notation func(a, b, c, keyword=d).


> It's not hard to check for None with a ternary statement. 

You're thinking in terms of trivial cases like 

    default if x is None else x

but replace x with a long, potentially expensive expression and 
you may think differently:

    default if spam(a, b, c, kw=d).eggs[cheese]() is None else spam(a, b c, kw=d).eggs[cheese]()

or a chain of tests:

    if spam(a, b, c, kw=d) is not None:
        if spam(a, b, c, kw=d).eggs is not None:
            if spam(a, b, c, kw=d).eggs[cheese] is not None:
                result = spam(a, b, kw=d).eggs[cheese]()


> Adding a '?'
> operator that goes at the end of an expression could do the same thing more
> elegantly 

I don't know that Python has any binary operators which are split 
into an infix part and a postfix part. I don't know any language with an 
operator like that. There is the ternary operator, of course, but this 
would be a binary operator.

So you're against:

    spam?.eggs?.cheese?.aardvark

but you prefer:

    spam.eggs?.cheese?.aardvark?

instead.



-- 
Steve

From steve at pearwood.info  Sun Jul 29 02:02:10 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 29 Jul 2018 16:02:10 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <5B5D0F09.5080209@canterbury.ac.nz>
References: <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAEbHw4Yzt1Wfr1iBw+mmTYWhh8b2SwyFLX1AHOL-4YP6H+_ChA@mail.gmail.com>
 <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>
 <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
 <CAKr=oZtfYM2_fxYnde76o+jvuc_Y3AH9R93BCY6BoNdriuKqKw@mail.gmail.com>
 <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
 <5B5D0F09.5080209@canterbury.ac.nz>
Message-ID: <20180729060210.GI22554@ando.pearwood.info>

On Sun, Jul 29, 2018 at 12:49:13PM +1200, Greg Ewing wrote:
> Abe Dillon wrote:
> >others countering that `person.name <http://person.name>` is not how 
> >periods are used in natural languages, so using other symbols in 
> >unintuitive ways is perfectly fine.
> 
> Dots have been used for attribute access in so many languages
> for so long that it has become the normal and expected syntax
> to use. ?. is much more recent. Maybe in another 30 years, if
> it has stood the test of time, it could be argued for on the
> basis of familiarity, but not now.

You're talking like the syntax is used only by a handful of experimental 
languages with a total user-base measured in the dozens.

?. is used by some of the most commonly used languages in the world, 
such as C++, Objective C and PHP, as well as up-and-coming "cool" 
languages getting lots of industry buzz, like Swift and Dart.

Its certainly more familiar now than Python's slicing syntax was when 
Python first started.



-- 
Steve

From rosuav at gmail.com  Sun Jul 29 02:03:51 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 29 Jul 2018 16:03:51 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180729055442.GH22554@ando.pearwood.info>
References: <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAEbHw4Yzt1Wfr1iBw+mmTYWhh8b2SwyFLX1AHOL-4YP6H+_ChA@mail.gmail.com>
 <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>
 <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
 <CAKr=oZtfYM2_fxYnde76o+jvuc_Y3AH9R93BCY6BoNdriuKqKw@mail.gmail.com>
 <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
 <CAPTjJmp2tN1TBz96EfnLRDsdO==QJ0xtYbCVkTBJDQ9+qOp6nA@mail.gmail.com>
 <CAKr=oZsTsR8Mv7_i386eOrEzQWTjiwbONt+7C=6D9V94f56kaQ@mail.gmail.com>
 <20180729055442.GH22554@ando.pearwood.info>
Message-ID: <CAPTjJmpV9i+WyJxhCf6bbh04uOrexspB+iv-ZYrSi5OOa1p38w@mail.gmail.com>

On Sun, Jul 29, 2018 at 3:54 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> Being an implementation detail, CPython is free to change it at any
> time, without warning or notice, even in a bug-fix release. If CPython
> ever gets a memory manager that can move objects around, as they can
> move in Jython and IronPython, CPython will also have to change id().
>

To clarify: If that were to happen, CPython would change *the
description of the id() function*, not the IDs returned. The IDs
returned must be constant for the lifetime of the object, as that's
what Python-the-language demands and guarantees.

ChrisA

From abedillon at gmail.com  Sun Jul 29 03:12:22 2018
From: abedillon at gmail.com (Abe Dillon)
Date: Sun, 29 Jul 2018 02:12:22 -0500
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPTjJmpV9i+WyJxhCf6bbh04uOrexspB+iv-ZYrSi5OOa1p38w@mail.gmail.com>
References: <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAEbHw4Yzt1Wfr1iBw+mmTYWhh8b2SwyFLX1AHOL-4YP6H+_ChA@mail.gmail.com>
 <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>
 <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
 <CAKr=oZtfYM2_fxYnde76o+jvuc_Y3AH9R93BCY6BoNdriuKqKw@mail.gmail.com>
 <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
 <CAPTjJmp2tN1TBz96EfnLRDsdO==QJ0xtYbCVkTBJDQ9+qOp6nA@mail.gmail.com>
 <CAKr=oZsTsR8Mv7_i386eOrEzQWTjiwbONt+7C=6D9V94f56kaQ@mail.gmail.com>
 <20180729055442.GH22554@ando.pearwood.info>
 <CAPTjJmpV9i+WyJxhCf6bbh04uOrexspB+iv-ZYrSi5OOa1p38w@mail.gmail.com>
Message-ID: <CAKr=oZu+Ke6Qc9j4mc3XNLySdjA0V5WXbDivHQZkWRMir6UxNw@mail.gmail.com>

First of all: Thanks to everyone for explaining everything about the id
function.
You are all so smart...

[Greg Ewing]

> Dots have been used for attribute access in so many languages
> for so long that it has become the normal and expected syntax
> to use. ?. is much more recent. Maybe in another 30 years, if
> it has stood the test of time, it could be argued for on the
> basis of familiarity, but not now.


My point exactly.

[Steve D'Aprano]

> You're thinking of Scratch and other languages aimed at school kids.


Nope. I'm thinking of Python.

[Steve D'Aprano]

> frankly I'm getting sick to the back teeth of
> people saying "Mustn't do that useful thing because it will make Python
> harder for beginners to learn".


> Fine. So it takes them an extra day to learn one more operator. Big
> deal.


It's not just about learning it. It's about the mental load of reading
code. little things add up.

If "sick through your back teeth" of people being cautious, that sounds
like a personal problem. I'm worried about the pace people are adding
functionality to Python. You start learning about concurrent.futures, then
asyncio gets added with "yield from", then async def and await and all that
gets added and it all seems to be a mess. Now there are how many ways to
format strings?

Should we just shovel features from other popular languages into Python
because they might be a little useful? I wouldn't wan't your back teeth to
rot from my prudishness...

Python is very little like natural language. Your earlier idea that
> attribute access spam.eggs is like natural language (in English) because
> its only a few characters different from "spam's eggs" really doesn't
> hold up. Apart from the word order, there's no similarity.


I feel like you're being willfully dense about this. There is a difference
between:

if any(thing.is_metalic for thing in pockets): raise Alarm("metal
> detected!")


and

if any(thing$!<>~.s_metal for thing in pockets): raise Alarm("metal
> detected!")


The syntax of this proposal is almost objectively disgusting. It's truly
horrid. I don't know how many ways to say it.

[Steve D'Aprano]

> I don't know that Python has any binary operators which are split
> into an infix part and a postfix part.


That's not at all what I proposed.

[Steve D'Aprano]

> So you're against:
>     spam?.eggs?.cheese?.aardvark
> but you prefer:
>     spam.eggs?.cheese?.aardvark?
> instead.


NO! I'm proposing:

spam.eggs.cheese.aardvark?

A single POSTFIX operator that has a high priority in the order of
operations.


On Sun, Jul 29, 2018 at 1:03 AM, Chris Angelico <rosuav at gmail.com> wrote:

> On Sun, Jul 29, 2018 at 3:54 PM, Steven D'Aprano <steve at pearwood.info>
> wrote:
> > Being an implementation detail, CPython is free to change it at any
> > time, without warning or notice, even in a bug-fix release. If CPython
> > ever gets a memory manager that can move objects around, as they can
> > move in Jython and IronPython, CPython will also have to change id().
> >
>
> To clarify: If that were to happen, CPython would change *the
> description of the id() function*, not the IDs returned. The IDs
> returned must be constant for the lifetime of the object, as that's
> what Python-the-language demands and guarantees.
>
> ChrisA
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180729/10d91efa/attachment.html>

From rosuav at gmail.com  Sun Jul 29 03:50:27 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 29 Jul 2018 17:50:27 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAKr=oZu+Ke6Qc9j4mc3XNLySdjA0V5WXbDivHQZkWRMir6UxNw@mail.gmail.com>
References: <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAEbHw4Yzt1Wfr1iBw+mmTYWhh8b2SwyFLX1AHOL-4YP6H+_ChA@mail.gmail.com>
 <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>
 <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
 <CAKr=oZtfYM2_fxYnde76o+jvuc_Y3AH9R93BCY6BoNdriuKqKw@mail.gmail.com>
 <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
 <CAPTjJmp2tN1TBz96EfnLRDsdO==QJ0xtYbCVkTBJDQ9+qOp6nA@mail.gmail.com>
 <CAKr=oZsTsR8Mv7_i386eOrEzQWTjiwbONt+7C=6D9V94f56kaQ@mail.gmail.com>
 <20180729055442.GH22554@ando.pearwood.info>
 <CAPTjJmpV9i+WyJxhCf6bbh04uOrexspB+iv-ZYrSi5OOa1p38w@mail.gmail.com>
 <CAKr=oZu+Ke6Qc9j4mc3XNLySdjA0V5WXbDivHQZkWRMir6UxNw@mail.gmail.com>
Message-ID: <CAPTjJmo1uL4CN8a6rqWLA8HpiTODGBEM82um3oYWfdcyvVpgMg@mail.gmail.com>

On Sun, Jul 29, 2018 at 5:12 PM, Abe Dillon <abedillon at gmail.com> wrote:
> The syntax of this proposal is almost objectively disgusting. It's truly
> horrid. I don't know how many ways to say it.

Almost. Except for the problem that "disgusting" is a subjective term.
If you want to say that this is *objectively disgusting*, you're going
to need some actual justification.

If you just mean "I find this disgusting", then that's fine, but we
already heard that argument. It's a statement of opinion, not of fact.

ChrisA

From mertz at gnosis.cx  Sun Jul 29 06:32:19 2018
From: mertz at gnosis.cx (David Mertz)
Date: Sun, 29 Jul 2018 06:32:19 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180729055442.GH22554@ando.pearwood.info>
References: <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAEbHw4Yzt1Wfr1iBw+mmTYWhh8b2SwyFLX1AHOL-4YP6H+_ChA@mail.gmail.com>
 <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>
 <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
 <CAKr=oZtfYM2_fxYnde76o+jvuc_Y3AH9R93BCY6BoNdriuKqKw@mail.gmail.com>
 <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
 <CAPTjJmp2tN1TBz96EfnLRDsdO==QJ0xtYbCVkTBJDQ9+qOp6nA@mail.gmail.com>
 <CAKr=oZsTsR8Mv7_i386eOrEzQWTjiwbONt+7C=6D9V94f56kaQ@mail.gmail.com>
 <20180729055442.GH22554@ando.pearwood.info>
Message-ID: <CAEbHw4Y0YFC0ttZuu3PDSjUJagC4uQ2XaOFC+6GNasG-OE+Gsw@mail.gmail.com>

On Sun, Jul 29, 2018, 2:00 AM Steven D'Aprano <steve at pearwood.info> wrote:

> Fine. So it takes them an extra day to learn one more operator. Big
> deal. It is commonly believed to take ten years to master a field or
> language. Amortize that one day over ten years and its virtually
> nothing.
>

This is where being wrong matters. The experience in this thread of most
supporters failing to get the semantics right shows that this isn't an
extra day to learn. It's something that experienced programmers would
continue to get won't in edge cases after years of use. The proposed
operators don't match how most experienced developers think about code.

Being glaringly ugly as written, and using an unintuitive or
counterintuitive symbol alternates that, of course.

I can hardly imagine a stronger bug magnet than PEP 505.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180729/ffbb7328/attachment-0001.html>

From contact at brice.xyz  Sun Jul 29 06:37:14 2018
From: contact at brice.xyz (Brice Parent)
Date: Sun, 29 Jul 2018 12:37:14 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180729060210.GI22554@ando.pearwood.info>
References: <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAEbHw4Yzt1Wfr1iBw+mmTYWhh8b2SwyFLX1AHOL-4YP6H+_ChA@mail.gmail.com>
 <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>
 <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
 <CAKr=oZtfYM2_fxYnde76o+jvuc_Y3AH9R93BCY6BoNdriuKqKw@mail.gmail.com>
 <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
 <5B5D0F09.5080209@canterbury.ac.nz>
 <20180729060210.GI22554@ando.pearwood.info>
Message-ID: <2bafaf65-8ac8-61ef-e859-1edfd1500c0e@brice.xyz>

Le 29/07/2018 ? 08:02, Steven D'Aprano a ?crit?:
> On Sun, Jul 29, 2018 at 12:49:13PM +1200, Greg Ewing wrote:
>> Abe Dillon wrote:
>>> others countering that `person.name <http://person.name>` is not how
>>> periods are used in natural languages, so using other symbols in
>>> unintuitive ways is perfectly fine.
>> Dots have been used for attribute access in so many languages
>> for so long that it has become the normal and expected syntax
>> to use. ?. is much more recent. Maybe in another 30 years, if
>> it has stood the test of time, it could be argued for on the
>> basis of familiarity, but not now.
> You're talking like the syntax is used only by a handful of experimental
> languages with a total user-base measured in the dozens.
>
> ?. is used by some of the most commonly used languages in the world,
> such as C++, Objective C and PHP, as well as up-and-coming "cool"
> languages getting lots of industry buzz, like Swift and Dart.
>
> Its certainly more familiar now than Python's slicing syntax was when
> Python first started.
But it's certainly less familiar than using 'foo++' and '++foo', but 
we're still writing 'foo += 1'. So familiarity in other languages is not 
the only point (alos, I'm convinced that if we were making a poll with 
average and advanced users of these languages about what these syntax 
were doing, a big proportion wouldn't know).

I believe the evolution should be done if the benefice are big enough 
(which I doubt, but I don't use Python in all the ways possible), not 
because others do it (whatever their reasons were). Then, when we know 
there's definitely a need to solve a problem, the solution should be 
chosen (??, but anything else could enter the discussion).

Here, we're mixing arguments about (and I confess I've done it too):
- how usefull it would be to have a solution to this problem
- if it should be solved by Python's syntax or by libraries (if it may 
be done, and if it may not, how spread are the use cases that can't be 
done this way)
- if other languages support something like that, if some are, how well 
this was accepted and if it's getting used in new code
- the proposed syntax itself

From contact at brice.xyz  Sun Jul 29 06:20:16 2018
From: contact at brice.xyz (Brice Parent)
Date: Sun, 29 Jul 2018 12:20:16 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAKr=oZu+Ke6Qc9j4mc3XNLySdjA0V5WXbDivHQZkWRMir6UxNw@mail.gmail.com>
References: <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAEbHw4Yzt1Wfr1iBw+mmTYWhh8b2SwyFLX1AHOL-4YP6H+_ChA@mail.gmail.com>
 <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>
 <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
 <CAKr=oZtfYM2_fxYnde76o+jvuc_Y3AH9R93BCY6BoNdriuKqKw@mail.gmail.com>
 <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
 <CAPTjJmp2tN1TBz96EfnLRDsdO==QJ0xtYbCVkTBJDQ9+qOp6nA@mail.gmail.com>
 <CAKr=oZsTsR8Mv7_i386eOrEzQWTjiwbONt+7C=6D9V94f56kaQ@mail.gmail.com>
 <20180729055442.GH22554@ando.pearwood.info>
 <CAPTjJmpV9i+WyJxhCf6bbh04uOrexspB+iv-ZYrSi5OOa1p38w@mail.gmail.com>
 <CAKr=oZu+Ke6Qc9j4mc3XNLySdjA0V5WXbDivHQZkWRMir6UxNw@mail.gmail.com>
Message-ID: <1e9ce8aa-fcee-26a0-70e9-cc34979b9ef8@brice.xyz>

Le 29/07/2018 ? 09:12, Abe Dillon a ?crit?:

>
[...]
>
> NO! I'm proposing:
>
> spam.eggs.cheese.aardvark?
>
> A single POSTFIX operator that has a high priority in the order of 
> operations.
I don't believe we need spam?.eggs.cheese?.aardvark, because I don't 
think it is justified by the small benefits it gets us.
For the same reason, I don't believe we need spam.eggs.cheese.aardvark? 
(there is exactly the same number of use cases). We win a bit in 
readability as it's closer to most spoken languages, but we lose in 
granularity as we're hiding any error that would raise if spam.eggs 
returned None, so it's not a full win on this side either...

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180729/0b3cd76d/attachment.html>

From steve at pearwood.info  Sun Jul 29 07:45:07 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 29 Jul 2018 21:45:07 +1000
Subject: [Python-ideas] Proposed alternative postfix syntax [was PEP 505:
 None-aware operators]
In-Reply-To: <CAKr=oZu+Ke6Qc9j4mc3XNLySdjA0V5WXbDivHQZkWRMir6UxNw@mail.gmail.com>
References: <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>
 <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
 <CAKr=oZtfYM2_fxYnde76o+jvuc_Y3AH9R93BCY6BoNdriuKqKw@mail.gmail.com>
 <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
 <CAPTjJmp2tN1TBz96EfnLRDsdO==QJ0xtYbCVkTBJDQ9+qOp6nA@mail.gmail.com>
 <CAKr=oZsTsR8Mv7_i386eOrEzQWTjiwbONt+7C=6D9V94f56kaQ@mail.gmail.com>
 <20180729055442.GH22554@ando.pearwood.info>
 <CAPTjJmpV9i+WyJxhCf6bbh04uOrexspB+iv-ZYrSi5OOa1p38w@mail.gmail.com>
 <CAKr=oZu+Ke6Qc9j4mc3XNLySdjA0V5WXbDivHQZkWRMir6UxNw@mail.gmail.com>
Message-ID: <20180729114507.GJ22554@ando.pearwood.info>

On Sun, Jul 29, 2018 at 02:12:22AM -0500, Abe Dillon wrote:

[I said this]
> > Python is very little like natural language. Your earlier idea that
> > attribute access spam.eggs is like natural language (in English) because
> > its only a few characters different from "spam's eggs" really doesn't
> > hold up. Apart from the word order, there's no similarity.
> 
> I feel like you're being willfully dense about this.

Am I? Your exact words were:

  My point is that '.' is unobtrusive, so it's not *that* different from 
  `person name' which isn't *that* different from `person's name`.

so you absolutely are claiming that dot access "isn't *that* different" 
from the possessive in English.

The dot might be small, but the last thing we want if for the dot to be 
unobtrusive. We want it to stand out and be obvious, so we can never 
mistake 1.5 for 15 or (say) "appbuild" for "app.build".

(The first is a variable called "appbuild", the second is a build method 
or attribute invoked on a variable called "app". Very different things 
indeed. We really don't want to mistake one for the other.)

This is one important reason that most programmers use monospaced fonts 
for code, so that the dot takes up as much space and any other character 
and stands out more.

The ?. syntax is obviously different from regular dot, but surely that 
is an *advantage* not a disadvantage? Things which do something 
different should look different.

One of the problems with David Mertz's counter-proposal for a magic 
None-aware proxy is that we can't tell the difference between 

    spam.eggs

and

    spam.eggs

where one of them is magic and the other is not. So we have a nice, 
clear, obvious difference in syntax so that at a glance the reader can 
instantly distinguish between dot and maybe-dot, and you are complaining 
that it is too clear, obvious and easily distinguishable compared to a 
regular dot.


> [Steve D'Aprano]
> > I don't know that Python has any binary operators which are split
> > into an infix part and a postfix part.
> 
> That's not at all what I proposed.

That's exactly what you proposed, you just didn't realise it. See 
below. Your syntax is

    foo.spam?

where the dot and the question mark together make a new binary operator. 
They aren't really delimiters like the parentheses in func( ... ) or the 
square brackets in seq[ ... ] but two halves of a single operator (or 
operator like construct) with an infix part (the dot) and a postfix part 
(the question mark). They certainly can't be two separate operators, a 
dot and a question mark.

I don't know how the Python parser itself will deal with this, but 
consider the syntax from the perspective of a reader. You suggest:


> I'm proposing:
> 
> spam.eggs.cheese.aardvark?

Given that, we read the code as:

   spam
   normal dot, so get the attribute eggs
   normal dot, so get the attribute cheese
   normal dot, so get the attribute aardvark 
   question mark, rewind to the beginning and start again:
   spam
   maybe-dot, so maybe get eggs, or None
   maybe-dot, so maybe get cheese, or None
   maybe-dot, so maybe get aardvark
   

I'm not saying that's what the parser will have to do (although I 
suspect it will need some sort of look-ahead, or backtracking, to parse 
this) but that's certainly how I'll read this.


> A single POSTFIX operator that has a high priority in the order of
> operations.

I don't think that will do what you think it will do. If ? is a normal 
unary postfix operator then its operand here:

    spam.eggs.cheese.aardvark?

will be (spam.eggs.cheese.aardvark), which defeats the purpose of having 
a maybe-dot operator. By the time the ? operator is called, the regular 
dot operators will have already been called.

It may be possible to get the syntax you prefer, but readability-wise, I 
think it is magical and confusing to have a postfix symbol at the end of 
an expression *retroactively* change the meaning of symbols occuring 
before it in the expression.

I expect it will also be a bug-magnet when we need to mix regular dot 
and maybe-dot in the same expression:

    spam.eggs?.cheese  # eggs might be None, but spam won't be

would need to be written as

    (spam.eggs).cheese?

but who is going to remember those parentheses? Instead I expect people 
will consistently write:

    spam.eggs.cheese?

thinking the ? applies only to *one* dot, not all of them, and wrongly 
suppress exceptions when spam is None (which ought to be treated as a 
bug).



-- 
Steve

From rosuav at gmail.com  Sun Jul 29 08:15:40 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 29 Jul 2018 22:15:40 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4Y0YFC0ttZuu3PDSjUJagC4uQ2XaOFC+6GNasG-OE+Gsw@mail.gmail.com>
References: <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAEbHw4Yzt1Wfr1iBw+mmTYWhh8b2SwyFLX1AHOL-4YP6H+_ChA@mail.gmail.com>
 <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>
 <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
 <CAKr=oZtfYM2_fxYnde76o+jvuc_Y3AH9R93BCY6BoNdriuKqKw@mail.gmail.com>
 <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
 <CAPTjJmp2tN1TBz96EfnLRDsdO==QJ0xtYbCVkTBJDQ9+qOp6nA@mail.gmail.com>
 <CAKr=oZsTsR8Mv7_i386eOrEzQWTjiwbONt+7C=6D9V94f56kaQ@mail.gmail.com>
 <20180729055442.GH22554@ando.pearwood.info>
 <CAEbHw4Y0YFC0ttZuu3PDSjUJagC4uQ2XaOFC+6GNasG-OE+Gsw@mail.gmail.com>
Message-ID: <CAPTjJmpbj9=xYff14SQdnb5O2_Ttj70JUd_aWpBx4ud4TXXnXQ@mail.gmail.com>

On Sun, Jul 29, 2018 at 8:32 PM, David Mertz <mertz at gnosis.cx> wrote:
> I can hardly imagine a stronger bug magnet than PEP 505.

The hyperbole in this thread is impressive. Not just "I can hardly
imagine anyone PROPOSING a stronger bug magnet". You cannot imagine
one even existing.

And this is after people have proposed that "if x = 1:" become the
spelling for assignment expressions - something so well known as a bug
magnet that C compilers specifically warn against it, and style guides
recommend always writing "if (1 == x)" instead. Claiming that "?." is
*the worst possible bug magnet* rather weakens your credibility here.

ChrisA

From mertz at gnosis.cx  Sun Jul 29 10:00:46 2018
From: mertz at gnosis.cx (David Mertz)
Date: Sun, 29 Jul 2018 10:00:46 -0400
Subject: [Python-ideas] Proposed alternative postfix syntax [was PEP
 505: None-aware operators]
In-Reply-To: <20180729114507.GJ22554@ando.pearwood.info>
References: <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>
 <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
 <CAKr=oZtfYM2_fxYnde76o+jvuc_Y3AH9R93BCY6BoNdriuKqKw@mail.gmail.com>
 <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
 <CAPTjJmp2tN1TBz96EfnLRDsdO==QJ0xtYbCVkTBJDQ9+qOp6nA@mail.gmail.com>
 <CAKr=oZsTsR8Mv7_i386eOrEzQWTjiwbONt+7C=6D9V94f56kaQ@mail.gmail.com>
 <20180729055442.GH22554@ando.pearwood.info>
 <CAPTjJmpV9i+WyJxhCf6bbh04uOrexspB+iv-ZYrSi5OOa1p38w@mail.gmail.com>
 <CAKr=oZu+Ke6Qc9j4mc3XNLySdjA0V5WXbDivHQZkWRMir6UxNw@mail.gmail.com>
 <20180729114507.GJ22554@ando.pearwood.info>
Message-ID: <CAEbHw4bWitUXu4S2nmf=JKji8151US0yFMFRVm7KS0GYR+n+Gg@mail.gmail.com>

On Sun, Jul 29, 2018, 7:46 AM Steven D'Aprano <steve at pearwood.info> wrote:

> One of the problems with David Mertz's counter-proposal for a magic
> None-aware proxy is that we can't tell the difference between  spam.eggs
> and spam.eggs where one of them is magic and the other is not. So we have a
> nice, clear, obvious difference in syntax so that at a glance the reader
> can instantly distinguish between dot and maybe-dot, and you are
> complaining that it is too clear, obvious and easily distinguishable
> compared to a regular dot.
>

That's exactly the strongest advantage in what I suggest!

The special domain where you want attribute access (or dictionary lookup)
to be None coalescing is something marked and different from most of what
you want in general attribute access. But once you're in that special world
of "semi-structured hierarchical data with leaves marked with None" you
don't want to be reminded in every operator. It's just what you do with
that kind of object.

Naming variable might be a good way to remind readers of the domain. Or
comments in the source code. Or putting the special behavior in an isolated
function. But effectively, this is a domain you want to enter for a while,
then return to the "real world" after some operations are done.

For a pretty good analogy, you cannot tell the difference between 'a+b' and
'a+b' easily either, in isolation.  But one of those is arithmetic over the
domain of integers, while the other is arithmetic over the domain of a
Z(65537) modular ring. Many operations are the same in both domains, but
I'd really hate to need a special operator to perform modular arithmetic
rather than integer arithmetic.  E.g. 'a + b' vs 'a ? b'.

It is MUCH more elegant and clear in both examples to put the work on the
*object* that has special behavior, not on the operator. I can see that if
you did a very special kind of programming where you frequently mixed
modular and integer arithmetic, you might want the reminder. But that's a
rare need, and the language itself should not grow that syntax (perhaps
something specialized like Sage or Octave might want a different balance,
very reasonably).

"Semi-structured hierarchical data with leafs marked with None" is just
altogether too special to impose the burden all Python programmers. A
None-aware proxy isn't the best possible thing for that special use by any
means. But it's pretty good without imposing any burden on the large
majority who will never need or use it.

Another similar case is the '@' operator. This is exactly a special domain
where an operator indicates special behavior. In principle, you could make
objects that did anything whatsoever when they see that, but the name
.__matmul__ is pretty suggestive of recommended use. However, nothing in
the builtins or standard library define any behavior at all. You can
happily program with Python your whole life and never think about this
operator. The only folks who will see it are folks who use NumPy and *also*
do linear algebra with it. By definition, the objects if this domain are
already special, as it should be.

Well, footnote. I think I've seen some other libraries that create a DSL
around email that use '@' also. That also feels fine. It's a very
different, but equally special, domain where the objects themselves define
the special behavior.

If PEP 505 were proposing a regular new protocol for .__none_attr__() or
some better dunder name, I would oppose it far less. Specifically, if the
behavior of this dunder operator were left to library authors who created
such special objects, the danger and bug-magneticism would be much less.
I'm well aware that dunders can't get you shortcutting, but modulo that,
such a proposal would be relatively innocuous (I'd still be -0, but not
really care).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180729/c743c7b4/attachment-0001.html>

From steve at pearwood.info  Sun Jul 29 10:57:38 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 30 Jul 2018 00:57:38 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4Y0YFC0ttZuu3PDSjUJagC4uQ2XaOFC+6GNasG-OE+Gsw@mail.gmail.com>
References: <CAEbHw4Yzt1Wfr1iBw+mmTYWhh8b2SwyFLX1AHOL-4YP6H+_ChA@mail.gmail.com>
 <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>
 <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
 <CAKr=oZtfYM2_fxYnde76o+jvuc_Y3AH9R93BCY6BoNdriuKqKw@mail.gmail.com>
 <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
 <CAPTjJmp2tN1TBz96EfnLRDsdO==QJ0xtYbCVkTBJDQ9+qOp6nA@mail.gmail.com>
 <CAKr=oZsTsR8Mv7_i386eOrEzQWTjiwbONt+7C=6D9V94f56kaQ@mail.gmail.com>
 <20180729055442.GH22554@ando.pearwood.info>
 <CAEbHw4Y0YFC0ttZuu3PDSjUJagC4uQ2XaOFC+6GNasG-OE+Gsw@mail.gmail.com>
Message-ID: <20180729145736.GK22554@ando.pearwood.info>

On Sun, Jul 29, 2018 at 06:32:19AM -0400, David Mertz wrote:
> On Sun, Jul 29, 2018, 2:00 AM Steven D'Aprano <steve at pearwood.info> wrote:
> 
> > Fine. So it takes them an extra day to learn one more operator. Big
> > deal. It is commonly believed to take ten years to master a field or
> > language. Amortize that one day over ten years and its virtually
> > nothing.
> >
> 
> This is where being wrong matters. The experience in this thread of most
> supporters failing to get the semantics right shows that this isn't an
> extra day to learn.

The difficulty one or two people had in coming up with a correct 
equivalent to none-aware operators on the spur of the moment is simply 
not relevant. Aside from whichever developers implements the feature, 
the rest of us merely *use* it, just as we already use import, 
comprehensions, yield from, operators, class inheritence, and other 
features which are exceedingly difficult to emulate precisely in pure 
Python code.

Even something as simple as the regular dot attribute lookup is 
difficult to emulate precisely. I doubt most people would be able to 
write a pure-Python version of getattr correctly the first time. Or even 
the fifth. I know I wouldn't.

I'm sure that you are fully aware that if this proposal is accepted, 
people will not need to reinvent the wheel by emulating these none-aware 
operators in pure Python, so your repeated argument that (allegedly) 
even the supporters can't implement it correctly is pure FUD. They won't 
have to implement it, that's the point.


-- 
Steve

From abedillon at gmail.com  Sun Jul 29 11:12:19 2018
From: abedillon at gmail.com (Abe Dillon)
Date: Sun, 29 Jul 2018 10:12:19 -0500
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAPTjJmpbj9=xYff14SQdnb5O2_Ttj70JUd_aWpBx4ud4TXXnXQ@mail.gmail.com>
References: <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAEbHw4Yzt1Wfr1iBw+mmTYWhh8b2SwyFLX1AHOL-4YP6H+_ChA@mail.gmail.com>
 <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>
 <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
 <CAKr=oZtfYM2_fxYnde76o+jvuc_Y3AH9R93BCY6BoNdriuKqKw@mail.gmail.com>
 <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
 <CAPTjJmp2tN1TBz96EfnLRDsdO==QJ0xtYbCVkTBJDQ9+qOp6nA@mail.gmail.com>
 <CAKr=oZsTsR8Mv7_i386eOrEzQWTjiwbONt+7C=6D9V94f56kaQ@mail.gmail.com>
 <20180729055442.GH22554@ando.pearwood.info>
 <CAEbHw4Y0YFC0ttZuu3PDSjUJagC4uQ2XaOFC+6GNasG-OE+Gsw@mail.gmail.com>
 <CAPTjJmpbj9=xYff14SQdnb5O2_Ttj70JUd_aWpBx4ud4TXXnXQ@mail.gmail.com>
Message-ID: <CAKr=oZv1Nibq=G+p442aF9Ce874HvLb60D18A3qMWoBJC41HDw@mail.gmail.com>

[Chris Angelico]

> Almost. Except for the problem that "disgusting" is a subjective term.


Yes. Thank you for explaining the joke.

 [Brice Parent]

> I don't believe we need spam.eggs.cheese.aardvark? (there is exactly the
> same number of use cases). We win a bit in readability as it's closer to
> most spoken languages, but we lose in granularity as we're hiding any error
> that would raise if spam.eggs returned None, so it's not a full win on
> this side either...


First, I mostly proposed that alternative to serve my argument that there
may be a better syntax that PEP 505 would make impossible down the road.
It's a half-baked proposal at best and I agree with your over-all sentiment
that it's probably not something we should consider either. That being
said; in the normal usage of '?.' and '?[]' it would make little sense to
only check the beginning of the expression:

spam?.eggs.cheese.aardvark  # why would you ever do this?

It makes some sense to do this:

spam.eggs.cheese?.aardvark

But in my proposal, you could force any part of the expression to evaluate
before the '?' with parentheses and expose whatever errors that might throw:

(spam.eggs.cheese).aardvark?

On Sun, Jul 29, 2018 at 7:15 AM, Chris Angelico <rosuav at gmail.com> wrote:

> On Sun, Jul 29, 2018 at 8:32 PM, David Mertz <mertz at gnosis.cx> wrote:
> > I can hardly imagine a stronger bug magnet than PEP 505.
>
> The hyperbole in this thread is impressive. Not just "I can hardly
> imagine anyone PROPOSING a stronger bug magnet". You cannot imagine
> one even existing.
>
> And this is after people have proposed that "if x = 1:" become the
> spelling for assignment expressions - something so well known as a bug
> magnet that C compilers specifically warn against it, and style guides
> recommend always writing "if (1 == x)" instead. Claiming that "?." is
> *the worst possible bug magnet* rather weakens your credibility here.
>
> ChrisA
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180729/83bbce09/attachment.html>

From abedillon at gmail.com  Sun Jul 29 13:08:46 2018
From: abedillon at gmail.com (Abe Dillon)
Date: Sun, 29 Jul 2018 12:08:46 -0500
Subject: [Python-ideas] Proposed alternative postfix syntax [was PEP
 505: None-aware operators]
In-Reply-To: <CAEbHw4bWitUXu4S2nmf=JKji8151US0yFMFRVm7KS0GYR+n+Gg@mail.gmail.com>
References: <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>
 <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
 <CAKr=oZtfYM2_fxYnde76o+jvuc_Y3AH9R93BCY6BoNdriuKqKw@mail.gmail.com>
 <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
 <CAPTjJmp2tN1TBz96EfnLRDsdO==QJ0xtYbCVkTBJDQ9+qOp6nA@mail.gmail.com>
 <CAKr=oZsTsR8Mv7_i386eOrEzQWTjiwbONt+7C=6D9V94f56kaQ@mail.gmail.com>
 <20180729055442.GH22554@ando.pearwood.info>
 <CAPTjJmpV9i+WyJxhCf6bbh04uOrexspB+iv-ZYrSi5OOa1p38w@mail.gmail.com>
 <CAKr=oZu+Ke6Qc9j4mc3XNLySdjA0V5WXbDivHQZkWRMir6UxNw@mail.gmail.com>
 <20180729114507.GJ22554@ando.pearwood.info>
 <CAEbHw4bWitUXu4S2nmf=JKji8151US0yFMFRVm7KS0GYR+n+Gg@mail.gmail.com>
Message-ID: <CAKr=oZspFj8q3OMjJxYG15ktoF0xykgddaVcyUApM_nsvJLdWQ@mail.gmail.com>

[Steve D'Aprano]

> so you absolutely are claiming that dot access "isn't *that* different"
> from the possessive in English.


Only in the context that I gave: `person.name`. In other contexts (e.g.
thing.is_metal) the possessive analogy is obviously inaccurate, but I
didn't make the broad claim that '.' means possessive.

[Steve D'Aprano]

> The dot might be small, but the last thing we want if for the dot to be
> unobtrusive. We want it to stand out and be obvious, so we can never
> mistake 1.5 for 15 or (say) "appbuild" for "app.build".


Unobtrusive doesn't mean invisible.

*ob?tru?sive adjective: *noticeable or prominent *in an unwelcome or
> intrusive way*.


I don't think '.' is invisible, nor that it should be. It's fairly elegant
because it communicates information without breaking the flow of reading.
It's as noticeable as it needs to be without getting in the way.

[Steve D'Aprano]

> Your syntax is
>     foo.spam?
> where the dot and the question mark together make a new binary operator.
> They aren't really delimiters like the parentheses in func( ... ) or the
> square brackets in seq[ ... ] but two halves of a single operator (or
> operator like construct) with an infix part (the dot) and a postfix part
> (the question mark). They certainly can't be two separate operators, a
> dot and a question mark.


They can be two separate operators.  The '?' operator would say, "evaluate
the preceding expression as though it was in a try-except block and if a
NoneType error is thrown, ignore it and evaluate to None." I imagine it
would work similarly to how ternary operators work: <expr_1> if
<condition_expr> else <expr_2> where neither expr_1 nor expr_2 are
evaluated before <condition_expr>.

[Steve D'Aprano]

> I'm not saying that's what the parser will have to do (although I
> suspect it will need some sort of look-ahead, or backtracking, to parse
> this) but that's certainly how I'll read this.


look-ahead and back-tracing is how people read punctuation like '?' and '!'
in natural language, and there are plenty of examples of Python following
this pattern in other expressionized syntax like in comprehensions and
ternary expressions. I actually think this makes sense for expressionized
syntax because it puts the "meat" before the "potatoes". When you're
reading code like:

initials = [person.name[0] ...

You can usually guess what goes in the `...` from context so it's
redundant, so it makes sense that it comes later. I made the argument a
while back that this same pattern should have been followed for lambda
expressions for the same reason: the argument list is almost always
redundant:

hand = sorted(cards, key=(<expression_based_on_individual_card> with
<parameters>))

Similarly, If I want to guard against None errors, that's something that
can come after the fact because obviously I'm not trying to access
attributes or index into None objects:

initial = person.name[0]?

[Steve D'Aprano]

> It may be possible to get the syntax you prefer, but readability-wise, I
> think it is magical and confusing to have a postfix symbol at the end of
> an expression *retroactively* change the meaning of symbols occuring
> before it in the expression.


1) That's the way it works in natural language
2) That's the way it should work in the given context because it's largely
a redundant check (from the reader's perspective) because you obviously
don't want to access or index into a None object. That can be inferred from
context by most humans, but needs to be explicit for the interpreter's sake.
3) there's plenty of precedent for behavior in which the order of execution
isn't left to right.

[Steve D'Aprano]
> I expect it will also be a bug-magnet when we need to mix regular dot
> and maybe-dot in the same expression:
>
 >   spam.eggs?.cheese  # eggs might be None, but spam won't be
>
> would need to be written as
>
>    (spam.eggs).cheese?

It may very well be a bug magnet. I'm not sure how often that usage will
come up. I don't find anything all that objectionable about the second
form. At least it looks like Python.

As you've pointed out many times before: people would have to learn it to
use it correctly. There's no way around that. I don't know why you'd all of
a sudden abandon that line of reasoning.

I believe the want for this functionality is symptomatic of a want for more
general functionality like deferred expression evaluation or an
expressionized try-except. I would rather try to crack those nuts than
implement a overly specific solution that would become cruft later on when
deffered evaluation or expressionized try-except are implemented.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180729/4277f4ec/attachment.html>

From nicholas.chammas at gmail.com  Sun Jul 29 13:43:47 2018
From: nicholas.chammas at gmail.com (Nicholas Chammas)
Date: Sun, 29 Jul 2018 13:43:47 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180729145736.GK22554@ando.pearwood.info>
References: <CAEbHw4Yzt1Wfr1iBw+mmTYWhh8b2SwyFLX1AHOL-4YP6H+_ChA@mail.gmail.com>
 <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>
 <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
 <CAKr=oZtfYM2_fxYnde76o+jvuc_Y3AH9R93BCY6BoNdriuKqKw@mail.gmail.com>
 <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
 <CAPTjJmp2tN1TBz96EfnLRDsdO==QJ0xtYbCVkTBJDQ9+qOp6nA@mail.gmail.com>
 <CAKr=oZsTsR8Mv7_i386eOrEzQWTjiwbONt+7C=6D9V94f56kaQ@mail.gmail.com>
 <20180729055442.GH22554@ando.pearwood.info>
 <CAEbHw4Y0YFC0ttZuu3PDSjUJagC4uQ2XaOFC+6GNasG-OE+Gsw@mail.gmail.com>
 <20180729145736.GK22554@ando.pearwood.info>
Message-ID: <CAOhmDzfihX9v_H=_UEq0STX6kvrQVSOy9okwLWHTXE38T=mDew@mail.gmail.com>

On Sun, Jul 29, 2018 at 10:58 AM Steven D'Aprano <steve at pearwood.info>
wrote:

> On Sun, Jul 29, 2018 at 06:32:19AM -0400, David Mertz wrote:
> > On Sun, Jul 29, 2018, 2:00 AM Steven D'Aprano <steve at pearwood.info>
> wrote:
> >
> > > Fine. So it takes them an extra day to learn one more operator. Big
> > > deal. It is commonly believed to take ten years to master a field or
> > > language. Amortize that one day over ten years and its virtually
> > > nothing.
> > >
> >
> > This is where being wrong matters. The experience in this thread of most
> > supporters failing to get the semantics right shows that this isn't an
> > extra day to learn.
>
> The difficulty one or two people had in coming up with a correct
> equivalent to none-aware operators on the spur of the moment is simply
> not relevant. Aside from whichever developers implements the feature,
> the rest of us merely *use* it, just as we already use import,
> comprehensions, yield from, operators, class inheritence, and other
> features which are exceedingly difficult to emulate precisely in pure
> Python code.
>
> Even something as simple as the regular dot attribute lookup is
> difficult to emulate precisely. I doubt most people would be able to
> write a pure-Python version of getattr correctly the first time. Or even
> the fifth. I know I wouldn't.
>
> I'm sure that you are fully aware that if this proposal is accepted,
> people will not need to reinvent the wheel by emulating these none-aware
> operators in pure Python, so your repeated argument that (allegedly)
> even the supporters can't implement it correctly is pure FUD. They won't
> have to implement it, that's the point.


+1 to this, though I agree with Raymond's post that perhaps a breather on
language changes would be helpful until some of the recently introduced
features have become more familiar to folks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180729/6fd39952/attachment-0001.html>

From tim.peters at gmail.com  Sun Jul 29 15:08:16 2018
From: tim.peters at gmail.com (Tim Peters)
Date: Sun, 29 Jul 2018 14:08:16 -0500
Subject: [Python-ideas] Start argument for itertools.accumulate() [Was:
 Proposal: A Reduce-Map Comprehension and a "last" builtin]
In-Reply-To: <AF12802D-AAB7-48D7-847A-C12D4901DA65@gmail.com>
References: <AF12802D-AAB7-48D7-847A-C12D4901DA65@gmail.com>
Message-ID: <CAExdVN=PTs-VV8xEpWZMXJ3yQxfHQoZAhHKvO558GfkMvaz=Kw@mail.gmail.com>

Since this started, I've been using variants of the following in my own
code, wrapping `accumulate`:

    from itertools import accumulate, chain, cycle, islice

    def accum(iterable, func=None, *, initial=None, skipfirst=False):
        if initial is not None:
            iterable = chain([initial], iterable)

        if func is None:
            iterable = accumulate(iterable)
        else:
            iterable = accumulate(iterable, func)

        if skipfirst:
            iterable = islice(iterable, 1, None)
        return iterable

I quite like it!  It handles everything that's come up pretty naturally.  A
difference from what was discussed before:  I opposed having a `skipfirst`
argument because I didn't want an optional argument that _only_ applied if
another optional argument was specified.  But `skipfirst` in the above
applies regardless of whether `initial` is specified.  It turned out to be
useful in cases where the _effect_ of `initial` had been gotten instead via
slamming the initial value into the first object returned by `iterable`,
and then ignored later by a special first case to throw away the first
result `accumulate` returned.

I'm writing this now because it turned out I used it eight times yesterday,
which raised it above background noise level ;-)

The first use captures a sequence countless thousands of Python programmers
have crafted in various ways by hand:

    def firstfactor(n, limit=1000):
        for p in chain([2, 3],
                       accum(cycle([2, 4]), initial=5)):
            # 2, 3, and integers not divisible by 2 or 3
            # 2, 3, 5, 7, 11, 13, 17, 19, 23, 25, 29, 31, ...
            if p*p > n:
                return n
            if p > limit:
                return 0
            if n % p == 0:
                return p

The same sequence can be gotten more obscurely via:

        for p in chain([2, 3],
            accum(cycle([4, 2]), initial=1, skipfirst=True)):

The remaining 7 uses were in transcribing's Lehman's elegant[1]
optimization of Fermat's "difference of squares" factorization method:

    def large(deltas, init):
        for k in accum(cycle(deltas), initial=init):
            ...

    # ints divisible by 30
    large([30], 30)

    # ints divisible by 24 but not by 30
    large([24, 24, 24, 48], 24)

    # ints divisible by 12 but not by 30 or 24
    large([24, 48, 24, 24], 12)

    # ints divisible by 18 but not by 30, 24, or 12
    large([36, 72, 36, 36], 18)

    # ints divisible by 6 but not by 30, 24, 12, or 18
    large([36, 24, 12, 24, 12, 24, 36, 12], 6)

    # ints divisible by 2 but not by 30, 24, 12, 18, or 6
    large([2, 4], 2)

    # ints not divisible by 30, 24, 12, 18, 6, or 2
    large([2], 1)

Lehman built his sequence generator "by hand" in Algol. where the delta
array (`c` in his code) is inherited from `large`'s enclosing scope, and
`large()` is passed just the number of deltas and the initial value.  A
more literal transcription of his code would have used:

    def large(deltas, init):
        for k in accum(cycle(deltas), initial=init, skipfirst=True):
            ...

instead with the delta lists rotated and with a different initial value.
For example, instead of

    large([24, 24, 24, 48], 24)

above his code uses

    large([48, 24, 24, 24], -24)

The latter needs to ignore the initial (-24) value (except to add it to the
first delta: -24 + 48 = 24).  That was more natural in context, due to the
way he advanced the generated in a `while True:` loop built out of gotos ;-)


[1]
https://math.boisestate.edu/~liljanab/Cryptology1Fall2013/LehmanFactoring.pdf

Historical gloss:  Lehman surprisingly transformed Fermat's O(n) method
into an O(cube_root(n)) method.  I believe that was the first deterministic
factoring method known beating trial division's O(sqrt(n)) time.  Alas for
Lehman, Pollard very soon after published his `rho` method, which runs in
probabilistic O(sqrt(p)) time where p is n's smallest prime factor, so in
worst-case probabilistic O(fourth_root(n)) time.  That relegated Lehman's
method to an academic curiosity, although it remains supremely effective if
N is the product of two odd primes whose ratio is close to a fraction with
"small" numerator and denominator.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180729/4c3cdc16/attachment.html>

From steve at pearwood.info  Sun Jul 29 20:48:57 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 30 Jul 2018 10:48:57 +1000
Subject: [Python-ideas] Proposed alternative postfix syntax [was PEP
 505: None-aware operators]
In-Reply-To: <CAKr=oZspFj8q3OMjJxYG15ktoF0xykgddaVcyUApM_nsvJLdWQ@mail.gmail.com>
References: <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
 <CAPTjJmp2tN1TBz96EfnLRDsdO==QJ0xtYbCVkTBJDQ9+qOp6nA@mail.gmail.com>
 <CAKr=oZsTsR8Mv7_i386eOrEzQWTjiwbONt+7C=6D9V94f56kaQ@mail.gmail.com>
 <20180729055442.GH22554@ando.pearwood.info>
 <CAPTjJmpV9i+WyJxhCf6bbh04uOrexspB+iv-ZYrSi5OOa1p38w@mail.gmail.com>
 <CAKr=oZu+Ke6Qc9j4mc3XNLySdjA0V5WXbDivHQZkWRMir6UxNw@mail.gmail.com>
 <20180729114507.GJ22554@ando.pearwood.info>
 <CAEbHw4bWitUXu4S2nmf=JKji8151US0yFMFRVm7KS0GYR+n+Gg@mail.gmail.com>
 <CAKr=oZspFj8q3OMjJxYG15ktoF0xykgddaVcyUApM_nsvJLdWQ@mail.gmail.com>
Message-ID: <20180730004856.GL22554@ando.pearwood.info>

On Sun, Jul 29, 2018 at 12:08:46PM -0500, Abe Dillon wrote:

> look-ahead and back-tracing is how people read punctuation like '?' and '!'
> in natural language

I really wish you would stop talking about "natural language" as if 
there were only one (and it were English). Many natural languages don't 
even have punctuation like '?', or if they do, they are a very recent 
innovation. Spanish uses a bracketing pair of marks:

   ?Qu? Mr Fawlty?

while many other European languages make do with just one at the end.
And not all of them use the same symbol. (Greek uses the semi-colon, 
which must make reading C-like languages a remarkable experience.)

But even in English, that isn't how people read questions in general, 
which is why we can still comprehend punctuation-less sentences

   what's for dinner

   where are you

   who was the German professor in Lord Palmerston's quote about the 
   Schleswig-Holstein business

as questions, despite the lack of a question mark. Questions really only 
need the question mark if they would otherwise be ambiguously either a 
statement or a question.

You may be able to find examples of English questions which do require 
back-tracking, but that is generally considered to be a bad thing, 
increasing surprise (not a good thing unless you are attempting humour) 
and reducing the speed and ease of comprehension. See my earlier link 
about garden path sentences.

In other words, reducing not increasing readability.


> and there are plenty of examples of Python following
> this pattern in other expressionized syntax like in comprehensions and
> ternary expressions.

But not binary operators which take only two arguments, in this case a 
name and an attribute name.


> I actually think this makes sense for expressionized
> syntax because it puts the "meat" before the "potatoes". When you're
> reading code like:
> 
> initials = [person.name[0] ...
> 
> You can usually guess what goes in the `...` from context

Can you?

In that example I can't even guess if that's the start of a list 
comprehension or a list display. If its a comprehension, I sure as hell 
can't guess the important question of what it being iterated over.


> so it's redundant, so it makes sense that it comes later.

If it's redundant, why don't we just leave it out?


-- 
Steve

From abedillon at gmail.com  Mon Jul 30 00:00:36 2018
From: abedillon at gmail.com (Abe Dillon)
Date: Sun, 29 Jul 2018 23:00:36 -0500
Subject: [Python-ideas] Proposed alternative postfix syntax [was PEP
 505: None-aware operators]
In-Reply-To: <20180730004856.GL22554@ando.pearwood.info>
References: <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
 <CAPTjJmp2tN1TBz96EfnLRDsdO==QJ0xtYbCVkTBJDQ9+qOp6nA@mail.gmail.com>
 <CAKr=oZsTsR8Mv7_i386eOrEzQWTjiwbONt+7C=6D9V94f56kaQ@mail.gmail.com>
 <20180729055442.GH22554@ando.pearwood.info>
 <CAPTjJmpV9i+WyJxhCf6bbh04uOrexspB+iv-ZYrSi5OOa1p38w@mail.gmail.com>
 <CAKr=oZu+Ke6Qc9j4mc3XNLySdjA0V5WXbDivHQZkWRMir6UxNw@mail.gmail.com>
 <20180729114507.GJ22554@ando.pearwood.info>
 <CAEbHw4bWitUXu4S2nmf=JKji8151US0yFMFRVm7KS0GYR+n+Gg@mail.gmail.com>
 <CAKr=oZspFj8q3OMjJxYG15ktoF0xykgddaVcyUApM_nsvJLdWQ@mail.gmail.com>
 <20180730004856.GL22554@ando.pearwood.info>
Message-ID: <CAKr=oZu23=pC5RdjFDRhj532hVpN9Bde74VQn6zSDCdmBp59Mw@mail.gmail.com>

[Steven D'Aprano]

> really wish you would stop talking about "natural language" as if
> there were only one (and it were English).


I'm fine with that. I've just had someone jump down my throat before about
being overly English-centric when talking about readability.

[Steven D'Aprano]

> even in English, that isn't how people read questions in general,
> which is why we can still comprehend punctuation-less sentences


Yes because you can make use of other context clues. So if I wrote:

def f(person=None):
    initial = person.name[0]?
    ...

You can anticipate the '?' at the end of the expression. Otherwise I would
be clearly missing an important edge case.
None checking is often done in very close proximity to some code that might
yield a None, so this should rarely be a surprise unless the coder is
writing the equivalent of a garden-path sentence.

[Steven D'Aprano]

> You may be able to find examples of English questions which do require
> back-tracking


It's easier, I'll admit, to find example's of exclamations that require
back-tracking, but it's not unheard of!

[Steven D'Aprano]

> > and there are plenty of examples of Python following
> > this pattern in other expressionized syntax like in comprehensions and
> > ternary expressions.
> But not binary operators which take only two arguments, in this case a
> name and an attribute name.


 Again, my proposal was never for a binary operator.

[Steven D'Aprano]

> > I actually think this makes sense for expressionized
> > syntax because it puts the "meat" before the "potatoes". When you're
> > reading code like:
> >
> > initials = [person.name[0] ...
> >
> > You can usually guess what goes in the `...` from context
> Can you?
> In that example I can't even guess if that's the start of a list
> comprehension or a list display. If its a comprehension, I sure as hell
> can't guess the important question of what it being iterated over.


Maybe I should have fleshed out my example a little more. Usually there are
plenty of context clues such that the latter part of a comprehension is
more a formality for the computer's sake because computers, unlike humans,
can't deal with ambiguity very well. So when you see:

def func(people):
    initials = [person.name[0] ...

You can usually guess that, since `person` isn't a local variable, we're
probably dealing with a generator so there's probably a `for` clause down
the line where `person` is declared and it probably iterates over, I don't
know, how about `people` and maybe there's a filter clause too. The point
is that the most important information, the "meat" of the expression is up
front.

In the same vein, In the spirit of EAFP, it's better to get the meat of
your business logic up front and do all the tedious-yet-necessary edge-case
checking later just like:

def func(person=None):
    initial = person.name[0] ?
    ...

The business logic comes first and the edge-case checking follows.

> so it's redundant, so it makes sense that it comes later.
> If it's redundant, why don't we just leave it out?


Because redundancy from the perspective of the reader doesn't imply
redundancy from the perspective of the computer. People can make sense of
ambiguous language in a way that computers cant. The sentence "time flies
like an arrow" has over 11 valid interpretations, yet few humans would
think "oh, you're commanding me to use a stop watch to measure some
behavior of common flying insects in the same way that I would use the same
instrument to measure the behavior of an arrow!". If I type:

def func(people):
    initials = [person.name[0] for person in people
    return Counter(initals).most_common(1)[0]

You, as a human, can probably guess with extremely high certainty that I
meant to put a ']' at the end of the second line. In that sense, it's
redundant to you, but not to the computer which doesn't know how to handle
that situation. Does that make sense?



On Sun, Jul 29, 2018 at 7:48 PM, Steven D'Aprano <steve at pearwood.info>
wrote:

> On Sun, Jul 29, 2018 at 12:08:46PM -0500, Abe Dillon wrote:
>
> > look-ahead and back-tracing is how people read punctuation like '?' and
> '!'
> > in natural language
>
> I really wish you would stop talking about "natural language" as if
> there were only one (and it were English). Many natural languages don't
> even have punctuation like '?', or if they do, they are a very recent
> innovation. Spanish uses a bracketing pair of marks:
>
>    ?Qu? Mr Fawlty?
>
> while many other European languages make do with just one at the end.
> And not all of them use the same symbol. (Greek uses the semi-colon,
> which must make reading C-like languages a remarkable experience.)
>
> But even in English, that isn't how people read questions in general,
> which is why we can still comprehend punctuation-less sentences
>
>    what's for dinner
>
>    where are you
>
>    who was the German professor in Lord Palmerston's quote about the
>    Schleswig-Holstein business
>
> as questions, despite the lack of a question mark. Questions really only
> need the question mark if they would otherwise be ambiguously either a
> statement or a question.
>
> You may be able to find examples of English questions which do require
> back-tracking, but that is generally considered to be a bad thing,
> increasing surprise (not a good thing unless you are attempting humour)
> and reducing the speed and ease of comprehension. See my earlier link
> about garden path sentences.
>
> In other words, reducing not increasing readability.
>
>
> > and there are plenty of examples of Python following
> > this pattern in other expressionized syntax like in comprehensions and
> > ternary expressions.
>
> But not binary operators which take only two arguments, in this case a
> name and an attribute name.
>
>
> > I actually think this makes sense for expressionized
> > syntax because it puts the "meat" before the "potatoes". When you're
> > reading code like:
> >
> > initials = [person.name[0] ...
> >
> > You can usually guess what goes in the `...` from context
>
> Can you?
>
> In that example I can't even guess if that's the start of a list
> comprehension or a list display. If its a comprehension, I sure as hell
> can't guess the important question of what it being iterated over.
>
>
> > so it's redundant, so it makes sense that it comes later.
>
> If it's redundant, why don't we just leave it out?
>
>
> --
> Steve
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180729/e5b8ad7a/attachment-0001.html>

From abedillon at gmail.com  Mon Jul 30 00:03:32 2018
From: abedillon at gmail.com (Abe Dillon)
Date: Sun, 29 Jul 2018 23:03:32 -0500
Subject: [Python-ideas] Proposed alternative postfix syntax [was PEP
 505: None-aware operators]
In-Reply-To: <CAKr=oZu23=pC5RdjFDRhj532hVpN9Bde74VQn6zSDCdmBp59Mw@mail.gmail.com>
References: <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
 <CAPTjJmp2tN1TBz96EfnLRDsdO==QJ0xtYbCVkTBJDQ9+qOp6nA@mail.gmail.com>
 <CAKr=oZsTsR8Mv7_i386eOrEzQWTjiwbONt+7C=6D9V94f56kaQ@mail.gmail.com>
 <20180729055442.GH22554@ando.pearwood.info>
 <CAPTjJmpV9i+WyJxhCf6bbh04uOrexspB+iv-ZYrSi5OOa1p38w@mail.gmail.com>
 <CAKr=oZu+Ke6Qc9j4mc3XNLySdjA0V5WXbDivHQZkWRMir6UxNw@mail.gmail.com>
 <20180729114507.GJ22554@ando.pearwood.info>
 <CAEbHw4bWitUXu4S2nmf=JKji8151US0yFMFRVm7KS0GYR+n+Gg@mail.gmail.com>
 <CAKr=oZspFj8q3OMjJxYG15ktoF0xykgddaVcyUApM_nsvJLdWQ@mail.gmail.com>
 <20180730004856.GL22554@ando.pearwood.info>
 <CAKr=oZu23=pC5RdjFDRhj532hVpN9Bde74VQn6zSDCdmBp59Mw@mail.gmail.com>
Message-ID: <CAKr=oZuzh+qD3M2hnHs0s0G=Pe5XgP9JWF6nr2MW50j5bjyr4g@mail.gmail.com>

Typo:
*You can usually guess that, since `person` isn't a local variable, we're
probably dealing with a generator *comprehension*

On Sun, Jul 29, 2018 at 11:00 PM, Abe Dillon <abedillon at gmail.com> wrote:

> [Steven D'Aprano]
>
>> really wish you would stop talking about "natural language" as if
>> there were only one (and it were English).
>
>
> I'm fine with that. I've just had someone jump down my throat before about
> being overly English-centric when talking about readability.
>
> [Steven D'Aprano]
>
>> even in English, that isn't how people read questions in general,
>> which is why we can still comprehend punctuation-less sentences
>
>
> Yes because you can make use of other context clues. So if I wrote:
>
> def f(person=None):
>     initial = person.name[0]?
>     ...
>
> You can anticipate the '?' at the end of the expression. Otherwise I would
> be clearly missing an important edge case.
> None checking is often done in very close proximity to some code that
> might yield a None, so this should rarely be a surprise unless the coder is
> writing the equivalent of a garden-path sentence.
>
> [Steven D'Aprano]
>
>> You may be able to find examples of English questions which do require
>> back-tracking
>
>
> It's easier, I'll admit, to find example's of exclamations that require
> back-tracking, but it's not unheard of!
>
> [Steven D'Aprano]
>
>> > and there are plenty of examples of Python following
>> > this pattern in other expressionized syntax like in comprehensions and
>> > ternary expressions.
>> But not binary operators which take only two arguments, in this case a
>> name and an attribute name.
>
>
>  Again, my proposal was never for a binary operator.
>
> [Steven D'Aprano]
>
>> > I actually think this makes sense for expressionized
>> > syntax because it puts the "meat" before the "potatoes". When you're
>> > reading code like:
>> >
>> > initials = [person.name[0] ...
>> >
>> > You can usually guess what goes in the `...` from context
>> Can you?
>> In that example I can't even guess if that's the start of a list
>> comprehension or a list display. If its a comprehension, I sure as hell
>> can't guess the important question of what it being iterated over.
>
>
> Maybe I should have fleshed out my example a little more. Usually there
> are plenty of context clues such that the latter part of a comprehension is
> more a formality for the computer's sake because computers, unlike humans,
> can't deal with ambiguity very well. So when you see:
>
> def func(people):
>     initials = [person.name[0] ...
>
> You can usually guess that, since `person` isn't a local variable, we're
> probably dealing with a generator so there's probably a `for` clause down
> the line where `person` is declared and it probably iterates over, I don't
> know, how about `people` and maybe there's a filter clause too. The point
> is that the most important information, the "meat" of the expression is up
> front.
>
> In the same vein, In the spirit of EAFP, it's better to get the meat of
> your business logic up front and do all the tedious-yet-necessary edge-case
> checking later just like:
>
> def func(person=None):
>     initial = person.name[0] ?
>     ...
>
> The business logic comes first and the edge-case checking follows.
>
> > so it's redundant, so it makes sense that it comes later.
>> If it's redundant, why don't we just leave it out?
>
>
> Because redundancy from the perspective of the reader doesn't imply
> redundancy from the perspective of the computer. People can make sense of
> ambiguous language in a way that computers cant. The sentence "time flies
> like an arrow" has over 11 valid interpretations, yet few humans would
> think "oh, you're commanding me to use a stop watch to measure some
> behavior of common flying insects in the same way that I would use the same
> instrument to measure the behavior of an arrow!". If I type:
>
> def func(people):
>     initials = [person.name[0] for person in people
>     return Counter(initals).most_common(1)[0]
>
> You, as a human, can probably guess with extremely high certainty that I
> meant to put a ']' at the end of the second line. In that sense, it's
> redundant to you, but not to the computer which doesn't know how to handle
> that situation. Does that make sense?
>
>
>
> On Sun, Jul 29, 2018 at 7:48 PM, Steven D'Aprano <steve at pearwood.info>
> wrote:
>
>> On Sun, Jul 29, 2018 at 12:08:46PM -0500, Abe Dillon wrote:
>>
>> > look-ahead and back-tracing is how people read punctuation like '?' and
>> '!'
>> > in natural language
>>
>> I really wish you would stop talking about "natural language" as if
>> there were only one (and it were English). Many natural languages don't
>> even have punctuation like '?', or if they do, they are a very recent
>> innovation. Spanish uses a bracketing pair of marks:
>>
>>    ?Qu? Mr Fawlty?
>>
>> while many other European languages make do with just one at the end.
>> And not all of them use the same symbol. (Greek uses the semi-colon,
>> which must make reading C-like languages a remarkable experience.)
>>
>> But even in English, that isn't how people read questions in general,
>> which is why we can still comprehend punctuation-less sentences
>>
>>    what's for dinner
>>
>>    where are you
>>
>>    who was the German professor in Lord Palmerston's quote about the
>>    Schleswig-Holstein business
>>
>> as questions, despite the lack of a question mark. Questions really only
>> need the question mark if they would otherwise be ambiguously either a
>> statement or a question.
>>
>> You may be able to find examples of English questions which do require
>> back-tracking, but that is generally considered to be a bad thing,
>> increasing surprise (not a good thing unless you are attempting humour)
>> and reducing the speed and ease of comprehension. See my earlier link
>> about garden path sentences.
>>
>> In other words, reducing not increasing readability.
>>
>>
>> > and there are plenty of examples of Python following
>> > this pattern in other expressionized syntax like in comprehensions and
>> > ternary expressions.
>>
>> But not binary operators which take only two arguments, in this case a
>> name and an attribute name.
>>
>>
>> > I actually think this makes sense for expressionized
>> > syntax because it puts the "meat" before the "potatoes". When you're
>> > reading code like:
>> >
>> > initials = [person.name[0] ...
>> >
>> > You can usually guess what goes in the `...` from context
>>
>> Can you?
>>
>> In that example I can't even guess if that's the start of a list
>> comprehension or a list display. If its a comprehension, I sure as hell
>> can't guess the important question of what it being iterated over.
>>
>>
>> > so it's redundant, so it makes sense that it comes later.
>>
>> If it's redundant, why don't we just leave it out?
>>
>>
>> --
>> Steve
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180729/e420cfbd/attachment-0001.html>

From encukou at gmail.com  Mon Jul 30 09:43:29 2018
From: encukou at gmail.com (Petr Viktorin)
Date: Mon, 30 Jul 2018 15:43:29 +0200
Subject: [Python-ideas] Idea: msgfmt.py and pygettext..py should be -m
 executable modules
In-Reply-To: <f047b0cb-fc29-d3b5-b68a-18bb2cef1fc8@redhat.com>
References: <f047b0cb-fc29-d3b5-b68a-18bb2cef1fc8@redhat.com>
Message-ID: <CA+=+wqCQd06s_p0M4ZE_3YPpM3UaLQQK_jH0SvkGpuE8uD9chQ@mail.gmail.com>

On Mon, Jul 23, 2018 at 2:16 PM, Miro Hron?ok <mhroncok at redhat.com> wrote:
> Currently, the documentation (as well as plenty of other places on the
> Internet, such as Stack Overflow answers) mentions msgfmt.py and
> pygettext.py.
>
> See https://docs.python.org/3/library/gettext.html
>
>> (Python also includes pure-Python versions of these programs, called
>> pygettext.py and msgfmt.py; some Python distributions will install
>> them for you. pygettext.py is similar to xgettext, but only
>> understands Python source code and cannot handle other programming
>> languages such as C or C++. pygettext.py supports a command-line
>> interface similar to xgettext; for details on its use, run
>> pygettext.py --help. msgfmt.py is binary compatible with GNU msgfmt.
>> With these two programs, you may not need the GNU gettext package to
>> internationalize your Python applications.)
>
> The problematic part is: "some Python distributions will install them for
> you"
>
> As a Python distributor, I may ask myself a question: Where do I install
> those?
>
> As a Python user, I may ask: Where did the distributors put them, and did
> they?
>
> So I suggest we make those modules instead and we document how to use them.

Not completely "instead" perhaps; the original Tools/i18n/*.py scripts
can be changed to just import the new module and call it.

> It might be:
>
>   $ python3 -m gettext

+1

> And:
>
>   $ python3 -m gettext.msgfmt

+1
Note that this means gettext will need to become a package.

> And (provided as a shortcut):
>
>   $ python3 -m msgfmt

-1. This would mean adding a new top-level module to the standard
library. Let's not pollute that namespace just to provide a shortcut.

>
> I feel that using -m is the general direction Python is going, considering
> this message:
>
>   $ pyvenv ...
>   WARNING: the pyenv script is deprecated in favour of `python3.7 -m venv`
>
> It also gives the user a choice for Python version on a system with multiple
> Python versions installed as the user could do:
>
>   $ python3.6 -m gettext
>
> Or
>
>   $ python3.7 -m gettext
>
> While with e.g. /usr/bin/pygettext.py this adds unnecessary burden on the
> distributor to package /usr/bin/pygettext-3.7.py etc...

Yes ? pygettext depends on the Python version, so being able to do
this sounds very useful. Less so for msgfmt of course.

I'm adding Barry Warsaw, who originally wrote pygettext and AFAIK
still does i18n in Python.
Barry, does this sound reasonable to you?

From rhodri at kynesim.co.uk  Mon Jul 30 09:54:04 2018
From: rhodri at kynesim.co.uk (Rhodri James)
Date: Mon, 30 Jul 2018 14:54:04 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
References: <CAPTjJmqoVCOfC+Hj041ON2kg1LPF9PJM8b+VHHbo0gsAvGJL7A@mail.gmail.com>
 <20180723165256.GB8744@ando.pearwood.info>
 <CAFYqXL8KJ2BD_YxVRoGY-wkb9jdOBAf5++YYfwBQoLD+2yRYOQ@mail.gmail.com>
 <20180724094917.GH8744@ando.pearwood.info>
 <CAEbHw4ZgPLUME1EwHR0heXaovtegeq5arqK+1sn=eN7ROv0R=w@mail.gmail.com>
 <20180725174041.GK8744@ando.pearwood.info>
 <CAEbHw4ZOVZ8KbnAuPxXy8xnrcNT3YfaqzA6rY4yZXHAg2z21ew@mail.gmail.com>
 <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAEbHw4Yzt1Wfr1iBw+mmTYWhh8b2SwyFLX1AHOL-4YP6H+_ChA@mail.gmail.com>
 <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>
 <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
 <CAKr=oZtfYM2_fxYnde76o+jvuc_Y3AH9R93BCY6BoNdriuKqKw@mail.gmail.com>
 <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
Message-ID: <56872da5-afba-8108-848c-38a5033485d8@kynesim.co.uk>

On 28/07/18 21:07, Abe Dillon wrote:
[>> = Me]
>> We could argue about how intuitive or not these operators are, but they
>> are used in other languages, so they clearly aren't completely unintuitive.
> 
> Other languages are other languages. Other languages use the "<condition> ?
> <expr_a> : <expr_b>" form of the the ternary operator. That doesn't mean we
> should adopt that into Python.

But it does and did mean that we should consider it when coming up with 
Python's version of the ternary operator.  *It isn't unintuitive* and 
neither is ??  None-aware operators may be bad for other reasons, but 
that argument doesn't wash.

>> Really?  Quick straw poll: how often do people care how in-place operators
>> are implemented?
> 
> The point I was trying to make is that it doesn't behave or compose like
> other expressions.

(This was about ??=, for context.)  Pardon?  It composes exactly like 
any other in-place operator.

[Abe previously wrote:]
 >>> val = person[0]?
 >>> val = person.name?
 >>
>> Um.  How is that supposed to parse?  "person[0]" has already been evaluated
>> by the time the "?" operator gets anywhere near it.
> 
> It parses by placing the '?' operator high in the order of operations; like
> I said. Do I really need to explain how order of operations works?

Apology elsewhere accepted, but yes, you do need to explain how merely 
being high precedence helps here.  Let's start small; please explain 
programmatically what

   name?

does as an expression.  Once we have that down, we can move on to

   person.(name?)

(parentheses for emphasis.)  Short of actual magic, I can't see how this 
is going to work.

-- 
Rhodri James *-* Kynesim Ltd

From rhodri at kynesim.co.uk  Mon Jul 30 10:10:28 2018
From: rhodri at kynesim.co.uk (Rhodri James)
Date: Mon, 30 Jul 2018 15:10:28 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAKr=oZv1Nibq=G+p442aF9Ce874HvLb60D18A3qMWoBJC41HDw@mail.gmail.com>
References: <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAEbHw4Yzt1Wfr1iBw+mmTYWhh8b2SwyFLX1AHOL-4YP6H+_ChA@mail.gmail.com>
 <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>
 <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
 <CAKr=oZtfYM2_fxYnde76o+jvuc_Y3AH9R93BCY6BoNdriuKqKw@mail.gmail.com>
 <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
 <CAPTjJmp2tN1TBz96EfnLRDsdO==QJ0xtYbCVkTBJDQ9+qOp6nA@mail.gmail.com>
 <CAKr=oZsTsR8Mv7_i386eOrEzQWTjiwbONt+7C=6D9V94f56kaQ@mail.gmail.com>
 <20180729055442.GH22554@ando.pearwood.info>
 <CAEbHw4Y0YFC0ttZuu3PDSjUJagC4uQ2XaOFC+6GNasG-OE+Gsw@mail.gmail.com>
 <CAPTjJmpbj9=xYff14SQdnb5O2_Ttj70JUd_aWpBx4ud4TXXnXQ@mail.gmail.com>
 <CAKr=oZv1Nibq=G+p442aF9Ce874HvLb60D18A3qMWoBJC41HDw@mail.gmail.com>
Message-ID: <e86b9886-4a15-d6f0-1c17-88196f422382@kynesim.co.uk>

On 29/07/18 16:12, Abe Dillon wrote:
> spam?.eggs.cheese.aardvark  # why would you ever do this?

If you knew that if you really have something in "spam", your program 
guarantees it will have an "eggs" attribute with a "cheese" attribute, 
etc, you just don't know if "spam" is not None.  It's the same insider 
knowledge you would use if you wrote it out as

spam.eggs.cheese.aardvark if spam is not None else None

The same sort of knowledge of your program structure could lead to you 
to use "?." instead of "." in any place or places in the chain.  If your 
program gives you strong enough guarantees, it is the sort of thing you 
can use to reduce the code's workload.

By way of example, I was working on some C code last night that was 
expanding an internal buffer, possibly from non-existence.  It did lot 
of taking the difference between various pointers to determine how big 
the new buffer needed to be, how much to copy from the old buffer, etc. 
It looked rather hairy until you realised that the code gave a strong 
guarantee that either all the pointers were meaningful or they were all 
NULL, so the pointer differences always made sense.  I rewrote that code 
so that it didn't take differences of NULL pointers since that was what 
the bug specified, but honestly it looks lumpier and less clear now.  A 
big comment explaining what was going on would probably be better.

-- 
Rhodri James *-* Kynesim Ltd

From jpic at yourlabs.org  Mon Jul 30 11:15:31 2018
From: jpic at yourlabs.org (Jamesie Pic)
Date: Mon, 30 Jul 2018 17:15:31 +0200
Subject: [Python-ideas] Redefining method
Message-ID: <CAC6Op19Z8oh-n=02qEcOH-uJ-2x89szx7Bk6oSa5YMpeuQGbpg@mail.gmail.com>

Hi all,

Not sure if this subject has been brought up already, but why can't we
redefine methods as such for example:

c = MyClass
o = c()

def c.foo(cls): ...
def o.bar(self): ...

Thanks in advance for sharing some of your insight

-- 
?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180730/b9575f80/attachment.html>

From prometheus235 at gmail.com  Mon Jul 30 11:22:39 2018
From: prometheus235 at gmail.com (Nick Timkovich)
Date: Mon, 30 Jul 2018 10:22:39 -0500
Subject: [Python-ideas] Idea: msgfmt.py and pygettext..py should be -m
 executable modules
In-Reply-To: <CA+=+wqCQd06s_p0M4ZE_3YPpM3UaLQQK_jH0SvkGpuE8uD9chQ@mail.gmail.com>
References: <f047b0cb-fc29-d3b5-b68a-18bb2cef1fc8@redhat.com>
 <CA+=+wqCQd06s_p0M4ZE_3YPpM3UaLQQK_jH0SvkGpuE8uD9chQ@mail.gmail.com>
Message-ID: <CAHkxive1-0UDHDQ4bnOVt=eFG3TDNmV2YajnZeRkJ2hRAA-K-g@mail.gmail.com>

On Mon, Jul 30, 2018 at 8:43 AM, Petr Viktorin <encukou at gmail.com> wrote:

> On Mon, Jul 23, 2018 at 2:16 PM, Miro Hron?ok <mhroncok at redhat.com> wrote:
>
> > It might be:
> >
> >   $ python3 -m gettext
>
> +1
>
> > And:
> >
> >   $ python3 -m gettext.msgfmt
>
> +1
> Note that this means gettext will need to become a package.
>
> > And (provided as a shortcut):
> >
> >   $ python3 -m msgfmt
>
> -1. This would mean adding a new top-level module to the standard
> library. Let's not pollute that namespace just to provide a shortcut.


Speaking of shortcuts, is there a reason that it should be "python -m
json.tool" versus "python -m json" (keep the old probably forever, but have
the shorter as a synonym)?

And as a broader question, is there opposition to generating patches to
make packages executable that may be useful as a CLI? Would be nice if the
UUID or random modules could directly spit something out, e.g. "python -m
uuid -4". Counterargument that you could "python -c "import
uuid;print(uuid.uuid4())".

Nick
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180730/afbd9d1b/attachment-0001.html>

From jfine2358 at gmail.com  Mon Jul 30 11:35:57 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Mon, 30 Jul 2018 16:35:57 +0100
Subject: [Python-ideas] Redefining method
In-Reply-To: <CAC6Op19Z8oh-n=02qEcOH-uJ-2x89szx7Bk6oSa5YMpeuQGbpg@mail.gmail.com>
References: <CAC6Op19Z8oh-n=02qEcOH-uJ-2x89szx7Bk6oSa5YMpeuQGbpg@mail.gmail.com>
Message-ID: <CALD=Yf_Rx9yyazBWLG_9tyzSY5_PmUOtEsoFNc9TX66y3eQwDQ@mail.gmail.com>

Hi Jamesie

Thank you for your question. You asked why not
> c = MyClass
> o = c()
>
> def c.foo(cls): ...
> def o.bar(self): ...

I've the same same query, but never had the courage to ask. So that
you for asking. And also giving me a chance to share my thoughts.

In Ruby, I believe, you can 'open up' an existing class, and add new
methods to it. You can even do with core classes. I think what you
asking might amount to a wish to do the same in Python.

I think the Python way to 'open up' an existing class is to create a
subclass, and add methods to that. Python design and development is
quite pragmatic. So instead of us saying why it's not there, why don't
you tell why you think it might be good idea. Based of course on
examples. Particularly code taken from successful real-world
applications.

By the way, your
>>> def o.var(self): ...
won't work as you expect. To get a bound method, the method must be an
attribute of the class (in other words type(o)) of the object o.
That's the way Python is. What you've written does work in JavaScript.

It's perhaps a bit technical and short of examples, but
https://docs.python.org/3/reference/datamodel.html would be a good
starting point. Can anyone else recommend some other URLs for this.

Once again, thank you for asking this question. I look forward to
reading other answers (and of course your response).

-- 
Jonathan

From ncoghlan at gmail.com  Mon Jul 30 12:10:12 2018
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 31 Jul 2018 02:10:12 +1000
Subject: [Python-ideas] Redefining method
In-Reply-To: <CALD=Yf_Rx9yyazBWLG_9tyzSY5_PmUOtEsoFNc9TX66y3eQwDQ@mail.gmail.com>
References: <CAC6Op19Z8oh-n=02qEcOH-uJ-2x89szx7Bk6oSa5YMpeuQGbpg@mail.gmail.com>
 <CALD=Yf_Rx9yyazBWLG_9tyzSY5_PmUOtEsoFNc9TX66y3eQwDQ@mail.gmail.com>
Message-ID: <CADiSq7eCrtp4AongeW=NAGZHjbf9bXh-R7iCghH-bjqM2ouqHg@mail.gmail.com>

On 31 July 2018 at 01:35, Jonathan Fine <jfine2358 at gmail.com> wrote:
> Hi Jamesie
>
> Thank you for your question. You asked why not
>> c = MyClass
>> o = c()
>>
>> def c.foo(cls): ...
>> def o.bar(self): ...
>
> I've the same same query, but never had the courage to ask. So that
> you for asking. And also giving me a chance to share my thoughts.

It's essentially due to the fact that while we deliberately allow
runtime monkeypatching (as it's sometimes the best available answer),
we also strongly encourage the idea of treating it as a last resort
option (outside specific use cases like testing).

So if you want to define methods on a class, the strongly preferred
place to define them is inside the class definition, where they're
easy for future maintainers of that class to find.

If you need to replace them for some reason, it will preferably be
within a temporary bounded scope, using a tool like
unittest.mock.patch, rather than as a permanent change that affects
every other use of the class within the process.

Cheers,
Nick.

P.S. While it's *not* explicitly part of Python's design rationale,
http://connascence.io/locality.html and the rest of that site provide
some good info on the kinds of problems that "action at a distance"
effects, like monkeypatching class definitions, can cause in a code
base.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From robertvandeneynde at hotmail.com  Mon Jul 30 12:11:24 2018
From: robertvandeneynde at hotmail.com (Robert Vanden Eynde)
Date: Mon, 30 Jul 2018 16:11:24 +0000
Subject: [Python-ideas] Idea: msgfmt.py and pygettext..py should be -m
 executable modules
In-Reply-To: <CAHkxive1-0UDHDQ4bnOVt=eFG3TDNmV2YajnZeRkJ2hRAA-K-g@mail.gmail.com>
References: <f047b0cb-fc29-d3b5-b68a-18bb2cef1fc8@redhat.com>
 <CA+=+wqCQd06s_p0M4ZE_3YPpM3UaLQQK_jH0SvkGpuE8uD9chQ@mail.gmail.com>
 <CAHkxive1-0UDHDQ4bnOVt=eFG3TDNmV2YajnZeRkJ2hRAA-K-g@mail.gmail.com>
Message-ID: <DB7PR03MB4379D8993429DEF91076882DB62F0@DB7PR03MB4379.eurprd03.prod.outlook.com>

Shortcuts designed for CLI are just to "be more mnemonic" have to be considered with caution.
If gettext is a package, it means the whole python community shoud agree on that.
msgfmt is part of gettext, so yes, python -m gettest.msgfmt is the best long lasting command. Or it could be 'python -m gettest msgfmt'.
If you're using CLI, 90% of chances you know that if you don't like "python -m gettest.msgfmt" because it's too long, you can add a `alias dotrans='python -m gettest.msgfmt'` in your .bashrc so that you can simply write "dotrans Something".
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180730/c56c1b0c/attachment.html>

From robertvandeneynde at hotmail.com  Mon Jul 30 12:16:34 2018
From: robertvandeneynde at hotmail.com (Robert Vanden Eynde)
Date: Mon, 30 Jul 2018 16:16:34 +0000
Subject: [Python-ideas] Idea: msgfmt.py and pygettext..py should be -m
 executable modules
In-Reply-To: <CAHkxive1-0UDHDQ4bnOVt=eFG3TDNmV2YajnZeRkJ2hRAA-K-g@mail.gmail.com>
References: <f047b0cb-fc29-d3b5-b68a-18bb2cef1fc8@redhat.com>
 <CA+=+wqCQd06s_p0M4ZE_3YPpM3UaLQQK_jH0SvkGpuE8uD9chQ@mail.gmail.com>
 <CAHkxive1-0UDHDQ4bnOVt=eFG3TDNmV2YajnZeRkJ2hRAA-K-g@mail.gmail.com>
Message-ID: <DB7PR03MB4379F50D2E6EC7F4806FC9F2B62F0@DB7PR03MB4379.eurprd03.prod.outlook.com>

Shortcuts designed for CLI are just to "be more mnemonic" have to be considered with caution.
If gettext is a package, it means the whole python community shoud agree on that.
msgfmt is part of gettext, so yes, python -m gettest.msgfmt is the best long lasting command. Or it could be 'python -m gettest msgfmt'.
If you're using CLI, 90% of chances you know that if you don't like "python -m gettest.msgfmt" because it's too long, you can add a `alias dotrans='python -m gettest.msgfmt'` in your .bashrc so that you can simply write "dotrans Something".
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180730/951846a8/attachment.html>

From abedillon at gmail.com  Mon Jul 30 13:41:20 2018
From: abedillon at gmail.com (Abe Dillon)
Date: Mon, 30 Jul 2018 12:41:20 -0500
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <e86b9886-4a15-d6f0-1c17-88196f422382@kynesim.co.uk>
References: <CAPTjJmqBGjrVJC62VJwkCzKi802+CxCijXN5Xec7AQ5gV9O9Vw@mail.gmail.com>
 <CAOhmDzerq3EkeOBP7kO3T-+ewF1XxAVzSXhibF4-WcoFsCWcyw@mail.gmail.com>
 <CAEbHw4Yzt1Wfr1iBw+mmTYWhh8b2SwyFLX1AHOL-4YP6H+_ChA@mail.gmail.com>
 <CAOhmDzd70HMN7J2grJNwUOR+MA1h3ytVXNMPWg2Cd-xyPF8kSg@mail.gmail.com>
 <CAEbHw4Y+BEXrVhH6QkddQn1mwQpaEVwUA6e1smHC9G0Et_wEcQ@mail.gmail.com>
 <CAKr=oZtfYM2_fxYnde76o+jvuc_Y3AH9R93BCY6BoNdriuKqKw@mail.gmail.com>
 <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
 <CAPTjJmp2tN1TBz96EfnLRDsdO==QJ0xtYbCVkTBJDQ9+qOp6nA@mail.gmail.com>
 <CAKr=oZsTsR8Mv7_i386eOrEzQWTjiwbONt+7C=6D9V94f56kaQ@mail.gmail.com>
 <20180729055442.GH22554@ando.pearwood.info>
 <CAEbHw4Y0YFC0ttZuu3PDSjUJagC4uQ2XaOFC+6GNasG-OE+Gsw@mail.gmail.com>
 <CAPTjJmpbj9=xYff14SQdnb5O2_Ttj70JUd_aWpBx4ud4TXXnXQ@mail.gmail.com>
 <CAKr=oZv1Nibq=G+p442aF9Ce874HvLb60D18A3qMWoBJC41HDw@mail.gmail.com>
 <e86b9886-4a15-d6f0-1c17-88196f422382@kynesim.co.uk>
Message-ID: <CAKr=oZuwNR2gim7znHuJtoW0bof5+jL3+dprKDREiU=Q_OxEXA@mail.gmail.com>

 [Rhodri James]

> We could argue about how intuitive or not these operators are, but they
>>> are used in other languages, so they clearly aren't completely
>>> unintuitive.
>>>
>>
>> Other languages are other languages. Other languages use the "<condition>
>> ?
>> <expr_a> : <expr_b>" form of the the ternary operator. That doesn't mean
>> we
>> should adopt that into Python.
>>
>
> But it does and did mean that we should consider it when coming up with
> Python's version of the ternary operator.  *It isn't unintuitive* and
> neither is ??  None-aware operators may be bad for other reasons, but that
> argument doesn't wash.


No. The fact that other languages use it doesn't mean that "it isn't
unintuitive". The fact that other languages use it doesn't mean that it
doesn't harm readability. The fact that other languages use it has pretty
much no bearing on any of that. What it actually means is: other languages
are trying it out so we have the luxury of seeing if they regret it in the
future or if it becomes a popular and well loved feature before integrating
it into Python.

I believe it's counter-intuitive because '?' always comes at the end of an
english sentence, so it reads like other clause-ending operators like the
")" on a function call or the "]" at the end of a index operation, then the
"." reads like a follow-on.

And no; you can't say the same thing about "." always coming at the end of
a sentence because:

a) that's not true (e.g. i.e, e.g., etc.)

b) "." attribute access has many decades of precedence

c) a "." is less visually obtrusive than a "?".

[Rhodri James]

>  Pardon?  It composes exactly like any other in-place operator.


Yes. I'll concede. I was wrong about that.

[Rhodri James]

> yes, you do need to explain how merely being high precedence helps here.
> Let's start small; please explain programmatically what


>   name?



does as an expression.  Once we have that down, we can move on to



  person.(name?)


Sure. First, I would suggest we define a new class
NoneException(BaseException), and derive new Exception multi-classes like:
class NoneAttributeError(AttributeError, NoneException).
Then I would make those the exceptions thrown by attribute errors on the
None object. That way you can conceptualize the '?' operator is to the
try-except statement as the ternary operator is to if-else statements
EXCEPT that the "except" part of the try-except block is implicitly filled
in with "except NoneException: return None" so in general:

<expression>?

Would parse to:

try:
    result = eval(expression)
except NoneException:
    result = None

Similar to how:

<expression_1> if <condition_expression> else <expression_2>

Parses to:

if eval(condition_expression):
    result = eval(expression_1)
else:
    result = eval(expression_2)

More specifically:

name?

Wouldn't do much:

try:
    result = name
except NoneException:
    result = None

And:

person.(name?)

would throw a SyntaxError because person.( is not valid syntax regardless
of my proposal.
Perhaps you meant something like:

(person.name).first?

In that case; person.name would evaluate to some result or throw an
un-guarded AttributeError, in the former case the <expression> to the left
of the '?' is an attribute access on the result of (person.name). In the
latter case; an AttributeError is thrown.

That seems pretty straight forward. No?

On 29/07/18 16:12, Abe Dillon wrote:

>  > spam?.eggs.cheese.aardvark  # why would you ever do this?



> If you knew that if you really have something in "spam", your program
> guarantees it will have an "eggs" attribute with a "cheese" attribute, etc,
> you just don't know if "spam" is not None.  It's the same insider knowledge
> you would use if you wrote it out as




> spam.eggs.cheese.aardvark if spam is not None else None


That's not the equivalent as described in PEP 505. If "spam" is None,
"spam?.eggs" evaluates to None without throwing an AttributeError, but the
rest of the expression will throw an AttributeError from trying to access
"None.cheese".

On Mon, Jul 30, 2018 at 9:10 AM, Rhodri James <rhodri at kynesim.co.uk> wrote:

> On 29/07/18 16:12, Abe Dillon wrote:
>
>> spam?.eggs.cheese.aardvark  # why would you ever do this?
>>
>
> If you knew that if you really have something in "spam", your program
> guarantees it will have an "eggs" attribute with a "cheese" attribute, etc,
> you just don't know if "spam" is not None.  It's the same insider knowledge
> you would use if you wrote it out as
>
> spam.eggs.cheese.aardvark if spam is not None else None
>
> The same sort of knowledge of your program structure could lead to you to
> use "?." instead of "." in any place or places in the chain.  If your
> program gives you strong enough guarantees, it is the sort of thing you can
> use to reduce the code's workload.
>
> By way of example, I was working on some C code last night that was
> expanding an internal buffer, possibly from non-existence.  It did lot of
> taking the difference between various pointers to determine how big the new
> buffer needed to be, how much to copy from the old buffer, etc. It looked
> rather hairy until you realised that the code gave a strong guarantee that
> either all the pointers were meaningful or they were all NULL, so the
> pointer differences always made sense.  I rewrote that code so that it
> didn't take differences of NULL pointers since that was what the bug
> specified, but honestly it looks lumpier and less clear now.  A big comment
> explaining what was going on would probably be better.
>
> --
> Rhodri James *-* Kynesim Ltd
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180730/897485b7/attachment-0001.html>

From chris.barker at noaa.gov  Mon Jul 30 14:12:54 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Mon, 30 Jul 2018 11:12:54 -0700
Subject: [Python-ideas] Redefining method
In-Reply-To: <CADiSq7eCrtp4AongeW=NAGZHjbf9bXh-R7iCghH-bjqM2ouqHg@mail.gmail.com>
References: <CAC6Op19Z8oh-n=02qEcOH-uJ-2x89szx7Bk6oSa5YMpeuQGbpg@mail.gmail.com>
 <CALD=Yf_Rx9yyazBWLG_9tyzSY5_PmUOtEsoFNc9TX66y3eQwDQ@mail.gmail.com>
 <CADiSq7eCrtp4AongeW=NAGZHjbf9bXh-R7iCghH-bjqM2ouqHg@mail.gmail.com>
Message-ID: <CALGmxELfDcLOHC657bRVcMO=QtMo4+_5ifDZFW287vD0Vzot6g@mail.gmail.com>

On Mon, Jul 30, 2018 at 9:10 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

If you need to replace them for some reason, it will preferably be
> within a temporary bounded scope, using a tool like
> unittest.mock.patch, rather than as a permanent change that affects
> every other use of the class within the process.


 yup -- but you can still do a raw monkey-patch anyway. You asked for:

> Thank you for your question. You asked why not
> >> c = MyClass
> >> o = c()
> >>
> >> def c.foo(cls): ...
>

do you mean this? you want a classmethod? or is this what you mean? --
which you can do:

C = MyClass

def foo(self, some_param):
    some_code()

C.foo = foo

(note that I used a capital C -- capitalized names are traditional for
classes -- see PEP 8)

In that case, any existing, and new instances of the C class will now have
the foo method.

Also -- that line: C = MyClass -- binds the name "C" to the very same class
object as MyClass -- so you will have just monkey=patched MyClass -- I"m
guessing you didn't want that. If not, and you wanted C t be a copy of
MyClass that you could then change/add/remove methods from, then you want
subclassing -- that is exactly what it is for.

Then there is this:


> >> def o.bar(self): ...
>

which is monkey patching an instance object, which you can also do, but you
don't get a bound method -- i.e. it doesn't get self passed in
automatically.

-CHB











> > I've the same same query, but never had the courage to ask. So that
> > you for asking. And also giving me a chance to share my thoughts.
>
> It's essentially due to the fact that while we deliberately allow
> runtime monkeypatching (as it's sometimes the best available answer),
> we also strongly encourage the idea of treating it as a last resort
> option (outside specific use cases like testing).
>
> So if you want to define methods on a class, the strongly preferred
> place to define them is inside the class definition, where they're
> easy for future maintainers of that class to find.
>
>
> Cheers,
> Nick.
>
> P.S. While it's *not* explicitly part of Python's design rationale,
> http://connascence.io/locality.html and the rest of that site provide
> some good info on the kinds of problems that "action at a distance"
> effects, like monkeypatching class definitions, can cause in a code
> base.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180730/5ff82107/attachment.html>

From rudy at matela.com.br  Mon Jul 30 15:15:17 2018
From: rudy at matela.com.br (Rudy Matela)
Date: Mon, 30 Jul 2018 16:15:17 -0300
Subject: [Python-ideas] Is this PEP-able? "with" statement inside genexps /
 list comprehensions
Message-ID: <20180730191517.GA27867@zero.localdomain>

Hello,

Do you think it would be nice to allow with statements inside genexps or
list comprehensions?  The functions __enter__ and __exit__ would be
automatically called as iterables are traversed.  I am thinking of
drafting a PEP about this.  Examples:


This 

	g = (f.read() for fn in filenames with open(fn) as f)

would be equivalent to the following use of a generator function:

	def __gen():
		for fn in filenames:
			with open(fn) as f:
				yield f.read()
	g = __gen()


This

	list = [f.read() for fn in filenames with open(fn) as f]

would be equivalent to the following:

	list = []
	for fn in filenames:
		with open(fn) as f:
			list.append(f.read())

--
Rudy

From klahnakoski at mozilla.com  Mon Jul 30 15:29:54 2018
From: klahnakoski at mozilla.com (Kyle Lahnakoski)
Date: Mon, 30 Jul 2018 15:29:54 -0400
Subject: [Python-ideas] Redefining method
In-Reply-To: <CALGmxELfDcLOHC657bRVcMO=QtMo4+_5ifDZFW287vD0Vzot6g@mail.gmail.com>
References: <CAC6Op19Z8oh-n=02qEcOH-uJ-2x89szx7Bk6oSa5YMpeuQGbpg@mail.gmail.com>
 <CALD=Yf_Rx9yyazBWLG_9tyzSY5_PmUOtEsoFNc9TX66y3eQwDQ@mail.gmail.com>
 <CADiSq7eCrtp4AongeW=NAGZHjbf9bXh-R7iCghH-bjqM2ouqHg@mail.gmail.com>
 <CALGmxELfDcLOHC657bRVcMO=QtMo4+_5ifDZFW287vD0Vzot6g@mail.gmail.com>
Message-ID: <43511866-6043-287a-33d0-9b33c6209fdb@mozilla.com>


I think C# calls methods that are added to a class "extension methods".
You can make a decorator that will do that for you:

> def extend(cls):
> ??? """
> ??? DECORATOR TO ADD METHODS TO CLASSES
> ??? :param cls: THE CLASS TO ADD THE METHOD TO
> ??? :return:
> ??? """
> ??? def extender(func):
> ??????? setattr(cls, get_function_name(func), func)
> ??????? return func
> ??? return extender

and use it

> C = MyClass
> @extend(C)
> def foo(self, schesome_param):
> ? ? some_code()



On 2018-07-30 14:12, Chris Barker via Python-ideas wrote:
> On Mon, Jul 30, 2018 at 9:10 AM, Nick Coghlan <ncoghlan at gmail.com
> <mailto:ncoghlan at gmail.com>> wrote:
>
>     If you need to replace them for some reason, it will preferably be
>     within a temporary bounded scope, using a tool like
>     unittest.mock.patch, rather than as a permanent change that affects
>     every other use of the class within the process.
>
>
> ?yup -- but you can still do a raw monkey-patch anyway. You asked for:
>
>     > Thank you for your question. You asked why not
>     >> c = MyClass
>     >> o = c()
>     >>
>     >> def c.foo(cls): ...
>
>
> do you mean this? you want a classmethod? or is this what you mean? --
> which you can do:
>
> C = MyClass
>
> def foo(self, some_param):
> ? ? some_code()
>
> C.foo = foo
>
> (note that I used a capital C -- capitalized names are traditional for
> classes -- see PEP 8)
>
> In that case, any existing, and new instances of the C class will now
> have the foo method.
>
> Also -- that line: C = MyClass -- binds the name "C" to the very same
> class object as MyClass -- so you will have just monkey=patched
> MyClass -- I"m guessing you didn't want that. If not, and you wanted C
> t be a copy of MyClass that you could then change/add/remove methods
> from, then you want subclassing -- that is exactly what it is for.
>
> Then there is this:
> ?
>
>     >> def o.bar(self): ...
>
>
> which is monkey patching an instance object, which you can also do,
> but you don't get a bound method -- i.e. it doesn't get self passed in
> automatically.
>
> -CHB
>
>
>
>
>
>
>
>
>
> ?
>
>     > I've the same same query, but never had the courage to ask. So that
>     > you for asking. And also giving me a chance to share my thoughts.
>
>     It's essentially due to the fact that while we deliberately allow
>     runtime monkeypatching (as it's sometimes the best available answer),
>     we also strongly encourage the idea of treating it as a last resort
>     option (outside specific use cases like testing).
>
>     So if you want to define methods on a class, the strongly preferred
>     place to define them is inside the class definition, where they're
>     easy for future maintainers of that class to find.
>
>
>     Cheers,
>     Nick.
>
>     P.S. While it's *not* explicitly part of Python's design rationale,
>     http://connascence.io/locality.html
>     <http://connascence.io/locality.html> and the rest of that site
>     provide
>     some good info on the kinds of problems that "action at a distance"
>     effects, like monkeypatching class definitions, can cause in a code
>     base.
>
>     -- 
>     Nick Coghlan? ?|? ?ncoghlan at gmail.com <mailto:ncoghlan at gmail.com>?
>     ?|? ?Brisbane, Australia
>     _______________________________________________
>     Python-ideas mailing list
>     Python-ideas at python.org <mailto:Python-ideas at python.org>
>     https://mail.python.org/mailman/listinfo/python-ideas
>     <https://mail.python.org/mailman/listinfo/python-ideas>
>     Code of Conduct: http://python.org/psf/codeofconduct/
>     <http://python.org/psf/codeofconduct/>
>
>
>
>
> -- 
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R ? ? ? ? ? ?(206) 526-6959?? voice
> 7600 Sand Point Way NE ??(206) 526-6329?? fax
> Seattle, WA ?98115 ? ? ??(206) 526-6317?? main reception
>
> Chris.Barker at noaa.gov <mailto:Chris.Barker at noaa.gov>
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180730/1e4619f5/attachment-0001.html>

From klahnakoski at mozilla.com  Mon Jul 30 15:54:19 2018
From: klahnakoski at mozilla.com (Kyle Lahnakoski)
Date: Mon, 30 Jul 2018 15:54:19 -0400
Subject: [Python-ideas] Is this PEP-able? "with" statement inside
 genexps / list comprehensions
In-Reply-To: <20180730191517.GA27867@zero.localdomain>
References: <20180730191517.GA27867@zero.localdomain>
Message-ID: <4fb1f54f-ea84-16c8-a445-017a4ad2d079@mozilla.com>

Rudy,

I think your proposal may be very specific to iterable context managers;
in which case, make a method that makes that assumption:

> def iter_with(obj):
>???? with obj as context:
>???????? yield from context

and use it

> g = (
>???? f.read()
>???? for fn in filenames
>???? for f in iter_with(open(fn))
> )

On 2018-07-30 15:15, Rudy Matela wrote:
> Hello,
>
> Do you think it would be nice to allow with statements inside genexps or
> list comprehensions?  The functions __enter__ and __exit__ would be
> automatically called as iterables are traversed.  I am thinking of
> drafting a PEP about this.  Examples:
>
>
> This 
>
> 	g = (f.read() for fn in filenames with open(fn) as f)
>
> would be equivalent to the following use of a generator function:
>
> 	def __gen():
> 		for fn in filenames:
> 			with open(fn) as f:
> 				yield f.read()
> 	g = __gen()
>
>
> This
>
> 	list = [f.read() for fn in filenames with open(fn) as f]
>
> would be equivalent to the following:
>
> 	list = []
> 	for fn in filenames:
> 		with open(fn) as f:
> 			list.append(f.read())
>
> --
> Rudy
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/


From oscar.j.benjamin at gmail.com  Mon Jul 30 17:26:39 2018
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Mon, 30 Jul 2018 22:26:39 +0100
Subject: [Python-ideas] Is this PEP-able? "with" statement inside
 genexps / list comprehensions
In-Reply-To: <20180730191517.GA27867@zero.localdomain>
References: <20180730191517.GA27867@zero.localdomain>
Message-ID: <CAHVvXxR3i60BJY39P8t+7kfuUmp+_H=ufQFqGywW_bVYxuqnig@mail.gmail.com>

On 30 July 2018 at 20:15, Rudy Matela <rudy at matela.com.br> wrote:
> Hello,

Hi Rudy,

> Do you think it would be nice to allow with statements inside genexps or
> list comprehensions?  The functions __enter__ and __exit__ would be
> automatically called as iterables are traversed.  I am thinking of
> drafting a PEP about this.  Examples:
>
>
> This
>
>         g = (f.read() for fn in filenames with open(fn) as f)
>
> would be equivalent to the following use of a generator function:
>
>         def __gen():
>                 for fn in filenames:
>                         with open(fn) as f:
>                                 yield f.read()
>         g = __gen()

Yielding from a with block should be discouraged rather than given
special syntax. There is essentially a contradiction between the
meaning/purpose of yield (suspend indefinitely) and with (definitely
call __exit__).

If I partially iterate over g as in

    for line in g:
        break

then at this point g is suspended and f.__exit__ has not been called,
so the file is not closed. I may choose to iterate over g later or
not, so it has to remain in suspension just in case. In practice if
you do this in CPython then f.__exit__ will *probably* be invoked
indirectly by g.__del__ if/when the gc collects g. This defeats the
main point of using with-open though which is to avoid depending on
the gc for closing files.

--
Oscar

From greg.ewing at canterbury.ac.nz  Mon Jul 30 18:10:32 2018
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 31 Jul 2018 10:10:32 +1200
Subject: [Python-ideas] Redefining method
In-Reply-To: <CAC6Op19Z8oh-n=02qEcOH-uJ-2x89szx7Bk6oSa5YMpeuQGbpg@mail.gmail.com>
References: <CAC6Op19Z8oh-n=02qEcOH-uJ-2x89szx7Bk6oSa5YMpeuQGbpg@mail.gmail.com>
Message-ID: <5B5F8CD8.3060002@canterbury.ac.nz>

Jamesie Pic wrote:
> def o.bar(self): ...

You could get almost the same effect with

    from functools import partial

    def bar(self, other_args):
       ...

    o.bar = partial(bar, o)

But IMO this is nowhere near being a common enough thing to
do to justify having special syntax for it.

-- 
Greg

From tjreedy at udel.edu  Mon Jul 30 18:59:50 2018
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 30 Jul 2018 18:59:50 -0400
Subject: [Python-ideas] Idea: msgfmt.py and pygettext..py should be -m
 executable modules
In-Reply-To: <CA+=+wqCQd06s_p0M4ZE_3YPpM3UaLQQK_jH0SvkGpuE8uD9chQ@mail.gmail.com>
References: <f047b0cb-fc29-d3b5-b68a-18bb2cef1fc8@redhat.com>
 <CA+=+wqCQd06s_p0M4ZE_3YPpM3UaLQQK_jH0SvkGpuE8uD9chQ@mail.gmail.com>
Message-ID: <pjo553$g4l$1@blaine.gmane.org>

On 7/30/2018 9:43 AM, Petr Viktorin wrote:
> On Mon, Jul 23, 2018 at 2:16 PM, Miro Hron?ok <mhroncok at redhat.com> wrote:

>>    $ python3 -m gettext.msgfmt
> 
> +1
> Note that this means gettext will need to become a package.

Once there is a command line interface, arguments can be supplied to the 
module, as in python -m gettext msgfmt. This only affects the newly 
added main function.

-- 
Terry Jan Reedy



From steve at pearwood.info  Mon Jul 30 19:28:28 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 31 Jul 2018 09:28:28 +1000
Subject: [Python-ideas] Redefining method
In-Reply-To: <5B5F8CD8.3060002@canterbury.ac.nz>
References: <CAC6Op19Z8oh-n=02qEcOH-uJ-2x89szx7Bk6oSa5YMpeuQGbpg@mail.gmail.com>
 <5B5F8CD8.3060002@canterbury.ac.nz>
Message-ID: <20180730232827.GC22431@ando.pearwood.info>

On Tue, Jul 31, 2018 at 10:10:32AM +1200, Greg Ewing wrote:
> Jamesie Pic wrote:
> >def o.bar(self): ...
> 
> You could get almost the same effect with
> 
>    from functools import partial
> 
>    def bar(self, other_args):
>       ...
> 
>    o.bar = partial(bar, o)

Why are you using functools.partial instead of types.MethodType? I'm 
wondering if there is some advantage to partial that I don't recognise.

I'm not sure if there's a functional difference between the two 
approaches, but it makes o.bar a different kind of callable and that 
will probably make a difference to somebody.


> But IMO this is nowhere near being a common enough thing to
> do to justify having special syntax for it.

This sort of thing isn't common because there's no neat, easy, obvious, 
built-in way to do it. If we allowed people to extend classes using the 
syntax

    def classobj.methodname(...): ...

    def instance.methodname(...): ...

people would use the technique more. For good or ill.

I don't question the utility of this technique, but I suspect we prefer 
to *slightly* discourage it by *not* providing a Batteries Included 
solution for this. If you want to do this, we won't stop you, but 
neither will we encourage it by supporting it in syntax or providing a 
standard decorator for it.



-- 
Steve

From stephanh42 at gmail.com  Mon Jul 30 22:03:07 2018
From: stephanh42 at gmail.com (Stephan Houben)
Date: Tue, 31 Jul 2018 04:03:07 +0200
Subject: [Python-ideas] Redefining method
In-Reply-To: <20180730232827.GC22431@ando.pearwood.info>
References: <CAC6Op19Z8oh-n=02qEcOH-uJ-2x89szx7Bk6oSa5YMpeuQGbpg@mail.gmail.com>
 <5B5F8CD8.3060002@canterbury.ac.nz>
 <20180730232827.GC22431@ando.pearwood.info>
Message-ID: <CAOOa=pPPFbUdiF=mf=cUjt9bqXm3HgM1+PCiCt6BZVHDsTWWkA@mail.gmail.com>

Here is an example of how it could be done.

https://gist.github.com/stephanh42/97b47506e5e416f97f5790c070be7878


Stephan

Op di 31 jul. 2018 01:29 schreef Steven D'Aprano <steve at pearwood.info>:

> On Tue, Jul 31, 2018 at 10:10:32AM +1200, Greg Ewing wrote:
> > Jamesie Pic wrote:
> > >def o.bar(self): ...
> >
> > You could get almost the same effect with
> >
> >    from functools import partial
> >
> >    def bar(self, other_args):
> >       ...
> >
> >    o.bar = partial(bar, o)
>
> Why are you using functools.partial instead of types.MethodType? I'm
> wondering if there is some advantage to partial that I don't recognise.
>
> I'm not sure if there's a functional difference between the two
> approaches, but it makes o.bar a different kind of callable and that
> will probably make a difference to somebody.
>
>
> > But IMO this is nowhere near being a common enough thing to
> > do to justify having special syntax for it.
>
> This sort of thing isn't common because there's no neat, easy, obvious,
> built-in way to do it. If we allowed people to extend classes using the
> syntax
>
>     def classobj.methodname(...): ...
>
>     def instance.methodname(...): ...
>
> people would use the technique more. For good or ill.
>
> I don't question the utility of this technique, but I suspect we prefer
> to *slightly* discourage it by *not* providing a Batteries Included
> solution for this. If you want to do this, we won't stop you, but
> neither will we encourage it by supporting it in syntax or providing a
> standard decorator for it.
>
>
>
> --
> Steve
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180731/96b474d0/attachment-0001.html>

From shoyer at gmail.com  Tue Jul 31 00:19:31 2018
From: shoyer at gmail.com (Stephan Hoyer)
Date: Mon, 30 Jul 2018 21:19:31 -0700
Subject: [Python-ideas] slice[] to get more complex slices
In-Reply-To: <CALD=Yf-E0mPkKEzbZkP-agRVTqfGJPppDN5zR1+taP2chSCdYA@mail.gmail.com>
References: <CAFpSVpKnE8b=QrjxLy6anAF3ammq2QbB4_K1Ey02Vk3n8EVwyQ@mail.gmail.com>
 <pj2lgk$c6c$1@blaine.gmane.org>
 <09cb9784-6a97-49b9-b5e7-1c9a9d8c8391@googlegroups.com>
 <CALD=Yf-MNLW7z8s=ruc_C13t7eWp8KC6OWvqf9durZgpN=8=sw@mail.gmail.com>
 <399ff7d70c204f679bdb5a79f9bc0460@xmail103.UGent.be>
 <4243953379d74d2aa5393be2434a99a7@xmail103.UGent.be>
 <5B55AE6A.7010305@UGent.be>
 <CACac1F_XOi+1gmEMgZdbmW_3UXs661JS5BF=OvWLrF47H0fJmg@mail.gmail.com>
 <CAEQ_Tvf2Y-up_uWoUdutyoreHeM6=X52bZzZcath-r1btYeqow@mail.gmail.com>
 <pj5nl6$p6b$1@blaine.gmane.org>
 <CAEbHw4YuS99=cVbhV6Riuwij4H_xNdcxp1Df6tMQfD-__CaO_w@mail.gmail.com>
 <CAEQ_Tvc3RzZrew324XZ-sauegHGssWC2sBk520X0T_QQyaeG0A@mail.gmail.com>
 <CALD=Yf-E0mPkKEzbZkP-agRVTqfGJPppDN5zR1+taP2chSCdYA@mail.gmail.com>
Message-ID: <CAEQ_TvfOVDRjTe_v5FZSuNAbnv1QRVccevkwQ2hRuQZfX9JnTg@mail.gmail.com>

On Sat, Jul 28, 2018 at 9:44 AM Jonathan Fine <jfine2358 at gmail.com> wrote:

> By all means start a PEP (on github) if you find it helps you. I think
> it's too early.
>
> If this problem has a pure Python solution, then I think it best to
> develop it as a third party module. I suggest the name slicetools.
> When slicetools is established, has users and is stable. That would be
> the time to submit the PEP. I think it would give a smoother and
> easier path.
>

For most proposals, I would agree that agree that a reference
implementation as a third-party library makes sense. But operator.subscript
so simple that it would be counter productive to write it in third-party
module.

The pure Python reference implementation for operator.subscript is
literally four lines (or three lines, with __class_getitem__ in Python 3.7):

class _Subscript:
    def __getitem__(self, key):
        return key
subscript = _Subscript()

Very few people would use an implementation in a third-party package,
because it seems unlikely to be worth the trouble of adding a dependency
for so few lines of code -- copy & paste is actually more maintainable.
This exact same operator also already exists in two popular third-party
Python packages by the names numpy.s_ and pandas.IndexSlice.

That said, the actual logic is not obvious, unless you already understand
the details of how Python's indexing works. So most user code does not
benefit in clarity by copying and pasting it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180730/62b2ea94/attachment.html>

From jpic at yourlabs.org  Tue Jul 31 03:56:39 2018
From: jpic at yourlabs.org (Jamesie Pic)
Date: Tue, 31 Jul 2018 09:56:39 +0200
Subject: [Python-ideas] Redefining method
In-Reply-To: <CALD=Yf_Rx9yyazBWLG_9tyzSY5_PmUOtEsoFNc9TX66y3eQwDQ@mail.gmail.com>
References: <CAC6Op19Z8oh-n=02qEcOH-uJ-2x89szx7Bk6oSa5YMpeuQGbpg@mail.gmail.com>
 <CALD=Yf_Rx9yyazBWLG_9tyzSY5_PmUOtEsoFNc9TX66y3eQwDQ@mail.gmail.com>
Message-ID: <CAC6Op1-vmeUo-DjxGBhw0NRbzdUmTci3mLF_Amqt4TPF=+SDWg@mail.gmail.com>

I like type() a lot, and the attributes dict it takes as argument works
with lambda.

My use case is just a python module for a framework provides a default
instance for some model, and I thought it would be cool to just change a
method without going through

I'm really bad at defending ideas, specially in English, but I have one
statement to try:

How exciting is life if you're not monkey patching runtime on a daily basis
?

Cheers
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180731/be7eba33/attachment.html>

From rhodri at kynesim.co.uk  Tue Jul 31 07:32:00 2018
From: rhodri at kynesim.co.uk (Rhodri James)
Date: Tue, 31 Jul 2018 12:32:00 +0100
Subject: [Python-ideas] Is this PEP-able? "with" statement inside
 genexps / list comprehensions
In-Reply-To: <20180730191517.GA27867@zero.localdomain>
References: <20180730191517.GA27867@zero.localdomain>
Message-ID: <025ca9fa-a8d3-2d3e-2462-8e5ad63b70e9@kynesim.co.uk>

On 30/07/18 20:15, Rudy Matela wrote:
> Hello,
> 
> Do you think it would be nice to allow with statements inside genexps or
> list comprehensions?  The functions __enter__ and __exit__ would be
> automatically called as iterables are traversed.  I am thinking of
> drafting a PEP about this.  Examples:

I was worried until I twigged that you meant adding a clause rather than 
a statement per se.  How does this interact with the other clauses in a 
comprehension?  I'm thinking of monstrosities like:

g = (f.read() for fn in filenames if fn.endswith(".txt") with open(fn) 
as f for filenames in get_directory_listing(dir) if .....)

-- 
Rhodri James *-* Kynesim Ltd

From storchaka at gmail.com  Tue Jul 31 08:49:08 2018
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 31 Jul 2018 15:49:08 +0300
Subject: [Python-ideas] Is this PEP-able? "with" statement inside
 genexps / list comprehensions
In-Reply-To: <CAHVvXxR3i60BJY39P8t+7kfuUmp+_H=ufQFqGywW_bVYxuqnig@mail.gmail.com>
References: <20180730191517.GA27867@zero.localdomain>
 <CAHVvXxR3i60BJY39P8t+7kfuUmp+_H=ufQFqGywW_bVYxuqnig@mail.gmail.com>
Message-ID: <pjplo1$pn5$1@blaine.gmane.org>

31.07.18 00:26, Oscar Benjamin ????:
> On 30 July 2018 at 20:15, Rudy Matela <rudy at matela.com.br> wrote:
> Yielding from a with block should be discouraged rather than given
> special syntax. There is essentially a contradiction between the
> meaning/purpose of yield (suspend indefinitely) and with (definitely
> call __exit__).
> 
> If I partially iterate over g as in
> 
>      for line in g:
>          break
> 
> then at this point g is suspended and f.__exit__ has not been called,
> so the file is not closed. I may choose to iterate over g later or
> not, so it has to remain in suspension just in case. In practice if
> you do this in CPython then f.__exit__ will *probably* be invoked
> indirectly by g.__del__ if/when the gc collects g. This defeats the
> main point of using with-open though which is to avoid depending on
> the gc for closing files.

Concur with Oscar.


From steve at pearwood.info  Tue Jul 31 13:46:47 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 1 Aug 2018 03:46:47 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAKr=oZuwNR2gim7znHuJtoW0bof5+jL3+dprKDREiU=Q_OxEXA@mail.gmail.com>
References: <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
 <CAPTjJmp2tN1TBz96EfnLRDsdO==QJ0xtYbCVkTBJDQ9+qOp6nA@mail.gmail.com>
 <CAKr=oZsTsR8Mv7_i386eOrEzQWTjiwbONt+7C=6D9V94f56kaQ@mail.gmail.com>
 <20180729055442.GH22554@ando.pearwood.info>
 <CAEbHw4Y0YFC0ttZuu3PDSjUJagC4uQ2XaOFC+6GNasG-OE+Gsw@mail.gmail.com>
 <CAPTjJmpbj9=xYff14SQdnb5O2_Ttj70JUd_aWpBx4ud4TXXnXQ@mail.gmail.com>
 <CAKr=oZv1Nibq=G+p442aF9Ce874HvLb60D18A3qMWoBJC41HDw@mail.gmail.com>
 <e86b9886-4a15-d6f0-1c17-88196f422382@kynesim.co.uk>
 <CAKr=oZuwNR2gim7znHuJtoW0bof5+jL3+dprKDREiU=Q_OxEXA@mail.gmail.com>
Message-ID: <20180731174646.GE22431@ando.pearwood.info>

On Mon, Jul 30, 2018 at 12:41:20PM -0500, Abe Dillon wrote:

[Rhodri James]
> > On 29/07/18 16:12, Abe Dillon wrote:
> >  > spam?.eggs.cheese.aardvark  # why would you ever do this?
> > 
> > If you knew that if you really have something in "spam", your program
> > guarantees it will have an "eggs" attribute with a "cheese" attribute, etc,
> > you just don't know if "spam" is not None.  It's the same insider knowledge
> > you would use if you wrote it out as
> > 
> > spam.eggs.cheese.aardvark if spam is not None else None
> 
> 
> That's not the equivalent as described in PEP 505. 

Yes it is. Rhodri is correct, although I admit that I hadn't realised 
this point myself until he pointed it out. (That is why until now I've 
been writing examples like "spam?.eggs?.cheese?.aardvark" which is 
redundant.)

The abstract says:

    The "None-aware attribute access" operator ?. ("maybe dot")
    evaluates the complete expression if the left hand side 
    evaluates to a value that is not None

and later on the PEP says:

    When a None-aware operator is present, the left-to-right 
    evaluation may be short-circuited. For example, 
    await a?.b(c).d?[e] is evaluated:

    _v = a
    if _v is not None:
        _v = _v.b
        _v = _v(c)
        _v = _v.d
        if _v is not None:
            _v = _v[e]
    await _v


I admit the example isn't the clearest. The inclusion of "await" doesn't 
seem helpful to me (if anything, the opposite, since "await None" will 
fail) and perhaps it would be clearer to write it as:

    _v = a
    if _v is not None:
        _v = _v.b(c).d
        if _v is not None:
            _v = _v[e]


The PEP also says:

    Parenthesised expressions are handled by the atom rule (not 
    shown above), which will implicitly terminate the short-
    circuiting behaviour of the above transformation.

which again isn't too clear, and the example given seems complicated 
enough to obfuscate the point rather than illustrate it, but if I'm 
reading it correctly, the behaviour you expected would have to be 
written as

    (spam?.eggs).cheese.aardvark

which would likely fail since None has no cheese attribute.

The bottom line is that in any unparenthesized chain of maybe-dot and 
maybe-subscript operations, only the first needs to be "maybe" as that 
will short-circuit the rest of the expression.

Figuratively speaking, it is as if we had:

    spam?(.eggs.cheese.aardvark)

where the () are used for grouping, not function call.



-- 
Steve

From mertz at gnosis.cx  Tue Jul 31 14:14:19 2018
From: mertz at gnosis.cx (David Mertz)
Date: Tue, 31 Jul 2018 14:14:19 -0400
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180731174646.GE22431@ando.pearwood.info>
References: <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
 <CAPTjJmp2tN1TBz96EfnLRDsdO==QJ0xtYbCVkTBJDQ9+qOp6nA@mail.gmail.com>
 <CAKr=oZsTsR8Mv7_i386eOrEzQWTjiwbONt+7C=6D9V94f56kaQ@mail.gmail.com>
 <20180729055442.GH22554@ando.pearwood.info>
 <CAEbHw4Y0YFC0ttZuu3PDSjUJagC4uQ2XaOFC+6GNasG-OE+Gsw@mail.gmail.com>
 <CAPTjJmpbj9=xYff14SQdnb5O2_Ttj70JUd_aWpBx4ud4TXXnXQ@mail.gmail.com>
 <CAKr=oZv1Nibq=G+p442aF9Ce874HvLb60D18A3qMWoBJC41HDw@mail.gmail.com>
 <e86b9886-4a15-d6f0-1c17-88196f422382@kynesim.co.uk>
 <CAKr=oZuwNR2gim7znHuJtoW0bof5+jL3+dprKDREiU=Q_OxEXA@mail.gmail.com>
 <20180731174646.GE22431@ando.pearwood.info>
Message-ID: <CAEbHw4a8im2Vo7NokkWRA3WJxvuirw2_gGAYzKk9RLbhqwqvyw@mail.gmail.com>

On Tue, Jul 31, 2018, 1:47 PM Steven D'Aprano <steve at pearwood.info> wrote:

> Yes it is. Rhodri is correct, although I admit that I hadn't realised
> this point myself until he pointed it out. (That is why until now I've
> been writing examples like "spam?.eggs?.cheese?.aardvark" which is
> redundant.)
>

Again, one of the most vocal advocates of the PEP gets the semantics wrong!

`spam?.eggs?.cheese?.aardvark` is NOT redundant for `
spam?.eggs.cheese.aardvark`.  The two expressions simply do different
things, but in a way guaranteed to assure that also everyone gets the
actual behaviors wrong.

Hint, run this before each expression:

    spam.eggs.cheese = None
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180731/8ba8190b/attachment.html>

From rhodri at kynesim.co.uk  Tue Jul 31 14:35:43 2018
From: rhodri at kynesim.co.uk (Rhodri James)
Date: Tue, 31 Jul 2018 19:35:43 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <20180731174646.GE22431@ando.pearwood.info>
References: <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
 <CAPTjJmp2tN1TBz96EfnLRDsdO==QJ0xtYbCVkTBJDQ9+qOp6nA@mail.gmail.com>
 <CAKr=oZsTsR8Mv7_i386eOrEzQWTjiwbONt+7C=6D9V94f56kaQ@mail.gmail.com>
 <20180729055442.GH22554@ando.pearwood.info>
 <CAEbHw4Y0YFC0ttZuu3PDSjUJagC4uQ2XaOFC+6GNasG-OE+Gsw@mail.gmail.com>
 <CAPTjJmpbj9=xYff14SQdnb5O2_Ttj70JUd_aWpBx4ud4TXXnXQ@mail.gmail.com>
 <CAKr=oZv1Nibq=G+p442aF9Ce874HvLb60D18A3qMWoBJC41HDw@mail.gmail.com>
 <e86b9886-4a15-d6f0-1c17-88196f422382@kynesim.co.uk>
 <CAKr=oZuwNR2gim7znHuJtoW0bof5+jL3+dprKDREiU=Q_OxEXA@mail.gmail.com>
 <20180731174646.GE22431@ando.pearwood.info>
Message-ID: <2850d720-4298-42e0-a225-51ab3f4f8d90@kynesim.co.uk>

On 31/07/18 18:46, Steven D'Aprano wrote:
> n Mon, Jul 30, 2018 at 12:41:20PM -0500, Abe Dillon wrote:
> 
> [Rhodri James]
>>> On 29/07/18 16:12, Abe Dillon wrote:
>>>   > spam?.eggs.cheese.aardvark  # why would you ever do this?
>>>
>>> If you knew that if you really have something in "spam", your program
>>> guarantees it will have an "eggs" attribute with a "cheese" attribute, etc,
>>> you just don't know if "spam" is not None.  It's the same insider knowledge
>>> you would use if you wrote it out as
>>>
>>> spam.eggs.cheese.aardvark if spam is not None else None
>>
>> That's not the equivalent as described in PEP 505.
> Yes it is. Rhodri is correct, although I admit that I hadn't realised
> this point myself until he pointed it out. (That is why until now I've
> been writing examples like "spam?.eggs?.cheese?.aardvark" which is
> redundant.)

Doh!  Yes, of course, the point was it was short-circuiting.  This keeps 
confusing me (easily done, I'll admit), which is probably not a point in 
its favour.

-- 
Rhodri James *-* Kynesim Ltd

From jfine2358 at gmail.com  Tue Jul 31 14:48:50 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Tue, 31 Jul 2018 19:48:50 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4a8im2Vo7NokkWRA3WJxvuirw2_gGAYzKk9RLbhqwqvyw@mail.gmail.com>
References: <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
 <CAPTjJmp2tN1TBz96EfnLRDsdO==QJ0xtYbCVkTBJDQ9+qOp6nA@mail.gmail.com>
 <CAKr=oZsTsR8Mv7_i386eOrEzQWTjiwbONt+7C=6D9V94f56kaQ@mail.gmail.com>
 <20180729055442.GH22554@ando.pearwood.info>
 <CAEbHw4Y0YFC0ttZuu3PDSjUJagC4uQ2XaOFC+6GNasG-OE+Gsw@mail.gmail.com>
 <CAPTjJmpbj9=xYff14SQdnb5O2_Ttj70JUd_aWpBx4ud4TXXnXQ@mail.gmail.com>
 <CAKr=oZv1Nibq=G+p442aF9Ce874HvLb60D18A3qMWoBJC41HDw@mail.gmail.com>
 <e86b9886-4a15-d6f0-1c17-88196f422382@kynesim.co.uk>
 <CAKr=oZuwNR2gim7znHuJtoW0bof5+jL3+dprKDREiU=Q_OxEXA@mail.gmail.com>
 <20180731174646.GE22431@ando.pearwood.info>
 <CAEbHw4a8im2Vo7NokkWRA3WJxvuirw2_gGAYzKk9RLbhqwqvyw@mail.gmail.com>
Message-ID: <CALD=Yf8=UqAcFxdPgmUvX9WQisSOMELHUPmv37ys5NxPm6fomA@mail.gmail.com>

David Mertz wrote:

> `spam?.eggs?.cheese?.aardvark` is NOT redundant for
> `spam?.eggs.cheese.aardvark`.  The two expressions simply do different
> things [...]

I agree, assuming ?. is a binary operator.  Given this, in Python (+
PEP 505) one can write

    tmp = spam ?. eggs
    val1 = tmp ?. cheese ?. aardvark    # For spam?.eggs?.cheese?.aardvark
    val2 = tmp . cheese . aardvark    # For spam?.eggs.cheese.aardvark

No special knowledge of PEP 505 is needed. If val1 is always equal to
val2, then the dot and None-dot operators must be the same. From the
assumptions, this is something that can be mathematically proved.

By the way, there's a widely used programming language in which
    val = a.method()
and
    tmp = a.method
    val = tmp()
are not always equivalent. Can you guess which language it is?

The answer is in:
https://www.slideshare.net/jonathanfine/javascript-the-easiest-quiz-in-the-world-ever
(question 6: Dot binds).

I'll now go back to following the example of Steve Bower and Raymond
Hettinger, which in my words is to wait until we have proper cover for
the BDFL's vacation.

-- 
Jonathan

From stephanh42 at gmail.com  Tue Jul 31 15:06:06 2018
From: stephanh42 at gmail.com (Stephan Houben)
Date: Tue, 31 Jul 2018 21:06:06 +0200
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CALD=Yf8=UqAcFxdPgmUvX9WQisSOMELHUPmv37ys5NxPm6fomA@mail.gmail.com>
References: <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
 <CAPTjJmp2tN1TBz96EfnLRDsdO==QJ0xtYbCVkTBJDQ9+qOp6nA@mail.gmail.com>
 <CAKr=oZsTsR8Mv7_i386eOrEzQWTjiwbONt+7C=6D9V94f56kaQ@mail.gmail.com>
 <20180729055442.GH22554@ando.pearwood.info>
 <CAEbHw4Y0YFC0ttZuu3PDSjUJagC4uQ2XaOFC+6GNasG-OE+Gsw@mail.gmail.com>
 <CAPTjJmpbj9=xYff14SQdnb5O2_Ttj70JUd_aWpBx4ud4TXXnXQ@mail.gmail.com>
 <CAKr=oZv1Nibq=G+p442aF9Ce874HvLb60D18A3qMWoBJC41HDw@mail.gmail.com>
 <e86b9886-4a15-d6f0-1c17-88196f422382@kynesim.co.uk>
 <CAKr=oZuwNR2gim7znHuJtoW0bof5+jL3+dprKDREiU=Q_OxEXA@mail.gmail.com>
 <20180731174646.GE22431@ando.pearwood.info>
 <CAEbHw4a8im2Vo7NokkWRA3WJxvuirw2_gGAYzKk9RLbhqwqvyw@mail.gmail.com>
 <CALD=Yf8=UqAcFxdPgmUvX9WQisSOMELHUPmv37ys5NxPm6fomA@mail.gmail.com>
Message-ID: <CAOOa=pOEQ5gYg-jQ_5FAnr_J-0AD6NZB1Ab=p7vGAqZcEtsZEw@mail.gmail.com>

Op di 31 jul. 2018 20:49 schreef Jonathan Fine <jfine2358 at gmail.com>:

> David Mertz wrote:
>
> > `spam?.eggs?.cheese?.aardvark` is NOT redundant for
> > `spam?.eggs.cheese.aardvark`.  The two expressions simply do different
> > things [...]
>
> I agree, assuming ?. is a binary operator.


It isn't.


Given this, in Python (+
> PEP 505) one can write
>
>     tmp = spam ?. eggs
>     val1 = tmp ?. cheese ?. aardvark    # For spam?.eggs?.cheese?.aardvark
>     val2 = tmp . cheese . aardvark    # For spam?.eggs.cheese.aardvark
>

Nope, the introduction of the tmp variable changed the semantics. It isn't
a "chain" anymore so it breaks shortcutting.

To be honest I didn't get this either until it was pointed out to me

>
> No special knowledge of PEP 505 is needed. If val1 is always equal to
> val2, then the dot and None-dot operators must be the same. From the
> assumptions, this is something that can be mathematically proved.
>

And false.


> By the way, there's a widely used programming language in which
>     val = a.method()
> and
>     tmp = a.method
>     val = tmp()
> are not always equivalent. Can you guess which language it is?


Javascript.
I suppose in the same way as x+2 and x*2 are " not always" equivalent.

Stephan



> The answer is in:
>
> https://www.slideshare.net/jonathanfine/javascript-the-easiest-quiz-in-the-world-ever
> (question 6: Dot binds).
>
> I'll now go back to following the example of Steve Bower and Raymond
> Hettinger, which in my words is to wait until we have proper cover for
> the BDFL's vacation.
>
> --
> Jonathan
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180731/699a2f27/attachment-0001.html>

From steve at pearwood.info  Tue Jul 31 15:50:06 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 1 Aug 2018 05:50:06 +1000
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAEbHw4a8im2Vo7NokkWRA3WJxvuirw2_gGAYzKk9RLbhqwqvyw@mail.gmail.com>
References: <CAPTjJmp2tN1TBz96EfnLRDsdO==QJ0xtYbCVkTBJDQ9+qOp6nA@mail.gmail.com>
 <CAKr=oZsTsR8Mv7_i386eOrEzQWTjiwbONt+7C=6D9V94f56kaQ@mail.gmail.com>
 <20180729055442.GH22554@ando.pearwood.info>
 <CAEbHw4Y0YFC0ttZuu3PDSjUJagC4uQ2XaOFC+6GNasG-OE+Gsw@mail.gmail.com>
 <CAPTjJmpbj9=xYff14SQdnb5O2_Ttj70JUd_aWpBx4ud4TXXnXQ@mail.gmail.com>
 <CAKr=oZv1Nibq=G+p442aF9Ce874HvLb60D18A3qMWoBJC41HDw@mail.gmail.com>
 <e86b9886-4a15-d6f0-1c17-88196f422382@kynesim.co.uk>
 <CAKr=oZuwNR2gim7znHuJtoW0bof5+jL3+dprKDREiU=Q_OxEXA@mail.gmail.com>
 <20180731174646.GE22431@ando.pearwood.info>
 <CAEbHw4a8im2Vo7NokkWRA3WJxvuirw2_gGAYzKk9RLbhqwqvyw@mail.gmail.com>
Message-ID: <20180731195005.GF22431@ando.pearwood.info>

On Tue, Jul 31, 2018 at 02:14:19PM -0400, David Mertz wrote:
> On Tue, Jul 31, 2018, 1:47 PM Steven D'Aprano <steve at pearwood.info> wrote:
> 
> > Yes it is. Rhodri is correct, although I admit that I hadn't realised
> > this point myself until he pointed it out. (That is why until now I've
> > been writing examples like "spam?.eggs?.cheese?.aardvark" which is
> > redundant.)
> >
> 
> Again, one of the most vocal advocates of the PEP gets the semantics wrong!
> 
> `spam?.eggs?.cheese?.aardvark` is NOT redundant for `
> spam?.eggs.cheese.aardvark`.

It is if *only* spam can be None, which is the scenario being discussed.

Please don't ignore the context of statements. I didn't say that

    spam?.eggs.cheese.aardvark

would guard against eggs or cheese being None. That interpretation of my 
comment is uncharitable and not supported by my re-write of the 
pseudo-code from the PEP.

It also goes completely against my comment "Figuratively speaking..." 
that the ?. operator groups to the right:

    spam?(.eggs.cheese.aardvark)  # grouping, not function call.

Perhaps you didn't read my post all the way to the end before firing off 
a triumphant post accusing me of getting it wrong.

In context, the question is whether:

    spam?.eggs.cheese.aardvark

might attempt to evaluate None.cheese.aardvark. It won't.

In context, the issue was not whether these two expressions are 
identical in every imaginable way:

    spam?.eggs.cheese.aardvark

    spam?.eggs?.cheese?.aardvark

In pseudo-code, they would be something close to:

    if spam is not None:
        return spam.eggs.cheese.aardvark

versus

    if spam is not None:
        if spam.eggs is not None:
            if spam.eggs.cheese is not None:
                return spam.eggs.cheese.aardvark

except without needing to evaluate each subexpression multiple times.

In context, we are discussing the behaviour when spam is None. When spam 
is None, there's no need for the additional maybe-dots, because the 
first short-circuits the rest. If additional attributes also could be 
None, then they too could be guarded with maybe-dot operators, but if 
they are never None, it is unnecessary and redundant to guard spam alone 
all the way through the expression with maybe-dots.


> The two expressions simply do different
> things, but in a way guaranteed to assure that also everyone gets the
> actual behaviors wrong.

Putting aside your prejudicial language about guaranteeing errors, I 
think that the PEP could be improved. I managed to read it multiple 
times without taking in the fact that ?. short-circuits to the right. I 
didn't explicitly say so (perhaps I should have) but I too initially 
made the same error as Abe, assuming that

    spam?.eggs.cheese

would fail if spam was None (evaluating None.cheese) but according to 
the PEP that's not the case.

With a working implementation, it would be the work of moments for
people to experiment in the REPL and clarify any lingering doubts about 
the operation.

Without it, there will always be some misunderstandings from those who 
(like me) failed to read the documentation carefully enough and made 
upjustified assumptions. That's no different from any other non-trivial 
feature.


-- 
Steve

From jfine2358 at gmail.com  Tue Jul 31 15:53:50 2018
From: jfine2358 at gmail.com (Jonathan Fine)
Date: Tue, 31 Jul 2018 20:53:50 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CAOOa=pOEQ5gYg-jQ_5FAnr_J-0AD6NZB1Ab=p7vGAqZcEtsZEw@mail.gmail.com>
References: <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
 <CAPTjJmp2tN1TBz96EfnLRDsdO==QJ0xtYbCVkTBJDQ9+qOp6nA@mail.gmail.com>
 <CAKr=oZsTsR8Mv7_i386eOrEzQWTjiwbONt+7C=6D9V94f56kaQ@mail.gmail.com>
 <20180729055442.GH22554@ando.pearwood.info>
 <CAEbHw4Y0YFC0ttZuu3PDSjUJagC4uQ2XaOFC+6GNasG-OE+Gsw@mail.gmail.com>
 <CAPTjJmpbj9=xYff14SQdnb5O2_Ttj70JUd_aWpBx4ud4TXXnXQ@mail.gmail.com>
 <CAKr=oZv1Nibq=G+p442aF9Ce874HvLb60D18A3qMWoBJC41HDw@mail.gmail.com>
 <e86b9886-4a15-d6f0-1c17-88196f422382@kynesim.co.uk>
 <CAKr=oZuwNR2gim7znHuJtoW0bof5+jL3+dprKDREiU=Q_OxEXA@mail.gmail.com>
 <20180731174646.GE22431@ando.pearwood.info>
 <CAEbHw4a8im2Vo7NokkWRA3WJxvuirw2_gGAYzKk9RLbhqwqvyw@mail.gmail.com>
 <CALD=Yf8=UqAcFxdPgmUvX9WQisSOMELHUPmv37ys5NxPm6fomA@mail.gmail.com>
 <CAOOa=pOEQ5gYg-jQ_5FAnr_J-0AD6NZB1Ab=p7vGAqZcEtsZEw@mail.gmail.com>
Message-ID: <CALD=Yf-Qc_dDKAPTQ1pTw5peLoHci+N5XES1mzJWMx5veUaBDg@mail.gmail.com>

Stephan Houben wrote:

> Nope, the introduction of the tmp variable changed the semantics. It isn't a
> "chain" anymore so it breaks shortcutting.

I'm confused. Assume 'a' is not defined.

With Python's dot (attribute access) we have
>>> a.b.c
NameError: name 'a' is not defined
>>> a.(b.c)
SyntaxError: invalid syntax
>>> (a.b).c
NameError: name 'a' is not defined

I'd expect, after PEP 505 is added to Python that we'd get
>>> a ?. b ?. c
NameError: name 'a' is not defined
>>> a ?. (b ?. c)
SyntaxError: invalid syntax
>>> (a ? . b) ?. c
NameError: name 'a' is not defined

If this is not correct, please some do tell me what we would get.

Now assume 'a' is defined. I'd also expect, for any value for 'a', that
>>> tmp = (a ?. b)
>>> val = tmp ?. c
give val the same value as
>>> val = (a ?. b) ?. c

If this is not so, then can the second val be computed from tmp? And if so, how?

-- 
Jonathan

From python at mrabarnett.plus.com  Tue Jul 31 21:04:32 2018
From: python at mrabarnett.plus.com (MRAB)
Date: Wed, 1 Aug 2018 02:04:32 +0100
Subject: [Python-ideas] PEP 505: None-aware operators
In-Reply-To: <CALD=Yf-Qc_dDKAPTQ1pTw5peLoHci+N5XES1mzJWMx5veUaBDg@mail.gmail.com>
References: <fbf18fe1-17c8-bbbd-7aa8-ad2978a18360@kynesim.co.uk>
 <CAKr=oZtKW1Hg+PSq-1qjCw9Mq_TOzrOLtOQd1LQHNHbAXr3RRg@mail.gmail.com>
 <CAPTjJmp2tN1TBz96EfnLRDsdO==QJ0xtYbCVkTBJDQ9+qOp6nA@mail.gmail.com>
 <CAKr=oZsTsR8Mv7_i386eOrEzQWTjiwbONt+7C=6D9V94f56kaQ@mail.gmail.com>
 <20180729055442.GH22554@ando.pearwood.info>
 <CAEbHw4Y0YFC0ttZuu3PDSjUJagC4uQ2XaOFC+6GNasG-OE+Gsw@mail.gmail.com>
 <CAPTjJmpbj9=xYff14SQdnb5O2_Ttj70JUd_aWpBx4ud4TXXnXQ@mail.gmail.com>
 <CAKr=oZv1Nibq=G+p442aF9Ce874HvLb60D18A3qMWoBJC41HDw@mail.gmail.com>
 <e86b9886-4a15-d6f0-1c17-88196f422382@kynesim.co.uk>
 <CAKr=oZuwNR2gim7znHuJtoW0bof5+jL3+dprKDREiU=Q_OxEXA@mail.gmail.com>
 <20180731174646.GE22431@ando.pearwood.info>
 <CAEbHw4a8im2Vo7NokkWRA3WJxvuirw2_gGAYzKk9RLbhqwqvyw@mail.gmail.com>
 <CALD=Yf8=UqAcFxdPgmUvX9WQisSOMELHUPmv37ys5NxPm6fomA@mail.gmail.com>
 <CAOOa=pOEQ5gYg-jQ_5FAnr_J-0AD6NZB1Ab=p7vGAqZcEtsZEw@mail.gmail.com>
 <CALD=Yf-Qc_dDKAPTQ1pTw5peLoHci+N5XES1mzJWMx5veUaBDg@mail.gmail.com>
Message-ID: <22124c7c-e0c6-5857-b2c7-4b162e908da2@mrabarnett.plus.com>

On 2018-07-31 20:53, Jonathan Fine wrote:
> Stephan Houben wrote:
> 
>> Nope, the introduction of the tmp variable changed the semantics. It isn't a
>> "chain" anymore so it breaks shortcutting.
> 
> I'm confused. Assume 'a' is not defined.
> 
> With Python's dot (attribute access) we have
>>>> a.b.c
> NameError: name 'a' is not defined
>>>> a.(b.c)
> SyntaxError: invalid syntax
>>>> (a.b).c
> NameError: name 'a' is not defined
> 
> I'd expect, after PEP 505 is added to Python that we'd get
>>>> a ?. b ?. c
> NameError: name 'a' is not defined
>>>> a ?. (b ?. c)
> SyntaxError: invalid syntax
>>>> (a ? . b) ?. c
> NameError: name 'a' is not defined
> 
> If this is not correct, please some do tell me what we would get.
> 
Correct.

> Now assume 'a' is defined. I'd also expect, for any value for 'a', that
>>>> tmp = (a ?. b)
>>>> val = tmp ?. c
> give val the same value as
>>>> val = (a ?. b) ?. c
> 
> If this is not so, then can the second val be computed from tmp? And if so, how?
> 
Also correct.

On the first point, Steven said that _figuratively speaking_ (his words) 
the series of attribute accesses would be _as though_ it was grouped 
a?.(b.c), so if a is None, then remainder of the attributes accessed 
would be short-circuited. He _wasn't_ suggesting that that was _actual_ 
syntax.

On the second point, you'd said:

     tmp = spam ?. eggs
     val2 = tmp . cheese . aardvark    # For spam?.eggs.cheese.aardvark

i.e. that:

     spam?.eggs.cheese.aardvark

could be rewritten as:

     tmp = spam ?. eggs
     val2 = tmp . cheese . aardvark

Steven pointed out that that was wrong.

In the first, if spam is None, then the remainder is short-circuited, 
and so the result of spam?.eggs.cheese.aardvark is None.

In the second, if spam is None, then tmp would be None, and tmp.cheese 
would fail.