From tjreedy at udel.edu  Sun Jan  1 18:23:39 2017
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 1 Jan 2017 18:23:39 -0500
Subject: [Python-Dev] SystemError: new style getargs format but argument is
 not a tuple
Message-ID: <o4c31n$kq1$1@blaine.gmane.org>

There are several recent question on Stackoverflow about

SystemError: new style getargs format but argument is not a tuple

PIL/pillow, 2.7, Nov 28, 2016
https://stackoverflow.com/questions/40844212/systemerror-new-style-getargs-format-but-argument-is-not-a-tuple-even-the-argum

ImageDraw.draw.line(), python, Sep 14, 2016
https://stackoverflow.com/questions/39497458/imagedraw-draw-line-systemerror-new-style-getargs-format-but-argument-is-no?rq=1

opencv, cv2.line, python, Sep 19, 2016
https://stackoverflow.com/questions/32673359/systemerror-new-style-getargs-format-but-argument-is-not-a-tuple?rq=1

opencv, cv2.imread/show, python, Nov 5, 2016
https://stackoverflow.com/questions/13225525/system-error-new-style-getargs-format-but-argument-is-not-a-tuple-when-using?rq=1
(code mostly from doc)

??.imread?, python, Jan 1, 2017
https://stackoverflow.com/questions/41419688/systemerror-new-style-getargs-format-but-argument-is-not-a-tuple-it-is-a-tup

(and maybe others)
[Would that people would identify OS and exact version of both python 
and imported module ;-)]

No one commenting has a clue.  Is message from Victor's new calling 
code?  Was it backported to 2.7?  Could error be a result of running old 
3rd party binary that needs to be recompiled?


-- 
Terry Jan Reedy


From storchaka at gmail.com  Sun Jan  1 18:40:38 2017
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 2 Jan 2017 01:40:38 +0200
Subject: [Python-Dev] SystemError: new style getargs format but argument
 is not a tuple
In-Reply-To: <o4c31n$kq1$1@blaine.gmane.org>
References: <o4c31n$kq1$1@blaine.gmane.org>
Message-ID: <o4c41i$ain$1@blaine.gmane.org>

On 02.01.17 01:23, Terry Reedy wrote:
> There are several recent question on Stackoverflow about
>
> SystemError: new style getargs format but argument is not a tuple
>
> PIL/pillow, 2.7, Nov 28, 2016
> https://stackoverflow.com/questions/40844212/systemerror-new-style-getargs-format-but-argument-is-not-a-tuple-even-the-argum
>
>
> ImageDraw.draw.line(), python, Sep 14, 2016
> https://stackoverflow.com/questions/39497458/imagedraw-draw-line-systemerror-new-style-getargs-format-but-argument-is-no?rq=1
>
>
> opencv, cv2.line, python, Sep 19, 2016
> https://stackoverflow.com/questions/32673359/systemerror-new-style-getargs-format-but-argument-is-not-a-tuple?rq=1
>
>
> opencv, cv2.imread/show, python, Nov 5, 2016
> https://stackoverflow.com/questions/13225525/system-error-new-style-getargs-format-but-argument-is-not-a-tuple-when-using?rq=1
>
> (code mostly from doc)
>
> ??.imread?, python, Jan 1, 2017
> https://stackoverflow.com/questions/41419688/systemerror-new-style-getargs-format-but-argument-is-not-a-tuple-it-is-a-tup
>
>
> (and maybe others)
> [Would that people would identify OS and exact version of both python
> and imported module ;-)]
>
> No one commenting has a clue.  Is message from Victor's new calling
> code?  Was it backported to 2.7?  Could error be a result of running old
> 3rd party binary that needs to be recompiled?

A system error "new style getargs format but argument is not a tuple" is 
not new. It means that PyArg_ParseTuple() is called with not a tuple as 
the first arguments. This is just a programmical error in third-party 
extension.



From njs at pobox.com  Mon Jan  2 03:27:42 2017
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 2 Jan 2017 00:27:42 -0800
Subject: [Python-Dev] --with-fpectl changes the CPython ABI
In-Reply-To: <CADiSq7f0NPouBWaCkfLnyQrDWYMq+_23gZDRaXE2ggcWG2AxJQ@mail.gmail.com>
References: <CAPJVwB=nUk7YaZTqk8gJcK5RymoPbcJqpZC8oBBFX=1Xdr9cfA@mail.gmail.com>
 <CADiSq7f0NPouBWaCkfLnyQrDWYMq+_23gZDRaXE2ggcWG2AxJQ@mail.gmail.com>
Message-ID: <CAPJVwB=6fwLZiPnEh6HEJ6iGW1i-bL11+4c6fO1uCejBVMHvhQ@mail.gmail.com>

On Sun, Dec 25, 2016 at 5:55 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 25 December 2016 at 09:48, Nathaniel Smith <njs at pobox.com> wrote:
>>
>> Or maybe make it so that even no-fpectl builds still export the
>> necessary symbols so that yes-fpectl extensions don't crash on import?
>> (This has the advantage that it can be done in a point release...)
>
>
> This seems like a sensible thing to do in 3.6, 3.5 and 2.7 regardless of
> what happens in 3.7.
>
> For 3.7, I don't understand the trade-offs well enough to have a strong
> opinion, but dropping the feature entirely does seem reasonable - folks that
> want fine-grained floating point exception control these days are likely to
> be much better served by the decimal module, or one of the third party
> computing libraries (numpy, gmpy, sympy, etc).

I looked into this a bit more. I think the way it's *supposed* to work
is that normally, various operations in Python might return inf or
nan, but if you call fpectl.turnon_sigfpe() then they switch to
raising exceptions instead. But AFAICT the fpectl module:

1) is totally broken on major platforms: There doesn't seem to be any
implementation at all for MacOS. On x86/x86-64 Linux it works by
fiddling with the x87 control word directly... which is okay for
traditional x86 with SSE disabled, but on x86-64, or x86 with SSE
enabled, then there are two separate floating point units on the
processor (the old x87 FPU, and the new SSE unit), and which one gets
used for any given operation is up to the C compiler. So on Linux,
whether fpectl actually affects any given floating point operation is
dependent on basically the phase of the moon. This is pretty bad.

2) doesn't seem to actually accomplish anything even when it does
work: Back in the old days, math.exp(1000) apparently returned inf
(there's a REPL transcript showing this at the top of the fpectl
documentation). But nowadays math.exp raises an exception in cases
where it used to return inf, regardless of fpectl. I haven't been able
to find any cases where fpectl actually... does anything?

3) ...except that it does break numpy and any other code that expects
the default IEEE-754 semantics: The way fpectl works is that it
twiddles with the FP control word, which is a thread-global variable.
After you call turnon_sigfpe(), then *any* floating point code in that
thread that happens to generate a nan or inf instead triggers a
SIGFPE, and if the code isn't specifically written to use the PyFPE_*
macros then this causes a process abort. For example:

~$ python
Python 3.5.2+ (default, Dec 13 2016, 14:16:35)
[GCC 6.2.1 20161124] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.longdouble(1) / np.longdouble(0)
__main__:1: RuntimeWarning: divide by zero encountered in longdouble_scalars
inf
>>> import fpectl
>>> fpectl.turnon_sigfpe()
>>> np.longdouble(1) / np.longdouble(0)
Fatal Python error: Unprotected floating point exception

Current thread 0x00007fea57a9f700 (most recent call first):
  File "<stdin>", line 1 in <module>
zsh: abort      python
~$

(I'm using np.longdouble to work around the Linux SSE bug -- using
long double forces the calculations to be done on the x87 unit. On
Windows I believe it would be sufficient to just do np.array(1.0) /
np.array(0.0).)

So I guess that yeah, my suggestion would be to drop this feature
entirely in 3.7, given that it's never been enabled by default and has
been largely broken for years. Or do we still need a full deprecation
cycle?

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From ncoghlan at gmail.com  Mon Jan  2 07:22:45 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 2 Jan 2017 22:22:45 +1000
Subject: [Python-Dev] --with-fpectl changes the CPython ABI
In-Reply-To: <CAPJVwB=6fwLZiPnEh6HEJ6iGW1i-bL11+4c6fO1uCejBVMHvhQ@mail.gmail.com>
References: <CAPJVwB=nUk7YaZTqk8gJcK5RymoPbcJqpZC8oBBFX=1Xdr9cfA@mail.gmail.com>
 <CADiSq7f0NPouBWaCkfLnyQrDWYMq+_23gZDRaXE2ggcWG2AxJQ@mail.gmail.com>
 <CAPJVwB=6fwLZiPnEh6HEJ6iGW1i-bL11+4c6fO1uCejBVMHvhQ@mail.gmail.com>
Message-ID: <CADiSq7cPquKUKO9bCvELbwjhq9xnt2XYsGpnTkOEanZ3DHfh_A@mail.gmail.com>

On 2 January 2017 at 18:27, Nathaniel Smith <njs at pobox.com> wrote:

> So I guess that yeah, my suggestion would be to drop this feature
> entirely in 3.7, given that it's never been enabled by default and has
> been largely broken for years. Or do we still need a full deprecation
> cycle?
>

I think the existing warning in the docs and the fact it's apparently been
fundamentally broken for years is sufficient justification for just
dropping it entirely. An explicit deprecation warning could be added in
3.6.1 and a Py3k warning in 2.7.x, though - those changes shouldn't be
difficult, and it's a nice courtesy for anyone that *is* somehow currently
getting it to work.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170102/b3496451/attachment.html>

From guido at python.org  Mon Jan  2 11:21:39 2017
From: guido at python.org (Guido van Rossum)
Date: Mon, 2 Jan 2017 08:21:39 -0800
Subject: [Python-Dev] --with-fpectl changes the CPython ABI
In-Reply-To: <CADiSq7cPquKUKO9bCvELbwjhq9xnt2XYsGpnTkOEanZ3DHfh_A@mail.gmail.com>
References: <CAPJVwB=nUk7YaZTqk8gJcK5RymoPbcJqpZC8oBBFX=1Xdr9cfA@mail.gmail.com>
 <CADiSq7f0NPouBWaCkfLnyQrDWYMq+_23gZDRaXE2ggcWG2AxJQ@mail.gmail.com>
 <CAPJVwB=6fwLZiPnEh6HEJ6iGW1i-bL11+4c6fO1uCejBVMHvhQ@mail.gmail.com>
 <CADiSq7cPquKUKO9bCvELbwjhq9xnt2XYsGpnTkOEanZ3DHfh_A@mail.gmail.com>
Message-ID: <CAP7+vJ+L1iOMSzX+q2ekYTjy__xE_kaC8S_562W4YfLuNczF9g@mail.gmail.com>

I am happy to see it go. It was contributed many, many years ago by people
(scientists from the early numpy world IIRC) who had a very specific use
for it, but weren't really into maintaining it long-term, and I wasn't
strong enough to refuse a well-meaning but poorly executed contribution at
the time -- so we compromised on having the whole thing enabled through
`#ifdef`. Clearly it started rotting the day I committed the code...

On Mon, Jan 2, 2017 at 4:22 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 2 January 2017 at 18:27, Nathaniel Smith <njs at pobox.com> wrote:
>
>> So I guess that yeah, my suggestion would be to drop this feature
>> entirely in 3.7, given that it's never been enabled by default and has
>> been largely broken for years. Or do we still need a full deprecation
>> cycle?
>>
>
> I think the existing warning in the docs and the fact it's apparently been
> fundamentally broken for years is sufficient justification for just
> dropping it entirely. An explicit deprecation warning could be added in
> 3.6.1 and a Py3k warning in 2.7.x, though - those changes shouldn't be
> difficult, and it's a nice courtesy for anyone that *is* somehow currently
> getting it to work.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170102/5b22ba0c/attachment.html>

From njs at pobox.com  Mon Jan  2 19:16:44 2017
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 2 Jan 2017 16:16:44 -0800
Subject: [Python-Dev] --with-fpectl changes the CPython ABI
In-Reply-To: <CAP7+vJ+L1iOMSzX+q2ekYTjy__xE_kaC8S_562W4YfLuNczF9g@mail.gmail.com>
References: <CAPJVwB=nUk7YaZTqk8gJcK5RymoPbcJqpZC8oBBFX=1Xdr9cfA@mail.gmail.com>
 <CADiSq7f0NPouBWaCkfLnyQrDWYMq+_23gZDRaXE2ggcWG2AxJQ@mail.gmail.com>
 <CAPJVwB=6fwLZiPnEh6HEJ6iGW1i-bL11+4c6fO1uCejBVMHvhQ@mail.gmail.com>
 <CADiSq7cPquKUKO9bCvELbwjhq9xnt2XYsGpnTkOEanZ3DHfh_A@mail.gmail.com>
 <CAP7+vJ+L1iOMSzX+q2ekYTjy__xE_kaC8S_562W4YfLuNczF9g@mail.gmail.com>
Message-ID: <CAPJVwBnQArYP0TrO_V+aKUq6QhQ2MDx6BpeQR_oThH3kCFqbJg@mail.gmail.com>

Great, sounds like we have a plan: https://bugs.python.org/issue29137

On Mon, Jan 2, 2017 at 8:21 AM, Guido van Rossum <guido at python.org> wrote:
> I am happy to see it go. It was contributed many, many years ago by people
> (scientists from the early numpy world IIRC) who had a very specific use for
> it, but weren't really into maintaining it long-term, and I wasn't strong
> enough to refuse a well-meaning but poorly executed contribution at the time
> -- so we compromised on having the whole thing enabled through `#ifdef`.
> Clearly it started rotting the day I committed the code...
>
> On Mon, Jan 2, 2017 at 4:22 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> On 2 January 2017 at 18:27, Nathaniel Smith <njs at pobox.com> wrote:
>>>
>>> So I guess that yeah, my suggestion would be to drop this feature
>>> entirely in 3.7, given that it's never been enabled by default and has
>>> been largely broken for years. Or do we still need a full deprecation
>>> cycle?
>>
>>
>> I think the existing warning in the docs and the fact it's apparently been
>> fundamentally broken for years is sufficient justification for just dropping
>> it entirely. An explicit deprecation warning could be added in 3.6.1 and a
>> Py3k warning in 2.7.x, though - those changes shouldn't be difficult, and
>> it's a nice courtesy for anyone that *is* somehow currently getting it to
>> work.
>>
>> Cheers,
>> Nick.
>>
>> --
>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)



-- 
Nathaniel J. Smith -- https://vorpus.org

From larry at hastings.org  Mon Jan  2 21:20:56 2017
From: larry at hastings.org (Larry Hastings)
Date: Mon, 2 Jan 2017 18:20:56 -0800
Subject: [Python-Dev] [RELEASED] Python 3.4.6rc1 and Python 3.5.3rc1 are now
 available
Message-ID: <4437910e-0f87-2d2d-062e-8e4fc0be2a58@hastings.org>


On behalf of the Python development community and the Python 3.4 and 
Python 3.5 release teams, I'm pleased to announce the availability of 
Python 3.4.6rc1 and Python 3.5.6rc1.

Python 3.4 is now in "security fixes only" mode.  This is the final 
stage of support for Python 3.4.  Python 3.4 now only receives security 
fixes, not bug fixes, and Python 3.4 releases are source code only--no 
more official binary installers will be produced.

Python 3.5 is still in active "bug fix" mode.  Python 3.5.3rc1 contains 
many incremental improvements over Python 3.5.2.

Both these releases are "release candidates".  They should not be 
considered the final releases, although the final releases should 
contain only minor differences.  Python users are encouraged to test 
with these releases and report any problems they encounter.


You can find Python 3.4.6rc1 here:

    https://www.python.org/downloads/release/python-346rc1/

And you can find Python 3.5.3rc1 here:

    https://www.python.org/downloads/release/python-353rc1/


Python 3.4.6 final and Python 3.5.3 final are both scheduled for release 
on January 16th, 2017.


Happy New Year,


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170102/b8ef3a57/attachment.html>

From rymg19 at gmail.com  Wed Jan  4 14:59:14 2017
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Wed, 4 Jan 2017 13:59:14 -0600
Subject: [Python-Dev] The "humor" section of the website is a bit outdated...
In-Reply-To: <CAO41-mNHyjXFr7sCQ52POS92=P-2EdDfrsZfrc-yrCptWbNM8A@mail.gmail.com>
References: <CAO41-mNYOx1P15_qrRf=T9Z2Dq_gEdvYZhFY0daC6mJQgE8Mcg@mail.gmail.com>
 <CAO41-mPistkJ=hLoo_m85P2vyYkRRL2chJJD4gNMvvYwD-oy7w@mail.gmail.com>
 <CAO41-mNPMxt1yjr0gZw-g-f-5t0aoBoSc-R+-du2sGGbti6azQ@mail.gmail.com>
 <CAO41-mMQ-FqMpKDxcH9+XvvihenF99kiw-gSOX27qpLQkmCxAQ@mail.gmail.com>
 <CAO41-mMOj1Lph=82m5muweT0dhKtMyMHQZUbegmq7Oi4Nw0TLQ@mail.gmail.com>
 <CAO41-mNVv7F=8njg7T5SD_mMsg6vpx1r98SF4rNAOWsJsR5jNw@mail.gmail.com>
 <CAO41-mN2YXzLYSRvd6iKWYbNL4xxeXH2CqNqpZcVghynfU+XGQ@mail.gmail.com>
 <CAO41-mNW=wiMuT=ifA=UFZPH0zPdp_ShRj2dcKP41GT0T4Jifg@mail.gmail.com>
 <CAO41-mO89AtwNMHQHKHhQUzRgu9YeW1C23mN1oMSNDEdDnJ8GA@mail.gmail.com>
 <CAO41-mPeBPdNw18yw-GjuLNBg+DLtpBFK1MRYkSbae3=+RsCqA@mail.gmail.com>
 <CAO41-mNHyjXFr7sCQ52POS92=P-2EdDfrsZfrc-yrCptWbNM8A@mail.gmail.com>
Message-ID: <CAO41-mN=tfbY==hvdMTS-tKS6nOkp36gbeDDqR1qdp_5fm50=g@mail.gmail.com>

Ok, in advance, my apologies for probably sending this to the wrong mailing
list...

Anyway, the most holy page in the entire Python website, containing all
sorts of deadly and wicked items, seems to have gathered a couple pounds of
dust:

https://www.python.org/doc/humor/

And by "pounds of dust", I mean that 2 of the links don't work anymore:

http://quotations.amk.ca/python-quotes/

seems to have been down for at least 3 years; available from archive.org (
http://web.archive.org/web/20111111175522/http://quotations.amk.ca/python-quotes/
)

The Python humor wiki page is also gone:

https://wiki.python.org/PythonHumor

Being that the primary purpose is for proposing additions to the humor
list, I don't think archive.org would be of much help for this one.

--
Ryan (????)
Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170104/acc438fe/attachment.html>

From songofacandy at gmail.com  Wed Jan  4 18:27:13 2017
From: songofacandy at gmail.com (INADA Naoki)
Date: Thu, 5 Jan 2017 08:27:13 +0900
Subject: [Python-Dev] IRC logs via BotBot.me
Message-ID: <CAEfz+Ty0=DBWow4oTxWiQr1gy5TxhMkrqVMYXkWaM8xVc9oq_Q@mail.gmail.com>

Hi.

IRC #python-dev channel is nice place to know what happens recently.
But I can't log in always because I use only laptop PC.

I found BotBot.me seems nice IRC log service and used by some major channels.
https://botbot.me/

I wonder if #python-dev is logged by BotBot.me.

I'm sorry if it had rejected already.

From senthil at uthcode.com  Wed Jan  4 18:43:38 2017
From: senthil at uthcode.com (Senthil Kumaran)
Date: Wed, 4 Jan 2017 15:43:38 -0800
Subject: [Python-Dev] IRC logs via BotBot.me
In-Reply-To: <CAEfz+Ty0=DBWow4oTxWiQr1gy5TxhMkrqVMYXkWaM8xVc9oq_Q@mail.gmail.com>
References: <CAEfz+Ty0=DBWow4oTxWiQr1gy5TxhMkrqVMYXkWaM8xVc9oq_Q@mail.gmail.com>
Message-ID: <CAPOVWORDSjNC3OXiWvOgt_W5SBZpLnUR1u75kTn8LKdVB_bvrw@mail.gmail.com>

On Wed, Jan 4, 2017 at 3:27 PM, INADA Naoki <songofacandy at gmail.com> wrote:
> I wonder if #python-dev is logged by BotBot.me.
>
> I'm sorry if it had rejected already.

I don't think so, but channel ops could request it.

Also, I found (https://www.irccloud.com) to be helpful to look at IRC
logs when away. (You can also permanently stay connected for a paid
service, afaict, they have put some interesting work into IRC over
web).

-- 
Senthil

From berker.peksag at gmail.com  Wed Jan  4 19:04:38 2017
From: berker.peksag at gmail.com (=?UTF-8?Q?Berker_Peksa=C4=9F?=)
Date: Thu, 5 Jan 2017 03:04:38 +0300
Subject: [Python-Dev] IRC logs via BotBot.me
In-Reply-To: <CAEfz+Ty0=DBWow4oTxWiQr1gy5TxhMkrqVMYXkWaM8xVc9oq_Q@mail.gmail.com>
References: <CAEfz+Ty0=DBWow4oTxWiQr1gy5TxhMkrqVMYXkWaM8xVc9oq_Q@mail.gmail.com>
Message-ID: <CAF4280LkhuGfPt2jefJsRNAFqiFZvqOXyQyvumi5aUb4ZEY5hA@mail.gmail.com>

On Thu, Jan 5, 2017 at 2:27 AM, INADA Naoki <songofacandy at gmail.com> wrote:
> Hi.
>
> IRC #python-dev channel is nice place to know what happens recently.
> But I can't log in always because I use only laptop PC.
>
> I found BotBot.me seems nice IRC log service and used by some major channels.
> https://botbot.me/
>
> I wonder if #python-dev is logged by BotBot.me.
>
> I'm sorry if it had rejected already.

I'm using my personal VPS to stay online on Freenode. This looks like
a tooling issue on your end and it can be solved without introducing a
public logging service.

--Berker

From berker.peksag at gmail.com  Wed Jan  4 19:09:18 2017
From: berker.peksag at gmail.com (=?UTF-8?Q?Berker_Peksa=C4=9F?=)
Date: Thu, 5 Jan 2017 03:09:18 +0300
Subject: [Python-Dev] The "humor" section of the website is a bit
 outdated...
In-Reply-To: <CAO41-mN=tfbY==hvdMTS-tKS6nOkp36gbeDDqR1qdp_5fm50=g@mail.gmail.com>
References: <CAO41-mNYOx1P15_qrRf=T9Z2Dq_gEdvYZhFY0daC6mJQgE8Mcg@mail.gmail.com>
 <CAO41-mPistkJ=hLoo_m85P2vyYkRRL2chJJD4gNMvvYwD-oy7w@mail.gmail.com>
 <CAO41-mNPMxt1yjr0gZw-g-f-5t0aoBoSc-R+-du2sGGbti6azQ@mail.gmail.com>
 <CAO41-mMQ-FqMpKDxcH9+XvvihenF99kiw-gSOX27qpLQkmCxAQ@mail.gmail.com>
 <CAO41-mMOj1Lph=82m5muweT0dhKtMyMHQZUbegmq7Oi4Nw0TLQ@mail.gmail.com>
 <CAO41-mNVv7F=8njg7T5SD_mMsg6vpx1r98SF4rNAOWsJsR5jNw@mail.gmail.com>
 <CAO41-mN2YXzLYSRvd6iKWYbNL4xxeXH2CqNqpZcVghynfU+XGQ@mail.gmail.com>
 <CAO41-mNW=wiMuT=ifA=UFZPH0zPdp_ShRj2dcKP41GT0T4Jifg@mail.gmail.com>
 <CAO41-mO89AtwNMHQHKHhQUzRgu9YeW1C23mN1oMSNDEdDnJ8GA@mail.gmail.com>
 <CAO41-mPeBPdNw18yw-GjuLNBg+DLtpBFK1MRYkSbae3=+RsCqA@mail.gmail.com>
 <CAO41-mNHyjXFr7sCQ52POS92=P-2EdDfrsZfrc-yrCptWbNM8A@mail.gmail.com>
 <CAO41-mN=tfbY==hvdMTS-tKS6nOkp36gbeDDqR1qdp_5fm50=g@mail.gmail.com>
Message-ID: <CAF4280+VuSiW57Qp3=NmrYPcZd5Nj70h8S-2XVHTr_PfOzww6A@mail.gmail.com>

On Wed, Jan 4, 2017 at 10:59 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
> Ok, in advance, my apologies for probably sending this to the wrong mailing
> list...

Hi,

https://github.com/python/pythondotorg/issues is a better place to
report content issues on python.org.

--Berker

From hpj at urpla.net  Wed Jan  4 19:28:27 2017
From: hpj at urpla.net (Hans-Peter Jansen)
Date: Thu, 05 Jan 2017 01:28:27 +0100
Subject: [Python-Dev] ctypes, memory mapped files and context manager
Message-ID: <1761734.RoB7TtUa4s@xrated>

Hi,

first of all, sorry for being such a pest, but all former attempts to solve 
this issue on other fora has been grinding to a halt.

In short: I try to combine a context manager with ctypes structures on memory 
mapped files in order to manage huge binary files. (An approach, that performs 
great, while easy to use and keeping the resource usage low).

FWIW, the code is targeted for Linux environments running Python3.

The smallest script demonstrating the issue (thanks to Peter Otten):

import ctypes
import mmap

from contextlib import contextmanager

class T(ctypes.Structure):
    _fields = [("foo", ctypes.c_uint32)]


@contextmanager
def map_struct(m, n):
    m.resize(n * mmap.PAGESIZE)
    yield T.from_buffer(m)

SIZE = mmap.PAGESIZE * 2
f = open("tmp.dat", "w+b")
f.write(b"\0" * SIZE)
f.seek(0)
m = mmap.mmap(f.fileno(), mmap.PAGESIZE)

with map_struct(m, 1) as a:
    a.foo = 1
with map_struct(m, 2) as b:
    b.foo = 2


resulting in:
$ python3 mmap_test.py 
Traceback (most recent call last):
  File "mmap_test.py", line 23, in <module>
    with map_struct(m, 2) as b:
  File "/usr/lib64/python3.4/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "mmap_test.py", line 12, in map_struct
    m.resize(n * mmap.PAGESIZE)
BufferError: mmap can't resize with extant buffers exported.


Python2 does not crash, but that's a different story. What happens here is: 
the context manager variable "a" keeps a reference to a memory mapped area 
alive, that results in a unresizable and not properly closable mmap.

Right now, this rather ugly and error prone workaround must be used, that 
renders the purpose of the context manager ad absurdum:

with map_struct(m, 1) as a:
    a.foo = 1
del a
with map_struct(m, 2) as b:
    b.foo = 2
del b

In order to get this working properly, the ctypes mapping needs a method to 
free the mapping actively. E.g.:

@contextmanager
def map_struct(m, n):
    m.resize(n * mmap.PAGESIZE)
    yield T.from_buffer(m)
	 T.unmap_buffer(m)

Other attempts with weakref and the like do not work due to the nature of the 
ctypes types.

My own investigations in the _ctypes module were unsuccessful so far.

Hopefully, somebody in the audience cares enough for this module in order to 
get this fixed up (or probably I'm missing something obvious..).

Any ideas are very much appreciated.

Pete

From ncoghlan at gmail.com  Wed Jan  4 21:37:20 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 5 Jan 2017 12:37:20 +1000
Subject: [Python-Dev] ctypes, memory mapped files and context manager
In-Reply-To: <1761734.RoB7TtUa4s@xrated>
References: <1761734.RoB7TtUa4s@xrated>
Message-ID: <CADiSq7fmug4fgXx0ogSL-v2sKZF2k-DtW9mP=PY79S95=SsxMA@mail.gmail.com>

On 5 January 2017 at 10:28, Hans-Peter Jansen <hpj at urpla.net> wrote:
> In order to get this working properly, the ctypes mapping needs a method to
> free the mapping actively. E.g.:
>
> @contextmanager
> def map_struct(m, n):
>     m.resize(n * mmap.PAGESIZE)
>     yield T.from_buffer(m)
>          T.unmap_buffer(m)
>
> Other attempts with weakref and the like do not work due to the nature of the
> ctypes types.

I don't know ctypes well enough myself to comment on the idea of
offering fully deterministic cleanup, but the closest you could get to
that without requiring a change to ctypes is to have the context
manager introduce a layer of indirection:

    class _T_data(ctypes.Structure):
        _fields = [("foo", ctypes.c_uint32)]

    class T:
        def __init__(self, buffer):
            self.data = _T_data.from_buffer(buffer)
        def close(self):
            self.data = None

    @contextmanager
    def map_struct(m, n):
        m.resize(n * mmap.PAGESIZE)
        mapped = T(m)
        try:
            yield mapped
        finally:
            mapped.close()

Client code would then need to consistently access the struct through
the data attribute:

    with map_struct(m, 1) as a:
        a.data.foo = 1
    with map_struct(m, 2) as b:
        b.data.foo = 2

Something like http://wrapt.readthedocs.io/en/latest/wrappers.html#object-proxy
would let you make the indirection to a contained object transparent,
but as far as I can tell, wrapt doesn't currently support "closing" a
proxy by replacing the reference to the internal object with a
reference to None (adding that might be possible, but I don't
personally know wrapt well enough to guess the feasibility of doing
so).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From tjreedy at udel.edu  Wed Jan  4 23:43:40 2017
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 4 Jan 2017 23:43:40 -0500
Subject: [Python-Dev] SystemError: new style getargs format but argument
 is not a tuple
In-Reply-To: <o4c41i$ain$1@blaine.gmane.org>
References: <o4c31n$kq1$1@blaine.gmane.org> <o4c41i$ain$1@blaine.gmane.org>
Message-ID: <o4kitp$m4j$1@blaine.gmane.org>

On 1/1/2017 6:40 PM, Serhiy Storchaka wrote:
> On 02.01.17 01:23, Terry Reedy wrote:
>> There are several recent question on Stackoverflow about
>>
>> SystemError: new style getargs format but argument is not a tuple

[snip]
Resulting from using 3rd party packages.

>> No one commenting has a clue.  Is message from Victor's new calling
>> code?  Was it backported to 2.7?  Could error be a result of running old
>> 3rd party binary that needs to be recompiled?
>
> A system error "new style getargs format but argument is not a tuple" is
> not new. It means that PyArg_ParseTuple() is called with not a tuple as
> the first arguments. This is just a programmical error in third-party
> extension.

Should the advice in the doc entry for SystemError be changed to include 
this case?

"    You should report this to the author or maintainer of your Python 
interpreter. Be sure to report the version of the Python interpreter 
(sys.version; it is also printed at the start of an interactive Python 
session), the exact error message (the exception?s associated value) and 
if possible the source of the program that triggered the error.
"

-- 
Terry Jan Reedy



From storchaka at gmail.com  Thu Jan  5 05:55:36 2017
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 5 Jan 2017 12:55:36 +0200
Subject: [Python-Dev] SystemError: new style getargs format but argument
 is not a tuple
In-Reply-To: <o4kitp$m4j$1@blaine.gmane.org>
References: <o4c31n$kq1$1@blaine.gmane.org> <o4c41i$ain$1@blaine.gmane.org>
 <o4kitp$m4j$1@blaine.gmane.org>
Message-ID: <o4l8n4$1oa$1@blaine.gmane.org>

On 05.01.17 06:43, Terry Reedy wrote:
> On 1/1/2017 6:40 PM, Serhiy Storchaka wrote:
>> On 02.01.17 01:23, Terry Reedy wrote:
>>> There are several recent question on Stackoverflow about
>>>
>>> SystemError: new style getargs format but argument is not a tuple
>
> [snip]
> Resulting from using 3rd party packages.
>
>>> No one commenting has a clue.  Is message from Victor's new calling
>>> code?  Was it backported to 2.7?  Could error be a result of running old
>>> 3rd party binary that needs to be recompiled?
>>
>> A system error "new style getargs format but argument is not a tuple" is
>> not new. It means that PyArg_ParseTuple() is called with not a tuple as
>> the first arguments. This is just a programmical error in third-party
>> extension.
>
> Should the advice in the doc entry for SystemError be changed to include
> this case?
>
> "    You should report this to the author or maintainer of your Python
> interpreter. Be sure to report the version of the Python interpreter
> (sys.version; it is also printed at the start of an interactive Python
> session), the exact error message (the exception?s associated value) and
> if possible the source of the program that triggered the error.
> "

Yes, third-party extensions should be mentioned as a source of 
SystemError. But note that nobody reported this error even on CPython 
bugtracker. I now found questions about this error (related to OpenCV or 
Pillow) asked many years ago on different forums.


From hpj at urpla.net  Thu Jan  5 09:09:48 2017
From: hpj at urpla.net (Hans-Peter Jansen)
Date: Thu, 05 Jan 2017 15:09:48 +0100
Subject: [Python-Dev] ctypes, memory mapped files and context manager
In-Reply-To: <CADiSq7fmug4fgXx0ogSL-v2sKZF2k-DtW9mP=PY79S95=SsxMA@mail.gmail.com>
References: <1761734.RoB7TtUa4s@xrated>
 <CADiSq7fmug4fgXx0ogSL-v2sKZF2k-DtW9mP=PY79S95=SsxMA@mail.gmail.com>
Message-ID: <4774442.SbkXyHSDfZ@xrated>

Hi Nick,

On Donnerstag, 5. Januar 2017 12:37:20 Nick Coghlan wrote:
> I don't know ctypes well enough myself to comment on the idea of
> offering fully deterministic cleanup, but the closest you could get to
> that without requiring a change to ctypes is to have the context
> manager introduce a layer of indirection:
> 
>     class _T_data(ctypes.Structure):
>         _fields = [("foo", ctypes.c_uint32)]
> 
>     class T:
>         def __init__(self, buffer):
>             self.data = _T_data.from_buffer(buffer)
>         def close(self):
>             self.data = None
> 
>     @contextmanager
>     def map_struct(m, n):
>         m.resize(n * mmap.PAGESIZE)
>         mapped = T(m)
>         try:
>             yield mapped
>         finally:
>             mapped.close()
> 
> Client code would then need to consistently access the struct through
> the data attribute:
> 
>     with map_struct(m, 1) as a:
>         a.data.foo = 1
>     with map_struct(m, 2) as b:
>         b.data.foo = 2

Thank you very much. Nice idea, indeed. 

Here's a slightly more complex example incorporating your idea:
https://gist.github.com/frispete/97c27e24a0aae1bcaf1375e2e463d239#file-ctypes_mmap_ctx2-py

I'm not sure, if I like the resulting code more than the dreaded dels.
Real code based on this approach tends to be much more complex, and 
suffers appropriately.

Anyway, your solution is working fine, and provides a choice.
Much appreciated, Nick.

@ctypes developers: with an unmap operation available, we wouldn't need 
to go through these hoops, and ctypes powers would become even more 
accessible for such cool use cases ;)...

For now, I'm about to resign from using a context manager at all, since 
it uglifies the code in one way or another without buying much..

Cheers,
Pete

From eryksun at gmail.com  Thu Jan  5 10:30:33 2017
From: eryksun at gmail.com (eryk sun)
Date: Thu, 5 Jan 2017 15:30:33 +0000
Subject: [Python-Dev] ctypes, memory mapped files and context manager
In-Reply-To: <CADiSq7fmug4fgXx0ogSL-v2sKZF2k-DtW9mP=PY79S95=SsxMA@mail.gmail.com>
References: <1761734.RoB7TtUa4s@xrated>
 <CADiSq7fmug4fgXx0ogSL-v2sKZF2k-DtW9mP=PY79S95=SsxMA@mail.gmail.com>
Message-ID: <CACL+1atSp4mC8o6OMHShVAWJfMp2W5DoSaRqByBypp0OtJq7Yw@mail.gmail.com>

On Thu, Jan 5, 2017 at 2:37 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 5 January 2017 at 10:28, Hans-Peter Jansen <hpj at urpla.net> wrote:
>> In order to get this working properly, the ctypes mapping needs a method to
>> free the mapping actively. E.g.:
>>
>> @contextmanager
>> def map_struct(m, n):
>>     m.resize(n * mmap.PAGESIZE)
>>     yield T.from_buffer(m)
>>          T.unmap_buffer(m)
>>
>> Other attempts with weakref and the like do not work due to the nature of the
>> ctypes types.
>
> I don't know ctypes well enough myself to comment on the idea of
> offering fully deterministic cleanup, but the closest you could get to
> that without requiring a change to ctypes is to have the context
> manager introduce a layer of indirection:

I think that's the best you can do with the current state of ctypes.

from_buffer was made safer in Python 3 by ensuring it keeps a
memoryview reference in the _objects attribute (i.e.
CDataObject.b_objects). Hans-Peter's problem is a consequence of this
reference. Simply calling release() on the underlying memoryview is
unsafe. For example:

    >>> b = bytearray(2**20)
    >>> a = ctypes.c_char.from_buffer(b)
    >>> a._objects
    <memory at 0x7f04283b8dc8>
    >>> a._objects.release()
    >>> del b
    >>> a.value
    Segmentation fault (core dumped)

A release() method on ctypes objects could release the memoryview and
also clear the CDataObject b_ptr field. In this case, any function
that accesses b_ptr would have to be modified to raise a ValueError
for a NULL value. Currently ctypes assumes b_ptr is valid, so this
would require adding a lot of checks.

On a related note, ctypes objects aren't tracking the number of
exported views like they should. resize() should raise a BufferError
in the following example:

    >>> b = (ctypes.c_char * (2**20))(255)
    >>> m = memoryview(b).cast('B')
    >>> m[0]
    255
    >>> ctypes.resize(b, 2**22)
    >>> m[0]
    Segmentation fault (core dumped)

From milan at oberkirch.org  Thu Jan  5 04:10:28 2017
From: milan at oberkirch.org (Milan Oberkirch)
Date: Thu, 5 Jan 2017 20:10:28 +1100
Subject: [Python-Dev] IRC logs via BotBot.me
In-Reply-To: <CAF4280LkhuGfPt2jefJsRNAFqiFZvqOXyQyvumi5aUb4ZEY5hA@mail.gmail.com>
References: <CAEfz+Ty0=DBWow4oTxWiQr1gy5TxhMkrqVMYXkWaM8xVc9oq_Q@mail.gmail.com>
 <CAF4280LkhuGfPt2jefJsRNAFqiFZvqOXyQyvumi5aUb4ZEY5hA@mail.gmail.com>
Message-ID: <6c1bff10-21c7-b35e-66a8-c04dfbb89eb4@oberkirch.org>

On 01/05/2017 11:04 AM, Berker Peksa? wrote:
> On Thu, Jan 5, 2017 at 2:27 AM, INADA Naoki <songofacandy at gmail.com> wrote:
>> Hi.
>>
>> IRC #python-dev channel is nice place to know what happens recently.
>> But I can't log in always because I use only laptop PC.
>>
>> I found BotBot.me seems nice IRC log service and used by some major channels.
>> https://botbot.me/
>>
>> I wonder if #python-dev is logged by BotBot.me.
>>
>> I'm sorry if it had rejected already.
> I'm using my personal VPS to stay online on Freenode. This looks like
> a tooling issue on your end and it can be solved without introducing a
> public logging service.
>
> --Berker

I think it makes sense to solve this issue with a public log instead of
requiring everyone to come up with their own solutions -- for the same
reasons we have public mailing-list archives.


From alexander.belopolsky at gmail.com  Thu Jan  5 15:37:18 2017
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Thu, 5 Jan 2017 15:37:18 -0500
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
Message-ID: <CAP7h-xY3s4Dm0Gb-rComBQnRZcL=DBhFf9iWaAkg82MYhkjd7A@mail.gmail.com>

On Wed, Oct 12, 2016 at 12:08 AM, INADA Naoki <songofacandy at gmail.com>
wrote:
>
> Now I'm sure about bytes.frombuffer() is worth enough.

I would like to revive this thread (taking a liberty to shorten the subject
line.)

The issue of how the bytes(x) constructor should behave when given objects
of various types have come up recently in issue 29159 (Regression in bytes
constructor). [1]

The regression was introduced in issue 27704  (bytes(x) is slow when x is
bytearray) which attempted to speed-up creating bytes and bytearray from
byte-like objects.

I think the core problem is that the bytes(x) constructor tries to be the
Jack of All Trades.  Here is how it is documented in the docstring:

 |  Construct an immutable array of bytes from:
 |    - an iterable yielding integers in range(256)
 |    - a text string encoded using the specified encoding
 |    - any object implementing the buffer API.
 |    - an integer
 |

On the other hand, the reference manual while not having this description
in the bytes section, has a similar list in the bytearray section. [3]

"""
The optional source parameter can be used to initialize the array in a few
different ways:

  * If it is a string, you must also give the encoding (and optionally,
errors) parameters; bytearray() then converts the string to bytes using
str.encode().
  * If it is an integer, the array will have that size and will be
initialized with null bytes.
  * If it is an object conforming to the buffer interface, a read-only
buffer of the object will be used to initialize the bytes array.
  * If it is an iterable, it must be an iterable of integers in the range 0
<= x < 256, which are used as the initial contents of the array.

Without an argument, an array of size 0 is created.
"""

Note that the integer case is listed before buffer interface.  Neither
document mentions the possibility that the source type has a __bytes__
method.

This ambiguity between integer-like and buffer-like sources causes a
problem in the case when a 3rd party type is both integer-like and
buffer-like.  This is what happens with numpy arrays:

>>> bytes(numpy.array([2], 'i1'))
b'\x00\x00'

>>> bytes(numpy.array([2, 2], 'i1'))
b'\x02\x02'

For better or worse, single-element numpy arrays have a working __index__
methods

>>> numpy.array([2], 'i1').__index__()
2

and are interpreted as integers by the bytes(X) constructor.

I propose the following:

1. For 3.6, restore and document 3.5 behavior.  Recommend that 3rd party
types that are both integer-like and buffer-like implement their own
__bytes__ method to resolve the bytes(x) ambiguity.

2. For 3.7, I would like to see a drastically simplified bytes(x):
2.1.  Accept only objects with a __bytes__ method or a sequence of ints in
range(256).
2.2.  Expand __bytes__ definition to accept optional encoding and errors
parameters.  Implement str.__bytes__(self, [encoding[, errors]]).
2.3.  Implement new specialized bytes.fromsize and bytes.frombuffer
constructors as per PEP 467 and Inada Naoki proposals.
2.4. Implement memoryview.__bytes__ method so that bytes(memoryview(x))
works ad before.
2.5.  Implement a fast bytearray.__bytes__ method.

3. Consider promoting __bytes__ to a tp_bytes type slot.

[1]: http://bugs.python.org/issue29159
[2]: http://bugs.python.org/issue27704
[3]: https://docs.python.org/3/library/functions.html#bytearray
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170105/b35edd74/attachment.html>

From storchaka at gmail.com  Thu Jan  5 17:54:57 2017
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Fri, 6 Jan 2017 00:54:57 +0200
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
In-Reply-To: <CAP7h-xY3s4Dm0Gb-rComBQnRZcL=DBhFf9iWaAkg82MYhkjd7A@mail.gmail.com>
References: <CAP7h-xY3s4Dm0Gb-rComBQnRZcL=DBhFf9iWaAkg82MYhkjd7A@mail.gmail.com>
Message-ID: <o4mirt$fuv$1@blaine.gmane.org>

On 05.01.17 22:37, Alexander Belopolsky wrote:
> I propose the following:
>
> 1. For 3.6, restore and document 3.5 behavior.  Recommend that 3rd party
> types that are both integer-like and buffer-like implement their own
> __bytes__ method to resolve the bytes(x) ambiguity.

The __bytes__ method is used only by the bytes constructor, not by the 
bytearray constructor.

> 2. For 3.7, I would like to see a drastically simplified bytes(x):
> 2.1.  Accept only objects with a __bytes__ method or a sequence of ints
> in range(256).
> 2.2.  Expand __bytes__ definition to accept optional encoding and errors
> parameters.  Implement str.__bytes__(self, [encoding[, errors]]).

I think it is better to use the encode() method if you want to encode 
from non-strings.

> 2.3.  Implement new specialized bytes.fromsize and bytes.frombuffer
> constructors as per PEP 467 and Inada Naoki proposals.

bytes.fromsize(n) is just b'\0'*n. I don't think this method is needed.

bytes.frombuffer(x) is bytes(memoryview(x)) or memoryview(x).tobytes().

> 2.4. Implement memoryview.__bytes__ method so that bytes(memoryview(x))
> works ad before.
> 2.5.  Implement a fast bytearray.__bytes__ method.

This wouldn't help for the bytearray constructor. And wouldn't allow to 
avoid double copying in the constructor of bytes subclass.

> 3. Consider promoting __bytes__ to a tp_bytes type slot.

The buffer protocol is more general than the __bytes__ method. It allows 
to avoid redundant memory copying in constructors of many types (bytes, 
bytearray, array.array, etc), not just bytes.



From hpj at urpla.net  Thu Jan  5 18:28:37 2017
From: hpj at urpla.net (Hans-Peter Jansen)
Date: Fri, 06 Jan 2017 00:28:37 +0100
Subject: [Python-Dev] ctypes, memory mapped files and context manager
In-Reply-To: <CACL+1atSp4mC8o6OMHShVAWJfMp2W5DoSaRqByBypp0OtJq7Yw@mail.gmail.com>
References: <1761734.RoB7TtUa4s@xrated>
 <CADiSq7fmug4fgXx0ogSL-v2sKZF2k-DtW9mP=PY79S95=SsxMA@mail.gmail.com>
 <CACL+1atSp4mC8o6OMHShVAWJfMp2W5DoSaRqByBypp0OtJq7Yw@mail.gmail.com>
Message-ID: <10601506.CGpYZZeef7@xrated>

Hi Eryk,

On Donnerstag, 5. Januar 2017 15:30:33 eryk sun wrote:
> 
> > manager introduce a layer of indirection:
> I think that's the best you can do with the current state of ctypes.
> 
> from_buffer was made safer in Python 3 by ensuring it keeps a
> memoryview reference in the _objects attribute (i.e.
> CDataObject.b_objects). Hans-Peter's problem is a consequence of this
> reference. Simply calling release() on the underlying memoryview is
> 
> unsafe. For example:
>     >>> b = bytearray(2**20)
>     >>> a = ctypes.c_char.from_buffer(b)
>     >>> a._objects
> 
>     <memory at 0x7f04283b8dc8>
> 
>     >>> a._objects.release()
>     >>> del b
>     >>> a.value
> 
>     Segmentation fault (core dumped)

This is exactly, what I was after:

@contextmanager
def cstructmap(cstruct, mm, offset = 0):
    # resize the mmap (and backing file), if structure exceeds mmap size
    # mmap size must be aligned to mmap.PAGESIZE
    cssize = ctypes.sizeof(cstruct)
    if offset + cssize > mm.size():
        newsize = align(offset + cssize, mmap.PAGESIZE)
        mm.resize(newsize)
    cmap = cstruct.from_buffer(mm, offset)
    try:
        yield cmap
    finally:
        for mv in cmap._objects.values():
            mv.release()

See also:
https://gist.github.com/frispete/97c27e24a0aae1bcaf1375e2e463d239#file-ctypes_mmap_ctx3-py

While technically possible (which is a surprise for me on its own), nothing 
should access the with variable after the block has finished. If that happens, 
a segfault is exactly was it deserves IMHO.

Leaves the question, how stable this "interface" is?
Accessing _objects here belongs to voodoo programming practices of course, but 
the magic is locally limited to just two lines of code, which is acceptable in 
order to get this context manager working without messing with the rest of the 
code.

Opinions?

Thanks,
Pete

From hpj at urpla.net  Thu Jan  5 19:03:30 2017
From: hpj at urpla.net (Hans-Peter Jansen)
Date: Fri, 06 Jan 2017 01:03:30 +0100
Subject: [Python-Dev] ctypes, memory mapped files and context manager
In-Reply-To: <10601506.CGpYZZeef7@xrated>
References: <1761734.RoB7TtUa4s@xrated>
 <CACL+1atSp4mC8o6OMHShVAWJfMp2W5DoSaRqByBypp0OtJq7Yw@mail.gmail.com>
 <10601506.CGpYZZeef7@xrated>
Message-ID: <2575279.lKMPEPTJM8@xrated>

On Freitag, 6. Januar 2017 00:28:37 Hans-Peter Jansen wrote:
> Hi Eryk,
> 
> This is exactly, what I was after:
> 
> @contextmanager
> def cstructmap(cstruct, mm, offset = 0):
>     # resize the mmap (and backing file), if structure exceeds mmap size
>     # mmap size must be aligned to mmap.PAGESIZE
>     cssize = ctypes.sizeof(cstruct)
>     if offset + cssize > mm.size():
>         newsize = align(offset + cssize, mmap.PAGESIZE)
>         mm.resize(newsize)
>     cmap = cstruct.from_buffer(mm, offset)
>     try:
>         yield cmap
>     finally:
>         for mv in cmap._objects.values():
            if isinstance(mv, memoryview):
                mv.release()

It happens, that _objects contain other objects as well...

Cheers,
Pete

From songofacandy at gmail.com  Thu Jan  5 19:11:54 2017
From: songofacandy at gmail.com (INADA Naoki)
Date: Fri, 6 Jan 2017 09:11:54 +0900
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
In-Reply-To: <o4mirt$fuv$1@blaine.gmane.org>
References: <CAP7h-xY3s4Dm0Gb-rComBQnRZcL=DBhFf9iWaAkg82MYhkjd7A@mail.gmail.com>
 <o4mirt$fuv$1@blaine.gmane.org>
Message-ID: <CAEfz+TzREV9ZdRfR16aPuwLrR9XZ-UoTi8D8qeyVQutqqWeXKA@mail.gmail.com>

>
> bytes.frombuffer(x) is bytes(memoryview(x)) or memoryview(x).tobytes().
>

There is pitfall: memoryview should be closed.
So b = bytes.frombuffer(x) is:

with memoryview(x) as m:
    b = bytes(m)
    # or b = m.tobytes()

From yselivanov.ml at gmail.com  Thu Jan  5 20:28:26 2017
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Thu, 5 Jan 2017 20:28:26 -0500
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
In-Reply-To: <CAEfz+TzREV9ZdRfR16aPuwLrR9XZ-UoTi8D8qeyVQutqqWeXKA@mail.gmail.com>
References: <CAP7h-xY3s4Dm0Gb-rComBQnRZcL=DBhFf9iWaAkg82MYhkjd7A@mail.gmail.com>
 <o4mirt$fuv$1@blaine.gmane.org>
 <CAEfz+TzREV9ZdRfR16aPuwLrR9XZ-UoTi8D8qeyVQutqqWeXKA@mail.gmail.com>
Message-ID: <292efaef-7fd1-eb8b-7cdc-fea7f0145d57@gmail.com>



On 2017-01-05 7:11 PM, INADA Naoki wrote:
>> bytes.frombuffer(x) is bytes(memoryview(x)) or memoryview(x).tobytes().
>>
> There is pitfall: memoryview should be closed.
> So b = bytes.frombuffer(x) is:
>
> with memoryview(x) as m:
>      b = bytes(m)
>      # or b = m.tobytes()


Thinking more about this, and after looking at my own code in asyncpg 
and uvloop, I'm now in favor of adding bytes.frombuffer() with the 
proposed signature: ``bytes.frombuffer(byteslike, length=-1, offset=0)``

Inada-san is right, the memoryview should be explicitly released, but 
few do that. Instead, a lot of people simply rely on CPython refcounting 
semantics, which will cause the temporary memoryview be GCed asap.  That 
won't work so flawlessly in PyPy and will cause hard to understand bugs.

Yury

From eryksun at gmail.com  Thu Jan  5 21:36:48 2017
From: eryksun at gmail.com (eryk sun)
Date: Fri, 6 Jan 2017 02:36:48 +0000
Subject: [Python-Dev] ctypes, memory mapped files and context manager
In-Reply-To: <10601506.CGpYZZeef7@xrated>
References: <1761734.RoB7TtUa4s@xrated>
 <CADiSq7fmug4fgXx0ogSL-v2sKZF2k-DtW9mP=PY79S95=SsxMA@mail.gmail.com>
 <CACL+1atSp4mC8o6OMHShVAWJfMp2W5DoSaRqByBypp0OtJq7Yw@mail.gmail.com>
 <10601506.CGpYZZeef7@xrated>
Message-ID: <CACL+1auAMwt0pPC9pKEBH8q_kahkvFiojoKT1Ykxdg2bV+a=0A@mail.gmail.com>

On Thu, Jan 5, 2017 at 11:28 PM, Hans-Peter Jansen <hpj at urpla.net> wrote:
> Leaves the question, how stable this "interface" is?
> Accessing _objects here belongs to voodoo programming practices of course, but
> the magic is locally limited to just two lines of code, which is acceptable in
> order to get this context manager working without messing with the rest of the
> code.

My intent was not to suggest that anyone directly use the _objects
value / dict  in production code. It's a private implementation
detail. I was demonstrating the problem of simply releasing the buffer
and the large number of checks that would be required if b_ptr is
cleared. It would be simpler for a release() method to allocate new
memory for the object and set the b_needsfree flag, but this may hide
bugs. Operating on a released object should raise an exception.

From songofacandy at gmail.com  Fri Jan  6 05:38:04 2017
From: songofacandy at gmail.com (INADA Naoki)
Date: Fri, 6 Jan 2017 19:38:04 +0900
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
In-Reply-To: <292efaef-7fd1-eb8b-7cdc-fea7f0145d57@gmail.com>
References: <CAP7h-xY3s4Dm0Gb-rComBQnRZcL=DBhFf9iWaAkg82MYhkjd7A@mail.gmail.com>
 <o4mirt$fuv$1@blaine.gmane.org>
 <CAEfz+TzREV9ZdRfR16aPuwLrR9XZ-UoTi8D8qeyVQutqqWeXKA@mail.gmail.com>
 <292efaef-7fd1-eb8b-7cdc-fea7f0145d57@gmail.com>
Message-ID: <CAEfz+TxbB-3vHT6X=K=x8RWsftP5ObMzoQHh=Dj4NTtWkp3Z2w@mail.gmail.com>

>
> Thinking more about this, and after looking at my own code in asyncpg and
> uvloop, I'm now in favor of adding bytes.frombuffer() with the proposed
> signature: ``bytes.frombuffer(byteslike, length=-1, offset=0)``
>

Do you prefer a signature proposed first?
I thought it from asyncio usage too.  Slicing from head of bytearray
is more common
than offset usage.

But in previous thread, Serhiy and Victor proposed signature same to
old buffer API.

> I like the idea with Serhiy's API (as Python 2 buffer constructor):
>
> bytes.frombuf(buffer, [offset, size])
> bytearray.frombuf(buffer, [offset, size])
> memoryview.frombuf(buffer, [offset, size])

From songofacandy at gmail.com  Fri Jan  6 06:17:57 2017
From: songofacandy at gmail.com (INADA Naoki)
Date: Fri, 6 Jan 2017 20:17:57 +0900
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
In-Reply-To: <CAEfz+TxbB-3vHT6X=K=x8RWsftP5ObMzoQHh=Dj4NTtWkp3Z2w@mail.gmail.com>
References: <CAP7h-xY3s4Dm0Gb-rComBQnRZcL=DBhFf9iWaAkg82MYhkjd7A@mail.gmail.com>
 <o4mirt$fuv$1@blaine.gmane.org>
 <CAEfz+TzREV9ZdRfR16aPuwLrR9XZ-UoTi8D8qeyVQutqqWeXKA@mail.gmail.com>
 <292efaef-7fd1-eb8b-7cdc-fea7f0145d57@gmail.com>
 <CAEfz+TxbB-3vHT6X=K=x8RWsftP5ObMzoQHh=Dj4NTtWkp3Z2w@mail.gmail.com>
Message-ID: <CAEfz+TyBSBvf=qywL5Y+k2RyqSK4X-V8ZODpHJ3wcjYECeysnA@mail.gmail.com>

I submit an issue about it.

See https://bugs.python.org/issue29178

On Fri, Jan 6, 2017 at 7:38 PM, INADA Naoki <songofacandy at gmail.com> wrote:
>>
>> Thinking more about this, and after looking at my own code in asyncpg and
>> uvloop, I'm now in favor of adding bytes.frombuffer() with the proposed
>> signature: ``bytes.frombuffer(byteslike, length=-1, offset=0)``
>>
>
> Do you prefer a signature proposed first?
> I thought it from asyncio usage too.  Slicing from head of bytearray
> is more common
> than offset usage.
>
> But in previous thread, Serhiy and Victor proposed signature same to
> old buffer API.
>
>> I like the idea with Serhiy's API (as Python 2 buffer constructor):
>>
>> bytes.frombuf(buffer, [offset, size])
>> bytearray.frombuf(buffer, [offset, size])
>> memoryview.frombuf(buffer, [offset, size])

From hpj at urpla.net  Fri Jan  6 06:39:29 2017
From: hpj at urpla.net (Hans-Peter Jansen)
Date: Fri, 06 Jan 2017 12:39:29 +0100
Subject: [Python-Dev] Context manager lifetime rules Was: Re: Adding
 bytes.frombuffer() constructor to PEP 467
In-Reply-To: <292efaef-7fd1-eb8b-7cdc-fea7f0145d57@gmail.com>
References: <CAP7h-xY3s4Dm0Gb-rComBQnRZcL=DBhFf9iWaAkg82MYhkjd7A@mail.gmail.com>
 <CAEfz+TzREV9ZdRfR16aPuwLrR9XZ-UoTi8D8qeyVQutqqWeXKA@mail.gmail.com>
 <292efaef-7fd1-eb8b-7cdc-fea7f0145d57@gmail.com>
Message-ID: <2575351.9Y0OTJj6po@xrated>

Hi Yury,

adjusted subject, since I'm dragging the discussion away from it.

On Donnerstag, 5. Januar 2017 20:28:26 Yury Selivanov wrote:
> On 2017-01-05 7:11 PM, INADA Naoki wrote:
> >> bytes.frombuffer(x) is bytes(memoryview(x)) or memoryview(x).tobytes().
> > 
> > There is pitfall: memoryview should be closed.
> > So b = bytes.frombuffer(x) is:
> > 
> > with memoryview(x) as m:
> >      b = bytes(m)
> >      # or b = m.tobytes()
> 
> Thinking more about this, and after looking at my own code in asyncpg
> and uvloop, I'm now in favor of adding bytes.frombuffer() with the
> proposed signature: ``bytes.frombuffer(byteslike, length=-1, offset=0)``
> 
> Inada-san is right, the memoryview should be explicitly released, but
> few do that. Instead, a lot of people simply rely on CPython refcounting
> semantics, which will cause the temporary memoryview be GCed asap.  That
> won't work so flawlessly in PyPy and will cause hard to understand bugs.

Did you noticed, that we're stirring in the same kind of soup lately?
(apart from my person, who's not that deep in the details)

Given above code snippet, my issue is caused from "m" surviving *too* long 
after the context manager block ended, resulting in a delayed release of the 
memoryview, which in turn interferes with subsequent code.

Simple minded, as I am, I would expect, that "m" is actively removed from 
local context as soon as the context manager block ends. I would even argue, 
that this is part of the contract, that the context manager offers here.

AFAICS, "m" is handed over the the GC, but:
	* this is somewhat surprising for the (probably simple minded) programmer
   * leads to problems *already*

A workaround is deleting "m" actively:

    with memoryview(x) as m:
        b = bytes(m)
    del m

I would like to discuss the rationals of this construct from a language design 
POV, not language implementation POV.

usually-hacking-del-free-code-ly y'rs,
Pete

From solipsis at pitrou.net  Fri Jan  6 07:03:46 2017
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 6 Jan 2017 13:03:46 +0100
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
References: <CAP7h-xY3s4Dm0Gb-rComBQnRZcL=DBhFf9iWaAkg82MYhkjd7A@mail.gmail.com>
 <o4mirt$fuv$1@blaine.gmane.org>
 <CAEfz+TzREV9ZdRfR16aPuwLrR9XZ-UoTi8D8qeyVQutqqWeXKA@mail.gmail.com>
 <292efaef-7fd1-eb8b-7cdc-fea7f0145d57@gmail.com>
Message-ID: <20170106130346.066721c8@fsol>

On Thu, 5 Jan 2017 20:28:26 -0500
Yury Selivanov <yselivanov.ml at gmail.com> wrote:
> On 2017-01-05 7:11 PM, INADA Naoki wrote:
> >> bytes.frombuffer(x) is bytes(memoryview(x)) or memoryview(x).tobytes().
> >>  
> > There is pitfall: memoryview should be closed.
> > So b = bytes.frombuffer(x) is:
> >
> > with memoryview(x) as m:
> >      b = bytes(m)
> >      # or b = m.tobytes()  
> 
> Thinking more about this, and after looking at my own code in asyncpg 
> and uvloop, I'm now in favor of adding bytes.frombuffer() with the 
> proposed signature: ``bytes.frombuffer(byteslike, length=-1, offset=0)``

+1

Note this matches numpy.frombuffer():

"""
frombuffer(buffer, dtype=float, count=-1, offset=0)

Interpret a buffer as a 1-dimensional array.
"""

Regards

Antoine.



From songofacandy at gmail.com  Fri Jan  6 07:27:30 2017
From: songofacandy at gmail.com (INADA Naoki)
Date: Fri, 6 Jan 2017 21:27:30 +0900
Subject: [Python-Dev] Context manager lifetime rules Was: Re: Adding
 bytes.frombuffer() constructor to PEP 467
In-Reply-To: <2575351.9Y0OTJj6po@xrated>
References: <CAP7h-xY3s4Dm0Gb-rComBQnRZcL=DBhFf9iWaAkg82MYhkjd7A@mail.gmail.com>
 <CAEfz+TzREV9ZdRfR16aPuwLrR9XZ-UoTi8D8qeyVQutqqWeXKA@mail.gmail.com>
 <292efaef-7fd1-eb8b-7cdc-fea7f0145d57@gmail.com> <2575351.9Y0OTJj6po@xrated>
Message-ID: <CAEfz+Txpr=UnMqedCUggMjXXBVr6UWtGq8_mi2RcYXFtV046pg@mail.gmail.com>

>> > with memoryview(x) as m:
>> >      b = bytes(m)
>> >      # or b = m.tobytes()
>>
>> Thinking more about this, and after looking at my own code in asyncpg
>> and uvloop, I'm now in favor of adding bytes.frombuffer() with the
>> proposed signature: ``bytes.frombuffer(byteslike, length=-1, offset=0)``
>>
>> Inada-san is right, the memoryview should be explicitly released, but
>> few do that. Instead, a lot of people simply rely on CPython refcounting
>> semantics, which will cause the temporary memoryview be GCed asap.  That
>> won't work so flawlessly in PyPy and will cause hard to understand bugs.
>
> Did you noticed, that we're stirring in the same kind of soup lately?
> (apart from my person, who's not that deep in the details)
>
> Given above code snippet, my issue is caused from "m" surviving *too* long
> after the context manager block ended, resulting in a delayed release of the
> memoryview, which in turn interferes with subsequent code.

In case of memoryview, it detaches x.  Like closed file object, it
doesn't hold any resource
other than memory used for the object itself.

>
> Simple minded, as I am, I would expect, that "m" is actively removed from
> local context as soon as the context manager block ends. I would even argue,
> that this is part of the contract, that the context manager offers here.

In case of general context manager, there are some use cases which needs "m"
after context ended.
For example, https://docs.python.org/3.5/library/unittest.html#unittest.TestCase.assertRaises

From status at bugs.python.org  Fri Jan  6 12:09:09 2017
From: status at bugs.python.org (Python tracker)
Date: Fri,  6 Jan 2017 18:09:09 +0100 (CET)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20170106170909.6AC3956AD5@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2016-12-30 - 2017-01-06)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    5685 (+22)
  closed 35253 (+45)
  total  40938 (+67)

Open issues with patches: 2460 


Issues opened (44)
==================

#24725: test_socket testFDPassEmpty fails on OS X 10.11+ with "Cannot 
http://bugs.python.org/issue24725  reopened by ned.deily

#29013: zipfile: inconsistent doc for ZIP64 file size
http://bugs.python.org/issue29013  reopened by serhiy.storchaka

#29116: Make str and bytes error messages on concatenation conform wit
http://bugs.python.org/issue29116  opened by Jim Fasarakis-Hilliard

#29117: dir(<bound method>) should include dunder attributes of the un
http://bugs.python.org/issue29117  opened by Antony.Lee

#29120: Move hash randomisation initialisation out of Python/random.c
http://bugs.python.org/issue29120  opened by ncoghlan

#29121: sqlite3 Controlling Transactions documentation not updated
http://bugs.python.org/issue29121  opened by palaviv

#29124: Freeze fails to compile on windows
http://bugs.python.org/issue29124  opened by Liran Ringel

#29125: Shell injection via TIX_LIBRARY when using tkinter.tix
http://bugs.python.org/issue29125  opened by symphorien

#29126: Fix comments about _PyThreadState_Current
http://bugs.python.org/issue29126  opened by Jim Nasby

#29127: Incorrect reference names in asyncio.subprocess documentation
http://bugs.python.org/issue29127  opened by Eric Ahn

#29132: shlex.shlex with punctuation_chars and posix doesn't handle pu
http://bugs.python.org/issue29132  opened by evan_

#29133: Minor inaccuracy in shlex.shlex punctuation_chars example
http://bugs.python.org/issue29133  opened by evan_

#29136: Add OP_NO_TLSv1_3
http://bugs.python.org/issue29136  opened by christian.heimes

#29137: Fix fpectl-induced ABI breakage
http://bugs.python.org/issue29137  opened by njs

#29139: operator.concat/iconcat could only work if left operand is a s
http://bugs.python.org/issue29139  opened by xiang.zhang

#29140: time_hash() reads the wrong bytes to get microseconds
http://bugs.python.org/issue29140  opened by haypo

#29141: error in 2to3
http://bugs.python.org/issue29141  opened by Sandeep Srinivasa

#29142: urllib: no_proxy variable values with leading dot not properly
http://bugs.python.org/issue29142  opened by tloetzer

#29143: Logger should ignore propagate property for disabled handlers.
http://bugs.python.org/issue29143  opened by Oleg Serov

#29144: Implicit namespace packages in Python 3.6
http://bugs.python.org/issue29144  opened by frenzy

#29145: failing overflow checks in replace_*
http://bugs.python.org/issue29145  opened by matejcik

#29147: registry value to be cleared when python is uninstalled
http://bugs.python.org/issue29147  opened by jha.amit6666

#29151: test_asyncore.TestAPI_UseIPv4Select / test_asyncore.TestAPI_Us
http://bugs.python.org/issue29151  opened by patrila

#29152: unittest subTest does not call addFailure
http://bugs.python.org/issue29152  opened by kristall

#29153: Test test.test_asynchat.TestAsynchat / test.test_asynchat.Test
http://bugs.python.org/issue29153  opened by patrila

#29154: 5 failures in test_doctest: ModuleNotFoundError: No module nam
http://bugs.python.org/issue29154  opened by Gerrit.Holl

#29155: test.test_spwd.TestSpwdNonRoot  failure with FileNotFoundError
http://bugs.python.org/issue29155  opened by Gerrit.Holl

#29157: random.c: Prefer getrandom() over getentropy(), handle ENOSYS 
http://bugs.python.org/issue29157  opened by haypo

#29158: Possible glitch in the interaction of a thread and a multiproc
http://bugs.python.org/issue29158  opened by luke_16

#29164: make test always fail at 218/405 ( AssertionError: ',' not fou
http://bugs.python.org/issue29164  opened by OO O

#29165: Use forward compatible macro in example code for creating new 
http://bugs.python.org/issue29165  opened by inada.naoki

#29167: Race condition in enum.py:_decompose()
http://bugs.python.org/issue29167  opened by simon.percivall

#29169: update zlib to 1.2.10
http://bugs.python.org/issue29169  opened by doko

#29171: blake2: Remove unused function to avoid undefined references
http://bugs.python.org/issue29171  opened by ericvw

#29172: blake2: Use lowest-common denominator signature of #pragma pac
http://bugs.python.org/issue29172  opened by ericvw

#29174: 'NoneType' object is not callable in subprocess.py
http://bugs.python.org/issue29174  opened by ita1024

#29175: Tutorial links to file object methods are broken.
http://bugs.python.org/issue29175  opened by jonroach at icloud.com

#29176: /tmp does not exist on Android and is used by curses.window.pu
http://bugs.python.org/issue29176  opened by xdegaye

#29177: skip tests using socketserver.UnixStreamServer when bind() rai
http://bugs.python.org/issue29177  opened by xdegaye

#29178: Adding bytes.frombuffer(byteslike) constructor
http://bugs.python.org/issue29178  opened by inada.naoki

#29179: Py_UNUSED is not documented
http://bugs.python.org/issue29179  opened by encukou

#29180: skip tests that raise PermissionError in test_os (non-root use
http://bugs.python.org/issue29180  opened by xdegaye

#29181: skip tests that raise PermissionError in test_tarfile (non-roo
http://bugs.python.org/issue29181  opened by xdegaye

#29182: Remove the warning in urllib docs that it doesn't do certifica
http://bugs.python.org/issue29182  opened by orsenthil



Most recent 15 issues with no replies (15)
==========================================

#29182: Remove the warning in urllib docs that it doesn't do certifica
http://bugs.python.org/issue29182

#29181: skip tests that raise PermissionError in test_tarfile (non-roo
http://bugs.python.org/issue29181

#29180: skip tests that raise PermissionError in test_os (non-root use
http://bugs.python.org/issue29180

#29179: Py_UNUSED is not documented
http://bugs.python.org/issue29179

#29171: blake2: Remove unused function to avoid undefined references
http://bugs.python.org/issue29171

#29165: Use forward compatible macro in example code for creating new 
http://bugs.python.org/issue29165

#29153: Test test.test_asynchat.TestAsynchat / test.test_asynchat.Test
http://bugs.python.org/issue29153

#29151: test_asyncore.TestAPI_UseIPv4Select / test_asyncore.TestAPI_Us
http://bugs.python.org/issue29151

#29144: Implicit namespace packages in Python 3.6
http://bugs.python.org/issue29144

#29127: Incorrect reference names in asyncio.subprocess documentation
http://bugs.python.org/issue29127

#29126: Fix comments about _PyThreadState_Current
http://bugs.python.org/issue29126

#29121: sqlite3 Controlling Transactions documentation not updated
http://bugs.python.org/issue29121

#29117: dir(<bound method>) should include dunder attributes of the un
http://bugs.python.org/issue29117

#29113: modulefinder no longer finds all required modules for Python i
http://bugs.python.org/issue29113

#29098: document minimum sqlite version
http://bugs.python.org/issue29098



Most recent 15 issues waiting for review (15)
=============================================

#29182: Remove the warning in urllib docs that it doesn't do certifica
http://bugs.python.org/issue29182

#29178: Adding bytes.frombuffer(byteslike) constructor
http://bugs.python.org/issue29178

#29176: /tmp does not exist on Android and is used by curses.window.pu
http://bugs.python.org/issue29176

#29172: blake2: Use lowest-common denominator signature of #pragma pac
http://bugs.python.org/issue29172

#29171: blake2: Remove unused function to avoid undefined references
http://bugs.python.org/issue29171

#29169: update zlib to 1.2.10
http://bugs.python.org/issue29169

#29157: random.c: Prefer getrandom() over getentropy(), handle ENOSYS 
http://bugs.python.org/issue29157

#29153: Test test.test_asynchat.TestAsynchat / test.test_asynchat.Test
http://bugs.python.org/issue29153

#29145: failing overflow checks in replace_*
http://bugs.python.org/issue29145

#29142: urllib: no_proxy variable values with leading dot not properly
http://bugs.python.org/issue29142

#29140: time_hash() reads the wrong bytes to get microseconds
http://bugs.python.org/issue29140

#29133: Minor inaccuracy in shlex.shlex punctuation_chars example
http://bugs.python.org/issue29133

#29132: shlex.shlex with punctuation_chars and posix doesn't handle pu
http://bugs.python.org/issue29132

#29126: Fix comments about _PyThreadState_Current
http://bugs.python.org/issue29126

#29125: Shell injection via TIX_LIBRARY when using tkinter.tix
http://bugs.python.org/issue29125



Top 10 most discussed issues (10)
=================================

#28180: sys.getfilesystemencoding() should default to utf-8
http://bugs.python.org/issue28180  23 msgs

#29145: failing overflow checks in replace_*
http://bugs.python.org/issue29145  12 msgs

#29057: Compiler failure on Mac OS X - sys/random.h
http://bugs.python.org/issue29057  11 msgs

#29102: Add an id field to PyInterpreterState.
http://bugs.python.org/issue29102   9 msgs

#29157: random.c: Prefer getrandom() over getentropy(), handle ENOSYS 
http://bugs.python.org/issue29157   9 msgs

#25750: tp_descr_get(self, obj, type) is called without owning a refer
http://bugs.python.org/issue25750   6 msgs

#29120: Move hash randomisation initialisation out of Python/random.c
http://bugs.python.org/issue29120   6 msgs

#27200: make doctest in CPython has failures
http://bugs.python.org/issue27200   5 msgs

#28518: execute("begin immediate") throwing OperationalError
http://bugs.python.org/issue28518   5 msgs

#28961: unittest.mock._Call ignores `name` parameter
http://bugs.python.org/issue28961   5 msgs



Issues closed (45)
==================

#15812: inspect.getframeinfo() cannot show first line
http://bugs.python.org/issue15812  closed by berker.peksag

#24932: Use proper command line parsing in _testembed
http://bugs.python.org/issue24932  closed by steve.dower

#26267: UUID docs should say how to get "standard form"
http://bugs.python.org/issue26267  closed by berker.peksag

#26851: android compilation and link flags
http://bugs.python.org/issue26851  closed by xdegaye

#28443: Logger methods never use kwargs
http://bugs.python.org/issue28443  closed by vinay.sajip

#28524: Set default argument of logging.disable() to logging.CRITICAL
http://bugs.python.org/issue28524  closed by python-dev

#28592: Installation freezes on C Runtime Install
http://bugs.python.org/issue28592  closed by steve.dower

#28768: Warning: implicit declaration of function '_setmode'
http://bugs.python.org/issue28768  closed by haypo

#28781: On Installation of 3.5 Python get error message
http://bugs.python.org/issue28781  closed by steve.dower

#28848: Add CopyingMock to mock.py
http://bugs.python.org/issue28848  closed by berker.peksag

#28985: sqlite3 authorizer codes constants not up to date
http://bugs.python.org/issue28985  closed by berker.peksag

#29012: __bases__ is a tuple (possibly empty or a singleton)
http://bugs.python.org/issue29012  closed by berker.peksag

#29014: Python 3.5.2 installer doesn't register IDLE .py extensions on
http://bugs.python.org/issue29014  closed by steve.dower

#29024: Add Kivy entry to Graphic User Interface FAQ
http://bugs.python.org/issue29024  closed by berker.peksag

#29035: regrtest: simplify regex to match test names for the --fromfil
http://bugs.python.org/issue29035  closed by haypo

#29059: Windows: Python not using ANSI compatible console
http://bugs.python.org/issue29059  closed by steve.dower

#29090: PySerial crash on Windows XP and Python 3.4
http://bugs.python.org/issue29090  closed by haypo

#29093: Windows launcher breaks history in Git Bash
http://bugs.python.org/issue29093  closed by evan_

#29094: Regression in zipfile writing in 2.7.13
http://bugs.python.org/issue29094  closed by serhiy.storchaka

#29105: code or doc improvement for logging.handlers.RotatingFileHandl
http://bugs.python.org/issue29105  closed by python-dev

#29118: Randit
http://bugs.python.org/issue29118  closed by terry.reedy

#29119: Unintentional hard reference assignment in Python version of O
http://bugs.python.org/issue29119  closed by rhettinger

#29122: set() bug
http://bugs.python.org/issue29122  closed by rhettinger

#29123: CheckSqlTimestamp is fragile
http://bugs.python.org/issue29123  closed by berker.peksag

#29128: No way to instsall win32com on python 3.6
http://bugs.python.org/issue29128  closed by zach.ware

#29129: Copy/paste error in "8.13.14.1.1. Using auto"
http://bugs.python.org/issue29129  closed by berker.peksag

#29130: Exit code 120 returned from Python unit test testing SystemExi
http://bugs.python.org/issue29130  closed by r.david.murray

#29131: Calling printf from the cdll does not print the full string
http://bugs.python.org/issue29131  closed by r.david.murray

#29134: contextlib doc bug
http://bugs.python.org/issue29134  closed by orsenthil

#29135: run_proces logs the command without escaping parmaeters such t
http://bugs.python.org/issue29135  closed by wgianopoulos

#29138: No __hash__() inheritance warning with -Werror
http://bugs.python.org/issue29138  closed by martin.panter

#29146: Confusing "invalid token" exception when integers have leading
http://bugs.python.org/issue29146  closed by berker.peksag

#29148: Inheritance behaviour ambiguos
http://bugs.python.org/issue29148  closed by xiang.zhang

#29149: SSL.py recursion limit crash
http://bugs.python.org/issue29149  closed by r.david.murray

#29150: Bad cast@ _mysql_ResultObject_Initialize() result in code exec
http://bugs.python.org/issue29150  closed by christian.heimes

#29156: remove superfluous pow test for (nonexistent) long
http://bugs.python.org/issue29156  closed by serhiy.storchaka

#29159: Regression in bytes constructor
http://bugs.python.org/issue29159  closed by inada.naoki

#29160: pow with three int arguments works like it had two arguments
http://bugs.python.org/issue29160  closed by mark.dickinson

#29161: random.SystemRandom(seed) documentation issue
http://bugs.python.org/issue29161  closed by rhettinger

#29162: pyshell.py: name 'sys' is not defined
http://bugs.python.org/issue29162  closed by OO O

#29163: shlex error in posix mode and empty quotes
http://bugs.python.org/issue29163  closed by berker.peksag

#29166: Spam
http://bugs.python.org/issue29166  closed by xiang.zhang

#29168: multiprocessing pickle error
http://bugs.python.org/issue29168  closed by vinay.sajip

#29170: Curses Module should check for is_keypad and not window flags
http://bugs.python.org/issue29170  closed by Roy Marples

#29173: Python 3.6 on Windows wastes a lot of CPU cycles in a while lo
http://bugs.python.org/issue29173  closed by prahladyeri

From alemos at plux.info  Fri Jan  6 12:39:20 2017
From: alemos at plux.info (=?UTF-8?B?QW5kcsOpIExlbW9z?=)
Date: Fri, 6 Jan 2017 17:39:20 +0000
Subject: [Python-Dev] enable-framework Vs. Naught
Message-ID: <CAJ4-95VgytiYCKhe5HZQzOWjgvQ=EDhUuzGfTzZ5O6jbJZUrYg@mail.gmail.com>

Hi,


I have a C++ module that I am compiling to use inside of my Python
installation under Mac OS.

If I compile & link it against a Framework enabled Python installation, it
works fine, but if I compile & link it against a *non* enabled Framework
installation that we use for distribution, I simply get a non inspiring:

Fatal Python error: PyThreadState_Get: no current thread


I am using python-config to get my flags on both the examples, but I simply
cannot get it to run (although it compiles fine) on a *non* enabled
Framework installation.


Thoughts/Help?



--
Andr? Lemos
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170106/210b3dfe/attachment.html>

From alexander.belopolsky at gmail.com  Fri Jan  6 14:31:18 2017
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Fri, 6 Jan 2017 14:31:18 -0500
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
In-Reply-To: <o4mirt$fuv$1@blaine.gmane.org>
References: <CAP7h-xY3s4Dm0Gb-rComBQnRZcL=DBhFf9iWaAkg82MYhkjd7A@mail.gmail.com>
 <o4mirt$fuv$1@blaine.gmane.org>
Message-ID: <CAP7h-xYSgyLnC_y9da=UTtJ=XW2obS_h1z5zmRXGquWbQhL4yQ@mail.gmail.com>

On Thu, Jan 5, 2017 at 5:54 PM, Serhiy Storchaka <storchaka at gmail.com>
wrote:

> On 05.01.17 22:37, Alexander Belopolsky wrote:
>
>> I propose the following:
>>
>> 1. For 3.6, restore and document 3.5 behavior.  Recommend that 3rd party
>> types that are both integer-like and buffer-like implement their own
>> __bytes__ method to resolve the bytes(x) ambiguity.
>>
>
> The __bytes__ method is used only by the bytes constructor, not by the
> bytearray constructor.


I am not sure this is deliberate.  See <
https://bugs.python.org/issue2415#msg71660>.

>
>
> 2. For 3.7, I would like to see a drastically simplified bytes(x):
>> 2.1.  Accept only objects with a __bytes__ method or a sequence of ints
>> in range(256).
>> 2.2.  Expand __bytes__ definition to accept optional encoding and errors
>> parameters.  Implement str.__bytes__(self, [encoding[, errors]]).
>>
>
> I think it is better to use the encode() method if you want to encode from
> non-strings.


Possibly, but the goal of my proposal is to lighten the logic in the
bytes(x, [encoding[, errors]])
constructor.  If it detects x.__bytes__, it should just call it with
whatever arguments are given.

>
>
> 2.3.  Implement new specialized bytes.fromsize and bytes.frombuffer
>> constructors as per PEP 467 and Inada Naoki proposals.
>>
>
> bytes.fromsize(n) is just b'\0'*n. I don't think this method is needed.
>

I don't care much about this.  If it helps removing bytes(int) case, I am
for it, otherwise ?0.


>
> bytes.frombuffer(x) is bytes(memoryview(x)) or memoryview(x).tobytes().


I've just tried Inada's patch < http://bugs.python.org/issue29178
<http://bugs.python.org/issue29178>>:

$ ./python.exe -m timeit -s "from array import array; x=array('f', [0])"
"bytes.frombuffer(x)"
2000000 loops, best of 5: 134 nsec per loop

$ ./python.exe -m timeit -s "from array import array; x=array('f', [0])"
"with memoryview(x) as m: bytes(m)"
500000 loops, best of 5: 436 nsec per loop

A 3x speed-up seems to be worth it.


>
>
> 2.4. Implement memoryview.__bytes__ method so that bytes(memoryview(x))
>> works ad before.
>> 2.5.  Implement a fast bytearray.__bytes__ method.
>>
>
> This wouldn't help for the bytearray constructor. And wouldn't allow to
> avoid double copying in the constructor of bytes subclass.


I don't see why bytearray constructor should behave differently from bytes.


>
> 3. Consider promoting __bytes__ to a tp_bytes type slot.
>>
>
> The buffer protocol is more general than the __bytes__ method. It allows
> to avoid redundant memory copying in constructors of many types (bytes,
> bytearray, array.array, etc), not just bytes.
>

It looks like there are two different views on what the bytes type
represents.  Is it a sequence of small integers or a blob of binary data?

Compare these two calls:

>>> from array import array
>>> bytes(array('h', [1, 2, 3]))
b'\x01\x00\x02\x00\x03\x00'

and

>>> bytes(array('f', [1, 2, 3]))
b'\x00\x00\x80?\x00\x00\x00@\x00\x00@@'

For me the __bytes__ method is a way for types to specify their bytes
representation that may or may not be the same as memoryview(x).tobytes().
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170106/51c4caa0/attachment.html>

From tjreedy at udel.edu  Fri Jan  6 15:57:01 2017
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 6 Jan 2017 15:57:01 -0500
Subject: [Python-Dev] enable-framework Vs. Naught
In-Reply-To: <CAJ4-95VgytiYCKhe5HZQzOWjgvQ=EDhUuzGfTzZ5O6jbJZUrYg@mail.gmail.com>
References: <CAJ4-95VgytiYCKhe5HZQzOWjgvQ=EDhUuzGfTzZ5O6jbJZUrYg@mail.gmail.com>
Message-ID: <o4p0aq$tjv$1@blaine.gmane.org>

On 1/6/2017 12:39 PM, Andr? Lemos wrote:

> I have a C++ module that I am compiling to use inside of my Python
> installation under Mac OS.
>
> If I compile & link it against a Framework enabled Python installation,
> it works fine, but if I compile & link it against a /non/ enabled
> Framework installation that we use for distribution, I simply get a non
> inspiring:
>
> Fatal Python error: PyThreadState_Get: no current thread
>
>
> I am using python-config to get my flags on both the examples, but I
> simply cannot get it to run (although it compiles fine) on a
> /non/ enabled Framework installation.

This list is for development of furture releases.  Questions about using 
current releases should go to python-list, Stackoverflow, or other help 
forums.

-- 
Terry Jan Reedy



From storchaka at gmail.com  Fri Jan  6 16:43:28 2017
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Fri, 6 Jan 2017 23:43:28 +0200
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
In-Reply-To: <CAP7h-xYSgyLnC_y9da=UTtJ=XW2obS_h1z5zmRXGquWbQhL4yQ@mail.gmail.com>
References: <CAP7h-xY3s4Dm0Gb-rComBQnRZcL=DBhFf9iWaAkg82MYhkjd7A@mail.gmail.com>
 <o4mirt$fuv$1@blaine.gmane.org>
 <CAP7h-xYSgyLnC_y9da=UTtJ=XW2obS_h1z5zmRXGquWbQhL4yQ@mail.gmail.com>
Message-ID: <o4p31s$sbb$1@blaine.gmane.org>

On 06.01.17 21:31, Alexander Belopolsky wrote:
> On Thu, Jan 5, 2017 at 5:54 PM, Serhiy Storchaka <storchaka at gmail.com
> <mailto:storchaka at gmail.com>> wrote:
>
>     On 05.01.17 22:37, Alexander Belopolsky wrote:
>
>         2. For 3.7, I would like to see a drastically simplified bytes(x):
>         2.1.  Accept only objects with a __bytes__ method or a sequence
>         of ints
>         in range(256).
>         2.2.  Expand __bytes__ definition to accept optional encoding
>         and errors
>         parameters.  Implement str.__bytes__(self, [encoding[, errors]]).
>
>
>     I think it is better to use the encode() method if you want to
>     encode from non-strings.
>
>
> Possibly, but the goal of my proposal is to lighten the logic in the
> bytes(x, [encoding[, errors]])
> constructor.  If it detects x.__bytes__, it should just call it with
> whatever arguments are given.

I think this would complicate the __bytes__ protocol. I don't know 
precedences of passing additional optional arguments to a special 
method. int() doesn't pass the base argument to __int__, str() doesn't 
pass encoding and errors to __str__, and pickle.dumps() passes the 
protocol argument to new special method __reduce_ex__ instead of __reduce__.

>     bytes.frombuffer(x) is bytes(memoryview(x)) or memoryview(x).tobytes().
>
>
> I've just tried Inada's patch < http://bugs.python.org/issue29178
> <http://bugs.python.org/issue29178>>:
>
> $ ./python.exe -m timeit -s "from array import array; x=array('f', [0])"
> "bytes..frombuffer(x)"
> 2000000 loops, best of 5: 134 nsec per loop
>
> $ ./python.exe -m timeit -s "from array import array; x=array('f', [0])"
> "with memoryview(x) as m: bytes(m)"
> 500000 loops, best of 5: 436 nsec per loop
>
> A 3x speed-up seems to be worth it.

There is a constant overhead for calling functions. It is dwarfen by 
memory copying for large arrays. I'm not sure that 300 ns is worth 
adding new method.

>         2.4. Implement memoryview.__bytes__ method so that
>         bytes(memoryview(x))
>         works ad before.
>         2.5.  Implement a fast bytearray.__bytes__ method.
>
>
>     This wouldn't help for the bytearray constructor. And wouldn't allow
>     to avoid double copying in the constructor of bytes subclass.
>
>
> I don't see why bytearray constructor should behave differently from bytes.

bytes constructor can just return the result of __bytes__. bytearray 
constructor needs to do a double copying if support __bytes__. First 
copy a data to a bytes object returned by __bytes__, then copy it's 
content to the newly created bytearray object. Creating a bytearray 
object using the buffer protocol needs only one copying.

Perhaps this is the cause why the support of __bytes__ was not added in 
bytearray constructor after all.

> Compare these two calls:
>
>>>> from array import array
>>>> bytes(array('h', [1, 2, 3]))
> b'\x01\x00\x02\x00\x03\x00'
>
> and
>
>>>> bytes(array('f', [1, 2, 3]))
> b'\x00\x00\x80?\x00\x00\x00@\x00\x00@@'

I don't see a difference.

> For me the __bytes__ method is a way for types to specify their bytes
> representation that may or may not be the same as memoryview(x).tobytes().

It would be confusing if some type that supports the buffer protocol 
would implement __bytes__ returning a result different from 
memoryview(x).tobytes(). If you want to get b'\1\2\3' from array('h', 
[1, 2, 3]), use bytes(list(x)).


From alexander.belopolsky at gmail.com  Fri Jan  6 17:33:27 2017
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Fri, 6 Jan 2017 17:33:27 -0500
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
In-Reply-To: <o4p31s$sbb$1@blaine.gmane.org>
References: <CAP7h-xY3s4Dm0Gb-rComBQnRZcL=DBhFf9iWaAkg82MYhkjd7A@mail.gmail.com>
 <o4mirt$fuv$1@blaine.gmane.org>
 <CAP7h-xYSgyLnC_y9da=UTtJ=XW2obS_h1z5zmRXGquWbQhL4yQ@mail.gmail.com>
 <o4p31s$sbb$1@blaine.gmane.org>
Message-ID: <CAP7h-xaty8fHbLAJrXH=-XhXTOnLxyfHk+=8HpLhRQ0dXiuwLQ@mail.gmail.com>

On Fri, Jan 6, 2017 at 4:43 PM, Serhiy Storchaka <storchaka at gmail.com>
wrote:

> Compare these two calls:
>>
>> from array import array
>>>>> bytes(array('h', [1, 2, 3]))
>>>>>
>>>> b'\x01\x00\x02\x00\x03\x00'
>>
>> and
>>
>> bytes(array('f', [1, 2, 3]))
>>>>>
>>>> b'\x00\x00\x80?\x00\x00\x00@\x00\x00@@'
>>
>
> I don't see a difference.


Hmm indeed.  Doesn't this contradict the documentation?

>>> help(bytes)
..
class bytes(object)
 |  bytes(iterable_of_ints) -> bytes
 |  bytes(string, encoding[, errors]) -> bytes
 |  bytes(bytes_or_buffer) -> immutable copy of bytes_or_buffer
..

This suggests that iterable_of_ints is tried first.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170106/37347723/attachment.html>

From armin.rigo at gmail.com  Sun Jan  8 03:25:13 2017
From: armin.rigo at gmail.com (Armin Rigo)
Date: Sun, 8 Jan 2017 09:25:13 +0100
Subject: [Python-Dev] ctypes, memory mapped files and context manager
In-Reply-To: <10601506.CGpYZZeef7@xrated>
References: <1761734.RoB7TtUa4s@xrated>
 <CADiSq7fmug4fgXx0ogSL-v2sKZF2k-DtW9mP=PY79S95=SsxMA@mail.gmail.com>
 <CACL+1atSp4mC8o6OMHShVAWJfMp2W5DoSaRqByBypp0OtJq7Yw@mail.gmail.com>
 <10601506.CGpYZZeef7@xrated>
Message-ID: <CAMSv6X0k6rx4gW2Rh1VQZHKtbtx-COcXLoJ-5Wr=WRSxDuFW0w@mail.gmail.com>

Hi Hans-Peter,

On 6 January 2017 at 00:28, Hans-Peter Jansen <hpj at urpla.net> wrote:
> Leaves the question, how stable this "interface" is?

Another way to jump through hoops:

    c_raw = ctypes.PYFUNCTYPE(ctypes.c_void_p, ctypes.c_void_p)(lambda p: p)

    addr = c_raw(ctypes.pointer(T.from_buffer(m)))
    b = ctypes.cast(addr, ctypes.POINTER(T)).contents

These lines give an object 'b' that is equivalent to
'T.from_buffer(m)', but doesn't hold any reference or any "opened
buffer" state to the original 'm'.  Your context manager can yield
that.  It should prevent all BufferErrors, at the price of segfaulting
if used incorrectly.  This means in your case that ``with
map_struct(..) as a:`` should not continue to use ``a`` after the
``with`` statement, which is pretty natural anyway.

(The same issue occurs with cffi instead of ctypes, but in this case a
simple cast is enough to detach the memoryview, instead of the hack
above.)


A bient?t,

Armin.

From eryksun at gmail.com  Sun Jan  8 10:21:14 2017
From: eryksun at gmail.com (eryk sun)
Date: Sun, 8 Jan 2017 15:21:14 +0000
Subject: [Python-Dev] ctypes, memory mapped files and context manager
In-Reply-To: <CAMSv6X0k6rx4gW2Rh1VQZHKtbtx-COcXLoJ-5Wr=WRSxDuFW0w@mail.gmail.com>
References: <1761734.RoB7TtUa4s@xrated>
 <CADiSq7fmug4fgXx0ogSL-v2sKZF2k-DtW9mP=PY79S95=SsxMA@mail.gmail.com>
 <CACL+1atSp4mC8o6OMHShVAWJfMp2W5DoSaRqByBypp0OtJq7Yw@mail.gmail.com>
 <10601506.CGpYZZeef7@xrated>
 <CAMSv6X0k6rx4gW2Rh1VQZHKtbtx-COcXLoJ-5Wr=WRSxDuFW0w@mail.gmail.com>
Message-ID: <CACL+1avUer9YYZswFFiyboTTSu7TGhLJqpo0HR6Aa=T6_iH-dw@mail.gmail.com>

On Sun, Jan 8, 2017 at 8:25 AM, Armin Rigo <armin.rigo at gmail.com> wrote:
>
>     c_raw = ctypes.PYFUNCTYPE(ctypes.c_void_p, ctypes.c_void_p)(lambda p: p)

Use ctypes.addressof.

>     addr = c_raw(ctypes.pointer(T.from_buffer(m)))
>     b = ctypes.cast(addr, ctypes.POINTER(T)).contents

ctypes.cast uses an FFI call. In this case you can more simply use from_address:

    b = T.from_address(ctypes.addressof(T.from_buffer(m)))

There's no supporting connection between b and m. If m was allocated
from a heap/pool/freelist, as opposed to a separate mmap
(VirtualAlloc) call, then you won't necessarily get a segfault (access
violation) if b is used after m has been deallocated or internally
realloc'd. It can lead to corrupt data and difficult to diagnose
errors. You're lucky if it segfaults.

From steve at holdenweb.com  Mon Jan  9 06:42:33 2017
From: steve at holdenweb.com (Steve Holden)
Date: Mon, 9 Jan 2017 11:42:33 +0000
Subject: [Python-Dev] Imports with underscores
Message-ID: <CAMofdRDZHknZz=PN_NAM2Mc=XPE1MDePc-FtHUdWJxZpR_FKVA@mail.gmail.com>

One of my developers recently submitted a pull request incuding a number of
lines like

import os as _os

When I asked him why he suggested a) this would improve encapsulation, and
b) the practice was supported in the stdlib. Further investigation reveals
that some modules (e.g. argparse, crypt, difflib, random) do use this
technique, but it is far from universal.

So I thought it would be useful to get input from current devs about the
value of this practice, since to me it seems somewhat anti-pythonic. What
advantages does it confer?

regards
Steve Holden
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170109/6fff6f50/attachment.html>

From barry at python.org  Mon Jan  9 11:20:16 2017
From: barry at python.org (Barry Warsaw)
Date: Mon, 9 Jan 2017 11:20:16 -0500
Subject: [Python-Dev] Imports with underscores
In-Reply-To: <CAMofdRDZHknZz=PN_NAM2Mc=XPE1MDePc-FtHUdWJxZpR_FKVA@mail.gmail.com>
References: <CAMofdRDZHknZz=PN_NAM2Mc=XPE1MDePc-FtHUdWJxZpR_FKVA@mail.gmail.com>
Message-ID: <20170109112016.6a2f32ff@subdivisions.wooz.org>

On Jan 09, 2017, at 11:42 AM, Steve Holden wrote:

>So I thought it would be useful to get input from current devs about the
>value of this practice, since to me it seems somewhat anti-pythonic. What
>advantages does it confer?

It just means you can't accidentally import it with a from-import-* since
those ignore underscored names by default.

(But then if you use __all__'s it won't happen anyway because you'll never put
'os' in __all__.)

(Aside from the fact that from-import-* is usually bad form, and the problem
with __all__ <wink> [1].)

Cheers,
-Barry

[1] http://public.readthedocs.io/en/latest/#the-problem

From p.f.moore at gmail.com  Mon Jan  9 11:31:46 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 9 Jan 2017 16:31:46 +0000
Subject: [Python-Dev] Imports with underscores
In-Reply-To: <CAMofdRDZHknZz=PN_NAM2Mc=XPE1MDePc-FtHUdWJxZpR_FKVA@mail.gmail.com>
References: <CAMofdRDZHknZz=PN_NAM2Mc=XPE1MDePc-FtHUdWJxZpR_FKVA@mail.gmail.com>
Message-ID: <CACac1F-R_TcnYeS6mHK4SS=6VTwgguQf0j0-jNY-_54yTy2QEA@mail.gmail.com>

On 9 January 2017 at 11:42, Steve Holden <steve at holdenweb.com> wrote:
> One of my developers recently submitted a pull request incuding a number of
> lines like
>
> import os as _os
>
> When I asked him why he suggested a) this would improve encapsulation, and
> b) the practice was supported in the stdlib. Further investigation reveals
> that some modules (e.g. argparse, crypt, difflib, random) do use this
> technique, but it is far from universal.
>
> So I thought it would be useful to get input from current devs about the
> value of this practice, since to me it seems somewhat anti-pythonic. What
> advantages does it confer?

As I understand it, it prevents the imports showing up in "import *"
imports, but using __all__ is better. I'd imagine usage in the stdlib
is mainly historical.

It's not a practice I'd recommend in user code. The needs of the
stdlib are somewhat special (and stdlib code can be many years old,
reflecting out of date design rules) and "because the stdlib does it"
is not necessarily a compelling argument in the absence of an actual
justification.

Regarding "improves encapsulation", as Barry mentioned, using __all__
(and avoiding import-* anyway) is a better approach. Furthermore,
Python does not encourage strict (as in, enforced, encapsulation).
This is very much a "consenting adults" situation - by hiding imports
like this, you make it harder to monkeypatch (for testing purposes,
for example). Users are meant to stick to the published API of a
module *because it's the right thing to do*, not because they are
forced to.

Paul

From ethan at stoneleaf.us  Mon Jan  9 11:32:30 2017
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 09 Jan 2017 08:32:30 -0800
Subject: [Python-Dev] Imports with underscores
In-Reply-To: <CAMofdRDZHknZz=PN_NAM2Mc=XPE1MDePc-FtHUdWJxZpR_FKVA@mail.gmail.com>
References: <CAMofdRDZHknZz=PN_NAM2Mc=XPE1MDePc-FtHUdWJxZpR_FKVA@mail.gmail.com>
Message-ID: <5873BB1E.80603@stoneleaf.us>

On 01/09/2017 03:42 AM, Steve Holden wrote:

> When I asked him why he suggested
>  a) this would improve encapsulation, and
>  b) the practice was supported in the stdlib.
>  Further investigation reveals that some modules (e.g. argparse, crypt,
>  difflib, random) do use this technique, but it is far from universal.
>
> So I thought it would be useful to get input from current devs about
>  the value of this practice, since to me it seems somewhat anti-pythonic.
>  What advantages does it confer?

Aside from what Barry said it also offers the (very) minor advantage of showing up as an implementation detail when someone does a dir() of the module.

Personally, I tend to use the technique for new code, but I wouldn't change old code for it (and new code in an old module should follow the old module's practices, etc.) .

--
~Ethan~

From guido at python.org  Mon Jan  9 11:48:03 2017
From: guido at python.org (Guido van Rossum)
Date: Mon, 9 Jan 2017 08:48:03 -0800
Subject: [Python-Dev] Imports with underscores
In-Reply-To: <5873BB1E.80603@stoneleaf.us>
References: <CAMofdRDZHknZz=PN_NAM2Mc=XPE1MDePc-FtHUdWJxZpR_FKVA@mail.gmail.com>
 <5873BB1E.80603@stoneleaf.us>
Message-ID: <CAP7+vJKuHPrg=Qjz9pDBXbObUWO7TmhJNsHvzZOD0_wvSaOQOA@mail.gmail.com>

I would focus on changing habits to discourage "import *" rather than
uglifying all new code with this "os as _os" pattern. Very occasionally one
designs a module to explicitly support "import *", and that usually entails
using __all__ (like it or not), making the problem go away without
uglifying the code.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170109/6be55041/attachment.html>

From tjreedy at udel.edu  Mon Jan  9 12:01:04 2017
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 9 Jan 2017 12:01:04 -0500
Subject: [Python-Dev] Imports with underscores
In-Reply-To: <CAMofdRDZHknZz=PN_NAM2Mc=XPE1MDePc-FtHUdWJxZpR_FKVA@mail.gmail.com>
References: <CAMofdRDZHknZz=PN_NAM2Mc=XPE1MDePc-FtHUdWJxZpR_FKVA@mail.gmail.com>
Message-ID: <o50fkd$b2s$1@blaine.gmane.org>

On 1/9/2017 6:42 AM, Steve Holden wrote:
> One of my developers recently submitted a pull request incuding a number
> of lines like
>
> import os as _os
>
> When I asked him why he suggested a) this would improve encapsulation,
> and b) the practice was supported in the stdlib. Further investigation
> reveals that some modules (e.g. argparse, crypt, difflib, random) do use
> this technique, but it is far from universal.

> So I thought it would be useful to get input from current devs about the
> value of this practice, since to me it seems somewhat anti-pythonic.
> What advantages does it confer?

If the module does not define __all__, it prevents * imports of the 
module from also importing the imported modules.  For instance:

 >>> sys
Traceback (most recent call last):
   File "<pyshell#1>", line 1, in <module>
     sys
NameError: name 'sys' is not defined
 >>> from tkinter import *
 >>> sys
<module 'sys' (built-in)>
 >>> enum
<module 'enum' from 'C:\\Programs\\Python36\\lib\\enum.py'>
 >>> itertools
Traceback (most recent call last):
   File "<pyshell#5>", line 1, in <module>
     itertools
NameError: name 'itertools' is not defined

Use of such undocumented and unintended attributes of a module is fragile.

1.  The imported module could be changed.  Tkinter's 'import enum' in 
only used in "class EventType(str, enum.Enum):".  The import could well 
be changed to 'from enum import Enum' or even better, 'from enum import 
Enum as _Enum' and the one use modified.

2. The importing module could be changed.  'from tkinter import *' might 
be changed to 'from tkinter import tk, ...' or 'import tkinter as tk' or 
even replaced by another module.

-- 
Terry Jan Reedy


From nd at perlig.de  Mon Jan  9 12:23:58 2017
From: nd at perlig.de (=?iso-8859-1?q?Andr=E9_Malo?=)
Date: Mon, 9 Jan 2017 18:23:58 +0100
Subject: [Python-Dev] Imports with underscores
In-Reply-To: <CAMofdRDZHknZz=PN_NAM2Mc=XPE1MDePc-FtHUdWJxZpR_FKVA@mail.gmail.com>
References: <CAMofdRDZHknZz=PN_NAM2Mc=XPE1MDePc-FtHUdWJxZpR_FKVA@mail.gmail.com>
Message-ID: <201701091823.58963@news.perlig.de>

* Steve Holden wrote:

> One of my developers recently submitted a pull request incuding a number
> of lines like
>
> import os as _os
>
> When I asked him why he suggested a) this would improve encapsulation,
> and b) the practice was supported in the stdlib. Further investigation
> reveals that some modules (e.g. argparse, crypt, difflib, random) do use
> this technique, but it is far from universal.
>
> So I thought it would be useful to get input from current devs about the
> value of this practice, since to me it seems somewhat anti-pythonic. What
> advantages does it confer?

For me (in favor of underscored imports), the following items apply:

- the imports are usually not part of the exported API. (If they are, I 
  specifically do not underscore them)

- __all__ was referenced in other answers. However, it only protects from a 
  single use case (import *). It does not help you directly with shells 
  (dir(), ipython tab expansion (?)) and it's easy to ignore if you look at 
  the source code itself (because, let's face it, documentation often 
  sucks).

- __all__ again: it's tedious and error-prone to maintain. I often found
  places in my own code where it was plain wrong. (pylint helps these days 
  against wrongness, but not against incompleteness)

- In my code (inside the module): I usually know exactly what 
  variable is a module and what is not (by the underscores)

- Also in my code - from time to time the modules steal good names for local 
  variables, underscoring also solved this problem for me.

Cheers,
nd
-- 
die (eval q-qq:Just Another Perl Hacker
:-)

# Andr? Malo, <http://pub.perlig.de/> #

From barry at python.org  Mon Jan  9 12:30:48 2017
From: barry at python.org (Barry Warsaw)
Date: Mon, 9 Jan 2017 12:30:48 -0500
Subject: [Python-Dev] Imports with underscores
In-Reply-To: <201701091823.58963@news.perlig.de>
References: <CAMofdRDZHknZz=PN_NAM2Mc=XPE1MDePc-FtHUdWJxZpR_FKVA@mail.gmail.com>
 <201701091823.58963@news.perlig.de>
Message-ID: <20170109123048.4b59e0cf@subdivisions.wooz.org>

On Jan 09, 2017, at 06:23 PM, Andr? Malo wrote:

>- __all__ again: it's tedious and error-prone to maintain.

Only if you use the wrong tools <wink>

http://public.readthedocs.io/en/latest/
http://bugs.python.org/issue26632

Cheers,
-Barry

From srkunze at mail.de  Mon Jan  9 13:03:06 2017
From: srkunze at mail.de (Sven R. Kunze)
Date: Mon, 9 Jan 2017 19:03:06 +0100
Subject: [Python-Dev] Imports with underscores
In-Reply-To: <CAMofdRDZHknZz=PN_NAM2Mc=XPE1MDePc-FtHUdWJxZpR_FKVA@mail.gmail.com>
References: <CAMofdRDZHknZz=PN_NAM2Mc=XPE1MDePc-FtHUdWJxZpR_FKVA@mail.gmail.com>
Message-ID: <046d3415-05b9-f910-1e51-8218c53d5a3d@mail.de>

Interesting to see that others have the same problem.

We also had this kind of "over-protective" behavior. As far as I know, 
our devs stopped doing it as it feels cumbersome.


Another argument for this is: when using PyCharm, this IDE will suggest 
imports from those modules which aren't the original ones. So, you might 
import from a third-party module. Over time, however, people learn to 
pick the "right" module to import from.

Cheers,
Sven


On 09.01.2017 12:42, Steve Holden wrote:
> One of my developers recently submitted a pull request incuding a 
> number of lines like
>
> import os as _os
>
> When I asked him why he suggested a) this would improve encapsulation, 
> and b) the practice was supported in the stdlib. Further investigation 
> reveals that some modules (e.g. argparse, crypt, difflib, random) do 
> use this technique, but it is far from universal.
>
> So I thought it would be useful to get input from current devs about 
> the value of this practice, since to me it seems somewhat 
> anti-pythonic. What advantages does it confer?
>
> regards
> Steve Holden
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170109/aba8b46a/attachment.html>

From dholth at gmail.com  Mon Jan  9 13:42:58 2017
From: dholth at gmail.com (Daniel Holth)
Date: Mon, 09 Jan 2017 18:42:58 +0000
Subject: [Python-Dev] Imports with underscores
In-Reply-To: <046d3415-05b9-f910-1e51-8218c53d5a3d@mail.de>
References: <CAMofdRDZHknZz=PN_NAM2Mc=XPE1MDePc-FtHUdWJxZpR_FKVA@mail.gmail.com>
 <046d3415-05b9-f910-1e51-8218c53d5a3d@mail.de>
Message-ID: <CAG8k2+7LztnTnLBA4gTPsL5ECVVQCy5H39btbfamRhMjpro6eA@mail.gmail.com>

Easily solved with the totally evil ninja mode pattern of module
initialization. It has yet to catch on.

def ninja():
    global exported
    import os
    def exported():
        # do something
ninja()
del ninja

On Mon, Jan 9, 2017 at 1:13 PM Sven R. Kunze <srkunze at mail.de> wrote:

> Interesting to see that others have the same problem.
>
> We also had this kind of "over-protective" behavior. As far as I know, our
> devs stopped doing it as it feels cumbersome.
>
>
> Another argument for this is: when using PyCharm, this IDE will suggest
> imports from those modules which aren't the original ones. So, you might
> import from a third-party module. Over time, however, people learn to pick
> the "right" module to import from.
>
> Cheers,
> Sven
>
>
> On 09.01.2017 12:42, Steve Holden wrote:
>
> One of my developers recently submitted a pull request incuding a number
> of lines like
>
> import os as _os
>
> When I asked him why he suggested a) this would improve encapsulation, and
> b) the practice was supported in the stdlib. Further investigation reveals
> that some modules (e.g. argparse, crypt, difflib, random) do use this
> technique, but it is far from universal.
>
> So I thought it would be useful to get input from current devs about the
> value of this practice, since to me it seems somewhat anti-pythonic. What
> advantages does it confer?
>
> regards
> Steve Holden
>
>
> _______________________________________________
> Python-Dev mailing listPython-Dev at python.orghttps://mail.python.org/mailman/listinfo/python-dev
>
>
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170109/7f37c7eb/attachment.html>

From tjreedy at udel.edu  Mon Jan  9 15:29:38 2017
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 9 Jan 2017 15:29:38 -0500
Subject: [Python-Dev] Imports with underscores
In-Reply-To: <CAP7+vJKuHPrg=Qjz9pDBXbObUWO7TmhJNsHvzZOD0_wvSaOQOA@mail.gmail.com>
References: <CAMofdRDZHknZz=PN_NAM2Mc=XPE1MDePc-FtHUdWJxZpR_FKVA@mail.gmail.com>
 <5873BB1E.80603@stoneleaf.us>
 <CAP7+vJKuHPrg=Qjz9pDBXbObUWO7TmhJNsHvzZOD0_wvSaOQOA@mail.gmail.com>
Message-ID: <o50rre$f6$1@blaine.gmane.org>

On 1/9/2017 11:48 AM, Guido van Rossum wrote:
> I would focus on changing habits to discourage "import *" rather than

The tkinter doc still has

...to use Tkinter all you need is a simple import statement:
   import tkinter
Or, more often:
   from tkinter import *

Should this be changed?

> uglifying all new code with this "os as _os" pattern. Very occasionally
> one designs a module to explicitly support "import *", and that usually
> entails using __all__ (like it or not), making the problem go away
> without uglifying the code.

tkinter does not have have __all__.  It would have 160 (in 3.6) minus at 
least 3 (for enum, re, and sys) entries.

-- 
Terry Jan Reedy


From chris.barker at noaa.gov  Tue Jan 10 11:49:28 2017
From: chris.barker at noaa.gov (Chris Barker)
Date: Tue, 10 Jan 2017 08:49:28 -0800
Subject: [Python-Dev] Imports with underscores
In-Reply-To: <o50rre$f6$1@blaine.gmane.org>
References: <CAMofdRDZHknZz=PN_NAM2Mc=XPE1MDePc-FtHUdWJxZpR_FKVA@mail.gmail.com>
 <5873BB1E.80603@stoneleaf.us>
 <CAP7+vJKuHPrg=Qjz9pDBXbObUWO7TmhJNsHvzZOD0_wvSaOQOA@mail.gmail.com>
 <o50rre$f6$1@blaine.gmane.org>
Message-ID: <CALGmxE+uTE+zOiDV8nqNm1XqBh6Cz17Yg3XwkNa0iF+hZT9yKg@mail.gmail.com>

On Mon, Jan 9, 2017 at 12:29 PM, Terry Reedy <tjreedy at udel.edu> wrote:

> The tkinter doc still has
>
> ...to use Tkinter all you need is a simple import statement:
>   import tkinter
> Or, more often:
>   from tkinter import *
>
> Should this be changed?
>

yes, it should.

I would suggest suggesting something like:

import tkinter as tk

following the similar convention of wxPython, numpy, matplotlib, (all
large widely used packages that started their lives with import *
recommendations...)

-CHB




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170110/9a2d0ad4/attachment.html>

From chris.barker at noaa.gov  Tue Jan 10 12:42:49 2017
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Tue, 10 Jan 2017 09:42:49 -0800
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
In-Reply-To: <CAP7h-xY3s4Dm0Gb-rComBQnRZcL=DBhFf9iWaAkg82MYhkjd7A@mail.gmail.com>
References: <CAP7h-xY3s4Dm0Gb-rComBQnRZcL=DBhFf9iWaAkg82MYhkjd7A@mail.gmail.com>
Message-ID: <381320226684483961@unknownmsgid>

>   This is what happens with numpy arrays:
>
> >>> bytes(numpy.array([2], 'i1'))
> b'\x00\x00'
>
> >>> bytes(numpy.array([2, 2], 'i1'))
> b'\x02\x02'
>
> For better or worse, single-element numpy arrays have a working __index__ methods

Ouch -- that probably is for the worse..

There are Numpy scalars that should be used for that.


> 1. For 3.6, restore and document 3.5 behavior.  Recommend that 3rd party types that are both integer-like and buffer-like implement their own __bytes__ method to resolve the bytes(x) ambiguity.

+1 -- though the default should be clear if there isn't one.

> 2.1.  Accept only objects with a __bytes__ method or a sequence of ints in range(256).

If frombuffer() is added, then yes.

> 2.2.  Expand __bytes__ definition to accept optional encoding and errors parameters.  Implement str.__bytes__(self, [encoding[, errors]]).

Ouch! I understand the desire to keep a simple API -- but I think
encoding clearly belongs with the strong object. What's wrong with:

s.encode() ?

IIUC, the ONLY object one could possibly encode anyway is a string -
'cause you have to know the internal representation. So bytes() would
simply be passing the encoding info off to the string anyway ( or
other object with and encode method).

> 2.3.  Implement new specialized bytes.fromsize and bytes.frombuffer constructors as per PEP 467 and Inada Naoki proposals.

Maybe not important -- but nice to have a obvious and perform any way
to do it. ( does this proposal allow an initializing value?).

For prior art, Numpy has:

zeros()
ones()
empty()

And I would like to see an explicit frombuffer() constructor. See
others' notes about how using an intermediary memoryview is not
obvious and straightforward.

-CHB

From chris.barker at noaa.gov  Tue Jan 10 12:51:04 2017
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Tue, 10 Jan 2017 09:51:04 -0800
Subject: [Python-Dev] enable-framework Vs. Naught
In-Reply-To: <o4p0aq$tjv$1@blaine.gmane.org>
References: <CAJ4-95VgytiYCKhe5HZQzOWjgvQ=EDhUuzGfTzZ5O6jbJZUrYg@mail.gmail.com>
 <o4p0aq$tjv$1@blaine.gmane.org>
Message-ID: <-1432114187368196844@unknownmsgid>

> This list is for development of furture releases.  Questions about using current releases should go to python-list, Stackoverflow, or other help forums.

The python-Mac SIG would be a good one for this.

CHB

>
> --
> Terry Jan Reedy
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov

From brett at snarky.ca  Mon Jan  9 15:50:00 2017
From: brett at snarky.ca (Brett Cannon)
Date: Mon, 09 Jan 2017 20:50:00 +0000
Subject: [Python-Dev] PEP 7 contradiction for comment style
Message-ID: <CAP1=2W7tEOyhNOQ4_satnQNK6zgmCUDKgGWBB55mSS1DxS-SsQ@mail.gmail.com>

https://bugs.python.org/issue29215 noticed that PEP 7 says "C++-style line
comments" are allowed, but then later says "Never use C++ style // one-line
comments." I'm assuming we are sticking with allowing C++-style comments
and the "never" link just needs an addendum to say that only applies to
code prior to Python 3.5, but I wanted to double-check before editing the
PEP.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170109/cc0bea6a/attachment.html>

From benjamin at python.org  Wed Jan 11 02:45:51 2017
From: benjamin at python.org (Benjamin Peterson)
Date: Tue, 10 Jan 2017 23:45:51 -0800
Subject: [Python-Dev] PEP 7 contradiction for comment style
In-Reply-To: <CAP1=2W7tEOyhNOQ4_satnQNK6zgmCUDKgGWBB55mSS1DxS-SsQ@mail.gmail.com>
References: <CAP1=2W7tEOyhNOQ4_satnQNK6zgmCUDKgGWBB55mSS1DxS-SsQ@mail.gmail.com>
Message-ID: <1484120751.2091812.844003824.5E5860CF@webmail.messagingengine.com>

Your assumption is correct. Perhaps the PEP 7 should be partitioned into
"< 3.6" and "3.6" sections where applicable.

On Mon, Jan 9, 2017, at 12:50, Brett Cannon wrote:
> https://bugs.python.org/issue29215 noticed that PEP 7 says "C++-style
> line
> comments" are allowed, but then later says "Never use C++ style //
> one-line
> comments." I'm assuming we are sticking with allowing C++-style comments
> and the "never" link just needs an addendum to say that only applies to
> code prior to Python 3.5, but I wanted to double-check before editing the
> PEP.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org

From brett at python.org  Wed Jan 11 14:33:28 2017
From: brett at python.org (Brett Cannon)
Date: Wed, 11 Jan 2017 19:33:28 +0000
Subject: [Python-Dev] PEP 7 contradiction for comment style
In-Reply-To: <1484120751.2091812.844003824.5E5860CF@webmail.messagingengine.com>
References: <CAP1=2W7tEOyhNOQ4_satnQNK6zgmCUDKgGWBB55mSS1DxS-SsQ@mail.gmail.com>
 <1484120751.2091812.844003824.5E5860CF@webmail.messagingengine.com>
Message-ID: <CAP1=2W4wjg7=k6rStm7OSyRdCdqbiRenKH-2PzK+F8CG3+NawA@mail.gmail.com>

https://github.com/python/peps/issues/176 is tracking the need to update
the PEP.

On Tue, 10 Jan 2017 at 23:45 Benjamin Peterson <benjamin at python.org> wrote:

> Your assumption is correct. Perhaps the PEP 7 should be partitioned into
> "< 3.6" and "3.6" sections where applicable.
>
> On Mon, Jan 9, 2017, at 12:50, Brett Cannon wrote:
> > https://bugs.python.org/issue29215 noticed that PEP 7 says "C++-style
> > line
> > comments" are allowed, but then later says "Never use C++ style //
> > one-line
> > comments." I'm assuming we are sticking with allowing C++-style comments
> > and the "never" link just needs an addendum to say that only applies to
> > code prior to Python 3.5, but I wanted to double-check before editing the
> > PEP.
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/benjamin%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170111/81f94ab8/attachment.html>

From cournape at gmail.com  Thu Jan 12 08:08:52 2017
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 12 Jan 2017 13:08:52 +0000
Subject: [Python-Dev] Making the new dtrace support work on OS X
Message-ID: <CAGY4rcXj8F2C=JR2zZeE0nOjxwfbd7HJt=7gUuj+kUV8HaJ5fQ@mail.gmail.com>

Hi,

I was excited to see official dtrace support for python 3.6.0 on OS X, but
I have not been able to make it work:

1. I built my own python from sources on OS X 10.9,  with the --with-dtrace
support
2. if I launch `python3.6 -q &` and then `sudo dtrace -l -P python$!`, I
get the following output:

   ID   PROVIDER            MODULE                          FUNCTION NAME
 2774 python48084         python3.6          _PyEval_EvalFrameDefault
function-entry
 2775 python48084         python3.6          _PyEval_EvalFrameDefault
function-return
 2776 python48084         python3.6                           collect
gc-done
 2777 python48084         python3.6                           collect
gc-start
 2778 python48084         python3.6          _PyEval_EvalFrameDefault line

Which looks similar but not the same as the example given in the doc at
https://docs.python.org/dev/howto/instrumentation.html#enabling-the-static-markers

3. When I try to test anything with the given call_stack.d example, I can't
make it work at all:

"""
# script.py
def start():
    foo()

def foo():
    pass

start()
"""

I am not very familiar with dtrace, so maybe I a missing a step, there is a
documentation bug, or it depends on which OS X version you are using ?

Thanks,
David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170112/d6e72985/attachment.html>

From victor.stinner at gmail.com  Fri Jan 13 08:17:48 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 13 Jan 2017 14:17:48 +0100
Subject: [Python-Dev] Add tp_fastcall to PyTypeObject
Message-ID: <CAMpsgwahH-+Z7kLojhhgQtvV7D84ZzNvWDUPovCEewbn07NCEA@mail.gmail.com>

Hi,

tl;dr Python 3.7 is going to be faster without breaking backward
compatibility, say hello to the new "tp_fastcall" slot!
=> http://bugs.python.org/issue29259


Python 3.6 got a new "FASTCALL" calling convention which allows to
avoid the creation a temporary tuple to pass positional arguments and
a temporary dictionary to pass keyword arguments. But callable objects
having a __call__() method implemented in Python don't benefit of
FASTCAL yet.

I tried to reuse the tp_call slot with a new flag in tp_flags, but I
had two major blocker issues:

* Deeply break the backward compatibility of the C API: calling
directly tp_call (with tuple/dict) would crash immediately if the
object uses FASTCALL

* Need to duplicate each "tp_call" function to get a new "tp_fastcall"
flavor. It wasn't easy to share the function body.


Good news, I found a new design which don't have any of these issues!
=> http://bugs.python.org/issue29259

I chose to add a new tp_fastcall field to PyTypeObject and use a tiny
wrapper calling tp_fastcall for tp_call, to keep the backward
compatibility.


The goal is to get optimizations "for free" when calling functions.
The best expected speedup on a microbenchmark is around 1.56x faster
(-36%) when calling an object supporting FASTCALL. Example with
property_descr_get() without its "cached args" hack, result without
fastcall ("py34") compared to fastcall ("fastcall_wrapper"):

Median +- std dev: [py34] 75.0 ns +- 1.7 ns -> [fastcall_wrapper] 48.2
ns +- 1.5 ns: 1.56x faster (-36%)

But please don't expect such large speedup on macro-benchmark.


tp_fastcall allows to remove the "cached args" optimization used in
various parts of Python core, old optimizations used in performance
critical code. This hack causes various kinds of complex bugs in
corner cases which can lead to crash in the worst case.


The patch to support tp_fastcall is tiny, but you should expect a long
list of tiny changes to replace tp_call with tp_fastcall in various
types.


Final bonus point: existing code (calling functions) doesn't need to
be modified (nor recompiled) to get speedup. Even if tp_call is
directly directly, fastcall will provide speedup, but only if it is
called only with positional arguments.

About the tp_call wrapper: keyword arguments require to convert a
Python dictionary to a C array which might be more expensive. I didn't
try to measure the performance, since this case is very rare. Almost
no C code calls functions with keyword arguments, just because it's
much more complex to pass keyword arguments, it requires too much C
code (and it's not simpler with fastcall, sorry).

Victor

From status at bugs.python.org  Fri Jan 13 12:09:11 2017
From: status at bugs.python.org (Python tracker)
Date: Fri, 13 Jan 2017 18:09:11 +0100 (CET)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20170113170911.AB96B567F7@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2017-01-06 - 2017-01-13)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    5719 (+34)
  closed 35300 (+47)
  total  41019 (+81)

Open issues with patches: 2474 


Issues opened (53)
==================

#10513: sqlite3.InterfaceError after commit
http://bugs.python.org/issue10513  reopened by benjamin.peterson

#26858: android: setting SO_REUSEPORT fails
http://bugs.python.org/issue26858  reopened by xdegaye

#28759: access to mkfifo, mknod and hard links is controled by SELinux
http://bugs.python.org/issue28759  reopened by xdegaye

#29183: Unintuitive error handling in wsgiref when a crash happens in 
http://bugs.python.org/issue29183  opened by jleclanche

#29184: skip tests of test_socketserver when bind() raises PermissionE
http://bugs.python.org/issue29184  opened by xdegaye

#29185: test_distutils fails on Android API level 24
http://bugs.python.org/issue29185  opened by xdegaye

#29187: Pickle failure is raising AttributeError and not PicklingError
http://bugs.python.org/issue29187  opened by Matt.Dodge

#29191: liblzma is missing from pcbuild.sln
http://bugs.python.org/issue29191  opened by Segev Finer

#29196: Remove old-deprecated plistlib features
http://bugs.python.org/issue29196  opened by serhiy.storchaka

#29197: Remove os.path.splitunc()
http://bugs.python.org/issue29197  opened by serhiy.storchaka

#29198: AsyncGenerator is missing from typing
http://bugs.python.org/issue29198  opened by Jelle Zijlstra

#29199: test_regrtest fails if PCBuild directory doesn't exist
http://bugs.python.org/issue29199  opened by ppperry

#29202: Improve dict iteration
http://bugs.python.org/issue29202  opened by rhettinger

#29204: Add code deprecations in ElementTree
http://bugs.python.org/issue29204  opened by serhiy.storchaka

#29205: col_offset for AsyncFunctionDef AST nodes is wrong
http://bugs.python.org/issue29205  opened by Jelle Zijlstra

#29207: make install fails on ARM
http://bugs.python.org/issue29207  opened by petevine

#29209: Remove old-deprecated ElementTree features
http://bugs.python.org/issue29209  opened by serhiy.storchaka

#29211: assertRaises with exceptions re-raised from a generator kills 
http://bugs.python.org/issue29211  opened by dalke

#29212: Python 3.6 logging thread name regression with concurrent.futu
http://bugs.python.org/issue29212  opened by desbma

#29213: python -m venv activate.bat has weird mix of line endings
http://bugs.python.org/issue29213  opened by Josh Wilson

#29216: Space saving step for the LRU cache
http://bugs.python.org/issue29216  opened by rhettinger

#29218: distutils: Remove unused install_misc class
http://bugs.python.org/issue29218  opened by ericvw

#29221: ABC Recursion Error on isinstance() with less than recursion l
http://bugs.python.org/issue29221  opened by Anthony Scopatz

#29225: distutils.command.install_lib.get_outputs() wrong with extensi
http://bugs.python.org/issue29225  opened by Elvis Stansvik

#29227: Reduce C stack consumption in function calls
http://bugs.python.org/issue29227  opened by haypo

#29230: Segfault in sqlite3
http://bugs.python.org/issue29230  opened by intchanter

#29232: Quiet Install
http://bugs.python.org/issue29232  opened by earl.maier

#29234: Disable inlining of _PyStack_AsTuple() to reduce the stack con
http://bugs.python.org/issue29234  opened by haypo

#29235: Allow profile/cProfile to be used as context managers
http://bugs.python.org/issue29235  opened by Thane Brimhall

#29236: 'an ASCII string of one or more PEM-encoded certificates' need
http://bugs.python.org/issue29236  opened by uzytkownik

#29237: Create enum for pstats sorting options
http://bugs.python.org/issue29237  opened by Thane Brimhall

#29238: Add more kwargs to cProfile's print_stats
http://bugs.python.org/issue29238  opened by Thane Brimhall

#29240: Implementation of the PEP 540: Add a new UTF-8 mode
http://bugs.python.org/issue29240  opened by haypo

#29241: sys._enablelegacywindowsfsencoding() don't apply to os.fsencod
http://bugs.python.org/issue29241  opened by JGoutin

#29242: Crash on GC when compiling PyPy
http://bugs.python.org/issue29242  opened by gumblex

#29243: --enable-optimizations makes common build commands always need
http://bugs.python.org/issue29243  opened by xiang.zhang

#29246: typing.Union raises RecursionError when comparing Union to oth
http://bugs.python.org/issue29246  opened by spiside

#29247: Document return value of epoll.poll
http://bugs.python.org/issue29247  opened by njs

#29248: os.readlink fails on Windows
http://bugs.python.org/issue29248  opened by eryksun

#29249: Pathlib glob ** bug
http://bugs.python.org/issue29249  opened by Jon Walsh

#29250: islink and stat follow_symlinks are inconsistent on Windows
http://bugs.python.org/issue29250  opened by eryksun

#29251: Class __dict__ is only a mapping proxy
http://bugs.python.org/issue29251  opened by martin.panter

#29252: self in classes missinterpreted as a string.
http://bugs.python.org/issue29252  opened by Decorater

#29253: Fix test_asyncore tests on Cygwin
http://bugs.python.org/issue29253  opened by erik.bray

#29255: selects.KqueueSelector behaves incorrectly when no fds are reg
http://bugs.python.org/issue29255  opened by njs

#29256: Windows select() errors out when given no fds to select on, wh
http://bugs.python.org/issue29256  opened by njs

#29257: Possible error in discussion of Abstract Base Classes and abst
http://bugs.python.org/issue29257  opened by gbritton

#29258: __init__.py required for pkgutil.walk_packages in python3
http://bugs.python.org/issue29258  opened by Anthony Sottile

#29259: Add tp_fastcall to PyTypeObject: support FASTCALL calling conv
http://bugs.python.org/issue29259  opened by haypo

#29260: Use designated initializers to define PyTypeObject types
http://bugs.python.org/issue29260  opened by haypo

#29261: Missing venv/scripts/common after "make install"
http://bugs.python.org/issue29261  opened by Dave Jones

#29262: Provide a way to check for *real* typing.Union instances
http://bugs.python.org/issue29262  opened by flying sheep

#29263: Implement LOAD_METHOD/CALL_METHOD for C functions
http://bugs.python.org/issue29263  opened by haypo



Most recent 15 issues with no replies (15)
==========================================

#29263: Implement LOAD_METHOD/CALL_METHOD for C functions
http://bugs.python.org/issue29263

#29262: Provide a way to check for *real* typing.Union instances
http://bugs.python.org/issue29262

#29261: Missing venv/scripts/common after "make install"
http://bugs.python.org/issue29261

#29258: __init__.py required for pkgutil.walk_packages in python3
http://bugs.python.org/issue29258

#29257: Possible error in discussion of Abstract Base Classes and abst
http://bugs.python.org/issue29257

#29256: Windows select() errors out when given no fds to select on, wh
http://bugs.python.org/issue29256

#29255: selects.KqueueSelector behaves incorrectly when no fds are reg
http://bugs.python.org/issue29255

#29253: Fix test_asyncore tests on Cygwin
http://bugs.python.org/issue29253

#29251: Class __dict__ is only a mapping proxy
http://bugs.python.org/issue29251

#29250: islink and stat follow_symlinks are inconsistent on Windows
http://bugs.python.org/issue29250

#29248: os.readlink fails on Windows
http://bugs.python.org/issue29248

#29247: Document return value of epoll.poll
http://bugs.python.org/issue29247

#29242: Crash on GC when compiling PyPy
http://bugs.python.org/issue29242

#29238: Add more kwargs to cProfile's print_stats
http://bugs.python.org/issue29238

#29236: 'an ASCII string of one or more PEM-encoded certificates' need
http://bugs.python.org/issue29236



Most recent 15 issues waiting for review (15)
=============================================

#29261: Missing venv/scripts/common after "make install"
http://bugs.python.org/issue29261

#29259: Add tp_fastcall to PyTypeObject: support FASTCALL calling conv
http://bugs.python.org/issue29259

#29253: Fix test_asyncore tests on Cygwin
http://bugs.python.org/issue29253

#29251: Class __dict__ is only a mapping proxy
http://bugs.python.org/issue29251

#29246: typing.Union raises RecursionError when comparing Union to oth
http://bugs.python.org/issue29246

#29240: Implementation of the PEP 540: Add a new UTF-8 mode
http://bugs.python.org/issue29240

#29234: Disable inlining of _PyStack_AsTuple() to reduce the stack con
http://bugs.python.org/issue29234

#29227: Reduce C stack consumption in function calls
http://bugs.python.org/issue29227

#29225: distutils.command.install_lib.get_outputs() wrong with extensi
http://bugs.python.org/issue29225

#29218: distutils: Remove unused install_misc class
http://bugs.python.org/issue29218

#29216: Space saving step for the LRU cache
http://bugs.python.org/issue29216

#29213: python -m venv activate.bat has weird mix of line endings
http://bugs.python.org/issue29213

#29212: Python 3.6 logging thread name regression with concurrent.futu
http://bugs.python.org/issue29212

#29209: Remove old-deprecated ElementTree features
http://bugs.python.org/issue29209

#29205: col_offset for AsyncFunctionDef AST nodes is wrong
http://bugs.python.org/issue29205



Top 10 most discussed issues (10)
=================================

#28870: Reduce stack consumption of PyObject_CallFunctionObjArgs() and
http://bugs.python.org/issue28870  20 msgs

#29099: sqlite3 timestamp converter cannot handle timezone
http://bugs.python.org/issue29099  18 msgs

#29240: Implementation of the PEP 540: Add a new UTF-8 mode
http://bugs.python.org/issue29240  13 msgs

#24875: pyvenv doesn??t install PIP inside a new venv with --system-si
http://bugs.python.org/issue24875   9 msgs

#28759: access to mkfifo, mknod and hard links is controled by SELinux
http://bugs.python.org/issue28759   9 msgs

#29259: Add tp_fastcall to PyTypeObject: support FASTCALL calling conv
http://bugs.python.org/issue29259   9 msgs

#28180: sys.getfilesystemencoding() should default to utf-8
http://bugs.python.org/issue28180   8 msgs

#29006: 2.7.13 _sqlite more prone to "database table is locked"
http://bugs.python.org/issue29006   8 msgs

#29178: Adding bytes.frombuffer(byteslike) constructor
http://bugs.python.org/issue29178   8 msgs

#15216: Support setting the encoding on a text stream after creation
http://bugs.python.org/issue15216   7 msgs



Issues closed (48)
==================

#16026: csv.DictReader argument names documented incorrectly
http://bugs.python.org/issue16026  closed by berker.peksag

#20804: Sentinels identity lost when pickled (unittest.mock)
http://bugs.python.org/issue20804  closed by serhiy.storchaka

#22343: Install bash activate script on Windows when using venv
http://bugs.python.org/issue22343  closed by vinay.sajip

#24699: TemporaryDirectory is cleaned up twice
http://bugs.python.org/issue24699  closed by Ilya.Kulakov

#28130: Document that time.tzset updates time module globals
http://bugs.python.org/issue28130  closed by python-dev

#28815: test_socket fails if /proc/modules is existent but not readabl
http://bugs.python.org/issue28815  closed by martin.panter

#28839: _PyFunction_FastCallDict(): replace PyTuple_New() with PyMem_M
http://bugs.python.org/issue28839  closed by haypo

#28969: lru_cache is not threadsafe
http://bugs.python.org/issue28969  closed by serhiy.storchaka

#29002: typing.AnyStr doc is unclear about python2 unicode support
http://bugs.python.org/issue29002  closed by gvanrossum

#29023: Results of random.seed() call with integer argument should be 
http://bugs.python.org/issue29023  closed by rhettinger

#29034: Fix memory leak and use-after-free in path_converter
http://bugs.python.org/issue29034  closed by xiang.zhang

#29050: xml.etree.ElementTree in Python 3.6 is incompatible with defus
http://bugs.python.org/issue29050  closed by serhiy.storchaka

#29082: In 2.7.13, _ctypes.LoadLibrary no longer accepts Unicode objec
http://bugs.python.org/issue29082  closed by serhiy.storchaka

#29133: Minor inaccuracy in shlex.shlex punctuation_chars example
http://bugs.python.org/issue29133  closed by python-dev

#29142: urllib: no_proxy variable values with leading dot not properly
http://bugs.python.org/issue29142  closed by xiang.zhang

#29145: failing overflow checks in replace_*
http://bugs.python.org/issue29145  closed by xiang.zhang

#29177: skip tests of test_logging when bind() raises PermissionError 
http://bugs.python.org/issue29177  closed by python-dev

#29186: TimeoutError isn't being raised?
http://bugs.python.org/issue29186  closed by YoSTEALTH

#29188: Backport random.c from Python 3.5 to Python 2.7
http://bugs.python.org/issue29188  closed by haypo

#29189: Broken indentation in FancyURLopener documentation
http://bugs.python.org/issue29189  closed by berker.peksag

#29190: Avoid possible errors in comparing strings
http://bugs.python.org/issue29190  closed by serhiy.storchaka

#29192: Remove deprecated features from http.cookies
http://bugs.python.org/issue29192  closed by serhiy.storchaka

#29193: Remove support of format_string as keyword argument in string.
http://bugs.python.org/issue29193  closed by serhiy.storchaka

#29194: importlib reload fails for module supplied as argument to comm
http://bugs.python.org/issue29194  closed by brett.cannon

#29195: Get rid of supporting outdated wrong keyword arguments in re m
http://bugs.python.org/issue29195  closed by serhiy.storchaka

#29200: is it a bug in `functools._HashedSeq`
http://bugs.python.org/issue29200  closed by rhettinger

#29201: Python 3.6 not working within VS 2015
http://bugs.python.org/issue29201  closed by steve.dower

#29203: With PEP 468, the lru cache not longer needs to sort keyword a
http://bugs.python.org/issue29203  closed by rhettinger

#29206: importlib reload fails for module supplied as argument to comm
http://bugs.python.org/issue29206  closed by brett.cannon

#29208: BlockingIOError during system startup
http://bugs.python.org/issue29208  closed by martin.panter

#29210: Remove the support of the exclude argument in tarfile.TarFile.
http://bugs.python.org/issue29210  closed by serhiy.storchaka

#29214: Standard open() does not allow to specify file permissions.
http://bugs.python.org/issue29214  closed by haypo

#29215: pyport.h uses non C90-style comment
http://bugs.python.org/issue29215  closed by benjamin.peterson

#29217: Documentation for uuid has wrong description for variant
http://bugs.python.org/issue29217  closed by xiang.zhang

#29219: TracebackException(capture_locals=True) may fail with Recursio
http://bugs.python.org/issue29219  closed by serhiy.storchaka

#29220: Python 3.6 regression/change using logging.addLevelName() to c
http://bugs.python.org/issue29220  closed by vinay.sajip

#29222: Python3 interactive shell hangs on
http://bugs.python.org/issue29222  closed by eryksun

#29223: Settable defaulting to decimal instead of float
http://bugs.python.org/issue29223  closed by haypo

#29224: OS related file operations (copy, move, delete, rename...) sho
http://bugs.python.org/issue29224  closed by haypo

#29226: Comment generates syntax error
http://bugs.python.org/issue29226  closed by ammar2

#29228: sqlite3 OperationalError on changing into WAL transaction mode
http://bugs.python.org/issue29228  closed by berker.peksag

#29229: incompatible: unittest.mock.sentinel and multiprocessing.Pool.
http://bugs.python.org/issue29229  closed by Jason Curtis

#29231: Broken MSIs in Python 3.5+
http://bugs.python.org/issue29231  closed by steve.dower

#29233: call_method(): call _PyObject_FastCall() rather than _PyObject
http://bugs.python.org/issue29233  closed by haypo

#29239: Fix wrong issue number in what's new entry
http://bugs.python.org/issue29239  closed by martin.panter

#29244: Python 3.6 unnecessarily requires inttypes.h
http://bugs.python.org/issue29244  closed by brett.cannon

#29245: Can't write to NamedTemporaryFile
http://bugs.python.org/issue29245  closed by r.david.murray

#29254: Documentation Error
http://bugs.python.org/issue29254  closed by ammar2

From lukasz at langa.pl  Fri Jan 13 16:12:52 2017
From: lukasz at langa.pl (Lukasz Langa)
Date: Fri, 13 Jan 2017 13:12:52 -0800
Subject: [Python-Dev] Making the new dtrace support work on OS X
In-Reply-To: <CAGY4rcXj8F2C=JR2zZeE0nOjxwfbd7HJt=7gUuj+kUV8HaJ5fQ@mail.gmail.com>
References: <CAGY4rcXj8F2C=JR2zZeE0nOjxwfbd7HJt=7gUuj+kUV8HaJ5fQ@mail.gmail.com>
Message-ID: <3888C0FC-7956-40C0-A16C-9D098D12EE49@langa.pl>

Looks like function-entry and function-return give you the C-level frame names for some reason. This was implemented on OS X 10.11 if that makes any difference. I will look at this in the evening, the laptop I'm on now is macOS Sierra with SIP which cripples dtrace.

> On Jan 12, 2017, at 5:08 AM, David Cournapeau <cournape at gmail.com> wrote:
> 
> Hi,
> 
> I was excited to see official dtrace support for python 3.6.0 on OS X, but I have not been able to make it work:
> 
> 1. I built my own python from sources on OS X 10.9,  with the --with-dtrace support
> 2. if I launch `python3.6 -q &` and then `sudo dtrace -l -P python$!`, I get the following output:
> 
>    ID   PROVIDER            MODULE                          FUNCTION NAME
>  2774 python48084         python3.6          _PyEval_EvalFrameDefault function-entry
>  2775 python48084         python3.6          _PyEval_EvalFrameDefault function-return
>  2776 python48084         python3.6                           collect gc-done
>  2777 python48084         python3.6                           collect gc-start
>  2778 python48084         python3.6          _PyEval_EvalFrameDefault line
> 
> Which looks similar but not the same as the example given in the doc at https://docs.python.org/dev/howto/instrumentation.html#enabling-the-static-markers <https://docs.python.org/dev/howto/instrumentation.html#enabling-the-static-markers>
> 
> 3. When I try to test anything with the given call_stack.d example, I can't make it work at all:
> 
> """
> # script.py
> def start():
>     foo()
> 
> def foo():
>     pass
> 
> start()
> """
> 
> I am not very familiar with dtrace, so maybe I a missing a step, there is a documentation bug, or it depends on which OS X version you are using ?
> 
> Thanks,
> David
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/lukasz%40langa.pl

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170113/21d58205/attachment.html>

From cournape at gmail.com  Fri Jan 13 19:11:49 2017
From: cournape at gmail.com (David Cournapeau)
Date: Sat, 14 Jan 2017 00:11:49 +0000
Subject: [Python-Dev] Making the new dtrace support work on OS X
In-Reply-To: <3888C0FC-7956-40C0-A16C-9D098D12EE49@langa.pl>
References: <CAGY4rcXj8F2C=JR2zZeE0nOjxwfbd7HJt=7gUuj+kUV8HaJ5fQ@mail.gmail.com>
 <3888C0FC-7956-40C0-A16C-9D098D12EE49@langa.pl>
Message-ID: <CAGY4rcUwQuBYAfTENdDihhNfigVbX0BqSMcb7O_-UAzwQr6sXA@mail.gmail.com>

On Fri, Jan 13, 2017 at 9:12 PM, Lukasz Langa <lukasz at langa.pl> wrote:

> Looks like function-entry and function-return give you the C-level frame
> names for some reason. This was implemented on OS X 10.11 if that makes any
> difference. I will look at this in the evening, the laptop I'm on now is
> macOS Sierra with SIP which cripples dtrace.
>

On that hint, I tried on OSX 11.1. sw_vers says

ProductName: Mac OS X
ProductVersion: 10.11.6
BuildVersion: 15G1108

And there, the example worked as advertised w/ my build of 3.6.0. I will
try on more versions of OS X in our test lab.

David

>
> On Jan 12, 2017, at 5:08 AM, David Cournapeau <cournape at gmail.com> wrote:
>
> Hi,
>
> I was excited to see official dtrace support for python 3.6.0 on OS X, but
> I have not been able to make it work:
>
> 1. I built my own python from sources on OS X 10.9,  with the
> --with-dtrace support
> 2. if I launch `python3.6 -q &` and then `sudo dtrace -l -P python$!`, I
> get the following output:
>
>    ID   PROVIDER            MODULE                          FUNCTION NAME
>  2774 python48084         python3.6          _PyEval_EvalFrameDefault
> function-entry
>  2775 python48084         python3.6          _PyEval_EvalFrameDefault
> function-return
>  2776 python48084         python3.6                           collect
> gc-done
>  2777 python48084         python3.6                           collect
> gc-start
>  2778 python48084         python3.6          _PyEval_EvalFrameDefault line
>
> Which looks similar but not the same as the example given in the doc at
> https://docs.python.org/dev/howto/instrumentation.
> html#enabling-the-static-markers
>
> 3. When I try to test anything with the given call_stack.d example, I
> can't make it work at all:
>
> """
> # script.py
> def start():
>     foo()
>
> def foo():
>     pass
>
> start()
> """
>
> I am not very familiar with dtrace, so maybe I a missing a step, there is
> a documentation bug, or it depends on which OS X version you are using ?
>
> Thanks,
> David
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> lukasz%40langa.pl
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170114/62906c49/attachment.html>

From xdegaye at gmail.com  Sat Jan 14 12:29:17 2017
From: xdegaye at gmail.com (Xavier de Gaye)
Date: Sat, 14 Jan 2017 18:29:17 +0100
Subject: [Python-Dev] support of the Android platform
Message-ID: <dbc927ac-5d5c-79ec-d73e-8ba48e186c04@gmail.com>

Only a few minor issues are left to be fixed before the support of the
Android platform may be considered. See the current status at msg285493 [1]
in the Android meta-issue 26865 [2].

Xavier

[1] http://bugs.python.org/issue26865#msg285493
[2] http://bugs.python.org/issue26865

From xdegaye at gmail.com  Sun Jan 15 05:17:12 2017
From: xdegaye at gmail.com (Xavier de Gaye)
Date: Sun, 15 Jan 2017 11:17:12 +0100
Subject: [Python-Dev] support of the Android platform
In-Reply-To: <CAMpsgwaLCjKjV5ux42_VfeRJb37Sdnkur26Obs4TZEDhHmw0AQ@mail.gmail.com>
References: <dbc927ac-5d5c-79ec-d73e-8ba48e186c04@gmail.com>
 <CAMpsgwaLCjKjV5ux42_VfeRJb37Sdnkur26Obs4TZEDhHmw0AQ@mail.gmail.com>
Message-ID: <6dd9b3b7-220f-947f-95e3-42ec0f56f4e9@gmail.com>

On 01/14/2017 09:56 PM, Victor Stinner wrote:
 > Great job! Thank you and Chi Hsuan Yen!
 >
 > Did you get feedback from users? Maybe from the Kivy community?
 >
 > Victor
 >
 > Le 14 janv. 2017 18:31, "Xavier de Gaye" a ?crit :
 >
 >     Only a few minor issues are left to be fixed before the support of the
 >     Android platform may be considered. See the current status at msg285493 [1]
 >     in the Android meta-issue 26865 [2].


Yes Chi Hsuan Yen is a great contributor to this project.

AFAIK Kivy uses the CrystaX NDK [1] for Python 3 instead of the
Android NDK, mainly because the Android NDK did not support wide
character until API level 21.

Xavier

[1] https://www.crystax.net/en


From armin.rigo at gmail.com  Sun Jan 15 05:43:24 2017
From: armin.rigo at gmail.com (Armin Rigo)
Date: Sun, 15 Jan 2017 11:43:24 +0100
Subject: [Python-Dev] Is there any remaining reason why weakref
 callbacks shouldn't be able to access the referenced object?
In-Reply-To: <CADiSq7c-kAyGBeFescTfoY_bz0X6P7xCmLDPrQQvHh7vV=ELTw@mail.gmail.com>
References: <CAPJVwB=KfLz1JJC+9Oxry9SR33YTqqZXdkhEJp=jiDU9TwDFoQ@mail.gmail.com>
 <CADiSq7c-kAyGBeFescTfoY_bz0X6P7xCmLDPrQQvHh7vV=ELTw@mail.gmail.com>
Message-ID: <CAMSv6X1GKLOZ5xpy=bLV3NYjAv5bRMSQZTFiLFVmswh=J1eAPg@mail.gmail.com>

Hi,

Sorry to reply in this old thread.  We just noticed this on #pypy:

On 22 October 2016 at 05:32, Nick Coghlan <ncoghlan at gmail.com> wrote:
> The weakref-before-__del__ ordering change in
> https://www.python.org/dev/peps/pep-0442/#disposal-of-cyclic-isolates
> only applies to cyclic garbage collection,so for normal refcount
> driven object cleanup in CPython, the __del__ still happens first:
>
>     >>> class C:
>     ...     def __del__(self):
>     ...         print("__del__ called")
>     ...
>     >>> c = C()
>     >>> import weakref
>     >>> def cb():
>     ...     print("weakref callback called")
>     ...
>     >>> weakref.finalize(c, cb)
>     <finalize object at 0x7f4300b710a0; for 'C' at 0x7f42f8ae3470>
>     >>> del c
>     __del__ called
>     weakref callback called

Nick, it seems you're suggesting that before PEP 442 (in CPython 3.4)
the __del__ happened before the weakref-clearing operation as well.
That's not the case: before CPython 3.4, weakrefs are always cleared
first.  The situation became more muddy in 3.4, where weakrefs are
usually cleared after the __del__ is called---in the absence of
reference cycles (so it's a backward-incompatible change).  If there
are reference cycles, then the weakref is cleared before the __del__
is called.

This can be shown in your example by replacing "weakref.finalize(c,
cb)" with an old-style "wr = weakref.ref(c, cb)".  Then CPython <= 3.3
and >= 3.4 print the two lines in opposite order.


A bient?t,

Armin.

From songofacandy at gmail.com  Tue Jan 17 01:32:26 2017
From: songofacandy at gmail.com (INADA Naoki)
Date: Tue, 17 Jan 2017 15:32:26 +0900
Subject: [Python-Dev] Can we use "designated initializer" widely in core
 modules?
Message-ID: <CAEfz+Tx9_Km03ZexEcY2vVqd4XvGCpRfMKxV3+q8TVj=bjmtrw@mail.gmail.com>

Hi.

---
This discussion is started in http://bugs.python.org/issue29259, and
there is separated issue at http://bugs.python.org/issue29260 .
But I want to see more comments, before issue 29259 is committed.
---

Since Python 3.6, some of C99 features are accepted in PEP 7.
"designated initializer" is one of them.

PEP7 says  "designated initializers (especially nice for type declarations)"

For example, issue 29259 is adding new slot named tp_fastcall in type structure.
Without designated initializer, there are many diffs like this:

     0,                                  /* tp_free */
+    0,                                  /* tp_is_gc */
+    0,                                  /* tp_bases */
+    0,                                  /* tp_mro */
+    0,                                  /* tp_cache */
+    0,                                  /* tp_subclasses */
+    0,                                  /* tp_weaklist */
+    0,                                  /* tp_del */
+    0,                                  /* tp_version_tag */
+    0,                                  /* tp_finalize */
+    (fastternaryfunc)attrgetter_call,   /* tp_fastcall */
 };

With designated initializer, it becomes:

     0,                                  /* tp_free */
+    .tp_fastcall = (fastternaryfunc)attrgetter_call,
 };

It's readable, and easy to review.
Without designated initializer, it's difficult to find copy & paste mistake.
(Can you find if I missed /* tp_is_gc */ line in first patch?)

On the other hand, this syntax is C99 specific.
C++ doesn't accept this syntax, even for newest C++14.
Using this feature make it's difficult to use (subset of) C++ features
in the future.


So, how widely can we use "designated initializer"?
Only in Modules (_asyncio uses it already)?
Or almost everywhere (Objects/ and Python/)?

I want to get decision before issue 29259 is committed, because it
will add many "0, /* tp_xxx */"

Regards,

From larry at hastings.org  Tue Jan 17 03:40:39 2017
From: larry at hastings.org (Larry Hastings)
Date: Tue, 17 Jan 2017 00:40:39 -0800
Subject: [Python-Dev] [RELEASED] Python 3.4.6 and Python 3.5.3 are now
 available
Message-ID: <ed1b85b6-ac63-8973-9134-c8e7d968f386@hastings.org>


On behalf of the Python development community and the Python 3.4 and 
Python 3.5 release teams, I'm delighted to announce the availability of 
Python 3.4.6 and Python 3.5.3.

Python 3.4 is now in "security fixes only" mode.  This is the final 
stage of support for Python 3.4.  Python 3.4 now only receives security 
fixes, not bug fixes, and Python 3.4 releases are source code only--no 
more official binary installers will be produced.

Python 3.5 is still in active "bug fix" mode.  Python 3.5.3 contains 
many incremental improvements over Python 3.5.2.

There were literally no code changes between rc1 and final for either 
release.  The only change--apart from the necessary updates from "rc1" 
to final--was a single copyright notice update for one of the OS X 
".plist" property list files in 3.5.3 final.

You can find Python 3.5.3 here:

    https://www.python.org/downloads/release/python-353/

And you can find Python 3.4.6 here:

    https://www.python.org/downloads/release/python-346/



Best wishes,


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170117/bac33443/attachment.html>

From victor.stinner at gmail.com  Tue Jan 17 08:46:49 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 17 Jan 2017 14:46:49 +0100
Subject: [Python-Dev] PyObject_CallFunction(func, "O", arg) special case
In-Reply-To: <CAMpsgwbHYAa_kshEBgZcZerywdMpV779P4jT4ncyiNPOdC1WJA@mail.gmail.com>
References: <CAMpsgwbHYAa_kshEBgZcZerywdMpV779P4jT4ncyiNPOdC1WJA@mail.gmail.com>
Message-ID: <CAMpsgwaUqMfQH62QtqTtpnR7hD7pLSpbPZtXF7iv3bwXT14tqQ@mail.gmail.com>

Oh, I found another recent bug fixed because of the "O" weird behaviour:
http://bugs.python.org/issue26478
"dict views don't implement subtraction correctly"

Victor

2016-12-09 18:46 GMT+01:00 Victor Stinner <victor.stinner at gmail.com>:
> Hi,
>
> The PyObject_CallFunction() function has a special case when the
> format string is "O", to pass exactly one Python object:
>
> * If the argument is a tuple, the tuple is unpacked: it behaves like func(*arg)
> * Otherwise, it behaves like func(arg)
>
> This case is not documented in the C API !
> https://docs.python.org/dev/c-api/object.html#c.PyObject_CallFunction
>
>
> The following C functions have the special case:
>
> * PyObject_CallFunction(), _PyObject_CallFunction_SizeT()
> * PyObject_CallMethod(), _PyObject_CallMethod_SizeT()
> * _PyObject_CallMethodId(), _PyObject_CallMethodId_SizeT()
>
>
> I guess that it's a side effect of the implementation: the code uses
> Py_BuildValue() and then checks if the value is a tuple or not.
> Py_BuildValue() is a little bit surprising:
>
> * "i" creates an integer object
> * "ii" creates a tuple
> * "(i)" and "(ii)" create a tuple.
>
> Getting a tuple or not depends on the length of the format string. It
> is not obvious when you have nested tuples like "O(OO)".
>
> Because of the special case, passing a tuple as the only argument
> requires to write "((...))" instead of just "(...)".
>
>
> In the past, this special behaviour caused a bug in
> generator.send(arg), probably because the author of the C code
> implementing generator.send() wasn't aware of the special case. See
> the issue:
> http://bugs.python.org/issue21209
>
> I found code using "O" format in the new _asyncio module, and I'm
> quite sure that unpacking special case is not expected. So I opened an
> issue:
> http://bugs.python.org/issue28920
>
>
> Last days, I patched functions of PyObject_CallFunction() family to
> use internally fast calls. I implemented the special case to keep
> backward compatibility.
>
> I replaced a lot of code using PyObject_CallFunction() with
> PyObject_CallFunctionObjArgs() when the format string was only made of
> "O", PyObject* arguments. I made this change to optimize the code, but
> indirectly, it avoids also the special case for code which used
> exactly "O" format. See:
> http://bugs.python.org/issue28915
>
> When I made these changes, I found some functions which rely the
> unpacking feature!
>
> * time_strptime() (change 49a7fdc0d40a)
> * unpickle() of _ctypes (change ceb22b8f6d32)
>
> I don't know well what we are supposed to do. I don't think that
> changing the behaviour of PyObject_CallFunction() to remove the
> special case is a good idea. It would be an obvious backward
> incompatible change which can break applications.
>
> I guess that the minimum is to document the special case?
>
> Victor

From robertomartinezp at gmail.com  Tue Jan 17 11:51:25 2017
From: robertomartinezp at gmail.com (=?UTF-8?Q?Roberto_Mart=C3=ADnez?=)
Date: Tue, 17 Jan 2017 16:51:25 +0000
Subject: [Python-Dev] collections.abc for data and non-data descriptors
Message-ID: <CAF0TK53XmG+X4VBo4OxuVEe+Xq6PAybLqfUWrbut8UoMir-GTw@mail.gmail.com>

Hi,

I miss abstract base classes in collections.abc implementing the descriptor
protocol. Any reason why they do not exist?

What do you think of adding NonDataDescriptor and DataDescriptor ABCs?

Best regards,
Roberto
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170117/08d6164b/attachment.html>

From brett at python.org  Tue Jan 17 11:55:50 2017
From: brett at python.org (Brett Cannon)
Date: Tue, 17 Jan 2017 16:55:50 +0000
Subject: [Python-Dev] Can we use "designated initializer" widely in core
 modules?
In-Reply-To: <CAEfz+Tx9_Km03ZexEcY2vVqd4XvGCpRfMKxV3+q8TVj=bjmtrw@mail.gmail.com>
References: <CAEfz+Tx9_Km03ZexEcY2vVqd4XvGCpRfMKxV3+q8TVj=bjmtrw@mail.gmail.com>
Message-ID: <CAP1=2W65qscErCM_eSvN_2-9bwYswZ+BDV0ogDeyEwZg0SXvRg@mail.gmail.com>

On Mon, 16 Jan 2017 at 22:34 INADA Naoki <songofacandy at gmail.com> wrote:

> Hi.
>
> ---
> This discussion is started in http://bugs.python.org/issue29259, and
> there is separated issue at http://bugs.python.org/issue29260 .
> But I want to see more comments, before issue 29259 is committed.
> ---
>
> Since Python 3.6, some of C99 features are accepted in PEP 7.
> "designated initializer" is one of them.
>
> PEP7 says  "designated initializers (especially nice for type
> declarations)"
>
> For example, issue 29259 is adding new slot named tp_fastcall in type
> structure.
> Without designated initializer, there are many diffs like this:
>
>      0,                                  /* tp_free */
> +    0,                                  /* tp_is_gc */
> +    0,                                  /* tp_bases */
> +    0,                                  /* tp_mro */
> +    0,                                  /* tp_cache */
> +    0,                                  /* tp_subclasses */
> +    0,                                  /* tp_weaklist */
> +    0,                                  /* tp_del */
> +    0,                                  /* tp_version_tag */
> +    0,                                  /* tp_finalize */
> +    (fastternaryfunc)attrgetter_call,   /* tp_fastcall */
>  };
>
> With designated initializer, it becomes:
>
>      0,                                  /* tp_free */
> +    .tp_fastcall = (fastternaryfunc)attrgetter_call,
>  };
>
> It's readable, and easy to review.
> Without designated initializer, it's difficult to find copy & paste
> mistake.
> (Can you find if I missed /* tp_is_gc */ line in first patch?)
>
> On the other hand, this syntax is C99 specific.
> C++ doesn't accept this syntax, even for newest C++14.
> Using this feature make it's difficult to use (subset of) C++ features
> in the future.
>

At this point I really doubt we will ever switch to C++ so I don't think
making any such theoretical transition is worth holding back cleaner code.
Plus if this isn't going into a header file that some C++ extension module
is going to be pulling in then third-party C++ code should be shielded from
the incompatibility.


>
>
> So, how widely can we use "designated initializer"?
> Only in Modules (_asyncio uses it already)?
> Or almost everywhere (Objects/ and Python/)?
>

I say everywhere we can (and that also means we should probably update the
xx* files in Modules/).

-Brett


>
> I want to get decision before issue 29259 is committed, because it
> will add many "0, /* tp_xxx */"
>
> Regards,
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170117/efecd37b/attachment.html>

From guido at python.org  Tue Jan 17 12:07:35 2017
From: guido at python.org (Guido van Rossum)
Date: Tue, 17 Jan 2017 09:07:35 -0800
Subject: [Python-Dev] collections.abc for data and non-data descriptors
In-Reply-To: <CAF0TK53XmG+X4VBo4OxuVEe+Xq6PAybLqfUWrbut8UoMir-GTw@mail.gmail.com>
References: <CAF0TK53XmG+X4VBo4OxuVEe+Xq6PAybLqfUWrbut8UoMir-GTw@mail.gmail.com>
Message-ID: <CAP7+vJ+FbtEs7Jnez4EAo5VQBbr1V-2Gr-CQMKJXK6mbeNnYig@mail.gmail.com>

Well, these are an example of the purest duck typing. Just implement
__get__ and/or __set__ (and __delete__) and you're good.

What's the situation where you miss them?

--Guido

On Tue, Jan 17, 2017 at 8:51 AM, Roberto Mart?nez <
robertomartinezp at gmail.com> wrote:

> Hi,
>
> I miss abstract base classes in collections.abc implementing the
> descriptor protocol. Any reason why they do not exist?
>
> What do you think of adding NonDataDescriptor and DataDescriptor ABCs?
>
> Best regards,
> Roberto
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170117/bb7d8eae/attachment.html>

From robertomartinezp at gmail.com  Tue Jan 17 12:41:25 2017
From: robertomartinezp at gmail.com (=?UTF-8?Q?Roberto_Mart=C3=ADnez?=)
Date: Tue, 17 Jan 2017 17:41:25 +0000
Subject: [Python-Dev] collections.abc for data and non-data descriptors
In-Reply-To: <CAP7+vJ+FbtEs7Jnez4EAo5VQBbr1V-2Gr-CQMKJXK6mbeNnYig@mail.gmail.com>
References: <CAF0TK53XmG+X4VBo4OxuVEe+Xq6PAybLqfUWrbut8UoMir-GTw@mail.gmail.com>
 <CAP7+vJ+FbtEs7Jnez4EAo5VQBbr1V-2Gr-CQMKJXK6mbeNnYig@mail.gmail.com>
Message-ID: <CAF0TK50rA=yFSOvuq0n23fLG2tApYgSFEdD9oH86nW2dZTkx2w@mail.gmail.com>

I need to check if some objects obey the descriptor protocol in the
construction of a metaclass. I expect to be able to write something like:

if isinstance(myobj, abc.DataDescriptor):
   # do something

The other idea crossing my mind is something like:

if all(hasattr(myobj, attr) for attr in ('__get__', '__set__')):
    # do something


El mar., 17 ene. 2017 a las 18:07, Guido van Rossum (<guido at python.org>)
escribi?:

> Well, these are an example of the purest duck typing. Just implement
> __get__ and/or __set__ (and __delete__) and you're good.
>
> What's the situation where you miss them?
>
> --Guido
>
> On Tue, Jan 17, 2017 at 8:51 AM, Roberto Mart?nez <
> robertomartinezp at gmail.com> wrote:
>
> Hi,
>
> I miss abstract base classes in collections.abc implementing the
> descriptor protocol. Any reason why they do not exist?
>
> What do you think of adding NonDataDescriptor and DataDescriptor ABCs?
>
> Best regards,
> Roberto
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170117/e9d57120/attachment.html>

From guido at python.org  Tue Jan 17 12:55:31 2017
From: guido at python.org (Guido van Rossum)
Date: Tue, 17 Jan 2017 09:55:31 -0800
Subject: [Python-Dev] collections.abc for data and non-data descriptors
In-Reply-To: <CAF0TK50rA=yFSOvuq0n23fLG2tApYgSFEdD9oH86nW2dZTkx2w@mail.gmail.com>
References: <CAF0TK53XmG+X4VBo4OxuVEe+Xq6PAybLqfUWrbut8UoMir-GTw@mail.gmail.com>
 <CAP7+vJ+FbtEs7Jnez4EAo5VQBbr1V-2Gr-CQMKJXK6mbeNnYig@mail.gmail.com>
 <CAF0TK50rA=yFSOvuq0n23fLG2tApYgSFEdD9oH86nW2dZTkx2w@mail.gmail.com>
Message-ID: <CAP7+vJ+OnLyc7y_YjWX2GF_s4zmkbdSr_miAjpzv5U9nx2Qtdg@mail.gmail.com>

For this use case I see nothing wrong with hasattr(myobj, '__set__').

On Tue, Jan 17, 2017 at 9:41 AM, Roberto Mart?nez <
robertomartinezp at gmail.com> wrote:

> I need to check if some objects obey the descriptor protocol in the
> construction of a metaclass. I expect to be able to write something like:
>
> if isinstance(myobj, abc.DataDescriptor):
>    # do something
>
> The other idea crossing my mind is something like:
>
> if all(hasattr(myobj, attr) for attr in ('__get__', '__set__')):
>     # do something
>
>
> El mar., 17 ene. 2017 a las 18:07, Guido van Rossum (<guido at python.org>)
> escribi?:
>
>> Well, these are an example of the purest duck typing. Just implement
>> __get__ and/or __set__ (and __delete__) and you're good.
>>
>> What's the situation where you miss them?
>>
>> --Guido
>>
>> On Tue, Jan 17, 2017 at 8:51 AM, Roberto Mart?nez <
>> robertomartinezp at gmail.com> wrote:
>>
>> Hi,
>>
>> I miss abstract base classes in collections.abc implementing the
>> descriptor protocol. Any reason why they do not exist?
>>
>> What do you think of adding NonDataDescriptor and DataDescriptor ABCs?
>>
>> Best regards,
>> Roberto
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
>> guido%40python.org
>>
>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170117/acc74cd1/attachment.html>

From robertomartinezp at gmail.com  Tue Jan 17 13:18:47 2017
From: robertomartinezp at gmail.com (=?UTF-8?Q?Roberto_Mart=C3=ADnez?=)
Date: Tue, 17 Jan 2017 18:18:47 +0000
Subject: [Python-Dev] collections.abc for data and non-data descriptors
In-Reply-To: <CAP7+vJ+OnLyc7y_YjWX2GF_s4zmkbdSr_miAjpzv5U9nx2Qtdg@mail.gmail.com>
References: <CAF0TK53XmG+X4VBo4OxuVEe+Xq6PAybLqfUWrbut8UoMir-GTw@mail.gmail.com>
 <CAP7+vJ+FbtEs7Jnez4EAo5VQBbr1V-2Gr-CQMKJXK6mbeNnYig@mail.gmail.com>
 <CAF0TK50rA=yFSOvuq0n23fLG2tApYgSFEdD9oH86nW2dZTkx2w@mail.gmail.com>
 <CAP7+vJ+OnLyc7y_YjWX2GF_s4zmkbdSr_miAjpzv5U9nx2Qtdg@mail.gmail.com>
Message-ID: <CAF0TK535RmB9fF14RvFbF=+0nyQZ4CGCB8LJnAQgwnd2eOX2ag@mail.gmail.com>

Well, for me having to check both __get__ and __str__ for a data descriptor
feels inelegant.

After read the documentation of collections.abc again "This module provides
abstract base classes that can be used to test whether a class provides a
particular interface; for example, whether it is hashable or whether it is
a mapping." I ask: what is the criteria to decide which interfaces are
worth implemented as an ABC in the standard library?

To my understanding, any well defined python protocol should have a proper
ABC.

El mar., 17 ene. 2017 a las 18:55, Guido van Rossum (<guido at python.org>)
escribi?:

> For this use case I see nothing wrong with hasattr(myobj, '__set__').
>
> On Tue, Jan 17, 2017 at 9:41 AM, Roberto Mart?nez <
> robertomartinezp at gmail.com> wrote:
>
> I need to check if some objects obey the descriptor protocol in the
> construction of a metaclass. I expect to be able to write something like:
>
> if isinstance(myobj, abc.DataDescriptor):
>    # do something
>
> The other idea crossing my mind is something like:
>
> if all(hasattr(myobj, attr) for attr in ('__get__', '__set__')):
>     # do something
>
>
> El mar., 17 ene. 2017 a las 18:07, Guido van Rossum (<guido at python.org>)
> escribi?:
>
> Well, these are an example of the purest duck typing. Just implement
> __get__ and/or __set__ (and __delete__) and you're good.
>
> What's the situation where you miss them?
>
> --Guido
>
> On Tue, Jan 17, 2017 at 8:51 AM, Roberto Mart?nez <
> robertomartinezp at gmail.com> wrote:
>
> Hi,
>
> I miss abstract base classes in collections.abc implementing the
> descriptor protocol. Any reason why they do not exist?
>
> What do you think of adding NonDataDescriptor and DataDescriptor ABCs?
>
> Best regards,
> Roberto
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170117/3c133f89/attachment.html>

From guido at python.org  Tue Jan 17 13:40:34 2017
From: guido at python.org (Guido van Rossum)
Date: Tue, 17 Jan 2017 10:40:34 -0800
Subject: [Python-Dev] collections.abc for data and non-data descriptors
In-Reply-To: <CAF0TK535RmB9fF14RvFbF=+0nyQZ4CGCB8LJnAQgwnd2eOX2ag@mail.gmail.com>
References: <CAF0TK53XmG+X4VBo4OxuVEe+Xq6PAybLqfUWrbut8UoMir-GTw@mail.gmail.com>
 <CAP7+vJ+FbtEs7Jnez4EAo5VQBbr1V-2Gr-CQMKJXK6mbeNnYig@mail.gmail.com>
 <CAF0TK50rA=yFSOvuq0n23fLG2tApYgSFEdD9oH86nW2dZTkx2w@mail.gmail.com>
 <CAP7+vJ+OnLyc7y_YjWX2GF_s4zmkbdSr_miAjpzv5U9nx2Qtdg@mail.gmail.com>
 <CAF0TK535RmB9fF14RvFbF=+0nyQZ4CGCB8LJnAQgwnd2eOX2ag@mail.gmail.com>
Message-ID: <CAP7+vJKhHw+YNjV4c5JqAhNHL2aFvfcPq2fqd+gf4woJWTZc6A@mail.gmail.com>

But you can never get people to use the new ABCs consistently.

Anyway, I'm not -1, I'm just -0 -- I don't think it'll make any difference
and you're better off just checking for __set__ (if it only has __set__ but
not __get__, do you care what it is?).

On Tue, Jan 17, 2017 at 10:18 AM, Roberto Mart?nez <
robertomartinezp at gmail.com> wrote:

> Well, for me having to check both __get__ and __str__ for a data
> descriptor feels inelegant.
>
> After read the documentation of collections.abc again "This module
> provides abstract base classes that can be used to test whether a class
> provides a particular interface; for example, whether it is hashable or
> whether it is a mapping." I ask: what is the criteria to decide which
> interfaces are worth implemented as an ABC in the standard library?
>
> To my understanding, any well defined python protocol should have a proper
> ABC.
>
> El mar., 17 ene. 2017 a las 18:55, Guido van Rossum (<guido at python.org>)
> escribi?:
>
>> For this use case I see nothing wrong with hasattr(myobj, '__set__').
>>
>> On Tue, Jan 17, 2017 at 9:41 AM, Roberto Mart?nez <
>> robertomartinezp at gmail.com> wrote:
>>
>> I need to check if some objects obey the descriptor protocol in the
>> construction of a metaclass. I expect to be able to write something like:
>>
>> if isinstance(myobj, abc.DataDescriptor):
>>    # do something
>>
>> The other idea crossing my mind is something like:
>>
>> if all(hasattr(myobj, attr) for attr in ('__get__', '__set__')):
>>     # do something
>>
>>
>> El mar., 17 ene. 2017 a las 18:07, Guido van Rossum (<guido at python.org>)
>> escribi?:
>>
>> Well, these are an example of the purest duck typing. Just implement
>> __get__ and/or __set__ (and __delete__) and you're good.
>>
>> What's the situation where you miss them?
>>
>> --Guido
>>
>> On Tue, Jan 17, 2017 at 8:51 AM, Roberto Mart?nez <
>> robertomartinezp at gmail.com> wrote:
>>
>> Hi,
>>
>> I miss abstract base classes in collections.abc implementing the
>> descriptor protocol. Any reason why they do not exist?
>>
>> What do you think of adding NonDataDescriptor and DataDescriptor ABCs?
>>
>> Best regards,
>> Roberto
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
>> guido%40python.org
>>
>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>>
>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170117/18d7919e/attachment-0001.html>

From steve.dower at python.org  Tue Jan 17 15:02:12 2017
From: steve.dower at python.org (Steve Dower)
Date: Tue, 17 Jan 2017 12:02:12 -0800
Subject: [Python-Dev] Can we use "designated initializer" widely in
 coremodules?
In-Reply-To: <CAP1=2W65qscErCM_eSvN_2-9bwYswZ+BDV0ogDeyEwZg0SXvRg@mail.gmail.com>
References: <CAEfz+Tx9_Km03ZexEcY2vVqd4XvGCpRfMKxV3+q8TVj=bjmtrw@mail.gmail.com>
 <CAP1=2W65qscErCM_eSvN_2-9bwYswZ+BDV0ogDeyEwZg0SXvRg@mail.gmail.com>
Message-ID: <E1cTZxZ-00085X-BF@se2-syd.hostedmail.net.au>

Avoiding header files would be my only request. As Brett says, the C99 requirement should not be enforced on all embedders or extenders, so we should try and keep the headers they'll use as compatible as possible.

Cheers,
Steve

Top-posted from my Windows Phone

-----Original Message-----
From: "Brett Cannon" <brett at python.org>
Sent: ?1/?17/?2017 8:58
To: "INADA Naoki" <songofacandy at gmail.com>; "Python-Dev" <python-dev at python.org>
Subject: Re: [Python-Dev] Can we use "designated initializer" widely in coremodules?




On Mon, 16 Jan 2017 at 22:34 INADA Naoki <songofacandy at gmail.com> wrote:

Hi.

---
This discussion is started in http://bugs.python.org/issue29259, and
there is separated issue at http://bugs.python.org/issue29260 .
But I want to see more comments, before issue 29259 is committed.
---

Since Python 3.6, some of C99 features are accepted in PEP 7.
"designated initializer" is one of them.

PEP7 says  "designated initializers (especially nice for type declarations)"

For example, issue 29259 is adding new slot named tp_fastcall in type structure.
Without designated initializer, there are many diffs like this:

     0,                                  /* tp_free */
+    0,                                  /* tp_is_gc */
+    0,                                  /* tp_bases */
+    0,                                  /* tp_mro */
+    0,                                  /* tp_cache */
+    0,                                  /* tp_subclasses */
+    0,                                  /* tp_weaklist */
+    0,                                  /* tp_del */
+    0,                                  /* tp_version_tag */
+    0,                                  /* tp_finalize */
+    (fastternaryfunc)attrgetter_call,   /* tp_fastcall */
 };

With designated initializer, it becomes:

     0,                                  /* tp_free */
+    .tp_fastcall = (fastternaryfunc)attrgetter_call,
 };

It's readable, and easy to review.
Without designated initializer, it's difficult to find copy & paste mistake.
(Can you find if I missed /* tp_is_gc */ line in first patch?)

On the other hand, this syntax is C99 specific.
C++ doesn't accept this syntax, even for newest C++14.
Using this feature make it's difficult to use (subset of) C++ features
in the future.



At this point I really doubt we will ever switch to C++ so I don't think making any such theoretical transition is worth holding back cleaner code. Plus if this isn't going into a header file that some C++ extension module is going to be pulling in then third-party C++ code should be shielded from the incompatibility.
 


So, how widely can we use "designated initializer"?
Only in Modules (_asyncio uses it already)?
Or almost everywhere (Objects/ and Python/)?



I say everywhere we can (and that also means we should probably update the xx* files in Modules/).


-Brett
 

I want to get decision before issue 29259 is committed, because it
will add many "0, /* tp_xxx */"

Regards,
_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170117/779c4633/attachment.html>

From robertomartinezp at gmail.com  Tue Jan 17 14:41:24 2017
From: robertomartinezp at gmail.com (=?UTF-8?Q?Roberto_Mart=C3=ADnez?=)
Date: Tue, 17 Jan 2017 19:41:24 +0000
Subject: [Python-Dev] collections.abc for data and non-data descriptors
In-Reply-To: <CAP7+vJKhHw+YNjV4c5JqAhNHL2aFvfcPq2fqd+gf4woJWTZc6A@mail.gmail.com>
References: <CAF0TK53XmG+X4VBo4OxuVEe+Xq6PAybLqfUWrbut8UoMir-GTw@mail.gmail.com>
 <CAP7+vJ+FbtEs7Jnez4EAo5VQBbr1V-2Gr-CQMKJXK6mbeNnYig@mail.gmail.com>
 <CAF0TK50rA=yFSOvuq0n23fLG2tApYgSFEdD9oH86nW2dZTkx2w@mail.gmail.com>
 <CAP7+vJ+OnLyc7y_YjWX2GF_s4zmkbdSr_miAjpzv5U9nx2Qtdg@mail.gmail.com>
 <CAF0TK535RmB9fF14RvFbF=+0nyQZ4CGCB8LJnAQgwnd2eOX2ag@mail.gmail.com>
 <CAP7+vJKhHw+YNjV4c5JqAhNHL2aFvfcPq2fqd+gf4woJWTZc6A@mail.gmail.com>
Message-ID: <CAF0TK53cKpB1TeBHBnJOyk6+HLMWrvBRPruwGQST0owZg8NRZQ@mail.gmail.com>

Oh, I understand. Maybe is not worth the effort anyway.

Thank you

El mar., 17 ene. 2017 a las 19:40, Guido van Rossum (<guido at python.org>)
escribi?:

But you can never get people to use the new ABCs consistently.

Anyway, I'm not -1, I'm just -0 -- I don't think it'll make any difference
and you're better off just checking for __set__ (if it only has __set__ but
not __get__, do you care what it is?).

On Tue, Jan 17, 2017 at 10:18 AM, Roberto Mart?nez <
robertomartinezp at gmail.com> wrote:

Well, for me having to check both __get__ and __str__ for a data descriptor
feels inelegant.

After read the documentation of collections.abc again "This module provides
abstract base classes that can be used to test whether a class provides a
particular interface; for example, whether it is hashable or whether it is
a mapping." I ask: what is the criteria to decide which interfaces are
worth implemented as an ABC in the standard library?

To my understanding, any well defined python protocol should have a proper
ABC.

El mar., 17 ene. 2017 a las 18:55, Guido van Rossum (<guido at python.org>)
escribi?:

For this use case I see nothing wrong with hasattr(myobj, '__set__').

On Tue, Jan 17, 2017 at 9:41 AM, Roberto Mart?nez <
robertomartinezp at gmail.com> wrote:

I need to check if some objects obey the descriptor protocol in the
construction of a metaclass. I expect to be able to write something like:

if isinstance(myobj, abc.DataDescriptor):
   # do something

The other idea crossing my mind is something like:

if all(hasattr(myobj, attr) for attr in ('__get__', '__set__')):
    # do something


El mar., 17 ene. 2017 a las 18:07, Guido van Rossum (<guido at python.org>)
escribi?:

Well, these are an example of the purest duck typing. Just implement
__get__ and/or __set__ (and __delete__) and you're good.

What's the situation where you miss them?

--Guido

On Tue, Jan 17, 2017 at 8:51 AM, Roberto Mart?nez <
robertomartinezp at gmail.com> wrote:

Hi,

I miss abstract base classes in collections.abc implementing the descriptor
protocol. Any reason why they do not exist?

What do you think of adding NonDataDescriptor and DataDescriptor ABCs?

Best regards,
Roberto

_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/guido%40python.org




-- 
--Guido van Rossum (python.org/~guido)




-- 
--Guido van Rossum (python.org/~guido)




-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170117/d8ecde7f/attachment.html>

From yselivanov.ml at gmail.com  Tue Jan 17 16:39:41 2017
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Tue, 17 Jan 2017 16:39:41 -0500
Subject: [Python-Dev] Can we use "designated initializer" widely in core
 modules?
In-Reply-To: <CAP1=2W65qscErCM_eSvN_2-9bwYswZ+BDV0ogDeyEwZg0SXvRg@mail.gmail.com>
References: <CAEfz+Tx9_Km03ZexEcY2vVqd4XvGCpRfMKxV3+q8TVj=bjmtrw@mail.gmail.com>
 <CAP1=2W65qscErCM_eSvN_2-9bwYswZ+BDV0ogDeyEwZg0SXvRg@mail.gmail.com>
Message-ID: <4d2a7c85-c1c0-8721-cdce-bda80b226428@gmail.com>



On 2017-01-17 11:55 AM, Brett Cannon wrote:
>> So, how widely can we use "designated initializer"?
>> Only in Modules (_asyncio uses it already)?
>> Or almost everywhere (Objects/ and Python/)?
>>
> I say everywhere we can (and that also means we should probably update the
> xx* files in Modules/).
+1.

Yury

From p.f.moore at gmail.com  Tue Jan 17 18:34:16 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 17 Jan 2017 23:34:16 +0000
Subject: [Python-Dev] Can we use "designated initializer" widely in
 coremodules?
In-Reply-To: <E1cTZxZ-00085X-BF@se2-syd.hostedmail.net.au>
References: <CAEfz+Tx9_Km03ZexEcY2vVqd4XvGCpRfMKxV3+q8TVj=bjmtrw@mail.gmail.com>
 <CAP1=2W65qscErCM_eSvN_2-9bwYswZ+BDV0ogDeyEwZg0SXvRg@mail.gmail.com>
 <E1cTZxZ-00085X-BF@se2-syd.hostedmail.net.au>
Message-ID: <CACac1F_chcmLMEU3ST0o7Aas1_YvmsBrnvsWrQQn8CgC9piPUA@mail.gmail.com>

On 17 January 2017 at 20:02, Steve Dower <steve.dower at python.org> wrote:
> Avoiding header files would be my only request. As Brett says, the C99
> requirement should not be enforced on all embedders or extenders, so we
> should try and keep the headers they'll use as compatible as possible.

Agreed, we probably shouldn't require users of Python.h to be
C99-compliant. (Disclaimer: I have no idea how many compilers *don't*
implement this feature).

Paul

From larry at hastings.org  Tue Jan 17 18:48:01 2017
From: larry at hastings.org (Larry Hastings)
Date: Tue, 17 Jan 2017 15:48:01 -0800
Subject: [Python-Dev] Can we use "designated initializer" widely in
 coremodules?
In-Reply-To: <E1cTZxZ-00085X-BF@se2-syd.hostedmail.net.au>
References: <CAEfz+Tx9_Km03ZexEcY2vVqd4XvGCpRfMKxV3+q8TVj=bjmtrw@mail.gmail.com>
 <CAP1=2W65qscErCM_eSvN_2-9bwYswZ+BDV0ogDeyEwZg0SXvRg@mail.gmail.com>
 <E1cTZxZ-00085X-BF@se2-syd.hostedmail.net.au>
Message-ID: <3f6ab374-f6df-e35c-c860-a232fe542395@hastings.org>


On 01/17/2017 12:02 PM, Steve Dower wrote:
> Avoiding header files would be my only request. As Brett says, the C99 
> requirement should not be enforced on all embedders or extenders, so 
> we should try and keep the headers they'll use as compatible as possible.


While that's a reasonable policy, unless we have a way to automatically 
detect that I suspect C99 stuff will creep into the header files and 
break the non-C99 customers.  Maybe we could get some sort of buildbot 
that exercises this scenario?


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170117/64ebad5b/attachment-0001.html>

From python at lucidity.plus.com  Tue Jan 17 19:00:05 2017
From: python at lucidity.plus.com (Erik)
Date: Wed, 18 Jan 2017 00:00:05 +0000
Subject: [Python-Dev] Can we use "designated initializer" widely in core
 modules?
In-Reply-To: <CAEfz+Tx9_Km03ZexEcY2vVqd4XvGCpRfMKxV3+q8TVj=bjmtrw@mail.gmail.com>
References: <CAEfz+Tx9_Km03ZexEcY2vVqd4XvGCpRfMKxV3+q8TVj=bjmtrw@mail.gmail.com>
Message-ID: <1c50fa66-249c-3263-7f59-1fcc0a17b563@lucidity.plus.com>

On 17/01/17 06:32, INADA Naoki wrote:
> With designated initializer, it becomes:
>
>      0,                                  /* tp_free */
> +    .tp_fastcall = (fastternaryfunc)attrgetter_call,
>  };
>
> It's readable, and easy to review.

FWIW, I dislike mixing the two forms (because it still prevents the 
structure layout changing - or introduces bugs if it does - though I 
guess in Python's case that's not likely to happen).

PEP 7 doesn't mention anything about this and I doubt a wholesale 
conversion effort to the C99 syntax would be desirable, but perhaps a 
recommendation that if a new initializer is being added then the whole 
expression should be changed to the C99 syntax is reasonable.

Regards, E.

From songofacandy at gmail.com  Tue Jan 17 19:48:01 2017
From: songofacandy at gmail.com (INADA Naoki)
Date: Wed, 18 Jan 2017 09:48:01 +0900
Subject: [Python-Dev] Can we use "designated initializer" widely in
 coremodules?
In-Reply-To: <3f6ab374-f6df-e35c-c860-a232fe542395@hastings.org>
References: <CAEfz+Tx9_Km03ZexEcY2vVqd4XvGCpRfMKxV3+q8TVj=bjmtrw@mail.gmail.com>
 <CAP1=2W65qscErCM_eSvN_2-9bwYswZ+BDV0ogDeyEwZg0SXvRg@mail.gmail.com>
 <E1cTZxZ-00085X-BF@se2-syd.hostedmail.net.au>
 <3f6ab374-f6df-e35c-c860-a232fe542395@hastings.org>
Message-ID: <CAEfz+Ty=RpaFhLKTiCm=p1eV-eP_Pw82yVH4m6GfKBX0DgNdJw@mail.gmail.com>

On Wed, Jan 18, 2017 at 8:48 AM, Larry Hastings <larry at hastings.org> wrote:
>
> On 01/17/2017 12:02 PM, Steve Dower wrote:
>
> Avoiding header files would be my only request. As Brett says, the C99
> requirement should not be enforced on all embedders or extenders, so we
> should try and keep the headers they'll use as compatible as possible.
>
>

C99 style comment is used in header file already.
see http://bugs.python.org/issue29215

>
> While that's a reasonable policy, unless we have a way to automatically
> detect that I suspect C99 stuff will creep into the header files and break
> the non-C99 customers.  Maybe we could get some sort of buildbot that
> exercises this scenario?
>

How about `gcc -ansi` ?

$ cat head.c
#include <Python.h>

$ gcc `python-config --cflags` -ansi -c head.c
In file included from
/home/inada-n/pyenv/versions/3.6.0/include/python3.6m/Python.h:50:0,
                 from head.c:1:
/home/inada-n/pyenv/versions/3.6.0/include/python3.6m/pyport.h:40:1:
error: C++ style comments are not allowed in ISO C90
 // long long is required. Ensure HAVE_LONG_LONG is defined for compatibility.
 ^
/home/inada-n/pyenv/versions/3.6.0/include/python3.6m/pyport.h:40:1:
error: (this will be reported only once per input file)
~


>
> /arry
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com
>

From larry at hastings.org  Tue Jan 17 19:59:33 2017
From: larry at hastings.org (Larry Hastings)
Date: Tue, 17 Jan 2017 16:59:33 -0800
Subject: [Python-Dev] Can we use "designated initializer" widely in
 coremodules?
In-Reply-To: <CAEfz+Ty=RpaFhLKTiCm=p1eV-eP_Pw82yVH4m6GfKBX0DgNdJw@mail.gmail.com>
References: <CAEfz+Tx9_Km03ZexEcY2vVqd4XvGCpRfMKxV3+q8TVj=bjmtrw@mail.gmail.com>
 <CAP1=2W65qscErCM_eSvN_2-9bwYswZ+BDV0ogDeyEwZg0SXvRg@mail.gmail.com>
 <E1cTZxZ-00085X-BF@se2-syd.hostedmail.net.au>
 <3f6ab374-f6df-e35c-c860-a232fe542395@hastings.org>
 <CAEfz+Ty=RpaFhLKTiCm=p1eV-eP_Pw82yVH4m6GfKBX0DgNdJw@mail.gmail.com>
Message-ID: <97c137c3-3f08-f905-045c-24e05713f48e@hastings.org>


On 01/17/2017 04:48 PM, INADA Naoki wrote:
> On Wed, Jan 18, 2017 at 8:48 AM, Larry Hastings<larry at hastings.org>  wrote:
>> While that's a reasonable policy, unless we have a way to automatically
>> detect that I suspect C99 stuff will creep into the header files and break
>> the non-C99 customers.  Maybe we could get some sort of buildbot that
>> exercises this scenario?
>>
> How about `gcc -ansi` ?

That seems like it should work fine.  Perhaps we could even add this gcc 
-ansi step to the normal Python build process.  It shouldn't take very 
long, so it wouldn't slow down the build very much, and since most 
Python development happens on UNIX platforms (I think) this would help 
prevent more C99 constructs from creeping into the header files.


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170117/81a326df/attachment.html>

From songofacandy at gmail.com  Tue Jan 17 19:59:20 2017
From: songofacandy at gmail.com (INADA Naoki)
Date: Wed, 18 Jan 2017 09:59:20 +0900
Subject: [Python-Dev] Can we use "designated initializer" widely in core
 modules?
In-Reply-To: <1c50fa66-249c-3263-7f59-1fcc0a17b563@lucidity.plus.com>
References: <CAEfz+Tx9_Km03ZexEcY2vVqd4XvGCpRfMKxV3+q8TVj=bjmtrw@mail.gmail.com>
 <1c50fa66-249c-3263-7f59-1fcc0a17b563@lucidity.plus.com>
Message-ID: <CAEfz+TyK7Eq0NoPxXbZO0chQNynk8_pY44PXwV6PVbm+8T+i=g@mail.gmail.com>

On Wed, Jan 18, 2017 at 9:00 AM, Erik <python at lucidity.plus.com> wrote:
> On 17/01/17 06:32, INADA Naoki wrote:
>>
>> With designated initializer, it becomes:
>>
>>      0,                                  /* tp_free */
>> +    .tp_fastcall = (fastternaryfunc)attrgetter_call,
>>  };
>>
>> It's readable, and easy to review.
>
>
> FWIW, I dislike mixing the two forms (because it still prevents the
> structure layout changing - or introduces bugs if it does - though I guess
> in Python's case that's not likely to happen).
>
> PEP 7 doesn't mention anything about this and I doubt a wholesale conversion
> effort to the C99 syntax would be desirable, but perhaps a recommendation
> that if a new initializer is being added then the whole expression should be
> changed to the C99 syntax is reasonable.
>
> Regards, E.
>

I think mixing two forms is OK only if new form is used only at bottom.
(like keyword arguments are allowed after all positional arguments in
Python function calling)

Complete rewriting makes diff huge.  And there is PyVarObject_HEAD_INIT already.

From victor.stinner at gmail.com  Tue Jan 17 20:16:28 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 18 Jan 2017 02:16:28 +0100
Subject: [Python-Dev] Can we use "designated initializer" widely in core
 modules?
In-Reply-To: <CAEfz+TyK7Eq0NoPxXbZO0chQNynk8_pY44PXwV6PVbm+8T+i=g@mail.gmail.com>
References: <CAEfz+Tx9_Km03ZexEcY2vVqd4XvGCpRfMKxV3+q8TVj=bjmtrw@mail.gmail.com>
 <1c50fa66-249c-3263-7f59-1fcc0a17b563@lucidity.plus.com>
 <CAEfz+TyK7Eq0NoPxXbZO0chQNynk8_pY44PXwV6PVbm+8T+i=g@mail.gmail.com>
Message-ID: <CAMpsgwZoFhXvoz5DJZ=8fW_2tEEekiZuBv7+jdaT41TE+xmtOg@mail.gmail.com>

2017-01-18 1:59 GMT+01:00 INADA Naoki <songofacandy at gmail.com>:
> I think mixing two forms is OK only if new form is used only at bottom.
> (like keyword arguments are allowed after all positional arguments in
> Python function calling)
>
> Complete rewriting makes diff huge.  And there is PyVarObject_HEAD_INIT already.

I'm in favor of replacing all long list of fields with the /* tp_xxx
*/ comments to use designated initializers. It would allow to remove a
lot of "0,     /* tp_xxx */" lines and make the code much more
readable! It should help to prevent bugs when the code is modified.

Victor

From raymond.hettinger at gmail.com  Tue Jan 17 20:52:54 2017
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Tue, 17 Jan 2017 17:52:54 -0800
Subject: [Python-Dev] Can we use "designated initializer" widely in core
 modules?
In-Reply-To: <CAMpsgwZoFhXvoz5DJZ=8fW_2tEEekiZuBv7+jdaT41TE+xmtOg@mail.gmail.com>
References: <CAEfz+Tx9_Km03ZexEcY2vVqd4XvGCpRfMKxV3+q8TVj=bjmtrw@mail.gmail.com>
 <1c50fa66-249c-3263-7f59-1fcc0a17b563@lucidity.plus.com>
 <CAEfz+TyK7Eq0NoPxXbZO0chQNynk8_pY44PXwV6PVbm+8T+i=g@mail.gmail.com>
 <CAMpsgwZoFhXvoz5DJZ=8fW_2tEEekiZuBv7+jdaT41TE+xmtOg@mail.gmail.com>
Message-ID: <1D156195-41CF-413B-A954-FDE8942741DF@gmail.com>


> On Jan 17, 2017, at 5:16 PM, Victor Stinner <victor.stinner at gmail.com> wrote:
> 
>  /* tp_xxx */" lines and make the code much more
> readable! It should help to prevent bugs when the code is modified.

I'm +0 on the change (it is a massive code churn with a huge diff and won't match any existing C extensions or published books on Python) but it does have advantages in the face of all the new tp slots being proposed which would be painful regardless.

I don't want to pretend that it really prevents bugs though.  Historically, this just hasn't been a problem for us in practice (I've written and reviewed many, many of these structs over the years).  In fact, the change may become a source of bugs as people forget to fill-in slots or lose their cues as to what slots are available (I for one use the current copy and paste as a checklist for what should and shouldn't be included).

I would really like to hear Guido's opinion on the subject (IIRC he designed the current system of slots and has had to live with its pros and cons over a long period time).  Also at one time, Guido opined that he was somewhat resistant to adding new slots (otherwise, we would already have a __length_hint__ slot which I proposed a few years ago).

Lastly, if we do go forward with this sweeping change, we should also explore the possibility of flattening the current tp_as_sequence, tp_as_mapping, and tp_as_number indirections. These have historically have been somewhat of a pain and have led to many otherwise unnecessary indirections.



Raymond


P.S.  I'm not sure if we care about the size of the types but they are growing at an unprecedented rate (amidst lots of other code churn as well).





From guido at python.org  Tue Jan 17 21:28:30 2017
From: guido at python.org (Guido van Rossum)
Date: Tue, 17 Jan 2017 18:28:30 -0800
Subject: [Python-Dev] Can we use "designated initializer" widely in core
 modules?
In-Reply-To: <1D156195-41CF-413B-A954-FDE8942741DF@gmail.com>
References: <CAEfz+Tx9_Km03ZexEcY2vVqd4XvGCpRfMKxV3+q8TVj=bjmtrw@mail.gmail.com>
 <1c50fa66-249c-3263-7f59-1fcc0a17b563@lucidity.plus.com>
 <CAEfz+TyK7Eq0NoPxXbZO0chQNynk8_pY44PXwV6PVbm+8T+i=g@mail.gmail.com>
 <CAMpsgwZoFhXvoz5DJZ=8fW_2tEEekiZuBv7+jdaT41TE+xmtOg@mail.gmail.com>
 <1D156195-41CF-413B-A954-FDE8942741DF@gmail.com>
Message-ID: <CAP7+vJLOcQJg=SmC_+nnYMdecjyPDQwD5GeT9q=MheJ2h+eO+A@mail.gmail.com>

I'm for allowing the new convention. If it had been supported 20 years ago
I would've used it.

I'm against changing any existing code to use it -- such massive changes
are high risk and low reward. Just do it for new fields or new types.

I recommend being reluctant to add new fields -- the number of type objects
is larger than you'd think and these are static bytes which I consider a
relatively expensive resource.

There's also an ABI issue with new fields -- new fields can only be
accessed after checking that the type object has been compiled with a
version of the headers that defines the field, else you'd be accessing
garbage bytes.

On Tue, Jan 17, 2017 at 5:52 PM, Raymond Hettinger <
raymond.hettinger at gmail.com> wrote:

>
> > On Jan 17, 2017, at 5:16 PM, Victor Stinner <victor.stinner at gmail.com>
> wrote:
> >
> >  /* tp_xxx */" lines and make the code much more
> > readable! It should help to prevent bugs when the code is modified.
>
> I'm +0 on the change (it is a massive code churn with a huge diff and
> won't match any existing C extensions or published books on Python) but it
> does have advantages in the face of all the new tp slots being proposed
> which would be painful regardless.
>
> I don't want to pretend that it really prevents bugs though.
> Historically, this just hasn't been a problem for us in practice (I've
> written and reviewed many, many of these structs over the years).  In fact,
> the change may become a source of bugs as people forget to fill-in slots or
> lose their cues as to what slots are available (I for one use the current
> copy and paste as a checklist for what should and shouldn't be included).
>
> I would really like to hear Guido's opinion on the subject (IIRC he
> designed the current system of slots and has had to live with its pros and
> cons over a long period time).  Also at one time, Guido opined that he was
> somewhat resistant to adding new slots (otherwise, we would already have a
> __length_hint__ slot which I proposed a few years ago).
>
> Lastly, if we do go forward with this sweeping change, we should also
> explore the possibility of flattening the current tp_as_sequence,
> tp_as_mapping, and tp_as_number indirections. These have historically have
> been somewhat of a pain and have led to many otherwise unnecessary
> indirections.
>
>
>
> Raymond
>
>
> P.S.  I'm not sure if we care about the size of the types but they are
> growing at an unprecedented rate (amidst lots of other code churn as well).
>
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170117/52070b7a/attachment.html>

From njs at pobox.com  Tue Jan 17 21:35:27 2017
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 17 Jan 2017 18:35:27 -0800
Subject: [Python-Dev] Can we use "designated initializer" widely in
 coremodules?
In-Reply-To: <CAEfz+Ty=RpaFhLKTiCm=p1eV-eP_Pw82yVH4m6GfKBX0DgNdJw@mail.gmail.com>
References: <CAEfz+Tx9_Km03ZexEcY2vVqd4XvGCpRfMKxV3+q8TVj=bjmtrw@mail.gmail.com>
 <CAP1=2W65qscErCM_eSvN_2-9bwYswZ+BDV0ogDeyEwZg0SXvRg@mail.gmail.com>
 <E1cTZxZ-00085X-BF@se2-syd.hostedmail.net.au>
 <3f6ab374-f6df-e35c-c860-a232fe542395@hastings.org>
 <CAEfz+Ty=RpaFhLKTiCm=p1eV-eP_Pw82yVH4m6GfKBX0DgNdJw@mail.gmail.com>
Message-ID: <CAPJVwBnEwhXPmdb81FMsbCX0T5-bgm=71YyahgAgnkyVaBw4ug@mail.gmail.com>

On Tue, Jan 17, 2017 at 4:48 PM, INADA Naoki <songofacandy at gmail.com> wrote:
> On Wed, Jan 18, 2017 at 8:48 AM, Larry Hastings <larry at hastings.org> wrote:
>>
>> On 01/17/2017 12:02 PM, Steve Dower wrote:
>>
>> Avoiding header files would be my only request. As Brett says, the C99
>> requirement should not be enforced on all embedders or extenders, so we
>> should try and keep the headers they'll use as compatible as possible.
>>
>>
>
> C99 style comment is used in header file already.
> see http://bugs.python.org/issue29215
>
>>
>> While that's a reasonable policy, unless we have a way to automatically
>> detect that I suspect C99 stuff will creep into the header files and break
>> the non-C99 customers.  Maybe we could get some sort of buildbot that
>> exercises this scenario?
>>
>
> How about `gcc -ansi` ?

I think the main concern isn't C90 compatibility, but C++
compatibility, right? The reason CPython is switching to allowing
(most of) C99 internally is that it seems like that it's now supported
as a matter of course on all the platforms we care about, so while
it's theoretically possible that someone uses C99 compiler to build
Python but then switches to a C90 compiler to build extensions, it
seems pretty unlikely. (Especially since the last hold-out on C99
support was MSVC, and on Windows we already force people to build
extensions using the same version of MSVC as was used to build
CPython.)

OTOH it is definitely important that the Python header files remain
polyglot C99-and-C++ compatible.

Even a simple check like:

echo '#include <Python.h>' > test.cc && g++ -c test.cc -o /dev/null

would probably catch most issues here.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From steve.dower at python.org  Tue Jan 17 21:37:43 2017
From: steve.dower at python.org (Steve Dower)
Date: Tue, 17 Jan 2017 18:37:43 -0800
Subject: [Python-Dev] Can we use "designated initializer" widely
 incoremodules?
In-Reply-To: <97c137c3-3f08-f905-045c-24e05713f48e@hastings.org>
References: <CAEfz+Tx9_Km03ZexEcY2vVqd4XvGCpRfMKxV3+q8TVj=bjmtrw@mail.gmail.com>
 <CAP1=2W65qscErCM_eSvN_2-9bwYswZ+BDV0ogDeyEwZg0SXvRg@mail.gmail.com>
 <E1cTZxZ-00085X-BF@se2-syd.hostedmail.net.au>
 <3f6ab374-f6df-e35c-c860-a232fe542395@hastings.org>
 <CAEfz+Ty=RpaFhLKTiCm=p1eV-eP_Pw82yVH4m6GfKBX0DgNdJw@mail.gmail.com>
 <97c137c3-3f08-f905-045c-24e05713f48e@hastings.org>
Message-ID: <E1cTg8F-0003gw-8D@se2-syd.hostedmail.net.au>

As well as pre-C99 compilers, there are also C++ compilers to think of.

It may be easier to identify the likely features we want to avoid and regex find them in the test suite. Combined with code reviews and the fact that we can change syntax in the header files whenever we want without impact (and if that's not true, let's definitely add checks for those cases), I think we can achieve this without excessive effort.

(And I'm fairly sure MSVC 9.0 in C mode is the most regressive compiler we have available ;) )

Cheers,
Steve

Top-posted from my Windows Phone

-----Original Message-----
From: "Larry Hastings" <larry at hastings.org>
Sent: ?1/?17/?2017 17:02
To: "Python-Dev" <python-dev at python.org>
Subject: Re: [Python-Dev] Can we use "designated initializer" widely incoremodules?


On 01/17/2017 04:48 PM, INADA Naoki wrote:

On Wed, Jan 18, 2017 at 8:48 AM, Larry Hastings <larry at hastings.org> wrote:
While that's a reasonable policy, unless we have a way to automatically
detect that I suspect C99 stuff will creep into the header files and break
the non-C99 customers.  Maybe we could get some sort of buildbot that
exercises this scenario?

How about `gcc -ansi` ?
That seems like it should work fine.  Perhaps we could even add this gcc -ansi step to the normal Python build process.  It shouldn't take very long, so it wouldn't slow down the build very much, and since most Python development happens on UNIX platforms (I think) this would help prevent more C99 constructs from creeping into the header files.


/arry
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170117/2688e308/attachment-0001.html>

From raymond.hettinger at gmail.com  Tue Jan 17 23:44:54 2017
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Tue, 17 Jan 2017 20:44:54 -0800
Subject: [Python-Dev] collections.abc for data and non-data descriptors
In-Reply-To: <CAF0TK53cKpB1TeBHBnJOyk6+HLMWrvBRPruwGQST0owZg8NRZQ@mail.gmail.com>
References: <CAF0TK53XmG+X4VBo4OxuVEe+Xq6PAybLqfUWrbut8UoMir-GTw@mail.gmail.com>
 <CAP7+vJ+FbtEs7Jnez4EAo5VQBbr1V-2Gr-CQMKJXK6mbeNnYig@mail.gmail.com>
 <CAF0TK50rA=yFSOvuq0n23fLG2tApYgSFEdD9oH86nW2dZTkx2w@mail.gmail.com>
 <CAP7+vJ+OnLyc7y_YjWX2GF_s4zmkbdSr_miAjpzv5U9nx2Qtdg@mail.gmail.com>
 <CAF0TK535RmB9fF14RvFbF=+0nyQZ4CGCB8LJnAQgwnd2eOX2ag@mail.gmail.com>
 <CAP7+vJKhHw+YNjV4c5JqAhNHL2aFvfcPq2fqd+gf4woJWTZc6A@mail.gmail.com>
 <CAF0TK53cKpB1TeBHBnJOyk6+HLMWrvBRPruwGQST0owZg8NRZQ@mail.gmail.com>
Message-ID: <140273B1-263B-48AB-93E5-713BC4DEEB11@gmail.com>


> On Jan 17, 2017, at 11:41 AM, Roberto Mart?nez <robertomartinezp at gmail.com> wrote:
> 
> Oh, I understand. Maybe is not worth the effort anyway.

FWIW, I'm also in the camp of thinking it is not worth the effort.  Until there is a demonstrated need (something than can't be met by checking for __set__), the default position should be to stick with a core usable set of ABCs that are know to have real value.  Each new ABC has a non-trivial maintenance burden and requires extra effort on the part of users to learn and remember.


Raymond

From victor.stinner at gmail.com  Wed Jan 18 02:44:29 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 18 Jan 2017 08:44:29 +0100
Subject: [Python-Dev] Can we use "designated initializer" widely in core
 modules?
In-Reply-To: <CAP7+vJLOcQJg=SmC_+nnYMdecjyPDQwD5GeT9q=MheJ2h+eO+A@mail.gmail.com>
References: <CAEfz+Tx9_Km03ZexEcY2vVqd4XvGCpRfMKxV3+q8TVj=bjmtrw@mail.gmail.com>
 <1c50fa66-249c-3263-7f59-1fcc0a17b563@lucidity.plus.com>
 <CAEfz+TyK7Eq0NoPxXbZO0chQNynk8_pY44PXwV6PVbm+8T+i=g@mail.gmail.com>
 <CAMpsgwZoFhXvoz5DJZ=8fW_2tEEekiZuBv7+jdaT41TE+xmtOg@mail.gmail.com>
 <1D156195-41CF-413B-A954-FDE8942741DF@gmail.com>
 <CAP7+vJLOcQJg=SmC_+nnYMdecjyPDQwD5GeT9q=MheJ2h+eO+A@mail.gmail.com>
Message-ID: <CAMpsgwbKxkJJvF9R5N2Hn8focXoKr-EEjxZA0QP_oCaVu5VwFA@mail.gmail.com>

2017-01-18 3:28 GMT+01:00 Guido van Rossum <guido at python.org>:
> I recommend being reluctant to add new fields -- the number of type objects
> is larger than you'd think and these are static bytes which I consider a
> relatively expensive resource.
>
> There's also an ABI issue with new fields -- new fields can only be accessed
> after checking that the type object has been compiled with a version of the
> headers that defines the field, else you'd be accessing garbage bytes.

A new t_finalize field was added to the PyTypeObject object:
https://www.python.org/dev/peps/pep-0442/

The ABI is protected by a new Py_TPFLAGS_HAVE_FINALIZE flag in
tp_flags. If a type doesn't have this flag, the tp_finalize is not
read nor written. If an extension was compiled for Pyhon 3.3, static
PyTypeObject types are smaller than Python 3.4 types, since the
compiler used Python 3.3 size without tp_finalize.

I'm working on adding a new tp_fastcall field with a new
Py_TPFLAGS_HAVE_FASTCALL flag. Same story, tp_fastcall is only used if
Py_TPFLAGS_HAVE_FASTCALL is set. A wrapper is used to call the
tp_fastcall field of base classes if needed:
http://bugs.python.org/issue29259

My patch adds Py_TPFLAGS_HAVE_FINALIZE and Py_TPFLAGS_HAVE_FASTCALL to
Py_TPFLAGS_DEFAULT. So all modules compiled with Python 3.7 will
announce that they have tp_finalize and tp_fastcall fields, even if
they are NULL. I don't know why this change wasn't done before for
tp_finalize. Having Py_TPFLAGS_HAVE_FINALIZE/FASTCALL allows to
inherit tp_finalize/fastcall in child classes.

I want to add tp_fastcall for performance, to avoid the creation of
temporary tupe/dict to pass arguments. I measured a speedup up of 22%
on the bm_telco benchmark.

Victor

From benjamin at python.org  Wed Jan 18 02:45:23 2017
From: benjamin at python.org (Benjamin Peterson)
Date: Tue, 17 Jan 2017 23:45:23 -0800
Subject: [Python-Dev] Can we use "designated initializer" widely in
 coremodules?
In-Reply-To: <CACac1F_chcmLMEU3ST0o7Aas1_YvmsBrnvsWrQQn8CgC9piPUA@mail.gmail.com>
References: <CAEfz+Tx9_Km03ZexEcY2vVqd4XvGCpRfMKxV3+q8TVj=bjmtrw@mail.gmail.com>
 <CAP1=2W65qscErCM_eSvN_2-9bwYswZ+BDV0ogDeyEwZg0SXvRg@mail.gmail.com>
 <E1cTZxZ-00085X-BF@se2-syd.hostedmail.net.au>
 <CACac1F_chcmLMEU3ST0o7Aas1_YvmsBrnvsWrQQn8CgC9piPUA@mail.gmail.com>
Message-ID: <1484725523.3326010.851319408.58159DE4@webmail.messagingengine.com>



On Tue, Jan 17, 2017, at 15:34, Paul Moore wrote:
> On 17 January 2017 at 20:02, Steve Dower <steve.dower at python.org> wrote:
> > Avoiding header files would be my only request. As Brett says, the C99
> > requirement should not be enforced on all embedders or extenders, so we
> > should try and keep the headers they'll use as compatible as possible.
> 
> Agreed, we probably shouldn't require users of Python.h to be
> C99-compliant. (Disclaimer: I have no idea how many compilers *don't*
> implement this feature).

We already require it through the use of several C99 features in headers
such as // comments and <stdint.h>. I agree they should be compatible
with C++, though.

From levkivskyi at gmail.com  Wed Jan 18 05:25:48 2017
From: levkivskyi at gmail.com (Ivan Levkivskyi)
Date: Wed, 18 Jan 2017 11:25:48 +0100
Subject: [Python-Dev] collections.abc for data and non-data descriptors
In-Reply-To: <140273B1-263B-48AB-93E5-713BC4DEEB11@gmail.com>
References: <CAF0TK53XmG+X4VBo4OxuVEe+Xq6PAybLqfUWrbut8UoMir-GTw@mail.gmail.com>
 <CAP7+vJ+FbtEs7Jnez4EAo5VQBbr1V-2Gr-CQMKJXK6mbeNnYig@mail.gmail.com>
 <CAF0TK50rA=yFSOvuq0n23fLG2tApYgSFEdD9oH86nW2dZTkx2w@mail.gmail.com>
 <CAP7+vJ+OnLyc7y_YjWX2GF_s4zmkbdSr_miAjpzv5U9nx2Qtdg@mail.gmail.com>
 <CAF0TK535RmB9fF14RvFbF=+0nyQZ4CGCB8LJnAQgwnd2eOX2ag@mail.gmail.com>
 <CAP7+vJKhHw+YNjV4c5JqAhNHL2aFvfcPq2fqd+gf4woJWTZc6A@mail.gmail.com>
 <CAF0TK53cKpB1TeBHBnJOyk6+HLMWrvBRPruwGQST0owZg8NRZQ@mail.gmail.com>
 <140273B1-263B-48AB-93E5-713BC4DEEB11@gmail.com>
Message-ID: <CAOMjWk=QKa35sXX3hG476wrLbGsdEY7CTVKyZu+-_0muOVD97w@mail.gmail.com>

A random thought: typing has ABCs for protocols, even more than
collections.abc has (e.g. typing.SupportsInt). Some support for descriptors
has been added to mypy recently.
Maybe it makes sense to add support for descriptor protocol in typing?

--
Ivan


On 18 January 2017 at 05:44, Raymond Hettinger <raymond.hettinger at gmail.com>
wrote:

>
> > On Jan 17, 2017, at 11:41 AM, Roberto Mart?nez <
> robertomartinezp at gmail.com> wrote:
> >
> > Oh, I understand. Maybe is not worth the effort anyway.
>
> FWIW, I'm also in the camp of thinking it is not worth the effort.  Until
> there is a demonstrated need (something than can't be met by checking for
> __set__), the default position should be to stick with a core usable set of
> ABCs that are know to have real value.  Each new ABC has a non-trivial
> maintenance burden and requires extra effort on the part of users to learn
> and remember.
>
>
> Raymond
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> levkivskyi%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170118/00236f4b/attachment.html>

From solipsis at pitrou.net  Wed Jan 18 06:33:52 2017
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 18 Jan 2017 12:33:52 +0100
Subject: [Python-Dev] Can we use "designated initializer" widely in core
 modules?
References: <CAEfz+Tx9_Km03ZexEcY2vVqd4XvGCpRfMKxV3+q8TVj=bjmtrw@mail.gmail.com>
 <1c50fa66-249c-3263-7f59-1fcc0a17b563@lucidity.plus.com>
 <CAEfz+TyK7Eq0NoPxXbZO0chQNynk8_pY44PXwV6PVbm+8T+i=g@mail.gmail.com>
 <CAMpsgwZoFhXvoz5DJZ=8fW_2tEEekiZuBv7+jdaT41TE+xmtOg@mail.gmail.com>
 <1D156195-41CF-413B-A954-FDE8942741DF@gmail.com>
 <CAP7+vJLOcQJg=SmC_+nnYMdecjyPDQwD5GeT9q=MheJ2h+eO+A@mail.gmail.com>
 <CAMpsgwbKxkJJvF9R5N2Hn8focXoKr-EEjxZA0QP_oCaVu5VwFA@mail.gmail.com>
Message-ID: <20170118123352.6c562a09@fsol>

On Wed, 18 Jan 2017 08:44:29 +0100
Victor Stinner <victor.stinner at gmail.com> wrote:
> 
> My patch adds Py_TPFLAGS_HAVE_FINALIZE and Py_TPFLAGS_HAVE_FASTCALL to
> Py_TPFLAGS_DEFAULT. So all modules compiled with Python 3.7 will
> announce that they have tp_finalize and tp_fastcall fields, even if
> they are NULL. I don't know why this change wasn't done before for
> tp_finalize.

I don't think I thought about that idea at the time.  tp_finalize
doesn't benefit many extension types, so it's not a huge burden to add
a FLAGS value for the few cases where you want to use it.

Regards

Antoine.



From ronaldoussoren at mac.com  Wed Jan 18 05:48:00 2017
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Wed, 18 Jan 2017 11:48:00 +0100
Subject: [Python-Dev] Can we use "designated initializer" widely in core
 modules?
In-Reply-To: <CAMpsgwZoFhXvoz5DJZ=8fW_2tEEekiZuBv7+jdaT41TE+xmtOg@mail.gmail.com>
References: <CAEfz+Tx9_Km03ZexEcY2vVqd4XvGCpRfMKxV3+q8TVj=bjmtrw@mail.gmail.com>
 <1c50fa66-249c-3263-7f59-1fcc0a17b563@lucidity.plus.com>
 <CAEfz+TyK7Eq0NoPxXbZO0chQNynk8_pY44PXwV6PVbm+8T+i=g@mail.gmail.com>
 <CAMpsgwZoFhXvoz5DJZ=8fW_2tEEekiZuBv7+jdaT41TE+xmtOg@mail.gmail.com>
Message-ID: <71A1C0BB-7ADE-4637-A730-233679208AC4@mac.com>


> On 18 Jan 2017, at 02:16, Victor Stinner <victor.stinner at gmail.com> wrote:
> 
> 2017-01-18 1:59 GMT+01:00 INADA Naoki <songofacandy at gmail.com>:
>> I think mixing two forms is OK only if new form is used only at bottom.
>> (like keyword arguments are allowed after all positional arguments in
>> Python function calling)
>> 
>> Complete rewriting makes diff huge.  And there is PyVarObject_HEAD_INIT already.
> 
> I'm in favor of replacing all long list of fields with the /* tp_xxx
> */ comments to use designated initializers. It would allow to remove a
> lot of "0,     /* tp_xxx */" lines and make the code much more
> readable! It should help to prevent bugs when the code is modified.

I agree. I?ve done this in my own projects and that made the code a lot easier to read.

Ronald


From victor.stinner at gmail.com  Wed Jan 18 09:52:48 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 18 Jan 2017 15:52:48 +0100
Subject: [Python-Dev] Can we use "designated initializer" widely in core
 modules?
In-Reply-To: <20170118123352.6c562a09@fsol>
References: <CAEfz+Tx9_Km03ZexEcY2vVqd4XvGCpRfMKxV3+q8TVj=bjmtrw@mail.gmail.com>
 <1c50fa66-249c-3263-7f59-1fcc0a17b563@lucidity.plus.com>
 <CAEfz+TyK7Eq0NoPxXbZO0chQNynk8_pY44PXwV6PVbm+8T+i=g@mail.gmail.com>
 <CAMpsgwZoFhXvoz5DJZ=8fW_2tEEekiZuBv7+jdaT41TE+xmtOg@mail.gmail.com>
 <1D156195-41CF-413B-A954-FDE8942741DF@gmail.com>
 <CAP7+vJLOcQJg=SmC_+nnYMdecjyPDQwD5GeT9q=MheJ2h+eO+A@mail.gmail.com>
 <CAMpsgwbKxkJJvF9R5N2Hn8focXoKr-EEjxZA0QP_oCaVu5VwFA@mail.gmail.com>
 <20170118123352.6c562a09@fsol>
Message-ID: <CAMpsgwah2X425j5Fe1MovNhHRJMSve+Frk_Z9hp1dnhv8H=z_A@mail.gmail.com>

2017-01-18 12:33 GMT+01:00 Antoine Pitrou <solipsis at pitrou.net>:
> I don't think I thought about that idea at the time.  tp_finalize
> doesn't benefit many extension types, so it's not a huge burden to add
> a FLAGS value for the few cases where you want to use it.

It isn't obvious to me that I have to define explicitly a
Py_TPFLAGS_HAVE_FINALIZE flag if I want to _inherit_ the finalizer
from the base class. The flag requires an #ifdef to only define it on
Python >= 3.4, which can be painful when you have to support older
Python versions and want a single code base for all Python versions.

If a C extension defines a type which inherit from a type which has a
finalizer, currently the finalizer is not inherited if the new type
doesn't set explicitly the Py_TPFLAGS_HAVE_FINALIZE flag.

I checked the _socket.socket type. Hopefully, sock_dealloc() calls
PyObject_CallFinalizerFromDealloc(), so I guess that it should "just
work" if tp_finalize is not inherited in the tp_finalize slot.

But for tp_fastcall, if a type doesn't have the
Py_TPFLAGS_HAVE_FASTCALL flag, the fastcall_wrapper() has to find the
first base class with a tp_fastcall slot which. The search has a
complexity of O(n), so it has an impact on performance (even if the
impact is low).

The flag only says that the C structure contains the field, not that
it's set. So I think that it's safe to add Py_TPFLAGS_HAVE_FINALIZE
and Py_TPFLAGS_HAVE_FASTCALL to Py_TPFLAGS_DEFAULT.

Victor

From guido at python.org  Wed Jan 18 10:35:47 2017
From: guido at python.org (Guido van Rossum)
Date: Wed, 18 Jan 2017 07:35:47 -0800
Subject: [Python-Dev] collections.abc for data and non-data descriptors
In-Reply-To: <CAOMjWk=QKa35sXX3hG476wrLbGsdEY7CTVKyZu+-_0muOVD97w@mail.gmail.com>
References: <CAF0TK53XmG+X4VBo4OxuVEe+Xq6PAybLqfUWrbut8UoMir-GTw@mail.gmail.com>
 <CAP7+vJ+FbtEs7Jnez4EAo5VQBbr1V-2Gr-CQMKJXK6mbeNnYig@mail.gmail.com>
 <CAF0TK50rA=yFSOvuq0n23fLG2tApYgSFEdD9oH86nW2dZTkx2w@mail.gmail.com>
 <CAP7+vJ+OnLyc7y_YjWX2GF_s4zmkbdSr_miAjpzv5U9nx2Qtdg@mail.gmail.com>
 <CAF0TK535RmB9fF14RvFbF=+0nyQZ4CGCB8LJnAQgwnd2eOX2ag@mail.gmail.com>
 <CAP7+vJKhHw+YNjV4c5JqAhNHL2aFvfcPq2fqd+gf4woJWTZc6A@mail.gmail.com>
 <CAF0TK53cKpB1TeBHBnJOyk6+HLMWrvBRPruwGQST0owZg8NRZQ@mail.gmail.com>
 <140273B1-263B-48AB-93E5-713BC4DEEB11@gmail.com>
 <CAOMjWk=QKa35sXX3hG476wrLbGsdEY7CTVKyZu+-_0muOVD97w@mail.gmail.com>
Message-ID: <CAP7+vJ+JLVJXfrOJLbCvxdPWo-3sFhaA1-=-D+Cgyw_ahShy0w@mail.gmail.com>

I'd rather wait until support for protocols has been added to PEP 484.

On Wed, Jan 18, 2017 at 2:25 AM, Ivan Levkivskyi <levkivskyi at gmail.com>
wrote:

> A random thought: typing has ABCs for protocols, even more than
> collections.abc has (e.g. typing.SupportsInt). Some support for descriptors
> has been added to mypy recently.
> Maybe it makes sense to add support for descriptor protocol in typing?
>
> --
> Ivan
>
>
> On 18 January 2017 at 05:44, Raymond Hettinger <
> raymond.hettinger at gmail.com> wrote:
>
>>
>> > On Jan 17, 2017, at 11:41 AM, Roberto Mart?nez <
>> robertomartinezp at gmail.com> wrote:
>> >
>> > Oh, I understand. Maybe is not worth the effort anyway.
>>
>> FWIW, I'm also in the camp of thinking it is not worth the effort.  Until
>> there is a demonstrated need (something than can't be met by checking for
>> __set__), the default position should be to stick with a core usable set of
>> ABCs that are know to have real value.  Each new ABC has a non-trivial
>> maintenance burden and requires extra effort on the part of users to learn
>> and remember.
>>
>>
>> Raymond
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/levkivsky
>> i%40gmail.com
>>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170118/cb787e91/attachment.html>

From levkivskyi at gmail.com  Wed Jan 18 10:45:53 2017
From: levkivskyi at gmail.com (Ivan Levkivskyi)
Date: Wed, 18 Jan 2017 16:45:53 +0100
Subject: [Python-Dev] collections.abc for data and non-data descriptors
In-Reply-To: <CAP7+vJ+JLVJXfrOJLbCvxdPWo-3sFhaA1-=-D+Cgyw_ahShy0w@mail.gmail.com>
References: <CAF0TK53XmG+X4VBo4OxuVEe+Xq6PAybLqfUWrbut8UoMir-GTw@mail.gmail.com>
 <CAP7+vJ+FbtEs7Jnez4EAo5VQBbr1V-2Gr-CQMKJXK6mbeNnYig@mail.gmail.com>
 <CAF0TK50rA=yFSOvuq0n23fLG2tApYgSFEdD9oH86nW2dZTkx2w@mail.gmail.com>
 <CAP7+vJ+OnLyc7y_YjWX2GF_s4zmkbdSr_miAjpzv5U9nx2Qtdg@mail.gmail.com>
 <CAF0TK535RmB9fF14RvFbF=+0nyQZ4CGCB8LJnAQgwnd2eOX2ag@mail.gmail.com>
 <CAP7+vJKhHw+YNjV4c5JqAhNHL2aFvfcPq2fqd+gf4woJWTZc6A@mail.gmail.com>
 <CAF0TK53cKpB1TeBHBnJOyk6+HLMWrvBRPruwGQST0owZg8NRZQ@mail.gmail.com>
 <140273B1-263B-48AB-93E5-713BC4DEEB11@gmail.com>
 <CAOMjWk=QKa35sXX3hG476wrLbGsdEY7CTVKyZu+-_0muOVD97w@mail.gmail.com>
 <CAP7+vJ+JLVJXfrOJLbCvxdPWo-3sFhaA1-=-D+Cgyw_ahShy0w@mail.gmail.com>
Message-ID: <CAOMjWk=_CZhg+_ze3tNKBf0tMdK1vX79DRv19BnhZqoJY8oZfA@mail.gmail.com>

On 18 January 2017 at 16:35, Guido van Rossum <guido at python.org> wrote:

> I'd rather wait until support for protocols has been added to PEP 484.
>

Sorry for offtopic but do you think this should be an addition to PEP 484
or a separate PEP?
I am interested in writing the specification (based on what Jukka proposed
on typing tracker)
and implementing it in mypy.

--
Ivan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170118/85819913/attachment.html>

From guido at python.org  Wed Jan 18 11:07:09 2017
From: guido at python.org (Guido van Rossum)
Date: Wed, 18 Jan 2017 08:07:09 -0800
Subject: [Python-Dev] collections.abc for data and non-data descriptors
In-Reply-To: <CAOMjWk=_CZhg+_ze3tNKBf0tMdK1vX79DRv19BnhZqoJY8oZfA@mail.gmail.com>
References: <CAF0TK53XmG+X4VBo4OxuVEe+Xq6PAybLqfUWrbut8UoMir-GTw@mail.gmail.com>
 <CAP7+vJ+FbtEs7Jnez4EAo5VQBbr1V-2Gr-CQMKJXK6mbeNnYig@mail.gmail.com>
 <CAF0TK50rA=yFSOvuq0n23fLG2tApYgSFEdD9oH86nW2dZTkx2w@mail.gmail.com>
 <CAP7+vJ+OnLyc7y_YjWX2GF_s4zmkbdSr_miAjpzv5U9nx2Qtdg@mail.gmail.com>
 <CAF0TK535RmB9fF14RvFbF=+0nyQZ4CGCB8LJnAQgwnd2eOX2ag@mail.gmail.com>
 <CAP7+vJKhHw+YNjV4c5JqAhNHL2aFvfcPq2fqd+gf4woJWTZc6A@mail.gmail.com>
 <CAF0TK53cKpB1TeBHBnJOyk6+HLMWrvBRPruwGQST0owZg8NRZQ@mail.gmail.com>
 <140273B1-263B-48AB-93E5-713BC4DEEB11@gmail.com>
 <CAOMjWk=QKa35sXX3hG476wrLbGsdEY7CTVKyZu+-_0muOVD97w@mail.gmail.com>
 <CAP7+vJ+JLVJXfrOJLbCvxdPWo-3sFhaA1-=-D+Cgyw_ahShy0w@mail.gmail.com>
 <CAOMjWk=_CZhg+_ze3tNKBf0tMdK1vX79DRv19BnhZqoJY8oZfA@mail.gmail.com>
Message-ID: <CAP7+vJL49AUj7zCzfYdJ0gR_5Tz+qr+Uv=fBCpUK6AGh2kMy3w@mail.gmail.com>

It's probably big enough to make it a new PEP.

On Wed, Jan 18, 2017 at 7:45 AM, Ivan Levkivskyi <levkivskyi at gmail.com>
wrote:

> On 18 January 2017 at 16:35, Guido van Rossum <guido at python.org> wrote:
>
>> I'd rather wait until support for protocols has been added to PEP 484.
>>
>
> Sorry for offtopic but do you think this should be an addition to PEP 484
> or a separate PEP?
> I am interested in writing the specification (based on what Jukka proposed
> on typing tracker)
> and implementing it in mypy.
>
> --
> Ivan
>
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170118/50627a4a/attachment-0001.html>

From brett at python.org  Wed Jan 18 15:20:20 2017
From: brett at python.org (Brett Cannon)
Date: Wed, 18 Jan 2017 20:20:20 +0000
Subject: [Python-Dev] Can we use "designated initializer" widely in
 coremodules?
In-Reply-To: <CAPJVwBnEwhXPmdb81FMsbCX0T5-bgm=71YyahgAgnkyVaBw4ug@mail.gmail.com>
References: <CAEfz+Tx9_Km03ZexEcY2vVqd4XvGCpRfMKxV3+q8TVj=bjmtrw@mail.gmail.com>
 <CAP1=2W65qscErCM_eSvN_2-9bwYswZ+BDV0ogDeyEwZg0SXvRg@mail.gmail.com>
 <E1cTZxZ-00085X-BF@se2-syd.hostedmail.net.au>
 <3f6ab374-f6df-e35c-c860-a232fe542395@hastings.org>
 <CAEfz+Ty=RpaFhLKTiCm=p1eV-eP_Pw82yVH4m6GfKBX0DgNdJw@mail.gmail.com>
 <CAPJVwBnEwhXPmdb81FMsbCX0T5-bgm=71YyahgAgnkyVaBw4ug@mail.gmail.com>
Message-ID: <CAP1=2W6te_5VTDLWtHff2YUFEipkMhVbgjmXWMQ=d0x3RZSntQ@mail.gmail.com>

On Tue, 17 Jan 2017 at 18:36 Nathaniel Smith <njs at pobox.com> wrote:

> On Tue, Jan 17, 2017 at 4:48 PM, INADA Naoki <songofacandy at gmail.com>
> wrote:
> > On Wed, Jan 18, 2017 at 8:48 AM, Larry Hastings <larry at hastings.org>
> wrote:
> >>
> >> On 01/17/2017 12:02 PM, Steve Dower wrote:
> >>
> >> Avoiding header files would be my only request. As Brett says, the C99
> >> requirement should not be enforced on all embedders or extenders, so we
> >> should try and keep the headers they'll use as compatible as possible.
> >>
> >>
> >
> > C99 style comment is used in header file already.
> > see http://bugs.python.org/issue29215
> >
> >>
> >> While that's a reasonable policy, unless we have a way to automatically
> >> detect that I suspect C99 stuff will creep into the header files and
> break
> >> the non-C99 customers.  Maybe we could get some sort of buildbot that
> >> exercises this scenario?
> >>
> >
> > How about `gcc -ansi` ?
>
> I think the main concern isn't C90 compatibility, but C++
> compatibility, right?


Correct. This is not about supporting old versions of C but making sure we
continue to support C++ extensions.


> The reason CPython is switching to allowing
> (most of) C99 internally is that it seems like that it's now supported
> as a matter of course on all the platforms we care about, so while
> it's theoretically possible that someone uses C99 compiler to build
> Python but then switches to a C90 compiler to build extensions, it
> seems pretty unlikely. (Especially since the last hold-out on C99
> support was MSVC, and on Windows we already force people to build
> extensions using the same version of MSVC as was used to build
> CPython.)
>
> OTOH it is definitely important that the Python header files remain
> polyglot C99-and-C++ compatible.
>
> Even a simple check like:
>
> echo '#include <Python.h>' > test.cc && g++ -c test.cc -o /dev/null
>
> would probably catch most issues here.
>

 I've gone ahead and added that to the Travis test matrix for when we move
to GitHub:
https://github.com/brettcannon/cpython-ci-test/blob/master/.travis.yml#L55
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170118/a28f6897/attachment.html>

From Dahui.Jiang at veritas.com  Thu Jan 19 06:54:16 2017
From: Dahui.Jiang at veritas.com (Dahui Jiang)
Date: Thu, 19 Jan 2017 11:54:16 +0000
Subject: [Python-Dev] Have problem when building python3.5.1 rpm with
 default SPEC file
Message-ID: <fcf13f7ce88a46599c11579856be56a6@vrtsxchclupin16.community.veritas.com>

Hi all:
I'm try to build a python rpm from source python3.5.1, and I use the spec file in the source tree.
But the building is not success as print the following error:
***********************************************
running build
running build_ext
error: [Errno 2] No such file or directory: 'Modules/Setup'
error: Bad exit status from /var/tmp/rpm-tmp.DDm3jI (%build)


RPM build errors:
Bad exit status from /var/tmp/rpm-tmp.DDm3jI (%build)
************************************************

Regards
Dahui
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170119/80c1b07d/attachment.html>

From cstratak at redhat.com  Thu Jan 19 13:40:12 2017
From: cstratak at redhat.com (Charalampos Stratakis)
Date: Thu, 19 Jan 2017 13:40:12 -0500 (EST)
Subject: [Python-Dev] Have problem when building python3.5.1 rpm with
 default SPEC file
In-Reply-To: <fcf13f7ce88a46599c11579856be56a6@vrtsxchclupin16.community.veritas.com>
References: <fcf13f7ce88a46599c11579856be56a6@vrtsxchclupin16.community.veritas.com>
Message-ID: <1459225640.92062396.1484851212581.JavaMail.zimbra@redhat.com>

Hello,

This is a distro specific issue so this list might not be the best for resolving that, you should contact your distro's package maintainers of python.

For Fedora 25 we currently ship Python 3.5.2, which builds fine with this SPEC file [0], so maybe you could give this a try.

[0] http://pkgs.fedoraproject.org/cgit/rpms/python3.git/tree/python3.spec?h=f25

Regards,

Charalampos Stratakis
Associate Software Engineer
Python Maintenance Team, Red Hat


----- Original Message -----
From: "Dahui Jiang" <Dahui.Jiang at veritas.com>
To: python-dev at python.org
Sent: Thursday, January 19, 2017 12:54:16 PM
Subject: [Python-Dev] Have problem when building python3.5.1 rpm with default SPEC file



Hi all: 

I?m try to build a python rpm from source python3.5.1, and I use the spec file in the source tree. 

But the building is not success as print the following error: 



*********************************************** 

running build 

running build_ext 

error: [Errno 2] No such file or directory: 'Modules/Setup' 

error: Bad exit status from /var/tmp/rpm-tmp.DDm3jI (%build) 





RPM build errors: 

Bad exit status from /var/tmp/rpm-tmp.DDm3jI (%build) 

************************************************ 



Regards 

Dahui 

_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/cstratak%40redhat.com

From valentin at too.gy  Thu Jan 19 13:59:42 2017
From: valentin at too.gy (Valentin Iovene)
Date: Thu, 19 Jan 2017 19:59:42 +0100
Subject: [Python-Dev] FunctionDef.returns - explicit 'None' return type hint
Message-ID: <20170119185942.GA18261@stewie>

With a ast.FunctionDef ast.AST node, is it possible to make the
difference between this function

    def hello_world():
        print('hello world')

and this one

    def hello_world() -> None:
        print('hello world')

?

In both cases, the FunctionDef node has its 'returns' (return type
hint) attribute set to None.

-- 
Valentin

From guido at python.org  Thu Jan 19 14:54:35 2017
From: guido at python.org (Guido van Rossum)
Date: Thu, 19 Jan 2017 11:54:35 -0800
Subject: [Python-Dev] FunctionDef.returns - explicit 'None' return type
 hint
In-Reply-To: <20170119185942.GA18261@stewie>
References: <20170119185942.GA18261@stewie>
Message-ID: <CAP7+vJ+oK5y4bDGDA_BTf-FyQmehQfFYZCGy=s_+Qk-mL4UUAQ@mail.gmail.com>

On Thu, Jan 19, 2017 at 10:59 AM, Valentin Iovene via Python-Dev <
python-dev at python.org> wrote:

> With a ast.FunctionDef ast.AST node, is it possible to make the
> difference between this function
>
>     def hello_world():
>         print('hello world')
>
> and this one
>
>     def hello_world() -> None:
>         print('hello world')
>
> ?
>
> In both cases, the FunctionDef node has its 'returns' (return type
> hint) attribute set to None.


>>> t = compile('def f(): pass', '', 'exec', ast.PyCF_ONLY_AST)
>>> print(t.body[0].returns)
None
>>> t = compile('def f() -> None: pass', '', 'exec', ast.PyCF_ONLY_AST)
>>> print(t.body[0].returns)
<_ast.NameConstant object at 0x10a900f28>
>>>  print(t.body[0].returns.value)
None
>>>

-- 
--Guido van Rossum (python.org/~guido <http://python.org/%7Eguido>)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170119/5654368a/attachment.html>

From valentin at too.gy  Thu Jan 19 16:34:35 2017
From: valentin at too.gy (Valentin Iovene)
Date: Thu, 19 Jan 2017 22:34:35 +0100
Subject: [Python-Dev] FunctionDef.returns - explicit 'None' return type
 hint
In-Reply-To: <CAP7+vJ+oK5y4bDGDA_BTf-FyQmehQfFYZCGy=s_+Qk-mL4UUAQ@mail.gmail.com>
References: <20170119185942.GA18261@stewie>
 <CAP7+vJ+oK5y4bDGDA_BTf-FyQmehQfFYZCGy=s_+Qk-mL4UUAQ@mail.gmail.com>
Message-ID: <20170119213435.GA32132@stewie>

> >>> t = compile('def f(): pass', '', 'exec', ast.PyCF_ONLY_AST)
> >>> print(t.body[0].returns)
> None
> >>> t = compile('def f() -> None: pass', '', 'exec', ast.PyCF_ONLY_AST)
> >>> print(t.body[0].returns)
> <_ast.NameConstant object at 0x10a900f28>
> >>>  print(t.body[0].returns.value)
> None

My bad, thank you my King. ;)

-- 
Valentin

From songofacandy at gmail.com  Fri Jan 20 05:49:01 2017
From: songofacandy at gmail.com (INADA Naoki)
Date: Fri, 20 Jan 2017 19:49:01 +0900
Subject: [Python-Dev] Investigating Python memory footprint of one real Web
 application
Message-ID: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>

Hi, all.

After reading Instagram's blog article [1], I?m thinking about how
Python can reduce memory usage of Web applications.
My company creating API server with Flask, SQLAlchemy and typing.
(sorry, it's closed source).
So I can get some data from it's codebase.

[1]: https://engineering.instagram.com/dismissing-python-garbage-collection-at-instagram-4dca40b29172#.lenebvdgn

Report is here
https://gist.github.com/methane/ce723adb9a4d32d32dc7525b738d3c31


My thoughts are:

* Interning (None,) seems worth enough.

* There are many empty dicts.  Allocating ma_keys lazily may reduce
memory usage.

* Most large strings are docstring.  Is it worth enough that option
for trim docstrings, without disabling asserts?

* typing may increase memory footprint, through functions
__attributes__ and abc.
   * Can we add option to remove or lazy evaluate __attributes__ ?
   * Using string literal for annotating generic type may reduce WeakRef usage.
   * Since typing will be used very widely in this year.  Need more
investigating.

From victor.stinner at gmail.com  Fri Jan 20 06:17:13 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 20 Jan 2017 12:17:13 +0100
Subject: [Python-Dev] Investigating Python memory footprint of one real
 Web application
In-Reply-To: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
References: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
Message-ID: <CAMpsgwYsVu3Yo3bfjg-+uy30Z7OG6Qr1q75jeAx9t3_gQvUEQQ@mail.gmail.com>

2017-01-20 11:49 GMT+01:00 INADA Naoki <songofacandy at gmail.com>:
> Report is here
> https://gist.github.com/methane/ce723adb9a4d32d32dc7525b738d3c31

Very interesting report, thanks!

> My thoughts are:
>
> * Interning (None,) seems worth enough.

I guess that (None,) comes from constants of code objects:

>>> def f(): pass
...
>>> f.__code__.co_consts
(None,)


> * There are many empty dicts.  Allocating ma_keys lazily may reduce
> memory usage.

Would you be able to estimate how many bytes would be saved by this
change? With the total memory usage to have an idea of the %.

By the way, it would help if you can add the total memory usage
computed by tracemalloc (get_traced_memory()[0]) in your report.

About empty dict, do you expect that they come from shared keys?
Anyway, if it has a negligible impact on the performance, go for it
:-)


> but other namespaces or annotations, like ('return',) or ('__wrapped__',) are not shared

Maybe we can intern all tuple which only contains one string?

Instead of interning, would it be possible to at least merge
duplicated immutable objects?


> * Most large strings are docstring.  Is it worth enough that option
> for trim docstrings, without disabling asserts?

Yeah, you are not the first one to propose. The problem is to decide
how to name the .pyc file.

My PEP 511 proposes to add a new -O command line option and a new
sys.implementation.optim_tag string to support this feature:
https://www.python.org/dev/peps/pep-0511/

Since the code transformer part of the PEP seems to be controversal,
maybe we should extract only these two changes from the PEP and
implement them? I also want -O noopt :-) (disable the peephole
optimizer)

Victor

From solipsis at pitrou.net  Fri Jan 20 06:43:33 2017
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 20 Jan 2017 12:43:33 +0100
Subject: [Python-Dev] Investigating Python memory footprint of one real
 Web application
References: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
Message-ID: <20170120124333.5f8f5e96@fsol>

On Fri, 20 Jan 2017 19:49:01 +0900
INADA Naoki <songofacandy at gmail.com> wrote:
> 
> Report is here
> https://gist.github.com/methane/ce723adb9a4d32d32dc7525b738d3c31

"this script counts static memory usage. It doesn?t care about dynamic
memory usage of processing real request"

You may be trying to optimize something which is only a very small
fraction of your actual memory footprint.  That said, the marshal
module could certainly try to intern some tuples and other immutable
structures.

> * Most large strings are docstring.  Is it worth enough that option
> for trim docstrings, without disabling asserts?

Perhaps docstrings may be compressed and then lazily decompressed when
accessed for the first time.  lz4 and zstd are good modern candidates
for that.  zstd also has a dictionary mode that helps for small data
(*).  See https://facebook.github.io/zstd/

(*) Even a 200-bytes docstring can be compressed this way:

>>> data = os.times.__doc__.encode()
>>> len(data)
211
>>> len(lz4.compress(data))
200
>>> c = zstd.ZstdCompressor()
>>> len(c.compress(data))
156
>>> c = zstd.ZstdCompressor(dict_data=dict_data)
>>> len(c.compress(data))
104

`dict_data` here is some 16KB dictionary I've trained on some Python
docstrings.  That 16KB dictionary could be computed while building
Python (or hand-generated from time to time, since it's unlikely to
change a lot) and put in a static array somewhere:

>>> samples = [(mod.__doc__ or '').encode() for mod in sys.modules.values()]
>>> sum(map(len, samples))
258113
>>> dict_data = zstd.train_dictionary(16384, samples)
>>> len(dict_data.as_bytes())
16384

Of course, compression is much more efficient on larger docstrings:

>>> import numpy as np
>>> data = np.__doc__.encode()
>>> len(data)
3140
>>> len(lz4.compress(data))
2271
>>> c = zstd.ZstdCompressor()
>>> len(c.compress(data))
1539
>>> c = zstd.ZstdCompressor(dict_data=dict_data)
>>> len(c.compress(data))
1348

>>> import pdb
>>> data = pdb.__doc__.encode()
>>> len(data)
12018
>>> len(lz4.compress(data))
6592
>>> c = zstd.ZstdCompressor()
>>> len(c.compress(data))
4502
>>> c = zstd.ZstdCompressor(dict_data=dict_data)
>>> len(c.compress(data))
4128


A similar strategy may be used for annotations and other
rarely-accessed metadata.

Another possibility, but probably much more costly in terms of initial
development and maintenance, is to put the docstrings (+ annotations,
etc.) in a separate file that's lazily read.

I think optimizing the footprint for everyone is much better than
adding command-line options to disable some specific metadata.

Regards

Antoine.



From levkivskyi at gmail.com  Fri Jan 20 06:52:37 2017
From: levkivskyi at gmail.com (Ivan Levkivskyi)
Date: Fri, 20 Jan 2017 12:52:37 +0100
Subject: [Python-Dev] Investigating Python memory footprint of one real
 Web application
In-Reply-To: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
References: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
Message-ID: <CAOMjWkkk2RZiZ754D5ggHoPDSp-gJ74itUSJukHN88JMbGJx9A@mail.gmail.com>

On 20 January 2017 at 11:49, INADA Naoki <songofacandy at gmail.com> wrote:

> * typing may increase memory footprint, through functions
> __attributes__ and abc.
>    * Can we add option to remove or lazy evaluate __attributes__ ?
>

This idea already appeared few times. I proposed to introduce a flag (e.g.
-OOO) to ignore function and variable annotations in compile.c
It was decide to postpone this, but maybe we can get back to this idea.

In 3.6, typing is already (quite heavily) optimized for both speed and
space.
I remember doing an experiment comparing a memory footprint with and
without annotations, the difference was few percent.
Do you have such comparison (with and without annotations) for your app?
It would be nice to have a realistic number to estimate what would the
additional optimization flag save.

--
Ivan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170120/56b18dfd/attachment.html>

From songofacandy at gmail.com  Fri Jan 20 07:15:42 2017
From: songofacandy at gmail.com (INADA Naoki)
Date: Fri, 20 Jan 2017 21:15:42 +0900
Subject: [Python-Dev] Investigating Python memory footprint of one real
 Web application
In-Reply-To: <20170120124333.5f8f5e96@fsol>
References: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
 <20170120124333.5f8f5e96@fsol>
Message-ID: <CAEfz+TwWEjMskjisHvgCBPZ2k7M7LrWA+EaoYH=03WWehNBZZg@mail.gmail.com>

>
> "this script counts static memory usage. It doesn?t care about dynamic
> memory usage of processing real request"
>
> You may be trying to optimize something which is only a very small
> fraction of your actual memory footprint.  That said, the marshal
> module could certainly try to intern some tuples and other immutable
> structures.
>

Yes.  I hadn't think static memory footprint is so important.

But Instagram tried to increase CoW efficiency of prefork application,
and got some success about memory usage and CPU throughput.
I surprised about it because prefork only shares static memory footprint.

Maybe, sharing some tuples which code object has may increase cache efficiency.
I'll try run pyperformance with the marshal patch.


>> * Most large strings are docstring.  Is it worth enough that option
>> for trim docstrings, without disabling asserts?
>
> Perhaps docstrings may be compressed and then lazily decompressed when
> accessed for the first time.  lz4 and zstd are good modern candidates
> for that.  zstd also has a dictionary mode that helps for small data
> (*).  See https://facebook.github.io/zstd/
>
> (*) Even a 200-bytes docstring can be compressed this way:
>
>>>> data = os.times.__doc__.encode()
>>>> len(data)
> 211
>>>> len(lz4.compress(data))
> 200
>>>> c = zstd.ZstdCompressor()
>>>> len(c.compress(data))
> 156
>>>> c = zstd.ZstdCompressor(dict_data=dict_data)
>>>> len(c.compress(data))
> 104
>
> `dict_data` here is some 16KB dictionary I've trained on some Python
> docstrings.  That 16KB dictionary could be computed while building
> Python (or hand-generated from time to time, since it's unlikely to
> change a lot) and put in a static array somewhere:
>

Interesting.  I noticed zstd is added to mercurial (current RC version).
But zstd (and brotli) are new project.  I stay tuned about them.

>
> A similar strategy may be used for annotations and other
> rarely-accessed metadata.
>
> Another possibility, but probably much more costly in terms of initial
> development and maintenance, is to put the docstrings (+ annotations,
> etc.) in a separate file that's lazily read.
>
> I think optimizing the footprint for everyone is much better than
> adding command-line options to disable some specific metadata.
>

I see.  Although -OO option exists, I can't strip only SQLAlchemy's docstrings.
I need to check all dependency libraries doesn't require __doc__ to use -OO
in production.

We have almost one year before 3.7beta1.  We can find and implement better way.

> Regards
>
> Antoine.
>

From songofacandy at gmail.com  Fri Jan 20 07:22:17 2017
From: songofacandy at gmail.com (INADA Naoki)
Date: Fri, 20 Jan 2017 21:22:17 +0900
Subject: [Python-Dev] Investigating Python memory footprint of one real
 Web application
In-Reply-To: <CAOMjWkkk2RZiZ754D5ggHoPDSp-gJ74itUSJukHN88JMbGJx9A@mail.gmail.com>
References: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
 <CAOMjWkkk2RZiZ754D5ggHoPDSp-gJ74itUSJukHN88JMbGJx9A@mail.gmail.com>
Message-ID: <CAEfz+TyVLM_=+oXrx7KOTnj-ry5_SPS+DFm1z2-2hZE91PaFew@mail.gmail.com>

On Fri, Jan 20, 2017 at 8:52 PM, Ivan Levkivskyi <levkivskyi at gmail.com> wrote:
> On 20 January 2017 at 11:49, INADA Naoki <songofacandy at gmail.com> wrote:
>>
>> * typing may increase memory footprint, through functions
>> __attributes__ and abc.
>>    * Can we add option to remove or lazy evaluate __attributes__ ?
>
>
> This idea already appeared few times. I proposed to introduce a flag (e.g.
> -OOO) to ignore function and variable annotations in compile.c
> It was decide to postpone this, but maybe we can get back to this idea.
>
> In 3.6, typing is already (quite heavily) optimized for both speed and
> space.
> I remember doing an experiment comparing a memory footprint with and without
> annotations, the difference was few percent.
> Do you have such comparison (with and without annotations) for your app?
> It would be nice to have a realistic number to estimate what would the
> additional optimization flag save.
>

I'm sorry.  I just read the blog article yesterday and investigate one
application today.
I don't have idea how to compare memory overhead of __annotations__ yet.

And the project I borrowed codebase start using typing very recently,
after reading
Dropbox's story.  So I don't know how % of functions are typed.

I'll survey more about it later, hopefully in this month.

From christian at python.org  Fri Jan 20 07:40:14 2017
From: christian at python.org (Christian Heimes)
Date: Fri, 20 Jan 2017 13:40:14 +0100
Subject: [Python-Dev] Investigating Python memory footprint of one real
 Web application
In-Reply-To: <CAEfz+TwWEjMskjisHvgCBPZ2k7M7LrWA+EaoYH=03WWehNBZZg@mail.gmail.com>
References: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
 <20170120124333.5f8f5e96@fsol>
 <CAEfz+TwWEjMskjisHvgCBPZ2k7M7LrWA+EaoYH=03WWehNBZZg@mail.gmail.com>
Message-ID: <1e8a03a3-afb6-fac1-b3ba-e011c4c87eea@python.org>

On 2017-01-20 13:15, INADA Naoki wrote:
>>
>> "this script counts static memory usage. It doesn?t care about dynamic
>> memory usage of processing real request"
>>
>> You may be trying to optimize something which is only a very small
>> fraction of your actual memory footprint.  That said, the marshal
>> module could certainly try to intern some tuples and other immutable
>> structures.
>>
> 
> Yes.  I hadn't think static memory footprint is so important.
> 
> But Instagram tried to increase CoW efficiency of prefork application,
> and got some success about memory usage and CPU throughput.
> I surprised about it because prefork only shares static memory footprint.
> 
> Maybe, sharing some tuples which code object has may increase cache efficiency.
> I'll try run pyperformance with the marshal patch.

IIRC Thomas Wouters (?) has been working on a patch to move the ref
counter out of the PyObject struct and into a dedicated memory area. He
proposed the idea to improve cache affinity, reduce cache evictions and
to make CoW more efficient. Especially modern ccNUMA machines with
multiple processors could benefit from the improvement, but also single
processor/multi core machines.

Christian


From christian at python.org  Fri Jan 20 07:41:03 2017
From: christian at python.org (Christian Heimes)
Date: Fri, 20 Jan 2017 13:41:03 +0100
Subject: [Python-Dev] Investigating Python memory footprint of one real
 Web application
In-Reply-To: <CAEfz+TwWEjMskjisHvgCBPZ2k7M7LrWA+EaoYH=03WWehNBZZg@mail.gmail.com>
References: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
 <20170120124333.5f8f5e96@fsol>
 <CAEfz+TwWEjMskjisHvgCBPZ2k7M7LrWA+EaoYH=03WWehNBZZg@mail.gmail.com>
Message-ID: <35cb0be4-5cc8-37f5-817a-97bc636d4b0f@python.org>

On 2017-01-20 13:15, INADA Naoki wrote:
>>
>> "this script counts static memory usage. It doesn?t care about dynamic
>> memory usage of processing real request"
>>
>> You may be trying to optimize something which is only a very small
>> fraction of your actual memory footprint.  That said, the marshal
>> module could certainly try to intern some tuples and other immutable
>> structures.
>>
> 
> Yes.  I hadn't think static memory footprint is so important.
> 
> But Instagram tried to increase CoW efficiency of prefork application,
> and got some success about memory usage and CPU throughput.
> I surprised about it because prefork only shares static memory footprint.
> 
> Maybe, sharing some tuples which code object has may increase cache efficiency.
> I'll try run pyperformance with the marshal patch.

IIRC Thomas Wouters (?) has been working on a patch to move the ref
counter out of the PyObject struct and into a dedicated memory area. He
proposed the idea to improve cache affinity, reduce cache evictions and
to make CoW more efficient. Especially modern ccNUMA machines with
multiple processors could benefit from the improvement, but also single
processor/multi core machines.

Christian



From songofacandy at gmail.com  Fri Jan 20 07:42:05 2017
From: songofacandy at gmail.com (INADA Naoki)
Date: Fri, 20 Jan 2017 21:42:05 +0900
Subject: [Python-Dev] Investigating Python memory footprint of one real
 Web application
In-Reply-To: <CAMpsgwYsVu3Yo3bfjg-+uy30Z7OG6Qr1q75jeAx9t3_gQvUEQQ@mail.gmail.com>
References: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
 <CAMpsgwYsVu3Yo3bfjg-+uy30Z7OG6Qr1q75jeAx9t3_gQvUEQQ@mail.gmail.com>
Message-ID: <CAEfz+TwWh+eFYPNR4LyFbncy0fmkjiu5CJ3KGA=RAEuE=ab2BA@mail.gmail.com>

On Fri, Jan 20, 2017 at 8:17 PM, Victor Stinner
<victor.stinner at gmail.com> wrote:
> 2017-01-20 11:49 GMT+01:00 INADA Naoki <songofacandy at gmail.com>:
>> Report is here
>> https://gist.github.com/methane/ce723adb9a4d32d32dc7525b738d3c31
>
> Very interesting report, thanks!
>
>> My thoughts are:
>>
>> * Interning (None,) seems worth enough.
>
> I guess that (None,) comes from constants of code objects:
>
>>>> def f(): pass
> ...
>>>> f.__code__.co_consts
> (None,)
>
>
>> * There are many empty dicts.  Allocating ma_keys lazily may reduce
>> memory usage.
>
> Would you be able to estimate how many bytes would be saved by this
> change? With the total memory usage to have an idea of the %.
>

Smallest dictkeysobject is 5*8 + 8 + (8 * 3 * 5) = 168 bytes.
1600 empty dicts = 268800 bytes.

Unlike tuples bound to code objects, I don't think this is so important
for cache hit rate.  So tuple is more important than dict.


> By the way, it would help if you can add the total memory usage
> computed by tracemalloc (get_traced_memory()[0]) in your report.
>

Oh, nice to know it.  I'll use it next time.

> About empty dict, do you expect that they come from shared keys?
> Anyway, if it has a negligible impact on the performance, go for it
> :-)
>
>
>> but other namespaces or annotations, like ('return',) or ('__wrapped__',) are not shared
>
> Maybe we can intern all tuple which only contains one string?

Ah, it's dict's key.  I used print(tuple(d.keys())) to count dicts.

>
> Instead of interning, would it be possible to at least merge
> duplicated immutable objects?
>

I meant sharing same object, I didn't meant using dict or adding bit
for interning like
interned strings.  So I think we have same idea.

>
>> * Most large strings are docstring.  Is it worth enough that option
>> for trim docstrings, without disabling asserts?
>
> Yeah, you are not the first one to propose. The problem is to decide
> how to name the .pyc file.
>
> My PEP 511 proposes to add a new -O command line option and a new
> sys.implementation.optim_tag string to support this feature:
> https://www.python.org/dev/peps/pep-0511/
>
> Since the code transformer part of the PEP seems to be controversal,
> maybe we should extract only these two changes from the PEP and
> implement them? I also want -O noopt :-) (disable the peephole
> optimizer)
>
> Victor

From solipsis at pitrou.net  Fri Jan 20 07:48:23 2017
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 20 Jan 2017 13:48:23 +0100
Subject: [Python-Dev] Investigating Python memory footprint of one real
 Web application
In-Reply-To: <1e8a03a3-afb6-fac1-b3ba-e011c4c87eea@python.org>
References: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
 <20170120124333.5f8f5e96@fsol>
 <CAEfz+TwWEjMskjisHvgCBPZ2k7M7LrWA+EaoYH=03WWehNBZZg@mail.gmail.com>
 <1e8a03a3-afb6-fac1-b3ba-e011c4c87eea@python.org>
Message-ID: <20170120134823.66f79971@fsol>

On Fri, 20 Jan 2017 13:40:14 +0100
Christian Heimes <christian at python.org> wrote:
> 
> IIRC Thomas Wouters (?) has been working on a patch to move the ref
> counter out of the PyObject struct and into a dedicated memory area. He
> proposed the idea to improve cache affinity, reduce cache evictions and
> to make CoW more efficient.

> Especially modern ccNUMA machines with
> multiple processors could benefit from the improvement, but also single
> processor/multi core machines.

Moving the refcount out of the PyObject will probably make increfs /
decrefs more costly, and there are a lot of them.  We'd have to see
actual measurements if a patch is written, but my intuition is that the
net result won't be positive.

Regards

Antoine.

From victor.stinner at gmail.com  Fri Jan 20 08:08:38 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 20 Jan 2017 14:08:38 +0100
Subject: [Python-Dev] Investigating Python memory footprint of one real
 Web application
In-Reply-To: <1e8a03a3-afb6-fac1-b3ba-e011c4c87eea@python.org>
References: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
 <20170120124333.5f8f5e96@fsol>
 <CAEfz+TwWEjMskjisHvgCBPZ2k7M7LrWA+EaoYH=03WWehNBZZg@mail.gmail.com>
 <1e8a03a3-afb6-fac1-b3ba-e011c4c87eea@python.org>
Message-ID: <CAMpsgwaZ752+4foSM81XTvjb7rLDQn=oJtEr1d7b7hjKhP8ecw@mail.gmail.com>

Larry Hastings' Gilectomy also moved the reference counter into a
separated memory block, no? (grouping all refcounts into large memory
blocks if I understood correctly.)

https://github.com/larryhastings/gilectomy

Victor

2017-01-20 13:40 GMT+01:00 Christian Heimes <christian at python.org>:
> On 2017-01-20 13:15, INADA Naoki wrote:
>>>
>>> "this script counts static memory usage. It doesn?t care about dynamic
>>> memory usage of processing real request"
>>>
>>> You may be trying to optimize something which is only a very small
>>> fraction of your actual memory footprint.  That said, the marshal
>>> module could certainly try to intern some tuples and other immutable
>>> structures.
>>>
>>
>> Yes.  I hadn't think static memory footprint is so important.
>>
>> But Instagram tried to increase CoW efficiency of prefork application,
>> and got some success about memory usage and CPU throughput.
>> I surprised about it because prefork only shares static memory footprint.
>>
>> Maybe, sharing some tuples which code object has may increase cache efficiency.
>> I'll try run pyperformance with the marshal patch.
>
> IIRC Thomas Wouters (?) has been working on a patch to move the ref
> counter out of the PyObject struct and into a dedicated memory area. He
> proposed the idea to improve cache affinity, reduce cache evictions and
> to make CoW more efficient. Especially modern ccNUMA machines with
> multiple processors could benefit from the improvement, but also single
> processor/multi core machines.
>
> Christian
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com

From songofacandy at gmail.com  Fri Jan 20 08:30:16 2017
From: songofacandy at gmail.com (INADA Naoki)
Date: Fri, 20 Jan 2017 22:30:16 +0900
Subject: [Python-Dev] Investigating Python memory footprint of one real
 Web application
In-Reply-To: <20170120134823.66f79971@fsol>
References: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
 <20170120124333.5f8f5e96@fsol>
 <CAEfz+TwWEjMskjisHvgCBPZ2k7M7LrWA+EaoYH=03WWehNBZZg@mail.gmail.com>
 <1e8a03a3-afb6-fac1-b3ba-e011c4c87eea@python.org>
 <20170120134823.66f79971@fsol>
Message-ID: <CAEfz+Txjk1ECUhovj9iZWM98vbSTnr5cu2BVJMHcioYB_DDMNg@mail.gmail.com>

>
> Moving the refcount out of the PyObject will probably make increfs /
> decrefs more costly, and there are a lot of them.  We'd have to see
> actual measurements if a patch is written, but my intuition is that the
> net result won't be positive.
>
> Regards
>
> Antoine.

I agree with you.  But I have similar idea: split only PyGc_Head (3 words).

SImple implementation may just use pointer to PyGc_Head instead of embedding it.
+1 word for tracked objects, and -2 words for untracked objects.

More complex implementation may use bitmap for tracking objects.
Memory pool has the bitmap.
It means GC module have own memory pool and allocator, or GC module
and obmalloc are
tightly coupled.

But it's too hard.  I don't think I can do it by Python 3.7.

Reducing number of tuples may be easier.

From solipsis at pitrou.net  Fri Jan 20 08:53:45 2017
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 20 Jan 2017 14:53:45 +0100
Subject: [Python-Dev] Investigating Python memory footprint of one real
 Web application
In-Reply-To: <CAEfz+Txjk1ECUhovj9iZWM98vbSTnr5cu2BVJMHcioYB_DDMNg@mail.gmail.com>
References: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
 <20170120124333.5f8f5e96@fsol>
 <CAEfz+TwWEjMskjisHvgCBPZ2k7M7LrWA+EaoYH=03WWehNBZZg@mail.gmail.com>
 <1e8a03a3-afb6-fac1-b3ba-e011c4c87eea@python.org>
 <20170120134823.66f79971@fsol>
 <CAEfz+Txjk1ECUhovj9iZWM98vbSTnr5cu2BVJMHcioYB_DDMNg@mail.gmail.com>
Message-ID: <20170120145345.48fa11c2@fsol>

On Fri, 20 Jan 2017 22:30:16 +0900
INADA Naoki <songofacandy at gmail.com> wrote:
> >
> > Moving the refcount out of the PyObject will probably make increfs /
> > decrefs more costly, and there are a lot of them.  We'd have to see
> > actual measurements if a patch is written, but my intuition is that the
> > net result won't be positive.
> >
> > Regards
> >
> > Antoine.  
> 
> I agree with you.  But I have similar idea: split only PyGc_Head (3 words).

That sounds like an interesting idea.  Once an object is created, the
GC header is rarely accessed.  Since the GC header has a small constant
size, it would probably be easy to make its allocation very fast
(e.g. using a freelist).  Then the GC header is out of the way which
increases the cache efficiency of GC-tracked objects.

Regards

Antoine.

From status at bugs.python.org  Fri Jan 20 12:09:10 2017
From: status at bugs.python.org (Python tracker)
Date: Fri, 20 Jan 2017 18:09:10 +0100 (CET)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20170120170910.8D99956C82@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2017-01-13 - 2017-01-20)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    5755 (+36)
  closed 35335 (+35)
  total  41090 (+71)

Open issues with patches: 2484 


Issues opened (49)
==================

#16623: argparse help formatter does not honor non-breaking space
http://bugs.python.org/issue16623  reopened by xiang.zhang

#29264: sparc/ffi.c:440 error: 'asm' undeclared
http://bugs.python.org/issue29264  opened by phantal

#29265: os.cpu_count is problematic on sparc/solaris
http://bugs.python.org/issue29265  opened by phantal

#29266: test_create_connection_service_name fails if "http" isn't list
http://bugs.python.org/issue29266  opened by phantal

#29267: Cannot override some flags in CFLAGS from the command-line
http://bugs.python.org/issue29267  opened by phantal

#29268: test_spwd fails on solaris using NIS users
http://bugs.python.org/issue29268  opened by phantal

#29269: test_socket failing in solaris
http://bugs.python.org/issue29269  opened by phantal

#29270: super call in ctypes sub-class fails in 3.6
http://bugs.python.org/issue29270  opened by Dave Jones

#29271: Task.current_task(None) returns unexpected result
http://bugs.python.org/issue29271  opened by yselivanov

#29272: test_logging hangs if /etc/hosts only aliases "localhost" to :
http://bugs.python.org/issue29272  opened by phantal

#29273: test___all__ alters utf8 locale setting
http://bugs.python.org/issue29273  opened by martin.panter

#29275: time module still has Y2K issues note
http://bugs.python.org/issue29275  opened by Elizacat

#29278: Python 3.6 build fails with parallel make
http://bugs.python.org/issue29278  opened by ugultopu

#29282: Fused multiply-add: proposal to add math.fma()
http://bugs.python.org/issue29282  opened by juraj.sukop

#29283: duplicate README in site-packages
http://bugs.python.org/issue29283  opened by matejcik

#29284: Include thread_name_prefix  in the concurrent.futures.ThreadPo
http://bugs.python.org/issue29284  opened by jftuga

#29286: Use METH_FASTCALL in str methods
http://bugs.python.org/issue29286  opened by haypo

#29287: IDLE needs syntax highlighting for f-strings
http://bugs.python.org/issue29287  opened by rhettinger

#29288: Lookup Error while importing idna from a worker thread
http://bugs.python.org/issue29288  opened by Ilya.Kulakov

#29289: Convert OrderedDict methods to Argument Clinic
http://bugs.python.org/issue29289  opened by haypo

#29290: argparse breaks long lines on NO-BREAK SPACE
http://bugs.python.org/issue29290  opened by steven.daprano

#29293: Missing parameter "n" on multiprocessing.Condition.notify()
http://bugs.python.org/issue29293  opened by Victor de la Fuente

#29298: argparse fails with required subparsers, un-named dest, and em
http://bugs.python.org/issue29298  opened by zachrahan

#29299: Argument Clinic: Fix signature of optional positional-only arg
http://bugs.python.org/issue29299  opened by haypo

#29300: Modify the _struct module to use FASTCALL and Argument Clinic
http://bugs.python.org/issue29300  opened by haypo

#29302: add contextlib.AsyncExitStack
http://bugs.python.org/issue29302  opened by thehesiod

#29304: dict: simplify lookup function
http://bugs.python.org/issue29304  opened by inada.naoki

#29306: Check usage of Py_EnterRecursiveCall() and Py_LeaveRecursiveCa
http://bugs.python.org/issue29306  opened by haypo

#29308: venv should match virtualenv VIRTUAL_ENV_DISABLE_PROMPT config
http://bugs.python.org/issue29308  opened by Jack Bennett

#29309: Interpreter hang when interrupting a loop.run_in_executor() fu
http://bugs.python.org/issue29309  opened by rsebille

#29310: Document typing.NamedTuple default argument syntax
http://bugs.python.org/issue29310  opened by Jelle Zijlstra

#29311: Argument Clinic: convert dict methods
http://bugs.python.org/issue29311  opened by haypo

#29313: msi by bdist_msi will fail execute install-scripts if space in
http://bugs.python.org/issue29313  opened by eszense

#29314: asyncio.async deprecation warning is missing stacklevel=2
http://bugs.python.org/issue29314  opened by r.david.murray

#29317: test_copyxattr_symlinks fails
http://bugs.python.org/issue29317  opened by marktay

#29318: Optimize _PyFunction_FastCallDict() for **kwargs
http://bugs.python.org/issue29318  opened by haypo

#29319: Embedded 3.6.0 distribution cannot run pyz files
http://bugs.python.org/issue29319  opened by paul.moore

#29320: bdist_msi install_script fail to execute if alt python locatio
http://bugs.python.org/issue29320  opened by eszense

#29321: Wrong documentation (Language Ref) for unicode and str compari
http://bugs.python.org/issue29321  opened by RK-5wWm9h

#29323: Wrong documentation (Library) for unicode and str comparison
http://bugs.python.org/issue29323  opened by RK-5wWm9h

#29324: test_aead_aes_gcm fails on Kernel 4.9
http://bugs.python.org/issue29324  opened by christian.heimes

#29326: Blank lines in ._pth file are not ignored
http://bugs.python.org/issue29326  opened by steve.dower

#29328: struct module should support variable-length strings
http://bugs.python.org/issue29328  opened by Elizacat

#29329: Incorrect documentation for custom `hex()` support on Python 2
http://bugs.python.org/issue29329  opened by pekka.klarck

#29330: __slots__ needs documentation
http://bugs.python.org/issue29330  opened by saumitra1978

#29331: Simplify argument parsing in sorted() and list.sort()
http://bugs.python.org/issue29331  opened by serhiy.storchaka

#29332: Uniform SelectSelector._select behavior
http://bugs.python.org/issue29332  opened by Wen Adam

#29333: ConfigParser calls Interpolation.before_read after reading
http://bugs.python.org/issue29333  opened by Anaphory

#29334: ssl.SSLObject method getpeercert() is buggy, do_handshake() is
http://bugs.python.org/issue29334  opened by Greg Stark



Most recent 15 issues with no replies (15)
==========================================

#29333: ConfigParser calls Interpolation.before_read after reading
http://bugs.python.org/issue29333

#29332: Uniform SelectSelector._select behavior
http://bugs.python.org/issue29332

#29326: Blank lines in ._pth file are not ignored
http://bugs.python.org/issue29326

#29320: bdist_msi install_script fail to execute if alt python locatio
http://bugs.python.org/issue29320

#29317: test_copyxattr_symlinks fails
http://bugs.python.org/issue29317

#29314: asyncio.async deprecation warning is missing stacklevel=2
http://bugs.python.org/issue29314

#29310: Document typing.NamedTuple default argument syntax
http://bugs.python.org/issue29310

#29308: venv should match virtualenv VIRTUAL_ENV_DISABLE_PROMPT config
http://bugs.python.org/issue29308

#29302: add contextlib.AsyncExitStack
http://bugs.python.org/issue29302

#29288: Lookup Error while importing idna from a worker thread
http://bugs.python.org/issue29288

#29287: IDLE needs syntax highlighting for f-strings
http://bugs.python.org/issue29287

#29284: Include thread_name_prefix  in the concurrent.futures.ThreadPo
http://bugs.python.org/issue29284

#29283: duplicate README in site-packages
http://bugs.python.org/issue29283

#29273: test___all__ alters utf8 locale setting
http://bugs.python.org/issue29273

#29272: test_logging hangs if /etc/hosts only aliases "localhost" to :
http://bugs.python.org/issue29272



Most recent 15 issues waiting for review (15)
=============================================

#29332: Uniform SelectSelector._select behavior
http://bugs.python.org/issue29332

#29331: Simplify argument parsing in sorted() and list.sort()
http://bugs.python.org/issue29331

#29313: msi by bdist_msi will fail execute install-scripts if space in
http://bugs.python.org/issue29313

#29311: Argument Clinic: convert dict methods
http://bugs.python.org/issue29311

#29306: Check usage of Py_EnterRecursiveCall() and Py_LeaveRecursiveCa
http://bugs.python.org/issue29306

#29304: dict: simplify lookup function
http://bugs.python.org/issue29304

#29300: Modify the _struct module to use FASTCALL and Argument Clinic
http://bugs.python.org/issue29300

#29299: Argument Clinic: Fix signature of optional positional-only arg
http://bugs.python.org/issue29299

#29290: argparse breaks long lines on NO-BREAK SPACE
http://bugs.python.org/issue29290

#29289: Convert OrderedDict methods to Argument Clinic
http://bugs.python.org/issue29289

#29286: Use METH_FASTCALL in str methods
http://bugs.python.org/issue29286

#29282: Fused multiply-add: proposal to add math.fma()
http://bugs.python.org/issue29282

#29273: test___all__ alters utf8 locale setting
http://bugs.python.org/issue29273

#29271: Task.current_task(None) returns unexpected result
http://bugs.python.org/issue29271

#29270: super call in ctypes sub-class fails in 3.6
http://bugs.python.org/issue29270



Top 10 most discussed issues (10)
=================================

#29259: Add tp_fastcall to PyTypeObject: support FASTCALL calling conv
http://bugs.python.org/issue29259  26 msgs

#29282: Fused multiply-add: proposal to add math.fma()
http://bugs.python.org/issue29282  19 msgs

#29262: Provide a way to check for *real* typing.Union instances
http://bugs.python.org/issue29262  15 msgs

#9216: FIPS support for hashlib
http://bugs.python.org/issue9216  13 msgs

#29286: Use METH_FASTCALL in str methods
http://bugs.python.org/issue29286  13 msgs

#20180: Derby #11: Convert 50 sites to Argument Clinic across 9 files
http://bugs.python.org/issue20180  10 msgs

#20186: Derby #18: Convert 31 sites to Argument Clinic across 23 files
http://bugs.python.org/issue20186   9 msgs

#29289: Convert OrderedDict methods to Argument Clinic
http://bugs.python.org/issue29289   9 msgs

#29311: Argument Clinic: convert dict methods
http://bugs.python.org/issue29311   9 msgs

#29319: Embedded 3.6.0 distribution cannot run pyz files
http://bugs.python.org/issue29319   9 msgs



Issues closed (33)
==================

#26110: Speedup method calls 1.2x
http://bugs.python.org/issue26110  closed by inada.naoki

#26296: colorsys rgb_to_hls algorithm error
http://bugs.python.org/issue26296  closed by serhiy.storchaka

#27596: Build failure with Xcode 8 beta on OSX 10.11
http://bugs.python.org/issue27596  closed by ned.deily

#29011: No entry Deque in typing.py
http://bugs.python.org/issue29011  closed by rhettinger

#29029: Faster positional arguments parsing in PyArg_ParseTupleAndKeyw
http://bugs.python.org/issue29029  closed by serhiy.storchaka

#29057: Compiler failure on Mac OS X - sys/random.h
http://bugs.python.org/issue29057  closed by ned.deily

#29132: shlex.shlex with punctuation_chars and posix doesn't handle pu
http://bugs.python.org/issue29132  closed by python-dev

#29197: Remove os.path.splitunc()
http://bugs.python.org/issue29197  closed by serhiy.storchaka

#29246: typing.Union raises RecursionError when comparing Union to oth
http://bugs.python.org/issue29246  closed by gvanrossum

#29261: Missing venv/scripts/common after "make install"
http://bugs.python.org/issue29261  closed by python-dev

#29274: Change ???tests cases??? ??? ???test cases???
http://bugs.python.org/issue29274  closed by martin.panter

#29276: HTMLParser in Python 2.7 doesn't recognize image tags wrapped 
http://bugs.python.org/issue29276  closed by Ari

#29277: os.getcwd failing on LOFS share
http://bugs.python.org/issue29277  closed by haypo

#29279: --with-pydebug optimizes too much
http://bugs.python.org/issue29279  closed by rhettinger

#29280: gdbm & ndbm support missing in Windows
http://bugs.python.org/issue29280  closed by r.david.murray

#29281: json.loads documentation missing "versionchanged" statement
http://bugs.python.org/issue29281  closed by rhettinger

#29285: Unicode errors occur inside of multi-line comments
http://bugs.python.org/issue29285  closed by zach.ware

#29291: Misleading text in the documentation of re library for non-gre
http://bugs.python.org/issue29291  closed by rhettinger

#29292: Missing a parameter in PyEval_EvalCodeEx doc
http://bugs.python.org/issue29292  closed by xiang.zhang

#29294: ctypes.windll.LoadLibrary refuses unicode argument
http://bugs.python.org/issue29294  closed by theller

#29295: dict: Optimize PyDict_GetItemString()
http://bugs.python.org/issue29295  closed by inada.naoki

#29296: convert print() to METH_FASTCALL
http://bugs.python.org/issue29296  closed by inada.naoki

#29297: python3 open() does not check argument type before attempting 
http://bugs.python.org/issue29297  closed by serhiy.storchaka

#29301: decimal: Use FASTCALL and/or Argument Clinic
http://bugs.python.org/issue29301  closed by haypo

#29303: asyncio.Lock.acquire() does not always yield
http://bugs.python.org/issue29303  closed by gvanrossum

#29305: encoding to ascii in http/client.py
http://bugs.python.org/issue29305  closed by martin.panter

#29307: ModuleNotFoundError when using literal string interpolation wi
http://bugs.python.org/issue29307  closed by r.david.murray

#29312: Use FASTCALL in dict.update()
http://bugs.python.org/issue29312  closed by haypo

#29315: \b requires raw strings or to be escaped. Update docs with tha
http://bugs.python.org/issue29315  closed by r.david.murray

#29316: Keep typing.py provisional for the duration of the Python 3.6 
http://bugs.python.org/issue29316  closed by ned.deily

#29322: SimpleCV error on Raspberry Pi
http://bugs.python.org/issue29322  closed by xiang.zhang

#29325: pysqlite: Evaluate removal of sqlite3_stmt_readonly
http://bugs.python.org/issue29325  closed by xiang.zhang

#29327: SystemError or crash in sorted(iterable=[])
http://bugs.python.org/issue29327  closed by serhiy.storchaka

From songofacandy at gmail.com  Fri Jan 20 22:42:27 2017
From: songofacandy at gmail.com (INADA Naoki)
Date: Sat, 21 Jan 2017 12:42:27 +0900
Subject: [Python-Dev] Investigating Python memory footprint of one real
 Web application
In-Reply-To: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
References: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
Message-ID: <CAEfz+TyNVy-iA43ennS19Q4ob5PNrdF1cHR4nsdiMVTeLKRmew@mail.gmail.com>

I've filed an issue about merging tuples: http://bugs.python.org/issue29336

I'll try the patch with my company's codebase again in next week.
But could someone try the patch with realworld large application too?

Or if you know OSS large application easy to install, could you share
requirements.txt + script imports many modules in the application?

From Dahui.Jiang at veritas.com  Sat Jan 21 20:58:39 2017
From: Dahui.Jiang at veritas.com (Dahui Jiang)
Date: Sun, 22 Jan 2017 01:58:39 +0000
Subject: [Python-Dev] Have problem when building python3.5.1 rpm with
 default SPEC file
In-Reply-To: <1459225640.92062396.1484851212581.JavaMail.zimbra@redhat.com>
References: <fcf13f7ce88a46599c11579856be56a6@vrtsxchclupin16.community.veritas.com>
 <1459225640.92062396.1484851212581.JavaMail.zimbra@redhat.com>
Message-ID: <e2189a8897494c4ab0a9903a7aa30d12@vrtsxchclupin16.community.veritas.com>

Hi Charalampos:

Thank you very much for your help, and I have built python3.5.1 rpm on redhat7 successfully.
But I find that there are quite a few content in SPEC file of fedora, and it seems that the file has been developed for a long time for patches and other supplementary, 
now that the SPEC file of fedora is open source, how about redhat's, how could I get it? 

Regards
Dahui

-----Original Message-----
From: Charalampos Stratakis [mailto:cstratak at redhat.com] 
Sent: Friday, January 20, 2017 2:40
To: Dahui Jiang <Dahui.Jiang at veritas.com>
Cc: python-dev at python.org
Subject: Re: [Python-Dev] Have problem when building python3.5.1 rpm with default SPEC file

Hello,

This is a distro specific issue so this list might not be the best for resolving that, you should contact your distro's package maintainers of python.

For Fedora 25 we currently ship Python 3.5.2, which builds fine with this SPEC file [0], so maybe you could give this a try.

[0] http://pkgs.fedoraproject.org/cgit/rpms/python3.git/tree/python3.spec?h=f25

Regards,

Charalampos Stratakis
Associate Software Engineer
Python Maintenance Team, Red Hat


----- Original Message -----
From: "Dahui Jiang" <Dahui.Jiang at veritas.com>
To: python-dev at python.org
Sent: Thursday, January 19, 2017 12:54:16 PM
Subject: [Python-Dev] Have problem when building python3.5.1 rpm with default SPEC file



Hi all: 

I?m try to build a python rpm from source python3.5.1, and I use the spec file in the source tree. 

But the building is not success as print the following error: 



*********************************************** 

running build 

running build_ext 

error: [Errno 2] No such file or directory: 'Modules/Setup' 

error: Bad exit status from /var/tmp/rpm-tmp.DDm3jI (%build) 





RPM build errors: 

Bad exit status from /var/tmp/rpm-tmp.DDm3jI (%build) 

************************************************ 



Regards 

Dahui 

_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/cstratak%40redhat.com

From stefanrin at gmail.com  Sun Jan 22 04:55:05 2017
From: stefanrin at gmail.com (Stefan Ring)
Date: Sun, 22 Jan 2017 10:55:05 +0100
Subject: [Python-Dev] Have problem when building python3.5.1 rpm with
 default SPEC file
In-Reply-To: <e2189a8897494c4ab0a9903a7aa30d12@vrtsxchclupin16.community.veritas.com>
References: <fcf13f7ce88a46599c11579856be56a6@vrtsxchclupin16.community.veritas.com>
 <1459225640.92062396.1484851212581.JavaMail.zimbra@redhat.com>
 <e2189a8897494c4ab0a9903a7aa30d12@vrtsxchclupin16.community.veritas.com>
Message-ID: <CAAxjCExUDpJ4tXN9=db2c9y2Odzp2ER8TFyMLNZzVCkK1UM1hw@mail.gmail.com>

> now that the SPEC file of fedora is open source, how about redhat's, how could I get it?

Fedora's spec files lives here:
http://pkgs.fedoraproject.org/cgit/rpms/python3.git

From brett at python.org  Sun Jan 22 13:19:22 2017
From: brett at python.org (Brett Cannon)
Date: Sun, 22 Jan 2017 18:19:22 +0000
Subject: [Python-Dev] Update on the GitHub migration
Message-ID: <CAP1=2W4Z5-pGPe8D-xJM-+EkYQ=cvfFSSCVOjPbZxXEzn9Md1w@mail.gmail.com>

The last blocker is updating issues on bugs.python.org when a commit is
made that references an issue (Maciej is working on it, although as usual
I'm sure help is welcome). Once that's in then it will be time to choose a
date to stop commits and do the conversion. Once that's done it will be a
bunch of little things to update due to the repo moving (e.g
 building the docs from git).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170122/e753e129/attachment.html>

From storchaka at gmail.com  Sun Jan 22 14:48:15 2017
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sun, 22 Jan 2017 21:48:15 +0200
Subject: [Python-Dev] Default formatting
In-Reply-To: <nun94q$64o$1@blaine.gmane.org>
References: <nun94q$64o$1@blaine.gmane.org>
Message-ID: <o6329r$v4d$1@blaine.gmane.org>

On 25.10.16 12:37, Serhiy Storchaka wrote:
> Classes that doesn't define the __format__ method for custom PEP 3101
> formatting inherits it from parents.
>
> Originally the object.__format__ method was designed as [1]:
>
>     def __format__(self, format_spec):
>         return format(str(self), format_spec)
>
> An instance is converted to string and resulting string is formatted
> according to format specifier.
>
> Later this design was reconsidered [2], and now object.__format__ is
> equivalent to:
>
>     def __format__(self, format_spec):
>         assert format_spec == ''
>         return format(str(self), '')
>
> Non-empty format specifier is rejected.
>
> But why call format() on resulting string? Why not return resulting
> string as is? object.__format__ could be simpler (not just
> implementation, but for understanding):
>
>     def __format__(self, format_spec):
>         assert format_spec == ''
>         return str(self)
>
> This can change the behaviour in corner case. str(self) can return not
> exact string, but string subclass with overloaded __format__. But I
> think we can ignore such subtle difference.
>
> [1] https://www.python.org/dev/peps/pep-3101/
> [2] http://bugs.python.org/issue7994

What is the decision about this?


From ethan at stoneleaf.us  Sun Jan 22 15:02:34 2017
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 22 Jan 2017 12:02:34 -0800
Subject: [Python-Dev] adding threaded tests to the test suite
Message-ID: <58850FDA.1060302@stoneleaf.us>

Question:  I need to add a threaded test to the enum test module [1] -- is there anything extra I
need to worry about besides the test itself?  Setting or resetting or using a tool library, etc?

--
~Ethan~


[1] The test to be added:

     def test_unique_composite(self):
         # override __eq__ to be identity only
         class TestFlag(IntFlag):
             one = auto()
             two = auto()
             three = auto()
             four = auto()
             five = auto()
             six = auto()
             seven = auto()
             eight = auto()
             def __eq__(self, other):
                 return self is other
             def __hash__(self):
                 return hash(self._value_)
         # have multiple threads competing to complete the composite members
         seen = set()
         failed = False
         def cycle_enum():
             nonlocal failed
             try:
                 for i in range(256):
                     seen.add(TestFlag(i))
             except (Exception, RuntimeError):
                 failed = True
         threads = []
         for i in range(8):
             threads.append(threading.Thread(target=cycle_enum))
         for t in threads:
             t.start()
         for t in threads:
             t.join()
         # check that only 248 members were created
         self.assertFalse(
                 failed,
                 'at least one thread failed while creating composite members')
         self.assertEqual(256, len(seen), 'too many composite members created')

From ethan at stoneleaf.us  Sun Jan 22 15:07:04 2017
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 22 Jan 2017 12:07:04 -0800
Subject: [Python-Dev] Default formatting
In-Reply-To: <o6329r$v4d$1@blaine.gmane.org>
References: <nun94q$64o$1@blaine.gmane.org> <o6329r$v4d$1@blaine.gmane.org>
Message-ID: <588510E8.2090200@stoneleaf.us>

On 01/22/2017 11:48 AM, Serhiy Storchaka wrote:
> On 25.10.16 12:37, Serhiy Storchaka wrote:
>> Classes that doesn't define the __format__ method for custom PEP 3101
>> formatting inherits it from parents.
>>
>> Originally the object.__format__ method was designed as [1]:
>>
>>     def __format__(self, format_spec):
>>         return format(str(self), format_spec)
>>
>> An instance is converted to string and resulting string is formatted
>> according to format specifier.
>>
>> Later this design was reconsidered [2], and now object.__format__ is
>> equivalent to:
>>
>>     def __format__(self, format_spec):
>>         assert format_spec == ''
>>         return format(str(self), '')
>>
>> Non-empty format specifier is rejected.
>>
>> But why call format() on resulting string? Why not return resulting
>> string as is? object.__format__ could be simpler (not just
>> implementation, but for understanding):
>>
>>     def __format__(self, format_spec):
>>         assert format_spec == ''
>>         return str(self)
>>
>> This can change the behaviour in corner case. str(self) can return not
>> exact string, but string subclass with overloaded __format__. But I
>> think we can ignore such subtle difference.
>>
>> [1] https://www.python.org/dev/peps/pep-3101/
>> [2] http://bugs.python.org/issue7994

Can you give an example of this corner case?

--
~Ethan~

From victor.stinner at gmail.com  Sun Jan 22 15:17:25 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sun, 22 Jan 2017 20:17:25 +0000
Subject: [Python-Dev] adding threaded tests to the test suite
In-Reply-To: <58850FDA.1060302@stoneleaf.us>
References: <58850FDA.1060302@stoneleaf.us>
Message-ID: <CAMpsgwaqnYJxRVazh=yOALsHm4uimAX+LFWROx6_jvwfKX_j=g@mail.gmail.com>

There is @support.reap_thread which can help.

Victor

Le dim. 22 janv. 2017 ? 21:04, Ethan Furman <ethan at stoneleaf.us> a ?crit :

> Question:  I need to add a threaded test to the enum test module [1] -- is
> there anything extra I
>
> need to worry about besides the test itself?  Setting or resetting or
> using a tool library, etc?
>
>
>
> --
>
> ~Ethan~
>
>
>
>
>
> [1] The test to be added:
>
>
>
>      def test_unique_composite(self):
>
>          # override __eq__ to be identity only
>
>          class TestFlag(IntFlag):
>
>              one = auto()
>
>              two = auto()
>
>              three = auto()
>
>              four = auto()
>
>              five = auto()
>
>              six = auto()
>
>              seven = auto()
>
>              eight = auto()
>
>              def __eq__(self, other):
>
>                  return self is other
>
>              def __hash__(self):
>
>                  return hash(self._value_)
>
>          # have multiple threads competing to complete the composite
> members
>
>          seen = set()
>
>          failed = False
>
>          def cycle_enum():
>
>              nonlocal failed
>
>              try:
>
>                  for i in range(256):
>
>                      seen.add(TestFlag(i))
>
>              except (Exception, RuntimeError):
>
>                  failed = True
>
>          threads = []
>
>          for i in range(8):
>
>              threads.append(threading.Thread(target=cycle_enum))
>
>          for t in threads:
>
>              t.start()
>
>          for t in threads:
>
>              t.join()
>
>          # check that only 248 members were created
>
>          self.assertFalse(
>
>                  failed,
>
>                  'at least one thread failed while creating composite
> members')
>
>          self.assertEqual(256, len(seen), 'too many composite members
> created')
>
> _______________________________________________
>
> Python-Dev mailing list
>
> Python-Dev at python.org
>
> https://mail.python.org/mailman/listinfo/python-dev
>
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170122/0d51607c/attachment.html>

From vadmium+py at gmail.com  Sun Jan 22 15:39:16 2017
From: vadmium+py at gmail.com (Martin Panter)
Date: Sun, 22 Jan 2017 20:39:16 +0000
Subject: [Python-Dev] adding threaded tests to the test suite
In-Reply-To: <CAMpsgwaqnYJxRVazh=yOALsHm4uimAX+LFWROx6_jvwfKX_j=g@mail.gmail.com>
References: <58850FDA.1060302@stoneleaf.us>
 <CAMpsgwaqnYJxRVazh=yOALsHm4uimAX+LFWROx6_jvwfKX_j=g@mail.gmail.com>
Message-ID: <CA+eR4cFGv=ZQ2On55zx9+66WNj2C923CRouhwyReLn82EqNLtg@mail.gmail.com>

> Le dim. 22 janv. 2017 ? 21:04, Ethan Furman <ethan at stoneleaf.us> a ?crit :
>> Question:  I need to add a threaded test to the enum test module [1] -- is
>> there anything extra I
>> need to worry about besides the test itself?  Setting or resetting or
>> using a tool library, etc?
>>
>> threads = []
>> for i in range(8):
>>     threads.append(threading.Thread(target=cycle_enum))
>> for t in threads:
>>     t.start()
>> for t in threads:
>>     t.join()

On 22 January 2017 at 20:17, Victor Stinner <victor.stinner at gmail.com> wrote:
> There is @support.reap_thread which can help.

As I understand, @reap_threads basically does a join() on each
background thread, with a total timeout of 1 s. So since your test is
unlikely to fail between starting threads and joining them, I don?t
think you need to use @reap_threads.

From zachary.ware+pydev at gmail.com  Sun Jan 22 15:46:05 2017
From: zachary.ware+pydev at gmail.com (Zachary Ware)
Date: Sun, 22 Jan 2017 14:46:05 -0600
Subject: [Python-Dev] adding threaded tests to the test suite
In-Reply-To: <58850FDA.1060302@stoneleaf.us>
References: <58850FDA.1060302@stoneleaf.us>
Message-ID: <CAKJDb-OVpsUYLnNQ1iF_CVZn2W4nAAdEa2RY3JOxasif+SWLbg@mail.gmail.com>

On Sun, Jan 22, 2017 at 2:02 PM, Ethan Furman <ethan at stoneleaf.us> wrote:
> Question:  I need to add a threaded test to the enum test module [1] -- is
> there anything extra I
> need to worry about besides the test itself?  Setting or resetting or using
> a tool library, etc?

As far as I know, the only extras to worry about are to use the
support.reap_threads decorator, and be sure to skip the test if
threading is not available.  Search through the tests for
'reap_threads' if you want other examples of how to handle threaded
tests.

-- 
Zach

From zachary.ware+pydev at gmail.com  Sun Jan 22 15:51:49 2017
From: zachary.ware+pydev at gmail.com (Zachary Ware)
Date: Sun, 22 Jan 2017 14:51:49 -0600
Subject: [Python-Dev] adding threaded tests to the test suite
In-Reply-To: <CA+eR4cFGv=ZQ2On55zx9+66WNj2C923CRouhwyReLn82EqNLtg@mail.gmail.com>
References: <58850FDA.1060302@stoneleaf.us>
 <CAMpsgwaqnYJxRVazh=yOALsHm4uimAX+LFWROx6_jvwfKX_j=g@mail.gmail.com>
 <CA+eR4cFGv=ZQ2On55zx9+66WNj2C923CRouhwyReLn82EqNLtg@mail.gmail.com>
Message-ID: <CAKJDb-MYtoh_yj8yQ0tc9WLXBeEL7E0H2ko5VHupyE0iCt1g-w@mail.gmail.com>

On Sun, Jan 22, 2017 at 2:39 PM, Martin Panter <vadmium+py at gmail.com> wrote:
> As I understand, @reap_threads basically does a join() on each
> background thread, with a total timeout of 1 s. So since your test is
> unlikely to fail between starting threads and joining them, I don?t
> think you need to use @reap_threads.

reap_threads is meant as a failsafe, in case your test case doesn't
clean up after itself properly.  Most of the time, reap_threads
shouldn't actually *do* anything.

-- 
Zach

From ethan at stoneleaf.us  Sun Jan 22 16:58:14 2017
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 22 Jan 2017 13:58:14 -0800
Subject: [Python-Dev] adding threaded tests to the test suite
In-Reply-To: <58850FDA.1060302@stoneleaf.us>
References: <58850FDA.1060302@stoneleaf.us>
Message-ID: <58852AF6.2070306@stoneleaf.us>

On 01/22/2017 12:02 PM, Ethan Furman wrote:

> Question:  I need to add a threaded test to the enum test module [1] -- is there anything extra I
> need to worry about besides the test itself?  Setting or resetting or using a tool library, etc?

Thanks everyone.

@support.reap_threads and skipping it is.

--
~Ethan~

From storchaka at gmail.com  Sun Jan 22 17:48:45 2017
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 23 Jan 2017 00:48:45 +0200
Subject: [Python-Dev] adding threaded tests to the test suite
In-Reply-To: <58850FDA.1060302@stoneleaf.us>
References: <58850FDA.1060302@stoneleaf.us>
Message-ID: <o63cs9$v6i$1@blaine.gmane.org>

On 22.01.17 22:02, Ethan Furman wrote:
> Question:  I need to add a threaded test to the enum test module [1] --
> is there anything extra I
> need to worry about besides the test itself?  Setting or resetting or
> using a tool library, etc?

You can use the test.support.start_threads() context manager.



From ramseydsilva at gmail.com  Sun Jan 22 21:31:20 2017
From: ramseydsilva at gmail.com (Ramsey D'silva)
Date: Sun, 22 Jan 2017 21:31:20 -0500
Subject: [Python-Dev] Hello World
Message-ID: <CAJLRRs_rdNmtqiTs_A90bqs5BYrp8iZ+XNSxqROoUjDkC54uVQ@mail.gmail.com>

Hello World,

I am a python developer that has lurked beyond my box.

Nice to meet all of you'll. I'm excited to learn and hopefully contribute
someday.

Best Regards,
Ramsey

https://linkedin.com/in/ramseydsilva
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170122/94a20553/attachment.html>

From songofacandy at gmail.com  Mon Jan 23 06:25:40 2017
From: songofacandy at gmail.com (INADA Naoki)
Date: Mon, 23 Jan 2017 20:25:40 +0900
Subject: [Python-Dev] Investigating Python memory footprint of one real
 Web application
In-Reply-To: <CAOMjWkkk2RZiZ754D5ggHoPDSp-gJ74itUSJukHN88JMbGJx9A@mail.gmail.com>
References: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
 <CAOMjWkkk2RZiZ754D5ggHoPDSp-gJ74itUSJukHN88JMbGJx9A@mail.gmail.com>
Message-ID: <CAEfz+TxoU0V+FHTWKfudDy+f+W8n7rBM9f-ZGD0zNR704csD3A@mail.gmail.com>

On Fri, Jan 20, 2017 at 8:52 PM, Ivan Levkivskyi <levkivskyi at gmail.com> wrote:
> On 20 January 2017 at 11:49, INADA Naoki <songofacandy at gmail.com> wrote:
>>
>> * typing may increase memory footprint, through functions
>> __attributes__ and abc.
>>    * Can we add option to remove or lazy evaluate __attributes__ ?
>
>
> This idea already appeared few times. I proposed to introduce a flag (e.g.
> -OOO) to ignore function and variable annotations in compile.c
> It was decide to postpone this, but maybe we can get back to this idea.
>
> In 3.6, typing is already (quite heavily) optimized for both speed and
> space.
> I remember doing an experiment comparing a memory footprint with and without
> annotations, the difference was few percent.
> Do you have such comparison (with and without annotations) for your app?
> It would be nice to have a realistic number to estimate what would the
> additional optimization flag save.
>
> --
> Ivan
>
>

Hi, Ivan.

I investigated why our app has so many WeakSet today.

We have dozen or hundreds of annotations like Iterable[User] or List[User].
(User is one example of application's domain object.  There are
hundreds of classes).

On the other hand, SQLAlchemy calls isinstance(obj,
collections.Iterable) many times,
in [sqlalchemy.util._collections.to_list](https://github.com/zzzeek/sqlalchemy/blob/master/lib/sqlalchemy/util/_collections.py#L795-L804)
method.

So there are (# of iterable subclasses) weaksets for negative cache,
and each weakset
contains (# of column types) entries.  That's why WeakSet ate much RAM.

It may be slowdown application startup too, because thousands of
__subclasscheck_ is called.

I gave advice to use 'List[User]' instead of List[User] to the team of
the project,
if the team think RAM usage or boot speed is important.

FWIW, stacktrace is like this:

  File "/Users/inada-n/local/py37dbg/lib/python3.7/_weakrefset.py", line 84
    self.data.add(ref(item, self._remove))
  File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 233
    cls._abc_negative_cache.add(subclass)
  File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 226
    if issubclass(subclass, scls):
  File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 226
    if issubclass(subclass, scls):
  File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 191
    return cls.__subclasscheck__(subclass)
  File "venv/lib/python3.7/site-packages/sqlalchemy/util/_collections.py",
line 803
    or not isinstance(x, collections.Iterable):
  File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line 1680
    columns = util.to_list(prop)
  File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line 1575
    prop = self._property_from_column(key, prop)
  File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line 1371
    setparent=True)
  File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line 675
    self._configure_properties()<PasteEnd>

Regards,

From victor.stinner at gmail.com  Mon Jan 23 06:33:36 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 23 Jan 2017 12:33:36 +0100
Subject: [Python-Dev] Investigating Python memory footprint of one real
 Web application
In-Reply-To: <CAEfz+TxoU0V+FHTWKfudDy+f+W8n7rBM9f-ZGD0zNR704csD3A@mail.gmail.com>
References: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
 <CAOMjWkkk2RZiZ754D5ggHoPDSp-gJ74itUSJukHN88JMbGJx9A@mail.gmail.com>
 <CAEfz+TxoU0V+FHTWKfudDy+f+W8n7rBM9f-ZGD0zNR704csD3A@mail.gmail.com>
Message-ID: <CAMpsgwbX7x3sHrc92bULTgJ5R1vsi8fagMU83Wct8OAShBDzmA@mail.gmail.com>

2017-01-23 12:25 GMT+01:00 INADA Naoki <songofacandy at gmail.com>:
> I gave advice to use 'List[User]' instead of List[User] to the team of
> the project,
> if the team think RAM usage or boot speed is important.

I would prefer a Python option (ex: "-o noannotation" command line
option) to opt-out annotations rather than having to write annotations
in strings, which is IMHO more "ugly".

Victor

From songofacandy at gmail.com  Mon Jan 23 07:25:33 2017
From: songofacandy at gmail.com (INADA Naoki)
Date: Mon, 23 Jan 2017 21:25:33 +0900
Subject: [Python-Dev] Investigating Python memory footprint of one real
 Web application
In-Reply-To: <CAMpsgwbX7x3sHrc92bULTgJ5R1vsi8fagMU83Wct8OAShBDzmA@mail.gmail.com>
References: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
 <CAOMjWkkk2RZiZ754D5ggHoPDSp-gJ74itUSJukHN88JMbGJx9A@mail.gmail.com>
 <CAEfz+TxoU0V+FHTWKfudDy+f+W8n7rBM9f-ZGD0zNR704csD3A@mail.gmail.com>
 <CAMpsgwbX7x3sHrc92bULTgJ5R1vsi8fagMU83Wct8OAShBDzmA@mail.gmail.com>
Message-ID: <CAEfz+Tzz1gc0KRxC79p55LKG2+YuuP9A77Z_k3HXGLWMAUm_Jw@mail.gmail.com>

On Mon, Jan 23, 2017 at 8:33 PM, Victor Stinner
<victor.stinner at gmail.com> wrote:
> 2017-01-23 12:25 GMT+01:00 INADA Naoki <songofacandy at gmail.com>:
>> I gave advice to use 'List[User]' instead of List[User] to the team of
>> the project,
>> if the team think RAM usage or boot speed is important.
>
> I would prefer a Python option (ex: "-o noannotation" command line
> option) to opt-out annotations rather than having to write annotations
> in strings, which is IMHO more "ugly".
>
> Victor

Personally speaking, I hope annotations are just static hint, and
makes zero overhead at runtime.
(startup time, memory consumption, and execution speed).

Anyway, many users are starting to use typing, for code completion or
static checking.
And very few user noticed it affects to performance of `isinstance(x,
collections.Sequence)`.
Python 3.7 may be too slow to help them.
Can't we skip abc registration of typing.List[MyClass] completely?

I'm sorry if it's silly idea.  I don't know about background of
current typing.py design. And I
don't use abc much too.

Naoki

From brett at python.org  Mon Jan 23 14:59:16 2017
From: brett at python.org (Brett Cannon)
Date: Mon, 23 Jan 2017 19:59:16 +0000
Subject: [Python-Dev] Hello World
In-Reply-To: <CAJLRRs_rdNmtqiTs_A90bqs5BYrp8iZ+XNSxqROoUjDkC54uVQ@mail.gmail.com>
References: <CAJLRRs_rdNmtqiTs_A90bqs5BYrp8iZ+XNSxqROoUjDkC54uVQ@mail.gmail.com>
Message-ID: <CAP1=2W4iHv99WF+TGuSX7Q9bA2okU9O4iMS5k=W3Firni=Gb=A@mail.gmail.com>

In case you haven't already done so, Ramsey, please consider reading
https://cpython-devguide.readthedocs.io if you want to contribute to the
project. You can also subscribe to the core-mentorship mailing list which
is specifically for people who want to help out but haven't done to
previously.

On Sun, 22 Jan 2017 at 18:53 Ramsey D'silva <ramseydsilva at gmail.com> wrote:

> Hello World,
>
> I am a python developer that has lurked beyond my box.
>
> Nice to meet all of you'll. I'm excited to learn and hopefully contribute
> someday.
>
> Best Regards,
> Ramsey
>
> https://linkedin.com/in/ramseydsilva
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170123/7ffce022/attachment.html>

From brett at python.org  Mon Jan 23 15:10:46 2017
From: brett at python.org (Brett Cannon)
Date: Mon, 23 Jan 2017 20:10:46 +0000
Subject: [Python-Dev] Investigating Python memory footprint of one real
 Web application
In-Reply-To: <CAEfz+Tzz1gc0KRxC79p55LKG2+YuuP9A77Z_k3HXGLWMAUm_Jw@mail.gmail.com>
References: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
 <CAOMjWkkk2RZiZ754D5ggHoPDSp-gJ74itUSJukHN88JMbGJx9A@mail.gmail.com>
 <CAEfz+TxoU0V+FHTWKfudDy+f+W8n7rBM9f-ZGD0zNR704csD3A@mail.gmail.com>
 <CAMpsgwbX7x3sHrc92bULTgJ5R1vsi8fagMU83Wct8OAShBDzmA@mail.gmail.com>
 <CAEfz+Tzz1gc0KRxC79p55LKG2+YuuP9A77Z_k3HXGLWMAUm_Jw@mail.gmail.com>
Message-ID: <CAP1=2W74k=YjKHQPyoo1q_FGKEeF8XT4rVscfscxSErbN=25eg@mail.gmail.com>

On Mon, 23 Jan 2017 at 04:27 INADA Naoki <songofacandy at gmail.com> wrote:

> On Mon, Jan 23, 2017 at 8:33 PM, Victor Stinner
> <victor.stinner at gmail.com> wrote:
> > 2017-01-23 12:25 GMT+01:00 INADA Naoki <songofacandy at gmail.com>:
> >> I gave advice to use 'List[User]' instead of List[User] to the team of
> >> the project,
> >> if the team think RAM usage or boot speed is important.
> >
> > I would prefer a Python option (ex: "-o noannotation" command line
> > option) to opt-out annotations rather than having to write annotations
> > in strings, which is IMHO more "ugly".
>

So basically the equivalent of -OO for docstrings? Maybe this can be the
final motivator for some of us to come up with a design to generalize -O or
something as it keeps coming up.


> >
> > Victor
>
> Personally speaking, I hope annotations are just static hint, and
> makes zero overhead at runtime.
> (startup time, memory consumption, and execution speed).
>

Local variable annotations are nothing but info in the AST. Parameter
annotations and class annotations are stored on their respective objects so
there's memory usage from that and the construction of them at object
creation time, but that's it (e.g. the cost of creating
func.__annotations__ when the function object is created is all you pay for
performance-wise). And using strings will make those introspective
attributes difficult to use, hence why I don't think people have said to do
that everywhere.


>
> Anyway, many users are starting to use typing, for code completion or
> static checking.
> And very few user noticed it affects to performance of `isinstance(x,
> collections.Sequence)`.
> Python 3.7 may be too slow to help them.
> Can't we skip abc registration of typing.List[MyClass] completely?
>
> I'm sorry if it's silly idea.  I don't know about background of
> current typing.py design. And I
> don't use abc much too.


Since isinstance() checks are expected to be rare I don't think anyone has
worried too much about the performance beyond the initial work to introduce
ABCs and __instancecheck__.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170123/0c993da6/attachment.html>

From lukasz at langa.pl  Mon Jan 23 15:37:15 2017
From: lukasz at langa.pl (Lukasz Langa)
Date: Mon, 23 Jan 2017 12:37:15 -0800
Subject: [Python-Dev] Investigating Python memory footprint of one real
 Web application
In-Reply-To: <CAP1=2W74k=YjKHQPyoo1q_FGKEeF8XT4rVscfscxSErbN=25eg@mail.gmail.com>
References: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
 <CAOMjWkkk2RZiZ754D5ggHoPDSp-gJ74itUSJukHN88JMbGJx9A@mail.gmail.com>
 <CAEfz+TxoU0V+FHTWKfudDy+f+W8n7rBM9f-ZGD0zNR704csD3A@mail.gmail.com>
 <CAMpsgwbX7x3sHrc92bULTgJ5R1vsi8fagMU83Wct8OAShBDzmA@mail.gmail.com>
 <CAEfz+Tzz1gc0KRxC79p55LKG2+YuuP9A77Z_k3HXGLWMAUm_Jw@mail.gmail.com>
 <CAP1=2W74k=YjKHQPyoo1q_FGKEeF8XT4rVscfscxSErbN=25eg@mail.gmail.com>
Message-ID: <B48E25F7-2D5B-4E0E-A1CF-3B5C83D81A77@langa.pl>


> On Jan 23, 2017, at 12:10 PM, Brett Cannon <brett at python.org> wrote:
> 
> 
> 
> On Mon, 23 Jan 2017 at 04:27 INADA Naoki <songofacandy at gmail.com <mailto:songofacandy at gmail.com>> wrote:
> On Mon, Jan 23, 2017 at 8:33 PM, Victor Stinner
> <victor.stinner at gmail.com <mailto:victor.stinner at gmail.com>> wrote:
> > 2017-01-23 12:25 GMT+01:00 INADA Naoki <songofacandy at gmail.com <mailto:songofacandy at gmail.com>>:
> >> I gave advice to use 'List[User]' instead of List[User] to the team of
> >> the project,
> >> if the team think RAM usage or boot speed is important.
> >
> > I would prefer a Python option (ex: "-o noannotation" command line
> > option) to opt-out annotations rather than having to write annotations
> > in strings, which is IMHO more "ugly".
> 
> So basically the equivalent of -OO for docstrings? Maybe this can be the final motivator for some of us to come up with a design to generalize -O or something as it keeps coming up.

Yes, please. We've talked about generalizing this for years now. FWIW, I know of projects that run with -OO for the memory wins stemming from docstrings and had to codemod assert statements into a "prod_assert" function call to achieve this. If you think docstrings aren't that much, multiply this by a few hundred processes on a box and it ends up being a substantial win to strip them out.

> >
> > Victor
> 
> Personally speaking, I hope annotations are just static hint, and
> makes zero overhead at runtime.
> (startup time, memory consumption, and execution speed).
> 
> Local variable annotations are nothing but info in the AST. Parameter annotations and class annotations are stored on their respective objects so there's memory usage from that and the construction of them at object creation time, but that's it (e.g. the cost of creating func.__annotations__ when the function object is created is all you pay for performance-wise). And using strings will make those introspective attributes difficult to use, hence why I don't think people have said to do that everywhere.

I suggested making all annotations just strings at runtime and PEP 484 still lists this as a possible course for the future. So far Guido blocked this on a legitimate question: how much do type hints actually cost? Nobody knows yet, the biggest annotated codebase is at Dropbox and this is using comments (so no runtime cost).

>  
> Anyway, many users are starting to use typing, for code completion or
> static checking.
> And very few user noticed it affects to performance of `isinstance(x,
> collections.Sequence)`.
> Python 3.7 may be too slow to help them.
> Can't we skip abc registration of typing.List[MyClass] completely?
> 
> I'm sorry if it's silly idea.  I don't know about background of
> current typing.py design. And I
> don't use abc much too.
> 
> Since isinstance() checks are expected to be rare I don't think anyone has worried too much about the performance beyond the initial work to introduce ABCs and __instancecheck__.

Similar to the above, I would advise against crippling functionality unless we prove this is affecting performance in a significant way.

- ?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170123/9283c330/attachment.html>

From raymond.hettinger at gmail.com  Mon Jan 23 16:12:10 2017
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Mon, 23 Jan 2017 13:12:10 -0800
Subject: [Python-Dev] Hello World
In-Reply-To: <CAJLRRs_rdNmtqiTs_A90bqs5BYrp8iZ+XNSxqROoUjDkC54uVQ@mail.gmail.com>
References: <CAJLRRs_rdNmtqiTs_A90bqs5BYrp8iZ+XNSxqROoUjDkC54uVQ@mail.gmail.com>
Message-ID: <EDFD752B-C600-4496-A3F9-55D75AB0D2D4@gmail.com>


> On Jan 22, 2017, at 6:31 PM, Ramsey D'silva <ramseydsilva at gmail.com> wrote:
> 
> Nice to meet all of you'll. I'm excited to learn and hopefully contribute someday.

Welcome aboard.


Raymond

From ramseydsilva at gmail.com  Mon Jan 23 17:15:15 2017
From: ramseydsilva at gmail.com (Ramsey D'silva)
Date: Mon, 23 Jan 2017 17:15:15 -0500
Subject: [Python-Dev] Hello World
In-Reply-To: <CAP1=2W4iHv99WF+TGuSX7Q9bA2okU9O4iMS5k=W3Firni=Gb=A@mail.gmail.com>
References: <CAJLRRs_rdNmtqiTs_A90bqs5BYrp8iZ+XNSxqROoUjDkC54uVQ@mail.gmail.com>
 <CAP1=2W4iHv99WF+TGuSX7Q9bA2okU9O4iMS5k=W3Firni=Gb=A@mail.gmail.com>
Message-ID: <CAJLRRs9UWrdax=WTb2-Mv+hR2Xh4z8wZRMLvBoSu+6syFLU+aQ@mail.gmail.com>

Thank you Brett. I will be doing as much reading as I can before seeking a
mentor. I have also subscribed to that mailing list.

On Mon, Jan 23, 2017 at 2:59 PM, Brett Cannon <brett at python.org> wrote:

> In case you haven't already done so, Ramsey, please consider reading
> https://cpython-devguide.readthedocs.io if you want to contribute to the
> project. You can also subscribe to the core-mentorship mailing list which
> is specifically for people who want to help out but haven't done to
> previously.
>
> On Sun, 22 Jan 2017 at 18:53 Ramsey D'silva <ramseydsilva at gmail.com>
> wrote:
>
>> Hello World,
>>
>> I am a python developer that has lurked beyond my box.
>>
>> Nice to meet all of you'll. I'm excited to learn and hopefully contribute
>> someday.
>>
>> Best Regards,
>> Ramsey
>>
>> https://linkedin.com/in/ramseydsilva
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
>> brett%40python.org
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170123/daa43c09/attachment.html>

From ramseydsilva at gmail.com  Mon Jan 23 17:15:29 2017
From: ramseydsilva at gmail.com (Ramsey D'silva)
Date: Mon, 23 Jan 2017 17:15:29 -0500
Subject: [Python-Dev] Hello World
In-Reply-To: <EDFD752B-C600-4496-A3F9-55D75AB0D2D4@gmail.com>
References: <CAJLRRs_rdNmtqiTs_A90bqs5BYrp8iZ+XNSxqROoUjDkC54uVQ@mail.gmail.com>
 <EDFD752B-C600-4496-A3F9-55D75AB0D2D4@gmail.com>
Message-ID: <CAJLRRs-Fsjm9AKFYLVKGPdTbJqYbWxHXt8pjdkG9y7XGPa_sTw@mail.gmail.com>

Thank you!

On Mon, Jan 23, 2017 at 4:12 PM, Raymond Hettinger <
raymond.hettinger at gmail.com> wrote:

>
> > On Jan 22, 2017, at 6:31 PM, Ramsey D'silva <ramseydsilva at gmail.com>
> wrote:
> >
> > Nice to meet all of you'll. I'm excited to learn and hopefully
> contribute someday.
>
> Welcome aboard.
>
>
> Raymond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170123/b6d212fa/attachment.html>

From songofacandy at gmail.com  Tue Jan 24 02:02:22 2017
From: songofacandy at gmail.com (INADA Naoki)
Date: Tue, 24 Jan 2017 16:02:22 +0900
Subject: [Python-Dev] Investigating Python memory footprint of one real
 Web application
In-Reply-To: <B48E25F7-2D5B-4E0E-A1CF-3B5C83D81A77@langa.pl>
References: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
 <CAOMjWkkk2RZiZ754D5ggHoPDSp-gJ74itUSJukHN88JMbGJx9A@mail.gmail.com>
 <CAEfz+TxoU0V+FHTWKfudDy+f+W8n7rBM9f-ZGD0zNR704csD3A@mail.gmail.com>
 <CAMpsgwbX7x3sHrc92bULTgJ5R1vsi8fagMU83Wct8OAShBDzmA@mail.gmail.com>
 <CAEfz+Tzz1gc0KRxC79p55LKG2+YuuP9A77Z_k3HXGLWMAUm_Jw@mail.gmail.com>
 <CAP1=2W74k=YjKHQPyoo1q_FGKEeF8XT4rVscfscxSErbN=25eg@mail.gmail.com>
 <B48E25F7-2D5B-4E0E-A1CF-3B5C83D81A77@langa.pl>
Message-ID: <CAEfz+Tx_gxsTAV92gTjj9a8caXOasLu+djwmt7ZtO4kdmANwRA@mail.gmail.com>

>> So basically the equivalent of -OO for docstrings? Maybe this can be the final motivator for some of us to come up with a design to generalize -O or something as it keeps coming up.
> Yes, please. We've talked about generalizing this for years now. FWIW, I know of projects that run with -OO for the memory wins stemming from docstrings and had to codemod assert statements into a "prod_assert" function call to achieve this. If you think docstrings aren't that much, multiply this by a few hundred processes on a box and it ends up being a substantial win to strip them out.

Strong +1.


> So far Guido blocked this on a legitimate question: how much do type hints actually cost? Nobody knows yet,

"Nobody knows yet" is difficult problem.

We may think "let's keep runtime cost, because nobody knows how large it is".
Users may think "let's use string/comment form annotation to avoid runtime cost,
because nobody knows how large it is."

And problem may happen in closed source application.
When building closed source application, the project can drop Python 2
support easily,
and buy PyCharm for all members.

(BTW, PyCharm's survey result [1] is very encouraging.
PyCharm users adopts Python 3 (relative) early.  I think they will
adopt typing early too.)

Early and large adopters of typing may be such teams (like my company).
And they may feel "Python is slow and fat!" if there are no easy way to check
runtime overhead of typing.

Optimize option to drop annotation will provide (1) easy way to check
runtime overhead of typing,
and (2) straightforward solution to remove the overhead, if it isn't negligible.


[1]: https://www.jetbrains.com/pycharm/python-developers-survey-2016/

From thomas at python.org  Tue Jan 24 06:33:27 2017
From: thomas at python.org (Thomas Wouters)
Date: Tue, 24 Jan 2017 12:33:27 +0100
Subject: [Python-Dev] Investigating Python memory footprint of one real
 Web application
In-Reply-To: <1e8a03a3-afb6-fac1-b3ba-e011c4c87eea@python.org>
References: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
 <20170120124333.5f8f5e96@fsol>
 <CAEfz+TwWEjMskjisHvgCBPZ2k7M7LrWA+EaoYH=03WWehNBZZg@mail.gmail.com>
 <1e8a03a3-afb6-fac1-b3ba-e011c4c87eea@python.org>
Message-ID: <CAPdQG2ogLrTqmuahAvccGr9LAqzZAiFYTcVgbyb_VUV_fkgtTw@mail.gmail.com>

On Fri, Jan 20, 2017 at 1:40 PM, Christian Heimes <christian at python.org>
wrote:

> On 2017-01-20 13:15, INADA Naoki wrote:
> >>
> >> "this script counts static memory usage. It doesn?t care about dynamic
> >> memory usage of processing real request"
> >>
> >> You may be trying to optimize something which is only a very small
> >> fraction of your actual memory footprint.  That said, the marshal
> >> module could certainly try to intern some tuples and other immutable
> >> structures.
> >>
> >
> > Yes.  I hadn't think static memory footprint is so important.
> >
> > But Instagram tried to increase CoW efficiency of prefork application,
> > and got some success about memory usage and CPU throughput.
> > I surprised about it because prefork only shares static memory footprint.
> >
> > Maybe, sharing some tuples which code object has may increase cache
> efficiency.
> > I'll try run pyperformance with the marshal patch.
>
> IIRC Thomas Wouters (?) has been working on a patch to move the ref
> counter out of the PyObject struct and into a dedicated memory area. He
> proposed the idea to improve cache affinity, reduce cache evictions and
> to make CoW more efficient. Especially modern ccNUMA machines with
> multiple processors could benefit from the improvement, but also single
> processor/multi core machines.
>

FWIW, I have a working patch for that (against trunk a few months back,
even though the original idea was for the gilectomy branch), moving just
the refcount and not PyGC_HEAD. Performance-wise, in the benchmarks it's a
small but consistent loss (2-5% on a noisy machine, as measured by
python-benchmarks, not perf), and it breaks the ABI as well as any code
that dereferences PyObject.ob_refcnt directly (the field was repurposed and
renamed, and exposed as a const* to avoid direct assignment). It also
exposes the API awkwardness that CPython doesn't *require* objects to go
through a specific mechanism for object initialisation, even though nearly
all extension modules do so. (That same API awkwardness made life a little
harder when experimenting with BDW GC :P.) I don't believe external
refcounts can be made the default without careful redesigning of a new set
of PyObject API calls and deprecation of the old ones.

-- 
Thomas Wouters <thomas at python.org>

Hi! I'm an email virus! Think twice before sending your email to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170124/cf8bd0f3/attachment.html>

From levkivskyi at gmail.com  Tue Jan 24 07:07:29 2017
From: levkivskyi at gmail.com (Ivan Levkivskyi)
Date: Tue, 24 Jan 2017 13:07:29 +0100
Subject: [Python-Dev] Investigating Python memory footprint of one real
 Web application
In-Reply-To: <CAPdQG2ogLrTqmuahAvccGr9LAqzZAiFYTcVgbyb_VUV_fkgtTw@mail.gmail.com>
References: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
 <20170120124333.5f8f5e96@fsol>
 <CAEfz+TwWEjMskjisHvgCBPZ2k7M7LrWA+EaoYH=03WWehNBZZg@mail.gmail.com>
 <1e8a03a3-afb6-fac1-b3ba-e011c4c87eea@python.org>
 <CAPdQG2ogLrTqmuahAvccGr9LAqzZAiFYTcVgbyb_VUV_fkgtTw@mail.gmail.com>
Message-ID: <CAOMjWknkm8ntOvgG3_eTFhN031O_4bH4-qdf1aUSnfoRjdxd9Q@mail.gmail.com>

1. It looks like there is still a room for performance improvement of
typing w.r.t. how ABCs and issubclass() works.
I will try to play with this soon. (the basic idea is that some steps could
be avoided for parameterized generics).

2. I am +1 on having three separate options to independently ignore
asserts, docstrings, and annotations.

3. I am -1 on ignoring annotations altogether. Sometimes they could be
helpful at runtime: typing.NamedTuple and mypy_extensions.TypedDict are two
examples.
Also some people use annotations for runtime checks or even for things
unrelated to typing. I think it would be a pity to lose these
functionalities for small performance gains.

--
Ivan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170124/3796a7b3/attachment.html>

From songofacandy at gmail.com  Tue Jan 24 07:23:50 2017
From: songofacandy at gmail.com (INADA Naoki)
Date: Tue, 24 Jan 2017 21:23:50 +0900
Subject: [Python-Dev] Investigating Python memory footprint of one real
 Web application
In-Reply-To: <CAOMjWknkm8ntOvgG3_eTFhN031O_4bH4-qdf1aUSnfoRjdxd9Q@mail.gmail.com>
References: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
 <20170120124333.5f8f5e96@fsol>
 <CAEfz+TwWEjMskjisHvgCBPZ2k7M7LrWA+EaoYH=03WWehNBZZg@mail.gmail.com>
 <1e8a03a3-afb6-fac1-b3ba-e011c4c87eea@python.org>
 <CAPdQG2ogLrTqmuahAvccGr9LAqzZAiFYTcVgbyb_VUV_fkgtTw@mail.gmail.com>
 <CAOMjWknkm8ntOvgG3_eTFhN031O_4bH4-qdf1aUSnfoRjdxd9Q@mail.gmail.com>
Message-ID: <CAEfz+TwnxJq9TT2Ji18KJt6-3dccQm0fxBdfCGk9S-jyA9kYUw@mail.gmail.com>

> 3. I am -1 on ignoring annotations altogether. Sometimes they could be
> helpful at runtime: typing.NamedTuple and mypy_extensions.TypedDict are two
> examples.

ignoring annotations doesn't mean ignoring typing at all.
You can use typing.NamedTuple even when functions doesn't have __annotations__.

> Also some people use annotations for runtime checks or even for things
> unrelated to typing. I think it would be a pity to lose these
> functionalities for small performance gains.
>

Sure.  It should be option, for backward compatibility.


Regards,

From songofacandy at gmail.com  Tue Jan 24 09:00:17 2017
From: songofacandy at gmail.com (INADA Naoki)
Date: Tue, 24 Jan 2017 23:00:17 +0900
Subject: [Python-Dev] Investigating Python memory footprint of one real
 Web application
In-Reply-To: <CAEfz+TwnxJq9TT2Ji18KJt6-3dccQm0fxBdfCGk9S-jyA9kYUw@mail.gmail.com>
References: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
 <20170120124333.5f8f5e96@fsol>
 <CAEfz+TwWEjMskjisHvgCBPZ2k7M7LrWA+EaoYH=03WWehNBZZg@mail.gmail.com>
 <1e8a03a3-afb6-fac1-b3ba-e011c4c87eea@python.org>
 <CAPdQG2ogLrTqmuahAvccGr9LAqzZAiFYTcVgbyb_VUV_fkgtTw@mail.gmail.com>
 <CAOMjWknkm8ntOvgG3_eTFhN031O_4bH4-qdf1aUSnfoRjdxd9Q@mail.gmail.com>
 <CAEfz+TwnxJq9TT2Ji18KJt6-3dccQm0fxBdfCGk9S-jyA9kYUw@mail.gmail.com>
Message-ID: <CAEfz+TyGE0YoQPwZg5wjMrxWmTQqLSnomGgGu7id8SwGekdtPQ@mail.gmail.com>

FWIW, I tried to skip compiler_visit_annotations() in Python/compile.c

a) default:                   41278060
b) remove annotations: 37140094
c) (b) + const merge: 35933436

(a-b)/a = 10%
(a-c)/a = 13%

And here are top 3 tracebacks from tracemalloc:

15109615 (/180598)
  File "<frozen importlib._bootstrap_external>", line 488
  File "<frozen importlib._bootstrap_external>", line 780
  File "<frozen importlib._bootstrap_external>", line 675
  File "<frozen importlib._bootstrap>", line 655

1255632 (/8316)
  File "/home/inada-n/local/cpython/lib/python3.7/_weakrefset.py", line 84
    self.data.add(ref(item, self._remove))
  File "/home/inada-n/local/cpython/lib/python3.7/abc.py", line 230
    cls._abc_negative_cache.add(subclass)
  File "/home/inada-n/local/cpython/lib/python3.7/abc.py", line 226
    if issubclass(subclass, scls):
  File "/home/inada-n/local/cpython/lib/python3.7/abc.py", line 226
    if issubclass(subclass, scls):

1056744 (/4020)
  File "/home/inada-n/local/cpython/lib/python3.7/abc.py", line 133
    cls = super().__new__(mcls, name, bases, namespace)
  File "/home/inada-n/local/cpython/lib/python3.7/typing.py", line 125
    return super().__new__(cls, name, bases, namespace)
  File "/home/inada-n/local/cpython/lib/python3.7/typing.py", line 977
    self = super().__new__(cls, name, bases, namespace, _root=True)
  File "/home/inada-n/local/cpython/lib/python3.7/typing.py", line 1105
    orig_bases=self.__orig_bases__)

Regards,

From victor.stinner at gmail.com  Tue Jan 24 09:08:03 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 24 Jan 2017 15:08:03 +0100
Subject: [Python-Dev] Investigating Python memory footprint of one real
 Web application
In-Reply-To: <CAEfz+TyGE0YoQPwZg5wjMrxWmTQqLSnomGgGu7id8SwGekdtPQ@mail.gmail.com>
References: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
 <20170120124333.5f8f5e96@fsol>
 <CAEfz+TwWEjMskjisHvgCBPZ2k7M7LrWA+EaoYH=03WWehNBZZg@mail.gmail.com>
 <1e8a03a3-afb6-fac1-b3ba-e011c4c87eea@python.org>
 <CAPdQG2ogLrTqmuahAvccGr9LAqzZAiFYTcVgbyb_VUV_fkgtTw@mail.gmail.com>
 <CAOMjWknkm8ntOvgG3_eTFhN031O_4bH4-qdf1aUSnfoRjdxd9Q@mail.gmail.com>
 <CAEfz+TwnxJq9TT2Ji18KJt6-3dccQm0fxBdfCGk9S-jyA9kYUw@mail.gmail.com>
 <CAEfz+TyGE0YoQPwZg5wjMrxWmTQqLSnomGgGu7id8SwGekdtPQ@mail.gmail.com>
Message-ID: <CAMpsgwbtbgSDNio1NVRT0zxLoU2dYnf12XxfqzMLztmnp9pokA@mail.gmail.com>

2017-01-24 15:00 GMT+01:00 INADA Naoki <songofacandy at gmail.com>:
> And here are top 3 tracebacks from tracemalloc:
>
> 15109615 (/180598)
>   File "<frozen importlib._bootstrap_external>", line 488
>   File "<frozen importlib._bootstrap_external>", line 780
>   File "<frozen importlib._bootstrap_external>", line 675
>   File "<frozen importlib._bootstrap>", line 655

FYI at Python startup, usually the largest memory block comes from the
dictionary used to intern all strings ("interned" in unicodeobject.c).
The traceback is never revelant for this specific object.

Victor

From songofacandy at gmail.com  Tue Jan 24 10:33:53 2017
From: songofacandy at gmail.com (INADA Naoki)
Date: Wed, 25 Jan 2017 00:33:53 +0900
Subject: [Python-Dev] Investigating Python memory footprint of one real
 Web application
In-Reply-To: <CAMpsgwbtbgSDNio1NVRT0zxLoU2dYnf12XxfqzMLztmnp9pokA@mail.gmail.com>
References: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
 <20170120124333.5f8f5e96@fsol>
 <CAEfz+TwWEjMskjisHvgCBPZ2k7M7LrWA+EaoYH=03WWehNBZZg@mail.gmail.com>
 <1e8a03a3-afb6-fac1-b3ba-e011c4c87eea@python.org>
 <CAPdQG2ogLrTqmuahAvccGr9LAqzZAiFYTcVgbyb_VUV_fkgtTw@mail.gmail.com>
 <CAOMjWknkm8ntOvgG3_eTFhN031O_4bH4-qdf1aUSnfoRjdxd9Q@mail.gmail.com>
 <CAEfz+TwnxJq9TT2Ji18KJt6-3dccQm0fxBdfCGk9S-jyA9kYUw@mail.gmail.com>
 <CAEfz+TyGE0YoQPwZg5wjMrxWmTQqLSnomGgGu7id8SwGekdtPQ@mail.gmail.com>
 <CAMpsgwbtbgSDNio1NVRT0zxLoU2dYnf12XxfqzMLztmnp9pokA@mail.gmail.com>
Message-ID: <CAEfz+TzHfAFHLz0RJn58oz_tr0bNKXtnh8ZoyrtfsyO6qWr0Kg@mail.gmail.com>

On Tue, Jan 24, 2017 at 11:08 PM, Victor Stinner
<victor.stinner at gmail.com> wrote:
> 2017-01-24 15:00 GMT+01:00 INADA Naoki <songofacandy at gmail.com>:
>> And here are top 3 tracebacks from tracemalloc:
>>
>> 15109615 (/180598)
>>   File "<frozen importlib._bootstrap_external>", line 488
>>   File "<frozen importlib._bootstrap_external>", line 780
>>   File "<frozen importlib._bootstrap_external>", line 675
>>   File "<frozen importlib._bootstrap>", line 655
>
> FYI at Python startup, usually the largest memory block comes from the
> dictionary used to intern all strings ("interned" in unicodeobject.c).
> The traceback is never revelant for this specific object.
>
> Victor

Yes! I used a few hours to notice it.

When PYTHONTRACEMALLOC=10, marshal.loads() of small module (15KB pyc)
looks eating 1.3MB.
I think small stacktrace depth (3~4) is better for showing summary of
large application.

BTW, about 1.3MB of 15MB (mashal.loads()) was for intern dict, as far
as I remember.

From njs at pobox.com  Tue Jan 24 13:21:45 2017
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 24 Jan 2017 10:21:45 -0800
Subject: [Python-Dev] Investigating Python memory footprint of one real
 Web application
In-Reply-To: <CAPdQG2ogLrTqmuahAvccGr9LAqzZAiFYTcVgbyb_VUV_fkgtTw@mail.gmail.com>
References: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
 <20170120124333.5f8f5e96@fsol>
 <CAEfz+TwWEjMskjisHvgCBPZ2k7M7LrWA+EaoYH=03WWehNBZZg@mail.gmail.com>
 <1e8a03a3-afb6-fac1-b3ba-e011c4c87eea@python.org>
 <CAPdQG2ogLrTqmuahAvccGr9LAqzZAiFYTcVgbyb_VUV_fkgtTw@mail.gmail.com>
Message-ID: <CAPJVwBm15RsLUHh=EZhh9y1LNkX1O2UZU2CqD8v-q8UD38+wpQ@mail.gmail.com>

On Jan 24, 2017 3:35 AM, "Thomas Wouters" <thomas at python.org> wrote:



On Fri, Jan 20, 2017 at 1:40 PM, Christian Heimes <christian at python.org>
wrote:

> On 2017-01-20 13:15, INADA Naoki wrote:
> >>
> >> "this script counts static memory usage. It doesn?t care about dynamic
> >> memory usage of processing real request"
> >>
> >> You may be trying to optimize something which is only a very small
> >> fraction of your actual memory footprint.  That said, the marshal
> >> module could certainly try to intern some tuples and other immutable
> >> structures.
> >>
> >
> > Yes.  I hadn't think static memory footprint is so important.
> >
> > But Instagram tried to increase CoW efficiency of prefork application,
> > and got some success about memory usage and CPU throughput.
> > I surprised about it because prefork only shares static memory footprint.
> >
> > Maybe, sharing some tuples which code object has may increase cache
> efficiency.
> > I'll try run pyperformance with the marshal patch.
>
> IIRC Thomas Wouters (?) has been working on a patch to move the ref
> counter out of the PyObject struct and into a dedicated memory area. He
> proposed the idea to improve cache affinity, reduce cache evictions and
> to make CoW more efficient. Especially modern ccNUMA machines with
> multiple processors could benefit from the improvement, but also single
> processor/multi core machines.
>

FWIW, I have a working patch for that (against trunk a few months back,
even though the original idea was for the gilectomy branch), moving just
the refcount and not PyGC_HEAD. Performance-wise, in the benchmarks it's a
small but consistent loss (2-5% on a noisy machine, as measured by
python-benchmarks, not perf), and it breaks the ABI as well as any code
that dereferences PyObject.ob_refcnt directly (the field was repurposed and
renamed, and exposed as a const* to avoid direct assignment). It also
exposes the API awkwardness that CPython doesn't *require* objects to go
through a specific mechanism for object initialisation, even though nearly
all extension modules do so. (That same API awkwardness made life a little
harder when experimenting with BDW GC :P.) I don't believe external
refcounts can be made the default without careful redesigning of a new set
of PyObject API calls and deprecation of the old ones.


The thing I found most surprising about that blog post was that contrary to
common wisdom, refcnt updates per se had essentially no effect on the
amount of memory shared between CoW processes, and the problems were all
due to the cycle collector. (Though I guess it's still possible that part
of the problems caused by the cycle collector are due to it touching
ob_refcnt.)

It's promising too though, because the GC metadata is much less exposed to
extension modules than PyObject_HEAD is, and the access patterns are
presumably (?) much more bursty. It'd be really interesting to see how
things performed if packing just PyGC_HEAD but *not* ob_refcnt into a
dedicated region.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170124/0c0baf57/attachment.html>

From solipsis at pitrou.net  Tue Jan 24 13:51:02 2017
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 24 Jan 2017 19:51:02 +0100
Subject: [Python-Dev] Investigating Python memory footprint of one real
 Web application
In-Reply-To: <CAPJVwBm15RsLUHh=EZhh9y1LNkX1O2UZU2CqD8v-q8UD38+wpQ@mail.gmail.com>
References: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
 <20170120124333.5f8f5e96@fsol>
 <CAEfz+TwWEjMskjisHvgCBPZ2k7M7LrWA+EaoYH=03WWehNBZZg@mail.gmail.com>
 <1e8a03a3-afb6-fac1-b3ba-e011c4c87eea@python.org>
 <CAPdQG2ogLrTqmuahAvccGr9LAqzZAiFYTcVgbyb_VUV_fkgtTw@mail.gmail.com>
 <CAPJVwBm15RsLUHh=EZhh9y1LNkX1O2UZU2CqD8v-q8UD38+wpQ@mail.gmail.com>
Message-ID: <20170124195102.510cb06e@fsol>

On Tue, 24 Jan 2017 10:21:45 -0800
Nathaniel Smith <njs at pobox.com> wrote:
> 
> The thing I found most surprising about that blog post was that contrary to
> common wisdom, refcnt updates per se had essentially no effect on the
> amount of memory shared between CoW processes, and the problems were all
> due to the cycle collector.

Indeed, it was unexpected, though it can be explained easily: refcount
updates touch only the live working set, while GC passes scan through
all existing objects, even those that are never actually used.

Regards

Antoine.

From rodrigc at freebsd.org  Wed Jan 25 00:38:44 2017
From: rodrigc at freebsd.org (Craig Rodrigues)
Date: Tue, 24 Jan 2017 21:38:44 -0800
Subject: [Python-Dev] Generator objects and list comprehensions?
Message-ID: <CAG=rPVco0cnKdA0qDbhnZUctzeVVuJTgN4YX7ua6x8Lb-Q-TUQ@mail.gmail.com>

Hi,

Glyph pointed this out to me here:
http://twistedmatrix.com/pipermail/twisted-python/2017-January/031106.html

If I do this on Python 3.6:

>>  [(yield 1) for x in range(10)]
<generator object <listcomp> at 0x10cd210f8>

If I understand this:
https://docs.python.org/3/reference/expressions.html#list-displays
then this is a list display and should give a list, not a generator object.
Is there a bug in Python, or does the documentation need to be updated?

--
Craig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170124/117a2eb4/attachment.html>

From rosuav at gmail.com  Wed Jan 25 01:01:18 2017
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 25 Jan 2017 17:01:18 +1100
Subject: [Python-Dev] Generator objects and list comprehensions?
In-Reply-To: <CAG=rPVco0cnKdA0qDbhnZUctzeVVuJTgN4YX7ua6x8Lb-Q-TUQ@mail.gmail.com>
References: <CAG=rPVco0cnKdA0qDbhnZUctzeVVuJTgN4YX7ua6x8Lb-Q-TUQ@mail.gmail.com>
Message-ID: <CAPTjJmrf+tAsJOAsiK=VZ2b-Qfzqi3z3f=+3LG8NrZaF9=4Tig@mail.gmail.com>

On Wed, Jan 25, 2017 at 4:38 PM, Craig Rodrigues <rodrigc at freebsd.org> wrote:
>
> Glyph pointed this out to me here:
> http://twistedmatrix.com/pipermail/twisted-python/2017-January/031106.html
>
> If I do this on Python 3.6:
>
>>>  [(yield 1) for x in range(10)]
> <generator object <listcomp> at 0x10cd210f8>
>
> If I understand this:
> https://docs.python.org/3/reference/expressions.html#list-displays
> then this is a list display and should give a list, not a generator object.
> Is there a bug in Python, or does the documentation need to be updated?

That looks like an odd interaction between yield expressions and list
comps. Since a list comprehension is actually implemented as a nested
function, your code actually looks more-or-less like this:

>>> def <listcomp>(iter):
    result = []
    for x in iter:
        result.append((yield 1))
    return result
>>> <listcomp>(iter(range(10))
<generator object <listcomp> at 0x10cd210f8>

This function is a generator, and calling it returns what you see
above. If you step that iterator, it'll yield 1 ten times, and then
raise StopIteration with the resulting list.

Based on a cursory examination of the issue at hand, I think what
you're working with might be functioning as a coroutine? If so, you
may find that using "await" instead of "yield" dodges the problem, as
it won't turn the list comp into a generator. But I can't be 100%
certain of that. (Also, that would definitely stop you from having
single-codebase 2.7/3.x code.)

ChrisA

From levkivskyi at gmail.com  Wed Jan 25 03:14:27 2017
From: levkivskyi at gmail.com (Ivan Levkivskyi)
Date: Wed, 25 Jan 2017 09:14:27 +0100
Subject: [Python-Dev] Generator objects and list comprehensions?
In-Reply-To: <CAPTjJmrf+tAsJOAsiK=VZ2b-Qfzqi3z3f=+3LG8NrZaF9=4Tig@mail.gmail.com>
References: <CAG=rPVco0cnKdA0qDbhnZUctzeVVuJTgN4YX7ua6x8Lb-Q-TUQ@mail.gmail.com>
 <CAPTjJmrf+tAsJOAsiK=VZ2b-Qfzqi3z3f=+3LG8NrZaF9=4Tig@mail.gmail.com>
Message-ID: <CAOMjWkk76oTqfebya4D_9EmkS+xZHWSBAE1NmgwZyP+j2LEjig@mail.gmail.com>

On 25 January 2017 at 07:01, Chris Angelico <rosuav at gmail.com> wrote:

> >>>  [(yield 1) for x in range(10)]
> > <generator object <listcomp> at 0x10cd210f8>
>

This is an old bug, see e.g. http://bugs.python.org/issue10544
The ``yield`` inside comprehensions is bound to the auxiliary function.
Instead it should be bound to an enclosing function, like it is done for
``await``.
The behaviour of ``await`` in comprehensions is intuitive
(since it is simply equivalent to a for-loop):

>>> async def f(i):
...     return i

>>> async def g_for():
...     lst = []
...     for i in range(5):
...         lst.append(await f(i))
...     print(lst)

>>> g_for().send(None)
[0, 1, 2, 3, 4]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

>>> async def g_comp():
...     print([await f(i) for i in range(5)])

>>> g_comp().send(None)    # exactly the same as g_for
[0, 1, 2, 3, 4]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

While current behaviour of ``yield`` in comprehensions is confusing:

>>> def c_for():
...     lst = []
...     for i in range(5):
...         lst.append((yield i))
...     print(lst)

>>> c_for().send(None)
0
>>> c_for().send(None)
1
# etc.

>>> def c_comp():
...     print([(yield i) for i in range(5)])

>>> c_comp().send(None)  # Naively this should be equivalent to the above,
but...
<generator object c_comp.<locals>.<listcomp> at 0x7f1fd1faa630>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'send'

I promised myself to write a patch, but didn't have time for this yet.
I hope I will do this at some point soon.

--
Ivan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170125/a423f1c6/attachment.html>

From songofacandy at gmail.com  Wed Jan 25 06:54:02 2017
From: songofacandy at gmail.com (INADA Naoki)
Date: Wed, 25 Jan 2017 20:54:02 +0900
Subject: [Python-Dev] Investigating Python memory footprint of one real
 Web application
In-Reply-To: <CAEfz+TzHfAFHLz0RJn58oz_tr0bNKXtnh8ZoyrtfsyO6qWr0Kg@mail.gmail.com>
References: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
 <20170120124333.5f8f5e96@fsol>
 <CAEfz+TwWEjMskjisHvgCBPZ2k7M7LrWA+EaoYH=03WWehNBZZg@mail.gmail.com>
 <1e8a03a3-afb6-fac1-b3ba-e011c4c87eea@python.org>
 <CAPdQG2ogLrTqmuahAvccGr9LAqzZAiFYTcVgbyb_VUV_fkgtTw@mail.gmail.com>
 <CAOMjWknkm8ntOvgG3_eTFhN031O_4bH4-qdf1aUSnfoRjdxd9Q@mail.gmail.com>
 <CAEfz+TwnxJq9TT2Ji18KJt6-3dccQm0fxBdfCGk9S-jyA9kYUw@mail.gmail.com>
 <CAEfz+TyGE0YoQPwZg5wjMrxWmTQqLSnomGgGu7id8SwGekdtPQ@mail.gmail.com>
 <CAMpsgwbtbgSDNio1NVRT0zxLoU2dYnf12XxfqzMLztmnp9pokA@mail.gmail.com>
 <CAEfz+TzHfAFHLz0RJn58oz_tr0bNKXtnh8ZoyrtfsyO6qWr0Kg@mail.gmail.com>
Message-ID: <CAEfz+TzDAJp9YdBa3dMyECzbr5RzyE=iQFip5jJA70tc7mJXCg@mail.gmail.com>

More detailed information:

## With annotations

=== tracemalloc stat ===
traced: (46969277, 46983753)
18,048,888 / 181112
  File "<frozen importlib._bootstrap_external>", line 488
  File "<frozen importlib._bootstrap_external>", line 780
  File "<frozen importlib._bootstrap_external>", line 675

=== size by types ===
dict                     9,083,816 (8,870.91KB) / 21846 = 415.811bytes (21.38%)
tuple                    6,420,960 (6,270.47KB) / 86781 = 73.990bytes (15.11%)
str                      6,050,213 (5,908.41KB) / 77527 = 78.040bytes (14.24%)
function                 2,772,224 (2,707.25KB) / 20384 = 136.000bytes (6.53%)
code                     2,744,888 (2,680.55KB) / 18987 = 144.567bytes (6.46%)
type                     2,713,552 (2,649.95KB) / 2769 = 979.975bytes (6.39%)
bytes                    2,650,838 (2,588.71KB) / 38723 = 68.456bytes (6.24%)
set                      2,445,280 (2,387.97KB) / 6969 = 350.880bytes (5.76%)
weakref                  1,255,600 (1,226.17KB) / 15695 = 80.000bytes (2.96%)
list                       707,336 (690.76KB) / 6628 = 106.719bytes (1.66%)

=== dict stat ===
t,         size,        total (%) / count
3,          256,    1,479,424 (15.68%) / 5779.0
3,        1,200,    1,330,800 (14.11%) / 1109.0
3,    1,310,832,    1,310,832 (13.90%) / 1.0
3,          664,    1,287,496 (13.65%) / 1939.0
7,          128,      756,352 (8.02%) / 5909.0
3,          384,      707,328 (7.50%) / 1842.0
3,        2,296,      642,880 (6.81%) / 280.0
0,          256,      378,112 (4.01%) / 1477.0
7,          168,      251,832 (2.67%) / 1499.0
3,        4,720,      221,840 (2.35%) / 47.0
3,        9,336,      130,704 (1.39%) / 14.0
7,           88,      105,072 (1.11%) / 1194.0

* t=7 key-sharing dict, t=3 interned string key only, t=1 string key
only, t=0 non string key is used


## Stripped annotations

=== tracemalloc stat ===
traced: (42383739, 42397983)

18,069,806 / 181346
  File "<frozen importlib._bootstrap_external>", line 488
  File "<frozen importlib._bootstrap_external>", line 780
  File "<frozen importlib._bootstrap_external>", line 675

=== size by types ===
dict                     7,913,144 (7,727.68KB) / 17598 = 449.662bytes (20.62%)
tuple                    6,149,120 (6,005.00KB) / 82734 = 74.324bytes (16.02%)
str                      6,070,083 (5,927.82KB) / 77741 = 78.081bytes (15.82%)
code                     2,744,312 (2,679.99KB) / 18983 = 144.567bytes (7.15%)
type                     2,713,552 (2,649.95KB) / 2769 = 979.975bytes (7.07%)
bytes                    2,650,464 (2,588.34KB) / 38715 = 68.461bytes (6.91%)
function                 2,547,280 (2,487.58KB) / 18730 = 136.000bytes (6.64%)
set                      1,423,520 (1,390.16KB) / 4627 = 307.655bytes (3.71%)
list                       634,472 (619.60KB) / 5454 = 116.331bytes (1.65%)
int                        608,784 (594.52KB) / 21021 = 28.961bytes (1.59%)

=== dict stat ===
t,         size,        total (%) / count
3,        1,200,    1,316,400 (16.06%) / 1097.0
3,    1,310,832,    1,310,832 (16.00%) / 1.0
3,          664,      942,216 (11.50%) / 1419.0
3,          256,      861,184 (10.51%) / 3364.0
3,          384,      657,024 (8.02%) / 1711.0
3,        2,296,      640,584 (7.82%) / 279.0
7,          128,      606,464 (7.40%) / 4738.0
0,          256,      379,904 (4.64%) / 1484.0
7,          168,      251,832 (3.07%) / 1499.0
3,        4,720,      221,840 (2.71%) / 47.0
3,        9,336,      130,704 (1.59%) / 14.0
7,           88,      105,248 (1.28%) / 1196.0
7,          256,       86,784 (1.06%) / 339.0


## Stripped annotation + without pydebug

=== tracemalloc stat ===
traced: (37371660, 40814265)
9,812,083 / 111082
  File "<frozen importlib._bootstrap>", line 205
  File "<frozen importlib._bootstrap_external>", line 742
  File "<frozen importlib._bootstrap_external>", line 782
6,761,207 / 85614
  File "<frozen importlib._bootstrap_external>", line 488
  File "<frozen importlib._bootstrap_external>", line 780
  File "<frozen importlib._bootstrap_external>", line 675


## Ideas about memory optimize

a) Split PyGC_Head from object

Reduces 2words (16byte) from each tuples.

>>> 82734 * 16 / 1024
1292.71875

So estimated -1.2MB


b) concat co_consts, co_names, co_varnames, co_freevars into one
tuple, or embed them into code.

Each tuple has 3 (gc head) + 3 (refcnt, *type, length) = 6 words
overhead. (or 4 words if (a) is applied)
If we can reduce 3 tuples, 18 words = 144byte (or 12 words=96byte) can
be reuduced.

>>> 18983 * 144
2733552
>>> 18983 * 96
1822368

But co_freevars is empty tuple in most cases.  So real effect is
smaller than 2.7MB.
If we can embed them into code object, we can estimate -2.7MB.

(There are co_cellvars too. But I don't know about it much, especially
it is GC tracked or not)


c) (interned) string key only dict.

20% of memory is used for dict, and 70% of dict has interned string keys.
Current DictKeyEntry is 3 words: {key, hash, value}.
But if we can assume all key is string, hash can be get from the key.

If we use 2 words entry: {key, value} for such dict, I think dict can
be 25% smaller.

>>> 7913144 * 0.25 / 1024
1931.919921875

So I estimate -1.9MB

If we can implement (a)~(c) I estimate memory usage on Python
(--without-pydebug)
can be reduced from 35.6MB to 30MB, roughly.

But I think -Onoannotation option is top priority.  It can reduce 4MB,
even we use
annotations only in our code.
If major libraries start using annotations, this number will be larger.

From jjevnik at quantopian.com  Wed Jan 25 01:28:26 2017
From: jjevnik at quantopian.com (Joe Jevnik)
Date: Wed, 25 Jan 2017 01:28:26 -0500
Subject: [Python-Dev] Generator objects and list comprehensions?
In-Reply-To: <CAG=rPVco0cnKdA0qDbhnZUctzeVVuJTgN4YX7ua6x8Lb-Q-TUQ@mail.gmail.com>
References: <CAG=rPVco0cnKdA0qDbhnZUctzeVVuJTgN4YX7ua6x8Lb-Q-TUQ@mail.gmail.com>
Message-ID: <CAFo6DPaP5dcDy=sm4QrauQJDfXgZQYLg0hFdNAwrQ=qM1OwkOw@mail.gmail.com>

List, set, and dict comprehensions compile like:

# input
result = [expression for lhs in iterator_expression]

# output
def comprehension(iterator):
    out = []
    for lhs in iterator:
        out.append(expression)
    return out

result = comprehension(iter(iterator_expression))

When you make `expression` a `yield` the compiler thinks that
`comprehension` is a generator function instead of a normal function.

We can manually translate the following comprehension:

result = [(yield n) for n in (0, 1)]

def comprehension(iterator):
    out = []
    for n in iterator:
        # (yield n) as an expression is the value sent into this generator
        out.append((yield n))
    return out

result = comprehension(iter((0, 1)))


We can see this in the behavior of `send` on the resulting generator:

In [1]: g = [(yield n) for n in (0, 1)]

In [2]: next(g)
Out[2]: 0

In [3]: g.send('hello')
Out[3]: 1

In [4]: g.send('world')
---------------------------------------------------------------------------
StopIteration                             Traceback (most recent call last)
<ipython-input-4-82c589e0c27d> in <module>()
----> 1 g.send('world')

StopIteration: ['hello', 'world']

The `return out` gets translated into `raise StopIteration(out)` because
the code is a generator.

The bytecode for this looks like:

In [5]: %%dis
   ...: [(yield n) for n in (0, 1)]
   ...:
<module>
--------
  1           0 LOAD_CONST               0 (<code object <listcomp> at
0x7f4bae68eed0, file "<show>", line 1>)
              3 LOAD_CONST               1 ('<listcomp>')
              6 MAKE_FUNCTION            0
              9 LOAD_CONST               5 ((0, 1))
             12 GET_ITER
             13 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
             16 POP_TOP
             17 LOAD_CONST               4 (None)
             20 RETURN_VALUE

<module>.<listcomp>
-------------------
  1           0 BUILD_LIST               0
              3 LOAD_FAST                0 (.0)
        >>    6 FOR_ITER                13 (to 22)
              9 STORE_FAST               1 (n)
             12 LOAD_FAST                1 (n)
             15 YIELD_VALUE
             16 LIST_APPEND              2
             19 JUMP_ABSOLUTE            6
        >>   22 RETURN_VALUE


In `<module>` you can see us create the <listcomp> function, call `iter` on
`(0, 1)`, and then call `<listcomp>(iter(0, 1))`. In `<listcomp>` you can
see the loop like we had above, but unlike a normal comprehension we have a
`YIELD_VALUE` (from the `(yield n)`) in the comprehension.

The reason that this is different in Python 2 is that list comprehension,
and only list comprehensions, compile inline. Instead of creating a new
function for the loop, it is done in the scope of the comprehension. For
example:

# input
result = [expression for lhs in iterator_expression]

# output
result = []
for lhs in iterator_expression:
    result.append(lhs)

This is why `lhs` bleeds out of the comprehension. In Python 2, adding
making `expression` a `yield` causes the _calling_ function to become a
generator:

def f():
    [(yield n) for n in range(3)]
    # no return

# py2
>>> type(f())
generator

# py3
>> type(f())
NoneType

In Python 2 the following will even raise a syntax error:

In [5]: def f():
   ...:     return [(yield n) for n in range(3)]
   ...:
   ...:
  File "<ipython-input-5-3602a9999f46>", line 2
    return [(yield n) for n in range(3)]
SyntaxError: 'return' with argument inside generator


Generator expressions are a little different because the compilation
already includes a `yield`.

# input
(expression for lhs in iterator_expression)

# output
def genexpr(iterator):
    for lhs in iterator:
        yield expression

You can actually abuse this to write a cute `flatten` function:

`flatten = lambda seq: (None for sub in seq if (yield from sub) and False)`

because it translates to:

def genexpr(seq):
    for sub in seq:
        # (yield from sub) as an expression returns the last sent in value
        if (yield from sub) and False:
            # we never enter this branch
            yield None


That was a long way to explain what the problem was. I think that that
solution is to stop using `yield` in comprehensions because it is
confusing, or to make `yield` in a comprehension a syntax error.

On Wed, Jan 25, 2017 at 12:38 AM, Craig Rodrigues <rodrigc at freebsd.org>
wrote:

> Hi,
>
> Glyph pointed this out to me here: http://twistedmatrix.
> com/pipermail/twisted-python/2017-January/031106.html
>
> If I do this on Python 3.6:
>
> >>  [(yield 1) for x in range(10)]
> <generator object <listcomp> at 0x10cd210f8>
>
> If I understand this: https://docs.python.org/
> 3/reference/expressions.html#list-displays
> then this is a list display and should give a list, not a generator object.
> Is there a bug in Python, or does the documentation need to be updated?
>
> --
> Craig
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> joe%40quantopian.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170125/ab9478f6/attachment-0001.html>

From cstratak at redhat.com  Wed Jan 25 12:09:43 2017
From: cstratak at redhat.com (Charalampos Stratakis)
Date: Wed, 25 Jan 2017 12:09:43 -0500 (EST)
Subject: [Python-Dev] Have problem when building python3.5.1 rpm with
 default SPEC file
In-Reply-To: <e2189a8897494c4ab0a9903a7aa30d12@vrtsxchclupin16.community.veritas.com>
References: <fcf13f7ce88a46599c11579856be56a6@vrtsxchclupin16.community.veritas.com>
 <1459225640.92062396.1484851212581.JavaMail.zimbra@redhat.com>
 <e2189a8897494c4ab0a9903a7aa30d12@vrtsxchclupin16.community.veritas.com>
Message-ID: <1566645094.94657335.1485364183867.JavaMail.zimbra@redhat.com>

Hello Dahui,

So I'll wear my downstream maintainer hat and try to elaborate on the situation that you are facing.

Currently Red Hat does not ship Python 3 with rhel 7, so it is not supported. However package maintainers have created a SPEC file for python 3 that works for EPEL7 (aka centos7 or rhel7 etc).

The name of the package is python34 [1] and you can install it by following the instructions at [0], it is at version 3.4.5

Currently there is no package for Python 3.5 in the EPEL repositories, you could request that at epel-devel mailing list [2], which is for these kind of queries, someone could be able to provide you with more info there.

[0] https://admin.fedoraproject.org/pkgdb/package/rpms/python34/
[1] https://fedoraproject.org/wiki/EPEL
[2] https://lists.fedoraproject.org/archives/list/epel-devel at lists.fedoraproject.org/

Regards,

Charalampos Stratakis
Associate Software Engineer
Python Maintenance Team, Red Hat


----- Original Message -----
From: "Dahui Jiang" <Dahui.Jiang at veritas.com>
To: "Charalampos Stratakis" <cstratak at redhat.com>
Cc: python-dev at python.org
Sent: Sunday, January 22, 2017 2:58:39 AM
Subject: RE: [Python-Dev] Have problem when building python3.5.1 rpm with default SPEC file

Hi Charalampos:

Thank you very much for your help, and I have built python3.5.1 rpm on redhat7 successfully.
But I find that there are quite a few content in SPEC file of fedora, and it seems that the file has been developed for a long time for patches and other supplementary, 
now that the SPEC file of fedora is open source, how about redhat's, how could I get it? 

Regards
Dahui

-----Original Message-----
From: Charalampos Stratakis [mailto:cstratak at redhat.com] 
Sent: Friday, January 20, 2017 2:40
To: Dahui Jiang <Dahui.Jiang at veritas.com>
Cc: python-dev at python.org
Subject: Re: [Python-Dev] Have problem when building python3.5.1 rpm with default SPEC file

Hello,

This is a distro specific issue so this list might not be the best for resolving that, you should contact your distro's package maintainers of python.

For Fedora 25 we currently ship Python 3.5.2, which builds fine with this SPEC file [0], so maybe you could give this a try.

[0] http://pkgs.fedoraproject.org/cgit/rpms/python3.git/tree/python3.spec?h=f25

Regards,

Charalampos Stratakis
Associate Software Engineer
Python Maintenance Team, Red Hat


----- Original Message -----
From: "Dahui Jiang" <Dahui.Jiang at veritas.com>
To: python-dev at python.org
Sent: Thursday, January 19, 2017 12:54:16 PM
Subject: [Python-Dev] Have problem when building python3.5.1 rpm with default SPEC file



Hi all: 

I?m try to build a python rpm from source python3.5.1, and I use the spec file in the source tree. 

But the building is not success as print the following error: 



*********************************************** 

running build 

running build_ext 

error: [Errno 2] No such file or directory: 'Modules/Setup' 

error: Bad exit status from /var/tmp/rpm-tmp.DDm3jI (%build) 





RPM build errors: 

Bad exit status from /var/tmp/rpm-tmp.DDm3jI (%build) 

************************************************ 



Regards 

Dahui 

_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/cstratak%40redhat.com

From solipsis at pitrou.net  Wed Jan 25 12:33:09 2017
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 25 Jan 2017 18:33:09 +0100
Subject: [Python-Dev] Investigating Python memory footprint of one real
 Web application
In-Reply-To: <CAEfz+TzDAJp9YdBa3dMyECzbr5RzyE=iQFip5jJA70tc7mJXCg@mail.gmail.com>
References: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
 <20170120124333.5f8f5e96@fsol>
 <CAEfz+TwWEjMskjisHvgCBPZ2k7M7LrWA+EaoYH=03WWehNBZZg@mail.gmail.com>
 <1e8a03a3-afb6-fac1-b3ba-e011c4c87eea@python.org>
 <CAPdQG2ogLrTqmuahAvccGr9LAqzZAiFYTcVgbyb_VUV_fkgtTw@mail.gmail.com>
 <CAOMjWknkm8ntOvgG3_eTFhN031O_4bH4-qdf1aUSnfoRjdxd9Q@mail.gmail.com>
 <CAEfz+TwnxJq9TT2Ji18KJt6-3dccQm0fxBdfCGk9S-jyA9kYUw@mail.gmail.com>
 <CAEfz+TyGE0YoQPwZg5wjMrxWmTQqLSnomGgGu7id8SwGekdtPQ@mail.gmail.com>
 <CAMpsgwbtbgSDNio1NVRT0zxLoU2dYnf12XxfqzMLztmnp9pokA@mail.gmail.com>
 <CAEfz+TzHfAFHLz0RJn58oz_tr0bNKXtnh8ZoyrtfsyO6qWr0Kg@mail.gmail.com>
 <CAEfz+TzDAJp9YdBa3dMyECzbr5RzyE=iQFip5jJA70tc7mJXCg@mail.gmail.com>
Message-ID: <20170125183309.2af20792@fsol>

On Wed, 25 Jan 2017 20:54:02 +0900
INADA Naoki <songofacandy at gmail.com> wrote:
> 
> ## Stripped annotation + without pydebug

Does this mean the other measurements were done with pydebug enabled?
pydebug is not meant to be used on production systems so, without
wanting to disparage the effort this went into these measurements, I'm
afraid that makes them not very useful.

Regards

Antoine.

From srkunze at mail.de  Wed Jan 25 12:34:36 2017
From: srkunze at mail.de (Sven R. Kunze)
Date: Wed, 25 Jan 2017 18:34:36 +0100
Subject: [Python-Dev] Generator objects and list comprehensions?
In-Reply-To: <CAFo6DPaP5dcDy=sm4QrauQJDfXgZQYLg0hFdNAwrQ=qM1OwkOw@mail.gmail.com>
References: <CAG=rPVco0cnKdA0qDbhnZUctzeVVuJTgN4YX7ua6x8Lb-Q-TUQ@mail.gmail.com>
 <CAFo6DPaP5dcDy=sm4QrauQJDfXgZQYLg0hFdNAwrQ=qM1OwkOw@mail.gmail.com>
Message-ID: <3de4361c-011c-ad73-787b-9a4af5fd588c@mail.de>

On 25.01.2017 07:28, Joe Jevnik wrote:
>
> That was a long way to explain what the problem was. I think that that 
> solution is to stop using `yield` in comprehensions because it is 
> confusing, or to make `yield` in a comprehension a syntax error.
>

Same here; mixing comprehensions and yield (from) can't be explained 
easily. A SyntaxError would be most appropriate.

Regards,
Sven

From songofacandy at gmail.com  Wed Jan 25 13:35:15 2017
From: songofacandy at gmail.com (INADA Naoki)
Date: Thu, 26 Jan 2017 03:35:15 +0900
Subject: [Python-Dev] Investigating Python memory footprint of one real
 Web application
In-Reply-To: <20170125183309.2af20792@fsol>
References: <CAEfz+TwjiYVXS191fqs_WV8-oSixfL16-c3FXPG+QCnjfZvTEw@mail.gmail.com>
 <20170120124333.5f8f5e96@fsol>
 <CAEfz+TwWEjMskjisHvgCBPZ2k7M7LrWA+EaoYH=03WWehNBZZg@mail.gmail.com>
 <1e8a03a3-afb6-fac1-b3ba-e011c4c87eea@python.org>
 <CAPdQG2ogLrTqmuahAvccGr9LAqzZAiFYTcVgbyb_VUV_fkgtTw@mail.gmail.com>
 <CAOMjWknkm8ntOvgG3_eTFhN031O_4bH4-qdf1aUSnfoRjdxd9Q@mail.gmail.com>
 <CAEfz+TwnxJq9TT2Ji18KJt6-3dccQm0fxBdfCGk9S-jyA9kYUw@mail.gmail.com>
 <CAEfz+TyGE0YoQPwZg5wjMrxWmTQqLSnomGgGu7id8SwGekdtPQ@mail.gmail.com>
 <CAMpsgwbtbgSDNio1NVRT0zxLoU2dYnf12XxfqzMLztmnp9pokA@mail.gmail.com>
 <CAEfz+TzHfAFHLz0RJn58oz_tr0bNKXtnh8ZoyrtfsyO6qWr0Kg@mail.gmail.com>
 <CAEfz+TzDAJp9YdBa3dMyECzbr5RzyE=iQFip5jJA70tc7mJXCg@mail.gmail.com>
 <20170125183309.2af20792@fsol>
Message-ID: <CAEfz+TyqHVbD+xLn+m+AC6h3uWHXzgStHm0mcTAi0vFVu=W+RA@mail.gmail.com>

On Thu, Jan 26, 2017 at 2:33 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Wed, 25 Jan 2017 20:54:02 +0900
> INADA Naoki <songofacandy at gmail.com> wrote:
>>
>> ## Stripped annotation + without pydebug
>
> Does this mean the other measurements were done with pydebug enabled?
> pydebug is not meant to be used on production systems so, without
> wanting to disparage the effort this went into these measurements, I'm
> afraid that makes them not very useful.
>
> Regards
>
> Antoine.

Yes.  I used sys.getobjects() API which is available only in pydebug mode.
Since it adds two words to all objects for doubly linked list,  I did
sys.getsizeof(o) - 16
when calculating memory used by each type.

While it may bit different from --without-pydebug, I believe it's
useful enough to
estimate how much memory is used by each types.

From steve at holdenweb.com  Wed Jan 25 17:23:10 2017
From: steve at holdenweb.com (Steve Holden)
Date: Wed, 25 Jan 2017 22:23:10 +0000
Subject: [Python-Dev] Generator objects and list comprehensions?
In-Reply-To: <CAFo6DPaP5dcDy=sm4QrauQJDfXgZQYLg0hFdNAwrQ=qM1OwkOw@mail.gmail.com>
References: <CAG=rPVco0cnKdA0qDbhnZUctzeVVuJTgN4YX7ua6x8Lb-Q-TUQ@mail.gmail.com>
 <CAFo6DPaP5dcDy=sm4QrauQJDfXgZQYLg0hFdNAwrQ=qM1OwkOw@mail.gmail.com>
Message-ID: <CAMofdRAag41xVaEHv+3gAZnfXuGQohpg-y5nUyj1M44k2PpzMg@mail.gmail.com>

On Wed, Jan 25, 2017 at 6:28 AM, Joe Jevnik <jjevnik at quantopian.com> wrote:

> That was a long way to explain what the problem was. I think that that
> solution is to stop using `yield` in comprehensions because it is
> confusing, or to make `yield` in a comprehension a syntax error.
>
> Thanks for the very clear explanation.

Would an addition to the documentation for 3.6 giving an abbreviated
version help while the devs consider whether any changes are appropriate
for 3.7?


Steve Holden
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170125/a0247d31/attachment.html>

From njs at pobox.com  Wed Jan 25 18:53:08 2017
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 25 Jan 2017 15:53:08 -0800
Subject: [Python-Dev] Generator objects and list comprehensions?
In-Reply-To: <CAFo6DPaP5dcDy=sm4QrauQJDfXgZQYLg0hFdNAwrQ=qM1OwkOw@mail.gmail.com>
References: <CAG=rPVco0cnKdA0qDbhnZUctzeVVuJTgN4YX7ua6x8Lb-Q-TUQ@mail.gmail.com>
 <CAFo6DPaP5dcDy=sm4QrauQJDfXgZQYLg0hFdNAwrQ=qM1OwkOw@mail.gmail.com>
Message-ID: <CAPJVwBkg4hPsJ9axhjDBRvUn20=yd0aOx_8eAJbvSD3R-qZr-g@mail.gmail.com>

On Jan 25, 2017 8:16 AM, "Joe Jevnik" <jjevnik at quantopian.com> wrote:

List, set, and dict comprehensions compile like:

# input
result = [expression for lhs in iterator_expression]

# output
def comprehension(iterator):
    out = []
    for lhs in iterator:
        out.append(expression)
    return out

result = comprehension(iter(iterator_expression))

When you make `expression` a `yield` the compiler thinks that
`comprehension` is a generator function instead of a normal function.

We can manually translate the following comprehension:

result = [(yield n) for n in (0, 1)]

def comprehension(iterator):
    out = []
    for n in iterator:
        # (yield n) as an expression is the value sent into this generator
        out.append((yield n))
    return out

result = comprehension(iter((0, 1)))


We can see this in the behavior of `send` on the resulting generator:

In [1]: g = [(yield n) for n in (0, 1)]

In [2]: next(g)
Out[2]: 0

In [3]: g.send('hello')
Out[3]: 1

In [4]: g.send('world')
---------------------------------------------------------------------------
StopIteration                             Traceback (most recent call last)
<ipython-input-4-82c589e0c27d> in <module>()
----> 1 g.send('world')

StopIteration: ['hello', 'world']

The `return out` gets translated into `raise StopIteration(out)` because
the code is a generator.

The bytecode for this looks like:

In [5]: %%dis
   ...: [(yield n) for n in (0, 1)]
   ...:
<module>
--------
  1           0 LOAD_CONST               0 (<code object <listcomp> at
0x7f4bae68eed0, file "<show>", line 1>)
              3 LOAD_CONST               1 ('<listcomp>')
              6 MAKE_FUNCTION            0
              9 LOAD_CONST               5 ((0, 1))
             12 GET_ITER
             13 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
             16 POP_TOP
             17 LOAD_CONST               4 (None)
             20 RETURN_VALUE

<module>.<listcomp>
-------------------
  1           0 BUILD_LIST               0
              3 LOAD_FAST                0 (.0)
        >>    6 FOR_ITER                13 (to 22)
              9 STORE_FAST               1 (n)
             12 LOAD_FAST                1 (n)
             15 YIELD_VALUE
             16 LIST_APPEND              2
             19 JUMP_ABSOLUTE            6
        >>   22 RETURN_VALUE


In `<module>` you can see us create the <listcomp> function, call `iter` on
`(0, 1)`, and then call `<listcomp>(iter(0, 1))`. In `<listcomp>` you can
see the loop like we had above, but unlike a normal comprehension we have a
`YIELD_VALUE` (from the `(yield n)`) in the comprehension.

The reason that this is different in Python 2 is that list comprehension,
and only list comprehensions, compile inline. Instead of creating a new
function for the loop, it is done in the scope of the comprehension. For
example:

# input
result = [expression for lhs in iterator_expression]

# output
result = []
for lhs in iterator_expression:
    result.append(lhs)

This is why `lhs` bleeds out of the comprehension. In Python 2, adding
making `expression` a `yield` causes the _calling_ function to become a
generator:

def f():
    [(yield n) for n in range(3)]
    # no return

# py2
>>> type(f())
generator

# py3
>> type(f())
NoneType

In Python 2 the following will even raise a syntax error:

In [5]: def f():
   ...:     return [(yield n) for n in range(3)]
   ...:
   ...:
  File "<ipython-input-5-3602a9999f46>", line 2
    return [(yield n) for n in range(3)]
SyntaxError: 'return' with argument inside generator


Generator expressions are a little different because the compilation
already includes a `yield`.

# input
(expression for lhs in iterator_expression)

# output
def genexpr(iterator):
    for lhs in iterator:
        yield expression

You can actually abuse this to write a cute `flatten` function:

`flatten = lambda seq: (None for sub in seq if (yield from sub) and False)`

because it translates to:

def genexpr(seq):
    for sub in seq:
        # (yield from sub) as an expression returns the last sent in value
        if (yield from sub) and False:
            # we never enter this branch
            yield None


That was a long way to explain what the problem was. I think that that
solution is to stop using `yield` in comprehensions because it is
confusing, or to make `yield` in a comprehension a syntax error.


Another option would be to restore the py2 behavior by inserting an
implicit 'yield from' whenever the embedded expression contains a yield. So
in your example where

result = [(yield n) for n in (0, 1)]

currently gets expanded to

result = comprehension(iter((0, 1)))

it could notice that the expanded function 'comprehension' is a generator,
and emit code like this instead:

result = yield from comprehension(iter((0, 1)))

At least, this would work for list/dict/set comprehensions; not so much for
generator expressions. I assume this is basically how 'await' in
comprehensions works in 3.6.

I'm not sure this is actually a good idea, given the potential for
ambiguity and backwards compatibility considerations -- I'm kind of leaning
towards the deprecate/error option on balance :-). But it's an option that
would probably be less confusing than the status quo, and would make it
easier to write the kind of straddling code the original poster was trying
to write.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170125/08fc26c0/attachment.html>

From levkivskyi at gmail.com  Thu Jan 26 07:09:05 2017
From: levkivskyi at gmail.com (Ivan Levkivskyi)
Date: Thu, 26 Jan 2017 13:09:05 +0100
Subject: [Python-Dev] Generator objects and list comprehensions?
In-Reply-To: <CAPJVwBkg4hPsJ9axhjDBRvUn20=yd0aOx_8eAJbvSD3R-qZr-g@mail.gmail.com>
References: <CAG=rPVco0cnKdA0qDbhnZUctzeVVuJTgN4YX7ua6x8Lb-Q-TUQ@mail.gmail.com>
 <CAFo6DPaP5dcDy=sm4QrauQJDfXgZQYLg0hFdNAwrQ=qM1OwkOw@mail.gmail.com>
 <CAPJVwBkg4hPsJ9axhjDBRvUn20=yd0aOx_8eAJbvSD3R-qZr-g@mail.gmail.com>
Message-ID: <CAOMjWkmCVZbex40Ls8nDfugQYyWMQTMeBPY+DD_nG9R7fa_QxA@mail.gmail.com>

On 26 January 2017 at 00:53, Nathaniel Smith <njs at pobox.com> wrote:

>
> I'm not sure this is actually a good idea, given the potential for
> ambiguity and backwards compatibility considerations -- I'm kind of leaning
> towards the deprecate/error option on balance :-). But it's an option that
> would probably be less confusing than the status quo, and would make it
> easier to write the kind of straddling code the original poster was trying
> to write.
>
>
Concerning list/set/dict comprehensions, I am much more in favor of making
comprehensions simply equivalent to for-loops (more or less like you
proposed using yield from). The only reason to introduce auxiliary function
scope was to prevent the loop variables from leaking outside comprehensions.
Formally, this is indeed backward incompatible, but I doubt many people
depend on the current counter-intuitive behavior.

Concerning generator expressions, probably it is indeed better to simply
prohibit yield inside them.

--
Ivan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170126/06dc8028/attachment.html>

From skip.montanaro at gmail.com  Thu Jan 26 11:00:38 2017
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Thu, 26 Jan 2017 10:00:38 -0600
Subject: [Python-Dev] Why is PyDateTimeAPI not initialized by datetime
 module import?
Message-ID: <CANc-5UxZ0a63Dvg0rapjVp5pnpL08L__cqpuMVeEnHAs0K10vg@mail.gmail.com>

I just got burned (wasted a good day or so) by the fact that PyDateTimeAPI
wasn't initialized. The datetime.rst doc states (emphasis mine):

Before using any of these functions, the header file :file:`datetime.h`
must be included in your source (note that this is not included by
:file:`Python.h`), and the macro :c:macro:`PyDateTime_IMPORT` must be
invoked, *usually as part of the module initialisation function*.

I thought that surely the datetime module itself would initialize that
stuff. Why not? Is it so terribly expensive that the C API requires this
rather weird hack? The code's been their for ages, so there must have been
a good reason at one time. Is that reason still valid today? (I haven't
programmed at the C API level for a good long while, or I'm sure I'd have
encountered this before.)

Thx,

Skip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170126/8f58ff1e/attachment-0001.html>

From alexander.belopolsky at gmail.com  Thu Jan 26 13:55:34 2017
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Thu, 26 Jan 2017 13:55:34 -0500
Subject: [Python-Dev] Why is PyDateTimeAPI not initialized by datetime
 module import?
In-Reply-To: <CANc-5UxZ0a63Dvg0rapjVp5pnpL08L__cqpuMVeEnHAs0K10vg@mail.gmail.com>
References: <CANc-5UxZ0a63Dvg0rapjVp5pnpL08L__cqpuMVeEnHAs0K10vg@mail.gmail.com>
Message-ID: <CAP7h-xZG38W-vc4BfVJSYSkD6utdDYaW8O3Ti=JewbEtoVW8Qw@mail.gmail.com>

On Thu, Jan 26, 2017 at 11:00 AM, Skip Montanaro <skip.montanaro at gmail.com>
wrote:

> I just got burned (wasted a good day or so) by the fact that PyDateTimeAPI
> wasn't initialized. The datetime.rst doc states (emphasis mine):
>
> Before using any of these functions, the header file :file:`datetime.h`
> must be included in your source (note that this is not included by
> :file:`Python.h`), and the macro :c:macro:`PyDateTime_IMPORT` must be
> invoked, *usually as part of the module initialisation function*.
>
> I thought that surely the datetime module itself would initialize that
> stuff. Why not?
>

It is fairly common for the modules that export C API in a capsule to only
initialize that capsule when explicitly asked.  For example, if you want to
use numpy C API, you need to call import_array() in your module
initialization function. [1]  I don't know how expensive PyDateTime_IMPORT
is, but it cannot be called in _datetimemodule.c because it is guarded by
the Py_BUILD_CORE macro and is not available when _datetimemodule.c itself
is compiled.  I don't know whether this can be easily circumvented, but it
is not hard to remember that one should initialize C API before using it.
Maybe we can improve the error message when
PyDateTime_IMPORT is forgotten.  Feel free to open an issue for that.

[1]:
https://docs.scipy.org/doc/numpy-1.12.0/user/c-info.how-to-extend.html#required-subroutine
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170126/bcaa3744/attachment.html>

From srkunze at mail.de  Thu Jan 26 16:13:11 2017
From: srkunze at mail.de (Sven R. Kunze)
Date: Thu, 26 Jan 2017 22:13:11 +0100
Subject: [Python-Dev] re performance
Message-ID: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>

Hi folks,

I recently refreshed regular expressions theoretical basics *indulging 
in reminiscences* So, I read https://swtch.com/~rsc/regexp/regexp1.html

However, reaching the chart in the lower third of the article, I saw 
Python 2.4 measured against a naive Thompson matching implementation. 
And I was surprised about how bad it performed compared to an 
unoptimized version of an older than dirt algorithm.

So, I decided to give it a try with Python2.7 and Python3.5. Eh, voil?, 
100% cpu and no results so far. Nothing has changed at all since 2007.

 >>> import re
 >>> re.match(r'a?'*30 + r'a'*30, 'a'*30)
.... (still waiting)

Quoting from the article: " Some might argue that this test is unfair to 
the backtracking implementations, since it focuses on an uncommon corner 
case. This argument misses the point: given a choice between an 
implementation with a predictable, consistent, fast running time on all 
inputs or one that usually runs quickly but can take years of CPU time 
(or more) on some inputs, the decision should be easy."

Victor, as the head of Python performance department, and Matthew, as 
the maintainer of the new regex module, what is your stance on this 
particular issue?

 From my perspective, I can say, that regular expressions might worth 
optimizing especially for web applications (url matching usually uses 
regexes) but also for other applications where I've seen many tight 
loops using regexes as well. So, I am probing interest on this topic here.

Best,
Sven


From vlastimil.brom at gmail.com  Thu Jan 26 16:33:57 2017
From: vlastimil.brom at gmail.com (Vlastimil Brom)
Date: Thu, 26 Jan 2017 22:33:57 +0100
Subject: [Python-Dev] re performance
In-Reply-To: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
Message-ID: <CAHzaPEO2+BcJUEy15j37d5O72eo1OWewy4PeR0_vcHu0AD6Msg@mail.gmail.com>

2017-01-26 22:13 GMT+01:00 Sven R. Kunze <srkunze at mail.de>:
> Hi folks,
>
> I recently refreshed regular expressions theoretical basics *indulging in
> reminiscences* So, I read https://swtch.com/~rsc/regexp/regexp1.html
>
> However, reaching the chart in the lower third of the article, I saw Python
> 2.4 measured against a naive Thompson matching implementation. And I was
> surprised about how bad it performed compared to an unoptimized version of
> an older than dirt algorithm.
>
> So, I decided to give it a try with Python2.7 and Python3.5. Eh, voil?, 100%
> cpu and no results so far. Nothing has changed at all since 2007.
>
>>>> import re
>>>> re.match(r'a?'*30 + r'a'*30, 'a'*30)
> .... (still waiting)
>
> Quoting from the article: " Some might argue that this test is unfair to the
> backtracking implementations, since it focuses on an uncommon corner case.
> This argument misses the point: given a choice between an implementation
> with a predictable, consistent, fast running time on all inputs or one that
> usually runs quickly but can take years of CPU time (or more) on some
> inputs, the decision should be easy."
>
> Victor, as the head of Python performance department, and Matthew, as the
> maintainer of the new regex module, what is your stance on this particular
> issue?
>
> From my perspective, I can say, that regular expressions might worth
> optimizing especially for web applications (url matching usually uses
> regexes) but also for other applications where I've seen many tight loops
> using regexes as well. So, I am probing interest on this topic here.
>
> Best,
> Sven
>

Hi,
I can't speak about the details of mrab's implementation, but using
regex, I get the resulting match instantly:

Python 3.6.0 (v3.6.0:41df79263a11, Dec 23 2016, 07:18:10) [MSC v.1900
32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> import regex
>>> regex.match(r'a?'*30 + r'a'*30, 'a'*30)
<regex.Match object; span=(0, 30), match='aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'>
>>>

(I personally prefer to use regex for other advantages, than this
artificial case, but it certainly doesn't hurt to have better
performance here too:)

regards,
    vbr

From srkunze at mail.de  Thu Jan 26 16:46:08 2017
From: srkunze at mail.de (Sven R. Kunze)
Date: Thu, 26 Jan 2017 22:46:08 +0100
Subject: [Python-Dev] re performance
In-Reply-To: <CAHzaPEO2+BcJUEy15j37d5O72eo1OWewy4PeR0_vcHu0AD6Msg@mail.gmail.com>
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
 <CAHzaPEO2+BcJUEy15j37d5O72eo1OWewy4PeR0_vcHu0AD6Msg@mail.gmail.com>
Message-ID: <d9e50354-8ed7-c347-a397-34f0b83927c6@mail.de>

On 26.01.2017 22:33, Vlastimil Brom wrote:
> Hi,
> I can't speak about the details of mrab's implementation, but using
> regex, I get the resulting match instantly: [...]

Nice! I focused on the stdlib re module as this is mainly used by other 
frameworks (like Django).

> (I personally prefer to use regex for other advantages, than this
> artificial case, but it certainly doesn't hurt to have better
> performance here too:)

Me, too.

So, it seems as if regex already uses a better algorithm although I 
couldn't find any reference to any regex theoretical framework like dfa, 
nfa, thompson multiple-state simulation or something.


Cheers,
Sven

From python at mrabarnett.plus.com  Thu Jan 26 20:14:50 2017
From: python at mrabarnett.plus.com (MRAB)
Date: Fri, 27 Jan 2017 01:14:50 +0000
Subject: [Python-Dev] re performance
In-Reply-To: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
Message-ID: <948dbd82-a30e-8e72-c9f9-3a9d987803fb@mrabarnett.plus.com>

On 2017-01-26 21:13, Sven R. Kunze wrote:
> Hi folks,
>
> I recently refreshed regular expressions theoretical basics *indulging
> in reminiscences* So, I read https://swtch.com/~rsc/regexp/regexp1.html
>
> However, reaching the chart in the lower third of the article, I saw
> Python 2.4 measured against a naive Thompson matching implementation.
> And I was surprised about how bad it performed compared to an
> unoptimized version of an older than dirt algorithm.
>
> So, I decided to give it a try with Python2.7 and Python3.5. Eh, voil?,
> 100% cpu and no results so far. Nothing has changed at all since 2007.
>
>   >>> import re
>   >>> re.match(r'a?'*30 + r'a'*30, 'a'*30)
> .... (still waiting)
>
> Quoting from the article: " Some might argue that this test is unfair to
> the backtracking implementations, since it focuses on an uncommon corner
> case. This argument misses the point: given a choice between an
> implementation with a predictable, consistent, fast running time on all
> inputs or one that usually runs quickly but can take years of CPU time
> (or more) on some inputs, the decision should be easy."
>
> Victor, as the head of Python performance department, and Matthew, as
> the maintainer of the new regex module, what is your stance on this
> particular issue?
>
>   From my perspective, I can say, that regular expressions might worth
> optimizing especially for web applications (url matching usually uses
> regexes) but also for other applications where I've seen many tight
> loops using regexes as well. So, I am probing interest on this topic here.
It's possible to add some checks to reduce the problem, as I've done in 
the regex module, but I'm not sure how easy it would be to retrofit it 
into the existing re code. (It wasn't pleasant in the regex module, so...).


From python at mrabarnett.plus.com  Thu Jan 26 20:16:52 2017
From: python at mrabarnett.plus.com (MRAB)
Date: Fri, 27 Jan 2017 01:16:52 +0000
Subject: [Python-Dev] re performance
In-Reply-To: <d9e50354-8ed7-c347-a397-34f0b83927c6@mail.de>
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
 <CAHzaPEO2+BcJUEy15j37d5O72eo1OWewy4PeR0_vcHu0AD6Msg@mail.gmail.com>
 <d9e50354-8ed7-c347-a397-34f0b83927c6@mail.de>
Message-ID: <f3414629-15b4-17ff-2570-ade73903bbdf@mrabarnett.plus.com>

On 2017-01-26 21:46, Sven R. Kunze wrote:
> On 26.01.2017 22:33, Vlastimil Brom wrote:
>> Hi,
>> I can't speak about the details of mrab's implementation, but using
>> regex, I get the resulting match instantly: [...]
>
> Nice! I focused on the stdlib re module as this is mainly used by other
> frameworks (like Django).
>
>> (I personally prefer to use regex for other advantages, than this
>> artificial case, but it certainly doesn't hurt to have better
>> performance here too:)
>
> Me, too.
>
> So, it seems as if regex already uses a better algorithm although I
> couldn't find any reference to any regex theoretical framework like dfa,
> nfa, thompson multiple-state simulation or something.
>
It still uses backtracking, like in the re module.


From lukasz at langa.pl  Fri Jan 27 12:03:34 2017
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Fri, 27 Jan 2017 09:03:34 -0800
Subject: [Python-Dev] re performance
In-Reply-To: <f3414629-15b4-17ff-2570-ade73903bbdf@mrabarnett.plus.com>
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
 <CAHzaPEO2+BcJUEy15j37d5O72eo1OWewy4PeR0_vcHu0AD6Msg@mail.gmail.com>
 <d9e50354-8ed7-c347-a397-34f0b83927c6@mail.de>
 <f3414629-15b4-17ff-2570-ade73903bbdf@mrabarnett.plus.com>
Message-ID: <11439969-DBE6-4031-9EC6-3B2452340942@langa.pl>


> On Jan 26, 2017, at 5:16 PM, MRAB <python at mrabarnett.plus.com> wrote:

>> So, it seems as if regex already uses a better algorithm although I
>> couldn't find any reference to any regex theoretical framework like dfa,
>> nfa, thompson multiple-state simulation or something.
>> 
> It still uses backtracking, like in the re module.

What?s the status of regex inclusion in the stdlib?

- ?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170127/270b1576/attachment.html>

From status at bugs.python.org  Fri Jan 27 12:09:10 2017
From: status at bugs.python.org (Python tracker)
Date: Fri, 27 Jan 2017 18:09:10 +0100 (CET)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20170127170910.2E7E956B8C@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2017-01-20 - 2017-01-27)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    5768 (+13)
  closed 35368 (+33)
  total  41136 (+46)

Open issues with patches: 2485 


Issues opened (33)
==================

#16899: Add support for C99 complex type (_Complex) as ctypes.c_comple
http://bugs.python.org/issue16899  reopened by mark.dickinson

#29301: decimal: Use FASTCALL and/or Argument Clinic
http://bugs.python.org/issue29301  reopened by haypo

#29335: subprocess module does not check WIFSTOPPED on SIGCHLD
http://bugs.python.org/issue29335  opened by Zach Riggle

#29336: merge tuples in module
http://bugs.python.org/issue29336  opened by inada.naoki

#29339: Interactive: Move to same indentation level as previous line
http://bugs.python.org/issue29339  opened by 12345 67890

#29340: SSL destructor segfaults in python3.6 threads when an unverifi
http://bugs.python.org/issue29340  opened by pirate

#29341: Missing accepting path-like object in docstrings of os module 
http://bugs.python.org/issue29341  opened by xiang.zhang

#29342: os.posix_fadvise misreports errors
http://bugs.python.org/issue29342  opened by enkore

#29343: sock.close() raises OSError EBADF when socket's fd is closed
http://bugs.python.org/issue29343  opened by christian.heimes

#29344: sock_recv not detected a coroutine
http://bugs.python.org/issue29344  opened by Jeremy Bustamante

#29347: Python 2.7.8 is crashing while creating weakref for a given ob
http://bugs.python.org/issue29347  opened by dhanavaths

#29349: Update old Python 2 code in Docs/tools/extensions/patchlevel.p
http://bugs.python.org/issue29349  opened by Jim Fasarakis-Hilliard

#29350: Add support of multiple signatures
http://bugs.python.org/issue29350  opened by serhiy.storchaka

#29351: absolute imports for logging
http://bugs.python.org/issue29351  opened by mgerstel

#29352: provide the authorative source for s[i:j] negative slice indic
http://bugs.python.org/issue29352  opened by akira

#29353: Incorrect handling of HTTP response with "Content-Type: messag
http://bugs.python.org/issue29353  opened by brokenenglish

#29354: Python 2.7.12: pydoc.help(lambda (a,): lambda x: a) raises Ind
http://bugs.python.org/issue29354  opened by Elias Vanderstuyft

#29355: sqlite3: remove sqlite3_stmt_readonly()
http://bugs.python.org/issue29355  opened by Ma Lin

#29362: regrtest: don't fail immediately if a child does crash
http://bugs.python.org/issue29362  opened by haypo

#29363: allow Python code to determine class help text
http://bugs.python.org/issue29363  opened by ethan.furman

#29364: msilib Fetch raises MSIError rather than return None
http://bugs.python.org/issue29364  opened by Jason Matthew

#29365: parallel make test or make install can fail building pgen when
http://bugs.python.org/issue29365  opened by gregory.p.smith

#29366: os.listdir has inconsistent behavior when run on a non-directo
http://bugs.python.org/issue29366  opened by raylu

#29367: python-gdb: display wrapper_call()
http://bugs.python.org/issue29367  opened by haypo

#29368: Optimize unpickling list-like objects: use extend() instead of
http://bugs.python.org/issue29368  opened by serhiy.storchaka

#29370: "./configure --enable-optimizations && make install" does not 
http://bugs.python.org/issue29370  opened by dilyan.palauzov

#29371: Typo in doctest documentation
http://bugs.python.org/issue29371  opened by marco.buttu

#29372: RotatingFileHandler rotates empty logfile if it emits a large 
http://bugs.python.org/issue29372  opened by Poddster

#29374: Doc bug: signal.sigwait and signal.sigtimedwait do not work ou
http://bugs.python.org/issue29374  opened by petr at motejlek.net

#29375: httplib: wrong Host header when connecting to IPv6 link-local 
http://bugs.python.org/issue29375  opened by JonathanGuthrie

#29376: threading._DummyThread.__repr__ raises AssertionError
http://bugs.python.org/issue29376  opened by Tom Myers

#29377: Add the 'wrapper_descriptor' type to the types module
http://bugs.python.org/issue29377  opened by Wheerd

#29378: Invalid example in documentation for PyErr_Fetch
http://bugs.python.org/issue29378  opened by Kaeptm Blaubaer



Most recent 15 issues with no replies (15)
==========================================

#29376: threading._DummyThread.__repr__ raises AssertionError
http://bugs.python.org/issue29376

#29370: "./configure --enable-optimizations && make install" does not 
http://bugs.python.org/issue29370

#29367: python-gdb: display wrapper_call()
http://bugs.python.org/issue29367

#29364: msilib Fetch raises MSIError rather than return None
http://bugs.python.org/issue29364

#29355: sqlite3: remove sqlite3_stmt_readonly()
http://bugs.python.org/issue29355

#29344: sock_recv not detected a coroutine
http://bugs.python.org/issue29344

#29333: ConfigParser calls Interpolation.before_read after reading
http://bugs.python.org/issue29333

#29320: bdist_msi install_script fail to execute if alt python locatio
http://bugs.python.org/issue29320

#29317: test_copyxattr_symlinks fails
http://bugs.python.org/issue29317

#29302: add contextlib.AsyncExitStack
http://bugs.python.org/issue29302

#29288: Lookup Error while importing idna from a worker thread
http://bugs.python.org/issue29288

#29287: IDLE needs syntax highlighting for f-strings
http://bugs.python.org/issue29287

#29283: duplicate README in site-packages
http://bugs.python.org/issue29283

#29272: test_logging hangs if /etc/hosts only aliases "localhost" to :
http://bugs.python.org/issue29272

#29269: test_socket failing in solaris
http://bugs.python.org/issue29269



Most recent 15 issues waiting for review (15)
=============================================

#29377: Add the 'wrapper_descriptor' type to the types module
http://bugs.python.org/issue29377

#29368: Optimize unpickling list-like objects: use extend() instead of
http://bugs.python.org/issue29368

#29367: python-gdb: display wrapper_call()
http://bugs.python.org/issue29367

#29354: Python 2.7.12: pydoc.help(lambda (a,): lambda x: a) raises Ind
http://bugs.python.org/issue29354

#29353: Incorrect handling of HTTP response with "Content-Type: messag
http://bugs.python.org/issue29353

#29352: provide the authorative source for s[i:j] negative slice indic
http://bugs.python.org/issue29352

#29351: absolute imports for logging
http://bugs.python.org/issue29351

#29349: Update old Python 2 code in Docs/tools/extensions/patchlevel.p
http://bugs.python.org/issue29349

#29336: merge tuples in module
http://bugs.python.org/issue29336

#29335: subprocess module does not check WIFSTOPPED on SIGCHLD
http://bugs.python.org/issue29335

#29326: Blank lines in ._pth file are not ignored
http://bugs.python.org/issue29326

#29314: asyncio.async deprecation warning is missing stacklevel=2
http://bugs.python.org/issue29314

#29313: msi by bdist_msi will fail execute install-scripts if space in
http://bugs.python.org/issue29313

#29311: Argument Clinic: convert dict methods
http://bugs.python.org/issue29311

#29310: Document typing.NamedTuple default argument syntax
http://bugs.python.org/issue29310



Top 10 most discussed issues (10)
=================================

#18235: _sysconfigdata.py wrong on AIX installations
http://bugs.python.org/issue18235  17 msgs

#29336: merge tuples in module
http://bugs.python.org/issue29336  14 msgs

#29300: Modify the _struct module to use FASTCALL and Argument Clinic
http://bugs.python.org/issue29300  13 msgs

#29335: subprocess module does not check WIFSTOPPED on SIGCHLD
http://bugs.python.org/issue29335  10 msgs

#29339: Interactive: Move to same indentation level as previous line
http://bugs.python.org/issue29339  10 msgs

#29259: Add tp_fastcall to PyTypeObject: support FASTCALL calling conv
http://bugs.python.org/issue29259   7 msgs

#29301: decimal: Use FASTCALL and/or Argument Clinic
http://bugs.python.org/issue29301   7 msgs

#29377: Add the 'wrapper_descriptor' type to the types module
http://bugs.python.org/issue29377   7 msgs

#10544: yield expression inside generator expression does nothing
http://bugs.python.org/issue10544   6 msgs

#29182: Remove the warning in urllib docs that it doesn't do certifica
http://bugs.python.org/issue29182   6 msgs



Issues closed (33)
==================

#18835: Add aligned memory variants to the suite of PyMem functions/ma
http://bugs.python.org/issue18835  closed by haypo

#22651: Open file in a+ mode reads from end of file in Python 3.4
http://bugs.python.org/issue22651  closed by haypo

#26149: Suggest PyCharm Community as an editor for Unix platforms
http://bugs.python.org/issue26149  closed by orsenthil

#26273: Expose TCP_CONGESTION and TCP_USER_TIMEOUT to the socket modul
http://bugs.python.org/issue26273  closed by haypo

#26729: Incorrect __text_signature__ for sorted
http://bugs.python.org/issue26729  closed by serhiy.storchaka

#28635: Update What's New for 3.6
http://bugs.python.org/issue28635  closed by ned.deily

#28735: Mock is equal to ANY but MagicMock is not
http://bugs.python.org/issue28735  closed by serhiy.storchaka

#28769: Make PyUnicode_AsUTF8 returning "const char *" rather of "char
http://bugs.python.org/issue28769  closed by serhiy.storchaka

#28784: shlex.shlex punctuation_chars documentation should use posix=T
http://bugs.python.org/issue28784  closed by python-dev

#28999: Use Py_RETURN_NONE and like
http://bugs.python.org/issue28999  closed by serhiy.storchaka

#29083: Readd PyArg_VaParse to the stable API
http://bugs.python.org/issue29083  closed by serhiy.storchaka

#29092: Sync os.stat's doc and doc string
http://bugs.python.org/issue29092  closed by xiang.zhang

#29167: Race condition in enum.py:_decompose()
http://bugs.python.org/issue29167  closed by python-dev

#29273: test___all__ alters utf8 locale setting
http://bugs.python.org/issue29273  closed by martin.panter

#29290: argparse breaks long lines on NO-BREAK SPACE
http://bugs.python.org/issue29290  closed by xiang.zhang

#29308: venv should match virtualenv VIRTUAL_ENV_DISABLE_PROMPT config
http://bugs.python.org/issue29308  closed by python-dev

#29331: Simplify argument parsing in sorted() and list.sort()
http://bugs.python.org/issue29331  closed by serhiy.storchaka

#29332: Uniform SelectSelector._select behavior
http://bugs.python.org/issue29332  closed by berker.peksag

#29337: BytesWarning at compile time
http://bugs.python.org/issue29337  closed by serhiy.storchaka

#29338: Output the text signature in the help of a class
http://bugs.python.org/issue29338  closed by serhiy.storchaka

#29345: More lost updates with multiprocessing.Value and .Array
http://bugs.python.org/issue29345  closed by eryksun

#29346: datetime doesn't check for min/max dates anymore in Python 3.6
http://bugs.python.org/issue29346  closed by haypo

#29348: this comment 'iff' should be 'if'?
http://bugs.python.org/issue29348  closed by inada.naoki

#29356: TypeError by typing for Ellipsis
http://bugs.python.org/issue29356  closed by ??d??m Szieberth

#29357: New NamedTuple syntax silently ignores method definitions
http://bugs.python.org/issue29357  closed by levkivskyi

#29358: Add tp_fastnew and tp_fastinit to PyTypeObject, 15-20% faster 
http://bugs.python.org/issue29358  closed by haypo

#29359: Deprecate string concatenation without plus
http://bugs.python.org/issue29359  closed by serhiy.storchaka

#29360: _PyStack_AsDict(): Don't check if all keys are strings nor if 
http://bugs.python.org/issue29360  closed by haypo

#29361: bug in pyserial: serialposix.py
http://bugs.python.org/issue29361  closed by berker.peksag

#29369: Use Py_IDENTIFIER in Python-ast.c
http://bugs.python.org/issue29369  closed by inada.naoki

#29373: subprocess.Popen os.close(p2cread) in _execute_child can cause
http://bugs.python.org/issue29373  closed by zmedico

#29379: Custom handlers not used when passing "context" argument to ur
http://bugs.python.org/issue29379  closed by martin.panter

#29380: A lot of warning while executing make install
http://bugs.python.org/issue29380  closed by zach.ware

From breamoreboy at yahoo.co.uk  Fri Jan 27 15:50:18 2017
From: breamoreboy at yahoo.co.uk (Mark Lawrence)
Date: Fri, 27 Jan 2017 20:50:18 +0000
Subject: [Python-Dev] re performance
In-Reply-To: <11439969-DBE6-4031-9EC6-3B2452340942@langa.pl>
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
 <CAHzaPEO2+BcJUEy15j37d5O72eo1OWewy4PeR0_vcHu0AD6Msg@mail.gmail.com>
 <d9e50354-8ed7-c347-a397-34f0b83927c6@mail.de>
 <f3414629-15b4-17ff-2570-ade73903bbdf@mrabarnett.plus.com>
 <11439969-DBE6-4031-9EC6-3B2452340942@langa.pl>
Message-ID: <o6gbq7$fk2$1@blaine.gmane.org>

On 27/01/2017 17:03, ?ukasz Langa wrote:
>
>> On Jan 26, 2017, at 5:16 PM, MRAB <python at mrabarnett.plus.com
>> <mailto:python at mrabarnett.plus.com>> wrote:
>
>>> So, it seems as if regex already uses a better algorithm although I
>>> couldn't find any reference to any regex theoretical framework like dfa,
>>> nfa, thompson multiple-state simulation or something.
>>>
>> It still uses backtracking, like in the re module.
>
> What?s the status of regex inclusion in the stdlib?
>
> - ?
>

I've asked about this in the past, but have now come to the conclusion 
that it is way better to leave regex, and many other third party 
modules, on pypi rather than have them tied into the Python release 
cycle.  If YMMV so be it.

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence


From python at mrabarnett.plus.com  Fri Jan 27 16:24:44 2017
From: python at mrabarnett.plus.com (MRAB)
Date: Fri, 27 Jan 2017 21:24:44 +0000
Subject: [Python-Dev] re performance
In-Reply-To: <11439969-DBE6-4031-9EC6-3B2452340942@langa.pl>
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
 <CAHzaPEO2+BcJUEy15j37d5O72eo1OWewy4PeR0_vcHu0AD6Msg@mail.gmail.com>
 <d9e50354-8ed7-c347-a397-34f0b83927c6@mail.de>
 <f3414629-15b4-17ff-2570-ade73903bbdf@mrabarnett.plus.com>
 <11439969-DBE6-4031-9EC6-3B2452340942@langa.pl>
Message-ID: <65cf9e6e-7962-d198-6aa7-4302818a8819@mrabarnett.plus.com>

On 2017-01-27 17:03, ?ukasz Langa wrote:
>
>> On Jan 26, 2017, at 5:16 PM, MRAB <python at mrabarnett.plus.com 
>> <mailto:python at mrabarnett.plus.com>> wrote:
>
>>> So, it seems as if regex already uses a better algorithm although I
>>> couldn't find any reference to any regex theoretical framework like dfa,
>>> nfa, thompson multiple-state simulation or something.
>>>
>> It still uses backtracking, like in the re module.
>
> What?s the status of regex inclusion in the stdlib?
>
>
I'm not bothered about it. It's quite a bit bigger than the re module, 
and, anyway, keeping it as a third-party module gives me more freedom to 
make updates, which are available for a range of Python versions.


From michael at felt.demon.nl  Fri Jan 27 17:15:36 2017
From: michael at felt.demon.nl (Michael Felt)
Date: Fri, 27 Jan 2017 23:15:36 +0100
Subject: [Python-Dev] re performance
In-Reply-To: <65cf9e6e-7962-d198-6aa7-4302818a8819@mrabarnett.plus.com>
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
 <CAHzaPEO2+BcJUEy15j37d5O72eo1OWewy4PeR0_vcHu0AD6Msg@mail.gmail.com>
 <d9e50354-8ed7-c347-a397-34f0b83927c6@mail.de>
 <f3414629-15b4-17ff-2570-ade73903bbdf@mrabarnett.plus.com>
 <11439969-DBE6-4031-9EC6-3B2452340942@langa.pl>
 <65cf9e6e-7962-d198-6aa7-4302818a8819@mrabarnett.plus.com>
Message-ID: <05fbabc1-df1b-c8d7-4a4e-de1cc9025fe8@felt.demon.nl>

On 27/01/2017 22:24, MRAB wrote:
> I'm not bothered about it. It's quite a bit bigger than the re module, 
> and, anyway, keeping it as a third-party module gives me more freedom 
> to make updates, which are available for a range of Python versions. 

I tried packaging it (pip build) and ran into a minor syntax error (on 
AIX using xlc). How do I go about reporting what I found?


From python at mrabarnett.plus.com  Fri Jan 27 18:39:26 2017
From: python at mrabarnett.plus.com (MRAB)
Date: Fri, 27 Jan 2017 23:39:26 +0000
Subject: [Python-Dev] re performance
In-Reply-To: <05fbabc1-df1b-c8d7-4a4e-de1cc9025fe8@felt.demon.nl>
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
 <CAHzaPEO2+BcJUEy15j37d5O72eo1OWewy4PeR0_vcHu0AD6Msg@mail.gmail.com>
 <d9e50354-8ed7-c347-a397-34f0b83927c6@mail.de>
 <f3414629-15b4-17ff-2570-ade73903bbdf@mrabarnett.plus.com>
 <11439969-DBE6-4031-9EC6-3B2452340942@langa.pl>
 <65cf9e6e-7962-d198-6aa7-4302818a8819@mrabarnett.plus.com>
 <05fbabc1-df1b-c8d7-4a4e-de1cc9025fe8@felt.demon.nl>
Message-ID: <fc4feadf-24c1-a629-3b5a-6f7a10642a84@mrabarnett.plus.com>

On 2017-01-27 22:15, Michael Felt wrote:
> On 27/01/2017 22:24, MRAB wrote:
> > I'm not bothered about it. It's quite a bit bigger than the re module,
> > and, anyway, keeping it as a third-party module gives me more freedom
> > to make updates, which are available for a range of Python versions.
>
> I tried packaging it (pip build) and ran into a minor syntax error (on
> AIX using xlc). How do I go about reporting what I found?
>
Report it on the project's issue tracker.


From armin.rigo at gmail.com  Sat Jan 28 06:44:37 2017
From: armin.rigo at gmail.com (Armin Rigo)
Date: Sat, 28 Jan 2017 12:44:37 +0100
Subject: [Python-Dev] re performance
In-Reply-To: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
Message-ID: <CAMSv6X3KzYBLuSL=7Zx5mPSaK9y7=-s5m+2EnLuXy6SRYFhPpA@mail.gmail.com>

Hi Sven,

On 26 January 2017 at 22:13, Sven R. Kunze <srkunze at mail.de> wrote:
> I recently refreshed regular expressions theoretical basics *indulging in
> reminiscences* So, I read https://swtch.com/~rsc/regexp/regexp1.html

Theoretical regular expressions and what Python/Perl/etc. call regular
expressions are a bit different.  You can read more about it at
https://en.wikipedia.org/wiki/Regular_expression#Implementations_and_running_times
.

Discussions about why they are different often focus on
backreferences, which is a rare feature.  Let me add two other points.

The theoretical kind of regexp is about giving a "yes/no" answer,
whereas the concrete "re" or "regexp" modules gives a match object,
which lets you ask for the subgroups' location, for example.  Strange
at it may seem, I am not aware of a way to do that using the
linear-time approach of the theory---if it answers "yes", then you
have no way of knowing *where* the subgroups matched.

Another issue is that the theoretical engine has no notion of
greedy/non-greedy matching.  Basically, you walk over the source
character and it answers "yes" or "no" after each of them.  This is
different from a typical backtracking implementation.  In Python:

>>> re.match(r'a*', 'aaa')
>>> re.match(r'a*?', 'aaa')

This matches either three or zero characters in Python.  The two
versions are however indistinguishable for the theoretical engine.


A bient?t,

Armin.

From ncoghlan at gmail.com  Sat Jan 28 09:43:42 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 28 Jan 2017 15:43:42 +0100
Subject: [Python-Dev] re performance
In-Reply-To: <65cf9e6e-7962-d198-6aa7-4302818a8819@mrabarnett.plus.com>
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
 <CAHzaPEO2+BcJUEy15j37d5O72eo1OWewy4PeR0_vcHu0AD6Msg@mail.gmail.com>
 <d9e50354-8ed7-c347-a397-34f0b83927c6@mail.de>
 <f3414629-15b4-17ff-2570-ade73903bbdf@mrabarnett.plus.com>
 <11439969-DBE6-4031-9EC6-3B2452340942@langa.pl>
 <65cf9e6e-7962-d198-6aa7-4302818a8819@mrabarnett.plus.com>
Message-ID: <CADiSq7dforv5fY9Y4hgRGJ_6z9R0AY0MM-T=Cai6UmzGmPJdVg@mail.gmail.com>

On 27 January 2017 at 22:24, MRAB <python at mrabarnett.plus.com> wrote:
> On 2017-01-27 17:03, ?ukasz Langa wrote:
>>
>>
>>> On Jan 26, 2017, at 5:16 PM, MRAB <python at mrabarnett.plus.com
>>> <mailto:python at mrabarnett.plus.com>> wrote:
>>
>>
>>>> So, it seems as if regex already uses a better algorithm although I
>>>> couldn't find any reference to any regex theoretical framework like dfa,
>>>> nfa, thompson multiple-state simulation or something.
>>>>
>>> It still uses backtracking, like in the re module.
>>
>>
>> What?s the status of regex inclusion in the stdlib?
>>
>>
> I'm not bothered about it. It's quite a bit bigger than the re module, and,
> anyway, keeping it as a third-party module gives me more freedom to make
> updates, which are available for a range of Python versions.

I still think it could be a good candidate for a first "bundled"
module, where we don't migrate it fully into the CPython development
process, but *do* officially bless it and provide it by default in the
form of a bundled wheel file (similar to what we do with pip).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From tjreedy at udel.edu  Sat Jan 28 10:30:42 2017
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 28 Jan 2017 10:30:42 -0500
Subject: [Python-Dev] re performance
In-Reply-To: <CADiSq7dforv5fY9Y4hgRGJ_6z9R0AY0MM-T=Cai6UmzGmPJdVg@mail.gmail.com>
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
 <CAHzaPEO2+BcJUEy15j37d5O72eo1OWewy4PeR0_vcHu0AD6Msg@mail.gmail.com>
 <d9e50354-8ed7-c347-a397-34f0b83927c6@mail.de>
 <f3414629-15b4-17ff-2570-ade73903bbdf@mrabarnett.plus.com>
 <11439969-DBE6-4031-9EC6-3B2452340942@langa.pl>
 <65cf9e6e-7962-d198-6aa7-4302818a8819@mrabarnett.plus.com>
 <CADiSq7dforv5fY9Y4hgRGJ_6z9R0AY0MM-T=Cai6UmzGmPJdVg@mail.gmail.com>
Message-ID: <o6ideu$oam$1@blaine.gmane.org>

On 1/28/2017 9:43 AM, Nick Coghlan wrote:
> On 27 January 2017 at 22:24, MRAB <python at mrabarnett.plus.com> wrote:
>> On 2017-01-27 17:03, ?ukasz Langa wrote:

>>> What?s the status of regex inclusion in the stdlib?
>>>
>> I'm not bothered about it. It's quite a bit bigger than the re module, and,
>> anyway, keeping it as a third-party module gives me more freedom to make
>> updates, which are available for a range of Python versions.
>
> I still think it could be a good candidate for a first "bundled"
> module, where we don't migrate it fully into the CPython development
> process, but *do* officially bless it and provide it by default in the
> form of a bundled wheel file (similar to what we do with pip).

So like pip, we would bundle the current version, install it in 
site-packages, and users who use it could upgrade as desired, even after 
x.y bugfix ends.

What would be nice would be to have a short entry in the docs for 
bundled modules that links to the real doc, wherever it is.

-- 
Terry Jan Reedy



From barry at python.org  Sat Jan 28 12:07:05 2017
From: barry at python.org (Barry Warsaw)
Date: Sat, 28 Jan 2017 12:07:05 -0500
Subject: [Python-Dev] re performance
In-Reply-To: <CADiSq7dforv5fY9Y4hgRGJ_6z9R0AY0MM-T=Cai6UmzGmPJdVg@mail.gmail.com>
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
 <CAHzaPEO2+BcJUEy15j37d5O72eo1OWewy4PeR0_vcHu0AD6Msg@mail.gmail.com>
 <d9e50354-8ed7-c347-a397-34f0b83927c6@mail.de>
 <f3414629-15b4-17ff-2570-ade73903bbdf@mrabarnett.plus.com>
 <11439969-DBE6-4031-9EC6-3B2452340942@langa.pl>
 <65cf9e6e-7962-d198-6aa7-4302818a8819@mrabarnett.plus.com>
 <CADiSq7dforv5fY9Y4hgRGJ_6z9R0AY0MM-T=Cai6UmzGmPJdVg@mail.gmail.com>
Message-ID: <20170128120705.3696f70c@subdivisions.wooz.org>

On Jan 28, 2017, at 03:43 PM, Nick Coghlan wrote:

>I still think it could be a good candidate for a first "bundled"
>module, where we don't migrate it fully into the CPython development
>process, but *do* officially bless it and provide it by default in the
>form of a bundled wheel file (similar to what we do with pip).

How would that work exactly.  I.e. is there a PEP?

While I think it could be a good idea to bundle (more?) third party packages
for a variety of reasons, I want to make sure it's done in a way that's still
friendly to downstream repackagers, as I'm sure you do to. :)

For Debian/Ubuntu, we already have some additional machinations to make
ensurepip and such work properly in a distro packaging environment.  We'd want
to be able to do the same for any bundled packages as well.

Cheers,
-Barry

From brett at python.org  Sat Jan 28 13:04:12 2017
From: brett at python.org (Brett Cannon)
Date: Sat, 28 Jan 2017 18:04:12 +0000
Subject: [Python-Dev] re performance
In-Reply-To: <65cf9e6e-7962-d198-6aa7-4302818a8819@mrabarnett.plus.com>
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
 <CAHzaPEO2+BcJUEy15j37d5O72eo1OWewy4PeR0_vcHu0AD6Msg@mail.gmail.com>
 <d9e50354-8ed7-c347-a397-34f0b83927c6@mail.de>
 <f3414629-15b4-17ff-2570-ade73903bbdf@mrabarnett.plus.com>
 <11439969-DBE6-4031-9EC6-3B2452340942@langa.pl>
 <65cf9e6e-7962-d198-6aa7-4302818a8819@mrabarnett.plus.com>
Message-ID: <CAP1=2W7bpP-1TN5=mjRS1Jk7Z-R71su2RhMiksJkDU9chK5ZZA@mail.gmail.com>

On Fri, 27 Jan 2017 at 13:26 MRAB <python at mrabarnett.plus.com> wrote:

> On 2017-01-27 17:03, ?ukasz Langa wrote:
> >
> >> On Jan 26, 2017, at 5:16 PM, MRAB <python at mrabarnett.plus.com
> >> <mailto:python at mrabarnett.plus.com>> wrote:
> >
> >>> So, it seems as if regex already uses a better algorithm although I
> >>> couldn't find any reference to any regex theoretical framework like
> dfa,
> >>> nfa, thompson multiple-state simulation or something.
> >>>
> >> It still uses backtracking, like in the re module.
> >
> > What?s the status of regex inclusion in the stdlib?
> >
> >
> I'm not bothered about it. It's quite a bit bigger than the re module,
> and, anyway, keeping it as a third-party module gives me more freedom to
> make updates, which are available for a range of Python versions.
>

Maybe regex should get a mention in the docs like requests does under
urllib.request?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170128/7659dcb8/attachment.html>

From storchaka at gmail.com  Sun Jan 29 01:13:16 2017
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sun, 29 Jan 2017 08:13:16 +0200
Subject: [Python-Dev] re performance
In-Reply-To: <CAP1=2W7bpP-1TN5=mjRS1Jk7Z-R71su2RhMiksJkDU9chK5ZZA@mail.gmail.com>
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
 <CAHzaPEO2+BcJUEy15j37d5O72eo1OWewy4PeR0_vcHu0AD6Msg@mail.gmail.com>
 <d9e50354-8ed7-c347-a397-34f0b83927c6@mail.de>
 <f3414629-15b4-17ff-2570-ade73903bbdf@mrabarnett.plus.com>
 <11439969-DBE6-4031-9EC6-3B2452340942@langa.pl>
 <65cf9e6e-7962-d198-6aa7-4302818a8819@mrabarnett.plus.com>
 <CAP1=2W7bpP-1TN5=mjRS1Jk7Z-R71su2RhMiksJkDU9chK5ZZA@mail.gmail.com>
Message-ID: <o6k15m$q6v$1@blaine.gmane.org>

On 28.01.17 20:04, Brett Cannon wrote:
> Maybe regex should get a mention in the docs like requests does under
> urllib.request?

https://bugs.python.org/issue22594


From jwilk at jwilk.net  Sun Jan 29 05:18:23 2017
From: jwilk at jwilk.net (Jakub Wilk)
Date: Sun, 29 Jan 2017 11:18:23 +0100
Subject: [Python-Dev] re performance
In-Reply-To: <CAMSv6X3KzYBLuSL=7Zx5mPSaK9y7=-s5m+2EnLuXy6SRYFhPpA@mail.gmail.com>
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
 <CAMSv6X3KzYBLuSL=7Zx5mPSaK9y7=-s5m+2EnLuXy6SRYFhPpA@mail.gmail.com>
Message-ID: <20170129101823.zyu62jccagdtq3tf@jwilk.net>

* Armin Rigo <armin.rigoatgmail.com>, 2017-01-28, 12:44:
>The theoretical kind of regexp is about giving a "yes/no" answer, whereas the 
>concrete "re" or "regexp" modules gives a match object, which lets you ask for 
>the subgroups' location, for example. Strange at it may seem, I am not aware 
>of a way to do that using the linear-time approach of the theory---if it 
>answers "yes", then you have no way of knowing *where* the subgroups matched.
>
>Another issue is that the theoretical engine has no notion of 
>greedy/non-greedy matching.

RE2 has linear execution time, and it supports both capture groups and 
greedy/non-greedy matching.

The implementation is explained in this article:
https://swtch.com/~rsc/regexp/regexp3.html

-- 
Jakub Wilk

From solipsis at pitrou.net  Sun Jan 29 11:13:29 2017
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 29 Jan 2017 17:13:29 +0100
Subject: [Python-Dev] re performance
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
 <CAHzaPEO2+BcJUEy15j37d5O72eo1OWewy4PeR0_vcHu0AD6Msg@mail.gmail.com>
 <d9e50354-8ed7-c347-a397-34f0b83927c6@mail.de>
 <f3414629-15b4-17ff-2570-ade73903bbdf@mrabarnett.plus.com>
 <11439969-DBE6-4031-9EC6-3B2452340942@langa.pl>
 <65cf9e6e-7962-d198-6aa7-4302818a8819@mrabarnett.plus.com>
 <CADiSq7dforv5fY9Y4hgRGJ_6z9R0AY0MM-T=Cai6UmzGmPJdVg@mail.gmail.com>
 <20170128120705.3696f70c@subdivisions.wooz.org>
Message-ID: <20170129171329.73dce91e@fsol>

On Sat, 28 Jan 2017 12:07:05 -0500
Barry Warsaw <barry at python.org> wrote:
> On Jan 28, 2017, at 03:43 PM, Nick Coghlan wrote:
> 
> >I still think it could be a good candidate for a first "bundled"
> >module, where we don't migrate it fully into the CPython development
> >process, but *do* officially bless it and provide it by default in the
> >form of a bundled wheel file (similar to what we do with pip).  
> 
> How would that work exactly.  I.e. is there a PEP?
> 
> While I think it could be a good idea to bundle (more?) third party packages
> for a variety of reasons, I want to make sure it's done in a way that's still
> friendly to downstream repackagers, as I'm sure you do to. :)

That sounds like a lot of effort and maintenance...  Don't we bundle pip
*exactly* so that we don't have to bundle other third-party packages
and instead tell users to "just use `pip install <some package>`"?

To sum it up, how about we simply add an official suggestion to use
regex in the docs?

Regards

Antoine.



From steve at holdenweb.com  Sun Jan 29 15:30:38 2017
From: steve at holdenweb.com (Steve Holden)
Date: Sun, 29 Jan 2017 20:30:38 +0000
Subject: [Python-Dev] re performance
In-Reply-To: <20170129171329.73dce91e@fsol>
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
 <CAHzaPEO2+BcJUEy15j37d5O72eo1OWewy4PeR0_vcHu0AD6Msg@mail.gmail.com>
 <d9e50354-8ed7-c347-a397-34f0b83927c6@mail.de>
 <f3414629-15b4-17ff-2570-ade73903bbdf@mrabarnett.plus.com>
 <11439969-DBE6-4031-9EC6-3B2452340942@langa.pl>
 <65cf9e6e-7962-d198-6aa7-4302818a8819@mrabarnett.plus.com>
 <CADiSq7dforv5fY9Y4hgRGJ_6z9R0AY0MM-T=Cai6UmzGmPJdVg@mail.gmail.com>
 <20170128120705.3696f70c@subdivisions.wooz.org> <20170129171329.73dce91e@fsol>
Message-ID: <CAMofdRDMbF0S99uEqL_Per3heyQHgO52E_thQG+_23fxQhBO2A@mail.gmail.com>

Why not declare re deprecated and remove it in Python 4? I am pretty sure
everyone wants to keep re in all 3.x releases, but that support need not
extend beyond. So Py4 would have no battery for re, but it would (should!)
be common knowledge that regex was the go-to module for general-purpose
pattern matching. If re has advantages in certain situations someone might
upgrade the 3.x implementation and provide it as a 3rd-party module, though
the effort involved would be significant, so someone would have to be
motivated to keep it.

regards
 Steve

Steve Holden

On Sun, Jan 29, 2017 at 4:13 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Sat, 28 Jan 2017 12:07:05 -0500
> Barry Warsaw <barry at python.org> wrote:
> > On Jan 28, 2017, at 03:43 PM, Nick Coghlan wrote:
> >
> > >I still think it could be a good candidate for a first "bundled"
> > >module, where we don't migrate it fully into the CPython development
> > >process, but *do* officially bless it and provide it by default in the
> > >form of a bundled wheel file (similar to what we do with pip).
> >
> > How would that work exactly.  I.e. is there a PEP?
> >
> > While I think it could be a good idea to bundle (more?) third party
> packages
> > for a variety of reasons, I want to make sure it's done in a way that's
> still
> > friendly to downstream repackagers, as I'm sure you do to. :)
>
> That sounds like a lot of effort and maintenance...  Don't we bundle pip
> *exactly* so that we don't have to bundle other third-party packages
> and instead tell users to "just use `pip install <some package>`"?
>
> To sum it up, how about we simply add an official suggestion to use
> regex in the docs?
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> steve%40holdenweb.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170129/1d0132cc/attachment.html>

From p.f.moore at gmail.com  Sun Jan 29 15:52:17 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 29 Jan 2017 20:52:17 +0000
Subject: [Python-Dev] re performance
In-Reply-To: <CAMofdRDMbF0S99uEqL_Per3heyQHgO52E_thQG+_23fxQhBO2A@mail.gmail.com>
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
 <CAHzaPEO2+BcJUEy15j37d5O72eo1OWewy4PeR0_vcHu0AD6Msg@mail.gmail.com>
 <d9e50354-8ed7-c347-a397-34f0b83927c6@mail.de>
 <f3414629-15b4-17ff-2570-ade73903bbdf@mrabarnett.plus.com>
 <11439969-DBE6-4031-9EC6-3B2452340942@langa.pl>
 <65cf9e6e-7962-d198-6aa7-4302818a8819@mrabarnett.plus.com>
 <CADiSq7dforv5fY9Y4hgRGJ_6z9R0AY0MM-T=Cai6UmzGmPJdVg@mail.gmail.com>
 <20170128120705.3696f70c@subdivisions.wooz.org> <20170129171329.73dce91e@fsol>
 <CAMofdRDMbF0S99uEqL_Per3heyQHgO52E_thQG+_23fxQhBO2A@mail.gmail.com>
Message-ID: <CACac1F8nnodRN7CyKUTNvqvz6Rq0adJumP9NPdk0gFShCprs5Q@mail.gmail.com>

On 29 January 2017 at 20:30, Steve Holden <steve at holdenweb.com> wrote:
> Why not declare re deprecated and remove it in Python 4? I am pretty sure
> everyone wants to keep re in all 3.x releases, but that support need not
> extend beyond. So Py4 would have no battery for re, but it would (should!)
> be common knowledge that regex was the go-to module for general-purpose
> pattern matching. If re has advantages in certain situations someone might
> upgrade the 3.x implementation and provide it as a 3rd-party module, though
> the effort involved would be significant, so someone would have to be
> motivated to keep it.

Not having regex capability distributed with Python is a pretty big
regression. There are still a lot of users who don't have access to
3rd party modules.

Paul

From storchaka at gmail.com  Sun Jan 29 16:00:34 2017
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sun, 29 Jan 2017 23:00:34 +0200
Subject: [Python-Dev] re performance
In-Reply-To: <CAMofdRDMbF0S99uEqL_Per3heyQHgO52E_thQG+_23fxQhBO2A@mail.gmail.com>
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
 <CAHzaPEO2+BcJUEy15j37d5O72eo1OWewy4PeR0_vcHu0AD6Msg@mail.gmail.com>
 <d9e50354-8ed7-c347-a397-34f0b83927c6@mail.de>
 <f3414629-15b4-17ff-2570-ade73903bbdf@mrabarnett.plus.com>
 <11439969-DBE6-4031-9EC6-3B2452340942@langa.pl>
 <65cf9e6e-7962-d198-6aa7-4302818a8819@mrabarnett.plus.com>
 <CADiSq7dforv5fY9Y4hgRGJ_6z9R0AY0MM-T=Cai6UmzGmPJdVg@mail.gmail.com>
 <20170128120705.3696f70c@subdivisions.wooz.org>
 <20170129171329.73dce91e@fsol>
 <CAMofdRDMbF0S99uEqL_Per3heyQHgO52E_thQG+_23fxQhBO2A@mail.gmail.com>
Message-ID: <o6ll5d$ip7$1@blaine.gmane.org>

On 29.01.17 22:30, Steve Holden wrote:
> Why not declare re deprecated and remove it in Python 4? I am pretty
> sure everyone wants to keep re in all 3.x releases, but that support
> need not extend beyond. So Py4 would have no battery for re, but it
> would (should!) be common knowledge that regex was the go-to module for
> general-purpose pattern matching. If re has advantages in certain
> situations someone might upgrade the 3.x implementation and provide it
> as a 3rd-party module, though the effort involved would be significant,
> so someone would have to be motivated to keep it.

Regular expressions are used in a number of standard modules and 
scripts. Excluding them from the stdlib would require excluding or 
rewriting the large part of the stdlib.



From storchaka at gmail.com  Sun Jan 29 16:08:20 2017
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sun, 29 Jan 2017 23:08:20 +0200
Subject: [Python-Dev] re performance
In-Reply-To: <20170129101823.zyu62jccagdtq3tf@jwilk.net>
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
 <CAMSv6X3KzYBLuSL=7Zx5mPSaK9y7=-s5m+2EnLuXy6SRYFhPpA@mail.gmail.com>
 <20170129101823.zyu62jccagdtq3tf@jwilk.net>
Message-ID: <o6lljv$fjh$1@blaine.gmane.org>

On 29.01.17 12:18, Jakub Wilk wrote:
> * Armin Rigo <armin.rigoatgmail.com>, 2017-01-28, 12:44:
>> The theoretical kind of regexp is about giving a "yes/no" answer,
>> whereas the concrete "re" or "regexp" modules gives a match object,
>> which lets you ask for the subgroups' location, for example. Strange
>> at it may seem, I am not aware of a way to do that using the
>> linear-time approach of the theory---if it answers "yes", then you
>> have no way of knowing *where* the subgroups matched.
>>
>> Another issue is that the theoretical engine has no notion of
>> greedy/non-greedy matching.
>
> RE2 has linear execution time, and it supports both capture groups and
> greedy/non-greedy matching.
>
> The implementation is explained in this article:
> https://swtch.com/~rsc/regexp/regexp3.html

Not all features of Python regular expressions can be implemented with 
linear complexity. It is possible to compile the part of regular 
expressions to the implementation with linear complexity. Patches are 
welcome.


From greg.ewing at canterbury.ac.nz  Sun Jan 29 16:38:22 2017
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 30 Jan 2017 10:38:22 +1300
Subject: [Python-Dev] re performance
In-Reply-To: <CAMSv6X3KzYBLuSL=7Zx5mPSaK9y7=-s5m+2EnLuXy6SRYFhPpA@mail.gmail.com>
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
 <CAMSv6X3KzYBLuSL=7Zx5mPSaK9y7=-s5m+2EnLuXy6SRYFhPpA@mail.gmail.com>
Message-ID: <588E60CE.1050305@canterbury.ac.nz>

Armin Rigo wrote:

> The theoretical kind of regexp is about giving a "yes/no" answer,
> whereas the concrete "re" or "regexp" modules gives a match object,
> which lets you ask for the subgroups' location, for example.
> 
> Another issue is that the theoretical engine has no notion of
> greedy/non-greedy matching.

These things aren't part of the classical theory of REs that is
usually taught, but it should be possible to do them in linear time.
They can be done for context-free languages using e.g. an LALR parser,
and regular languages are a subset of context-free languages.

-- 
Greg

From rodrigc at freebsd.org  Sun Jan 29 19:14:12 2017
From: rodrigc at freebsd.org (Craig Rodrigues)
Date: Sun, 29 Jan 2017 16:14:12 -0800
Subject: [Python-Dev] Generator objects and list comprehensions?
In-Reply-To: <CAOMjWkmCVZbex40Ls8nDfugQYyWMQTMeBPY+DD_nG9R7fa_QxA@mail.gmail.com>
References: <CAG=rPVco0cnKdA0qDbhnZUctzeVVuJTgN4YX7ua6x8Lb-Q-TUQ@mail.gmail.com>
 <CAFo6DPaP5dcDy=sm4QrauQJDfXgZQYLg0hFdNAwrQ=qM1OwkOw@mail.gmail.com>
 <CAPJVwBkg4hPsJ9axhjDBRvUn20=yd0aOx_8eAJbvSD3R-qZr-g@mail.gmail.com>
 <CAOMjWkmCVZbex40Ls8nDfugQYyWMQTMeBPY+DD_nG9R7fa_QxA@mail.gmail.com>
Message-ID: <CAG=rPVcab4amWESWhu4cHBUfmsE9vHLL7G-oYeQA2p83bwmZDQ@mail.gmail.com>

On Thu, Jan 26, 2017 at 4:09 AM, Ivan Levkivskyi <levkivskyi at gmail.com>
wrote:

>
>
> Concerning list/set/dict comprehensions, I am much more in favor of making
> comprehensions simply equivalent to for-loops (more or less like you
> proposed using yield from). The only reason to introduce auxiliary function
> scope was to prevent the loop variables from leaking outside comprehensions.
> Formally, this is indeed backward incompatible, but I doubt many people
> depend on the current counter-intuitive behavior.
>
> Concerning generator expressions, probably it is indeed better to simply
> prohibit yield inside them.
>
>
Thank you to everyone who responded to my post and provided excellent
analysis.

For Python, I don't know what the best way to proceed is:

OPTION 1
========

Make a SyntaxError:  [(yield 1) for x in range(10)]
and update the documentation to explain that this is an invalid construct.

This would have certainly helped me identify the source of the problem as I
tried porting buildbot 0.9 to Python 3.

However, while not very common, there is Python 2.x code that uses that.
I found these cases in the buildbot code which I changed so as to work on
Python 2 and 3:

https://github.com/buildbot/buildbot/pull/2661
https://github.com/buildbot/buildbot/pull/2673


 OPTION 2
=========
Make this return a list on Python 3, like in Python 2:  [
(yield 1) for x in range(10)]

As pointed out by others on the this mailing list, there are some
problems associated with that.  I don't know if there are many Python 2
codebases out there
with this construct, but it would be nice to have one less Python 2 -> 3
porting gotcha.


I'm OK with either approach.  Leaving things the way they are in Python 3
is no good, IMHO.
--
Craig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170129/da3909bd/attachment.html>

From ncoghlan at gmail.com  Mon Jan 30 06:38:11 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 30 Jan 2017 12:38:11 +0100
Subject: [Python-Dev] re performance
In-Reply-To: <20170128120705.3696f70c@subdivisions.wooz.org>
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
 <CAHzaPEO2+BcJUEy15j37d5O72eo1OWewy4PeR0_vcHu0AD6Msg@mail.gmail.com>
 <d9e50354-8ed7-c347-a397-34f0b83927c6@mail.de>
 <f3414629-15b4-17ff-2570-ade73903bbdf@mrabarnett.plus.com>
 <11439969-DBE6-4031-9EC6-3B2452340942@langa.pl>
 <65cf9e6e-7962-d198-6aa7-4302818a8819@mrabarnett.plus.com>
 <CADiSq7dforv5fY9Y4hgRGJ_6z9R0AY0MM-T=Cai6UmzGmPJdVg@mail.gmail.com>
 <20170128120705.3696f70c@subdivisions.wooz.org>
Message-ID: <CADiSq7dYxHka1s7f35hHzHLHfYC+tBWBvuoC-QBKRfrUvPW2Yw@mail.gmail.com>

On 28 January 2017 at 18:07, Barry Warsaw <barry at python.org> wrote:
> On Jan 28, 2017, at 03:43 PM, Nick Coghlan wrote:
>
>>I still think it could be a good candidate for a first "bundled"
>>module, where we don't migrate it fully into the CPython development
>>process, but *do* officially bless it and provide it by default in the
>>form of a bundled wheel file (similar to what we do with pip).
>
> How would that work exactly.  I.e. is there a PEP?

There's no PEP, and I don't think it would make sense to have one
without a specific concrete example to drive the discussion.

I think there are 3 main candidates that could fit that bill:

- requests
- setuptools
- regex

Using requests to drive it would bring in a lot of incidental
complexity around SSL/TLS certificate management and integrating with
platform trust stores, which I think would overshadow the core policy
discussion of switching from a pure co-development model for the
standard library to one where some components are independently
versioned with recent snapshots being included in CPython feature and
maintenance branches.

For setuptools, one of our main goals in PyPA is to make it just one
build system option amongst many, as well as ensuring that it's easy
to do service deployments that don't have any build system present at
all, so bundling it by default with the stdlib would run counter to
those goals

regex by contrast is a relatively simple standalone module that offers
a more actively developed alternative to the standard library's re
module, but would still require a PEP to address the following
technical and policy questions that are normally handled by copying
code wholesale into the CPython tree under a contributor license
agreement:

- what licensing and other IP assurances would the PSF need to be
comfortable redistributing the bundled component?
- what maintenance sustainability questions would need to be
addressed? (e.g. succession planning for the ability to publish new
releases)
- how would bundled modules be handled in the CPython test suite?
- how would bundled modules be handled during local development,
during installation on supported OSes, and when creating virtual
environments?
- how does platform support in bundled modules relate to the platform
support guidelines given in PEP 11?
- how are additional build requirements for bundled modules (e.g. C++
compilers) handled?
- would bundled modules be permitted to gain features in CPython
maintenance releases, or would we expect any bundled modules to offer
conservative maintenance branches for the life of a given CPython
feature release?
- how would maintenance collaboration and issue escalation work? Would
bundled module maintainers need to follow both their own issue tracker
and the CPython one?

In many respects, PEP 453 and the bundling of pip already offers
precedent for how these are handled (e.g. it's collectively maintained
by PyPA, and that group is as closely associated with the PSF from a
legal and support perspective as python-dev is), so any PEP along
these lines would be about deciding which parts of what we did for pip
were a technical and policy precedent for 3rd party module bundling in
general, and which are unique to pip.

> While I think it could be a good idea to bundle (more?) third party packages
> for a variety of reasons, I want to make sure it's done in a way that's still
> friendly to downstream repackagers, as I'm sure you do to. :)

Indeed :)

> For Debian/Ubuntu, we already have some additional machinations to make
> ensurepip and such work properly in a distro packaging environment.  We'd want
> to be able to do the same for any bundled packages as well.

Yeah, I suspect any development along these lines would make it even
more important to have something like PEP 534 in place so that the
bundled modules could be omitted by default in some cases (such as in
virtual environments), and require people to depend on them explicitly
if they needed them.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Mon Jan 30 06:56:41 2017
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 30 Jan 2017 12:56:41 +0100
Subject: [Python-Dev] re performance
In-Reply-To: <CAMofdRDMbF0S99uEqL_Per3heyQHgO52E_thQG+_23fxQhBO2A@mail.gmail.com>
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
 <CAHzaPEO2+BcJUEy15j37d5O72eo1OWewy4PeR0_vcHu0AD6Msg@mail.gmail.com>
 <d9e50354-8ed7-c347-a397-34f0b83927c6@mail.de>
 <f3414629-15b4-17ff-2570-ade73903bbdf@mrabarnett.plus.com>
 <11439969-DBE6-4031-9EC6-3B2452340942@langa.pl>
 <65cf9e6e-7962-d198-6aa7-4302818a8819@mrabarnett.plus.com>
 <CADiSq7dforv5fY9Y4hgRGJ_6z9R0AY0MM-T=Cai6UmzGmPJdVg@mail.gmail.com>
 <20170128120705.3696f70c@subdivisions.wooz.org>
 <20170129171329.73dce91e@fsol>
 <CAMofdRDMbF0S99uEqL_Per3heyQHgO52E_thQG+_23fxQhBO2A@mail.gmail.com>
Message-ID: <20170130125641.7525fc2f@fsol>

On Sun, 29 Jan 2017 20:30:38 +0000
Steve Holden <steve at holdenweb.com> wrote:
> Why not declare re deprecated and remove it in Python 4?

Why deprecate and remove a library that's perfectly usable and
satisfactory for the vast majority of regular expression usage?  It's
not like regex presents a radically different API...

Do we really have to face another wave of stdlib bashing on this
mailing-list?

Regards

Antoine.

From berker.peksag at gmail.com  Mon Jan 30 07:35:34 2017
From: berker.peksag at gmail.com (=?UTF-8?Q?Berker_Peksa=C4=9F?=)
Date: Mon, 30 Jan 2017 15:35:34 +0300
Subject: [Python-Dev] re performance
In-Reply-To: <20170130125641.7525fc2f@fsol>
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
 <CAHzaPEO2+BcJUEy15j37d5O72eo1OWewy4PeR0_vcHu0AD6Msg@mail.gmail.com>
 <d9e50354-8ed7-c347-a397-34f0b83927c6@mail.de>
 <f3414629-15b4-17ff-2570-ade73903bbdf@mrabarnett.plus.com>
 <11439969-DBE6-4031-9EC6-3B2452340942@langa.pl>
 <65cf9e6e-7962-d198-6aa7-4302818a8819@mrabarnett.plus.com>
 <CADiSq7dforv5fY9Y4hgRGJ_6z9R0AY0MM-T=Cai6UmzGmPJdVg@mail.gmail.com>
 <20170128120705.3696f70c@subdivisions.wooz.org> <20170129171329.73dce91e@fsol>
 <CAMofdRDMbF0S99uEqL_Per3heyQHgO52E_thQG+_23fxQhBO2A@mail.gmail.com>
 <20170130125641.7525fc2f@fsol>
Message-ID: <CAF4280JdBc0XougSVdyekWATdyrHSLHU1kF4WHLkpCsXJw9NaQ@mail.gmail.com>

On Mon, Jan 30, 2017 at 2:56 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sun, 29 Jan 2017 20:30:38 +0000
> Steve Holden <steve at holdenweb.com> wrote:
>> Why not declare re deprecated and remove it in Python 4?
>
> Why deprecate and remove a library that's perfectly usable and
> satisfactory for the vast majority of regular expression usage?  It's
> not like regex presents a radically different API...
>
> Do we really have to face another wave of stdlib bashing on this
> mailing-list?

I agree with Antoine. And note that re has an active maintainer who
has fixed a bunch of bugs and added some new features in the last few
years.

--Berker

From cournape at gmail.com  Mon Jan 30 08:53:15 2017
From: cournape at gmail.com (David Cournapeau)
Date: Mon, 30 Jan 2017 13:53:15 +0000
Subject: [Python-Dev] SSL certificates recommendations for downstream python
 packagers
Message-ID: <CAGY4rcUfNpcyX2fB=h-AAkL1tTZm7zsx840Rpie55NNYF3OV2Q@mail.gmail.com>

Hi,

I am managing the team responsible for providing python packaging at
Enthought, and I would like to make sure we are providing a good (and
secure) out of the box experience for SSL.

My understanding is that PEP 476 is the latest PEP that concerns this
issue, and that PEP recommends using the system store:
https://www.python.org/dev/peps/pep-0476/#trust-database. But looking at
binary python distributions from python.org, that does not seem to a.ways
be the case. I looked at the following:

* 3.5.3 from python.org for OS X (64 bits): this uses the old, system
openssl
* 3.6.0 from python.org for OS X: this embeds a recent openssl, but ssl
seems to be configured to use non existing paths
(ssl..get_default_verify_paths()), and indeed, cert validation seems to
fail by default with those installers
* 3.6.0 from python.org for windows: I have not found how the ssl module
finds the certificate, but certification seems to work

Are there any official recommendations for downstream packagers beyond PEP
476 ? Is it "acceptable" for downstream packagers to patch python's default
cert locations ?

David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170130/5f51d0d4/attachment-0001.html>

From barry at python.org  Mon Jan 30 09:26:37 2017
From: barry at python.org (Barry Warsaw)
Date: Mon, 30 Jan 2017 09:26:37 -0500
Subject: [Python-Dev] re performance
In-Reply-To: <CADiSq7dYxHka1s7f35hHzHLHfYC+tBWBvuoC-QBKRfrUvPW2Yw@mail.gmail.com>
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
 <CAHzaPEO2+BcJUEy15j37d5O72eo1OWewy4PeR0_vcHu0AD6Msg@mail.gmail.com>
 <d9e50354-8ed7-c347-a397-34f0b83927c6@mail.de>
 <f3414629-15b4-17ff-2570-ade73903bbdf@mrabarnett.plus.com>
 <11439969-DBE6-4031-9EC6-3B2452340942@langa.pl>
 <65cf9e6e-7962-d198-6aa7-4302818a8819@mrabarnett.plus.com>
 <CADiSq7dforv5fY9Y4hgRGJ_6z9R0AY0MM-T=Cai6UmzGmPJdVg@mail.gmail.com>
 <20170128120705.3696f70c@subdivisions.wooz.org>
 <CADiSq7dYxHka1s7f35hHzHLHfYC+tBWBvuoC-QBKRfrUvPW2Yw@mail.gmail.com>
Message-ID: <20170130092637.4665eaea@presto.wooz.org>

On Jan 30, 2017, at 12:38 PM, Nick Coghlan wrote:

>I think there are 3 main candidates that could fit that bill:
>
>- requests
>- setuptools
>- regex

Actually, I think pkg_resources would make an excellent candidate.  The
setuptools crew is working on a branch that would allow for setuptools and
pkg_resources to be split, which would be great for other reasons.  Splitting
them may mean that pkg_resources could eventually be added to the stdlib, but
as an intermediate step, it could also test out this idea.  It probably has a
lot less of the baggage that you outline.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 801 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170130/a2f4c2f0/attachment.sig>

From brett at python.org  Mon Jan 30 13:05:30 2017
From: brett at python.org (Brett Cannon)
Date: Mon, 30 Jan 2017 18:05:30 +0000
Subject: [Python-Dev] Generator objects and list comprehensions?
In-Reply-To: <CAG=rPVcab4amWESWhu4cHBUfmsE9vHLL7G-oYeQA2p83bwmZDQ@mail.gmail.com>
References: <CAG=rPVco0cnKdA0qDbhnZUctzeVVuJTgN4YX7ua6x8Lb-Q-TUQ@mail.gmail.com>
 <CAFo6DPaP5dcDy=sm4QrauQJDfXgZQYLg0hFdNAwrQ=qM1OwkOw@mail.gmail.com>
 <CAPJVwBkg4hPsJ9axhjDBRvUn20=yd0aOx_8eAJbvSD3R-qZr-g@mail.gmail.com>
 <CAOMjWkmCVZbex40Ls8nDfugQYyWMQTMeBPY+DD_nG9R7fa_QxA@mail.gmail.com>
 <CAG=rPVcab4amWESWhu4cHBUfmsE9vHLL7G-oYeQA2p83bwmZDQ@mail.gmail.com>
Message-ID: <CAP1=2W6UZX6J1Zyd9L6UTtHC-a88mwQ3P9k_JgRwJvZ4Dqqh7A@mail.gmail.com>

On Sun, 29 Jan 2017 at 16:39 Craig Rodrigues <rodrigc at freebsd.org> wrote:

> On Thu, Jan 26, 2017 at 4:09 AM, Ivan Levkivskyi <levkivskyi at gmail.com>
> wrote:
>
>
>
> Concerning list/set/dict comprehensions, I am much more in favor of making
> comprehensions simply equivalent to for-loops (more or less like you
> proposed using yield from). The only reason to introduce auxiliary function
> scope was to prevent the loop variables from leaking outside comprehensions.
> Formally, this is indeed backward incompatible, but I doubt many people
> depend on the current counter-intuitive behavior.
>
> Concerning generator expressions, probably it is indeed better to simply
> prohibit yield inside them.
>
>
> Thank you to everyone who responded to my post and provided excellent
> analysis.
>
> For Python, I don't know what the best way to proceed is:
>
> OPTION 1
> ========
>
> Make a SyntaxError:  [(yield 1) for x in range(10)]
> and update the documentation to explain that this is an invalid construct.
>
> This would have certainly helped me identify the source of the problem as
> I tried porting buildbot 0.9 to Python 3.
>
> However, while not very common, there is Python 2.x code that uses that.
> I found these cases in the buildbot code which I changed so as to work on
> Python 2 and 3:
>
> https://github.com/buildbot/buildbot/pull/2661
> https://github.com/buildbot/buildbot/pull/2673
>
>
>  OPTION 2
> =========
> Make this return a list on Python 3, like in Python 2:  [
> (yield 1) for x in range(10)]
>
> As pointed out by others on the this mailing list, there are some
> problems associated with that.  I don't know if there are many Python 2
> codebases out there
> with this construct, but it would be nice to have one less Python 2 -> 3
> porting gotcha.
>
>
> I'm OK with either approach.  Leaving things the way they are in Python 3
> is no good, IMHO.
>

My vote is it be a SyntaxError since you're not getting what you expect
from the syntax.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170130/ab1a3c2a/attachment.html>

From brett at python.org  Mon Jan 30 13:14:35 2017
From: brett at python.org (Brett Cannon)
Date: Mon, 30 Jan 2017 18:14:35 +0000
Subject: [Python-Dev] modules people want installed automatically (was: Re:
 re performance)
In-Reply-To: <20170130092637.4665eaea@presto.wooz.org>
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
 <CAHzaPEO2+BcJUEy15j37d5O72eo1OWewy4PeR0_vcHu0AD6Msg@mail.gmail.com>
 <d9e50354-8ed7-c347-a397-34f0b83927c6@mail.de>
 <f3414629-15b4-17ff-2570-ade73903bbdf@mrabarnett.plus.com>
 <11439969-DBE6-4031-9EC6-3B2452340942@langa.pl>
 <65cf9e6e-7962-d198-6aa7-4302818a8819@mrabarnett.plus.com>
 <CADiSq7dforv5fY9Y4hgRGJ_6z9R0AY0MM-T=Cai6UmzGmPJdVg@mail.gmail.com>
 <20170128120705.3696f70c@subdivisions.wooz.org>
 <CADiSq7dYxHka1s7f35hHzHLHfYC+tBWBvuoC-QBKRfrUvPW2Yw@mail.gmail.com>
 <20170130092637.4665eaea@presto.wooz.org>
Message-ID: <CAP1=2W6oraajQt+7wiXcKv0Pec64BiES_8zBu40rfFKWuyox7Q@mail.gmail.com>

On Mon, 30 Jan 2017 at 06:27 Barry Warsaw <barry at python.org> wrote:

> On Jan 30, 2017, at 12:38 PM, Nick Coghlan wrote:
>
> >I think there are 3 main candidates that could fit that bill:
> >
> >- requests
> >- setuptools
> >- regex
>
> Actually, I think pkg_resources would make an excellent candidate.  The
> setuptools crew is working on a branch that would allow for setuptools and
> pkg_resources to be split, which would be great for other reasons.
> Splitting
> them may mean that pkg_resources could eventually be added to the stdlib,
> but
> as an intermediate step, it could also test out this idea.  It probably
> has a
> lot less of the baggage that you outline.


What functionality are you after here from pkg_resources? If it's the
reading of data files then you will get that in the stdlib through
importlib as soon as I'm not working on getting our workflow to work
through GitHub's web UI (which obviously includes the migration itself).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170130/2dbfa78e/attachment.html>

From barry at python.org  Mon Jan 30 13:26:41 2017
From: barry at python.org (Barry Warsaw)
Date: Mon, 30 Jan 2017 13:26:41 -0500
Subject: [Python-Dev] modules people want installed automatically (was:
 Re: re performance)
In-Reply-To: <CAP1=2W6oraajQt+7wiXcKv0Pec64BiES_8zBu40rfFKWuyox7Q@mail.gmail.com>
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
 <CAHzaPEO2+BcJUEy15j37d5O72eo1OWewy4PeR0_vcHu0AD6Msg@mail.gmail.com>
 <d9e50354-8ed7-c347-a397-34f0b83927c6@mail.de>
 <f3414629-15b4-17ff-2570-ade73903bbdf@mrabarnett.plus.com>
 <11439969-DBE6-4031-9EC6-3B2452340942@langa.pl>
 <65cf9e6e-7962-d198-6aa7-4302818a8819@mrabarnett.plus.com>
 <CADiSq7dforv5fY9Y4hgRGJ_6z9R0AY0MM-T=Cai6UmzGmPJdVg@mail.gmail.com>
 <20170128120705.3696f70c@subdivisions.wooz.org>
 <CADiSq7dYxHka1s7f35hHzHLHfYC+tBWBvuoC-QBKRfrUvPW2Yw@mail.gmail.com>
 <20170130092637.4665eaea@presto.wooz.org>
 <CAP1=2W6oraajQt+7wiXcKv0Pec64BiES_8zBu40rfFKWuyox7Q@mail.gmail.com>
Message-ID: <20170130132641.6375da72@presto>

On Jan 30, 2017, at 06:14 PM, Brett Cannon wrote:

>What functionality are you after here from pkg_resources? If it's the
>reading of data files then you will get that in the stdlib through
>importlib as soon as I'm not working on getting our workflow to work
>through GitHub's web UI (which obviously includes the migration itself).

http://setuptools.readthedocs.io/en/latest/pkg_resources.html#basic-resource-access

Mostly I use:

* resource_filename()
* resource_string() (really, resource_bytes!)
* resource_stream() (although I'd really like a more open()-like API)

This might fall under a simpler PEP 365.  Also, while I would love to have
these available say in importlib, I also like to consider a backward
compatible API where the stdlib provides the `pkg_resources` module name.
That's not totally required though because you can play ImportError games
until Python 3.7 (presumably) is in widespread -and only- use.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 801 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170130/b1b0a1be/attachment.sig>

From lukasz at langa.pl  Mon Jan 30 13:32:33 2017
From: lukasz at langa.pl (Lukasz Langa)
Date: Mon, 30 Jan 2017 10:32:33 -0800
Subject: [Python-Dev] re performance
In-Reply-To: <20170130092637.4665eaea@presto.wooz.org>
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
 <CAHzaPEO2+BcJUEy15j37d5O72eo1OWewy4PeR0_vcHu0AD6Msg@mail.gmail.com>
 <d9e50354-8ed7-c347-a397-34f0b83927c6@mail.de>
 <f3414629-15b4-17ff-2570-ade73903bbdf@mrabarnett.plus.com>
 <11439969-DBE6-4031-9EC6-3B2452340942@langa.pl>
 <65cf9e6e-7962-d198-6aa7-4302818a8819@mrabarnett.plus.com>
 <CADiSq7dforv5fY9Y4hgRGJ_6z9R0AY0MM-T=Cai6UmzGmPJdVg@mail.gmail.com>
 <20170128120705.3696f70c@subdivisions.wooz.org>
 <CADiSq7dYxHka1s7f35hHzHLHfYC+tBWBvuoC-QBKRfrUvPW2Yw@mail.gmail.com>
 <20170130092637.4665eaea@presto.wooz.org>
Message-ID: <622FFA29-2974-4248-B4AF-E1531DA26710@langa.pl>


> On Jan 30, 2017, at 6:26 AM, Barry Warsaw <barry at python.org> wrote:
> 
> Actually, I think pkg_resources would make an excellent candidate.  The
> setuptools crew is working on a branch that would allow for setuptools and
> pkg_resources to be split, which would be great for other reasons.  Splitting
> them may mean that pkg_resources could eventually be added to the stdlib, but
> as an intermediate step, it could also test out this idea.  It probably has a
> lot less of the baggage that you outline.

FWIW this is what we do at Facebook. We ship pkg_resources with everything but not setuptools. So there's definitely value in that. It's a little surprising we haven't been doing that yet, given that zipped applications need it.

- ?

From christian at python.org  Mon Jan 30 13:44:38 2017
From: christian at python.org (Christian Heimes)
Date: Mon, 30 Jan 2017 19:44:38 +0100
Subject: [Python-Dev] SSL certificates recommendations for downstream
 python packagers
In-Reply-To: <CAGY4rcUfNpcyX2fB=h-AAkL1tTZm7zsx840Rpie55NNYF3OV2Q@mail.gmail.com>
References: <CAGY4rcUfNpcyX2fB=h-AAkL1tTZm7zsx840Rpie55NNYF3OV2Q@mail.gmail.com>
Message-ID: <adb14e71-a9b3-41b7-f5e2-b9aec046ffd8@python.org>

On 2017-01-30 14:53, David Cournapeau wrote:
> Hi,
> 
> I am managing the team responsible for providing python packaging at
> Enthought, and I would like to make sure we are providing a good (and
> secure) out of the box experience for SSL.
> 
> My understanding is that PEP 476 is the latest PEP that concerns this
> issue, and that PEP recommends using the system store:
> https://www.python.org/dev/peps/pep-0476/#trust-database. But looking at
> binary python distributions from python.org <http://python.org>, that
> does not seem to a.ways be the case. I looked at the following:
> 
> * 3.5.3 from python.org <http://python.org> for OS X (64 bits): this
> uses the old, system openssl

On macOS, system OpenSSL uses TEA to look up trust anchors from the
keyring, https://hynek.me/articles/apple-openssl-verification-surprises/

> * 3.6.0 from python.org <http://python.org> for OS X: this embeds a
> recent openssl, but ssl seems to be configured to use non existing paths
> (ssl..get_default_verify_paths()), and indeed, cert validation seems to
> fail by default with those installers

I guess you haven't read the warning on the download page,
https://www.python.org/downloads/release/python-360/ :)

tl;dr Python on macOS can no longer use the keyring for certificates.
You have to install certifi. Personally I (as a maintainer of the ssl
module) do neither like certifi nor the approach taken. It's wrong and
dangerous because certifi by-passes system policies and does not include
local CAs. But it's the best we got.

> * 3.6.0 from python.org <http://python.org> for windows: I have not
> found how the ssl module finds the certificate, but certification seems
> to work

The magic happens in load_default_certs(),
https://hg.python.org/cpython/file/tip/Lib/ssl.py#l445 . The approach
does work most of the time. Sometimes it fails because Windows download
root CA's on demand. The code does not trigger a download.

Steve and I have some ideas to improve the situation,
https://bugs.python.org/issue28747 .

> 
> Are there any official recommendations for downstream packagers beyond
> PEP 476 ? Is it "acceptable" for downstream packagers to patch python's
> default cert locations ?

My semi-official recommendations are:

* On Linux and BSD, use OpenSSL from your vendor (Red Hat, SuSE, Ubuntu,
Debian, Arch, Gentoo, ...). Vendors compile OpenSSL with special flags
to make get_default_verify_paths() work.

* If your vendor prefers LibreSSL, you might run into minor issue.
LibreSSL is not fully compatible to OpenSSL. Your vendor also compiles
LibresSL with special flags.

* Windows kinda works. We are going to improve the situation for 3.7.

* On macOS ... well, please contact Apple service and demand a solution.
Cory Benfield's TLS ABC PEP and Paul Kehrer's work on SecureTransport
bindings for Python will let you use Apple's native TLS lib (with some
restrictions regarding fork()). In the mean time you are screwed and
have to rely on certifi.

As last resort, use certifi. You HAVE to make sure to keep it up to date
and you have to deal with local policies and CAs yourself.

Christian

From cory at lukasa.co.uk  Mon Jan 30 15:50:59 2017
From: cory at lukasa.co.uk (Cory Benfield)
Date: Mon, 30 Jan 2017 20:50:59 +0000
Subject: [Python-Dev] SSL certificates recommendations for downstream
 python packagers
In-Reply-To: <CAGY4rcUfNpcyX2fB=h-AAkL1tTZm7zsx840Rpie55NNYF3OV2Q@mail.gmail.com>
References: <CAGY4rcUfNpcyX2fB=h-AAkL1tTZm7zsx840Rpie55NNYF3OV2Q@mail.gmail.com>
Message-ID: <8296B2E6-5444-43B0-8C71-6398DDDCB74A@lukasa.co.uk>



> On 30 Jan 2017, at 13:53, David Cournapeau <cournape at gmail.com> wrote:
> 
> Are there any official recommendations for downstream packagers beyond PEP 476 ? Is it "acceptable" for downstream packagers to patch python's default cert locations ?

There *are* no default cert locations on Windows or macOS that can be accessed by OpenSSL.

I cannot stress this strongly enough: you cannot provide a platform-native certificate validation logic for Python *and* use OpenSSL for certificate validation on Windows or macOS. (macOS can technically do this when you link against the system OpenSSL, at the cost of using a catastrophically insecure version of OpenSSL.) 

The only program I am aware of that does platform-native certificate validation on all three major desktop OS platforms is Chrome. It does this using a fork of OpenSSL to do the actual TLS, but the platform-native crypto library to do the certificate validation. This is the only acceptable way to do this, and Python does not expose the appropriate hooks to do it from within Python code. This would require that you carry substantial patches to the standard library to achieve this, all of which would be custom code. I strongly recommend you don't undertake to do this unless you are very confident of your ability to write this code correctly.

The best long term solution to this is to stop using OpenSSL on platforms that don't consider it the 'blessed' approach. If you're interested in following that work, we're currently discussing it on the security-SIG, and you'd be welcome to join. 

Cory

From cournape at gmail.com  Mon Jan 30 16:00:21 2017
From: cournape at gmail.com (David Cournapeau)
Date: Mon, 30 Jan 2017 21:00:21 +0000
Subject: [Python-Dev] SSL certificates recommendations for downstream
 python packagers
In-Reply-To: <8296B2E6-5444-43B0-8C71-6398DDDCB74A@lukasa.co.uk>
References: <CAGY4rcUfNpcyX2fB=h-AAkL1tTZm7zsx840Rpie55NNYF3OV2Q@mail.gmail.com>
 <8296B2E6-5444-43B0-8C71-6398DDDCB74A@lukasa.co.uk>
Message-ID: <CAGY4rcUV1iqxdeqcy52AmHX1BAWEAAsYnpS--CSDkMtyOdSd0Q@mail.gmail.com>

On Mon, Jan 30, 2017 at 8:50 PM, Cory Benfield <cory at lukasa.co.uk> wrote:

>
>
> > On 30 Jan 2017, at 13:53, David Cournapeau <cournape at gmail.com> wrote:
> >
> > Are there any official recommendations for downstream packagers beyond
> PEP 476 ? Is it "acceptable" for downstream packagers to patch python's
> default cert locations ?
>
> There *are* no default cert locations on Windows or macOS that can be
> accessed by OpenSSL.
>
> I cannot stress this strongly enough: you cannot provide a platform-native
> certificate validation logic for Python *and* use OpenSSL for certificate
> validation on Windows or macOS. (macOS can technically do this when you
> link against the system OpenSSL, at the cost of using a catastrophically
> insecure version of OpenSSL.)
>

Ah, thanks, that's already useful information.

Just making sure I understand: this means there is no way to use python's
SSL library to use the system store on windows, in particular private
certifications that are often deployed by internal ITs in large orgs ?


> The only program I am aware of that does platform-native certificate
> validation on all three major desktop OS platforms is Chrome. It does this
> using a fork of OpenSSL to do the actual TLS, but the platform-native
> crypto library to do the certificate validation. This is the only
> acceptable way to do this, and Python does not expose the appropriate hooks
> to do it from within Python code. This would require that you carry
> substantial patches to the standard library to achieve this, all of which
> would be custom code. I strongly recommend you don't undertake to do this
> unless you are very confident of your ability to write this code correctly.
>

That's exactly what I was afraid of and why I asked before attempting
anything.


>
> The best long term solution to this is to stop using OpenSSL on platforms
> that don't consider it the 'blessed' approach. If you're interested in
> following that work, we're currently discussing it on the security-SIG, and
> you'd be welcome to join.
>

Thanks, I will see if it looks like I have anything to contribute.

David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170130/c88d16cc/attachment.html>

From cournape at gmail.com  Mon Jan 30 16:05:33 2017
From: cournape at gmail.com (David Cournapeau)
Date: Mon, 30 Jan 2017 21:05:33 +0000
Subject: [Python-Dev] SSL certificates recommendations for downstream
 python packagers
In-Reply-To: <8296B2E6-5444-43B0-8C71-6398DDDCB74A@lukasa.co.uk>
References: <CAGY4rcUfNpcyX2fB=h-AAkL1tTZm7zsx840Rpie55NNYF3OV2Q@mail.gmail.com>
 <8296B2E6-5444-43B0-8C71-6398DDDCB74A@lukasa.co.uk>
Message-ID: <CAGY4rcVTt5=b2_07L85ea8U2mqK0BZm6_awx3vug9KNHd+bp=Q@mail.gmail.com>

On Mon, Jan 30, 2017 at 8:50 PM, Cory Benfield <cory at lukasa.co.uk> wrote:

>
>
> > On 30 Jan 2017, at 13:53, David Cournapeau <cournape at gmail.com> wrote:
> >
> > Are there any official recommendations for downstream packagers beyond
> PEP 476 ? Is it "acceptable" for downstream packagers to patch python's
> default cert locations ?
>
> There *are* no default cert locations on Windows or macOS that can be
> accessed by OpenSSL.
>

Also, doesn't that contradict the wording of PEP 476, specifically " Python
would use the system provided certificate database on all platforms.
Failure to locate such a database would be an error, and users would need
to explicitly specify a location to fix it." ?

Or is that PEP a long term goal, and not a description of the current
status ?

David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170130/27da721e/attachment-0001.html>

From christian at python.org  Mon Jan 30 16:10:21 2017
From: christian at python.org (Christian Heimes)
Date: Mon, 30 Jan 2017 22:10:21 +0100
Subject: [Python-Dev] SSL certificates recommendations for downstream
 python packagers
In-Reply-To: <8296B2E6-5444-43B0-8C71-6398DDDCB74A@lukasa.co.uk>
References: <CAGY4rcUfNpcyX2fB=h-AAkL1tTZm7zsx840Rpie55NNYF3OV2Q@mail.gmail.com>
 <8296B2E6-5444-43B0-8C71-6398DDDCB74A@lukasa.co.uk>
Message-ID: <4cece125-61c5-ec85-7137-f425badb1bc6@python.org>

On 2017-01-30 21:50, Cory Benfield wrote:
> 
> 
>> On 30 Jan 2017, at 13:53, David Cournapeau <cournape at gmail.com> wrote:
>>
>> Are there any official recommendations for downstream packagers beyond PEP 476 ? Is it "acceptable" for downstream packagers to patch python's default cert locations ?
> 
> There *are* no default cert locations on Windows or macOS that can be accessed by OpenSSL.
> 
> I cannot stress this strongly enough: you cannot provide a platform-native certificate validation logic for Python *and* use OpenSSL for certificate validation on Windows or macOS. (macOS can technically do this when you link against the system OpenSSL, at the cost of using a catastrophically insecure version of OpenSSL.) 

In theory it is possible for Python and OpenSSL, too. I looked into a
custom X509_LOOKUP_METHOD to locate trust anchors by subject. Steve is
trying an alternative approach in https://bugs.python.org/issue28747. It
ain't pretty and we are not there yet, too. Native support for SChannel
and SecureTransport has some benefits.

It's too bad OpenSSL lacks support for PKCS#11 Trust Assertion Objects.
We could use
https://p11-glue.freedesktop.org/doc/pkcs11-trust-assertions/#pkcs11-objects
under Linux and the PKCS#11 under Windows and macOS.

Christian

From christian at python.org  Mon Jan 30 16:14:13 2017
From: christian at python.org (Christian Heimes)
Date: Mon, 30 Jan 2017 22:14:13 +0100
Subject: [Python-Dev] SSL certificates recommendations for downstream
 python packagers
In-Reply-To: <CAGY4rcUV1iqxdeqcy52AmHX1BAWEAAsYnpS--CSDkMtyOdSd0Q@mail.gmail.com>
References: <CAGY4rcUfNpcyX2fB=h-AAkL1tTZm7zsx840Rpie55NNYF3OV2Q@mail.gmail.com>
 <8296B2E6-5444-43B0-8C71-6398DDDCB74A@lukasa.co.uk>
 <CAGY4rcUV1iqxdeqcy52AmHX1BAWEAAsYnpS--CSDkMtyOdSd0Q@mail.gmail.com>
Message-ID: <b8b24e2a-4d45-9531-a4de-52a8850caabc@python.org>

On 2017-01-30 22:00, David Cournapeau wrote:
> 
> 
> On Mon, Jan 30, 2017 at 8:50 PM, Cory Benfield <cory at lukasa.co.uk
> <mailto:cory at lukasa.co.uk>> wrote:
> 
> 
> 
>     > On 30 Jan 2017, at 13:53, David Cournapeau <cournape at gmail.com <mailto:cournape at gmail.com>> wrote:
>     >
>     > Are there any official recommendations for downstream packagers beyond PEP 476 ? Is it "acceptable" for downstream packagers to patch python's default cert locations ?
> 
>     There *are* no default cert locations on Windows or macOS that can
>     be accessed by OpenSSL.
> 
>     I cannot stress this strongly enough: you cannot provide a
>     platform-native certificate validation logic for Python *and* use
>     OpenSSL for certificate validation on Windows or macOS. (macOS can
>     technically do this when you link against the system OpenSSL, at the
>     cost of using a catastrophically insecure version of OpenSSL.)
> 
> 
> Ah, thanks, that's already useful information.
> 
> Just making sure I understand: this means there is no way to use
> python's SSL library to use the system store on windows, in particular
> private certifications that are often deployed by internal ITs in large
> orgs ?

That works with CPython because we get all trust anchors from the cert
store. However Python is not able to retrieve *additional* certificates.
A new installation of Windows starts off with a minimal set of trust
anchors. Chrome, IE and Edge use the proper APIs.

There is no way to get internal CA on macOS with 3.6, with certifi, or
with self-build OpenSSL.

Christian

From cournape at gmail.com  Mon Jan 30 16:19:12 2017
From: cournape at gmail.com (David Cournapeau)
Date: Mon, 30 Jan 2017 21:19:12 +0000
Subject: [Python-Dev] SSL certificates recommendations for downstream
 python packagers
In-Reply-To: <b8b24e2a-4d45-9531-a4de-52a8850caabc@python.org>
References: <CAGY4rcUfNpcyX2fB=h-AAkL1tTZm7zsx840Rpie55NNYF3OV2Q@mail.gmail.com>
 <8296B2E6-5444-43B0-8C71-6398DDDCB74A@lukasa.co.uk>
 <CAGY4rcUV1iqxdeqcy52AmHX1BAWEAAsYnpS--CSDkMtyOdSd0Q@mail.gmail.com>
 <b8b24e2a-4d45-9531-a4de-52a8850caabc@python.org>
Message-ID: <CAGY4rcWv1QkJESbzSZFjvegHcH-nZzev_873yZ=vRNDE_ts6bg@mail.gmail.com>

On Mon, Jan 30, 2017 at 9:14 PM, Christian Heimes <christian at python.org>
wrote:

> On 2017-01-30 22:00, David Cournapeau wrote:
> >
> >
> > On Mon, Jan 30, 2017 at 8:50 PM, Cory Benfield <cory at lukasa.co.uk
> > <mailto:cory at lukasa.co.uk>> wrote:
> >
> >
> >
> >     > On 30 Jan 2017, at 13:53, David Cournapeau <cournape at gmail.com
> <mailto:cournape at gmail.com>> wrote:
> >     >
> >     > Are there any official recommendations for downstream packagers
> beyond PEP 476 ? Is it "acceptable" for downstream packagers to patch
> python's default cert locations ?
> >
> >     There *are* no default cert locations on Windows or macOS that can
> >     be accessed by OpenSSL.
> >
> >     I cannot stress this strongly enough: you cannot provide a
> >     platform-native certificate validation logic for Python *and* use
> >     OpenSSL for certificate validation on Windows or macOS. (macOS can
> >     technically do this when you link against the system OpenSSL, at the
> >     cost of using a catastrophically insecure version of OpenSSL.)
> >
> >
> > Ah, thanks, that's already useful information.
> >
> > Just making sure I understand: this means there is no way to use
> > python's SSL library to use the system store on windows, in particular
> > private certifications that are often deployed by internal ITs in large
> > orgs ?
>
> That works with CPython because we get all trust anchors from the cert
> store. However Python is not able to retrieve *additional* certificates.
> A new installation of Windows starts off with a minimal set of trust
> anchors. Chrome, IE and Edge use the proper APIs.
>

Hm. Is this documented anywhere ? We have customers needing
"private/custom" certificates, and I am unsure where to look for.

David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170130/c1841466/attachment.html>

From christian at python.org  Mon Jan 30 16:28:00 2017
From: christian at python.org (Christian Heimes)
Date: Mon, 30 Jan 2017 22:28:00 +0100
Subject: [Python-Dev] SSL certificates recommendations for downstream
 python packagers
In-Reply-To: <CAGY4rcWv1QkJESbzSZFjvegHcH-nZzev_873yZ=vRNDE_ts6bg@mail.gmail.com>
References: <CAGY4rcUfNpcyX2fB=h-AAkL1tTZm7zsx840Rpie55NNYF3OV2Q@mail.gmail.com>
 <8296B2E6-5444-43B0-8C71-6398DDDCB74A@lukasa.co.uk>
 <CAGY4rcUV1iqxdeqcy52AmHX1BAWEAAsYnpS--CSDkMtyOdSd0Q@mail.gmail.com>
 <b8b24e2a-4d45-9531-a4de-52a8850caabc@python.org>
 <CAGY4rcWv1QkJESbzSZFjvegHcH-nZzev_873yZ=vRNDE_ts6bg@mail.gmail.com>
Message-ID: <a92f5478-f047-6392-a6dc-c046a59d9793@python.org>

On 2017-01-30 22:19, David Cournapeau wrote:
> Hm. Is this documented anywhere ? We have customers needing
> "private/custom" certificates, and I am unsure where to look for.

For full control it is advised to use a custom SSLContext that only
loads the internal CA.
https://docs.python.org/3/library/ssl.html#ssl.create_default_context

With OpenSSL you can also set SSL_CERT_FILE and SSL_CERT_DIR env vars.
It doesn't work with LibreSSL, though.

import os, ssl
os.environ['SSL_CERT_FILE'] = '/path/to/internalca.pem'
os.environ['SSL_CERT_DIR'] = os.devnull

ctx = ssl.create_default_context()

Christian

From anglian at mag.mhglobalmc1.co.uk  Mon Jan 30 18:12:46 2017
From: anglian at mag.mhglobalmc1.co.uk (Anglian Architectural)
Date: Mon, 30 Jan 2017 23:12:46 +0000
Subject: [Python-Dev] January's almost gone but there's still time to book
 your project capacity now
Message-ID: <3vC5Rl4x44zFqyJ@mail.python.org>

 



 

 

No Images? Click here
 


 


Glazing & Rainscreen Systems  -  Concept to Completion
 

 

 

 
 


 


CASE STUDIES
 


ROOFLIGHTS
 


CONTACT
 

 


ANGLIAN ARCHITECTURAL GLAZING & RAINSCREEN SYSTEMS
**FIRE RATED ALUMINIUM AND STEEL GLAZING SYSTEMS**
 
 


 


Dear Sirs


JANUARY HAS FLOWN BY...

...BUT YOU CAN BOOK THAT JOB IN WHILE WE'VE STILL GOT CAPACITY

Where has January gone to? Yes, we're now into the last week of the month and wish to say thanks to those clients who took up my offer to look at fast track work following my last email ? we still have some immediate capacity at the present time but the 2017 order book is filling fast. So please keep those enquiries flowing - and existing customers - please don?t leave it till the last minute because we always want to say ?YES WE CAN? to you especially !!
 
 


 


 


MTV Studios This month we feature more of our work in London - this time at MTV Studios in Camden, where we have installed curtain walling, complete with bris soleil and various feature cladding, as well as recreating and refurbishing feature elements from previous phases and a new rooflight. Good photographs have been difficult to source due to the extremely tight
 

 


nature of this site but we hope to get this job up on our website very soon - especially given the prestigious nature of the work carried out.
 
 


 


Portfolio Youcan see Anglian?s wide portfolio of work here on our website and we are happy to quote Aluminium Glazing and Rainscreen Systems projects from ?100k to ?2m value.

And finally, throughout 2017 please remember the website WWW.DIYROOFLIGHTS.COM

New customers are finding our SLIMGLAZE and SG2 Rooflights every month. We hope you will take a look because at ?298 net ex works for a 1m x 1m double glazed stock unit they are competitively priced and available fast. We still believe that these are the BEST PRICES in the marketplace, but if you know otherwise, please let me know. Triple glazed Rooflights are also available and you can see the range of Rooflights here.

Please bookmark the website WWW.DIYROOFLIGHTS.COM for when you need it !

Best wishes for 2017 and we hope we can do business with you in this new year.

Kind regards,

Trevor Kirby

Managing Director
 
 


 


Ask Anglian! UK fabricators and installers that can manufacture bespoke solutions for refurbs and new builds, including curved head and fire rated windows.
Visit the website for more information.
 
 


 


Slimglaze and SG2 Rooflights
Anglian Architectural Slimglaze Rooflights are competitively priced and designed for installation on flat roofs. They come glazed and silicone sealed ready for installation.
 
Please follow this link for pricing and more information.

You can also see a number of Slimglaze examples here.
 

 
 


 


"ANGLIAN ARCHITECTURAL MANUFACTURE AND INSTALL FIRE RATED GLAZING SYSTEMS!!"
 
 


 


10 good reasons you should ask us to tender for your next project:

?    Expert sub-contractor with big reputation for quality and efficiency
?    Integrated capability includes design, surveying, manufacturing and installation
?    Complete start-to-finish solution
?    In-house design, fabrication and project management
?    Critical product specification
?    Problem solving
?    Industry recognised systems and components
?    All Quality Accreditations ISO 9001, ISO 14001, ISO 18001, CHAS 
?    Long experience working with major Contractors and High Street brands 
?    Competitive prices and value for money, guaranteed.

For more information please visit the website, phone 01485 520860
 
 


 

 

Quality
Service 
Experience
 
 


 


ANGLIAN ARCHITECTURAL LIMITED
Unit 1, Mill Lane, Waterford Industrial Est.
Gt. Massingham Norfolk PE32 2HT

Tel: 01485 520860 Fax: 01485 521196
 


Visit the Anglian Architectural website

Email for more information

Follow us on Twitter 
 
 


 


Anglian Architectural are approved suppliers of Aluminium and Steel FIRE RATED GLAZING SYSTEMS and registered with Constructionline
 

 
 


 

 
 



 



Anglian Architectural Ltd, Unit 1 Mill Lane, Waterford Industrial Estate, Great Massingham, Norfolk PE32 2HT  Company No: 04346652             


 


 

 


Preferences  |  Unsubscribe 
 
 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170130/97c2f666/attachment-0001.html>

From brett at python.org  Mon Jan 30 20:44:18 2017
From: brett at python.org (Brett Cannon)
Date: Tue, 31 Jan 2017 01:44:18 +0000
Subject: [Python-Dev] modules people want installed automatically (was:
 Re: re performance)
In-Reply-To: <20170130132641.6375da72@presto>
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
 <CAHzaPEO2+BcJUEy15j37d5O72eo1OWewy4PeR0_vcHu0AD6Msg@mail.gmail.com>
 <d9e50354-8ed7-c347-a397-34f0b83927c6@mail.de>
 <f3414629-15b4-17ff-2570-ade73903bbdf@mrabarnett.plus.com>
 <11439969-DBE6-4031-9EC6-3B2452340942@langa.pl>
 <65cf9e6e-7962-d198-6aa7-4302818a8819@mrabarnett.plus.com>
 <CADiSq7dforv5fY9Y4hgRGJ_6z9R0AY0MM-T=Cai6UmzGmPJdVg@mail.gmail.com>
 <20170128120705.3696f70c@subdivisions.wooz.org>
 <CADiSq7dYxHka1s7f35hHzHLHfYC+tBWBvuoC-QBKRfrUvPW2Yw@mail.gmail.com>
 <20170130092637.4665eaea@presto.wooz.org>
 <CAP1=2W6oraajQt+7wiXcKv0Pec64BiES_8zBu40rfFKWuyox7Q@mail.gmail.com>
 <20170130132641.6375da72@presto>
Message-ID: <CAP1=2W65pBeZqo7Q1vG6kJccq5kPGBeYNO=mNWMHQ90qX1KqFg@mail.gmail.com>

Proper support in the loader API is possible and was on its way to being
coded up until GitHub took precedence (realize this project has put all
other major python projects of mine on hold for nearly 2 years hence the
delay). And pkg_resources I'm sure could be updated to use any new API when
available, so I don't think it's appropriate to bring the name into the
stdlib.

On Mon, Jan 30, 2017, 10:28 Barry Warsaw, <barry at python.org> wrote:

> On Jan 30, 2017, at 06:14 PM, Brett Cannon wrote:
>
> >What functionality are you after here from pkg_resources? If it's the
> >reading of data files then you will get that in the stdlib through
> >importlib as soon as I'm not working on getting our workflow to work
> >through GitHub's web UI (which obviously includes the migration itself).
>
>
> http://setuptools.readthedocs.io/en/latest/pkg_resources.html#basic-resource-access
>
> Mostly I use:
>
> * resource_filename()
> * resource_string() (really, resource_bytes!)
> * resource_stream() (although I'd really like a more open()-like API)
>
> This might fall under a simpler PEP 365.  Also, while I would love to have
> these available say in importlib, I also like to consider a backward
> compatible API where the stdlib provides the `pkg_resources` module name.
> That's not totally required though because you can play ImportError games
> until Python 3.7 (presumably) is in widespread -and only- use.
>
> Cheers,
> -Barry
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170131/842d4088/attachment.html>

From cory at lukasa.co.uk  Tue Jan 31 04:19:34 2017
From: cory at lukasa.co.uk (Cory Benfield)
Date: Tue, 31 Jan 2017 09:19:34 +0000
Subject: [Python-Dev] SSL certificates recommendations for downstream
 python packagers
In-Reply-To: <CAGY4rcUV1iqxdeqcy52AmHX1BAWEAAsYnpS--CSDkMtyOdSd0Q@mail.gmail.com>
References: <CAGY4rcUfNpcyX2fB=h-AAkL1tTZm7zsx840Rpie55NNYF3OV2Q@mail.gmail.com>
 <8296B2E6-5444-43B0-8C71-6398DDDCB74A@lukasa.co.uk>
 <CAGY4rcUV1iqxdeqcy52AmHX1BAWEAAsYnpS--CSDkMtyOdSd0Q@mail.gmail.com>
Message-ID: <2B2F8AFC-F3CA-4290-A182-E02B4BA73982@lukasa.co.uk>


> On 30 Jan 2017, at 21:00, David Cournapeau <cournape at gmail.com> wrote:
> 
> 
> 
> On Mon, Jan 30, 2017 at 8:50 PM, Cory Benfield <cory at lukasa.co.uk <mailto:cory at lukasa.co.uk>> wrote:
> 
> 
> > On 30 Jan 2017, at 13:53, David Cournapeau <cournape at gmail.com <mailto:cournape at gmail.com>> wrote:
> >
> > Are there any official recommendations for downstream packagers beyond PEP 476 ? Is it "acceptable" for downstream packagers to patch python's default cert locations ?
> 
> There *are* no default cert locations on Windows or macOS that can be accessed by OpenSSL.
> 
> I cannot stress this strongly enough: you cannot provide a platform-native certificate validation logic for Python *and* use OpenSSL for certificate validation on Windows or macOS. (macOS can technically do this when you link against the system OpenSSL, at the cost of using a catastrophically insecure version of OpenSSL.)
> 
> Ah, thanks, that's already useful information.
> 
> Just making sure I understand: this means there is no way to use python's SSL library to use the system store on windows, in particular private certifications that are often deployed by internal ITs in large orgs ?

If only it were that simple!

No, you absolutely *can* do that. You can extract the trust roots from the system trust store, convert them into PEM/DER-encoded files, and load them into OpenSSL. That will work.

The problem is that both SecureTransport and SChannel have got a number of differences from OpenSSL. In no particular order:

1. Their chain building logic is different. This means that, given a collection of certificates presented by a server and a bundle of already-trusted certs, each implementation may build a different trust chain. This may cause one implementation to refuse to validate where the others do, or vice versa. This is very common with older OpenSSLs.
2. SecureTransport and SChannel both use the system trust DB, which on both Windows and mac allows the setting of custom policies. OpenSSL won?t respect these policies, which means you can fail-open (that is, export and use a root certificate that the OS believes should not be trusted for a given use case). There is no way to export these trust policies into OpenSSL.
3. SecureTransport, SChannel, and OpenSSL all support different X.509 extensions and understand them differently. This means that some certs may be untrusted for certain uses by Windows but trusted for those uses by OpenSSL, for example.

In general, it is unwise to mix trust stores. If you want to use your OS?s trust store, the best approach is to use the OS?s TLS stack as well. At least that way when a user says ?It works in my browser?, you know it should work for you too.

Cory

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170131/73393a4b/attachment.html>

From christian at python.org  Tue Jan 31 04:33:16 2017
From: christian at python.org (Christian Heimes)
Date: Tue, 31 Jan 2017 10:33:16 +0100
Subject: [Python-Dev] SSL certificates recommendations for downstream
 python packagers
In-Reply-To: <2B2F8AFC-F3CA-4290-A182-E02B4BA73982@lukasa.co.uk>
References: <CAGY4rcUfNpcyX2fB=h-AAkL1tTZm7zsx840Rpie55NNYF3OV2Q@mail.gmail.com>
 <8296B2E6-5444-43B0-8C71-6398DDDCB74A@lukasa.co.uk>
 <CAGY4rcUV1iqxdeqcy52AmHX1BAWEAAsYnpS--CSDkMtyOdSd0Q@mail.gmail.com>
 <2B2F8AFC-F3CA-4290-A182-E02B4BA73982@lukasa.co.uk>
Message-ID: <e826d15e-3e3b-0413-c5d6-e765a853dc43@python.org>

On 2017-01-31 10:19, Cory Benfield wrote:
> 
>> On 30 Jan 2017, at 21:00, David Cournapeau <cournape at gmail.com
>> <mailto:cournape at gmail.com>> wrote:
>>
>>
>>
>> On Mon, Jan 30, 2017 at 8:50 PM, Cory Benfield <cory at lukasa.co.uk
>> <mailto:cory at lukasa.co.uk>> wrote:
>>
>>
>>
>>     > On 30 Jan 2017, at 13:53, David Cournapeau <cournape at gmail.com <mailto:cournape at gmail.com>> wrote:
>>     >
>>     > Are there any official recommendations for downstream packagers beyond PEP 476 ? Is it "acceptable" for downstream packagers to patch python's default cert locations ?
>>
>>     There *are* no default cert locations on Windows or macOS that can
>>     be accessed by OpenSSL.
>>
>>     I cannot stress this strongly enough: you cannot provide a
>>     platform-native certificate validation logic for Python *and* use
>>     OpenSSL for certificate validation on Windows or macOS. (macOS can
>>     technically do this when you link against the system OpenSSL, at
>>     the cost of using a catastrophically insecure version of OpenSSL.)
>>
>>
>> Ah, thanks, that's already useful information.
>>
>> Just making sure I understand: this means there is no way to use
>> python's SSL library to use the system store on windows, in particular
>> private certifications that are often deployed by internal ITs in
>> large orgs ?
> 
> If only it were that simple!
> 
> No, you absolutely *can* do that. You can extract the trust roots from
> the system trust store, convert them into PEM/DER-encoded files, and
> load them into OpenSSL. That will work.
> 
> The problem is that both SecureTransport and SChannel have got a number
> of differences from OpenSSL. In no particular order:
> 
> 1. Their chain building logic is different. This means that, given a
> collection of certificates presented by a server and a bundle of
> already-trusted certs, each implementation may build a different trust
> chain. This may cause one implementation to refuse to validate where the
> others do, or vice versa. This is very common with older OpenSSLs.
> 2. SecureTransport and SChannel both use the system trust DB, which on
> both Windows and mac allows the setting of custom policies. OpenSSL
> won?t respect these policies, which means you can fail-open (that is,
> export and use a root certificate that the OS believes should not be
> trusted for a given use case). There is no way to export these trust
> policies into OpenSSL.

One small correction, it is possible to export some of the trust
settings to a TRUSTED CERTIFICATE and import them into OpenSSL. It works
correctly in 1.0.1 and since 1.0.2e or f. Trust settings are stored in
X509_AUX extension after the actual certificate and signature. OpenSSL's
default loaders for cert dir and cert file do load auxiliary trust
information.

(Of course the feature is experimental and was broken in 1.0.2 for a
long time until I discovered the issue and fixed it...)


https://linux.die.net/man/1/x509

TRUST SETTINGS

Please note these options are currently experimental and may well change.

A trusted certificate is an ordinary certificate which has several
additional pieces of information attached to it such as the permitted
and prohibited uses of the certificate and an "alias".

Normally when a certificate is being verified at least one certificate
must be "trusted". By default a trusted certificate must be stored
locally and must be a root CA: any certificate chain ending in this CA
is then usable for any purpose.

Trust settings currently are only used with a root CA . They allow a
finer control over the purposes the root CA can be used for. For example
a CA may be trusted for SSL client but not SSL server use.

From cory at lukasa.co.uk  Tue Jan 31 04:37:12 2017
From: cory at lukasa.co.uk (Cory Benfield)
Date: Tue, 31 Jan 2017 09:37:12 +0000
Subject: [Python-Dev] SSL certificates recommendations for downstream
 python packagers
In-Reply-To: <e826d15e-3e3b-0413-c5d6-e765a853dc43@python.org>
References: <CAGY4rcUfNpcyX2fB=h-AAkL1tTZm7zsx840Rpie55NNYF3OV2Q@mail.gmail.com>
 <8296B2E6-5444-43B0-8C71-6398DDDCB74A@lukasa.co.uk>
 <CAGY4rcUV1iqxdeqcy52AmHX1BAWEAAsYnpS--CSDkMtyOdSd0Q@mail.gmail.com>
 <2B2F8AFC-F3CA-4290-A182-E02B4BA73982@lukasa.co.uk>
 <e826d15e-3e3b-0413-c5d6-e765a853dc43@python.org>
Message-ID: <B85C6092-7404-4A92-87C9-2ED3DFFE932C@lukasa.co.uk>


> On 31 Jan 2017, at 09:33, Christian Heimes <christian at python.org> wrote:
> 
> One small correction, it is possible to export some of the trust
> settings to a TRUSTED CERTIFICATE and import them into OpenSSL. It works
> correctly in 1.0.1 and since 1.0.2e or f. Trust settings are stored in
> X509_AUX extension after the actual certificate and signature. OpenSSL's
> default loaders for cert dir and cert file do load auxiliary trust
> information.

Ah, good spot.

I suspect the code you?d need to write to safely extract that functionality is pretty subtle. I definitely don?t trust myself to get it right.

Cory

From ncoghlan at gmail.com  Tue Jan 31 04:48:52 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 31 Jan 2017 10:48:52 +0100
Subject: [Python-Dev] Generator objects and list comprehensions?
In-Reply-To: <CAP1=2W6UZX6J1Zyd9L6UTtHC-a88mwQ3P9k_JgRwJvZ4Dqqh7A@mail.gmail.com>
References: <CAG=rPVco0cnKdA0qDbhnZUctzeVVuJTgN4YX7ua6x8Lb-Q-TUQ@mail.gmail.com>
 <CAFo6DPaP5dcDy=sm4QrauQJDfXgZQYLg0hFdNAwrQ=qM1OwkOw@mail.gmail.com>
 <CAPJVwBkg4hPsJ9axhjDBRvUn20=yd0aOx_8eAJbvSD3R-qZr-g@mail.gmail.com>
 <CAOMjWkmCVZbex40Ls8nDfugQYyWMQTMeBPY+DD_nG9R7fa_QxA@mail.gmail.com>
 <CAG=rPVcab4amWESWhu4cHBUfmsE9vHLL7G-oYeQA2p83bwmZDQ@mail.gmail.com>
 <CAP1=2W6UZX6J1Zyd9L6UTtHC-a88mwQ3P9k_JgRwJvZ4Dqqh7A@mail.gmail.com>
Message-ID: <CADiSq7eSQb-fcwoXvO=GhLoCeh5O2PHtMZKqyRC+q_7jR46BFA@mail.gmail.com>

On 30 January 2017 at 19:05, Brett Cannon <brett at python.org> wrote:
> On Sun, 29 Jan 2017 at 16:39 Craig Rodrigues <rodrigc at freebsd.org> wrote:
>> I'm OK with either approach.  Leaving things the way they are in Python 3
>> is no good, IMHO.
>
> My vote is it be a SyntaxError since you're not getting what you expect from
> the syntax.

I'd agree that's a sensible place for us to end up, as any code
relying on the current behaviour is really too clever to be
maintainable.

In terms of getting there, we'll likely want:

- SyntaxWarning or DeprecationWarning in 3.7
- Py3k warning in 2.7.x
- SyntaxError in 3.8

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Tue Jan 31 04:54:26 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 31 Jan 2017 10:54:26 +0100
Subject: [Python-Dev] re performance
In-Reply-To: <20170130092637.4665eaea@presto.wooz.org>
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
 <CAHzaPEO2+BcJUEy15j37d5O72eo1OWewy4PeR0_vcHu0AD6Msg@mail.gmail.com>
 <d9e50354-8ed7-c347-a397-34f0b83927c6@mail.de>
 <f3414629-15b4-17ff-2570-ade73903bbdf@mrabarnett.plus.com>
 <11439969-DBE6-4031-9EC6-3B2452340942@langa.pl>
 <65cf9e6e-7962-d198-6aa7-4302818a8819@mrabarnett.plus.com>
 <CADiSq7dforv5fY9Y4hgRGJ_6z9R0AY0MM-T=Cai6UmzGmPJdVg@mail.gmail.com>
 <20170128120705.3696f70c@subdivisions.wooz.org>
 <CADiSq7dYxHka1s7f35hHzHLHfYC+tBWBvuoC-QBKRfrUvPW2Yw@mail.gmail.com>
 <20170130092637.4665eaea@presto.wooz.org>
Message-ID: <CADiSq7fDZzehQvG+Y7aTbYvFs5jSRbZ1tGdxXFGUbgdVGLLWKA@mail.gmail.com>

On 30 January 2017 at 15:26, Barry Warsaw <barry at python.org> wrote:
> On Jan 30, 2017, at 12:38 PM, Nick Coghlan wrote:
>
>>I think there are 3 main candidates that could fit that bill:
>>
>>- requests
>>- setuptools
>>- regex
>
> Actually, I think pkg_resources would make an excellent candidate.  The
> setuptools crew is working on a branch that would allow for setuptools and
> pkg_resources to be split, which would be great for other reasons.  Splitting
> them may mean that pkg_resources could eventually be added to the stdlib, but
> as an intermediate step, it could also test out this idea.  It probably has a
> lot less of the baggage that you outline.

Yep, if/when pkg_resources is successfully split out from the rest of
setuptools, I agree it would also be a good candidate for stdlib
bundling - version independent runtime access to the database of
installed packages is a key capability for many use cases, and not
currently something we support especially well.

It's also far more analogous to the existing pip bundling, since
setuptools/pkg_resources are also maintained under the PyPA structure.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From p.f.moore at gmail.com  Tue Jan 31 04:56:28 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 31 Jan 2017 09:56:28 +0000
Subject: [Python-Dev] SSL certificates recommendations for downstream
 python packagers
In-Reply-To: <2B2F8AFC-F3CA-4290-A182-E02B4BA73982@lukasa.co.uk>
References: <CAGY4rcUfNpcyX2fB=h-AAkL1tTZm7zsx840Rpie55NNYF3OV2Q@mail.gmail.com>
 <8296B2E6-5444-43B0-8C71-6398DDDCB74A@lukasa.co.uk>
 <CAGY4rcUV1iqxdeqcy52AmHX1BAWEAAsYnpS--CSDkMtyOdSd0Q@mail.gmail.com>
 <2B2F8AFC-F3CA-4290-A182-E02B4BA73982@lukasa.co.uk>
Message-ID: <CACac1F8C6j0abYmN_pdewDb1BStefvS-GCstD4U6+oumBZQZYQ@mail.gmail.com>

On 31 January 2017 at 09:19, Cory Benfield <cory at lukasa.co.uk> wrote:
>
> In general, it is unwise to mix trust stores. If you want to use your OS?s
> trust store, the best approach is to use the OS?s TLS stack as well. At
> least that way when a user says ?It works in my browser?, you know it should
> work for you too.

As a bystander (and an "end user" of this stuff) the message I'm
getting here is a bit worrying. To take a step back from the sysadmin
issues here, is the statement

    It's safe to use Python (either via the stdlib, or various 3rd
party libraries like requests) to access https URLs

correct? I understand that "safe" is a complex concept here, but in
terms of promoting Python, I'd be using the term in the sense of "at
least as acceptable as using something like C# or Java" - in other
words I'm not introducing any new vulnerabilities if I argue for
Python over one of those languages?

Paul

From cournape at gmail.com  Tue Jan 31 08:41:08 2017
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 31 Jan 2017 13:41:08 +0000
Subject: [Python-Dev] SSL certificates recommendations for downstream
 python packagers
In-Reply-To: <2B2F8AFC-F3CA-4290-A182-E02B4BA73982@lukasa.co.uk>
References: <CAGY4rcUfNpcyX2fB=h-AAkL1tTZm7zsx840Rpie55NNYF3OV2Q@mail.gmail.com>
 <8296B2E6-5444-43B0-8C71-6398DDDCB74A@lukasa.co.uk>
 <CAGY4rcUV1iqxdeqcy52AmHX1BAWEAAsYnpS--CSDkMtyOdSd0Q@mail.gmail.com>
 <2B2F8AFC-F3CA-4290-A182-E02B4BA73982@lukasa.co.uk>
Message-ID: <CAGY4rcVucdbEQGu=m0TLQYP40-DmBmJYBQr2_Ph5SL-mJDbnew@mail.gmail.com>

On Tue, Jan 31, 2017 at 9:19 AM, Cory Benfield <cory at lukasa.co.uk> wrote:

>
> On 30 Jan 2017, at 21:00, David Cournapeau <cournape at gmail.com> wrote:
>
>
>
> On Mon, Jan 30, 2017 at 8:50 PM, Cory Benfield <cory at lukasa.co.uk> wrote:
>
>>
>>
>> > On 30 Jan 2017, at 13:53, David Cournapeau <cournape at gmail.com> wrote:
>> >
>> > Are there any official recommendations for downstream packagers beyond
>> PEP 476 ? Is it "acceptable" for downstream packagers to patch python's
>> default cert locations ?
>>
>> There *are* no default cert locations on Windows or macOS that can be
>> accessed by OpenSSL.
>>
>> I cannot stress this strongly enough: you cannot provide a
>> platform-native certificate validation logic for Python *and* use OpenSSL
>> for certificate validation on Windows or macOS. (macOS can technically do
>> this when you link against the system OpenSSL, at the cost of using a
>> catastrophically insecure version of OpenSSL.)
>>
>
> Ah, thanks, that's already useful information.
>
> Just making sure I understand: this means there is no way to use python's
> SSL library to use the system store on windows, in particular private
> certifications that are often deployed by internal ITs in large orgs ?
>
>
> If only it were that simple!
>
> No, you absolutely *can* do that. You can extract the trust roots from the
> system trust store, convert them into PEM/DER-encoded files, and load them
> into OpenSSL. That will work.
>

Right, I guess it depends on what one means by "can". In my context, it was
to be taken as "can it work without the end user having to do anything". We
provide them a python-based tool, and it has to work with the system store
out of the box. If the system store is updated through e.g. group policy,
our python tool automatically get that update.

>From the sound of it, it looks like this is simply not possible ATM with
python, at least not without 3rd party libraries.

David


> The problem is that both SecureTransport and SChannel have got a number of
> differences from OpenSSL. In no particular order:
>
> 1. Their chain building logic is different. This means that, given a
> collection of certificates presented by a server and a bundle of
> already-trusted certs, each implementation may build a different trust
> chain. This may cause one implementation to refuse to validate where the
> others do, or vice versa. This is very common with older OpenSSLs.
> 2. SecureTransport and SChannel both use the system trust DB, which on
> both Windows and mac allows the setting of custom policies. OpenSSL won?t
> respect these policies, which means you can fail-open (that is, export and
> use a root certificate that the OS believes should not be trusted for a
> given use case). There is no way to export these trust policies into
> OpenSSL.
> 3. SecureTransport, SChannel, and OpenSSL all support different X.509
> extensions and understand them differently. This means that some certs may
> be untrusted for certain uses by Windows but trusted for those uses by
> OpenSSL, for example.
>
> In general, it is unwise to mix trust stores. If you want to use your OS?s
> trust store, the best approach is to use the OS?s TLS stack as well. At
> least that way when a user says ?It works in my browser?, you know it
> should work for you too.
>
> Cory
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170131/de284137/attachment-0001.html>

From cory at lukasa.co.uk  Tue Jan 31 09:54:42 2017
From: cory at lukasa.co.uk (Cory Benfield)
Date: Tue, 31 Jan 2017 14:54:42 +0000
Subject: [Python-Dev] SSL certificates recommendations for downstream
 python packagers
In-Reply-To: <CACac1F8C6j0abYmN_pdewDb1BStefvS-GCstD4U6+oumBZQZYQ@mail.gmail.com>
References: <CAGY4rcUfNpcyX2fB=h-AAkL1tTZm7zsx840Rpie55NNYF3OV2Q@mail.gmail.com>
 <8296B2E6-5444-43B0-8C71-6398DDDCB74A@lukasa.co.uk>
 <CAGY4rcUV1iqxdeqcy52AmHX1BAWEAAsYnpS--CSDkMtyOdSd0Q@mail.gmail.com>
 <2B2F8AFC-F3CA-4290-A182-E02B4BA73982@lukasa.co.uk>
 <CACac1F8C6j0abYmN_pdewDb1BStefvS-GCstD4U6+oumBZQZYQ@mail.gmail.com>
Message-ID: <4C7A4DFF-8C09-47B3-A45D-722A8F092D28@lukasa.co.uk>


> On 31 Jan 2017, at 09:56, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> On 31 January 2017 at 09:19, Cory Benfield <cory at lukasa.co.uk> wrote:
>> 
>> In general, it is unwise to mix trust stores. If you want to use your OS?s
>> trust store, the best approach is to use the OS?s TLS stack as well. At
>> least that way when a user says ?It works in my browser?, you know it should
>> work for you too.
> 
> As a bystander (and an "end user" of this stuff) the message I'm
> getting here is a bit worrying. To take a step back from the sysadmin
> issues here, is the statement
> 
>    It's safe to use Python (either via the stdlib, or various 3rd
> party libraries like requests) to access https URLs
> 
> correct? I understand that "safe" is a complex concept here, but in
> terms of promoting Python, I'd be using the term in the sense of "at
> least as acceptable as using something like C# or Java" - in other
> words I'm not introducing any new vulnerabilities if I argue for
> Python over one of those languages?

My answer would be ?Yes*?.

The asterisk is that it works best when you understand how you are rooting your trust stores. On Linux, if you use your distro provided OpenSSL and its trust store it is as safe as the rest of your system. On macOS, if you use the system Python it is unsafe. If you use a Python compiled against Homebrew?s OpenSSL, it is safe, though it is only safe based on a snapshot in time (this is for boring Homebrew-related reasons that are nothing to do with Python). On Windows, it is safe, though potentially less functional. Requests is basically safe, though it is definitely possible.

If you are trying to reach websites on the open web using Requests, that?s safe, subject to the criteria below.

The answer gets murkier if you do substantial configuration of a trust database. If you have added or removed root certificates from a system trust store, or you have configured the trust database on a Windows or Apple machine, then it all gets murky fast. If you?ve only added or expanded trust then Python is still safe on all those platforms, it is just less functional. If you?ve removed trust then it is possible that a Python process will trust something that other processes on your machine will not.

However, you should be aware that that?s basically the default state of being for TLS. Java is basically in the same place as Python today, but with the added fun of having a hand-rolled TLS implementation (JSSE). C# is a different story: the stdlib uses SSPI for certificate validation, which basically means SChannel *on Windows*. In the wider .NET ecosystem I have no idea how this is being done.

So C# applications are Windows-native safe on Windows, and are a crapshoot elsewhere. For Java vs Python, I?d say we?re slightly ahead right now.

Again, the long-term solution to this fix is to allow us to use SChannel and SecureTransport to provide TLS on the relevant platforms. This will also let people use GnuTLS or NSS or whatever other TLS implementations float their boat on other Unices. I?ll be bringing a draft PEP in front of python-dev sometime in the next month to start this work: if you?re interested, I recommend helping out with that process! 

Cory


From p.f.moore at gmail.com  Tue Jan 31 10:16:13 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 31 Jan 2017 15:16:13 +0000
Subject: [Python-Dev] SSL certificates recommendations for downstream
 python packagers
In-Reply-To: <4C7A4DFF-8C09-47B3-A45D-722A8F092D28@lukasa.co.uk>
References: <CAGY4rcUfNpcyX2fB=h-AAkL1tTZm7zsx840Rpie55NNYF3OV2Q@mail.gmail.com>
 <8296B2E6-5444-43B0-8C71-6398DDDCB74A@lukasa.co.uk>
 <CAGY4rcUV1iqxdeqcy52AmHX1BAWEAAsYnpS--CSDkMtyOdSd0Q@mail.gmail.com>
 <2B2F8AFC-F3CA-4290-A182-E02B4BA73982@lukasa.co.uk>
 <CACac1F8C6j0abYmN_pdewDb1BStefvS-GCstD4U6+oumBZQZYQ@mail.gmail.com>
 <4C7A4DFF-8C09-47B3-A45D-722A8F092D28@lukasa.co.uk>
Message-ID: <CACac1F_HfGiVZti3aUPziBtFHk=ZLvOB9HLK3utDNU+T5Pr5Sg@mail.gmail.com>

On 31 January 2017 at 14:54, Cory Benfield <cory at lukasa.co.uk> wrote:
>
> So C# applications are Windows-native safe on Windows, and are a crapshoot elsewhere. For Java vs Python, I?d say we?re slightly ahead right now.

That's precisely the sort of answer I was after. Many thanks. The
additional detail is interesting, but starts being scary again. I
think the "advantage" languages like Java has is that no-one really
discusses the details - so it seems like things are fine - but it
devolves into a "how do we get this to work?" mess if you try to do
anything hard. That's not a real advantage, but unfortunately politics
often trumps technical accuracy in my area of work :-( (My job is
often to make technically correct politically acceptable - who knew
that's what "coding" really was?)

> Again, the long-term solution to this fix is to allow us to use SChannel and SecureTransport to provide TLS on the relevant platforms. This will also let people use GnuTLS or NSS or whatever other TLS implementations float their boat on other Unices. I?ll be bringing a draft PEP in front of python-dev sometime in the next month to start this work: if you?re interested, I recommend helping out with that process!

That sounds fantastic. I'm not sure how much I'd be able to help,
beyond whining "but I don't care about all this, just make it work and
make it eeeeasyyyyyy" :-) but I'll certainly watch the discussions and
do what I can.

Thanks,
Paul

From brett at python.org  Tue Jan 31 13:18:24 2017
From: brett at python.org (Brett Cannon)
Date: Tue, 31 Jan 2017 18:18:24 +0000
Subject: [Python-Dev] Heads up: possible double-comments on bpo for commits
Message-ID: <CAP1=2W7vvz2t0X=vpyccA+nWawWXA+X5wSbw9ZAq1Bk+r93dDA@mail.gmail.com>

I've activated the webhook for receiving comments on issues when a commit
lands mentioning an issue, so if you see a commit from our hg integration
and another from GitHub, understand that's why (mention issues as "bpo
NNNN" in commit messages if you want to see it in action). If it becomes
too much of a hassle to see the duplicates before we migrate I can turn off
the notifications, but obviously more testing the better. :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170131/39c1b5f9/attachment.html>

From steve.dower at python.org  Tue Jan 31 13:26:52 2017
From: steve.dower at python.org (Steve Dower)
Date: Tue, 31 Jan 2017 10:26:52 -0800
Subject: [Python-Dev] SSL certificates recommendations for downstream
 python packagers
In-Reply-To: <4cece125-61c5-ec85-7137-f425badb1bc6@python.org>
References: <CAGY4rcUfNpcyX2fB=h-AAkL1tTZm7zsx840Rpie55NNYF3OV2Q@mail.gmail.com>
 <8296B2E6-5444-43B0-8C71-6398DDDCB74A@lukasa.co.uk>
 <4cece125-61c5-ec85-7137-f425badb1bc6@python.org>
Message-ID: <781d78df-f5e5-6fba-136d-a2967bbbdf52@python.org>

On 30Jan2017 1310, Christian Heimes wrote:
> On 2017-01-30 21:50, Cory Benfield wrote:
>>
>>
>>> On 30 Jan 2017, at 13:53, David Cournapeau <cournape at gmail.com> wrote:
>>>
>>> Are there any official recommendations for downstream packagers beyond PEP 476 ? Is it "acceptable" for downstream packagers to patch python's default cert locations ?
>>
>> There *are* no default cert locations on Windows or macOS that can be accessed by OpenSSL.
>>
>> I cannot stress this strongly enough: you cannot provide a platform-native certificate validation logic for Python *and* use OpenSSL for certificate validation on Windows or macOS. (macOS can technically do this when you link against the system OpenSSL, at the cost of using a catastrophically insecure version of OpenSSL.)
>
> In theory it is possible for Python and OpenSSL, too. I looked into a
> custom X509_LOOKUP_METHOD to locate trust anchors by subject. Steve is
> trying an alternative approach in https://bugs.python.org/issue28747. It
> ain't pretty and we are not there yet, too. Native support for SChannel
> and SecureTransport has some benefits.

My approach there is certainly not pretty, but IMHO it's the only 
feasible way to enable this on 3.6 (that is, until we get the proper TLS 
API).

In short, I want to allow Python code to set OpenSSL's certificate 
validation callback. Basically, given a raw certificate, return 
True/False based on whether it should be trusted. I then have separate 
code (yet to be published) implementing the callback on Windows by 
calling into the WinVerifyTrust API with the HTTPS provider, which 
(allegedly) behaves identically to browsers using the same API (again, 
allegedly - I have absolutely no evidence to support this other than 
some manual testing).

The big cons are that all the OpenSSL configuration you may do becomes 
useless (and as we know, there's essentially no configuration available 
for Windows's validation), and you're delegating the most 
interesting-to-exploit part of the process to someone else,perhaps 
unknowingly, if you import something that patches SSLContext... You 
still end up with an OpenSSL-wrapped socket, but I guess it could be 
with a certificate it doesn't like? (Also a big con is that Christian 
doesn't like this approach, and his risk assessment is certainly better 
than mine :) )

Native support for secure sockets is the best approach, but it won't 
land before 3.7.

Cheers,
Steve


From peter.xihong.wang at intel.com  Tue Jan 31 14:40:48 2017
From: peter.xihong.wang at intel.com (Wang, Peter Xihong)
Date: Tue, 31 Jan 2017 19:40:48 +0000
Subject: [Python-Dev] re performance
In-Reply-To: <CADiSq7fDZzehQvG+Y7aTbYvFs5jSRbZ1tGdxXFGUbgdVGLLWKA@mail.gmail.com>
References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de>
 <CAHzaPEO2+BcJUEy15j37d5O72eo1OWewy4PeR0_vcHu0AD6Msg@mail.gmail.com>
 <d9e50354-8ed7-c347-a397-34f0b83927c6@mail.de>
 <f3414629-15b4-17ff-2570-ade73903bbdf@mrabarnett.plus.com>
 <11439969-DBE6-4031-9EC6-3B2452340942@langa.pl>
 <65cf9e6e-7962-d198-6aa7-4302818a8819@mrabarnett.plus.com>
 <CADiSq7dforv5fY9Y4hgRGJ_6z9R0AY0MM-T=Cai6UmzGmPJdVg@mail.gmail.com>
 <20170128120705.3696f70c@subdivisions.wooz.org>
 <CADiSq7dYxHka1s7f35hHzHLHfYC+tBWBvuoC-QBKRfrUvPW2Yw@mail.gmail.com>
 <20170130092637.4665eaea@presto.wooz.org>
 <CADiSq7fDZzehQvG+Y7aTbYvFs5jSRbZ1tGdxXFGUbgdVGLLWKA@mail.gmail.com>
Message-ID: <371EBC7881C7844EAAF5556BFF21BCCC42EFF68E@ORSMSX105.amr.corp.intel.com>

Regarding to the performance difference between "re" and "regex" and packaging related options, we did a performance comparison using Python 3.6.0 to run some micro-benchmarks in the Python Benchmark Suite (https://github.com/python/performance):

Results in ms, and the lower the better (running on Ubuntu 15.10)
				re		regex (via pip install regex, and a replacement of "import re" with "import regex as re")
bm_regex_compile.py		229		298
bm_regex_dna.py		171		267
bm_regex_effbot.py		2.77		3.04
bm_regex_v8.py		24.8		14.1
This data shows "re" is better than "regex" in term of performance in 3 out of 4 above micro-benchmarks.

Anyone searching for "regular expression python" will get a first hit at the Python documentation on "re".  Naturally, any new developer could start with "re" since day 1 and not bother to look elsewhere for alternatives later on.

We did a query for "import re" against the big cloud computing software application, OpenStack (with 3.7 million lines of source codes and majority of them written in Python), and got ~1000 hits.

With that being said, IMHO, it would be nice to capture ("borrow") the performance benefit from "regex" and merged into "re", without knowing or worrying about packaging/installing stuff.

Cheers,

Peter

?

-----Original Message-----
From: Python-Dev [mailto:python-dev-bounces+peter.xihong.wang=intel.com at python.org] On Behalf Of Nick Coghlan
Sent: Tuesday, January 31, 2017 1:54 AM
To: Barry Warsaw <barry at python.org>
Cc: python-dev at python.org
Subject: Re: [Python-Dev] re performance

On 30 January 2017 at 15:26, Barry Warsaw <barry at python.org> wrote:
> On Jan 30, 2017, at 12:38 PM, Nick Coghlan wrote:
>
>>I think there are 3 main candidates that could fit that bill:
>>
>>- requests
>>- setuptools
>>- regex
>
> Actually, I think pkg_resources would make an excellent candidate.  
> The setuptools crew is working on a branch that would allow for 
> setuptools and pkg_resources to be split, which would be great for 
> other reasons.  Splitting them may mean that pkg_resources could 
> eventually be added to the stdlib, but as an intermediate step, it 
> could also test out this idea.  It probably has a lot less of the baggage that you outline.

Yep, if/when pkg_resources is successfully split out from the rest of setuptools, I agree it would also be a good candidate for stdlib bundling - version independent runtime access to the database of installed packages is a key capability for many use cases, and not currently something we support especially well.

It's also far more analogous to the existing pip bundling, since setuptools/pkg_resources are also maintained under the PyPA structure.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/peter.xihong.wang%40intel.com