From benjamin at python.org Thu Jun 1 02:36:26 2017 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 31 May 2017 23:36:26 -0700 Subject: [Python-Dev] PEP 7 and braces { .... } on if In-Reply-To: <592F4AC5.5040802@canterbury.ac.nz> References: <20170531110457.5b4ab7ac@subdivisions.wooz.org> <592F4AC5.5040802@canterbury.ac.nz> Message-ID: <1496298986.1342619.995043544.5F93FAFA@webmail.messagingengine.com> Modern GCC can defend against these kinds of problems. If I introduce a "goto fail" bug somewhere in Python, I get a nice warning: ../Objects/abstract.c: In function ?PyObject_Type?: ../Objects/abstract.c:35:5: warning: this ?if? clause does not guard... [-Wmisleading-indentation] if (o == NULL) ^~ ../Objects/abstract.c:37:9: note: ...this statement, but the latter is misleadingly indented as if it is guarded by the ?if? 2 + 3; ^ This is not to say that simply bracing everything isn't the right way to go. On Wed, May 31, 2017, at 15:59, Greg Ewing wrote: > Seems like a good idea to tighten it up. > > If a style guide is going to say "you can either do X or > not do X", it might as well not mention X at all. :-) > > -- > Greg > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/benjamin%40python.org From storchaka at gmail.com Thu Jun 1 02:53:16 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 1 Jun 2017 09:53:16 +0300 Subject: [Python-Dev] PEP 7 and braces { .... } on if In-Reply-To: References: Message-ID: 31.05.17 20:27, Guido van Rossum ????: > I interpret the PEP as saying that you should use braces everywhere but > not to add them in code that you're not modifying otherwise. (I.e. don't > go on a brace-adding rampage.) If author and reviewer of a PR disagree I > would go with "add braces" since that's clearly the PEP's preference. > This is C code. We should play it safe. Thank you. I'll be interpreting it the same way. From storchaka at gmail.com Thu Jun 1 02:52:16 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 1 Jun 2017 09:52:16 +0300 Subject: [Python-Dev] PEP 7 and braces { .... } on if In-Reply-To: <1496298986.1342619.995043544.5F93FAFA@webmail.messagingengine.com> References: <20170531110457.5b4ab7ac@subdivisions.wooz.org> <592F4AC5.5040802@canterbury.ac.nz> <1496298986.1342619.995043544.5F93FAFA@webmail.messagingengine.com> Message-ID: 01.06.17 09:36, Benjamin Peterson ????: > Modern GCC can defend against these kinds of problems. If I introduce a > "goto fail" bug somewhere in Python, I get a nice warning: > ../Objects/abstract.c: In function ?PyObject_Type?: > ../Objects/abstract.c:35:5: warning: this ?if? clause does not guard... > [-Wmisleading-indentation] > if (o == NULL) > ^~ > ../Objects/abstract.c:37:9: note: ...this statement, but the latter is > misleadingly indented as if it is guarded by the ?if? > 2 + 3; > ^ > > This is not to say that simply bracing everything isn't the right way to > go. Actually a bug with misleadingly indented statement in CPython sources was fixed just few months ago (thanks to modern GCC). From victor.stinner at gmail.com Thu Jun 1 03:17:15 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 1 Jun 2017 09:17:15 +0200 Subject: [Python-Dev] PEP 7 and braces { .... } on if In-Reply-To: References: Message-ID: 2017-05-31 19:27 GMT+02:00 Guido van Rossum : > I interpret the PEP (...) Right, the phrasing requires to "interpret" it :-) > (...) as saying that you should use braces everywhere but not > to add them in code that you're not modifying otherwise. (I.e. don't go on a > brace-adding rampage.) If author and reviewer of a PR disagree I would go > with "add braces" since that's clearly the PEP's preference. This is C code. > We should play it safe. Would someone be nice enough to try to rephrase the PEP 7 to explain that? Just to avoid further boring discussion on the C coding style... Victor From larry at hastings.org Thu Jun 1 03:38:09 2017 From: larry at hastings.org (Larry Hastings) Date: Thu, 1 Jun 2017 00:38:09 -0700 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE Message-ID: When CPython's small block allocator was first merged in late February 2001, it allocated memory in gigantic chunks it called "arenas". These arenas were a massive 256 KILOBYTES apiece. This tunable parameter has not been touched in the intervening 16 years. Yet CPython's memory consumption continues to grow. By the time a current "trunk" build of CPython reaches the REPL prompt it's already allocated 16 arenas. I propose we make the arena size larger. By how much? I asked Victor to run some benchmarks with arenas of 1mb, 2mb, and 4mb. The results with 1mb and 2mb were mixed, but his benchmarks with a 4mb arena size showed measurable (>5%) speedups on ten benchmarks and no slowdowns. What would be the result of making the arena size 4mb? * CPython could no longer run on a computer where at startup it could not allocate at least one 4mb continguous block of memory. * CPython programs would die slightly sooner in out-of-memory conditions. * CPython programs would use more memory. How much? Hard to say. It depends on their allocation strategy. I suspect most of the time it would be < 3mb additional memory. But for pathological allocation strategies the difference could be significant. (e.g: lots of allocs, followed by lots of frees, but the occasional object lives forever, which means that the arena it's in can never be freed. If 1 out of ever 16 256k arenas is kept alive this way, and the objects are spaced out precisely such that now it's 1 for every 4mb arena, max memory use would be the same but later stable memory use would hypothetically be 16x current) * Many programs would be slightly faster now and then, simply because we call malloc() 1/16 as often. What say you? Vote for your favorite color of bikeshed now! //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Thu Jun 1 03:39:05 2017 From: larry at hastings.org (Larry Hastings) Date: Thu, 1 Jun 2017 00:39:05 -0700 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: References: Message-ID: On 06/01/2017 12:38 AM, Larry Hastings wrote: > I propose we make the arena size larger. By how much? I asked Victor > to run some benchmarks with arenas of 1mb, 2mb, and 4mb. The results > with 1mb and 2mb were mixed, but his benchmarks with a 4mb arena size > showed measurable (>5%) speedups on ten benchmarks and no slowdowns. Oh, sorry! Meant to add: thanks, Victor, for running these benchmarks for me! Where are my manners?! //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Thu Jun 1 03:57:04 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 1 Jun 2017 09:57:04 +0200 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: References: Message-ID: 2017-06-01 9:38 GMT+02:00 Larry Hastings : > When CPython's small block allocator was first merged in late February 2001, > it allocated memory in gigantic chunks it called "arenas". These arenas > were a massive 256 KILOBYTES apiece. The arena size defines the strict minimum memory usage of Python. With 256 kB, it means that the smallest memory usage is 256 kB. > What would be the result of making the arena size 4mb? A minimum memory usage of 4 MB. It also means that if you allocate 4 MB + 1 byte, Python will eat 8 MB from the operating system. The GNU libc malloc uses a variable threshold to choose between sbrk() (heap memory) or mmap(). It starts at 128 kB or 256 kB, and then is adapted depending on the workload (I don't know how exactly). I would prefer to have an adaptative arena size. For example start at 256 kB and then double the arena size while the memory usage grows. The problem is that pymalloc is currently designed for a fixed arena size. I have no idea how hard it would be to make the size per allocated arena. I already read that CPU support "large pages" between 2 MB and 1 GB, instead of just 4 kB. Using large pages can have a significant impact on performance. I don't know if we can do something to help the Linux kernel to use large pages for our memory? I don't know neither how we could do that :-) Maybe using mmap() closer to large pages will help Linux to join them to build a big page? (Linux has something magic to make applications use big pages transparently.) More generally: I'm strongly in favor of making our memory allocator more efficient :-D When I wrote my tracemalloc PEP 454, I counted that Python calls malloc() , realloc() or free() 270,000 times per second in average when running the Python test suite: https://www.python.org/dev/peps/pep-0454/#log-calls-to-the-memory-allocator (now I don't recall if it was really "malloc" or PyObject_Malloc, but well, we do a lot of memory allocations and deallocations ;-)) When I analyzed the timeline of CPython master performance, I was surprised to see that my change on PyMem_Malloc() to make it use pymalloc was one of the most significant "optimization" of the Python 3.6! http://pyperformance.readthedocs.io/cpython_results_2017.html#pymalloc-allocator The CPython performance heavily depends on the performance of our memory allocator, at least of the performance of pymalloc (the specialized allocator for blocks <= 512 bytes). By the way, Naoki INADA also proposed a different idea: "Global freepool: Many types has it?s own freepool. Sharing freepool can increase memory and cache efficiency. Add PyMem_FastFree(void* ptr, size_t size) to store memory block to freepool, and PyMem_Malloc can check global freepool first." http://faster-cpython.readthedocs.io/cpython37.html IMHO It's also worth it to investigate this change as well. Victor From larry at hastings.org Thu Jun 1 04:07:02 2017 From: larry at hastings.org (Larry Hastings) Date: Thu, 1 Jun 2017 01:07:02 -0700 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: References: Message-ID: On 06/01/2017 12:57 AM, Victor Stinner wrote: > I would prefer to have an adaptative arena size. For example start at > 256 kB and then double the arena size while the memory usage grows. > The problem is that pymalloc is currently designed for a fixed arena > size. I have no idea how hard it would be to make the size per > allocated arena. It's not hard. The major pain point is that it'd make the address_in_range() inline function slightly more expensive. Currently that code has ARENA_SIZE hardcoded inside it; if the size was dynamic we'd have to look up the size of the arena every time. This function is called every time we free a pointer, so it's done hundreds of thousands of times per second (as you point out). It's worth trying the experiment to see if dynamic arena sizes would make programs notably faster. However... why not both? Changing to 4mb arenas now is a one-line change, and on first examination seems mostly harmless, and yields an easy (if tiny) performance win. If someone wants to experiment with dynamic arenas, they could go right ahead, and if it works well we could merge that too. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Jun 1 04:19:06 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 1 Jun 2017 10:19:06 +0200 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE References: Message-ID: <20170601101906.60a7e65c@fsol> On Thu, 1 Jun 2017 00:38:09 -0700 Larry Hastings wrote: > * CPython programs would use more memory. How much? Hard to say. It > depends on their allocation strategy. I suspect most of the time it > would be < 3mb additional memory. But for pathological allocation > strategies the difference could be significant. (e.g: lots of > allocs, followed by lots of frees, but the occasional object lives > forever, which means that the arena it's in can never be freed. If > 1 out of ever 16 256k arenas is kept alive this way, and the objects > are spaced out precisely such that now it's 1 for every 4mb arena, > max memory use would be the same but later stable memory use would > hypothetically be 16x current) Yes, this is the same kind of reason the default page size is still 4KB on many platforms today, despite typical memory size having grown by a huge amount. Apart from the cost of fragmentation as you mentioned, another issue is when many small Python processes are running on a machine: a 2MB overhead per process can compound to large numbers if you have many (e.g. hundreds) such processes. I would suggest we exert caution here. Small benchmarks generally have a nice memory behaviour: not only they do not allocate a lot of memory a, but often they will release it all at once after a single run. Perhaps some of those benchmarks would even be better off if we allocated 64MB up front and never released it :-) Long-running applications can be less friendly than that, having various pieces of internal with unpredictable lifetimes (especially when it's talking over the network with other peers which come and go). And long-running applications are typically where Python memory usage is a sensitive matter. If you'd like to go that way anyway, I would suggest 1MB as a starting point in 3.7. > * Many programs would be slightly faster now and then, simply because > we call malloc() 1/16 as often. malloc() you said? Arenas are allocated using mmap() nowadays, right? Regards Antoine. From songofacandy at gmail.com Thu Jun 1 04:23:04 2017 From: songofacandy at gmail.com (INADA Naoki) Date: Thu, 1 Jun 2017 17:23:04 +0900 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: References: Message-ID: Hello. AFAIK, allocating arena doesn't eat real (physical) memory. * On Windows, VirtualAlloc is used for arena. Real memory page is assigned when the page is used first time. * On Linux and some other *nix, anonymous mmap is used. Real page is assigned when first touch, like Windows. Arena size is more important for **freeing** memory. Python returns memory to system when: 1. When no block in pool is used, it returned to arena. 2. When no pool is used, return the arena to system. So only one memory block can disturb returning the whole arena. Some VMs (e.g. mono) uses special APIs to return "real page" from allocated space. * On Windows, VirtualFree() + VirtualAlloc() can be used to unassign pages. * On Linux, madvice(..., MADV_DONTNEED) can be used. * On other *nix, madvice(..., MADV_DONTNEED) + madvice(..., MADV_FREE) can be used. See also: https://github.com/corngood/mono/blob/ef186403b5e95a5c95c38f1f19d0c8d061f2ac37/mono/utils/mono-mmap.c#L204-L208 (Windows) https://github.com/corngood/mono/blob/ef186403b5e95a5c95c38f1f19d0c8d061f2ac37/mono/utils/mono-mmap.c#L410-L424 (Unix) I think we can return not recently used free pools to system in same way. So more large arena size + more memory efficient can be achieved. But I need more experiment. Regards, From victor.stinner at gmail.com Thu Jun 1 04:34:23 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 1 Jun 2017 10:34:23 +0200 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: <20170601101906.60a7e65c@fsol> References: <20170601101906.60a7e65c@fsol> Message-ID: 2017-06-01 10:19 GMT+02:00 Antoine Pitrou : > Yes, this is the same kind of reason the default page size is still 4KB > on many platforms today, despite typical memory size having grown by a > huge amount. Apart from the cost of fragmentation as you mentioned, > another issue is when many small Python processes are running on a > machine: a 2MB overhead per process can compound to large numbers if > you have many (e.g. hundreds) such processes. > > I would suggest we exert caution here. Small benchmarks generally have > a nice memory behaviour: not only they do not allocate a lot of memory a, > but often they will release it all at once after a single run. Perhaps > some of those benchmarks would even be better off if we allocated 64MB > up front and never released it :-) By the way, the benchmark suite performance supports different ways to trace memory usage: * using tracemalloc * using /proc/pid/smaps * using VmPeak of /proc/pid/status (max RSS memory) I wrote the code but I didn't try it yet :-) Maybe we should check the memory usage before deciding to change the arena size? Victor From victor.stinner at gmail.com Thu Jun 1 04:37:01 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 1 Jun 2017 10:37:01 +0200 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: References: Message-ID: 2017-06-01 10:23 GMT+02:00 INADA Naoki : > AFAIK, allocating arena doesn't eat real (physical) memory. > > * On Windows, VirtualAlloc is used for arena. Real memory page is assigned > when the page is used first time. > * On Linux and some other *nix, anonymous mmap is used. Real page is > assigned when first touch, like Windows. Memory fragmentation is also a real problem in pymalloc. I don't think that pymalloc is designed to reduce the memory fragmentation. I know one worst case: the Python parser which allocates small objects which will be freed when the parser completes, while other objects living longer are created. https://github.com/haypo/misc/blob/master/memory/python_memleak.py In a perfect world, the parser should use a different memory allocator for that. But currently, the Python API doesn't offer this level of granularity. Victor From solipsis at pitrou.net Thu Jun 1 04:40:21 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 1 Jun 2017 10:40:21 +0200 Subject: [Python-Dev] "Global freepool" References: Message-ID: <20170601104021.31aee844@fsol> On Thu, 1 Jun 2017 09:57:04 +0200 Victor Stinner wrote: > > By the way, Naoki INADA also proposed a different idea: > > "Global freepool: Many types has it?s own freepool. Sharing freepool > can increase memory and cache efficiency. Add PyMem_FastFree(void* > ptr, size_t size) to store memory block to freepool, and PyMem_Malloc > can check global freepool first." This is already exactly how PyObject_Malloc() works. Really, the fast path for PyObject_Malloc() is: size = (uint)(nbytes - 1) >> ALIGNMENT_SHIFT; pool = usedpools[size + size]; if (pool != pool->nextpool) { /* * There is a used pool for this size class. * Pick up the head block of its free list. */ ++pool->ref.count; bp = pool->freeblock; assert(bp != NULL); if ((pool->freeblock = *(block **)bp) != NULL) { UNLOCK(); return (void *)bp; // <- fast path! } I don't think you can get much faster than that in a generic allocation routine (unless you have a compacting GC where allocating memory is basically bumping a single global pointer). IMHO the main thing the private freelists have is that they're *private* precisely, so they can avoid a couple of conditional branches. Regards Antoine. From larry at hastings.org Thu Jun 1 04:41:15 2017 From: larry at hastings.org (Larry Hastings) Date: Thu, 1 Jun 2017 01:41:15 -0700 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: <20170601101906.60a7e65c@fsol> References: <20170601101906.60a7e65c@fsol> Message-ID: <58f3233b-2b69-cf68-49dc-5549a1f32512@hastings.org> On 06/01/2017 01:19 AM, Antoine Pitrou wrote: > If you'd like to go that way anyway, I would suggest 1MB as a starting > point in 3.7. I understand the desire for caution. But I was hoping maybe we could experiment with 4mb in trunk for a while? We could change it to 1mb--or even 256k--before beta 1 if we get anxious. >> * Many programs would be slightly faster now and then, simply because >> we call malloc() 1/16 as often. > malloc() you said? Arenas are allocated using mmap() nowadays, right? malloc() and free(). See _PyObject_ArenaMalloc (etc) in Objects/obmalloc.c. On Windows Python uses VirtualAlloc(), and I don't know what the implications are of that. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Thu Jun 1 04:45:56 2017 From: larry at hastings.org (Larry Hastings) Date: Thu, 1 Jun 2017 01:45:56 -0700 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: <58f3233b-2b69-cf68-49dc-5549a1f32512@hastings.org> References: <20170601101906.60a7e65c@fsol> <58f3233b-2b69-cf68-49dc-5549a1f32512@hastings.org> Message-ID: On 06/01/2017 01:41 AM, Larry Hastings wrote: > On 06/01/2017 01:19 AM, Antoine Pitrou wrote: >> malloc() you said? Arenas are allocated using mmap() nowadays, right? > malloc() and free(). See _PyObject_ArenaMalloc (etc) in > Objects/obmalloc.c. Oh, sorry, I forgot how to read. If ARENAS_USE_MMAP is on it uses mmap(). I can't figure out when or how MAP_ANONYMOUS gets set, but if I step into the _PyObject_Arena.alloc() it indeed calls _PyObject_ArenaMmap() which uses mmap(). So, huzzah!, we use mmap() to allocate our enormous 256kb arenas. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Jun 1 04:52:15 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 1 Jun 2017 10:52:15 +0200 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE References: <20170601101906.60a7e65c@fsol> <58f3233b-2b69-cf68-49dc-5549a1f32512@hastings.org> Message-ID: <20170601105215.60abe547@fsol> On Thu, 1 Jun 2017 01:41:15 -0700 Larry Hastings wrote: > On 06/01/2017 01:19 AM, Antoine Pitrou wrote: > > If you'd like to go that way anyway, I would suggest 1MB as a starting > > point in 3.7. > > I understand the desire for caution. But I was hoping maybe we could > experiment with 4mb in trunk for a while? We could change it to 1mb--or > even 256k--before beta 1 if we get anxious. Almost nobody tests "trunk" (or "master" :-)) on production systems. At best a couple rare open source projects will run their test suite on the pre-release betas, but that's all. So we are unlikely to spot memory usage ballooning problems before the final release. > >> * Many programs would be slightly faster now and then, simply because > >> we call malloc() 1/16 as often. > > malloc() you said? Arenas are allocated using mmap() nowadays, right? > > malloc() and free(). See _PyObject_ArenaMalloc (etc) in Objects/obmalloc.c. _PyObject_ArenaMalloc should only be used if the OS doesn't support mmap() or MAP_ANONYMOUS (see ARENAS_USE_MMAP). Otherwise _PyObject_ArenaMmap is used. Apparently OS X doesn't have MAP_ANONYMOUS but it has the synonymous MAP_ANON: https://github.com/HaxeFoundation/hashlink/pull/12 Regards Antoine. From songofacandy at gmail.com Thu Jun 1 04:56:58 2017 From: songofacandy at gmail.com (INADA Naoki) Date: Thu, 1 Jun 2017 17:56:58 +0900 Subject: [Python-Dev] "Global freepool" In-Reply-To: <20170601104021.31aee844@fsol> References: <20170601104021.31aee844@fsol> Message-ID: Hi, As you said, I think PyObject_Malloc() is fast enough. But PyObject_Free() is somewhat complex. Actually, there are some freelists (e.g. tuple, dict, frame) and they improve performance significantly. My "global unified freelist" idea is unify them. The merit is: * Unify _PyXxx_DebugMallocStats(). Some freelists provide it but some doesn't. * Unify PyXxx_ClearFreeList(). Some freelists doesn't provide it and it may disturb returning memory to system. * Potential better CPU cache hit ratio by unifying LRU if some freelists has same memory block size. This idea is partially implemented in https://github.com/methane/cpython/pull/3 But there are no significant difference about speed or memory usage. Regards, On Thu, Jun 1, 2017 at 5:40 PM, Antoine Pitrou wrote: > On Thu, 1 Jun 2017 09:57:04 +0200 > Victor Stinner wrote: >> >> By the way, Naoki INADA also proposed a different idea: >> >> "Global freepool: Many types has it?s own freepool. Sharing freepool >> can increase memory and cache efficiency. Add PyMem_FastFree(void* >> ptr, size_t size) to store memory block to freepool, and PyMem_Malloc >> can check global freepool first." > > This is already exactly how PyObject_Malloc() works. Really, the fast > path for PyObject_Malloc() is: > > size = (uint)(nbytes - 1) >> ALIGNMENT_SHIFT; > pool = usedpools[size + size]; > if (pool != pool->nextpool) { > /* > * There is a used pool for this size class. > * Pick up the head block of its free list. > */ > ++pool->ref.count; > bp = pool->freeblock; > assert(bp != NULL); > if ((pool->freeblock = *(block **)bp) != NULL) { > UNLOCK(); > return (void *)bp; // <- fast path! > } > > > I don't think you can get much faster than that in a generic allocation > routine (unless you have a compacting GC where allocating memory is > basically bumping a single global pointer). IMHO the main thing the > private freelists have is that they're *private* precisely, so they can > avoid a couple of conditional branches. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com From solipsis at pitrou.net Thu Jun 1 04:57:03 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 1 Jun 2017 10:57:03 +0200 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 References: Message-ID: <20170601105703.753551fd@fsol> On Wed, 31 May 2017 14:09:20 -0600 Jim Baker wrote: > > But I object to a completely new feature being added to 2.7 to support the > implementation of event loop SSL usage. This feature cannot be construed as > a security fix, and therefore does not qualify as a feature that can be > added to CPython 2.7 at this point in its lifecycle. I agree with this sentiment. Also see comments by Ben Darnell and others here: https://github.com/python/peps/pull/272#pullrequestreview-41388700 Moreover I think that a 2.7 policy decision shouldn't depend on whatever future plans there are for Requests. The slippery slope of relaxing maintenance policy on 2.7 has come to absurd extremities. If Requests is to remain 2.7-compatible, it's up to Requests to do the necessary work to do so. Regards Antoine. From victor.stinner at gmail.com Thu Jun 1 05:03:04 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 1 Jun 2017 11:03:04 +0200 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: <58f3233b-2b69-cf68-49dc-5549a1f32512@hastings.org> References: <20170601101906.60a7e65c@fsol> <58f3233b-2b69-cf68-49dc-5549a1f32512@hastings.org> Message-ID: 2017-06-01 10:41 GMT+02:00 Larry Hastings : > On 06/01/2017 01:19 AM, Antoine Pitrou wrote: > If you'd like to go that way anyway, I would suggest 1MB as a starting > point in 3.7. > > I understand the desire for caution. But I was hoping maybe we could > experiment with 4mb in trunk for a while? We could change it to 1mb--or > even 256k--before beta 1 if we get anxious. While I fail to explain why in depth, I would prefer to *not* touch the default arena size at this point. We need more data, for example measure the memory usage on different workloads using different arena sizes. It's really hard to tune a memory allocator for *any* use cases. A simple enhancement would be to add an environment variable to change the arena size at Python startup. Example: PYTHONARENASIZE=1M. If you *know* that your application will allocate at least 2 GB, you may even want to try PYTHONARENASIZE=1G which is more likely to use a single large page... Such parameter cannot be used by default: it would make the default Python memory usage insane ;-) Victor From victor.stinner at gmail.com Thu Jun 1 05:13:52 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 1 Jun 2017 11:13:52 +0200 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <20170601105703.753551fd@fsol> References: <20170601105703.753551fd@fsol> Message-ID: 2017-06-01 10:57 GMT+02:00 Antoine Pitrou : > If Requests is to remain 2.7-compatible, it's up to Requests to do the > necessary work to do so. In practice, CPython does include Requests in ensurepip. Because of that, it means that Requests cannot use any C extension. CPython 2.7 ensurepip prevents evolutions of Requests on Python 3.7. Is my rationale broken somehow? The root issue is to get a very secure TLS connection in pip to download packages from pypi.python.org. On CPython 3.6, we made multiple small steps to include more and more features in the stdlib ssl module, but I understand that the lack of root certificate authorities (CA) on Windows and macOS is still a major blocker issue for pip. That's why pip uses Requests which uses certifi (Mozilla bundled root certificate authorities.) pip and so Requests are part of the current success of the Python community. I disagree that Requests pratical isssues are not our problems. -- Moreover, the PEP 546 Rationale not only include Requests, but also the important PEP 543 to make CPython 3.7 more secure in the long term. Do you also disagree on the need of the need of the PEP 546 (backport) to make the PEP 543 (new TLS API) feasible in practice? Victor From songofacandy at gmail.com Thu Jun 1 05:14:15 2017 From: songofacandy at gmail.com (INADA Naoki) Date: Thu, 1 Jun 2017 18:14:15 +0900 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: References: Message-ID: > * On Linux, madvice(..., MADV_DONTNEED) can be used. Recent Linux has MADV_FREE. It is faster than MADV_DONTNEED, https://lwn.net/Articles/591214/ From victor.stinner at gmail.com Thu Jun 1 05:20:05 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 1 Jun 2017 11:20:05 +0200 Subject: [Python-Dev] "Global freepool" In-Reply-To: <20170601104021.31aee844@fsol> References: <20170601104021.31aee844@fsol> Message-ID: 2017-06-01 10:40 GMT+02:00 Antoine Pitrou : > This is already exactly how PyObject_Malloc() works. (...) Oh ok, good to know... > IMHO the main thing the > private freelists have is that they're *private* precisely, so they can > avoid a couple of conditional branches. I would like to understand how private free lists are "so much" faster. In fact, I don't recall if someone *measured* the performance speedup of these free lists :-) By the way, the Linux kernel uses a "SLAB" allocator for the most common object types like inode. I'm curious to know if CPython would benefit of a similar allocator for our most common object types? For example types which already use a free list. https://en.wikipedia.org/wiki/Slab_allocation Victor From songofacandy at gmail.com Thu Jun 1 05:21:55 2017 From: songofacandy at gmail.com (INADA Naoki) Date: Thu, 1 Jun 2017 18:21:55 +0900 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: <171c176c-a099-20d4-c2a6-626bbce2eff2@gotplt.org> References: <171c176c-a099-20d4-c2a6-626bbce2eff2@gotplt.org> Message-ID: Thanks for detailed info. But I don't think it's a big problem. Arenas are returned to system by chance. So other processes shouldn't relying to it. And I don't propose to stop returning arena to system. I just mean per pool (part of arena) MADV_DONTNEED can reduce RSS. If we use very large arena, or stop returning arena to system, it can be problem. Regards, On Thu, Jun 1, 2017 at 6:05 PM, Siddhesh Poyarekar wrote: > On Thursday 01 June 2017 01:53 PM, INADA Naoki wrote: >> * On Linux, madvice(..., MADV_DONTNEED) can be used. > > madvise does not reduce the commit charge in the Linux kernel, so in > high consumption scenarios (and where memory overcommit is disabled or > throttled) you'll see programs dying with OOM despite the MADV_DONTNEED. > The way we solved it in glibc was to use mprotect to drop PROT_READ and > PROT_WRITE in blocks that we don't need when we detect that the system > is not configured to overcommit (using /proc/sys/vm/overcommit_memory). > You'll need to fix the protection again though if you want to reuse the > block. > > Siddhesh From antoine at python.org Thu Jun 1 05:23:25 2017 From: antoine at python.org (Antoine Pitrou) Date: Thu, 1 Jun 2017 11:23:25 +0200 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> Message-ID: <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> Le 01/06/2017 ? 11:13, Victor Stinner a ?crit : > That's why pip uses Requests which uses certifi (Mozilla > bundled root certificate authorities.) pip could use certifi without using Requests. My guess is that Requests is used mostly because it eases coding. > pip and so Requests are part of the current success of the Python > community. pip is, but I'm not convinced about Requests. If Requests didn't exist, people (including pip's developers) would use another HTTP-fetching library, they wouldn't switch to Go or Ruby. > Do you also disagree on the need of the need of the PEP 546 > (backport) to make the PEP 543 (new TLS API) feasible in practice? Yes, I disagree. We needn't backport that new API to Python 2.7. Perhaps it's time to be reasonable: Python 2.7 has been in bugfix-only mode for a very long time. Python 3.6 is out. We should move on. Regards Antoine. From thomas at python.org Thu Jun 1 05:25:42 2017 From: thomas at python.org (Thomas Wouters) Date: Thu, 1 Jun 2017 11:25:42 +0200 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: References: <20170601101906.60a7e65c@fsol> <58f3233b-2b69-cf68-49dc-5549a1f32512@hastings.org> Message-ID: On Thu, Jun 1, 2017 at 10:45 AM, Larry Hastings wrote: > On 06/01/2017 01:41 AM, Larry Hastings wrote: > > On 06/01/2017 01:19 AM, Antoine Pitrou wrote: > > malloc() you said? Arenas are allocated using mmap() nowadays, right? > > malloc() and free(). See _PyObject_ArenaMalloc (etc) in > Objects/obmalloc.c. > > > Oh, sorry, I forgot how to read. If ARENAS_USE_MMAP is on it uses > mmap(). I can't figure out when or how MAP_ANONYMOUS gets set, > MAP_ANONYMOUS is set by sys/mman.h (where the system supports it), just like the other MAP_* defines. > but if I step into the _PyObject_Arena.alloc() it indeed calls > _PyObject_ArenaMmap() which uses mmap(). So, huzzah!, we use mmap() to > allocate our enormous 256kb arenas. > > > */arry* > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > thomas%40python.org > > -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From songofacandy at gmail.com Thu Jun 1 05:37:17 2017 From: songofacandy at gmail.com (INADA Naoki) Date: Thu, 1 Jun 2017 18:37:17 +0900 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: References: Message-ID: x86's hugepage is 2MB. And some Linux enables "Transparent Huge Page" feature. Maybe, 2MB arena size is better for TLB efficiency. Especially, for servers having massive memory. On Thu, Jun 1, 2017 at 4:38 PM, Larry Hastings wrote: > > > When CPython's small block allocator was first merged in late February 2001, > it allocated memory in gigantic chunks it called "arenas". These arenas > were a massive 256 KILOBYTES apiece. > > This tunable parameter has not been touched in the intervening 16 years. > Yet CPython's memory consumption continues to grow. By the time a current > "trunk" build of CPython reaches the REPL prompt it's already allocated 16 > arenas. > > I propose we make the arena size larger. By how much? I asked Victor to > run some benchmarks with arenas of 1mb, 2mb, and 4mb. The results with 1mb > and 2mb were mixed, but his benchmarks with a 4mb arena size showed > measurable (>5%) speedups on ten benchmarks and no slowdowns. > > What would be the result of making the arena size 4mb? > > CPython could no longer run on a computer where at startup it could not > allocate at least one 4mb continguous block of memory. > CPython programs would die slightly sooner in out-of-memory conditions. > CPython programs would use more memory. How much? Hard to say. It depends > on their allocation strategy. I suspect most of the time it would be < 3mb > additional memory. But for pathological allocation strategies the > difference could be significant. (e.g: lots of allocs, followed by lots of > frees, but the occasional object lives forever, which means that the arena > it's in can never be freed. If 1 out of ever 16 256k arenas is kept alive > this way, and the objects are spaced out precisely such that now it's 1 for > every 4mb arena, max memory use would be the same but later stable memory > use would hypothetically be 16x current) > Many programs would be slightly faster now and then, simply because we call > malloc() 1/16 as often. > > > What say you? Vote for your favorite color of bikeshed now! > > > /arry > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com > From me at louie.lu Thu Jun 1 05:42:55 2017 From: me at louie.lu (Louie Lu) Date: Thu, 1 Jun 2017 17:42:55 +0800 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: References: Message-ID: For the ARENA_SIZE, will that be better to setting by ./configure first, and without hard code in c files? 2017-06-01 17:37 GMT+08:00 INADA Naoki : > x86's hugepage is 2MB. > And some Linux enables "Transparent Huge Page" feature. > > Maybe, 2MB arena size is better for TLB efficiency. > Especially, for servers having massive memory. > > > On Thu, Jun 1, 2017 at 4:38 PM, Larry Hastings wrote: >> >> >> When CPython's small block allocator was first merged in late February 2001, >> it allocated memory in gigantic chunks it called "arenas". These arenas >> were a massive 256 KILOBYTES apiece. >> >> This tunable parameter has not been touched in the intervening 16 years. >> Yet CPython's memory consumption continues to grow. By the time a current >> "trunk" build of CPython reaches the REPL prompt it's already allocated 16 >> arenas. >> >> I propose we make the arena size larger. By how much? I asked Victor to >> run some benchmarks with arenas of 1mb, 2mb, and 4mb. The results with 1mb >> and 2mb were mixed, but his benchmarks with a 4mb arena size showed >> measurable (>5%) speedups on ten benchmarks and no slowdowns. >> >> What would be the result of making the arena size 4mb? >> >> CPython could no longer run on a computer where at startup it could not >> allocate at least one 4mb continguous block of memory. >> CPython programs would die slightly sooner in out-of-memory conditions. >> CPython programs would use more memory. How much? Hard to say. It depends >> on their allocation strategy. I suspect most of the time it would be < 3mb >> additional memory. But for pathological allocation strategies the >> difference could be significant. (e.g: lots of allocs, followed by lots of >> frees, but the occasional object lives forever, which means that the arena >> it's in can never be freed. If 1 out of ever 16 256k arenas is kept alive >> this way, and the objects are spaced out precisely such that now it's 1 for >> every 4mb arena, max memory use would be the same but later stable memory >> use would hypothetically be 16x current) >> Many programs would be slightly faster now and then, simply because we call >> malloc() 1/16 as often. >> >> >> What say you? Vote for your favorite color of bikeshed now! >> >> >> /arry >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com >> > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/me%40louie.lu From solipsis at pitrou.net Thu Jun 1 05:50:14 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 1 Jun 2017 11:50:14 +0200 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE References: Message-ID: <20170601115014.18dcb755@fsol> On Thu, 1 Jun 2017 18:37:17 +0900 INADA Naoki wrote: > x86's hugepage is 2MB. > And some Linux enables "Transparent Huge Page" feature. > > Maybe, 2MB arena size is better for TLB efficiency. > Especially, for servers having massive memory. But, since Linux is able to merge pages transparently, we perhaps needn't allocate large pages explicitly. Another possible strategy is: allocate several arenas at once (using a larger mmap() call), and use MADV_DONTNEED to relinquish individual arenas. Regards Antoine. From rosuav at gmail.com Thu Jun 1 05:50:22 2017 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 1 Jun 2017 19:50:22 +1000 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> Message-ID: On Thu, Jun 1, 2017 at 7:23 PM, Antoine Pitrou wrote: >> Do you also disagree on the need of the need of the PEP 546 >> (backport) to make the PEP 543 (new TLS API) feasible in practice? > > Yes, I disagree. We needn't backport that new API to Python 2.7. > Perhaps it's time to be reasonable: Python 2.7 has been in bugfix-only > mode for a very long time. Python 3.6 is out. We should move on. But it is in *security fix* mode for at least another three years (ish). Proper use of TLS certificates is a security question. How hard would it be for the primary codebase of Requests to be written to use a C extension, but with a fallback *for pip's own bootstrapping only* that provides one single certificate - the authority that signs pypi.python.org? The point of the new system is that back-ends can be switched out; a stub back-end that authorizes only one certificate would theoretically be possible, right? Or am I completely misreading which part needs C? ChrisA From solipsis at pitrou.net Thu Jun 1 06:01:31 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 1 Jun 2017 12:01:31 +0200 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> Message-ID: <20170601120131.0e1cafc6@fsol> On Thu, 1 Jun 2017 19:50:22 +1000 Chris Angelico wrote: > On Thu, Jun 1, 2017 at 7:23 PM, Antoine Pitrou wrote: > >> Do you also disagree on the need of the need of the PEP 546 > >> (backport) to make the PEP 543 (new TLS API) feasible in practice? > > > > Yes, I disagree. We needn't backport that new API to Python 2.7. > > Perhaps it's time to be reasonable: Python 2.7 has been in bugfix-only > > mode for a very long time. Python 3.6 is out. We should move on. > > But it is in *security fix* mode for at least another three years > (ish). Proper use of TLS certificates is a security question. Why are you bringing "proper use of TLS certificates"? Python 2.7 doesn't need another backport for that. The certifi package is available for Python 2.7 and can be integrated simply with the existing ssl module. Regards Antoine. From rosuav at gmail.com Thu Jun 1 06:05:48 2017 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 1 Jun 2017 20:05:48 +1000 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <20170601120131.0e1cafc6@fsol> References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601120131.0e1cafc6@fsol> Message-ID: On Thu, Jun 1, 2017 at 8:01 PM, Antoine Pitrou wrote: > On Thu, 1 Jun 2017 19:50:22 +1000 > Chris Angelico wrote: >> On Thu, Jun 1, 2017 at 7:23 PM, Antoine Pitrou wrote: >> >> Do you also disagree on the need of the need of the PEP 546 >> >> (backport) to make the PEP 543 (new TLS API) feasible in practice? >> > >> > Yes, I disagree. We needn't backport that new API to Python 2.7. >> > Perhaps it's time to be reasonable: Python 2.7 has been in bugfix-only >> > mode for a very long time. Python 3.6 is out. We should move on. >> >> But it is in *security fix* mode for at least another three years >> (ish). Proper use of TLS certificates is a security question. > > Why are you bringing "proper use of TLS certificates"? Python 2.7 > doesn't need another backport for that. The certifi package is > available for Python 2.7 and can be integrated simply with the existing > ssl module. As stated in this thread, OS-provided certificates are not handled by that. For instance, if a local administrator distributes a self-signed cert for the intranet server, web browsers will use it, but pip will not. ChrisA From songofacandy at gmail.com Thu Jun 1 06:08:15 2017 From: songofacandy at gmail.com (INADA Naoki) Date: Thu, 1 Jun 2017 19:08:15 +0900 Subject: [Python-Dev] "Global freepool" In-Reply-To: References: <20170601104021.31aee844@fsol> Message-ID: I thought pymalloc is SLAB allocator. What is the difference between SLAB and pymalloc allocator? On Thu, Jun 1, 2017 at 6:20 PM, Victor Stinner wrote: > 2017-06-01 10:40 GMT+02:00 Antoine Pitrou : >> This is already exactly how PyObject_Malloc() works. (...) > > Oh ok, good to know... > >> IMHO the main thing the >> private freelists have is that they're *private* precisely, so they can >> avoid a couple of conditional branches. > > I would like to understand how private free lists are "so much" > faster. In fact, I don't recall if someone *measured* the performance > speedup of these free lists :-) > > By the way, the Linux kernel uses a "SLAB" allocator for the most > common object types like inode. I'm curious to know if CPython would > benefit of a similar allocator for our most common object types? For > example types which already use a free list. > > https://en.wikipedia.org/wiki/Slab_allocation > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com From storchaka at gmail.com Thu Jun 1 06:09:23 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 1 Jun 2017 13:09:23 +0300 Subject: [Python-Dev] "Global freepool" In-Reply-To: References: <20170601104021.31aee844@fsol> Message-ID: 01.06.17 12:20, Victor Stinner ????: > 2017-06-01 10:40 GMT+02:00 Antoine Pitrou : >> This is already exactly how PyObject_Malloc() works. (...) > > Oh ok, good to know... > >> IMHO the main thing the >> private freelists have is that they're *private* precisely, so they can >> avoid a couple of conditional branches. > > I would like to understand how private free lists are "so much" > faster. In fact, I don't recall if someone *measured* the performance > speedup of these free lists :-) I measured the performance boost of adding the free list for dict keys structures. [1] This proposition was withdraw in the favor of using PyObject_Malloc(). The latter solution is slightly slower, but simpler. But even private free lists are not fast enough. That is why some functions (zip, dict.items iterator, property getter, etc) have private caches for tuples and the FASTCALL protocol added so much speedup. At end we have multiple levels of free lists and caches, and every level adds good speedup (otherwise it wouldn't used). I also have found many times is spent in dealloc functions for tuples called before placing an object back in a free list or memory pool. They use the trashcan mechanism for guarding from stack overflow, and it is costly in comparison with clearing 1-element tuple. [1] https://bugs.python.org/issue16465 From solipsis at pitrou.net Thu Jun 1 06:18:12 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 1 Jun 2017 12:18:12 +0200 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601120131.0e1cafc6@fsol> Message-ID: <20170601121812.2c3270cd@fsol> On Thu, 1 Jun 2017 20:05:48 +1000 Chris Angelico wrote: > > As stated in this thread, OS-provided certificates are not handled by > that. For instance, if a local administrator distributes a self-signed > cert for the intranet server, web browsers will use it, but pip will > not. That's true. But: 1) pip could grow a config entry to set an alternative or additional CA path 2) it is not a "security fix", as not being able to recognize privately-signed certificates is not a security breach. It's a new feature Regards Antoine. From cory at lukasa.co.uk Thu Jun 1 06:22:21 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 1 Jun 2017 11:22:21 +0100 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> Message-ID: > On 1 Jun 2017, at 10:23, Antoine Pitrou wrote: > > Yes, I disagree. We needn't backport that new API to Python 2.7. > Perhaps it's time to be reasonable: Python 2.7 has been in bugfix-only > mode for a very long time. Python 3.6 is out. We should move on. Who is the ?we? that should move on? Python core dev? Or the Python ecosystem? Because if it?s the latter, then I?m going to tell you right now that the ecosystem did not get the memo. If you check the pip download numbers for Requests in the last month you?ll see that 80% of our downloads (9.4 million) come from Python 2. That is an enormous proportion: far too many to consider not supporting that user-base. So Requests is basically bound to support that userbase. Requests is stuck in a place from which it cannot move. We feel we cannot drop 2.7 support. We want to support as many TLS backends as possible. We want to enable the pip developers to focus on their features, rather than worrying about HTTP and TLS. And we want people to adopt the async/await keywords as much as possible. It turns out that we cannot satisfy all of those desires with the status quo, so we proposed an alternative that involves backporting MemoryBIO. So, to the notion of ?we need to move on?, I say this: we?re trying. We really, genuinely, are. I don?t know how much stronger of a signal I can give about how much Requests cares about Python 3 than to signal that we?re trying to adopt async/await and be compatible with asyncio. I believe that Python 3 is the present and future of this language. But right now, we can?t properly adopt it because we have a userbase that you want to leave behind, and we don?t. I want to move on, but I want to bring that 80% of our userbase with us when we do. My reading of your post is that you would rather Requests not adopt the async/await paradigm than backport MemoryBIO: is my understanding correct? If so, fair enough. If not, I?d like to try to work with you to a place where we can all get what we want. Cory From cory at lukasa.co.uk Thu Jun 1 06:23:03 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 1 Jun 2017 11:23:03 +0100 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <20170601121812.2c3270cd@fsol> References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601120131.0e1cafc6@fsol> <20170601121812.2c3270cd@fsol> Message-ID: <08AB00FA-E0AB-4D45-89C6-7AB6C6250F1D@lukasa.co.uk> > On 1 Jun 2017, at 11:18, Antoine Pitrou wrote: > > On Thu, 1 Jun 2017 20:05:48 +1000 > Chris Angelico wrote: >> >> As stated in this thread, OS-provided certificates are not handled by >> that. For instance, if a local administrator distributes a self-signed >> cert for the intranet server, web browsers will use it, but pip will >> not. > > That's true. But: > 1) pip could grow a config entry to set an alternative or additional CA > path No it can?t. Exporting the Windows or macOS security store to a big file of PEM is a security vulnerability because the macOS and Windows security stores expect to work with their own certificate chain building algorithms. OpenSSL builds chains differently, and disregards some metadata that Windows and macOS store, which means that cert validation will work differently than in the system store. This can lead to pip accepting a cert marked as ?untrusted for SSL?, for example, which would be pretty bad. Cory From antoine at python.org Thu Jun 1 06:28:57 2017 From: antoine at python.org (Antoine Pitrou) Date: Thu, 1 Jun 2017 12:28:57 +0200 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <08AB00FA-E0AB-4D45-89C6-7AB6C6250F1D@lukasa.co.uk> References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601120131.0e1cafc6@fsol> <20170601121812.2c3270cd@fsol> <08AB00FA-E0AB-4D45-89C6-7AB6C6250F1D@lukasa.co.uk> Message-ID: <0f8b1097-c060-ac32-8fc0-69dc95a1bb91@python.org> Le 01/06/2017 ? 12:23, Cory Benfield a ?crit : > > No it can?t. > > OpenSSL builds chains differently, and disregards some metadata that Windows and macOS store, which means that cert validation will work differently than in the system store. This can lead to pip accepting a cert marked as ?untrusted for SSL?, for example, which would be pretty bad. Are you claiming that OpenSSL certificate validation is insecure and shouldn't be used at all? I have never heard that claim before. Regards Antoine. From dw+python-dev at hmmz.org Thu Jun 1 06:39:14 2017 From: dw+python-dev at hmmz.org (David Wilson) Date: Thu, 1 Jun 2017 10:39:14 +0000 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> Message-ID: <20170601103914.GA8121@k3> Hi Cory, On Thu, Jun 01, 2017 at 11:22:21AM +0100, Cory Benfield wrote: > We want to support as many TLS backends as possible. Just a wild idea, but have you investigated a pure-Python fallback for 2.7 such as TLSlite? Of course the fallback need only be used during bootstrapping, and the solution would be compatible with every stable LTS Linux distribution release that was not shipping the latest and greatest 2.7. David From solipsis at pitrou.net Thu Jun 1 06:39:48 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 1 Jun 2017 12:39:48 +0200 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> Message-ID: <20170601123948.1c55678f@fsol> On Thu, 1 Jun 2017 11:22:21 +0100 Cory Benfield wrote: > > Who is the ?we? that should move on? Python core dev? Or the Python ecosystem? Sorry. Python core dev certainly. As for the rest of the ecosystem, it is moving on as well. > Requests is stuck in a place from which it cannot move. > We feel we cannot drop 2.7 support. We want to support as many TLS > backends as possible. Well, certain features could be 3.x-only, couldn't they? > We want to enable the pip developers to focus on > their features, rather than worrying about HTTP and TLS. And we want > people to adopt the async/await keywords as much as possible. I don't get what async/await keywords have to do with this. We're talking about backporting the ssl memory BIO object... (also, as much as I think asyncio is a good thing, I'm not sure it will do much for the problem of downloading packages from HTTP, even in parallel) > I want to move on, but I want to bring that 80% of our userbase with us when we do. My reading of your post is that you would rather Requests not adopt the async/await paradigm than backport MemoryBIO: is my understanding correct? Well you cannot use async/await on 2.7 in any case, and you cannot use asyncio on 2.7 (Trollius, which was maintained by Victor, has been abandoned AFAIK). If you want to use coroutines in 2.7, you need to use Tornado or Twisted. Twisted may not, but Tornado works fine with the stdlib ssl module. Regards Antoine. From cory at lukasa.co.uk Thu Jun 1 06:45:14 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 1 Jun 2017 11:45:14 +0100 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <0f8b1097-c060-ac32-8fc0-69dc95a1bb91@python.org> References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601120131.0e1cafc6@fsol> <20170601121812.2c3270cd@fsol> <08AB00FA-E0AB-4D45-89C6-7AB6C6250F1D@lukasa.co.uk> <0f8b1097-c060-ac32-8fc0-69dc95a1bb91@python.org> Message-ID: > On 1 Jun 2017, at 11:28, Antoine Pitrou wrote: > > > Le 01/06/2017 ? 12:23, Cory Benfield a ?crit : >> >> No it can?t. >> >> OpenSSL builds chains differently, and disregards some metadata that Windows and macOS store, which means that cert validation will work differently than in the system store. This can lead to pip accepting a cert marked as ?untrusted for SSL?, for example, which would be pretty bad. > > Are you claiming that OpenSSL certificate validation is insecure and > shouldn't be used at all? I have never heard that claim before. Of course I?m not. I am claiming that using OpenSSL certificate validation with root stores that are not intended for OpenSSL can be. This is because trust of a certificate is non-binary. For example, consider WoSign. The Windows TLS implementation will distrust certificates that chain up to WoSign as a root certificate that were issued after October 21 2016. This is not something that can currently be represented as a PEM file. Therefore, the person exporting the certs needs to choose: should that be exported or not? If it is, then OpenSSL will happily trust it even in situations where the system trust store would not. More generally, macOS allows the administrator to configure graduated trust: that is, to override whether or not a root should be trusted for certificate validation in some circumstances. Again, exporting this to a PEM does not persist this information. Cory From cory at lukasa.co.uk Thu Jun 1 06:47:31 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 1 Jun 2017 11:47:31 +0100 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <20170601103914.GA8121@k3> References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> Message-ID: <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> > On 1 Jun 2017, at 11:39, David Wilson wrote: > > Hi Cory, > > On Thu, Jun 01, 2017 at 11:22:21AM +0100, Cory Benfield wrote: > >> We want to support as many TLS backends as possible. > > Just a wild idea, but have you investigated a pure-Python fallback for > 2.7 such as TLSlite? Of course the fallback need only be used during > bootstrapping, and the solution would be compatible with every stable > LTS Linux distribution release that was not shipping the latest and > greatest 2.7. I have, but discarded the idea. There are no pure-Python TLS implementations that are both feature-complete and actively maintained. Additionally, doing crypto operations in pure-Python is a bad idea, so any implementation that did crypto in Python code would be ruled out immediately (which rules out TLSLite), so I?d need what amounts to a custom library: pure-Python TLS with crypto from OpenSSL, which is not currently exposed by any Python module. Ultimately it?s just not a winner. Cory From solipsis at pitrou.net Thu Jun 1 06:51:41 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 1 Jun 2017 12:51:41 +0200 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601120131.0e1cafc6@fsol> <20170601121812.2c3270cd@fsol> <08AB00FA-E0AB-4D45-89C6-7AB6C6250F1D@lukasa.co.uk> <0f8b1097-c060-ac32-8fc0-69dc95a1bb91@python.org> Message-ID: <20170601125141.5629e955@fsol> On Thu, 1 Jun 2017 11:45:14 +0100 Cory Benfield wrote: > > I am claiming that using OpenSSL certificate validation with root stores that are not intended for OpenSSL can be. This is because trust of a certificate is non-binary. For example, consider WoSign. The Windows TLS implementation will distrust certificates that chain up to WoSign as a root certificate that were issued after October 21 2016. This is not something that can currently be represented as a PEM file. Therefore, the person exporting the certs needs to choose: should that be exported or not? If it is, then OpenSSL will happily trust it even in situations where the system trust store would not. I was not talking about exporting the whole system CA as a PEM file, I was talking about adding an option for system adminstrators to configure an extra CA certificate to be recognized by pip. > More generally, macOS allows the administrator to configure graduated trust: that is, to override whether or not a root should be trusted for certificate validation in some circumstances. Again, exporting this to a PEM does not persist this information. How much of this is relevant to pip? Regards Antoine. From cory at lukasa.co.uk Thu Jun 1 07:01:41 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 1 Jun 2017 12:01:41 +0100 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <20170601123948.1c55678f@fsol> References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601123948.1c55678f@fsol> Message-ID: <48E229E5-B355-4953-BBA4-A09A0DF725F0@lukasa.co.uk> > On 1 Jun 2017, at 11:39, Antoine Pitrou wrote: > > On Thu, 1 Jun 2017 11:22:21 +0100 > Cory Benfield wrote: >> >> Who is the ?we? that should move on? Python core dev? Or the Python ecosystem? > > Sorry. Python core dev certainly. As for the rest of the ecosystem, it > is moving on as well. Moving, sure, but slowly. Again, I point to the 80% download number. >> Requests is stuck in a place from which it cannot move. >> We feel we cannot drop 2.7 support. We want to support as many TLS >> backends as possible. > > Well, certain features could be 3.x-only, couldn't they? In principle, sure. In practice, that means most of our users don?t use those features and so we don?t get any feedback on whether they?re good solutions to the problem. This is not great. Ideally we want features to be available across as wide a deploy base as possible, otherwise we risk shipping features that don?t solve the actual problem very well. Good software comes, in part, from getting user feedback. >> We want to enable the pip developers to focus on >> their features, rather than worrying about HTTP and TLS. And we want >> people to adopt the async/await keywords as much as possible. > > I don't get what async/await keywords have to do with this. We're > talking about backporting the ssl memory BIO object? All of this is related. I wrote a very, very long email initially and deleted it all because it was just too long to expect any normal human being to read it, but the TL;DR here is that we also want to support async/await, and doing so requires a memory BIO object. >> I want to move on, but I want to bring that 80% of our userbase with us when we do. My reading of your post is that you would rather Requests not adopt the async/await paradigm than backport MemoryBIO: is my understanding correct? > > Well you cannot use async/await on 2.7 in any case, and you cannot use > asyncio on 2.7 (Trollius, which was maintained by Victor, has been > abandoned AFAIK). If you want to use coroutines in 2.7, you need to > use Tornado or Twisted. Twisted may not, but Tornado works fine with > the stdlib ssl module. I can use Twisted on 2.7, and Twisted has great integration with async/await and asyncio when they are available. Great and getting greater, in fact, thanks to the work of the Twisted and asyncio teams. As to Tornado, the biggest concern there is that there is no support for composing the TLS over non-TCP sockets as far as I am aware. The wrapped socket approach is not suitable for some kinds of stream-based I/O that users really should be able to use with Requests (e.g. UNIX pipes). Not a complete non-starter, but also not something I?d like to forego. Cory From cory at lukasa.co.uk Thu Jun 1 07:05:04 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 1 Jun 2017 12:05:04 +0100 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <20170601125141.5629e955@fsol> References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601120131.0e1cafc6@fsol> <20170601121812.2c3270cd@fsol> <08AB00FA-E0AB-4D45-89C6-7AB6C6250F1D@lukasa.co.uk> <0f8b1097-c060-ac32-8fc0-69dc95a1bb91@python.org> <20170601125141.5629e955@fsol> Message-ID: <560F49D1-1430-4ED8-8BF5-CDF426DF3C49@lukasa.co.uk> > On 1 Jun 2017, at 11:51, Antoine Pitrou wrote: > > On Thu, 1 Jun 2017 11:45:14 +0100 > Cory Benfield wrote: >> >> I am claiming that using OpenSSL certificate validation with root stores that are not intended for OpenSSL can be. This is because trust of a certificate is non-binary. For example, consider WoSign. The Windows TLS implementation will distrust certificates that chain up to WoSign as a root certificate that were issued after October 21 2016. This is not something that can currently be represented as a PEM file. Therefore, the person exporting the certs needs to choose: should that be exported or not? If it is, then OpenSSL will happily trust it even in situations where the system trust store would not. > > I was not talking about exporting the whole system CA as a PEM file, I > was talking about adding an option for system adminstrators to > configure an extra CA certificate to be recognized by pip. Generally speaking system administrators aren?t wild about this option, as it means that they can only add to the trust store, not remove from it. So, while possible, it?s not a complete solution to this issue. I say this because the option *already* exists, at least in part, via the REQUESTS_CA_BUNDLE environment variable, and we nonetheless still get many complaints from system administrators. >> More generally, macOS allows the administrator to configure graduated trust: that is, to override whether or not a root should be trusted for certificate validation in some circumstances. Again, exporting this to a PEM does not persist this information. > > How much of this is relevant to pip? Depends. If the design goal is ?pip respects the system administrator?, then the answer is ?all of it?. An administrator wants to be able to configure their system trust settings. Ideally they want to do this once, and once only, such that all applications on their system respect it. Cory From dw+python-dev at hmmz.org Thu Jun 1 07:09:12 2017 From: dw+python-dev at hmmz.org (David Wilson) Date: Thu, 1 Jun 2017 11:09:12 +0000 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> Message-ID: <20170601110912.GB8121@k3> On Thu, Jun 01, 2017 at 11:47:31AM +0100, Cory Benfield wrote: > I have, but discarded the idea. I'm glad to hear it was given sufficent thought. :) I have one final 'crazy' idea, and actually it does not seem to bad at all: can't you just fork a subprocess or spawn threads to handle the blocking SSL APIs? Sure it wouldn't be beautiful, but it is more appealing than forcing an upgrade on all 2.7 users just so they can continue to use pip. (Which, ironically, seems to resonate strongly with the motivation behind all of this work -- allowing users to continue with their old environments without forcing an upgrade to 3.x!) David From cory at lukasa.co.uk Thu Jun 1 07:18:48 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 1 Jun 2017 12:18:48 +0100 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <20170601110912.GB8121@k3> References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> Message-ID: <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> > On 1 Jun 2017, at 12:09, David Wilson wrote: > > On Thu, Jun 01, 2017 at 11:47:31AM +0100, Cory Benfield wrote: > >> I have, but discarded the idea. > > I'm glad to hear it was given sufficent thought. :) > > I have one final 'crazy' idea, and actually it does not seem to bad at > all: can't you just fork a subprocess or spawn threads to handle the > blocking SSL APIs? > > Sure it wouldn't be beautiful, but it is more appealing than forcing an > upgrade on all 2.7 users just so they can continue to use pip. (Which, > ironically, seems to resonate strongly with the motivation behind all of > this work -- allowing users to continue with their old environments > without forcing an upgrade to 3.x!) So, this will work, but at a performance and code cleanliness cost. This essentially becomes a Python-2-only code-path, and a very large and complex one at that. This has the combined unfortunate effects of meaning a) a proportionally small fraction of our users get access to the code path we want to take forward into the future, and b) the majority of our users get an inferior experience of having a library either spawn threads or processes under their feet, which in Python has a tendency to get nasty fast (I for one have experienced the joy of having to ctrl+c multiple times to get a program using paramiko to actually die). Again, it?s worth noting that this change will not just affect pip but also the millions of Python 2 applications using Requests. I am ok with giving those users access to only part of the functionality that the Python 3 users get, but I?m not ok with that smaller part also being objectively worse than what we do today. Cory From solipsis at pitrou.net Thu Jun 1 07:20:52 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 1 Jun 2017 13:20:52 +0200 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <48E229E5-B355-4953-BBA4-A09A0DF725F0@lukasa.co.uk> References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601123948.1c55678f@fsol> <48E229E5-B355-4953-BBA4-A09A0DF725F0@lukasa.co.uk> Message-ID: <20170601132052.3e14466b@fsol> On Thu, 1 Jun 2017 12:01:41 +0100 Cory Benfield wrote: > In principle, sure. In practice, that means most of our users don?t use those features and so we don?t get any feedback on whether they?re good solutions to the problem. On bugs.python.org we get plenty of feedback from people using Python 3's features, and we have been for years. Your concern would have been very valid in the Python 3.2 timeframe, but I don't think it is anymore. > All of this is related. I wrote a very, very long email initially and deleted it all because it was just too long to expect any normal human being to read it, but the TL;DR here is that we also want to support async/await, and doing so requires a memory BIO object. async/await doesn't require a memory BIO object. For example, Tornado supports async/await (*) even though it doesn't use a memory BIO object for its SSL layer. And asyncio started with a non-memory BIO SSL implementation while still using "yield from". (*) Despite the fact that Tornado's own coroutines are yield-based generators. > As to Tornado, the biggest concern there is that there is no support for composing the TLS over non-TCP sockets as far as I am aware. The wrapped socket approach is not suitable for some kinds of stream-based I/O that users really should be able to use with Requests (e.g. UNIX pipes). Hmm, why would you use TLS on UNIX pipes except as an academic experiment? Tornado is far from a full-fledged networking package like Twisted, but its HTTP(S) support should be very sufficient (understandably, since it is the core use case for it). Regards Antoine. From cory at lukasa.co.uk Thu Jun 1 09:12:41 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 1 Jun 2017 14:12:41 +0100 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <20170601132052.3e14466b@fsol> References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601123948.1c55678f@fsol> <48E229E5-B355-4953-BBA4-A09A0DF725F0@lukasa.co.uk> <20170601132052.3e14466b@fsol> Message-ID: > On 1 Jun 2017, at 12:20, Antoine Pitrou wrote: > > On Thu, 1 Jun 2017 12:01:41 +0100 > Cory Benfield wrote: >> In principle, sure. In practice, that means most of our users don?t use those features and so we don?t get any feedback on whether they?re good solutions to the problem. > > On bugs.python.org we get plenty of feedback from people using Python > 3's features, and we have been for years. > > Your concern would have been very valid in the Python 3.2 timeframe, > but I don't think it is anymore. Ok? I guess? I don?t know what to do with that answer, really. I gave you some data (80%+ of requests downloads over the last month were Python 2), and you responded with ?it doesn?t cause us problems?. That?s good for you, I suppose, and well done, but it doesn?t seem immediately applicable to the concern I have. >> All of this is related. I wrote a very, very long email initially and deleted it all because it was just too long to expect any normal human being to read it, but the TL;DR here is that we also want to support async/await, and doing so requires a memory BIO object. > > async/await doesn't require a memory BIO object. For example, Tornado > supports async/await (*) even though it doesn't use a memory BIO object > for its SSL layer. And asyncio started with a non-memory BIO SSL > implementation while still using "yield from". > > (*) Despite the fact that Tornado's own coroutines are yield-based > generators. You are right, sorry. I should not have used the word ?require?. Allow me to rephrase. MemoryBIO objects are vastly, vastly more predictable and tractable than wrapped sockets when combined with non-blocking I/O. Using wrapped sockets and select/poll/epoll/kqueue, while possible, requires extremely subtle code that is easy to get wrong, and can nonetheless still have awkward bugs in it. I would be extremely loathe to use such an implementation, but you are correct, such an implementation can exist. >> As to Tornado, the biggest concern there is that there is no support for composing the TLS over non-TCP sockets as far as I am aware. The wrapped socket approach is not suitable for some kinds of stream-based I/O that users really should be able to use with Requests (e.g. UNIX pipes). > > Hmm, why would you use TLS on UNIX pipes except as an academic > experiment? Tornado is far from a full-fledged networking package like > Twisted, but its HTTP(S) support should be very sufficient > (understandably, since it is the core use case for it). Let me be clear that there is no intention to use either Tornado or Twisted?s HTTP/1.1 parsers or engines. With all due respect to both projects, I have concerns about both their client implementations. Tornado?s default is definitely not suitable for use in Requests, and the curl backend is but, surprise surprise, requires a C extension and oh god we?re back here again. I have similar concerns about Twisted?s default HTTP/1.1 client. Tornado?s HTTP/1.1 server is certainly sufficient, but also not of much use to Requests. Requests very much intends to use our own HTTP logic, not least because we?re sick of relying on someone else?s. Literally what we want is to have an event loop backing us that we can integrate with async/await and that requires us to reinvent as few wheels as possible while giving an overall better end-user experience. If I were to use Tornado, because I would want to integrate PEP 543 support into Tornado I?d ultimately have to rewrite Tornado?s TLS implementation *anyway* to replace it with a PEP 543 version. If I?m doing that, I?d much rather do it with MemoryBIO than wrapped sockets, for all of the reasons above. As a final note, because I think we?re getting into the weeds here: this is not *necessary*. None of this is *necessary*. Requests exists, and works today. We?ll get Windows TLS support regardless of anything that?s done here, because I?ll just shim it into urllib3 like we did for macOS. What I am pushing for with PEP 543 is an improvement that would benefit the whole ecosystem: all I want to do is to make it possible for me to actually use it and ship it to users in the tools I maintain. It is reasonable and coherent for python-dev to say ?well, good luck, but no backports to help you out?. The result of that is that I put PEP 543 on the backburner (because it doesn?t solve Requests/urllib3?s problems, and ultimately my day job is about resolving those issues), and probably that we shutter the async discussion for Requests until we drop Python 2 support. That?s fine, Python is your project, not mine. But I don?t see that there?s any reason for us not to ask for this backport. After all, the worst you can do is say no, and my problems remain the same. Cory From antoine at python.org Thu Jun 1 09:21:33 2017 From: antoine at python.org (Antoine Pitrou) Date: Thu, 1 Jun 2017 15:21:33 +0200 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601123948.1c55678f@fsol> <48E229E5-B355-4953-BBA4-A09A0DF725F0@lukasa.co.uk> <20170601132052.3e14466b@fsol> Message-ID: <5a1e869c-893f-04d6-68f5-89395453d12a@python.org> Le 01/06/2017 ? 15:12, Cory Benfield a ?crit : > > I don?t know what to do with that answer, really. I gave you some data (80%+ of requests downloads over the last month were Python 2), and you responded with ?it doesn?t cause us problems?. And indeed it doesn't. Unless the target user base for pip is widely different than Python's, it shouldn't cause you any problems either. > As a final note, because I think we?re getting into the weeds here: this is not *necessary*. None of this is *necessary*. Requests exists, and works today. And pip could even bundle a frozen 2.7-compatible version of Requests if it wanted/needed to... > Let me be clear that there is no intention to use either Tornado or Twisted?s HTTP/1.1 parsers or engines. [...] Requests very much intends to use our own HTTP logic, not least because we?re sick of relying on someone else?s. Then the PEP is really wrong or misleading in the way it states its own motivations. Regards Antoine. From cory at lukasa.co.uk Thu Jun 1 09:37:55 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 1 Jun 2017 14:37:55 +0100 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <5a1e869c-893f-04d6-68f5-89395453d12a@python.org> References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601123948.1c55678f@fsol> <48E229E5-B355-4953-BBA4-A09A0DF725F0@lukasa.co.uk> <20170601132052.3e14466b@fsol> <5a1e869c-893f-04d6-68f5-89395453d12a@python.org> Message-ID: <9D0A9679-D0D0-47B4-A8C3-2C95A40F924B@lukasa.co.uk> > On 1 Jun 2017, at 14:21, Antoine Pitrou wrote: > > > Le 01/06/2017 ? 15:12, Cory Benfield a ?crit : >> >> I don?t know what to do with that answer, really. I gave you some data (80%+ of requests downloads over the last month were Python 2), and you responded with ?it doesn?t cause us problems?. > > And indeed it doesn't. Unless the target user base for pip is widely > different than Python's, it shouldn't cause you any problems either. Maybe not now, but I think it?s fair to say that it did, right? As I recall, Python spent a long time with two fully supported Python versions, and then an even longer time with a version that was getting bugfixes. Tell me, which did you get more feedback on during that time? Generally speaking it is fair to say that at this point *every line of code in Requests* is exercised or depended on by one of our users. If we write new code available to a small fraction of them, and it is in any way sizeable, then that stops being true. Again, we should look at the fact that most libraries that successfully support Python 2 and Python 3 do so through having codebases that share as much code as possible between the two implementations. Each line of code that is exercised in only one implementation becomes a vector for a long, lingering bug. Anyway, all I know is that the last big project to do this kind of hard cut was Python, and while many of us are glad that Python 3 is real and glad that we pushed through the pain, I don?t think anyone would argue that the move was painless. A lesson can be learned there, especially for Requests which is not currently nursing a problem as fundamental to it as Python was. >> As a final note, because I think we?re getting into the weeds here: this is not *necessary*. None of this is *necessary*. Requests exists, and works today. > > And pip could even bundle a frozen 2.7-compatible version of Requests if > it wanted/needed to? Sure, if pip wants to internalise supporting and maintaining that version. One of the advantages of the pip/Requests relationship is that pip gets to stop worrying about HTTP: if there?s a HTTP problem, that?s on someone else to fix. Bundling that would remove that advantage. > >> Let me be clear that there is no intention to use either Tornado or > Twisted?s HTTP/1.1 parsers or engines. [...] Requests very much intends > to use our own HTTP logic, not least because we?re sick of relying on > someone else?s. > > Then the PEP is really wrong or misleading in the way it states its own > motivations. How so? TLS is not a part of the HTTP parser. It?s an intermediary layer between the transport (resolutely owned by the network layer in Twisted/Tornado) and the parsing layer (resolutely owned by Requests). Ideally we would not roll our own. Cory From solipsis at pitrou.net Thu Jun 1 09:53:19 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 1 Jun 2017 15:53:19 +0200 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601123948.1c55678f@fsol> <48E229E5-B355-4953-BBA4-A09A0DF725F0@lukasa.co.uk> <20170601132052.3e14466b@fsol> <5a1e869c-893f-04d6-68f5-89395453d12a@python.org> <9D0A9679-D0D0-47B4-A8C3-2C95A40F924B@lukasa.co.uk> Message-ID: <20170601155319.3375b044@fsol> On Thu, 1 Jun 2017 14:37:55 +0100 Cory Benfield wrote: > > > > And indeed it doesn't. Unless the target user base for pip is widely > > different than Python's, it shouldn't cause you any problems either. > > Maybe not now, but I think it?s fair to say that it did, right? Until Python 3.2 and perhaps 3.3, yes. Since 3.4, definitely not. For example asyncio quickly grew a sizable community around it, even though it had established Python 2-compatible competitors. > > Then the PEP is really wrong or misleading in the way it states its own > > motivations. > > How so? In the sentence "There are plans afoot to look at moving Requests to a more event-loop-y model, and doing so basically mandates a MemoryBIO", and also in the general feeling it gives that the backport is motivated by security reasons primarily. I understand that some users would like more features in Python 2.7. That has been the case since it was decided that feature development in the 2.x line would end in favour of Python 3 development. But our maintenance policy has been and is to develop new features on Python 3 (which some people have described as a "carrot" for migrating, which is certainly true). Regards Antoine. From cory at lukasa.co.uk Thu Jun 1 10:09:41 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 1 Jun 2017 15:09:41 +0100 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <20170601155319.3375b044@fsol> References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601123948.1c55678f@fsol> <48E229E5-B355-4953-BBA4-A09A0DF725F0@lukasa.co.uk> <20170601132052.3e14466b@fsol> <5a1e869c-893f-04d6-68f5-89395453d12a@python.org> <9D0A9679-D0D0-47B4-A8C3-2C95A40F924B@lukasa.co.uk> <20170601155319.3375b044@fsol> Message-ID: <26A86E7A-53E2-498E-B343-CD61941345DF@lukasa.co.uk> > On 1 Jun 2017, at 14:53, Antoine Pitrou wrote: > > On Thu, 1 Jun 2017 14:37:55 +0100 > Cory Benfield wrote: >>> >>> And indeed it doesn't. Unless the target user base for pip is widely >>> different than Python's, it shouldn't cause you any problems either. >> >> Maybe not now, but I think it?s fair to say that it did, right? > > Until Python 3.2 and perhaps 3.3, yes. Since 3.4, definitely not. For > example asyncio quickly grew a sizable community around it, even though > it had established Python 2-compatible competitors. Sure, but ?until 3.2? covers a long enough time to take us from now to ?deprecation of Python 2?. Given that the Requests team is 4 people, unlike python-dev?s much larger number, I suspect we?d have at least as much pain proportionally as Python did. I?m not wild about signing up for that. >>> Then the PEP is really wrong or misleading in the way it states its own >>> motivations. >> >> How so? > > In the sentence "There are plans afoot to look at moving Requests to a > more event-loop-y model, and doing so basically mandates a MemoryBIO", > and also in the general feeling it gives that the backport is motivated > by security reasons primarily. Ok, let?s address those together. There are security reasons to do the backport, but they are ?it helps us build a pathway to PEP 543?. Right now there are a lot of people interested in seeing PEP 543 happen, but vastly fewer in a position to do the work. I am, but only if I can actually use it for the things that are in my job. If I can?t, then PEP 543 becomes an ?evenings and weekends? activity for me *at best*, and something I have to drop entirely at worst. Adopting PEP 543 *would* be a security benefit, so while this PEP itself is not directly in and of itself a security benefit, it builds a pathway to something that is. As to the plans to move Requests to a more event loop-y model, I think that it does stand in the way of this, but only insomuch as, again, we want our event loopy model to be as bug-free as possible. But I can concede that rewording on that point would be valuable. *However*, it?s my understanding that even if I did that rewording, you?d still be against it. Is that correct? Cory From dw+python-dev at hmmz.org Thu Jun 1 10:10:00 2017 From: dw+python-dev at hmmz.org (David Wilson) Date: Thu, 1 Jun 2017 14:10:00 +0000 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> Message-ID: <20170601141000.GC8121@k3> On Thu, Jun 01, 2017 at 12:18:48PM +0100, Cory Benfield wrote: > So, this will work, but at a performance and code cleanliness cost. > This essentially becomes a Python-2-only code-path, and a very large > and complex one at that. "Doctor, it hurts when I do this .." Fine, then how about rather than exporting pip's problems on to the rest of the world (which an API change to a security module in a stable branch most certainly is, 18 months later when every other Python library starts depending on it), just drop SSL entirely, it will almost certainly cost less pain in the long run, and you can even arrange for the same code to run in both major versions. Drop SSL? But that's madness! Serve the assets over plain HTTP and tack a signature somewhere alongside it, either side-by-side in a file, embedded in a URL query string, or whatever. Here[0] is 1000 lines of pure Python that can validate a public key signature over a hash of the asset as it's downloaded. Embed the 32 byte public key in the pip source and hey presto. [0] https://github.com/jfindlay/pure_pynacl/blob/master/pure_pynacl/tweetnacl.py Finding someone to audit the signature checking capabilities of [0] will have vastly lower net cost than getting the world into a situation where pip no longer runs on the >1e6 EC2 instances that will be running Ubuntu 14.04/16.04 LTS until the turn of the next decade. Requests can't be installed without a working SSL implementation? Then drop requests, it's not like it does much for pip anyway. Downloads worldwide get a huge speedup due to lack of TLS handshake latency, a million Squid caching reverse proxies worldwide jump into action caching tarballs they previously couldn't see, pip's _vendor directory drops by 4.2MB, and Python package security depends on 1k lines of memory-safe code rather than possibly *the* worst example of security-unconcious C to come into existence since the birth of our industry. Sounds like a win to me. Maybe set a standard rather than blindly follow everyone else, at the cost of.. everyone else. David From ben at bendarnell.com Thu Jun 1 09:47:21 2017 From: ben at bendarnell.com (Ben Darnell) Date: Thu, 01 Jun 2017 13:47:21 +0000 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 Message-ID: Trying to transfer github comments from https://github.com/python/peps/pull/272#pullrequestreview-41388700: I said: > Tornado has been doing TLS in an event-loop model in python 2.5+ with just wrap_socket, no MemoryBIO necessary. What am I missing? MemoryBIO certainly gives some extra flexibility, but nothing I can see that's strictly required for an HTTP client. (Maybe it comes up in some proxy scenarios that Tornado hasn't implemented?) There were three main responses: - MemoryBIO is necessary to support TLS on windows with IOCP. Tornado's approach requires the less-efficient select() interface. This is valid and IMHO the biggest argument against using Tornado instead of Twisted in requests. Even if requests is willing to accept the limitation of not being able to use IOCP on Python 2, it may be tricky to arrange things so it can support both Tornado's select-based event loop on Python 2 and the IOCP-based interfaces in Python 3's asyncio (I'd volunteer to help with this if the requests team is interested in pursuing it, though). - wrap_socket is difficult to use correctly with an event loop; Twisted was happy to move away from it to the MemoryBIO model. My response: MemoryBIO is certainly a *better* solution for this problem, but it's not a *requirement*. Twisted prefers to do as little buffering as possible, which contributes to the difficulty of using wrap_socket. The buffering in Tornado's SSLIOStream simplifies this. Glyph reports that there are still some difficult-to-reproduce bugs; that may be but I haven't heard any other reports of this. I believe that whatever bugs might remain in this area are resolvable. - MemoryBIO supports a wider variety of transports, including pipes. There's a question about unix domain sockets - Tornado supports these generally but I haven't tried them with TLS. I would expect it to work. -------------- next part -------------- An HTML attachment was scrubbed... URL: From siddhesh at gotplt.org Thu Jun 1 05:05:00 2017 From: siddhesh at gotplt.org (Siddhesh Poyarekar) Date: Thu, 1 Jun 2017 14:35:00 +0530 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: References: Message-ID: <171c176c-a099-20d4-c2a6-626bbce2eff2@gotplt.org> On Thursday 01 June 2017 01:53 PM, INADA Naoki wrote: > * On Linux, madvice(..., MADV_DONTNEED) can be used. madvise does not reduce the commit charge in the Linux kernel, so in high consumption scenarios (and where memory overcommit is disabled or throttled) you'll see programs dying with OOM despite the MADV_DONTNEED. The way we solved it in glibc was to use mprotect to drop PROT_READ and PROT_WRITE in blocks that we don't need when we detect that the system is not configured to overcommit (using /proc/sys/vm/overcommit_memory). You'll need to fix the protection again though if you want to reuse the block. Siddhesh From victor.stinner at gmail.com Thu Jun 1 10:58:17 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 1 Jun 2017 16:58:17 +0200 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: References: <171c176c-a099-20d4-c2a6-626bbce2eff2@gotplt.org> Message-ID: It seems very complex and not portable at all to "free" a part of an arena. We already support freeing a whole arena using munmap(). I was a huge enhancement in our memory allocator. Change made in Python 2.5? I don't recall, ask Evan Jones: http://www.evanjones.ca/memoryallocator/ :-) I'm not sure that it's worth it to increase the arena size and try to implement the MADV_DONTNEED / MADV_FREE thing. Victor 2017-06-01 11:21 GMT+02:00 INADA Naoki : > Thanks for detailed info. > > But I don't think it's a big problem. > Arenas are returned to system by chance. So other processes > shouldn't relying to it. > > And I don't propose to stop returning arena to system. > I just mean per pool (part of arena) MADV_DONTNEED can reduce RSS. > > If we use very large arena, or stop returning arena to system, > it can be problem. > > Regards, > > On Thu, Jun 1, 2017 at 6:05 PM, Siddhesh Poyarekar wrote: >> On Thursday 01 June 2017 01:53 PM, INADA Naoki wrote: >>> * On Linux, madvice(..., MADV_DONTNEED) can be used. >> >> madvise does not reduce the commit charge in the Linux kernel, so in >> high consumption scenarios (and where memory overcommit is disabled or >> throttled) you'll see programs dying with OOM despite the MADV_DONTNEED. >> The way we solved it in glibc was to use mprotect to drop PROT_READ and >> PROT_WRITE in blocks that we don't need when we detect that the system >> is not configured to overcommit (using /proc/sys/vm/overcommit_memory). >> You'll need to fix the protection again though if you want to reuse the >> block. >> >> Siddhesh > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com From cory at lukasa.co.uk Thu Jun 1 11:01:54 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 1 Jun 2017 16:01:54 +0100 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <20170601141000.GC8121@k3> References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> Message-ID: > On 1 Jun 2017, at 15:10, David Wilson wrote: > Finding someone to audit the signature checking capabilities of [0] will > have vastly lower net cost than getting the world into a situation where > pip no longer runs on the >1e6 EC2 instances that will be running Ubuntu > 14.04/16.04 LTS until the turn of the next decade. So for the record I?m assuming most of the previous email was a joke: certainly it?s not going to happen. ;) But this is a real concern that does need to be addressed: Requests can?t meaningfully use this as its only TLS backend until it propagates to the wider 2.7 ecosystem, at least far enough such that pip can drop Python 2.7 releases lower than 2.7.14 (or wherever MemoryBIO ends up, if backported). So a concern emerges: if you grant my other premises about the utility of the backport, is it worth backporting at all? The answer to that is honestly not clear to me. I chatted with the pip developers, and they have 90%+ of their users currently on Python 2, but more than half of those are on 2.7.9 or later. This shows some interest in upgrading to newer Python 2s. The question, I think, is: do we end up in a position where a good number of developers are on 2.7.14 or later and only a very small fraction on 2.7.13 or earlier before the absolute number of Python 2 devs drops low enough to just drop Python 2? I don?t have an answer to that question. I have a gut instinct that says yes, probably, but a lack of certainty. My suspicion is that most of the core dev community believe the answer to that is ?no?. But I?d say that from my perspective this is the crux of the problem. We can hedge against this by just choosing to backport and accepting that it may never become useful, but a reasonable person can disagree and say that it?s just not worth the effort. Frankly, I think that amidst all the other arguments this is the one that most concretely needs answering, because if we don?t think Requests can ever meaningfully rely on the presence of MemoryBIO on 2.7 (where ?rely on? can be approximated to 90%+ of 2.7 users having access to it AND 2.7 still having non-trivial usage numbers) then ultimately this PEP doesn?t grant me much benefit. There are others who believe there are a few other benefits we could get from it (helping out Twisted etc.), but I don?t know that I?m well placed to make those arguments. (I also suspect I?d get accused of moving the goalposts.) Cory From solipsis at pitrou.net Thu Jun 1 11:56:57 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 1 Jun 2017 17:56:57 +0200 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <26A86E7A-53E2-498E-B343-CD61941345DF@lukasa.co.uk> References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601123948.1c55678f@fsol> <48E229E5-B355-4953-BBA4-A09A0DF725F0@lukasa.co.uk> <20170601132052.3e14466b@fsol> <5a1e869c-893f-04d6-68f5-89395453d12a@python.org> <9D0A9679-D0D0-47B4-A8C3-2C95A40F924B@lukasa.co.uk> <20170601155319.3375b044@fsol> <26A86E7A-53E2-498E-B343-CD61941345DF@lukasa.co.uk> Message-ID: <20170601175657.635cd828@fsol> On Thu, 1 Jun 2017 15:09:41 +0100 Cory Benfield wrote: > > As to the plans to move Requests to a more event loop-y model, I think that it does stand in the way of this, but only insomuch as, again, we want our event loopy model to be as bug-free as possible. But I can concede that rewording on that point would be valuable. > > *However*, it?s my understanding that even if I did that rewording, > you?d still be against it. Is that correct? Yes. It's just that it would more fairly inform the people reading it. Regards Antoine. From rosuav at gmail.com Thu Jun 1 12:14:17 2017 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 2 Jun 2017 02:14:17 +1000 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> Message-ID: On Fri, Jun 2, 2017 at 1:01 AM, Cory Benfield wrote: > The answer to that is honestly not clear to me. I chatted with the pip developers, and they have 90%+ of their users currently on Python 2, but more than half of those are on 2.7.9 or later. This shows some interest in upgrading to newer Python 2s. The question, I think, is: do we end up in a position where a good number of developers are on 2.7.14 or later and only a very small fraction on 2.7.13 or earlier before the absolute number of Python 2 devs drops low enough to just drop Python 2? > > I don?t have an answer to that question. I have a gut instinct that says yes, probably, but a lack of certainty. My suspicion is that most of the core dev community believe the answer to that is ?no?. > Let's see. Python 2 users include people on Windows who install it themselves, and then have no mechanism for automatic updates. They'll probably stay on whatever 2.7.x they first got, until something forces them to update. But it also includes people on stable Linux distros, where they have automatic updates provided by Red Hat or Debian or whomever, so a change like this WILL propagate - particularly (a) as the window is three entire years, and (b) if the change is considered important by the distro managers, which is a smaller group of people to convince than the users themselves. By 2020, Windows 7 will be out of support. By various estimates, Win 7 represents roughly half of all current Windows users. That means that, by 2020, at least half of today's Windows users will either have upgraded to a new OS (likely with a wipe-and-fresh-install, so they'll get a newer Python), or be on an unsupported OS, on par with people still running XP today. The same is true for probably close to 100% of Linux users, since any supported Linux distro will be shipping updates between now and 2020, and I don't know much about Mac OS updates, but I rather suspect that they'll also be updating. (Can anyone confirm?) So I'd be in the "yes" category. Across the next few years, I strongly suspect that 2.7.14 will propagate reasonably well. And I also strongly suspect that, even once 2020 hits and Python 2 stops getting updates, it will still be important to a lot of people. These numbers aren't backed by much, but it's slightly better than mere gut instinct. Do you have figures for how many people use pip on Windows vs Linux vs Mac OS? ChrisA From cory at lukasa.co.uk Thu Jun 1 12:35:43 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 1 Jun 2017 17:35:43 +0100 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> Message-ID: > On 1 Jun 2017, at 17:14, Chris Angelico wrote: > > > Do you have figures for how many people use pip on Windows vs Linux vs Mac OS? I have figures for the download numbers, which are an awkward proxy because most people don?t CI on Windows and macOS, but they?re the best we have. Linux has approximately 20x the download numbers of either Windows or macOS, and both Windows and macOS are pretty close together. These numbers are a bit confounded due to the fact that 1/4 of Linux?s downloads are made up of systems that don?t report their platform, so the actual ratio could be anywhere from about 25:1 to 3:1 in favour of Linux for either Windows or macOS. All of this is based on the downloads made in the last month. Again, an enormous number of these downloads are going to be CI downloads which overwhelmingly favour Linux systems. For some extra perspective, the next highest platform by download count is FreeBSD, with 0.04% of the downloads of Linux. HTH, Cory From rosuav at gmail.com Thu Jun 1 12:51:08 2017 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 2 Jun 2017 02:51:08 +1000 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> Message-ID: On Fri, Jun 2, 2017 at 2:35 AM, Cory Benfield wrote: > I have figures for the download numbers, which are an awkward proxy because most people don?t CI on Windows and macOS, but they?re the best we have. Linux has approximately 20x the download numbers of either Windows or macOS, and both Windows and macOS are pretty close together. These numbers are a bit confounded due to the fact that 1/4 of Linux?s downloads are made up of systems that don?t report their platform, so the actual ratio could be anywhere from about 25:1 to 3:1 in favour of Linux for either Windows or macOS. All of this is based on the downloads made in the last month. > > Again, an enormous number of these downloads are going to be CI downloads which overwhelmingly favour Linux systems. Hmm. So it's really hard to know. Pity. I suppose it's too much to ask for IP-based stat exclusion for the most commonly-used CI systems (Travis, Circle, etc)? Still, it does look like most pip usage happens on Linux. Also, it seems likely that the people who use Python and pip heavily are going to be the ones who most care about keeping up-to-date with point releases, so I still stand by my belief that yes, 2.7.14+ could take the bulk of 2.7's marketshare before 2.7 itself stops being significant. Thanks for the figures. ChrisA From victor.stinner at gmail.com Thu Jun 1 13:05:58 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 1 Jun 2017 19:05:58 +0200 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> Message-ID: 2017-06-01 18:51 GMT+02:00 Chris Angelico : > Hmm. So it's really hard to know. Pity. I suppose it's too much to ask > for IP-based stat exclusion for the most commonly-used CI systems > (Travis, Circle, etc)? Still, it does look like most pip usage happens > on Linux. Also, it seems likely that the people who use Python and pip > heavily are going to be the ones who most care about keeping > up-to-date with point releases, so I still stand by my belief that > yes, 2.7.14+ could take the bulk of 2.7's marketshare before 2.7 > itself stops being significant. It sems like PyPI statistics are public: https://langui.sh/2016/12/09/data-driven-decisions/ Another article on PyPI stats: https://hynek.me/articles/python3-2016/ 2.7: 419 millions (89%) 3.3+3.4+3.5+3.6: 51 millions (11%) (I ignored 2.6) Victor From barry at python.org Thu Jun 1 13:09:22 2017 From: barry at python.org (Barry Warsaw) Date: Thu, 1 Jun 2017 13:09:22 -0400 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> Message-ID: <20170601130922.3332361c@subdivisions.wooz.org> On Jun 02, 2017, at 02:14 AM, Chris Angelico wrote: >But it also includes people on stable Linux distros, where they have >automatic updates provided by Red Hat or Debian or whomever, so a change like >this WILL propagate - particularly (a) as the window is three entire years, >and (b) if the change is considered important by the distro managers, which >is a smaller group of people to convince than the users themselves. [...] >So I'd be in the "yes" category. Across the next few years, I strongly >suspect that 2.7.14 will propagate reasonably well. I'm not so sure about that, given long term support releases. For Ubuntu, LTS releases live for 5 years: https://www.ubuntu.com/info/release-end-of-life By 2020, only Ubuntu 16.04 and 18.04 will still be maintained, so while 18.04 will likely contain whatever the latest 2.7 is available at that time, 16.04 won't track upstream point releases, but instead will get select cherry picks. For good reason, there's a lot of overhead to backporting fixes into stable releases, and something as big as being suggested here would, in my best guess, have a very low chance of showing up in stable releases. -Barry From njs at pobox.com Thu Jun 1 13:10:51 2017 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 1 Jun 2017 10:10:51 -0700 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> Message-ID: On Jun 1, 2017 9:20 AM, "Chris Angelico" wrote: On Fri, Jun 2, 2017 at 1:01 AM, Cory Benfield wrote: > The answer to that is honestly not clear to me. I chatted with the pip developers, and they have 90%+ of their users currently on Python 2, but more than half of those are on 2.7.9 or later. This shows some interest in upgrading to newer Python 2s. The question, I think, is: do we end up in a position where a good number of developers are on 2.7.14 or later and only a very small fraction on 2.7.13 or earlier before the absolute number of Python 2 devs drops low enough to just drop Python 2? > > I don?t have an answer to that question. I have a gut instinct that says yes, probably, but a lack of certainty. My suspicion is that most of the core dev community believe the answer to that is ?no?. > Let's see. Python 2 users include people on Windows who install it themselves, and then have no mechanism for automatic updates. They'll probably stay on whatever 2.7.x they first got, until something forces them to update. But it also includes people on stable Linux distros, where they have automatic updates provided by Red Hat or Debian or whomever, so a change like this WILL propagate - particularly (a) as the window is three entire years, and (b) if the change is considered important by the distro managers, which is a smaller group of people to convince than the users themselves. I believe that for answering this question about the ssl module, it's really only Linux users that matter, since pip/requests/everyone else pushing for this only want to use ssl.MemoryBIO on Linux. Their plan on Windows/MacOS (IIUC) is to stop using the ssl module entirely in favor of new ctypes bindings for their respective native TLS libraries. (And yes, in principle it might be possible to write new ctypes-based bindings for openssl, but (a) this whole project is already teetering on the verge of being impossible given the resources available, so adding any major extra deliverable is likely to sink the whole thing, and (b) compared to the proprietary libraries, openssl is *much* harder and riskier to wrap at the ctypes level because it has different/incompatible ABIs depending on its micro version and the vendor who distributed it. This is why manylinux packages that need openssl have to ship their own, but pip can't and shouldn't ship its own openssl for many hopefully obvious reasons.) -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Thu Jun 1 13:22:06 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 1 Jun 2017 19:22:06 +0200 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <20170601130922.3332361c@subdivisions.wooz.org> References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <20170601130922.3332361c@subdivisions.wooz.org> Message-ID: 2017-06-01 19:09 GMT+02:00 Barry Warsaw : > By 2020, only Ubuntu 16.04 and 18.04 will still be maintained, so while 18.04 > will likely contain whatever the latest 2.7 is available at that time, 16.04 > won't track upstream point releases, but instead will get select cherry > picks. For good reason, there's a lot of overhead to backporting fixes into > stable releases, and something as big as being suggested here would, in my > best guess, have a very low chance of showing up in stable releases. I can help Canonical to backport MemoryBIO *if they want* to cherry-pick this feature ;-) Victor From victor.stinner at gmail.com Thu Jun 1 13:30:36 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 1 Jun 2017 19:30:36 +0200 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> Message-ID: 2017-06-01 19:10 GMT+02:00 Nathaniel Smith : > (...) since pip/requests/everyone else pushing for > this only want to use ssl.MemoryBIO on Linux. Their plan on Windows/MacOS > (IIUC) is to stop using the ssl module entirely in favor of new ctypes > bindings for their respective native TLS libraries. The long term plan include one Windows implementation, one macOS implementation and one implementation using the stdlib ssl module. But it seems like right now, Cory is working alone and has limited time to implement his PEP 543 (new TLS API). The short term plans is to implement the strict minimum implementation, the one relying on the existing stdlib ssl module. Backporting MemoryBIO makes it possible to get the new TLS API "for free" on Python 2.7. IMHO Python 2.7 support is a requirement to make the PEP popular enough to make it successful. The backport is supposed to fix a chicken-and-egg issue :-) > (And yes, in principle it might be possible to write new ctypes-based > bindings for openssl, but (...)) A C extension can also be considered, but I trust more code in CPython stdlib, since it would be well tested by our big farm of buildbots and have more eyes looking to the code. -- It seems like the PEP 546 (backport MemoryBIO) should make it more explicit that MemoryBIO support will be "optional": it's ok if Jython or PyPy doesn't implement it. It's ok if old Python 2.7 versions don't implement it. I expect anyway to use a fallback for those. It's just that I would prefer to avoid a fallback (likely a C extension) whenever possible, since it would cause various issues, especially for C code using OpenSSL: OpenSSL API changed many times :-/ Victor From larry at hastings.org Thu Jun 1 14:17:12 2017 From: larry at hastings.org (Larry Hastings) Date: Thu, 1 Jun 2017 11:17:12 -0700 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: References: <20170601101906.60a7e65c@fsol> <58f3233b-2b69-cf68-49dc-5549a1f32512@hastings.org> Message-ID: <86bce169-4299-bdb0-c4b7-b0cc581473c4@hastings.org> On 06/01/2017 02:03 AM, Victor Stinner wrote: > 2017-06-01 10:41 GMT+02:00 Larry Hastings : >> On 06/01/2017 01:19 AM, Antoine Pitrou wrote: >>> If you'd like to go that way anyway, I would suggest 1MB as a starting >>> point in 3.7. >> >> I understand the desire for caution. But I was hoping maybe we could >> experiment with 4mb in trunk for a while? We could change it to 1mb--or >> even 256k--before beta 1 if we get anxious. > While I fail to explain why in depth, I would prefer to *not* touch > the default arena size at this point. > > We need more data, for example measure the memory usage on different > workloads using different arena sizes. I can't argue with collecting data at this point in the process. My thesis is simply "the correct value for this tunable parameter in 2001 is probably not the same value in 2017". I don't mind proceeding *slowly* or gathering more data or what have you for now. But I would like to see it change somehow between now and 3.7.0b1, because my sense is that we can get some performance for basically free by updating the value. If ARENA_SIZE tracked Moore's Law, meaning that we doubled it every 18 months like clockwork, it'd currently be 2**10 times bigger: 256MB, and we'd be changing it to 512MB at the end of August. (And yes, as a high school student I was once bitten by a radioactive optimizer, so these days when I'm near possible optimizations my spider-sense--uh, I mean, my optimization-sense--starts tingling.) > A simple enhancement would be to add an environment variable to change > the arena size at Python startup. Example: PYTHONARENASIZE=1M. Implementing this would slow down address_in_range which currently compiles in arena size. It'd be by a tiny amount, but this inline function gets called very very frequently. It's possible this wouldn't hurt performance, but my guess is it'd offset the gains we got from larger arenas, and the net result would be no faster or slightly slower. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Thu Jun 1 14:22:48 2017 From: barry at python.org (Barry Warsaw) Date: Thu, 1 Jun 2017 14:22:48 -0400 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <20170601130922.3332361c@subdivisions.wooz.org> Message-ID: <20170601142248.2f46ce36@presto> On Jun 01, 2017, at 07:22 PM, Victor Stinner wrote: >I can help Canonical to backport MemoryBIO *if they want* to >cherry-pick this feature ;-) (Pedantically speaking, this falls under the Ubuntu project's responsibility, not directly Canonical.) Writing the patch is only part of the process: https://wiki.ubuntu.com/StableReleaseUpdates There's also Debian to consider. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 801 bytes Desc: OpenPGP digital signature URL: From larry at hastings.org Thu Jun 1 14:44:11 2017 From: larry at hastings.org (Larry Hastings) Date: Thu, 1 Jun 2017 11:44:11 -0700 Subject: [Python-Dev] "Global freepool" In-Reply-To: References: <20170601104021.31aee844@fsol> Message-ID: On 06/01/2017 02:20 AM, Victor Stinner wrote: > I would like to understand how private free lists are "so much" > faster. In fact, I don't recall if someone *measured* the performance > speedup of these free lists :-) I have, recently, kind of by accident. When working on the Gilectomy I turned off some freelists as they were adding "needless" complexity and presumably weren't helping performance that much. Then I turned them back on because it turned out they really did help. My intuition is that the help in two major ways: * Since they're a known size, you don't need to go through the general-case code of looking up the right spot in usedpools (etc) to get one / put one back in malloc/free. * The code that recycles these objects assumes that objects from its freelist are already mostly initialized, so it doesn't need to initialize them. The really crazy one is PyFrameObjects. The global freelist for these stores up to 200 (I think) in a stack, implemented as a simple linked list. When CPython wants a new frame object, it takes the top one off the stack and uses it. Where it gets crazy is: PyFrameObjects are dynamically sized, based on the number of arguments + local variables + stack + freevars + cellvars. So the frame you pull off the free list might not be big enough. If it isn't big enough, the code calls *realloc* on it then uses it. This seems like such a weird approach to me. But it's obviously a successful approach, and I've learned not to argue with success. p.s. Speaking of freelists, at one point Serhiy had a patch adding a freelist for single- and I think two-digit ints. Right now the only int creation optimization we have is the array of constant "small ints"; if the int you're constructing isn't one of those, we use the normal slow allocation path with PyObject_Alloc etc. IIRC this patch made things faster. Serhiy, what happened to that patch? Was it actually a bad idea, or did it just get forgotten? //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Jun 1 15:51:28 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 1 Jun 2017 20:51:28 +0100 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> Message-ID: On 1 June 2017 at 17:14, Chris Angelico wrote: > Python 2 users include people on Windows who install it themselves, > and then have no mechanism for automatic updates. They'll probably > stay on whatever 2.7.x they first got, until something forces them to > update. But it also includes people on stable Linux distros, where > they have automatic updates provided by Red Hat or Debian or whomever, > so a change like this WILL propagate - particularly (a) as the window > is three entire years, and (b) if the change is considered important > by the distro managers, which is a smaller group of people to convince > than the users themselves. However, it is trivial for Windows users to upgrade if asked to, as there's no issue around system packages depending on a particular version (or indeed, much of anything depending - 3rd party applications on Windows bundle their own Python, they don't use the globally installed one). So in principle, there should be no problem expecting Windows users to be on the latest version of 2.7.x. In fact, I suspect that the proportion of Windows users on Python 3 is noticeably higher than the proportion of Linux/Mac OS users on Python 3 (for the same reason). So this problem may overall be less pressing for Windows users. I have no evidence that isn't anecdotal to back this last assertion up, though. Linux users often use the OS-supplied Python, and so getting the distributions to upgrade, and to backport upgrades to old versions of their OS and (push those backports as required updates) is the route to get the bulk of the users there. Experience on pip seems to indicate this is unlikely to happen, in practice. Mac OS users who use the system Python are, as I understand it, stuck with a pretty broken version (I don't know if newer versions of the OS change that). But distributions like Macports are more common and more up to date. Apart from the Windows details, these are purely my impressions. > Do you have figures for how many people use pip on Windows vs Linux vs Mac OS? No. But we do get plenty of bug reports from Windows users, so I don't think there's any reason to assume it's particularly low (given the relative numbers of *python* users - in fact, it may be proportionately higher as Windows users don't have alternative options like yum). Paul From dw+python-dev at hmmz.org Thu Jun 1 15:55:46 2017 From: dw+python-dev at hmmz.org (David Wilson) Date: Thu, 1 Jun 2017 19:55:46 +0000 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> Message-ID: <20170601195546.GA12874@k3> On Thu, Jun 01, 2017 at 04:01:54PM +0100, Cory Benfield wrote: > > lower net cost than getting the world into a situation where pip no > > longer runs on the >1e6 EC2 instances that will be running Ubuntu > > 14.04/16.04 LTS until the turn of the next decade. > So for the record I?m assuming most of the previous email was a joke: > certainly it?s not going to happen. ;) > But this is a real concern that does need to be addressed Unfortunately it wasn't, but at least I'm glad to have accidentally made a valid point amidst the cloud of caffeine-fuelled irritability :/ Apologies for the previous post, it was hardly constructive. David From donald at stufft.io Thu Jun 1 15:20:01 2017 From: donald at stufft.io (Donald Stufft) Date: Thu, 1 Jun 2017 15:20:01 -0400 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <20170601130922.3332361c@subdivisions.wooz.org> References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <20170601130922.3332361c@subdivisions.wooz.org> Message-ID: <211FC21E-DBFA-48CF-9B63-70D2E47BF609@stufft.io> > On Jun 1, 2017, at 1:09 PM, Barry Warsaw wrote: > > On Jun 02, 2017, at 02:14 AM, Chris Angelico wrote: > >> But it also includes people on stable Linux distros, where they have >> automatic updates provided by Red Hat or Debian or whomever, so a change like >> this WILL propagate - particularly (a) as the window is three entire years, >> and (b) if the change is considered important by the distro managers, which >> is a smaller group of people to convince than the users themselves. > [...] >> So I'd be in the "yes" category. Across the next few years, I strongly >> suspect that 2.7.14 will propagate reasonably well. > > I'm not so sure about that, given long term support releases. For Ubuntu, LTS > releases live for 5 years: > > https://www.ubuntu.com/info/release-end-of-life > > By 2020, only Ubuntu 16.04 and 18.04 will still be maintained, so while 18.04 > will likely contain whatever the latest 2.7 is available at that time, 16.04 > won't track upstream point releases, but instead will get select cherry > picks. For good reason, there's a lot of overhead to backporting fixes into > stable releases, and something as big as being suggested here would, in my > best guess, have a very low chance of showing up in stable releases. > Using 2.7.9 as a sort of benchmark here, currently 26% of downloads from PyPI are using a version of Python older than 2.7.9, 2 months ago that number was 31%. (That?s total across all Python versions). Python >= 2.7.9, <3 is at 43% (previously 53%). So in ~2.5 years 2.7.9+ has become > 50% of all downloads from PyPI while older versions of Python 2.7 are down to only ~25% of the total number of downloads made by pip. I was also curious about how this had changed over the past year instead of just the past two months, a year ago >=2.7,<2.7.9 accounted for almost 50% of all downloads from PyPI (compared to the 25% today). It *looks* like on average we?re dropping somewhere between 1.5% and 2% each month so a conservative estimate if these numbers hold, we?re be looking at single digit numbers for >=2.7,<2.7.9 in roughly 11 months, or 3.5 years after the release of 2.7.9. If we assume that the hypothetical 2.7.14 w/ MemoryBio support would follow a similar adoption curve, we would expect to be able to mandate it for pip/etc in at a worst case scenario, 3-4 years after release. In addition to that, pip 9 comes with a new feature that makes it easier to sunset support for versions of Python without breaking the world [1]. The likely scenario is that while pip 9+ is increasing in share Python <2.7.14 will be decreasing, and that would mean that we could *likely* start mandating it earlier, maybe at the 2 year mark or so. [1] An astute reader might ask, why could you not use this same mechanism to simply move on to only supporting Python 3? It?s true we could do that, however as a rule we generally try to keep support for Pythons until the usage drops below some threshold, where that threshold varies based on how hard it is to continue supporting that version of Python and what the ?win? is in terms of dropping it. Since we?re still at 90% of downloads from PyPI being done using Python 2, that suggests the threshold for Python 3.x is very far away and will extend beyond 2020 (I mean, we?re just *now* finally able to drop support for Python 2.6). In case it?s not obvious, I am very much in support of this PEP. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Jun 1 15:57:27 2017 From: donald at stufft.io (Donald Stufft) Date: Thu, 1 Jun 2017 15:57:27 -0400 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> Message-ID: <8AFC9059-96F2-4CD1-9493-85C7407B0119@stufft.io> > On Jun 1, 2017, at 3:51 PM, Paul Moore wrote: > > Linux users often use the OS-supplied Python, and so getting the > distributions to upgrade, and to backport upgrades to old versions of > their OS and (push those backports as required updates) is the route > to get the bulk of the users there. Experience on pip seems to > indicate this is unlikely to happen, in practice. Mac OS users who use > the system Python are, as I understand it, stuck with a pretty broken > version (I don't know if newer versions of the OS change that). But > distributions like Macports are more common and more up to date. > Note that on macOS, within the next year macOS users using the system Python are going to be unable to talk to PyPI anyways (unless Apple does something here, which I think they will), but in either case, Apple was pretty good about upgrading to 2.7.9 (I think they had the first OS released that supported 2.7.9?). ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Jun 1 16:14:37 2017 From: donald at stufft.io (Donald Stufft) Date: Thu, 1 Jun 2017 16:14:37 -0400 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <8AFC9059-96F2-4CD1-9493-85C7407B0119@stufft.io> References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <8AFC9059-96F2-4CD1-9493-85C7407B0119@stufft.io> Message-ID: > On Jun 1, 2017, at 3:57 PM, Donald Stufft wrote: > > Note that on macOS, within the next year macOS users using the system Python are going to be unable to talk to PyPI anyways (unless Apple does something here, which I think they will), but in either case, Apple was pretty good about upgrading to 2.7.9 (I think they had the first OS released that supported 2.7.9?). Forgot to mention that pip 10.0 will work around this, thus forcing macOS users to upgrade or be cut off. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Thu Jun 1 16:16:53 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 1 Jun 2017 23:16:53 +0300 Subject: [Python-Dev] "Global freepool" In-Reply-To: References: <20170601104021.31aee844@fsol> Message-ID: 01.06.17 21:44, Larry Hastings ????: > p.s. Speaking of freelists, at one point Serhiy had a patch adding a > freelist for single- and I think two-digit ints. Right now the only int > creation optimization we have is the array of constant "small ints"; if > the int you're constructing isn't one of those, we use the normal slow > allocation path with PyObject_Alloc etc. IIRC this patch made things > faster. Serhiy, what happened to that patch? Was it actually a bad > idea, or did it just get forgotten? The issue [1] still is open. Patches neither applied nor rejected. They exposes the speed up in microbenchmarks, but it is not large. Up to 40% for iterating over enumerate() and 5-7% for hard integer computations like base85 encoding or spectral_norm benchmark. [1] https://bugs.python.org/issue25324 From steve.dower at python.org Thu Jun 1 15:59:15 2017 From: steve.dower at python.org (Steve Dower) Date: Thu, 1 Jun 2017 12:59:15 -0700 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> Message-ID: <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> On 01Jun2017 1010, Nathaniel Smith wrote: > I believe that for answering this question about the ssl module, it's > really only Linux users that matter, since pip/requests/everyone else > pushing for this only want to use ssl.MemoryBIO on Linux. Their plan on > Windows/MacOS (IIUC) is to stop using the ssl module entirely in favor > of new ctypes bindings for their respective native TLS libraries. > > (And yes, in principle it might be possible to write new ctypes-based > bindings for openssl, but (a) this whole project is already teetering on > the verge of being impossible given the resources available, so adding > any major extra deliverable is likely to sink the whole thing, and (b) > compared to the proprietary libraries, openssl is *much* harder and > riskier to wrap at the ctypes level because it has > different/incompatible ABIs depending on its micro version and the > vendor who distributed it. This is why manylinux packages that need > openssl have to ship their own, but pip can't and shouldn't ship its own > openssl for many hopefully obvious reasons.) How much of a stop-gap would it be (for Windows at least) to override OpenSSL's certificate validation with a call into the OS? This leaves most of the work with OpenSSL, but lets the OS say yes/no to the certificates based on its own configuration. For Windows, this is under 100 lines of C code in (probably) _ssl, and while I think an SChannel based approach is the better way to go long-term,[1] offering platform-specific certificate validation as the default in 2.7 is far more palatable than backporting new public API. I can't speak to whether there is an equivalent function for Mac (validate a certificate chain given the cert blob). Cheers, Steve [1]: though I've now spent hours looking at it and still have no idea how it's supposed to actually work... From siddhesh at gotplt.org Thu Jun 1 04:55:22 2017 From: siddhesh at gotplt.org (Siddhesh Poyarekar) Date: Thu, 1 Jun 2017 14:25:22 +0530 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: References: Message-ID: On Thursday 01 June 2017 01:27 PM, Victor Stinner wrote: > The GNU libc malloc uses a variable threshold to choose between sbrk() > (heap memory) or mmap(). It starts at 128 kB or 256 kB, and then is > adapted depending on the workload (I don't know how exactly). The threshold starts at 128K and increases whenever an mmap'd block is freed. For example, if the program allocates 2M (which is returned using mmap) and then frees that block, glibc malloc assumes that 2M blocks will be needed again and optimizes that allocation by increasing the threshold to 2M. This works well in practice for common programs but it has been known to cause issues in some cases, which is why there's MALLOC_MMAP_THRESHOLD_ to fix the threshold. > I already read that CPU support "large pages" between 2 MB and 1 GB, > instead of just 4 kB. Using large pages can have a significant impact > on performance. I don't know if we can do something to help the Linux > kernel to use large pages for our memory? I don't know neither how we > could do that :-) Maybe using mmap() closer to large pages will help > Linux to join them to build a big page? (Linux has something magic to > make applications use big pages transparently.) There's MAP_HUGETLB and friends for mmap flags, but it's generally better to just let the kernel do this for you transparently (using Transparent Huge Pages) by making sure that your arena allocations are either contiguous or big enough. Siddhesh From victor.stinner at gmail.com Thu Jun 1 17:24:52 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 1 Jun 2017 23:24:52 +0200 Subject: [Python-Dev] "Global freepool" In-Reply-To: References: <20170601104021.31aee844@fsol> Message-ID: 2017-06-01 22:16 GMT+02:00 Serhiy Storchaka : > The issue [1] still is open. Patches neither applied nor rejected. They > exposes the speed up in microbenchmarks, but it is not large. Up to 40% for > iterating over enumerate() and 5-7% for hard integer computations like > base85 encoding or spectral_norm benchmark. > > [1] https://bugs.python.org/issue25324 Hum, I think that the right issue is: http://bugs.python.org/issue24165 Victor From brett at python.org Thu Jun 1 17:08:59 2017 From: brett at python.org (Brett Cannon) Date: Thu, 01 Jun 2017 21:08:59 +0000 Subject: [Python-Dev] PEP 7 and braces { .... } on if In-Reply-To: References: Message-ID: If you create an issue at github.com/python/peps and assign it to me I will get to it someday. :) On Thu, 1 Jun 2017 at 00:19 Victor Stinner wrote: > 2017-05-31 19:27 GMT+02:00 Guido van Rossum : > > I interpret the PEP (...) > > Right, the phrasing requires to "interpret" it :-) > > > (...) as saying that you should use braces everywhere but not > > to add them in code that you're not modifying otherwise. (I.e. don't go > on a > > brace-adding rampage.) If author and reviewer of a PR disagree I would go > > with "add braces" since that's clearly the PEP's preference. This is C > code. > > We should play it safe. > > Would someone be nice enough to try to rephrase the PEP 7 to explain > that? Just to avoid further boring discussion on the C coding style... > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Thu Jun 1 18:21:35 2017 From: barry at python.org (Barry Warsaw) Date: Thu, 1 Jun 2017 18:21:35 -0400 Subject: [Python-Dev] PEP 7 and braces { .... } on if References: Message-ID: <20170601182135.373d6d43@subdivisions.wooz.org> https://github.com/python/peps/pull/280/files On Jun 01, 2017, at 09:08 PM, Brett Cannon wrote: >If you create an issue at github.com/python/peps and assign it to me I will >get to it someday. :) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 801 bytes Desc: OpenPGP digital signature URL: From jake at lwn.net Thu Jun 1 21:27:23 2017 From: jake at lwn.net (Jake Edge) Date: Thu, 1 Jun 2017 19:27:23 -0600 Subject: [Python-Dev] 2017 Python Language Summit coverage -- Round 2 Message-ID: <20170601192723.1030f02e@chukar.edge2.net> Hola python-dev, Thanks for all the positive feedback on the coverage (and the corrections/clarifications in the comments too)! There is, it seems, always more to do, but I do have three additional articles from the summit up now and should complete the coverage over the next week. The starting point is the overview article, here: https://lwn.net/Articles/723251/ which should now be free for anyone to see (and the first four articles too). LWN subscribers can see the content right away, but one week after they are published in the weekly edition, they become freely available for everyone. Until then, though, feel free to share the SubscriberLinks I am posting here. I have been asked about our policy on appropriate places to share SubscriberLinks; blogs, tweets, social media, mailing lists, etc. are all perfectly fine with us. The new articles are: Keeping Python competitive: https://lwn.net/Articles/723949/ or https://lwn.net/SubscriberLink/723949/56a392defaae995c/ Trio and the future of asynchronous execution in Python: https://lwn.net/Articles/724082/ or https://lwn.net/SubscriberLink/724082/43c399adca8006f0/ Python ssl module update: https://lwn.net/Articles/724209/ or https://lwn.net/SubscriberLink/724209/8460ca8b51c00634/ stay tuned sometime next week for the thrilling conclusion :) jake -- Jake Edge - LWN - jake at lwn.net - http://lwn.net From larry at hastings.org Thu Jun 1 23:21:12 2017 From: larry at hastings.org (Larry Hastings) Date: Thu, 1 Jun 2017 20:21:12 -0700 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: <20170601115014.18dcb755@fsol> References: <20170601115014.18dcb755@fsol> Message-ID: <32428072-366e-dd64-5b71-254fd58deade@hastings.org> On 06/01/2017 02:50 AM, Antoine Pitrou wrote: > Another possible strategy is: allocate several arenas at once (using a > larger mmap() call), and use MADV_DONTNEED to relinquish individual > arenas. Thus adding a *fourth* layer of abstraction over memory we get from the OS? block -> pool -> arena -> "multi-arena" -> OS Y'know, this might actually make things faster. These multi-arenas could be the dynamically growing thing Victor wants to try. We allocate 16mb, then carve it up into arenas (however big those are), then next time allocate 32mb or what have you. Since the arenas remain a fixed size, we don't make the frequently-used code path (address_in_range) any slower. The code to deal with the multi-arenas would add a little complexity--to an admittedly already complex allocator implementation, but then what allocator isn't complex internally?--but it'd be an infrequent code path and I bet it'd be an improvement over simply calling malloc / mmap / VirtualAlloc. What do you think, Victor? And to think I started this reply ironically, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Jun 2 01:55:29 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 2 Jun 2017 15:55:29 +1000 Subject: [Python-Dev] Aligning the packaging.python.org theme with the rest of the docs In-Reply-To: <20170530140827.1ca65f00@fsol> References: <20170530140827.1ca65f00@fsol> Message-ID: On 30 May 2017 at 22:08, Antoine Pitrou wrote: > On Tue, 30 May 2017 21:49:19 +1000 > Nick Coghlan wrote: >> >> Here's an alternate wording for the README that would focus on those >> considerations without explicitly asking folks not to use the theme: >> >> "Note that when adopting this theme, you're also borrowing an element >> of the trust and credibility established by the CPython core >> developers over the years, as well as the legal credibility arising >> from their close association with the Python Software Foundation. > > The statement about "legal credibility" sounds wishy-washy and could > lure users into thinking that they're doing something illegal by > borrowing the theme. > > Also I'm not sure what is that "legal credibility" you're talking > about. If it's about the PSF license and the Python CLA then > better to voice that explicitly, IMO. It's probably better to just drop that clause and call the repository "cpython-docs-theme" rather than "psf-docs-theme". Explicitly affiliating the theme with the PSF made sense if we were reserving the right to seek trade dress protections in the future, but it sounds like folks are pretty solidly against that idea, so we can instead leave the PSF out of it entirely. >> That's fine, and you're welcome to do so for other Python community >> projects if you so choose, but please keep in mind that in doing so >> you're also choosing to become a co-steward of that collective trust >> :)" > > "Becoming a co-steward of that collective trust" sounds serious enough > (even though I don't understand what it means concretely), so why > the smiley? Mainly to convey that the situation isn't necessarily as profound as that wording might suggest. Rephrasing that part, and incorporating the amendment from above: "Note that when adopting this theme, you're also borrowing an element of the trust and credibility established by the CPython core developers over the years. That's fine, and you're welcome to do so for other Python community projects if you so choose, but please keep in mind that in doing so you're also choosing to accept some of the responsibility for maintaining that collective trust." Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From cory at lukasa.co.uk Fri Jun 2 05:08:43 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Fri, 2 Jun 2017 10:08:43 +0100 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> Message-ID: <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> > On 1 Jun 2017, at 20:59, Steve Dower wrote: > > On 01Jun2017 1010, Nathaniel Smith wrote: >> I believe that for answering this question about the ssl module, it's really only Linux users that matter, since pip/requests/everyone else pushing for this only want to use ssl.MemoryBIO on Linux. Their plan on Windows/MacOS (IIUC) is to stop using the ssl module entirely in favor of new ctypes bindings for their respective native TLS libraries. >> (And yes, in principle it might be possible to write new ctypes-based bindings for openssl, but (a) this whole project is already teetering on the verge of being impossible given the resources available, so adding any major extra deliverable is likely to sink the whole thing, and (b) compared to the proprietary libraries, openssl is *much* harder and riskier to wrap at the ctypes level because it has different/incompatible ABIs depending on its micro version and the vendor who distributed it. This is why manylinux packages that need openssl have to ship their own, but pip can't and shouldn't ship its own openssl for many hopefully obvious reasons.) > > How much of a stop-gap would it be (for Windows at least) to override OpenSSL's certificate validation with a call into the OS? This leaves most of the work with OpenSSL, but lets the OS say yes/no to the certificates based on its own configuration. > > For Windows, this is under 100 lines of C code in (probably) _ssl, and while I think an SChannel based approach is the better way to go long-term,[1] offering platform-specific certificate validation as the default in 2.7 is far more palatable than backporting new public API. It?s entirely do-able. This is where I reveal just how long I?ve been fretting over this problem: https://pypi.python.org/pypi/certitude . Ignore the description, it?s wildly out-of-date: let me summarise the library instead. Certitude is a Python library that uses CFFI and Rust to call into the system certificate validation libraries on macOS and Windows using a single unified API. Under the covers it has a whole bunch of Rust code that translates from what OpenSSL can give you (a list of certificates in the peer cert chain in DER format) and into what those two operating systems expect. The Rust code for Windows is here[1] and is about as horrifying a chunk of Rust as you can imagine seeing (the Windows API does not translate very neatly into Rust so the word ?unsafe? appears a lot), but it does appear to work, at least in the mainline cases and in the few tests I have. The macOS code is here[2] and is moderately less horrifying, containing no instances of the word ?unsafe?. I lifted this approach from Chrome, because at the moment this is what they do: they use their custom fork of OpenSSL (BoringSSL) to do the actual TLS protocol manipulation, but hand the cert chain verification off to platform-native libraries on Windows and macOS. I have never productised this library because ultimately I never had the time to spend writing a sufficiently broad test-case to confirm to me that it worked in all cases. There are very real risks in calling these APIs directly because if you get it wrong it?s easy to fail open. It should be noted that right now certitude only works with, surprise, PyOpenSSL. Partly this is because the standard library does not expose SSL_get_peer_cert_chain, but even if it did that wouldn?t be enough as OpenSSL with VERIFY_NONE does not actually *save* the peer cert chain anywhere. That means even with PyOpenSSL the only way to get the peer cert chain is to hook into the verify callback and save off the certs as they come in, a gloriously absurd solution that is impossible with pure-Python code from the ssl module. While this approach could work with _ssl.c, it ultimately doesn?t resolve the issue. It involves writing a substantial amount of new code which needs to be maintained by the ssl module maintainers. All of this code needs to be tested *thoroughly*, because python-dev would be accepting responsibility for the incredibly damaging potential CVEs in that code. And it doesn?t get python-dev out of the business of shipping OpenSSL on macOS and Windows, meaning that python-dev continues to bear the burden of OpenSSL CVEs, as well as the brand new CVEs that it is at risk of introducing. Oh, and it can?t be backported to Python 2.7 or any of the bugfix-only Python 3 releases, and as I just noted the ssl module has never made it possible to use this approach from outside CPython. So it?s strictly just as bad as the situation PEP 543 is in, but with more C code. Doesn?t sound like a winning description to me. ;) Cory [1]: https://github.com/Lukasa/rust-certitude/blob/master/rust-certitude/src/windows.rs [2]: https://github.com/Lukasa/rust-certitude/blob/master/rust-certitude/src/osx.rs -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Fri Jun 2 05:38:16 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 2 Jun 2017 11:38:16 +0200 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE References: <20170601115014.18dcb755@fsol> <32428072-366e-dd64-5b71-254fd58deade@hastings.org> Message-ID: <20170602113816.0176b09b@fsol> On Thu, 1 Jun 2017 20:21:12 -0700 Larry Hastings wrote: > On 06/01/2017 02:50 AM, Antoine Pitrou wrote: > > Another possible strategy is: allocate several arenas at once (using a > > larger mmap() call), and use MADV_DONTNEED to relinquish individual > > arenas. > > Thus adding a *fourth* layer of abstraction over memory we get from the OS? > > block -> pool -> arena -> "multi-arena" -> OS Not a layer of abstraction, just over-allocation. You would over-allocate arenas like you over-allocate a list object's elements storage (though probably using a different over-allocation strategy ;-)). That would reduce the number of mmap() calls (though not necessarily the number of munmap() or madvise() calls), and also provide more opportunities for the kernel to allocate a large page. But you would still handle individual arenas in the allocation code. > Y'know, this might actually make things faster. These multi-arenas > could be the dynamically growing thing Victor wants to try. We allocate > 16mb, then carve it up into arenas (however big those are), then next > time allocate 32mb or what have you. I hope those are not the actual numbers you're intending to use ;-) I still think that allocating more than 1 or 2MB at once would be foolish. Remember this is data that's going to be carved up into (tens of) thousands of small objects. Large objects eschew the small object allocator (not to mention that third-party libraries like Numpy may be using different allocation routines when they allocate very large data). Regards Antoine. From victor.stinner at gmail.com Fri Jun 2 05:42:58 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 2 Jun 2017 11:42:58 +0200 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> Message-ID: Thanks Cory for the long explanation. Let me try to summarize (tell me if I'm wrong). We have 3 options: * Do nothing: reject the PEP 546 and let each project handles security on its own (current status co) * Write *new* C code, maybe using certitude as a starting point, to offload certifcate validation on Windows and macOS * Backport existing code from master to 2.7: MemoryBIO and SSLObject Writing new code seems more risky and error-prone than backporting already "battle-tested" MemoryBIO from master. I also expect that writing code to validate certificate will be longer than the "100 lines of C code in (probably)" expected by Steve Dower. rust-certitude counts around 700 lines of Rust and 80 lines of Python code. But maybe I misunderstood the purpose of certitude: Steve Dower asked to only validate a certificate, not load or export CA. I counted 150 Python lines for SSLObject and 230 C lines for MemoryBIO. Since the long term plan is to not use stdlib ssl but a new implementation on Windows and macOS, it seems worthless to backport MemoryBIO on Python 2.7. The PEP 546 (backport MemoryBIO) is a practical solution to provide a *smooth* transition from ssl to a new TLS API. The experience showed that hard changes like "run 2to3 and drop your Python 2 code" doesn't work in practice. Users want a transition plan with small steps. Victor 2017-06-02 11:08 GMT+02:00 Cory Benfield : > > On 1 Jun 2017, at 20:59, Steve Dower wrote: > > On 01Jun2017 1010, Nathaniel Smith wrote: > > I believe that for answering this question about the ssl module, it's really > only Linux users that matter, since pip/requests/everyone else pushing for > this only want to use ssl.MemoryBIO on Linux. Their plan on Windows/MacOS > (IIUC) is to stop using the ssl module entirely in favor of new ctypes > bindings for their respective native TLS libraries. > (And yes, in principle it might be possible to write new ctypes-based > bindings for openssl, but (a) this whole project is already teetering on the > verge of being impossible given the resources available, so adding any major > extra deliverable is likely to sink the whole thing, and (b) compared to the > proprietary libraries, openssl is *much* harder and riskier to wrap at the > ctypes level because it has different/incompatible ABIs depending on its > micro version and the vendor who distributed it. This is why manylinux > packages that need openssl have to ship their own, but pip can't and > shouldn't ship its own openssl for many hopefully obvious reasons.) > > > How much of a stop-gap would it be (for Windows at least) to override > OpenSSL's certificate validation with a call into the OS? This leaves most > of the work with OpenSSL, but lets the OS say yes/no to the certificates > based on its own configuration. > > For Windows, this is under 100 lines of C code in (probably) _ssl, and while > I think an SChannel based approach is the better way to go long-term,[1] > offering platform-specific certificate validation as the default in 2.7 is > far more palatable than backporting new public API. > > > It?s entirely do-able. This is where I reveal just how long I?ve been > fretting over this problem: https://pypi.python.org/pypi/certitude. Ignore > the description, it?s wildly out-of-date: let me summarise the library > instead. > > Certitude is a Python library that uses CFFI and Rust to call into the > system certificate validation libraries on macOS and Windows using a single > unified API. Under the covers it has a whole bunch of Rust code that > translates from what OpenSSL can give you (a list of certificates in the > peer cert chain in DER format) and into what those two operating systems > expect. The Rust code for Windows is here[1] and is about as horrifying a > chunk of Rust as you can imagine seeing (the Windows API does not translate > very neatly into Rust so the word ?unsafe? appears a lot), but it does > appear to work, at least in the mainline cases and in the few tests I have. > The macOS code is here[2] and is moderately less horrifying, containing no > instances of the word ?unsafe?. > > I lifted this approach from Chrome, because at the moment this is what they > do: they use their custom fork of OpenSSL (BoringSSL) to do the actual TLS > protocol manipulation, but hand the cert chain verification off to > platform-native libraries on Windows and macOS. > > I have never productised this library because ultimately I never had the > time to spend writing a sufficiently broad test-case to confirm to me that > it worked in all cases. There are very real risks in calling these APIs > directly because if you get it wrong it?s easy to fail open. > > It should be noted that right now certitude only works with, surprise, > PyOpenSSL. Partly this is because the standard library does not expose > SSL_get_peer_cert_chain, but even if it did that wouldn?t be enough as > OpenSSL with VERIFY_NONE does not actually *save* the peer cert chain > anywhere. That means even with PyOpenSSL the only way to get the peer cert > chain is to hook into the verify callback and save off the certs as they > come in, a gloriously absurd solution that is impossible with pure-Python > code from the ssl module. > > While this approach could work with _ssl.c, it ultimately doesn?t resolve > the issue. It involves writing a substantial amount of new code which needs > to be maintained by the ssl module maintainers. All of this code needs to be > tested *thoroughly*, because python-dev would be accepting responsibility > for the incredibly damaging potential CVEs in that code. And it doesn?t get > python-dev out of the business of shipping OpenSSL on macOS and Windows, > meaning that python-dev continues to bear the burden of OpenSSL CVEs, as > well as the brand new CVEs that it is at risk of introducing. > > Oh, and it can?t be backported to Python 2.7 or any of the bugfix-only > Python 3 releases, and as I just noted the ssl module has never made it > possible to use this approach from outside CPython. So it?s strictly just as > bad as the situation PEP 543 is in, but with more C code. Doesn?t sound like > a winning description to me. ;) > > Cory > > [1]: > https://github.com/Lukasa/rust-certitude/blob/master/rust-certitude/src/windows.rs > [2]: > https://github.com/Lukasa/rust-certitude/blob/master/rust-certitude/src/osx.rs > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > From victor.stinner at gmail.com Fri Jun 2 05:46:43 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 2 Jun 2017 11:46:43 +0200 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: References: Message-ID: I would be curious of another test: use pymalloc for objects larger than 512 bytes. For example, allocate up to 4 KB? In the past, we already changed the maximum size from 256 to 512 to support most common Python objects on 64-bit platforms. Since Python objects contain many pointers: switching from 32 bit to 64 bit can double the size of the object in the worst case. Victor 2017-06-01 9:38 GMT+02:00 Larry Hastings : > > > When CPython's small block allocator was first merged in late February 2001, > it allocated memory in gigantic chunks it called "arenas". These arenas > were a massive 256 KILOBYTES apiece. > > This tunable parameter has not been touched in the intervening 16 years. > Yet CPython's memory consumption continues to grow. By the time a current > "trunk" build of CPython reaches the REPL prompt it's already allocated 16 > arenas. > > I propose we make the arena size larger. By how much? I asked Victor to > run some benchmarks with arenas of 1mb, 2mb, and 4mb. The results with 1mb > and 2mb were mixed, but his benchmarks with a 4mb arena size showed > measurable (>5%) speedups on ten benchmarks and no slowdowns. > > What would be the result of making the arena size 4mb? > > CPython could no longer run on a computer where at startup it could not > allocate at least one 4mb continguous block of memory. > CPython programs would die slightly sooner in out-of-memory conditions. > CPython programs would use more memory. How much? Hard to say. It depends > on their allocation strategy. I suspect most of the time it would be < 3mb > additional memory. But for pathological allocation strategies the > difference could be significant. (e.g: lots of allocs, followed by lots of > frees, but the occasional object lives forever, which means that the arena > it's in can never be freed. If 1 out of ever 16 256k arenas is kept alive > this way, and the objects are spaced out precisely such that now it's 1 for > every 4mb arena, max memory use would be the same but later stable memory > use would hypothetically be 16x current) > Many programs would be slightly faster now and then, simply because we call > malloc() 1/16 as often. > > > What say you? Vote for your favorite color of bikeshed now! > > > /arry > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > From cory at lukasa.co.uk Fri Jun 2 06:39:32 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Fri, 2 Jun 2017 11:39:32 +0100 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> Message-ID: > On 2 Jun 2017, at 10:42, Victor Stinner wrote: > > Writing new code seems more risky and error-prone than backporting > already "battle-tested" MemoryBIO from master. I also expect that > writing code to validate certificate will be longer than the "100 > lines of C code in (probably)" expected by Steve Dower. > > rust-certitude counts around 700 lines of Rust and 80 lines of Python > code. But maybe I misunderstood the purpose of certitude: Steve Dower > asked to only validate a certificate, not load or export CA. That?s all certitude does. The docs of certitude are from an older version of the project when I was considering just doing a live-export to PEM file, before I realised the security concerns of that approach. We?d also require some other code that lives outside certitude. In particular, code needs to be written to handle the OpenSSL verify callback to save off the cert chain and to translate errors appropriately. There?s also a follow-on problem: the ssl module allows you to call SSLContext.load_default_certs and then SSLContext.load_verify_locations. If you do that, those two behave *additively*: both the default certs and custom verify locations are trusted. Certitude doesn?t allow you to do that: it says that if you choose to use the system certs then darn it that?s all you get. Working out how to do that without just importing random stuff into the user?s keychain would be?tricky. Do-able, for sure, but would require code I haven?t written for Certitude (I may have written it using ctypes elsewhere though). Cory From songofacandy at gmail.com Fri Jun 2 07:56:31 2017 From: songofacandy at gmail.com (INADA Naoki) Date: Fri, 2 Jun 2017 20:56:31 +0900 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: References: Message-ID: > I would be curious of another test: use pymalloc for objects larger > than 512 bytes. For example, allocate up to 4 KB? Since current pool size is 4KB and there is pool_header in pool, we can't allocate 4KB block from pool. And if support 1KB block, only 3KB of 4KB can be actually used. I think 512 bytes / 4KB (1/8) is good ratio. Do you mean increase pool size? How about adding configure option like server-mode? SMALL_REQUEST_THRESHOLD 1024 // 2x POOL_SIZE (16*1024) // 4x ARENA_SIZE (2*1024*1024) // 8x, and same to huge page size. > > In the past, we already changed the maximum size from 256 to 512 to > support most common Python objects on 64-bit platforms. Since Python > objects contain many pointers: switching from 32 bit to 64 bit can > double the size of the object in the worst case. > Make sense. Naoki From solipsis at pitrou.net Fri Jun 2 08:06:22 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 2 Jun 2017 14:06:22 +0200 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> Message-ID: <20170602140622.64cbfca4@fsol> Thanks you for all the explanations. So to summarize my opinion, I'm still -0.5 on this PEP. I would also like to see the issues Jython, Ubuntu et al. have mentioned solved before this is accepted. Regards Antoine. On Fri, 2 Jun 2017 11:42:58 +0200 Victor Stinner wrote: > Thanks Cory for the long explanation. Let me try to summarize (tell me > if I'm wrong). > > We have 3 options: > > * Do nothing: reject the PEP 546 and let each project handles security > on its own (current status co) > * Write *new* C code, maybe using certitude as a starting point, to > offload certifcate validation on Windows and macOS > * Backport existing code from master to 2.7: MemoryBIO and SSLObject > > Writing new code seems more risky and error-prone than backporting > already "battle-tested" MemoryBIO from master. I also expect that > writing code to validate certificate will be longer than the "100 > lines of C code in (probably)" expected by Steve Dower. > > rust-certitude counts around 700 lines of Rust and 80 lines of Python > code. But maybe I misunderstood the purpose of certitude: Steve Dower > asked to only validate a certificate, not load or export CA. > > I counted 150 Python lines for SSLObject and 230 C lines for MemoryBIO. > > Since the long term plan is to not use stdlib ssl but a new > implementation on Windows and macOS, it seems worthless to backport > MemoryBIO on Python 2.7. The PEP 546 (backport MemoryBIO) is a > practical solution to provide a *smooth* transition from ssl to a new > TLS API. The experience showed that hard changes like "run 2to3 and > drop your Python 2 code" doesn't work in practice. Users want a > transition plan with small steps. > > Victor > > 2017-06-02 11:08 GMT+02:00 Cory Benfield : > > > > On 1 Jun 2017, at 20:59, Steve Dower wrote: > > > > On 01Jun2017 1010, Nathaniel Smith wrote: > > > > I believe that for answering this question about the ssl module, it's really > > only Linux users that matter, since pip/requests/everyone else pushing for > > this only want to use ssl.MemoryBIO on Linux. Their plan on Windows/MacOS > > (IIUC) is to stop using the ssl module entirely in favor of new ctypes > > bindings for their respective native TLS libraries. > > (And yes, in principle it might be possible to write new ctypes-based > > bindings for openssl, but (a) this whole project is already teetering on the > > verge of being impossible given the resources available, so adding any major > > extra deliverable is likely to sink the whole thing, and (b) compared to the > > proprietary libraries, openssl is *much* harder and riskier to wrap at the > > ctypes level because it has different/incompatible ABIs depending on its > > micro version and the vendor who distributed it. This is why manylinux > > packages that need openssl have to ship their own, but pip can't and > > shouldn't ship its own openssl for many hopefully obvious reasons.) > > > > > > How much of a stop-gap would it be (for Windows at least) to override > > OpenSSL's certificate validation with a call into the OS? This leaves most > > of the work with OpenSSL, but lets the OS say yes/no to the certificates > > based on its own configuration. > > > > For Windows, this is under 100 lines of C code in (probably) _ssl, and while > > I think an SChannel based approach is the better way to go long-term,[1] > > offering platform-specific certificate validation as the default in 2.7 is > > far more palatable than backporting new public API. > > > > > > It?s entirely do-able. This is where I reveal just how long I?ve been > > fretting over this problem: https://pypi.python.org/pypi/certitude. Ignore > > the description, it?s wildly out-of-date: let me summarise the library > > instead. > > > > Certitude is a Python library that uses CFFI and Rust to call into the > > system certificate validation libraries on macOS and Windows using a single > > unified API. Under the covers it has a whole bunch of Rust code that > > translates from what OpenSSL can give you (a list of certificates in the > > peer cert chain in DER format) and into what those two operating systems > > expect. The Rust code for Windows is here[1] and is about as horrifying a > > chunk of Rust as you can imagine seeing (the Windows API does not translate > > very neatly into Rust so the word ?unsafe? appears a lot), but it does > > appear to work, at least in the mainline cases and in the few tests I have. > > The macOS code is here[2] and is moderately less horrifying, containing no > > instances of the word ?unsafe?. > > > > I lifted this approach from Chrome, because at the moment this is what they > > do: they use their custom fork of OpenSSL (BoringSSL) to do the actual TLS > > protocol manipulation, but hand the cert chain verification off to > > platform-native libraries on Windows and macOS. > > > > I have never productised this library because ultimately I never had the > > time to spend writing a sufficiently broad test-case to confirm to me that > > it worked in all cases. There are very real risks in calling these APIs > > directly because if you get it wrong it?s easy to fail open. > > > > It should be noted that right now certitude only works with, surprise, > > PyOpenSSL. Partly this is because the standard library does not expose > > SSL_get_peer_cert_chain, but even if it did that wouldn?t be enough as > > OpenSSL with VERIFY_NONE does not actually *save* the peer cert chain > > anywhere. That means even with PyOpenSSL the only way to get the peer cert > > chain is to hook into the verify callback and save off the certs as they > > come in, a gloriously absurd solution that is impossible with pure-Python > > code from the ssl module. > > > > While this approach could work with _ssl.c, it ultimately doesn?t resolve > > the issue. It involves writing a substantial amount of new code which needs > > to be maintained by the ssl module maintainers. All of this code needs to be > > tested *thoroughly*, because python-dev would be accepting responsibility > > for the incredibly damaging potential CVEs in that code. And it doesn?t get > > python-dev out of the business of shipping OpenSSL on macOS and Windows, > > meaning that python-dev continues to bear the burden of OpenSSL CVEs, as > > well as the brand new CVEs that it is at risk of introducing. > > > > Oh, and it can?t be backported to Python 2.7 or any of the bugfix-only > > Python 3 releases, and as I just noted the ssl module has never made it > > possible to use this approach from outside CPython. So it?s strictly just as > > bad as the situation PEP 543 is in, but with more C code. Doesn?t sound like > > a winning description to me. ;) > > > > Cory > > > > [1]: > > https://github.com/Lukasa/rust-certitude/blob/master/rust-certitude/src/windows.rs > > [2]: > > https://github.com/Lukasa/rust-certitude/blob/master/rust-certitude/src/osx.rs > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org From ben at bendarnell.com Fri Jun 2 09:43:21 2017 From: ben at bendarnell.com (Ben Darnell) Date: Fri, 02 Jun 2017 13:43:21 +0000 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <20170602140622.64cbfca4@fsol> References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> <20170602140622.64cbfca4@fsol> Message-ID: The PEP's rationale is now "This PEP will help facilitate the future adoption of :pep:`543` across all supported Python versions, which will improve security for both Python 2 and Python 3 users." What exactly are these security improvements? My understanding (which may well be incorrect) is that the security improvements come in the future when the PEP 543 APIs are implemented on top of the various platform-native security libraries. These changes will not be present until some future 3.x release, and will not be backported to 2.7 (without another PEP, which I expect would be even more contentious than this one). What then is the security improvement for Python 2 users? In Tornado, I have not felt any urgency to replace wrap_socket with MemoryBIO. Is there a security-related reason I should do so sooner rather than later? (I'd also urge Cory and any other wrap_socket skeptics on the requests team to reconsider - Tornado's SSLIOStream works well. The asynchronous use of wrap_socket isn't so subtle and risky with buffering) -Ben On Fri, Jun 2, 2017 at 8:07 AM Antoine Pitrou wrote: > > Thanks you for all the explanations. So to summarize my opinion, I'm > still -0.5 on this PEP. I would also like to see the issues Jython, > Ubuntu et al. have mentioned solved before this is accepted. > > Regards > > Antoine. > > > > On Fri, 2 Jun 2017 11:42:58 +0200 > Victor Stinner wrote: > > Thanks Cory for the long explanation. Let me try to summarize (tell me > > if I'm wrong). > > > > We have 3 options: > > > > * Do nothing: reject the PEP 546 and let each project handles security > > on its own (current status co) > > * Write *new* C code, maybe using certitude as a starting point, to > > offload certifcate validation on Windows and macOS > > * Backport existing code from master to 2.7: MemoryBIO and SSLObject > > > > Writing new code seems more risky and error-prone than backporting > > already "battle-tested" MemoryBIO from master. I also expect that > > writing code to validate certificate will be longer than the "100 > > lines of C code in (probably)" expected by Steve Dower. > > > > rust-certitude counts around 700 lines of Rust and 80 lines of Python > > code. But maybe I misunderstood the purpose of certitude: Steve Dower > > asked to only validate a certificate, not load or export CA. > > > > I counted 150 Python lines for SSLObject and 230 C lines for MemoryBIO. > > > > Since the long term plan is to not use stdlib ssl but a new > > implementation on Windows and macOS, it seems worthless to backport > > MemoryBIO on Python 2.7. The PEP 546 (backport MemoryBIO) is a > > practical solution to provide a *smooth* transition from ssl to a new > > TLS API. The experience showed that hard changes like "run 2to3 and > > drop your Python 2 code" doesn't work in practice. Users want a > > transition plan with small steps. > > > > Victor > > > > 2017-06-02 11:08 GMT+02:00 Cory Benfield : > > > > > > On 1 Jun 2017, at 20:59, Steve Dower wrote: > > > > > > On 01Jun2017 1010, Nathaniel Smith wrote: > > > > > > I believe that for answering this question about the ssl module, it's > really > > > only Linux users that matter, since pip/requests/everyone else pushing > for > > > this only want to use ssl.MemoryBIO on Linux. Their plan on > Windows/MacOS > > > (IIUC) is to stop using the ssl module entirely in favor of new ctypes > > > bindings for their respective native TLS libraries. > > > (And yes, in principle it might be possible to write new ctypes-based > > > bindings for openssl, but (a) this whole project is already teetering > on the > > > verge of being impossible given the resources available, so adding any > major > > > extra deliverable is likely to sink the whole thing, and (b) compared > to the > > > proprietary libraries, openssl is *much* harder and riskier to wrap at > the > > > ctypes level because it has different/incompatible ABIs depending on > its > > > micro version and the vendor who distributed it. This is why manylinux > > > packages that need openssl have to ship their own, but pip can't and > > > shouldn't ship its own openssl for many hopefully obvious reasons.) > > > > > > > > > How much of a stop-gap would it be (for Windows at least) to override > > > OpenSSL's certificate validation with a call into the OS? This leaves > most > > > of the work with OpenSSL, but lets the OS say yes/no to the > certificates > > > based on its own configuration. > > > > > > For Windows, this is under 100 lines of C code in (probably) _ssl, and > while > > > I think an SChannel based approach is the better way to go > long-term,[1] > > > offering platform-specific certificate validation as the default in > 2.7 is > > > far more palatable than backporting new public API. > > > > > > > > > It?s entirely do-able. This is where I reveal just how long I?ve been > > > fretting over this problem: https://pypi.python.org/pypi/certitude. > Ignore > > > the description, it?s wildly out-of-date: let me summarise the library > > > instead. > > > > > > Certitude is a Python library that uses CFFI and Rust to call into the > > > system certificate validation libraries on macOS and Windows using a > single > > > unified API. Under the covers it has a whole bunch of Rust code that > > > translates from what OpenSSL can give you (a list of certificates in > the > > > peer cert chain in DER format) and into what those two operating > systems > > > expect. The Rust code for Windows is here[1] and is about as > horrifying a > > > chunk of Rust as you can imagine seeing (the Windows API does not > translate > > > very neatly into Rust so the word ?unsafe? appears a lot), but it does > > > appear to work, at least in the mainline cases and in the few tests I > have. > > > The macOS code is here[2] and is moderately less horrifying, > containing no > > > instances of the word ?unsafe?. > > > > > > I lifted this approach from Chrome, because at the moment this is what > they > > > do: they use their custom fork of OpenSSL (BoringSSL) to do the actual > TLS > > > protocol manipulation, but hand the cert chain verification off to > > > platform-native libraries on Windows and macOS. > > > > > > I have never productised this library because ultimately I never had > the > > > time to spend writing a sufficiently broad test-case to confirm to me > that > > > it worked in all cases. There are very real risks in calling these APIs > > > directly because if you get it wrong it?s easy to fail open. > > > > > > It should be noted that right now certitude only works with, surprise, > > > PyOpenSSL. Partly this is because the standard library does not expose > > > SSL_get_peer_cert_chain, but even if it did that wouldn?t be enough as > > > OpenSSL with VERIFY_NONE does not actually *save* the peer cert chain > > > anywhere. That means even with PyOpenSSL the only way to get the peer > cert > > > chain is to hook into the verify callback and save off the certs as > they > > > come in, a gloriously absurd solution that is impossible with > pure-Python > > > code from the ssl module. > > > > > > While this approach could work with _ssl.c, it ultimately doesn?t > resolve > > > the issue. It involves writing a substantial amount of new code which > needs > > > to be maintained by the ssl module maintainers. All of this code needs > to be > > > tested *thoroughly*, because python-dev would be accepting > responsibility > > > for the incredibly damaging potential CVEs in that code. And it > doesn?t get > > > python-dev out of the business of shipping OpenSSL on macOS and > Windows, > > > meaning that python-dev continues to bear the burden of OpenSSL CVEs, > as > > > well as the brand new CVEs that it is at risk of introducing. > > > > > > Oh, and it can?t be backported to Python 2.7 or any of the bugfix-only > > > Python 3 releases, and as I just noted the ssl module has never made it > > > possible to use this approach from outside CPython. So it?s strictly > just as > > > bad as the situation PEP 543 is in, but with more C code. Doesn?t > sound like > > > a winning description to me. ;) > > > > > > Cory > > > > > > [1]: > > > > https://github.com/Lukasa/rust-certitude/blob/master/rust-certitude/src/windows.rs > > > [2]: > > > > https://github.com/Lukasa/rust-certitude/blob/master/rust-certitude/src/osx.rs > > > > > > _______________________________________________ > > > Python-Dev mailing list > > > Python-Dev at python.org > > > https://mail.python.org/mailman/listinfo/python-dev > > > Unsubscribe: > > > > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ben%40bendarnell.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From naomi at seyfer.org Fri Jun 2 11:34:43 2017 From: naomi at seyfer.org (Naomi Seyfer) Date: Fri, 2 Jun 2017 08:34:43 -0700 Subject: [Python-Dev] PEP 484 update proposal: annotating decorated declarations In-Reply-To: References: Message-ID: Yep, interested in implementing it! I will put implementation time on my schedule and tell y'all when it is, for holding myself accountable -- it turns out I never do anything not on my schedule. On Wed, May 31, 2017 at 3:17 PM, Guido van Rossum wrote: > On Wed, May 31, 2017 at 6:16 AM, Ivan Levkivskyi > wrote: > >> On 30 May 2017 at 23:02, Guido van Rossum wrote: >> >>> All in all I'm still leaning towards Naomi's original proposal -- it >>> looks simpler to implement as well. >>> >> >> OK, I think having a bit of verbosity is absolutely fine if we win >> simplicity of implementation (for both static and runtime purposes). >> > > Then I propose to do it this way. We can always add Jukka's way as an > alternative notation later. I'd like to hear from Jukka before I merge the > PR for PEP-484. > > In the meantime, Naomi, are you interested in trying to implement this? > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From status at bugs.python.org Fri Jun 2 12:09:11 2017 From: status at bugs.python.org (Python tracker) Date: Fri, 2 Jun 2017 18:09:11 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20170602160911.77C9956B79@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2017-05-26 - 2017-06-02) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 6025 (+26) closed 36284 (+43) total 42309 (+69) Open issues with patches: 2370 Issues opened (54) ================== #30486: Allow setting cell value http://bugs.python.org/issue30486 opened by pitrou #30487: DOC: automatically create a venv and install Sphinx when runni http://bugs.python.org/issue30487 opened by cjrh #30489: Add CmdLineTest to standard library http://bugs.python.org/issue30489 opened by Santiago Castro #30490: Allow pass an exception to the Event.set method http://bugs.python.org/issue30490 opened by pfreixes #30491: Add a lightweight mechanism for detecting un-awaited coroutine http://bugs.python.org/issue30491 opened by njs #30492: 'make clinic' does not work for out of tree builds / clinic.py http://bugs.python.org/issue30492 opened by gregory.p.smith #30493: Increase coverage of base64 http://bugs.python.org/issue30493 opened by leecannon #30495: IDLE: modernize textview module http://bugs.python.org/issue30495 opened by terry.reedy #30498: Run Python's slowest tests in the first 3/4 of tests when usin http://bugs.python.org/issue30498 opened by brett.cannon #30500: urllib connects to a wrong host http://bugs.python.org/issue30500 opened by Nam.Nguyen #30501: Produce optimized code for boolean conditions http://bugs.python.org/issue30501 opened by serhiy.storchaka #30502: Fix buffer handling of OBJ_obj2txt http://bugs.python.org/issue30502 opened by christian.heimes #30504: Allow inspecting buffering attribute of IO objects http://bugs.python.org/issue30504 opened by pitrou #30505: Performance of typing._ProtocolMeta._get_protocol_attrs and is http://bugs.python.org/issue30505 opened by orenbenkiki #30508: "Task exception was never retrieved" reported for a canceled t http://bugs.python.org/issue30508 opened by Miguel Grinberg #30509: Optimize calling type slots http://bugs.python.org/issue30509 opened by serhiy.storchaka #30510: c_bool type not supported for BigEndianStructure on little-end http://bugs.python.org/issue30510 opened by Hassan El Karouni #30511: shutil.make_archive should not need to chdir (alternatively: m http://bugs.python.org/issue30511 opened by Alex Gaynor #30512: CAN Socket support for NetBSD http://bugs.python.org/issue30512 opened by wiz #30513: getrusage returns platform-dependent value http://bugs.python.org/issue30513 opened by sam-s #30514: test_poplib replace asyncore http://bugs.python.org/issue30514 opened by grzgrzgrz3 #30516: Documentation for datetime substract operation incorrect? http://bugs.python.org/issue30516 opened by Ren?? Hern??ndez Remedios #30518: Import type aliases from another module http://bugs.python.org/issue30518 opened by Paragape #30519: Add daemon argument to Timer http://bugs.python.org/issue30519 opened by awolokita #30520: loggers can't be pickled http://bugs.python.org/issue30520 opened by pitrou #30521: IDLE: Add navigate bar and replace current goto dialog http://bugs.python.org/issue30521 opened by louielu #30522: Allow replacing a logging.StreamHandler's stream http://bugs.python.org/issue30522 opened by pitrou #30523: unittest: add --list-tests option to only display the list of http://bugs.python.org/issue30523 opened by haypo #30524: iter(classmethod, sentinel) broken for Argument Clinic class m http://bugs.python.org/issue30524 opened by mjpieters #30525: Expose SCTs on TLS connections http://bugs.python.org/issue30525 opened by alex #30526: Allow setting line_buffering on existing TextIOWrapper http://bugs.python.org/issue30526 opened by pitrou #30528: ipaddress.IPv{4,6}Network.reverse_pointer is broken http://bugs.python.org/issue30528 opened by h.venev #30529: Incorrect error messages for invalid whitespaces in f-string s http://bugs.python.org/issue30529 opened by serhiy.storchaka #30530: Descriptors HowTo: Example on function.__get__ needs update http://bugs.python.org/issue30530 opened by Mariano Anaya #30532: email.policy.SMTP.fold() mangles long headers http://bugs.python.org/issue30532 opened by chrisb at emergence.com #30533: missing feature in inspect module: getmembers_static http://bugs.python.org/issue30533 opened by carljm #30534: error message for incorrect call degraded in 3.7 http://bugs.python.org/issue30534 opened by scoder #30535: Warn that meta_path is not empty http://bugs.python.org/issue30535 opened by xmorel #30536: [EASY] SubinterpThreadingTests.test_threads_join_2() of test_t http://bugs.python.org/issue30536 opened by haypo #30537: Using PyNumber_AsSsize_t in itertools.islice http://bugs.python.org/issue30537 opened by MSeifert #30538: Functional Programming HOWTO describes one argument itertools. http://bugs.python.org/issue30538 opened by csabella #30539: Make Proactor public in asyncio.ProactorEventLoop http://bugs.python.org/issue30539 opened by jabdoa #30540: regrtest: add --matchfile option http://bugs.python.org/issue30540 opened by haypo #30541: Add restricted mocks to the python unittest mocking framework http://bugs.python.org/issue30541 opened by mariocj89 #30542: [EASY] test_files() of test_tools.test_unparse.DirectoryTestCa http://bugs.python.org/issue30542 opened by haypo #30543: test_timeout fails on AMD64 FreeBSD CURRENT Debug 3.x: Connect http://bugs.python.org/issue30543 opened by haypo #30544: _io._WindowsConsoleIO.write raises the wrong error when WriteC http://bugs.python.org/issue30544 opened by Segev Finer #30546: [EASY][Windows] test_uname_win32_ARCHITEW6432() of test_platfo http://bugs.python.org/issue30546 opened by haypo #30547: [EASY][Windows] SubinterpreterTest.test_callbacks_leak() of te http://bugs.python.org/issue30547 opened by haypo #30548: typo in documentation for create_autospec http://bugs.python.org/issue30548 opened by Erik Bystr??m #30549: ProcessPoolExecutor hangs forever if the object raises on __ge http://bugs.python.org/issue30549 opened by Roberto Mart??nez #30550: Document order-preserving dictionary output in json http://bugs.python.org/issue30550 opened by lebigot #30553: Add HTTP Response code 421 http://bugs.python.org/issue30553 opened by julienc #30554: Inaccessible attribute characters_written on OSError instances http://bugs.python.org/issue30554 opened by Igor Kozyrenko (ikseek) Most recent 15 issues with no replies (15) ========================================== #30554: Inaccessible attribute characters_written on OSError instances http://bugs.python.org/issue30554 #30553: Add HTTP Response code 421 http://bugs.python.org/issue30553 #30550: Document order-preserving dictionary output in json http://bugs.python.org/issue30550 #30549: ProcessPoolExecutor hangs forever if the object raises on __ge http://bugs.python.org/issue30549 #30548: typo in documentation for create_autospec http://bugs.python.org/issue30548 #30543: test_timeout fails on AMD64 FreeBSD CURRENT Debug 3.x: Connect http://bugs.python.org/issue30543 #30539: Make Proactor public in asyncio.ProactorEventLoop http://bugs.python.org/issue30539 #30530: Descriptors HowTo: Example on function.__get__ needs update http://bugs.python.org/issue30530 #30528: ipaddress.IPv{4,6}Network.reverse_pointer is broken http://bugs.python.org/issue30528 #30525: Expose SCTs on TLS connections http://bugs.python.org/issue30525 #30521: IDLE: Add navigate bar and replace current goto dialog http://bugs.python.org/issue30521 #30519: Add daemon argument to Timer http://bugs.python.org/issue30519 #30514: test_poplib replace asyncore http://bugs.python.org/issue30514 #30512: CAN Socket support for NetBSD http://bugs.python.org/issue30512 #30510: c_bool type not supported for BigEndianStructure on little-end http://bugs.python.org/issue30510 Most recent 15 issues waiting for review (15) ============================================= #30534: error message for incorrect call degraded in 3.7 http://bugs.python.org/issue30534 #30530: Descriptors HowTo: Example on function.__get__ needs update http://bugs.python.org/issue30530 #30529: Incorrect error messages for invalid whitespaces in f-string s http://bugs.python.org/issue30529 #30526: Allow setting line_buffering on existing TextIOWrapper http://bugs.python.org/issue30526 #30519: Add daemon argument to Timer http://bugs.python.org/issue30519 #30512: CAN Socket support for NetBSD http://bugs.python.org/issue30512 #30509: Optimize calling type slots http://bugs.python.org/issue30509 #30502: Fix buffer handling of OBJ_obj2txt http://bugs.python.org/issue30502 #30501: Produce optimized code for boolean conditions http://bugs.python.org/issue30501 #30495: IDLE: modernize textview module http://bugs.python.org/issue30495 #30487: DOC: automatically create a venv and install Sphinx when runni http://bugs.python.org/issue30487 #30485: Element.findall(path, dict) doesn't insert null namespace http://bugs.python.org/issue30485 #30466: Tutorial doesn't explain the use of classes http://bugs.python.org/issue30466 #30465: FormattedValue expressions have wrong lineno and col_offset in http://bugs.python.org/issue30465 #30455: Generate C code from token.py and not vice versa http://bugs.python.org/issue30455 Top 10 most discussed issues (10) ================================= #30177: pathlib.resolve(strict=False) only includes first child http://bugs.python.org/issue30177 15 msgs #25324: Importing tokenize modifies token http://bugs.python.org/issue25324 12 msgs #30495: IDLE: modernize textview module http://bugs.python.org/issue30495 12 msgs #30509: Optimize calling type slots http://bugs.python.org/issue30509 12 msgs #30300: asyncio.Controller http://bugs.python.org/issue30300 11 msgs #30535: Warn that meta_path is not empty http://bugs.python.org/issue30535 8 msgs #30534: error message for incorrect call degraded in 3.7 http://bugs.python.org/issue30534 7 msgs #11783: email parseaddr and formataddr should be IDNA aware http://bugs.python.org/issue11783 6 msgs #30501: Produce optimized code for boolean conditions http://bugs.python.org/issue30501 6 msgs #30526: Allow setting line_buffering on existing TextIOWrapper http://bugs.python.org/issue30526 6 msgs Issues closed (41) ================== #3062: Turtle speed() function has no effect under Mac OS X http://bugs.python.org/issue3062 closed by willingc #13349: Non-informative error message in index() and remove() function http://bugs.python.org/issue13349 closed by rhettinger #16500: Allow registering at-fork handlers http://bugs.python.org/issue16500 closed by pitrou #17188: Document 'from None' in raise statement doc. http://bugs.python.org/issue17188 closed by Mariatta #20210: Support the *disabled* marker in Setup files http://bugs.python.org/issue20210 closed by xdegaye #22702: to improve documentation for join() (str method) http://bugs.python.org/issue22702 closed by rhettinger #24692: types.coroutines() idempotence documentation http://bugs.python.org/issue24692 closed by seirl #27618: docs for threading.Lock claim it's a class (since 3.3), but it http://bugs.python.org/issue27618 closed by Mariatta #29660: Document that print/format_exception ignore etype http://bugs.python.org/issue29660 closed by Mariatta #29766: --with-lto still implied by --enable-optimizations in Python 2 http://bugs.python.org/issue29766 closed by gregory.p.smith #29828: Allow registering after-fork initializers in multiprocessing http://bugs.python.org/issue29828 closed by pitrou #29960: _random.Random state corrupted on exception http://bugs.python.org/issue29960 closed by Mariatta #30216: xdrlib.Unpacker.unpack_string returns bytes (docs say should b http://bugs.python.org/issue30216 closed by serhiy.storchaka #30245: possible overflow when organize struct.pack_into error message http://bugs.python.org/issue30245 closed by xiang.zhang #30248: Using boolean arguments in the _json module http://bugs.python.org/issue30248 closed by serhiy.storchaka #30354: Change data model documentation to zero-argument super() http://bugs.python.org/issue30354 closed by Mariatta #30361: Docs example: converting mixed types to floating point http://bugs.python.org/issue30361 closed by terry.reedy #30378: SysLogHandler does not support IPv6 destinations http://bugs.python.org/issue30378 closed by xiang.zhang #30398: Add a docstring for re.error http://bugs.python.org/issue30398 closed by serhiy.storchaka #30433: Devguide lacks instructions for building docs http://bugs.python.org/issue30433 closed by Mariatta #30446: Embedded 3.6.1 distribution cannot find _socket module http://bugs.python.org/issue30446 closed by ThomasS #30468: Propagate zipfile.py pypy issue #905 patch to CPython 3.7 http://bugs.python.org/issue30468 closed by serhiy.storchaka #30470: Deprecate invalid ctypes call protection on Windows http://bugs.python.org/issue30470 closed by Mariatta #30475: Docs for PyDict_GetItemWithError() should say it returns a bor http://bugs.python.org/issue30475 closed by eric.snow #30476: Add _GenerateCRCTable() to zipfile.py to pre-compute CRC Tab http://bugs.python.org/issue30476 closed by martin.panter #30477: tuple.index error message improvement http://bugs.python.org/issue30477 closed by brett.cannon #30488: Documentation for subprocess.STDOUT needs clarification http://bugs.python.org/issue30488 closed by eryksun #30494: python.exe crashed when open long path in windows 7 http://bugs.python.org/issue30494 closed by eryksun #30496: Incomplete traceback with `exec` on SyntaxError http://bugs.python.org/issue30496 closed by stefan #30497: Line number of docstring in AST http://bugs.python.org/issue30497 closed by serhiy.storchaka #30499: Deprecated note in set documentation http://bugs.python.org/issue30499 closed by Mariatta #30503: It should be possible to use a module name with the same name http://bugs.python.org/issue30503 closed by ncoghlan #30506: Replace 'list' with 'array' in array.remove and array.index http://bugs.python.org/issue30506 closed by serhiy.storchaka #30507: Elements reports it is a list on Element.remove http://bugs.python.org/issue30507 closed by Jim Fasarakis-Hilliard #30515: unittest: assertAlmostEqual rounding error http://bugs.python.org/issue30515 closed by sam-s #30517: Enum does not recognize enum.auto as unique values http://bugs.python.org/issue30517 closed by max #30527: PyMemoryView_FromBuffer memory leak http://bugs.python.org/issue30527 closed by lazka #30531: Windows 10 3.6.1 install disallows pip installs of packages http://bugs.python.org/issue30531 closed by eryksun #30545: Enum equality across modules: comparing objects instead of val http://bugs.python.org/issue30545 closed by ethan.furman #30551: Attribute Error http://bugs.python.org/issue30551 closed by Mahira #30552: Bugs http://bugs.python.org/issue30552 closed by Mahira From ncoghlan at gmail.com Fri Jun 2 12:10:50 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 Jun 2017 02:10:50 +1000 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> Message-ID: On 2 June 2017 at 19:42, Victor Stinner wrote: > Thanks Cory for the long explanation. Let me try to summarize (tell me > if I'm wrong). > > We have 3 options: > > * Do nothing: reject the PEP 546 and let each project handles security > on its own (current status co) > * Write *new* C code, maybe using certitude as a starting point, to > offload certifcate validation on Windows and macOS > * Backport existing code from master to 2.7: MemoryBIO and SSLObject There's also a 4th option: * Introduce a dependency from requests onto PyOpenSSL when running in async mode on Python 2.7 in the general case, and figure out some other pip-specific option for ensurepip bootstrapping (like a *private* MemoryBIO implementation, or falling back to synchronous mode in requests) During the pre-publication PEP discussions, I kinda dismissed the PyOpenSSL dependency option out of hand due to the ensurepip bootstrapping issues it may introduce, but I think we need to discuss it further in the PEP as it would avoid some of the other challenges brought up here (Linux distro update latencies, potential complications for alternate Python 2.7 implementations, etc). For example: * if requests retains a synchronous mode fallback implementation, then ensurepip could use that in the absence of PyOpenSSL * even if requests drops synchronous mode entirely, we could leave the public ssl module API alone, and add an _ensurepip bootstrap module specifically for use in the absence of a full PyOpenSSL module If we adopted the latter approach, then for almost all intents and purposes, ssl.MemoryBIO and ssl.SSLObject would remain a Python 3.5+ only API, and anyone wanting access to it on 2.7 would still need to depend on PyOpenSSL. The benefit of making any backport a private API is that it would mean we weren't committing to support that API for general use: it would be supported *solely* for the use case discussed in the PEP (i.e. helping to advance the development of PEP 543 without breaking pip bootstrapping in the process). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From barry at python.org Fri Jun 2 12:21:17 2017 From: barry at python.org (Barry Warsaw) Date: Fri, 2 Jun 2017 12:21:17 -0400 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> Message-ID: <20170602122117.5f86e422@subdivisions.wooz.org> On Jun 03, 2017, at 02:10 AM, Nick Coghlan wrote: >* Introduce a dependency from requests onto PyOpenSSL when running in >async mode on Python 2.7 in the general case, and figure out some >other pip-specific option for ensurepip bootstrapping (like a >*private* MemoryBIO implementation, or falling back to synchronous >mode in requests) [...] > >If we adopted the latter approach, then for almost all intents and >purposes, ssl.MemoryBIO and ssl.SSLObject would remain a Python 3.5+ >only API, and anyone wanting access to it on 2.7 would still need to >depend on PyOpenSSL. > >The benefit of making any backport a private API is that it would mean >we weren't committing to support that API for general use: it would be >supported *solely* for the use case discussed in the PEP (i.e. helping >to advance the development of PEP 543 without breaking pip >bootstrapping in the process). That sounds like a good compromise. My own major objection was in exposing a new public API in Python 2.7, which would clearly be a new feature. Cheers, -Barry From donald at stufft.io Fri Jun 2 12:22:06 2017 From: donald at stufft.io (Donald Stufft) Date: Fri, 2 Jun 2017 12:22:06 -0400 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> Message-ID: > On Jun 2, 2017, at 12:10 PM, Nick Coghlan wrote: > > On 2 June 2017 at 19:42, Victor Stinner wrote: >> Thanks Cory for the long explanation. Let me try to summarize (tell me >> if I'm wrong). >> >> We have 3 options: >> >> * Do nothing: reject the PEP 546 and let each project handles security >> on its own (current status co) >> * Write *new* C code, maybe using certitude as a starting point, to >> offload certifcate validation on Windows and macOS >> * Backport existing code from master to 2.7: MemoryBIO and SSLObject > > There's also a 4th option: > > * Introduce a dependency from requests onto PyOpenSSL when running in > async mode on Python 2.7 in the general case, and figure out some > other pip-specific option for ensurepip bootstrapping (like a > *private* MemoryBIO implementation, or falling back to synchronous > mode in requests) > > During the pre-publication PEP discussions, I kinda dismissed the > PyOpenSSL dependency option out of hand due to the ensurepip > bootstrapping issues it may introduce, but I think we need to discuss > it further in the PEP as it would avoid some of the other challenges > brought up here (Linux distro update latencies, potential > complications for alternate Python 2.7 implementations, etc). > It?s not just bootstrapping that pip has a problem with for C extensions, it also prevents upgrading PyOpenSSL on Windows because having pip import PyOpenSSL locks the .dll, and we can?t delete it or overwrite it until the pip process exits and no longer imports PyOpenSSL. This isn?t a problem on Linux or macOS or the other *nix clients though. We patch requests as it is today to prevent it from importing simplejson and cryptography for this reason. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Jun 2 12:39:44 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 Jun 2017 02:39:44 +1000 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> Message-ID: On 3 June 2017 at 02:22, Donald Stufft wrote: > It?s not just bootstrapping that pip has a problem with for C extensions, it > also prevents upgrading PyOpenSSL on Windows because having pip import > PyOpenSSL locks the .dll, and we can?t delete it or overwrite it until the > pip process exits and no longer imports PyOpenSSL. This isn?t a problem on > Linux or macOS or the other *nix clients though. We patch requests as it is > today to prevent it from importing simplejson and cryptography for this > reason. Would requests be loading PyOpenSSL on Windows, though? If the aim is to facilitate PEP 543, then I'd expect it to be using the SChannel backend in that case. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Fri Jun 2 12:41:04 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 2 Jun 2017 18:41:04 +0200 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> Message-ID: <20170602184104.6b3ed724@fsol> On Fri, 2 Jun 2017 12:22:06 -0400 Donald Stufft wrote: > > It?s not just bootstrapping that pip has a problem with for C extensions, it also prevents upgrading PyOpenSSL on Windows because having pip import PyOpenSSL locks the .dll, and we can?t delete it or overwrite it until the pip process exits and no longer imports PyOpenSSL. This isn?t a problem on Linux or macOS or the other *nix clients though. We patch requests as it is today to prevent it from importing simplejson and cryptography for this reason. Does pip use any advanced features in Requests, at least when it comes to downloading packages (which is where the bootstrapping issue lies AFAIU)? Because at this point it should like you may be better off with a simple pure Python HTTP downloader. Regards Antoine. From k7hoven at gmail.com Fri Jun 2 12:41:16 2017 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Fri, 2 Jun 2017 19:41:16 +0300 Subject: [Python-Dev] PEP 484 update proposal: annotating decorated declarations In-Reply-To: References: Message-ID: On Fri, Jun 2, 2017 at 6:34 PM, Naomi Seyfer wrote: > Yep, interested in implementing it! I will put implementation time on my > schedule and tell y'all when it is, for holding myself accountable -- it > turns out I never do anything not on my schedule. > I still don't understand what would happen with __annotations__. If the decorator returns a non-function, one would expect the annotations to be in the __annotations__ attribute of the enclosing class or module. If it returns a function, they would be in the __annotations__ attribute of the function. And I'm talking about the runtime behavior in Python as explained in PEP484 and PEP526. I would expect these declarations to behave according to the same principles as other ways to annotate variables/functions. If there is no runtime behavior, a comment-based syntax might be more appropriate. Or have I missed something? ?Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + From donald at stufft.io Fri Jun 2 12:49:06 2017 From: donald at stufft.io (Donald Stufft) Date: Fri, 2 Jun 2017 12:49:06 -0400 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> Message-ID: <33873A8B-A0AC-41E9-90AB-58ACA277AE8A@stufft.io> > On Jun 2, 2017, at 12:39 PM, Nick Coghlan wrote: > > On 3 June 2017 at 02:22, Donald Stufft wrote: >> It?s not just bootstrapping that pip has a problem with for C extensions, it >> also prevents upgrading PyOpenSSL on Windows because having pip import >> PyOpenSSL locks the .dll, and we can?t delete it or overwrite it until the >> pip process exits and no longer imports PyOpenSSL. This isn?t a problem on >> Linux or macOS or the other *nix clients though. We patch requests as it is >> today to prevent it from importing simplejson and cryptography for this >> reason. > > Would requests be loading PyOpenSSL on Windows, though? If the aim is > to facilitate PEP 543, then I'd expect it to be using the SChannel > backend in that case. > I?m not sure! The exact specifics of how it?d get implemented and the transition from what we have now to that could be yes or no (particularly the transition period). I?m just making sure that the constraint we have in pip is clearly defined here to make sure we don?t accept something that ends up not actually being suitable. I don?t have an opinion on the private bootstrap module (well I do, I like it less than just back porting MemoryBio ?for real", but not one on whether it?d work or not). ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From dw+python-dev at hmmz.org Fri Jun 2 12:51:41 2017 From: dw+python-dev at hmmz.org (David Wilson) Date: Fri, 2 Jun 2017 16:51:41 +0000 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> Message-ID: <20170602165141.GA26265@k3> On Sat, Jun 03, 2017 at 02:10:50AM +1000, Nick Coghlan wrote: > * and figure out some other pip-specific option for ensurepip > bootstrapping (like a *private* MemoryBIO implementation, or falling > back to synchronous mode in requests) Ignoring Ben's assertion regarding the legitimacy of async wrap_socket() (which seems to render this entire conversation moot), if you still really want to go this route, could ctypes be abused to provide the missing implementation from the underlying libs? It'd be a hack, but it would only be necessary during bootstrapping. David From donald at stufft.io Fri Jun 2 12:59:42 2017 From: donald at stufft.io (Donald Stufft) Date: Fri, 2 Jun 2017 12:59:42 -0400 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <20170602184104.6b3ed724@fsol> References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> <20170602184104.6b3ed724@fsol> Message-ID: <5E4A70A9-D5C4-4081-9746-0CD8696642ED@stufft.io> > On Jun 2, 2017, at 12:41 PM, Antoine Pitrou wrote: > > On Fri, 2 Jun 2017 12:22:06 -0400 > Donald Stufft wrote: >> >> It?s not just bootstrapping that pip has a problem with for C extensions, it also prevents upgrading PyOpenSSL on Windows because having pip import PyOpenSSL locks the .dll, and we can?t delete it or overwrite it until the pip process exits and no longer imports PyOpenSSL. This isn?t a problem on Linux or macOS or the other *nix clients though. We patch requests as it is today to prevent it from importing simplejson and cryptography for this reason. > > Does pip use any advanced features in Requests, at least when it comes > to downloading packages (which is where the bootstrapping issue lies > AFAIU)? Because at this point it should like you may be better off with > a simple pure Python HTTP downloader. > It?s hard to fully answer the question because it sort of depends? Could we switch to just like, urllib2 or something? Yea we could, in fact we used to use that and switched to using requests because we had to backport security work around / fixes ourselves (the big one at the time was host name matching/verification) and we were really bad at keeping up with tracking what patches needed applying and when. Switching to requests let us offload that work to the requests team who are doing a phenomenal job at it. Beyond that though, getting HTTP right is hard, and pip used to have to try and implement work arounds to either broken or less optimal urllib2 behavior whereas requests generally gets it right for us out of the box. Closer to your specific questions about features, we?re using the requests session support to handle connection pooling to speed up downloading (since we don?t need to open a new connection for every download then), the adapter API to handle transparently allowing file:// URLs, the auth framework to handle holding authentication for multiple domains at once, and the third party library cachecontrol to handle our HTTP caching using a browser style cache. I suspect (though I?d let him speak for himself) that Cory would rather continue to be sync only than require pip to go back to not using requests. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at vorpus.org Fri Jun 2 13:11:09 2017 From: njs at vorpus.org (Nathaniel Smith) Date: Fri, 2 Jun 2017 10:11:09 -0700 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> <20170602140622.64cbfca4@fsol> Message-ID: On Jun 2, 2017 7:24 AM, "Ben Darnell" wrote: The PEP's rationale is now "This PEP will help facilitate the future adoption of :pep:`543` across all supported Python versions, which will improve security for both Python 2 and Python 3 users." What exactly are these security improvements? My understanding (which may well be incorrect) is that the security improvements come in the future when the PEP 543 APIs are implemented on top of the various platform-native security libraries. These changes will not be present until some future 3.x release, and will not be backported to 2.7 (without another PEP, which I expect would be even more contentious than this one). What then is the security improvement for Python 2 users? My understanding is that PEP 543 would be released as a library on pypi, as well as targeting stdlib inclusion in some future release. The goal of the MemoryBIO PEP is to allow the PEP 543 package to be pure-python, support all the major platforms, and straddle py2/py3 In Tornado, I have not felt any urgency to replace wrap_socket with MemoryBIO. Is there a security-related reason I should do so sooner rather than later? (I'd also urge Cory and any other wrap_socket skeptics on the requests team to reconsider - Tornado's SSLIOStream works well. The asynchronous use of wrap_socket isn't so subtle and risky with buffering) I'll leave the discussion of wrap_socket's reliability to others, because I don't have any experience there, but I do want to point out that that which primitive you pick has major system design consequences. MemoryBIO is an abstraction that can implement wrap_socket, but not vice-versa; if you use wrap_socket as your fundamental primitive then that leaks all over your abstraction hierarchy. Twisted has to have a MemoryBIO? implementation of their TLS code, because they want to support TLS over arbitrary transports. Carrying a wrap_socket implementation as well would mean twice the tricky and security-sensitive code to maintain, plus breaking their current abstractions to expose whether any particular transport object is actually just a raw socket. The problem gets worse when you add PEP 543's pluggable TLS backends. If you can require a MemoryBIO-like API, then adding a backend becomes a matter of defining like 4 methods with relatively simple, testable interfaces, and it automatically works everywhere. Implementing a wrap_socket interface on top of this is complicated because the socket API is complicated and full of subtle corners, but it only has to be done once and libraries like tornado can adapt to the quirks of the one implementation. OTOH if you have to also support backends that only have wrap_socket, then this multiplies the complexity of everything. Now we need a way to negotiate which APIs each backend supports, and we need to somehow document all the weird corners of how wrapped sockets are supposed to behave in edge cases, and when different wrapped sockets inevitably behave differently then libraries like tornado need to discover those differences and support all the options. We need a way to explain to users why some backends work with some libraries but not others. And so on. And we still need to support MemoryBIOs in as many cases as possible. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Jun 2 13:57:39 2017 From: guido at python.org (Guido van Rossum) Date: Fri, 2 Jun 2017 10:57:39 -0700 Subject: [Python-Dev] PEP 484 update proposal: annotating decorated declarations In-Reply-To: References: Message-ID: On Fri, Jun 2, 2017 at 9:41 AM, Koos Zevenhoven wrote: > On Fri, Jun 2, 2017 at 6:34 PM, Naomi Seyfer wrote: > > Yep, interested in implementing it! I will put implementation time on my > > schedule and tell y'all when it is, for holding myself accountable -- it > > turns out I never do anything not on my schedule. > > I still don't understand what would happen with __annotations__. If > the decorator returns a non-function, one would expect the annotations > to be in the __annotations__ attribute of the enclosing class or > module. If it returns a function, they would be in the __annotations__ > attribute of the function. And I'm talking about the runtime behavior > in Python as explained in PEP484 and PEP526. I would expect these > declarations to behave according to the same principles as other ways > to annotate variables/functions. If there is no runtime behavior, a > comment-based syntax might be more appropriate. Or have I missed > something? > So when returning a function, the runtime version of the decorator can easily update the function's __annotations__. But when returning a non-function, the decorator would have a hard time updating __annotations__ of the containing class/module without "cheating" (e.g. sys._getframe()). I think the latter is similar to e.g. attributes defined with @property -- those don't end up in __annotations__ either. I think this is an acceptable deficiency. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Fri Jun 2 14:23:05 2017 From: tim.peters at gmail.com (Tim Peters) Date: Fri, 2 Jun 2017 13:23:05 -0500 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: References: Message-ID: [INADA Naoki ] > ... > Since current pool size is 4KB and there is pool_header in pool, > we can't allocate 4KB block from pool. > And if support 1KB block, only 3KB of 4KB can be actually used. > I think 512 bytes / 4KB (1/8) is good ratio. > > Do you mean increase pool size? > > How about adding configure option like server-mode? > > SMALL_REQUEST_THRESHOLD 1024 // 2x > POOL_SIZE (16*1024) // 4x > ARENA_SIZE (2*1024*1024) // 8x, and same to huge page size. While I would like to increase the pool size, it's fraught with danger. The problem: Py_ADDRESS_IN_RANGE has to figure out whether an "arbitrary" address is - or is not - controlled by Python's small-object allocator. The excruciating logic is spelled out at length in obmalloc.c. As is, the code reads up memory near "the start" of a pool to get the pool's belief about which arena it's in. If the memory is not in fact controlled by pymalloc, this can be anything, even uninitialized trash. That's fine (as explained in the comments) - but that memory _must_ be readable. And that's why POOL_SIZE is set to 4K: it's the smallest page size across all the OSes Python is known to run on. If pymalloc is handed an "arbitrary" (but valid) address, the entire OS page containing that address is readable. If, e.g., an OS allocates 4K pages, but Python's POOL_SIZE is 8K (anything bigger than the OS's page allocation unit), then perhaps the OS page at 16K is unallocated, but the page at 20K is. pymalloc sees an address at 21K. As is, pymalloc reads the putative arena index at 20K, which is fine. But if POOL_SIZE were 8K, it would try to read the putative arena index at 16K, and - boom! - segfault. This failure mode would be rare but - of course - catastrophic when it occurs. It would be nice to find a different way for pymalloc to figure out which addresses belong to it. The excruciating Py_ADDRESS_IN_RANGE manages to do it in small constant (independent of the number of arenas and pools in use) time, which is its only virtue ;-) From solipsis at pitrou.net Fri Jun 2 14:37:34 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 2 Jun 2017 20:37:34 +0200 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE References: Message-ID: <20170602203734.79329dfe@fsol> On Fri, 2 Jun 2017 13:23:05 -0500 Tim Peters wrote: > > While I would like to increase the pool size, it's fraught with > danger. What would be the point of increasing the pool size? Apart from being able to allocate 4KB objects out of it, I mean. Since 4KB+ objects are relatively uncommon (I mean we don't allocate hundreds of thousands of them per second), I don't think it's really worthwhile trying to have the small object allocator handle them. > It would be nice to find a different way for pymalloc to figure out > which addresses belong to it. The excruciating Py_ADDRESS_IN_RANGE > manages to do it in small constant (independent of the number of > arenas and pools in use) time, which is its only virtue ;-) So, to sum it up, it's excruciating but fast and works reliably. Why change it? Regards Antoine. From tim.peters at gmail.com Fri Jun 2 15:09:42 2017 From: tim.peters at gmail.com (Tim Peters) Date: Fri, 2 Jun 2017 14:09:42 -0500 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: <20170602203734.79329dfe@fsol> References: <20170602203734.79329dfe@fsol> Message-ID: [Tim] >> While I would like to increase the pool size, it's fraught with >> danger. [Antoine Pitrou ] > What would be the point of increasing the pool size? Apart from being > able to allocate 4KB objects out of it, I mean. > > Since 4KB+ objects are relatively uncommon (I mean we don't allocate > hundreds of thousands of them per second), I don't think it's really > worthwhile trying to have the small object allocator handle them. I don't care about "large" objects here. It's educated intuition about speed: so long as pymalloc is working within a pool, it's blazing fast. When it has to muck with a new pool, much slower (compared to staying within a pool) code is required. When it has to muck with a new arena, slower still. So the intuition is simple: the larger a pool, the more object operations it can handle staying within its best-case (fastest) code paths. >> It would be nice to find a different way for pymalloc to figure out >> which addresses belong to it. The excruciating Py_ADDRESS_IN_RANGE >> manages to do it in small constant (independent of the number of >> arenas and pools in use) time, which is its only virtue ;-) > So, to sum it up, it's excruciating but fast and works reliably. Why > change it? To enable using pools larger than the greatest common divisor of all OS's native pool sizes. There's also that Py_ADDRESS_IN_RANGE is responsible for countless hours of head-scratching by poor developers trying to use magical memory debuggers - it's at best unusual for code to read up memory without caring a bit whether the memory has ever been stored to. I should note that Py_ADDRESS_IN_RANGE is my code - this isn't a backhanded swipe at someone else. It's always been near the top of the "code stink" scale. So I thought I'd mention it in case someone has been sitting on a cleaner idea but has held back because they didn't want to offend me ;-) From larry at hastings.org Fri Jun 2 15:31:19 2017 From: larry at hastings.org (Larry Hastings) Date: Fri, 2 Jun 2017 12:31:19 -0700 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: References: Message-ID: On 06/02/2017 02:46 AM, Victor Stinner wrote: > I would be curious of another test: use pymalloc for objects larger > than 512 bytes. For example, allocate up to 4 KB? > > In the past, we already changed the maximum size from 256 to 512 to > support most common Python objects on 64-bit platforms. Since Python > objects contain many pointers: switching from 32 bit to 64 bit can > double the size of the object in the worst case. You've already seen Tim Peters' post about why we must leave pool size set to 4k. Obviously This in turn means using obmalloc for larger objects will mean more and more wasted memory. For example, let's say we use obmalloc for allocations of 2048 bytes. Pool size is 4096 bytes, and there's a 48-byte "pool_header" structure on the front (on 64-bit platforms, if I counted right). So there are only 4048 bytes usable per pool. After the first 2048 allocation, we're left with 2000 bytes at the end. You can't use that memory for another allocation class, that's impossible given obmalloc's design. So that 2000 bytes is just wasted. Currently obmalloc's maximum allocation size is 512 bytes; after 7 allocations, this leaves 464 wasted bytes at the end. Which isn't *great* exactly but it's only 11% of the overall allocated memory. Anyway, I'm not super excited by the prospect of using obmalloc for larger objects. There's an inverse relation between the size of allocation and the frequency of allocation. In Python there are lots of tiny allocations, but fewer and fewer as the size increases. (A similarly-shaped graph to what retailers call the "long tail".) By no small coincidence, obmalloc is great at small objects, which is where we needed the help most. Let's leave it at that. A more fruitful endeavor might be to try one of these fancy new third-party allocators in CPython, e.g. tcmalloc, jemalloc. Try each with both obmalloc turned on and turned off, and see what happens to performance and memory usage. (I'd try it myself, but I'm already so far behind on watching funny cat videos.) //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Fri Jun 2 15:33:41 2017 From: larry at hastings.org (Larry Hastings) Date: Fri, 2 Jun 2017 12:33:41 -0700 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: References: <20170602203734.79329dfe@fsol> Message-ID: <3a652def-a01a-c01f-b6d3-bb9d0baf24e2@hastings.org> On 06/02/2017 12:09 PM, Tim Peters wrote: > I should note that Py_ADDRESS_IN_RANGE is my code - this isn't a > backhanded swipe at someone else. One minor note. During the development of 3.6, CPython started permitting some C99-isms, including static inline functions. Py_ADDRESS_IN_RANGE was therefore converted from a macro into a static inline function, and its new name is address_in_range. Just so we're all on the same (4k!) page, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Fri Jun 2 16:05:21 2017 From: larry at hastings.org (Larry Hastings) Date: Fri, 2 Jun 2017 13:05:21 -0700 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: <20170602113816.0176b09b@fsol> References: <20170601115014.18dcb755@fsol> <32428072-366e-dd64-5b71-254fd58deade@hastings.org> <20170602113816.0176b09b@fsol> Message-ID: <873daa82-14f8-9e3d-d1d2-779160e80ed3@hastings.org> On 06/02/2017 02:38 AM, Antoine Pitrou wrote: > I hope those are not the actual numbers you're intending to use ;-) > I still think that allocating more than 1 or 2MB at once would be > foolish. Remember this is data that's going to be carved up into > (tens of) thousands of small objects. Large objects eschew the small > object allocator (not to mention that third-party libraries like Numpy > may be using different allocation routines when they allocate very > large data). Honest, I'm well aware of what obmalloc does and how it works. I bet I've spent more time crawling around in it in the last year than anybody else on the planet. Mainly because it works so well for CPython, nobody else needed to bother! I'm also aware, for example, that if your process grows to consume gigabytes of memory, you're going to have tens of thousands of allocated arenas. The idea that on systems with gigabytes of memory--90%+? of current systems running CPython--we should allocate memory forever in 256kb chunks is faintly ridiculous. I agree that we should start small, and ramp up slowly, so Python continues to run well on small computers and not allocate tons of memory for small programs. But I also think we should ramp up *ever*, for programs that use tens or hundreds of megabytes. Also note that if we don't touch the allocated memory, smart modern OSes won't actually commit any resources to it. All that happens when your process allocates 1GB is that the OS changes some integers around. It doesn't actually commit any memory to your process until you attempt to write to that memory, at which point it gets mapped in in local-page-size chunks (4k? 8k? something in that neighborhood and power-of-2 sized). So if we allocate 32mb, and only touch the first 1mb, the other 31mb doesn't consume any real resources. I was planning on making the multi-arena code only touch memory when it actually needs to, similarly to the way obmalloc lazily consumes memory inside an allocated pool (see the nextoffset field in pool_header), to take advantage of this ubiquitous behavior. If I write this multi-arena code, which I might, I was thinking I'd try this approach: * leave arenas themselves at 256k * start with a 1MB multi-arena size * every time I allocate a new multi-arena, multiply the size of the next multi-arena by 1.5 (rounding up to 256k each time) * every time I free a multi-arena, divide the size of the next multi-arena by 2 (rounding up to 256k each time) * if allocation of a multi-arena fails, use a binary search algorithm to allocate the largest multi-arena possible (rounding up to 256k at each step) * cap the size of multi arenas at, let's say, 32mb So multi-arenas would be 1mb, 1.5mb, 2.25mb, 3.5mb (round up!), etc. Fun fact: Python allocates 16 arenas at the start of the program, just to initialize obmalloc. That consumes 4mb of memory. With the above multi-arena approach, that'd allocate the first three multi-arenas, pre-allocating 19 arenas, leaving 3 unused. It's *mildly* tempting to make the first multi-arena be 4mb, just so this is exactly right-sized, but... naah. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Fri Jun 2 16:07:01 2017 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Fri, 2 Jun 2017 23:07:01 +0300 Subject: [Python-Dev] PEP 484 update proposal: annotating decorated declarations In-Reply-To: References: Message-ID: On Fri, Jun 2, 2017 at 8:57 PM, Guido van Rossum wrote: > On Fri, Jun 2, 2017 at 9:41 AM, Koos Zevenhoven wrote: >> >> I still don't understand what would happen with __annotations__. If >> the decorator returns a non-function, one would expect the annotations >> to be in the __annotations__ attribute of the enclosing class or >> module. If it returns a function, they would be in the __annotations__ >> attribute of the function. And I'm talking about the runtime behavior >> in Python as explained in PEP484 and PEP526. I would expect these >> declarations to behave according to the same principles as other ways >> to annotate variables/functions. If there is no runtime behavior, a >> comment-based syntax might be more appropriate. Or have I missed >> something? > > > So when returning a function, the runtime version of the decorator can > easily update the function's __annotations__. But when returning a > non-function, the decorator would have a hard time updating __annotations__ > of the containing class/module without "cheating" (e.g. sys._getframe()). I > think the latter is similar to e.g. attributes defined with @property -- > those don't end up in __annotations__ either. I think this is an acceptable > deficiency. > I suppose it is, especially because there seems to be nothing that prevents you from getting runtime annotations in the enclosing class/module ?: number: int @call def number(): return 42 But for functions one could have ( ?using the context manager example): def session(url: str) -> ContextManager[DatabaseSession]: ... @predeclared @contextmanager def session(url: str) -> Iterator[DatabaseSession]: s = DatabaseSession(url) try: yield s finally: s.close() This makes it clear that the function is declared elsewhere. But the `predeclared` decorator would need tricks like sys._getframe(1) to set session.__annotations__ according to the predeclaration. -- Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Fri Jun 2 16:29:57 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 2 Jun 2017 16:29:57 -0400 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <20170602122117.5f86e422@subdivisions.wooz.org> References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> <20170602122117.5f86e422@subdivisions.wooz.org> Message-ID: On 6/2/2017 12:21 PM, Barry Warsaw wrote: > On Jun 03, 2017, at 02:10 AM, Nick Coghlan wrote: >> The benefit of making any backport a private API is that it would mean >> we weren't committing to support that API for general use: it would be >> supported *solely* for the use case discussed in the PEP (i.e. helping >> to advance the development of PEP 543 without breaking pip >> bootstrapping in the process). > > That sounds like a good compromise. My own major objection was in exposing a > new public API in Python 2.7, which would clearly be a new feature. Which would likely be seen by someone as justifying other requests to add to 2.7 'just this one more essential new feature' ;-). -- Terry Jan Reedy From levkivskyi at gmail.com Fri Jun 2 18:10:06 2017 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Sat, 3 Jun 2017 00:10:06 +0200 Subject: [Python-Dev] PEP 544: Protocols - second round In-Reply-To: References: <8b064822-14bb-6800-810a-c14dc97f565a@hotpy.org> Message-ID: On 1 June 2017 at 00:10, Guido van Rossum wrote: > > On Wed, May 31, 2017 at 2:16 AM, Ivan Levkivskyi > wrote: > >> On 31 May 2017 at 00:58, Guido van Rossum wrote: >> [...] >> >> Thank you for very detailed answers! I have practically nothing to add. >> It seems to me that most of the Kevin's questions stem from unnecessary >> focus >> on runtime type checking. Here are two ideas about how to fix this: >> >> * Add the word "static" somewhere in the PEP title. >> > > So the title could become "Protocols: Static structural subtyping (duck > typing)" -- long, but not record-setting. > I am thinking about "Protocols: Structural subtyping (static duck typing)". The reason is that subtyping is already a mostly static concept (in contrast to subclassing), while duck typing is typically associated with the runtime behaviour. This might seem minor, but this version of the title sounds much more naturally to me. -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Jun 2 18:16:10 2017 From: guido at python.org (Guido van Rossum) Date: Fri, 2 Jun 2017 15:16:10 -0700 Subject: [Python-Dev] PEP 484 update proposal: annotating decorated declarations In-Reply-To: References: Message-ID: On Fri, Jun 2, 2017 at 1:07 PM, Koos Zevenhoven wrote: > [...] > I suppose it is, especially because there seems to be nothing that > prevents you from getting runtime annotations in the enclosing class/module > ?: > > > number: int > > @call > def number(): > return 42 > Well mypy actually gives an error for that, "Name 'number' already defined". > > But for functions one could have ( > ?using > the context manager example): > > > def session(url: str) -> ContextManager[DatabaseSession]: ... > > @predeclared > @contextmanager > def session(url: str) -> Iterator[DatabaseSession]: > s = DatabaseSession(url) > try: > yield s > finally: > s.close() > > > This makes it clear that the function is declared elsewhere. But the > `predeclared` decorator would need tricks like sys._getframe(1) to set > session.__annotations__ according to the predeclaration. > I'm not excited about that. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Jun 2 18:21:53 2017 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 2 Jun 2017 15:21:53 -0700 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> <20170602122117.5f86e422@subdivisions.wooz.org> Message-ID: On Fri, Jun 2, 2017 at 1:29 PM, Terry Reedy wrote: > On 6/2/2017 12:21 PM, Barry Warsaw wrote: >> >> On Jun 03, 2017, at 02:10 AM, Nick Coghlan wrote: > > >>> The benefit of making any backport a private API is that it would mean >>> we weren't committing to support that API for general use: it would be >>> supported *solely* for the use case discussed in the PEP (i.e. helping >>> to advance the development of PEP 543 without breaking pip >>> bootstrapping in the process). >> >> >> That sounds like a good compromise. My own major objection was in >> exposing a >> new public API in Python 2.7, which would clearly be a new feature. > > > Which would likely be seen by someone as justifying other requests to add to > 2.7 'just this one more essential new feature' ;-). Whatever the eventual outcome, I don't think there's any danger someone will read this thread and think "wow, it's so easy to get new features into 2.7". -n -- Nathaniel J. Smith -- https://vorpus.org From ncoghlan at gmail.com Sat Jun 3 02:25:06 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 Jun 2017 16:25:06 +1000 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> <20170602122117.5f86e422@subdivisions.wooz.org> Message-ID: On 3 June 2017 at 08:21, Nathaniel Smith wrote: > On Fri, Jun 2, 2017 at 1:29 PM, Terry Reedy wrote: >> On 6/2/2017 12:21 PM, Barry Warsaw wrote: >>> >>> On Jun 03, 2017, at 02:10 AM, Nick Coghlan wrote: >> >> >>>> The benefit of making any backport a private API is that it would mean >>>> we weren't committing to support that API for general use: it would be >>>> supported *solely* for the use case discussed in the PEP (i.e. helping >>>> to advance the development of PEP 543 without breaking pip >>>> bootstrapping in the process). >>> >>> >>> That sounds like a good compromise. My own major objection was in >>> exposing a >>> new public API in Python 2.7, which would clearly be a new feature. >> >> >> Which would likely be seen by someone as justifying other requests to add to >> 2.7 'just this one more essential new feature' ;-). > > Whatever the eventual outcome, I don't think there's any danger > someone will read this thread and think "wow, it's so easy to get new > features into 2.7". Aye, when "Be related to the Python ecosystem's collective handling of one of the essential network protocols that underpins the security of the entire internet" is a criterion to even *start* the backport discussion (let alone successfully make the case), I don't think "We might get deluged with further backport proposals" is really something we need to worry about :) Another key factor is being able to convince folks with some level of influence with commercial redistributors that the idea is worth supporting - the upstream backport in CPython is only step one in rolling out this kind of change, and we need to get it all the way through into the redistributor channels for it to really have the desired impact (hence PEP 493 as a follow-up to PEP 476 - at least some redistributors, including Red Hat, said "No" to the original hard compatibility break, but were OK with a smoother transition plan that gave end users more control over the timing of the change in behaviour). For the MemoryBIO and SSLObject backport, I'm still in the "Maybe" state myself. While I'm definitely sympathetic to the proposal (SSL/TLS is incredibly important, but cross-distro and cross-version SSL/TLS is already painful, and it's even worse once you bring cross-platform considerations and the sync/async split into the mix), it's an unfortunate fact of redistributor life that we don't typically have the luxury of pitching backport proposals to product management based on the benefits they offer to the upstream development community - we need to be able to articulate a clear and relatively immediate benefit to customers or commercial partners. For this PEP, "We want to make the new Python TLS abstraction API work" would potentially qualify as such a benefit (although I can't make any promises about that), but we're not even at that stage yet - that's a point we'd only reach if the library was initially built with a dependency on 3.5+ and 2.7.14+, *and* we either saw significant demand for that new API amongst customers, or else we genuinely needed to bring it in as a dependency of something else (and even then, we prefer approaches like a clearly distinct PyOpenSSL dependency that can be handled in a higher level product like OpenStack rather than needing to be baked directly into the operating system itself). As a result, as awkward and annoying as it would be, I'm wondering if the structure of the requests migration may need to be something like: - switch unconditionally to an async backend on Py3 - switch conditionally to an async backend on Py2 when PyOpenSSL is available - retain sufficient sync-only support on Py2 to allow pip to disable the PyOpenSSL dependency As a redistributor, ensuring that sufficiently recent versions of PyOpenSSL are available is likely to be a *far* more tractable problem than getting backports implemented at the standard library level (even as a private API), and even if that doesn't happen, developers that are using 3rd party modules like requests usually have significantly more freedom to install additional Python level dependencies than they do to upgrade their production Python runtime. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Sat Jun 3 06:07:06 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 3 Jun 2017 12:07:06 +0200 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE References: Message-ID: <20170603120706.75fa7ba8@fsol> On Fri, 2 Jun 2017 12:31:19 -0700 Larry Hastings wrote: > > Anyway, I'm not super excited by the prospect of using obmalloc for > larger objects. There's an inverse relation between the size of > allocation and the frequency of allocation. In Python there are lots of > tiny allocations, but fewer and fewer as the size increases. (A > similarly-shaped graph to what retailers call the "long tail".) By no > small coincidence, obmalloc is great at small objects, which is where we > needed the help most. Let's leave it at that. +1 to that and nice explanation. > A more fruitful endeavor might be to try one of these fancy new > third-party allocators in CPython, e.g. tcmalloc, jemalloc. Try each > with both obmalloc turned on and turned off, and see what happens to > performance and memory usage. (I'd try it myself, but I'm already so > far behind on watching funny cat videos.) We should lobby for a ban on funny cat videos so that you spend more time on CPython. Regards Antoine. From solipsis at pitrou.net Sat Jun 3 06:31:55 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 3 Jun 2017 12:31:55 +0200 Subject: [Python-Dev] Extremely slow test modules Message-ID: <20170603123155.128494e7@fsol> Hi, Is there a reason some of our tests are excruciatingly slow in `-uall` mode? `test_multiprocessing_spawn` is understandable (after all, it will spawn a new executable for each subprocess), but other tests leave me baffled: - test_tools: 7 min 41 sec - test_tokenize: 6 min 23 sec - test_datetime: 6 min 3 sec - test_lib2to3: 5 min 25 sec [excerpt from recent Travis CI logs] Why does datetime, 2to3 or tokenize testing take so long? And do we have so many tools that it should take 7 minutes to run all of them? I must admit, I don't understand how we got to such a point. Regards Antoine. From storchaka at gmail.com Sat Jun 3 08:28:18 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 3 Jun 2017 15:28:18 +0300 Subject: [Python-Dev] Extremely slow test modules In-Reply-To: <20170603123155.128494e7@fsol> References: <20170603123155.128494e7@fsol> Message-ID: 03.06.17 13:31, Antoine Pitrou ????: > Is there a reason some of our tests are excruciatingly slow in `-uall` > mode? `test_multiprocessing_spawn` is understandable (after all, it > will spawn a new executable for each subprocess), but other tests leave > me baffled: > > - test_tools: 7 min 41 sec > - test_tokenize: 6 min 23 sec > - test_datetime: 6 min 3 sec > - test_lib2to3: 5 min 25 sec > [excerpt from recent Travis CI logs] > > Why does datetime, 2to3 or tokenize testing take so long? And do we > have so many tools that it should take 7 minutes to run all of them? > I must admit, I don't understand how we got to such a point. test_tools (in particular the test for the unparse.py script), test_tokenize, and test_lib2to3 read and proceed every Python file in the stdlib. This is necessary in full test run because some syntax constructs are very rarely used. This is controlled by the cpy resource. I suggested to disable it on the slowest buildbots (-uall,-cpu). In that case tests are ran only for few random files. test_datetime generates tests for all possible timezones. This is controlled by the tzdata resource and also can be disabled on the slowest buildbots. From solipsis at pitrou.net Sat Jun 3 09:01:07 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 3 Jun 2017 15:01:07 +0200 Subject: [Python-Dev] Extremely slow test modules References: <20170603123155.128494e7@fsol> Message-ID: <20170603150107.691ba9a7@fsol> On Sat, 3 Jun 2017 15:28:18 +0300 Serhiy Storchaka wrote: > > test_tools (in particular the test for the unparse.py script), > test_tokenize, and test_lib2to3 read and proceed every Python file in > the stdlib. This is necessary in full test run because some syntax > constructs are very rarely used. There's no need to parse the whole stdlib for that. Just parse a couple files with the required syntax constructs (for example the test suite, which by construction should have all of them). > This is controlled by the cpy resource. > I suggested to disable it on the slowest buildbots (-uall,-cpu). In that > case tests are ran only for few random files. I don't really care about the buildbots, but I care about CI turnaround. A Travis-CI test run takes 24 minutes. Assuming it uses 4 cores and those 4 tests take more than 6 minutes each, that means we could almost shave 6 minutes (25%) on the duration of the Travis-CI test run. Regards Antoine. From solipsis at pitrou.net Sat Jun 3 09:07:48 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 3 Jun 2017 15:07:48 +0200 Subject: [Python-Dev] Extremely slow test modules References: <20170603123155.128494e7@fsol> <20170603150107.691ba9a7@fsol> Message-ID: <20170603150748.1de948e0@fsol> On Sat, 3 Jun 2017 15:01:07 +0200 Antoine Pitrou wrote: > > > This is controlled by the cpy resource. > > I suggested to disable it on the slowest buildbots (-uall,-cpu). In that > > case tests are ran only for few random files. > > I don't really care about the buildbots, but I care about CI > turnaround. A Travis-CI test run takes 24 minutes. Assuming it uses 4 > cores and those 4 tests take more than 6 minutes each, that means we > could almost shave 6 minutes (25%) on the duration of the Travis-CI > test run. And if, as it is likely, Travis-CI only exposes 2 CPU cores, we could actually shave 12 minutes off of each 24 minute CI run... Regards Antoine. From storchaka at gmail.com Sat Jun 3 10:43:23 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 3 Jun 2017 17:43:23 +0300 Subject: [Python-Dev] Extremely slow test modules In-Reply-To: <20170603150107.691ba9a7@fsol> References: <20170603123155.128494e7@fsol> <20170603150107.691ba9a7@fsol> Message-ID: 03.06.17 16:01, Antoine Pitrou ????: > On Sat, 3 Jun 2017 15:28:18 +0300 > Serhiy Storchaka wrote: >> >> test_tools (in particular the test for the unparse.py script), >> test_tokenize, and test_lib2to3 read and proceed every Python file in >> the stdlib. This is necessary in full test run because some syntax >> constructs are very rarely used. > > There's no need to parse the whole stdlib for that. Just parse a > couple files with the required syntax constructs (for example the test > suite, which by construction should have all of them). We don't know what these files are. It may be possible (and even likely), that parsing the whole stdlib is not enough. Ideally the tests should parse the whole word, but this is impossible for some reasons. >> This is controlled by the cpy resource. >> I suggested to disable it on the slowest buildbots (-uall,-cpu). In that >> case tests are ran only for few random files. > > I don't really care about the buildbots, but I care about CI > turnaround. A Travis-CI test run takes 24 minutes. Assuming it uses 4 > cores and those 4 tests take more than 6 minutes each, that means we > could almost shave 6 minutes (25%) on the duration of the Travis-CI > test run. They could be disabled on Travis-CI too. From storchaka at gmail.com Sat Jun 3 12:25:25 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 3 Jun 2017 19:25:25 +0300 Subject: [Python-Dev] PEP 7 and braces { .... } on if In-Reply-To: References: Message-ID: Yet about braces. PEP 7 implicitly forbids breaking the line before an opening brace. An opening brace should stay at the end the line of the outer compound statement. if (mro != NULL) { ... } else { ... } if (type->tp_dictoffset != 0 && base->tp_dictoffset == 0 && type->tp_dictoffset == b_size && (size_t)t_size == b_size + sizeof(PyObject *)) { return 0; /* "Forgive" adding a __dict__ only */ } But the latter example continuation lines are intended at the same level as the following block of code. I propose to make exception for that case and allow moving an open brace to the start of the next line. if (type->tp_dictoffset != 0 && base->tp_dictoffset == 0 && type->tp_dictoffset == b_size && (size_t)t_size == b_size + sizeof(PyObject *)) { return 0; /* "Forgive" adding a __dict__ only */ } This adds a visual separation of a multiline condition from the following code. From barry at python.org Sat Jun 3 16:30:24 2017 From: barry at python.org (Barry Warsaw) Date: Sat, 3 Jun 2017 16:30:24 -0400 Subject: [Python-Dev] PEP 7 and braces { .... } on if In-Reply-To: References: Message-ID: <20170603163024.6688f086@subdivisions.wooz.org> On Jun 03, 2017, at 07:25 PM, Serhiy Storchaka wrote: >But the latter example continuation lines are intended at the same level as >the following block of code. I propose to make exception for that case and >allow moving an open brace to the start of the next line. > > if (type->tp_dictoffset != 0 && base->tp_dictoffset == 0 && > type->tp_dictoffset == b_size && > (size_t)t_size == b_size + sizeof(PyObject *)) > { > return 0; /* "Forgive" adding a __dict__ only */ > } Agreed! https://github.com/python/peps/issues/283 https://github.com/python/peps/pull/284 Cheers, -Barry From tim.peters at gmail.com Sat Jun 3 22:46:09 2017 From: tim.peters at gmail.com (Tim Peters) Date: Sat, 3 Jun 2017 21:46:09 -0500 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: <20170603120706.75fa7ba8@fsol> References: <20170603120706.75fa7ba8@fsol> Message-ID: For fun, let's multiply everything by 256: - A "pool" becomes 1 MB. - An "arena" becomes 64 MB. As briefly suggested before, then for any given size class a pool could pass out hundreds of times more objects before needing to fall back on the slower code creating new pools or new arenas. As an added bonus, programs would finish much sooner due to the flurry of segfaults from Py_ADDRESS_IN_RANGE ;-) But greatly increasing the pool size also makes a different implementation of that much more attractive: an obvious one. That is, obmalloc could model its address space with a bit vector, one bit per pool-aligned address. For a given address, shift it right by 20 bits (divide by 1MB) and use what remains as the bit vector index. If the bit is set, obmalloc manages that MB, else (or if the bit address is out of the vector's domain) it doesn't. The system page size would become irrelevant to its operation, and it would play nice with magical memory debuggers (it would never access memory obmalloc hadn't first allocated and initialized itself). A virtual address space span of a terabyte could hold 1M pools, so would "only" need a 1M/8 = 128KB bit vector. That's minor compared to a terabyte (one bit per megabyte). Of course using a bit per 4KB (the current pool size) is less attractive - by a factor of 256. Which is why that wasn't even tried. Note that trying to play the same trick with arenas instead would be at best more complicated. The system calls can't be relied on to return arena-aligned _or_ pool-aligned addresses. obmalloc itself forces pool-alignment of pool base addresses, by (if necessary) ignoring some number of the leading bytes in an arena. That makes useful arithmetic on pool addresses uniform and simple. From solipsis at pitrou.net Sun Jun 4 04:18:13 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 4 Jun 2017 10:18:13 +0200 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE References: <20170603120706.75fa7ba8@fsol> Message-ID: <20170604101813.0141930a@fsol> On Sat, 3 Jun 2017 21:46:09 -0500 Tim Peters wrote: > > A virtual address space span of a terabyte could hold 1M pools, so > would "only" need a 1M/8 = 128KB bit vector. That's minor compared to > a terabyte (one bit per megabyte). The virtual address space currently supported by x86-64 is 48 bits wide (spanning an address space of 2**48 bytes, that is 256 TB), so you would need a 2**(48-20) bits vector, i.e. 256 Mbits or 32 MB. Note Intel has plans to extend the virtual address space to 2**57 bytes: https://www.phoronix.com/scan.php?page=news_item&px=Intel-5-Level-Paging > The system calls can't be relied on to > return arena-aligned _or_ pool-aligned addresses. Unless using mmap() for pools with a size equal to or an exact divisor of the system's page size, that is ;-) Regards Antoine. From tim.peters at gmail.com Sun Jun 4 10:46:10 2017 From: tim.peters at gmail.com (Tim Peters) Date: Sun, 4 Jun 2017 09:46:10 -0500 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: <20170604101813.0141930a@fsol> References: <20170603120706.75fa7ba8@fsol> <20170604101813.0141930a@fsol> Message-ID: [Tim] >> A virtual address space span of a terabyte could hold 1M pools, so >> would "only" need a 1M/8 = 128KB bit vector. That's minor compared to >> a terabyte (one bit per megabyte). [Antoine] > The virtual address space currently supported by x86-64 is 48 bits > wide (spanning an address space of 2**48 bytes, that is 256 TB), so you > would need a 2**(48-20) bits vector, i.e. 256 Mbits or 32 MB. > > Note Intel has plans to extend the virtual address space to 2**57 bytes: > https://www.phoronix.com/scan.php?page=news_item&px=Intel-5-Level-Paging Fill in the blanks ;-) There's only a need for the bit vector to cover the range of addresses _actually returned_ by the system for arenas allocated so far (we're only trying to identify the memory obmalloc _does_ control) . I didn't spell that out, but it was implicit in glosses like "(or if the bit address is out of the vector's domain)". That is, it's up to the bit vector implementation to intelligently represent what's almost always going to be a relatively tiny slice of a theoretically massive address space. So my observation about a terabyte is to be read as applying to a case in which obmalloc has _actually allocated_ arenas spanning a terabyte of address space.(2**14 arenas of 64MB each, and if the system is kind enough to keep them contiguous). Since that's about 100 times more address space than any Python program I've run actually needed, it's a bare minimum for a thought experiment. From solipsis at pitrou.net Sun Jun 4 13:19:34 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 4 Jun 2017 19:19:34 +0200 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: References: <20170603120706.75fa7ba8@fsol> <20170604101813.0141930a@fsol> Message-ID: <20170604191934.191fc364@fsol> On Sun, 4 Jun 2017 09:46:10 -0500 Tim Peters wrote: > [Tim] > >> A virtual address space span of a terabyte could hold 1M pools, so > >> would "only" need a 1M/8 = 128KB bit vector. That's minor compared to > >> a terabyte (one bit per megabyte). > > [Antoine] > > The virtual address space currently supported by x86-64 is 48 bits > > wide (spanning an address space of 2**48 bytes, that is 256 TB), so you > > would need a 2**(48-20) bits vector, i.e. 256 Mbits or 32 MB. > > > > Note Intel has plans to extend the virtual address space to 2**57 bytes: > > https://www.phoronix.com/scan.php?page=news_item&px=Intel-5-Level-Paging > > Fill in the blanks ;-) There's only a need for the bit vector to > cover the range of addresses _actually returned_ by the system for > arenas allocated so far (we're only trying to identify the memory > obmalloc _does_ control) . I didn't spell that out, but it was > implicit in glosses like "(or if the bit address is out of the > vector's domain)". That is, it's up to the bit vector implementation > to intelligently represent what's almost always going to be a > relatively tiny slice of a theoretically massive address space. True. That works if the operating system doesn't go too wild in address space randomization, though ;-) Regards Antoine. From tim.peters at gmail.com Sun Jun 4 14:50:09 2017 From: tim.peters at gmail.com (Tim Peters) Date: Sun, 4 Jun 2017 13:50:09 -0500 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: <20170604191934.191fc364@fsol> References: <20170603120706.75fa7ba8@fsol> <20170604101813.0141930a@fsol> <20170604191934.191fc364@fsol> Message-ID: [Tim] >> ... That is, it's up to the bit vector implementation >> to intelligently represent what's almost always going to be a >> relatively tiny slice of a theoretically massive address space. [Antoine] > True. That works if the operating system doesn't go too wild in > address space randomization, though ;-) I don't know that it's a real problem, but if it is I'm sure you know of efficient ways to implement sparse sets (for example, picture Python's set implementation, but slimmed down to handle just C uint64 members). I was hoping to spur a discussion of much higher level issues. I bet Larry was too ;-) From tim.peters at gmail.com Sun Jun 4 16:18:20 2017 From: tim.peters at gmail.com (Tim Peters) Date: Sun, 4 Jun 2017 15:18:20 -0500 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: References: Message-ID: [Larry Hastings ] > ... > Yet CPython's memory consumption continues to grow. By the time a current > "trunk" build of CPython reaches the REPL prompt it's already allocated 16 > arenas. I'd be surprised if that's true ;-) The first time `new_arena()` is called, it allocates space for a vector of 16 (INITIAL_ARENA_OBJECTS) `arena_object` structs. Those are tiny, and hold bookkeeping info for the actual arenas, none of which are allocated at first. `new_arena()` asks the system for memory for just one arena (and 15 of its 16 initial arena_object structs remain unused, awaiting future calls). At the time, 16 wasn't picked because it was the minimum Python needed, but because it was the _most_ the bulk of a variety of casual Python scripts needed to complete. It's certainly gotten worse on a 64-bit box. Here on a 64-bit Win10 under Python 3.6.1 upon reaching the interactive prompt in a DOS box (set envar PYTHONMALLOCSTATS to get this kind of output whenever `new_arena()` is called): # arenas allocated total = 14 # arenas reclaimed = 5 # arenas highwater mark = 9 # arenas allocated current = 9 9 arenas * 262144 bytes/arena = 2,359,296 # bytes in allocated blocks = 2,166,656 # bytes in available blocks = 140,104 0 unused pools * 4096 bytes = 0 # bytes lost to pool headers = 27,648 # bytes lost to quantization = 24,888 # bytes lost to arena alignment = 0 Total = 2,359,296 So at most 9 arenas ("highwater mark") were ever simultaneously allocated.. Change the arena size to 64MB, and most programs would stop with the first arena allocated ;-) From storchaka at gmail.com Mon Jun 5 01:41:22 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 5 Jun 2017 08:41:22 +0300 Subject: [Python-Dev] PEP 7 and braces { .... } on if In-Reply-To: <20170603163024.6688f086@subdivisions.wooz.org> References: <20170603163024.6688f086@subdivisions.wooz.org> Message-ID: 03.06.17 23:30, Barry Warsaw ????: > On Jun 03, 2017, at 07:25 PM, Serhiy Storchaka wrote: > >> But the latter example continuation lines are intended at the same level as >> the following block of code. I propose to make exception for that case and >> allow moving an open brace to the start of the next line. >> >> if (type->tp_dictoffset != 0 && base->tp_dictoffset == 0 && >> type->tp_dictoffset == b_size && >> (size_t)t_size == b_size + sizeof(PyObject *)) >> { >> return 0; /* "Forgive" adding a __dict__ only */ >> } > > Agreed! > > https://github.com/python/peps/issues/283 > https://github.com/python/peps/pull/284 Thank you for opening a PR Barry! But there is some disputation. Barry and Victor prefer moving a brace on a new line in all multiline conditional cases. I think that it should be done only when the condition continuation lines and the following block of the code have the same indentation (as in the example above), and the following code is enough readable: if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1, "invalid escape sequence '\\%c'", *first_invalid_escape) < 0) { Py_DECREF(result); return NULL; } What other core developers think about this? From stefan_ml at behnel.de Mon Jun 5 03:14:12 2017 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 5 Jun 2017 09:14:12 +0200 Subject: [Python-Dev] PEP 7 and braces { .... } on if In-Reply-To: References: Message-ID: Serhiy Storchaka schrieb am 03.06.2017 um 18:25: > Yet about braces. PEP 7 implicitly forbids breaking the line before an > opening brace. An opening brace should stay at the end the line of the > outer compound statement. > > if (mro != NULL) { > ... > } > else { > ... > } > > if (type->tp_dictoffset != 0 && base->tp_dictoffset == 0 && > type->tp_dictoffset == b_size && > (size_t)t_size == b_size + sizeof(PyObject *)) { > return 0; /* "Forgive" adding a __dict__ only */ > } > > But the latter example continuation lines are intended at the same level as > the following block of code. I propose to make exception for that case and > allow moving an open brace to the start of the next line. > > if (type->tp_dictoffset != 0 && base->tp_dictoffset == 0 && > type->tp_dictoffset == b_size && > (size_t)t_size == b_size + sizeof(PyObject *)) > { > return 0; /* "Forgive" adding a __dict__ only */ > } > > This adds a visual separation of a multiline condition from the following > code. Python itself has a similar problem and solves it differently. Why not take a look at PEP-8 here? """ Yes: # Aligned with opening delimiter. foo = long_function_name(var_one, var_two, var_three, var_four) # More indentation included to distinguish this from the rest. def long_function_name( var_one, var_two, var_three, var_four): print(var_one) # Hanging indents should add a level. foo = long_function_name( var_one, var_two, var_three, var_four) No: # Arguments on first line forbidden when not using vertical alignment. foo = long_function_name(var_one, var_two, var_three, var_four) # Further indentation required as indentation is not distinguishable. def long_function_name( var_one, var_two, var_three, var_four): print(var_one) """ ISTM that overindenting the conditions (i.e. following the "more indentation" example) solves this problem and makes it very readable. The same can be done in C code and avoids having to remember two different special ways to do it for core devs in C and Python. Stefan From cory at lukasa.co.uk Mon Jun 5 04:35:32 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Mon, 5 Jun 2017 09:35:32 +0100 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> Message-ID: > On 2 Jun 2017, at 17:39, Nick Coghlan wrote: > > On 3 June 2017 at 02:22, Donald Stufft wrote: >> It?s not just bootstrapping that pip has a problem with for C extensions, it >> also prevents upgrading PyOpenSSL on Windows because having pip import >> PyOpenSSL locks the .dll, and we can?t delete it or overwrite it until the >> pip process exits and no longer imports PyOpenSSL. This isn?t a problem on >> Linux or macOS or the other *nix clients though. We patch requests as it is >> today to prevent it from importing simplejson and cryptography for this >> reason. > > Would requests be loading PyOpenSSL on Windows, though? If the aim is > to facilitate PEP 543, then I'd expect it to be using the SChannel > backend in that case. Only assuming the SChannel backend is available. This would also likely be a third-party Python library so you can just search and replace ?PyOpenSSL? above with ?SChannel? and hit the exact same problem. Cory From cory at lukasa.co.uk Mon Jun 5 04:37:02 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Mon, 5 Jun 2017 09:37:02 +0100 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <5E4A70A9-D5C4-4081-9746-0CD8696642ED@stufft.io> References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> <20170602184104.6b3ed724@fsol> <5E4A70A9-D5C4-4081-9746-0CD8696642ED@stufft.io> Message-ID: > On 2 Jun 2017, at 17:59, Donald Stufft wrote: > > I suspect (though I?d let him speak for himself) that Cory would rather continue to be sync only than require pip to go back to not using requests. We are not wedded to supporting pip, but I think the interaction between the two tools is positive. I think pip gains a lot from depending on us, and we get a lot of value from having pip as a dependency. So the cost of supporting pip would have to be pretty darn high for us to want to stop doing that, and so on this issue I think Donald is right. Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Mon Jun 5 04:42:41 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Mon, 5 Jun 2017 09:42:41 +0100 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> <20170602122117.5f86e422@subdivisions.wooz.org> Message-ID: <84A2C0B0-8B3C-400A-96D2-42C368429B3B@lukasa.co.uk> > On 3 Jun 2017, at 07:25, Nick Coghlan wrote: > > As a result, as awkward and annoying as it would be, I'm wondering if > the structure of the requests migration may need to be something like: > > - switch unconditionally to an async backend on Py3 > - switch conditionally to an async backend on Py2 when PyOpenSSL is available > - retain sufficient sync-only support on Py2 to allow pip to disable > the PyOpenSSL dependency Ultimately we don?t have the resources to do this. The requests core development team is 4 people, 3 of whom are part time best-effort maintainers and one of whom is me, who has foolishly decided to also work on PEP 543 as well as take over the lead maintenance role on urllib3. urllib3 has another two additional best-effort maintainers. We simply do not have the development resources to support two parallel branches of code. Even if we did, doing so is a recipe for adding bugs and inconsistencies between the two, leading to increased workload as we break our users and fail to keep the two branches in sync. It also makes it much harder for users to migrate from our synchronous backend to our async one, as those two no longer use identical underlying implementations and so will have subtle inconsistencies. The TL;DR there is: no, we?d rather stay sync only than do that. Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Jun 5 07:00:25 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 5 Jun 2017 21:00:25 +1000 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <84A2C0B0-8B3C-400A-96D2-42C368429B3B@lukasa.co.uk> References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> <20170602122117.5f86e422@subdivisions.wooz.org> <84A2C0B0-8B3C-400A-96D2-42C368429B3B@lukasa.co.uk> Message-ID: On 5 June 2017 at 18:42, Cory Benfield wrote: > On 3 Jun 2017, at 07:25, Nick Coghlan wrote: > As a result, as awkward and annoying as it would be, I'm wondering if > the structure of the requests migration may need to be something like: > > - switch unconditionally to an async backend on Py3 > - switch conditionally to an async backend on Py2 when PyOpenSSL is > available > - retain sufficient sync-only support on Py2 to allow pip to disable > the PyOpenSSL dependency > > Ultimately we don?t have the resources to do this. > > The requests core development team is 4 people, 3 of whom are part time > best-effort maintainers and one of whom is me, who has foolishly decided to > also work on PEP 543 as well as take over the lead maintenance role on > urllib3. urllib3 has another two additional best-effort maintainers. > > We simply do not have the development resources to support two parallel > branches of code. Even if we did, doing so is a recipe for adding bugs and > inconsistencies between the two, leading to increased workload as we break > our users and fail to keep the two branches in sync. It also makes it much > harder for users to migrate from our synchronous backend to our async one, > as those two no longer use identical underlying implementations and so will > have subtle inconsistencies. > > The TL;DR there is: no, we?d rather stay sync only than do that. Would you be OK with the notion of a "just for pip bootstrapping" private backend in _ensurepip_ssl? That is, the only officially supported async-backed requests configuration on Py2 would be with the PyOpenSSL dependency installed, but in collaboration with the pip devs we'd also plumb in the pieces to let a new async-backed requests work without any extension modules other than those in the standard library. That way, the only thing truly gated on the backport would be *pip* updating its bundled version of requests to the async-backed version - for normal third party use, the change would be "you need PyOpenSSL", rather than "you need a newer version of Python". We'd still effectively end up with two different code execution paths (one backed by PyOpenSSL, one backed by the new private _ensurepip_ssl extension module), but the level of divergence would be much lower (it's just a question of where MemoryBIO and SSLObject are coming from) and the support scope for the less frequently used path would be much narrower (i.e. if a problem report isn't related to pip bootstrapping, it can be closed as "won't fix") Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From cory at lukasa.co.uk Mon Jun 5 07:36:08 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Mon, 5 Jun 2017 12:36:08 +0100 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> <20170602122117.5f86e422@subdivisions.wooz.org> <84A2C0B0-8B3C-400A-96D2-42C368429B3B@lukasa.co.uk> Message-ID: <1304264E-2F99-4A1C-BD52-5E4AFE9F72E4@lukasa.co.uk> > On 5 Jun 2017, at 12:00, Nick Coghlan wrote: > > Would you be OK with the notion of a "just for pip bootstrapping" > private backend in _ensurepip_ssl? > > That is, the only officially supported async-backed requests > configuration on Py2 would be with the PyOpenSSL dependency installed, > but in collaboration with the pip devs we'd also plumb in the pieces > to let a new async-backed requests work without any extension modules > other than those in the standard library. > > That way, the only thing truly gated on the backport would be *pip* > updating its bundled version of requests to the async-backed version - > for normal third party use, the change would be "you need PyOpenSSL", > rather than "you need a newer version of Python". > > We'd still effectively end up with two different code execution paths > (one backed by PyOpenSSL, one backed by the new private _ensurepip_ssl > extension module), but the level of divergence would be much lower > (it's just a question of where MemoryBIO and SSLObject are coming > from) and the support scope for the less frequently used path would be > much narrower (i.e. if a problem report isn't related to pip > bootstrapping, it can be closed as "won't fix? It?s not clear to me what the structure of that looks like, or what work is required to achieve it. Right now Twisted never uses MemoryBIO and SSLObject: it always uses PyOpenSSL. That seems like we?d need to add MemoryBIO and SSLObject support to Twisted which can be enabled in some way other than feature detection (that is, so it can be installed). Without PEP 543 this is pretty gross. With PEP 543 it sucks less, but it also gates this work behind PEP 543 being successfully implemented and landed. I guess we are *open* to that approach? It?s not clear to me how beneficial that is, and it doesn?t gain any of the ecosystem benefits (no-one else on Py 2 can ever use this chunk of tested, understood code), but it?s certainly an option. The indirection gives me pause though. Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Jun 5 07:44:17 2017 From: donald at stufft.io (Donald Stufft) Date: Mon, 5 Jun 2017 07:44:17 -0400 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> <20170602122117.5f86e422@subdivisions.wooz.org> <84A2C0B0-8B3C-400A-96D2-42C368429B3B@lukasa.co.uk> Message-ID: > On Jun 5, 2017, at 7:00 AM, Nick Coghlan wrote: > > On 5 June 2017 at 18:42, Cory Benfield > wrote: >> On 3 Jun 2017, at 07:25, Nick Coghlan wrote: >> As a result, as awkward and annoying as it would be, I'm wondering if >> the structure of the requests migration may need to be something like: >> >> - switch unconditionally to an async backend on Py3 >> - switch conditionally to an async backend on Py2 when PyOpenSSL is >> available >> - retain sufficient sync-only support on Py2 to allow pip to disable >> the PyOpenSSL dependency >> >> Ultimately we don?t have the resources to do this. >> >> The requests core development team is 4 people, 3 of whom are part time >> best-effort maintainers and one of whom is me, who has foolishly decided to >> also work on PEP 543 as well as take over the lead maintenance role on >> urllib3. urllib3 has another two additional best-effort maintainers. >> >> We simply do not have the development resources to support two parallel >> branches of code. Even if we did, doing so is a recipe for adding bugs and >> inconsistencies between the two, leading to increased workload as we break >> our users and fail to keep the two branches in sync. It also makes it much >> harder for users to migrate from our synchronous backend to our async one, >> as those two no longer use identical underlying implementations and so will >> have subtle inconsistencies. >> >> The TL;DR there is: no, we?d rather stay sync only than do that. > > Would you be OK with the notion of a "just for pip bootstrapping" > private backend in _ensurepip_ssl? > Is pip allowed to use the hypothetical _ensurepip_ssl outside of ensurepip? Thinking about this reminded me about the *other* reason pip avoids dependencies? avoiding making assertions about what versions of software can actually be installed. IOW if pip depends on pyOpenSSL >= X.Y, then you essentially can?t install any other version of pyOpenSSL and you?d be forced to upgrade. This isn?t end of the world and pyOpenSSL is generally stable enough we could *probably* get away with a unversioned dependency on a non Windows platform. On Windows we?d have to uh, I guess use a SChannel c-types backend? Not having our TLS library in the stdlib makes it a bit more difficult, but from pip?s own POV if there?s a MemoryBio that we?re allowed to use and then requests uses pyOpenSSL normally (we?d apply a small patch to pip?s copy of requests to make it use it, but that?s easy to do with our setup) I think that would be workable. I think that?s a less optimal solution than just accepting PEP 546 which I think is beneficial to the Python ecosystem as a whole, making it easier to port more code onto 3.x (and allowing unlocking 3.x?s capabilities) and will make it easier to maintain the ssl library on 2.7, given it will have less of a divergence from 3.x again. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip.montanaro at gmail.com Mon Jun 5 08:00:18 2017 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Mon, 5 Jun 2017 07:00:18 -0500 Subject: [Python-Dev] PEP 7 and braces { .... } on if In-Reply-To: References: <20170603163024.6688f086@subdivisions.wooz.org> Message-ID: On Mon, Jun 5, 2017 at 12:41 AM, Serhiy Storchaka wrote: > Barry and Victor prefer moving a brace on a new line in all multiline > conditional cases. I think that it should be done only when the condition > continuation lines and the following block of the code have the same > indentation (as in the example above), and the following code is enough > readable: > > if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1, > "invalid escape sequence '\\%c'", > *first_invalid_escape) < 0) { > Py_DECREF(result); > return NULL; > } > > What other core developers think about this? Wow, this discussion takes me back. Glad I don't have to check out comp.lang.c to get my brace placement fix. Skip From ncoghlan at gmail.com Mon Jun 5 09:59:18 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 5 Jun 2017 23:59:18 +1000 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> <20170602122117.5f86e422@subdivisions.wooz.org> <84A2C0B0-8B3C-400A-96D2-42C368429B3B@lukasa.co.uk> Message-ID: On 5 June 2017 at 21:44, Donald Stufft wrote: > Is pip allowed to use the hypothetical _ensurepip_ssl outside of ensurepip? Yes, so something like _tls_bootstrap would probably be a better name for the helper module (assuming we go down the "private API to bootstrap 3rd party SSL modules" path). The leading underscore would be akin to the one on _thread - it isn't that the API would be undocumented or inaccessible, it's that it wouldn't have any backwards compatibility guarantees for general use, so you shouldn't use it without a really good reason. > Thinking about this reminded me about the *other* reason pip avoids > dependencies? avoiding making assertions about what versions of software can > actually be installed. IOW if pip depends on pyOpenSSL >= X.Y, then you > essentially can?t install any other version of pyOpenSSL and you?d be forced > to upgrade. Indeed, and I think that's sufficient justification for at least adding the private TLS bootstrapping API. > This isn?t end of the world and pyOpenSSL is generally stable enough we > could *probably* get away with a unversioned dependency on a non Windows > platform. On Windows we?d have to uh, I guess use a SChannel c-types > backend? Not having our TLS library in the stdlib makes it a bit more > difficult, but from pip?s own POV if there?s a MemoryBio that we?re allowed > to use and then requests uses pyOpenSSL normally (we?d apply a small patch > to pip?s copy of requests to make it use it, but that?s easy to do with our > setup) I think that would be workable. > > I think that?s a less optimal solution than just accepting PEP 546 which I > think is beneficial to the Python ecosystem as a whole, making it easier to > port more code onto 3.x (and allowing unlocking 3.x?s capabilities) and will > make it easier to maintain the ssl library on 2.7, given it will have less > of a divergence from 3.x again. Right, I'm just trying to make sure we clearly separate the two arguments: pip bootstrapping (which can potentially be addressed through a private alternate SSL/TLS API) and ecosystem advancement (making it easier for SSL/TLS capable applications to retain Python 2.7.x compatibility at least until 2020, while still supporting the improvements in asynchronous IO capabilities in the 3.x series). Separating the two main options like that gives us two future scenarios: 1. Private TLS bootstrapping API - requests & any other libraries that want this functionality would need a dependency on PyOpenSSL for Python versions prior to 3.5 - the base SSL/TLS support in Python 2.7 remains focused on synchronous IO, with only limited support for sans-io style protocol library implementations - some time in 2017 or 2018, pip starts requiring that environments provide either an externally managed _tls_bootstrap module, *or* a sufficiently recent PyOpenSSL - the first CPython maintenance release to include that pip update would also provide the required _tls_bootstrap module by default - redistributors will be able to backport _tls_bootstrap as an independent module (e.g. as part of their system pip packages) rather than necessarily including it directly in their Python runtime - this distribution model should be resilient over time, allowing _tls_bootstrap to be rebased freely without risking breaking end user applications that weren't expecting it (this approach would likely even be useful in LTS 3.x environments as they age - e.g. Ubuntu 14.04 and 16.04) - the private _tls_bootstrap API would either be a straight backport of the 3.6 ssl module, or else a shim around the corresponding standard library's ssl module to add the required features, whichever was judged easiest to implement and maintain 2. Public standard library ssl module API update - libraries needing the functionality *either* depend on PyOpenSSL *or* set their minimum Python version to 2.7.14+ - if libraries add a Python version check to their installation metadata, LTS distributions are out of luck, since we typically don't rebase Python (e.g. the RHEL 7 system Python still advertises itself as 2.7.5, even though assorted backports have been applied on top of that) - even if libraries rely on API feature detection rather than explicit version checks, redistributors still need to actually patch the standard library - we can't just add a guaranteed-side-effect-free module as a separate post-installation step - while some system administrators may be willing and able to deploy their own variant a _tls_boostrap module in advance of their redistributors providing one, relatively few are going to be willing to apply custom patches to Python before deploying it So honestly, I don't think the current "ecosystem advancement" argument in PEP 546 actually works, as the enhancement is *significantly* easier to roll out to older 2.7.x releases if the adoption model is "Supporting SSL/TLS with sans-io style protocol implementations on Python versions prior to Python 3.5 requires the use of PyOpenSSL or a suitable PEP 543 compatible library (unless you're a package management tool, in which case you should check for a separate platform provided Python version independent _tls_bootstrap module before falling back to PyOpenSSL)". As long as we actually document the _tls_bootstrap API (even if that's just a cross-reference to the Python 3.6 ssl module documentation), then folks wanting to share asynchronous IO code between Python 2 & 3 will have the following options: - use wrap_socket() (as Tornado does today) - depend on PyOpenSSL (as Twisted does today) - use _tls_bootstrap if available, fall back to PyOpenSSL otherwise <-- new piece added by PEP 546 due to the problems around pip attempting to upgrade its own dependencies Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ethan at stoneleaf.us Mon Jun 5 11:19:49 2017 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 05 Jun 2017 08:19:49 -0700 Subject: [Python-Dev] PEP 7 and braces { .... } on if In-Reply-To: References: <20170603163024.6688f086@subdivisions.wooz.org> Message-ID: <59357695.8060409@stoneleaf.us> > if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1, > "invalid escape sequence '\\%c'", > *first_invalid_escape) < 0) { > Py_DECREF(result); > return NULL; > } > > What other core developers think about this? I would format that as: if (PyErr_WarnFormat( PyExc_DeprecationWarning, 1, "invalid escape sequence '\\%c'", *first_invalid_escape) < 0) { Py_DECREF(result); return NULL; } Because: - having all the arguments on separate lines means - the function and first argument don't get run together - it's easy to pick out the individual arguments - having the opening brace on its own line means - a little extra white space to buffer the condition and the body - it's easier to read the function name and then drop down to the body All in all, it becomes easier for (at least my) suboptimal eyes to read the code. -- ~Ethan~ From python at mrabarnett.plus.com Mon Jun 5 12:54:23 2017 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 5 Jun 2017 17:54:23 +0100 Subject: [Python-Dev] PEP 7 and braces { .... } on if In-Reply-To: References: <20170603163024.6688f086@subdivisions.wooz.org> Message-ID: <56824bac-1938-2abb-ae17-b09271264e59@mrabarnett.plus.com> On 2017-06-05 13:00, Skip Montanaro wrote: > On Mon, Jun 5, 2017 at 12:41 AM, Serhiy Storchaka wrote: >> Barry and Victor prefer moving a brace on a new line in all multiline >> conditional cases. I think that it should be done only when the condition >> continuation lines and the following block of the code have the same >> indentation (as in the example above), and the following code is enough >> readable: >> >> if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1, >> "invalid escape sequence '\\%c'", >> *first_invalid_escape) < 0) { >> Py_DECREF(result); >> return NULL; >> } >> >> What other core developers think about this? > > Wow, this discussion takes me back. Glad I don't have to check out > comp.lang.c to get my brace placement fix. > FWIW, I half-indent continuation lines (an advantage of using spaces instead of tabs). From greg at krypto.org Mon Jun 5 16:53:57 2017 From: greg at krypto.org (Gregory P. Smith) Date: Mon, 05 Jun 2017 20:53:57 +0000 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: References: Message-ID: On Fri, Jun 2, 2017 at 12:33 PM Larry Hastings wrote: > > On 06/02/2017 02:46 AM, Victor Stinner wrote: > > I would be curious of another test: use pymalloc for objects larger > than 512 bytes. For example, allocate up to 4 KB? > > In the past, we already changed the maximum size from 256 to 512 to > support most common Python objects on 64-bit platforms. Since Python > objects contain many pointers: switching from 32 bit to 64 bit can > double the size of the object in the worst case. > > > You've already seen Tim Peters' post about why we must leave pool size set > to 4k. Obviously This in turn means using obmalloc for larger objects will > mean more and more wasted memory. > > For example, let's say we use obmalloc for allocations of 2048 bytes. > Pool size is 4096 bytes, and there's a 48-byte "pool_header" structure on > the front (on 64-bit platforms, if I counted right). So there are only > 4048 bytes usable per pool. After the first 2048 allocation, we're left > with 2000 bytes at the end. You can't use that memory for another > allocation class, that's impossible given obmalloc's design. So that 2000 > bytes is just wasted. > > Currently obmalloc's maximum allocation size is 512 bytes; after 7 > allocations, this leaves 464 wasted bytes at the end. Which isn't *great* > exactly but it's only 11% of the overall allocated memory. > > Anyway, I'm not super excited by the prospect of using obmalloc for larger > objects. There's an inverse relation between the size of allocation and > the frequency of allocation. In Python there are lots of tiny allocations, > but fewer and fewer as the size increases. (A similarly-shaped graph to > what retailers call the "long tail".) By no small coincidence, obmalloc is > great at small objects, which is where we needed the help most. Let's > leave it at that. > > > A more fruitful endeavor might be to try one of these fancy new > third-party allocators in CPython, e.g. tcmalloc, jemalloc. Try each with > both obmalloc turned on and turned off, and see what happens to performance > and memory usage. (I'd try it myself, but I'm already so far behind on > watching funny cat videos.) > FYI - in CPython using a different malloc instead of CPython's own obmalloc is effectively a simple addition of ~three lines of code to something Py_InitializeEx()-ish thanks to tracemalloc: PyMemAllocator pma; ... PyMem_GetAllocator(PYMEM_DOMAIN_RAW, &pma); PyMem_SetAllocator(PYMEM_DOMAIN_OBJ, &pma); That sets the object allocator (normally obmalloc) to be the "system" malloc which I assume you are having your alternate-malloc (tcmalloc, jemalloc?, etc..) override. As to where exactly I'd have to walk through the code... I see that it was just refactored beyond what I used to recognize as part of cleaning up interpreter startup. (yay!) :) For CPython at large, I don't want us to be in the business of shipping a malloc implementation (any more than we already do). But it does seem worth making what we've got tune-able without having to recompile to do it. I liked the environment variable arena size setting idea. But there are likely more things that could be offered up for tuning by people deploying applications who have tested things to determine what is best for their specific needs. A note about OS/HW page sizes: While hugepages and transparent hugepages *can* be a large performance increase due to less TLB cache misses, they don't fit every application. They can be painful for anyone who depends on a fork()'ed workers application model as a copy on write for a 2mb or 1gb page due to a single refcount update is... costly. -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Jun 5 17:59:29 2017 From: guido at python.org (Guido van Rossum) Date: Mon, 5 Jun 2017 14:59:29 -0700 Subject: [Python-Dev] PEP 544: Protocols - second round In-Reply-To: References: <8b064822-14bb-6800-810a-c14dc97f565a@hotpy.org> Message-ID: On Fri, Jun 2, 2017 at 3:10 PM, Ivan Levkivskyi wrote: > On 1 June 2017 at 00:10, Guido van Rossum wrote: > >> >> On Wed, May 31, 2017 at 2:16 AM, Ivan Levkivskyi >> wrote: >> >>> On 31 May 2017 at 00:58, Guido van Rossum wrote: >>> [...] >>> >>> Thank you for very detailed answers! I have practically nothing to add. >>> It seems to me that most of the Kevin's questions stem from unnecessary >>> focus >>> on runtime type checking. Here are two ideas about how to fix this: >>> >>> * Add the word "static" somewhere in the PEP title. >>> >> >> So the title could become "Protocols: Static structural subtyping (duck >> typing)" -- long, but not record-setting. >> > > I am thinking about "Protocols: Structural subtyping (static duck > typing)". The reason is that subtyping is already a mostly static concept > (in contrast to subclassing), > while duck typing is typically associated with the runtime behaviour. > > This might seem minor, but this version of the title sounds much more > naturally to me. > +1 -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Mon Jun 5 18:55:40 2017 From: larry at hastings.org (Larry Hastings) Date: Mon, 5 Jun 2017 15:55:40 -0700 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: References: <20170603120706.75fa7ba8@fsol> <20170604101813.0141930a@fsol> <20170604191934.191fc364@fsol> Message-ID: <6eb84ff4-9561-80ef-a11a-ebdf017f0b73@hastings.org> On 06/04/2017 01:18 PM, Tim Peters wrote: > [Larry Hastings ] >> ... >> Yet CPython's memory consumption continues to grow. By the time a current >> "trunk" build of CPython reaches the REPL prompt it's already allocated 16 >> arenas. > I'd be surprised if that's true ;-) The first time `new_arena()` is > called, it allocates space for a vector of 16 (INITIAL_ARENA_OBJECTS) > `arena_object` structs. Those are tiny, and hold bookkeeping info for > the actual arenas, none of which are allocated at first. Oh! I thought it also allocated the arenas themselves, in a loop. I thought I saw that somewhere. Happy to be proved wrong... > So at most 9 arenas ("highwater mark") were ever simultaneously allocated.. ... though not completely off-base. On 06/04/2017 11:50 AM, Tim Peters wrote: > I was hoping to spur a discussion of much higher level issues. I bet > Larry was too ;-) Actually I was hoping everyone would just tell me how right I was and thank me for my profound insights. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon Jun 5 20:59:43 2017 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 5 Jun 2017 17:59:43 -0700 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> <20170602122117.5f86e422@subdivisions.wooz.org> <84A2C0B0-8B3C-400A-96D2-42C368429B3B@lukasa.co.uk> Message-ID: On Jun 5, 2017 7:01 AM, "Nick Coghlan" wrote: On 5 June 2017 at 21:44, Donald Stufft wrote: > Is pip allowed to use the hypothetical _ensurepip_ssl outside of ensurepip? Yes, so something like _tls_bootstrap would probably be a better name for the helper module (assuming we go down the "private API to bootstrap 3rd party SSL modules" path). It seems like there's a risk here that we end up with two divergent copies of ssl.py and _ssl.c inside the python 2 tree, and that this will make it harder to do future bug fixing and backports. Is this going to cause problems? Is there any way to mitigate them? -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Mon Jun 5 23:10:49 2017 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 5 Jun 2017 22:10:49 -0500 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: <6eb84ff4-9561-80ef-a11a-ebdf017f0b73@hastings.org> References: <20170603120706.75fa7ba8@fsol> <20170604101813.0141930a@fsol> <20170604191934.191fc364@fsol> <6eb84ff4-9561-80ef-a11a-ebdf017f0b73@hastings.org> Message-ID: [Larry] > ... > Oh! I thought it also allocated the arenas themselves, in a loop. I > thought I saw that somewhere. Happy to be proved wrong... There is a loop in `new_arena()`, but it doesn't do what a casual glance may assume it's doing ;-) It's actually looping over the newly-allocated teensy arena descriptor structs, linking them in to a freelist and recording that they're _not_ (yet) associated with any address space. [Tim] >> So at most 9 arenas ("highwater mark") were ever simultaneously allocated [by the >> time the REPL prompt appeared in a 64-bit 3.6.1].. > ... though not completely off-base. Yes, 9 is in the ballpark of 16. >> I was hoping to spur a discussion of much higher level issues. I bet >> Larry was too ;-) > Actually I was hoping everyone would just tell me how right I was and thank > me for my profound insights. Thank you! It should be thought about again. I think _some_ increase of arena size should be a no-brainer, but I don't expect it to help a lot. For reasons already partly explained, I expect we'd get much better bang for the buck by increasing the pool size: - Roughly speaking, we bash into a slows-the-code pool boundary 64 times as frequently as an arena boundary. If the arena size increased, that ratio only gets worse. - On 64-bit boxes the bytes lost to pool headers increased, but the pool size did not. Thus we're guaranteed to "waste" a higher _percentage_ of allocated bytes than we did on a 32-bit box. - The small object threshold also doubled. Generally (not always), the bytes lost to quantization increase the larger the size class. For example, for the largest size class now, on a 64-bit box we can only fit 7 512-byte objects into the 4096 - 48 = 4048 bytes that remain in a pool. So, in addition to the 48 pool header bytes, we _also_ lose the 464 leftover bytes. So we've lost 512 of the 4096 bytes: 12.5%. That's certainly not typical, but in any case quantization losses as percentage of total bytes decrease the larger the pool. Alas, I haven't thought of a plausibly practical way to replace Py_ADDRESS_IN_RANGE unless the pool size increases a whole frickin' lot :-( From ncoghlan at gmail.com Mon Jun 5 23:49:23 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 6 Jun 2017 13:49:23 +1000 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> <20170602122117.5f86e422@subdivisions.wooz.org> <84A2C0B0-8B3C-400A-96D2-42C368429B3B@lukasa.co.uk> Message-ID: On 6 June 2017 at 10:59, Nathaniel Smith wrote: > On Jun 5, 2017 7:01 AM, "Nick Coghlan" wrote: > > On 5 June 2017 at 21:44, Donald Stufft wrote: >> Is pip allowed to use the hypothetical _ensurepip_ssl outside of >> ensurepip? > > Yes, so something like _tls_bootstrap would probably be a better name > for the helper module (assuming we go down the "private API to > bootstrap 3rd party SSL modules" path). > > > It seems like there's a risk here that we end up with two divergent copies > of ssl.py and _ssl.c inside the python 2 tree, and that this will make it > harder to do future bug fixing and backports. Is this going to cause > problems? Is there any way to mitigate them? Aye, I spent some time thinking about potentially viable implementation architectures, and realised that any "private API" style solution pretty much *has* to be structured as a monkeypatching API to be effective. Otherwise there is too much risk of divergence in things like exception definitions that end up causing cryptic "Why is my SSL exception handler not catching my SSL connection error?" type bugs. The gist of the approach would be that for libraries and non-bootstrapped applications, their SSL/TLS feature detection code would ultimately look something like this: try: # Don't do anything special if we don't need to from ssl import MemoryBIO, SSLObject expect ImportError: # Otherwise fall back to using PyOpenSSL try: from OpenSSL.SSL import Connection except ImportError: raise ImportError("Failed to import asynchronous SSL/TLS support:
") It's currently more complex than that in practice (since PyOpenSSL doesn't natively offer an emulation of the affected parts of the standard library's ssl module API), but that's what it would effectively boil down to at the lowest level of any compatibility wrapper. Conveniently, this is *already* what libraries and non-bootstrapped modules have to do if they're looking to support older Python versions without requiring PyOpenSSL on newer releases, so there wouldn't actually be any changes on that front. By contrast, for bootstrapped applications (including pip) the oversimplified compatibility import summary would instead look more like this: try: # Don't do anything special if we don't need to from ssl import MemoryBIO, SSLObject expect ImportError: # See if we can do a runtime in-place upgrade of the standard library try: import _tls_bootstrap except ImportError: # Otherwise fall back to using PyOpenSSL try: from OpenSSL.SSL import Connection except ImportError: raise ImportError("Failed to bootstrap asynchronous SSL/TLS support:
") else: _tls_bootstrap.monkeypatch_ssl() The reason this kind of approach is really attractive to redistributors from a customer risk management perspective is that like gevent's monkeypatching of synchronous networking APIs, it's *opt-in at runtime*, so the risk of our accidentally inflicting it on a customer that doesn't want it and doesn't need it is almost exactly zero - if none of their own code includes the "import _tls_bootstrap; _tls_bootstrap.monkeypatch_ssl()" invocation and none of their dependencies start enabling it as an implicit side effect of some other operation, they'll never even know the enhancement is there. Instead, the compatibility risks get concentrated in the applications relying on the bootstrapping API, since the monkeypatching process is a potentially new source of bugs that don't exist in the more conventional execution models. The reason that's still preferable though, is that it means the monkeypatching demonstrably doesn't have to be 100% perfect: it just has to be close enough to the way the Python 3 standard library API works to meet the needs of the applications that actually use it. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From njs at pobox.com Tue Jun 6 06:08:47 2017 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 6 Jun 2017 03:08:47 -0700 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> <20170602122117.5f86e422@subdivisions.wooz.org> <84A2C0B0-8B3C-400A-96D2-42C368429B3B@lukasa.co.uk> Message-ID: On Mon, Jun 5, 2017 at 8:49 PM, Nick Coghlan wrote: > The reason this kind of approach is really attractive to > redistributors from a customer risk management perspective is that > like gevent's monkeypatching of synchronous networking APIs, it's > *opt-in at runtime*, so the risk of our accidentally inflicting it on > a customer that doesn't want it and doesn't need it is almost exactly > zero - if none of their own code includes the "import _tls_bootstrap; > _tls_bootstrap.monkeypatch_ssl()" invocation and none of their > dependencies start enabling it as an implicit side effect of some > other operation, they'll never even know the enhancement is there. > Instead, the compatibility risks get concentrated in the applications > relying on the bootstrapping API, since the monkeypatching process is > a potentially new source of bugs that don't exist in the more > conventional execution models. OK. To unpack that, I think it would mean: 2.7's ssl.py and _ssl.c remain exactly as they are. We create a new _tls_bootstrap.py and __tls_bootstrap.c, which start out as ~copies of the current ssl.py and _ssl.c code in master. (Unfortunately the SSLObject code can't just be separated out from _ssl.c into a new file, because the implementations of SSLObject and SSLSocket are heavily intertwined.) Then the copied files are hacked up to work in this monkeypatching context (so that e.g. instead of defining the SSLError hierarchy in C code, it gets imported from the main ssl.py). So: we end up with two copies of the ssl code in the py27 tree, both diverged from the copy that in the py3 tree, in different ways. I know that the ssl maintainers won't be *happy* about this given that keeping the py2 and py3 code similar is an ongoing issue and was one of the stated advantages of backporting SSLObject, but I don't know whether it's a showstopper or not; it'd be good to hear their perspective. -n -- Nathaniel J. Smith -- https://vorpus.org From jim.baker at python.org Tue Jun 6 13:49:49 2017 From: jim.baker at python.org (Jim Baker) Date: Tue, 6 Jun 2017 11:49:49 -0600 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> <20170602122117.5f86e422@subdivisions.wooz.org> <84A2C0B0-8B3C-400A-96D2-42C368429B3B@lukasa.co.uk> Message-ID: With Nick's suggestion of _tls_bootstrap, this has certainly become more complicated. I'm still thinking of the ramifications for future Jython 2.7 support, if Python dev goes down this path. It still seems easier to me to avoid exposing the SSLObject/MemoryBIO model to 2.7 and instead concentrate on just having native TLS support. Yes, this might require more work than a simple backport, but everyone is resource constrained, not just the Requests or Jython teams. My concrete suggestion is that any work on PEP 543 will benefit from improving the testing currently found in test_ssl as part of its implementation. For a variety of reasons, test functions like ssl_io_loop ( https://github.com/python/cpython/blob/master/Lib/test/test_ssl.py#L1691) avoid asserting more than it can properly manage the framing of wrapped/unwrapped data, so for example one doesn't want to state explicitly when SSL_ERROR_WANT_READ would be called (too much white box at that point). On the other hand, the lack of unit testing of, for example, SSLObject.read, but instead only doing at the functional test level, means that there's nothing explicit testing "will raise an SSLWantReadError if it needs more data than the incoming BIO has available" (per the docs), or similar use of signaling (SSLEOFError comes to mind). The problem we have seen with Jython supporting similar functionality in the past in socket/select/ssl is not the happy path aspects like what ssl_io_loop tests, but underdocumented/undertested error states. So what benefits SChannel support, for example, should benefit Jython 3's implementation that would use Java's counterpart to SSLObject, SSLEngine. That would be a very good outcome of these proposed PEPs. - Jim On Tue, Jun 6, 2017 at 4:08 AM, Nathaniel Smith wrote: > On Mon, Jun 5, 2017 at 8:49 PM, Nick Coghlan wrote: > > The reason this kind of approach is really attractive to > > redistributors from a customer risk management perspective is that > > like gevent's monkeypatching of synchronous networking APIs, it's > > *opt-in at runtime*, so the risk of our accidentally inflicting it on > > a customer that doesn't want it and doesn't need it is almost exactly > > zero - if none of their own code includes the "import _tls_bootstrap; > > _tls_bootstrap.monkeypatch_ssl()" invocation and none of their > > dependencies start enabling it as an implicit side effect of some > > other operation, they'll never even know the enhancement is there. > > Instead, the compatibility risks get concentrated in the applications > > relying on the bootstrapping API, since the monkeypatching process is > > a potentially new source of bugs that don't exist in the more > > conventional execution models. > > OK. To unpack that, I think it would mean: > > 2.7's ssl.py and _ssl.c remain exactly as they are. > > We create a new _tls_bootstrap.py and __tls_bootstrap.c, which start > out as ~copies of the current ssl.py and _ssl.c code in master. > (Unfortunately the SSLObject code can't just be separated out from > _ssl.c into a new file, because the implementations of SSLObject and > SSLSocket are heavily intertwined.) Then the copied files are hacked > up to work in this monkeypatching context (so that e.g. instead of > defining the SSLError hierarchy in C code, it gets imported from the > main ssl.py). > > So: we end up with two copies of the ssl code in the py27 tree, both > diverged from the copy that in the py3 tree, in different ways. > > I know that the ssl maintainers won't be *happy* about this given that > keeping the py2 and py3 code similar is an ongoing issue and was one > of the stated advantages of backporting SSLObject, but I don't know > whether it's a showstopper or not; it'd be good to hear their > perspective. > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > jbaker%40zyasoft.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Jun 6 18:35:17 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 6 Jun 2017 15:35:17 -0700 Subject: [Python-Dev] The untuned tunable parameter ARENA_SIZE In-Reply-To: References: <20170603120706.75fa7ba8@fsol> <20170604101813.0141930a@fsol> <20170604191934.191fc364@fsol> <6eb84ff4-9561-80ef-a11a-ebdf017f0b73@hastings.org> Message-ID: On Mon, Jun 5, 2017 at 8:10 PM, Tim Peters wrote: > [Tim] > >> So at most 9 arenas ("highwater mark") were ever simultaneously > allocated [by the > >> time the REPL prompt appeared in a 64-bit 3.6.1].. > > > ... though not completely off-base. > > Yes, 9 is in the ballpark of 16. > > > I think _some_ increase of arena size should be a no-brainer, Maybe big enough to hold a bare, just started up interpreter? > but I > don't expect it to help a lot. I was wondering about that -- in my experience with making re-sizable numpy arrays, (not the same use-case, I know!), I found that allocating memory wasn't all that slow, really. IN that use case, if you re-allocated every time you added a single element, it was dog-slow. But once you allocated less than, say, about every 10 or so, it started to make little difference how much you over-allocated. In this case, I'm thinking that as long as there is a not-tiny arena, it just isn't going to make that much difference. But only profiling will tell us. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Tue Jun 6 20:30:04 2017 From: barry at python.org (Barry Warsaw) Date: Tue, 6 Jun 2017 17:30:04 -0700 Subject: [Python-Dev] PEP 7 and braces { .... } on if In-Reply-To: References: <20170603163024.6688f086@subdivisions.wooz.org> Message-ID: <20170606173004.476ff6b2@presto> On Jun 05, 2017, at 08:41 AM, Serhiy Storchaka wrote: >the example above), and the following code is enough readable: > > if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1, > "invalid escape sequence '\\%c'", > *first_invalid_escape) < 0) { > Py_DECREF(result); > return NULL; > } My rationale for placing the opening brace on a separate line, indented to under the `if` instead of hanging is that it's easier to miss the opening brace in the example you posted above. Visually I (we?) tend to have a harder time recognizing characters sitting way out to the right. On Jun 05, 2017, at 08:19 AM, Ethan Furman wrote: >I would format that as: > > if (PyErr_WarnFormat( > PyExc_DeprecationWarning, > 1, > "invalid escape sequence '\\%c'", > *first_invalid_escape) < 0) > { > Py_DECREF(result); > return NULL; > } In this case I'd *still* indent the opening brace to under the `if`. The mismatched indentation between the open and close braces is jarring to me. >- having all the arguments on separate lines means > - the function and first argument don't get run together > - it's easy to pick out the individual arguments That's fine with me, but so is hanging the arguments, so I'd tend to leave this up to the individual devs. >- having the opening brace on its own line means > - a little extra white space to buffer the condition and the body > - it's easier to read the function name and then drop down to the > body Agreed with the rationale for the open brace being on a separate line, but did you mean to indent the opening and closing braces to different levels? Cheers, -Barry From barry at python.org Tue Jun 6 20:45:16 2017 From: barry at python.org (Barry Warsaw) Date: Tue, 6 Jun 2017 17:45:16 -0700 Subject: [Python-Dev] PEP 7 and braces { .... } on if In-Reply-To: References: <20170603163024.6688f086@subdivisions.wooz.org> Message-ID: <20170606174516.74f349be@presto> On Jun 05, 2017, at 07:00 AM, Skip Montanaro wrote: >Wow, this discussion takes me back. Glad I don't have to check out >comp.lang.c to get my brace placement fix. Life is better without braces! -Barry From rob.cliffe at btinternet.com Tue Jun 6 20:42:23 2017 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Wed, 7 Jun 2017 01:42:23 +0100 Subject: [Python-Dev] PEP 7 and braces { .... } on if In-Reply-To: <20170606173004.476ff6b2@presto> References: <20170603163024.6688f086@subdivisions.wooz.org> <20170606173004.476ff6b2@presto> Message-ID: <8a4ab76a-abc2-89cd-c730-cd88850bda1a@btinternet.com> On 07/06/2017 01:30, Barry Warsaw wrote: > On Jun 05, 2017, at 08:41 AM, Serhiy Storchaka wrote: > >> the example above), and the following code is enough readable: >> >> if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1, >> "invalid escape sequence '\\%c'", >> *first_invalid_escape) < 0) { >> Py_DECREF(result); >> return NULL; >> } > My rationale for placing the opening brace on a separate line, indented to > under the `if` instead of hanging is that it's easier to miss the opening > brace in the example you posted above. Visually I (we?) tend to have a harder > time recognizing characters sitting way out to the right. > > On Jun 05, 2017, at 08:19 AM, Ethan Furman wrote: > >> I would format that as: >> >> if (PyErr_WarnFormat( >> PyExc_DeprecationWarning, >> 1, >> "invalid escape sequence '\\%c'", >> *first_invalid_escape) < 0) >> { >> Py_DECREF(result); >> return NULL; >> } > In this case I'd *still* indent the opening brace to under the `if`. The > mismatched indentation between the open and close braces is jarring to me. > > FWIW I feel exactly the same. Rob Cliffe From ncoghlan at gmail.com Wed Jun 7 00:39:45 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 7 Jun 2017 14:39:45 +1000 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> <20170602122117.5f86e422@subdivisions.wooz.org> <84A2C0B0-8B3C-400A-96D2-42C368429B3B@lukasa.co.uk> Message-ID: On 6 June 2017 at 20:08, Nathaniel Smith wrote: > On Mon, Jun 5, 2017 at 8:49 PM, Nick Coghlan wrote: >> The reason this kind of approach is really attractive to >> redistributors from a customer risk management perspective is that >> like gevent's monkeypatching of synchronous networking APIs, it's >> *opt-in at runtime*, so the risk of our accidentally inflicting it on >> a customer that doesn't want it and doesn't need it is almost exactly >> zero - if none of their own code includes the "import _tls_bootstrap; >> _tls_bootstrap.monkeypatch_ssl()" invocation and none of their >> dependencies start enabling it as an implicit side effect of some >> other operation, they'll never even know the enhancement is there. >> Instead, the compatibility risks get concentrated in the applications >> relying on the bootstrapping API, since the monkeypatching process is >> a potentially new source of bugs that don't exist in the more >> conventional execution models. > > OK. To unpack that, I think it would mean: > > 2.7's ssl.py and _ssl.c remain exactly as they are. > > We create a new _tls_bootstrap.py and __tls_bootstrap.c, which start > out as ~copies of the current ssl.py and _ssl.c code in master. > (Unfortunately the SSLObject code can't just be separated out from > _ssl.c into a new file, because the implementations of SSLObject and > SSLSocket are heavily intertwined.) Then the copied files are hacked > up to work in this monkeypatching context (so that e.g. instead of > defining the SSLError hierarchy in C code, it gets imported from the > main ssl.py). > > So: we end up with two copies of the ssl code in the py27 tree, both > diverged from the copy that in the py3 tree, in different ways. If we went down this path, I think the only way to keep it maintainable would be to do one of two things: 1. Do the refactoring in Py3 first, but with the namespace pairing being "ssl/_enable_ssl_backport" rather than "_tls_bootstrap/ssl" 2. Don't try to share any implementation details with Py2, and instead provide a full "_ssl_backport" module more along the lines of importlib2 With the second option, we'd just do a wholesale backport of the 3.6 SSL module under a different name, and projects like pip would use it as follows: # See if there is an updated SSL module backport available try: import _ssl_backport except ImportError: pass else: _ssl_backport.install() # Fails if ssl or _ssl were already imported # At this point, sys.modules["ssl"] and sys.modules["_ssl"] would match the # API of the Python 3.6 versions, *not* the Python 2.7 versions. # At the first 2.7.x maintenance release after 3.7, they'd be updated # to match 3.7, and then the same again for 3.8. # (3.9 is likely to be after the end of 2.7 maintenance) # After that, proceed as usual with feature-based checks try: from ssl import MemoryBIO, SSLObject expect ImportError: # Otherwise fall back to using PyOpenSSL try: from OpenSSL.SSL import Connection except ImportError: raise ImportError("Failed to bootstrap asynchronous SSL/TLS support:
") Since there's no chance of breaking applications that don't opt in, while still making the newer network security features available to customers that want them, I think I could make a reasonable case for backporting such an "_ssl_backport.install()" implementation into the RHEL 7 system Python, but even if I wasn't able to do that, this kind of runtime injection into sys.modules could still be shipped as an add-on library in a way that an updated standard library SSL module can't. Such an approach would make the 2.7 regression tests noticeably slower (since we'd need to re-run a subset of them with the backport being injected by regrtest before anything loads the ssl module), and means backporting SSL fixes to 2.7 would become a two step process (once to resync the _ssl_backport module, and the second to update the default implementation), but I think there would be enough benefits to make that worth the pain: - we'd have a baseline 2.7 SSL/TLS implementation that we can readily keep in line with its 3.x counterparts - this approach has a reasonable chance of making it through the review processes for long term support distributions - system and application integrators can more readily do their own backports independently of redistributors Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ethan at stoneleaf.us Wed Jun 7 00:45:00 2017 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 06 Jun 2017 21:45:00 -0700 Subject: [Python-Dev] PEP 7 and braces { .... } on if In-Reply-To: <20170606173004.476ff6b2@presto> References: <20170603163024.6688f086@subdivisions.wooz.org> <20170606173004.476ff6b2@presto> Message-ID: <593784CC.1020307@stoneleaf.us> On 06/06/2017 05:30 PM, Barry Warsaw wrote: > On Jun 05, 2017, at 08:19 AM, Ethan Furman wrote: >> I would format that as: >> >> if (PyErr_WarnFormat( >> PyExc_DeprecationWarning, >> 1, >> "invalid escape sequence '\\%c'", >> *first_invalid_escape) < 0) >> { >> Py_DECREF(result); >> return NULL; >> } > > In this case I'd *still* indent the opening brace to under the `if`. The > mismatched indentation between the open and close braces is jarring to me. > >> - having all the arguments on separate lines means >> - the function and first argument don't get run together >> - it's easy to pick out the individual arguments > > That's fine with me, but so is hanging the arguments, so I'd tend to leave > this up to the individual devs. > >> - having the opening brace on its own line means >> - a little extra white space to buffer the condition and the body >> - it's easier to read the function name and then drop down to the >> body > > Agreed with the rationale for the open brace being on a separate line, but did > you mean to indent the opening and closing braces to different levels? It's what I see. Left to my own devices I would leave the opening brace where it is and indent the closing brace to match. That way when I see code at the same level as the opening `if` I know I'm out of that block. -- ~Ethan~ From cory at lukasa.co.uk Wed Jun 7 04:31:58 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Wed, 7 Jun 2017 09:31:58 +0100 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> <20170602122117.5f86e422@subdivisions.wooz.org> <84A2C0B0-8B3C-400A-96D2-42C368429B3B@lukasa.co.uk> Message-ID: <6E697036-ECBC-452F-BCF5-09A8F0A51375@lukasa.co.uk> > On 6 Jun 2017, at 18:49, Jim Baker wrote: > > With Nick's suggestion of _tls_bootstrap, this has certainly become more complicated. I'm still thinking of the ramifications for future Jython 2.7 support, if Python dev goes down this path. It still seems easier to me to avoid exposing the SSLObject/MemoryBIO model to 2.7 and instead concentrate on just having native TLS support. Yes, this might require more work than a simple backport, but everyone is resource constrained, not just the Requests or Jython teams. Oh. This hadn?t occurred to me until just now, but doesn?t PyOpenSSL (or anything built on CFFI) just basically not run on Jython? Or am I mistaken? > My concrete suggestion is that any work on PEP 543 will benefit from improving the testing currently found in test_ssl as part of its implementation. For a variety of reasons, test functions like ssl_io_loop (https://github.com/python/cpython/blob/master/Lib/test/test_ssl.py#L1691 ) avoid asserting more than it can properly manage the framing of wrapped/unwrapped data, so for example one doesn't want to state explicitly when SSL_ERROR_WANT_READ would be called (too much white box at that point). On the other hand, the lack of unit testing of, for example, SSLObject.read, but instead only doing at the functional test level, means that there's nothing explicit testing "will raise an SSLWantReadError if it needs more data than the incoming BIO has available" (per the docs), or similar use of signaling (SSLEOFError comes to mind). Yeah, PEP 543 just basically assumes the stdlib?s testing of TLS doesn?t exist and that it?ll have to manufacture its own. Unfortunately, because it is attempting to abstract across many different implementations the tests will need to be fairly high-level: for example, there is no consistent way to discuss the actual size of the buffers in the buffer-based TLS implementation, and they are allowed to be unbounded buffers, so tests cannot validate that TLSWantReadError and TLSWantWriteError are ever actually raised: all they can do is run tests that will handle them in the even that they are raised and confirm the data is transferred appropriately. Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Jun 7 04:56:01 2017 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 7 Jun 2017 01:56:01 -0700 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> <20170602122117.5f86e422@subdivisions.wooz.org> <84A2C0B0-8B3C-400A-96D2-42C368429B3B@lukasa.co.uk> Message-ID: On Tue, Jun 6, 2017 at 10:49 AM, Jim Baker wrote: > With Nick's suggestion of _tls_bootstrap, this has certainly become more > complicated. I'm still thinking of the ramifications for future Jython 2.7 > support, if Python dev goes down this path. It still seems easier to me to > avoid exposing the SSLObject/MemoryBIO model to 2.7 and instead concentrate > on just having native TLS support. Yes, this might require more work than a > simple backport, but everyone is resource constrained, not just the Requests > or Jython teams. Yeah, presumably the preferred way to do PEP 543 on Jython would be to have a native backend based on javax.net.ssl.SSLEngine. Not sure how that affects the MemoryBIO discussion ? maybe it means that Jython just doesn't need a MemoryBIO backport in the stdlib ssl module, because everyone who might use it will find themselves using SSLEngine instead. > My concrete suggestion is that any work on PEP 543 will benefit from > improving the testing currently found in test_ssl as part of its > implementation. For a variety of reasons, test functions like ssl_io_loop > (https://github.com/python/cpython/blob/master/Lib/test/test_ssl.py#L1691) > avoid asserting more than it can properly manage the framing of > wrapped/unwrapped data, so for example one doesn't want to state explicitly > when SSL_ERROR_WANT_READ would be called (too much white box at that point). > On the other hand, the lack of unit testing of, for example, SSLObject.read, > but instead only doing at the functional test level, means that there's > nothing explicit testing "will raise an SSLWantReadError if it needs more > data than the incoming BIO has available" (per the docs), or similar use of > signaling (SSLEOFError comes to mind). Another testing challenge is that the stdlib ssl module has no way to trigger a renegotiation, and therefore there's no way to write tests to check that it properly handles a renegotiation, even though renegotiation is by far the trickiest part of the protocol to get right. (In particular, renegotiation is the only case where attempting to read can give WantWrite and vice-versa.) -n -- Nathaniel J. Smith -- https://vorpus.org From victor.stinner at gmail.com Wed Jun 7 09:29:19 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 7 Jun 2017 15:29:19 +0200 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> <20170602122117.5f86e422@subdivisions.wooz.org> <84A2C0B0-8B3C-400A-96D2-42C368429B3B@lukasa.co.uk> Message-ID: 2017-06-07 10:56 GMT+02:00 Nathaniel Smith : > Another testing challenge is that the stdlib ssl module has no way to > trigger a renegotiation, and therefore there's no way to write tests > to check that it properly handles a renegotiation, even though > renegotiation is by far the trickiest part of the protocol to get > right. (In particular, renegotiation is the only case where attempting > to read can give WantWrite and vice-versa.) Renegociation was the source of a vulnerability in SSL/TLS protocols, so maybe it's a good thing that it's not implemented :-) https://www.rapid7.com/db/vulnerabilities/tls-sess-renegotiation Renegociation was removed from the new TLS 1.3 protocol: https://tlswg.github.io/tls13-spec/ "TLS 1.3 forbids renegotiation" Victor From cory at lukasa.co.uk Wed Jun 7 09:33:34 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Wed, 7 Jun 2017 14:33:34 +0100 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> <20170602122117.5f86e422@subdivisions.wooz.org> <84A2C0B0-8B3C-400A-96D2-42C368429B3B@lukasa.co.uk> Message-ID: <35D560D8-D194-46C4-B768-29FBDF340980@lukasa.co.uk> > On 7 Jun 2017, at 14:29, Victor Stinner wrote: > > 2017-06-07 10:56 GMT+02:00 Nathaniel Smith : >> Another testing challenge is that the stdlib ssl module has no way to >> trigger a renegotiation, and therefore there's no way to write tests >> to check that it properly handles a renegotiation, even though >> renegotiation is by far the trickiest part of the protocol to get >> right. (In particular, renegotiation is the only case where attempting >> to read can give WantWrite and vice-versa.) > > Renegociation was the source of a vulnerability in SSL/TLS protocols, > so maybe it's a good thing that it's not implemented :-) > https://www.rapid7.com/db/vulnerabilities/tls-sess-renegotiation > > Renegociation was removed from the new TLS 1.3 protocol: > https://tlswg.github.io/tls13-spec/ > "TLS 1.3 forbids renegotiation" Renegotiation remains extremely widely deployed with TLS client certificates in enterprise environments, sadly. Cory From sbellem at gmail.com Wed Jun 7 10:36:56 2017 From: sbellem at gmail.com (Sylvain Bellemare) Date: Wed, 7 Jun 2017 16:36:56 +0200 Subject: [Python-Dev] Support for RFC 6920: Naming Things with Hashes? Message-ID: Hi, I hope this is the right place to post this kind of question. If not I apologize. I was simply wondering if anyone had been looking into supporting RFC 6920 ( https://tools.ietf.org/html/rfc6920). For a simple example of what this is about see https://tools.ietf.org/html/rfc6920#section-8 Thanks! Sylvain -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Jun 7 14:49:38 2017 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 7 Jun 2017 11:49:38 -0700 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> <20170602122117.5f86e422@subdivisions.wooz.org> <84A2C0B0-8B3C-400A-96D2-42C368429B3B@lukasa.co.uk> Message-ID: On Jun 7, 2017 6:29 AM, "Victor Stinner" wrote: 2017-06-07 10:56 GMT+02:00 Nathaniel Smith : > Another testing challenge is that the stdlib ssl module has no way to > trigger a renegotiation, and therefore there's no way to write tests > to check that it properly handles a renegotiation, even though > renegotiation is by far the trickiest part of the protocol to get > right. (In particular, renegotiation is the only case where attempting > to read can give WantWrite and vice-versa.) Renegociation was the source of a vulnerability in SSL/TLS protocols, so maybe it's a good thing that it's not implemented :-) https://www.rapid7.com/db/vulnerabilities/tls-sess-renegotiation Renegociation was removed from the new TLS 1.3 protocol: https://tlswg.github.io/tls13-spec/ "TLS 1.3 forbids renegotiation" Oh, sure, renegotiation is awful, no question. The HTTP/2 spec also forbids it. But it does still get used and python totally implements it :-). If python is talking to some peer and the peer says "hey, let's renegotiate now" then it will. There just aren't any tests for what happens next. Not that this has much to do with MemoryBIOs. Sorry for the tangent. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim.baker at python.org Wed Jun 7 15:06:39 2017 From: jim.baker at python.org (Jim Baker) Date: Wed, 7 Jun 2017 13:06:39 -0600 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <6E697036-ECBC-452F-BCF5-09A8F0A51375@lukasa.co.uk> References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> <20170602122117.5f86e422@subdivisions.wooz.org> <84A2C0B0-8B3C-400A-96D2-42C368429B3B@lukasa.co.uk> <6E697036-ECBC-452F-BCF5-09A8F0A51375@lukasa.co.uk> Message-ID: On Wed, Jun 7, 2017 at 2:31 AM, Cory Benfield wrote: > > On 6 Jun 2017, at 18:49, Jim Baker wrote: > > With Nick's suggestion of _tls_bootstrap, this has certainly become more > complicated. I'm still thinking of the ramifications for future Jython 2.7 > support, if Python dev goes down this path. It still seems easier to me to > avoid exposing the SSLObject/MemoryBIO model to 2.7 and instead concentrate > on just having native TLS support. Yes, this might require more work than a > simple backport, but everyone is resource constrained, not just the > Requests or Jython teams. > > > Oh. This hadn?t occurred to me until just now, but doesn?t PyOpenSSL (or > anything built on CFFI) just basically not run on Jython? Or am I mistaken? > Sorry, I wish this were true, but PyOpenSSL is not available on Jython, because we do not yet support CFFI for Jython. CFFI support is something we have looked at, but we have not implemented. (There is a related and far more ambitious project to implement support for the C Extension API, http://jyni.org/, which Stefan Richthofer is working on with me under the Google Summer of Code.) Having said that, bundling PyOpenSSL for use by pip is something that we would *not* want to do for Jython itself. We want to use the native Java ecosystem where possible for built-in functionality, in part because we already have native support already to the underlying system trust store on *all platforms*. (Java development did all the heavy lifting for us.) Instead our current implementation of socket/ssl/select is built on Netty, plus some additional work for managing SSLContext in such a way that is compatible with Python. There is an out-of-date doc I prepared describing what was done (but broad aspects are still current): https://github.com/ jimbaker/socket-reboot > > My concrete suggestion is that any work on PEP 543 will benefit from > improving the testing currently found in test_ssl as part of its > implementation. For a variety of reasons, test functions like ssl_io_loop ( > https://github.com/python/cpython/blob/master/Lib/test/test_ssl.py#L1691) > avoid asserting more than it can properly manage the framing of > wrapped/unwrapped data, so for example one doesn't want to state explicitly > when SSL_ERROR_WANT_READ would be called (too much white box at that > point). On the other hand, the lack of unit testing of, for example, > SSLObject.read, but instead only doing at the functional test level, means > that there's nothing explicit testing "will raise an SSLWantReadError if it > needs more data than the incoming BIO has available" (per the docs), or > similar use of signaling (SSLEOFError comes to mind). > > > Yeah, PEP 543 just basically assumes the stdlib?s testing of TLS doesn?t > exist and that it?ll have to manufacture its own. Unfortunately, because it > is attempting to abstract across many different implementations the tests > will need to be fairly high-level: for example, there is no consistent way > to discuss the actual size of the buffers in the buffer-based TLS > implementation, and they are allowed to be unbounded buffers, so tests > cannot validate that TLSWantReadError and TLSWantWriteError are ever > actually raised: all they can do is run tests that will handle them in the > even that they are raised and confirm the data is transferred appropriately. > Right, so such improved testing, regardless of level, will still be an improvement, and we look forward to seeing it for use by Jython as well. I should mention that we sometimes see undocumented functionality leak out. For example, https://docs.python.org/3/library/socket.html#socket.socket.listen doesn't say anything about backlog=0, but the requests test suite (last time I looked on Jython) now depends on it. We assumed it was something like http://pubs.opengroup.org/onlinepubs/009695399/functions/listen.html, but as described in http://bugs.python.org/issue8498, it now means that "a server application in python that accepts exactly 1 connection", by passing to the underlying C. More tests, more docs, please, even though of course that's a lot of dev effort. - Jim -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike_computinker at coffeebird.net Wed Jun 7 15:47:20 2017 From: mike_computinker at coffeebird.net (CompuTinker) Date: Wed, 7 Jun 2017 14:47:20 -0500 Subject: [Python-Dev] Extracting python bytecode from a linux core dump? Message-ID: <20170607194720.GA18806@coffeebird.net> I hope this is the proper place for internals questions... I have a core file (produced via the gcore command) of a linux python2.6 process. I need to extract the byte code and de-compile it. I looked at https://wiki.python.org/moin/DebuggingWithGdb and related pages. However, these all seem to require either a running process and/or a binary with debugging symbols. I'm thinking that the compiled bytecode is likely in an array or contiguous set of memory within the python executable's image and that there's probably a way to pull it out with gdb. Unsurprisingly, the pyc 0xd1f20d0a magic number isn't kept in memory. So, how do I find the memory holding the compiled byte-code ? From cory at lukasa.co.uk Thu Jun 8 04:30:37 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 8 Jun 2017 09:30:37 +0100 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> <20170602122117.5f86e422@subdivisions.wooz.org> <84A2C0B0-8B3C-400A-96D2-42C368429B3B@lukasa.co.uk> <6E697036-ECBC-452F-BCF5-09A8F0A51375@lukasa.co.uk> Message-ID: > On 7 Jun 2017, at 20:06, Jim Baker wrote: > > Sorry, I wish this were true, but PyOpenSSL is not available on Jython, because we do not yet support CFFI for Jython. CFFI support is something we have looked at, but we have not implemented. (There is a related and far more ambitious project to implement support for the C Extension API, http://jyni.org/ , which Stefan Richthofer is working on with me under the Google Summer of Code.) This is what I was worried about. Moving to require PyOpenSSL *also* locks us out of Jython support, at least for the time being. That?s another point in the ?con? column for making PyOpenSSL a mandatory dependency. > I should mention that we sometimes see undocumented functionality leak out. For example, https://docs.python.org/3/library/socket.html#socket.socket.listen doesn't say anything about backlog=0, but the requests test suite (last time I looked on Jython) now depends on it. We assumed it was something like http://pubs.opengroup.org/onlinepubs/009695399/functions/listen.html , but as described in http://bugs.python.org/issue8498 , it now means that "a server application in python that accepts exactly 1 connection", by passing to the underlying C. More tests, more docs, please, even though of course that's a lot of dev effort. Ah, yes, we do. In our defense, this is the semantic of the listen syscall, and the socket module is generally a pretty thin wrapper around most of the socket syscalls. But in hindsight this is definitely the kind of thing that gets tricky for someone trying to reimplement the socket module in terms of a different abstraction. I don?t want to dive down this rabbit hole because if we do I?ll have to start talking about how the complexity of the socket API is one of the other implementation hurdles for PEP 543, but I think that?s a conversation for another time. Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Thu Jun 8 06:36:42 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 8 Jun 2017 12:36:42 +0200 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> <20170602122117.5f86e422@subdivisions.wooz.org> <84A2C0B0-8B3C-400A-96D2-42C368429B3B@lukasa.co.uk> <6E697036-ECBC-452F-BCF5-09A8F0A51375@lukasa.co.uk> Message-ID: 2017-06-08 10:30 GMT+02:00 Cory Benfield : > This is what I was worried about. Moving to require PyOpenSSL *also* locks > us out of Jython support, at least for the time being. That?s another point > in the ?con? column for making PyOpenSSL a mandatory dependency. Even if we do backport MemoryBIO to the next Python 2.7.14, I don't think that you can require MemoryBIO. What about all existing operating systems which provide a Python 2.7 without MemoryBIO? You need to have a workaround anyway. For example, make the new asynchronous API optional and use the old blocking mode in the meanwhile. Victor From turnbull.stephen.fw at u.tsukuba.ac.jp Thu Jun 8 06:37:31 2017 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Thu, 8 Jun 2017 19:37:31 +0900 Subject: [Python-Dev] Support for RFC 6920: Naming Things with Hashes? In-Reply-To: References: Message-ID: <22841.10475.411064.968180@turnbull.sk.tsukuba.ac.jp> Sylvain Bellemare writes: > Hi, > > I hope this is the right place to post this kind of question. If not I > apologize. > > I was simply wondering if anyone had been looking into supporting RFC 6920 ( > https://tools.ietf.org/html/rfc6920). > > For a simple example of what this is about see > https://tools.ietf.org/html/rfc6920#section-8 This is not the best place. My first take would be "This is a great idea! But it doesn't need to be in core. Write it up and post it to PyPI." My second would be "there's probably a special interest group (SIG) mailing list" that's more appropriate. See https://mail.python.org/; WebSIG list looks like a good bet. Third take, try python-list at python.org and see what the general Python users say. Many are web developers. Finally, I am not a web expert, just a long time follower of discussions on these lists. If you are really sure this belongs in the stdlib right now, you could try python-ideas at python.org. That list is for proposing and refining new ideas, this one is more for discussion of stuff that has an implementation in progress and needs fine-tuning. Happy hacking! Steve From donald at stufft.io Thu Jun 8 07:06:03 2017 From: donald at stufft.io (Donald Stufft) Date: Thu, 8 Jun 2017 07:06:03 -0400 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> <20170602122117.5f86e422@subdivisions.wooz.org> <84A2C0B0-8B3C-400A-96D2-42C368429B3B@lukasa.co.uk> <6E697036-ECBC-452F-BCF5-09A8F0A51375@lukasa.co.uk> Message-ID: <2E0E1D87-4AB3-42B8-9788-AED19B64DE1E@stufft.io> > On Jun 8, 2017, at 6:36 AM, Victor Stinner wrote: > > 2017-06-08 10:30 GMT+02:00 Cory Benfield : >> This is what I was worried about. Moving to require PyOpenSSL *also* locks >> us out of Jython support, at least for the time being. That?s another point >> in the ?con? column for making PyOpenSSL a mandatory dependency. > > Even if we do backport MemoryBIO to the next Python 2.7.14, I don't > think that you can require MemoryBIO. What about all existing > operating systems which provide a Python 2.7 without MemoryBIO? You > need to have a workaround anyway. For example, make the new > asynchronous API optional and use the old blocking mode in the > meanwhile. > I mentioned it earlier, but using the current download numbers from PyPI, <2.7.9 is rapidly dropping and is likely going to be single digit %age within the next 6-8 months IIRC. If 2.7.14 follows a similar trajectory, requests can depend on it within like? 2 years? Maybe 3? Likely it will depend on whether 2.7.14 gets into the next Ubuntu LTS or not. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Thu Jun 8 07:30:52 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 8 Jun 2017 13:30:52 +0200 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <2E0E1D87-4AB3-42B8-9788-AED19B64DE1E@stufft.io> References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> <20170602122117.5f86e422@subdivisions.wooz.org> <84A2C0B0-8B3C-400A-96D2-42C368429B3B@lukasa.co.uk> <6E697036-ECBC-452F-BCF5-09A8F0A51375@lukasa.co.uk> <2E0E1D87-4AB3-42B8-9788-AED19B64DE1E@stufft.io> Message-ID: Maybe the intent of my PEP is unclear: the goal is not to allow Requests to require MemoryBIO, but to get a wide adoption of a future implementation of the new TLS API (PEP). IMHO having an implementation working on the latest Python 2.7 version should make it possible to use it on some kinds of applications. I should take time to read the last messages in this thread and try to summarize them in the PEP ;-) Victor 2017-06-08 13:06 GMT+02:00 Donald Stufft : > > On Jun 8, 2017, at 6:36 AM, Victor Stinner wrote: > > 2017-06-08 10:30 GMT+02:00 Cory Benfield : > > This is what I was worried about. Moving to require PyOpenSSL *also* locks > us out of Jython support, at least for the time being. That?s another point > in the ?con? column for making PyOpenSSL a mandatory dependency. > > > Even if we do backport MemoryBIO to the next Python 2.7.14, I don't > think that you can require MemoryBIO. What about all existing > operating systems which provide a Python 2.7 without MemoryBIO? You > need to have a workaround anyway. For example, make the new > asynchronous API optional and use the old blocking mode in the > meanwhile. > > > I mentioned it earlier, but using the current download numbers from PyPI, > <2.7.9 is rapidly dropping and is likely going to be single digit %age > within the next 6-8 months IIRC. If 2.7.14 follows a similar trajectory, > requests can depend on it within like? 2 years? Maybe 3? Likely it will > depend on whether 2.7.14 gets into the next Ubuntu LTS or not. > > ? > Donald Stufft > > > From solipsis at pitrou.net Thu Jun 8 07:54:14 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 8 Jun 2017 13:54:14 +0200 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 References: <84A2C0B0-8B3C-400A-96D2-42C368429B3B@lukasa.co.uk> <6E697036-ECBC-452F-BCF5-09A8F0A51375@lukasa.co.uk> Message-ID: <20170608135414.6fda167f@fsol> On Thu, 8 Jun 2017 09:30:37 +0100 Cory Benfield wrote: > > Ah, yes, we do. In our defense, this is the semantic of the listen syscall,[...] According to POSIX, the backlog is only a hint, i.e. Jython is probably ok in not observing its value: """The backlog argument provides a hint to the implementation which the implementation shall use to limit the number of outstanding connections in the socket's listen queue.""" http://pubs.opengroup.org/onlinepubs/9699919799/functions/listen.html Regards Antoine. From victor.stinner at gmail.com Thu Jun 8 08:01:12 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 8 Jun 2017 14:01:12 +0200 Subject: [Python-Dev] Buildbot report, june 2017 Message-ID: Hi, With the help of Zachary Ware, we succeed to set up a mailing list getting email notifications when a buildbot fails whereas it previously succeeded (green/success or orange/warnings): https://mail.python.org/mm3/mailman3/lists/buildbot-status.python.org/ Please subscribe if you want to stay tuned. I started to create bug reports for each buildbot failures on 2.7, 3.5, 3.6 and master (future 3.7) branches. I try to reply to buildbot emails with a reference to the issue (I create a new issue if needed). The good news is that these issues are being slowly fixed one by one! In general, buildbots are now stable and at least able to catch the obvious regressions. Zachary Ware added two new "Refleaks" buildbot slaves, on Windows 8 and Gentoo, to catch reference leaks. We are fixing leaks. It seems like some of them are recent. Previously, we add a server ran by Antoine Pitrou, but Antoine didn't update its configuration for Git (when CPython migrated from Mercurial). I like the idea of an integrated "Refleaks" server in our buildbot farm. Information on CPython buildbots: https://www.python.org/dev/buildbot/ Victor From ncoghlan at gmail.com Thu Jun 8 09:33:50 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 8 Jun 2017 23:33:50 +1000 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <20170601105703.753551fd@fsol> <86c00b75-60f7-4dda-e0ba-92d9b8a9e214@python.org> <20170601103914.GA8121@k3> <7B74D214-F781-423A-AC60-7C6A4CA97074@lukasa.co.uk> <20170601110912.GB8121@k3> <9D5664F2-05EF-44F1-873A-164928F628CC@lukasa.co.uk> <20170601141000.GC8121@k3> <37c5b34f-c5ae-53a5-3fc2-730701d7f1f8@python.org> <90D05B5C-F81B-48CF-99E5-B400EA2CEB90@lukasa.co.uk> <20170602122117.5f86e422@subdivisions.wooz.org> <84A2C0B0-8B3C-400A-96D2-42C368429B3B@lukasa.co.uk> <6E697036-ECBC-452F-BCF5-09A8F0A51375@lukasa.co.uk> <2E0E1D87-4AB3-42B8-9788-AED19B64DE1E@stufft.io> Message-ID: On 8 June 2017 at 21:30, Victor Stinner wrote: > Maybe the intent of my PEP is unclear: the goal is not to allow > Requests to require MemoryBIO, but to get a wide adoption of a future > implementation of the new TLS API (PEP). IMHO having an implementation > working on the latest Python 2.7 version should make it possible to > use it on some kinds of applications. > > I should take time to read the last messages in this thread and try to > summarize them in the PEP ;-) I think the short version of the main unanswered questions would be: 1. How hard would it be to define a full wheel-installable "backport.ssl" module that made the latest 3.x ssl module API available to Python 2 applications? 2. Given such a module, how hard would it be to add an "install()" function that added it to sys.modules as the "ssl" module? While that wouldn't allow pip or requests to depend on it without some solution to the bootstrapping problem, it would clearly separate the two questions, and also give us an architecture closer to the way importlib2 works (i.e. if you want to make that the implementation of import in a Python 2 application, you have to explicitly enable it at runtime, but you can also install it whenever you want, rather than having to rely on your Python implementation to provide it). We'd also still be left with the probably of figuring out how to handle Jython, but I figure we can consider that part of the bootstrapping problem: how to let applications reliably get hold of "backport.ssl" under a name provided by the host Python 2.7 implementation, rather than the normal pip-installable name. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From jake at lwn.net Thu Jun 8 11:52:53 2017 From: jake at lwn.net (Jake Edge) Date: Thu, 8 Jun 2017 09:52:53 -0600 Subject: [Python-Dev] 2017 Python Language Summit coverage finale Message-ID: <20170608095253.3444f4af@redtail> Hola python-dev, I (finally!) finished up the coverage from the language summit ... The starting point is the overview article, here: https://lwn.net/Articles/723251/ which should now be free for anyone to see (and the first seven articles too). LWN subscribers can see the articles right away, but one week after they are published in the weekly edition, they become freely available for everyone. Until then, though, feel free to share the SubscriberLinks I am posting here. I have been asked about our policy on appropriate places to share SubscriberLinks; blogs, tweets, social media, mailing lists, etc. are all perfectly fine with us. The new articles are: Classes and types in the Python typing module https://lwn.net/Articles/724639/ or https://lwn.net/SubscriberLink/724639/b055ef642e22abfc/ Status of mypy and type checking https://lwn.net/Articles/724740/ or https://lwn.net/SubscriberLink/724740/5d0b0158880097cf/ Language summit lightning talks https://lwn.net/Articles/723823/ or https://lwn.net/SubscriberLink/723823/695e5554c4d68f1b/ Thanks again to Larry and Barry for inviting me! jake -- Jake Edge - LWN - jake at lwn.net - http://lwn.net From steve.dower at python.org Thu Jun 8 12:40:27 2017 From: steve.dower at python.org (Steve Dower) Date: Thu, 8 Jun 2017 09:40:27 -0700 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <6E697036-ECBC-452F-BCF5-09A8F0A51375@lukasa.co.uk> <2E0E1D87-4AB3-42B8-9788-AED19B64DE1E@stufft.io> Message-ID: <07e12379-84e0-b519-22a2-1a75c4d958b6@python.org> I'm just going to straight up admit that I've lost track of the point of this thread. It sounds like we don't *need* to backport any of ssl into the Python 2.7 standard library, as long as we can bundle a 3rd-party backport for pip? I assume that, at a high level, the operation needed is to download content over https using the system trust stores. Is that what we're trying to achieve here? If it is, can we just provide an enhanced urlretrieve()? Certainly on Windows, and presumably on macOS, it's much easier to do the high-level GET operation than to reimplement it using primitives. As far as I can tell (bearing in mind that my brain can't keep track of this thread anymore), the only argument against this is if someone wants sensible default behaviour with local configuration tweaks: treat one specific certificate as valid while also treating the system (and user) certificate stores as valid without actually putting your certificate into the system (or user) store. My gut reaction to that is to say "too bad - choices are 100% system configuration or 100% custom configuration". I suspect that's not a suitable reaction, but I can't explain why. (And bear in mind that the current state of the ssl module on Windows will never get 100% system configuration.) Can someone explain to me why pip can't use a simple "system store urlretrieve()" function without configuration and "OpenSSL urlretrieve()" function with fully custom configuration? Even if only to bootstrap something that *can* merge two entirely different configuration systems? Cheers, Steve From sbellem at gmail.com Thu Jun 8 14:11:15 2017 From: sbellem at gmail.com (Sylvain Bellemare) Date: Thu, 8 Jun 2017 20:11:15 +0200 Subject: [Python-Dev] Support for RFC 6920: Naming Things with Hashes? In-Reply-To: <22841.10475.411064.968180@turnbull.sk.tsukuba.ac.jp> References: <22841.10475.411064.968180@turnbull.sk.tsukuba.ac.jp> Message-ID: On 8 June 2017 at 12:37, Stephen J. Turnbull < turnbull.stephen.fw at u.tsukuba.ac.jp> wrote: > Sylvain Bellemare writes: > > Hi, > > > > I hope this is the right place to post this kind of question. If not I > > apologize. > > > > I was simply wondering if anyone had been looking into supporting RFC > 6920 ( > > https://tools.ietf.org/html/rfc6920). > > > > For a simple example of what this is about see > > https://tools.ietf.org/html/rfc6920#section-8 > > This is not the best place. My first take would be "This is a great > idea! But it doesn't need to be in core. Write it up and post it to > PyPI." > > My second would be "there's probably a special interest group (SIG) > mailing list" that's more appropriate. See https://mail.python.org/; > WebSIG list looks like a good bet. Third take, try > python-list at python.org and see what the general Python users say. > Many are web developers. > > Finally, I am not a web expert, just a long time follower of > discussions on these lists. If you are really sure this belongs in > the stdlib right now, you could try python-ideas at python.org. That > list is for proposing and refining new ideas, this one is more for > discussion of stuff that has an implementation in progress and needs > fine-tuning. > > Happy hacking! > > Steve > > Thanks for the detailed answer! I will follow your tips! -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Jun 8 15:35:34 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 8 Jun 2017 20:35:34 +0100 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <07e12379-84e0-b519-22a2-1a75c4d958b6@python.org> References: <6E697036-ECBC-452F-BCF5-09A8F0A51375@lukasa.co.uk> <2E0E1D87-4AB3-42B8-9788-AED19B64DE1E@stufft.io> <07e12379-84e0-b519-22a2-1a75c4d958b6@python.org> Message-ID: On 8 June 2017 at 17:40, Steve Dower wrote: > I'm just going to straight up admit that I've lost track of the point of > this thread. You have my sympathies - I'm not really following it either :-( > It sounds like we don't *need* to backport any of ssl into the Python 2.7 > standard library, as long as we can bundle a 3rd-party backport for pip? My understanding is that the PEP is asking to backport a new feature. The problem is that as a new feature, this received some (justifiable) pushback. The arguments for why this new feature is needed then got messy - as I understand it, it's something to do with how the requests library moves forward - they are blocked from supporting newer async features from 3.x because they need to support 2.7. This feature would relieve that logjam for them. Obviously, as a 3rd party library, their issues aren't particularly compelling for the stdlib - but because pip uses requests, and pip is shipped with Python, things get complicated. > I assume that, at a high level, the operation needed is to download content > over https using the system trust stores. Is that what we're trying to > achieve here? > > If it is, can we just provide an enhanced urlretrieve()? Certainly on > Windows, and presumably on macOS, it's much easier to do the high-level GET > operation than to reimplement it using primitives. The problem is that pip uses more features of requests than just issuing GET requests. We aren't going to be in a position to switch to a simple urlretrieve operation, even as some sort of fallback. What I'm personally not at all clear on is why we can't just ship a version of pip that supports 2.7 with 2.7, and a later version with 3.x. That doesn't make the problem for pip and requests any easier, but it does make it not python-dev's problem. The issue is that the gulf between 2.7 and 3.x is getting wider, and it's starting to cause real pain to 3rd party projects like requests. Does that justify backporting this specific feature to 2.7? I don't know. Note that I haven't actually read the original PEP. I don't have a view on networking issues, security, or Python 2.7 support. So I didn't really feel the need to more than skim this thread. My only interest really is where pip gets involved - and that's where I get confused as I don't really see why (ensure)pip makes the problem so much more complicated. Paul PS I'd be amazed if my summary above isn't wrong in at least some key points. Corrections welcome! From donald at stufft.io Thu Jun 8 15:37:53 2017 From: donald at stufft.io (Donald Stufft) Date: Thu, 8 Jun 2017 15:37:53 -0400 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <07e12379-84e0-b519-22a2-1a75c4d958b6@python.org> References: <6E697036-ECBC-452F-BCF5-09A8F0A51375@lukasa.co.uk> <2E0E1D87-4AB3-42B8-9788-AED19B64DE1E@stufft.io> <07e12379-84e0-b519-22a2-1a75c4d958b6@python.org> Message-ID: <8287CE3C-687E-4BE6-9747-39E651B7178B@stufft.io> > On Jun 8, 2017, at 12:40 PM, Steve Dower wrote: > > I'm just going to straight up admit that I've lost track of the point of this thread. > > It sounds like we don't *need* to backport any of ssl into the Python 2.7 standard library, as long as we can bundle a 3rd-party backport for pip? > > I assume that, at a high level, the operation needed is to download content over https using the system trust stores. Is that what we're trying to achieve here? The basic yak stak here is: * PEP 543 should be the future, it is a much much better way of handling TLS than our current ssl module is. * Cory can?t spend his work time on PEP 543 unless he can say it is useful for requests. * In order for PEP 543 to be useful for requests, he needs a way to provide a backport for it for Python 2.7. * This backport *CAN* be OpenSSL only, but needs to be able to provide the same API. * PEP 543 wants to work with MemoryBIOs instead of sockets, because a MemoryBio is a much much better way of implementing this problem for a variety of reasons, and it would be a mistake to use a socket primitive again. * Indepently, requests also wants to be able to provide the ability for people to use it with asyncio, however it can?t drop support for Python 2.7 in the quest for doing that. Twisted provides a way forward that lets requests work on both 2.x and 3.x and integrate with asyncio, but Twisted requires MemoryBio to do so. * pyOpenSSL *could* be used to provide the MemoryBio needed on 2.7 for both cases from up above, however, pip cannot depend on a C library that isn?t part of the standard library - in addition this would break alternative runtimes like Jython where pyOpenSSL doesn?t work. Thus, adding MemoryBio to 2.7?s ssl means that requests can use it instead of depending on a C package (which it can?t because of pip), which means that Cory can then justify working on PEP 543 as part of his requests work, because he can say on Python 3.7, We use PEP 543 natively, and on Python < 3.7 we either no longer support that Python or we can wrap ssl.MemoryBio using a pure Python backport shim that provides the same API as PEP 543. Indendently of that, adding MemoryBio to 2.7?s ssl means that Twisted can use it instead of depending on a C package, which means requests can depend on Twisted (which it otherwise can?t, again because of pip). This then means that requests can refactor itself internally to use Twisted to write asyncio compatible code that provides a synchronous API and a asynchronous API that works on 2.7 and 3.x with asyncio. All of the other options require effectively forking the code or the ecosystem by either having ?this library you use for sync? and ?the library you use for async? largely duplicating code OR requires all of the network libraries to drop support for 2.7 (can?t, most of their users are on 2.7 still) or requires forking the library to have a 2.x and a 3.x version (we tried this before early on in the 3.x split and we settled on the fact that a single code base is a much better way to handle straddling the 2.x/3.x line). So basically back porting MemoryBio unlocks two important things for the health of the Python ecosystem: * Allows forward progress on PEP 543, which provides a wealth of great benefits like using the platform trust model and removing the need for pip, requests, etc to bundle a CA bundle internally and removing the need (long term anyways) for Python to ship a copy of OpenSSL on platforms that don?t provide it. * Allows requests and other libraries to continue to straddle the 2.x/3.x line where they need to, while still providing people who are using Python 3.x a way to use asyncio without having to fork the entire ecosystem into having an aio* copy of every single network library that exists. > > Can someone explain to me why pip can't use a simple "system store urlretrieve()" function without configuration and "OpenSSL urlretrieve()" function with fully custom configuration? Even if only to bootstrap something that *can* merge two entirely different configuration systems? It would require rewriting significant parts of our code that interfaces with HTTP, including the fact that we rely on some additional requests libraries (like cache control) to implement HTTP caching in pip, so unless urlretireve supported that as well we?d have to completely rewrite that from scratch. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Jun 8 15:51:24 2017 From: donald at stufft.io (Donald Stufft) Date: Thu, 8 Jun 2017 15:51:24 -0400 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <8287CE3C-687E-4BE6-9747-39E651B7178B@stufft.io> References: <6E697036-ECBC-452F-BCF5-09A8F0A51375@lukasa.co.uk> <2E0E1D87-4AB3-42B8-9788-AED19B64DE1E@stufft.io> <07e12379-84e0-b519-22a2-1a75c4d958b6@python.org> <8287CE3C-687E-4BE6-9747-39E651B7178B@stufft.io> Message-ID: <8932A107-0D8C-4B6C-A32F-8861D7B19EF1@stufft.io> > On Jun 8, 2017, at 3:37 PM, Donald Stufft wrote: > > So basically back porting MemoryBio unlocks two important things for the health of the Python ecosystem: > > * Allows forward progress on PEP 543, which provides a wealth of great benefits like using the platform trust model and removing the need for pip, requests, etc to bundle a CA bundle internally and removing the need (long term anyways) for Python to ship a copy of OpenSSL on platforms that don?t provide it. > * Allows requests and other libraries to continue to straddle the 2.x/3.x line where they need to, while still providing people who are using Python 3.x a way to use asyncio without having to fork the entire ecosystem into having an aio* copy of every single network library that exists. Sorry I forgot one other important benefit: * It reduces the delta between the 3.x and the 2.x ssl and _ssl modules, which makes actually maintaining those modules easier because this code is fiddly and hard to get right, so the more we can just directly backport security fixes from one to the other rather than having to rewrite the patch, the better off we are. And the downside here is pretty small honestly? It?s not changing behavior of anything that currently exists since it?s adding a new thing inside the ssl module and Alex has already written the patch so there?s little extra work to do and it actually makes maintenance easier since patches can more readily be applied straight from `master`. The primary argument I can see against it, is one of purity, that 2.7 shouldn?t get new features but as we know, practicality beats purity ;) (and we?ve already accepted that TLS is a special case, special enough to break the rules, so the main question is whether this specific thing is worthwhile? which given it?s benefits to the Python ecosystem and to maintaining Python, I think it is). ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Thu Jun 8 15:57:51 2017 From: steve.dower at python.org (Steve Dower) Date: Thu, 8 Jun 2017 12:57:51 -0700 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <8287CE3C-687E-4BE6-9747-39E651B7178B@stufft.io> References: <6E697036-ECBC-452F-BCF5-09A8F0A51375@lukasa.co.uk> <2E0E1D87-4AB3-42B8-9788-AED19B64DE1E@stufft.io> <07e12379-84e0-b519-22a2-1a75c4d958b6@python.org> <8287CE3C-687E-4BE6-9747-39E651B7178B@stufft.io> Message-ID: <40d4782d-a4de-8cb1-cf5b-a4ad7d30f9c2@python.org> On 08Jun2017 1237, Donald Stufft wrote: > > The basic yak stak here is: > > * PEP 543 should be the future, it is a much much better way of handling > TLS than our current ssl module is. > * Cory can?t spend his work time on PEP 543 unless he can say it is > useful for requests. > * In order for PEP 543 to be useful for requests, he needs a way to > provide a backport for it for Python 2.7. > * This backport *CAN* be OpenSSL only, but needs to be able to > provide the same API. > * PEP 543 wants to work with MemoryBIOs instead of sockets, because a > MemoryBio is a much much better way of implementing this problem for a > variety of reasons, and it would be a mistake to use a socket primitive > again. > * Indepently, requests also wants to be able to provide the ability for > people to use it with asyncio, however it can?t drop support for Python > 2.7 in the quest for doing that. Twisted provides a way forward that > lets requests work on both 2.x and 3.x and integrate with asyncio, but > Twisted requires MemoryBio to do so. > * pyOpenSSL *could* be used to provide the MemoryBio needed on 2.7 for > both cases from up above, however, pip cannot depend on a C library that > isn?t part of the standard library - in addition this would break > alternative runtimes like Jython where pyOpenSSL doesn?t work. Awesome, this is exactly what I needed to see. What if Python 2.7 just exposed the OpenSSL primitives necessary so that ctypes could use them? Is that a viable approach here? Presumably then a MemoryBIO backport could be pure-Python. It doesn't help the other *ythons, but unless they have MemoryBIO implementations ready to backport then I can't think of anything that will help them short of a completely new API. Cheers, Steve From donald at stufft.io Thu Jun 8 16:17:07 2017 From: donald at stufft.io (Donald Stufft) Date: Thu, 8 Jun 2017 16:17:07 -0400 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <40d4782d-a4de-8cb1-cf5b-a4ad7d30f9c2@python.org> References: <6E697036-ECBC-452F-BCF5-09A8F0A51375@lukasa.co.uk> <2E0E1D87-4AB3-42B8-9788-AED19B64DE1E@stufft.io> <07e12379-84e0-b519-22a2-1a75c4d958b6@python.org> <8287CE3C-687E-4BE6-9747-39E651B7178B@stufft.io> <40d4782d-a4de-8cb1-cf5b-a4ad7d30f9c2@python.org> Message-ID: <2F297FAD-EFFF-4B26-BABE-C0EE86FDBB94@stufft.io> > On Jun 8, 2017, at 3:57 PM, Steve Dower wrote: > > On 08Jun2017 1237, Donald Stufft wrote: >> The basic yak stak here is: >> * PEP 543 should be the future, it is a much much better way of handling TLS than our current ssl module is. >> * Cory can?t spend his work time on PEP 543 unless he can say it is useful for requests. >> * In order for PEP 543 to be useful for requests, he needs a way to provide a backport for it for Python 2.7. >> * This backport *CAN* be OpenSSL only, but needs to be able to provide the same API. >> * PEP 543 wants to work with MemoryBIOs instead of sockets, because a MemoryBio is a much much better way of implementing this problem for a variety of reasons, and it would be a mistake to use a socket primitive again. >> * Indepently, requests also wants to be able to provide the ability for people to use it with asyncio, however it can?t drop support for Python 2.7 in the quest for doing that. Twisted provides a way forward that lets requests work on both 2.x and 3.x and integrate with asyncio, but Twisted requires MemoryBio to do so. >> * pyOpenSSL *could* be used to provide the MemoryBio needed on 2.7 for both cases from up above, however, pip cannot depend on a C library that isn?t part of the standard library - in addition this would break alternative runtimes like Jython where pyOpenSSL doesn?t work. > > Awesome, this is exactly what I needed to see. > > What if Python 2.7 just exposed the OpenSSL primitives necessary so that ctypes could use them? Is that a viable approach here? Presumably then a MemoryBIO backport could be pure-Python. > > It doesn't help the other *ythons, but unless they have MemoryBIO implementations ready to backport then I can't think of anything that will help them short of a completely new API. > I would have to let Cory answer the question about feasibility here since he?s much more familiar with OpenSSL?s API (and even binding something like this with ctypes) than I am. The first thing that really stands out to me though is it just feels a bit like shuffling deckchairs? At least I don?t see what adding the new feature of ?thing you can use ctypes with to get a MemoryBio? does that adding the ssl.MemoryBio feature doesn?t other than downsides: * It creates yet another divergence between 3.x and 2.x that makes it harder to maintain ssl and _ssl. * It?s harder to use for requests/Twisted/etc. * It?s not already written (Ok this is minor, but still!). What do you see the benefits of that approach being over just backporting ssl.MemoryBio? ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Thu Jun 8 16:49:49 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 8 Jun 2017 21:49:49 +0100 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <2F297FAD-EFFF-4B26-BABE-C0EE86FDBB94@stufft.io> References: <6E697036-ECBC-452F-BCF5-09A8F0A51375@lukasa.co.uk> <2E0E1D87-4AB3-42B8-9788-AED19B64DE1E@stufft.io> <07e12379-84e0-b519-22a2-1a75c4d958b6@python.org> <8287CE3C-687E-4BE6-9747-39E651B7178B@stufft.io> <40d4782d-a4de-8cb1-cf5b-a4ad7d30f9c2@python.org> <2F297FAD-EFFF-4B26-BABE-C0EE86FDBB94@stufft.io> Message-ID: <2951782D-6851-43A4-8E93-B72471F7731E@lukasa.co.uk> > On 8 Jun 2017, at 21:17, Donald Stufft wrote: > >> >> On Jun 8, 2017, at 3:57 PM, Steve Dower > wrote: >> >> Awesome, this is exactly what I needed to see. >> >> What if Python 2.7 just exposed the OpenSSL primitives necessary so that ctypes could use them? Is that a viable approach here? Presumably then a MemoryBIO backport could be pure-Python. >> >> It doesn't help the other *ythons, but unless they have MemoryBIO implementations ready to backport then I can't think of anything that will help them short of a completely new API. >> > > > I would have to let Cory answer the question about feasibility here since he?s much more familiar with OpenSSL?s API (and even binding something like this with ctypes) than I am. The first thing that really stands out to me though is it just feels a bit like shuffling deckchairs? The short answer is that, while it?s do-able, we have some problems with ABI compatibility. OpenSSL 1.1 and 1.0 are ABI incompatible, so I have to write divergent ctypes code to handle each case. It may also be relevant to support OpenSSL 0.9.x so we roll into the same ABI compatibility concern all over again. Doubly annoyingly a bunch of OpenSSL code in 1.0 is actually macros which don?t work in ctypes so there?ll be a lot of futzing about in structures to get what I need to do done. This also doesn?t get into the difficulty of some operating systems shipping a LibreSSL masquerading as an OpenSSL, which is subtly incompatible in ways I don?t fully understand at this time. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Thu Jun 8 19:55:19 2017 From: steve.dower at python.org (Steve Dower) Date: Thu, 8 Jun 2017 16:55:19 -0700 Subject: [Python-Dev] Extracting python bytecode from a linux core dump? In-Reply-To: <20170607194720.GA18806@coffeebird.net> References: <20170607194720.GA18806@coffeebird.net> Message-ID: I can?t help with the gdb commands, but I?d suggest you want to start from one of the global variables for the sys module, probably the modules dict. You?ll have to reverse engineer the memory structures to find its values and each of their dicts, eventually finding function objects pointing to code objects which will point to bytecode blobs. All of these structures are in the sources, so it shouldn?t be that hard, just time consuming (I?ve done it on Windows before with different tools). If you know the code was being executed when the dump was made you could look at the stack to find calls in the EvalFrame function. Those should have a local or argument to the code object or bytecode (my memory on names and structures in 2.6 isn?t that good). A final alternative would be to find the function type object address and search memory for that to locate function objects and then code objects. That might be the best one, if you can locate the type object in the dump. Hope that helps, Steve Top-posted from my Windows phone From: CompuTinker Sent: Wednesday, June 7, 2017 13:26 To: python-dev at python.org Subject: [Python-Dev] Extracting python bytecode from a linux core dump? I hope this is the proper place for internals questions... I have a core file (produced via the gcore command) of a linux python2.6 process. I need to extract the byte code and de-compile it. I looked at https://wiki.python.org/moin/DebuggingWithGdb and related pages. However, these all seem to require either a running process and/or a binary with debugging symbols. I'm thinking that the compiled bytecode is likely in an array or contiguous set of memory within the python executable's image and that there's probably a way to pull it out with gdb. Unsurprisingly, the pyc 0xd1f20d0a magic number isn't kept in memory. So, how do I find the memory holding the compiled byte-code ? _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From nad at python.org Thu Jun 8 23:34:45 2017 From: nad at python.org (Ned Deily) Date: Thu, 8 Jun 2017 23:34:45 -0400 Subject: [Python-Dev] IMPORTANT: Python 3.6.2 Maintenance Release Release Candidate in 3+ days (Monday 2017-06-12 12:00 UTC) Message-ID: <9F658EB5-9DF3-4E1F-87A4-41ECF303924F@python.org> We are approaching the end of the second calendar quarter of 2017 and, according to PEP 494, it's time to start producing the second maintenance release for the 3.6 series. The schedule calls for the release candidate to be produced on Monday 2017-06-12 UTC. As was the case with previous 3.6.x releases, the plan is for the release candidate to be the same as the final release, that is, no additional changes go in after the release candidate except for any showstopper critical problems that might be discovered with rc1. So please plan to get any security fixes, bug fixes, and documentation changes you think should be in 3.6.2 merged in ASAP. The 3.6.2 final is planned for two weeks following rc1, that is, on 2017-06-26. The next 3.6 maintenance release (3.6.3) is planned to follow about 3 months later, so most likely in 2017-09. A reminder that the 3.6 branch will remain open for checkins throughout the 3.6.2rc and final cycle. Once 3.6.2rc1 is tagged, new commits to the 3.6 branch will release with 3.6.3. As always, if you find any problem in 3.6.2rc1 that you may believe should be corrected in 3.6.2, please ensure the problem is documented in a new or existing open issue on bugs.python.org, ensure that the Priority field of the issue is set to "release blocker", and that "Python 3.6" is included in the selected Versions. Comments or tags on github Pull Requests are NOT sufficient! Thanks again for all of your efforts in bringing 3.6 into the world and for helping now to make it even better! https://www.python.org/dev/peps/pep-0494/ http://cpython-devguide.readthedocs.io -- Ned Deily nad at python.org -- [] From ncoghlan at gmail.com Fri Jun 9 06:10:59 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 9 Jun 2017 20:10:59 +1000 Subject: [Python-Dev] 2017 Python Language Summit coverage finale In-Reply-To: <20170608095253.3444f4af@redtail> References: <20170608095253.3444f4af@redtail> Message-ID: On 9 June 2017 at 01:52, Jake Edge wrote: > > Hola python-dev, > > I (finally!) finished up the coverage from the language summit ... > > The starting point is the overview article, here: > https://lwn.net/Articles/723251/ which should now be free for anyone to > see (and the first seven articles too). Thanks once again for these - very helpful for those of us that weren't able to attend! Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Fri Jun 9 06:43:16 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 9 Jun 2017 20:43:16 +1000 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <8932A107-0D8C-4B6C-A32F-8861D7B19EF1@stufft.io> References: <6E697036-ECBC-452F-BCF5-09A8F0A51375@lukasa.co.uk> <2E0E1D87-4AB3-42B8-9788-AED19B64DE1E@stufft.io> <07e12379-84e0-b519-22a2-1a75c4d958b6@python.org> <8287CE3C-687E-4BE6-9747-39E651B7178B@stufft.io> <8932A107-0D8C-4B6C-A32F-8861D7B19EF1@stufft.io> Message-ID: On 9 June 2017 at 05:51, Donald Stufft wrote: > And the downside here is pretty small honestly? Unfortunately, one of the downsides is "Doesn't provide any clearly compelling benefits to users of LTS distributions, so even those of us in a position to advocate for such backports probably won't win the related arguments". So the question is whether or not we're OK with saying that users affected by that will need to switch to a different set of Python binaries to get the latest pip & requests (e.g. upgrading their base distro, or else adopting Red Hat SCLs or one of the cross-platform, and hence cross-distro, distributions). > It?s not changing behavior of anything that currently exists since it?s > adding a new thing inside the ssl module and Alex has already written the > patch so there?s little extra work to do and it actually makes maintenance > easier since patches can more readily be applied straight from `master`. > > The primary argument I can see against it, is one of purity, that 2.7 > shouldn?t get new features but as we know, practicality beats purity ;) (and > we?ve already accepted that TLS is a special case, special enough to break > the rules, so the main question is whether this specific thing is > worthwhile? which given it?s benefits to the Python ecosystem and to > maintaining Python, I think it is). It isn't a purity argument so much as a "Will the change reach a sufficiently large proportion of 2.7 users to actually solve the compatibility problem it is aiming to solve?" argument (There's a *bit* of a purity argument behind that, in that the PEP 466 backport *did* break customer applications due to the incompatibilities with some of the async frameworks that were using underscore prefixed APIs that changed behaviour, so our credibility for "trust us, it won't break anything" is low in this context, but it's not the primary objection) I'm 100% confident that we'd reach a large enough audience with a compatibility shim that gets installed on disk as something other than "ssl" (e.g. "backport.ssl" or "_ssl_bootstrap"), and that has the virtue of enabling a multi-tiered distribution approach: - backport.ssl on PyPI as an independently installed module - _ssl_bootstrap as a runtime or redistributor provided module This approach also allows the API to be updated as necessary to meet the needs of PEP 543, without needing to go through the full PEP process again. The downside is adding a 3rd stack to maintain (Py3, Py2-compatible Py3 backport, Py2) without making maintenance on the Py2 stack any easier. I'm markedly less confident of reaching a sufficiently large audience with a "So stop using the system Python, then" approach (Don't get me wrong, I'd love for people to stop using the system Python for things that aren't part of the operating system, but people also *really like* the convenience of doing that). OTOH, if the aim is to make the change now, so it gets into Ubuntu 18.04, with a view to projects only starting to fully rely on it in mid-to-late 2018 or so? That timeline might actually work, and this approach has the benefit of actually making the Py2 SSL stack easier to maintain between now and 2020. So honestly, I'd be +1 for either approach: - stdlib backport to make dual-stack maintenance easier for the current volunteers, and we'll see how things work out on the ease-of-adoption front - PyPI backport to make 2.7 adoption easier, and we'll continue pestering redistributors to actually fund maintenance of Python 2.7's SSL stack properly (and encourage customers of those redistributors to do the same) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From victor.stinner at gmail.com Fri Jun 9 07:30:51 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 9 Jun 2017 13:30:51 +0200 Subject: [Python-Dev] [python-committers] IMPORTANT: Python 3.6.2 Maintenance Release Release Candidate in 3+ days (Monday 2017-06-12 12:00 UTC) In-Reply-To: <9F658EB5-9DF3-4E1F-87A4-41ECF303924F@python.org> References: <9F658EB5-9DF3-4E1F-87A4-41ECF303924F@python.org> Message-ID: Oh, about very annoying 3.6 bug, there was a regression caused by FASTCALL optimizations. It's now fixed in the 3.6 branch: https://github.com/python/cpython/commit/f0ff849adc6b4a01f9d1f08d9ad0f1511ff84541 Victor 2017-06-09 5:34 GMT+02:00 Ned Deily : > We are approaching the end of the second calendar quarter of 2017 and, according to PEP 494, it's time to start producing the second maintenance release for the 3.6 series. The schedule calls for the release candidate to be produced on Monday 2017-06-12 UTC. As was the case with previous 3.6.x releases, the plan is for the release candidate to be the same as the final release, that is, no additional changes go in after the release candidate except for any showstopper critical problems that might be discovered with rc1. So please plan to get any security fixes, bug fixes, and documentation changes you think should be in 3.6.2 merged in ASAP. The 3.6.2 final is planned for two weeks following rc1, that is, on 2017-06-26. The next 3.6 maintenance release (3.6.3) is planned to follow about 3 months later, so most likely in 2017-09. > > A reminder that the 3.6 branch will remain open for checkins throughout the 3.6.2rc and final cycle. Once 3.6.2rc1 is tagged, new commits to the 3.6 branch will release with 3.6.3. As always, if you find any problem in 3.6.2rc1 that you may believe should be corrected in 3.6.2, please ensure the problem is documented in a new or existing open issue on bugs.python.org, ensure that the Priority field of the issue is set to "release blocker", and that "Python 3.6" is included in the selected Versions. Comments or tags on github Pull Requests are NOT sufficient! > > Thanks again for all of your efforts in bringing 3.6 into the world and for helping now to make it even better! > > https://www.python.org/dev/peps/pep-0494/ > http://cpython-devguide.readthedocs.io > > -- > Ned Deily > nad at python.org -- [] > > _______________________________________________ > python-committers mailing list > python-committers at python.org > https://mail.python.org/mailman/listinfo/python-committers > Code of Conduct: https://www.python.org/psf/codeofconduct/ From victor.stinner at gmail.com Fri Jun 9 08:43:06 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 9 Jun 2017 14:43:06 +0200 Subject: [Python-Dev] Remove embedded expat library? Message-ID: Hi, Python embeds a copy of the expat library which already got two major security vulnerabilities: "CVE-2016-0718: expat bug #537" http://python-security.readthedocs.io/vuln/cve-2016-0718_expat_bug_537.html "Issue #26556: Expat 2.1.1" http://python-security.readthedocs.io/vuln/issue_26556_expat_2.1.1.html Would it be possible to maintain this dependency on an external repository which would be easier to maintain? Like http://svn.python.org/projects/external/ used to build Python on Windows. I expect that all Linux distributions build Python using --with-system-expat. It may become the default? What about macOS and other operating systems? By the way, Zachary Ware is working on converting this repository to Git. I don't know his progress: - https://github.com/python/cpython-bin-deps - https://github.com/python/cpython-source-deps Victor From guido at python.org Fri Jun 9 11:14:55 2017 From: guido at python.org (Guido van Rossum) Date: Fri, 9 Jun 2017 08:14:55 -0700 Subject: [Python-Dev] 2017 Python Language Summit coverage finale In-Reply-To: References: <20170608095253.3444f4af@redtail> Message-ID: Also a good summary for those of us that *did* attend! Thanks Jake from all participants. (And Nick, I hope you won't make a habit of skipping the summit. :-) On Fri, Jun 9, 2017 at 3:10 AM, Nick Coghlan wrote: > On 9 June 2017 at 01:52, Jake Edge wrote: > > > > Hola python-dev, > > > > I (finally!) finished up the coverage from the language summit ... > > > > The starting point is the overview article, here: > > https://lwn.net/Articles/723251/ which should now be free for anyone to > > see (and the first seven articles too). > > Thanks once again for these - very helpful for those of us that > weren't able to attend! > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From nad at python.org Fri Jun 9 12:03:26 2017 From: nad at python.org (Ned Deily) Date: Fri, 9 Jun 2017 12:03:26 -0400 Subject: [Python-Dev] Remove embedded expat library? In-Reply-To: References: Message-ID: On Jun 9, 2017, at 08:43, Victor Stinner wrote: > Python embeds a copy of the expat library which already got two major > security vulnerabilities: > > "CVE-2016-0718: expat bug #537" > http://python-security.readthedocs.io/vuln/cve-2016-0718_expat_bug_537.html > > "Issue #26556: Expat 2.1.1" > http://python-security.readthedocs.io/vuln/issue_26556_expat_2.1.1.html > > Would it be possible to maintain this dependency on an external > repository which would be easier to maintain? Like > http://svn.python.org/projects/external/ used to build Python on > Windows. Considering how infrequently there have been releases of upstream expat over the past decade or so, even if we had caught every release, I don't see that the benefit to maintaining our embedded copy as a separate repo rather than have to apply the upstreams copies a few time (to each Python branch) is worth the disruptive effort in changing the current Python build process (assuming we don't change the default, see below). Now, as to whether we should continue to provide our own copy of expat, that's a different question. Are there differences between our bundled expat and upstream? > I expect that all Linux distributions build Python using > --with-system-expat. It may become the default? What about macOS and > other operating systems? The current default is --with-system-expat=no so, unless builders of Python take explicit action, the bundled version of expat is used. Using the bundled version is also currently the case for the python.org macOS installer, no idea what other distributors do. Apple supplies a version of expat with macOS so we presumably we could use the system version for the installer. Presumably (Zach?) we would need to continue to supply a version of expat for Windows builds. But do we need to for others? If it were only Windows, *then* perhaps it might make sense to make all the changes to move expat out of cpython into the common repo for third-party Windows libs. > By the way, Zachary Ware is working on converting this repository to > Git. I don't know his progress: > - https://github.com/python/cpython-bin-deps > - https://github.com/python/cpython-source-deps It appears that the project itself is moving from Sourceforge to Github: https://github.com/libexpat/libexpat -- Ned Deily nad at python.org -- [] From status at bugs.python.org Fri Jun 9 12:09:19 2017 From: status at bugs.python.org (Python tracker) Date: Fri, 9 Jun 2017 18:09:19 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20170609160919.365D711A869@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2017-06-02 - 2017-06-09) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 6035 (+10) closed 36335 (+51) total 42370 (+61) Open issues with patches: 2369 Issues opened (40) ================== #30555: _io._WindowsConsoleIO breaks in the face of fd redirection http://bugs.python.org/issue30555 opened by Segev Finer #30556: asyncio.wait very slow with FIRST_COMPLETED http://bugs.python.org/issue30556 opened by samuelcolvin #30560: Py_SetFatalErrorAbortFunc: Allow embedding program to handle f http://bugs.python.org/issue30560 opened by thomas.perl #30561: sync-up gammavariate and expovariate code http://bugs.python.org/issue30561 opened by leodema #30563: multiprocessing module with pool object issue http://bugs.python.org/issue30563 opened by Julien Verger #30565: PEP 538: default to skipping warning for implicit locale coerc http://bugs.python.org/issue30565 opened by ncoghlan #30566: punycode codec raises IndexError in decode_generalized_number( http://bugs.python.org/issue30566 opened by Vikram Hegde #30569: Tutorial section 2.1 has *nix example at 3.7, but Windows at 3 http://bugs.python.org/issue30569 opened by csabella #30570: issubclass segfaults on objects with weird __getattr__ http://bugs.python.org/issue30570 opened by Daniel Lepage #30571: Add integer formatting code for fixed-width signed arithmetic http://bugs.python.org/issue30571 opened by rhettinger #30576: http.server should support HTTP compression (gzip) http://bugs.python.org/issue30576 opened by quentel #30578: Misleading example in sys.set_coroutine_wrapper docs http://bugs.python.org/issue30578 opened by ncoghlan #30579: Allow traceback objects to be instantiated/mutated/annotated http://bugs.python.org/issue30579 opened by njs #30581: os.cpu_count() returns wrong number of processors on system wi http://bugs.python.org/issue30581 opened by robbuckley #30585: [security][3.3] Backport smtplib fix for TLS stripping vulnera http://bugs.python.org/issue30585 opened by haypo #30586: Encode to EBCDIC doesn't take into account conversion table ir http://bugs.python.org/issue30586 opened by Vladimir Filippov #30587: Mock with spec object does not ensure method call signatures http://bugs.python.org/issue30587 opened by cbelu #30588: Missing documentation for codecs.escape_decode http://bugs.python.org/issue30588 opened by mdartiailh #30589: With forkserver, Process.exitcode does not get signal number http://bugs.python.org/issue30589 opened by pitrou #30593: sqlite3 executescript does not respect isolation_level? http://bugs.python.org/issue30593 opened by Noah Levitt #30594: Refcounting mistake in _ssl.c http://bugs.python.org/issue30594 opened by njs #30595: test_queue_feeder_donot_stop_onexc() of test_multiprocessing_s http://bugs.python.org/issue30595 opened by haypo #30596: Add close() to multiprocessing.Process http://bugs.python.org/issue30596 opened by pitrou #30597: Show expected input in custom "print" error message http://bugs.python.org/issue30597 opened by ncoghlan #30598: Py_NewInterpreter() leaks a reference on warnoptions in _PySys http://bugs.python.org/issue30598 opened by haypo #30599: os.register_at_fork(): allow to unregister callbacks -- test_t http://bugs.python.org/issue30599 opened by haypo #30600: Error message incorrect when index is called with keyword argu http://bugs.python.org/issue30600 opened by SylvainDe #30602: [Windows] os.spawn*() tests of test_os leak references on Wind http://bugs.python.org/issue30602 opened by haypo #30603: textwrap: declining indent level has no test case http://bugs.python.org/issue30603 opened by jonathaneunice #30604: co_extra_freefuncs is stored thread locally and can lead to cr http://bugs.python.org/issue30604 opened by dino.viehland #30605: re.compile fails when compiling bytes under `-bb` mode http://bugs.python.org/issue30605 opened by Roy Williams #30607: Extract documentation theme into a separate package http://bugs.python.org/issue30607 opened by Jon Wayne Parrott #30608: argparse calculates string widths incorrectly http://bugs.python.org/issue30608 opened by Vanessa McHale #30609: Python 3.6.1 fails to generate 256 colors on Cygwin based 64-b http://bugs.python.org/issue30609 opened by rigordo #30610: Python's libexpat vulnerable to CVE-2016-0718 http://bugs.python.org/issue30610 opened by Duy Phan Thanh #30611: Windows HTML Help always opens maximized http://bugs.python.org/issue30611 opened by Christian.Ullrich #30612: Unusual Windows registry path syntax http://bugs.python.org/issue30612 opened by Christian.Ullrich #30613: Distutils register command creates non-standard multipart data http://bugs.python.org/issue30613 opened by Kaeptm Blaubaer #30614: [EASY][2.7] testInitNonExistentFile() of test_bz2 leaks refere http://bugs.python.org/issue30614 opened by haypo #30615: [EASY][2.7] test_recursive_repr() of test_xml_etree_c leaks re http://bugs.python.org/issue30615 opened by haypo Most recent 15 issues with no replies (15) ========================================== #30615: [EASY][2.7] test_recursive_repr() of test_xml_etree_c leaks re http://bugs.python.org/issue30615 #30614: [EASY][2.7] testInitNonExistentFile() of test_bz2 leaks refere http://bugs.python.org/issue30614 #30613: Distutils register command creates non-standard multipart data http://bugs.python.org/issue30613 #30612: Unusual Windows registry path syntax http://bugs.python.org/issue30612 #30607: Extract documentation theme into a separate package http://bugs.python.org/issue30607 #30603: textwrap: declining indent level has no test case http://bugs.python.org/issue30603 #30593: sqlite3 executescript does not respect isolation_level? http://bugs.python.org/issue30593 #30589: With forkserver, Process.exitcode does not get signal number http://bugs.python.org/issue30589 #30587: Mock with spec object does not ensure method call signatures http://bugs.python.org/issue30587 #30578: Misleading example in sys.set_coroutine_wrapper docs http://bugs.python.org/issue30578 #30571: Add integer formatting code for fixed-width signed arithmetic http://bugs.python.org/issue30571 #30569: Tutorial section 2.1 has *nix example at 3.7, but Windows at 3 http://bugs.python.org/issue30569 #30565: PEP 538: default to skipping warning for implicit locale coerc http://bugs.python.org/issue30565 #30563: multiprocessing module with pool object issue http://bugs.python.org/issue30563 #30561: sync-up gammavariate and expovariate code http://bugs.python.org/issue30561 Most recent 15 issues waiting for review (15) ============================================= #30605: re.compile fails when compiling bytes under `-bb` mode http://bugs.python.org/issue30605 #30604: co_extra_freefuncs is stored thread locally and can lead to cr http://bugs.python.org/issue30604 #30589: With forkserver, Process.exitcode does not get signal number http://bugs.python.org/issue30589 #30560: Py_SetFatalErrorAbortFunc: Allow embedding program to handle f http://bugs.python.org/issue30560 #30519: Add daemon argument to Timer http://bugs.python.org/issue30519 #30512: CAN Socket support for NetBSD http://bugs.python.org/issue30512 #30509: Optimize calling type slots http://bugs.python.org/issue30509 #30502: Fix buffer handling of OBJ_obj2txt http://bugs.python.org/issue30502 #30501: Produce optimized code for boolean conditions http://bugs.python.org/issue30501 #30495: IDLE: modernize textview module http://bugs.python.org/issue30495 #30487: DOC: automatically create a venv and install Sphinx when runni http://bugs.python.org/issue30487 #30485: Element.findall(path, dict) doesn't insert null namespace http://bugs.python.org/issue30485 #30466: Tutorial doesn't explain the use of classes http://bugs.python.org/issue30466 #30465: FormattedValue expressions have wrong lineno and col_offset in http://bugs.python.org/issue30465 #30455: Generate C code from token.py and not vice versa http://bugs.python.org/issue30455 Top 10 most discussed issues (10) ================================= #25910: Fixing links in documentation http://bugs.python.org/issue25910 12 msgs #27321: Email parser creates a message object that can't be flattened http://bugs.python.org/issue27321 9 msgs #30542: test_files() of test_tools.test_unparse.DirectoryTestCase leak http://bugs.python.org/issue30542 8 msgs #30417: [buildbot] Disable the cpu resources on slow buildbots http://bugs.python.org/issue30417 7 msgs #30597: Show expected input in custom "print" error message http://bugs.python.org/issue30597 7 msgs #26103: Contradiction in definition of "data descriptor" between (dott http://bugs.python.org/issue26103 6 msgs #29464: Specialize FASTCALL for functions with positional-only paramet http://bugs.python.org/issue29464 6 msgs #30038: Race condition in how trip_signal writes to wakeup fd http://bugs.python.org/issue30038 6 msgs #30599: os.register_at_fork(): allow to unregister callbacks -- test_t http://bugs.python.org/issue30599 6 msgs #30610: Python's libexpat vulnerable to CVE-2016-0718 http://bugs.python.org/issue30610 6 msgs Issues closed (50) ================== #19180: some RFC references could be updated http://bugs.python.org/issue19180 closed by ncoghlan #21783: smtpd.py does not allow multiple helo/ehlo commands http://bugs.python.org/issue21783 closed by barry #24755: asyncio.wrap_future undocumented http://bugs.python.org/issue24755 closed by Mariatta #24899: Add an os.path <=> pathlib equivalence table in pathlib docs http://bugs.python.org/issue24899 closed by brett.cannon #25324: Importing tokenize modifies token http://bugs.python.org/issue25324 closed by serhiy.storchaka #25444: Py Launch Icon http://bugs.python.org/issue25444 closed by eryksun #29596: Unfinished sentence in howto/clinic.rst http://bugs.python.org/issue29596 closed by Mariatta #29822: inspect.isabstract does not work on abstract base classes duri http://bugs.python.org/issue29822 closed by serhiy.storchaka #30052: URL Quoting page links to function Bytes instead of defintion http://bugs.python.org/issue30052 closed by Mariatta #30095: HTMLCalendar allow custom classes http://bugs.python.org/issue30095 closed by doerwalter #30177: pathlib.resolve(strict=False) only includes first child http://bugs.python.org/issue30177 closed by steve.dower #30418: test_communicate_epipe() of test_subprocess fails randomly on http://bugs.python.org/issue30418 closed by haypo #30463: Add __slots__ to ABC convenience class http://bugs.python.org/issue30463 closed by serhiy.storchaka #30486: Allow setting cell value http://bugs.python.org/issue30486 closed by serhiy.storchaka #30489: Add CmdLineTest to standard library http://bugs.python.org/issue30489 closed by terry.reedy #30518: Import type aliases from another module http://bugs.python.org/issue30518 closed by Paragape #30520: loggers can't be pickled http://bugs.python.org/issue30520 closed by vinay.sajip #30526: Allow setting line_buffering on existing TextIOWrapper http://bugs.python.org/issue30526 closed by pitrou #30529: Incorrect error messages for invalid whitespaces in f-string s http://bugs.python.org/issue30529 closed by serhiy.storchaka #30530: Descriptors HowTo: Example on function.__get__ needs update http://bugs.python.org/issue30530 closed by Mariatta #30534: error message for incorrect call degraded in 3.7 http://bugs.python.org/issue30534 closed by serhiy.storchaka #30536: [EASY] SubinterpThreadingTests.test_threads_join_2() of test_t http://bugs.python.org/issue30536 closed by haypo #30537: Using PyNumber_AsSsize_t in itertools.islice http://bugs.python.org/issue30537 closed by rhettinger #30538: Functional Programming HOWTO describes one argument itertools. http://bugs.python.org/issue30538 closed by rhettinger #30540: regrtest: add --matchfile option http://bugs.python.org/issue30540 closed by haypo #30544: _io._WindowsConsoleIO.write raises the wrong error when WriteC http://bugs.python.org/issue30544 closed by steve.dower #30546: [EASY][Windows] test_uname_win32_ARCHITEW6432() of test_platfo http://bugs.python.org/issue30546 closed by haypo #30547: [Windows] SubinterpreterTest.test_callbacks_leak() of test_ate http://bugs.python.org/issue30547 closed by haypo #30554: Inaccessible attribute characters_written on OSError instances http://bugs.python.org/issue30554 closed by xiang.zhang #30557: faulthandler does not correctly filter fatal exceptions on Win http://bugs.python.org/issue30557 closed by steve.dower #30558: [Suggestion] Improve documentation for set() API http://bugs.python.org/issue30558 closed by martin.panter #30559: Bugs in Web app http://bugs.python.org/issue30559 closed by mark.dickinson #30562: SSL socket does not respect SO_RCVTIME0 timeouts http://bugs.python.org/issue30562 closed by christian.heimes #30564: Base64 decoding gives incorrect outputs. http://bugs.python.org/issue30564 closed by r.david.murray #30567: Leak in sys.getwindowsversion http://bugs.python.org/issue30567 closed by serhiy.storchaka #30568: README Formatting http://bugs.python.org/issue30568 closed by Mariatta #30572: pip is broken http://bugs.python.org/issue30572 closed by Dingo64 #30573: How to install cmake? http://bugs.python.org/issue30573 closed by Dingo64 #30574: Document why embedding the interpreter can sometimes crash. http://bugs.python.org/issue30574 closed by r.david.murray #30575: Python interpreter crashes on macOS http://bugs.python.org/issue30575 closed by Yibo Wang #30577: Multidimensional comprehensions cannot access class variables http://bugs.python.org/issue30577 closed by zach.ware #30580: WSGI examples raise AttributeError: __exit__ http://bugs.python.org/issue30580 closed by r.david.murray #30582: Incorrect propagation in logging.wheb creating lighters in a c http://bugs.python.org/issue30582 closed by vinay.sajip #30583: docs mention datetuil presumably it should be dateutil http://bugs.python.org/issue30583 closed by Mariatta #30584: test_os fails on non-English (Russian) Windows http://bugs.python.org/issue30584 closed by denis-osipov #30590: str.format no longer accepts unpacked defaultdict for default http://bugs.python.org/issue30590 closed by serhiy.storchaka #30591: textwrap: placeholder backtracking special case lacks test cov http://bugs.python.org/issue30591 closed by serhiy.storchaka #30592: Bad error message 'bool()() takes no keyword arguments' http://bugs.python.org/issue30592 closed by serhiy.storchaka #30601: [Windows] test_winconsoleio leaks references http://bugs.python.org/issue30601 closed by haypo #30606: Spam http://bugs.python.org/issue30606 closed by berker.peksag From donald at stufft.io Fri Jun 9 12:35:56 2017 From: donald at stufft.io (Donald Stufft) Date: Fri, 9 Jun 2017 12:35:56 -0400 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <6E697036-ECBC-452F-BCF5-09A8F0A51375@lukasa.co.uk> <2E0E1D87-4AB3-42B8-9788-AED19B64DE1E@stufft.io> <07e12379-84e0-b519-22a2-1a75c4d958b6@python.org> <8287CE3C-687E-4BE6-9747-39E651B7178B@stufft.io> <8932A107-0D8C-4B6C-A32F-8861D7B19EF1@stufft.io> Message-ID: <563F18F1-A9C9-4997-8346-6397C49C227F@stufft.io> > On Jun 9, 2017, at 6:43 AM, Nick Coghlan wrote: > > OTOH, if the aim is to make the change now, so it gets into Ubuntu > 18.04, with a view to projects only starting to fully rely on it in > mid-to-late 2018 or so? That timeline might actually work, and this > approach has the benefit of actually making the Py2 SSL stack easier > to maintain between now and 2020. Cory can correct me if I?m wrong, but yea that?s the general idea here. We need a tree, and the best time to plant a tree is 20 years ago, but the second best time to plant a tree is today ;) I don?t think anyone is going to immediately start depending on a hypothetical 2.7.14 with MemoryBio support, but getting it out now means that Cory can start working on PEP 543 as part of a longer term effort for requests as well as starting to work on async-ifying requests. Neither of those efforts are going to be quick or simple tasks, so the work to make them possible can happen while we wait for adoption to naturally grow on 2.7.14. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachary.ware+pydev at gmail.com Fri Jun 9 12:40:16 2017 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Fri, 9 Jun 2017 11:40:16 -0500 Subject: [Python-Dev] Remove embedded expat library? In-Reply-To: References: Message-ID: On Fri, Jun 9, 2017 at 11:03 AM, Ned Deily wrote: > On Jun 9, 2017, at 08:43, Victor Stinner wrote: >> I expect that all Linux distributions build Python using >> --with-system-expat. It may become the default? What about macOS and >> other operating systems? > > The current default is --with-system-expat=no so, unless builders of Python take explicit action, the bundled version of expat is used. Using the bundled version is also currently the case for the python.org macOS installer, no idea what other distributors do. Apple supplies a version of expat with macOS so we presumably we could use the system version for the installer. Presumably (Zach?) we would need to continue to supply a version of expat for Windows builds. But do we need to for others? If it were only Windows, *then* perhaps it might make sense to make all the changes to move expat out of cpython into the common repo for third-party Windows libs. Yes, we would need to continue providing a version for Windows. It would be a relatively small change to move it to the externals repository. I would be fine with switching to `--with-system-expat=yes` by default and building from externals on Windows in 3.7, and removing the bundled expat in 3.8. >> By the way, Zachary Ware is working on converting this repository to >> Git. I don't know his progress: >> - https://github.com/python/cpython-bin-deps >> - https://github.com/python/cpython-source-deps PR 1783 (https://github.com/python/cpython/pull/1783); needs another review from Steve to make sure I haven't made a complete mess of things, then it should be ready to go. Anyone else on Windows (Terry Reedy?) who can test it and provide feedback, please do! -- Zach From steve.dower at python.org Fri Jun 9 13:07:21 2017 From: steve.dower at python.org (Steve Dower) Date: Fri, 9 Jun 2017 10:07:21 -0700 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <6E697036-ECBC-452F-BCF5-09A8F0A51375@lukasa.co.uk> <2E0E1D87-4AB3-42B8-9788-AED19B64DE1E@stufft.io> <07e12379-84e0-b519-22a2-1a75c4d958b6@python.org> <8287CE3C-687E-4BE6-9747-39E651B7178B@stufft.io> <8932A107-0D8C-4B6C-A32F-8861D7B19EF1@stufft.io> Message-ID: <0a086a16-6c20-a721-e7a5-b4b17665b50b@python.org> On 09Jun2017 0343, Nick Coghlan wrote: > So honestly, I'd be +1 for either approach: > > - stdlib backport to make dual-stack maintenance easier for the > current volunteers, and we'll see how things work out on the > ease-of-adoption front > - PyPI backport to make 2.7 adoption easier, and we'll continue > pestering redistributors to actually fund maintenance of Python 2.7's > SSL stack properly (and encourage customers of those redistributors to > do the same) My draft reply to Donald sat overnight, so I abandoned it in favour of agreeing with Nick. I'm in principle in favour of anything that makes 2.7 less of a burden to maintain (up to and including EOL :) ), so if backporting parts of ssl/_ssl makes that easier then I'm +0. However, I do prefer the PyPI backport with some tool bundled in order to obtain it. In fact, given the nature of OpenSSL, I'd be in favour of that approach for all versions of Python (at least on Windows it would likely work well - probably less so on other platforms where we couldn't include a prebuilt fallback easily, though those tend to include compilers...). That hypothetical "_ensuressl" module in my mind really doesn't have to do much other than determine which file to download and then download and extract it, which can be done with OS level tools rather than needing our own stack. It may also be the necessary mechanism to make ssl pip-updateable, since we have the locking problem that prevents it being possible normally. Cheers, Steve From donald at stufft.io Fri Jun 9 14:18:11 2017 From: donald at stufft.io (Donald Stufft) Date: Fri, 9 Jun 2017 14:18:11 -0400 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <0a086a16-6c20-a721-e7a5-b4b17665b50b@python.org> References: <6E697036-ECBC-452F-BCF5-09A8F0A51375@lukasa.co.uk> <2E0E1D87-4AB3-42B8-9788-AED19B64DE1E@stufft.io> <07e12379-84e0-b519-22a2-1a75c4d958b6@python.org> <8287CE3C-687E-4BE6-9747-39E651B7178B@stufft.io> <8932A107-0D8C-4B6C-A32F-8861D7B19EF1@stufft.io> <0a086a16-6c20-a721-e7a5-b4b17665b50b@python.org> Message-ID: > On Jun 9, 2017, at 1:07 PM, Steve Dower wrote: > > On 09Jun2017 0343, Nick Coghlan wrote: >> So honestly, I'd be +1 for either approach: >> - stdlib backport to make dual-stack maintenance easier for the >> current volunteers, and we'll see how things work out on the >> ease-of-adoption front >> - PyPI backport to make 2.7 adoption easier, and we'll continue >> pestering redistributors to actually fund maintenance of Python 2.7's >> SSL stack properly (and encourage customers of those redistributors to >> do the same) > > My draft reply to Donald sat overnight, so I abandoned it in favour of agreeing with Nick. > > I'm in principle in favour of anything that makes 2.7 less of a burden to maintain (up to and including EOL :) ), so if backporting parts of ssl/_ssl makes that easier then I'm +0. > > However, I do prefer the PyPI backport with some tool bundled in order to obtain it. In fact, given the nature of OpenSSL, I'd be in favour of that approach for all versions of Python (at least on Windows it would likely work well - probably less so on other platforms where we couldn't include a prebuilt fallback easily, though those tend to include compilers...). > > That hypothetical "_ensuressl" module in my mind really doesn't have to do much other than determine which file to download and then download and extract it, which can be done with OS level tools rather than needing our own stack. It may also be the necessary mechanism to make ssl pip-updateable, since we have the locking problem that prevents it being possible normally. > A ensuressl style module that tries to install an OpenSSL module is actually fairly hard to do securely. The fundamental issue being that fetching a file securely from the network before you have the primary tool for fetching things securely from a network gets to be a bit of a chicken and egg problem. There are *possible* ways around that, but they essentially boil down to having to ship something akin to a CA Bundle inside of Python which we just shift the responsibility from needing to have an updated OpenSSL in all versions of Python. The good news here is that PEP 543 is the thing that will enable using SChannel on Windows, SecureTransport on macOS, and OpenSSL on Linux getting python-dev out of the business of shipping an OpenSSL at all [1]. Backporting ssl.MemoryBio represents the smallest reasonable change to 2.7 we can make [2] that paves the path forward to being able to do that in some number of years, because we can?t do that until PEP 543 is a thing and people are using it. [1] Ok, the ``ssl`` module is still going to be ssl dependent, and there is a lot of code out there using it so we would need to keep it for awhile to maintain compatibility?. but at some point we can stop bundling it in our installers under the guise of ?we have a newer, better API for handling this, and if you still need 2.7 compatibility, here?s the pip installable libraries you need to install to get it?. [2] Where reasonable means, it?s not sacrificing good things or making things harder to use for the sake of being smaller. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Fri Jun 9 14:41:46 2017 From: barry at python.org (Barry Warsaw) Date: Fri, 9 Jun 2017 11:41:46 -0700 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <6E697036-ECBC-452F-BCF5-09A8F0A51375@lukasa.co.uk> <2E0E1D87-4AB3-42B8-9788-AED19B64DE1E@stufft.io> <07e12379-84e0-b519-22a2-1a75c4d958b6@python.org> <8287CE3C-687E-4BE6-9747-39E651B7178B@stufft.io> <8932A107-0D8C-4B6C-A32F-8861D7B19EF1@stufft.io> Message-ID: <20170609114146.00165bac@presto> On Jun 09, 2017, at 08:43 PM, Nick Coghlan wrote: >So honestly, I'd be +1 for either approach: > >- stdlib backport to make dual-stack maintenance easier for the >current volunteers, and we'll see how things work out on the >ease-of-adoption front As I've said, I'm okay with this approach as long as we don't expose new public APIs in 2.7. That won't prevent other folks from using the private APIs, but we have no responsibility to support them. >- PyPI backport to make 2.7 adoption easier, and we'll continue >pestering redistributors to actually fund maintenance of Python 2.7's >SSL stack properly (and encourage customers of those redistributors to >do the same) Of course, lots of distributions are completely voluntary, so that kind of pestering falls on underfunded ears. ;) But a PyPI backport might still be useful since those third party packages can be distro-fied into current and backported channels. -Barry From skip.montanaro at gmail.com Fri Jun 9 14:38:09 2017 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Fri, 9 Jun 2017 13:38:09 -0500 Subject: [Python-Dev] Extracting python bytecode from a linux core dump? In-Reply-To: <20170607194720.GA18806@coffeebird.net> References: <20170607194720.GA18806@coffeebird.net> Message-ID: > I have a core file (produced via the gcore command) of a linux python2.6 process. I need to extract the byte code and de-compile it. Following on Steve's comment, you might want to take a look at Misc/gdbinit for some GDB command inspiration. You are correct, you won't have a running process, but I think you should be able source that file (maybe with tweaks, depending on the Python version you are debugging), then move up and down the C call stack, poke around in the C locals, then follow pointers to the currently active functions, then for those which are Python functions, follow the func_code attribute to get the code object. I can't remember what the actual bytecode attribute is called in the code object. (It's been too many years.) > However, these all seem to require either a running process and/or a binary with debugging symbols. Yeah, you're going to have *a lot* of fun with a stripped executable. If you're debugging a core file from an interpreter compiled with much in the way of compiler optimization, many of the local variables will have been optimized out. You'll likely be stuck rummaging around until you figure out the pattern of where the compiler put things (register-wise). > I'm thinking that the compiled bytecode is likely in an array or contiguous set of memory within the python executable's image and that there's probably a way to pull it out with gdb. Unsurprisingly, the pyc 0xd1f20d0a magic number isn't kept in memory. So, how do I find the memory holding the compiled byte-code ? Correct. The module level bytecode is executed once at import time, then discarded, at least that used to be how it was done. Skip From steve.dower at python.org Fri Jun 9 15:41:33 2017 From: steve.dower at python.org (Steve Dower) Date: Fri, 9 Jun 2017 12:41:33 -0700 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <6E697036-ECBC-452F-BCF5-09A8F0A51375@lukasa.co.uk> <2E0E1D87-4AB3-42B8-9788-AED19B64DE1E@stufft.io> <07e12379-84e0-b519-22a2-1a75c4d958b6@python.org> <8287CE3C-687E-4BE6-9747-39E651B7178B@stufft.io> <8932A107-0D8C-4B6C-A32F-8861D7B19EF1@stufft.io> <0a086a16-6c20-a721-e7a5-b4b17665b50b@python.org> Message-ID: On 09Jun2017 1118, Donald Stufft wrote: > A ensuressl style module that tries to install an OpenSSL module is > actually fairly hard to do securely. The fundamental issue being that > fetching a file securely from the network before you have the primary > tool for fetching things securely from a network gets to be a bit of a > chicken and egg problem. There are *possible* ways around that, but they > essentially boil down to having to ship something akin to a CA Bundle > inside of Python which we just shift the responsibility from needing to > have an updated OpenSSL in all versions of Python. > The good news here is that PEP 543 is the thing that will enable using > SChannel on Windows, SecureTransport on macOS, and OpenSSL on Linux > getting python-dev out of the business of shipping an OpenSSL at all > [1]. Backporting ssl.MemoryBio represents the smallest reasonable change > to 2.7 we can make [2] that paves the path forward to being able to do > that in some number of years, because we can?t do that until PEP 543 is > a thing and people are using it. One of the reasons I'm wanting to push this way is that there are other APIs on Windows to do the core client use cases we seem to care about. For example, https://msdn.microsoft.com/en-us/library/windows/desktop/aa384106(v=vs.85).aspx could easily let us implement the requests API with a direct call to the operating system, without having to get into the SChannel world of hurt. I assume macOS has similarly high-level APIs. These are totally fine for implementing a requests-like API that relies on system configuration for HTTPS connections. They are not sufficient for building a server, but they should be sufficient for downloading the packages you need to build a server. This is starting to feel to me like we're stuck on "building the thing right" and have skipped over "building the right thing". But maybe it's just me... Cheers, Steve From cory at lukasa.co.uk Fri Jun 9 15:48:53 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Fri, 9 Jun 2017 20:48:53 +0100 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <6E697036-ECBC-452F-BCF5-09A8F0A51375@lukasa.co.uk> <2E0E1D87-4AB3-42B8-9788-AED19B64DE1E@stufft.io> <07e12379-84e0-b519-22a2-1a75c4d958b6@python.org> <8287CE3C-687E-4BE6-9747-39E651B7178B@stufft.io> <8932A107-0D8C-4B6C-A32F-8861D7B19EF1@stufft.io> <0a086a16-6c20-a721-e7a5-b4b17665b50b@python.org> Message-ID: <6C094262-1C4C-4EB7-972D-D92E9531036A@lukasa.co.uk> > On 9 Jun 2017, at 20:41, Steve Dower wrote: > > These are totally fine for implementing a requests-like API that relies on system configuration for HTTPS connections. They are not sufficient for building a server, but they should be sufficient for downloading the packages you need to build a server. > > This is starting to feel to me like we're stuck on "building the thing right" and have skipped over "building the right thing". But maybe it's just me... For the purposes of this PEP I think we should exclude this. The only way this works on a decent number of platforms (hi there BSDs and Linuxes) is if the option on those platforms is libcurl. Linux does not ship a high level HTTP client library: libcurl is basically the option. That would require pip binding libcurl, NSURLSession, and the Windows API. It?s not clear to me that the solution to ?we don?t want to backport some SSL code? is ?write a bunch of bindings to other third-party libraries?. Cory From donald at stufft.io Fri Jun 9 15:54:10 2017 From: donald at stufft.io (Donald Stufft) Date: Fri, 9 Jun 2017 15:54:10 -0400 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: References: <6E697036-ECBC-452F-BCF5-09A8F0A51375@lukasa.co.uk> <2E0E1D87-4AB3-42B8-9788-AED19B64DE1E@stufft.io> <07e12379-84e0-b519-22a2-1a75c4d958b6@python.org> <8287CE3C-687E-4BE6-9747-39E651B7178B@stufft.io> <8932A107-0D8C-4B6C-A32F-8861D7B19EF1@stufft.io> <0a086a16-6c20-a721-e7a5-b4b17665b50b@python.org> Message-ID: <197F5FF4-28BF-4DF1-ACCC-3E07AD5FDD1C@stufft.io> > On Jun 9, 2017, at 3:41 PM, Steve Dower wrote: > > On 09Jun2017 1118, Donald Stufft wrote: >> A ensuressl style module that tries to install an OpenSSL module is actually fairly hard to do securely. The fundamental issue being that fetching a file securely from the network before you have the primary tool for fetching things securely from a network gets to be a bit of a chicken and egg problem. There are *possible* ways around that, but they essentially boil down to having to ship something akin to a CA Bundle inside of Python which we just shift the responsibility from needing to have an updated OpenSSL in all versions of Python. > >> The good news here is that PEP 543 is the thing that will enable using SChannel on Windows, SecureTransport on macOS, and OpenSSL on Linux getting python-dev out of the business of shipping an OpenSSL at all [1]. Backporting ssl.MemoryBio represents the smallest reasonable change to 2.7 we can make [2] that paves the path forward to being able to do that in some number of years, because we can?t do that until PEP 543 is a thing and people are using it. > > One of the reasons I'm wanting to push this way is that there are other APIs on Windows to do the core client use cases we seem to care about. > > For example, https://msdn.microsoft.com/en-us/library/windows/desktop/aa384106(v=vs.85).aspx could easily let us implement the requests API with a direct call to the operating system, without having to get into the SChannel world of hurt. I assume macOS has similarly high-level APIs. > > These are totally fine for implementing a requests-like API that relies on system configuration for HTTPS connections. They are not sufficient for building a server, but they should be sufficient for downloading the packages you need to build a server. > > This is starting to feel to me like we're stuck on "building the thing right" and have skipped over "building the right thing". But maybe it's just me? > I?m personally not super interested in having platform specific HTTP handling. I honestly want as little platform specific code as I can get away with, and the primary reason why I?m okay with drawing the line at the TLS stack (instead of the HTTP stack) is because being able to get operating system updates for the TLS stack is of immense benefit in addition to the fact that it?s really the *only* reasonable way to get access to the platform specific trust store. Relying on the OS to handle HTTP means that we cannot depend on anything that isn?t the lowest common denominator across all of the variety of platforms we support, which (as I see Cory has mentioned) isn?t really a thing outside of macOS/Windows. For those platforms we?d just be adding yet another C dependency to Python (or pip, or requests, or whatever). There?s a lot of HTTP handling code in pip that would need to be rewritten to support this, and I don?t think it would be of a particular benefit much less be doable in any reasonable timeframe. This isn?t really being particularly innovative in this area, it?s essentially what the browsers are doing. It also matches what curl is doing. All that being said, if someone *does* want pip to use WinHTTP, requests provides a mechanism where you can plug in your own network handling code, so someone could write a requests-winhttp adapter that did that, and we could possibly even expose the ability to ask pip to use that. However that?s entirely in the domain of pip/requests feature requests and not really related to python-dev/this PEP itself. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Fri Jun 9 18:32:37 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 9 Jun 2017 18:32:37 -0400 Subject: [Python-Dev] Remove embedded expat library? In-Reply-To: References: Message-ID: On 6/9/2017 12:40 PM, Zachary Ware wrote: > PR 1783 (https://github.com/python/cpython/pull/1783); needs another > review from Steve to make sure I haven't made a complete mess of > things, then it should be ready to go. Anyone else on Windows (Terry > Reedy?) who can test it and provide feedback, please do! Something like the following condensed log? f:\dev>git clone https://github.com/python/cpython tem f:\dev>cd tem f:\dev\tem>git remote add upstream https://github.com/python/cpython f:\dev\tem>git pr 1783 f:\dev\tem>pcbuild\build.bat -e Fetching external libraries... Fetching bzip2-1.0.6... Fetching openssl-1.0.2k... Fetching sqlite-3.14.2.0... Fetching tcl-core-8.6.6.0... Fetching tk-8.6.6.0... Fetching tix-8.4.3.6... Fetching xz-5.2.2... Fetching external binaries... Fetching nasm-2.11.06... ... # Appeared to finish with usual warnings, but no errors. f:\dev\tem>python -m test -j14 ... 367 tests OK. 38 tests skipped: test_crypt test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll test_fcntl test_fork1 test_gdb test_grp test_ioctl test_kqueue test_nis test_openpty test_ossaudiodev test_pipes test_poll test_posix test_pty test_pwd test_readline test_resource test_smtpnet test_socketserver test_spwd test_syslog test_threadsignals test_timeout test_tix test_tk test_ttk_guionly test_urllib2net test_urllibnet test_wait3 test_wait4 test_winsound test_xmlrpc_net test_zipfile64 Total duration: 2 min 15 sec Tests result: SUCCESS ----- Duration aside, my working clone master, without the patch, gives the same result. (The usual debug build takes twice as long.) In this respect, the patch seems ready to go. If you have any questions about the compile or test logs, I will try to keep them around at least a few hours. -- Terry Jan Reedy From benjamin at python.org Fri Jun 9 19:56:32 2017 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 09 Jun 2017 16:56:32 -0700 Subject: [Python-Dev] =?utf-8?q?On_=22PEP_546_=E2=80=94_Backport_ssl=2EMe?= =?utf-8?q?moryBIO_and_ssl=2ESSLObject_to_Python_2=2E7=22?= Message-ID: <1497052592.1924088.1004692328.777E6622@webmail.messagingengine.com> The reason we're having this conversation at all is probably a matter of timing. If MemoryBIO was in Python 3 when PEP 466 was accepted, it surely would have come along for the ride to 2.7. I believe PEP 466 is generally considered to have produced positive results. PEP 546, carrying no breaking changes, is less risky than PEP 466. The reluctance to bend 2.7 rules is healthy. This PEP is part of the price we pay, though, for making a backwards-incompatible release. The security landscape has and will change over the 10+ python-dev-supported life span of 2.7. During that time, we have an obligation to keep Python 2 secure. Part of that is supporting modern security interfaces, which are features. This change is needed to make another stdlib feature, ensurepip (which is itself yet another 2.7.x backport) work well. Therefore, as 2.7 release manager, I'm accepting the PEP. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Jun 9 21:03:57 2017 From: guido at python.org (Guido van Rossum) Date: Fri, 9 Jun 2017 18:03:57 -0700 Subject: [Python-Dev] =?utf-8?q?On_=22PEP_546_=E2=80=94_Backport_ssl=2EMe?= =?utf-8?q?moryBIO_and_ssl=2ESSLObject_to_Python_2=2E7=22?= In-Reply-To: <1497052592.1924088.1004692328.777E6622@webmail.messagingengine.com> References: <1497052592.1924088.1004692328.777E6622@webmail.messagingengine.com> Message-ID: Well reasoned! On Fri, Jun 9, 2017 at 4:56 PM, Benjamin Peterson wrote: > The reason we're having this conversation at all is probably a matter of > timing. If MemoryBIO was in Python 3 when PEP 466 was accepted, it surely > would have come along for the ride to 2.7. I believe PEP 466 is generally > considered to have produced positive results. PEP 546, carrying no breaking > changes, is less risky than PEP 466. > > The reluctance to bend 2.7 rules is healthy. This PEP is part of the price > we pay, though, for making a backwards-incompatible release. The security > landscape has and will change over the 10+ python-dev-supported life span > of 2.7. During that time, we have an obligation to keep Python 2 secure. > Part of that is supporting modern security interfaces, which are features. > This change is needed to make another stdlib feature, ensurepip (which is > itself yet another 2.7.x backport) work well. > > Therefore, as 2.7 release manager, I'm accepting the PEP. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Sat Jun 10 05:09:58 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 10 Jun 2017 11:09:58 +0200 Subject: [Python-Dev] =?utf-8?q?On_=22PEP_546_=E2=80=94_Backport_ssl=2EMe?= =?utf-8?q?moryBIO_and_ssl=2ESSLObject_to_Python_2=2E7=22?= In-Reply-To: References: <1497052592.1924088.1004692328.777E6622@webmail.messagingengine.com> Message-ID: > On Fri, Jun 9, 2017 at 4:56 PM, Benjamin Peterson > wrote: > Therefore, as 2.7 release manager, I'm accepting the PEP. 2017-06-10 3:03 GMT+02:00 Guido van Rossum : > Well reasoned! Guido: by default, you are the only one who pronounces officially on a PEP. Should I understand that you approved the PEP? I would prefer a more explicit email, just to prevent confusion :-) Victor From christian at python.org Sat Jun 10 07:49:21 2017 From: christian at python.org (Christian Heimes) Date: Sat, 10 Jun 2017 13:49:21 +0200 Subject: [Python-Dev] =?utf-8?q?On_=22PEP_546_=E2=80=94_Backport_ssl=2EMe?= =?utf-8?q?moryBIO_and_ssl=2ESSLObject_to_Python_2=2E7=22?= In-Reply-To: <1497052592.1924088.1004692328.777E6622@webmail.messagingengine.com> References: <1497052592.1924088.1004692328.777E6622@webmail.messagingengine.com> Message-ID: <1ab4fcd7-1df9-321b-b752-10d439cd99f9@python.org> On 2017-06-10 01:56, Benjamin Peterson wrote: > The reason we're having this conversation at all is probably a matter of > timing. If MemoryBIO was in Python 3 when PEP 466 was accepted, it > surely would have come along for the ride to 2.7. I believe PEP 466 is > generally considered to have produced positive results. PEP 546, > carrying no breaking changes, is less risky than PEP 466. > > The reluctance to bend 2.7 rules is healthy. This PEP is part of the > price we pay, though, for making a backwards-incompatible release. The > security landscape has and will change over the 10+ python-dev-supported > life span of 2.7. During that time, we have an obligation to keep Python > 2 secure. Part of that is supporting modern security interfaces, which > are features. This change is needed to make another stdlib feature, > ensurepip (which is itself yet another 2.7.x backport) work well. > > Therefore, as 2.7 release manager, I'm accepting the PEP. That's fantastic news. Thanks Benjamin! Christian From ncoghlan at gmail.com Sat Jun 10 12:30:20 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 11 Jun 2017 02:30:20 +1000 Subject: [Python-Dev] 2017 Python Language Summit coverage finale In-Reply-To: References: <20170608095253.3444f4af@redtail> Message-ID: On 10 June 2017 at 01:14, Guido van Rossum wrote: > Also a good summary for those of us that *did* attend! Thanks Jake from all > participants. > > (And Nick, I hope you won't make a habit of skipping the summit. :-) I've learned a fair bit about where my limits are for the amount of travel I can handle in the last couple of years, so I hope the same :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Jun 10 12:53:56 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 11 Jun 2017 02:53:56 +1000 Subject: [Python-Dev] =?utf-8?q?On_=22PEP_546_=E2=80=94_Backport_ssl=2EMe?= =?utf-8?q?moryBIO_and_ssl=2ESSLObject_to_Python_2=2E7=22?= In-Reply-To: <1497052592.1924088.1004692328.777E6622@webmail.messagingengine.com> References: <1497052592.1924088.1004692328.777E6622@webmail.messagingengine.com> Message-ID: On 10 June 2017 at 09:56, Benjamin Peterson wrote: > The reason we're having this conversation at all is probably a matter of > timing. If MemoryBIO was in Python 3 when PEP 466 was accepted, it surely > would have come along for the ride to 2.7. I believe PEP 466 is generally > considered to have produced positive results. PEP 546, carrying no breaking > changes, is less risky than PEP 466. > > The reluctance to bend 2.7 rules is healthy. This PEP is part of the price > we pay, though, for making a backwards-incompatible release. The security > landscape has and will change over the 10+ python-dev-supported life span of > 2.7. During that time, we have an obligation to keep Python 2 secure. Part > of that is supporting modern security interfaces, which are features. This > change is needed to make another stdlib feature, ensurepip (which is itself > yet another 2.7.x backport) work well. > > Therefore, as 2.7 release manager, I'm accepting the PEP. Thanks Benjamin. I was just about to post in the other thread to say I thought this was the right way to go, as I think our experience with PEP 476 (the switch to validating HTTPS certificates by default) is illustrative here: we (Red Hat) technically *didn't* backport that PEP as originally written into RHEL (and hence into CentOS etc). Instead, we had folks primarily from Red Hat, eGenix, and Canonical figure out the variant covered in PEP 493 that eventually became the system Python behaviour in RHEL 7.3+. So even if we eventually decide we can't backport PEP 546 into the RHEL system Python as written: - it will still be in Ubuntu 18.04+ - it will still make its way into future versions of other long term support distributions (whether community driven or commercial) - it will still make its way into Red Hat Software Collections at some point - we're still free to write a follow-up PEP for an opt-in _ssl_backport bootstrapping module if/when there's a clearer benefit to justify the additional effort Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From cory at lukasa.co.uk Sat Jun 10 14:00:51 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Sat, 10 Jun 2017 19:00:51 +0100 Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 In-Reply-To: <197F5FF4-28BF-4DF1-ACCC-3E07AD5FDD1C@stufft.io> References: <6E697036-ECBC-452F-BCF5-09A8F0A51375@lukasa.co.uk> <2E0E1D87-4AB3-42B8-9788-AED19B64DE1E@stufft.io> <07e12379-84e0-b519-22a2-1a75c4d958b6@python.org> <8287CE3C-687E-4BE6-9747-39E651B7178B@stufft.io> <8932A107-0D8C-4B6C-A32F-8861D7B19EF1@stufft.io> <0a086a16-6c20-a721-e7a5-b4b17665b50b@python.org> <197F5FF4-28BF-4DF1-ACCC-3E07AD5FDD1C@stufft.io> Message-ID: <9E46F781-867C-42E2-B153-805CE9AFC829@lukasa.co.uk> > On 9 Jun 2017, at 20:54, Donald Stufft wrote: > > > All that being said, if someone *does* want pip to use WinHTTP, requests provides a mechanism where you can plug in your own network handling code, so someone could write a requests-winhttp adapter that did that, and we could possibly even expose the ability to ask pip to use that. However that?s entirely in the domain of pip/requests feature requests and not really related to python-dev/this PEP itself. As an FYI I played with this on Mac far enough to prove that it?d work: https://github.com/Lukasa/requests-darwin It?s not anywhere near feature complete, but it basically works pretty ok. Of course, ideally Requests would just get entirely out of the way at this point because it duplicates a lot of NSURLSession?s logic, but it?s an approach that can work. Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat Jun 10 16:08:42 2017 From: guido at python.org (Guido van Rossum) Date: Sat, 10 Jun 2017 13:08:42 -0700 Subject: [Python-Dev] =?utf-8?q?On_=22PEP_546_=E2=80=94_Backport_ssl=2EMe?= =?utf-8?q?moryBIO_and_ssl=2ESSLObject_to_Python_2=2E7=22?= In-Reply-To: References: <1497052592.1924088.1004692328.777E6622@webmail.messagingengine.com> Message-ID: On Sat, Jun 10, 2017 at 2:09 AM, Victor Stinner wrote: > > On Fri, Jun 9, 2017 at 4:56 PM, Benjamin Peterson > > wrote: > > Therefore, as 2.7 release manager, I'm accepting the PEP. > > 2017-06-10 3:03 GMT+02:00 Guido van Rossum : > > Well reasoned! > > Guido: by default, you are the only one who pronounces officially on a > PEP. Should I understand that you approved the PEP? I would prefer a > more explicit email, just to prevent confusion :-) > Oh, sorry! Let's retroactively make Benjamin the BDFL-delegate for this PEP. The effect is the same: the PEP is officially accepted. You can update the PEP headers to reflect that. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Sat Jun 10 16:38:08 2017 From: steve.dower at python.org (Steve Dower) Date: Sat, 10 Jun 2017 13:38:08 -0700 Subject: [Python-Dev] =?utf-8?q?On_=22PEP_546_=E2=80=94_Backport_ssl=2EMe?= =?utf-8?q?moryBIO_and_ssl=2ESSLObject_to_Python_2=2E7=22?= In-Reply-To: <1497052592.1924088.1004692328.777E6622@webmail.messagingengine.com> References: <1497052592.1924088.1004692328.777E6622@webmail.messagingengine.com> Message-ID: Agreed, that?s good reasoning. Thanks for short-circuiting the discussion! Cheers, Steve Top-posted from my Windows phone From: Benjamin Peterson Sent: Friday, June 9, 2017 16:59 To: python-dev at python.org Subject: [Python-Dev] On "PEP 546 ? Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7" The reason we're having this conversation at all is probably a matter of timing. If MemoryBIO was in Python 3 when PEP 466 was accepted, it surely would have come along for the ride to 2.7. I believe PEP 466 is generally considered to have produced positive results. PEP 546, carrying no breaking changes, is less risky than PEP 466. The reluctance to bend 2.7 rules is healthy. This PEP is part of the price we pay, though, for making a backwards-incompatible release. The security landscape has and will change over the 10+ python-dev-supported life span of 2.7. During that time, we have an obligation to keep Python 2 secure. Part of that is supporting modern security interfaces, which are features. This change is needed to make another stdlib feature, ensurepip (which is itself yet another 2.7.x backport) work well. Therefore, as 2.7 release manager, I'm accepting the PEP. -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Sat Jun 10 17:56:30 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 10 Jun 2017 23:56:30 +0200 Subject: [Python-Dev] =?utf-8?q?On_=22PEP_546_=E2=80=94_Backport_ssl=2EMe?= =?utf-8?q?moryBIO_and_ssl=2ESSLObject_to_Python_2=2E7=22?= In-Reply-To: References: <1497052592.1924088.1004692328.777E6622@webmail.messagingengine.com> Message-ID: Le 10 juin 2017 22:09, "Guido van Rossum" a ?crit : Let's retroactively make Benjamin the BDFL-delegate for this PEP. The effect is the same: the PEP is officially accepted. Ok fine, I will update the PEP and then start to work on review the old implementation written by Alex Gaynor. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Sat Jun 10 17:57:03 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 10 Jun 2017 23:57:03 +0200 Subject: [Python-Dev] =?utf-8?q?On_=22PEP_546_=E2=80=94_Backport_ssl=2EMe?= =?utf-8?q?moryBIO_and_ssl=2ESSLObject_to_Python_2=2E7=22?= In-Reply-To: <1497052592.1924088.1004692328.777E6622@webmail.messagingengine.com> References: <1497052592.1924088.1004692328.777E6622@webmail.messagingengine.com> Message-ID: Thank you Benjamin ;-) Victor Le 10 juin 2017 01:58, "Benjamin Peterson" a ?crit : > The reason we're having this conversation at all is probably a matter of > timing. If MemoryBIO was in Python 3 when PEP 466 was accepted, it surely > would have come along for the ride to 2.7. I believe PEP 466 is generally > considered to have produced positive results. PEP 546, carrying no breaking > changes, is less risky than PEP 466. > > The reluctance to bend 2.7 rules is healthy. This PEP is part of the price > we pay, though, for making a backwards-incompatible release. The security > landscape has and will change over the 10+ python-dev-supported life span > of 2.7. During that time, we have an obligation to keep Python 2 secure. > Part of that is supporting modern security interfaces, which are features. > This change is needed to make another stdlib feature, ensurepip (which is > itself yet another 2.7.x backport) work well. > > Therefore, as 2.7 release manager, I'm accepting the PEP. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > victor.stinner%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Sun Jun 11 02:35:58 2017 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sun, 11 Jun 2017 08:35:58 +0200 Subject: [Python-Dev] Remove embedded expat library? In-Reply-To: References: Message-ID: <92BBD834-10BB-4D76-AC1E-AE821BCB1F16@mac.com> > On 9 Jun 2017, at 18:03, Ned Deily wrote: > > On Jun 9, 2017, at 08:43, Victor Stinner wrote: > >> I expect that all Linux distributions build Python using >> --with-system-expat. It may become the default? What about macOS and >> other operating systems? > > The current default is --with-system-expat=no so, unless builders of Python take explicit action, the bundled version of expat is used. Using the bundled version is also currently the case for the python.org macOS installer, no idea what other distributors do. Apple supplies a version of expat with macOS so we presumably we could use the system version for the installer. Presumably (Zach?) we would need to continue to supply a version of expat for Windows builds. But do we need to for others? If it were only Windows, *then* perhaps it might make sense to make all the changes to move expat out of cpython into the common repo for third-party Windows libs. I don?t think it would be a good idea to rely on the system provided libexpat on macOS, as Apple is not exactly fast w.r.t. upgrading their external dependencies and could easily stop updating libraries when the no longer need them (see for example the mess w.r.t. OpenSSL). Ronald From victor.stinner at gmail.com Sun Jun 11 06:10:59 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 11 Jun 2017 12:10:59 +0200 Subject: [Python-Dev] Remove embedded expat library? In-Reply-To: <92BBD834-10BB-4D76-AC1E-AE821BCB1F16@mac.com> References: <92BBD834-10BB-4D76-AC1E-AE821BCB1F16@mac.com> Message-ID: Le 11 juin 2017 09:38, "Ronald Oussoren" a ?crit : I don?t think it would be a good idea to rely on the system provided libexpat on macOS, as Apple is not exactly fast w.r.t. upgrading their external dependencies and could easily stop updating libraries when the no longer need them (see for example the mess w.r.t. OpenSSL). Ok, but can't we download expat instead of keeping an old copy in our repisitory? Having a copy is useful when we modify it. I don't that it is the case here. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Sun Jun 11 10:23:33 2017 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sun, 11 Jun 2017 16:23:33 +0200 Subject: [Python-Dev] Remove embedded expat library? In-Reply-To: References: <92BBD834-10BB-4D76-AC1E-AE821BCB1F16@mac.com> Message-ID: <224EBBAE-A497-4BB9-95C2-513CD14C4D5F@mac.com> > On 11 Jun 2017, at 12:10, Victor Stinner wrote: > > Le 11 juin 2017 09:38, "Ronald Oussoren" > a ?crit : > I don?t think it would be a good idea to rely on the system provided libexpat on macOS, as Apple is not exactly fast w.r.t. upgrading their external dependencies and could easily stop updating libraries when the no longer need them (see for example the mess w.r.t. OpenSSL). > > > Ok, but can't we download expat instead of keeping an old copy in our repisitory? Sure. The script that creates the installer already downloads a number of libraries, adding one more shouldn?t be a problem. > > Having a copy is useful when we modify it. I don't that it is the case here. I don?t know why expat was included in the CPython tree and if those reasons are still valid. I therefore have no opinion on keeping it, other than that expat shouldn?t be kept in the CPython tree unless there?s a good reason for doing so. BTW. Removing 3th-party libraries from the source tree doesn?t fully isolate us from security issues in those libraries due to shipping the libraries in binary installers on Windows and macOS. The removal does make maintenance easier (no need to guess whether or not there are local patches). Ronald > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From gzlist at googlemail.com Sun Jun 11 20:05:23 2017 From: gzlist at googlemail.com (Martin (gzlist)) Date: Mon, 12 Jun 2017 01:05:23 +0100 Subject: [Python-Dev] PEP 538 (review round 2): Coercing the legacy C locale to a UTF-8 based locale In-Reply-To: References: Message-ID: On 09/05/2017, Nick Coghlan wrote: > > Enough changes have accumulated in PEP 538 since the start of the > previous thread that it seems sensible to me to start a new thread > specifically covering the current design (which aims to address all > the concerns raised in the previous thread). > > I haven't requoted the PEP in full since it's so long, but will > instead refer readers to the web version: > https://www.python.org/dev/peps/pep-0538/ I did try to follow along via the mailing list threads, and have now read over the PEP again. Responding now as I'm actually touching code relevent to this again. Broadly the proposal looks good to me. It does help one of the two cases I care about, and does no serious harm. For a command line Python script, making sure Python itself uses UTF-8 for the C locale is sufficient, and setting LC_CTYPE so spawned processes that aren't Python have a chance at doing the right thing too is a reasonable upgrade. This is probably good enough to drop one hack[1] rather than porting it to Python 3. For hosted Python code this does nothing (apart from print to stderr), so mod_wsgi for instance is still going to need the same kind dance to get users to set LANG as configuration themselves. Ideally this PEP would have a C api or something so I could file bugs to make it just do the right thing. A few notes on specifics, I'll try not to stray too much into choices already made. The PEP doesn't persuade me that Py_Initialize actually is too late to switch *specifically* from ascii to utf-8. Any preceeding operations that operate on unicode would have been a safe subset. There might be issues with other internals, or surrogateescape, or it's just a pain? I don't like the side effect of changing the standard stream error handler to surrogateescape if LANG=C.UTF-8 is actually set. Obviously bad data vs exception is a trade off anyway, but means to get a Python script that will always output valid data or exit, you have to set an arbitrary language like en_US. Yes, that's true of the change as implemented in 3.5 anyway. Not setting LANG and only setting LC_CTYPE seems fine. Obviously, things can go wrong based on odd behaviours of spawned processes, but it works for the normal idioms. I'm not sold on adding the PYTHONCOERCECLOCALE runtime configuration. All it really does is turn off stderr kipple if you must use the C locale for other reasons? Anyone with the ability to set that variable could just set LANG instead. I was going to suggest just documenting LC_ALL=C as the override instead of adding a python specific variable, but note looking around that Debian discourage that[3]. That's all, though I will also grumble a bit about how long the PEP is. Martin [1] Override Py_FileSystemDefaultEncoding to utf-8 from ascii for the bzr script [2] WSGIDaemonProcess lang and locale options [3] "Using LC_ALL is strongly discouraged as it overrides everything" From ncoghlan at gmail.com Sun Jun 11 21:48:38 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 12 Jun 2017 11:48:38 +1000 Subject: [Python-Dev] PEP 538 (review round 2): Coercing the legacy C locale to a UTF-8 based locale In-Reply-To: References: Message-ID: On 12 June 2017 at 10:05, Martin (gzlist) via Python-Dev wrote: > I don't like the side effect of changing the standard stream error > handler to surrogateescape if LANG=C.UTF-8 is actually set. Obviously > bad data vs exception is a trade off anyway, but means to get a Python > script that will always output valid data or exit, you have to set an > arbitrary language like en_US. Yes, that's true of the change as > implemented in 3.5 anyway. `PYTHONIOENCODING=:strict` remains the preferred way of forcing strict encoding checks on the standard streams, regardless of locale. > I'm not sold on adding the PYTHONCOERCECLOCALE runtime configuration. > All it really does is turn off stderr kipple if you must use the C > locale for other reasons? Anyone with the ability to set that variable > could just set LANG instead. I was going to suggest just documenting > LC_ALL=C as the override instead of adding a python specific variable, > but note looking around that Debian discourage that[3]. In addition to providing a reliable escape hatch with no other potentially unwanted side effects (for when folks actually want the current behaviour), the entry for the off switch in the CLI usage docs also provides us with a convenient place to document the *default* behaviour. > That's all, though I will also grumble a bit about how long the PEP is. The ASCII-to-Unicode migration has been in progress for almost as long as Python has been around, and ASCII has been the default encoding in C for almost twice as long as that, so it takes a bit of text to explain why *now* is a good time to break with 50+ years of precedent :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From storchaka at gmail.com Mon Jun 12 03:10:28 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 12 Jun 2017 10:10:28 +0300 Subject: [Python-Dev] Handle errors in cleanup code Message-ID: There is an idiomatic Python code: do_something() try: do_something_other() except: undo_something() raise If an error is raised when execute do_something_other(), then we should first restore the state that was before calling do_something(), and then reraise the exception. It is important that the bare "except" (or "except BaseException") is used, because undo_something() should be executed on any exception. And this is one of few cases where using the bare "except" is correct. But if an error is raised when execute undo_something(), it replaces the original exception which become chaining as the __context__ attribute. The problem is that this can change the type of the exception. If do_something_other() raises SystemExit and undo_something() raises KeyError, the final exception has type KeyError. Yury in the comment for PR 2108 [1] suggested more complicated code: do_something() try: do_something_other() except BaseException as ex: try: undo_something() finally: raise ex Does it mean that we should rewrite every chunk of code similar to the above? And if such cumbersome code is necessary and become common, maybe it needs syntax support in the language? Or changing the semantic of exceptions raised in error handlers and finally blocks? [1] https://github.com/python/cpython/pull/2108 From gzlist at googlemail.com Mon Jun 12 03:31:02 2017 From: gzlist at googlemail.com (Martin (gzlist)) Date: Mon, 12 Jun 2017 08:31:02 +0100 Subject: [Python-Dev] Handle errors in cleanup code In-Reply-To: References: Message-ID: On 12/06/2017, Serhiy Storchaka wrote: > > But if an error is raised when execute undo_something(), it replaces the > original exception which become chaining as the __context__ attribute. > The problem is that this can change the type of the exception. If > do_something_other() raises SystemExit and undo_something() raises > KeyError, the final exception has type KeyError. For Python 2.7, I've used the following idiom, which always masks errors from the undo: do_something() try: do_something_other() except: try: undo_something() finally: raise Unfortunately, that breaks down on Python 3, as the new exception propogates with the original as context.. > Does it mean that we should rewrite every chunk of code similar to the > above? And if such cumbersome code is necessary and become common, maybe > it needs syntax support in the language? Or changing the semantic of > exceptions raised in error handlers and finally blocks? What I want is a way to chain exceptions in the other direction, raising the original error, but attaching a related one. Unfortunately neither __cause__ or __context__ really help. Martin From ncoghlan at gmail.com Mon Jun 12 03:46:22 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 12 Jun 2017 17:46:22 +1000 Subject: [Python-Dev] Handle errors in cleanup code In-Reply-To: References: Message-ID: On 12 June 2017 at 17:10, Serhiy Storchaka wrote: > Does it mean that we should rewrite every chunk of code similar to the > above? And if such cumbersome code is necessary and become common, maybe it > needs syntax support in the language? Or changing the semantic of exceptions > raised in error handlers and finally blocks? No, as in general there's no way for us to reliably tell the difference between intentionally changing the exception type (e.g. KeyError->AttributeError and vice-versa), unintentionally changing the exception type due to a bug in the cleanup code (e.g. SystemExit->KeyError), and the exception type changing due to external events (e.g. KeyError -> KeyboardInterrupt). So in the example given if an expected-and-allowed kind of exception can escape from "undo_something()", it's not really cleanup code, it's normal code, and needs to be refactored more like the way most close() methods work: particular known-to-be-possible exceptions get caught and reported (or silently ignored), but not propagated. That last example also shows that the suggested replacement idiom is incorrect, as it can suppress SystemExit, KeyboardInterrupt, etc. That said, as far as better builtin support for dynamic resource management goes: that would currently be contextlib.ExitStack. I'm also completely open to adding more hooks to let users tweak how that reports exceptions, including adding a contextlib.ResourceSet variant, which *doesn't* semantically nest the callback handling the way ExitStack does, and instead collects any exceptions raised into a new __cleanup_errors__ attribute on the original exception (thus leaving the original exception chain untouched). That would still need some fiddling to work out what to do in the "exception that doesn't inherit from Exception" case (probably clean up all the registered resources anyway then raise the first such exception caught, while still attaching all the others to __cleanup_errors__ on the original exception), but it's likely to be significantly more feasible than doing anything at the syntax level. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From gzlist at googlemail.com Mon Jun 12 03:47:07 2017 From: gzlist at googlemail.com (Martin (gzlist)) Date: Mon, 12 Jun 2017 08:47:07 +0100 Subject: [Python-Dev] PEP 538 (review round 2): Coercing the legacy C locale to a UTF-8 based locale In-Reply-To: References: Message-ID: Thanks for replying to my points! On 12/06/2017, Nick Coghlan wrote: > > `PYTHONIOENCODING=:strict` remains the preferred way of forcing strict > encoding checks on the standard streams, regardless of locale. Then the user of my script has to care that it's written in Python and set that specifically in their crontab or so on... > In addition to providing a reliable escape hatch with no other > potentially unwanted side effects (for when folks actually want the > current behaviour), the entry for the off switch in the CLI usage docs > also provides us with a convenient place to document the *default* > behaviour. The documentation aspect is an interesting consideration. Having thought about it a bit more, my preferred option is having the disable be if either LC_ALL or LC_CTYPE vars are exactly 'C', then don't override. Otherwise (including for LANG=C), force C.UTF-8. The CLI usage docs could have a LC_CTYPE entry that goes into details of why. Martin From ncoghlan at gmail.com Mon Jun 12 04:07:18 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 12 Jun 2017 18:07:18 +1000 Subject: [Python-Dev] Handle errors in cleanup code In-Reply-To: References: Message-ID: On 12 June 2017 at 17:31, Martin (gzlist) via Python-Dev wrote: > On 12/06/2017, Serhiy Storchaka wrote: >> >> But if an error is raised when execute undo_something(), it replaces the >> original exception which become chaining as the __context__ attribute. >> The problem is that this can change the type of the exception. If >> do_something_other() raises SystemExit and undo_something() raises >> KeyError, the final exception has type KeyError. > > For Python 2.7, I've used the following idiom, which always masks > errors from the undo: > > do_something() > try: > do_something_other() > except: > try: > undo_something() > finally: > raise > > Unfortunately, that breaks down on Python 3, as the new exception > propogates with the original as context.. Relevant tracker issue for that problem: https://bugs.python.org/issue18861 >> Does it mean that we should rewrite every chunk of code similar to the >> above? And if such cumbersome code is necessary and become common, maybe >> it needs syntax support in the language? Or changing the semantic of >> exceptions raised in error handlers and finally blocks? > > What I want is a way to chain exceptions in the other direction, > raising the original error, but attaching a related one. Unfortunately > neither __cause__ or __context__ really help. Aye, agreed. The key challenge we have is that we're trying to represent the exception state as a linked list, when what we really have once we start taking cleanup errors into account is an exception *tree*. The thing that finally clicked for me in this thread is that if we add contextlib.ResourceSet, and have it add a new attribute to the original exception (e.g. "__cleanup_errors__") with a list of exceptions raised while attempt to cleanup after the earlier exception, then that lets us actually start modelling that tree at runtime. Once we understand the *behaviour* we want, *then* we can consider whether we might want to add a context manager to have any raised exceptions be attached to the currently active exception as cleanup errors rather than as new exceptions in their own right. For example: do_something() try: do_something_other() except BaseException as exc: with contextlib.cleanup(exc) as reraise: # Exceptions raised in here would be added to # exc.__cleanup_errors__, rather than being # propagated normally undo_something() reraise() The need for the "as reraise:"/"reraise()" trick arises from the need to special case the situation where the original exception inherits from Exception, but one of the raised exceptions *doesn't* - we want SystemExit/KeyboardInterrupt/etc to take precedence in that case, and a bare raise statement won't handle that for us (it *could* in a hypothetical future version of Python that's natively `__cleanup_errors__` aware, but that's not going to be useful for existing versions). Since I don't see anything in the discussion so far that *requires* changes to the standard library (aside from "we may want to use this ourselves"), the right place to thrash out the design details is probably contextlib2: https://github.com/jazzband/contextlib2 That's where contextlib.ExitStack was born, and I prefer using it to iterate on context management design concepts, since we can push updates out faster, and if we make bad choices anywhere along the way, they can just sit around in contextlib2, rather than polluting the standard library indefinitely. Cheers, Nick. P.S. trio's MultiError is also likely worth looking into in this context -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From songofacandy at gmail.com Mon Jun 12 04:46:57 2017 From: songofacandy at gmail.com (INADA Naoki) Date: Mon, 12 Jun 2017 17:46:57 +0900 Subject: [Python-Dev] PEP 538 (review round 2): Coercing the legacy C locale to a UTF-8 based locale In-Reply-To: References: Message-ID: > On 12/06/2017, Nick Coghlan wrote: >> >> `PYTHONIOENCODING=:strict` remains the preferred way of forcing strict >> encoding checks on the standard streams, regardless of locale. > > Then the user of my script has to care that it's written in Python and > set that specifically in their crontab or so on... > That's why I think https://bugs.python.org/issue15216 should be fixed in Python 3.7 too. Python should have one preferable way to specify encoding and error handler from inside of the program, not from envvar or command line argument. Regards, From victor.stinner at gmail.com Mon Jun 12 04:56:31 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 12 Jun 2017 10:56:31 +0200 Subject: [Python-Dev] PEP 538 warning at startup: please remove it Message-ID: Hi, Nick Coghlan pushed his implementation of his PEP 538: nice! Nice step forward to UTF-8 everywhere ;-) I would prefer to not be annoyed by warning messages about encodings at startup if possible: "Python detected LC_CTYPE=C: LC_CTYPE coerced to C.UTF-8 (set another locale or PYTHONCOERCECLOCALE=0 to disable this locale coercion behavior)." Python 3.7 is the only programming language that I know that complains about encoding at startup. Even if my code is 100% valid, PEP 8 compliant, don't emit any warning, etc. I will get the warning. It is *not* possible to ignore the warning... Like "yeah, I know that my locale is C, but it's not like I'm able to configure it." haypo at selma$ export LANG= haypo at selma$ locale LANG= LC_CTYPE="POSIX" ... haypo at selma$ ./python -c 'print("Hello World!")' # Python 3.7 Python detected LC_CTYPE=C: LC_CTYPE coerced to C.UTF-8 (set another locale or PYTHONCOERCECLOCALE=0 to disable this locale coercion behavior). Hello World! haypo at selma$ python2 -c 'print("Hello World!")' # Python 2.7 Hello World! haypo at selma$ perl -e 'print "Hello, world!\n"' # Perl 5.24 Hello, world! haypo at selma$ ./c # C Hello World! ... Note: I don't consider that 2>/dev/null is a good practice to ignore a single warning, since it will ignore *all* message written into stderr, including useful warnings like ResourceWarning or "Exception ignored ...". Victor From solipsis at pitrou.net Mon Jun 12 05:08:09 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 12 Jun 2017 11:08:09 +0200 Subject: [Python-Dev] PEP 538 warning at startup: please remove it References: Message-ID: <20170612110809.70cd7354@fsol> On Mon, 12 Jun 2017 10:56:31 +0200 Victor Stinner wrote: > Hi, > > Nick Coghlan pushed his implementation of his PEP 538: nice! Nice step > forward to UTF-8 everywhere ;-) > > I would prefer to not be annoyed by warning messages about encodings > at startup if possible: > > "Python detected LC_CTYPE=C: LC_CTYPE coerced to C.UTF-8 (set another > locale or PYTHONCOERCECLOCALE=0 to disable this locale coercion > behavior)." +1 for muting this warning. Regards Antoine. From songofacandy at gmail.com Mon Jun 12 05:35:11 2017 From: songofacandy at gmail.com (INADA Naoki) Date: Mon, 12 Jun 2017 18:35:11 +0900 Subject: [Python-Dev] PEP 538 warning at startup: please remove it In-Reply-To: References: Message-ID: On Mon, Jun 12, 2017 at 5:56 PM, Victor Stinner wrote: > Hi, > > Nick Coghlan pushed his implementation of his PEP 538: nice! Nice step > forward to UTF-8 everywhere ;-) > > I would prefer to not be annoyed by warning messages about encodings > at startup if possible: > > "Python detected LC_CTYPE=C: LC_CTYPE coerced to C.UTF-8 (set another > locale or PYTHONCOERCECLOCALE=0 to disable this locale coercion > behavior)." > This is intentional behavior, to motivate people to use UTF-8 locale without coercion. > Note: I don't consider that 2>/dev/null is a good practice to ignore a > single warning, since it will ignore *all* message written into > stderr, including useful warnings like ResourceWarning or "Exception > ignored ...". I think "Good practice" is set `LC_CTYPE=C.UTF-8` environment variable, as warning says. # When PEP 540 is accepted, it can be used to disable the warning too. Regards, From victor.stinner at gmail.com Mon Jun 12 05:46:19 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 12 Jun 2017 11:46:19 +0200 Subject: [Python-Dev] PEP 538 warning at startup: please remove it In-Reply-To: References: Message-ID: 2017-06-12 11:35 GMT+02:00 INADA Naoki : > I think "Good practice" is set `LC_CTYPE=C.UTF-8` environment variable, > as warning says. I like using LANG=C to display a manual page in english. Technically, I know that it's wrong, but it works. I don't see the point of the warning. I'm able to render my hello world with the wrong locale :-) If you want to use C.utf-8, fine, just do it, but don't bug me with locales please :-) I consider that I understand well locales and encodings. But now try to imagine someone who don't know anything about programming and starts learning a new language, Python, and see this warning... Victor From ncoghlan at gmail.com Mon Jun 12 08:05:29 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 12 Jun 2017 22:05:29 +1000 Subject: [Python-Dev] PEP 538 (review round 2): Coercing the legacy C locale to a UTF-8 based locale In-Reply-To: References: Message-ID: On 12 June 2017 at 17:47, Martin (gzlist) wrote: > Thanks for replying to my points! > > On 12/06/2017, Nick Coghlan wrote: >> >> `PYTHONIOENCODING=:strict` remains the preferred way of forcing strict >> encoding checks on the standard streams, regardless of locale. > > Then the user of my script has to care that it's written in Python and > set that specifically in their crontab or so on... As Inada-san wrote, we think the right way to fix that is to make it easier and safer for application developers to override the default settings on the standard streams. At the moment, doing so requires rebinding sys.stdin/out/err, which means you end up with multiple Python level streams sharing the one underlying C stream, which can cause problems. The basic API for that was recently merged (`TextIOWrapper.reconfigure()`), so it's now a matter of extending it to also allow updating `encoding` and `errors`. >> In addition to providing a reliable escape hatch with no other >> potentially unwanted side effects (for when folks actually want the >> current behaviour), the entry for the off switch in the CLI usage docs >> also provides us with a convenient place to document the *default* >> behaviour. > > The documentation aspect is an interesting consideration. > > Having thought about it a bit more, my preferred option is having the > disable be if either LC_ALL or LC_CTYPE vars are exactly 'C', then > don't override. Otherwise (including for LANG=C), force C.UTF-8. The > CLI usage docs could have a LC_CTYPE entry that goes into details of > why. LC_ALL=C doesn't actually disable the locale coercion (i.e. we still set LC_CTYPE). The coercion just doesn't have any effect, since LC_ALL takes precedence. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Jun 12 08:24:56 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 12 Jun 2017 22:24:56 +1000 Subject: [Python-Dev] PEP 538 warning at startup: please remove it In-Reply-To: References: Message-ID: On 12 June 2017 at 18:56, Victor Stinner wrote: > Hi, > > Nick Coghlan pushed his implementation of his PEP 538: nice! Nice step > forward to UTF-8 everywhere ;-) > > I would prefer to not be annoyed by warning messages about encodings > at startup if possible: > > "Python detected LC_CTYPE=C: LC_CTYPE coerced to C.UTF-8 (set another > locale or PYTHONCOERCECLOCALE=0 to disable this locale coercion > behavior)." Note that there's an open issue for this linked from the What's New entry: * https://docs.python.org/dev/whatsnew/3.7.html#pep-538-legacy-c-locale-coercion * https://bugs.python.org/issue30565 I suspect the eventual outcome is going to be dropping that particular warning (since it's been problematic for Fedora's 3.6 backport as well, and the problems are due to the warning itself, *not* the locale coercion), but I'd prefer to keep the notification at least for a while (potentially even until alpha 1). OTOH, I'm also open to being persuaded otherwise if enough folks are running into problems with it just while working on CPython (I'd still like to turn it back on for alpha 1 even if we turn off in the meantime, though). Cheers, Nick. P.S. Part of my rationale for doing it this way is that I'm certain that after 3.7's release next year we're going to get at least a few users genuinely upset at our decision to move the ASCII-based C locale explicitly into the "legacy partially-supported environment" category, and even more upset that we're "silently ignoring their explicit configuration settings" by implicitly coercing it to something else. Those kinds of concerns are much easier to address effectively if we can say "We tried it with an explicit warning, and it was too annoying to be usable; see if you want more details" than if we're in the situation of having to say "We assumed an explicit warning would be too annoying, so we never even tried it". -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From jsbueno at python.org.br Mon Jun 12 08:50:53 2017 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Mon, 12 Jun 2017 14:50:53 +0200 Subject: [Python-Dev] PEP 538 warning at startup: please remove it In-Reply-To: References: Message-ID: On 12 June 2017 at 14:24, Nick Coghlan wrote: > On 12 June 2017 at 18:56, Victor Stinner wrote: >> Hi, >> >> Nick Coghlan pushed his implementation of his PEP 538: nice! Nice step >> forward to UTF-8 everywhere ;-) >> >> I would prefer to not be annoyed by warning messages about encodings >> at startup if possible: >> >> "Python detected LC_CTYPE=C: LC_CTYPE coerced to C.UTF-8 (set another >> locale or PYTHONCOERCECLOCALE=0 to disable this locale coercion >> behavior)." > > Note that there's an open issue for this linked from the What's New entry: > > * https://docs.python.org/dev/whatsnew/3.7.html#pep-538-legacy-c-locale-coercion > * https://bugs.python.org/issue30565 > > I suspect the eventual outcome is going to be dropping that particular > warning (since it's been problematic for Fedora's 3.6 backport as > well, and the problems are due to the warning itself, *not* the locale > coercion), but I'd prefer to keep the notification at least for a > while (potentially even until alpha 1). > > OTOH, I'm also open to being persuaded otherwise if enough folks are > running into problems with it just while working on CPython (I'd still > like to turn it back on for alpha 1 even if we turn off in the > meantime, though). > > Cheers, > Nick. > > P.S. Part of my rationale for doing it this way is that I'm certain > that after 3.7's release next year we're going to get at least a few > users genuinely upset at our decision to move the ASCII-based C locale > explicitly into the "legacy partially-supported environment" category, > and even more upset that we're "silently ignoring their explicit > configuration settings" by implicitly coercing it to something else. Yes - all these 15 users can be quite noisy - they are not quite users of "whatever there is is good" like the other 800 million or so users that will be bothered by the warning. (Ok - I guess most of the 800 million users won't even be seeing a terminal when running their Python - but still it would be a couple million users and let's suppose there are 150 and not 15 genuinely worried about that coercion) > > Those kinds of concerns are much easier to address effectively if we > can say "We tried it with an explicit warning, and it was too annoying > to be usable; see if you want more details" than if we're > in the situation of having to say "We assumed an explicit warning > would be too annoying, so we never even tried it". Perfect - and guess what? It looks like it already happened, as you can see by these e-mail messages. Therefore we are good to remove the warning now. > > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/jsbueno%40python.org.br From songofacandy at gmail.com Mon Jun 12 09:28:31 2017 From: songofacandy at gmail.com (INADA Naoki) Date: Mon, 12 Jun 2017 22:28:31 +0900 Subject: [Python-Dev] PEP 538 warning at startup: please remove it In-Reply-To: References: Message-ID: On Mon, Jun 12, 2017 at 6:46 PM, Victor Stinner wrote: > 2017-06-12 11:35 GMT+02:00 INADA Naoki : >> I think "Good practice" is set `LC_CTYPE=C.UTF-8` environment variable, >> as warning says. > > I like using LANG=C to display a manual page in english. Me too. But we can use LC_CTYPE=C.UTF-8 with LANG=C. And LC_CTYPE=C.UTF-8 is much easier (and ideal) than PYTHONLOCALECOERCIONWARNING=0. > Technically, I know that it's wrong, but it works. I don't see the point of the > warning. I'm able to render my hello world with the wrong locale :-) > If you want to use C.utf-8, fine, just do it, but don't bug me with > locales please :-) Do you mean which? a) make locale coercion off by default. b) don't show warning when locale is coerced. > I consider that I understand well locales and encodings. But now try > to imagine someone who don't know anything about programming and > starts learning a new language, Python, and see this warning... Locale coercion is for them. Currently (Python 3.6), they will see UnicodeEncodeError. This warning is much helpful than UnicodeEncodeError, because it teaches how to configure UTF-8 locale. And I fear about overriding locale silently. AFAIK, locale coercion is very nice for most cases. But there may be someone who prefer strict, ascii-only C locale. This warning tells how to disable coercion for them. And there may be some surprising side effect of coercing location we can't expect for now. That's another reason why coercion shows warning. But we use `surrogateescape` error handler for stdout in C locale already. I may be too nervous about it. So my vote is: -1 for disable coercion by default: It's too unhelpful for beginners. -0.5 for disable warning NOW: I don't think we know all side effects well. Regards, From stefanrin at gmail.com Mon Jun 12 09:29:00 2017 From: stefanrin at gmail.com (Stefan Ring) Date: Mon, 12 Jun 2017 15:29:00 +0200 Subject: [Python-Dev] Handle errors in cleanup code In-Reply-To: References: Message-ID: > Yury in the comment for PR 2108 [1] suggested more complicated code: > > do_something() > try: > do_something_other() > except BaseException as ex: > try: > undo_something() > finally: > raise ex And this is still bad, because it loses the back trace. The way we do it is: do_something() try: do_something_other() except BaseException as ex: tb = sys.exc_info()[2] try: undo_something() finally: raise ex, None, tb From victor.stinner at gmail.com Mon Jun 12 10:00:14 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 12 Jun 2017 16:00:14 +0200 Subject: [Python-Dev] PEP 538 warning at startup: please remove it In-Reply-To: References: Message-ID: 2017-06-12 15:28 GMT+02:00 INADA Naoki : >> I like using LANG=C to display a manual page in english. > > Me too. But we can use LC_CTYPE=C.UTF-8 with LANG=C. My point is that LANG=C is easy to remember and "it just works". > And LC_CTYPE=C.UTF-8 is much easier (and ideal) than > PYTHONLOCALECOERCIONWARNING=0. I just propose to remove the warning, so LANG=C would "just works" as well with Python 3.7. >> Technically, I know that it's wrong, but it works. I don't see the point of the >> warning. I'm able to render my hello world with the wrong locale :-) >> If you want to use C.utf-8, fine, just do it, but don't bug me with >> locales please :-) > > Do you mean which? > > b) don't show warning when locale is coerced. Always remove the warning. >> I consider that I understand well locales and encodings. But now try >> to imagine someone who don't know anything about programming and >> starts learning a new language, Python, and see this warning... > > Locale coercion is for them. Currently (Python 3.6), they will see > UnicodeEncodeError. > This warning is much helpful than UnicodeEncodeError, because > it teaches how to configure UTF-8 locale. Sorry, I don't understand. Currently, Python 3.7 "just works": it uses UTF-8 when the locale is C and so we don't get any UnicodeError anymore. Why would user need to be annoyed by a warning while Python just do the "right thing" for them? > And I fear about overriding locale silently. The thing is nobody understand locales :-) No need to annoy users about locales. Just fix them silently ;-) > But there may be someone who prefer strict, > ascii-only C locale. This warning tells how to disable coercion for them. Users who understand locales don't need such warning. They know how to read a documentation and how to use properly locales :-) > -1 for disable coercion by default: It's too unhelpful for beginners. Are you proposing to reject the PEP that you approved? Now I'm confused. Victor From ethan at stoneleaf.us Mon Jun 12 11:48:07 2017 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 12 Jun 2017 08:48:07 -0700 Subject: [Python-Dev] PEP 538 warning at startup: please remove it In-Reply-To: References: Message-ID: <593EB7B7.8010403@stoneleaf.us> On 06/12/2017 05:24 AM, Nick Coghlan wrote: > I suspect the eventual outcome is going to be dropping that particular > warning (since it's been problematic for Fedora's 3.6 backport as > well, and the problems are due to the warning itself, *not* the locale > coercion), but I'd prefer to keep the notification at least for a > while (potentially even until alpha 1). As long as we take it out for 3.7.0 I'll be happy. At that point it can be enabled by whichever flag we have that says emit all warnings instead of silently suppressing them. This reminds me of pip with it's 20 lines of ssl and sudo warnings -- it makes it nearly unusable as I have to search and search to find the two lines somewhere in the middle that I'm interested in. -- ~Ethan~ From victor.stinner at gmail.com Mon Jun 12 12:01:15 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 12 Jun 2017 18:01:15 +0200 Subject: [Python-Dev] =?utf-8?q?On_=22PEP_546_=E2=80=94_Backport_ssl=2EMe?= =?utf-8?q?moryBIO_and_ssl=2ESSLObject_to_Python_2=2E7=22?= In-Reply-To: <1497052592.1924088.1004692328.777E6622@webmail.messagingengine.com> References: <1497052592.1924088.1004692328.777E6622@webmail.messagingengine.com> Message-ID: FYI I started to work on the implementation: I rebased Alex Gaynor's patch written in 2014 and converted it to a pull request. http://bugs.python.org/issue22559 Victor 2017-06-10 1:56 GMT+02:00 Benjamin Peterson : > The reason we're having this conversation at all is probably a matter of > timing. If MemoryBIO was in Python 3 when PEP 466 was accepted, it surely > would have come along for the ride to 2.7. I believe PEP 466 is generally > considered to have produced positive results. PEP 546, carrying no breaking > changes, is less risky than PEP 466. > > The reluctance to bend 2.7 rules is healthy. This PEP is part of the price > we pay, though, for making a backwards-incompatible release. The security > landscape has and will change over the 10+ python-dev-supported life span of > 2.7. During that time, we have an obligation to keep Python 2 secure. Part > of that is supporting modern security interfaces, which are features. This > change is needed to make another stdlib feature, ensurepip (which is itself > yet another 2.7.x backport) work well. > > Therefore, as 2.7 release manager, I'm accepting the PEP. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > From greg at krypto.org Mon Jun 12 13:48:39 2017 From: greg at krypto.org (Gregory P. Smith) Date: Mon, 12 Jun 2017 17:48:39 +0000 Subject: [Python-Dev] PEP 538 warning at startup: please remove it In-Reply-To: References: Message-ID: On Mon, Jun 12, 2017 at 5:25 AM Nick Coghlan wrote: > On 12 June 2017 at 18:56, Victor Stinner wrote: > > Hi, > > > > Nick Coghlan pushed his implementation of his PEP 538: nice! Nice step > > forward to UTF-8 everywhere ;-) > > > > I would prefer to not be annoyed by warning messages about encodings > > at startup if possible: > > > > "Python detected LC_CTYPE=C: LC_CTYPE coerced to C.UTF-8 (set another > > locale or PYTHONCOERCECLOCALE=0 to disable this locale coercion > > behavior)." > > Note that there's an open issue for this linked from the What's New entry: > > * > https://docs.python.org/dev/whatsnew/3.7.html#pep-538-legacy-c-locale-coercion > * https://bugs.python.org/issue30565 > > I suspect the eventual outcome is going to be dropping that particular > warning (since it's been problematic for Fedora's 3.6 backport as > well, and the problems are due to the warning itself, *not* the locale > coercion), but I'd prefer to keep the notification at least for a > while (potentially even until alpha 1). > > OTOH, I'm also open to being persuaded otherwise if enough folks are > running into problems with it just while working on CPython (I'd still > like to turn it back on for alpha 1 even if we turn off in the > meantime, though). > > Cheers, > Nick. > > P.S. Part of my rationale for doing it this way is that I'm certain > that after 3.7's release next year we're going to get at least a few > users genuinely upset at our decision to move the ASCII-based C locale > explicitly into the "legacy partially-supported environment" category, > and even more upset that we're "silently ignoring their explicit > configuration settings" by implicitly coercing it to something else. > > Those kinds of concerns are much easier to address effectively if we > can say "We tried it with an explicit warning, and it was too annoying > to be usable; see if you want more details" than if we're > in the situation of having to say "We assumed an explicit warning > would be too annoying, so we never even tried it". > The problem, as with all warnings, is that it isn't the user who has control over the problem who sees the warning. It is the end use of an application on a system that sees it. So they way I see it, we have no choice but to disable this warning by default before we exit 3.7 beta. -gps - "infamous" author of the bad idea of a DeprecationWarning in the old md5 and sha modules... > > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/greg%40krypto.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon Jun 12 13:02:31 2017 From: brett at python.org (Brett Cannon) Date: Mon, 12 Jun 2017 17:02:31 +0000 Subject: [Python-Dev] Handle errors in cleanup code In-Reply-To: References: Message-ID: On Mon, 12 Jun 2017 at 01:08 Nick Coghlan wrote: > On 12 June 2017 at 17:31, Martin (gzlist) via Python-Dev > wrote: > > On 12/06/2017, Serhiy Storchaka wrote: > >> > >> But if an error is raised when execute undo_something(), it replaces the > >> original exception which become chaining as the __context__ attribute. > >> The problem is that this can change the type of the exception. If > >> do_something_other() raises SystemExit and undo_something() raises > >> KeyError, the final exception has type KeyError. > > > > For Python 2.7, I've used the following idiom, which always masks > > errors from the undo: > > > > do_something() > > try: > > do_something_other() > > except: > > try: > > undo_something() > > finally: > > raise > > > > Unfortunately, that breaks down on Python 3, as the new exception > > propogates with the original as context.. > > Relevant tracker issue for that problem: > https://bugs.python.org/issue18861 > > >> Does it mean that we should rewrite every chunk of code similar to the > >> above? And if such cumbersome code is necessary and become common, maybe > >> it needs syntax support in the language? Or changing the semantic of > >> exceptions raised in error handlers and finally blocks? > > > > What I want is a way to chain exceptions in the other direction, > > raising the original error, but attaching a related one. Unfortunately > > neither __cause__ or __context__ really help. > > Aye, agreed. The key challenge we have is that we're trying to > represent the exception state as a linked list, when what we really > have once we start taking cleanup errors into account is an exception > *tree*. > > The thing that finally clicked for me in this thread is that if we add > contextlib.ResourceSet, and have it add a new attribute to the > original exception (e.g. "__cleanup_errors__") with a list of > exceptions raised while attempt to cleanup after the earlier > exception, then that lets us actually start modelling that tree at > runtime. > You might want to go back and read Jake's LWN article for this year's language summit as Nathaniel Smith said he wanted some sort of multi-exception type which would go along with this idea. -Brett > > Once we understand the *behaviour* we want, *then* we can consider > whether we might want to add a context manager to have any raised > exceptions be attached to the currently active exception as cleanup > errors rather than as new exceptions in their own right. > > For example: > > do_something() > try: > do_something_other() > except BaseException as exc: > with contextlib.cleanup(exc) as reraise: > # Exceptions raised in here would be added to > # exc.__cleanup_errors__, rather than being > # propagated normally > undo_something() > reraise() > > The need for the "as reraise:"/"reraise()" trick arises from the need > to special case the situation where the original exception inherits > from Exception, but one of the raised exceptions *doesn't* - we want > SystemExit/KeyboardInterrupt/etc to take precedence in that case, and > a bare raise statement won't handle that for us (it *could* in a > hypothetical future version of Python that's natively > `__cleanup_errors__` aware, but that's not going to be useful for > existing versions). > > Since I don't see anything in the discussion so far that *requires* > changes to the standard library (aside from "we may want to use this > ourselves"), the right place to thrash out the design details is > probably contextlib2: https://github.com/jazzband/contextlib2 > > That's where contextlib.ExitStack was born, and I prefer using it to > iterate on context management design concepts, since we can push > updates out faster, and if we make bad choices anywhere along the way, > they can just sit around in contextlib2, rather than polluting the > standard library indefinitely. > > Cheers, > Nick. > > P.S. trio's MultiError is also likely worth looking into in this context > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From songofacandy at gmail.com Mon Jun 12 14:42:43 2017 From: songofacandy at gmail.com (INADA Naoki) Date: Tue, 13 Jun 2017 03:42:43 +0900 Subject: [Python-Dev] PEP 538 warning at startup: please remove it In-Reply-To: References: Message-ID: >> -1 for disable coercion by default: It's too unhelpful for beginners. > > Are you proposing to reject the PEP that you approved? Now I'm confused. > No, I just wanted to clarify your propose. You said "I'm able to render my hello world with the wrong locale :-)". So I want to clarify you didn't mean "Python 3.6 can print ASCII in C locale, so stop coercion and warning by default". Now I think I understand your point. > > Why would user need to be annoyed by a warning while Python just do > the "right thing" for them? Hmm, I didn't have enough confident about locale coercion is the "right thing" always. But since current PEP 538 changes only LC_CTYPE, there are very small chance for unwanted side-effect. I agree it's small enough to do the coercion silently. There are some unhappy feedback about warning already. Unless I see some unhappy feedback about coercion, I'm +1 to remove the warning at some point before 3.7.0. And I change my -0.5 to -0 about removing it right now. Regards, From tjreedy at udel.edu Mon Jun 12 16:11:12 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 12 Jun 2017 16:11:12 -0400 Subject: [Python-Dev] PEP 538 warning at startup: please remove it In-Reply-To: References: Message-ID: On 6/12/2017 5:35 AM, INADA Naoki wrote: > On Mon, Jun 12, 2017 at 5:56 PM, Victor Stinner > wrote: >> Hi, >> >> Nick Coghlan pushed his implementation of his PEP 538: nice! Nice step >> forward to UTF-8 everywhere ;-) >> >> I would prefer to not be annoyed by warning messages about encodings >> at startup if possible: >> >> "Python detected LC_CTYPE=C: LC_CTYPE coerced to C.UTF-8 (set another >> locale or PYTHONCOERCECLOCALE=0 to disable this locale coercion >> behavior)." I do not see this with my fresh 3.7 repository debug build on my Win 10 machine. If I did, I would be -1000. This warning should be treated as a deprecation warning, off by default. > This is intentional behavior, to motivate people to use UTF-8 locale without > coercion. I thought the idea of the PEP was to do the right thing without making people do anything. >> Note: I don't consider that 2>/dev/null is a good practice to ignore a >> single warning, since it will ignore *all* message written into >> stderr, including useful warnings like ResourceWarning or "Exception >> ignored ...". > > I think "Good practice" is set `LC_CTYPE=C.UTF-8` environment variable, > as warning says. I do not have any locale-related env vars. You should check whether the warning is off on all Win10 systems, as well as Win7 and Win8. Many Windows users know nothing about ENV VARS, and even if they do, they may not know how (I don't know the details) or MAY NOT BE ABLE to set one permanently. -- Terry Jan Reedy From p.f.moore at gmail.com Mon Jun 12 16:24:31 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 12 Jun 2017 21:24:31 +0100 Subject: [Python-Dev] PEP 538 warning at startup: please remove it In-Reply-To: References: Message-ID: On 12 June 2017 at 21:11, Terry Reedy wrote: > I do not see this with my fresh 3.7 repository debug build on my Win 10 > machine. If I did, I would be -1000. This warning should be treated > as a deprecation warning, off by default. My understanding is that this is a Unix-only change. If it affects Windows, I'll add my voice to the complaints that we don't want a warning by default. (But if it's Unix-only, I'm happy to leave the comments to the people who are affected). Paul From v+python at g.nevcal.com Mon Jun 12 16:30:14 2017 From: v+python at g.nevcal.com (Glenn Linderman) Date: Mon, 12 Jun 2017 13:30:14 -0700 Subject: [Python-Dev] PEP 538 warning at startup: please remove it In-Reply-To: References: Message-ID: <35dc10bd-8ee1-349c-e0bb-76caba99a2b2@g.nevcal.com> On 6/12/2017 1:11 PM, Terry Reedy wrote: > I do not have any locale-related env vars. You should check whether > the warning is off on all Win10 systems, as well as Win7 and Win8. > Many Windows users know nothing about ENV VARS, and even if they do, > they may not know how (I don't know the details) or MAY NOT BE ABLE to > set one permanently. where (latest Win10): in the control panel, System and Security, System, Advanced system settings (left edge menu item), Advanced Tab, Environment Variables (button). Two sets of variables. User should always be able to set User variables that are in effect for all new processes launched after creation of the variable. System variables may be locked down if not administrator. Both types are permanent, stored in the registry. From a command line, batch file, or program launcher, temporary, local environment variables can be set that only apply to other programs in that session and that session's child processes. So there is no fear of being unable to set an environment variable on Windows. Naturally, some users may not know what they are or how to set them, but it is well-documented in various places... just google "how to set windows environment variables". -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon Jun 12 16:48:37 2017 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 12 Jun 2017 13:48:37 -0700 Subject: [Python-Dev] PEP 538 warning at startup: please remove it In-Reply-To: References: Message-ID: On Jun 12, 2017 10:50, "Gregory P. Smith" wrote: The problem, as with all warnings, is that it isn't the user who has control over the problem who sees the warning. It is the end use of an application on a system that sees it. I don't think I understand the distinction you're making here. This isn't like DeprecationWarnings where python is trying to tell the application developer that they need to fix something, and the application user gets caught in the crossfire. In this case there's nothing the application developer can do ? the problem is that the end user has their system misconfigured, and probably should fix their .bashrc or crontab something before running the application. Or maybe they shouldn't and python should just do its thing and it's fine. But if anyone's going to do something it would have to be the end user, right? (I don't have an opinion on the warning itself; I'd be mildly interested to discover which of my systems are broken in this fashion but it doesn't affect me much either way. It does occur to me though that it will probably make some sysadmins *extremely* cranky: the ones that have crontabs with broken locales and which are set to email if their script generates any output ? I think these are both still cron defaults ? and who after upgrading to py37 will suddenly find copies of this warning pouring into their inboxes at a rate of hundreds per hour.) -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Mon Jun 12 16:58:40 2017 From: greg at krypto.org (Gregory P. Smith) Date: Mon, 12 Jun 2017 20:58:40 +0000 Subject: [Python-Dev] PEP 538 warning at startup: please remove it In-Reply-To: References: Message-ID: On Mon, Jun 12, 2017 at 1:48 PM Nathaniel Smith wrote: > On Jun 12, 2017 10:50, "Gregory P. Smith" wrote: > > > The problem, as with all warnings, is that it isn't the user who has > control over the problem who sees the warning. It is the end use of an > application on a system that sees it. > > > I don't think I understand the distinction you're making here. This isn't > like DeprecationWarnings where python is trying to tell the application > developer that they need to fix something, and the application user gets > caught in the crossfire. In this case there's nothing the application > developer can do ? the problem is that the end user has their system > misconfigured, and probably should fix their .bashrc or crontab something > before running the application. Or maybe they shouldn't and python should > just do its thing and it's fine. But if anyone's going to do something it > would have to be the end user, right? > > (I don't have an opinion on the warning itself; I'd be mildly interested > to discover which of my systems are broken in this fashion but it doesn't > affect me much either way. It does occur to me though that it will probably > make some sysadmins *extremely* cranky: the ones that have crontabs with > broken locales and which are set to email if their script generates any > output ? I think these are both still cron defaults ? and who after > upgrading to py37 will suddenly find copies of this warning pouring into > their inboxes at a rate of hundreds per hour.) > > -n > I guess so. It is a system environment configuration problem. But I'd still imagine that end users never touch their LANG or LC_* environment variables themselves. (the only time I touch anything is to explicitly set LANG=C on occasion when I want text process shell tools to run a lot faster; but I hardly count as I live in ascii-land, aka the US) I'm inclined to suggest it should be silent, but I'm curious if we can get some linux distros testing with it enabled during the beta period to see how much impact it actually has on users in the real world. We could be talking about a minor non-issue. -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon Jun 12 20:26:07 2017 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 12 Jun 2017 17:26:07 -0700 Subject: [Python-Dev] Handle errors in cleanup code In-Reply-To: References: Message-ID: On Mon, Jun 12, 2017 at 6:29 AM, Stefan Ring wrote: > > > Yury in the comment for PR 2108 [1] suggested more complicated code: > > > > do_something() > > try: > > do_something_other() > > except BaseException as ex: > > try: > > undo_something() > > finally: > > raise ex > > And this is still bad, because it loses the back trace. The way we do it is: > > do_something() > try: > do_something_other() > except BaseException as ex: > tb = sys.exc_info()[2] > try: > undo_something() > finally: > raise ex, None, tb Are you testing on python 2? On Python 3 just plain 'raise ex' seems to give a sensible traceback for me... -n -- Nathaniel J. Smith -- https://vorpus.org From nad at python.org Tue Jun 13 00:09:04 2017 From: nad at python.org (Ned Deily) Date: Tue, 13 Jun 2017 00:09:04 -0400 Subject: [Python-Dev] IMPORTANT: Python 3.6.2 Maintenance Release Release Candidate in 3+ days (Monday 2017-06-12 12:00 UTC) In-Reply-To: <9F658EB5-9DF3-4E1F-87A4-41ECF303924F@python.org> References: <9F658EB5-9DF3-4E1F-87A4-41ECF303924F@python.org> Message-ID: <542D74F2-85FF-4B5F-A5D7-553A394BE6F5@python.org> An update on 3.6.2rc1: we have been working through a few critical and release blocker issues that needed to be fixed for 3.6.2. Thanks to everyone who has helped with them! At the moment, we have one security-related release blocker (http://bugs.python.org/issue29591) which I think we need to get addressed in 3.6.2. So, I'm delaying 3.6.2rc1 until we can get that resolved. That means that, for the moment, changes going into the 3.6 branch will likely make in into 3.6.2. I'll let you know when we are ready to tag. Thanks! --Ned On Jun 8, 2017, at 23:34, Ned Deily wrote: > We are approaching the end of the second calendar quarter of 2017 and, according to PEP 494, it's time to start producing the second maintenance release for the 3.6 series. The schedule calls for the release candidate to be produced on Monday 2017-06-12 UTC. As was the case with previous 3.6.x releases, the plan is for the release candidate to be the same as the final release, that is, no additional changes go in after the release candidate except for any showstopper critical problems that might be discovered with rc1. So please plan to get any security fixes, bug fixes, and documentation changes you think should be in 3.6.2 merged in ASAP. The 3.6.2 final is planned for two weeks following rc1, that is, on 2017-06-26. The next 3.6 maintenance release (3.6.3) is planned to follow about 3 months later, so most likely in 2017-09. > > A reminder that the 3.6 branch will remain open for checkins throughout the 3.6.2rc and final cycle. Once 3.6.2rc1 is tagged, new commits to the 3.6 branch will release with 3.6.3. As always, if you find any problem in 3.6.2rc1 that you may believe should be corrected in 3.6.2, please ensure the problem is documented in a new or existing open issue on bugs.python.org, ensure that the Priority field of the issue is set to "release blocker", and that "Python 3.6" is included in the selected Versions. Comments or tags on github Pull Requests are NOT sufficient! > > Thanks again for all of your efforts in bringing 3.6 into the world and for helping now to make it even better! > > https://www.python.org/dev/peps/pep-0494/ > http://cpython-devguide.readthedocs.io -- Ned Deily nad at python.org -- [] From njs at pobox.com Tue Jun 13 00:10:05 2017 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 12 Jun 2017 21:10:05 -0700 Subject: [Python-Dev] Handle errors in cleanup code In-Reply-To: References: Message-ID: On Mon, Jun 12, 2017 at 1:07 AM, Nick Coghlan wrote: > Aye, agreed. The key challenge we have is that we're trying to > represent the exception state as a linked list, when what we really > have once we start taking cleanup errors into account is an exception > *tree*. [...] > P.S. trio's MultiError is also likely worth looking into in this context Huh, yeah, this is some interesting convergent evolution. trio.MultiError is exactly a way of representing a tree of exceptions, though it's designed to do that for "live" exceptions rather than just context chaining. Briefly... the motivation here is that if you have multiple concurrent call-stacks (threads/tasks/whatever-your-abstraction-is-called) running at the same time, then it means you can literally have multiple uncaught exceptions propagating out at the same time, so you need some strategy for dealing with that. One popular option is to avoid the problem by throwing away exceptions that propagate "too far" (e.g., in the stdlib threading module, if an exception hits the top of the call stack of a non-main thread, then it gets printed to stderr and then the program continues normally). Trio takes a different approach: its tasks are arranged in a tree, and if a child task exits with an exception then that exception gets propagated into the parent task. This allows us avoid throwing away exceptions, but it means that we need some way to represent the situation when a parent task has *multiple* live exceptions propagate into it at the same time. That's what trio.MultiError is for. Structurally, MultiError is just an Exception that holds a list of child exceptions, like MultiError([TypeError(), OSError(), KeyboardInterrupt()]) so that they can propagate together. One design decision though is that if you put a MultiError inside a MultiError, it *isn't* collapsed, so it's also legal to have something like MultiError([ TypeError(), MultiError([OSError(), KeyboardInterrupt()]), ]) Semantically, these two MultiErrors are mostly the same; they both represent a situation where there are 3 unhandled exceptions propagating together. The reason for keeping the tree structure is that if the inner MultiError propagated for a while on its own before meeting the TypeError, then it accumulated some traceback and we need somewhere to store that information. (This generally happens when the task tree has multiple levels of nesting.) The other option would be to make two copies of this part of the traceback and attach one copy onto each of the two exceptions inside it, (a) but that's potentially expensive, and (b) if we eventually print the traceback then it's much more readable if we can say "here's where OSError was raised, and where KeyboardInterrupt was raised, and here's where they traveled together" and only print the common frames once. There's some examples of how this works on pages 38-49 of my language summit slides here: https://vorpus.org/~njs/misc/trio-language-summit-2017.pdf And here's the source for the toy example programs that I used in the slides, in case anyone wants to play with them: https://gist.github.com/njsmith/634b596e5765d5ed8b819a4f8e56d306 Then the other piece of the MultiError design is catching them. This is done with a context manager called MultiError.catch, which "maps" an exception handler (represented as a callable) over all the exceptions that propagate through it, and then simplifies the MultiError tree to collapse empty and singleton nodes. Since the exceptions inside a MultiError represent independent, potentially unrelated errors, you definitely don't want to accidentally throw away that KeyboardInterrupt just because your code knows how to handle the OSError. Or if you have something like MultiError([OSError(), TypeError()]) then trio has no idea which of those exceptions might be the error you know how to catch and handle which might be the error that indicates some terrible bug that should abort the program, so neither is allowed to mask the other - you have to decide how to handle them independently. If anyone wants to dig into it the code is here: https://github.com/python-trio/trio/blob/master/trio/_core/_multierror.py Anyway, compared to the __cleanup_errors__ idea: - Both involve a collection object that holds exceptions, but in the MultiError version the collection subclasses BaseException. One consequence is that you can put the exception collection object directly into __context__ or __cause__ instead of using a new attribute. - MultiError allows for a tree structure *within* a single collection of parallel exceptions. (And then of course on top of that each individual exception within the collection can have its own chain attached.) Since the motivation for this is wanting to preserve traceback structure accumulated while the collection was propagating, and __cleanup_errors__ is only intended for "dead" expections that don't propagate, this is solving an issue that __cleanup_errors__ doesn't have. - OTOH, it's not clear to me that you *want* to always stick cleanup errors into a __context__-like attribute where they'll be mostly ignored. Forcing the developer to guess ahead of time whether it's the original error or the cleanup errors that are most important seems pretty, well, error-prone. Maybe it would be more useful to have a version of ExitStack that collects up the errors from inside the block and from cleanup handlers and then raises them all together as a MultiError. (If nothing else, this would let you avoid having to guess which exceptions are more important than others, like you mention with your reraise() idea for trying to prioritize KeyboardInterrupt over other exceptions.) > Since I don't see anything in the discussion so far that *requires* > changes to the standard library (aside from "we may want to use this > ourselves"), the right place to thrash out the design details is so > probably contextlib2: https://github.com/jazzband/contextlib2 > > That's where contextlib.ExitStack was born, and I prefer using it to > iterate on context management design concepts, since we can push > updates out faster, and if we make bad choices anywhere along the way, > they can just sit around in contextlib2, rather than polluting the > standard library indefinitely. I'd also be open to extracting MultiError into a standalone library that trio and contextlib2 both consume, if there was interest in going that way. -n -- Nathaniel J. Smith -- https://vorpus.org From ncoghlan at gmail.com Tue Jun 13 02:51:23 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 13 Jun 2017 16:51:23 +1000 Subject: [Python-Dev] PEP 538 warning at startup: please remove it In-Reply-To: References: Message-ID: On 13 June 2017 at 06:58, Gregory P. Smith wrote: > I guess so. It is a system environment configuration problem. But I'd still > imagine that end users never touch their LANG or LC_* environment variables > themselves. (the only time I touch anything is to explicitly set LANG=C on > occasion when I want text process shell tools to run a lot faster; but I > hardly count as I live in ascii-land, aka the US) > > I'm inclined to suggest it should be silent, but I'm curious if we can get > some linux distros testing with it enabled during the beta period to see how > much impact it actually has on users in the real world. We could be talking > about a minor non-issue. It's at least affecting Fedora's build system (where we've backported PEP 538 to the system Python in F26): https://lists.fedorahosted.org/archives/list/python-devel at lists.fedoraproject.org/thread/VSEGOF76XMBJOAO4C2MORNK3I2VIPOTU/ However, Fedora python-devel's current view on that is that it's a sign that the build system's configuration is outdated, and we should be setting the locale to C.UTF-8 instead. (Ditto for the Fedora Docker images, which are currently expecting systemd to sort out the default locale settings, so it ends up being left as the C default when there's no init system running inside the container). So I think the most compelling argument in favour of making the warning silent by default is Victor's one: so that "LANG=C python" "just works", rather than attempting to persuade end users to change what they've become accustomed to typing in order to get software to emit otherwise internationalised text in English. After all, one of the long term goals of PEP 538 is to get to the point where at least the glibc developers, and hopefully other C standard library implementation developers, are comfortable having UTF-8 as the default text encoding anyway, at which point CPython's locale coercion would essentially become a no-op (and folks would have to type something like "LANG=C.ASCII" or "LANG=C.latin-1" to get the old platform dependent non-UTF-8 behaviour back). If we turned the warning off by default, but retained an easy "on-switch" for the warning to help track places where locale coercion is triggering unexpectedly (e.g. "PYTHONCOERCECLOCALE=warn"), that would probably give us the best of both worlds. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Jun 13 03:10:57 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 13 Jun 2017 17:10:57 +1000 Subject: [Python-Dev] Handle errors in cleanup code In-Reply-To: References: Message-ID: On 13 June 2017 at 14:10, Nathaniel Smith wrote: > On Mon, Jun 12, 2017 at 1:07 AM, Nick Coghlan wrote: >> Since I don't see anything in the discussion so far that *requires* >> changes to the standard library (aside from "we may want to use this >> ourselves"), the right place to thrash out the design details is so >> probably contextlib2: https://github.com/jazzband/contextlib2 >> >> That's where contextlib.ExitStack was born, and I prefer using it to >> iterate on context management design concepts, since we can push >> updates out faster, and if we make bad choices anywhere along the way, >> they can just sit around in contextlib2, rather than polluting the >> standard library indefinitely. > > I'd also be open to extracting MultiError into a standalone library > that trio and contextlib2 both consume, if there was interest in going > that way. I think that would make sense, as it occurred to me while reading your post that a construct like MultiError may also be useful when reporting failures from concurrent.futures.wait: https://pythonhosted.org/futures/#concurrent.futures.wait At the moment, it isn't at all clear how best to throw an exception that reports *all* of the raised exceptions in a concurrent map call, rather than just the first failure. So the recurring element we have that's common across all 3 scenarios is the notion of "peer exceptions", where we're unwinding the stack for multiple reasons, and we want to keep track of all of them. The "failed resource cleanup" case then becomes an asymmetric variant of that, where there's one primary exception (the one that triggered the cleanup), and one or more secondary failures. Given MultiError, that could be modelled as a two-tiered tree: class CleanupError(MultiError): pass CleanupError([ original_exc, MultiError([*cleanup_errors]), ]) And if there was no original exception and the resource cleanup failed unprovoked, you'd indicated that by having just a single child rather than two: CleanupError([ MultiError([*cleanup_errors]), ]) Figuring out how to *display* an exception tree coherently is going to be a pain (it's already problematic with just the linked list), but if we can at least model exception trees consistently, then we'd be able to share that display logic, even if the scenarios resulting in MultiErrors varied. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From stefanrin at gmail.com Tue Jun 13 04:32:48 2017 From: stefanrin at gmail.com (Stefan Ring) Date: Tue, 13 Jun 2017 10:32:48 +0200 Subject: [Python-Dev] Handle errors in cleanup code In-Reply-To: References: Message-ID: On Tue, Jun 13, 2017 at 2:26 AM, Nathaniel Smith wrote: > On Mon, Jun 12, 2017 at 6:29 AM, Stefan Ring wrote: >> >> > Yury in the comment for PR 2108 [1] suggested more complicated code: >> > >> > do_something() >> > try: >> > do_something_other() >> > except BaseException as ex: >> > try: >> > undo_something() >> > finally: >> > raise ex >> >> And this is still bad, because it loses the back trace. The way we do it is: >> >> do_something() >> try: >> do_something_other() >> except BaseException as ex: >> tb = sys.exc_info()[2] >> try: >> undo_something() >> finally: >> raise ex, None, tb > > Are you testing on python 2? On Python 3 just plain 'raise ex' seems > to give a sensible traceback for me... Yes, on Python 2. Interesting to know that this has changed in Python 3. I'll check this out immediately. From njs at pobox.com Tue Jun 13 04:43:54 2017 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 13 Jun 2017 01:43:54 -0700 Subject: [Python-Dev] Handle errors in cleanup code In-Reply-To: References: Message-ID: On Tue, Jun 13, 2017 at 12:10 AM, Nick Coghlan wrote: > > reporting failures from concurrent.futures.wait: > https://pythonhosted.org/futures/#concurrent.futures.wait Yeah, and asyncio.gather is another example in the stdlib. Or there's twisted's DeferredList. Trio is unusual in effectively forcing all tasks to be run under a gather(), but the basic concept isn't unique at all. > Figuring out how to *display* an exception tree coherently is going to > be a pain (it's already problematic with just the linked list), but if > we can at least model exception trees consistently, then we'd be able > to share that display logic, even if the scenarios resulting in > MultiErrors varied. > It's true, it was a pain :-). And I'm sure it can be refined. But I think this is a reasonable first cut: https://github.com/python-trio/trio/blob/9e0df6159e55fe5e389ae5e24f9bbe51e9b77943/trio/_core/_multierror.py#L341-L388 Basically the algorithm there is: if there's a __cause__ or __context__: print it (recursively using this algorithm) print the description line ("this exception was the direct cause" or whatever is appropriate) print the traceback and str() attached to this exception itself for each embedded exception: print "Details of embedded exception {i}:" with extra indentation: print the embedded exception (recursively using this algorithm) (+ some memoization to avoid infinite recursion on loopy structures) Of course really complicated trainwrecks that send exception shrapnel flying everywhere can still be complicated to read, but ... that's why we make the big bucks, I guess. (My fingers initially typoed that as "that's why we make the big bugs". Just throwing that out there.) Currently that code has a hard-coded assumption that the only kind of exception-container is MultiError, but I guess it wouldn't be too hard to extend that into some kind of protocol that your asymmetric CleanupError could also participate in? Like: an abstract exception container has 0 or 1 (predecessor exception, description) pairs [i.e., __cause__ or __context__], plus 0 or more (embedded exception, description) pairs, and then the printing code just walks the resulting tree using the above algorithm? If this all works out at the library level, then one can start to imagine ways that the interpreter could potentially get benefits from participating: - right now, if an exception that already has a __cause__ or __context__ is re-raised, then implicit exception chaining will overwrite the old __cause__ or __context__. Instead, it could check to see if there's already a non-None value there, and if so do exc.__context__ = MultiError([exc.__context__, new_exc]). - standardizing the protocol for walking over an exception container would let us avoid fighting over who owns sys.excepthook, and make it easier for third-party exception printers (like IPython, pytest, etc.) to play along. - MultiError.catch is a bit awkward. If we were going all-in on this then one can imagine some construct like multitry: ... do stuff ... raise MultiError([ValueError(1), MultiError([TypeError(), ValueError(2)])]) multiexcept ValueError as exc: print("caught a ValueError", exc) multiexcept TypeError: print("caught a TypeError and raising RuntimeError") raise RuntimeError which prints caught a ValueError 1 caught a TypeError and raising RuntimeError caught a ValueError 2 and then raises a RuntimeError with its __context__ pointing to the TypeError. ...but of course that's preeeeetty speculative at this point; definitely more python-ideas territory. -n -- Nathaniel J. Smith -- https://vorpus.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Tue Jun 13 05:07:05 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 13 Jun 2017 11:07:05 +0200 Subject: [Python-Dev] PEP 538 warning at startup: please remove it In-Reply-To: References: Message-ID: Ok, if you want to have a more concrete example of regression introduced by this warning, look at test_tracemalloc failures on this FreeBSD buildbot: http://buildbot.python.org/all/builders/AMD64%20FreeBSD%20CURRENT%20Debug%203.x/builds/419/steps/test/logs/stdio I expect that a *lot* of tests using stderr will be broken by this warning. I'm not interested to discuss if the CI is properly configured or not. The thing is the warning can easily be seen as a regression: the test pass on 3.6 on the same platform ;-) One example: ====================================================================== FAIL: test_conflicting_envvar_and_command_line (test.test_warnings.CEnvironmentVariableTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_warnings/__init__.py", line 1062, in test_conflicting_envvar_and_command_line b"DeprecationWarning: Message"]) AssertionError: Lists differ: [b'Python runtime initialized with LC_CTYPE=[305 chars]age'] != [b'Traceback (most recent call last):', b' [65 chars]age'] First differing element 0: b'Python runtime initialized with LC_CTYPE=[190 chars]ded.' b'Traceback (most recent call last):' First list contains 1 additional elements. First extra element 3: b'DeprecationWarning: Message' - [b'Python runtime initialized with LC_CTYPE=C (a locale with default ASCII enco' - b'ding), which may cause Unicode compatibility problems. Using C.UTF-8, C.utf8' - b', or UTF-8 (if available) as alternative Unicode-compatible locales is recom' - b'mended.', - b'Traceback (most recent call last):', ? ^ + [b'Traceback (most recent call last):', ? ^ b' File "", line 1, in ', b'DeprecationWarning: Message'] Victor 2017-06-12 10:56 GMT+02:00 Victor Stinner : > Hi, > > Nick Coghlan pushed his implementation of his PEP 538: nice! Nice step > forward to UTF-8 everywhere ;-) > > I would prefer to not be annoyed by warning messages about encodings > at startup if possible: > > "Python detected LC_CTYPE=C: LC_CTYPE coerced to C.UTF-8 (set another > locale or PYTHONCOERCECLOCALE=0 to disable this locale coercion > behavior)." > > > Python 3.7 is the only programming language that I know that complains > about encoding at startup. Even if my code is 100% valid, PEP 8 > compliant, don't emit any warning, etc. I will get the warning. It is > *not* possible to ignore the warning... Like "yeah, I know that my > locale is C, but it's not like I'm able to configure it." > > haypo at selma$ export LANG= > haypo at selma$ locale > LANG= > LC_CTYPE="POSIX" > ... > > haypo at selma$ ./python -c 'print("Hello World!")' # Python 3.7 > Python detected LC_CTYPE=C: LC_CTYPE coerced to C.UTF-8 (set another > locale or PYTHONCOERCECLOCALE=0 to disable this locale coercion > behavior). > Hello World! > > haypo at selma$ python2 -c 'print("Hello World!")' # Python 2.7 > Hello World! > > haypo at selma$ perl -e 'print "Hello, world!\n"' # Perl 5.24 > Hello, world! > > haypo at selma$ ./c # C > Hello World! > > ... > > Note: I don't consider that 2>/dev/null is a good practice to ignore a > single warning, since it will ignore *all* message written into > stderr, including useful warnings like ResourceWarning or "Exception > ignored ...". > > Victor From victor.stinner at gmail.com Tue Jun 13 05:48:11 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 13 Jun 2017 11:48:11 +0200 Subject: [Python-Dev] PEP 538 warning at startup: please remove it In-Reply-To: References: Message-ID: 2017-06-13 8:51 GMT+02:00 Nick Coghlan : > If we turned the warning off by default, but retained an easy > "on-switch" for the warning to help track places where locale coercion > is triggering unexpectedly (e.g. "PYTHONCOERCECLOCALE=warn"), that > would probably give us the best of both worlds. Hum, it reminds me when I proposed to add a new "locale" encoding which would be used to explicitly use the locale encoding. My intent was to emit a warning when the locale encoding was used implicitely... The problem is that the locale encoding is basically used *everywhere* in the standard library and *it's fine* :-) Really, using the locale encoding is the right encoding is many use cases. Especially when you work on Windows! Ok, when I proposed the idea, distutils still needed to use UTF-8 explicitly instead of the implicit locale encoding, but you may know how painful it was to change anything in distutils: write a PEP and/or discuss it on distutils-sig, handle backward compatibility, etc. The good news is that these issues are now fixed :-) -- While I'm not opposed to PYTHONCOERCECLOCALE=warn, I'm not sure that *developers* will use it. Usually, developers don't care of Unicode until enough users complain that their application don't work on a specific configuration on a specific platform :-) So I'm not sure that it's useful. We already have many PYTHON* environment variables. I'm not sure that all of them are used. We should be careful when adding new ones ;-) https://docs.python.org/dev/using/cmdline.html#environment-variables It's already possible to implement manually a check in an application to complain if the LC_CTYPE locale "C", no? haypo at selma$ cat > x.py import locale if locale.setlocale(locale.LC_CTYPE, None) == 'C': print("Don't use the C locale") haypo at selma$ LANG= python2 x.py Don't use the C locale haypo at selma$ LANG= python3 x.py Don't use the C locale Victor From ncoghlan at gmail.com Tue Jun 13 09:07:19 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 13 Jun 2017 23:07:19 +1000 Subject: [Python-Dev] PEP 538 warning at startup: please remove it In-Reply-To: References: Message-ID: On 13 June 2017 at 19:07, Victor Stinner wrote: > Ok, if you want to have a more concrete example of regression > introduced by this warning, look at test_tracemalloc failures on this > FreeBSD buildbot: > http://buildbot.python.org/all/builders/AMD64%20FreeBSD%20CURRENT%20Debug%203.x/builds/419/steps/test/logs/stdio > > I expect that a *lot* of tests using stderr will be broken by this warning. > > I'm not interested to discuss if the CI is properly configured or not. > The thing is the warning can easily be seen as a regression: the test > pass on 3.6 on the same platform ;-) Those failures aren't due to the successful coercion warning, they're due to the "Python 3.7 has dropped support for your ASCII-only platform" warning that the interpreter ends up emitting after coercion fails :) You're already working on the more user-friendly solution to that case, by way of PEP 540 (and if we decide it's a more expedient interim solution, I'd be fine with switching the Py_WARN_ON_C_LOCALE config setting to "no" until you've had more time to work on that). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Jun 13 09:10:37 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 13 Jun 2017 23:10:37 +1000 Subject: [Python-Dev] PEP 538 warning at startup: please remove it In-Reply-To: References: Message-ID: On 13 June 2017 at 19:48, Victor Stinner wrote: > While I'm not opposed to PYTHONCOERCECLOCALE=warn, I'm not sure that > *developers* will use it. Usually, developers don't care of Unicode > until enough users complain that their application don't work on a > specific configuration on a specific platform :-) So I'm not sure that > it's useful. It isn't cross-platform app developers that I expect to find it useful, it's system integrators and operating system developers. They're users too (and those categories are largely a better description of my own Python development background than user facing app development would be). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From me at louie.lu Wed Jun 14 01:48:43 2017 From: me at louie.lu (Louie Lu) Date: Wed, 14 Jun 2017 13:48:43 +0800 Subject: [Python-Dev] IDLE internal layout Message-ID: (Due to idle-dev mailing list low traffic, this mail was sent to python-dev and idle-dev) Hi all, I'm now trying to figure out IDLE internal relation. I got some module dependency graph and count. The work is at HackMD: https://hackmd.io/IwNg7AphDGBmsFoAsBOCAmZAGMBDBKsArEgruuiEkQCa5YBGsNQA Is there any previous material on this topic? thanks Louie. From tjreedy at udel.edu Wed Jun 14 08:45:17 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 14 Jun 2017 08:45:17 -0400 Subject: [Python-Dev] IDLE internal layout In-Reply-To: References: Message-ID: On 6/14/2017 1:48 AM, Louie Lu wrote: > (Due to idle-dev mailing list low traffic, this mail was sent to > python-dev and idle-dev) > I'm now trying to figure out IDLE internal relation. > I got some module dependency graph and count. > > The work is at HackMD: > > https://hackmd.io/IwNg7AphDGBmsFoAsBOCAmZAGMBDBKsArEgruuiEkQCa5YBGsNQA > > Is there any previous material on this topic? Not that I know of. There are many delayed imports in idlelib. At least one is to avoid circular import error, as I discovered when I tried to move it to the top as suggested by PEP8. Thanks for finding the site and producing the .pngs. Even though missing a few links from importlib.import_module calls, they will be helpful. I saved the 2nd one locally so I can view it at full size. (The left column links such as debugger.py linking to 'debugger.py' are nonsensical.) -- Terry Jan Reedy From me at louie.lu Wed Jun 14 08:59:08 2017 From: me at louie.lu (Louie Lu) Date: Wed, 14 Jun 2017 20:59:08 +0800 Subject: [Python-Dev] IDLE internal layout In-Reply-To: References: Message-ID: I generate these picture with no-external, so it won't show up about importlib.import_module calls (I think) I'll try to modify the code, and re-generated the pic with importlib. 2017-06-14 20:45 GMT+08:00 Terry Reedy : > On 6/14/2017 1:48 AM, Louie Lu wrote: >> >> (Due to idle-dev mailing list low traffic, this mail was sent to >> python-dev and idle-dev) > > >> I'm now trying to figure out IDLE internal relation. >> I got some module dependency graph and count. >> >> The work is at HackMD: >> >> https://hackmd.io/IwNg7AphDGBmsFoAsBOCAmZAGMBDBKsArEgruuiEkQCa5YBGsNQA >> >> Is there any previous material on this topic? > > > Not that I know of. There are many delayed imports in idlelib. At least one > is to avoid circular import error, as I discovered when I tried to move it > to the top as suggested by PEP8. > > Thanks for finding the site and producing the .pngs. Even though missing a > few links from importlib.import_module calls, they will be helpful. I saved > the 2nd one locally so I can view it at full size. > > (The left column links such as debugger.py linking to 'debugger.py' are > nonsensical.) > > -- > Terry Jan Reedy > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/me%40louie.lu From nad at python.org Thu Jun 15 01:27:00 2017 From: nad at python.org (Ned Deily) Date: Thu, 15 Jun 2017 01:27:00 -0400 Subject: [Python-Dev] UPDATE: Python 3.6.2rc1 cutoff now scheduled for 2017-06-16 12:00 UTC In-Reply-To: <542D74F2-85FF-4B5F-A5D7-553A394BE6F5@python.org> References: <9F658EB5-9DF3-4E1F-87A4-41ECF303924F@python.org> <542D74F2-85FF-4B5F-A5D7-553A394BE6F5@python.org> Message-ID: <2D8BD598-4579-4EAD-AEA0-1A2916B02698@python.org> The security-related release blocker (bpo-29591) has been resolved; my thanks to Victor for leading the effort. As of now, I'm not aware of any other release blocker issues for the 3.6.2 release candidate so I'm rescheduling the code cutoff for 2017-06-16 12:00 UTC, approximately 30 hours from now. That gives you one last opportunity to get important fixes in for 3.6.2. As always, if you know of issues that you think need to be resolved before the 3.6.2 release, please ensure there is an open tracker issue for it on bugs.python.org and mark the issue as "release blocker". To still give 2 weeks of exposure for the rc, the new target release date for 3.6.2 final will be 2017-06-30. --Ned On Jun 13, 2017, at 00:09, Ned Deily wrote: > An update on 3.6.2rc1: we have been working through a few critical and release blocker issues that needed to be fixed for 3.6.2. Thanks to everyone who has helped with them! At the moment, we have one security-related release blocker (http://bugs.python.org/issue29591) which I think we need to get addressed in 3.6.2. So, I'm delaying 3.6.2rc1 until we can get that resolved. That means that, for the moment, changes going into the 3.6 branch will likely make in into 3.6.2. I'll let you know when we are ready to tag. > > On Jun 8, 2017, at 23:34, Ned Deily wrote: >> We are approaching the end of the second calendar quarter of 2017 and, according to PEP 494, it's time to start producing the second maintenance release for the 3.6 series. The schedule calls for the release candidate to be produced on Monday 2017-06-12 UTC. As was the case with previous 3.6.x releases, the plan is for the release candidate to be the same as the final release, that is, no additional changes go in after the release candidate except for any showstopper critical problems that might be discovered with rc1. So please plan to get any security fixes, bug fixes, and documentation changes you think should be in 3.6.2 merged in ASAP. The 3.6.2 final is planned for two weeks following rc1, that is, on 2017-06-26. The next 3.6 maintenance release (3.6.3) is planned to follow about 3 months later, so most likely in 2017-09. >> >> A reminder that the 3.6 branch will remain open for checkins throughout the 3.6.2rc and final cycle. Once 3.6.2rc1 is tagged, new commits to the 3.6 branch will release with 3.6.3. As always, if you find any problem in 3.6.2rc1 that you may believe should be corrected in 3.6.2, please ensure the problem is documented in a new or existing open issue on bugs.python.org, ensure that the Priority field of the issue is set to "release blocker", and that "Python 3.6" is included in the selected Versions. Comments or tags on github Pull Requests are NOT sufficient! >> >> Thanks again for all of your efforts in bringing 3.6 into the world and for helping now to make it even better! >> >> https://www.python.org/dev/peps/pep-0494/ >> http://cpython-devguide.readthedocs.io -- Ned Deily nad at python.org -- [] From bdaboy at up.edu.ph Thu Jun 15 22:40:58 2017 From: bdaboy at up.edu.ph (Blaine Corwyn Aboy) Date: Fri, 16 Jun 2017 10:40:58 +0800 Subject: [Python-Dev] Inquiry on Buildbot Message-ID: Hello Ms/Mr Python People, I'm Corwyn Aboy and am a student of the University of the Philippines and currently working on a project to create a buildbot system on a Windows 10 machine and would like to ask for guidance in doing so. I've read the BuildbotOnWindows in your wiki.python.org website but have not yet tried it out. I would like to ask if the same steps would still apply to installing it on a Windows 10 machine. It would be really great to hear some feedback as soon as possible. Thank you for your time. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rodrigc at freebsd.org Thu Jun 15 23:49:45 2017 From: rodrigc at freebsd.org (Craig Rodrigues) Date: Thu, 15 Jun 2017 20:49:45 -0700 Subject: [Python-Dev] Inquiry on Buildbot In-Reply-To: References: Message-ID: Corwyn, This mailing list is related to development on Python itself, and not really oriented to support questions for buildbot. For setting up buildbot under Windows 10, I recommend that you go to https://buildbot.net and read the documentation there for installing buildbot. If you need further help, that web site has pointers to buildbot-specific mailing lists. The wiki page you read for BuildbotOnWindows is very old and outdated, so it is not a good guide to follow. -- Craig On Thu, Jun 15, 2017 at 7:40 PM, Blaine Corwyn Aboy wrote: > Hello Ms/Mr Python People, > > I'm Corwyn Aboy and am a student of the University of the Philippines and > currently working on a project to create a buildbot system on a Windows 10 > machine and would like to ask for guidance in doing so. I've read the > BuildbotOnWindows in your wiki.python.org website but have not yet tried > it out. > > I would like to ask if the same steps would still apply to installing it > on a Windows 10 machine. It would be really great to hear some feedback as > soon as possible. > > Thank you for your time. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > rodrigc%40freebsd.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Fri Jun 16 00:19:23 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 16 Jun 2017 07:19:23 +0300 Subject: [Python-Dev] Issue #21071: change struct.Struct.format type from bytes to str In-Reply-To: References: Message-ID: 27.03.17 15:12, Victor Stinner ????: > I would like to change struct.Struct.format type from bytes to str. I > don't expect that anyone uses this attribute, and struct.Struct() > constructor accepts both bytes and str. > > http://bugs.python.org/issue21071 > > It's just to be convenient: more functions accept str than bytes in > Python 3. Example: print() (python3 -bb raises an exceptions if you > pass bytes to print). > > Is anything opposed to breaking the backward compatibility? If nobody opposed to this change it will be made in short time. From ncoghlan at gmail.com Fri Jun 16 04:40:02 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 16 Jun 2017 18:40:02 +1000 Subject: [Python-Dev] Issue #21071: change struct.Struct.format type from bytes to str In-Reply-To: References: Message-ID: On 16 June 2017 at 14:19, Serhiy Storchaka wrote: > 27.03.17 15:12, Victor Stinner ????: >> >> I would like to change struct.Struct.format type from bytes to str. I >> don't expect that anyone uses this attribute, and struct.Struct() >> constructor accepts both bytes and str. >> >> http://bugs.python.org/issue21071 >> >> It's just to be convenient: more functions accept str than bytes in >> Python 3. Example: print() (python3 -bb raises an exceptions if you >> pass bytes to print). >> >> Is anything opposed to breaking the backward compatibility? > > > If nobody opposed to this change it will be made in short time. As long as it's noted in the "Porting to Python 3.7" section of the 3.7 What's New guide, this seems like a sensible change to me. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From victor.stinner at gmail.com Fri Jun 16 06:46:11 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 16 Jun 2017 12:46:11 +0200 Subject: [Python-Dev] Issue #21071: change struct.Struct.format type from bytes to str In-Reply-To: References: Message-ID: 2017-06-16 10:40 GMT+02:00 Nick Coghlan : > As long as it's noted in the "Porting to Python 3.7" section of the > 3.7 What's New guide, this seems like a sensible change to me. Yes, the change is already documented there: https://github.com/python/cpython/pull/845/files Victor From victor.stinner at gmail.com Fri Jun 16 12:05:12 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 16 Jun 2017 18:05:12 +0200 Subject: [Python-Dev] New work-in-progress bisection tool for the Python test suite (in regrtest) Message-ID: Hi, Last weeks, I worked on a new tool to bisect failing tests because it's painful to bisect manually reference leaks (I remove as much code as possible until the code is small enough to be reviewable manually). See the bisect_test.py script attached to this issue: http://bugs.python.org/issue29512 With the help of Louie Lu, I added new --list-cases option to "python -m test", so you can now list all test cases and write it into a text file: ./python -m test --list-cases test_os > tests I also added a new --matchfile option, to filter tests using a text file which contains one pattern per line: ./python -m test --matchfile=tests test_os fnmatch is used to match test names, so "*" joker character can be used in test names. My bisection tool takes a text file with the --matchfile format (one pattern per line) and creates a random subset of tests with half of the tests. If tests still fail, use the subset. Otherwise, create a new random subset. Loop until the subset contains a single test (configurable threshold, -n command line option). The long term plan is to integrate the bisection feature directly into regrtest. Right now, my script is hardcoded to bisect reference leak bugs, but it should be easy to modify it to bisect other test issues like test creating files without removing it ("ENV_CHANGED" failure in regrtest). For example, a core file is dumped when running test_subprocess on FreeBSD buildbots: http://bugs.python.org/issue30448 But I'm unable to reproduce the issue on my FreeBSD. It would be nice to be able to automate the bisection on the buildbot directly. --list-cases and --matchfile options are now available in 2.7, 3.5, 3.6 and master (3.7) branches. TODO: doctest tests are only partially supported, see: http://bugs.python.org/issue30683 Victor From status at bugs.python.org Fri Jun 16 12:09:16 2017 From: status at bugs.python.org (Python tracker) Date: Fri, 16 Jun 2017 18:09:16 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20170616160916.AD69156BD5@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2017-06-09 - 2017-06-16) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 6034 ( -1) closed 36405 (+70) total 42439 (+69) Open issues with patches: 2366 Issues opened (48) ================== #17870: Python does not provide PyLong_FromIntMax_t() or PyLong_FromUi http://bugs.python.org/issue17870 reopened by Devin Jeanpierre #30616: Cannot use functional API to create enum with zero values http://bugs.python.org/issue30616 opened by Gerrit.Holl #30617: IDLE: Add docstrings and unittests to outwin.py http://bugs.python.org/issue30617 opened by csabella #30618: readlink for pathlib paths http://bugs.python.org/issue30618 opened by smheidrich #30619: typing.Union doc incoherence in case a class and its subclass http://bugs.python.org/issue30619 opened by khyox #30622: Fix NPN guard for OpenSSL 1.1 http://bugs.python.org/issue30622 opened by msopacua #30623: python-nightly import numpy fails since recently http://bugs.python.org/issue30623 opened by Arek Bulski #30625: Documentation is unclear how "y*" and "y#" format units vary http://bugs.python.org/issue30625 opened by Gregory.Szorc #30629: lower() is called twice http://bugs.python.org/issue30629 opened by motoki #30630: Missing MSI files http://bugs.python.org/issue30630 opened by Mark Airey #30633: Python 3.6.1 installation issues on OpenSuse 42.1: ModuleNotFo http://bugs.python.org/issue30633 opened by zopyx #30637: Syntax error reported on compile(...), but not on compile(..., http://bugs.python.org/issue30637 opened by hniksic #30638: Additional dependencies and rule for `make regen-all` http://bugs.python.org/issue30638 opened by serhiy.storchaka #30639: inspect.getfile(obj) calls object repr on failure http://bugs.python.org/issue30639 opened by takluyver #30640: NULL + 1 in _PyFunction_FastCallDict() http://bugs.python.org/issue30640 opened by serhiy.storchaka #30645: imp.py: load_package() appends to its own loop variable http://bugs.python.org/issue30645 opened by Alexandru Ardelean #30647: CODESET error on AMD64 FreeBSD 10.x Shared 3.x caused by the P http://bugs.python.org/issue30647 opened by haypo #30648: test_logout() of test_imaplib.RemoteIMAP_STARTTLSTest failed r http://bugs.python.org/issue30648 opened by haypo #30649: test_utime_current_old() of test_os fails randomy on x86 Windo http://bugs.python.org/issue30649 opened by haypo #30651: test_poplib.test_stls_context() access violation on x86 Window http://bugs.python.org/issue30651 opened by haypo #30652: test_threading_not_handled() of test_socketserver hangs random http://bugs.python.org/issue30652 opened by haypo #30653: test_socket hangs in ThreadableTest.tearDown() on AMD64 FreeBS http://bugs.python.org/issue30653 opened by haypo #30654: signal module always overwrites SIGINT on interpreter shutdown http://bugs.python.org/issue30654 opened by pkerling #30657: Unsafe arithmetic in PyString_DecodeEscape http://bugs.python.org/issue30657 opened by jaybosamiya #30658: Buildbot: don't sent email notification for custom builders http://bugs.python.org/issue30658 opened by haypo #30660: Lossless Optimization of PNG files http://bugs.python.org/issue30660 opened by tfs #30661: Support tarfile.PAX_FORMAT in shutil.make_archive http://bugs.python.org/issue30661 opened by ncoghlan #30662: fix OrderedDict.__init__ docstring to reflect PEP 468 http://bugs.python.org/issue30662 opened by jonathaneunice #30663: IDLE: Add lineno sidebar to editor window http://bugs.python.org/issue30663 opened by louielu #30664: Change unittest's _SubTest to not sort its params when printin http://bugs.python.org/issue30664 opened by eric.smith #30665: pass big values for arg to fcntl.ioctl http://bugs.python.org/issue30665 opened by ukl #30666: IDLE: add tests for autocomplete window. http://bugs.python.org/issue30666 opened by terry.reedy #30667: IDLE: revise doc subsections 'Completions' http://bugs.python.org/issue30667 opened by terry.reedy #30668: DOC: missing word in license.rst http://bugs.python.org/issue30668 opened by csabella #30670: pprint for dict in sorted order or insert order? http://bugs.python.org/issue30670 opened by josephsmeng #30671: dict: improve lookup function http://bugs.python.org/issue30671 opened by Dmitry Rubanovich #30672: PEP 538: Unexpected locale behaviour on Mac OS X http://bugs.python.org/issue30672 opened by ncoghlan #30673: Add -t option (timeout) to Tools/buildbot/test.bat for "AMD64 http://bugs.python.org/issue30673 opened by haypo #30674: IDLE: add docstrings to grep.py http://bugs.python.org/issue30674 opened by csabella #30675: test_zipfile leaks references on Python 3.5 on the wo Refleaks http://bugs.python.org/issue30675 opened by haypo #30677: Enhance documentation of os.mkdir() http://bugs.python.org/issue30677 opened by ebianchi #30678: Widget variable binding does not work if mainloop is called fr http://bugs.python.org/issue30678 opened by Javier Dehesa #30679: __aexit__ not called when `run_until_complete` is interrupted http://bugs.python.org/issue30679 opened by rthr #30680: textwrap should treat Unicode em-dash like ASCII em-dash http://bugs.python.org/issue30680 opened by jonathaneunice #30681: email.utils.parsedate_to_datetime() should return None when da http://bugs.python.org/issue30681 opened by timb07 #30682: f-string assert is too restrictive http://bugs.python.org/issue30682 opened by eric.smith #30683: Enhance doctest support in regrtest --list-cases http://bugs.python.org/issue30683 opened by haypo #30684: datetime.fromtimestamp raises OSError on first day after epoch http://bugs.python.org/issue30684 opened by lazka Most recent 15 issues with no replies (15) ========================================== #30672: PEP 538: Unexpected locale behaviour on Mac OS X http://bugs.python.org/issue30672 #30668: DOC: missing word in license.rst http://bugs.python.org/issue30668 #30667: IDLE: revise doc subsections 'Completions' http://bugs.python.org/issue30667 #30665: pass big values for arg to fcntl.ioctl http://bugs.python.org/issue30665 #30663: IDLE: Add lineno sidebar to editor window http://bugs.python.org/issue30663 #30654: signal module always overwrites SIGINT on interpreter shutdown http://bugs.python.org/issue30654 #30653: test_socket hangs in ThreadableTest.tearDown() on AMD64 FreeBS http://bugs.python.org/issue30653 #30629: lower() is called twice http://bugs.python.org/issue30629 #30622: Fix NPN guard for OpenSSL 1.1 http://bugs.python.org/issue30622 #30619: typing.Union doc incoherence in case a class and its subclass http://bugs.python.org/issue30619 #30618: readlink for pathlib paths http://bugs.python.org/issue30618 #30616: Cannot use functional API to create enum with zero values http://bugs.python.org/issue30616 #30613: Distutils register command creates non-standard multipart data http://bugs.python.org/issue30613 #30607: Extract documentation theme into a separate package http://bugs.python.org/issue30607 #30587: Mock with spec object does not ensure method call signatures http://bugs.python.org/issue30587 Most recent 15 issues waiting for review (15) ============================================= #30680: textwrap should treat Unicode em-dash like ASCII em-dash http://bugs.python.org/issue30680 #30665: pass big values for arg to fcntl.ioctl http://bugs.python.org/issue30665 #30657: Unsafe arithmetic in PyString_DecodeEscape http://bugs.python.org/issue30657 #30604: co_extra_freefuncs is stored thread locally and can lead to cr http://bugs.python.org/issue30604 #30593: sqlite3 executescript does not respect isolation_level? http://bugs.python.org/issue30593 #30560: Py_SetFatalErrorAbortFunc: Allow embedding program to handle f http://bugs.python.org/issue30560 #30541: Add restricted mocks to the python unittest mocking framework http://bugs.python.org/issue30541 #30519: Add daemon argument to Timer http://bugs.python.org/issue30519 #30512: CAN Socket support for NetBSD http://bugs.python.org/issue30512 #30509: Optimize calling type slots http://bugs.python.org/issue30509 #30502: Fix buffer handling of OBJ_obj2txt http://bugs.python.org/issue30502 #30495: IDLE: modernize textview module http://bugs.python.org/issue30495 #30487: DOC: automatically create a venv and install Sphinx when runni http://bugs.python.org/issue30487 #30485: Element.findall(path, dict) doesn't insert null namespace http://bugs.python.org/issue30485 #30484: Garbage Collector can cause Segfault whilst iterating dictiona http://bugs.python.org/issue30484 Top 10 most discussed issues (10) ================================= #29406: asyncio SSL contexts leak sockets after calling close with cer http://bugs.python.org/issue29406 18 msgs #28180: Implementation of the PEP 538: coerce C locale to C.utf-8 http://bugs.python.org/issue28180 15 msgs #17870: Python does not provide PyLong_FromIntMax_t() or PyLong_FromUi http://bugs.python.org/issue17870 10 msgs #28994: Misc fixes and cleanups in error handling C code http://bugs.python.org/issue28994 10 msgs #30523: regrtest: add --list-cases option to only display test case id http://bugs.python.org/issue30523 10 msgs #30673: Add -t option (timeout) to Tools/buildbot/test.bat for "AMD64 http://bugs.python.org/issue30673 10 msgs #29591: Various security vulnerabilities in bundled expat (CVE-2016-07 http://bugs.python.org/issue29591 9 msgs #30623: python-nightly import numpy fails since recently http://bugs.python.org/issue30623 9 msgs #30671: dict: improve lookup function http://bugs.python.org/issue30671 8 msgs #30571: Add integer formatting code for fixed-width signed arithmetic http://bugs.python.org/issue30571 7 msgs Issues closed (69) ================== #6519: Reorder 'with' statement for files in Python Tutorial http://bugs.python.org/issue6519 closed by Mariatta #11822: Improve disassembly to show embedded code objects http://bugs.python.org/issue11822 closed by serhiy.storchaka #15786: IDLE code completion window can hang or misbehave with mouse http://bugs.python.org/issue15786 closed by terry.reedy #20485: Enable non-ASCII extension module names http://bugs.python.org/issue20485 closed by ncoghlan #23890: assertRaises increases reference counter http://bugs.python.org/issue23890 closed by haypo #24484: multiprocessing cleanup occasionally throws exception http://bugs.python.org/issue24484 closed by pitrou #24744: Lack of type checks in pkgutil.walk_packages and friends http://bugs.python.org/issue24744 closed by r.david.murray #25409: fnmatch.fnmatch normalizes slashes/backslashes on Windows http://bugs.python.org/issue25409 closed by Mariatta #27425: Tests fail because of git's newline preferences on Windows http://bugs.python.org/issue27425 closed by zach.ware #27585: asyncio.Lock deadlock after cancellation http://bugs.python.org/issue27585 closed by yselivanov #28837: 2to3 does not wrap zip correctly http://bugs.python.org/issue28837 closed by Mariatta #29324: test_aead_aes_gcm fails on Kernel 4.9 http://bugs.python.org/issue29324 closed by nascheme #29478: email.policy.Compat32(max_line_length=None) not as documented http://bugs.python.org/issue29478 closed by Mariatta #29514: Add a test case that prevents magic number changes in minor re http://bugs.python.org/issue29514 closed by ncoghlan #29550: Mac build-installer touch step fails after github conversion http://bugs.python.org/issue29550 closed by ned.deily #29581: __init_subclass__ causes TypeError when used with standard lib http://bugs.python.org/issue29581 closed by serhiy.storchaka #29679: Add @contextlib.asynccontextmanager http://bugs.python.org/issue29679 closed by yselivanov #29721: "abort: repository . not found!" during the build of Python 2. http://bugs.python.org/issue29721 closed by serhiy.storchaka #29743: Closing transport during handshake process leaks open socket http://bugs.python.org/issue29743 closed by yselivanov #29783: Modify codecs.open() to use the io module instead of codecs.St http://bugs.python.org/issue29783 closed by haypo #29905: TypeErrors not formatting values correctly http://bugs.python.org/issue29905 closed by haypo #29931: ipaddress.ip_interface __lt__ check seems to be broken http://bugs.python.org/issue29931 closed by serhiy.storchaka #30039: Resuming a 'yield from' stack is broken if a signal arrives in http://bugs.python.org/issue30039 closed by yselivanov #30149: inspect.signature() doesn't support partialmethod without expl http://bugs.python.org/issue30149 closed by serhiy.storchaka #30192: hashlib module breaks with 64-bit kernel and 32-bit user space http://bugs.python.org/issue30192 closed by Mariatta #30217: Missing entry for the tilde (~) operator in the Index http://bugs.python.org/issue30217 closed by Mariatta #30231: test_imaplib needs a TLS server accepting self-signed certific http://bugs.python.org/issue30231 closed by haypo #30266: AbstractContextManager should support __method__ = None http://bugs.python.org/issue30266 closed by Mariatta #30284: Build CPython out of tree with a read-only source tree http://bugs.python.org/issue30284 closed by haypo #30312: Small correction in set code sample http://bugs.python.org/issue30312 closed by Mariatta #30335: Document deprecated alias of assertNotRegex http://bugs.python.org/issue30335 closed by Mariatta #30395: deadlocked child process after forking on pystate.c's head_mut http://bugs.python.org/issue30395 closed by lukasz.langa #30417: [buildbot] Disable the cpu resources on slow buildbots http://bugs.python.org/issue30417 closed by haypo #30501: Produce optimized code for boolean conditions http://bugs.python.org/issue30501 closed by serhiy.storchaka #30508: "Task exception was never retrieved" reported for a canceled t http://bugs.python.org/issue30508 closed by yselivanov #30524: iter(classmethod, sentinel) broken for Argument Clinic class m http://bugs.python.org/issue30524 closed by haypo #30589: With forkserver, Process.exitcode does not get signal number http://bugs.python.org/issue30589 closed by pitrou #30594: Refcounting mistake in _ssl.c http://bugs.python.org/issue30594 closed by serhiy.storchaka #30600: Error message incorrect when index is called with keyword argu http://bugs.python.org/issue30600 closed by serhiy.storchaka #30603: textwrap: declining indent level has no test case http://bugs.python.org/issue30603 closed by Mariatta #30605: re.compile fails when compiling bytes under `-bb` mode http://bugs.python.org/issue30605 closed by serhiy.storchaka #30608: argparse calculates string widths incorrectly http://bugs.python.org/issue30608 closed by terry.reedy #30609: Python 3.6.1 fails to generate 256 colors on Cygwin based 64-b http://bugs.python.org/issue30609 closed by terry.reedy #30610: Python's libexpat vulnerable to CVE-2016-0718 http://bugs.python.org/issue30610 closed by ned.deily #30611: Windows HTML Help always opens maximized http://bugs.python.org/issue30611 closed by Christian.Ullrich #30614: [EASY][2.7] testInitNonExistentFile() of test_bz2 leaks refere http://bugs.python.org/issue30614 closed by haypo #30615: [EASY][2.7] test_recursive_repr() of test_xml_etree_c leaks re http://bugs.python.org/issue30615 closed by matrixise #30620: textwrap: dedent contains logic that can never execute http://bugs.python.org/issue30620 closed by Mariatta #30621: import json before using json module in tutorial Input and Out http://bugs.python.org/issue30621 closed by Mariatta #30624: selectors should use bare except clauses http://bugs.python.org/issue30624 closed by giampaolo.rodola #30626: "SystemError: returned NULL withou http://bugs.python.org/issue30626 closed by serhiy.storchaka #30627: Incorrect error message for a few functions called with keywod http://bugs.python.org/issue30627 closed by serhiy.storchaka #30628: why venv install old pip? http://bugs.python.org/issue30628 closed by ncoghlan #30631: Warnings in VS2015 Update 3 32-Bit http://bugs.python.org/issue30631 closed by zach.ware #30632: IDLE: add unittests to test_autocomplete http://bugs.python.org/issue30632 closed by louielu #30634: ctypes.cast(obj,ctypes.c_void_p) invalid return in linux_x64 http://bugs.python.org/issue30634 closed by eryksun #30635: Leak in test_c_locale_coercion http://bugs.python.org/issue30635 closed by ncoghlan #30636: Add PYTHONCOERCECLOCALE to the help of the command line http://bugs.python.org/issue30636 closed by ncoghlan #30641: No way to specify "File name too long" error in except stateme http://bugs.python.org/issue30641 closed by r.david.murray #30642: Fix leaks in idlelib http://bugs.python.org/issue30642 closed by terry.reedy #30643: test_basic_script_no_suffix() of test_multiprocessing_main_han http://bugs.python.org/issue30643 closed by pitrou #30644: [EASY][doc] Document that signal.set_wakeup_fd(-1), unregister http://bugs.python.org/issue30644 closed by pitrou #30646: SQLite: sqlite3_enable_shared_cache() is deprecated http://bugs.python.org/issue30646 closed by ned.deily #30650: lack of right parentheses http://bugs.python.org/issue30650 closed by serhiy.storchaka #30655: speech module http://bugs.python.org/issue30655 closed by serhiy.storchaka #30656: typo in PyModule_New documentation http://bugs.python.org/issue30656 closed by Mariatta #30659: Use ** correctly for "kwargs" in signature of namedtuple._repl http://bugs.python.org/issue30659 closed by Mariatta #30669: json.tool does not accept an --indent flag http://bugs.python.org/issue30669 closed by serhiy.storchaka #30676: [Windows] Potential for GIL deadlock on Windows in threadmodul http://bugs.python.org/issue30676 closed by sdeibel From brett at python.org Fri Jun 16 13:00:49 2017 From: brett at python.org (Brett Cannon) Date: Fri, 16 Jun 2017 17:00:49 +0000 Subject: [Python-Dev] New work-in-progress bisection tool for the Python test suite (in regrtest) In-Reply-To: References: Message-ID: Great idea! I had thought about doing something similar to this years ago to help diagnose when a test passes in isolation but somehow fails due to one of the bazillion of other tests that ran previously (which can be hard to narrow down if the failure is not early on in a test run). On Fri, 16 Jun 2017 at 09:06 Victor Stinner wrote: > Hi, > > Last weeks, I worked on a new tool to bisect failing tests because > it's painful to bisect manually reference leaks (I remove as much code > as possible until the code is small enough to be reviewable manually). > > See the bisect_test.py script attached to this issue: > http://bugs.python.org/issue29512 > > With the help of Louie Lu, I added new --list-cases option to "python > -m test", so you can now list all test cases and write it into a text > file: > > ./python -m test --list-cases test_os > tests > > I also added a new --matchfile option, to filter tests using a text > file which contains one pattern per line: > > ./python -m test --matchfile=tests test_os > > fnmatch is used to match test names, so "*" joker character can be > used in test names. > > > My bisection tool takes a text file with the --matchfile format (one > pattern per line) and creates a random subset of tests with half of > the tests. If tests still fail, use the subset. Otherwise, create a > new random subset. Loop until the subset contains a single test > (configurable threshold, -n command line option). > > The long term plan is to integrate the bisection feature directly into > regrtest. > > > > Right now, my script is hardcoded to bisect reference leak bugs, but > it should be easy to modify it to bisect other test issues like test > creating files without removing it ("ENV_CHANGED" failure in > regrtest). > > For example, a core file is dumped when running test_subprocess on > FreeBSD buildbots: > > http://bugs.python.org/issue30448 > > But I'm unable to reproduce the issue on my FreeBSD. It would be nice > to be able to automate the bisection on the buildbot directly. > > > --list-cases and --matchfile options are now available in 2.7, 3.5, > 3.6 and master (3.7) branches. > > TODO: doctest tests are only partially supported, see: > > http://bugs.python.org/issue30683 > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertvandeneynde at hotmail.com Fri Jun 16 07:32:19 2017 From: robertvandeneynde at hotmail.com (Robert Vanden Eynde) Date: Fri, 16 Jun 2017 11:32:19 +0000 Subject: [Python-Dev] Language proposal: variable assignment in functional context In-Reply-To: References: Message-ID: Hello, I would like to propose an idea for the language but I don't know where I can talk about it. In a nutshell, I would like to be able to write: y = (b+2 for b = a + 1) Or in list comprehension: Y = [b+2 for a in L for b = a+1] Which can already be done like this: Y = [b+2 for a in L for b in [a+1]] Which is less obvious, has a small overhead (iterating over a list) and get messy with multiple assignment: Y = [b+c+2 for a in L for b,c in [(a+1,a+2)]] New syntax would allow to write: Y = [b+c+2 for a in L for b,c = (a+1,a+2)] The parenthesis are needed for syntax reason I think. One could have successive assignements My first example (b+2 for b = a+1) can already be done using ugly syntax using lambda y = (lambda b: b+2)(b=a+1) y = (lambda b: b+2)(a+1) y = (lambda b=a+1: b+2)() Choice of syntax: for is good because it uses current keyword, and the analogy for x = 5 vs for x in [5] is natural. But the "for" loses the meaning of iteration. The use of "with" would maybe sound more logical. Python already have the "functional if", lambdas, list comprehension, but not simple assignment functional style. -------------- next part -------------- An HTML attachment was scrubbed... URL: From shloub at gmail.com Fri Jun 16 16:18:56 2017 From: shloub at gmail.com (Shloub) Date: Fri, 16 Jun 2017 22:18:56 +0200 Subject: [Python-Dev] Language proposal: variable assignment in functional context In-Reply-To: References: Message-ID: 2017-06-16 13:32 UTC+02:00, Robert Vanden Eynde : > Hello, I would like to propose an idea for the language but I don't know > where I can talk about it. Hi, May I suggest Python-ideas? https://mail.python.org/mailman/listinfo/python-ideas Thanks. From victor.stinner at gmail.com Fri Jun 16 16:35:48 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 16 Jun 2017 22:35:48 +0200 Subject: [Python-Dev] New work-in-progress bisection tool for the Python test suite (in regrtest) In-Reply-To: References: Message-ID: 2017-06-16 19:00 GMT+02:00 Brett Cannon : > Great idea! I had thought about doing something similar to this years ago to > help diagnose when a test passes in isolation but somehow fails due to one > of the bazillion of other tests that ran previously (which can be hard to > narrow down if the failure is not early on in a test run). Yeah, sometimes a test only fails if it is run after other tests. The well known issue of side effects ;-) Each time that my script bisect_test.py script succeed to reduce the number of tests, it writes the test list on disk. So you can interrupt it the script is stuck and you consider that the output is small enough to a manual review. Victor > On Fri, 16 Jun 2017 at 09:06 Victor Stinner > wrote: >> >> Hi, >> >> Last weeks, I worked on a new tool to bisect failing tests because >> it's painful to bisect manually reference leaks (I remove as much code >> as possible until the code is small enough to be reviewable manually). >> >> See the bisect_test.py script attached to this issue: >> http://bugs.python.org/issue29512 >> >> With the help of Louie Lu, I added new --list-cases option to "python >> -m test", so you can now list all test cases and write it into a text >> file: >> >> ./python -m test --list-cases test_os > tests >> >> I also added a new --matchfile option, to filter tests using a text >> file which contains one pattern per line: >> >> ./python -m test --matchfile=tests test_os >> >> fnmatch is used to match test names, so "*" joker character can be >> used in test names. >> >> >> My bisection tool takes a text file with the --matchfile format (one >> pattern per line) and creates a random subset of tests with half of >> the tests. If tests still fail, use the subset. Otherwise, create a >> new random subset. Loop until the subset contains a single test >> (configurable threshold, -n command line option). >> >> The long term plan is to integrate the bisection feature directly into >> regrtest. >> >> >> >> Right now, my script is hardcoded to bisect reference leak bugs, but >> it should be easy to modify it to bisect other test issues like test >> creating files without removing it ("ENV_CHANGED" failure in >> regrtest). >> >> For example, a core file is dumped when running test_subprocess on >> FreeBSD buildbots: >> >> http://bugs.python.org/issue30448 >> >> But I'm unable to reproduce the issue on my FreeBSD. It would be nice >> to be able to automate the bisection on the buildbot directly. >> >> >> --list-cases and --matchfile options are now available in 2.7, 3.5, >> 3.6 and master (3.7) branches. >> >> TODO: doctest tests are only partially supported, see: >> >> http://bugs.python.org/issue30683 >> >> Victor >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/brett%40python.org From python at lucidity.plus.com Fri Jun 16 19:38:08 2017 From: python at lucidity.plus.com (Erik) Date: Sat, 17 Jun 2017 00:38:08 +0100 Subject: [Python-Dev] Language proposal: variable assignment in functional context In-Reply-To: References: Message-ID: <9362b6a9-8808-ef99-63a7-9e065a4fb52d@lucidity.plus.com> On 16/06/17 12:32, Robert Vanden Eynde wrote: > Hello, I would like to propose an idea for the language but I don't know > where I can talk about it. Robert, I have replied to this on the "python-ideas" ML. Regards, E. From steve at pearwood.info Fri Jun 16 20:27:56 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 17 Jun 2017 10:27:56 +1000 Subject: [Python-Dev] Language proposal: variable assignment in functional context In-Reply-To: References: Message-ID: <20170617002754.GH3149@ando.pearwood.info> Welcome Robert. My response below. Follow-ups to Python-Ideas, thanks. You'll need to subscribe to see any further discussion. On Fri, Jun 16, 2017 at 11:32:19AM +0000, Robert Vanden Eynde wrote: > In a nutshell, I would like to be able to write: > y = (b+2 for b = a + 1) I think this is somewhat similar to a suggestion of Nick Coghlan's. One possible syntax as a statement might be: y = b + 2 given: b = a + 1 https://www.python.org/dev/peps/pep-3150/ In mathematics, I might write: y = b + 2 where b = a + 1 although of course I wouldn't do so for anything so simple. Here's a better example, the quadratic formula: -b ? ?? x = ????????? 2a where ? = b? - 4ac although even there I'd usually write ? in place. > Python already have the "functional if", lambdas, list comprehension, > but not simple assignment functional style. I think you mean "if *expression*" rather than "functional if". The term "functional" in programming usually refers to a particular paradigm: https://en.wikipedia.org/wiki/Functional_programming -- Steve From robertvandeneynde at hotmail.com Fri Jun 16 16:25:24 2017 From: robertvandeneynde at hotmail.com (Robert Vanden Eynde) Date: Fri, 16 Jun 2017 20:25:24 +0000 Subject: [Python-Dev] Language proposal: variable assignment in functional context In-Reply-To: References: Message-ID: Thanks! Le 17 juin 2017 06:19, "Shloub" > a ?crit : 2017-06-16 13:32 UTC+02:00, Robert Vanden Eynde >: > Hello, I would like to propose an idea for the language but I don't know > where I can talk about it. Hi, May I suggest Python-ideas? https://mail.python.org/mailman/listinfo/python-ideas Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Jun 17 02:39:49 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 17 Jun 2017 16:39:49 +1000 Subject: [Python-Dev] PEP 538 (review round 2): Coercing the legacy C locale to a UTF-8 based locale In-Reply-To: References: Message-ID: On 12 June 2017 at 22:05, Nick Coghlan wrote: > On 12 June 2017 at 17:47, Martin (gzlist) wrote: >> Having thought about it a bit more, my preferred option is having the >> disable be if either LC_ALL or LC_CTYPE vars are exactly 'C', then >> don't override. Otherwise (including for LANG=C), force C.UTF-8. The >> CLI usage docs could have a LC_CTYPE entry that goes into details of >> why. > > LC_ALL=C doesn't actually disable the locale coercion (i.e. we still > set LC_CTYPE). The coercion just doesn't have any effect, since LC_ALL > takes precedence. After improving the test suite to better cover this case, it seems my assumptions regarding the behaviour of setlocale() when LC_ALL is set may have been incorrect - when LC_ALL=C is set, we *only* get the legacy locale warning, *not* the locale coercion warning (at least on Fedora - we'll know more about the behaviour on other platforms once I test my proposed resolution for https://bugs.python.org/issue30565 across the buildbot fleet). So if we chose to, we could treat an explicit "LC_CTYPE=C" the same way we treat "PYTHONCOERCECLOCALE=0" - it's definitely worth considering, so please file an RFE for that. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From robertvandeneynde at hotmail.com Fri Jun 16 16:25:25 2017 From: robertvandeneynde at hotmail.com (Robert Vanden Eynde) Date: Fri, 16 Jun 2017 20:25:25 +0000 Subject: [Python-Dev] Language proposal: variable assignment in functional context In-Reply-To: References: Message-ID: Thanks! Le 17 juin 2017 06:19, "Shloub" > a ?crit : 2017-06-16 13:32 UTC+02:00, Robert Vanden Eynde >: > Hello, I would like to propose an idea for the language but I don't know > where I can talk about it. Hi, May I suggest Python-ideas? https://mail.python.org/mailman/listinfo/python-ideas Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nad at python.org Sat Jun 17 22:31:05 2017 From: nad at python.org (Ned Deily) Date: Sat, 17 Jun 2017 22:31:05 -0400 Subject: [Python-Dev] [RELEASE] Python 3.6.2rc1 is now available for testing Message-ID: On behalf of the Python development community and the Python 3.6 release team, I would like to announce the availability of Python 3.6.2rc1. 3.6.2rc1 is the first release candidate for Python 3.6.2, the next maintenance release of Python 3.6. While 3.6.2rc1 is a preview release and, thus, not intended for production environments, we encourage you to explore it and provide feedback via the Python bug tracker (https://bugs.python.org). Please see "What?s New In Python 3.6" for more information: https://docs.python.org/3.6/whatsnew/3.6.html You can find Python 3.6.2rc1 here: https://www.python.org/downloads/release/python-362rc1/ and its change log here: https://docs.python.org/3.6/whatsnew/changelog.html#python-3-6-2-release-candidate-1 3.6.2 is planned for final release on 2017-06-30 with the next maintenance release expected to follow in about 3 months. More information about the 3.6 release schedule can be found here: https://www.python.org/dev/peps/pep-0494/ -- Ned Deily nad at python.org -- [] From steve at holdenweb.com Sun Jun 18 17:28:50 2017 From: steve at holdenweb.com (Steve Holden) Date: Sun, 18 Jun 2017 22:28:50 +0100 Subject: [Python-Dev] Inquiry on Buildbot In-Reply-To: References: Message-ID: On Fri, Jun 16, 2017 at 4:49 AM, Craig Rodrigues wrote: > This mailing list is related to development on Python itself, and not > really oriented to support > questions for buildbot. For setting up buildbot under Windows 10, I > recommend > that you go to https://buildbot.net and read the documentation there for > installing buildbot. > ?I've added a note to the Wiki page informing readers of the preferred link. S? Steve Holden -------------- next part -------------- An HTML attachment was scrubbed... URL: From damien.p.george at gmail.com Mon Jun 19 02:35:45 2017 From: damien.p.george at gmail.com (Damien George) Date: Mon, 19 Jun 2017 16:35:45 +1000 Subject: [Python-Dev] Inconsistency between PEP492, documentation and behaviour of "async with" Message-ID: Hi all, Regarding the behaviour of the "async with" statement: it seems that the description of it in PEP492, and the language documentation, do not match the behaviour of CPython (v3.6.1). The PEP and the docs here: https://www.python.org/dev/peps/pep-0492/#asynchronous-context-managers-and-async-with https://docs.python.org/3/reference/compound_stmts.html#async-with say that "async with" is equivalent to a particular use of try/except/else. But the implementation seems more like a try/except/finally, because the __aexit__ is always executed, even if a return statement is in the try block ("else" won't be executed if there's a "return" in the "try"). Also, as per normal "with", the implementation is a bit more complex than try/except/finally because you don't want to execute the __aexit__ method twice if there is an exception in the try. Can someone please clarify the exact behaviour of "async with"? Background: in implementing "async with" in MicroPython, we went by the PEP/docs, and now our behaviour doesn't match that of CPython. Cheers, Damien. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Jun 19 08:27:40 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 19 Jun 2017 22:27:40 +1000 Subject: [Python-Dev] Inconsistency between PEP492, documentation and behaviour of "async with" In-Reply-To: References: Message-ID: On 19 June 2017 at 16:35, Damien George wrote: > Hi all, > > Regarding the behaviour of the "async with" statement: it seems that the > description of it in PEP492, and the language documentation, do not match > the behaviour of CPython (v3.6.1). > > The PEP and the docs here: > https://www.python.org/dev/peps/pep-0492/#asynchronous-context-managers-and-async-with > https://docs.python.org/3/reference/compound_stmts.html#async-with > say that "async with" is equivalent to a particular use of try/except/else. > > But the implementation seems more like a try/except/finally, because the > __aexit__ is always executed, even if a return statement is in the try block > ("else" won't be executed if there's a "return" in the "try"). Also, as per > normal "with", the implementation is a bit more complex than > try/except/finally because you don't want to execute the __aexit__ method > twice if there is an exception in the try. > > Can someone please clarify the exact behaviour of "async with"? "async with" is expected to behave essentially the same way that normal "with" does as far as return, break, and continue are concerned (i.e. calling __aexit__ without an exception set, so it's more like try/finally than it is try/else). Would you mind filing a documentation bug for that? We clearly missed that the semantics described in the new documentation didn't actually match the original with statement semantics (even though matching those semantics is the intended behaviour). Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From damien.p.george at gmail.com Tue Jun 20 00:38:37 2017 From: damien.p.george at gmail.com (Damien George) Date: Tue, 20 Jun 2017 14:38:37 +1000 Subject: [Python-Dev] Inconsistency between PEP492, documentation and behaviour of "async with" In-Reply-To: References: Message-ID: > > Can someone please clarify the exact behaviour of "async with"? > > "async with" is expected to behave essentially the same way that > normal "with" does as far as return, break, and continue are concerned > (i.e. calling __aexit__ without an exception set, so it's more like > try/finally than it is try/else). > > Would you mind filing a documentation bug for that? We clearly missed > that the semantics described in the new documentation didn't actually > match the original with statement semantics (even though matching > those semantics is the intended behaviour). Ok, bug filed at: http://bugs.python.org/issue30707 -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Tue Jun 20 14:34:24 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 20 Jun 2017 14:34:24 -0400 Subject: [Python-Dev] bugs.python.org is down at the moment (503) Message-ID: and had been for at least a few minutes, so it is not just you ;-) ----------------------- Service Temporarily Unavailable The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later. Apache/2.2.16 (Debian) Server at bugs.python.org Port 443 -- Terry Jan Reedy From phd at phdru.name Tue Jun 20 14:54:09 2017 From: phd at phdru.name (Oleg Broytman) Date: Tue, 20 Jun 2017 20:54:09 +0200 Subject: [Python-Dev] bugs.python.org is down at the moment (503) In-Reply-To: References: Message-ID: <20170620185409.GA10461@phdru.name> Works for me, no problem. On Tue, Jun 20, 2017 at 02:34:24PM -0400, Terry Reedy wrote: > and had been for at least a few minutes, so it is not just you ;-) > > ----------------------- > Service Temporarily Unavailable > > The server is temporarily unable to service your request due to maintenance > downtime or capacity problems. Please try again later. > Apache/2.2.16 (Debian) Server at bugs.python.org Port 443 > -- > Terry Jan Reedy Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From tjreedy at udel.edu Tue Jun 20 14:57:01 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 20 Jun 2017 14:57:01 -0400 Subject: [Python-Dev] bugs.python.org is down at the moment (503) In-Reply-To: References: Message-ID: On 6/20/2017 2:34 PM, Terry Reedy wrote: > and had been for at least a few minutes, so it is not just you ;-) > > ----------------------- > Service Temporarily Unavailable > > The server is temporarily unable to service your request due to > maintenance downtime or capacity problems. Please try again later. > Apache/2.2.16 (Debian) Server at bugs.python.org Port 443 Slightly slow, but working again. -- Terry Jan Reedy From victor.stinner at gmail.com Tue Jun 20 18:35:13 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 21 Jun 2017 00:35:13 +0200 Subject: [Python-Dev] Python FTP Injections Allow for Firewall Bypass (oss-security advisory) In-Reply-To: <20170224043657.GE5689@ando.pearwood.info> References: <87wpcktw4e.fsf@example.com> <20170224043657.GE5689@ando.pearwood.info> Message-ID: Hi, Re: "[Python-Dev] Python FTP Injections Allow for Firewall Bypass (oss-security advisory)" 2017-02-24 5:36 GMT+01:00 Steven D'Aprano : > I am not qualified to judge the merits of this, but it does seem > worrying that (alledgedly) the Python security team hasn't responded for > over 12 months. > > Is anyone able to comment? I don't have the archives of the PSRT mailing list and I'm not sure that I was subscribed when "the" email was sent. Does someone have the date of this email? It's to complete the new entry in my doc: http://python-security.readthedocs.io/vuln/urllib_ftp_protocol_stream_injection.html#urllib-ftp-protocol-stream-injection I don't want to blame anyone, I just want to collect data to help us to enhance our process to handle security vulnerabilities. FYI I tried to take care of a few security vulnerabilities recently, and as expected, each issue is more tricky than expected :-) While fixing http://bugs.python.org/issue30500 I noticed that urllib accepts newline characters in URLs. I don't know if it's deliberate or not... So I created a new issue http://bugs.python.org/issue30713 I updated expat from 2.1.1 to 2.2.0, but now the compilation fails in 2.7 on Windows with Visual Studio 2008. And just when I was done, expat 2.2.1 was released. I have to do the same job again :-) Victor From guido at python.org Tue Jun 20 19:06:49 2017 From: guido at python.org (Guido van Rossum) Date: Tue, 20 Jun 2017 16:06:49 -0700 Subject: [Python-Dev] Python FTP Injections Allow for Firewall Bypass (oss-security advisory) In-Reply-To: References: <87wpcktw4e.fsf@example.com> <20170224043657.GE5689@ando.pearwood.info> Message-ID: I think that the first email about this was received from Timothy D. Morgan on 1/15/16. You should be able to get confirmation of this from Christian Heimes. I think that was a dark year for the PSRT. On Tue, Jun 20, 2017 at 3:35 PM, Victor Stinner wrote: > Hi, > > Re: "[Python-Dev] Python FTP Injections Allow for Firewall Bypass > (oss-security advisory)" > > 2017-02-24 5:36 GMT+01:00 Steven D'Aprano : > > I am not qualified to judge the merits of this, but it does seem > > worrying that (alledgedly) the Python security team hasn't responded for > > over 12 months. > > > > Is anyone able to comment? > > I don't have the archives of the PSRT mailing list and I'm not sure > that I was subscribed when "the" email was sent. Does someone have the > date of this email? It's to complete the new entry in my doc: > http://python-security.readthedocs.io/vuln/urllib_ > ftp_protocol_stream_injection.html#urllib-ftp-protocol-stream-injection > > I don't want to blame anyone, I just want to collect data to help us > to enhance our process to handle security vulnerabilities. > > FYI I tried to take care of a few security vulnerabilities recently, > and as expected, each issue is more tricky than expected :-) > > While fixing http://bugs.python.org/issue30500 I noticed that urllib > accepts newline characters in URLs. I don't know if it's deliberate or > not... So I created a new issue http://bugs.python.org/issue30713 > > I updated expat from 2.1.1 to 2.2.0, but now the compilation fails in > 2.7 on Windows with Visual Studio 2008. And just when I was done, > expat 2.2.1 was released. I have to do the same job again :-) > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Tue Jun 20 19:57:41 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 21 Jun 2017 01:57:41 +0200 Subject: [Python-Dev] Python FTP Injections Allow for Firewall Bypass (oss-security advisory) In-Reply-To: References: <87wpcktw4e.fsf@example.com> <20170224043657.GE5689@ando.pearwood.info> Message-ID: Thank you. Now you can admire the beautiful timeline :-) http://python-security.readthedocs.io/vuln/urllib_ftp_protocol_stream_injection.html#timeline Timeline using the disclosure date 2017-02-20 as reference: 2016-01-15 (-402 days): Reported (email sent to the PSRT list) 2017-02-20: Disclosure date (blog post, mail to oss-security) 2017-02-20 (+0 days): Python issue #29606 reported by ecbftw 2017-06-21 1:06 GMT+02:00 Guido van Rossum : > I think that the first email about this was received from Timothy D. Morgan > on 1/15/16. You should be able to get confirmation of this from Christian > Heimes. I think that was a dark year for the PSRT. > > On Tue, Jun 20, 2017 at 3:35 PM, Victor Stinner > wrote: >> >> Hi, >> >> Re: "[Python-Dev] Python FTP Injections Allow for Firewall Bypass >> (oss-security advisory)" >> >> 2017-02-24 5:36 GMT+01:00 Steven D'Aprano : >> > I am not qualified to judge the merits of this, but it does seem >> > worrying that (alledgedly) the Python security team hasn't responded for >> > over 12 months. >> > >> > Is anyone able to comment? >> >> I don't have the archives of the PSRT mailing list and I'm not sure >> that I was subscribed when "the" email was sent. Does someone have the >> date of this email? It's to complete the new entry in my doc: >> >> http://python-security.readthedocs.io/vuln/urllib_ftp_protocol_stream_injection.html#urllib-ftp-protocol-stream-injection >> >> I don't want to blame anyone, I just want to collect data to help us >> to enhance our process to handle security vulnerabilities. >> >> FYI I tried to take care of a few security vulnerabilities recently, >> and as expected, each issue is more tricky than expected :-) >> >> While fixing http://bugs.python.org/issue30500 I noticed that urllib >> accepts newline characters in URLs. I don't know if it's deliberate or >> not... So I created a new issue http://bugs.python.org/issue30713 >> >> I updated expat from 2.1.1 to 2.2.0, but now the compilation fails in >> 2.7 on Windows with Visual Studio 2008. And just when I was done, >> expat 2.2.1 was released. I have to do the same job again :-) >> >> Victor >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > -- > --Guido van Rossum (python.org/~guido) From victor.stinner at gmail.com Wed Jun 21 05:26:05 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 21 Jun 2017 11:26:05 +0200 Subject: [Python-Dev] test_sax and test_random fail on Python 3.6.2rc1 on Windows Message-ID: Hi, The end-of-line hell is not over, test_sax and test_random tests are still failing if you install Python 3.6.2rc1 on Windows: http://bugs.python.org/issue27425#msg296519 These tests rely on files in Lib/test/. The end-of-line of these files is controlled by .gitattributes, but it seems like the Windows installer changed the end-of-line or the file were not correctly created on the Git clone (.gitattributes ignored)? In my Git checkout on Windows, Lib/test/xmltestdata/test.xml.out and Lib/test/randv2_32.pck use UNIX end-of-line (\n). While it's possible to fix test_sax to translate the end-of-line, I would prefer to not have to modify test_random which opens a pickle binary file (Lib/test/randv2_32.pck). So, can someone look how the Windows installer stores and then installs these files? Victor From steve.dower at python.org Wed Jun 21 10:10:53 2017 From: steve.dower at python.org (Steve Dower) Date: Wed, 21 Jun 2017 07:10:53 -0700 Subject: [Python-Dev] test_sax and test_random fail on Python 3.6.2rc1 on Windows In-Reply-To: References: Message-ID: I?m looking. Nobody in any of their reports has actually said what the line endings *are* in they install yet (I assume \n), and I haven?t had a chance to check yet. Do we have a minimum git version requirement? Maybe I need to update my build machine. Top-posted from my Windows phone From: Victor Stinner Sent: Wednesday, June 21, 2017 2:27 To: Python Dev; Steve Dower Subject: test_sax and test_random fail on Python 3.6.2rc1 on Windows Hi, The end-of-line hell is not over, test_sax and test_random tests are still failing if you install Python 3.6.2rc1 on Windows: http://bugs.python.org/issue27425#msg296519 These tests rely on files in Lib/test/. The end-of-line of these files is controlled by .gitattributes, but it seems like the Windows installer changed the end-of-line or the file were not correctly created on the Git clone (.gitattributes ignored)? In my Git checkout on Windows, Lib/test/xmltestdata/test.xml.out and Lib/test/randv2_32.pck use UNIX end-of-line (\n). While it's possible to fix test_sax to translate the end-of-line, I would prefer to not have to modify test_random which opens a pickle binary file (Lib/test/randv2_32.pck). So, can someone look how the Windows installer stores and then installs these files? Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Wed Jun 21 10:22:22 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 21 Jun 2017 16:22:22 +0200 Subject: [Python-Dev] test_sax and test_random fail on Python 3.6.2rc1 on Windows In-Reply-To: References: Message-ID: 2017-06-21 16:10 GMT+02:00 Steve Dower : > Do we have a minimum git version requirement? Maybe I need to update my > build machine. After we added .gitattributes, we had to force a fresh Git checkout on buildbots. Otherwise, they kept the old and wrong end of line. Maybe try to move my checkout and create a new checkout? Victor From steve.dower at python.org Wed Jun 21 10:31:56 2017 From: steve.dower at python.org (Steve Dower) Date: Wed, 21 Jun 2017 07:31:56 -0700 Subject: [Python-Dev] test_sax and test_random fail on Python 3.6.2rc1 on Windows In-Reply-To: References: Message-ID: The last release from the same checkout was fine. I?m more inclined to think something went wrong with the pull from Ned?s fork and checking out his tag. (Or the fact that it took me three goes to get to that point ? my least favourite part of the git migration is git...) When I get to work I?ll clean it up and try again to see if it repros or was a random occurrence. Top-posted from my Windows phone From: Victor Stinner Sent: Wednesday, June 21, 2017 7:23 To: Steve Dower Cc: Python Dev Subject: Re: test_sax and test_random fail on Python 3.6.2rc1 on Windows 2017-06-21 16:10 GMT+02:00 Steve Dower : > Do we have a minimum git version requirement? Maybe I need to update my > build machine. After we added .gitattributes, we had to force a fresh Git checkout on buildbots. Otherwise, they kept the old and wrong end of line. Maybe try to move my checkout and create a new checkout? Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Wed Jun 21 10:36:49 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 21 Jun 2017 16:36:49 +0200 Subject: [Python-Dev] test_sax and test_random fail on Python 3.6.2rc1 on Windows In-Reply-To: References: Message-ID: Thank you Steve for looking at this issue and to work on our Windows installer :-) Victor 2017-06-21 16:31 GMT+02:00 Steve Dower : > The last release from the same checkout was fine. I?m more inclined to think > something went wrong with the pull from Ned?s fork and checking out his tag. > (Or the fact that it took me three goes to get to that point ? my least > favourite part of the git migration is git...) > > > > When I get to work I?ll clean it up and try again to see if it repros or was > a random occurrence. > > > > Top-posted from my Windows phone > > > > From: Victor Stinner > Sent: Wednesday, June 21, 2017 7:23 > To: Steve Dower > Cc: Python Dev > Subject: Re: test_sax and test_random fail on Python 3.6.2rc1 on Windows > > > > 2017-06-21 16:10 GMT+02:00 Steve Dower : > >> Do we have a minimum git version requirement? Maybe I need to update my > >> build machine. > > > > After we added .gitattributes, we had to force a fresh Git checkout on > > buildbots. Otherwise, they kept the old and wrong end of line. Maybe > > try to move my checkout and create a new checkout? > > > > Victor > > From steve.dower at python.org Wed Jun 21 12:42:41 2017 From: steve.dower at python.org (Steve Dower) Date: Wed, 21 Jun 2017 09:42:41 -0700 Subject: [Python-Dev] test_sax and test_random fail on Python 3.6.2rc1 on Windows In-Reply-To: References: Message-ID: On 21Jun2017 0736, Victor Stinner wrote: > Thank you Steve for looking at this issue and to work on our Windows > installer :-) No worries. So we didn't ship 3.6.2rc1 with LF line endings instead of CRLF - those are fine. These issues are due to .gitattributes changing for test data files, but those files didn't change and so the line endings in my repo didn't get updated. I've posted more details and recommendations at http://bugs.python.org/issue30716. Thanks, Steve From brett at python.org Wed Jun 21 21:14:51 2017 From: brett at python.org (Brett Cannon) Date: Thu, 22 Jun 2017 01:14:51 +0000 Subject: [Python-Dev] https://devguide.python.org now exists Message-ID: While the redirect for https://docs.python.org/devguide does not exist yet, the infrastructure team has been nice enough to set up https://devguide.python.org for us so that the devguide is served from Read the Docs. This not only will (eventually) let the infrastructure team have one less custom piece of doc infrastructure to maintain, but it means updates to the devguide will be live instantly. Plus the URL is a lot easier to remember. :) So a win-win for everyone! -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Wed Jun 21 22:58:13 2017 From: larry at hastings.org (Larry Hastings) Date: Wed, 21 Jun 2017 19:58:13 -0700 Subject: [Python-Dev] Proposed release schedule for Python 3.5.4 Message-ID: It's time to start planning the next 3.5 release, 3.5.4. Note that this will be the last 3.5 "bugfix" release; after 3.5.4, the 3.5 branch will only be open for security fixes. 3.5.4 will also be the last release of 3.5 with binary installers. I propose to tag and release 3.5.4 on these dates: 3.5.4rc1 tag Sat July 22 2017 release Sun July 23 2017 3.5.4 final tag Sun Aug 6 2017 release Mon Aug 7 2017 Thus rc1 would be tagged in just over four weeks. As for 3.4-- Lately I've been releasing new versions of 3.5 and 3.4 at the same time. But I'm not sure it's worth the effort to release another 3.4 right now. There have only been two (2) checkins into the 3.4 branch since 3.4.6 was released back in January: f37b0cb230069481609b0bb06891b5dd26320504 bpo-25008: Deprecate smtpd and point to aiosmtpd fa53dbdec818b0f2a0e22ca12a49d83ec948fc91 Issues #27850 and #27766: Remove 3DES from ssl default cipher list and add ChaCha20 Poly1305. The first was a documentation-only change which is already live on docs.python.org. The second changes the _DEFAULT_CIPHERS and _RESTRICTED_SERVER_CIPHERS constants in Lib/ssl.py. A reasonable change, but minor. I'm not convinced it's worth spending the time of many people in the community at large to update 3.4 just for this. If you have any feedback / concerns about this schedule, or if you think it's important that I release 3.4.7 with these minor changes, please reply here. If I don't hear anything back in a day or two I'll go ahead and make this the official schedule. Your friendly neighborhood release manager, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed Jun 21 23:36:33 2017 From: brett at python.org (Brett Cannon) Date: Thu, 22 Jun 2017 03:36:33 +0000 Subject: [Python-Dev] https://devguide.python.org now exists In-Reply-To: References: Message-ID: FYI we're aware of the internal link problem and I've notified the infrastructure team about it (looks like some URL rewrite rule is misconfigured ?). On Wed, Jun 21, 2017, 18:14 Brett Cannon, wrote: > While the redirect for https://docs.python.org/devguide does not exist > yet, the infrastructure team has been nice enough to set up > https://devguide.python.org for us so that the devguide is served from > Read the Docs. This not only will (eventually) let the infrastructure team > have one less custom piece of doc infrastructure to maintain, but it means > updates to the devguide will be live instantly. Plus the URL is a lot > easier to remember. :) > > So a win-win for everyone! > -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Thu Jun 22 01:07:51 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 22 Jun 2017 08:07:51 +0300 Subject: [Python-Dev] [python-committers] Proposed release schedule for Python 3.5.4 In-Reply-To: References: Message-ID: 2017-06-22 5:58 GMT+03:00 Larry Hastings : > There have only been two (2) checkins into the 3.4 branch since 3.4.6 was > released back in January: > > f37b0cb230069481609b0bb06891b5dd26320504 > bpo-25008: Deprecate smtpd and point to aiosmtpd > > fa53dbdec818b0f2a0e22ca12a49d83ec948fc91 > Issues #27850 and #27766: Remove 3DES from ssl > default cipher list and add ChaCha20 Poly1305. Please look also at issue30484. The PR is blocked, seems only the RM can merge it. Maybe this is a cause of small number of changes in 3.4 since moving to GitHub? https://bugs.python.org/issue30484 There were several security issues fixed recent times. Perhaps the fixes should be backported to 3.4? From victor.stinner at gmail.com Thu Jun 22 04:04:01 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 22 Jun 2017 10:04:01 +0200 Subject: [Python-Dev] [python-committers] Proposed release schedule for Python 3.5.4 In-Reply-To: References: Message-ID: For 3.4, please review my pending security fixes :-) There are more of them. About the cipher list in ssl, the change itself is simple but it's to blacklist DES and 3DES since it has been proved that these ciphers are really too weak nowadays: http://python-security.readthedocs.io/vuln/cve-2016-2183_sweet32_attack_des_3des.html By the way, is Larry the only one to be able to merge changes in 3.4? Before GitHub, all core dev were technically allowed to push in security-only branches. I would be interested to be allowed to push my own security fixes, but also to enable Travis CI and maybe AppVeyor on the 3.4 and 3.3 branches. Victor Le 22 juin 2017 04:58, "Larry Hastings" a ?crit : > > > It's time to start planning the next 3.5 release, 3.5.4. Note that this > will be the last 3.5 "bugfix" release; after 3.5.4, the 3.5 branch will > only be open for security fixes. 3.5.4 will also be the last release of > 3.5 with binary installers. > > I propose to tag and release 3.5.4 on these dates: > > 3.5.4rc1 > tag Sat July 22 2017 > release Sun July 23 2017 > > 3.5.4 final > tag Sun Aug 6 2017 > release Mon Aug 7 2017 > > Thus rc1 would be tagged in just over four weeks. > > > As for 3.4-- > > Lately I've been releasing new versions of 3.5 and 3.4 at the same time. > But I'm not sure it's worth the effort to release another 3.4 right now. > There have only been two (2) checkins into the 3.4 branch since 3.4.6 was > released back in January: > > f37b0cb230069481609b0bb06891b5dd26320504 > bpo-25008: Deprecate smtpd and point to aiosmtpd > > fa53dbdec818b0f2a0e22ca12a49d83ec948fc91 > Issues #27850 and #27766: Remove 3DES from ssl > default cipher list and add ChaCha20 Poly1305. > > > The first was a documentation-only change which is already live on > docs.python.org. The second changes the _DEFAULT_CIPHERS and > _RESTRICTED_SERVER_CIPHERS constants in Lib/ssl.py. A reasonable change, > but minor. I'm not convinced it's worth spending the time of many people > in the community at large to update 3.4 just for this. > > If you have any feedback / concerns about this schedule, or if you think > it's important that I release 3.4.7 with these minor changes, please reply > here. If I don't hear anything back in a day or two I'll go ahead and make > this the official schedule. > > > Your friendly neighborhood release manager, > > > */arry* > > _______________________________________________ > python-committers mailing list > python-committers at python.org > https://mail.python.org/mailman/listinfo/python-committers > Code of Conduct: https://www.python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Thu Jun 22 05:31:31 2017 From: larry at hastings.org (Larry Hastings) Date: Thu, 22 Jun 2017 02:31:31 -0700 Subject: [Python-Dev] [python-committers] Proposed release schedule for Python 3.5.4 In-Reply-To: References: Message-ID: On 06/22/2017 01:04 AM, Victor Stinner wrote: > About the cipher list in ssl, the change itself is simple but it's to > blacklist DES and 3DES since it has been proved that these ciphers are > really too weak nowadays: > http://python-security.readthedocs.io/vuln/cve-2016-2183_sweet32_attack_des_3des.html Not "blacklist"--IIUC the user can still manually specify whatever cipher suites they like. And not DES... who knows how long ago that was removed from the list. This change in 3.4 removes 3DES from the /default/ permissible cipher list, changing those entries to use "HIGH cipher suites" instead (OpenSSL's term for "cipher suites with key sizes >= 128 bytes"). It also adds ChaCha20 to the default cipher list. > By the way, is Larry the only one to be able to merge changes in 3.4? > Before GitHub, all core dev were technically allowed to push in > security-only branches. Oh? Am I? **insert evil laugh** Ladies and gentlemen, get out your checkbooks! 3.4 is about to get... expensive. Seriously, though, I was mostly hoping other people would handle the security stuff and just keep me informed. If I'm the only one permitted to accept PRs into 3.4 (and soon 3.5), okay, I can work with that. I'm still probably gonna delegate the actual judgment of the validity of the PRs. But obviously it'll mean I'll have to be more hands-on, where so far I was assuming I could just let other people handle it. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From markus.wissinger at gmail.com Thu Jun 22 04:44:12 2017 From: markus.wissinger at gmail.com (Markus Wissinger) Date: Thu, 22 Jun 2017 10:44:12 +0200 Subject: [Python-Dev] PEP 544: Protocols - second round In-Reply-To: References: <8b064822-14bb-6800-810a-c14dc97f565a@hotpy.org> Message-ID: Hi, I have to admit I am not happy with separating the concepts of 'runtime' and 'static' types as implied by pep544. I am currently exploring a type hint generator that produces hints out of types used in unit tests. It debugs the tests and collects the parameter types of call and return events. It ignores a type when a supertype is present. Failing isinstance/issubclass calls for protocols would hurt there. I understand that any type checker code that could provide isinstance functionality for pep544 protocols would rely on method signatures that my hint generator is just producing. proof of concept implementation (writes method docstrings, no pep484 type hints yet): https://github.com/markuswissinger/ducktestpy This is currently just some personal project that some of you will consider a strange idea. I just want to mention that this use case might not play well together with pep544. Regards Markus 2017-06-05 23:59 GMT+02:00 Guido van Rossum : > On Fri, Jun 2, 2017 at 3:10 PM, Ivan Levkivskyi > wrote: > >> On 1 June 2017 at 00:10, Guido van Rossum wrote: >> >>> >>> On Wed, May 31, 2017 at 2:16 AM, Ivan Levkivskyi >>> wrote: >>> >>>> On 31 May 2017 at 00:58, Guido van Rossum wrote: >>>> [...] >>>> >>>> Thank you for very detailed answers! I have practically nothing to add. >>>> It seems to me that most of the Kevin's questions stem from unnecessary >>>> focus >>>> on runtime type checking. Here are two ideas about how to fix this: >>>> >>>> * Add the word "static" somewhere in the PEP title. >>>> >>> >>> So the title could become "Protocols: Static structural subtyping (duck >>> typing)" -- long, but not record-setting. >>> >> >> I am thinking about "Protocols: Structural subtyping (static duck >> typing)". The reason is that subtyping is already a mostly static concept >> (in contrast to subclassing), >> while duck typing is typically associated with the runtime behaviour. >> >> This might seem minor, but this version of the title sounds much more >> naturally to me. >> > > +1 > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > markus.wissinger%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bhavishyagopesh at gmail.com Thu Jun 22 09:54:36 2017 From: bhavishyagopesh at gmail.com (Bhavishya) Date: Thu, 22 Jun 2017 19:24:36 +0530 Subject: [Python-Dev] [Core-mentorship] C version of weakrefset.py. In-Reply-To: References: <557E27CF-936C-4E78-8977-C98C88BE74DC@gmail.com> Message-ID: Hello, 1)I might be totally wrong here, but even if we go with stripping annotation(in .pyc)...still the "lag" that comes from ABCs needs to be addressed. 2) I been reading for past few days about your fat-optimizer project and the corresponding TO-DO list, if you think that PEP-0511 should be improved, I can work on that. 3)Also I was seeing to existing repos which implement some-kind of optimizaton, like numpy,snake-oil.... Finally I wanted to decide upon a roadmap, so that I could put more specific efforts.(fat optimizer?) Thank You Regards, Bhavishya On Wed, Jun 21, 2017 at 8:05 PM, Victor Stinner wrote: > 2017-06-21 15:21 GMT+02:00 INADA Naoki : > > ABC slowdown Python startup only 2ms. But importing typing module take > 11ms. > > While typing is not imported from site.py, many new Python application > > will import it. > > It may take over 100ms for applications or libraries heavily depending > on ABCs. > > When typing is not used in the application, only used for static > checks, you can try to "strip" annotations to avoid any overhead on > the application startup. It's not only a matter of "import typing", > it's also the cost of instanciating types like "List[int]" (or even > more complex ones). > > I discussed with Jukka Lehtosalo at Pycon US about stripping > completely annotations. He told me that my PEP 511 may be a good > solution to keep annotation in the .py code, but strip them for > "production code", in the cached .pyc files: > > "PEP 511 -- API for code transformers" > https://www.python.org/dev/peps/pep-0511/ > > This PEP is somehow controversal. Some people fear that it would allow > people to hack the Python language to write their own incompatible > variant of Python. I don't think that my PEP adds anything new, it's > already possible to do that, importlib made it even more easy. I used > my FAT Python optimizer project to sell this PEP. Since FAT Python is > also controversal (it hasn't been proved to be actually faster), the > PEP didn't go far at my last attempt. > > Note: Is core-menthorship the best place for such performance > discussion? :-) Maybe we should open a thread on python-dev@ or speed@ > mailing list. > > Victor > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Thu Jun 22 11:56:29 2017 From: brett at python.org (Brett Cannon) Date: Thu, 22 Jun 2017 15:56:29 +0000 Subject: [Python-Dev] [python-committers] Proposed release schedule for Python 3.5.4 In-Reply-To: References: Message-ID: On Thu, 22 Jun 2017 at 02:32 Larry Hastings wrote: > > > On 06/22/2017 01:04 AM, Victor Stinner wrote: > > About the cipher list in ssl, the change itself is simple but it's to > blacklist DES and 3DES since it has been proved that these ciphers are > really too weak nowadays: > > http://python-security.readthedocs.io/vuln/cve-2016-2183_sweet32_attack_des_3des.html > > > Not "blacklist"--IIUC the user can still manually specify whatever cipher > suites they like. And not DES... who knows how long ago that was removed > from the list. > > This change in 3.4 removes 3DES from the *default* permissible cipher > list, changing those entries to use "HIGH cipher suites" instead (OpenSSL's > term for "cipher suites with key sizes >= 128 bytes"). It also adds > ChaCha20 to the default cipher list. > > > > By the way, is Larry the only one to be able to merge changes in 3.4? > Before GitHub, all core dev were technically allowed to push in > security-only branches. > > > Oh? Am I? **insert evil laugh** Ladies and gentlemen, get out your > checkbooks! 3.4 is about to get... expensive. > > Seriously, though, I was mostly hoping other people would handle the > security stuff and just keep me informed. If I'm the only one permitted to > accept PRs into 3.4 (and soon 3.5), okay, I can work with that. I'm still > probably gonna delegate the actual judgment of the validity of the PRs. > But obviously it'll mean I'll have to be more hands-on, where so far I was > assuming I could just let other people handle it. > Currently the security-only branches are set so that only release managers can merge PRs since they technically are on the hook if some compatibility breaks due to some patch (e.g. I expect Ned to use this for 3.7 once we hit rc to really control what goes in last minute). It's easy enough to turn this protection off, though, if people want. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sf at fermigier.com Thu Jun 22 12:09:22 2017 From: sf at fermigier.com (=?UTF-8?Q?St=C3=A9fane_Fermigier?=) Date: Thu, 22 Jun 2017 18:09:22 +0200 Subject: [Python-Dev] PEP 544: Protocols - second round In-Reply-To: References: <8b064822-14bb-6800-810a-c14dc97f565a@hotpy.org> Message-ID: On Thu, Jun 22, 2017 at 10:44 AM, Markus Wissinger < markus.wissinger at gmail.com> wrote: > Hi, > > I have to admit I am not happy with separating the concepts of 'runtime' > and 'static' types as implied by pep544. > > I am currently exploring a type hint generator that produces hints out of > types used in unit tests. It debugs the tests and collects the parameter > types of call and return events. It ignores a type when a supertype is > present. Failing isinstance/issubclass calls for protocols would hurt > there. I understand that any type checker code that could provide > isinstance functionality for pep544 protocols would rely on method > signatures that my hint generator is just producing. > > proof of concept implementation (writes method docstrings, no pep484 type > hints yet): > https://github.com/markuswissinger/ducktestpy > > This is currently just some personal project that some of you will > consider a strange idea. > Not a strange idea, I've had a similar idea and played a bit with it ~10 years ago (inspired by a Java project whose name eludes me now). Also, I think PyCharm is able to do similar things (see https://blog.jetbrains.com/pycharm/2013/02/dynamic-runtime-type-inference-in-pycharm-2-7/ ). S. -- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Chairman, Free&OSS Group / Systematic Cluster - http://www.gt-logiciel-libre.org/ Co-Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Founder & Organiser, PyData Paris - http://pydata.fr/ --- ?You never change things by ?ghting the existing reality. To change something, build a new model that makes the existing model obsolete.? ? R. Buckminster Fuller -------------- next part -------------- An HTML attachment was scrubbed... URL: From levkivskyi at gmail.com Thu Jun 22 17:53:20 2017 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Thu, 22 Jun 2017 23:53:20 +0200 Subject: [Python-Dev] PEP 544: Protocols - second round In-Reply-To: References: <8b064822-14bb-6800-810a-c14dc97f565a@hotpy.org> Message-ID: On 22 June 2017 at 10:44, Markus Wissinger wrote: > I have to admit I am not happy with separating the concepts of 'runtime' > and 'static' types as implied by pep544. > This is not something new, already PEP 483 makes a clear distinction between types (a static concept) and classes (a runtime concept). > Failing isinstance/issubclass calls for protocols would hurt there. I > understand that any type checker code that could provide isinstance > functionality for pep544 protocols would rely on method signatures that my > hint generator is just producing. > isinstance(obj, T) and issubclass(Cls, T) already fail if T is a subscripted generic like List[int], so that again nothing new here. To check runtime subtyping with such types one can write a third party introspection tool based on typing_inspect package on PyPI (which potentially might in future become an official wrapper for currently internal typing API). -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From Stefan.Richthofer at gmx.de Thu Jun 22 19:08:44 2017 From: Stefan.Richthofer at gmx.de (Stefan Richthofer) Date: Fri, 23 Jun 2017 01:08:44 +0200 Subject: [Python-Dev] PEP 544: Protocols - second round In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From bhavishyagopesh at gmail.com Thu Jun 22 19:04:38 2017 From: bhavishyagopesh at gmail.com (Bhavishya) Date: Fri, 23 Jun 2017 04:34:38 +0530 Subject: [Python-Dev] [Core-mentorship] C version of weakrefset.py. In-Reply-To: References: <557E27CF-936C-4E78-8977-C98C88BE74DC@gmail.com> Message-ID: Also I saw your conversation with "Brett Cannon" on lazy-loading some modules at startup, and also doing so using ModuleProxy(also several implementations that exist like importlib's Lazyloader , PEAK ). So your suggestion's on this too? Thank You regards, Bhavishya On Thu, Jun 22, 2017 at 7:24 PM, Bhavishya wrote: > Hello, > 1)I might be totally wrong here, but even if we go with stripping > annotation(in .pyc)...still the "lag" that comes from ABCs needs to be > addressed. > > 2) I been reading for past few days about your fat-optimizer project and > the corresponding TO-DO list, if you think that PEP-0511 should be > improved, I can work on that. > > 3)Also I was seeing to existing repos which implement some-kind of > optimizaton, like > numpy,snake-oil.... > > Finally I wanted to decide upon a roadmap, so that I could put more > specific efforts.(fat optimizer?) > > Thank You > Regards, > Bhavishya > > On Wed, Jun 21, 2017 at 8:05 PM, Victor Stinner > wrote: > >> 2017-06-21 15:21 GMT+02:00 INADA Naoki : >> > ABC slowdown Python startup only 2ms. But importing typing module take >> 11ms. >> > While typing is not imported from site.py, many new Python application >> > will import it. >> > It may take over 100ms for applications or libraries heavily depending >> on ABCs. >> >> When typing is not used in the application, only used for static >> checks, you can try to "strip" annotations to avoid any overhead on >> the application startup. It's not only a matter of "import typing", >> it's also the cost of instanciating types like "List[int]" (or even >> more complex ones). >> >> I discussed with Jukka Lehtosalo at Pycon US about stripping >> completely annotations. He told me that my PEP 511 may be a good >> solution to keep annotation in the .py code, but strip them for >> "production code", in the cached .pyc files: >> >> "PEP 511 -- API for code transformers" >> https://www.python.org/dev/peps/pep-0511/ >> >> This PEP is somehow controversal. Some people fear that it would allow >> people to hack the Python language to write their own incompatible >> variant of Python. I don't think that my PEP adds anything new, it's >> already possible to do that, importlib made it even more easy. I used >> my FAT Python optimizer project to sell this PEP. Since FAT Python is >> also controversal (it hasn't been proved to be actually faster), the >> PEP didn't go far at my last attempt. >> >> Note: Is core-menthorship the best place for such performance >> discussion? :-) Maybe we should open a thread on python-dev@ or speed@ >> mailing list. >> >> Victor >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Fri Jun 23 04:55:57 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 23 Jun 2017 10:55:57 +0200 Subject: [Python-Dev] [python-committers] Proposed release schedule for Python 3.5.4 In-Reply-To: References: Message-ID: 2017-06-22 17:56 GMT+02:00 Brett Cannon : > On Thu, 22 Jun 2017 at 02:32 Larry Hastings wrote: >> Seriously, though, I was mostly hoping other people would handle the >> security stuff and just keep me informed. If I'm the only one permitted to >> accept PRs into 3.4 (and soon 3.5), okay, I can work with that. I'm still >> probably gonna delegate the actual judgment of the validity of the PRs. But >> obviously it'll mean I'll have to be more hands-on, where so far I was >> assuming I could just let other people handle it. > > Currently the security-only branches are set so that only release managers > can merge PRs since they technically are on the hook if some compatibility > breaks due to some patch (e.g. I expect Ned to use this for 3.7 once we hit > rc to really control what goes in last minute). It's easy enough to turn > this protection off, though, if people want. Larry: would you be ok to turn this protection off on the 3.4 branch? Or would you feel more confortable if only a few people would be allowed to push to the 3.4 branch, so add me a whitelist group or something like that? As I wrote, my plan is not only to merge my security fixes, but also to work on a CI. I would feel more confortable to see the Travis CI test result on my security PRs. Victor From larry at hastings.org Fri Jun 23 09:19:45 2017 From: larry at hastings.org (Larry Hastings) Date: Fri, 23 Jun 2017 06:19:45 -0700 Subject: [Python-Dev] [python-committers] Proposed release schedule for Python 3.5.4 In-Reply-To: References: Message-ID: On 06/23/2017 01:55 AM, Victor Stinner wrote: > Larry: would you be ok to turn this protection off on the 3.4 branch? > Or would you feel more confortable if only a few people would be > allowed to push to the 3.4 branch, so add me a whitelist group or > something like that? Actually I kind of like the idea of the branch being restricted. Let's try it for now and see if it works. > As I wrote, my plan is not only to merge my security fixes, but also > to work on a CI. I would feel more confortable to see the Travis CI > test result on my security PRs. Do you need write access to the branch in order to get Travis CI working? //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Fri Jun 23 09:20:35 2017 From: larry at hastings.org (Larry Hastings) Date: Fri, 23 Jun 2017 06:20:35 -0700 Subject: [Python-Dev] Proposed release schedule for Python 3.5.4 In-Reply-To: References: Message-ID: <25b2e757-9598-91af-2cea-d369ba060236@hastings.org> On 06/21/2017 07:58 PM, Larry Hastings wrote: > If you have any feedback / concerns about this schedule, or if you > think it's important that I release 3.4.7 with these minor changes, > please reply here. If I don't hear anything back in a day or two I'll > go ahead and make this the official schedule. I haven't heard any concerns, so I'm declaring that the official schedule for 3.5.4, and I'm not scheduling 3.4.7 at this time. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Fri Jun 23 09:24:18 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 23 Jun 2017 15:24:18 +0200 Subject: [Python-Dev] [python-committers] Proposed release schedule for Python 3.5.4 In-Reply-To: References: Message-ID: 2017-06-23 15:19 GMT+02:00 Larry Hastings : > Do you need write access to the branch in order to get Travis CI working? As soon as someone reviews my proposed 3.4 patches, no :-) I will work on a PR. Victor From markus.wissinger at gmail.com Fri Jun 23 05:53:29 2017 From: markus.wissinger at gmail.com (markus.wissinger) Date: Fri, 23 Jun 2017 11:53:29 +0200 Subject: [Python-Dev] PEP 544: Protocols - second round Message-ID: <594ce518.5788500a.49ea7.e2a6@mx.google.com> What I try to do is a new workflow: - 'declare' types of method parameters by usage in a unit test. Therefore ensure the interface is obeyed inside.- publish the used types via hints to help callers of the method.?-Do express all hints in test code. Do not insert any hints manually. Any type used in any test will result in a hint (with some clever exceptions). This requires some discipline while writing tests, but worked quite well already. I think I just realized pep544 protocols will not break that scheme, just make it a little more difficult to handle. I can create intermediate signatures that still contain superfluous subtypes and remove those only in the end when protocol hints are sure to exist. Thanks.Markus? -------- Urspr?ngliche Nachricht --------Von: Stefan Richthofer Datum: 23.06.17 01:08 (GMT+01:00) An: Ivan Levkivskyi Cc: Markus Wissinger , Python-Dev Betreff: Re: [Python-Dev] PEP 544: Protocols - second round >> I am currently exploring a type hint generator that produces hints out of types used in unit tests. ? Note that pytypes (https://github.com/Stewori/pytypes) already supports this. It can dump PEP 484 style stubfiles from runtime type observations (made via decorators or profiler hook). ? >> isinstance(obj, T) and issubclass(Cls, T) already fail if T is a subscripted generic like List[int], so that again nothing new here. To check runtime subtyping with such types one can write a third party introspection tool based on typing_inspect package on PyPI ? There are public API functions in pytypes for isinstance and issubclass with support for most PEP 484 types, see https://github.com/Stewori/pytypes#is_of_typeobj-cls and https://github.com/Stewori/pytypes#is_subtypesubclass-superclass respectively. We also use them internally for pytypes' runtime typechecking features. ? Unfortunately there is no release finalized yet, but it's mostly cleanup work and pip integration left to do. Hope to get a beta release done soon. ? Best ? -Stefan ? Gesendet:?Donnerstag, 22. Juni 2017 um 23:53 Uhr Von:?"Ivan Levkivskyi" An:?"Markus Wissinger" Cc:?Python-Dev Betreff:?Re: [Python-Dev] PEP 544: Protocols - second round On 22 June 2017 at 10:44, Markus Wissinger wrote: I have to admit I am not happy with separating the concepts of 'runtime' and 'static' types as implied by pep544. ? This is not something new, already PEP 483 makes a clear distinction between types (a static concept) and classes (a runtime concept). ? Failing isinstance/issubclass calls for protocols would hurt there. I understand that any type checker code that could provide isinstance functionality for pep544 protocols would rely on method signatures that my hint generator is just producing. ? isinstance(obj, T) and issubclass(Cls, T) already fail if T is a subscripted generic like List[int], so that again nothing new here. To check runtime subtyping with such types one can write a third party introspection tool based on typing_inspect package on PyPI (which potentially might in future become an official wrapper for currently internal typing API). -- Ivan ? _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/stefan.richthofer%40gmx.de -------------- next part -------------- An HTML attachment was scrubbed... URL: From status at bugs.python.org Fri Jun 23 12:09:17 2017 From: status at bugs.python.org (Python tracker) Date: Fri, 23 Jun 2017 18:09:17 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20170623160917.C695056BD7@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2017-06-16 - 2017-06-23) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 6026 ( -8) closed 36470 (+65) total 42496 (+57) Open issues with patches: 2361 Issues opened (49) ================== #27425: Tests fail because of git's newline preferences on Windows http://bugs.python.org/issue27425 reopened by haypo #30383: [3.5] Backport regrtest features from master to Python 3.5 http://bugs.python.org/issue30383 reopened by haypo #30597: Show expected input in custom "print" error message http://bugs.python.org/issue30597 reopened by ncoghlan #30685: Multiprocessing Send to Manager Fails for Large Payload http://bugs.python.org/issue30685 opened by maxehr #30686: make `make install` faster http://bugs.python.org/issue30686 opened by inada.naoki #30688: support named Unicode escapes (\N{name}) in re http://bugs.python.org/issue30688 opened by jonathaneunice #30691: WeakSet is not pickleable http://bugs.python.org/issue30691 opened by serhiy.storchaka #30692: Automatic upcast does not occur for custom classes http://bugs.python.org/issue30692 opened by rharke #30693: tarfile add uses random order http://bugs.python.org/issue30693 opened by bmwiedemann #30694: Update embedded copy of expat to 2.2.1 http://bugs.python.org/issue30694 opened by ned.deily #30695: add a nomemory_allocator to the _testcapi module http://bugs.python.org/issue30695 opened by xdegaye #30696: infinite loop in PyRun_InteractiveLoopFlags() http://bugs.python.org/issue30696 opened by xdegaye #30697: segfault in PyErr_NormalizeException() after memory exhaustion http://bugs.python.org/issue30697 opened by xdegaye #30698: asyncio sslproto do not shutdown ssl layer cleanly http://bugs.python.org/issue30698 opened by grzgrzgrz3 #30699: Misleading class names in datetime.tzinfo usage examples http://bugs.python.org/issue30699 opened by Robert.Tasarz #30700: fileinput inplace clobbers file without leaving backup on deco http://bugs.python.org/issue30700 opened by switchnode #30701: Exception parsing certain invalid email address headers http://bugs.python.org/issue30701 opened by timb07 #30703: test_multiprocessing_forkserver hangs on the master branch http://bugs.python.org/issue30703 opened by haypo #30704: test_free_different_thread() of test_code leaks references on http://bugs.python.org/issue30704 opened by haypo #30705: python2.7 -m test -R 3:3 hangs on Refleaks 2.7 buildbots http://bugs.python.org/issue30705 opened by haypo #30706: EmailMessage Object Creation http://bugs.python.org/issue30706 opened by Michael Salsone #30707: Incorrect description of "async with" in PEP492 and documentat http://bugs.python.org/issue30707 opened by Damien George #30708: Ensure that the result of PyUnicode_AsWideCharString() doesn't http://bugs.python.org/issue30708 opened by serhiy.storchaka #30710: getaddrinfo raises OverflowError http://bugs.python.org/issue30710 opened by smejkar #30711: getaddrinfo invalid port number http://bugs.python.org/issue30711 opened by smejkar #30713: Reject newline character (U+000A) in URLs in urllib.parse http://bugs.python.org/issue30713 opened by haypo #30714: test_ssl fails with openssl 1.1.0f http://bugs.python.org/issue30714 opened by cstratak #30715: Test_winreg, test_dynamic_key hangs on my Win 10 http://bugs.python.org/issue30715 opened by terry.reedy #30716: Failing tests with installed 3.6.2rc1 on Win 10-64 http://bugs.python.org/issue30716 opened by terry.reedy #30717: str.center() is not unicode aware http://bugs.python.org/issue30717 opened by Guillaume Sanchez #30718: open builtin function: specifying the size of buffer has no ef http://bugs.python.org/issue30718 opened by direprobs #30719: IDLE: Make PyShell visible upon error. http://bugs.python.org/issue30719 opened by terry.reedy #30721: Show expected input for right shift operator usage in custom " http://bugs.python.org/issue30721 opened by CuriousLearner #30722: Tools/demo/redemo.py broken http://bugs.python.org/issue30722 opened by Christoph Sarnowski2 #30723: IDLE parenmatch - left and right highlighting, independent fla http://bugs.python.org/issue30723 opened by wohlganger #30724: ZipFile.open treats directory path as empty file http://bugs.python.org/issue30724 opened by Kit Yan Choi #30725: Windows installer for 2.7.13 doesn't install pip when installi http://bugs.python.org/issue30725 opened by KevKeating #30726: [Windows] Warnings in elementtree due to new expat http://bugs.python.org/issue30726 opened by Segev Finer #30728: IDLE: Modernize configdialog code. http://bugs.python.org/issue30728 opened by terry.reedy #30730: Injecting environment variable in subprocess on Windows http://bugs.python.org/issue30730 opened by serhiy.storchaka #30731: Use correct executable manifest for windows http://bugs.python.org/issue30731 opened by tumagonx #30732: json.dumps() lacks information about RecursionError related to http://bugs.python.org/issue30732 opened by Krzysztof Nazarewski #30733: Typo in Document What's New: Calendar http://bugs.python.org/issue30733 opened by Jonathon Vandezande #30734: 200000 indexes crashes eval and python (without eval) http://bugs.python.org/issue30734 opened by george-shuklin #30735: Python 3.6.1 test_asyncio fails on Solaris 11 http://bugs.python.org/issue30735 opened by petriborg #30737: Update devguide link to the new URL http://bugs.python.org/issue30737 opened by Mariatta #30739: pypi ssl errors [CERTIFICATE_VERIFY_FAILED] http://bugs.python.org/issue30739 opened by Luc Zimmermann #30740: SSLError when cancelling an SSL connection http://bugs.python.org/issue30740 opened by mehaase #30741: https://www.pypi-mirrors.org/ error 503 http://bugs.python.org/issue30741 opened by Luc Zimmermann Most recent 15 issues with no replies (15) ========================================== #30740: SSLError when cancelling an SSL connection http://bugs.python.org/issue30740 #30739: pypi ssl errors [CERTIFICATE_VERIFY_FAILED] http://bugs.python.org/issue30739 #30735: Python 3.6.1 test_asyncio fails on Solaris 11 http://bugs.python.org/issue30735 #30734: 200000 indexes crashes eval and python (without eval) http://bugs.python.org/issue30734 #30733: Typo in Document What's New: Calendar http://bugs.python.org/issue30733 #30732: json.dumps() lacks information about RecursionError related to http://bugs.python.org/issue30732 #30730: Injecting environment variable in subprocess on Windows http://bugs.python.org/issue30730 #30724: ZipFile.open treats directory path as empty file http://bugs.python.org/issue30724 #30718: open builtin function: specifying the size of buffer has no ef http://bugs.python.org/issue30718 #30715: Test_winreg, test_dynamic_key hangs on my Win 10 http://bugs.python.org/issue30715 #30711: getaddrinfo invalid port number http://bugs.python.org/issue30711 #30705: python2.7 -m test -R 3:3 hangs on Refleaks 2.7 buildbots http://bugs.python.org/issue30705 #30704: test_free_different_thread() of test_code leaks references on http://bugs.python.org/issue30704 #30700: fileinput inplace clobbers file without leaving backup on deco http://bugs.python.org/issue30700 #30699: Misleading class names in datetime.tzinfo usage examples http://bugs.python.org/issue30699 Most recent 15 issues waiting for review (15) ============================================= #30730: Injecting environment variable in subprocess on Windows http://bugs.python.org/issue30730 #30714: test_ssl fails with openssl 1.1.0f http://bugs.python.org/issue30714 #30711: getaddrinfo invalid port number http://bugs.python.org/issue30711 #30710: getaddrinfo raises OverflowError http://bugs.python.org/issue30710 #30708: Ensure that the result of PyUnicode_AsWideCharString() doesn't http://bugs.python.org/issue30708 #30696: infinite loop in PyRun_InteractiveLoopFlags() http://bugs.python.org/issue30696 #30695: add a nomemory_allocator to the _testcapi module http://bugs.python.org/issue30695 #30693: tarfile add uses random order http://bugs.python.org/issue30693 #30688: support named Unicode escapes (\N{name}) in re http://bugs.python.org/issue30688 #30680: textwrap should treat Unicode em-dash like ASCII em-dash http://bugs.python.org/issue30680 #30665: pass big values for arg to fcntl.ioctl http://bugs.python.org/issue30665 #30593: sqlite3 executescript does not respect isolation_level? http://bugs.python.org/issue30593 #30560: Py_SetFatalErrorAbortFunc: Allow embedding program to handle f http://bugs.python.org/issue30560 #30541: Add restricted mocks to the python unittest mocking framework http://bugs.python.org/issue30541 #30519: Add daemon argument to Timer http://bugs.python.org/issue30519 Top 10 most discussed issues (10) ================================= #30500: [security] urllib connects to a wrong host http://bugs.python.org/issue30500 13 msgs #30695: add a nomemory_allocator to the _testcapi module http://bugs.python.org/issue30695 10 msgs #29591: expat 2.2.0: Various security vulnerabilities in bundled expat http://bugs.python.org/issue29591 9 msgs #30725: Windows installer for 2.7.13 doesn't install pip when installi http://bugs.python.org/issue30725 9 msgs #16258: test_local.TestEnUSCollection failures on Solaris 10 http://bugs.python.org/issue16258 8 msgs #30368: [2.7] OpenSSL compilation fails on AMD64 Windows7 SP1 VS9.0 2. http://bugs.python.org/issue30368 8 msgs #30694: Update embedded copy of expat to 2.2.1 http://bugs.python.org/issue30694 8 msgs #30710: getaddrinfo raises OverflowError http://bugs.python.org/issue30710 8 msgs #30726: [Windows] Warnings in elementtree due to new expat http://bugs.python.org/issue30726 8 msgs #30263: regrtest: log the system load? http://bugs.python.org/issue30263 7 msgs Issues closed (63) ================== #4765: IDLE fails to "Delete Custom Key Set" properly http://bugs.python.org/issue4765 closed by terry.reedy #10079: idlelib for Python 3 with Guilherme Polo GSoC enhancements http://bugs.python.org/issue10079 closed by terry.reedy #12420: distutils tests fail if PATH is not defined http://bugs.python.org/issue12420 closed by terry.reedy #14326: IDLE - allow shell to support different locales http://bugs.python.org/issue14326 closed by terry.reedy #15308: IDLE - add an "Interrupt Execution" to shell menu http://bugs.python.org/issue15308 closed by terry.reedy #17060: IDLE -- jump to home should not go past the PS1 and PS2 prompt http://bugs.python.org/issue17060 closed by terry.reedy #17187: Python segfaults from improperly formed and called function http://bugs.python.org/issue17187 closed by serhiy.storchaka #17773: test_pydoc fails with the installed testsuite (2.7) http://bugs.python.org/issue17773 closed by terry.reedy #18152: Idle: add 2.7 backport script http://bugs.python.org/issue18152 closed by terry.reedy #18316: Idle 2.7: update to simplify cross-version patches http://bugs.python.org/issue18316 closed by terry.reedy #18967: Find a less conflict prone approach to Misc/NEWS http://bugs.python.org/issue18967 closed by brett.cannon #20663: Introduce exception argument to iter http://bugs.python.org/issue20663 closed by terry.reedy #21071: struct.Struct.format is bytes, but should be str http://bugs.python.org/issue21071 closed by haypo #21632: Idle: sychronize text files across versions as appropriate. http://bugs.python.org/issue21632 closed by terry.reedy #22028: Python 3.4.1 Installer ended prematurely (Windows msi) http://bugs.python.org/issue22028 closed by tim.golden #22897: IDLE hangs on OS X with Cocoa Tk if encoding dialog is require http://bugs.python.org/issue22897 closed by terry.reedy #23337: [IDLE/Windows] Run python with restricted rights http://bugs.python.org/issue23337 closed by terry.reedy #23431: Idle Application Not Responding http://bugs.python.org/issue23431 closed by terry.reedy #24212: Idle, 2.7, backport idlelib.__main__, enable py -m idlelib http://bugs.python.org/issue24212 closed by terry.reedy #24718: Specify interpreter when running in IDLE http://bugs.python.org/issue24718 closed by terry.reedy #24936: Idle: handle 'raise' properly when running with subprocess (2. http://bugs.python.org/issue24936 closed by terry.reedy #25020: IDLE - centralize ui policies and small utilities http://bugs.python.org/issue25020 closed by terry.reedy #25125: "Edit with IDLE" does not work for shortcuts http://bugs.python.org/issue25125 closed by terry.reedy #25224: Replace Idle's README.txt with annotated file list http://bugs.python.org/issue25224 closed by terry.reedy #25303: Add option to py_compile to compile for syntax checking withou http://bugs.python.org/issue25303 closed by terry.reedy #26627: IDLE incorrectly labeling error as internal http://bugs.python.org/issue26627 closed by terry.reedy #26993: Copy idlelib *.py files with new names http://bugs.python.org/issue26993 closed by terry.reedy #27162: Add idlelib.interface module http://bugs.python.org/issue27162 closed by terry.reedy #28889: IDLE needs the ability to pass in command-line arguments http://bugs.python.org/issue28889 closed by terry.reedy #29366: os.listdir has inconsistent behavior when run on a non-directo http://bugs.python.org/issue29366 closed by terry.reedy #29523: latest setuptools breaks virtualenvs due to unbundling depende http://bugs.python.org/issue29523 closed by cstratak #29702: Error 0x80070003: Failed to launch elevated child process http://bugs.python.org/issue29702 closed by eryksun #29755: python3 gettext.lgettext sometimes returns bytes, not string http://bugs.python.org/issue29755 closed by serhiy.storchaka #30016: No sideways scrolling in IDLE http://bugs.python.org/issue30016 closed by terry.reedy #30054: Expose tracemalloc C API to track/untrack memory blocks http://bugs.python.org/issue30054 closed by haypo #30176: curses attribute constants list is incomplete http://bugs.python.org/issue30176 closed by xiang.zhang #30303: IDLE: Add _utest to textview and add textview tests http://bugs.python.org/issue30303 closed by terry.reedy #30360: getpass.getpass() does not accept stdin from an Expect script http://bugs.python.org/issue30360 closed by terry.reedy #30473: defaultdict raises SystemError, __missing__ returned NULL in t http://bugs.python.org/issue30473 closed by haypo #30565: PEP 538: silence locale coercion and compatibility warnings by http://bugs.python.org/issue30565 closed by ncoghlan #30598: Py_NewInterpreter() leaks a reference on warnoptions in _PySys http://bugs.python.org/issue30598 closed by ncoghlan #30602: [Windows] os.spawn*() tests of test_os leak references on Wind http://bugs.python.org/issue30602 closed by haypo #30604: co_extra_freefuncs is stored thread locally and can lead to cr http://bugs.python.org/issue30604 closed by yselivanov #30619: typing.Union doc incoherence in case a class and its subclass http://bugs.python.org/issue30619 closed by Mariatta #30629: lower() is called twice http://bugs.python.org/issue30629 closed by terry.reedy #30630: Missing MSI files http://bugs.python.org/issue30630 closed by terry.reedy #30657: Unsafe arithmetic in PyString_DecodeEscape http://bugs.python.org/issue30657 closed by serhiy.storchaka #30670: pprint for dict in sorted order or insert order? http://bugs.python.org/issue30670 closed by terry.reedy #30671: dict: improve lookup function http://bugs.python.org/issue30671 closed by tim.peters #30682: f-string assert is too restrictive http://bugs.python.org/issue30682 closed by eric.smith #30684: datetime.fromtimestamp raises OSError on first day after epoch http://bugs.python.org/issue30684 closed by r.david.murray #30687: build.bat should locate msbuild.exe rather than vcvarsall.bat http://bugs.python.org/issue30687 closed by steve.dower #30689: len() and iter() of ChainMap don't work with unhashable keys http://bugs.python.org/issue30689 closed by rhettinger #30690: ChainMap doesn't preserve the order of ordered mappings http://bugs.python.org/issue30690 closed by rhettinger #30702: pickle uses getattr(obj, '__reduce__') instead of getattr(type http://bugs.python.org/issue30702 closed by serhiy.storchaka #30709: Bad getter from Descriptor#Properties example http://bugs.python.org/issue30709 closed by Mariatta #30712: struct.unpack generates wrong error in certain conditions http://bugs.python.org/issue30712 closed by serhiy.storchaka #30720: re.sub substitution match group contains wrong value after unm http://bugs.python.org/issue30720 closed by serhiy.storchaka #30727: [2.7] test_threading.ConditionTests.test_notify() hangs random http://bugs.python.org/issue30727 closed by serhiy.storchaka #30729: Swap doesn't work in some circumstances http://bugs.python.org/issue30729 closed by rhettinger #30736: Support Unicode 10.0 http://bugs.python.org/issue30736 closed by benjamin.peterson #30738: __next__() method in iterators 9.9 http://bugs.python.org/issue30738 closed by eryksun #1080387: Making IDLE Themes and Keys Config more Robust http://bugs.python.org/issue1080387 closed by terry.reedy From tjreedy at udel.edu Fri Jun 23 12:36:04 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 23 Jun 2017 12:36:04 -0400 Subject: [Python-Dev] Appveyor builds fail on Windows for 3.6 backports. Message-ID: Example: appveyor passes for https://github.com/python/cpython/pull/2335#discussion_r123724857 Appveyor build fails for 3.6 backport https://github.com/python/cpython/pull/2359 https://ci.appveyor.com/project/python/cpython/build/3.6.1+.3673 The compile and build errors have nothing to do with the patch. This failure is routine; turning off appveyor for 3.6, until fixed, would reduce noise. I presume this is a workflow issue, but I don't know if Windows experts would see it. -- Terry Jan Reedy From brett at python.org Fri Jun 23 12:19:46 2017 From: brett at python.org (Brett Cannon) Date: Fri, 23 Jun 2017 16:19:46 +0000 Subject: [Python-Dev] [Core-mentorship] C version of weakrefset.py. In-Reply-To: References: <557E27CF-936C-4E78-8977-C98C88BE74DC@gmail.com> Message-ID: On Thu, 22 Jun 2017 at 16:26 Bhavishya wrote: > Also I saw your conversation with "Brett Cannon" on lazy-loading some > modules at startup, and also doing so using ModuleProxy(also several > implementations that exist like importlib's Lazyloader > , > PEAK ). > So your suggestion's on this too? > As stated earlier, if you wish to discuss seeing if lazy loading will help in Python's startup performance then please start a thread over at speed at python.org and we can discuss it there. -Brett > > Thank You > > regards, > Bhavishya > > On Thu, Jun 22, 2017 at 7:24 PM, Bhavishya > wrote: > >> Hello, >> 1)I might be totally wrong here, but even if we go with stripping >> annotation(in .pyc)...still the "lag" that comes from ABCs needs to be >> addressed. >> >> 2) I been reading for past few days about your fat-optimizer project and >> the corresponding TO-DO list, if you think that PEP-0511 should be >> improved, I can work on that. >> >> 3)Also I was seeing to existing repos which implement some-kind of >> optimizaton, like >> numpy,snake-oil.... >> >> Finally I wanted to decide upon a roadmap, so that I could put more >> specific efforts.(fat optimizer?) >> >> Thank You >> Regards, >> Bhavishya >> >> On Wed, Jun 21, 2017 at 8:05 PM, Victor Stinner > > wrote: >> >>> 2017-06-21 15:21 GMT+02:00 INADA Naoki : >>> > ABC slowdown Python startup only 2ms. But importing typing module >>> take 11ms. >>> > While typing is not imported from site.py, many new Python application >>> > will import it. >>> > It may take over 100ms for applications or libraries heavily depending >>> on ABCs. >>> >>> When typing is not used in the application, only used for static >>> checks, you can try to "strip" annotations to avoid any overhead on >>> the application startup. It's not only a matter of "import typing", >>> it's also the cost of instanciating types like "List[int]" (or even >>> more complex ones). >>> >>> I discussed with Jukka Lehtosalo at Pycon US about stripping >>> completely annotations. He told me that my PEP 511 may be a good >>> solution to keep annotation in the .py code, but strip them for >>> "production code", in the cached .pyc files: >>> >>> "PEP 511 -- API for code transformers" >>> https://www.python.org/dev/peps/pep-0511/ >>> >>> This PEP is somehow controversal. Some people fear that it would allow >>> people to hack the Python language to write their own incompatible >>> variant of Python. I don't think that my PEP adds anything new, it's >>> already possible to do that, importlib made it even more easy. I used >>> my FAT Python optimizer project to sell this PEP. Since FAT Python is >>> also controversal (it hasn't been proved to be actually faster), the >>> PEP didn't go far at my last attempt. >>> >>> Note: Is core-menthorship the best place for such performance >>> discussion? :-) Maybe we should open a thread on python-dev@ or speed@ >>> mailing list. >>> >>> Victor >>> >> >> > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachary.ware+pydev at gmail.com Fri Jun 23 14:24:25 2017 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Fri, 23 Jun 2017 13:24:25 -0500 Subject: [Python-Dev] Appveyor builds fail on Windows for 3.6 backports. In-Reply-To: References: Message-ID: On Fri, Jun 23, 2017 at 11:36 AM, Terry Reedy wrote: > Example: appveyor passes for > https://github.com/python/cpython/pull/2335#discussion_r123724857 > > Appveyor build fails for 3.6 backport > https://github.com/python/cpython/pull/2359 > https://ci.appveyor.com/project/python/cpython/build/3.6.1+.3673 > The compile and build errors have nothing to do with the patch. > > This failure is routine; turning off appveyor for 3.6, until fixed, would > reduce noise. > > I presume this is a workflow issue, but I don't know if Windows experts > would see it. I had not seen that failure before, but it is troubling. For AppVeyor, it looks like we might be able to get around it by using the `cache` feature (which we should probably do anyway), but if that failure happens routinely otherwise, we may have to completely rethink the approach I've taken in https://bugs.python.org/issue30450 -- Zach From tjreedy at udel.edu Fri Jun 23 14:41:44 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 23 Jun 2017 14:41:44 -0400 Subject: [Python-Dev] Appveyor builds fail on Windows for 3.6 backports. In-Reply-To: References: Message-ID: On 6/23/2017 2:24 PM, Zachary Ware wrote: > On Fri, Jun 23, 2017 at 11:36 AM, Terry Reedy wrote: >> Example: appveyor passes for >> https://github.com/python/cpython/pull/2335#discussion_r123724857 >> >> Appveyor build fails for 3.6 backport >> https://github.com/python/cpython/pull/2359 >> https://ci.appveyor.com/project/python/cpython/build/3.6.1+.3673 >> The compile and build errors have nothing to do with the patch. >> >> This failure is routine; turning off appveyor for 3.6, until fixed, would >> reduce noise. >> >> I presume this is a workflow issue, but I don't know if Windows experts >> would see it. > > I had not seen that failure before, but it is troubling. For > AppVeyor, it looks like we might be able to get around it by using the > `cache` feature (which we should probably do anyway), but if that > failure happens routinely otherwise, we may have to completely rethink > the approach I've taken in https://bugs.python.org/issue30450 If by 'cache', you mean keep a current build of the main x.y branch, and only recompile what is needed, and skip compile for .py and .rst only patches, thereby speeding up CI testing, I would be all in favor. -- Terry Jan Reedy From steve.dower at python.org Fri Jun 23 16:14:03 2017 From: steve.dower at python.org (Steve Dower) Date: Fri, 23 Jun 2017 13:14:03 -0700 Subject: [Python-Dev] Appveyor builds fail on Windows for 3.6 backports. In-Reply-To: References: Message-ID: We could improve life for everyone even more if we built Tcl/Tk once per update and made it a binary dependency. Nobody is updating our version of it regularly anyway, and this would significantly improve build time in CI. The externals script probably just needs some retry logic. Doesn?t look like a ?throw it all out? problem to me. Cheers, Steve Top-posted from my Windows phone From: Terry Reedy Sent: Friday, June 23, 2017 11:44 To: python-dev at python.org Subject: Re: [Python-Dev] Appveyor builds fail on Windows for 3.6 backports. On 6/23/2017 2:24 PM, Zachary Ware wrote: > On Fri, Jun 23, 2017 at 11:36 AM, Terry Reedy wrote: >> Example: appveyor passes for >> https://github.com/python/cpython/pull/2335#discussion_r123724857 >> >> Appveyor build fails for 3.6 backport >> https://github.com/python/cpython/pull/2359 >> https://ci.appveyor.com/project/python/cpython/build/3.6.1+.3673 >> The compile and build errors have nothing to do with the patch. >> >> This failure is routine; turning off appveyor for 3.6, until fixed, would >> reduce noise. >> >> I presume this is a workflow issue, but I don't know if Windows experts >> would see it. > > I had not seen that failure before, but it is troubling. For > AppVeyor, it looks like we might be able to get around it by using the > `cache` feature (which we should probably do anyway), but if that > failure happens routinely otherwise, we may have to completely rethink > the approach I've taken in https://bugs.python.org/issue30450 If by 'cache', you mean keep a current build of the main x.y branch, and only recompile what is needed, and skip compile for .py and .rst only patches, thereby speeding up CI testing, I would be all in favor. -- Terry Jan Reedy _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Fri Jun 23 20:04:24 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 23 Jun 2017 20:04:24 -0400 Subject: [Python-Dev] Appveyor builds fail on Windows for 3.6 backports. In-Reply-To: References: Message-ID: On 6/23/2017 4:14 PM, Steve Dower wrote: > We could improve life for everyone even more if we built Tcl/Tk once per > update and made it a binary dependency. Nobody is updating our version > of it regularly anyway, and this would significantly improve build time > in CI. On Windows, I believe compiling tcl/tk takes longer to compile than python. It is certainly close. -- Terry Jan Reedy From larry at hastings.org Fri Jun 23 23:24:05 2017 From: larry at hastings.org (Larry Hastings) Date: Fri, 23 Jun 2017 20:24:05 -0700 Subject: [Python-Dev] New workflow change: Welcome to blurb Message-ID: One minor but ongoing problem we've had in CPython core development has been the mess of updating Misc/NEWS. Day-to-day developers may have a conflict if they lose a push race, which means a little editing. You'll have a similar, if slightly worse, problem when cherry-picking a fix between versions. Worst of all used to be the manual merges necessary after cutting a release--this was the bane of a CPython release manager's existence. (Though the new git-based workflow may have obviated the worst of this.) The real problem is that we have one central file that everybody continually edits in a haphazard way. We aren't actually editing the same information, we aren't actually changing the same lines. But our revision control systems and diff algorithms don't understand the structure of Misc/NEWS and so they get confused. And for what? It's not like there's a tremendous benefit to having this central file everyone's fighting over. We've been talking about addressing this for years. Fixing this was one of the goals of the new workflow. And finally, as of right now, the future is here. Ladies and gentlemen, I present: blurb. https://github.com/python/core-workflow/tree/master/blurb blurb is an interactive command-line tool that helps you write Misc/NEWS entries. You simply run blurb from anywhere inside a CPython repo. blurb runs an editor for you with a template open. You fill in these three pieces of information: * the bugs.python.org or "bpo" issue number, * what "section" of Misc/NEWS this entry should go in (by uncommenting the correct line), and * the text of the Misc/NEWS entry, in ReST format. You save and exit and you're done. blurb even stages the Misc/NEWS entry in git for you! Behind the scenes, blurb writes your information here: Misc/NEWS.d/next// The "" is the name of the section in Misc/NEWS where your entry should go. contains the current date and time, the bpo number, and a nonce to prevent collisions. These "next" files get merged together into a single aggregate .rst file by the release manager when cutting a release (using "blurb release"). One nice feature of this approach: when you cherry-pick a change, its Misc/NEWS entry in "next" gets cherry-picked along with it. One important change: Misc/NEWS will no longer be checked in. It'll still be present in CPython tarballs; it will be generated by the release manager as part of cutting a release. But as a repository of information, it's been superseded by the various blurb data files. And by regenerating it from data files, we ensure that we'll never ever have a Misc/NEWS conflict ever again! The plan is to leave Misc/NEWS in the CPython repo for maybe another week, to let the current crop of PRs get merged. But new work should switch to using blurb immediately. You can install blurb from pip: % pip3.6 install blurb In fact--please do! //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rodrigc at freebsd.org Sat Jun 24 00:01:50 2017 From: rodrigc at freebsd.org (Craig Rodrigues) Date: Fri, 23 Jun 2017 21:01:50 -0700 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: References: Message-ID: On Fri, Jun 23, 2017 at 8:24 PM, Larry Hastings wrote: > > > We've been talking about addressing this for years. Fixing this was one > of the goals of the new workflow. And finally, as of right now, the future > is here. Ladies and gentlemen, I present: blurb. > > https://github.com/python/core-workflow/tree/master/blurb > > Yes, when you have a single NEWS file that needs to get updated, the process begins to fall apart when you have a pull request type of workflow which results in many merge conflicts to the single NEWS file. Twisted and Buildbot use towncrier ( https://github.com/hawkowl/towncrier ) which tries to solve the same problem as blurb. The developer needs to commit individual newsfragment files to the tree along with their pull request. At release time, the release engineer runs towncrier which aggregates the individual newsfragments into a single NEWS file. The NEWS file is not checked in, but the release engineer makes it available as part of official release tarballs. -- Craig -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Sat Jun 24 01:18:28 2017 From: steve.dower at python.org (Steve Dower) Date: Fri, 23 Jun 2017 22:18:28 -0700 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: References: Message-ID: One quick heads up ? the NEWS file is included in the docs build (if not in the html docs, certainly in the CHM for Windows releases). You may have to do some extra work to keep that from breaking when you remove it. We might also include it as plain text in the installers, I forget right now. Is blurb going to be embedded in the main repository? Not necessarily a problem if not, but I'd rather not have the build process depend on pip. Though I guess Sphinx is dependency already, so perhaps I should just integrate it better into the build? Top-posted from my Windows phone From: Larry Hastings Sent: Friday, June 23, 2017 20:26 To: Python Dev; python-committers Subject: [Python-Dev] New workflow change: Welcome to blurb One minor but ongoing problem we've had in CPython core development has been the mess of updating Misc/NEWS.? Day-to-day developers may have a conflict if they lose a push race, which means a little editing.? You'll have a similar, if slightly worse, problem when cherry-picking a fix between versions.? Worst of all used to be the manual merges necessary after cutting a release--this was the bane of a CPython release manager's existence.? (Though the new git-based workflow may have obviated the worst of this.) The real problem is that we have one central file that everybody continually edits in a haphazard way.? We aren't actually editing the same information, we aren't actually changing the same lines.? But our revision control systems and diff algorithms don't understand the structure of Misc/NEWS and so they get confused.? And for what? It's not like there's a tremendous benefit to having this central file everyone's fighting over. We've been talking about addressing this for years.? Fixing this was one of the goals of the new workflow.? And finally, as of right now, the future is here.? Ladies and gentlemen, I present: blurb. https://github.com/python/core-workflow/tree/master/blurb blurb is an interactive command-line tool that helps you write Misc/NEWS entries.? You simply run blurb from anywhere inside a CPython repo.? blurb runs an editor for you with a template open.? You fill in these three pieces of information: ? the bugs.python.org or "bpo" issue number, ? what "section" of Misc/NEWS this entry should go in (by uncommenting the correct line), and ? the text of the Misc/NEWS entry, in ReST format. You save and exit and you're done.? blurb even stages the Misc/NEWS entry in git for you! Behind the scenes, blurb writes your information here: Misc/NEWS.d/next// The "" is the name of the section in Misc/NEWS where your entry should go.? contains the current date and time, the bpo number, and a nonce to prevent collisions. These "next" files get merged together into a single aggregate .rst file by the release manager when cutting a release (using "blurb release").? One nice feature of this approach: when you cherry-pick a change, its Misc/NEWS entry in "next" gets cherry-picked along with it. One important change: Misc/NEWS will no longer be checked in.? It'll still be present in CPython tarballs; it will be generated by the release manager as part of cutting a release.? But as a repository of information, it's been superseded by the various blurb data files.? And by regenerating it from data files, we ensure that we'll never ever have a Misc/NEWS conflict ever again! The plan is to leave Misc/NEWS in the CPython repo for maybe another week, to let the current crop of PRs get merged.? But new work should switch to using blurb immediately. You can install blurb from pip: % pip3.6 install blurb In fact--please do! /arry -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Jun 24 01:48:01 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 24 Jun 2017 15:48:01 +1000 Subject: [Python-Dev] [python-committers] New workflow change: Welcome to blurb In-Reply-To: References: Message-ID: On 24 June 2017 at 13:24, Larry Hastings wrote: > > > One minor but ongoing problem we've had in CPython core development has been > the mess of updating Misc/NEWS. Day-to-day developers may have a conflict > if they lose a push race, which means a little editing. You'll have a > similar, if slightly worse, problem when cherry-picking a fix between > versions. Worst of all used to be the manual merges necessary after cutting > a release--this was the bane of a CPython release manager's existence. > (Though the new git-based workflow may have obviated the worst of this.) > > The real problem is that we have one central file that everybody continually > edits in a haphazard way. We aren't actually editing the same information, > we aren't actually changing the same lines. But our revision control > systems and diff algorithms don't understand the structure of Misc/NEWS and > so they get confused. And for what? It's not like there's a tremendous > benefit to having this central file everyone's fighting over. > > We've been talking about addressing this for years. Fixing this was one of > the goals of the new workflow. And finally, as of right now, the future is > here. Ladies and gentlemen, I present: blurb. > > https://github.com/python/core-workflow/tree/master/blurb Thanks Larry, great to see this go live! > Behind the scenes, blurb writes your information here: > > Misc/NEWS.d/next// > > The "" is the name of the section in Misc/NEWS where your > entry should go. contains the current date and time, the bpo > number, and a nonce to prevent collisions. Folks are also free to handcraft these files if they really want to do so. The Developer Guide has the necessary details: https://docs.python.org/devguide/committing.html#what-s-new-and-news-entries Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Jun 24 01:55:55 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 24 Jun 2017 15:55:55 +1000 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: References: Message-ID: On 24 June 2017 at 14:01, Craig Rodrigues wrote: > On Fri, Jun 23, 2017 at 8:24 PM, Larry Hastings wrote: >> >> >> >> We've been talking about addressing this for years. Fixing this was one >> of the goals of the new workflow. And finally, as of right now, the future >> is here. Ladies and gentlemen, I present: blurb. >> >> https://github.com/python/core-workflow/tree/master/blurb > > > Yes, when you have a single NEWS file that needs to get updated, > the process begins to fall apart when you have a pull request type of > workflow > which results in many merge conflicts to the single NEWS file. > > Twisted and Buildbot use towncrier ( https://github.com/hawkowl/towncrier ) > which tries > to solve the same problem as blurb. Aye, towncrier and OpenStack's reno were the two main alternatives we looked at in addition to Larry's offer of creating a tool specifically for CPython: https://github.com/python/core-workflow/issues/6 We ultimately settled on blurb mainly because we wanted to be able to customise various details differently from the way towncrier works, and we figure they're close enough in spirit that folks familiar with one won't have any problems adapting to the other. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Jun 24 02:00:11 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 24 Jun 2017 16:00:11 +1000 Subject: [Python-Dev] Appveyor builds fail on Windows for 3.6 backports. In-Reply-To: References: Message-ID: On 24 June 2017 at 06:14, Steve Dower wrote: > We could improve life for everyone even more if we built Tcl/Tk once per > update and made it a binary dependency. Nobody is updating our version of it > regularly anyway, and this would significantly improve build time in CI. I'm pretty sure the *nix builds already default to using the system Tcl/Tk, so figuring out a way to do something similar on Windows (and then bundling the DLLs in the installer) would make sense to me. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From storchaka at gmail.com Sat Jun 24 02:25:16 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 24 Jun 2017 09:25:16 +0300 Subject: [Python-Dev] [python-committers] New workflow change: Welcome to blurb In-Reply-To: References: Message-ID: 2017-06-24 6:24 GMT+03:00 Larry Hastings : > One minor but ongoing problem we've had in CPython core development has been > the mess of updating Misc/NEWS. Day-to-day developers may have a conflict > if they lose a push race, which means a little editing. You'll have a > similar, if slightly worse, problem when cherry-picking a fix between > versions. Worst of all used to be the manual merges necessary after cutting > a release--this was the bane of a CPython release manager's existence. > (Though the new git-based workflow may have obviated the worst of this.) Thanks Larry! With the old hg-based workflow this was only slightly annoying, but the new git-based workflow turned this into a hell. If you have several PRs that updates the same Misc/NEWS section you needed to spent many time for just resolving conflicts one by one and waiting CI tests. And be lucky if other core developer trying to do the same withis PRs at the same time. > You can install blurb from pip: > > % pip3.6 install blurb > > In fact--please do! I have installed it, but how to use it? $ python3 -m pip install --user blurb Collecting blurb Using cached blurb-1.0-py3-none-any.whl Installing collected packages: blurb Successfully installed blurb-1.0 $ python3 -m blurb /usr/bin/python3: No module named blurb From solipsis at pitrou.net Sat Jun 24 04:53:43 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 24 Jun 2017 10:53:43 +0200 Subject: [Python-Dev] New workflow change: Welcome to blurb References: Message-ID: <20170624105343.6cca748b@fsol> On Fri, 23 Jun 2017 20:24:05 -0700 Larry Hastings wrote: > > We've been talking about addressing this for years. Fixing this was one > of the goals of the new workflow. And finally, as of right now, the > future is here. Ladies and gentlemen, I present: blurb. > > https://github.com/python/core-workflow/tree/master/blurb "blurb's only dependency is Python 3.6+." Would you like to make it 3.5-compatible? 3.6 is quite young and not all systems have it (e.g. Ubuntu 16.04, which many people use, has 3.5). Regards Antoine. From storchaka at gmail.com Sat Jun 24 06:03:12 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 24 Jun 2017 13:03:12 +0300 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: <20170624105343.6cca748b@fsol> References: <20170624105343.6cca748b@fsol> Message-ID: 24.06.17 11:53, Antoine Pitrou ????: > On Fri, 23 Jun 2017 20:24:05 -0700 > Larry Hastings wrote: >> >> We've been talking about addressing this for years. Fixing this was one >> of the goals of the new workflow. And finally, as of right now, the >> future is here. Ladies and gentlemen, I present: blurb. >> >> https://github.com/python/core-workflow/tree/master/blurb > > "blurb's only dependency is Python 3.6+." > > Would you like to make it 3.5-compatible? 3.6 is quite young and not > all systems have it (e.g. Ubuntu 16.04, which many people use, has 3.5). +1 From larry at hastings.org Sat Jun 24 11:53:50 2017 From: larry at hastings.org (Larry Hastings) Date: Sat, 24 Jun 2017 08:53:50 -0700 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: References: Message-ID: On 06/23/2017 10:55 PM, Nick Coghlan wrote: > Aye, towncrier and OpenStack's reno were the two main alternatives we > looked at in addition to Larry's offer of creating a tool specifically > for CPython: https://github.com/python/core-workflow/issues/6 Fun fact: all three tools started at about the same time, at least according to publicly visible commits. Towncrier started December 2015, reno in August 2015, and my first commits to "mergenews" (which eventually became blurb) were in September of 2015. We'd actually been discussing it on bpo since at least September 2013: https://bugs.python.org/issue18967 //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Sat Jun 24 11:57:24 2017 From: larry at hastings.org (Larry Hastings) Date: Sat, 24 Jun 2017 08:57:24 -0700 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: <20170624105343.6cca748b@fsol> References: <20170624105343.6cca748b@fsol> Message-ID: On 06/24/2017 01:53 AM, Antoine Pitrou wrote: > Would you like to make it 3.5-compatible? 3.6 is quite young and not > all systems have it (e.g. Ubuntu 16.04, which many people use, has 3.5). Well, tbh I think that's a bit silly. First of all, it shouldn't be installed in your system-wide python3 interpreter. Second, I assumed core devs were compiling their own Python interpreters locally in their account--in no small part so they can install any packages they want. And finally, you're asking me to give up f-strings. If someone else wants to send me a PR getting it to work on 3.5, I'd look at it. I'm guessing it's really just f-strings but I can't say for sure what else is a 3.6 dependency. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sat Jun 24 12:10:36 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 24 Jun 2017 18:10:36 +0200 Subject: [Python-Dev] New workflow change: Welcome to blurb References: <20170624105343.6cca748b@fsol> Message-ID: <20170624181036.5d180830@fsol> On Sat, 24 Jun 2017 08:57:24 -0700 Larry Hastings wrote: > On 06/24/2017 01:53 AM, Antoine Pitrou wrote: > > Would you like to make it 3.5-compatible? 3.6 is quite young and not > > all systems have it (e.g. Ubuntu 16.04, which many people use, has 3.5). > > Well, tbh I think that's a bit silly. First of all, it shouldn't be > installed in your system-wide python3 interpreter. Well, you can always create a venv using the system python3. Nowadays, to be honest, I use conda and Anaconda a lot, so I can very quickly create a local Python 3.6 environment on my work machine, even if my OS doesn't provide a Python 3.6. I'm just not sure everyone has the same habits as me, though I'm all for more core developers adopting conda :-) > Second, I assumed > core devs were compiling their own Python interpreters locally in their > account--in no small part so they can install any packages they want. I used to do that, but in the end it feels too confusing and cumbersome. So I don't install self-compiled Pythons anymore. Of course, since you are doing the work, it's ok for me if you don't want to make blurb Python 3.5-compatible. I'm just pointing out that it would ideally be better if it were :-) Regards Antoine. From tjreedy at udel.edu Sat Jun 24 12:27:21 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 24 Jun 2017 12:27:21 -0400 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: References: <20170624105343.6cca748b@fsol> Message-ID: On 6/24/2017 11:57 AM, Larry Hastings wrote: > On 06/24/2017 01:53 AM, Antoine Pitrou wrote: >> Would you like to make it 3.5-compatible? 3.6 is quite young and not >> all systems have it (e.g. Ubuntu 16.04, which many people use, has 3.5). > > Well, tbh I think that's a bit silly. First of all, it shouldn't be > installed in your system-wide python3 interpreter. Second, I assumed > core devs were compiling their own Python interpreters locally in their > account--in no small part so they can install any packages they want. > And finally, you're asking me to give up f-strings. > > If someone else wants to send me a PR getting it to work on 3.5, I'd > look at it. I'm guessing it's really just f-strings but I can't say for > sure what else is a 3.6 dependency. I believe cherry-picker also requires 3.6, for the same reason. -- Terry Jan Reedy From larry at hastings.org Sat Jun 24 12:38:05 2017 From: larry at hastings.org (Larry Hastings) Date: Sat, 24 Jun 2017 09:38:05 -0700 Subject: [Python-Dev] [python-committers] New workflow change: Welcome to blurb In-Reply-To: References: Message-ID: On 06/23/2017 11:25 PM, Serhiy Storchaka wrote: > I have installed it, but how to use it? > > $ python3 -m pip install --user blurb > Collecting blurb > Using cached blurb-1.0-py3-none-any.whl > Installing collected packages: blurb > Successfully installed blurb-1.0 > $ python3 -m blurb > /usr/bin/python3: No module named blurb It's on your path. Just run % blurb from inside a CPython repo. I'm amazed that your first thought was "python -m blurb". //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sat Jun 24 12:40:38 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 24 Jun 2017 12:40:38 -0400 Subject: [Python-Dev] [python-committers] New workflow change: Welcome to blurb In-Reply-To: References: Message-ID: <154d618d-f72e-25cd-efdf-dd278d87e73b@udel.edu> On 6/23/2017 11:24 PM, Larry Hastings wrote: > You can install blurb from pip: > > % pip3.6 install blurb This does not seem to work right. On Windows: C:\Users\Terry>py -3 -m pip install blurb Collecting blurb Downloading blurb-1.0-py3-none-any.whl Installing collected packages: blurb Successfully installed blurb-1.0 Explorer shows that 3.6 site-packages has a 'blurb-1.0.dist-info' directory but neither blurb.py nor 'blurb/' is present. So the following are to be expected. C:\Users\Terry>py -3 -m blurb C:\Programs\Python36\python.exe: No module named blurb > py -3 >>> import blurb ... ModuleNotFoundError: No module named 'blurb' Serhiy reported a similar problem on, I presume, some flavor of Linux. tjr From larry at hastings.org Sat Jun 24 12:45:41 2017 From: larry at hastings.org (Larry Hastings) Date: Sat, 24 Jun 2017 09:45:41 -0700 Subject: [Python-Dev] [python-committers] New workflow change: Welcome to blurb In-Reply-To: <154d618d-f72e-25cd-efdf-dd278d87e73b@udel.edu> References: <154d618d-f72e-25cd-efdf-dd278d87e73b@udel.edu> Message-ID: <84adf4d7-8e3e-3b58-3e45-71f7c6135083@hastings.org> On 06/24/2017 09:40 AM, Terry Reedy wrote: > On 6/23/2017 11:24 PM, Larry Hastings wrote: > > > You can install blurb from pip: > > > > % pip3.6 install blurb > > This does not seem to work right. On Windows: > > C:\Users\Terry>py -3 -m pip install blurb > Collecting blurb > Downloading blurb-1.0-py3-none-any.whl > Installing collected packages: blurb > Successfully installed blurb-1.0 > > Explorer shows that 3.6 site-packages has a 'blurb-1.0.dist-info' > directory but neither blurb.py nor 'blurb/' is present. So the > following are to be expected. > > C:\Users\Terry>py -3 -m blurb > C:\Programs\Python36\python.exe: No module named blurb > > > py -3 > >>> import blurb > ... > ModuleNotFoundError: No module named 'blurb' > > Serhiy reported a similar problem on, I presume, some flavor of Linux. I replied to Serhiy; it's just "blurb", it's a command-line tool, it's not a package or a module. It should be a command on your path. TBH I don't know if installation of a command-line tool like that works on Windows. The tool itself was ported to Windows by Zach at the PyCon core dev sprints last month, though that predates the PyPI work, and in any case I could have broken the Windows support since then. Unfortunately I'm no longer a qualified Windows developer, so if it doesn't work on Windows I fear someone will have to send me a PR. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Jun 24 12:54:59 2017 From: brett at python.org (Brett Cannon) Date: Sat, 24 Jun 2017 16:54:59 +0000 Subject: [Python-Dev] [python-committers] New workflow change: Welcome to blurb In-Reply-To: <84adf4d7-8e3e-3b58-3e45-71f7c6135083@hastings.org> References: <154d618d-f72e-25cd-efdf-dd278d87e73b@udel.edu> <84adf4d7-8e3e-3b58-3e45-71f7c6135083@hastings.org> Message-ID: On Sat, 24 Jun 2017 at 09:46 Larry Hastings wrote: > On 06/24/2017 09:40 AM, Terry Reedy wrote: > > On 6/23/2017 11:24 PM, Larry Hastings wrote: > > > You can install blurb from pip: > > > > % pip3.6 install blurb > > This does not seem to work right. On Windows: > > C:\Users\Terry>py -3 -m pip install blurb > Collecting blurb > Downloading blurb-1.0-py3-none-any.whl > Installing collected packages: blurb > Successfully installed blurb-1.0 > > Explorer shows that 3.6 site-packages has a 'blurb-1.0.dist-info' > directory but neither blurb.py nor 'blurb/' is present. So the following > are to be expected. > > C:\Users\Terry>py -3 -m blurb > C:\Programs\Python36\python.exe: No module named blurb > > > py -3 > >>> import blurb > ... > ModuleNotFoundError: No module named 'blurb' > > Serhiy reported a similar problem on, I presume, some flavor of Linux. > > > I replied to Serhiy; it's just "blurb", it's a command-line tool, it's not > a package or a module. It should be a command on your path. > > TBH I don't know if installation of a command-line tool like that works on > Windows. The tool itself was ported to Windows by Zach at the PyCon core > dev sprints last month, though that predates the PyPI work, and in any case > I could have broken the Windows support since then. Unfortunately I'm no > longer a qualified Windows developer, so if it doesn't work on Windows I > fear someone will have to send me a PR. > One of the great perks of `python3 -m blurb` is it avoids needing to care about your PATH on any platform. Anyway, the next release of blurb -- whether that's 1.0.0.post1 or a bigger release -- will have a blurb.py as well as the entry point giving people the `blurb` command. And people can also use pipsi if they want to install blurb as more of a self-contained command-line app (at least on UNIX; don't know about its support on Windows). -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Jun 24 12:59:51 2017 From: brett at python.org (Brett Cannon) Date: Sat, 24 Jun 2017 16:59:51 +0000 Subject: [Python-Dev] [python-committers] New workflow change: Welcome to blurb In-Reply-To: References: <154d618d-f72e-25cd-efdf-dd278d87e73b@udel.edu> <84adf4d7-8e3e-3b58-3e45-71f7c6135083@hastings.org> Message-ID: I just pushed blurb 1.0.0.post1 which re-packages everything using flit so there's a blurb.py and an entry point for the `blurb` command. That should meet everyone's needs for launching the tool. On Sat, 24 Jun 2017 at 09:54 Brett Cannon wrote: > On Sat, 24 Jun 2017 at 09:46 Larry Hastings wrote: > >> On 06/24/2017 09:40 AM, Terry Reedy wrote: >> >> On 6/23/2017 11:24 PM, Larry Hastings wrote: >> >> > You can install blurb from pip: >> > >> > % pip3.6 install blurb >> >> This does not seem to work right. On Windows: >> >> C:\Users\Terry>py -3 -m pip install blurb >> Collecting blurb >> Downloading blurb-1.0-py3-none-any.whl >> Installing collected packages: blurb >> Successfully installed blurb-1.0 >> >> Explorer shows that 3.6 site-packages has a 'blurb-1.0.dist-info' >> directory but neither blurb.py nor 'blurb/' is present. So the following >> are to be expected. >> >> C:\Users\Terry>py -3 -m blurb >> C:\Programs\Python36\python.exe: No module named blurb >> >> > py -3 >> >>> import blurb >> ... >> ModuleNotFoundError: No module named 'blurb' >> >> Serhiy reported a similar problem on, I presume, some flavor of Linux. >> >> >> I replied to Serhiy; it's just "blurb", it's a command-line tool, it's >> not a package or a module. It should be a command on your path. >> >> TBH I don't know if installation of a command-line tool like that works >> on Windows. The tool itself was ported to Windows by Zach at the PyCon >> core dev sprints last month, though that predates the PyPI work, and in any >> case I could have broken the Windows support since then. Unfortunately I'm >> no longer a qualified Windows developer, so if it doesn't work on Windows I >> fear someone will have to send me a PR. >> > > One of the great perks of `python3 -m blurb` is it avoids needing to care > about your PATH on any platform. > > Anyway, the next release of blurb -- whether that's 1.0.0.post1 or a > bigger release -- will have a blurb.py as well as the entry point giving > people the `blurb` command. And people can also use pipsi if they want to > install blurb as more of a self-contained command-line app (at least on > UNIX; don't know about its support on Windows). > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sat Jun 24 13:06:31 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 24 Jun 2017 19:06:31 +0200 Subject: [Python-Dev] New workflow change: Welcome to blurb References: Message-ID: <20170624190631.58dac153@fsol> Larry, I have just used blurb on https://github.com/python/cpython/pull/2010 and it was absolutely painless. Thank you! One minor thing: `make patchcheck` doesn't know to examine the NEWS.d directory when checking for NEWS entries. Regards Antoine. On Fri, 23 Jun 2017 20:24:05 -0700 Larry Hastings wrote: > One minor but ongoing problem we've had in CPython core development has > been the mess of updating Misc/NEWS. Day-to-day developers may have a > conflict if they lose a push race, which means a little editing. You'll > have a similar, if slightly worse, problem when cherry-picking a fix > between versions. Worst of all used to be the manual merges necessary > after cutting a release--this was the bane of a CPython release > manager's existence. (Though the new git-based workflow may have > obviated the worst of this.) > > The real problem is that we have one central file that everybody > continually edits in a haphazard way. We aren't actually editing the > same information, we aren't actually changing the same lines. But our > revision control systems and diff algorithms don't understand the > structure of Misc/NEWS and so they get confused. And for what? It's not > like there's a tremendous benefit to having this central file everyone's > fighting over. > > We've been talking about addressing this for years. Fixing this was one > of the goals of the new workflow. And finally, as of right now, the > future is here. Ladies and gentlemen, I present: blurb. > > https://github.com/python/core-workflow/tree/master/blurb > > > blurb is an interactive command-line tool that helps you write Misc/NEWS > entries. You simply run blurb from anywhere inside a CPython repo. > blurb runs an editor for you with a template open. You fill in these > three pieces of information: > > * the bugs.python.org or "bpo" issue number, > * what "section" of Misc/NEWS this entry should go in (by uncommenting > the correct line), and > * the text of the Misc/NEWS entry, in ReST format. > > You save and exit and you're done. blurb even stages the Misc/NEWS > entry in git for you! > > > Behind the scenes, blurb writes your information here: > > Misc/NEWS.d/next// > > The "" is the name of the section in Misc/NEWS where your > entry should go. contains the current date and time, the bpo > number, and a nonce to prevent collisions. > > These "next" files get merged together into a single aggregate .rst file > by the release manager when cutting a release (using "blurb release"). > One nice feature of this approach: when you cherry-pick a change, its > Misc/NEWS entry in "next" gets cherry-picked along with it. > > > One important change: Misc/NEWS will no longer be checked in. It'll > still be present in CPython tarballs; it will be generated by the > release manager as part of cutting a release. But as a repository of > information, it's been superseded by the various blurb data files. And > by regenerating it from data files, we ensure that we'll never ever have > a Misc/NEWS conflict ever again! > > The plan is to leave Misc/NEWS in the CPython repo for maybe another > week, to let the current crop of PRs get merged. But new work should > switch to using blurb immediately. > > > You can install blurb from pip: > > % pip3.6 install blurb > > In fact--please do! > > > //arry/ > From solipsis at pitrou.net Sat Jun 24 13:21:42 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 24 Jun 2017 19:21:42 +0200 Subject: [Python-Dev] For all branches? [was Re: New workflow change: Welcome to blurb] References: Message-ID: <20170624192142.315f4ef8@fsol> One thing you didn't mention: is the switch on all branches or only master? Regards Antoine. On Fri, 23 Jun 2017 20:24:05 -0700 Larry Hastings wrote: > One minor but ongoing problem we've had in CPython core development has > been the mess of updating Misc/NEWS. Day-to-day developers may have a > conflict if they lose a push race, which means a little editing. You'll > have a similar, if slightly worse, problem when cherry-picking a fix > between versions. Worst of all used to be the manual merges necessary > after cutting a release--this was the bane of a CPython release > manager's existence. (Though the new git-based workflow may have > obviated the worst of this.) > > The real problem is that we have one central file that everybody > continually edits in a haphazard way. We aren't actually editing the > same information, we aren't actually changing the same lines. But our > revision control systems and diff algorithms don't understand the > structure of Misc/NEWS and so they get confused. And for what? It's not > like there's a tremendous benefit to having this central file everyone's > fighting over. > > We've been talking about addressing this for years. Fixing this was one > of the goals of the new workflow. And finally, as of right now, the > future is here. Ladies and gentlemen, I present: blurb. > > https://github.com/python/core-workflow/tree/master/blurb > > > blurb is an interactive command-line tool that helps you write Misc/NEWS > entries. You simply run blurb from anywhere inside a CPython repo. > blurb runs an editor for you with a template open. You fill in these > three pieces of information: > > * the bugs.python.org or "bpo" issue number, > * what "section" of Misc/NEWS this entry should go in (by uncommenting > the correct line), and > * the text of the Misc/NEWS entry, in ReST format. > > You save and exit and you're done. blurb even stages the Misc/NEWS > entry in git for you! > > > Behind the scenes, blurb writes your information here: > > Misc/NEWS.d/next// > > The "" is the name of the section in Misc/NEWS where your > entry should go. contains the current date and time, the bpo > number, and a nonce to prevent collisions. > > These "next" files get merged together into a single aggregate .rst file > by the release manager when cutting a release (using "blurb release"). > One nice feature of this approach: when you cherry-pick a change, its > Misc/NEWS entry in "next" gets cherry-picked along with it. > > > One important change: Misc/NEWS will no longer be checked in. It'll > still be present in CPython tarballs; it will be generated by the > release manager as part of cutting a release. But as a repository of > information, it's been superseded by the various blurb data files. And > by regenerating it from data files, we ensure that we'll never ever have > a Misc/NEWS conflict ever again! > > The plan is to leave Misc/NEWS in the CPython repo for maybe another > week, to let the current crop of PRs get merged. But new work should > switch to using blurb immediately. > > > You can install blurb from pip: > > % pip3.6 install blurb > > In fact--please do! > > > //arry/ > From larry at hastings.org Sat Jun 24 13:25:54 2017 From: larry at hastings.org (Larry Hastings) Date: Sat, 24 Jun 2017 10:25:54 -0700 Subject: [Python-Dev] For all branches? [was Re: New workflow change: Welcome to blurb] In-Reply-To: <20170624192142.315f4ef8@fsol> References: <20170624192142.315f4ef8@fsol> Message-ID: <0e5058ed-1fa1-ff55-db90-a9483c0482bd@hastings.org> On 06/24/2017 10:21 AM, Antoine Pitrou wrote: > One thing you didn't mention: is the switch on all branches or only > master? All active branches: 2.7 3.3 3.4 3.5 3.6 master You should see a Misc/NEWS.d directory on each of those with an up-to-date clone. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sat Jun 24 13:30:45 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 24 Jun 2017 13:30:45 -0400 Subject: [Python-Dev] [python-committers] New workflow change: Welcome to blurb In-Reply-To: <84adf4d7-8e3e-3b58-3e45-71f7c6135083@hastings.org> References: <154d618d-f72e-25cd-efdf-dd278d87e73b@udel.edu> <84adf4d7-8e3e-3b58-3e45-71f7c6135083@hastings.org> Message-ID: On 6/24/2017 12:45 PM, Larry Hastings wrote: > On 06/24/2017 09:40 AM, Terry Reedy wrote: >> On 6/23/2017 11:24 PM, Larry Hastings wrote: >> >> > You can install blurb from pip: >> > >> > % pip3.6 install blurb >> >> This does not seem to work right. On Windows: >> >> C:\Users\Terry>py -3 -m pip install blurb >> Collecting blurb >> Downloading blurb-1.0-py3-none-any.whl >> Installing collected packages: blurb >> Successfully installed blurb-1.0 >> >> Explorer shows that 3.6 site-packages has a 'blurb-1.0.dist-info' >> directory but neither blurb.py nor 'blurb/' is present. So the >> following are to be expected. >> >> C:\Users\Terry>py -3 -m blurb >> C:\Programs\Python36\python.exe: No module named blurb >> >> > py -3 >> >>> import blurb >> ... >> ModuleNotFoundError: No module named 'blurb' >> >> Serhiy reported a similar problem on, I presume, some flavor of Linux. > > I replied to Serhiy; it's just "blurb", it's a command-line tool, it's > not a package or a module. It should be a command on your path. The reason I tried " -m blurb" is because that is the standard and recommended way to run installed scripts on Windows. That is how I run pip and cherry_picker, for instance. I found 'blurb' in <36dir>/Scripts/. The name and location are errors. 1. On Windows, python files need the .py extension. 2. That directory is not currently on the path on my machine. I believe it once was, but installing 3.5.3 replaced it with the 3.5 /Scripts. On Windows, 3rd party installers must not presume that any /Scripts directory is on the path. By default, none are. Solution: name the file blurb.py and put it in site-packages. This is standard and what is done by all other pip-installs that I have run. Put a copy in /Scripts if you want, but that is really optional and only sometimes effective. > TBH I don't know if installation of a command-line tool like that works > on Windows. The tool itself was ported to Windows by Zach at the PyCon > core dev sprints last month, though that predates the PyPI work, and in > any case I could have broken the Windows support since then. > Unfortunately I'm no longer a qualified Windows developer, so if it > doesn't work on Windows I fear someone will have to send me a PR. I only know what the end result should be. Pip-installed Cherry_picker works on Windows, so copy from the spec files for that, or ask whoever wrote the pip-upload. From larry at hastings.org Sat Jun 24 18:15:54 2017 From: larry at hastings.org (Larry Hastings) Date: Sat, 24 Jun 2017 15:15:54 -0700 Subject: [Python-Dev] [python-committers] New workflow change: Welcome to blurb In-Reply-To: References: <154d618d-f72e-25cd-efdf-dd278d87e73b@udel.edu> <84adf4d7-8e3e-3b58-3e45-71f7c6135083@hastings.org> Message-ID: On 06/24/2017 10:30 AM, Terry Reedy wrote: > Solution: name the file blurb.py and put it in site-packages. This is > standard and what is done by all other pip-installs that I have run. > Put a copy in /Scripts if you want, but that is really optional and > only sometimes effective. Brett redid the installer with "flit" and pushed, and he says you should now be able to run blurb via "python3 -m blurb". Please update blurb (via pip3.6) and let us know if it now works for you on Windows. Cheers, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Jun 24 19:48:03 2017 From: brett at python.org (Brett Cannon) Date: Sat, 24 Jun 2017 23:48:03 +0000 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: <20170624190631.58dac153@fsol> References: <20170624190631.58dac153@fsol> Message-ID: On Sat, 24 Jun 2017 at 10:07 Antoine Pitrou wrote: > > Larry, > > I have just used blurb on https://github.com/python/cpython/pull/2010 > and it was absolutely painless. Thank you! > > One minor thing: `make patchcheck` doesn't know to examine the NEWS.d > directory when checking for NEWS entries. > I opened https://bugs.python.org/issue30750 for that if someone wants to make the change. I should also mention that after Larry blows up Misc/NEWS into individual files in about a week I will add a check to Bedevere for a news file unless the PR is labeled as trivial. -Brett > > Regards > > Antoine. > > > > On Fri, 23 Jun 2017 20:24:05 -0700 > Larry Hastings wrote: > > One minor but ongoing problem we've had in CPython core development has > > been the mess of updating Misc/NEWS. Day-to-day developers may have a > > conflict if they lose a push race, which means a little editing. You'll > > have a similar, if slightly worse, problem when cherry-picking a fix > > between versions. Worst of all used to be the manual merges necessary > > after cutting a release--this was the bane of a CPython release > > manager's existence. (Though the new git-based workflow may have > > obviated the worst of this.) > > > > The real problem is that we have one central file that everybody > > continually edits in a haphazard way. We aren't actually editing the > > same information, we aren't actually changing the same lines. But our > > revision control systems and diff algorithms don't understand the > > structure of Misc/NEWS and so they get confused. And for what? It's not > > like there's a tremendous benefit to having this central file everyone's > > fighting over. > > > > We've been talking about addressing this for years. Fixing this was one > > of the goals of the new workflow. And finally, as of right now, the > > future is here. Ladies and gentlemen, I present: blurb. > > > > https://github.com/python/core-workflow/tree/master/blurb > > > > > > blurb is an interactive command-line tool that helps you write Misc/NEWS > > entries. You simply run blurb from anywhere inside a CPython repo. > > blurb runs an editor for you with a template open. You fill in these > > three pieces of information: > > > > * the bugs.python.org or "bpo" issue number, > > * what "section" of Misc/NEWS this entry should go in (by uncommenting > > the correct line), and > > * the text of the Misc/NEWS entry, in ReST format. > > > > You save and exit and you're done. blurb even stages the Misc/NEWS > > entry in git for you! > > > > > > Behind the scenes, blurb writes your information here: > > > > Misc/NEWS.d/next// > > > > The "" is the name of the section in Misc/NEWS where your > > entry should go. contains the current date and time, the bpo > > number, and a nonce to prevent collisions. > > > > These "next" files get merged together into a single aggregate .rst file > > by the release manager when cutting a release (using "blurb release"). > > One nice feature of this approach: when you cherry-pick a change, its > > Misc/NEWS entry in "next" gets cherry-picked along with it. > > > > > > One important change: Misc/NEWS will no longer be checked in. It'll > > still be present in CPython tarballs; it will be generated by the > > release manager as part of cutting a release. But as a repository of > > information, it's been superseded by the various blurb data files. And > > by regenerating it from data files, we ensure that we'll never ever have > > a Misc/NEWS conflict ever again! > > > > The plan is to leave Misc/NEWS in the CPython repo for maybe another > > week, to let the current crop of PRs get merged. But new work should > > switch to using blurb immediately. > > > > > > You can install blurb from pip: > > > > % pip3.6 install blurb > > > > In fact--please do! > > > > > > //arry/ > > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sat Jun 24 19:54:54 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 25 Jun 2017 01:54:54 +0200 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: References: <20170624190631.58dac153@fsol> Message-ID: <20170625015454.5aad28e0@fsol> On Sat, 24 Jun 2017 23:48:03 +0000 Brett Cannon wrote: > On Sat, 24 Jun 2017 at 10:07 Antoine Pitrou wrote: > > > > > Larry, > > > > I have just used blurb on https://github.com/python/cpython/pull/2010 > > and it was absolutely painless. Thank you! > > > > One minor thing: `make patchcheck` doesn't know to examine the NEWS.d > > directory when checking for NEWS entries. > > > > I opened https://bugs.python.org/issue30750 for that if someone wants to > make the change. Ok, then I edited the title for https://github.com/python/cpython/pull/2381 :-) Regards Antoine. From ncoghlan at gmail.com Sat Jun 24 21:51:29 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 25 Jun 2017 11:51:29 +1000 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: <20170624181036.5d180830@fsol> References: <20170624105343.6cca748b@fsol> <20170624181036.5d180830@fsol> Message-ID: On 25 June 2017 at 02:10, Antoine Pitrou wrote: > On Sat, 24 Jun 2017 08:57:24 -0700 > Larry Hastings wrote: >> On 06/24/2017 01:53 AM, Antoine Pitrou wrote: >> > Would you like to make it 3.5-compatible? 3.6 is quite young and not >> > all systems have it (e.g. Ubuntu 16.04, which many people use, has 3.5). >> >> Well, tbh I think that's a bit silly. First of all, it shouldn't be >> installed in your system-wide python3 interpreter. > > Well, you can always create a venv using the system python3. > > Nowadays, to be honest, I use conda and Anaconda a lot, so I can > very quickly create a local Python 3.6 environment on my work machine, > even if my OS doesn't provide a Python 3.6. I'm just not sure everyone > has the same habits as me, though I'm all for more core developers > adopting conda :-) I already have a cpicker venv set up in vex to run cherry-picker [1], so I'll probably just add blurb to that (or create a new "cpydev" one with all the CPython workflow tools). So count me in as a +1 for standardising on a model where: - client-side core-workflow tools are free to use features from the latest released version of Python - we expect core devs to set up a venv or conda env to run those tools if their system Python is too old to let them just use "pip install --user" or "pipsi install" Cheers, Nick. [1] Based on a "make altinstall"ed 3.6 for now, although I'll be able to drop that once I upgrade to Fedora 26 -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tjreedy at udel.edu Sat Jun 24 21:56:47 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 24 Jun 2017 21:56:47 -0400 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: References: <20170624190631.58dac153@fsol> Message-ID: On 6/24/2017 7:48 PM, Brett Cannon wrote: > I should also mention that after Larry blows up Misc/NEWS into > individual files in about a week I will add a check to Bedevere for a > news file unless the PR is labeled as trivial. And what if there isn't? There are sometimes good reasons to have multiple patches on an issue, whereas an issue usually gets only one news item. -- Terry Jan Reedy From larry at hastings.org Sat Jun 24 22:10:13 2017 From: larry at hastings.org (Larry Hastings) Date: Sat, 24 Jun 2017 19:10:13 -0700 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: References: <20170624190631.58dac153@fsol> Message-ID: On 06/24/2017 06:56 PM, Terry Reedy wrote: > And what if there isn't? There are sometimes good reasons to have > multiple patches on an issue, whereas an issue usually gets only one > news item. There's nothing about blurb--or anything else in the workflow IIUC--that says you can't have multiple PRs / checkins for a single issue. Can you describe the problem you're having? //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Jun 24 23:04:28 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 25 Jun 2017 13:04:28 +1000 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: References: <20170624190631.58dac153@fsol> Message-ID: On 25 June 2017 at 11:56, Terry Reedy wrote: > On 6/24/2017 7:48 PM, Brett Cannon wrote: > >> I should also mention that after Larry blows up Misc/NEWS into individual >> files in about a week I will add a check to Bedevere for a news file unless >> the PR is labeled as trivial. > > > And what if there isn't? There are sometimes good reasons to have multiple > patches on an issue, whereas an issue usually gets only one news item. All the PRs share an issue number in that case, so Bedevere should be able to pick up that there's a pre-existing NEWS entry (even if it isn't being added by that particular PR). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From storchaka at gmail.com Sun Jun 25 00:14:08 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 25 Jun 2017 07:14:08 +0300 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: References: <20170624105343.6cca748b@fsol> Message-ID: 24.06.17 18:57, Larry Hastings ????: > On 06/24/2017 01:53 AM, Antoine Pitrou wrote: >> Would you like to make it 3.5-compatible? 3.6 is quite young and not >> all systems have it (e.g. Ubuntu 16.04, which many people use, has 3.5). > > Well, tbh I think that's a bit silly. First of all, it shouldn't be > installed in your system-wide python3 interpreter. Second, I assumed > core devs were compiling their own Python interpreters locally in their > account--in no small part so they can install any packages they want. Not only core developers make PRs for CPython. Since all non-trivial changes need to be mentioned in Misc/NEWS, blurb becomes a required tool for all committers. From storchaka at gmail.com Sun Jun 25 00:19:48 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 25 Jun 2017 07:19:48 +0300 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: References: <20170624105343.6cca748b@fsol> <20170624181036.5d180830@fsol> Message-ID: 25.06.17 04:51, Nick Coghlan ????: > So count me in as a +1 for standardising on a model where: > > - client-side core-workflow tools are free to use features from the > latest released version of Python > - we expect core devs to set up a venv or conda env to run those tools > if their system Python is too old to let them just use "pip install > --user" or "pipsi install" Don't forget that not only core developers but all committers need to use blurb. This makes committing to CPython more difficult for new committers. From storchaka at gmail.com Sun Jun 25 00:30:52 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 25 Jun 2017 07:30:52 +0300 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: References: <20170624190631.58dac153@fsol> Message-ID: 25.06.17 06:04, Nick Coghlan ????: > On 25 June 2017 at 11:56, Terry Reedy wrote: >> On 6/24/2017 7:48 PM, Brett Cannon wrote: >>> I should also mention that after Larry blows up Misc/NEWS into individual >>> files in about a week I will add a check to Bedevere for a news file unless >>> the PR is labeled as trivial. >> >> And what if there isn't? There are sometimes good reasons to have multiple >> patches on an issue, whereas an issue usually gets only one news item. > > All the PRs share an issue number in that case, so Bedevere should be > able to pick up that there's a pre-existing NEWS entry (even if it > isn't being added by that particular PR). What if the patch removes a NEWS entry? Or fixes a number in an existing entry? From larry at hastings.org Sun Jun 25 00:37:46 2017 From: larry at hastings.org (Larry Hastings) Date: Sat, 24 Jun 2017 21:37:46 -0700 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: References: <20170624105343.6cca748b@fsol> Message-ID: <02c890c4-48b4-af21-97f1-a3ade3cc6e6d@hastings.org> On 06/24/2017 09:14 PM, Serhiy Storchaka wrote: > Not only core developers make PRs for CPython. Since all non-trivial > changes need to be mentioned in Misc/NEWS, blurb becomes a required > tool for all committers. Well, no. *Writing blurb-compatible files* becomes a required step for all committers. And blurb makes that easy. But it's pretty easy to write them by hand; that's why we pre-created the "next" directories, and there are instructions in the dev guide. Also it should be easy for some JavaScript expert to write a static page that generates blurb files for you--it provides a form, you fill it out, and you "download" the blurb file. (I've seen pages that do that sort of thing, though I don't know how to write that kind of JavaScript myself.) //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sun Jun 25 02:16:33 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 25 Jun 2017 02:16:33 -0400 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: References: <20170624105343.6cca748b@fsol> <20170624181036.5d180830@fsol> Message-ID: On 6/25/2017 12:19 AM, Serhiy Storchaka wrote: > 25.06.17 04:51, Nick Coghlan ????: >> So count me in as a +1 for standardising on a model where: >> >> - client-side core-workflow tools are free to use features from the >> latest released version of Python >> - we expect core devs to set up a venv or conda env to run those tools >> if their system Python is too old to let them just use "pip install >> --user" or "pipsi install" > > Don't forget that not only core developers but all committers need to With rare exception, 'committer' == 'core developer'. Do you mean 'contributor'? > use blurb. This makes committing to CPython more difficult for new > committers. Brand-new contributors may leave the news item for someone else, but we will no longer have to ask contributors to skip it because of merge conflicts. -- Terry Jan Reedy From tjreedy at udel.edu Sun Jun 25 02:31:54 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 25 Jun 2017 02:31:54 -0400 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: References: <20170624190631.58dac153@fsol> Message-ID: On 6/25/2017 12:30 AM, Serhiy Storchaka wrote: > 25.06.17 06:04, Nick Coghlan ????: >> On 25 June 2017 at 11:56, Terry Reedy wrote: >>> On 6/24/2017 7:48 PM, Brett Cannon wrote: >>>> I should also mention that after Larry blows up Misc/NEWS into >>>> individual >>>> files in about a week I will add a check to Bedevere for a news file >>>> unless >>>> the PR is labeled as trivial. >>> >>> And what if there isn't? In particular, will Belvedere suggest adding a news entry or mandate one? >>> There are sometimes good reasons to have multiple >>> patches on an issue, whereas an issue usually gets only one news item. >> All the PRs share an issue number in that case, so Bedevere should be >> able to pick up that there's a pre-existing NEWS entry (even if it >> isn't being added by that particular PR). You seem to be suggesting adding a possibly incomplete entry with the first patch. If that is so, it would be helpful if blurb could pull an existing file for an issue into the editor. > What if the patch removes a NEWS entry? If the news entry is proper, it should only be removed as part of reverting the patch, but only if it is the last remaining PR for the issue. > Or fixes a number in an existing entry? I believe bpo numbers are part of the file *name*, not part of the news entry. (The consolidator should do the formatting.) So the issue is changing file names. Belvedere should flag mismatches between PR title and news file names. Changing the number in the title, which is sometimes necessary, will create a mismatch. -- Terry Jan Reedy From ncoghlan at gmail.com Sun Jun 25 02:44:55 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 25 Jun 2017 16:44:55 +1000 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: References: <20170624190631.58dac153@fsol> Message-ID: On 25 June 2017 at 16:31, Terry Reedy wrote: >> 25.06.17 06:04, Nick Coghlan ????: >>> All the PRs share an issue number in that case, so Bedevere should be >>> able to pick up that there's a pre-existing NEWS entry (even if it >>> isn't being added by that particular PR). > > You seem to be suggesting adding a possibly incomplete entry with the first > patch. If that is so, it would be helpful if blurb could pull an existing > file for an issue into the editor. I was mainly referring to cases where either an initial commit (with a complete NEWS entry) requires follow-up fixes (no NEWS edit necessary). However, another case did occur to me, which is fixing bugs reported against never-released features (which is what I'm currently working on for PEP 538). So trying to automate that particular check may end up proving to be more hassle than it's worth. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Sun Jun 25 04:33:20 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 25 Jun 2017 10:33:20 +0200 Subject: [Python-Dev] Helping contributors with chores (do we have to?) References: <20170624105343.6cca748b@fsol> <02c890c4-48b4-af21-97f1-a3ade3cc6e6d@hastings.org> Message-ID: <20170625103320.4ab6b244@fsol> On Sat, 24 Jun 2017 21:37:46 -0700 Larry Hastings wrote: > On 06/24/2017 09:14 PM, Serhiy Storchaka wrote: > > Not only core developers make PRs for CPython. Since all non-trivial > > changes need to be mentioned in Misc/NEWS, blurb becomes a required > > tool for all committers. > > Well, no. *Writing blurb-compatible files* becomes a required step for > all committers. And blurb makes that easy. But it's pretty easy to > write them by hand; that's why we pre-created the "next" directories, > and there are instructions in the dev guide. Hmm. If it were so easy, you wouldn't have felt the need to add that functionality to blurb, right? :-) This is touching a more general problem, though. Before GitHub, we (core developers) would take the patch submitted by a contributor, make whatever minor changes were needed (e.g. Misc/NEWS) and push the aggregate ourselves. With GitHub, while it's possible to edit someone else's PR, it's frankly a PITA (I've tried to do it once, I don't want to try a second time unless GitHub makes it massively easier and less footgunning-prone). So we have to rely on contributors to make the PR merge-ready by themselves... which means spending some time guiding them through the oh-so-exciting steps necessary to add the right Misc/NEWS entry, or fix the occasional bit of reStructuredText mis-syntax. Regards Antoine. From pmiscml at gmail.com Sun Jun 25 04:47:10 2017 From: pmiscml at gmail.com (Paul Sokolovsky) Date: Sun, 25 Jun 2017 11:47:10 +0300 Subject: [Python-Dev] Helping contributors with chores (do we have to?) In-Reply-To: <20170625103320.4ab6b244@fsol> References: <20170624105343.6cca748b@fsol> <02c890c4-48b4-af21-97f1-a3ade3cc6e6d@hastings.org> <20170625103320.4ab6b244@fsol> Message-ID: <20170625114710.170cee15@x230> Hello, On Sun, 25 Jun 2017 10:33:20 +0200 Antoine Pitrou wrote: [] > Hmm. If it were so easy, you wouldn't have felt the need to add that > functionality to blurb, right? :-) > > This is touching a more general problem, though. Before GitHub, we > (core developers) would take the patch submitted by a contributor, > make whatever minor changes were needed (e.g. Misc/NEWS) and push the > aggregate ourselves. With GitHub, while it's possible to edit someone > else's PR, it's frankly a PITA (I've tried to do it once, I don't want > to try a second time unless GitHub makes it massively easier and less > footgunning-prone). Sorry, but how that can be true? A GitHub PR is just a git branch (in somebody else's repository, but also in the repository it's submitted to). So, like any git branch, you can fetch it, re-branch to your local branch, apply any changes to it, rebase, push anywhere. There're also various tools for dealing specifically with git branch layout as used by Github, and every real man writes their own (because it's easier to shoot a 5-liner than to review whether somebody else's tool do what you need or not, it's all trivial git commands anyway). [] -- Best regards, Paul mailto:pmiscml at gmail.com From solipsis at pitrou.net Sun Jun 25 05:25:17 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 25 Jun 2017 11:25:17 +0200 Subject: [Python-Dev] Helping contributors with chores (do we have to?) In-Reply-To: <20170625114710.170cee15@x230> References: <20170624105343.6cca748b@fsol> <02c890c4-48b4-af21-97f1-a3ade3cc6e6d@hastings.org> <20170625103320.4ab6b244@fsol> <20170625114710.170cee15@x230> Message-ID: <20170625112517.1bbaf43c@fsol> On Sun, 25 Jun 2017 11:47:10 +0300 Paul Sokolovsky wrote: > > > > This is touching a more general problem, though. Before GitHub, we > > (core developers) would take the patch submitted by a contributor, > > make whatever minor changes were needed (e.g. Misc/NEWS) and push the > > aggregate ourselves. With GitHub, while it's possible to edit someone > > else's PR, it's frankly a PITA (I've tried to do it once, I don't want > > to try a second time unless GitHub makes it massively easier and less > > footgunning-prone). > > Sorry, but how that can be true? A GitHub PR is just a git branch [...] Well, git usage can still be a PITA, at least for a non-neligible proportion of its users. I remember trying to push some changes to someone else's PR, only to find that GitHub considered the PR had an empty diff to master. I'm not sure what produced it, but I have other things to do than deal with obnoxious tooling (be it git or GitHub) on my volunteer time. So, my current policy with PRs where pushing changes would be required is just to look elsewhere in the hope that another core developer comes and deals with it ;-) > There're also various tools for dealing specifically with git branch > layout as used by Github, and every real man writes their own (because > it's easier to shoot a 5-liner than to review whether somebody else's > tool do what you need or not, it's all trivial git commands anyway). I guess I'm not a "real man" who likes to "shoot 5-liners" made of "trivial git commands" on my free time, then. For some reason I'm not even interested in becoming one. The part of computing where people posture as "real men" (or "wizards") by sequencing arcane commands on ill-conceived UIs has always felt uninteresting and hostile to me. Regards Antoine. From storchaka at gmail.com Sun Jun 25 08:06:14 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 25 Jun 2017 15:06:14 +0300 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: References: <20170624105343.6cca748b@fsol> Message-ID: 24.06.17 18:57, Larry Hastings ????: > On 06/24/2017 01:53 AM, Antoine Pitrou wrote: >> Would you like to make it 3.5-compatible? 3.6 is quite young and not >> all systems have it (e.g. Ubuntu 16.04, which many people use, has 3.5). > > Well, tbh I think that's a bit silly. First of all, it shouldn't be > installed in your system-wide python3 interpreter. Second, I assumed > core devs were compiling their own Python interpreters locally in their > account--in no small part so they can install any packages they want. > And finally, you're asking me to give up f-strings. > > If someone else wants to send me a PR getting it to work on 3.5, I'd > look at it. I'm guessing it's really just f-strings but I can't say for > sure what else is a 3.6 dependency. https://github.com/python/core-workflow/pull/143 From tismer at stackless.com Sun Jun 25 08:06:31 2017 From: tismer at stackless.com (Christian Tismer) Date: Sun, 25 Jun 2017 14:06:31 +0200 Subject: [Python-Dev] __qualname__ format question Message-ID: <12e46b36-b18f-5712-350e-90c6f150de8a@stackless.com> Hi friends, by chance, I stumbled over meth_get__qualname__ in methodobject.c and calculate_qualname in descrobject.c . The first uses res = PyUnicode_FromFormat("%S.%s", type_qualname, m->m_ml->ml_name); and the latter uses res = PyUnicode_FromFormat("%S.%S", type_qualname, descr->d_name); To my knowledge, the "%S" character is undefined in C99 and C11. Q: Why this character, and why this difference? cheers - Chris -- Christian Tismer :^) tismer at stackless.com Software Consulting : http://www.stackless.com/ Karl-Liebknecht-Str. 121 : https://github.com/PySide 14482 Potsdam : GPG key -> 0xFB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 496 bytes Desc: OpenPGP digital signature URL: From storchaka at gmail.com Sun Jun 25 08:41:55 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 25 Jun 2017 15:41:55 +0300 Subject: [Python-Dev] __qualname__ format question In-Reply-To: <12e46b36-b18f-5712-350e-90c6f150de8a@stackless.com> References: <12e46b36-b18f-5712-350e-90c6f150de8a@stackless.com> Message-ID: 25.06.17 15:06, Christian Tismer ????: > by chance, I stumbled over > > meth_get__qualname__ > > in methodobject.c and > > calculate_qualname > > in descrobject.c . > > The first uses > > res = PyUnicode_FromFormat("%S.%s", type_qualname, m->m_ml->ml_name); > > and the latter uses > > res = PyUnicode_FromFormat("%S.%S", type_qualname, descr->d_name); > > To my knowledge, the "%S" character is undefined in C99 and C11. > > Q: Why this character, and why this difference? Se the documentation of PyUnicode_FromFormat(). https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_FromFormat From jwilk at jwilk.net Sun Jun 25 10:35:11 2017 From: jwilk at jwilk.net (Jakub Wilk) Date: Sun, 25 Jun 2017 16:35:11 +0200 Subject: [Python-Dev] Helping contributors with chores (do we have to?) In-Reply-To: <20170625114710.170cee15@x230> References: <20170624105343.6cca748b@fsol> <02c890c4-48b4-af21-97f1-a3ade3cc6e6d@hastings.org> <20170625103320.4ab6b244@fsol> <20170625114710.170cee15@x230> Message-ID: <20170625143511.jmf5axc6fzxuynry@jwilk.net> * Paul Sokolovsky , 2017-06-25, 11:47: >A GitHub PR is just a git branch (in somebody else's repository, but also in >the repository it's submitted to). So, like any git branch, you can fetch it, >re-branch to your local branch, apply any changes to it, rebase, push >anywhere. Right, this is documented here: https://help.github.com/articles/checking-out-pull-requests-locally/ >There're also various tools for dealing specifically with git branch layout as >used by Github, and every real man writes their own I have this in my gitconfig: [alias] hub-pr = ! "_g() { set -e -u; git fetch origin \"pull/$1/head:gh-$1\" && git checkout \"gh-$1\"; }; _g" If I want to checkout PR#42, I do: $ git hub-pr 42 -- Jakub Wilk From tjreedy at udel.edu Sun Jun 25 11:14:53 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 25 Jun 2017 11:14:53 -0400 Subject: [Python-Dev] PEP 538 warning at startup: please remove it In-Reply-To: <35dc10bd-8ee1-349c-e0bb-76caba99a2b2@g.nevcal.com> References: <35dc10bd-8ee1-349c-e0bb-76caba99a2b2@g.nevcal.com> Message-ID: On 6/12/2017 4:30 PM, Glenn Linderman wrote: > On 6/12/2017 1:11 PM, Terry Reedy wrote: >> I do not have any locale-related env vars. You should check whether >> the warning is off on all Win10 systems, as well as Win7 and Win8. >> Many Windows users know nothing about ENV VARS, and even if they do, >> they may not know how (I don't know the details) or MAY NOT BE ABLE to >> set one permanently. > > where (latest Win10): in the control panel, System and Security, System, > Advanced system settings (left edge menu item), On my system, accessing Advanced system settings requires an admin password. > Advanced Tab, Environment Variables (button). > > Two sets of variables. User should always be able to set User variables > that are in effect for all new processes launched after creation of the > variable. On my auto-updated Win 10 system, one may only set User variables for the admin account one gave the password for. > System variables may be locked down if not administrator. Are locked down. > Both types are permanent, stored in the registry. As I said, I seem to only be able to do so permanently with admin access. -- Terry Jan Reedy From tismer at stackless.com Sun Jun 25 11:25:44 2017 From: tismer at stackless.com (Christian Tismer) Date: Sun, 25 Jun 2017 17:25:44 +0200 Subject: [Python-Dev] __qualname__ format question In-Reply-To: References: <12e46b36-b18f-5712-350e-90c6f150de8a@stackless.com> Message-ID: On 25.06.17 14:41, Serhiy Storchaka wrote: > 25.06.17 15:06, Christian Tismer ????: >> by chance, I stumbled over >> >> meth_get__qualname__ >> >> in methodobject.c and >> >> calculate_qualname >> >> in descrobject.c . >> >> The first uses >> >> res = PyUnicode_FromFormat("%S.%s", type_qualname, >> m->m_ml->ml_name); >> >> and the latter uses >> >> res = PyUnicode_FromFormat("%S.%S", type_qualname, descr->d_name); >> >> To my knowledge, the "%S" character is undefined in C99 and C11. >> >> Q: Why this character, and why this difference? > > Se the documentation of PyUnicode_FromFormat(). > > https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_FromFormat > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/tismer%40stackless.com Ah, thank you very much. Cheers - Chris -- Christian Tismer :^) tismer at stackless.com Software Consulting : http://www.stackless.com/ Karl-Liebknecht-Str. 121 : https://github.com/PySide 14482 Potsdam : GPG key -> 0xFB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 496 bytes Desc: OpenPGP digital signature URL: From rosuav at gmail.com Sun Jun 25 11:27:20 2017 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 26 Jun 2017 01:27:20 +1000 Subject: [Python-Dev] Helping contributors with chores (do we have to?) In-Reply-To: <20170625112517.1bbaf43c@fsol> References: <20170624105343.6cca748b@fsol> <02c890c4-48b4-af21-97f1-a3ade3cc6e6d@hastings.org> <20170625103320.4ab6b244@fsol> <20170625114710.170cee15@x230> <20170625112517.1bbaf43c@fsol> Message-ID: On Sun, Jun 25, 2017 at 7:25 PM, Antoine Pitrou wrote: >> There're also various tools for dealing specifically with git branch >> layout as used by Github, and every real man writes their own (because >> it's easier to shoot a 5-liner than to review whether somebody else's >> tool do what you need or not, it's all trivial git commands anyway). > > I guess I'm not a "real man" who likes to "shoot 5-liners" made of > "trivial git commands" on my free time, then. For some reason I'm not > even interested in becoming one. The part of computing where people > posture as "real men" (or "wizards") by sequencing arcane commands on > ill-conceived UIs has always felt uninteresting and hostile to me. In the web programming bootcamp that I'm involved with, git is taught in the very first week. It's not some arcane and hostile thing; the command line is a fundamental tool that everyone is expected to become friends with. The students learn about branching and merging (including merge conflicts) and the pull-request workflow on the second day of bootcamp. Are we "real men" (and real women - we're not sexist here) because we know how to type commands into a terminal? If so, we're making sure the next generation of programmers is exclusively real men and women. ChrisA From solipsis at pitrou.net Sun Jun 25 11:52:49 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 25 Jun 2017 17:52:49 +0200 Subject: [Python-Dev] Helping contributors with chores (do we have to?) References: <20170624105343.6cca748b@fsol> <02c890c4-48b4-af21-97f1-a3ade3cc6e6d@hastings.org> <20170625103320.4ab6b244@fsol> <20170625114710.170cee15@x230> <20170625112517.1bbaf43c@fsol> Message-ID: <20170625175249.17a8f2a0@fsol> On Mon, 26 Jun 2017 01:27:20 +1000 Chris Angelico wrote: > On Sun, Jun 25, 2017 at 7:25 PM, Antoine Pitrou wrote: > >> There're also various tools for dealing specifically with git branch > >> layout as used by Github, and every real man writes their own (because > >> it's easier to shoot a 5-liner than to review whether somebody else's > >> tool do what you need or not, it's all trivial git commands anyway). > > > > I guess I'm not a "real man" who likes to "shoot 5-liners" made of > > "trivial git commands" on my free time, then. For some reason I'm not > > even interested in becoming one. The part of computing where people > > posture as "real men" (or "wizards") by sequencing arcane commands on > > ill-conceived UIs has always felt uninteresting and hostile to me. > > In the web programming bootcamp that I'm involved with, git is taught > in the very first week. It's not some arcane and hostile thing; the > command line is a fundamental tool that everyone is expected to become > friends with. The students learn about branching and merging > (including merge conflicts) and the pull-request workflow on the > second day of bootcamp. > > Are we "real men" (and real women - we're not sexist here) because we > know how to type commands into a terminal? The patronizing and not addressing the concrete issue at hand doesn't make your opinion about "real men and women" very interesting to me, sorry :-/ Regards Antoine. From rosuav at gmail.com Sun Jun 25 12:09:19 2017 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 26 Jun 2017 02:09:19 +1000 Subject: [Python-Dev] Helping contributors with chores (do we have to?) In-Reply-To: <20170625175249.17a8f2a0@fsol> References: <20170624105343.6cca748b@fsol> <02c890c4-48b4-af21-97f1-a3ade3cc6e6d@hastings.org> <20170625103320.4ab6b244@fsol> <20170625114710.170cee15@x230> <20170625112517.1bbaf43c@fsol> <20170625175249.17a8f2a0@fsol> Message-ID: On Mon, Jun 26, 2017 at 1:52 AM, Antoine Pitrou wrote: > On Mon, 26 Jun 2017 01:27:20 +1000 > Chris Angelico wrote: >> On Sun, Jun 25, 2017 at 7:25 PM, Antoine Pitrou wrote: >> >> There're also various tools for dealing specifically with git branch >> >> layout as used by Github, and every real man writes their own (because >> >> it's easier to shoot a 5-liner than to review whether somebody else's >> >> tool do what you need or not, it's all trivial git commands anyway). >> > >> > I guess I'm not a "real man" who likes to "shoot 5-liners" made of >> > "trivial git commands" on my free time, then. For some reason I'm not >> > even interested in becoming one. The part of computing where people >> > posture as "real men" (or "wizards") by sequencing arcane commands on >> > ill-conceived UIs has always felt uninteresting and hostile to me. >> >> In the web programming bootcamp that I'm involved with, git is taught >> in the very first week. It's not some arcane and hostile thing; the >> command line is a fundamental tool that everyone is expected to become >> friends with. The students learn about branching and merging >> (including merge conflicts) and the pull-request workflow on the >> second day of bootcamp. >> >> Are we "real men" (and real women - we're not sexist here) because we >> know how to type commands into a terminal? > > The patronizing and not addressing the concrete issue at hand doesn't > make your opinion about "real men and women" very interesting to me, > sorry :-/ My point is that EVERY developer needs to understand source control. Since git is the dominant system these days, that basically means everyone needs to understand git, although if you know (say) Mercurial instead, that's adequate for 99% of situations (the other 1% being where you're specifically collaborating on a git project). It's not about some kind of supermen who know git, and everyone else who doesn't. The command line is not an alternative to a GUI; it is augmented by one. It is not "arcane" and "hostile"; it is a basic fundamental of any sort of serious use of a computer. ChrisA From stefan at bytereef.org Sun Jun 25 12:13:12 2017 From: stefan at bytereef.org (Stefan Krah) Date: Sun, 25 Jun 2017 18:13:12 +0200 Subject: [Python-Dev] Helping contributors with chores (do we have to?) In-Reply-To: <20170625175249.17a8f2a0@fsol> References: <20170624105343.6cca748b@fsol> <02c890c4-48b4-af21-97f1-a3ade3cc6e6d@hastings.org> <20170625103320.4ab6b244@fsol> <20170625114710.170cee15@x230> <20170625112517.1bbaf43c@fsol> <20170625175249.17a8f2a0@fsol> Message-ID: <20170625161312.GA5451@bytereef.org> On Sun, Jun 25, 2017 at 05:52:49PM +0200, Antoine Pitrou wrote: > On Mon, 26 Jun 2017 01:27:20 +1000 > Chris Angelico wrote: > > In the web programming bootcamp that I'm involved with, git is taught > > in the very first week. It's not some arcane and hostile thing; the > > command line is a fundamental tool that everyone is expected to become > > friends with. The students learn about branching and merging > > (including merge conflicts) and the pull-request workflow on the > > second day of bootcamp. > > > > Are we "real men" (and real women - we're not sexist here) because we > > know how to type commands into a terminal? > > The patronizing and not addressing the concrete issue at hand doesn't > make your opinion about "real men and women" very interesting to me, > sorry :-/ Indeed, perhaps all core devs should take a course at this "web programming bootcamp" (whatever that is), so we finally know how to use the command line. ;) Linus should also attend the "bootcamp", so he can learn git and the command line: https://github.com/torvalds/linux/pull/17#issuecomment-5654674 Stefan Krah From tjreedy at udel.edu Sun Jun 25 12:16:10 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 25 Jun 2017 12:16:10 -0400 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: References: <20170624105343.6cca748b@fsol> Message-ID: On 6/25/2017 8:06 AM, Serhiy Storchaka wrote: > 24.06.17 18:57, Larry Hastings ????: >> On 06/24/2017 01:53 AM, Antoine Pitrou wrote: >>> Would you like to make it 3.5-compatible? 3.6 is quite young and not >>> all systems have it (e.g. Ubuntu 16.04, which many people use, has 3.5). >> >> Well, tbh I think that's a bit silly. First of all, it shouldn't be >> installed in your system-wide python3 interpreter. Second, I assumed >> core devs were compiling their own Python interpreters locally in >> their account--in no small part so they can install any packages they >> want. And finally, you're asking me to give up f-strings. >> >> If someone else wants to send me a PR getting it to work on 3.5, I'd >> look at it. I'm guessing it's really just f-strings but I can't say >> for sure what else is a 3.6 dependency. > > https://github.com/python/core-workflow/pull/143 The diff amounts to regressing all f-strings back to %-strings. If the patch is applied, an issue should be opened to revert it sometime in the future, such as when 3.7 is released. But I really think we should be able to use currently released python to write tools to develop future releases. My alternative proposal: modify the tooling so that one can install blurb, cherry_picker, and anything else into local cloned repositories without blocking proper use of git. As near as I can tell, we just need to .gitignore Scripts/ and Lib/site-packages/, which are empty in the master repository. Am I missing something? F:\dev\3x>python -m ensurepip Running Debug|Win32 interpreter... Collecting setuptools Collecting pip Installing collected packages: setuptools, pip Successfully installed pip-9.0.1 setuptools-28.8.0 F:\dev\3x>git status On branch pr_2377 Untracked files: (use "git add ..." to include in what will be committed) Lib/site-packages/easy_install.py Lib/site-packages/pip-9.0.1.dist-info/ Lib/site-packages/pip/ Lib/site-packages/pkg_resources/ Lib/site-packages/setuptools-28.8.0.dist-info/ Lib/site-packages/setuptools/ Scripts/ nothing added to commit but untracked files present (use "git add" to track) F:\dev\3x>python -m pip install blurb Running Debug|Win32 interpreter... Collecting blurb Downloading blurb-1.0.0.post1-py3-none-any.whl (56kB) 100% |????????????????????????????????| 61kB 596kB/s Installing collected packages: blurb Successfully installed blurb-1.0.0.post1 F:\dev\3x>python -m blurb Running Debug|Win32 interpreter... F:\dev\3x>git status ... new file: Misc/NEWS.d/next/IDLE/2017-06-25-12-00-03.bpo-12345.2Oj8ot.rst There it is, staged and ready to commit, with the message I entered ('test'). (So, by the way, the revised blurb installer works.) The same change should allow people to clone cpython and install and easily test other packages, such as numpy, with tip. -- Terry Jan Reedy From tjreedy at udel.edu Sun Jun 25 12:30:58 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 25 Jun 2017 12:30:58 -0400 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: References: <20170624105343.6cca748b@fsol> Message-ID: On 6/25/2017 12:16 PM, Terry Reedy wrote: > On 6/25/2017 8:06 AM, Serhiy Storchaka wrote: >> 24.06.17 18:57, Larry Hastings ????: >>> On 06/24/2017 01:53 AM, Antoine Pitrou wrote: >>>> Would you like to make it 3.5-compatible? 3.6 is quite young and not >>>> all systems have it (e.g. Ubuntu 16.04, which many people use, has >>>> 3.5). >>> >>> Well, tbh I think that's a bit silly. First of all, it shouldn't be >>> installed in your system-wide python3 interpreter. Second, I assumed >>> core devs were compiling their own Python interpreters locally in >>> their account--in no small part so they can install any packages they >>> want. And finally, you're asking me to give up f-strings. >>> >>> If someone else wants to send me a PR getting it to work on 3.5, I'd >>> look at it. I'm guessing it's really just f-strings but I can't say >>> for sure what else is a 3.6 dependency. >> >> https://github.com/python/core-workflow/pull/143 > > The diff amounts to regressing all f-strings back to %-strings. If the > patch is applied, an issue should be opened to revert it sometime in the > future, such as when 3.7 is released. But I really think we should be > able to use currently released python to write tools to develop future > releases. > > My alternative proposal: modify the tooling so that one can install > blurb, cherry_picker, and anything else into local cloned repositories > without blocking proper use of git. As near as I can tell, we just need > to .gitignore Scripts/ and Lib/site-packages/, which are empty in the > master repository. Am I missing something? The repository does not have an empty Scripts/ (and maybe this is Windows specific). It does have Lib/site-packages. > F:\dev\3x>python -m ensurepip > Running Debug|Win32 interpreter... > Collecting setuptools > Collecting pip > Installing collected packages: setuptools, pip > Successfully installed pip-9.0.1 setuptools-28.8.0 > > F:\dev\3x>git status > On branch pr_2377 > Untracked files: > (use "git add ..." to include in what will be committed) > > Lib/site-packages/easy_install.py > Lib/site-packages/pip-9.0.1.dist-info/ > Lib/site-packages/pip/ > Lib/site-packages/pkg_resources/ > Lib/site-packages/setuptools-28.8.0.dist-info/ > Lib/site-packages/setuptools/ > Scripts/ > > nothing added to commit but untracked files present (use "git add" to > track) > > F:\dev\3x>python -m pip install blurb > Running Debug|Win32 interpreter... > Collecting blurb > Downloading blurb-1.0.0.post1-py3-none-any.whl (56kB) > 100% |????????????????????????????????| 61kB 596kB/s > Installing collected packages: blurb > Successfully installed blurb-1.0.0.post1 Removing added files is different from reverting changes to existing files. To clean up, after running 'git clean -d -n', I ran F:\dev\3x>git clean -d -f Removing Lib/site-packages/blurb-1.0.0.post1.dist-info/ Removing Lib/site-packages/blurb.py Removing Lib/site-packages/easy_install.py Removing Lib/site-packages/pip-9.0.1.dist-info/ Removing Lib/site-packages/pip/ Removing Lib/site-packages/pkg_resources/ Removing Lib/site-packages/setuptools-28.8.0.dist-info/ Removing Lib/site-packages/setuptools/ Removing Misc/NEWS.d/next/IDLE/2017-06-25-12-00-03.bpo-12345.2Oj8ot.rst Removing Scripts/ -- Terry Jan Reedy From brett at python.org Sun Jun 25 12:39:27 2017 From: brett at python.org (Brett Cannon) Date: Sun, 25 Jun 2017 16:39:27 +0000 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: References: <20170624190631.58dac153@fsol> Message-ID: On Sat, Jun 24, 2017, 23:45 Nick Coghlan, wrote: > On 25 June 2017 at 16:31, Terry Reedy wrote: > >> 25.06.17 06:04, Nick Coghlan ????: > >>> All the PRs share an issue number in that case, so Bedevere should be > >>> able to pick up that there's a pre-existing NEWS entry (even if it > >>> isn't being added by that particular PR). > > > > You seem to be suggesting adding a possibly incomplete entry with the > first > > patch. If that is so, it would be helpful if blurb could pull an > existing > > file for an issue into the editor. > > I was mainly referring to cases where either an initial commit (with a > complete NEWS entry) requires follow-up fixes (no NEWS edit > necessary). > > However, another case did occur to me, which is fixing bugs reported > against never-released features (which is what I'm currently working > on for PEP 538). > https://github.com/python/bedevere/pull/22 is the current implementation of the check. All it does is see if the PR has a path that contains a news file. It can also easily be changed to also support some "no news" label if people want it to. > So trying to automate that particular check may end up proving to be > more hassle than it's worth. > The entire reason I want to have this check is I don't want or have to ask another PR submitter to include a draft of a news item. It's much easier to edit an existing news entry through the web UI than either ask for it and deal with the turnaround time to get the PR submitter to add it. -brett > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sun Jun 25 13:02:44 2017 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 25 Jun 2017 10:02:44 -0700 Subject: [Python-Dev] Helping contributors with chores (do we have to?) In-Reply-To: <20170625143511.jmf5axc6fzxuynry@jwilk.net> References: <20170624105343.6cca748b@fsol> <02c890c4-48b4-af21-97f1-a3ade3cc6e6d@hastings.org> <20170625103320.4ab6b244@fsol> <20170625114710.170cee15@x230> <20170625143511.jmf5axc6fzxuynry@jwilk.net> Message-ID: On Jun 25, 2017 08:12, "Jakub Wilk" wrote: * Paul Sokolovsky , 2017-06-25, 11:47: A GitHub PR is just a git branch (in somebody else's repository, but also > in the repository it's submitted to). So, like any git branch, you can > fetch it, re-branch to your local branch, apply any changes to it, rebase, > push anywhere. > Right, this is documented here: https://help.github.com/articles/checking-out-pull-requests-locally/ There're also various tools for dealing specifically with git branch layout > as used by Github, and every real man writes their own > I have this in my gitconfig: [alias] hub-pr = ! "_g() { set -e -u; git fetch origin \"pull/$1/head:gh-$1\" && git checkout \"gh-$1\"; }; _g" If I want to checkout PR#42, I do: $ git hub-pr 42 I believe you and Paul are missing the specific problem that Antoine was talking about, which is: how can we easily make changes *to someone else's PR*, i.e. these changes should show up in the diff view if you go to the PR's web page. This requires not just getting a copy of the PR branch locally, but also pushing it back to the original submitter's branch on GitHub. Allegedly this is possible in most cases (there's a permissions toggle the submitter can set, but it's set by default): https://help.github.com/articles/committing-changes-to-a-pull-request-branch-created-from-a-fork/ However, like Antoine, when I've tried to do this then all I've managed is to get obscure errors from GitHub. Did I have the wrong incantations? Was the permissions toggle set wrong? (I thought the web ui said it wasn't, but maybe I misunderstood.) It's a mystery. Has anyone figured out how to make *this* work reliably or ergonomically? Also, as a general comment, not directed at Jakub: the posturing about how easy git is reminds me of the posturing about how much better language X is than others described here: http://blog.aurynn.com/contempt-culture-2. My dudes, in a previous life I helped invent distributed VCS, but I still get confused by fiddly git BS just like everyone else. I know you probably aren't meaning to go around telling people that they're not Real Programmers because they get confused like me, but you are and it's not kind; please stop. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Sun Jun 25 13:13:18 2017 From: larry at hastings.org (Larry Hastings) Date: Sun, 25 Jun 2017 10:13:18 -0700 Subject: [Python-Dev] Helping contributors with chores (do we have to?) In-Reply-To: <20170625103320.4ab6b244@fsol> References: <20170624105343.6cca748b@fsol> <02c890c4-48b4-af21-97f1-a3ade3cc6e6d@hastings.org> <20170625103320.4ab6b244@fsol> Message-ID: <6bfb5e0f-84d6-e708-570a-2543e19892d0@hastings.org> On 06/25/2017 01:33 AM, Antoine Pitrou wrote: > On Sat, 24 Jun 2017 21:37:46 -0700 > Larry Hastings wrote: >> Well, no. *Writing blurb-compatible files* becomes a required step for >> all committers. And blurb makes that easy. But it's pretty easy to >> write them by hand; that's why we pre-created the "next" directories, >> and there are instructions in the dev guide. > Hmm. If it were so easy, you wouldn't have felt the need to add that > functionality to blurb, right? :-) No, I'm not contradicting myself. It's easy enough to do without blurb, but blurb makes it effortless. For core developers who are adding NEWS entries all the time, it's little enough burden to ask them to install an extra tool. There were enough constraints on the process that asking people to continually do it by hand seemed worse. For non-core developers who may have never contributed to CPython before, and are doing this for probably the one time in their lives, having instructions on a web page telling them how to name and format a special file is probably better than simply requiring them to install a special tool. Again, it's my hope that someone can make a static web page, to be checked in to the Dev Guide, that uses JS wizardry to create a downloadable file for you. Also it might work to hide a simple "blurb-add.py" in the CPython library so that users of 3.7+ don't need an external tool in order to simply add a NEWS item. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Sun Jun 25 13:18:23 2017 From: larry at hastings.org (Larry Hastings) Date: Sun, 25 Jun 2017 10:18:23 -0700 Subject: [Python-Dev] Helping contributors with chores (do we have to?) In-Reply-To: References: <20170624105343.6cca748b@fsol> <02c890c4-48b4-af21-97f1-a3ade3cc6e6d@hastings.org> <20170625103320.4ab6b244@fsol> <20170625114710.170cee15@x230> <20170625143511.jmf5axc6fzxuynry@jwilk.net> Message-ID: <456bcefc-bd1a-f1e4-6e82-989c1d2bc8cd@hastings.org> On 06/25/2017 10:02 AM, Nathaniel Smith wrote: > My dudes, in a previous life I helped invent distributed VCS, but I > still get confused by fiddly git BS just like everyone else. Really? I thought Bitkeeper was out before the monotone project even started--and TeamWare had monotone beat by most of a decade. Not to downplay your achievements, but I don't think you "helped invent DVCS". Or did you work on Bitkeeper / TeamWare? //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Jun 25 13:25:57 2017 From: donald at stufft.io (Donald Stufft) Date: Sun, 25 Jun 2017 13:25:57 -0400 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: References: <20170624190631.58dac153@fsol> Message-ID: <560FA727-F472-456B-BAEE-773A0E4B53D2@stufft.io> > On Jun 25, 2017, at 12:39 PM, Brett Cannon wrote: > > > > On Sat, Jun 24, 2017, 23:45 Nick Coghlan, > wrote: > On 25 June 2017 at 16:31, Terry Reedy > wrote: > >> 25.06.17 06:04, Nick Coghlan ????: > >>> All the PRs share an issue number in that case, so Bedevere should be > >>> able to pick up that there's a pre-existing NEWS entry (even if it > >>> isn't being added by that particular PR). > > > > You seem to be suggesting adding a possibly incomplete entry with the first > > patch. If that is so, it would be helpful if blurb could pull an existing > > file for an issue into the editor. > > I was mainly referring to cases where either an initial commit (with a > complete NEWS entry) requires follow-up fixes (no NEWS edit > necessary). > > However, another case did occur to me, which is fixing bugs reported > against never-released features (which is what I'm currently working > on for PEP 538). > > https://github.com/python/bedevere/pull/22 is the current implementation of the check. All it does is see if the PR has a path that contains a news file. It can also easily be changed to also support some "no news" label if people want it to. > > > So trying to automate that particular check may end up proving to be > more hassle than it's worth. > > The entire reason I want to have this check is I don't want or have to ask another PR submitter to include a draft of a news item. It's much easier to edit an existing news entry through the web UI than either ask for it and deal with the turnaround time to get the PR submitter to add it. > We?ve had a similar check on pip for awhile now and we?ve had no real complaints. The label to mark a change as not needing a news file is useful, we also let you add a empty news file marked as trivial for the PR authors to indicate that their change doesn?t deserve a news file entry. It has worked well so far and we?ve gotten better news file coverage then we previously had. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Jun 25 13:31:00 2017 From: donald at stufft.io (Donald Stufft) Date: Sun, 25 Jun 2017 13:31:00 -0400 Subject: [Python-Dev] Helping contributors with chores (do we have to?) In-Reply-To: References: <20170624105343.6cca748b@fsol> <02c890c4-48b4-af21-97f1-a3ade3cc6e6d@hastings.org> <20170625103320.4ab6b244@fsol> <20170625114710.170cee15@x230> <20170625143511.jmf5axc6fzxuynry@jwilk.net> Message-ID: <6ECB9B5F-35AC-40B6-94CA-352C8A8820BD@stufft.io> > On Jun 25, 2017, at 1:02 PM, Nathaniel Smith wrote: > > However, like Antoine, when I've tried to do this then all I've managed is to get obscure errors from GitHub. Did I have the wrong incantations? Was the permissions toggle set wrong? (I thought the web ui said it wasn't, but maybe I misunderstood.) It's a mystery. Has anyone figured out how to make *this* work reliably or ergonomically? I have used it. I don?t use it every day but I?ve never had it fail on me unless the contributor has unchecked the flag. I just ``git remote add `` then checkout their branch, add more commits, and push to their branch. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sun Jun 25 14:01:08 2017 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 25 Jun 2017 11:01:08 -0700 Subject: [Python-Dev] Helping contributors with chores (do we have to?) In-Reply-To: <456bcefc-bd1a-f1e4-6e82-989c1d2bc8cd@hastings.org> References: <20170624105343.6cca748b@fsol> <02c890c4-48b4-af21-97f1-a3ade3cc6e6d@hastings.org> <20170625103320.4ab6b244@fsol> <20170625114710.170cee15@x230> <20170625143511.jmf5axc6fzxuynry@jwilk.net> <456bcefc-bd1a-f1e4-6e82-989c1d2bc8cd@hastings.org> Message-ID: On Jun 25, 2017 10:27, "Larry Hastings" wrote: On 06/25/2017 10:02 AM, Nathaniel Smith wrote: My dudes, in a previous life I helped invent distributed VCS, but I still get confused by fiddly git BS just like everyone else. Really? I thought Bitkeeper was out before the monotone project even started--and TeamWare had monotone beat by most of a decade. Not to downplay your achievements, but I don't think you "helped invent DVCS". Or did you work on Bitkeeper / TeamWare? If your response to the previous message is to send an email to all of python-dev nitpicking credentials, then I think you missed the point... (There are many projects that attempted to somehow combine the terms "distributed" and "VCS", but nowadays the overwhelmingly dominant use of that term is to refer to system designs in the monotone/git/hg lineage. Perhaps I could have phrased it better; feel free to pretend I wrote a several paragraph history tracing which systems influenced which instead. I don't think any of this affects the actual point, which is that you can be arbitrarily familiar with how a software system works and still be lost and confused on a regular basis, so people should pay attention and try to avoid making comments that imply this indicates some personal deficit in the asker.) ((Maybe I should clarify that the reason I'm being cranky about this isn't really on my or Antoine's behalf, but on behalf of anyone lurking and thinking "oh, python-dev says I suck, I guess I'm not cut out to be a python contributor".)) (((Though also I *would* genuinely appreciate it if anyone could explain how to make GitHub do the thing I couldn't figure out. Probably it's something silly and obvious...))) -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sun Jun 25 14:23:04 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 25 Jun 2017 14:23:04 -0400 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: References: <20170624190631.58dac153@fsol> Message-ID: On 6/25/2017 12:39 PM, Brett Cannon wrote: > The entire reason I want to have this check is I don't want or have to > ask another PR submitter to include a draft of a news item. It's much > easier to edit an existing news entry through the web UI than either ask > for it and deal with the turnaround time to get the PR submitter to add it. I was not aware that PRs could be edited via a web UI and have no idea how. I search a PR for something, searched the devguide index for 'web', and searched 3 Lifecycle of a pull request. Did I miss something. Also, 6. Helping with Documentation should say something about submitting typo PRs via the web, if indeed that is possible yet. -- Terry Jan Reedy From stefan at bytereef.org Sun Jun 25 14:38:51 2017 From: stefan at bytereef.org (Stefan Krah) Date: Sun, 25 Jun 2017 20:38:51 +0200 Subject: [Python-Dev] Helping contributors with chores (do we have to?) In-Reply-To: References: <20170624105343.6cca748b@fsol> <02c890c4-48b4-af21-97f1-a3ade3cc6e6d@hastings.org> <20170625103320.4ab6b244@fsol> <20170625114710.170cee15@x230> <20170625143511.jmf5axc6fzxuynry@jwilk.net> Message-ID: <20170625183851.GA3954@bytereef.org> On Sun, Jun 25, 2017 at 10:02:44AM -0700, Nathaniel Smith wrote: > https://help.github.com/articles/committing-changes-to-a-pull-request-branch-created-from-a-fork/ > > However, like Antoine, when I've tried to do this then all I've managed is > to get obscure errors from GitHub. Did I have the wrong incantations? Was > the permissions toggle set wrong? (I thought the web ui said it wasn't, but > maybe I misunderstood.) It's a mystery. Has anyone figured out how to make > *this* work reliably or ergonomically? I don't know either. Adding .patch to the PR URL, downloading the thing, editing as necessary and crediting the author in the commit message would be much much faster. Stefan Krah From larry at hastings.org Sun Jun 25 14:50:31 2017 From: larry at hastings.org (Larry Hastings) Date: Sun, 25 Jun 2017 11:50:31 -0700 Subject: [Python-Dev] Helping contributors with chores (do we have to?) In-Reply-To: References: <20170624105343.6cca748b@fsol> <02c890c4-48b4-af21-97f1-a3ade3cc6e6d@hastings.org> <20170625103320.4ab6b244@fsol> <20170625114710.170cee15@x230> <20170625143511.jmf5axc6fzxuynry@jwilk.net> <456bcefc-bd1a-f1e4-6e82-989c1d2bc8cd@hastings.org> Message-ID: <528b5d55-deba-c8d9-7d20-e20a52141bf7@hastings.org> On 06/25/2017 11:01 AM, Nathaniel Smith wrote: > On Jun 25, 2017 10:27, "Larry Hastings" > wrote: > > > On 06/25/2017 10:02 AM, Nathaniel Smith wrote: >> My dudes, in a previous life I helped invent distributed VCS, but >> I still get confused by fiddly git BS just like everyone else. > > Really? I thought Bitkeeper was out before the monotone project > even started--and TeamWare had monotone beat by most of a decade. > Not to downplay your achievements, but I don't think you "helped > invent DVCS". Or did you work on Bitkeeper / TeamWare? > > > If your response to the previous message is to send an email to all of > python-dev nitpicking credentials, then I think you missed the point... Actually, I agreed with the point, I just wanted you to clarify about your claimed bona fides here. And I notice your reply was sent "to all of python-dev" and contained a lot of "nitpicking"; either this behavior is permissible or it isn't. DVCSes were a conceptual breakthrough, and the people behind it deserve all the praise and admiration we can heap on them. If you "helped invent" them--that's fantastic! Thank you! If you didn't, then it's misleading to claim that you did, so please don't. > Perhaps I could have phrased it better; feel free to pretend I wrote a > several paragraph history tracing which systems influenced which instead. I don't think that's necessary. "I was a core developer on a DVCS" would have handily made your point--you deeply understand DVCSes from both a conceptual and practical perspective--while also being brief and (IIUC) 100% accurate. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun Jun 25 14:51:04 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 25 Jun 2017 20:51:04 +0200 Subject: [Python-Dev] Helping contributors with chores (do we have to?) References: <20170624105343.6cca748b@fsol> <02c890c4-48b4-af21-97f1-a3ade3cc6e6d@hastings.org> <20170625103320.4ab6b244@fsol> <20170625114710.170cee15@x230> <20170625143511.jmf5axc6fzxuynry@jwilk.net> <20170625183851.GA3954@bytereef.org> Message-ID: <20170625205104.60e65992@fsol> On Sun, 25 Jun 2017 20:38:51 +0200 Stefan Krah wrote: > On Sun, Jun 25, 2017 at 10:02:44AM -0700, Nathaniel Smith wrote: > > https://help.github.com/articles/committing-changes-to-a-pull-request-branch-created-from-a-fork/ > > > > However, like Antoine, when I've tried to do this then all I've managed is > > to get obscure errors from GitHub. Did I have the wrong incantations? Was > > the permissions toggle set wrong? (I thought the web ui said it wasn't, but > > maybe I misunderstood.) It's a mystery. Has anyone figured out how to make > > *this* work reliably or ergonomically? > > I don't know either. Adding .patch to the PR URL, downloading the thing, > editing as necessary and crediting the author in the commit message would > be much much faster. I might do that as well (actually did on another project after I borked an attempt to push to the submitter's branch). Of course it doesn't work if your aim is to collaborate with the submitter, only if you want to take over and apply some polish before pushing. Regards Antoine. From brett at python.org Sun Jun 25 15:23:16 2017 From: brett at python.org (Brett Cannon) Date: Sun, 25 Jun 2017 19:23:16 +0000 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: References: <20170624190631.58dac153@fsol> Message-ID: On Sun, Jun 25, 2017, 11:24 Terry Reedy, wrote: > On 6/25/2017 12:39 PM, Brett Cannon wrote: > > > The entire reason I want to have this check is I don't want or have to > > ask another PR submitter to include a draft of a news item. It's much > > easier to edit an existing news entry through the web UI than either ask > > for it and deal with the turnaround time to get the PR submitter to add > it. > > I was not aware that PRs could be edited via a web UI and have no idea > how. I search a PR for something, searched the devguide index for > 'web', and searched 3 Lifecycle of a pull request. Did I miss > something. Also, 6. Helping with Documentation should say something > about submitting typo PRs via the web, if indeed that is possible yet. > You can edit anything via the web, it's just part of GitHub and so we have not documented it explicitly to avoid just duplicating GitHub's own docs. -brett > > -- > Terry Jan Reedy > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Jun 25 16:09:32 2017 From: guido at python.org (Guido van Rossum) Date: Sun, 25 Jun 2017 13:09:32 -0700 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: References: <20170624190631.58dac153@fsol> Message-ID: On Sun, Jun 25, 2017 at 12:23 PM, Brett Cannon wrote: > > > On Sun, Jun 25, 2017, 11:24 Terry Reedy, wrote: > >> On 6/25/2017 12:39 PM, Brett Cannon wrote: >> >> > The entire reason I want to have this check is I don't want or have to >> > ask another PR submitter to include a draft of a news item. It's much >> > easier to edit an existing news entry through the web UI than either ask >> > for it and deal with the turnaround time to get the PR submitter to add >> it. >> >> I was not aware that PRs could be edited via a web UI and have no idea >> how. I search a PR for something, searched the devguide index for >> 'web', and searched 3 Lifecycle of a pull request. Did I miss >> something. Also, 6. Helping with Documentation should say something >> about submitting typo PRs via the web, if indeed that is possible yet. >> > > You can edit anything via the web, it's just part of GitHub and so we have > not documented it explicitly to avoid just duplicating GitHub's own docs. > If you're still not sure how, from the PR UI go to the "Files changed" tab and click the "pencil" icon on the heading for the file you want to change. The PR creator must have enabled this (but I think it's on by default). When you save, GitHub creates a new commit in the PR's branch and all the usual tests (Travis-CI etc.) get kicked off. Where it breaks down is if you want to edit multiple files (it creates a new commit for each) or if the files are large (scrolling around is awkward) or if you want to make extensive changes (the web editor is limited in its capabilities, it's based on https://codemirror.net/). -- --Guido van Rossum (python.org/~guido ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Jun 25 17:39:36 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 25 Jun 2017 22:39:36 +0100 Subject: [Python-Dev] Helping contributors with chores (do we have to?) In-Reply-To: <6ECB9B5F-35AC-40B6-94CA-352C8A8820BD@stufft.io> References: <20170624105343.6cca748b@fsol> <02c890c4-48b4-af21-97f1-a3ade3cc6e6d@hastings.org> <20170625103320.4ab6b244@fsol> <20170625114710.170cee15@x230> <20170625143511.jmf5axc6fzxuynry@jwilk.net> <6ECB9B5F-35AC-40B6-94CA-352C8A8820BD@stufft.io> Message-ID: On 25 June 2017 at 18:31, Donald Stufft wrote: > > I have used it. I don?t use it every day but I?ve never had it fail on me > unless the contributor has unchecked the flag. I just ``git remote add > `` then checkout their branch, add more > commits, and push to their branch. That's relatively simple, but not immediately obvious (at least to me). There's a lot of concepts in here that are not exactly basic: 1. Being allowed to have multiple remotes in one repository 2. Naming of branches in non-default remotes, and how to translate the name in the remote to the local name you need to use 3. Pushing to non-default remotes There's also the point noted that by default, github doesn't permit this usage, and the contributor has to explicitly allow it - which probably means the core dev need to know how to do it, and how to explain that process to the contributor. And probably others. I'm not interested in debating what constitutes stuff that "everyone should know", or how "easy" or not git is. But for someone coming from a familiarity with Mercurial (which means many core devs) the learning curve is pretty steep (I'd consider that self-evident, because of the differences between the 2 systems). The decision to move to git/github has been made. It's not up for debate whether core devs need to learn to deal with it. But we do need to acknowledge that there's a significant learning curve, and while core devs are contributing from their free time, learning the new tooling is a major distraction from what they actually want to do, which is work on Python code. I don't have a good solution, but I do think we need to acknowledge the issue and address it without making judgements on what other people are familiar with or not. Paul PS There's a *lot* of stuff going by in the core-workflow list - the new blurb and cherry-picker tools, lots of workflow recommendations, etc. I haven't even looked at most of them yet, which means I'm accumulating an increasing load of work I can do before I get back to the point where I can contribute (and I'm familiar with git). Finding the time to pay off that debt is getting harder and harder... From phd at phdru.name Sun Jun 25 17:59:42 2017 From: phd at phdru.name (Oleg Broytman) Date: Sun, 25 Jun 2017 23:59:42 +0200 Subject: [Python-Dev] Helping contributors with chores (do we have to?) In-Reply-To: References: <20170624105343.6cca748b@fsol> <02c890c4-48b4-af21-97f1-a3ade3cc6e6d@hastings.org> <20170625103320.4ab6b244@fsol> <20170625114710.170cee15@x230> <20170625143511.jmf5axc6fzxuynry@jwilk.net> <6ECB9B5F-35AC-40B6-94CA-352C8A8820BD@stufft.io> Message-ID: <20170625215942.GA30108@phdru.name> On Sun, Jun 25, 2017 at 10:39:36PM +0100, Paul Moore wrote: > for someone coming from a familiarity with Mercurial (which means many > core devs) the learning curve is pretty steep (I'd consider that I switched to git after had been using hg for about 3 years. The first few months after the switch were rather painful. It took me a year to start understanding git and another year to master it. But I'm a slow thinker, I must admit. > self-evident, because of the differences between the 2 systems). Not sure about self-evidence. At the first glance they look similar. Both are DVCSes with DAG. The devil is in the details. In a lot of small quite different details. > Paul Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From timothy.c.delaney at gmail.com Sun Jun 25 18:20:20 2017 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Mon, 26 Jun 2017 08:20:20 +1000 Subject: [Python-Dev] Helping contributors with chores (do we have to?) In-Reply-To: References: <20170624105343.6cca748b@fsol> <02c890c4-48b4-af21-97f1-a3ade3cc6e6d@hastings.org> <20170625103320.4ab6b244@fsol> <20170625114710.170cee15@x230> <20170625143511.jmf5axc6fzxuynry@jwilk.net> <6ECB9B5F-35AC-40B6-94CA-352C8A8820BD@stufft.io> Message-ID: On 26 June 2017 at 07:39, Paul Moore wrote: > On 25 June 2017 at 18:31, Donald Stufft wrote: > > > > I have used it. I don?t use it every day but I?ve never had it fail on me > > unless the contributor has unchecked the flag. I just ``git remote add > > `` then checkout their branch, add more > > commits, and push to their branch. > > The decision to move to git/github has been made. It's not up for > debate whether core devs need to learn to deal with it. But we do need > to acknowledge that there's a significant learning curve, and while > core devs are contributing from their free time, learning the new > tooling is a major distraction from what they actually want to do, > which is work on Python code. > I went through this transition a few years ago when I changed employment (and didn't enjoy it in the slightest). Coming from Mercurial, Git feels primitive (I mean that literally - common functionality often requires low-level, arcane invocations). I can keep all of the Mercurial command-line that I use regularly in my head, whereas with Git I always have to go back to the manual even for things that I use all the time, and I'm often unsure if I'll get the result I expect. As a result, I've avoided using Git directly as much as possible. Instead, my personal recommendation for people who are experienced with Mercurial but not Git is to use Mercurial with the hggit plugin. It's not ideal, but since Mercurial functionality is almost a superset of Git functionality, it works so long as you don't use things that Git can't handle. The most important things to be aware of IMO are: 1. Avoid the use of named branches and instead use bookmarks (a workflow I personally hate, but it's the closest match to git branches, and I know I'm an anomaly in preferring named branches). 2. Last I checked hggit can't force-push to a git repository after history-modifying actions (e.g. rebase) so after such actions it's necessary to delete any existing branch in a local git repo, hg push to that then force-push to Github. This wnew branch head. So the workflow I use for working with Github is (after enabling hggit): git clone .git hg clone git+.git cd .hg ... cd .git git branch -d cd .hg hg push -B .git cd .git git push --force Although I use a Git GUI to avoid having to remember the git commands ... Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothy.c.delaney at gmail.com Sun Jun 25 18:22:34 2017 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Mon, 26 Jun 2017 08:22:34 +1000 Subject: [Python-Dev] Helping contributors with chores (do we have to?) In-Reply-To: References: <20170624105343.6cca748b@fsol> <02c890c4-48b4-af21-97f1-a3ade3cc6e6d@hastings.org> <20170625103320.4ab6b244@fsol> <20170625114710.170cee15@x230> <20170625143511.jmf5axc6fzxuynry@jwilk.net> <6ECB9B5F-35AC-40B6-94CA-352C8A8820BD@stufft.io> Message-ID: On 26 June 2017 at 08:20, Tim Delaney wrote: > > 2. Last I checked hggit can't force-push to a git repository after > history-modifying actions (e.g. rebase) so after such actions it's > necessary to delete any existing branch in a local git repo, hg push to > that then force-push to Github. This wnew branch head. > Not sure what happened there - that last line should have been: This will update any PR for that branch to the new branch head. Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From v+python at g.nevcal.com Sun Jun 25 19:47:16 2017 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sun, 25 Jun 2017 16:47:16 -0700 Subject: [Python-Dev] PEP 538 warning at startup: please remove it In-Reply-To: References: <35dc10bd-8ee1-349c-e0bb-76caba99a2b2@g.nevcal.com> Message-ID: <4a87fef3-153a-1721-db59-12ee04052720@g.nevcal.com> On 6/25/2017 8:14 AM, Terry Reedy wrote: > On 6/12/2017 4:30 PM, Glenn Linderman wrote: >> On 6/12/2017 1:11 PM, Terry Reedy wrote: >>> I do not have any locale-related env vars. You should check whether >>> the warning is off on all Win10 systems, as well as Win7 and Win8. >>> Many Windows users know nothing about ENV VARS, and even if they do, >>> they may not know how (I don't know the details) or MAY NOT BE ABLE >>> to set one permanently. >> >> where (latest Win10): in the control panel, System and Security, >> System, Advanced system settings (left edge menu item), > > On my system, accessing Advanced system settings requires an admin > password. Interesting and surprising. I guess you've concluded that my testing/reporting was from an admin account. Not only did I not get a request for an admin password, I did not get a confirmation dialog that I should proceed with admin access (like happens with installing programs, and running .reg files). > >> Advanced Tab, Environment Variables (button). >> >> Two sets of variables. User should always be able to set User >> variables that are in effect for all new processes launched after >> creation of the variable. > > On my auto-updated Win 10 system, one may only set User variables for > the admin account one gave the password for. Interesting and surprising again. User variables for the admin account, or user variables for the user account, once you achieve admin access. Perhaps the only way to discover the difference would be to create a new, junk variable when you get the dialog, and then reboot, and login as that user, and see if the variable persisted in user mode (can be seen by starting a CMD prompt, and doing set command). > >> System variables may be locked down if not administrator. > > Are locked down. > >> Both types are permanent, stored in the registry. > > As I said, I seem to only be able to do so permanently with admin access. > This seems extremely limiting. User variables should be for the user, not the admin, and should be able to be set/saved by the user without having admin access. I wonder in what version of Windows this limitation was introduced... it is ridiculous, especially when people are preaching that you shouldn't need to run as admin all the time, for better security in Windows. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Jun 25 21:16:00 2017 From: donald at stufft.io (Donald Stufft) Date: Sun, 25 Jun 2017 21:16:00 -0400 Subject: [Python-Dev] Helping contributors with chores (do we have to?) In-Reply-To: References: <20170624105343.6cca748b@fsol> <02c890c4-48b4-af21-97f1-a3ade3cc6e6d@hastings.org> <20170625103320.4ab6b244@fsol> <20170625114710.170cee15@x230> <20170625143511.jmf5axc6fzxuynry@jwilk.net> <6ECB9B5F-35AC-40B6-94CA-352C8A8820BD@stufft.io> Message-ID: <54ADB602-0547-4626-818D-9EDC93BDD6F9@stufft.io> > On Jun 25, 2017, at 5:39 PM, Paul Moore wrote: > > On 25 June 2017 at 18:31, Donald Stufft wrote: >> >> I have used it. I don?t use it every day but I?ve never had it fail on me >> unless the contributor has unchecked the flag. I just ``git remote add >> `` then checkout their branch, add more >> commits, and push to their branch. > > That's relatively simple, but not immediately obvious (at least to me). I?m completely willing to agree that because git was the first VCS I used seriously (I tried Mercural out early on, but switched quickly before I got too deep in it b/c of Github) that my brain has successfully been broken in a git shaped way ;) > > There's a lot of concepts in here that are not exactly basic: > > 1. Being allowed to have multiple remotes in one repository > 2. Naming of branches in non-default remotes, and how to translate the > name in the remote to the local name you need to use > 3. Pushing to non-default remotes > > There's also the point noted that by default, github doesn't permit > this usage, and the contributor has to explicitly allow it - which > probably means the core dev need to know how to do it, and how to > explain that process to the contributor. Just a point of clarification, as far as I am aware Github defaults that checkmark to on, and PR authors have to explicitly turn it off to disable it. Although I think older PRs were all set to act as if the author did not grant that permission. > > And probably others. I'm not interested in debating what constitutes > stuff that "everyone should know", or how "easy" or not git is. But > for someone coming from a familiarity with Mercurial (which means many > core devs) the learning curve is pretty steep (I'd consider that > self-evident, because of the differences between the 2 systems). Not sure if this was aimed at me or not, but I don?t think that everyone should know that off the bat! I was just giving the steps I use to use it, hopefully in a useful way for other people. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Jun 25 23:56:16 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 26 Jun 2017 13:56:16 +1000 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: References: <20170624190631.58dac153@fsol> Message-ID: On 26 June 2017 at 02:39, Brett Cannon wrote: > On Sat, Jun 24, 2017, 23:45 Nick Coghlan, wrote: >> I was mainly referring to cases where either an initial commit (with a >> complete NEWS entry) requires follow-up fixes (no NEWS edit >> necessary). >> >> However, another case did occur to me, which is fixing bugs reported >> against never-released features (which is what I'm currently working >> on for PEP 538). > > https://github.com/python/bedevere/pull/22 is the current implementation of > the check. All it does is see if the PR has a path that contains a news > file. It can also easily be changed to also support some "no news" label if > people want it to. Yeah, I think I'd prefer to see the current "trivial" label split into the more explicit "no issue needed" and "no NEWS needed". It isn't necessarily the case that the changes in such PRs are trivial, it's that the usual checks don't apply for some reason (and the change being genuinely trivial is only one possibility) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Jun 26 00:08:12 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 26 Jun 2017 14:08:12 +1000 Subject: [Python-Dev] PEP 538 warning at startup: please remove it In-Reply-To: References: Message-ID: On 12 June 2017 at 22:24, Nick Coghlan wrote: > On 12 June 2017 at 18:56, Victor Stinner wrote: >> Hi, >> >> Nick Coghlan pushed his implementation of his PEP 538: nice! Nice step >> forward to UTF-8 everywhere ;-) >> >> I would prefer to not be annoyed by warning messages about encodings >> at startup if possible: >> >> "Python detected LC_CTYPE=C: LC_CTYPE coerced to C.UTF-8 (set another >> locale or PYTHONCOERCECLOCALE=0 to disable this locale coercion >> behavior)." > > Note that there's an open issue for this linked from the What's New entry: > > * https://docs.python.org/dev/whatsnew/3.7.html#pep-538-legacy-c-locale-coercion > * https://bugs.python.org/issue30565 > > I suspect the eventual outcome is going to be dropping that particular > warning (since it's been problematic for Fedora's 3.6 backport as > well, and the problems are due to the warning itself, *not* the locale > coercion), but I'd prefer to keep the notification at least for a > while (potentially even until alpha 1). > > OTOH, I'm also open to being persuaded otherwise if enough folks are > running into problems with it just while working on CPython (I'd still > like to turn it back on for alpha 1 even if we turn off in the > meantime, though). Following up here for the sake of folks following this thread, but not all the related issues: with the locale warnings enabled by default, we(/I) couldn't even get CPython's own test suite to pass reliably across the full fleet of stable buildbots. As a result, https://bugs.python.org/issue30565 was implemented in https://github.com/python/cpython/pull/2260, switching us to an opt-in warning model where you have to set `PYTHONCOERCECLOCALE=warn` to get the notifications on stderr both when locale coercion triggers and when the interpreter runtime itself is initialized in a locale that would have triggered coercion in the CLI. That then becomes a runtime debugging switch (akin to `-X faulthandler` or `-X tracemalloc`) for folks investigating surprising locale-related behaviour in 3.7+, rather than something we do by default. The original PEP has also been updated with a note indicating that the originally accepted approach was found to be impractical: https://www.python.org/dev/peps/pep-0538/#implementation-notes Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Jun 26 00:21:39 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 26 Jun 2017 14:21:39 +1000 Subject: [Python-Dev] Helping contributors with chores (do we have to?) In-Reply-To: References: <20170624105343.6cca748b@fsol> <02c890c4-48b4-af21-97f1-a3ade3cc6e6d@hastings.org> <20170625103320.4ab6b244@fsol> <20170625114710.170cee15@x230> <20170625143511.jmf5axc6fzxuynry@jwilk.net> <6ECB9B5F-35AC-40B6-94CA-352C8A8820BD@stufft.io> Message-ID: On 26 June 2017 at 07:39, Paul Moore wrote: > On 25 June 2017 at 18:31, Donald Stufft wrote: >> >> I have used it. I don?t use it every day but I?ve never had it fail on me >> unless the contributor has unchecked the flag. I just ``git remote add >> `` then checkout their branch, add more >> commits, and push to their branch. > > That's relatively simple, but not immediately obvious (at least to me). This seems like a good point to remind folks that we have a "Git Bootcamp and Cheat Sheet" as part of the developer guide: https://docs.python.org/devguide/gitbootcamp.html PR editing is one of the topics covered: https://docs.python.org/devguide/gitbootcamp.html#editing-a-pull-request-prior-to-merging However, many of us editing it tend to be on Linux or Mac OS X (and are often experienced git users as well), so issues and PRs clarifying steps that are currently omitted, platform-specific, or otherwise don't work as advertised are definitely appreciated. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From victor.stinner at gmail.com Mon Jun 26 12:36:38 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 26 Jun 2017 18:36:38 +0200 Subject: [Python-Dev] New work-in-progress bisection tool for the Python test suite (in regrtest) In-Reply-To: References: Message-ID: Hi, I updated my bisect_test.py script to simplify its command line: https://github.com/haypo/misc/blob/master/python/bisect_test.py To debug a reference leak in test_os, now just type: $ ./python bisect_test.py -R 3:3 test_os Example of output: --- (...) Failing tests (1): * test.test_os.ExecTests.test_execve_invalid_env Bisection completed in 24 iterations and 0:02:11 --- In 2 minutes, you get the guilty: test_execve_invalid_env()! It's now fully automated! I also added --fail-env-changed option to regrtest. Using this option, you can find which test leaked a file, and modified any other resource tracked by regrtest. For examle, I used this command to find which test method of test_subprocess created a core dump file: $ ./python bisect_test.py --fail-env-changed test_subprocess Later, I will try to integrate this tiny tool in Python. Maybe somewhere in Tools/. Or better, maybe directly in regrtest? Victor From brett at python.org Mon Jun 26 15:57:26 2017 From: brett at python.org (Brett Cannon) Date: Mon, 26 Jun 2017 19:57:26 +0000 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: References: <20170624190631.58dac153@fsol> Message-ID: On Sun, 25 Jun 2017 at 20:56 Nick Coghlan wrote: > On 26 June 2017 at 02:39, Brett Cannon wrote: > > On Sat, Jun 24, 2017, 23:45 Nick Coghlan, wrote: > >> I was mainly referring to cases where either an initial commit (with a > >> complete NEWS entry) requires follow-up fixes (no NEWS edit > >> necessary). > >> > >> However, another case did occur to me, which is fixing bugs reported > >> against never-released features (which is what I'm currently working > >> on for PEP 538). > > > > https://github.com/python/bedevere/pull/22 is the current > implementation of > > the check. All it does is see if the PR has a path that contains a news > > file. It can also easily be changed to also support some "no news" label > if > > people want it to. > > Yeah, I think I'd prefer to see the current "trivial" label split into > the more explicit "no issue needed" and "no NEWS needed". It isn't > necessarily the case that the changes in such PRs are trivial, it's > that the usual checks don't apply for some reason (and the change > being genuinely trivial is only one possibility) > I've opened https://github.com/python/bedevere/issues/25 to change this for the issue number check (although I will name the label "skipped issue" so all check-skipping labels and start with the same prefix for sorting purpose). -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon Jun 26 15:58:48 2017 From: brett at python.org (Brett Cannon) Date: Mon, 26 Jun 2017 19:58:48 +0000 Subject: [Python-Dev] New work-in-progress bisection tool for the Python test suite (in regrtest) In-Reply-To: References: Message-ID: On Mon, 26 Jun 2017 at 09:38 Victor Stinner wrote: > Hi, > > I updated my bisect_test.py script to simplify its command line: > > https://github.com/haypo/misc/blob/master/python/bisect_test.py > > To debug a reference leak in test_os, now just type: > > $ ./python bisect_test.py -R 3:3 test_os > > Example of output: > --- > (...) > Failing tests (1): > * test.test_os.ExecTests.test_execve_invalid_env > > Bisection completed in 24 iterations and 0:02:11 > --- > > In 2 minutes, you get the guilty: test_execve_invalid_env()! It's now > fully automated! > > > I also added --fail-env-changed option to regrtest. Using this option, > you can find which test leaked a file, and modified any other resource > tracked by regrtest. For examle, I used this command to find which > test method of test_subprocess created a core dump file: > > $ ./python bisect_test.py --fail-env-changed test_subprocess > > > Later, I will try to integrate this tiny tool in Python. Maybe > somewhere in Tools/. Or better, maybe directly in regrtest? > I don't see why regrtest isn't the right place for this. -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Mon Jun 26 16:37:43 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 26 Jun 2017 22:37:43 +0200 Subject: [Python-Dev] New work-in-progress bisection tool for the Python test suite (in regrtest) In-Reply-To: References: Message-ID: 2017-06-26 21:58 GMT+02:00 Brett Cannon : > I don't see why regrtest isn't the right place for this. The current regrest CLI isn't designed for subcommands, and I don't want to "pollute" regrtest with multiple options for bisect. Currently, my script has already 4 options: haypo at selma$ python3 ~/prog/GIT/misc/python/bisect_test.py --help usage: bisect_test.py [-h] [-i INPUT] [-o OUTPUT] [-n MAX_TESTS] [-N MAX_ITER] optional arguments: -h, --help show this help message and exit -i INPUT, --input INPUT Test names produced by --list-tests written into a file. If not set, run --list-tests -o OUTPUT, --output OUTPUT Result of the bisection -n MAX_TESTS, --max-tests MAX_TESTS Maximum number of tests to stop the bisection (default: 1) -N MAX_ITER, --max-iter MAX_ITER Maximum number of bisection iterations (default: 100) I really like subcommands, it's a nice way to design complex CLI ;-) Victor From tjreedy at udel.edu Mon Jun 26 17:25:09 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 26 Jun 2017 17:25:09 -0400 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: References: <20170624190631.58dac153@fsol> Message-ID: On 6/25/2017 4:09 PM, Guido van Rossum wrote: > On Sun, Jun 25, 2017 at 12:23 PM, Brett Cannon > wrote: > > > > On Sun, Jun 25, 2017, 11:24 Terry Reedy, > wrote: > > I was not aware that PRs could be edited via a web UI and have > no idea > how. I search a PR for something, searched the devguide index for > 'web', and searched 3 Lifecycle of a pull request. Did I miss > something. Also, 6. Helping with Documentation should say something > about submitting typo PRs via the web, if indeed that is > possible yet. > > > You can edit anything via the web, it's just part of GitHub and so > we have not documented it explicitly to avoid just duplicating > GitHub's own docs. There is a different between 'duplicating' the docs and providing short pointers. > If you're still not sure how, from the PR UI go to the "Files changed" > tab and click the "pencil" icon on the heading for the file you want to > change. The PR creator must have enabled this (but I think it's on by > default). When you save, GitHub creates a new commit in the PR's branch > and all the usual tests (Travis-CI etc.) get kicked off. Where it breaks > down is if you want to edit multiple files (it creates a new commit for > each) or if the files are large (scrolling around is awkward) or if you > want to make extensive changes (the web editor is limited in its > capabilities, it's based on https://codemirror.net/). Thank you. This will make reviewing more fun. This should speed up editing doc strings and comments and making other small changes (in a single file) while reviewing. I just used this to edit a news item I pushed previously. It seems strange that such an edit triggers a travis download of 3.6 to run the cherry_picker test, but that is another issue. Maybe I will use the web editor to add a a couple of lines to the devguide about using the web editor ;-). -- Terry Jan Reedy From brett at python.org Mon Jun 26 17:34:29 2017 From: brett at python.org (Brett Cannon) Date: Mon, 26 Jun 2017 21:34:29 +0000 Subject: [Python-Dev] New workflow change: Welcome to blurb In-Reply-To: References: <20170624190631.58dac153@fsol> Message-ID: On Mon, 26 Jun 2017 at 14:28 Terry Reedy wrote: > On 6/25/2017 4:09 PM, Guido van Rossum wrote: > > On Sun, Jun 25, 2017 at 12:23 PM, Brett Cannon > > wrote: > > > > > > > > On Sun, Jun 25, 2017, 11:24 Terry Reedy, > > wrote: > > > > > I was not aware that PRs could be edited via a web UI and have > > no idea > > how. I search a PR for something, searched the devguide index > for > > 'web', and searched 3 Lifecycle of a pull request. Did I miss > > something. Also, 6. Helping with Documentation should say > something > > about submitting typo PRs via the web, if indeed that is > > possible yet. > > > > > > You can edit anything via the web, it's just part of GitHub and so > > we have not documented it explicitly to avoid just duplicating > > GitHub's own docs. > > There is a different between 'duplicating' the docs and providing short > pointers. > Yes, but I've been using GitHub for years so I don't know what is obvious and isn't anymore. And there is also a balance when documenting things that we don't control and thus can go stale in the devguide. This is the sort of thing we could point out in a "pointers on GitHub" page once someone has time to re-organize the devguide around use-cases (e.g. external contributor, core dev who doesn't use GitHub day-to-day, and How we differ from other GitHub projects). > > > If you're still not sure how, from the PR UI go to the "Files changed" > > tab and click the "pencil" icon on the heading for the file you want to > > change. The PR creator must have enabled this (but I think it's on by > > default). When you save, GitHub creates a new commit in the PR's branch > > and all the usual tests (Travis-CI etc.) get kicked off. Where it breaks > > down is if you want to edit multiple files (it creates a new commit for > > each) or if the files are large (scrolling around is awkward) or if you > > want to make extensive changes (the web editor is limited in its > > capabilities, it's based on https://codemirror.net/). > > Thank you. This will make reviewing more fun. This should speed up > editing doc strings and comments and making other small changes (in a > single file) while reviewing. I just used this to edit a news item I > pushed previously. > > It seems strange that such an edit triggers a travis download of 3.6 to > run the cherry_picker test, but that is another issue. > Why is it strange? You changed something in the repository and Travis is there to test that repository (and Travis doesn't know what you do and do not want tested). And Python 3.6 doesn't come in the version of Ubuntu they use so they have to get it somehow (and by downloading they can do stuff like run release candidates or let people run against development versions of Python). > > Maybe I will use the web editor to add a a couple of lines to the > devguide about using the web editor ;-). > :) Seems appropriate if there's a place to put that ATM. -Brett > > -- > Terry Jan Reedy > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon Jun 26 17:49:08 2017 From: brett at python.org (Brett Cannon) Date: Mon, 26 Jun 2017 21:49:08 +0000 Subject: [Python-Dev] Helping contributors with chores (do we have to?) In-Reply-To: <20170625103320.4ab6b244@fsol> References: <20170624105343.6cca748b@fsol> <02c890c4-48b4-af21-97f1-a3ade3cc6e6d@hastings.org> <20170625103320.4ab6b244@fsol> Message-ID: [It seems to me most of this thread has gone off-topic since it now includes what I consider a sexist comment, complaining about git, and discussing people's credentials in terms of being able to comment on something, I'm basically ignoring everything that came after this email as none of that has anything to do with what Antoine is complaining about.] On Sun, 25 Jun 2017 at 01:34 Antoine Pitrou wrote: > On Sat, 24 Jun 2017 21:37:46 -0700 > Larry Hastings wrote: > > On 06/24/2017 09:14 PM, Serhiy Storchaka wrote: > > > Not only core developers make PRs for CPython. Since all non-trivial > > > changes need to be mentioned in Misc/NEWS, blurb becomes a required > > > tool for all committers. > > > > Well, no. *Writing blurb-compatible files* becomes a required step for > > all committers. And blurb makes that easy. But it's pretty easy to > > write them by hand; that's why we pre-created the "next" directories, > > and there are instructions in the dev guide. > > Hmm. If it were so easy, you wouldn't have felt the need to add that > functionality to blurb, right? :-) > Actually a good amount of blurb is to handle blowing up a Misc/NEWS file into individual news files and then re-combining them back into a Misc/NEWS file for release. The format also simplified over time while we refined the design so that the file name contains the most common metadata, thus not really needing to check the formatting as much. > > This is touching a more general problem, though. Before GitHub, we > (core developers) would take the patch submitted by a contributor, make > whatever minor changes were needed (e.g. Misc/NEWS) and push the > aggregate ourselves. With GitHub, while it's possible to edit someone > else's PR, it's frankly a PITA (I've tried to do it once, I don't want > to try a second time unless GitHub makes it massively easier and less > footgunning-prone). So there's editing a PR by adding/removing files and then there's editing files that have already been touched by the PR. In the latter case you can just edit the files through a browser while reviewing them no problem (I do this regularly rather than ask a PR submitter to fix e.g. a spelling mistake, just click the pencil icon at the top-right corner of the diff for the file). In the former case where you want to add/delete a file is where it's less obvious on how to do it. Donald has said how to pull down a pull request as a branch in your clone, editing, and then pushing again. But it turns out you can *also* do it entirely through the web UI. For instance, if you look at your https://github.com/pitrou/cpython/tree/signal_pyatomic branch you had for your https://github.com/python/cpython/pull/2417 PR (sorry about that, but I needed someone's branch to test with that wasn't my own), you will see I created and deleted a news file, and I did that entirely through my browser. The biggest pain, though, is navigating back to someone's branch as I don't think there's a direct link to be found in a PR back to the repo:branch it's based on. IOW everything you used to do by hand can still be done by hand, just through a web browser ... on a tablet at the beach with WiFi (which was one of my goals of this whole transition to begin with :) . > So we have to rely on contributors to make the PR > merge-ready by themselves... which means spending some time guiding > them through the oh-so-exciting steps necessary to add the right > Misc/NEWS entry, My hope is that having a news entry status check through Bedevere with help with that. I'm also open to ideas on how to make it more obvious to first-time contributors that things need to be done, e.g. maybe a default PR message that is nothing more than "" (making it an HTML comment means it won't' ever show up in the rendered output of the PR, plus that comment is short and thus easy to ignore)? We could also make the CONTRIBUTING.md file have a bullet-point list of the key things we expect people to do that is different from most projects. (The only other thing I can think of is a comment for first-time contributors pointing all of this out, but I'm leery of that as all of our bots are stateless and this would make at least Bedevere stateful.) And if you want to make it easier to add/remove files in a PR, we could look at adding a link to the branch the PR comes from so that it's easy to get to to do things like add a news file through the browser. If any of this sounds reasonable we can discuss it over on core-workflow. > or fix the occasional bit of reStructuredText > mis-syntax. > > You can make those sorts of changes in the browser yourself. -Brett > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon Jun 26 17:51:38 2017 From: brett at python.org (Brett Cannon) Date: Mon, 26 Jun 2017 21:51:38 +0000 Subject: [Python-Dev] New work-in-progress bisection tool for the Python test suite (in regrtest) In-Reply-To: References: Message-ID: On Mon, 26 Jun 2017 at 13:38 Victor Stinner wrote: > 2017-06-26 21:58 GMT+02:00 Brett Cannon : > > I don't see why regrtest isn't the right place for this. > > The current regrest CLI isn't designed for subcommands, and I don't > want to "pollute" regrtest with multiple options for bisect. > Currently, my script has already 4 options: > > haypo at selma$ python3 ~/prog/GIT/misc/python/bisect_test.py --help > usage: bisect_test.py [-h] [-i INPUT] [-o OUTPUT] [-n MAX_TESTS] [-N > MAX_ITER] > > optional arguments: > -h, --help show this help message and exit > -i INPUT, --input INPUT > Test names produced by --list-tests written into a > file. If not set, run --list-tests > -o OUTPUT, --output OUTPUT > Result of the bisection > -n MAX_TESTS, --max-tests MAX_TESTS > Maximum number of tests to stop the bisection > (default: 1) > -N MAX_ITER, --max-iter MAX_ITER > Maximum number of bisection iterations (default: > 100) > > I really like subcommands, it's a nice way to design complex CLI ;-) > Then I think you answered your own question about whether this should go into regrtest ;) (Unless you want to rewrite regrtest's CLI to support subcommands.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Tue Jun 27 01:33:34 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 27 Jun 2017 08:33:34 +0300 Subject: [Python-Dev] New work-in-progress bisection tool for the Python test suite (in regrtest) In-Reply-To: References: Message-ID: 26.06.17 23:37, Victor Stinner ????: > 2017-06-26 21:58 GMT+02:00 Brett Cannon : >> I don't see why regrtest isn't the right place for this. > > The current regrest CLI isn't designed for subcommands, and I don't > want to "pollute" regrtest with multiple options for bisect. > Currently, my script has already 4 options: > > haypo at selma$ python3 ~/prog/GIT/misc/python/bisect_test.py --help > usage: bisect_test.py [-h] [-i INPUT] [-o OUTPUT] [-n MAX_TESTS] [-N MAX_ITER] > > optional arguments: > -h, --help show this help message and exit > -i INPUT, --input INPUT > Test names produced by --list-tests written into a > file. If not set, run --list-tests > -o OUTPUT, --output OUTPUT > Result of the bisection > -n MAX_TESTS, --max-tests MAX_TESTS > Maximum number of tests to stop the bisection > (default: 1) > -N MAX_ITER, --max-iter MAX_ITER > Maximum number of bisection iterations (default: 100) > > I really like subcommands, it's a nice way to design complex CLI ;-) You could make it just a submodule in the test package. ./python -m test.bisect -R 3:3 test_os From ncoghlan at gmail.com Tue Jun 27 07:32:10 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 27 Jun 2017 21:32:10 +1000 Subject: [Python-Dev] Helping contributors with chores (do we have to?) In-Reply-To: References: <20170624105343.6cca748b@fsol> <02c890c4-48b4-af21-97f1-a3ade3cc6e6d@hastings.org> <20170625103320.4ab6b244@fsol> Message-ID: On 27 June 2017 at 07:49, Brett Cannon wrote: > On Sun, 25 Jun 2017 at 01:34 Antoine Pitrou wrote: >> So we have to rely on contributors to make the PR >> merge-ready by themselves... which means spending some time guiding >> them through the oh-so-exciting steps necessary to add the right >> Misc/NEWS entry, > > My hope is that having a news entry status check through Bedevere with help > with that. I'm also open to ideas on how to make it more obvious to > first-time contributors that things need to be done, e.g. maybe a default PR > message that is nothing more than "" (making it an HTML comment means it won't' ever show up in > the rendered output of the PR, plus that comment is short and thus easy to > ignore)? We could also make the CONTRIBUTING.md file have a bullet-point > list of the key things we expect people to do that is different from most > projects. (The only other thing I can think of is a comment for first-time > contributors pointing all of this out, but I'm leery of that as all of our > bots are stateless and this would make at least Bedevere stateful.) This reminded me of another relevant option for contributors making their first drafts of NEWS entries: they/we can take an existing NEWS snippet like https://github.com/python/cpython/blob/master/Misc/NEWS.d/next/Core%20and%20Builtins/2017-06-26-14-29-50.bpo-30765.Q5iBmf.rst, save it under an appropriate filename, and then edit the content. To be honest, I suspect that's how I'll end up writing most of my own NEWS entries rather than using blurb to generate them :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From victor.stinner at gmail.com Tue Jun 27 07:50:39 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 27 Jun 2017 13:50:39 +0200 Subject: [Python-Dev] Helping contributors with chores (do we have to?) In-Reply-To: <20170625103320.4ab6b244@fsol> References: <20170624105343.6cca748b@fsol> <02c890c4-48b4-af21-97f1-a3ade3cc6e6d@hastings.org> <20170625103320.4ab6b244@fsol> Message-ID: Small enhancement: I added Python 3.5 support to blurb with the help of Serhiy Storchaka ;-) Victor 2017-06-25 10:33 GMT+02:00 Antoine Pitrou : > On Sat, 24 Jun 2017 21:37:46 -0700 > Larry Hastings wrote: >> On 06/24/2017 09:14 PM, Serhiy Storchaka wrote: >> > Not only core developers make PRs for CPython. Since all non-trivial >> > changes need to be mentioned in Misc/NEWS, blurb becomes a required >> > tool for all committers. >> >> Well, no. *Writing blurb-compatible files* becomes a required step for >> all committers. And blurb makes that easy. But it's pretty easy to >> write them by hand; that's why we pre-created the "next" directories, >> and there are instructions in the dev guide. > > Hmm. If it were so easy, you wouldn't have felt the need to add that > functionality to blurb, right? :-) > > This is touching a more general problem, though. Before GitHub, we > (core developers) would take the patch submitted by a contributor, make > whatever minor changes were needed (e.g. Misc/NEWS) and push the > aggregate ourselves. With GitHub, while it's possible to edit someone > else's PR, it's frankly a PITA (I've tried to do it once, I don't want > to try a second time unless GitHub makes it massively easier and less > footgunning-prone). So we have to rely on contributors to make the PR > merge-ready by themselves... which means spending some time guiding > them through the oh-so-exciting steps necessary to add the right > Misc/NEWS entry, or fix the occasional bit of reStructuredText > mis-syntax. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com From steve.dower at python.org Tue Jun 27 10:36:23 2017 From: steve.dower at python.org (Steve Dower) Date: Tue, 27 Jun 2017 07:36:23 -0700 Subject: [Python-Dev] Proposal for C++ metaclasses Message-ID: Thought this might be interesting for those of us who live deeper in the language than most ? this is the formal proposal to add metaclasses to C++. http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2017/p0707r0.pdf Given the differences between Python and C++, it?s obviously got a different approach, though I am struck by the similarities. I think it?s also a good presentation on the value and use of metaclasses, so likely also interesting for those of us who occasionally teach or explain the concept. Cheers, Steve -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Jun 27 12:54:14 2017 From: guido at python.org (Guido van Rossum) Date: Tue, 27 Jun 2017 09:54:14 -0700 Subject: [Python-Dev] Proposal for C++ metaclasses In-Reply-To: References: Message-ID: Heh, I'm not totally surprised -- I took Python's metaclass design from a book named Putting Metaclasses to Work, by Ira R. Forman and Scott H. Danforth ( https://www.amazon.com/Putting-Metaclasses-Work-Ira-Forman/dp/0201433052). The book describes a custom metaclass extension to C++ supporting metaclasses, from which I took everything I could given that Python is a dynamic language (the key thing I left out was automatic synthesis of combined metaclasses when multiple inheritance sees two unrelated metaclasses). Hopefully the authors get some credit in the current C++ standard proposal. On Tue, Jun 27, 2017 at 7:36 AM, Steve Dower wrote: > Thought this might be interesting for those of us who live deeper in the > language than most ? this is the formal proposal to add metaclasses to C++. > > > > http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2017/p0707r0.pdf > > > > Given the differences between Python and C++, it?s obviously got a > different approach, though I am struck by the similarities. I think it?s > also a good presentation on the value and use of metaclasses, so likely > also interesting for those of us who occasionally teach or explain the > concept. > > > > Cheers, > > Steve > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Tue Jun 27 20:39:39 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 28 Jun 2017 02:39:39 +0200 Subject: [Python-Dev] New work-in-progress bisection tool for the Python test suite (in regrtest) In-Reply-To: References: Message-ID: 2017-06-27 7:33 GMT+02:00 Serhiy Storchaka : > You could make it just a submodule in the test package. > > ./python -m test.bisect -R 3:3 test_os I like the idea :-) I proposed a PR which was approved by Yury Selivanov, and so I just merged it! It means that you can now play with "./python -m test.bisect" in the master branch. Enjoy ;-) Example: ./python -m test.bisect -R 3:3 test_multiprocessing_forkserver This command should give you the name of the two failing test methods which are the last known reference leaks! => http://bugs.python.org/issue30775 This specific bisection is very slow since running the 286 tests of test_multiprocessing_forkserver using -R 3:3 takes longer than 3 minutes on my laptop! And the answer is already know, see the bpo ;-) FYI, apart of the bpo-30775, all other reference leaks now seem to be fixed on 2.7, 3.5, 3.6 and master branches! Tested on Windows and Linux. Victor From eric at trueblade.com Tue Jun 27 20:54:45 2017 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 27 Jun 2017 20:54:45 -0400 Subject: [Python-Dev] New work-in-progress bisection tool for the Python test suite (in regrtest) In-Reply-To: References: Message-ID: <26ea5a32-a2bd-cd03-238c-e425a4361ca4@trueblade.com> That's great, Victor. Thanks for all of your work on this. Eric. On 6/27/2017 8:39 PM, Victor Stinner wrote: > 2017-06-27 7:33 GMT+02:00 Serhiy Storchaka : >> You could make it just a submodule in the test package. >> >> ./python -m test.bisect -R 3:3 test_os > > I like the idea :-) I proposed a PR which was approved by Yury > Selivanov, and so I just merged it! It means that you can now play > with "./python -m test.bisect" in the master branch. Enjoy ;-) > Example: > > ./python -m test.bisect -R 3:3 test_multiprocessing_forkserver > > This command should give you the name of the two failing test methods > which are the last known reference leaks! > => http://bugs.python.org/issue30775 > > This specific bisection is very slow since running the 286 tests of > test_multiprocessing_forkserver using -R 3:3 takes longer than 3 > minutes on my laptop! And the answer is already know, see the bpo ;-) > > FYI, apart of the bpo-30775, all other reference leaks now seem to be > fixed on 2.7, 3.5, 3.6 and master branches! Tested on Windows and > Linux. > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/eric%2Ba-python-dev%40trueblade.com > From tjreedy at udel.edu Wed Jun 28 00:52:55 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 28 Jun 2017 00:52:55 -0400 Subject: [Python-Dev] New work-in-progress bisection tool for the Python test suite (in regrtest) In-Reply-To: References: Message-ID: On 6/27/2017 8:39 PM, Victor Stinner wrote: > 2017-06-27 7:33 GMT+02:00 Serhiy Storchaka : >> You could make it just a submodule in the test package. >> >> ./python -m test.bisect -R 3:3 test_os > > I like the idea :-) I proposed a PR which was approved by Yury > Selivanov, and so I just merged it! It means that you can now play > with "./python -m test.bisect" in the master branch. Enjoy ;-) > Example: > > ./python -m test.bisect -R 3:3 test_multiprocessing_forkserver > > This command should give you the name of the two failing test methods > which are the last known reference leaks! > => http://bugs.python.org/issue30775 > > This specific bisection is very slow since running the 286 tests of > test_multiprocessing_forkserver using -R 3:3 takes longer than 3 > minutes on my laptop! And the answer is already know, see the bpo ;-) > > FYI, apart of the bpo-30775, all other reference leaks now seem to be > fixed on 2.7, 3.5, 3.6 and master branches! Tested on Windows and > Linux. Are you testing for refleaks with gui enabled? A few weeks ago I discovered that some had crept in IDLE, but the gui-less leak tests never saw them. I fixed the ones that existed then, but as I add tests, it will be easy enough to add more leaks. Can test.bisect run a test module in idlelib.idle_test? Unittest.main will, but as far as I know, it does not run leak tests. Test.regrtest has a leak test, but will not modules in other directories, at least not that one. I wrote a custom script to narrow a leak in test.test_idle to one or more files in idlelib.idle_test, but from then on it was manual work, commenting out parts of the file until I identified the test function or functions. One of the things I had to do is test setUpClass and tearDownClass by themselves by commenting out all test_xyz functions and inserting test_dummy(self): pass. -- Terry Jan Reedy From zachary.ware+pydev at gmail.com Wed Jun 28 01:27:26 2017 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Wed, 28 Jun 2017 00:27:26 -0500 Subject: [Python-Dev] New work-in-progress bisection tool for the Python test suite (in regrtest) In-Reply-To: References: Message-ID: On Tue, Jun 27, 2017 at 11:52 PM, Terry Reedy wrote: > Are you testing for refleaks with gui enabled? Yes; the refleak builders are running on my Gentoo and Windows workers, both of which (should have, at least) GUI available and enabled. However, I have caught Xvfb not running properly on the Gentoo bot a time or two, so there may be occasional outages in that coverage. -- Zach From benhoyt at gmail.com Wed Jun 28 09:50:10 2017 From: benhoyt at gmail.com (Ben Hoyt) Date: Wed, 28 Jun 2017 09:50:10 -0400 Subject: [Python-Dev] Micro-optimizations by adding special-case bytecodes? In-Reply-To: References: Message-ID: I've now got a patch to add new COMPARE_IS_NONE and COMPARE_IS_NOT_NONE bytecodes to CPython to attempt to speed up the very common "is None" and "is not None" expressions. It's a very simple change overall. Here is my patch: https://gist.github.com/benhoyt/e5ba19afe7b869fd743c1c39fc2afdf8 Note: The intent here is a proof of concept, not a CPython-quality implementation. I've put the code to generate these in compile.c, though it may be better off in peephole.c. Also, I'm not sure two new bytecodes are needed -- it'd be simpler and probably almost as fast to have one bytecode instruction COMPARE_NONE with an argument that determines the "is" vs "is not" part. Anyway, it looks like a nice improvement on a micro-benchmark, with a 20% speed increase for "is None" and "is not None" expressions using timeit. See results at [1] below. However, I'm not very familiar with the python/performance benchmarks and with my first try at running those I'm getting wildly varying results (most faster, but some tests say 1.7x slower, so I'm pretty sure this is an artifact of how I'm running them). See [2] below for my results. Would someone with more experience running those be able to run the performance tests and post results? Any other thoughts welcome as well. -Ben [1] Here are the micro-benchmarks using timeit against the latest master (normal build and also PGO) as well as with my patch: latest master (e613e6add5f07ff6aad5802924596b631b707d2a, June 27) ./configure make -j ../test_none.sh x = 1234; x is None -- 20000000 loops, best of 5: 19.8 nsec per loop x = 1234; x is not None -- 10000000 loops, best of 5: 20 nsec per loop x = None; x is None -- 10000000 loops, best of 5: 20.7 nsec per loop x = None; x is not None -- 10000000 loops, best of 5: 20.8 nsec per loop avg 20.3 nsec per loop ./configure --enable-optimizations make -j ../test_none.sh x = 1234; x is None -- 20000000 loops, best of 5: 18.6 nsec per loop x = 1234; x is not None -- 20000000 loops, best of 5: 19.2 nsec per loop x = None; x is None -- 10000000 loops, best of 5: 19.5 nsec per loop x = None; x is not None -- 20000000 loops, best of 5: 19.4 nsec per loop avg 19.2 nsec per loop is_none_bytecode branch (with my patch) ./configure make -j ../test_none.sh x = 1234; x is None -- 20000000 loops, best of 5: 16 nsec per loop x = 1234; x is not None -- 20000000 loops, best of 5: 16.4 nsec per loop x = None; x is None -- 20000000 loops, best of 5: 16.6 nsec per loop x = None; x is not None -- 20000000 loops, best of 5: 15.5 nsec per loop avg 16.1 nsec per loop (21% faster than master) ./configure --enable-optimizations make -j ../test_none.sh x = 1234; x is None -- 20000000 loops, best of 5: 15.6 nsec per loop x = 1234; x is not None -- 20000000 loops, best of 5: 16.4 nsec per loop x = None; x is None -- 20000000 loops, best of 5: 15.7 nsec per loop x = None; x is not None -- 20000000 loops, best of 5: 15.4 nsec per loop avg 15.8 nsec per loop (18% faster than master) [2] Benchmarks comparing master and is_none_bytecode patch (each compiled with --enable-optimizations) using python/performance: +-------------------------+------------+------------------------------+ | Benchmark | master_opt | is_none_bytecode_opt | +=========================+============+==============================+ | 2to3 | 617 ms | 541 ms: 1.14x faster (-12%) | +-------------------------+------------+------------------------------+ | chameleon | 19.9 ms | 18.6 ms: 1.07x faster (-7%) | +-------------------------+------------+------------------------------+ | crypto_pyaes | 208 ms | 201 ms: 1.04x faster (-3%) | +-------------------------+------------+------------------------------+ | deltablue | 13.8 ms | 12.9 ms: 1.07x faster (-7%) | +-------------------------+------------+------------------------------+ | django_template | 245 ms | 233 ms: 1.05x faster (-5%) | +-------------------------+------------+------------------------------+ | dulwich_log | 132 ms | 126 ms: 1.05x faster (-5%) | +-------------------------+------------+------------------------------+ | fannkuch | 898 ms | 849 ms: 1.06x faster (-5%) | +-------------------------+------------+------------------------------+ | float | 204 ms | 191 ms: 1.07x faster (-7%) | +-------------------------+------------+------------------------------+ | genshi_text | 57.5 ms | 54.8 ms: 1.05x faster (-5%) | +-------------------------+------------+------------------------------+ | genshi_xml | 122 ms | 114 ms: 1.07x faster (-6%) | +-------------------------+------------+------------------------------+ | hexiom | 18.4 ms | 26.9 ms: 1.46x slower (+46%) | +-------------------------+------------+------------------------------+ | html5lib | 155 ms | 265 ms: 1.72x slower (+72%) | +-------------------------+------------+------------------------------+ | json_dumps | 23.0 ms | 30.6 ms: 1.33x slower (+33%) | +-------------------------+------------+------------------------------+ | logging_format | 23.1 us | 21.6 us: 1.07x faster (-6%) | +-------------------------+------------+------------------------------+ | logging_simple | 19.6 us | 18.1 us: 1.09x faster (-8%) | +-------------------------+------------+------------------------------+ | mako | 35.4 ms | 33.5 ms: 1.06x faster (-5%) | +-------------------------+------------+------------------------------+ | meteor_contest | 177 ms | 169 ms: 1.04x faster (-4%) | +-------------------------+------------+------------------------------+ | nqueens | 184 ms | 174 ms: 1.06x faster (-5%) | +-------------------------+------------+------------------------------+ | pathlib | 46.7 ms | 45.1 ms: 1.03x faster (-3%) | +-------------------------+------------+------------------------------+ | pickle | 19.5 us | 17.4 us: 1.12x faster (-11%) | +-------------------------+------------+------------------------------+ | pickle_dict | 58.6 us | 49.9 us: 1.17x faster (-15%) | +-------------------------+------------+------------------------------+ | pickle_list | 7.45 us | 6.86 us: 1.09x faster (-8%) | +-------------------------+------------+------------------------------+ | python_startup | 28.1 ms | 31.0 ms: 1.10x slower (+10%) | +-------------------------+------------+------------------------------+ | regex_compile | 335 ms | 353 ms: 1.05x slower (+5%) | +-------------------------+------------+------------------------------+ | regex_dna | 230 ms | 247 ms: 1.07x slower (+7%) | +-------------------------+------------+------------------------------+ | regex_effbot | 4.16 ms | 4.49 ms: 1.08x slower (+8%) | +-------------------------+------------+------------------------------+ | regex_v8 | 33.0 ms | 36.2 ms: 1.10x slower (+10%) | +-------------------------+------------+------------------------------+ | richards | 120 ms | 124 ms: 1.03x slower (+3%) | +-------------------------+------------+------------------------------+ | scimark_fft | 539 ms | 593 ms: 1.10x slower (+10%) | +-------------------------+------------+------------------------------+ | scimark_monte_carlo | 183 ms | 172 ms: 1.07x faster (-6%) | +-------------------------+------------+------------------------------+ | scimark_sparse_mat_mult | 6.42 ms | 6.54 ms: 1.02x slower (+2%) | +-------------------------+------------+------------------------------+ | spectral_norm | 251 ms | 233 ms: 1.08x faster (-7%) | +-------------------------+------------+------------------------------+ | sqlalchemy_declarative | 246 ms | 229 ms: 1.07x faster (-7%) | +-------------------------+------------+------------------------------+ | sqlalchemy_imperative | 48.8 ms | 45.9 ms: 1.06x faster (-6%) | +-------------------------+------------+------------------------------+ | sqlite_synth | 4.56 us | 4.69 us: 1.03x slower (+3%) | +-------------------------+------------+------------------------------+ | sympy_integrate | 31.4 ms | 33.5 ms: 1.07x slower (+7%) | +-------------------------+------------+------------------------------+ | telco | 9.76 ms | 9.99 ms: 1.02x slower (+2%) | +-------------------------+------------+------------------------------+ | unpack_sequence | 73.8 ns | 76.9 ns: 1.04x slower (+4%) | +-------------------------+------------+------------------------------+ | unpickle | 24.1 us | 24.8 us: 1.03x slower (+3%) | +-------------------------+------------+------------------------------+ | unpickle_list | 6.04 us | 6.46 us: 1.07x slower (+7%) | +-------------------------+------------+------------------------------+ | xml_etree_parse | 223 ms | 245 ms: 1.10x slower (+10%) | +-------------------------+------------+------------------------------+ | xml_etree_iterparse | 175 ms | 203 ms: 1.16x slower (+16%) | +-------------------------+------------+------------------------------+ | xml_etree_process | 144 ms | 149 ms: 1.03x slower (+3%) | +-------------------------+------------+------------------------------+ Not significant (15): chaos; go; json_loads; nbody; pickle_pure_python; pidigits; python_startup_no_site; raytrace; scimark_lu; scimark_sor; sympy_expand; sympy_sum; sympy_str; unpickle_pure_python; xml_etree_generate On Thu, May 25, 2017 at 10:28 AM, Ben Hoyt wrote: > Thanks, Victor. That's very helpful. So RETURN_NONE (and probably > RETURN_SMALL_CONST) are not worth it, based on your empirical tests. Your > patch shows how (relatively) straight-forward it is to test out new opcodes. > > I'm still optimistic about the value of COMPARE_IS_NONE and > COMPARE_IS_NOT_NONE, though. Mainly because I've done a quick expansion of > LOAD_CONST(None) + COMPARE_OP and it's quite a bit more code and many more > instructions than COMPARE_IS_NONE would be: > > LOAD_CONST(None) > COMPARE_OP > PyObject *value = ((PyTupleObject *)(consts))->ob_item[oparg]; > value->ob_refcnt++; > *stack_pointer++ = value; > FAST_DISPATCH(); > PyObject *right = *--stack_pointer; > PyObject *left = stack_pointer[-1] > > // cmp_outcome(), presumably inlined > int r = 0; > switch (compare_oparg) { > case PyCmp_IS: > r = (left == right); > break; > case PyCmp_IS_NOT: > r = (left != right); > break; > case ... > } > PyObject *res = r ? Py_True : Py_False; > res->ob_refcnt++; > > if (--(left)->ob_refcnt == 0) > _Py_Dealloc(left); > if (--(right)->ob_refcnt == 0) > _Py_Dealloc(right); > stack_pointer[-1] = res; > if (res == NULL) > goto error; > > PREDICT(POP_JUMP_IF_FALSE); > PREDICT(POP_JUMP_IF_TRUE); > DISPATCH(); > > > COMPARE_IS_NONE > PyObject* left = stack_pointer[-1]; // TOP() > PyObject* res = (left == Py_None) ? Py_True : Py_False; > res->ob_refcnt++; > if (--(left)->ob_refcnt == 0) > _Py_Dealloc(left); > stack_pointer[-1] = res; // SET_TOP(res) > PREDICT(POP_JUMP_IF_FALSE); > PREDICT(POP_JUMP_IF_TRUE); > DISPATCH(); > > You don't have to get the const arg, there are fewer increfs/decrefs, you > skip a pop, you don't have to test res==NULL (because it's Py_True or > Py_False, which are never NULL), and if there are separate COMPARE_IS_NONE > and COMPARE_IS_NOT_NONE you don't have to switch on the compare arg (though > I'm not sure if that part will be worth it). > > For reference, based on a grep, " is None" occurs 2737 times in the CPython > source tree, and " is not None" 2010 times. And I know personally I often > use them in loops as well is at the start of functions (for mutable default > arg handling). > > Still, the performance proof will be in the pudding! I might hack these two > opcodes together and test it at some point. > > -Ben > > On Thu, May 25, 2017 at 6:47 AM, Victor Stinner > wrote: >> >> Hi Ben, >> >> I am not convinced that combining operations will have a significant >> impact in term of performance. Mark Shanon implemented that in his HotPy >> project. >> >> I proposed a RETURN_NONE opcode to combine LOAD_CONST with RETURN_VALUE. >> The issue was rejected because I failed to show any speedup. >> >> https://bugs.python.org/issue28800 >> >> I would be interested to restart/finish my registervm project to use >> register-based bytecode. It allows to implement more optmisations and reduce >> the number of instructions. In my experience, less instructions = faster >> code. >> >> http://faster-cpython.readthedocs.io/registervm.html >> >> Mark's bytecode uses registers but also a stack. >> >> Victor >> >> Le 24 mai 2017 8:09 PM, "Ben Hoyt" a ?crit : >>> >>> Hi folks, >>> >>> I was looking at some `dis` output today, and I was wondering if anyone >>> has investigated optimizing Python (slightly) by adding special-case >>> bytecodes for common expressions or statements involving constants? >>> >>> For example, I (and, based on a quick grep of the stdlib, many others) >>> write "x is None" and "x is not None" very often. Or "return True" or >>> "return None" or "return 1" and things like that. These all expand into two >>> bytecodes, which seems pretty non-optimal (LOAD_CONST + COMPARE_OP or >>> LOAD_CONST + RETURN_VALUE). It seems we could get an easy speedup for these >>> common cases by adding a peephole optimization and some new opcodes (maybe >>> COMPARE_IS_SMALL_CONST and RETURN_SMALL_CONST for these cases). >>> >>> I'm not proposing to do this yet, as I'd need to benchmark to see how >>> much of a gain (if any) it would amount to, but I'm just wondering if >>> there's any previous work on this kind of thing. Or, if not, any other >>> thoughts before I try it? >>> >>> Thanks, >>> Ben >>> >>> >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev at python.org >>> https://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: >>> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com >>> > From victor.stinner at gmail.com Wed Jun 28 10:20:06 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 28 Jun 2017 16:20:06 +0200 Subject: [Python-Dev] Micro-optimizations by adding special-case bytecodes? In-Reply-To: References: Message-ID: (Victor wears his benchmark hat.) 2017-06-28 15:50 GMT+02:00 Ben Hoyt : > ../test_none.sh > x = 1234; x is None -- 20000000 loops, best of 5: 19.8 nsec per loop > x = 1234; x is not None -- 10000000 loops, best of 5: 20 nsec per loop > x = None; x is None -- 10000000 loops, best of 5: 20.7 nsec per loop > x = None; x is not None -- 10000000 loops, best of 5: 20.8 nsec per loop > avg 20.3 nsec per loop Hum, please use perf timeit instead of timeit, it's more reliable. See also: "How to get reproductible benchmark results" http://perf.readthedocs.io/en/latest/run_benchmark.html#how-to-get-reproductible-benchmark-results > [2] Benchmarks comparing master and is_none_bytecode patch (each > compiled with --enable-optimizations) using python/performance: > > +-------------------------+------------+------------------------------+ > | Benchmark | master_opt | is_none_bytecode_opt | > +=========================+============+==============================+ > | 2to3 | 617 ms | 541 ms: 1.14x faster (-12%) | > +-------------------------+------------+------------------------------+ > | chameleon | 19.9 ms | 18.6 ms: 1.07x faster (-7%) | > +-------------------------+------------+------------------------------+ > | crypto_pyaes | 208 ms | 201 ms: 1.04x faster (-3%) | > +-------------------------+------------+------------------------------+ > | deltablue | 13.8 ms | 12.9 ms: 1.07x faster (-7%) | FYI you can add the -G option to perf compare_to to sort results by speed (group faster & slower). It gives a more readable table. I also like using --min-speed=5 to ignore changes smaller than 5%, it reduces the noise and makes the stable more readable. Victor From solipsis at pitrou.net Wed Jun 28 10:20:54 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 28 Jun 2017 16:20:54 +0200 Subject: [Python-Dev] Micro-optimizations by adding special-case bytecodes? References: Message-ID: <20170628162054.0b4469df@fsol> On Wed, 28 Jun 2017 09:50:10 -0400 Ben Hoyt wrote: > I've now got a patch to add new COMPARE_IS_NONE and > COMPARE_IS_NOT_NONE bytecodes to CPython to attempt to speed up the > very common "is None" and "is not None" expressions. It's a very > simple change overall. Here is my patch: > https://gist.github.com/benhoyt/e5ba19afe7b869fd743c1c39fc2afdf8 I don't want to discourage you, but this is unlikely to produce significant speedups on real-world code. The simple reason is that comparing to None is not a bottleneck for any real application -- comparing to None is probably damn fast compared to everything else the real application does. That said, it would be nice if you could get stable benchmark results to validate that theory (or not!) ;-) Regards Antoine. From benhoyt at gmail.com Wed Jun 28 10:30:00 2017 From: benhoyt at gmail.com (Ben Hoyt) Date: Wed, 28 Jun 2017 10:30:00 -0400 Subject: [Python-Dev] Micro-optimizations by adding special-case bytecodes? In-Reply-To: References: Message-ID: Thanks for the pointers -- I wasn't aware of perf timeit. I'll try to get some more stable benchmark results. -Ben On Wed, Jun 28, 2017 at 10:20 AM, Victor Stinner wrote: > (Victor wears his benchmark hat.) > > 2017-06-28 15:50 GMT+02:00 Ben Hoyt : >> ../test_none.sh >> x = 1234; x is None -- 20000000 loops, best of 5: 19.8 nsec per loop >> x = 1234; x is not None -- 10000000 loops, best of 5: 20 nsec per loop >> x = None; x is None -- 10000000 loops, best of 5: 20.7 nsec per loop >> x = None; x is not None -- 10000000 loops, best of 5: 20.8 nsec per loop >> avg 20.3 nsec per loop > > Hum, please use perf timeit instead of timeit, it's more reliable. See also: > > "How to get reproductible benchmark results" > http://perf.readthedocs.io/en/latest/run_benchmark.html#how-to-get-reproductible-benchmark-results > >> [2] Benchmarks comparing master and is_none_bytecode patch (each >> compiled with --enable-optimizations) using python/performance: >> >> +-------------------------+------------+------------------------------+ >> | Benchmark | master_opt | is_none_bytecode_opt | >> +=========================+============+==============================+ >> | 2to3 | 617 ms | 541 ms: 1.14x faster (-12%) | >> +-------------------------+------------+------------------------------+ >> | chameleon | 19.9 ms | 18.6 ms: 1.07x faster (-7%) | >> +-------------------------+------------+------------------------------+ >> | crypto_pyaes | 208 ms | 201 ms: 1.04x faster (-3%) | >> +-------------------------+------------+------------------------------+ >> | deltablue | 13.8 ms | 12.9 ms: 1.07x faster (-7%) | > > FYI you can add the -G option to perf compare_to to sort results by > speed (group faster & slower). It gives a more readable table. I also > like using --min-speed=5 to ignore changes smaller than 5%, it reduces > the noise and makes the stable more readable. > > Victor From benhoyt at gmail.com Wed Jun 28 12:11:18 2017 From: benhoyt at gmail.com (Ben Hoyt) Date: Wed, 28 Jun 2017 12:11:18 -0400 Subject: [Python-Dev] Micro-optimizations by adding special-case bytecodes? In-Reply-To: <20170628162054.0b4469df@fsol> References: <20170628162054.0b4469df@fsol> Message-ID: > I don't want to discourage you, but this is unlikely to produce > significant speedups on real-world code. The simple reason is that > comparing to None is not a bottleneck for any real application -- > comparing to None is probably damn fast compared to everything else > the real application does. That's true. However, I don't think that makes small optimizations worthless. For example, the core devs speed up memory allocation or method calls or whatever by a few percent, and consider it worthwhile, because it all adds up. > That said, it would be nice if you could get stable benchmark results > to validate that theory (or not!) ;-) Yep, I'm more than willing to accept that outcome if that's what the results show! :-) -Ben From victor.stinner at gmail.com Thu Jun 29 11:09:08 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 29 Jun 2017 17:09:08 +0200 Subject: [Python-Dev] Buildbot report (almost July) Message-ID: Hi, Our buildbots are now even more stable than in my previous buildbot report. Many random failures have been fixed, even if there are still many rare random failures (most of them are related to multiprocessing). Search for issues created by "haypo" (me!) with the Component "Tests" to see all remaining issues. I'm trying to open an issue for *each* buildbot failure! So yeah, we have a long list of 44 open issues! Correct me if I'm wrong, but, for the first time, *all reference leaks* have been fixed on *all branches* (2.7, 3.5, 3.6 and master), on *Linux and Windows*! Before, we mostly focused on the master branch (called "default" in Mercurial) on Linux. I also started to fix a few "memory block" leaks, most (or all?) of them should also be fixed (on all branches, on Linux and Windows). A new policy has been decided (on python-committers): if a commit breaks too many buildbots and cannot fixed easily nor quickly, the commit will be reverted, just to keep our buildbots "green". It doesn't mean that the commit will be rejected forever. It's a practical solution to give more time to write a proper fix, take time to review it, and not have to be stressed to fix buildbots "ASAP". Moreover, keeping green buildbots all the time allows to catch regressions more quickly, which ease debug in many ways. You have be warned! Now I will not hesitate to revert your change if you break my little buildbots ;-) I mostly care of Linux, Windows, macOS and FreeBSD (10 and CURRENT). Breaking other platforms is less important, since other platforms already have issues, but not enough developers interested to fix them. Obviously, I'm interested in keeping more buildbots green, if someone shows up and help me to fix remaining issues! Victor From tjreedy at udel.edu Thu Jun 29 13:13:05 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 29 Jun 2017 13:13:05 -0400 Subject: [Python-Dev] Buildbot report (almost July) In-Reply-To: References: Message-ID: On 6/29/2017 11:09 AM, Victor Stinner wrote: > Hi, > > Our buildbots are now even more stable than in my previous buildbot report. > > Many random failures have been fixed, even if there are still many > rare random failures (most of them are related to multiprocessing). > > Search for issues created by "haypo" (me!) with the Component "Tests" > to see all remaining issues. I'm trying to open an issue for *each* > buildbot failure! So yeah, we have a long list of 44 open issues! > > Correct me if I'm wrong, but, for the first time, *all reference > leaks* have been fixed on *all branches* (2.7, 3.5, 3.6 and master), > on *Linux and Windows*! Before, we mostly focused on the master branch > (called "default" in Mercurial) on Linux. > > I also started to fix a few "memory block" leaks, most (or all?) of > them should also be fixed (on all branches, on Linux and Windows). > > A new policy has been decided (on python-committers): if a commit > breaks too many buildbots and cannot fixed easily nor quickly, the > commit will be reverted, just to keep our buildbots "green". It > doesn't mean that the commit will be rejected forever. It's a > practical solution to give more time to write a proper fix, take time > to review it, and not have to be stressed to fix buildbots "ASAP". > Moreover, keeping green buildbots all the time allows to catch > regressions more quickly, which ease debug in many ways. > > You have be warned! Now I will not hesitate to revert your change if > you break my little buildbots ;-) > > I mostly care of Linux, Windows, macOS and FreeBSD (10 and CURRENT). > Breaking other platforms is less important, since other platforms > already have issues, but not enough developers interested to fix them. > Obviously, I'm interested in keeping more buildbots green, if someone > shows up and help me to fix remaining issues! Great work. How about compiler warnings (and errors)? When I compile on Windows, there are a boatload of orange-yellow warnings. Some are about using a deprecated featured; some about dodgy casts; some (I presume) about other things. Should 'no warnings' be a goal? I believe you once fixed some, but new commits added more. Could 'no additional warnings' be a CI requirement for merging? I expect that a new "How to avoid compiler warnings on Windows" section of the devguide would be needed. -- Terry Jan Reedy From steve.dower at python.org Thu Jun 29 13:54:10 2017 From: steve.dower at python.org (Steve Dower) Date: Thu, 29 Jun 2017 10:54:10 -0700 Subject: [Python-Dev] Buildbot report (almost July) In-Reply-To: References: Message-ID: <969460ba-30c7-44b0-9d38-5895518d4285@python.org> On 29Jun2017 1013, Terry Reedy wrote: > Great work. Agreed. Thanks, Victor! > How about compiler warnings (and errors)? When I compile on Windows, > there are a boatload of orange-yellow warnings. Some are about using a > deprecated featured; some about dodgy casts; some (I presume) about > other things. Should 'no warnings' be a goal? Yes, I think that's a good goal. We're quickly getting there as well - I just merged two contributions yesterday that should significantly reduce the number of warnings. > I believe you once fixed some, but new commits added more. Could 'no > additional warnings' be a CI requirement for merging? I expect that a > new "How to avoid compiler warnings on Windows" section of the devguide > would be needed. This is probably not feasible without a *really good* section in the devguide. I would rather have warnings in the output than global suppressions, and suppressing warnings globally is the usual instinct. Locally suppressing warnings can be fairly hideous and is usually not portable. Some warnings are also complicated because of the nature of CPython. For example, the socket module exposes deprecated CRT functions (on Windows) directly because the API of the socket module promises to provide the function directly. Changing to the safer function would break the API guarantee (except sometimes it won't... hence "complicated"). Noting in PR builds that there are new warnings would be great if possible. I'd be concerned about it becoming a hard requirement though - I much prefer to leave the final decision in the hands of trusted people and provide them enough information to make a good decision. Cheers, Steve From tjreedy at udel.edu Thu Jun 29 14:57:04 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 29 Jun 2017 14:57:04 -0400 Subject: [Python-Dev] Buildbot report (almost July) In-Reply-To: <969460ba-30c7-44b0-9d38-5895518d4285@python.org> References: <969460ba-30c7-44b0-9d38-5895518d4285@python.org> Message-ID: On 6/29/2017 1:54 PM, Steve Dower wrote: > On 29Jun2017 1013, Terry Reedy wrote: >> Great work. > > Agreed. Thanks, Victor! > >> How about compiler warnings (and errors)? When I compile on Windows, >> there are a boatload of orange-yellow warnings. Some are about using >> a deprecated featured; some about dodgy casts; some (I presume) about >> other things. Should 'no warnings' be a goal? > > Yes, I think that's a good goal. We're quickly getting there as well - I > just merged two contributions yesterday that should significantly reduce > the number of warnings. Great. I recompiled changes for the last day or two (about 100 c files) and see none. _socket aside, I presume some of those files were the ones you fixed. >> I believe you once fixed some, but new commits added more. Could 'no >> additional warnings' be a CI requirement for merging? I expect that a >> new "How to avoid compiler warnings on Windows" section of the >> devguide would be needed. > > This is probably not feasible without a *really good* section in the > devguide. I would rather have warnings in the output than global > suppressions, and suppressing warnings globally is the usual instinct. > Locally suppressing warnings can be fairly hideous and is usually not > portable. > > Some warnings are also complicated because of the nature of CPython. For > example, the socket module exposes deprecated CRT functions (on Windows) > directly because the API of the socket module promises to provide the > function directly. Changing to the safer function would break the API > guarantee (except sometimes it won't... hence "complicated"). Thanks for the explanation. Does 'deprecated' mean future removal in ms-speak? Perhaps after the compile, build.bat should print something like [Ignore any socket deprecation warnings] > Noting in PR builds that there are new warnings would be great if > possible. I'd be concerned about it becoming a hard requirement though - > I much prefer to leave the final decision in the hands of trusted people > and provide them enough information to make a good decision. A robot that could scan the appveyor compile log, compare with a 'current warnings' set, and post a comment requesting a fix and possible help sources would be better than leaving them unseen, which I expect is the current situation. -- Terry Jan Reedy From steve.dower at python.org Thu Jun 29 15:22:00 2017 From: steve.dower at python.org (Steve Dower) Date: Thu, 29 Jun 2017 12:22:00 -0700 Subject: [Python-Dev] Buildbot report (almost July) In-Reply-To: References: <969460ba-30c7-44b0-9d38-5895518d4285@python.org> Message-ID: <3fe4fea2-8ae0-a5be-0906-4b7540a67088@python.org> On 29Jun2017 1157, Terry Reedy wrote: > On 6/29/2017 1:54 PM, Steve Dower wrote: >> Some warnings are also complicated because of the nature of CPython. >> For example, the socket module exposes deprecated CRT functions (on >> Windows) directly because the API of the socket module promises to >> provide the function directly. Changing to the safer function would >> break the API guarantee (except sometimes it won't... hence >> "complicated"). > > Thanks for the explanation. Does 'deprecated' mean future removal in > ms-speak? Perhaps after the compile, build.bat should print something like > [Ignore any socket deprecation warnings] Probably not future removal, considering the vast quantities of legacy code that we don't want to break, but renaming is possible (generally for standards compliance, such as the timezone->_timezone rename) and new platforms (e.g. ARM) are not guaranteed to have the APIs when there is essentially no existing code to worry about. In many (most?) cases, deprecations in MS libraries indicate that there are known security issues either inherently in the function itself or with many/most uses of it, and there is a better option (e.g. strcpy is deprecated in favour of strcpy_s, which requires/infers the length of the destination buffer and ensures you never overflow). I'm still in favour of local suppression over ignoring warnings - we just need to promote the macro at https://github.com/python/cpython/blob/master/Modules/socketmodule.c#L510 to somewhere global to make it consistent (and maybe extend it to handle other compilers where it matters). But this is why I'm not against the general idea, but it requires that the devguide entry be well written. Let's clean up all the warnings, but in the right way, not the way people use when faced with arbitrary rules. Cheers, Steve From victor.stinner at gmail.com Thu Jun 29 16:04:06 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 29 Jun 2017 22:04:06 +0200 Subject: [Python-Dev] Buildbot report (almost July) In-Reply-To: References: Message-ID: 2017-06-29 19:13 GMT+02:00 Terry Reedy : > How about compiler warnings (and errors)? That would be nice to have, but I don't have the bandwidth to handle this goal. > When I compile on Windows, there > are a boatload of orange-yellow warnings. Some are about using a deprecated > featured; some about dodgy casts; some (I presume) about other things. > Should 'no warnings' be a goal? Last years, I tried to fix a few integer downcast warnings on Windows 64-bit, but they are still a few ones. It's not easy to distinguish warnings from real bugs (integer overflow). There are a few open issues about Windows 64-bit warnings. Segev Finer just wrote an interesting but big PR: https://github.com/python/cpython/pull/2492 The deprecated warnings is another interesting topic :-) > I believe you once fixed some, but new commits added more. Could 'no > additional warnings' be a CI requirement for merging? I expect that a new > "How to avoid compiler warnings on Windows" section of the devguide would be > needed. My policy is more to start by fixing all warnings, before starting to check for regressions. I began by fixing the most common test failures, then references leaks, and now memory leaks. So it becomes much simpler to check for regressions on these checks. Victor From robb at datalogics.com Thu Jun 29 17:34:03 2017 From: robb at datalogics.com (Rob Boehne) Date: Thu, 29 Jun 2017 21:34:03 +0000 Subject: [Python-Dev] Buildbot report (almost July) In-Reply-To: References: Message-ID: Hello, I?m new to the list, and contributing to Python specifically, and I?m interested in getting master and 3.6 branches building and working ?better? on UNIX. I?ve been looking at a problem building 3.6 on HP-UX and see a PR was merged into master, https://github.com/python/cpython/pull/1351 and I?d like to see it applied to 3.6. I?m happy to create a PR with a cherry-picked commit, and/or test. Rob Boehne Senior Software Architect | Datalogics, Inc. +1.312.853.8351 | robb at datalogics.com datalogics.com | blogs.datalogics.com Connect with us: Facebook | Twitter | LinkedIn | YouTube PROUDLY CELEBRATING 50 YEARS! On 6/29/17, 10:09 AM, "Python-Dev on behalf of Victor Stinner" wrote: >Hi, > >Our buildbots are now even more stable than in my previous buildbot >report. > >Many random failures have been fixed, even if there are still many >rare random failures (most of them are related to multiprocessing). > >Search for issues created by "haypo" (me!) with the Component "Tests" >to see all remaining issues. I'm trying to open an issue for *each* >buildbot failure! So yeah, we have a long list of 44 open issues! > >Correct me if I'm wrong, but, for the first time, *all reference >leaks* have been fixed on *all branches* (2.7, 3.5, 3.6 and master), >on *Linux and Windows*! Before, we mostly focused on the master branch >(called "default" in Mercurial) on Linux. > >I also started to fix a few "memory block" leaks, most (or all?) of >them should also be fixed (on all branches, on Linux and Windows). > >A new policy has been decided (on python-committers): if a commit >breaks too many buildbots and cannot fixed easily nor quickly, the >commit will be reverted, just to keep our buildbots "green". It >doesn't mean that the commit will be rejected forever. It's a >practical solution to give more time to write a proper fix, take time >to review it, and not have to be stressed to fix buildbots "ASAP". >Moreover, keeping green buildbots all the time allows to catch >regressions more quickly, which ease debug in many ways. > >You have be warned! Now I will not hesitate to revert your change if >you break my little buildbots ;-) > >I mostly care of Linux, Windows, macOS and FreeBSD (10 and CURRENT). >Breaking other platforms is less important, since other platforms >already have issues, but not enough developers interested to fix them. >Obviously, I'm interested in keeping more buildbots green, if someone >shows up and help me to fix remaining issues! > >Victor >_______________________________________________ >Python-Dev mailing list >Python-Dev at python.org >https://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: >https://mail.python.org/mailman/options/python-dev/robb%40datalogics.com From victor.stinner at gmail.com Fri Jun 30 03:29:42 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 30 Jun 2017 09:29:42 +0200 Subject: [Python-Dev] (no subject) Message-ID: Hi Robb, 2017-06-29 23:34 GMT+02:00 Rob Boehne : > I?m new to the list, and contributing to Python specifically, and I?m > interested in getting master and 3.6 branches building and working > ?better? on UNIX. > I?ve been looking at a problem building 3.6 on HP-UX and see a PR was > merged into master, https://github.com/python/cpython/pull/1351 and I?d > like to see it applied to 3.6. I?m happy to create a PR with a > cherry-picked commit, and/or test. Sure, this change can be backported to 3.6, maybe also to 3.5. But hum, I may suggest to first focus on the master branch and fix most HP-UX issues, before spending time on backport. I would prefer to see most tests of the important modules pass on HP-UX (ex: test_os). For a backport, you can directly comment http://bugs.python.org/issue30183 Victor From solipsis at pitrou.net Fri Jun 30 04:05:57 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 30 Jun 2017 10:05:57 +0200 Subject: [Python-Dev] Bug in buildbot view (changes column) Message-ID: <20170630100557.1e0c8493@fsol> Hello, If you look at this page (waterfall view of 3.x unstable buildbots), you'll notice the "changes" column on the left lists the recent changesets on master: http://buildbot.python.org/all/waterfall?tag=3.x.unstable However, on the equivalent view for 3.x stable buildbots, the "changes" column on the left is empty: http://buildbot.python.org/all/waterfall?category=3.x.stable Is there a particular reason? Oh wait... The first URL lists "3.x.unstable" as a *tag* while the second URL lists "3.x.stable" as a *category*. If I change the second URL to use "tag" instead of "category", the "changes" column miraculously lists the recent changesets: http://buildbot.python.org/all/waterfall?tag=3.x.stable Finicky buildbot :-/ Regards Antoine. From tjreedy at udel.edu Fri Jun 30 04:22:49 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 30 Jun 2017 04:22:49 -0400 Subject: [Python-Dev] Buildbot report (almost July) In-Reply-To: References: Message-ID: On 6/29/2017 5:34 PM, Rob Boehne wrote: > Hello, > > I?m new to the list, and contributing to Python specifically, and I?m > interested in getting master and 3.6 branches building and working > ?better? on UNIX. > I?ve been looking at a problem building 3.6 on HP-UX and see a PR was > merged into master, https://github.com/python/cpython/pull/1351 and The issue is https://bugs.python.org/issue30183 3.6 and 3.5 are marked as possible backport targets. I?d > like to see it applied to 3.6. I?m happy to create a PR with a > cherry-picked commit, and/or test. Ask on the issue or PR whether David of Victor are planning to do a backport or if they know some reason why it would be difficult, and repeat you offer to try it. -- Terry Jan Reedy From victor.stinner at gmail.com Fri Jun 30 05:06:28 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 30 Jun 2017 11:06:28 +0200 Subject: [Python-Dev] Bug in buildbot view (changes column) In-Reply-To: <20170630100557.1e0c8493@fsol> References: <20170630100557.1e0c8493@fsol> Message-ID: FYI there is an ongoing migration from the old patched Buildbot 0.8 to a vanilla Buildbot 0.9: https://github.com/python/buildmaster-config/pull/12 Sorry, I don't know much more. There are private discussion between 2 buildbot developers, Zachary Ware and me, but Zach is in vacation :-) We will try to keep you in touch (include you in the loop, or just try to discuss in public). Moreover, Zach also has a larger project: rewrite the buildbot configuration from scratch to use tags. But I don't know much more, again, you should discuss that with Zach when he will be back ;-) Victor 2017-06-30 10:05 GMT+02:00 Antoine Pitrou : > > Hello, > > If you look at this page (waterfall view of 3.x unstable buildbots), > you'll notice the "changes" column on the left lists the recent > changesets on master: > http://buildbot.python.org/all/waterfall?tag=3.x.unstable > > However, on the equivalent view for 3.x stable buildbots, the "changes" > column on the left is empty: > http://buildbot.python.org/all/waterfall?category=3.x.stable > > Is there a particular reason? > > Oh wait... The first URL lists "3.x.unstable" as a *tag* while the > second URL lists "3.x.stable" as a *category*. > > If I change the second URL to use "tag" instead of "category", the > "changes" column miraculously lists the recent changesets: > http://buildbot.python.org/all/waterfall?tag=3.x.stable > > Finicky buildbot :-/ > > Regards > > Antoine. > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com From victor.stinner at gmail.com Fri Jun 30 06:50:34 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 30 Jun 2017 12:50:34 +0200 Subject: [Python-Dev] New work-in-progress bisection tool for the Python test suite (in regrtest) In-Reply-To: References: Message-ID: Another example to bisect "Running test_warnings twice fails" bug: http://bugs.python.org/issue30812 --- haypo at selma$ ./python -m test.bisect -o bisect test_warnings test_warnings (...) Tests (1): * test.test_warnings._WarningsTests.test_showwarnmsg_missing (...) Bisection completed in 14 iterations and 0:00:14 --- So the bisection also works to find a failing test method when running a test file twice ;-) Victor 2017-06-28 2:39 GMT+02:00 Victor Stinner : > 2017-06-27 7:33 GMT+02:00 Serhiy Storchaka : >> You could make it just a submodule in the test package. >> >> ./python -m test.bisect -R 3:3 test_os > > I like the idea :-) I proposed a PR which was approved by Yury > Selivanov, and so I just merged it! It means that you can now play > with "./python -m test.bisect" in the master branch. Enjoy ;-) > Example: > > ./python -m test.bisect -R 3:3 test_multiprocessing_forkserver > > This command should give you the name of the two failing test methods > which are the last known reference leaks! > => http://bugs.python.org/issue30775 > > This specific bisection is very slow since running the 286 tests of > test_multiprocessing_forkserver using -R 3:3 takes longer than 3 > minutes on my laptop! And the answer is already know, see the bpo ;-) > > FYI, apart of the bpo-30775, all other reference leaks now seem to be > fixed on 2.7, 3.5, 3.6 and master branches! Tested on Windows and > Linux. > > Victor From victor.stinner at gmail.com Fri Jun 30 09:59:25 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 30 Jun 2017 15:59:25 +0200 Subject: [Python-Dev] Star the CPython GitHub project if you like Python! Message-ID: Hi, GitHub has a showcase page of hosted programming languages: https://github.com/showcases/programming-languages Python is only #11 with 8,539 stars, behind PHP and Ruby! Hey, you should "like" ("star"?) the CPython project if you like Python! https://github.com/python/cpython/ Click on "Star" at the top right. Thank you! Victor From robb at datalogics.com Fri Jun 30 10:10:34 2017 From: robb at datalogics.com (Rob Boehne) Date: Fri, 30 Jun 2017 14:10:34 +0000 Subject: [Python-Dev] (no subject) In-Reply-To: References: Message-ID: Victor, Thanks - I will comment on the issue WRT backporting the fix. If you have particular issues you?d like me to look into, just point me in the right direction. Thanks, Rob On 6/30/17, 2:29 AM, "Victor Stinner" wrote: >Hi Robb, > >2017-06-29 23:34 GMT+02:00 Rob Boehne : >> I?m new to the list, and contributing to Python specifically, and I?m >> interested in getting master and 3.6 branches building and working >> ?better? on UNIX. >> I?ve been looking at a problem building 3.6 on HP-UX and see a PR was >> merged into master, https://github.com/python/cpython/pull/1351 and I?d >> like to see it applied to 3.6. I?m happy to create a PR with a >> cherry-picked commit, and/or test. > >Sure, this change can be backported to 3.6, maybe also to 3.5. But >hum, I may suggest to first focus on the master branch and fix most >HP-UX issues, before spending time on backport. I would prefer to see >most tests of the important modules pass on HP-UX (ex: test_os). > >For a backport, you can directly comment http://bugs.python.org/issue30183 > >Victor From rymg19 at gmail.com Fri Jun 30 10:13:37 2017 From: rymg19 at gmail.com (rymg19 at gmail.com) Date: Fri, 30 Jun 2017 07:13:37 -0700 Subject: [Python-Dev] Star the CPython GitHub project if you like Python! In-Reply-To: <> Message-ID: I had tried to get people to star it after the GitHub migration, but I don't think too many did... FWIW Python's doing pretty well on GitHub considering it's only been there a couple of months. Only other project that did so well was Swift, and that was partly because it was previously closed-source. -- Ryan (????) Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone elsehttp://refi64.com On Jun 30, 2017 at 9:01 AM, > wrote: Hi, GitHub has a showcase page of hosted programming languages: https://github.com/showcases/programming-languages Python is only #11 with 8,539 stars, behind PHP and Ruby! Hey, you should "like" ("star"?) the CPython project if you like Python! https://github.com/python/cpython/ Click on "Star" at the top right. Thank you! Victor _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From status at bugs.python.org Fri Jun 30 12:09:20 2017 From: status at bugs.python.org (Python tracker) Date: Fri, 30 Jun 2017 18:09:20 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20170630160920.087DD11A882@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2017-06-23 - 2017-06-30) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 6006 (-20) closed 36565 (+95) total 42571 (+75) Open issues with patches: 2344 Issues opened (53) ================== #14094: ntpath.realpath() should use GetFinalPathNameByHandle() http://bugs.python.org/issue14094 reopened by eryksun #30744: Local variable assignment is broken when combined with threads http://bugs.python.org/issue30744 opened by njs #30747: _Py_atomic_* not actually atomic on Windows with MSVC http://bugs.python.org/issue30747 opened by Paxxi #30748: Upgrade tuples in socket to named tuple like http://bugs.python.org/issue30748 opened by Ben Lewis #30751: IDLE: Display entry errors for key sequence entry in entry box http://bugs.python.org/issue30751 opened by terry.reedy #30754: textwrap.dedent mishandles empty lines http://bugs.python.org/issue30754 opened by Julian #30755: locale.normalize() and getdefaultlocale() convert C.UTF-8 to e http://bugs.python.org/issue30755 opened by mattheww #30756: ttk: GUI tests fail on Ubuntu http://bugs.python.org/issue30756 opened by csabella #30757: pyinstaller can be added to docs, py2exe ref can be updated http://bugs.python.org/issue30757 opened by denfromufa #30758: regrtest hangs sometimes on the master branch (test_pydoc? tes http://bugs.python.org/issue30758 opened by haypo #30759: [2.7] Fix python2 -m test --list-cases test_multibytecodec_sup http://bugs.python.org/issue30759 opened by haypo #30760: configparse module in python3 can not write '%' to config file http://bugs.python.org/issue30760 opened by quanyechavshuo #30761: pdb: Add step / next count arguments http://bugs.python.org/issue30761 opened by louielu #30762: Misleading message ???can't concat bytes to str??? http://bugs.python.org/issue30762 opened by antoine.pietri #30766: Make CHECK_STATUS_PTHREAD signal-safe http://bugs.python.org/issue30766 opened by pitrou #30767: logging must check exc_info correctly http://bugs.python.org/issue30767 opened by krivushinme #30768: PyThread_acquire_lock_timed() should recompute the timeout whe http://bugs.python.org/issue30768 opened by haypo #30772: Normalise non-ASCII variable names in __all__ http://bugs.python.org/issue30772 opened by Nate Soares #30773: async generator receives wrong value when shared between corou http://bugs.python.org/issue30773 opened by Dima.Tisnek #30774: list_repr not safe against concurrent mutation http://bugs.python.org/issue30774 opened by pitrou #30777: IDLE: configdialog -- add docstrings and improve comments http://bugs.python.org/issue30777 opened by terry.reedy #30778: test_bsddb3 crash on x86 Windows XP 2.7 http://bugs.python.org/issue30778 opened by haypo #30779: IDLE: configdialog -- factor out Changes class http://bugs.python.org/issue30779 opened by terry.reedy #30780: IDLE: configdialog - add tests for ConfigDialog GUI. http://bugs.python.org/issue30780 opened by terry.reedy #30781: IDLE: configdialog -- switch to ttk widgets. http://bugs.python.org/issue30781 opened by terry.reedy #30782: Allow limiting the number of concurrent tasks in asyncio.as_co http://bugs.python.org/issue30782 opened by andybalaam #30784: IDLE: separate editor window and text http://bugs.python.org/issue30784 opened by louielu #30786: getaddrinfo emulation does not support AI_NUMERICSERV http://bugs.python.org/issue30786 opened by smejkar #30788: email.policy.SMTP.fold() issue for long filenames with spaces http://bugs.python.org/issue30788 opened by jhillacre #30789: Redesign PyCodeObject.co_extras to use a single memory block, http://bugs.python.org/issue30789 opened by haypo #30790: Can't use proxy to connect internet on windows http://bugs.python.org/issue30790 opened by Cordius #30791: tkinter.Tk() adds suffix to window class name when launching m http://bugs.python.org/issue30791 opened by hakonhagland #30792: Add contextlib.convert_exception manager http://bugs.python.org/issue30792 opened by steven.daprano #30793: Parsing error on f-string-expressions containing strings with http://bugs.python.org/issue30793 opened by Anselm Kiefner #30794: Add multiprocessing.Process.kill() http://bugs.python.org/issue30794 opened by pitrou #30795: OS X failures in test_site http://bugs.python.org/issue30795 opened by pitrou #30797: ./pyconfig.h:1438:0: warning: "_GNU_SOURCE" redefined [enabled http://bugs.python.org/issue30797 opened by Segev Finer #30798: Document that subprocess.Popen does not set PWD http://bugs.python.org/issue30798 opened by jamesdlin #30799: Improved test coverage Lib/_osx_support.py 99% http://bugs.python.org/issue30799 opened by chexex #30800: zlib.compressobj took too much memory on window http://bugs.python.org/issue30800 opened by dennisding #30801: shoutdown process error with python 3.4 and pyqt/PySide http://bugs.python.org/issue30801 opened by alberfontan1 #30802: datetime.datetime.strptime('200722', '%Y%U') http://bugs.python.org/issue30802 opened by winton #30803: Truth value of sets not properly documented http://bugs.python.org/issue30803 opened by thomassen #30804: bolen-dmg-3.x build-installer.py failed http://bugs.python.org/issue30804 opened by haypo #30805: asyncio: race condition with debug and subprocess http://bugs.python.org/issue30805 opened by wg #30806: netrc.__repr__() is broken for writing to file http://bugs.python.org/issue30806 opened by Bezier89 #30808: Use _Py_atomic API for concurrency-sensitive signal state http://bugs.python.org/issue30808 opened by pitrou #30809: IDLE parenmatch - highlighting options http://bugs.python.org/issue30809 opened by wohlganger #30811: A venv created and activated from within a virtualenv uses the http://bugs.python.org/issue30811 opened by Antony.Lee #30814: Import dotted name as alias breaks with concurrency http://bugs.python.org/issue30814 opened by Samuele Santi2 #30816: test_open() of test_eintr timeout after 10 min on "x86-64 El C http://bugs.python.org/issue30816 opened by haypo #30817: Abort in PyErr_PrintEx() when no memory http://bugs.python.org/issue30817 opened by xdegaye #30818: Warning -- asyncore.socket_map was modified by test_ftplib on http://bugs.python.org/issue30818 opened by haypo Most recent 15 issues with no replies (15) ========================================== #30816: test_open() of test_eintr timeout after 10 min on "x86-64 El C http://bugs.python.org/issue30816 #30814: Import dotted name as alias breaks with concurrency http://bugs.python.org/issue30814 #30811: A venv created and activated from within a virtualenv uses the http://bugs.python.org/issue30811 #30806: netrc.__repr__() is broken for writing to file http://bugs.python.org/issue30806 #30805: asyncio: race condition with debug and subprocess http://bugs.python.org/issue30805 #30804: bolen-dmg-3.x build-installer.py failed http://bugs.python.org/issue30804 #30801: shoutdown process error with python 3.4 and pyqt/PySide http://bugs.python.org/issue30801 #30800: zlib.compressobj took too much memory on window http://bugs.python.org/issue30800 #30799: Improved test coverage Lib/_osx_support.py 99% http://bugs.python.org/issue30799 #30798: Document that subprocess.Popen does not set PWD http://bugs.python.org/issue30798 #30797: ./pyconfig.h:1438:0: warning: "_GNU_SOURCE" redefined [enabled http://bugs.python.org/issue30797 #30794: Add multiprocessing.Process.kill() http://bugs.python.org/issue30794 #30791: tkinter.Tk() adds suffix to window class name when launching m http://bugs.python.org/issue30791 #30790: Can't use proxy to connect internet on windows http://bugs.python.org/issue30790 #30789: Redesign PyCodeObject.co_extras to use a single memory block, http://bugs.python.org/issue30789 Most recent 15 issues waiting for review (15) ============================================= #30817: Abort in PyErr_PrintEx() when no memory http://bugs.python.org/issue30817 #30808: Use _Py_atomic API for concurrency-sensitive signal state http://bugs.python.org/issue30808 #30747: _Py_atomic_* not actually atomic on Windows with MSVC http://bugs.python.org/issue30747 #30714: test_ssl fails with openssl 1.1.0f http://bugs.python.org/issue30714 #30711: getaddrinfo invalid port number http://bugs.python.org/issue30711 #30710: getaddrinfo raises OverflowError http://bugs.python.org/issue30710 #30696: infinite loop in PyRun_InteractiveLoopFlags() http://bugs.python.org/issue30696 #30695: add a nomemory_allocator to the _testcapi module http://bugs.python.org/issue30695 #30693: tarfile add uses random order http://bugs.python.org/issue30693 #30688: support named Unicode escapes (\N{name}) in re http://bugs.python.org/issue30688 #30680: textwrap should treat Unicode em-dash like ASCII em-dash http://bugs.python.org/issue30680 #30665: pass big values for arg to fcntl.ioctl http://bugs.python.org/issue30665 #30593: sqlite3 executescript does not respect isolation_level? http://bugs.python.org/issue30593 #30560: Py_SetFatalErrorAbortFunc: Allow embedding program to handle f http://bugs.python.org/issue30560 #30541: Add restricted mocks to the python unittest mocking framework http://bugs.python.org/issue30541 Top 10 most discussed issues (10) ================================= #24813: Redesign Help => About IDLE, make it non-modal http://bugs.python.org/issue24813 21 msgs #30703: Non-reentrant signal handler (test_multiprocessing_forkserver http://bugs.python.org/issue30703 18 msgs #29926: IDLE: in shell, time.sleep ignores _thread.interrupt_main() http://bugs.python.org/issue29926 16 msgs #30744: Local variable assignment is broken when combined with threads http://bugs.python.org/issue30744 13 msgs #30422: Add roadmap.txt section to idlelib http://bugs.python.org/issue30422 11 msgs #21998: asyncio: support fork http://bugs.python.org/issue21998 9 msgs #30730: [security] Injecting environment variable in subprocess on Win http://bugs.python.org/issue30730 9 msgs #29988: (async) with blocks and try/finally are not as KeyboardInterru http://bugs.python.org/issue29988 8 msgs #30302: Improve .__repr__ implementation for datetime.timedelta http://bugs.python.org/issue30302 7 msgs #30726: [Windows] Warnings in elementtree due to new expat http://bugs.python.org/issue30726 7 msgs Issues closed (87) ================== #10909: IDLE: thread hang, possibly related to print http://bugs.python.org/issue10909 closed by terry.reedy #10972: zipfile: add "unicode" option to the force the filename encodi http://bugs.python.org/issue10972 closed by haypo #12215: TextIOWrapper: issues with interlaced read-write http://bugs.python.org/issue12215 closed by haypo #12512: codecs: StreamWriter issues with stateful codecs after a seek http://bugs.python.org/issue12512 closed by haypo #12513: codec.StreamReaderWriter: issues with interlaced read-write http://bugs.python.org/issue12513 closed by haypo #13552: Compilation issues of the curses module on Solaris and OpenInd http://bugs.python.org/issue13552 closed by haypo #13617: Reject embedded null characters in wchar* strings http://bugs.python.org/issue13617 closed by serhiy.storchaka #19150: IDLE shell fails: "ModifiedInterpreter instance has no attribu http://bugs.python.org/issue19150 closed by terry.reedy #20615: Replace PyDict_GetItem() with PyDict_GetItemWithError() http://bugs.python.org/issue20615 closed by haypo #20718: OpenBSD/AIX: tests passing a file descriptor with sendmsg/recv http://bugs.python.org/issue20718 closed by haypo #21080: asyncio.subprocess: connect pipes of two programs http://bugs.python.org/issue21080 closed by haypo #21519: IDLE : Bug in keybinding validity check http://bugs.python.org/issue21519 closed by terry.reedy #22207: Test for integer overflow on Py_ssize_t: explicitly cast to si http://bugs.python.org/issue22207 closed by haypo #22271: Deprecate PyUnicode_AsUnicode(): emit a DeprecationWarning http://bugs.python.org/issue22271 closed by haypo #23587: asyncio: use the new traceback.TracebackException class http://bugs.python.org/issue23587 closed by haypo #23763: Chain exceptions in C http://bugs.python.org/issue23763 closed by haypo #24598: asyncio: add background task detecting reference cycles http://bugs.python.org/issue24598 closed by haypo #24999: Segfault in test_re.test_sre_character_class_literals() when P http://bugs.python.org/issue24999 closed by haypo #25606: asyncio doc: 'socket' transport extra info is not mandatory http://bugs.python.org/issue25606 closed by haypo #25796: Running test_multiprocessing_spawn is slow (more than 8 minute http://bugs.python.org/issue25796 closed by haypo #25868: test_eintr.test_sigwaitinfo() hangs on "AMD64 FreeBSD CURRENT http://bugs.python.org/issue25868 closed by haypo #26373: asyncio: add support for async context manager on streams? http://bugs.python.org/issue26373 closed by haypo #26643: regrtest: rework libregrtest.save_env submodule http://bugs.python.org/issue26643 closed by haypo #26757: test_urllib2net.test_http_basic() timeout after 15 min on http://bugs.python.org/issue26757 closed by haypo #27103: regrtest: capture stdout (-W) option is incompatible with refl http://bugs.python.org/issue27103 closed by haypo #27847: os.set_inheritable() looks to be broken on OpenIndiana, regres http://bugs.python.org/issue27847 closed by haypo #27940: xml.etree: Avoid XML declaration for the "ascii" encoding http://bugs.python.org/issue27940 closed by haypo #28223: test_tools fails with timeout on AMD64 Snow Leop 3.x buildbot http://bugs.python.org/issue28223 closed by haypo #28224: Compilation warnings on Windows: export 'PyInit_xx' specified http://bugs.python.org/issue28224 closed by haypo #28232: asyncio: wrap_future() doesn't handle cancellation correctly http://bugs.python.org/issue28232 closed by haypo #28554: Windows: _socket module fails to compile on "AMD64 Windows7 SP http://bugs.python.org/issue28554 closed by haypo #28572: IDLE: add tests for config dialog. http://bugs.python.org/issue28572 closed by terry.reedy #28711: IDLE doesn't open http://bugs.python.org/issue28711 closed by terry.reedy #29461: Experiment usage of likely/unlikely in CPython core http://bugs.python.org/issue29461 closed by haypo #29512: regrtest refleak: implement bisection feature http://bugs.python.org/issue29512 closed by haypo #29530: Windows buildbots broken by the migration to GitHub (meta issu http://bugs.python.org/issue29530 closed by haypo #29585: site.py imports relatively large `sysconfig` module. http://bugs.python.org/issue29585 closed by inada.naoki #29910: Ctrl-D eats a character on IDLE http://bugs.python.org/issue29910 closed by terry.reedy #30038: Race condition in how trip_signal writes to wakeup fd http://bugs.python.org/issue30038 closed by njs #30062: datetime in Python 3.6+ no longer respects 'TZ' environment va http://bugs.python.org/issue30062 closed by ammar2 #30172: test_tools takes longer than 5 minutes on some buildbots http://bugs.python.org/issue30172 closed by haypo #30263: regrtest: log the system load? http://bugs.python.org/issue30263 closed by haypo #30280: Warning -- threading._dangling was modified by test_asyncio on http://bugs.python.org/issue30280 closed by haypo #30314: Buildbots: 15 min is too low for test_tools on x86 Tiger 3.6 b http://bugs.python.org/issue30314 closed by haypo #30329: poplib and imaplib should catch "OSError: [WinError 10022] An http://bugs.python.org/issue30329 closed by haypo #30362: Launcher add list and list with paths options http://bugs.python.org/issue30362 closed by steve.dower #30368: [2.7] OpenSSL compilation fails on AMD64 Windows7 SP1 VS9.0 2. http://bugs.python.org/issue30368 closed by haypo #30523: regrtest: add --list-cases option to only display test case id http://bugs.python.org/issue30523 closed by haypo #30542: test_files() of test_tools.test_unparse.DirectoryTestCase leak http://bugs.python.org/issue30542 closed by haypo #30596: Add close() to multiprocessing.Process http://bugs.python.org/issue30596 closed by pitrou #30599: test_threaded_import leaks references http://bugs.python.org/issue30599 closed by haypo #30645: imp.py: load_package() appends to its own loop variable http://bugs.python.org/issue30645 closed by brett.cannon #30647: CODESET error on AMD64 FreeBSD 10.x Shared 3.x caused by the P http://bugs.python.org/issue30647 closed by ncoghlan #30664: Change unittest's _SubTest to not sort its params when printin http://bugs.python.org/issue30664 closed by serhiy.storchaka #30673: Add -t option (timeout) to Tools/buildbot/test.bat for "AMD64 http://bugs.python.org/issue30673 closed by haypo #30674: IDLE: add docstrings to grep.py http://bugs.python.org/issue30674 closed by terry.reedy #30675: test_zipfile leaks references on Python 3.5 on the wo Refleaks http://bugs.python.org/issue30675 closed by haypo #30704: test_free_different_thread() of test_code leaks references on http://bugs.python.org/issue30704 closed by haypo #30705: python2.7 -m test -R 3:3 hangs on Refleaks 2.7 buildbots http://bugs.python.org/issue30705 closed by haypo #30706: EmailMessage Object Creation http://bugs.python.org/issue30706 closed by r.david.murray #30708: Ensure that the result of PyUnicode_AsWideCharString() doesn't http://bugs.python.org/issue30708 closed by serhiy.storchaka #30723: IDLE parenmatch - left and right highlighting, independent fla http://bugs.python.org/issue30723 closed by terry.reedy #30733: Typo in Document What's New: Calendar http://bugs.python.org/issue30733 closed by doerwalter #30734: 200000 indexes crashes eval and python (without eval) http://bugs.python.org/issue30734 closed by serhiy.storchaka #30742: VS2015 support for 2.7 branch http://bugs.python.org/issue30742 closed by steve.dower #30743: unittest discover does not mention module file must be named w http://bugs.python.org/issue30743 closed by alex.75 #30745: Warnings in Modules/_winapi.c http://bugs.python.org/issue30745 closed by serhiy.storchaka #30746: Reject environment variable names containing '=' http://bugs.python.org/issue30746 closed by serhiy.storchaka #30749: Non-atomic and unusual (wrong) rename behavior under OS X http://bugs.python.org/issue30749 closed by Alex Groce #30750: Update `make patchcheck` to understand Misc/NEWS.d http://bugs.python.org/issue30750 closed by brett.cannon #30752: Basic subtraction is wrong ( 1.83 - 1.52 == 0.3100000000000000 http://bugs.python.org/issue30752 closed by mark.dickinson #30753: not able to execute print command on page 16 of documentation http://bugs.python.org/issue30753 closed by steven.daprano #30763: There is functionality bug in linecache library. http://bugs.python.org/issue30763 closed by gvanrossum #30764: regrtest: Add --fail-env-changed option http://bugs.python.org/issue30764 closed by haypo #30765: PyThread_acquire_lock can block even when asked not ot http://bugs.python.org/issue30765 closed by pitrou #30769: [EASY (C)] test_execve_invalid_env() of test_os leaks referenc http://bugs.python.org/issue30769 closed by haypo #30775: test_multiprocessing_forkserver leaks references on Python 3 http://bugs.python.org/issue30775 closed by pitrou #30776: regrtest: change -R/--huntrleaks rule to decide if a test leak http://bugs.python.org/issue30776 closed by haypo #30783: Spawned subprocesses don't respect environment http://bugs.python.org/issue30783 closed by eryksun #30785: ast.c duplicates STR(CHILD) http://bugs.python.org/issue30785 closed by emilyemorehouse #30787: Please add a comprehensive index of decorators to the document http://bugs.python.org/issue30787 closed by brett.cannon #30796: Failures/crashes in test_signal on some buildbots http://bugs.python.org/issue30796 closed by pitrou #30807: setitimer() can disable timer by mistake http://bugs.python.org/issue30807 closed by pitrou #30810: Germany made the upper case ?? official. '??'.upper() should n http://bugs.python.org/issue30810 closed by haypo #30812: Running test_warnings twice fails http://bugs.python.org/issue30812 closed by haypo #30813: test_unittest fails when hunting reference leaks http://bugs.python.org/issue30813 closed by haypo #30815: StringIO uses inefficient PyUnicode_AsUCS4 http://bugs.python.org/issue30815 closed by inada.naoki