From victor.stinner at gmail.com  Sun Sep  1 00:24:30 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sun, 1 Sep 2013 00:24:30 +0200
Subject: [Python-Dev] Add a new tracemalloc module to trace memory
	allocations
In-Reply-To: <CAGE7PNJ9DrKQ8Zb-E-F7QPRTP=t+S8StBvm3OQgvct8539gFPg@mail.gmail.com>
References: <CAMpsgwaa3pABNrMSw2S7-Og111u8DdXK6dYnWVJzS4TAKbFNEQ@mail.gmail.com>
 <CAMpsgwatDdFQg4q=b3s7W6BZ+WQmpo=KTeJZd7ESbuGG+RzaKA@mail.gmail.com>
 <CAGE7PNJ9DrKQ8Zb-E-F7QPRTP=t+S8StBvm3OQgvct8539gFPg@mail.gmail.com>
Message-ID: <CAMpsgwZBeN7W+GK6hLSHp4VYXAYgWFQBTx_-K3J7KzOMDB3H=g@mail.gmail.com>

Le 31 ao?t 2013 19:09, "Gregory P. Smith" <greg at krypto.org> a ?crit :
> First, I really like this.  +1

Current votes: +3 (i also want tracemalloc!). No opposition against such
addition.

> We should be consistent with faulthandler's options. Why do you not want
to support both the env var and enable()/disable() functions?

The reason was technical but I have issues with enabling tracemalloc before
Python is fully initialized. Enabling tracemalloc when Python is almost
initialized and disabling it before Python exit is more reliable.

In the last implementation, enable() and disable() are back.
PYTHONTRACEMALLOC is still available. I will also add -X tracemalloc
command line.

> Users are likely to want snapshots captured by enable()/disable() around
particular pieces of code just as much as whole program information.

enable()/disable() are useful for tests of the tracemalloc module!

> Taking that further: file and line information is great, but what if you
extend the concept: could you allow for C API or even Python hooks to
gather additional information at the time of each allocation or free? for
example: Gathering the actual C and Python stack traces for correlation to
figure out what call patterns lead allocations is powerful.

There is no portable function to retrieve the C traceback.

For the Python traceback: it may be possible to get it but you have to
remember that the hook is on PyMem_Malloc(), a low level memory allocator.
The GIL is hold. It is possible to call Python code in some cases, but they
are corner cases where it would lead to a loop, deadlock or worse.

I prefer to only read available Python data without calling any Python code
and try to write a reliable debug tool. I still have some issues like
issues with reentrant calls to the hook, but I'm working on them.

A nice improvement compared to the implementation on PyPI would be to hook
PyMem_RawMalloc(), to trace also the memory used by gzip, bz2, lzma and
other C modules. Locking the GIL in PyMem_RawMalloc() to get the filename
and locking internals tracemalloc structures causes also new issues (like a
funny deadlock related to subinterpreter).

> (Yes, this gets messy fast as hooks should not trigger calls back into
themselves when they allocate or free, similar to the "fun" involved in
writing coverage tools)

tracemalloc uses a "reentrant" variable to do nothing on a reentrant call
to the hook.

> let me know if you think i'm crazy. :)

You are not crazy. It is just hard to implement it. I'm not sure that it is
possible.

Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130901/d9caffd6/attachment.html>

From ncoghlan at gmail.com  Sun Sep  1 03:28:36 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 1 Sep 2013 11:28:36 +1000
Subject: [Python-Dev] Pre-PEP: Redesigning extension modules
In-Reply-To: <kvtfek$23p$1@ger.gmane.org>
References: <kv77o2$f96$1@ger.gmane.org> <20130823111822.49cba700@pitrou.net>
 <CADiSq7dhzpTWMw7gcBRrzZot7jaS=Aj=aukq0+jh_cK2+F_kGQ@mail.gmail.com>
 <CADiSq7dBruZiWhObnjKbcneji9+ziXmyRpg1+72a8YeMWbyO3A@mail.gmail.com>
 <kvabt8$jfc$1@ger.gmane.org>
 <CADiSq7c6OjFa8zo7sMB=k2zLSmaf1w-tAwhYKSQ26T3caLyj6A@mail.gmail.com>
 <kvaka3$sbu$1@ger.gmane.org>
 <CADiSq7dzQqyH9i6g9QYWp6_ZHHUD4+4npYo6of6mCNahQ+w8kw@mail.gmail.com>
 <kvcr9f$4gs$1@ger.gmane.org>
 <CADiSq7cfNH1M29W19BW88-BBQaWEpaMkE=s09wNyT5VGOrPB+g@mail.gmail.com>
 <kvtfek$23p$1@ger.gmane.org>
Message-ID: <CADiSq7d3L1+Jc5E4dWYDRUQPNDCmxqyc3GzbgXJvGMe4+OnWrQ@mail.gmail.com>

On 1 Sep 2013 05:18, "Stefan Behnel" <stefan_ml at behnel.de> wrote:
>
> Nick Coghlan, 31.08.2013 18:49:
> > On 25 Aug 2013 21:56, "Stefan Behnel" wrote:
> >>>>> One key point to note is that it *doesn't* call
> >>>>> _PyImport_FixupExtensionObject, which is the API that handles all
the
> >>>>> PEP 3121 per-module state stuff. Instead, the idea will be for
modules
> >>>>> that don't need additional C level state to just implement
> >>>>> PyImportExec_NAME, while those that *do* need C level state
implement
> >>>>> PyImportCreate_NAME and return a custom object (which may or may not
> >>>>> be a module subtype).
> >>>>
> >>>> Is it really a common case for an extension module not to need any C
> >>>> level
> >>>> state at all? I mean, this might work for very simple accelerator
> >>>> modules
> >>>> with only a few stand-alone functions. But anything non-trivial will
> >>>> almost
> >>>> certainly have some kind of global state, cache, external library,
> >>>> etc.,
> >>>> and that state is best stored at the C level for safety reasons.
> >
> > In my experience, most extension authors aren't writing high
performance C
> > accelerators, they're exposing an existing C API to Python. It's the
cffi
> > use case rather than the Cython use case.
>
> Interesting. I can't really remember a case where I could afford the
> runtime overhead of implementing a wrapper in Python and going through
> something like ctypes or cffi. I mean, testing C libraries with Python
> tools would be one, but then, you wouldn't want to write an extension
> module for that and instead want to call it directly from the test code as
> directly as possible.
>
> I'm certainly aware that that use case exists, though, and also the case
of
> just wanting to get things done as quickly and easily as possible.

Keep in mind I first came to Python as a tool for test automation of custom
C++ hardware APIs that could be written to be SWIG friendly.

I now work for an OS vendor where the 3 common languages for system
utilities are C, C++ and Python.

For those use cases, dropping a bunch of standard Python objects in a
module dict is often going to be a quick and easy solution that avoids a
lot of nasty pointer lifecycle issues at the C level.

This style of extension code would suffer similar runtime checking overhead
as Python, including for function calls, but, like CPython itself, would
still often be "fast enough".

However, as soon as you want to manually optimise for *speed* at all,
you're going to want to remove those module internal indirections through
the Python API. There are at least three ways to do this (internally,
CPython uses all of them in various places):

* type checks followed by direct function calls on the optimised path and
falling back to the abstract object APIs on the compatibility path
* type checks followed by an exception for unknown types
* hidden state that isn't exposed directly at the Python level and hence
can be trusted to only be changed through the module APIs.

The third approach can be implemented in three ways, with various
consequences:

* C static variables. For mutable state, including pointers to Python
types, this breaks subinterpreters, reloading in place and loading a fresh
copy of the module
* PEP 3121 per-interpreter shared state. Handles subinterpreters, *may*
handle reloading (but may segfault if references are held to old types and
functions from before the reload), doesn't handle loading a fresh copy at
all.
* PEP 3121 with a size of "0". As above, but avoids the module state APIs
in order to support reloading. All module state (including type
cross-references) is stored in hidden state (e.g. an instance of a custom
type not exposed to Python, with a reference stored on each custom type
object defined in the module, and any module level "functions" actually
being methods of a hidden object). Still doesn't support loading a *fresh*
copy due to the hidden PEP 3121 module cache.

The proposed new approach is to bypass the PEP 3121 cache entirely, and
instead recommend providing an instance of a custom type to be placed in
sys.modules. Extension modules will be given the ability to explicitly
disallow in-place reloading *or* to make it work reliably, rather than the
status quo where the import system assumes it will work, and instead may
fail in unusual ways.

> > Mutable module global state is always a recipe for obscure bugs, and not
> > something I will ever let through code review without a really good
> > rationale. Hidden process global state is never good, just sometimes a
> > necessary evil.
>
> I'm not necessarily talking about mutable state. Rather about things like
> pre-initialised data or imported functionality. For example, I often have
a
> bound method of a compiled regex lying around somewhere in my Python
> modules as a utility function. And the same kind of stuff exists in C
code,
> some may be local to a class, but other things can well be module global.
> And given that we are talking about module internals here I'd always keep
> them at the C level rather than exposing them through the module dict. The
> module dict involves a much higher access overhead, in addition to the
> reduced safety due to user accessibility.
>
> Exported C-APIs are also a use case. You'd import the C-API of another
> module at init time and from that point on only go through function
> pointers etc. Those are (sub-)interpreter specific, i.e. they are module
> global state that is specific to the currently loaded module instances.

Due to refcounting, all instances of Python objects qualify as mutable
state.

Hopefully my elaboration above helps make it clear why I think it's
worthwhile to clearly separate out the "no custom C level state needed"
case.

> > However, keep in mind my patch is currently just the part I can
implement
> > without PEP 451 module spec objects.
>
> Understood.
>
>
> >> Note that even global functions usually hold state, be it in the form
of
> >> globally imported modules, global caches, constants, ...
> >
> > If they can be shared safely across multiple instances of the module
(e.g.
> > immutable constants), then these can be shared at the C level.
Otherwise, a
> > custom Python type will be needed to make them instance specific.
>
> I assume you meant a custom module (extension) type here.

Not sure yet. For PEP 451, we still need to support arbitrary objects in
sys.modules, so it's still possible that freedom will be made available to
extension modules.

>
> Just to be clear, the "module state at the C-level" is meant to be stored
> in the object struct fields of the extension type that implements the
> module, at least for modules that want to support reloading and
> sub-interpreters. Obviously, nothing should be stored in static (global)
> variables etc.

Right.

> >>> We also need the create/exec split to properly support reloading.
Reload
> >>> *must* reinitialize the object already in sys.modules instead of
> >>> inserting
> >>> a different object or it completely misses the point of reloading
> >>> modules
> >>> over deleting and reimporting them (i.e. implicitly affecting the
> >>> references from other modules that imported the original object).
> >>
> >> Interesting. I never thought of it that way.
> >>
> >> I'm not sure this can be done in general. What if the module has
threads
> >> running that access the global state? In that case, reinitialising the
> >> module object itself would almost certainly lead to a crash.

That's why I want a way for loaders in general (and extension modules in
particular) to clearly say "in-place reloading not supported", rather than
Python blundering ahead with it and risking a crash.

> > My current proposal on import-sig is to make the first hook
> > "prepare_module", and pass in the existing object in the reload case.
For
> > the extension loader, this would be reflected in the signature of the C
> > level hook as well, so the module could decide for itself if it
supported
> > reloading.
>
> I really don't like the idea of reloading by replacing module state. It
> would be much simpler if the module itself would be replaced, then the
> original module could stay alive and could still be used by those who hold
> a reference to it or parts of its contents. Especially the from-import
case
> would benefit from this. Obviously, you could still run into obscure bugs
> where a function you call rejects the input because it expects an older
> version of a type, for example. But I can't see that being worse (or even
> just different) from the reload-by-refilling-dict case.

Sure, this is what we do in the test suite in
"test.support.import_fresh_module". It was actually Eli trying to use that
in the etree tests that triggered our recent investigation of the limits of
PEP 3121 (it breaks for stateful extension modules due to the
per-interpreter caching).

It's a different operation from imp.reload, though. Assuming we can get
this stable and reliable in the new API, I expect we'll be able to add
"imp.reload_fresh" as a supported API in 3.5.

> You seemed to be ok with my idea of making the loader return a wrapped
> extension module instead of the module itself. We should actually try
that.

Sure, that's just a variant of the "hidden state object" idea I described
above. It should actually work today with the PEP 3121 custom storage size
set to zero.

> > This is actually my primary motivation for trying to improve the
> > "can this be reloaded or not?" aspects of the loader API in PEP 451.
>
> I assume you mean that the extension module would be able to clearly
signal
> that it can't be reloaded, right? I agree that that's helpful. If you're
> wrapping a C library, then the way that library is implemented might
simply
> force you to prevent any attempts at reloading the wrapper module. But if
> reloading is possible at all, it would be even more helpful if we could
> make it really easy to properly support it.

Yep, that's my goal (and why it's really good to be having this discussion
while PEP 451 is still in development).

>
>
> > (keep in mind existing extension modules using the existing API will
still
> > never be reloaded)
>
> Sure, that's the cool thing. We can really design this totally from
scratch
> without looking back.

Well, not *quite*. We need to ensure a module can implement both APIs can
coexist in the same module for source compatibility without nasty ifdef
hacks, and that there is a reasonable migration path for existing
handwritten extension modules.

> >>> Take a look at the current example - everything gets stored in the
> >>> module dict for the simple case with no C level global state.
> >>
> >> Well, you're storing types there. And those types are your module API.
I
> >> understand that it's just an example, but I don't think it matches a
> >> common
> >> case. As far as I can see, the types are not even interacting with each
> >> other, let alone doing any C-level access of each other. We should try
to
> >> focus on the normal case that needs C-level state and C-level field
access
> >> of extension types. Once that's solved, we can still think about how to
> >> make the really simple cases simpler, if it turns out that they are not
> >> simple enough.
> >
> > Our experience is very different - my perspective is that the normal
case
> > either eschews C level global state in the extension module, because it
> > causes so many problems, or else just completely ignores subinterpreter
> > support and proper module cleanup.
>
> As soon as you have more than one extension type in your module, and they
> interact with each other, they will almost certainly have to do type
checks
> against each other to make sure users haven't passed them rubbish before
> they access any C struct fields of the object. Doing a type check means
> that at least one type has a pointer to the other, meaning that it holds
> global module state.

Sure, but you can use the CPython API rather than writing normal C code. We
do this fairly often in CPython when we're dealing with things stored in
modules that can be manipulated from Python.

It incurs CPython's dynamic dispatch overhead, but sometimes that's worth
it to avoid needing to deal with C level lifecycle issues.

> I really think that having some kind of global module state is the
> exceedingly common case for an extension module.

I wouldn't be willing to make the call about which of stateless vs stateful
is more common without a lot more research :)

They're both common enough that I think they should both be well supported,
and making the "no custom C level state" case as simple as possible.

> >> I didn't know about PyType_FromSpec(), BTW. It looks like a nice
addition
> >> for manually written code (although useless for Cython).
> >
> > This is the only way to create custom types when using the stable ABI.
Can
> > I take your observation to mean that Cython doesn't currently offer the
> > option of limiting itself to the stable ABI?
>
> Correct. I've taken a bird's view at it back then, and keep stumbling over
> "wow - I couldn't even use that?" kind of declarations in the header
files.
> I don't think it makes sense for Cython. Existing CPython versions are
easy
> to support because they don't change anymore, and new major releases most
> likely need adaptations anyway, if only to adapt to new features and
> performance changes. Cython actually knows quite a lot about the inner
> workings of CPython and its various releases. Going only through the
stable
> ABI parts of the C-API would make the code horribly slow in comparison, so
> there are huge drawbacks for the benefit it might give.
>
> The Cython way of doing it is more like: you want your code to run on a
new
> CPython version, then use a recent Cython release to compile it. It may
> still work with older ones, but what you actually want is the newest
> anyway, and you also want to compile the C code for the specific CPython
> version at hand to get the most out of it. It's the C code that adapts,
not
> the runtime code (or Cython itself).
>
> We run continuous integration tests with all of CPython's development
> branches since 2.4, so we usually support new CPython releases long before
> they are out. And new releases of CPython rarely affect Cython user code.

The main advantage of the stable ABI is being able to publish cross-version
binary extension modules. I guess if Cython already supports generating
binaries for each new version of CPython before we release it, that
capability is indeed less useful than it is for those that are maintaining
extension modules by hand.

Cheers,
Nick.

>
> Stefan
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130901/aa8d7cd8/attachment.html>

From ncoghlan at gmail.com  Sun Sep  1 03:36:51 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 1 Sep 2013 11:36:51 +1000
Subject: [Python-Dev] Add a new tracemalloc module to trace memory
	allocations
In-Reply-To: <CAMpsgwZBeN7W+GK6hLSHp4VYXAYgWFQBTx_-K3J7KzOMDB3H=g@mail.gmail.com>
References: <CAMpsgwaa3pABNrMSw2S7-Og111u8DdXK6dYnWVJzS4TAKbFNEQ@mail.gmail.com>
 <CAMpsgwatDdFQg4q=b3s7W6BZ+WQmpo=KTeJZd7ESbuGG+RzaKA@mail.gmail.com>
 <CAGE7PNJ9DrKQ8Zb-E-F7QPRTP=t+S8StBvm3OQgvct8539gFPg@mail.gmail.com>
 <CAMpsgwZBeN7W+GK6hLSHp4VYXAYgWFQBTx_-K3J7KzOMDB3H=g@mail.gmail.com>
Message-ID: <CADiSq7dKojZ5f1bprhZZxu6DjovjWtwy-8o=-BYjYRvHZWqoFw@mail.gmail.com>

+1 from me for both tracemalloc and failmalloc in the same vein as
faulthandler (and using similar API concepts to faulthandler).

While I like the flat top level naming structure, we should clearly
document these as implementation dependent runtime services. Other
implementations may not provide them at all, and even if they do, many
details will likely be different.

The gc module may even fall into the same category.

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130901/4bed69d1/attachment.html>

From stefan_ml at behnel.de  Sun Sep  1 10:11:36 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 01 Sep 2013 10:11:36 +0200
Subject: [Python-Dev] Pre-PEP: Redesigning extension modules
In-Reply-To: <CADiSq7d3L1+Jc5E4dWYDRUQPNDCmxqyc3GzbgXJvGMe4+OnWrQ@mail.gmail.com>
References: <kv77o2$f96$1@ger.gmane.org> <20130823111822.49cba700@pitrou.net>
 <CADiSq7dhzpTWMw7gcBRrzZot7jaS=Aj=aukq0+jh_cK2+F_kGQ@mail.gmail.com>
 <CADiSq7dBruZiWhObnjKbcneji9+ziXmyRpg1+72a8YeMWbyO3A@mail.gmail.com>
 <kvabt8$jfc$1@ger.gmane.org>
 <CADiSq7c6OjFa8zo7sMB=k2zLSmaf1w-tAwhYKSQ26T3caLyj6A@mail.gmail.com>
 <kvaka3$sbu$1@ger.gmane.org>
 <CADiSq7dzQqyH9i6g9QYWp6_ZHHUD4+4npYo6of6mCNahQ+w8kw@mail.gmail.com>
 <kvcr9f$4gs$1@ger.gmane.org>
 <CADiSq7cfNH1M29W19BW88-BBQaWEpaMkE=s09wNyT5VGOrPB+g@mail.gmail.com>
 <kvtfek$23p$1@ger.gmane.org>
 <CADiSq7d3L1+Jc5E4dWYDRUQPNDCmxqyc3GzbgXJvGMe4+OnWrQ@mail.gmail.com>
Message-ID: <kvusrg$2vm$1@ger.gmane.org>

Nick Coghlan, 01.09.2013 03:28:
> On 1 Sep 2013 05:18, "Stefan Behnel" wrote:
>> I can't really remember a case where I could afford the
>> runtime overhead of implementing a wrapper in Python and going through
>> something like ctypes or cffi. I mean, testing C libraries with Python
>> tools would be one, but then, you wouldn't want to write an extension
>> module for that and instead want to call it directly from the test code as
>> directly as possible.
>>
>> I'm certainly aware that that use case exists, though, and also the case
>> of just wanting to get things done as quickly and easily as possible.
> 
> Keep in mind I first came to Python as a tool for test automation of custom
> C++ hardware APIs that could be written to be SWIG friendly.

Interesting again. Would you still do it that way? I recently had a
discussion with Holger Krekel of py.test fame about testing C code with
Cython, and we quickly agreed that wrapping the code in an extension module
was both too cumbersome and too inflexible for testing purposes.
Specifically, neither of Cython's top selling points fits here, not speed,
not clarity, not API design. It's most likely different for SWIG, which
involves less (not no, just less) manual work and gives you API-wise more
of less exactly what you put in. However, cffi is almost certainly the
better way to do it, because it gives you all sorts of flexibility for your
test code without having to think about the wrapper design all the time.

The situation is also different for C++ where you have less options for
wrapping it. I can imagine SWIG still being the tool of choice on that
front when it comes to bare and direct testing of large code bases.


> I now work for an OS vendor where the 3 common languages for system
> utilities are C, C++ and Python.
> 
> For those use cases, dropping a bunch of standard Python objects in a
> module dict is often going to be a quick and easy solution that avoids a
> lot of nasty pointer lifecycle issues at the C level.

That's yet another use case, BTW. When you control the whole application,
then safety doesn't really matter at these points and keeping a bunch of
stuff in a dict will usually work just fine. I'm mainly used to writing
libraries for (sometimes tons of) other people, in which case the
requirements are so diverse on user side that safety is a top thing to care
about. Anything you can keep inside of C code should stay there.
(Especially when dealing with libxml2&friends in lxml which continuously
present their 'interesting' usability characteristics.)


> * PEP 3121 with a size of "0". As above, but avoids the module state APIs
> in order to support reloading. All module state (including type
> cross-references) is stored in hidden state (e.g. an instance of a custom
> type not exposed to Python, with a reference stored on each custom type
> object defined in the module, and any module level "functions" actually
> being methods of a hidden object).

Thanks for elaborating. I had completely failed to make the mental link
that you could simply stick bound methods as functions into the module
dict, i.e. that they don't even have to be methods of the module itself.
That's something that Cython could already use in older CPythons, even as a
preparation for any future import protocol changes. The object that they
are methods of would then eventually become the module instance.

You'd still suffer a slight performance hit from going from a static global
C variable to a pointer indirection - for everything: string constants,
cached Python objects, all user defined global C variables would have to go
there as Cython cannot know if they are module instance specific state or
not (they usually will be, I guess). But that has to be done anyway if the
goal is to get rid of static state to enable sub-interpreters. I can't wait
seeing lxml run threaded in mod_wsgi... ;-)


>> You seemed to be ok with my idea of making the loader return a wrapped
>> extension module instead of the module itself. We should actually try
>> that.
> 
> Sure, that's just a variant of the "hidden state object" idea I described
> above. It should actually work today with the PEP 3121 custom storage size
> set to zero.

True. The only difference is whether you leave it to the extension type
itself or make it a part of the loader architecture.

Anyway, I promise I'll give it a try in Cython. Will be some work, though,
to rewrite Cython's use of global variables, create a module state type,
migrate everything to heap types, ... I had wanted to do that for a couple
of years, but it's clearly not something for a happy afternoon or two.

Plus, it would even have to be optional in the compiler to avoid
performance regressions for modules that want to continue using fast static
globals simply because they cannot support multiple instances anyway (e.g.
due to external C library dependencies). Let's see if we can solve that at
C compilation time by throwing in a couple of macros. That would at least
help keeping the Cython compiler itself simple in that regard... (I guess
it would also help with testing as we could just duplicate the test suite
runs for both design modes)


>> As soon as you have more than one extension type in your module, and they
>> interact with each other, they will almost certainly have to do type
>> checks
>> against each other to make sure users haven't passed them rubbish before
>> they access any C struct fields of the object. Doing a type check means
>> that at least one type has a pointer to the other, meaning that it holds
>> global module state.
> 
> Sure, but you can use the CPython API rather than writing normal C code. We
> do this fairly often in CPython when we're dealing with things stored in
> modules that can be manipulated from Python.
> 
> It incurs CPython's dynamic dispatch overhead, but sometimes that's worth
> it to avoid needing to deal with C level lifecycle issues.

Not so much of a problem in Cython, because all you usually have to do to
get fast C level access to something is to change a "def" into a "cdef"
somewhere, or add a decorator, or an assignment to a known extension type
variable. Once the module global state is 'virtualised', this will also be
a safe thing to do in the face of multiple module instances, and still be
much faster than going through Python calls.


>> I really think that having some kind of global module state is the
>> exceedingly common case for an extension module.
> 
> I wouldn't be willing to make the call about which of stateless vs stateful
> is more common without a lot more research :)
> 
> They're both common enough that I think they should both be well supported,
> and making the "no custom C level state" case as simple as possible.

Agreed.


>>>> I didn't know about PyType_FromSpec(), BTW. It looks like a nice
>>>> addition for manually written code (although useless for Cython).
>>>
>>> This is the only way to create custom types when using the stable ABI.

I actually think I recall reading about it in the PEP back when it was
designed, decided that it made sense in the given context, and then forgot
about it as I didn't consider it relevant.


> The main advantage of the stable ABI is being able to publish cross-version
> binary extension modules. I guess if Cython already supports generating
> binaries for each new version of CPython before we release it, that
> capability is indeed less useful than it is for those that are maintaining
> extension modules by hand.

I consider it mostly interesting for Linux distributions and closed source
module vendors as it reduces the build/support overhead. But compiling the
generated C code for the specific CPython version at hand really has some
major advantages.

Stefan



From ncoghlan at gmail.com  Sun Sep  1 14:23:02 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 1 Sep 2013 22:23:02 +1000
Subject: [Python-Dev] Pre-PEP: Redesigning extension modules
In-Reply-To: <kvusrg$2vm$1@ger.gmane.org>
References: <kv77o2$f96$1@ger.gmane.org> <20130823111822.49cba700@pitrou.net>
 <CADiSq7dhzpTWMw7gcBRrzZot7jaS=Aj=aukq0+jh_cK2+F_kGQ@mail.gmail.com>
 <CADiSq7dBruZiWhObnjKbcneji9+ziXmyRpg1+72a8YeMWbyO3A@mail.gmail.com>
 <kvabt8$jfc$1@ger.gmane.org>
 <CADiSq7c6OjFa8zo7sMB=k2zLSmaf1w-tAwhYKSQ26T3caLyj6A@mail.gmail.com>
 <kvaka3$sbu$1@ger.gmane.org>
 <CADiSq7dzQqyH9i6g9QYWp6_ZHHUD4+4npYo6of6mCNahQ+w8kw@mail.gmail.com>
 <kvcr9f$4gs$1@ger.gmane.org>
 <CADiSq7cfNH1M29W19BW88-BBQaWEpaMkE=s09wNyT5VGOrPB+g@mail.gmail.com>
 <kvtfek$23p$1@ger.gmane.org>
 <CADiSq7d3L1+Jc5E4dWYDRUQPNDCmxqyc3GzbgXJvGMe4+OnWrQ@mail.gmail.com>
 <kvusrg$2vm$1@ger.gmane.org>
Message-ID: <CADiSq7cN17Maf5wdrmyVR=xG4EUqxeuzDfm7ZWa6=6oMFS_iLA@mail.gmail.com>

On 1 September 2013 18:11, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Nick Coghlan, 01.09.2013 03:28:
>> On 1 Sep 2013 05:18, "Stefan Behnel" wrote:
>>> I can't really remember a case where I could afford the
>>> runtime overhead of implementing a wrapper in Python and going through
>>> something like ctypes or cffi. I mean, testing C libraries with Python
>>> tools would be one, but then, you wouldn't want to write an extension
>>> module for that and instead want to call it directly from the test code as
>>> directly as possible.
>>>
>>> I'm certainly aware that that use case exists, though, and also the case
>>> of just wanting to get things done as quickly and easily as possible.
>>
>> Keep in mind I first came to Python as a tool for test automation of custom
>> C++ hardware APIs that could be written to be SWIG friendly.
>
> Interesting again. Would you still do it that way? I recently had a
> discussion with Holger Krekel of py.test fame about testing C code with
> Cython, and we quickly agreed that wrapping the code in an extension module
> was both too cumbersome and too inflexible for testing purposes.
> Specifically, neither of Cython's top selling points fits here, not speed,
> not clarity, not API design. It's most likely different for SWIG, which
> involves less (not no, just less) manual work and gives you API-wise more
> of less exactly what you put in. However, cffi is almost certainly the
> better way to do it, because it gives you all sorts of flexibility for your
> test code without having to think about the wrapper design all the time.
>
> The situation is also different for C++ where you have less options for
> wrapping it. I can imagine SWIG still being the tool of choice on that
> front when it comes to bare and direct testing of large code bases.

To directly wrap C++, I'd still use SWIG. It makes a huge difference
when you can tweak the C++ side of the API to be SWIG friendly rather
than having to live with whatever a third party C++ library provides.
Having classes in C++ map directly to classes in Python is the main
benefit of doing it this way over using a C wrapper and cffi.

However, for an existing C API, or a custom API where I didn't need
the direct object mapping that C++ can provide, using cffi would be a
more attractive option than SWIG these days (the stuff I was doing
with SWIG was back around 2003 or so).

I think this is getting a little off topic for the list, though :)

>> I now work for an OS vendor where the 3 common languages for system
>> utilities are C, C++ and Python.
>>
>> For those use cases, dropping a bunch of standard Python objects in a
>> module dict is often going to be a quick and easy solution that avoids a
>> lot of nasty pointer lifecycle issues at the C level.
>
> That's yet another use case, BTW. When you control the whole application,
> then safety doesn't really matter at these points and keeping a bunch of
> stuff in a dict will usually work just fine. I'm mainly used to writing
> libraries for (sometimes tons of) other people, in which case the
> requirements are so diverse on user side that safety is a top thing to care
> about. Anything you can keep inside of C code should stay there.
> (Especially when dealing with libxml2&friends in lxml which continuously
> present their 'interesting' usability characteristics.)

I don't think it's a coincidence that it was the etree interface with
expat that highlighted the deficiencies of the current extension
module hooks when it comes to working properly with
test.support.import_fresh_module :)

>> * PEP 3121 with a size of "0". As above, but avoids the module state APIs
>> in order to support reloading. All module state (including type
>> cross-references) is stored in hidden state (e.g. an instance of a custom
>> type not exposed to Python, with a reference stored on each custom type
>> object defined in the module, and any module level "functions" actually
>> being methods of a hidden object).
>
> Thanks for elaborating. I had completely failed to make the mental link
> that you could simply stick bound methods as functions into the module
> dict, i.e. that they don't even have to be methods of the module itself.
> That's something that Cython could already use in older CPythons, even as a
> preparation for any future import protocol changes. The object that they
> are methods of would then eventually become the module instance.
>
> You'd still suffer a slight performance hit from going from a static global
> C variable to a pointer indirection - for everything: string constants,
> cached Python objects, all user defined global C variables would have to go
> there as Cython cannot know if they are module instance specific state or
> not (they usually will be, I guess). But that has to be done anyway if the
> goal is to get rid of static state to enable sub-interpreters. I can't wait
> seeing lxml run threaded in mod_wsgi... ;-)

To be honest, I didn't realise that such a trick might already be
possible until I was writing down this list of alternatives. If you
manage to turn it into a real solution for lxml (or Cython in
general), it would be great to hear more about how you turned the
general idea into something real :)

That means the powers any new extension initialisation API will offer
will be limited to:

* letting the module know its own name (and other details)
* letting the module explicitly block reloading
* letting the module support loading multiple copies at once by taking
the initial import out of sys.modules (but keeping a separate
reference to it alive)

<snip>
>>> As soon as you have more than one extension type in your module, and they
>>> interact with each other, they will almost certainly have to do type
>>> checks
>>> against each other to make sure users haven't passed them rubbish before
>>> they access any C struct fields of the object. Doing a type check means
>>> that at least one type has a pointer to the other, meaning that it holds
>>> global module state.
>>
>> Sure, but you can use the CPython API rather than writing normal C code. We
>> do this fairly often in CPython when we're dealing with things stored in
>> modules that can be manipulated from Python.
>>
>> It incurs CPython's dynamic dispatch overhead, but sometimes that's worth
>> it to avoid needing to deal with C level lifecycle issues.
>
> Not so much of a problem in Cython, because all you usually have to do to
> get fast C level access to something is to change a "def" into a "cdef"
> somewhere, or add a decorator, or an assignment to a known extension type
> variable. Once the module global state is 'virtualised', this will also be
> a safe thing to do in the face of multiple module instances, and still be
> much faster than going through Python calls.

It's kinda cool how designing a next generation API can sometimes
reveal hidden possibilities of an *existing* API :)

We had another one of those recently on distutils-sig, when I realised
the much-maligned .pth modules are actually a decent solution to
sharing distributions between virtual environments. I'm so used to
disliking their global side effects when used with the system Python
that it took me a long time to recognise the validity of using them to
make implicit path additions in a more controlled virtual environment
:)

In terms of where we go from here - do you mind if I use your pre-PEP
as the initial basis for a PEP of my own some time in the next week or
two (listing you as co-author)? Improving extension module
initialisation has been the driver for most of the PEP 451 feedback
I've been giving to Eric over on import-sig, so I have some definite
ideas on how I think that API should look :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From stefan_ml at behnel.de  Sun Sep  1 14:41:50 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 01 Sep 2013 14:41:50 +0200
Subject: [Python-Dev] Pre-PEP: Redesigning extension modules
In-Reply-To: <CADiSq7cN17Maf5wdrmyVR=xG4EUqxeuzDfm7ZWa6=6oMFS_iLA@mail.gmail.com>
References: <kv77o2$f96$1@ger.gmane.org> <20130823111822.49cba700@pitrou.net>
 <CADiSq7dhzpTWMw7gcBRrzZot7jaS=Aj=aukq0+jh_cK2+F_kGQ@mail.gmail.com>
 <CADiSq7dBruZiWhObnjKbcneji9+ziXmyRpg1+72a8YeMWbyO3A@mail.gmail.com>
 <kvabt8$jfc$1@ger.gmane.org>
 <CADiSq7c6OjFa8zo7sMB=k2zLSmaf1w-tAwhYKSQ26T3caLyj6A@mail.gmail.com>
 <kvaka3$sbu$1@ger.gmane.org>
 <CADiSq7dzQqyH9i6g9QYWp6_ZHHUD4+4npYo6of6mCNahQ+w8kw@mail.gmail.com>
 <kvcr9f$4gs$1@ger.gmane.org>
 <CADiSq7cfNH1M29W19BW88-BBQaWEpaMkE=s09wNyT5VGOrPB+g@mail.gmail.com>
 <kvtfek$23p$1@ger.gmane.org>
 <CADiSq7d3L1+Jc5E4dWYDRUQPNDCmxqyc3GzbgXJvGMe4+OnWrQ@mail.gmail.com>
 <kvusrg$2vm$1@ger.gmane.org>
 <CADiSq7cN17Maf5wdrmyVR=xG4EUqxeuzDfm7ZWa6=6oMFS_iLA@mail.gmail.com>
Message-ID: <kvvcm6$okb$1@ger.gmane.org>

Nick Coghlan, 01.09.2013 14:23:
> That means the powers any new extension initialisation API will offer
> will be limited to:
> 
> * letting the module know its own name (and other details)
> * letting the module explicitly block reloading
> * letting the module support loading multiple copies at once by taking
> the initial import out of sys.modules (but keeping a separate
> reference to it alive)

Which, all by themselves, can be considered a huge benefit, IMHO.

Plus, if we design the protocol broad enough now, specifically as a two-way
interface (info in, module out), we won't have to make any major changes to
it again anywhere in the near future, because incremental changes can just
be integrated into what's there then, in case we need any. It's sad that we
didn't see these requirements for Py3.0.


> In terms of where we go from here - do you mind if I use your pre-PEP
> as the initial basis for a PEP of my own some time in the next week or
> two (listing you as co-author)? Improving extension module
> initialisation has been the driver for most of the PEP 451 feedback
> I've been giving to Eric over on import-sig, so I have some definite
> ideas on how I think that API should look :)

Go for it. I'm not sure how much time I can actively spend on this during
the next weeks anyway, so I'm happy if this continues to get pushed onwards
in the meantime.

Stefan



From solipsis at pitrou.net  Sun Sep  1 15:03:51 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 1 Sep 2013 15:03:51 +0200
Subject: [Python-Dev] Pre-PEP: Redesigning extension modules
References: <kv77o2$f96$1@ger.gmane.org> <20130823111822.49cba700@pitrou.net>
 <CADiSq7dhzpTWMw7gcBRrzZot7jaS=Aj=aukq0+jh_cK2+F_kGQ@mail.gmail.com>
 <CADiSq7dBruZiWhObnjKbcneji9+ziXmyRpg1+72a8YeMWbyO3A@mail.gmail.com>
 <kvabt8$jfc$1@ger.gmane.org>
 <CADiSq7c6OjFa8zo7sMB=k2zLSmaf1w-tAwhYKSQ26T3caLyj6A@mail.gmail.com>
 <kvaka3$sbu$1@ger.gmane.org>
 <CADiSq7dzQqyH9i6g9QYWp6_ZHHUD4+4npYo6of6mCNahQ+w8kw@mail.gmail.com>
 <kvcr9f$4gs$1@ger.gmane.org>
 <CADiSq7cfNH1M29W19BW88-BBQaWEpaMkE=s09wNyT5VGOrPB+g@mail.gmail.com>
 <kvtfek$23p$1@ger.gmane.org>
 <CADiSq7d3L1+Jc5E4dWYDRUQPNDCmxqyc3GzbgXJvGMe4+OnWrQ@mail.gmail.com>
Message-ID: <20130901150351.2f2aa44c@fsol>

On Sun, 1 Sep 2013 11:28:36 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> * PEP 3121 with a size of "0". As above, but avoids the module state APIs
> in order to support reloading. All module state (including type
> cross-references) is stored in hidden state (e.g. an instance of a custom
> type not exposed to Python, with a reference stored on each custom type
> object defined in the module, and any module level "functions" actually
> being methods of a hidden object). Still doesn't support loading a *fresh*
> copy due to the hidden PEP 3121 module cache.

Not sure what you mean by that:

>>> import atexit
>>> id(atexit)
140031896222680
>>> import sys
>>> del sys.modules['atexit']
>>> import atexit
>>> id(atexit)
140031896221400


> Due to refcounting, all instances of Python objects qualify as mutable
> state.

That's an overly broad definition. Many objects are shared between
subinterpreters without any problems (None, the empty tuple, built-in
types and most C extension types, etc.). As long as the state is
an internal implementation detail, there shouldn't be any problem.

> I wouldn't be willing to make the call about which of stateless vs stateful
> is more common without a lot more research :)
> 
> They're both common enough that I think they should both be well supported,
> and making the "no custom C level state" case as simple as possible.

Agreed.

Regards

Antoine.



From ncoghlan at gmail.com  Sun Sep  1 16:10:08 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 2 Sep 2013 00:10:08 +1000
Subject: [Python-Dev] Pre-PEP: Redesigning extension modules
In-Reply-To: <20130901150351.2f2aa44c@fsol>
References: <kv77o2$f96$1@ger.gmane.org> <20130823111822.49cba700@pitrou.net>
 <CADiSq7dhzpTWMw7gcBRrzZot7jaS=Aj=aukq0+jh_cK2+F_kGQ@mail.gmail.com>
 <CADiSq7dBruZiWhObnjKbcneji9+ziXmyRpg1+72a8YeMWbyO3A@mail.gmail.com>
 <kvabt8$jfc$1@ger.gmane.org>
 <CADiSq7c6OjFa8zo7sMB=k2zLSmaf1w-tAwhYKSQ26T3caLyj6A@mail.gmail.com>
 <kvaka3$sbu$1@ger.gmane.org>
 <CADiSq7dzQqyH9i6g9QYWp6_ZHHUD4+4npYo6of6mCNahQ+w8kw@mail.gmail.com>
 <kvcr9f$4gs$1@ger.gmane.org>
 <CADiSq7cfNH1M29W19BW88-BBQaWEpaMkE=s09wNyT5VGOrPB+g@mail.gmail.com>
 <kvtfek$23p$1@ger.gmane.org>
 <CADiSq7d3L1+Jc5E4dWYDRUQPNDCmxqyc3GzbgXJvGMe4+OnWrQ@mail.gmail.com>
 <20130901150351.2f2aa44c@fsol>
Message-ID: <CADiSq7e=E+WCstffBsREc6KKrBU5mbKFWiriDRt6orAKsQWnVQ@mail.gmail.com>

On 1 September 2013 23:03, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sun, 1 Sep 2013 11:28:36 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
>> * PEP 3121 with a size of "0". As above, but avoids the module state APIs
>> in order to support reloading. All module state (including type
>> cross-references) is stored in hidden state (e.g. an instance of a custom
>> type not exposed to Python, with a reference stored on each custom type
>> object defined in the module, and any module level "functions" actually
>> being methods of a hidden object). Still doesn't support loading a *fresh*
>> copy due to the hidden PEP 3121 module cache.
>
> Not sure what you mean by that:
>
>>>> import atexit
>>>> id(atexit)
> 140031896222680
>>>> import sys
>>>> del sys.modules['atexit']
>>>> import atexit
>>>> id(atexit)
> 140031896221400

Ah, you're right - I misremembered the exact problem that broke
xml.etree.ElementTree testing. PyModule_GetState is actually fine
(since that pointer is hidden state on the module object), it's only
PyState_GetModule that is broken when you import a second copy. So,
here, when the second import happens, it breaks the original atexit
module's callbacks, even though the two callback registries are
properly isolated:

$ ./python
Python 3.4.0a1+ (default:575071257c92+, Aug 25 2013, 00:42:17)
[GCC 4.7.2 20121109 (Red Hat 4.7.2-8)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import atexit
>>> atexit.register(print, "Hello World!")
<built-in function print>
>>> import sys
>>> del sys.modules["atexit"]
>>> import atexit as atexit2
>>> atexit2.register(print, "Goodbye World!")
<built-in function print>
>>>
Goodbye World!

So I think PEP 3121 is actually as good as we can get on the hidden
state front, but the important point is that it is the
*PyState_GetModule* API that can't handle fresh imports - the second
import will always replace the first one. So anyone affected needs to
find some other way of passing the state, like using bound methods of
a hidden type rather than ordinary callables. If you have to
interoperate with a C API that only accepts a C callback without
allowing additional state arguments, you're going to have trouble.

I think atexit serves as a good example, though - that _Py_PyAtExit
call will *always* be destructive (even if you still have a reference
to the original module), so there should be a way for the module to
explicitly indicate to the import system "you can only create this
module once, and then you're committed - unloading it and importing it
again won't work properly due to side effects on the process state".

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From rdmurray at bitdance.com  Sun Sep  1 18:28:06 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Sun, 01 Sep 2013 12:28:06 -0400
Subject: [Python-Dev] Completing the email6 API changes.
In-Reply-To: <878uzhdh24.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <20130831052136.42D4B2507F5@webabinitio.net>
 <87bo4echcr.fsf@uwakimon.sk.tsukuba.ac.jp>
 <20130831142352.600462507F5@webabinitio.net>
 <878uzhdh24.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <20130901162807.0C345250813@webabinitio.net>

On Sun, 01 Sep 2013 00:18:59 +0900, "Stephen J. Turnbull" <stephen at xemacs.org> wrote:
> R. David Murray writes:
> 
>  > Full validation is something that is currently a "future
>  > objective".
> 
> I didn't mean it to be anything else. :-)
> 
>  > There's infrastructure to do it, but not all of the necessary knowledge
>  > has been coded in yet.
> 
> Well, I assume you already know that there's no way that can ever
> happen (at least until we abandon messaging entirely): new RFCs will
> continue to be published.  So it needs to be an extensible mechanism,
> a "pipeline" of checks (Barry would say a "chain of rules", I think).

My idea was to encode as much of the current known rules as as we have
the stomach for, and to have a validation flag that you turn on if you
want to check your message against those standards.  But without that
flag the code allows you to set arbitrary parameters and headers.

As you say, an extensible mechanism for the validators is a good idea.
So I take it back that the infrastructure is in place, since extensibility
doesn't exist for that feature yet.

--David

From solipsis at pitrou.net  Sun Sep  1 19:40:48 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 1 Sep 2013 19:40:48 +0200
Subject: [Python-Dev] Pre-PEP: Redesigning extension modules
In-Reply-To: <CADiSq7e=E+WCstffBsREc6KKrBU5mbKFWiriDRt6orAKsQWnVQ@mail.gmail.com>
References: <kv77o2$f96$1@ger.gmane.org> <20130823111822.49cba700@pitrou.net>
 <CADiSq7dhzpTWMw7gcBRrzZot7jaS=Aj=aukq0+jh_cK2+F_kGQ@mail.gmail.com>
 <CADiSq7dBruZiWhObnjKbcneji9+ziXmyRpg1+72a8YeMWbyO3A@mail.gmail.com>
 <kvabt8$jfc$1@ger.gmane.org>
 <CADiSq7c6OjFa8zo7sMB=k2zLSmaf1w-tAwhYKSQ26T3caLyj6A@mail.gmail.com>
 <kvaka3$sbu$1@ger.gmane.org>
 <CADiSq7dzQqyH9i6g9QYWp6_ZHHUD4+4npYo6of6mCNahQ+w8kw@mail.gmail.com>
 <kvcr9f$4gs$1@ger.gmane.org>
 <CADiSq7cfNH1M29W19BW88-BBQaWEpaMkE=s09wNyT5VGOrPB+g@mail.gmail.com>
 <kvtfek$23p$1@ger.gmane.org>
 <CADiSq7d3L1+Jc5E4dWYDRUQPNDCmxqyc3GzbgXJvGMe4+OnWrQ@mail.gmail.com>
 <20130901150351.2f2aa44c@fsol>
 <CADiSq7e=E+WCstffBsREc6KKrBU5mbKFWiriDRt6orAKsQWnVQ@mail.gmail.com>
Message-ID: <20130901194048.0d7d1b75@fsol>

On Mon, 2 Sep 2013 00:10:08 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> $ ./python
> Python 3.4.0a1+ (default:575071257c92+, Aug 25 2013, 00:42:17)
> [GCC 4.7.2 20121109 (Red Hat 4.7.2-8)] on linux
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import atexit
> >>> atexit.register(print, "Hello World!")
> <built-in function print>
> >>> import sys
> >>> del sys.modules["atexit"]
> >>> import atexit as atexit2
> >>> atexit2.register(print, "Goodbye World!")
> <built-in function print>
> >>>
> Goodbye World!

Yeah, atexit is a very particular example, because it interacts with
global state by design (the main interpreter instance), and no amount
of module initialization magic can prevent that :-)

Speaking of which, it also doesn't work (well) with subinterpreters:
http://bugs.python.org/issue18618

Regards

Antoine.

From tjreedy at udel.edu  Sun Sep  1 22:02:33 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 01 Sep 2013 16:02:33 -0400
Subject: [Python-Dev] 'Subinterpreter' (was Re: Pre-PEP: Redesigning
	extension modules)
In-Reply-To: <20130901194048.0d7d1b75@fsol>
References: <kv77o2$f96$1@ger.gmane.org> <20130823111822.49cba700@pitrou.net>
 <CADiSq7dhzpTWMw7gcBRrzZot7jaS=Aj=aukq0+jh_cK2+F_kGQ@mail.gmail.com>
 <CADiSq7dBruZiWhObnjKbcneji9+ziXmyRpg1+72a8YeMWbyO3A@mail.gmail.com>
 <kvabt8$jfc$1@ger.gmane.org>
 <CADiSq7c6OjFa8zo7sMB=k2zLSmaf1w-tAwhYKSQ26T3caLyj6A@mail.gmail.com>
 <kvaka3$sbu$1@ger.gmane.org>
 <CADiSq7dzQqyH9i6g9QYWp6_ZHHUD4+4npYo6of6mCNahQ+w8kw@mail.gmail.com>
 <kvcr9f$4gs$1@ger.gmane.org>
 <CADiSq7cfNH1M29W19BW88-BBQaWEpaMkE=s09wNyT5VGOrPB+g@mail.gmail.com>
 <kvtfek$23p$1@ger.gmane.org>
 <CADiSq7d3L1+Jc5E4dWYDRUQPNDCmxqyc3GzbgXJvGMe4+OnWrQ@mail.gmail.com>
 <20130901150351.2f2aa44c@fsol>
 <CADiSq7e=E+WCstffBsREc6KKrBU5mbKFWiriDRt6orAKsQWnVQ@mail.gmail.com>
 <20130901194048.0d7d1b75@fsol>
Message-ID: <l006gp$jb7$1@ger.gmane.org>


> Speaking of which, it also doesn't work (well) with subinterpreters:

Could someone briefly explain 'subinterpreter' or point me somewhere in 
the docs? It appears throughout this thread but there is no index or 
glossary entry.

-- 
Terry Jan Reedy


From solipsis at pitrou.net  Sun Sep  1 22:06:49 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 1 Sep 2013 22:06:49 +0200
Subject: [Python-Dev] 'Subinterpreter' (was Re: Pre-PEP: Redesigning
 extension modules)
References: <kv77o2$f96$1@ger.gmane.org> <20130823111822.49cba700@pitrou.net>
 <CADiSq7dhzpTWMw7gcBRrzZot7jaS=Aj=aukq0+jh_cK2+F_kGQ@mail.gmail.com>
 <CADiSq7dBruZiWhObnjKbcneji9+ziXmyRpg1+72a8YeMWbyO3A@mail.gmail.com>
 <kvabt8$jfc$1@ger.gmane.org>
 <CADiSq7c6OjFa8zo7sMB=k2zLSmaf1w-tAwhYKSQ26T3caLyj6A@mail.gmail.com>
 <kvaka3$sbu$1@ger.gmane.org>
 <CADiSq7dzQqyH9i6g9QYWp6_ZHHUD4+4npYo6of6mCNahQ+w8kw@mail.gmail.com>
 <kvcr9f$4gs$1@ger.gmane.org>
 <CADiSq7cfNH1M29W19BW88-BBQaWEpaMkE=s09wNyT5VGOrPB+g@mail.gmail.com>
 <kvtfek$23p$1@ger.gmane.org>
 <CADiSq7d3L1+Jc5E4dWYDRUQPNDCmxqyc3GzbgXJvGMe4+OnWrQ@mail.gmail.com>
 <20130901150351.2f2aa44c@fsol>
 <CADiSq7e=E+WCstffBsREc6KKrBU5mbKFWiriDRt6orAKsQWnVQ@mail.gmail.com>
 <20130901194048.0d7d1b75@fsol> <l006gp$jb7$1@ger.gmane.org>
Message-ID: <20130901220649.3e9c3ab4@fsol>

On Sun, 01 Sep 2013 16:02:33 -0400
Terry Reedy <tjreedy at udel.edu> wrote:
> 
> > Speaking of which, it also doesn't work (well) with subinterpreters:
> 
> Could someone briefly explain 'subinterpreter' or point me somewhere in 
> the docs? It appears throughout this thread but there is no index or 
> glossary entry.

http://docs.python.org/dev/c-api/init.html#sub-interpreter-support

Subinterpreters are a somewhat borderline feature that allows embedding
applications to host multiple Python programs in a single process.  A
well-known example is mod_wsgi.

Regards

Antoine.



From solipsis at pitrou.net  Sun Sep  1 23:04:14 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 1 Sep 2013 23:04:14 +0200
Subject: [Python-Dev] cpython (merge 3.3 -> default): Merge fix from 3.3
 into default.
References: <3cSn1P710Hz7LjX@mail.python.org>
Message-ID: <20130901230414.44af5426@fsol>

On Sun,  1 Sep 2013 23:02:17 +0200 (CEST)
tim.peters <python-checkins at python.org> wrote:
> http://hg.python.org/cpython/rev/25211a22228b
> changeset:   85495:25211a22228b
> parent:      85493:267e09700978
> parent:      85494:8efcf3c823f9
> user:        Tim Peters <tim at python.org>
> date:        Sun Sep 01 16:01:46 2013 -0500
> summary:
>   Merge fix from 3.3 into default.
> 
> Fix issue 18889: test_sax: multiple failures on Windows desktop.
> 
> "The fix" is to tell Mercurial that the test files are binary.
> 
> Windows developers:  to get the correct line endings in your checkout,
> delete Lib\test\xmltestdata, and then "hg revert" that directory.
> 
> Why the Windows buildbots didn't fail test_sax remains a mystery :-(

Probably because they don't have the hgeol extension enabled.

Regards

Antoine.



From stefan_ml at behnel.de  Sun Sep  1 23:13:36 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 01 Sep 2013 23:13:36 +0200
Subject: [Python-Dev] 'Subinterpreter' (was Re: Pre-PEP: Redesigning
	extension modules)
In-Reply-To: <20130901220649.3e9c3ab4@fsol>
References: <kv77o2$f96$1@ger.gmane.org> <20130823111822.49cba700@pitrou.net>
 <CADiSq7dhzpTWMw7gcBRrzZot7jaS=Aj=aukq0+jh_cK2+F_kGQ@mail.gmail.com>
 <CADiSq7dBruZiWhObnjKbcneji9+ziXmyRpg1+72a8YeMWbyO3A@mail.gmail.com>
 <kvabt8$jfc$1@ger.gmane.org>
 <CADiSq7c6OjFa8zo7sMB=k2zLSmaf1w-tAwhYKSQ26T3caLyj6A@mail.gmail.com>
 <kvaka3$sbu$1@ger.gmane.org>
 <CADiSq7dzQqyH9i6g9QYWp6_ZHHUD4+4npYo6of6mCNahQ+w8kw@mail.gmail.com>
 <kvcr9f$4gs$1@ger.gmane.org>
 <CADiSq7cfNH1M29W19BW88-BBQaWEpaMkE=s09wNyT5VGOrPB+g@mail.gmail.com>
 <kvtfek$23p$1@ger.gmane.org>
 <CADiSq7d3L1+Jc5E4dWYDRUQPNDCmxqyc3GzbgXJvGMe4+OnWrQ@mail.gmail.com>
 <20130901150351.2f2aa44c@fsol>
 <CADiSq7e=E+WCstffBsREc6KKrBU5mbKFWiriDRt6orAKsQWnVQ@mail.gmail.com>
 <20130901194048.0d7d1b75@fsol> <l006gp$jb7$1@ger.gmane.org>
 <20130901220649.3e9c3ab4@fsol>
Message-ID: <l00alo$qlc$1@ger.gmane.org>

Antoine Pitrou, 01.09.2013 22:06:
> On Sun, 01 Sep 2013 16:02:33 -0400
> Terry Reedy wrote:
>>> Speaking of which, it also doesn't work (well) with subinterpreters:
>>
>> Could someone briefly explain 'subinterpreter' or point me somewhere in 
>> the docs? It appears throughout this thread but there is no index or 
>> glossary entry.
> 
> http://docs.python.org/dev/c-api/init.html#sub-interpreter-support
> 
> Subinterpreters are a somewhat borderline feature that allows embedding
> applications to host multiple Python programs in a single process.  A
> well-known example is mod_wsgi.

And extension modules usually don't play well with subinterpreters because
each subinterpreter requires its own separate version of the module and
extension modules are rarely designed to keep their state completely local
to an interpreter, let alone being prepared for having their module init
function be called more than once.

Stefan



From rdmurray at bitdance.com  Mon Sep  2 00:10:39 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Sun, 01 Sep 2013 18:10:39 -0400
Subject: [Python-Dev] Completing the email6 API changes.
In-Reply-To: <87bo4echcr.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <20130831052136.42D4B2507F5@webabinitio.net>
 <87bo4echcr.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <20130901221040.DC76E250818@webabinitio.net>

On Sat, 31 Aug 2013 18:57:56 +0900, "Stephen J. Turnbull" <stephen at xemacs.org> wrote:
> R. David Murray writes:
> 
>  > But I would certainly appreciate review from anyone so moved, since I
>  > haven't gotten any yet.
> 
> I'll try to make time for a serious (but obviously partial) review by
> Monday.
> 
> I don't know if this is "serious" bikeshedding, but I have a comment
> or two on the example:
> 
>  >     from email.message import MIMEMessage
>  >     from email.headerregistry import Address
>  >     fullmsg = MIMEMessage()
>  >     fullmsg['To'] = Address('Fo?? Bar', 'fbar at example.com')
>  >     fullmsg['From'] = "m???? <me at example.com>"
>  >     fullmsg['Subject'] = "j'ai un probl??me de python."
> 
> This is very nice!  *I* *love* it.
> 
> But (sorry!) I worry that it's not obvious to "naive" users.  Maybe it
> would be useful to have a Message() factory which has one semantic
> difference from MIMEMessage: it "requires" RFC 822-required headers
> (or optionally RFC 1036 for news).  Eg:
> 
>     # This message will be posted and mailed
>     # These would conform to the latest Draft Standards
>     # and be DKIM-signed
>     fullmsg = Message('rfc822', 'rfc1036', 'dmarc')
> 
> I'm not sure how "required" would be implemented (perhaps through a
> .validate() method).  So the signature of the API suggested above is
> Message(*validators, **kw).

Adding new constructor arguments to the existing Message class is
possible.  However, given the new architecture, the more logical way
to do this is to put it in the policy.  So currently the idea would be
for this to be spelled like this:

    fullmsg = Message(policy=policy.SMTP+policy.strict)

Then what would happen is that when the message is serialized (be it
via str(), bytes(), by passing it to smtplib.sendmail or
smtplib.sendmessage, or by an explicit call to a Generator), an
error would be raised if the minimum required headers are not
present.

As I said in an earlier message, currently there's no extensibility
mechanism for the validation.  If the parser recognizes a defect, whether
or not an error is raised is controlled by the policy.  But there's
no mechanism for adding new defect checking that the parser doesn't
already know about, or for issues that are not parse-time defects.
(There is currently one non-parsing defect for which there is a custom
control: the maximum number of headers of a given type that are allowed
to be added to a Message object.)

So we need some way to add additional constraints as well.  Probably a
list of validation functions that take a Message/MIMEPart as the
argument and do a raise if they want to reject the message.

The tricky bit is that currently raise_on_defect means you get an error
as soon as a (parsing) defect is discovered.  Likewise, if max_count
is being enforced for headers, the error is raised as soon as the
duplicate header is added.

Generating errors early when building messages was one of or original
design goals, and *only* detecting problems via validators runs counter
to that unless all the validators are called every time an operation
is performed that modifies a message.  Maybe that would be OK, but it
feels ugly.

For the missing header problem, the custom solution could be to add a
'headers' argument to Message that would allow you to write:

     fullmsg = Message(header=(
                        Header('Date', email.utils.localtime()),
                        Header('To', Address('Fred', 'abc at xyz.com')),
                        Header('From', Address('Sally, 'fgd at xyz.com')),
                        Header('Subject', 'Foo'),
                        ),
                    policy=policy.SMTP+policy.Strict)

This call could then immediately raise an error if not all of the
required headers are present.  (Header is unfortunately not a good
choice of name here because we already have a 'Header' class that has a
different API).

Aside: I could also imagine adding a 'content' argument that would let
you generate a simple text message via a single call...which means you
could also extend this model to specifying the entire message in a single
call, if you wrote a suitable content manager function for tuples:

     fullmsg = Message(
                policy=policy.SMTP+policy.Strict,
                header=(
                   Header('Date', datetime.datetime.now()),
                   Header('To', Address('Fred', 'abc at xyz.com')),
                   Header('From', Address('Sally, 'fgd at xyz.com')),
                   Header('Subject', 'Foo'),
                   ),
                content=(
                        (
                            'This is the text part',
                            (
                              '<p>Here is the html</p><img src="image1" \>',
                              {'image1': b'image data'},
                              ),
                        ),
                    b'attachment data',
                    )
                    
But that is probably a little bit crazy...easier to just write a custom
function for your application that calls the message building APIs.

Well, anyway, coming back from that little tangent, it seems to me in
summary that if we want to raise errors as soon as possible, we need
to add custom mechanisms for the error detection as we figure out each
class of error we want to handle, so that the validation gets done right
away without adding too much overhead.  For parsing defects, that means
if you want to control which ones are raised you hook handle_defect
and make your decision there (there's no way to *add* parsing defects
via that hook).  For duplicate headers, you hook header_max_count (or
set max_count on your custom header classes).  For missing headers we
*could* introduce something like the above headers argument to Message,
which would be required in 'strict' mode.

But clearly we should also have a general "validation function" list on
the policy, where all the functions would be called before serialization
and have an opportunity to reject the message by raising errors.
That would provide a generalized, if heavier handed, hook to use when
there is no specific hook that provides extendability.

For missing headers I'm inclined to use a serialization-time check
rather than the Message constructor check I speculated about above.
My logic is that the message isn't complete until you are ready to send
it, so generating an error earlier probably isn't wanted in many cases,
nor will generating it at serialization time loose you much debugging
information.  So I'm inclined to implement it as a default validator in
the proposed list of serialization time validators.

It is still probably worthwhile providing a utility function for creating
a message in a single call.  I think there's even an open issue in the
tracker for that.  But I'm inclined to postpone that until 3.5.

> For MIMEMessage, I think I prefer the name "MIMEPart".  To naive
> users, the idea of MIMEMessages containing MIMEMessages is a bit
> disconcerting, except in the case of multipart/digest, I think.

Or message/rfc822:

    [...]
    try:
        smtplib.sendmessage(msg)
    except Exception as exc:
        errormsg = MIMEMEssage()
        errormsg['To'] = msg['sender'] or msg['from']
        errormsg['From'] = 'robot at mydomain.org'
        errormsg.set_content("I'm sorry, sending failed: {}".format(exc))
        errormsg.add_attachment(orig_msg, disposition='inline')
        smtplib.sendmessage(errormsg)

(Terrible code, but you get the idea :)

>  >     fullmsg.set_content("et la il est mont?? sur moi et il commence"
>  >                        " a m'????touffer.")
>  >     htmlmsg = MIMEMessage()
>  >     htmlmsg.set_content("<p>et la il est mont?? sur moi et il commence"
>  >                         " a m'??touffer.</p><img src='image1' />",
>  >                         subtype='html')
> 
> I think I'd like to express the suite above as
> 
>     fullmsg.payload.add_alternative(...)
>     fullmsg.payload.add_alternative(..., subtype='html')
> 
> This would automatically convert the MIME type of fullmsg to
> 'multipart/alternative', and .payload to a list where necessary.
> .set_content() would be available but it's "dangerous" (it could
> replace an arbitrary multipart -- this would be useful operation to
> replace it with a textual URL or external-body part).

Having an attribute with methods that affect the parent object is not a
natural act in Python.  (Is it in OO in general?)  But aside from that,
I'm not sure I see the point of a two level name here.  You are still
talking about mutating the message model object, which is exactly what
is going on in my example: if you call add_alternative on a text part,
it gets turned into a multipart/alternative with the first part being
the text part and the second part being the thing you just added.
That's in the docs but I didn't refer to it explicitly in my example.
Does that make things clearer?

It is a good point about unintentionally overwriting the existing
contents.  That would be simple to fix: make it an error to call
set_content if payload is not None.  Possibly a 'clear' method would
then be useful, especially since so many other Python datatypes have one.

> Aside: it occurs to me that the .payload attribute (and other such
> attributes) could be avoided by the device of using names prefixed by
> ":" such as ":payload" as keys: "fullmsg[':payload']" since colon is
> illegal in field names (cf RFC 5322).  Probably I've just been writing
> too much Common Lisp, though.<wink/>

I think maybe so :).  I'd rather write 'msg.payload' than 'msg[:payload'].
So I don't have a motivation to "avoid" those attributes.  (Nor would
I find msg[':payload'].sommethod mutating the parent object any less
surprising.)

> I'm not sure whether "payload" is a better name than "content" for
> that attribute.
> 
> Now the suite
> 
>  >     with open('python.jpg', 'rb') as python:
>  >         htmlmsg.add_related(python.read(), 'image', 'jpg', cid='image1'
>  >                             disposition='inline')
>  >     fullmsg.make_alternative()
>  >     fullmsg.attach(htmlmsg)
> 
> becomes just
> 
>     with open('python.jpg', 'rb') as python:
>         fullmsg.payload['text/html'].add_related(...)

This doesn't work, though, because you could (although you usually
won't) have more than one 'text/html' part in a single multipart.

The reason behind the structure of my example is to avoid having
to think about indexing into already existing parts.  It *could*
have been written:

    fullmsg.add_alternative(...)
    fullmsg.add_alternative(...., subtype='html')
    with open('python.jpg', 'rb') as python:
        fullmsg.get_payload()[1].add_related(...)

But that means you have to think about payload lists and order of
parts and figure out the right index to use.  But perhaps some people
will find that more congenial than the make_alternative() dance, and I
should probably make sure that I include that alternative in the examples
(that I haven't yet written for the docs).  (I dislike the 'get_payload'
API, by the way.)

The awkwardness of the code in my example arises from the fact that
a Message object is a valid thing to want to use as content (see
message/rfc822 above), so I can't have the code just assume that:

    fullmsg.add_alternative(htmlmsg)

means "make fullmsg into a multipart/alternative and attach htmlmsg",
since in the logic of my API what it naturally means is "make_fullmsg
into a multipart/alternative and attach a new message/rfc822 part
containing htmlmsg".  Thus the make/attach dance instead.

However, what I really want, as I mentioned in my original proposal but
have dropped from my 3.4 proposed code addition, is:


    with open('python.jpg', 'rb') as python:
        fullmsg.add_alternative(
            Webpage('<p>example<\p><img src=image1 \>',
                    {'image1': python.read()}
                   ))

I dropped it from the code proposal because it seems to me that we
should give the community an opportunity to experiment with the content
manager interface before we decide what the best stuff is to include in
the stdlib.

> At this point, "fullmsg.add_related()" without the .payload attribute
> would be an error, unless a "insertPart=True" keyword argument were
> present.  With "insertPart=True", a new top-level multipart/related
> would be interposed with the existing multipart/alternative as its
> first child, and the argument of add_related as the second.  Maybe
> that's too complicated, but I suspect it's harder for people who think
> of MIME messages as trees, than for people who think of messages as
> documents and don't want to hear about mimes other than Marcel
> Marceau.<wink/>

I *think* I see what you are suggesting, but I don't see that it is
easier for the non-tree thinker than my proposal.  My proposal doesn't
even let you make the conversion your insertPart=True would produce.
(That could indeed be a bug, so we may want a "I know what I'm doing"
keyword for 'make_related' and friends).

> The indexing of the .payload attribute by part type is perhaps too
> smart for my own good, haven't thought carefully about it.  It's
> plausible, though, since a message with multiple parts of the same
> type can only have one displayed -- normally that shouldn't happen.
> OTOH, this wouldn't work without modification for multipart/mixed or
> multipart/related.  Could use Yet Another Keyword Argument, maybe.

Ah, good point.  But the mechanism doesn't generalize to other types (eg:
you can have any number of 'image/jpg'), which would make the API quite
inconsistent (you can index by 'text/html' but not by 'image/jpeg'? Wat?)

> (BTW, it's really annoying when the text/plain part refers to images
> etc that are attached only to the text/html part.  AFAICT from RFC
> 2387 it ought to be possible to put the multipart/related part at the
> top so both text/html and text/plain can refer to it.)

Is there a text part format that can actually refer to related parts?
text/plain isn't one, as far as I know.  As I said in another message, it
is worth thinking about supporting more unusual structures.  The tradeoff
is not getting an error when you do something that is in most cases
a mistake.  Absent information about a reasonably common text format
that actually embeds images, I'm inclined to keep the restriction.
We can always remove it later, but we can't add it back later.

>  >     with open('police-report.txt') as report:
>  >         fullmsg.add_attachment(report.read(), filename='p????lice-report.txt',
>  >                                params=dict(wrap='flow'), headers=(
>  >                                     'X-Secret-Level: top',
>  >                                     'X-Authorization: Monty'))
> 
> I can't find an RFC that specifies a "wrap" parameter to text/plain.
> Do you mean RFC 2646 'format="flowed"' here?

Yes, I did.  I just didn't bother to look it up (my apologies), since the
point of the example was that you could add arbitrary extra parameters.

> (A "validate" method could raise a warning on unregistered parameters.)

Yes, which if we want it to happen at the time the parameter is set, means
we probably want a custom (extendible) mechanism for this in the policy.

>  > (Hmm.  Looking at that I see I didn't fully fix a bug I had meant to fix:
>  > some of the parts have a MIME-Version header that don't need it.)
> 
> Another reason why the top-level part should be treated differently in
> the API.

True.  And if we add MIMEPart but keep MIMEMessage as the top level part
rather than re-using the existing Message, we can make the MIMEMessage
policy strict by default.

--David

From eliben at gmail.com  Mon Sep  2 00:57:02 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Sun, 1 Sep 2013 15:57:02 -0700
Subject: [Python-Dev] [Python-checkins] cpython: Further reduce the cost
 of hash collisions by inspecting an additional nearby
In-Reply-To: <3cSLzl490rzSc9@mail.python.org>
References: <3cSLzl490rzSc9@mail.python.org>
Message-ID: <CAF-Rda8zmjwLTVsAZ+hoT1hrUEqEFU0fXcQ=Hqgq1UFrUcM-OA@mail.gmail.com>

On Sat, Aug 31, 2013 at 9:29 PM, raymond.hettinger <
python-checkins at python.org> wrote:

> http://hg.python.org/cpython/rev/d40a65658ff0
> changeset:   85486:d40a65658ff0
> parent:      85482:4d604f1f0219
> user:        Raymond Hettinger <python at rcn.com>
> date:        Sat Aug 31 21:27:08 2013 -0700
> summary:
>   Further reduce the cost of hash collisions by inspecting an additional
> nearby entry.
>


Hi Raymond,

I'm curious about the benchmarks used to test such micro-optimizations. Do
you use one of the existing Python benchmark suites or do you have a
particular set of micro-benchmarks you're running this on?

Eli




>
> files:
>   Objects/setobject.c |  43 +++++++++++++++++++++++++++++---
>   1 files changed, 39 insertions(+), 4 deletions(-)
>
>
> diff --git a/Objects/setobject.c b/Objects/setobject.c
> --- a/Objects/setobject.c
> +++ b/Objects/setobject.c
> @@ -65,10 +65,11 @@
>  The initial probe index is computed as hash mod the table size. Subsequent
>  probe indices are computed as explained in Objects/dictobject.c.
>
> -To improve cache locality, each probe is done in pairs.
> -After the probe is examined, an adjacent entry is then examined as well.
> -The likelihood is that an adjacent entry is in the same cache line and
> -can be examined more cheaply than another probe elsewhere in memory.
> +To improve cache locality, each probe inspects nearby entries before
> +moving on to probes elsewhere in memory.  Depending on alignment and the
> +size of a cache line, the nearby entries are cheaper to inspect than
> +other probes elsewhere in memory.  This probe strategy reduces the cost
> +of hash collisions.
>
>  All arithmetic on hash should ignore overflow.
>
> @@ -130,6 +131,26 @@
>          if (entry->key == dummy && freeslot == NULL)
>              freeslot = entry;
>
> +        entry = &table[j ^ 2];
> +        if (entry->key == NULL)
> +            break;
> +        if (entry->key == key)
> +            return entry;
> +        if (entry->hash == hash && entry->key != dummy) {
> +            PyObject *startkey = entry->key;
> +            Py_INCREF(startkey);
> +            cmp = PyObject_RichCompareBool(startkey, key, Py_EQ);
> +            Py_DECREF(startkey);
> +            if (cmp < 0)
> +                return NULL;
> +            if (table != so->table || entry->key != startkey)
> +                return set_lookkey(so, key, hash);
> +            if (cmp > 0)
> +                return entry;
> +        }
> +        if (entry->key == dummy && freeslot == NULL)
> +            freeslot = entry;
> +
>          i = i * 5 + perturb + 1;
>          j = i & mask;
>          perturb >>= PERTURB_SHIFT;
> @@ -190,6 +211,17 @@
>          if (entry->key == dummy && freeslot == NULL)
>              freeslot = entry;
>
> +        entry = &table[j ^ 2];
> +        if (entry->key == NULL)
> +            break;
> +        if (entry->key == key
> +            || (entry->hash == hash
> +            && entry->key != dummy
> +            && unicode_eq(entry->key, key)))
> +            return entry;
> +        if (entry->key == dummy && freeslot == NULL)
> +            freeslot = entry;
> +
>          i = i * 5 + perturb + 1;
>          j = i & mask;
>          perturb >>= PERTURB_SHIFT;
> @@ -258,6 +290,9 @@
>          entry = &table[j ^ 1];
>          if (entry->key == NULL)
>              break;
> +        entry = &table[j ^ 2];
> +        if (entry->key == NULL)
> +            break;
>          i = i * 5 + perturb + 1;
>          j = i & mask;
>          perturb >>= PERTURB_SHIFT;
>
> --
> Repository URL: http://hg.python.org/cpython
>
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> http://mail.python.org/mailman/listinfo/python-checkins
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130901/1873bb9b/attachment.html>

From eliben at gmail.com  Mon Sep  2 00:58:54 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Sun, 1 Sep 2013 15:58:54 -0700
Subject: [Python-Dev] [Python-checkins] cpython: Issue #11798: fix tests
 for regrtest -R :
In-Reply-To: <3cSMds1ZCzzPrk@mail.python.org>
References: <3cSMds1ZCzzPrk@mail.python.org>
Message-ID: <CAF-Rda_5sPNkM+pOh5VB3S43t7+dh5jSQSpQ0q_bemMesgRjrA@mail.gmail.com>

On Sat, Aug 31, 2013 at 9:58 PM, andrew.svetlov
<python-checkins at python.org>wrote:

> http://hg.python.org/cpython/rev/39781c3737f8
> changeset:   85490:39781c3737f8
> user:        Andrew Svetlov <andrew.svetlov at gmail.com>
> date:        Sun Sep 01 07:58:41 2013 +0300
> summary:
>   Issue #11798: fix tests for regrtest -R :
>
> files:
>   Lib/test/regrtest.py            |  5 +++++
>   Lib/unittest/suite.py           |  8 ++++++--
>   Lib/unittest/test/test_suite.py |  8 ++++++++
>   3 files changed, 19 insertions(+), 2 deletions(-)
>
>
>
Hi Andrew,

It would help if you could add more details into the commit message. This
would make both post-commit reviews and future code archeology simpler.

Eli



> diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py
> --- a/Lib/test/regrtest.py
> +++ b/Lib/test/regrtest.py
> @@ -496,6 +496,8 @@
>
>      if ns.slaveargs is not None:
>          args, kwargs = json.loads(ns.slaveargs)
> +        if kwargs.get('huntrleaks'):
> +            unittest.BaseTestSuite._cleanup = False
>          try:
>              result = runtest(*args, **kwargs)
>          except KeyboardInterrupt:
> @@ -528,6 +530,9 @@
>              #gc.set_debug(gc.DEBUG_SAVEALL)
>              found_garbage = []
>
> +    if ns.huntrleaks:
> +        unittest.BaseTestSuite._cleanup = False
> +
>      if ns.single:
>          filename = os.path.join(TEMPDIR, 'pynexttest')
>          try:
> diff --git a/Lib/unittest/suite.py b/Lib/unittest/suite.py
> --- a/Lib/unittest/suite.py
> +++ b/Lib/unittest/suite.py
> @@ -16,6 +16,8 @@
>  class BaseTestSuite(object):
>      """A simple test suite that doesn't provide class or module shared
> fixtures.
>      """
> +    _cleanup = True
> +
>      def __init__(self, tests=()):
>          self._tests = []
>          self.addTests(tests)
> @@ -61,7 +63,8 @@
>              if result.shouldStop:
>                  break
>              test(result)
> -            self._removeTestAtIndex(index)
> +            if self._cleanup:
> +                self._removeTestAtIndex(index)
>          return result
>
>      def _removeTestAtIndex(self, index):
> @@ -115,7 +118,8 @@
>              else:
>                  test.debug()
>
> -            self._removeTestAtIndex(index)
> +            if self._cleanup:
> +                self._removeTestAtIndex(index)
>
>          if topLevel:
>              self._tearDownPreviousClass(None, result)
> diff --git a/Lib/unittest/test/test_suite.py
> b/Lib/unittest/test/test_suite.py
> --- a/Lib/unittest/test/test_suite.py
> +++ b/Lib/unittest/test/test_suite.py
> @@ -303,6 +303,9 @@
>          suite.run(unittest.TestResult())
>
>      def test_remove_test_at_index(self):
> +        if not unittest.BaseTestSuite._cleanup:
> +            raise unittest.SkipTest("Suite cleanup is disabled")
> +
>          suite = unittest.TestSuite()
>
>          suite._tests = [1, 2, 3]
> @@ -311,6 +314,9 @@
>          self.assertEqual([1, None, 3], suite._tests)
>
>      def test_remove_test_at_index_not_indexable(self):
> +        if not unittest.BaseTestSuite._cleanup:
> +            raise unittest.SkipTest("Suite cleanup is disabled")
> +
>          suite = unittest.TestSuite()
>          suite._tests = None
>
> @@ -318,6 +324,8 @@
>          suite._removeTestAtIndex(2)
>
>      def assert_garbage_collect_test_after_run(self, TestSuiteClass):
> +        if not unittest.BaseTestSuite._cleanup:
> +            raise unittest.SkipTest("Suite cleanup is disabled")
>
>          class Foo(unittest.TestCase):
>              def test_nothing(self):
>
> --
> Repository URL: http://hg.python.org/cpython
>
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> http://mail.python.org/mailman/listinfo/python-checkins
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130901/71a4d40b/attachment.html>

From eliben at gmail.com  Mon Sep  2 00:59:12 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Sun, 1 Sep 2013 15:59:12 -0700
Subject: [Python-Dev] [Python-checkins] cpython: Update whatsnew/3.4.rst
 wrt. the socket constants switch to IntEnum
In-Reply-To: <522281B1.50004@udel.edu>
References: <3cSBmf31XfzPkT@mail.python.org> <522281B1.50004@udel.edu>
Message-ID: <CAF-Rda8ho7KwWrQSSRxe6THq9zTzwQWCbc+jDg=TTE=NOUq=Ug@mail.gmail.com>

On Sat, Aug 31, 2013 at 4:52 PM, Terry Reedy <tjreedy at udel.edu> wrote:

> On 8/31/2013 6:19 PM, eli.bendersky wrote:
>
>> http://hg.python.org/cpython/**rev/4d604f1f0219<http://hg.python.org/cpython/rev/4d604f1f0219>
>> changeset:   85482:4d604f1f0219
>> user:        Eli Bendersky <eliben at gmail.com>
>> date:        Sat Aug 31 15:18:48 2013 -0700
>> summary:
>>    Update whatsnew/3.4.rst wrt. the socket constants switch to IntEnum
>>
>> [issue #18730]
>>
>
>
> Wrong issue number I think.
> _______________________________________________
>

Oops, yes. Sorry about that.

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130901/2c77d9d2/attachment.html>

From v+python at g.nevcal.com  Mon Sep  2 01:08:18 2013
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Sun, 01 Sep 2013 16:08:18 -0700
Subject: [Python-Dev] Completing the email6 API changes.
In-Reply-To: <20130901221040.DC76E250818@webabinitio.net>
References: <20130831052136.42D4B2507F5@webabinitio.net>
 <87bo4echcr.fsf@uwakimon.sk.tsukuba.ac.jp>
 <20130901221040.DC76E250818@webabinitio.net>
Message-ID: <5223C8E2.2000006@g.nevcal.com>

On 9/1/2013 3:10 PM, R. David Murray wrote:
> This doesn't work, though, because you could (although you usually
> won't) have more than one 'text/html' part in a single multipart.

I was traveling and your original message is still unread in my queue of 
"things to look at later" :(  I haven't caught up with old stuff yet, 
but am trying to stay current on current stuff...

The quoted issue was mentioned in another message in this thread, though 
in different terms.

I recall being surprised when first seeing messages generated by Apple 
Mail software, that are multipart/related, having a sequence of 
intermixed text/plain and image/jpeg parts. This is apparently how Apple 
Mail generates messages that have inline pictures, without resorting to 
use of HTML mail. Other email clients handle this relatively better or 
worse, depending on the expectations of their authors! Several of them 
treat all the parts after the initial text/html part as attachments; 
some of them display inline attachments if they are text/html or 
image/jpeg and others do not. I can't say for sure if there are other 
ways they are treated; I rather imagine that Apple Mail displays the 
whole message with interspersed pictures quite effectively, without 
annoying the user with attachment "markup", but I'm not an Apple Mail 
user so I couldn't say for sure.

You should, of course, ensure that it is possible to create such a message.

Whether Apple Mail does that with other embedded image/* formats, or 
with other text/* formats, or other non-image, non-text formats, I 
couldn't say. I did attempt to determine if it was non-standard usage: 
it is certainly non-common usage, but I found nothing in the email/MIME 
RFCs that precludes such usage.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130901/d4f7c39c/attachment-0001.html>

From tjreedy at udel.edu  Mon Sep  2 01:25:31 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 01 Sep 2013 19:25:31 -0400
Subject: [Python-Dev] cpython (merge 3.3 -> default): Merge fix from 3.3
	into default.
In-Reply-To: <20130901230414.44af5426@fsol>
References: <3cSn1P710Hz7LjX@mail.python.org> <20130901230414.44af5426@fsol>
Message-ID: <l00ida$1v5$1@ger.gmane.org>

On 9/1/2013 5:04 PM, Antoine Pitrou wrote:
> On Sun,  1 Sep 2013 23:02:17 +0200 (CEST)
> tim.peters <python-checkins at python.org> wrote:

>> Windows developers:  to get the correct line endings in your checkout,
>> delete Lib\test\xmltestdata, and then "hg revert" that directory.

Or, in Tortoisehg Workbenck, select the four Working Directory patch 
entries, right click, and revert.

>> Why the Windows buildbots didn't fail test_sax remains a mystery :-(
>
> Probably because they don't have the hgeol extension enabled.

Since the tests also failed on installed Python, it seems that the .msi 
installer is created in a repository with the extension enabled. If so, 
I think that the buildbots should have the extension enabled also, so 
that they are testing Python as it will be distributed.

Or we could stop supporting Notepad and change what we distribute ;-).

-- 
Terry Jan Reedy


From tjreedy at udel.edu  Mon Sep  2 01:38:58 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 01 Sep 2013 19:38:58 -0400
Subject: [Python-Dev] 'Subinterpreter' (was Re: Pre-PEP: Redesigning
	extension modules)
In-Reply-To: <l00alo$qlc$1@ger.gmane.org>
References: <kv77o2$f96$1@ger.gmane.org> <20130823111822.49cba700@pitrou.net>
 <CADiSq7dhzpTWMw7gcBRrzZot7jaS=Aj=aukq0+jh_cK2+F_kGQ@mail.gmail.com>
 <CADiSq7dBruZiWhObnjKbcneji9+ziXmyRpg1+72a8YeMWbyO3A@mail.gmail.com>
 <kvabt8$jfc$1@ger.gmane.org>
 <CADiSq7c6OjFa8zo7sMB=k2zLSmaf1w-tAwhYKSQ26T3caLyj6A@mail.gmail.com>
 <kvaka3$sbu$1@ger.gmane.org>
 <CADiSq7dzQqyH9i6g9QYWp6_ZHHUD4+4npYo6of6mCNahQ+w8kw@mail.gmail.com>
 <kvcr9f$4gs$1@ger.gmane.org>
 <CADiSq7cfNH1M29W19BW88-BBQaWEpaMkE=s09wNyT5VGOrPB+g@mail.gmail.com>
 <kvtfek$23p$1@ger.gmane.org>
 <CADiSq7d3L1+Jc5E4dWYDRUQPNDCmxqyc3GzbgXJvGMe4+OnWrQ@mail.gmail.com>
 <20130901150351.2f2aa44c@fsol>
 <CADiSq7e=E+WCstffBsREc6KKrBU5mbKFWiriDRt6orAKsQWnVQ@mail.gmail.com>
 <20130901194048.0d7d1b75@fsol> <l006gp$jb7$1@ger.gmane.org>
 <20130901220649.3e9c3ab4@fsol> <l00alo$qlc$1@ger.gmane.org>
Message-ID: <l00j6i$97n$1@ger.gmane.org>

On 9/1/2013 5:13 PM, Stefan Behnel wrote:
> Antoine Pitrou, 01.09.2013 22:06:
>> On Sun, 01 Sep 2013 16:02:33 -0400
>> Terry Reedy wrote:
>>>> Speaking of which, it also doesn't work (well) with subinterpreters:
>>>
>>> Could someone briefly explain 'subinterpreter' or point me somewhere in
>>> the docs? It appears throughout this thread but there is no index or
>>> glossary entry.
>>
>> http://docs.python.org/dev/c-api/init.html#sub-interpreter-support

So cpython specific.

>> Subinterpreters are a somewhat borderline feature that allows embedding
>> applications to host multiple Python programs in a single process.  A
>> well-known example is mod_wsgi.

Thank you for both the link *and* the explanatory example, which just 
what I needed to make the past discussion more intelligible. I imagine 
that wsgi uses a sub-interpreter for each user connection.

> And extension modules usually don't play well with subinterpreters because
> each subinterpreter requires its own separate version of the module and
> extension modules are rarely designed to keep their state completely local
> to an interpreter, let alone being prepared for having their module init
> function be called more than once.

I can see now why this is a bit of a 'hair-puller';-).

-- 
Terry Jan Reedy


From tjreedy at udel.edu  Mon Sep  2 01:42:10 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 01 Sep 2013 19:42:10 -0400
Subject: [Python-Dev] [Python-checkins] cpython: Update whatsnew/3.4.rst
 wrt. the socket constants switch to IntEnum
In-Reply-To: <CAF-Rda8ho7KwWrQSSRxe6THq9zTzwQWCbc+jDg=TTE=NOUq=Ug@mail.gmail.com>
References: <3cSBmf31XfzPkT@mail.python.org> <522281B1.50004@udel.edu>
 <CAF-Rda8ho7KwWrQSSRxe6THq9zTzwQWCbc+jDg=TTE=NOUq=Ug@mail.gmail.com>
Message-ID: <l00jch$97n$2@ger.gmane.org>

On 9/1/2013 6:59 PM, Eli Bendersky wrote:
>
>
>
> On Sat, Aug 31, 2013 at 4:52 PM, Terry Reedy <tjreedy at udel.edu
> <mailto:tjreedy at udel.edu>> wrote:
>
>     On 8/31/2013 6:19 PM, eli.bendersky wrote:

>         [issue #18730]

>     Wrong issue number I think.
>     _______________________________________________
>
>
> Oops, yes. Sorry about that.

I unlinked it from 18730.


-- 
Terry Jan Reedy


From eliben at gmail.com  Mon Sep  2 03:02:30 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Sun, 1 Sep 2013 18:02:30 -0700
Subject: [Python-Dev] SEEK_* constants in io and os
Message-ID: <CAF-Rda_2j=HTDJ5nEdKWxX1mCmbHLgUNe_zyDYXzGHE6-LUBmg@mail.gmail.com>

Hello,

I was looking at the possibility of replacing the SEEK_* constants by
IntEnums, and the first thing that catches attention is that these
constants are defined in both Lib/os.py and Lib/io.py; both places also
recently started supporting SEEK_HOLE and SEEK_DATA (though here io refers
to os.SEEK_HOLE and os.SEEK_DATA).

Additional data points: other modules take these constants as arguments -
* mmap: directs to use os.SEEK_*
* chunk and fcntk: spell out the numeric values.

os seems to import io in some functions; can this be done always? If yes,
we can just define the constants once and os.SEEK_* will alias io.SEEK_*?
The other way (io taking from os) is also a possibility (maybe the
preferred one because io already refers to os.SEEK_HOLE/DATA, at least in
the documentation).

Any ideas and suggestions are welcome,

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130901/e9a9c8cb/attachment.html>

From db3l.net at gmail.com  Mon Sep  2 03:08:57 2013
From: db3l.net at gmail.com (David Bolen)
Date: Sun, 01 Sep 2013 21:08:57 -0400
Subject: [Python-Dev] cpython (merge 3.3 -> default): Merge fix from 3.3
	into default.
References: <3cSn1P710Hz7LjX@mail.python.org> <20130901230414.44af5426@fsol>
 <l00ida$1v5$1@ger.gmane.org>
Message-ID: <m2sixouj12.fsf@valheru.db3l.homeip.net>

Terry Reedy <tjreedy at udel.edu> writes:

> On 9/1/2013 5:04 PM, Antoine Pitrou wrote:
>> Probably because they don't have the hgeol extension enabled.

Yes, I believe that's correct, at least for my Windows buildbots.

> Since the tests also failed on installed Python, it seems that the
> .msi installer is created in a repository with the extension
> enabled. If so, I think that the buildbots should have the extension
> enabled also, so that they are testing Python as it will be
> distributed.

If it should be enabled, I believe it will have to be done globally,
so affecting all buildbot branches (not that it sounds like that would
be an issue).  The individual branch repositories are physically
removed at times (such as during a failure to update, which escalates
to a removal followed by full clone, as well as I believe during a
custom build), so I don't believe persistent local changes to the
repository's own hgrc are possible in a reliable way.

In the event we did want per-repository control, then I think we'd
need to have some sort of additional script that ran as part of the
buildbot build process to ensure the right contents for the local
repository hgrc following a clone, or just prior to the update step.

-- David


From andrew.svetlov at gmail.com  Mon Sep  2 04:37:20 2013
From: andrew.svetlov at gmail.com (Andrew Svetlov)
Date: Mon, 2 Sep 2013 05:37:20 +0300
Subject: [Python-Dev] [Python-checkins] cpython: Issue #11798: fix tests
 for regrtest -R :
In-Reply-To: <CAF-Rda_5sPNkM+pOh5VB3S43t7+dh5jSQSpQ0q_bemMesgRjrA@mail.gmail.com>
References: <3cSMds1ZCzzPrk@mail.python.org>
 <CAF-Rda_5sPNkM+pOh5VB3S43t7+dh5jSQSpQ0q_bemMesgRjrA@mail.gmail.com>
Message-ID: <CAL3CFcUoq2ihwQ4GEOopCsVNC29Dyvu7wx8ofhuBcJa-fUK=oQ@mail.gmail.com>

regrtest -R runs test suites several times. That's why test cleanup
should be disabled for this case.
Details discussed in issue.
I'll do more expressive commit messages next time.
Thanks.


On Mon, Sep 2, 2013 at 1:58 AM, Eli Bendersky <eliben at gmail.com> wrote:
>
>
>
> On Sat, Aug 31, 2013 at 9:58 PM, andrew.svetlov <python-checkins at python.org>
> wrote:
>>
>> http://hg.python.org/cpython/rev/39781c3737f8
>> changeset:   85490:39781c3737f8
>> user:        Andrew Svetlov <andrew.svetlov at gmail.com>
>> date:        Sun Sep 01 07:58:41 2013 +0300
>> summary:
>>   Issue #11798: fix tests for regrtest -R :
>>
>> files:
>>   Lib/test/regrtest.py            |  5 +++++
>>   Lib/unittest/suite.py           |  8 ++++++--
>>   Lib/unittest/test/test_suite.py |  8 ++++++++
>>   3 files changed, 19 insertions(+), 2 deletions(-)
>>
>>
>
> Hi Andrew,
>
> It would help if you could add more details into the commit message. This
> would make both post-commit reviews and future code archeology simpler.
>
> Eli
>
>
>>
>> diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py
>> --- a/Lib/test/regrtest.py
>> +++ b/Lib/test/regrtest.py
>> @@ -496,6 +496,8 @@
>>
>>      if ns.slaveargs is not None:
>>          args, kwargs = json.loads(ns.slaveargs)
>> +        if kwargs.get('huntrleaks'):
>> +            unittest.BaseTestSuite._cleanup = False
>>          try:
>>              result = runtest(*args, **kwargs)
>>          except KeyboardInterrupt:
>> @@ -528,6 +530,9 @@
>>              #gc.set_debug(gc.DEBUG_SAVEALL)
>>              found_garbage = []
>>
>> +    if ns.huntrleaks:
>> +        unittest.BaseTestSuite._cleanup = False
>> +
>>      if ns.single:
>>          filename = os.path.join(TEMPDIR, 'pynexttest')
>>          try:
>> diff --git a/Lib/unittest/suite.py b/Lib/unittest/suite.py
>> --- a/Lib/unittest/suite.py
>> +++ b/Lib/unittest/suite.py
>> @@ -16,6 +16,8 @@
>>  class BaseTestSuite(object):
>>      """A simple test suite that doesn't provide class or module shared
>> fixtures.
>>      """
>> +    _cleanup = True
>> +
>>      def __init__(self, tests=()):
>>          self._tests = []
>>          self.addTests(tests)
>> @@ -61,7 +63,8 @@
>>              if result.shouldStop:
>>                  break
>>              test(result)
>> -            self._removeTestAtIndex(index)
>> +            if self._cleanup:
>> +                self._removeTestAtIndex(index)
>>          return result
>>
>>      def _removeTestAtIndex(self, index):
>> @@ -115,7 +118,8 @@
>>              else:
>>                  test.debug()
>>
>> -            self._removeTestAtIndex(index)
>> +            if self._cleanup:
>> +                self._removeTestAtIndex(index)
>>
>>          if topLevel:
>>              self._tearDownPreviousClass(None, result)
>> diff --git a/Lib/unittest/test/test_suite.py
>> b/Lib/unittest/test/test_suite.py
>> --- a/Lib/unittest/test/test_suite.py
>> +++ b/Lib/unittest/test/test_suite.py
>> @@ -303,6 +303,9 @@
>>          suite.run(unittest.TestResult())
>>
>>      def test_remove_test_at_index(self):
>> +        if not unittest.BaseTestSuite._cleanup:
>> +            raise unittest.SkipTest("Suite cleanup is disabled")
>> +
>>          suite = unittest.TestSuite()
>>
>>          suite._tests = [1, 2, 3]
>> @@ -311,6 +314,9 @@
>>          self.assertEqual([1, None, 3], suite._tests)
>>
>>      def test_remove_test_at_index_not_indexable(self):
>> +        if not unittest.BaseTestSuite._cleanup:
>> +            raise unittest.SkipTest("Suite cleanup is disabled")
>> +
>>          suite = unittest.TestSuite()
>>          suite._tests = None
>>
>> @@ -318,6 +324,8 @@
>>          suite._removeTestAtIndex(2)
>>
>>      def assert_garbage_collect_test_after_run(self, TestSuiteClass):
>> +        if not unittest.BaseTestSuite._cleanup:
>> +            raise unittest.SkipTest("Suite cleanup is disabled")
>>
>>          class Foo(unittest.TestCase):
>>              def test_nothing(self):
>>
>> --
>> Repository URL: http://hg.python.org/cpython
>>
>> _______________________________________________
>> Python-checkins mailing list
>> Python-checkins at python.org
>> http://mail.python.org/mailman/listinfo/python-checkins
>>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com
>



-- 
Thanks,
Andrew Svetlov

From stephen at xemacs.org  Mon Sep  2 05:03:40 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Mon, 02 Sep 2013 12:03:40 +0900
Subject: [Python-Dev] Completing the email6 API changes.
In-Reply-To: <5223C8E2.2000006@g.nevcal.com>
References: <20130831052136.42D4B2507F5@webabinitio.net>
 <87bo4echcr.fsf@uwakimon.sk.tsukuba.ac.jp>
 <20130901221040.DC76E250818@webabinitio.net>
 <5223C8E2.2000006@g.nevcal.com>
Message-ID: <874na3diwj.fsf@uwakimon.sk.tsukuba.ac.jp>

This is getting off-topic IMO; we should probably take this thread to
email-sig.

Glenn Linderman writes:

 > I recall being surprised when first seeing messages generated by
 > Apple Mail software, that are multipart/related, having a sequence
 > of intermixed text/plain and image/jpeg parts. This is apparently
 > how Apple Mail generates messages that have inline pictures,
 > without resorting to use of HTML mail.

(Are you sure you mean "text/plain" above?  I've not seen this form of
message.  And you mention only "text/html" below.)

This practice (like my suggestion) is based on the conjecture that
MUAs that implement multipart/related will treat it as multipart/mixed
if the "main" subpart isn't known to implement links to external
entities.

 > Other email clients handle this relatively better or worse,
 > depending on the expectations of their authors!

Sure.  After all, this is a world in which some MUAs have a long
history of happily executing virus executables.

 > I did attempt to determine if it was non-standard usage: it is
 > certainly non-common usage, but I found nothing in the email/MIME
 > RFCs that precludes such usage.

Clearly RFCs 2046 and 2387 envision a fallback to multipart/mixed, but
are silent on how to do it for MUAs that implement multipart/related.
RFC 2387 says:

    MIME User Agents that do recognize Multipart/Related entities but
    are unable to process the given type should give the user the
    option of suppressing the entire Multipart/Related body part shall
    be. [...]  Handling Multipart/Related differs [from handling of
    existing composite subtypes] in that processing cannot be reduced
    to handling the individual entities.

I think that the sane policy is that when processing multipart/related
internally, the MUA should treat the whole as multipart/mixed, unless
it knows how links are implemented in the "start" part.  But the RFC
doesn't say that.

 > Several of them treat all the parts after the initial text/html
 > part as attachments;

They don't implement RFC 2387 (promoted to draft standard in 1998,
following two others, the earlier being RFC 1872 from 1995).  Too bad
for their users.  But what I'm worried about is a different issue,
which is how to ensure that multipart/alternative messages present all
relevant content entities in both presentations.  For example, the
following hypothetical structure is efficient:

    multipart/alternative
        text/plain
        multipart/related
            text/html
            application/x-opentype-font

because the text/plain can't use the font.  But this

    multipart/alternative
        text/plain
        multipart/related
            text/html
            image/png
            image/png

often cost the text/plain receiver a view of the images, and I don't
see any way to distinguish the two cases.  (The images might be
character glyphs, for example, effectively a "poor man's font".)
OTOH, if the message is structured

    multipart/related
        multipart/alternative
            text/plain
            text/html
        image/png
        image/png

the receiver can infer that the images are related to both text/*
parts and DTRT for each.

Steve


From v+python at g.nevcal.com  Mon Sep  2 07:55:58 2013
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Sun, 01 Sep 2013 22:55:58 -0700
Subject: [Python-Dev] Completing the email6 API changes.
In-Reply-To: <874na3diwj.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <20130831052136.42D4B2507F5@webabinitio.net>
 <87bo4echcr.fsf@uwakimon.sk.tsukuba.ac.jp>
 <20130901221040.DC76E250818@webabinitio.net> <5223C8E2.2000006@g.nevcal.com>
 <874na3diwj.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <5224286E.4040409@g.nevcal.com>

On 9/1/2013 8:03 PM, Stephen J. Turnbull wrote:
> This is getting off-topic IMO; we should probably take this thread to
> email-sig.

Probably, but you didn't :)

> Glenn Linderman writes:
>
>   > I recall being surprised when first seeing messages generated by
>   > Apple Mail software, that are multipart/related, having a sequence
>   > of intermixed text/plain and image/jpeg parts. This is apparently
>   > how Apple Mail generates messages that have inline pictures,
>   > without resorting to use of HTML mail.
>
> (Are you sure you mean "text/plain" above?  I've not seen this form of
> message.  And you mention only "text/html" below.)

Yes, I'm sure it was text/plain. I may be able to access the archived 
discussion from a non-Python mailing list about it, to verify, if that 
becomes important.  But now that you mention mulitpart/mixed, I'm not 
sure if it was multipart/related or mulitpart/mixed for the grouping 
MIME part. Perhaps someone with Apple Mail could produce one... probably 
by composing a message as text/plain, and dragging in a picture or two.

The other references to text/html was in error.

> This practice (like my suggestion) is based on the conjecture that
> MUAs that implement multipart/related will treat it as multipart/mixed
> if the "main" subpart isn't known to implement links to external
> entities.
>
>   > Other email clients handle this relatively better or worse,
>   > depending on the expectations of their authors!
>
> Sure.  After all, this is a world in which some MUAs have a long
> history of happily executing virus executables.
>
>   > I did attempt to determine if it was non-standard usage: it is
>   > certainly non-common usage, but I found nothing in the email/MIME
>   > RFCs that precludes such usage.
>
> Clearly RFCs 2046 and 2387 envision a fallback to multipart/mixed, but
> are silent on how to do it for MUAs that implement multipart/related.
> RFC 2387 says:
>
>      MIME User Agents that do recognize Multipart/Related entities but
>      are unable to process the given type should give the user the
>      option of suppressing the entire Multipart/Related body part shall
>      be. [...]  Handling Multipart/Related differs [from handling of
>      existing composite subtypes] in that processing cannot be reduced
>      to handling the individual entities.
>
> I think that the sane policy is that when processing multipart/related
> internally, the MUA should treat the whole as multipart/mixed, unless
> it knows how links are implemented in the "start" part.  But the RFC
> doesn't say that.
>
>   > Several of them treat all the parts after the initial text/html
>   > part as attachments;
>
> They don't implement RFC 2387 (promoted to draft standard in 1998,
> following two others, the earlier being RFC 1872 from 1995).  Too bad
> for their users.

Correct... but the MUA receiving the Apple Mail message I was talking 
about being a text-mostly MUA, it is probably a reasonable method of 
handling them.

> But what I'm worried about is a different issue,
> which is how to ensure that multipart/alternative messages present all
> relevant content entities in both presentations.  For example, the
> following hypothetical structure is efficient:
>
>      multipart/alternative
>          text/plain
>          multipart/related
>              text/html
>              application/x-opentype-font
>
> because the text/plain can't use the font.  But this
>
>      multipart/alternative
>          text/plain
>          multipart/related
>              text/html
>              image/png
>              image/png
>
> often cost the text/plain receiver a view of the images, and I don't
> see any way to distinguish the two cases.  (The images might be
> character glyphs, for example, effectively a "poor man's font".)

Yes, that issue is handled by some text MUA by showing the image/png (or 
anything in such a position) as attachments. Again, being text-mostly, 
that might be a reasonable way of handling them. Perhaps the standard 
says they should be ignored, when displaying text/plain alternative.

> OTOH, if the message is structured
>
>      multipart/related
>          multipart/alternative
>              text/plain
>              text/html
>          image/png
>          image/png
>
> the receiver can infer that the images are related to both text/*
> parts and DTRT for each.
>
With the images being treated as attachments. Or is there a syntax to 
allow the text/html to embed the images and the text/plain to see them 
as attachments?  I think the text/html wants to refer to things within 
its containing multipart/related, but am not sure if that allows the 
intervening multipart/alternative.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130901/874f57b8/attachment.html>

From stephen at xemacs.org  Mon Sep  2 09:06:53 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Mon, 02 Sep 2013 16:06:53 +0900
Subject: [Python-Dev] Completing the email6 API changes.
In-Reply-To: <5224286E.4040409@g.nevcal.com>
References: <20130831052136.42D4B2507F5@webabinitio.net>
 <87bo4echcr.fsf@uwakimon.sk.tsukuba.ac.jp>
 <20130901221040.DC76E250818@webabinitio.net>
 <5223C8E2.2000006@g.nevcal.com>
 <874na3diwj.fsf@uwakimon.sk.tsukuba.ac.jp>
 <5224286E.4040409@g.nevcal.com>
Message-ID: <87y57fbt2q.fsf@uwakimon.sk.tsukuba.ac.jp>

>>>>> Glenn writes:
> >>>>> Steve writes:

>> OTOH, if the message is structured
>>
>>      multipart/related
>>          multipart/alternative
>>              text/plain
>>              text/html
>>          image/png
>>          image/png
>>
>> the receiver can infer that the images are related to both text/*
>> parts and DTRT for each.

>With the images being treated as attachments. Or is there a syntax to 
>allow the text/html to embed the images and the text/plain to see them 
>as attachments?

I believe the above is that syntax.  But the standard doesn't say
anything about this.  The standard for multipart/alternative is RFC
2046, which doesn't know about multipart/related.  RFC 2387 doesn't
update RFC 2046, so it doesn't say anything about
multipart/alternative within multipart/related, either.

>I think the text/html wants to refer to things within its containing
>multipart/related, but am not sure if that allows the intervening
>multipart/alternative.

I don't see why not.  But it would depend on the implementations,
which we'll have to test before recommending the structure I
(theoretically :-) prefer.e

From solipsis at pitrou.net  Mon Sep  2 10:16:42 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 2 Sep 2013 10:16:42 +0200
Subject: [Python-Dev] Pre-PEP: Redesigning extension modules
References: <kv77o2$f96$1@ger.gmane.org>
 <CALeMXf5oxi0oX13zSfHwgVi1eXF_z_coY23MfBP-YOK8WVkWsg@mail.gmail.com>
 <kvctoj$qsv$1@ger.gmane.org> <kvstdj$jo0$1@ger.gmane.org>
 <CADiSq7c5xtEcPBXGPWqQosfgGcb2n+2AnHiAWFoATeYfQTUcEQ@mail.gmail.com>
Message-ID: <20130902101642.04b69588@pitrou.net>

Le Sun, 1 Sep 2013 02:19:48 +1000,
Nick Coghlan <ncoghlan at gmail.com> a ?crit :
> On 1 Sep 2013 00:10, "Stefan Behnel" <stefan_ml at behnel.de> wrote:
> >
> > *bump*
> >
> > Does this sound like a viable solution?
> 
> This isn't likely to progress until we have Eric's PEP 451 to a point
> where it's ready for python-dev discussion and pronouncement.
> 
> However, the revised loader API is being designed to allow for the
> loader returning arbitrary objects, so something along these lines
> should work. There will likely be some adjustments to the API
> signature to allow extension modules to optionally support reloading
> if they so desire.

I think the biggest challenge here is to propose an API that's simple
and easy to use (i.e. that doesn't make extension module writing more
complicated than it currently is).

The basic concept of putting custom module objects in sys.modules is
sound, IMHO.

As for "extension module as a wrapper", though, it shounds like the
kind of complication I would personally prefer to stay away from. Also,
it would make extension modules less like Python modules, rather than
more.

Regards

Antoine.



> 
> Cheers,
> Nick.
> 
> >
> > Stefan
> >
> >
> > Stefan Behnel, 25.08.2013 14:36:
> > > Hi,
> > >
> > > thanks for bringing this up. It clearly shows that there is more
> > > to this problem than I initially thought.
> > >
> > > Let me just add one idea that your post gave me.
> > >
> > > PJ Eby, 25.08.2013 06:12:
> > >> My "Importing" package offers lazy imports by creating module
> > >> objects in sys.modules that are a subtype of ModuleType, and use
> > >> a __getattribute__ hook so that trying to use them fires off a
> > >> reload() of the module.
> > >
> > > I wonder if this wouldn't be an approach to fix the reloading
> > > problem in general. What if extension module loading, at least
> > > with the new scheme, didn't return the module object itself and
> > > put it into sys.modules but created a wrapper that redirects its
> > > __getattr__ and __setattr__ to the actual module object? That
> > > would have a tiny performance impact on attribute access, but I'd
> > > expect that to be negligible given that the
> usual
> > > reason for the extension module to exist is that it does
> > > non-trivial
> stuff
> > > in whatever its API provides. Reloading could then really create a
> > > completely new module object and replace the reference inside of
> > > the
> wrapper.
> > >
> > > That way, code that currently uses "from extmodule import xyz"
> > > would continue to see the original version of the module as of
> > > the time of its import, and code that just did "import extmodule"
> > > and then used
> attribute
> > > access at need would always see the current content of the module
> > > as it
> was
> > > last loaded. I think that, together with keeping module global
> > > state in
> the
> > > module object itself, would nicely fix both cases.
> > >
> > > Stefan
> >
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > http://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
> 



From solipsis at pitrou.net  Mon Sep  2 10:18:17 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 2 Sep 2013 10:18:17 +0200
Subject: [Python-Dev] Add a new tracemalloc module to trace memory
	allocations
References: <CAMpsgwaa3pABNrMSw2S7-Og111u8DdXK6dYnWVJzS4TAKbFNEQ@mail.gmail.com>
 <CAMpsgwatDdFQg4q=b3s7W6BZ+WQmpo=KTeJZd7ESbuGG+RzaKA@mail.gmail.com>
 <CAGE7PNJ9DrKQ8Zb-E-F7QPRTP=t+S8StBvm3OQgvct8539gFPg@mail.gmail.com>
 <CAMpsgwZBeN7W+GK6hLSHp4VYXAYgWFQBTx_-K3J7KzOMDB3H=g@mail.gmail.com>
 <CADiSq7dKojZ5f1bprhZZxu6DjovjWtwy-8o=-BYjYRvHZWqoFw@mail.gmail.com>
Message-ID: <20130902101817.1a6d1c5d@pitrou.net>

Le Sun, 1 Sep 2013 11:36:51 +1000,
Nick Coghlan <ncoghlan at gmail.com> a ?crit :
> +1 from me for both tracemalloc and failmalloc in the same vein as
> faulthandler (and using similar API concepts to faulthandler).
> 
> While I like the flat top level naming structure, we should clearly
> document these as implementation dependent runtime services. Other
> implementations may not provide them at all, and even if they do, many
> details will likely be different.

So, it sounds like Victor may write a new PEP soon :-)

+1 from me on a flat namespace. Non-designed naming hierarchies are
just a pain to work with.

Regards

Antoine.



From solipsis at pitrou.net  Mon Sep  2 10:24:37 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 2 Sep 2013 10:24:37 +0200
Subject: [Python-Dev] SEEK_* constants in io and os
References: <CAF-Rda_2j=HTDJ5nEdKWxX1mCmbHLgUNe_zyDYXzGHE6-LUBmg@mail.gmail.com>
Message-ID: <20130902102437.197c1708@pitrou.net>

Le Sun, 1 Sep 2013 18:02:30 -0700,
Eli Bendersky <eliben at gmail.com> a ?crit :
> Hello,
> 
> I was looking at the possibility of replacing the SEEK_* constants by
> IntEnums, and the first thing that catches attention is that these
> constants are defined in both Lib/os.py and Lib/io.py; both places
> also recently started supporting SEEK_HOLE and SEEK_DATA (though here
> io refers to os.SEEK_HOLE and os.SEEK_DATA).

What is the runtime cost of doing so? os is a fundamental module that is
imported by almost every Python program.

Regards

Antoine.



From ncoghlan at gmail.com  Mon Sep  2 15:02:24 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 2 Sep 2013 23:02:24 +1000
Subject: [Python-Dev] Pre-PEP: Redesigning extension modules
In-Reply-To: <20130902101642.04b69588@pitrou.net>
References: <kv77o2$f96$1@ger.gmane.org>
 <CALeMXf5oxi0oX13zSfHwgVi1eXF_z_coY23MfBP-YOK8WVkWsg@mail.gmail.com>
 <kvctoj$qsv$1@ger.gmane.org> <kvstdj$jo0$1@ger.gmane.org>
 <CADiSq7c5xtEcPBXGPWqQosfgGcb2n+2AnHiAWFoATeYfQTUcEQ@mail.gmail.com>
 <20130902101642.04b69588@pitrou.net>
Message-ID: <CADiSq7c+R92r17c8g3ygW0T4p=VG8LiSf5XSGuOJsFHrOtOJDg@mail.gmail.com>

On 2 September 2013 18:16, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Le Sun, 1 Sep 2013 02:19:48 +1000,
> Nick Coghlan <ncoghlan at gmail.com> a ?crit :
>> On 1 Sep 2013 00:10, "Stefan Behnel" <stefan_ml at behnel.de> wrote:
>> >
>> > *bump*
>> >
>> > Does this sound like a viable solution?
>>
>> This isn't likely to progress until we have Eric's PEP 451 to a point
>> where it's ready for python-dev discussion and pronouncement.
>>
>> However, the revised loader API is being designed to allow for the
>> loader returning arbitrary objects, so something along these lines
>> should work. There will likely be some adjustments to the API
>> signature to allow extension modules to optionally support reloading
>> if they so desire.
>
> I think the biggest challenge here is to propose an API that's simple
> and easy to use (i.e. that doesn't make extension module writing more
> complicated than it currently is).
>
> The basic concept of putting custom module objects in sys.modules is
> sound, IMHO.

The hook API I currently have in mind is a two step initialisation:

    PyImport_PrepareNAME (optional)
    PyImport_ExecNAME

If you don't define prepare, the import system takes care of creating
a module object for you, and passing it in to the exec hook. The
return result from that is just an integer indicating success or
failure (on failure, an exception should be set).

If you *do* define the prepare hook, then it's similar to the existing
init hook, but receives a PEP 451 module spec object with info about
the module being imported (see PEP 451 for the draft details) and is
permitted to return an arbitrary PyObject reference.

The main open questions I have are how to deal with clearly indicating
whether modules support in-place reloading, unloading, loading in
subinterpreters and/or loading a second copy in the same interpreter.
That's actually more a question for PEP 451 though, so I'll post some
more detailed thoughts on that over on import-sig, which may
eventually make their way into a PEP 451 draft.

> As for "extension module as a wrapper", though, it shounds like the
> kind of complication I would personally prefer to stay away from. Also,
> it would make extension modules less like Python modules, rather than
> more.

Yeah, that will be allowed (since we'll probably permit returning
arbitrary objects), but definitely not required.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From eliben at gmail.com  Mon Sep  2 15:18:31 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Mon, 2 Sep 2013 06:18:31 -0700
Subject: [Python-Dev] SEEK_* constants in io and os
In-Reply-To: <20130902102437.197c1708@pitrou.net>
References: <CAF-Rda_2j=HTDJ5nEdKWxX1mCmbHLgUNe_zyDYXzGHE6-LUBmg@mail.gmail.com>
 <20130902102437.197c1708@pitrou.net>
Message-ID: <CAF-Rda-_Z7QTQNf+jonLC1FqWj4pdLVk2h5r5ORTPaiUCwGauA@mail.gmail.com>

On Mon, Sep 2, 2013 at 1:24 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> Le Sun, 1 Sep 2013 18:02:30 -0700,
> Eli Bendersky <eliben at gmail.com> a ?crit :
> > Hello,
> >
> > I was looking at the possibility of replacing the SEEK_* constants by
> > IntEnums, and the first thing that catches attention is that these
> > constants are defined in both Lib/os.py and Lib/io.py; both places
> > also recently started supporting SEEK_HOLE and SEEK_DATA (though here
> > io refers to os.SEEK_HOLE and os.SEEK_DATA).
>
> What is the runtime cost of doing so? os is a fundamental module that is
> imported by almost every Python program.
>

Theoretically, it should be very low given that we just need to add an
import and define one class. os already does a number of things in its
toplevel (mostly a few imports which transitively do other things).
Compounded with import caching, since this is done just once per run,
doesn't seem like a problem.

Empirically, I tried measuring it but I can't discern a difference
with/without translating SEEK_* to enums. There's a fluctuation of ~1usec
which I can't distinguish from noise. Let me know if you have a good
methodology of benchmarking these things

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130902/144c8345/attachment.html>

From solipsis at pitrou.net  Mon Sep  2 15:45:22 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 2 Sep 2013 15:45:22 +0200
Subject: [Python-Dev] SEEK_* constants in io and os
References: <CAF-Rda_2j=HTDJ5nEdKWxX1mCmbHLgUNe_zyDYXzGHE6-LUBmg@mail.gmail.com>
 <20130902102437.197c1708@pitrou.net>
 <CAF-Rda-_Z7QTQNf+jonLC1FqWj4pdLVk2h5r5ORTPaiUCwGauA@mail.gmail.com>
Message-ID: <20130902154522.495b1697@pitrou.net>

Le Mon, 2 Sep 2013 06:18:31 -0700,
Eli Bendersky <eliben at gmail.com> a ?crit :
> On Mon, Sep 2, 2013 at 1:24 AM, Antoine Pitrou <solipsis at pitrou.net>
> wrote:
> 
> > Le Sun, 1 Sep 2013 18:02:30 -0700,
> > Eli Bendersky <eliben at gmail.com> a ?crit :
> > > Hello,
> > >
> > > I was looking at the possibility of replacing the SEEK_*
> > > constants by IntEnums, and the first thing that catches attention
> > > is that these constants are defined in both Lib/os.py and
> > > Lib/io.py; both places also recently started supporting SEEK_HOLE
> > > and SEEK_DATA (though here io refers to os.SEEK_HOLE and
> > > os.SEEK_DATA).
> >
> > What is the runtime cost of doing so? os is a fundamental module
> > that is imported by almost every Python program.
> >
> 
> Theoretically, it should be very low given that we just need to add an
> import and define one class. os already does a number of things in its
> toplevel (mostly a few imports which transitively do other things).
> Compounded with import caching, since this is done just once per run,
> doesn't seem like a problem.
> 
> Empirically, I tried measuring it but I can't discern a difference
> with/without translating SEEK_* to enums. There's a fluctuation of
> ~1usec which I can't distinguish from noise. Let me know if you have
> a good methodology of benchmarking these things

How did you get that result? You have to remove to "os" from
sys.modules before importing it again, otherwise "import os" will
simply return the already imported module.

Regards

Antoine.



From solipsis at pitrou.net  Mon Sep  2 15:51:09 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 2 Sep 2013 15:51:09 +0200
Subject: [Python-Dev] SEEK_* constants in io and os
References: <CAF-Rda_2j=HTDJ5nEdKWxX1mCmbHLgUNe_zyDYXzGHE6-LUBmg@mail.gmail.com>
 <20130902102437.197c1708@pitrou.net>
 <CAF-Rda-_Z7QTQNf+jonLC1FqWj4pdLVk2h5r5ORTPaiUCwGauA@mail.gmail.com>
 <20130902154522.495b1697@pitrou.net>
Message-ID: <20130902155109.7da869bd@pitrou.net>

Le Mon, 2 Sep 2013 15:45:22 +0200,
Antoine Pitrou <solipsis at pitrou.net> a ?crit :
> Le Mon, 2 Sep 2013 06:18:31 -0700,
> Eli Bendersky <eliben at gmail.com> a ?crit :
> > On Mon, Sep 2, 2013 at 1:24 AM, Antoine Pitrou <solipsis at pitrou.net>
> > wrote:
> > 
> > > Le Sun, 1 Sep 2013 18:02:30 -0700,
> > > Eli Bendersky <eliben at gmail.com> a ?crit :
> > > > Hello,
> > > >
> > > > I was looking at the possibility of replacing the SEEK_*
> > > > constants by IntEnums, and the first thing that catches
> > > > attention is that these constants are defined in both Lib/os.py
> > > > and Lib/io.py; both places also recently started supporting
> > > > SEEK_HOLE and SEEK_DATA (though here io refers to os.SEEK_HOLE
> > > > and os.SEEK_DATA).
> > >
> > > What is the runtime cost of doing so? os is a fundamental module
> > > that is imported by almost every Python program.
> > >
> > 
> > Theoretically, it should be very low given that we just need to add
> > an import and define one class. os already does a number of things
> > in its toplevel (mostly a few imports which transitively do other
> > things). Compounded with import caching, since this is done just
> > once per run, doesn't seem like a problem.
> > 
> > Empirically, I tried measuring it but I can't discern a difference
> > with/without translating SEEK_* to enums. There's a fluctuation of
> > ~1usec which I can't distinguish from noise. Let me know if you have
> > a good methodology of benchmarking these things
> 
> How did you get that result? You have to remove to "os" from
> sys.modules before importing it again, otherwise "import os" will
> simply return the already imported module.

Oh and remove "enum" too...

Regards

Antoine.



From eliben at gmail.com  Mon Sep  2 16:49:31 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Mon, 2 Sep 2013 07:49:31 -0700
Subject: [Python-Dev] SEEK_* constants in io and os
In-Reply-To: <20130902155109.7da869bd@pitrou.net>
References: <CAF-Rda_2j=HTDJ5nEdKWxX1mCmbHLgUNe_zyDYXzGHE6-LUBmg@mail.gmail.com>
 <20130902102437.197c1708@pitrou.net>
 <CAF-Rda-_Z7QTQNf+jonLC1FqWj4pdLVk2h5r5ORTPaiUCwGauA@mail.gmail.com>
 <20130902154522.495b1697@pitrou.net> <20130902155109.7da869bd@pitrou.net>
Message-ID: <CAF-Rda86kyZg+ZuJcw4jff8w6LgCYo__N=VDyR_GQqdEcQ6ePg@mail.gmail.com>

On Mon, Sep 2, 2013 at 6:51 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> Le Mon, 2 Sep 2013 15:45:22 +0200,
> Antoine Pitrou <solipsis at pitrou.net> a ?crit :
> > Le Mon, 2 Sep 2013 06:18:31 -0700,
> > Eli Bendersky <eliben at gmail.com> a ?crit :
> > > On Mon, Sep 2, 2013 at 1:24 AM, Antoine Pitrou <solipsis at pitrou.net>
> > > wrote:
> > >
> > > > Le Sun, 1 Sep 2013 18:02:30 -0700,
> > > > Eli Bendersky <eliben at gmail.com> a ?crit :
> > > > > Hello,
> > > > >
> > > > > I was looking at the possibility of replacing the SEEK_*
> > > > > constants by IntEnums, and the first thing that catches
> > > > > attention is that these constants are defined in both Lib/os.py
> > > > > and Lib/io.py; both places also recently started supporting
> > > > > SEEK_HOLE and SEEK_DATA (though here io refers to os.SEEK_HOLE
> > > > > and os.SEEK_DATA).
> > > >
> > > > What is the runtime cost of doing so? os is a fundamental module
> > > > that is imported by almost every Python program.
> > > >
> > >
> > > Theoretically, it should be very low given that we just need to add
> > > an import and define one class. os already does a number of things
> > > in its toplevel (mostly a few imports which transitively do other
> > > things). Compounded with import caching, since this is done just
> > > once per run, doesn't seem like a problem.
> > >
> > > Empirically, I tried measuring it but I can't discern a difference
> > > with/without translating SEEK_* to enums. There's a fluctuation of
> > > ~1usec which I can't distinguish from noise. Let me know if you have
> > > a good methodology of benchmarking these things
> >
> > How did you get that result? You have to remove to "os" from
> > sys.modules before importing it again, otherwise "import os" will
> > simply return the already imported module.
>
> Oh and remove "enum" too...
>

Yes, now I see a 500 usec difference timed within the Python script. When
timing the whole execution of Python:

$ sudo nice -n -20 perf stat -r 100 python -c 'import os'

It still gets lost in the noise, though. I usually get around 34 ms per run
with 1 - 1.5% jitter. Since 1.5% of 34 ms is ~0.5 ms, it's difficult to
distinguish the runs clearly.

That 0.5 ms seems to be a one-time penalty for the importing of enum (most
time goes to importing, not defining the new class based on IntEnum), no
matter where/how it's used in the stdlib and the user code.

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130902/77095dfd/attachment.html>

From victor.stinner at gmail.com  Mon Sep  2 17:48:01 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 2 Sep 2013 17:48:01 +0200
Subject: [Python-Dev] SEEK_* constants in io and os
In-Reply-To: <CAF-Rda86kyZg+ZuJcw4jff8w6LgCYo__N=VDyR_GQqdEcQ6ePg@mail.gmail.com>
References: <CAF-Rda_2j=HTDJ5nEdKWxX1mCmbHLgUNe_zyDYXzGHE6-LUBmg@mail.gmail.com>
 <20130902102437.197c1708@pitrou.net>
 <CAF-Rda-_Z7QTQNf+jonLC1FqWj4pdLVk2h5r5ORTPaiUCwGauA@mail.gmail.com>
 <20130902154522.495b1697@pitrou.net> <20130902155109.7da869bd@pitrou.net>
 <CAF-Rda86kyZg+ZuJcw4jff8w6LgCYo__N=VDyR_GQqdEcQ6ePg@mail.gmail.com>
Message-ID: <CAMpsgwYoR9p+F5Q9rXw-UbLKV2fFyEsZARR=VKXvjSLwGv8i6A@mail.gmail.com>

2013/9/2 Eli Bendersky <eliben at gmail.com>:
> Yes, now I see a 500 usec difference timed within the Python script. When
> timing the whole execution of Python: (...)

Can you please provide the list of imported modules by:
python -c 'import sys; print(sys.modules)'

For python with default options and for python with -S (no site
module) options? And also with your patch?

Python should be fast to write "hello world" (or "python -c pass),
it's a dummy but common benchmark (to compare Python to other VM /
other programming languages).

Victor

From eliben at gmail.com  Mon Sep  2 18:02:00 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Mon, 2 Sep 2013 09:02:00 -0700
Subject: [Python-Dev] SEEK_* constants in io and os
In-Reply-To: <CAMpsgwYoR9p+F5Q9rXw-UbLKV2fFyEsZARR=VKXvjSLwGv8i6A@mail.gmail.com>
References: <CAF-Rda_2j=HTDJ5nEdKWxX1mCmbHLgUNe_zyDYXzGHE6-LUBmg@mail.gmail.com>
 <20130902102437.197c1708@pitrou.net>
 <CAF-Rda-_Z7QTQNf+jonLC1FqWj4pdLVk2h5r5ORTPaiUCwGauA@mail.gmail.com>
 <20130902154522.495b1697@pitrou.net> <20130902155109.7da869bd@pitrou.net>
 <CAF-Rda86kyZg+ZuJcw4jff8w6LgCYo__N=VDyR_GQqdEcQ6ePg@mail.gmail.com>
 <CAMpsgwYoR9p+F5Q9rXw-UbLKV2fFyEsZARR=VKXvjSLwGv8i6A@mail.gmail.com>
Message-ID: <CAF-Rda_msMu--d-8Q8kPy8ReaJdNavKQ8keLrVQPOjMxpA41Yg@mail.gmail.com>

On Mon, Sep 2, 2013 at 8:48 AM, Victor Stinner <victor.stinner at gmail.com>wrote:

> 2013/9/2 Eli Bendersky <eliben at gmail.com>:
> > Yes, now I see a 500 usec difference timed within the Python script. When
> > timing the whole execution of Python: (...)
>
> Can you please provide the list of imported modules by:
> python -c 'import sys; print(sys.modules)'
>
> For python with default options and for python with -S (no site
> module) options? And also with your patch?
>
> Python should be fast to write "hello world" (or "python -c pass),
> it's a dummy but common benchmark (to compare Python to other VM /
> other programming languages).
>

The sorted list for both default and -S (they're identical) is
http://pastebin.com/4vzSMCu7 - there are 55 entries there, including things
like itertools, heapq, functools and collections. With my patch, enum is
also added (which makes sense since os is in the list). So the 0.5ms
increase for the 34ms runtime kind-of makes sense.

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130902/d7f87272/attachment.html>

From rdmurray at bitdance.com  Mon Sep  2 23:40:55 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Mon, 02 Sep 2013 17:40:55 -0400
Subject: [Python-Dev] Completing the email6 API changes.
In-Reply-To: <87y57fbt2q.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <20130831052136.42D4B2507F5@webabinitio.net>
 <87bo4echcr.fsf@uwakimon.sk.tsukuba.ac.jp>
 <20130901221040.DC76E250818@webabinitio.net> <5223C8E2.2000006@g.nevcal.com>
 <874na3diwj.fsf@uwakimon.sk.tsukuba.ac.jp> <5224286E.4040409@g.nevcal.com>
 <87y57fbt2q.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <20130902214056.ADC342502EA@webabinitio.net>

On Mon, 02 Sep 2013 16:06:53 +0900, "Stephen J. Turnbull" <stephen at xemacs.org> wrote:
> >>>>> Glenn writes:
> > >>>>> Steve writes:
> 
> >> OTOH, if the message is structured
> >>
> >>      multipart/related
> >>          multipart/alternative
> >>              text/plain
> >>              text/html
> >>          image/png
> >>          image/png
> >>
> >> the receiver can infer that the images are related to both text/*
> >> parts and DTRT for each.
> 
> >With the images being treated as attachments. Or is there a syntax to 
> >allow the text/html to embed the images and the text/plain to see them 
> >as attachments?
> 
> I believe the above is that syntax.  But the standard doesn't say
> anything about this.  The standard for multipart/alternative is RFC
> 2046, which doesn't know about multipart/related.  RFC 2387 doesn't
> update RFC 2046, so it doesn't say anything about
> multipart/alternative within multipart/related, either.
> 
> >I think the text/html wants to refer to things within its containing
> >multipart/related, but am not sure if that allows the intervening
> >multipart/alternative.
> 
> I don't see why not.  But it would depend on the implementations,
> which we'll have to test before recommending the structure I
> (theoretically :-) prefer.e

I'm still not understanding how the text/plain part *refers* to the
related parts.  I can understand the structure Glen found in Applemail:
a series of text/plain parts interspersed with image/jpg, with all parts
after the first being marked 'Contentent-Disposition: inline'.  Any MUA
that can display text and images *ought* to handle that correctly and
produce the expected result.  But that isn't what your structure above
would produce.  If you did:

    multipart/related
        multipart/alternative
            text/html
            text/plain
        image/png
        text/plain
        image/png
        text/plain

and only referred to the png parts in the text/html part and marked all
the parts as 'inline' (even though that is irrelevant in the text/html
related case), an MUA that *knew* about this technique *could* display it
"correctly", but an MUA that is just following the standards most
likely won't.

I don't see any way short of duplicating the image parts to make it
likely that a typical MUA would display images for both a text/plain
sequence and a text/html related part.  On the other hand, my experience
with MUAs is actually quite limited :)

Unless there is some standard for referring to related parts in a
text/plain part?  I'm not aware of any, but you have much more experience
with this stuff than I do.  (Even text/enriched (RFC 1896) doesn't seem
to have one, though of course there could be "extensions" that
define both that and the font support you used as an example.)

--David

From v+python at g.nevcal.com  Tue Sep  3 00:52:59 2013
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Mon, 02 Sep 2013 15:52:59 -0700
Subject: [Python-Dev] Completing the email6 API changes.
In-Reply-To: <20130902214056.ADC342502EA@webabinitio.net>
References: <20130831052136.42D4B2507F5@webabinitio.net>
 <87bo4echcr.fsf@uwakimon.sk.tsukuba.ac.jp>
 <20130901221040.DC76E250818@webabinitio.net> <5223C8E2.2000006@g.nevcal.com>
 <874na3diwj.fsf@uwakimon.sk.tsukuba.ac.jp> <5224286E.4040409@g.nevcal.com>
 <87y57fbt2q.fsf@uwakimon.sk.tsukuba.ac.jp>
 <20130902214056.ADC342502EA@webabinitio.net>
Message-ID: <522516CB.1030705@g.nevcal.com>

On 9/2/2013 2:40 PM, R. David Murray wrote:
> I'm still not understanding how the text/plain part*refers*  to the
> related parts.
I don't think the text/plain part can refer to the related parts, but, 
like you, am willing to be educated if there is a way; but while the 
text/html may be able to if things like cid: URIs can reach up a level 
in a given MUA, the text/plain would be left with the additional parts 
being attachments, methinks. This is less interesting than the technique 
Apple Mail uses, but more interesting than not even seeing the attached 
pictures.

MUAs tend to be able to display what they produce themselves, but I have 
situations where they don't handle what other MUAs produce.

One nice thing about this email6 toolkit might be the ability to 
produce, more easily than before, a variety of MIME combinations to 
exercise and test a variety of MUAs. While perhaps most of them have 
been tested with some obviously standard MIME combinations, I suspect 
most of them will produce strange results with combinations that are out 
of the ordinary.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130902/7d746b6b/attachment.html>

From rdmurray at bitdance.com  Tue Sep  3 03:27:15 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Mon, 02 Sep 2013 21:27:15 -0400
Subject: [Python-Dev] Completing the email6 API changes.
In-Reply-To: <522516CB.1030705@g.nevcal.com>
References: <20130831052136.42D4B2507F5@webabinitio.net>
 <87bo4echcr.fsf@uwakimon.sk.tsukuba.ac.jp>
 <20130901221040.DC76E250818@webabinitio.net> <5223C8E2.2000006@g.nevcal.com>
 <874na3diwj.fsf@uwakimon.sk.tsukuba.ac.jp> <5224286E.4040409@g.nevcal.com>
 <87y57fbt2q.fsf@uwakimon.sk.tsukuba.ac.jp>
 <20130902214056.ADC342502EA@webabinitio.net> <522516CB.1030705@g.nevcal.com>
Message-ID: <20130903012716.1E3FF250166@webabinitio.net>

On Mon, 02 Sep 2013 15:52:59 -0700, Glenn Linderman <v+python at g.nevcal.com> wrote:
> MUAs tend to be able to display what they produce themselves, but I have 
> situations where they don't handle what other MUAs produce.
> 
> One nice thing about this email6 toolkit might be the ability to 
> produce, more easily than before, a variety of MIME combinations to 
> exercise and test a variety of MUAs. While perhaps most of them have 
> been tested with some obviously standard MIME combinations, I suspect 
> most of them will produce strange results with combinations that are out 
> of the ordinary.

Yeah, RFC compliance and other types of testing is something I want this
package to be good for.  The API under discussion here, though, is
oriented toward people using the library for easily generating emails
from their application and/or easily accessing the information from
emails sent to their application.

--David

From stephen at xemacs.org  Tue Sep  3 03:56:36 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Tue, 03 Sep 2013 10:56:36 +0900
Subject: [Python-Dev] Completing the email6 API changes.
In-Reply-To: <20130902214056.ADC342502EA@webabinitio.net>
References: <20130831052136.42D4B2507F5@webabinitio.net>
 <87bo4echcr.fsf@uwakimon.sk.tsukuba.ac.jp>
 <20130901221040.DC76E250818@webabinitio.net>
 <5223C8E2.2000006@g.nevcal.com>
 <874na3diwj.fsf@uwakimon.sk.tsukuba.ac.jp>
 <5224286E.4040409@g.nevcal.com>
 <87y57fbt2q.fsf@uwakimon.sk.tsukuba.ac.jp>
 <20130902214056.ADC342502EA@webabinitio.net>
Message-ID: <87r4d6brcb.fsf@uwakimon.sk.tsukuba.ac.jp>

R. David Murray writes:

 > I'm still not understanding how the text/plain part *refers* to the
 > related parts.

Like this: "Check out this picture of my dog!"  Or this: "The terms of
the contract are found in the attached PDF.  Please print it and sign
it, then return it by carrier pigeon (attached)."  With this structure

    multipart/alternative
        text/plain
        multipart/related
            text/html
            application/pdf
            application/rfc6214-transport

the rendering of the text/plain part will not show evidence of the PDF
at all (eg, a view/download button), at least in some of the MUAs I've
tested.  And it *should* not, in an RFC-conforming MUA.

 > I can understand the structure Glen found in Applemail:
 > a series of text/plain parts interspersed with image/jpg, with all parts
 > after the first being marked 'Contentent-Disposition: inline'.  Any MUA
 > that can display text and images *ought* to handle that correctly and
 > produce the expected result.  But that isn't what your structure above
 > would produce.  If you did:
 > 
 >     multipart/related
 >         multipart/alternative
 >             text/html
 >             text/plain
 >         image/png
 >         text/plain
 >         image/png
 >         text/plain
 > 
 > and only referred to the png parts in the text/html part and marked all
 > the parts as 'inline' (even though that is irrelevant in the text/html
 > related case), an MUA that *knew* about this technique *could* display it
 > "correctly", but an MUA that is just following the standards most
 > likely won't.

OK, I see that now.  It requires non-MIME information about the
treatment of the root entity by the implementation.  On the other
hand, it shouldn't *hurt*.  RFC 2387 explicitly specifies that at
least some parts of a contained multipart/related part should be able
to refer to entities related via the containing multipart/related.
Since it does not mention *any* restrictions on contained root
entities, I take it that it implicitly specifies that any contained
multipart may make such references.  But I suspect it's not
implemented by most MUAs.  I'll have to test.

 > I don't see any way short of duplicating the image parts to make it
 > likely that a typical MUA would display images for both a text/plain
 > sequence and a text/html related part.  On the other hand, my experience
 > with MUAs is actually quite limited :)
 > 
 > Unless there is some standard for referring to related parts in a
 > text/plain part?

No, the whole point is that we MUA implementers *know* that there is
no machine-parsable way to refer to the related parts in text/plain,
and therefore the only way to communicate even the *presence* of the
attachment in

    multipart/related
        text/plain
        image/jpeg; name="dog-photo.jpg"

to the receiving user is to make an exception in the implementation
and treat it as multipart/mixed.[1]

It *does* make sense, i.e., doesn't require any information not
already available to the implementation.

I wonder if use of external bodies could avoid the duplication in
current implementations.  Probably too fragile, though.

Footnotes: 
[1]  This is conformant to the RFC, as the mechanism of "relation" is
explicitly application-dependent.


From rdmurray at bitdance.com  Tue Sep  3 16:01:42 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Tue, 03 Sep 2013 10:01:42 -0400
Subject: [Python-Dev] Completing the email6 API changes.
In-Reply-To: <87r4d6brcb.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <20130831052136.42D4B2507F5@webabinitio.net>
 <87bo4echcr.fsf@uwakimon.sk.tsukuba.ac.jp>
 <20130901221040.DC76E250818@webabinitio.net> <5223C8E2.2000006@g.nevcal.com>
 <874na3diwj.fsf@uwakimon.sk.tsukuba.ac.jp> <5224286E.4040409@g.nevcal.com>
 <87y57fbt2q.fsf@uwakimon.sk.tsukuba.ac.jp>
 <20130902214056.ADC342502EA@webabinitio.net>
 <87r4d6brcb.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <20130903140143.4EC862502EA@webabinitio.net>

On Tue, 03 Sep 2013 10:56:36 +0900, "Stephen J. Turnbull" <stephen at xemacs.org> wrote:
> R. David Murray writes:
>  > I can understand the structure Glen found in Applemail:
>  > a series of text/plain parts interspersed with image/jpg, with all parts
>  > after the first being marked 'Contentent-Disposition: inline'.  Any MUA
>  > that can display text and images *ought* to handle that correctly and
>  > produce the expected result.  But that isn't what your structure above
>  > would produce.  If you did:
>  > 
>  >     multipart/related
>  >         multipart/alternative
>  >             text/html
>  >             text/plain
>  >         image/png
>  >         text/plain
>  >         image/png
>  >         text/plain
>  > 
>  > and only referred to the png parts in the text/html part and marked all
>  > the parts as 'inline' (even though that is irrelevant in the text/html
>  > related case), an MUA that *knew* about this technique *could* display it
>  > "correctly", but an MUA that is just following the standards most
>  > likely won't.
> 
> OK, I see that now.  It requires non-MIME information about the
> treatment of the root entity by the implementation.  On the other
> hand, it shouldn't *hurt*.  RFC 2387 explicitly specifies that at
> least some parts of a contained multipart/related part should be able
> to refer to entities related via the containing multipart/related.
> Since it does not mention *any* restrictions on contained root
> entities, I take it that it implicitly specifies that any contained
> multipart may make such references.  But I suspect it's not
> implemented by most MUAs.  I'll have to test.

OK, I see what you are driving at now.  Whether or not it works is
dependent on whether or not typical MUAs handle a multipart/related with
a text/plain root part by treating it as if it were a multipart/mixed
with inline or attachment sub-parts.  So yes, whether or not we should
support and/or document this technique very much depends on whether or
not typical MUAs do so.  I will, needless to say, be very interested in
the results of your research :)

--David

From rdmurray at bitdance.com  Tue Sep  3 17:28:32 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Tue, 03 Sep 2013 11:28:32 -0400
Subject: [Python-Dev] Completing the email6 API changes.
In-Reply-To: <20130903140143.4EC862502EA@webabinitio.net>
References: <20130831052136.42D4B2507F5@webabinitio.net>
 <87bo4echcr.fsf@uwakimon.sk.tsukuba.ac.jp>
 <20130901221040.DC76E250818@webabinitio.net> <5223C8E2.2000006@g.nevcal.com>
 <874na3diwj.fsf@uwakimon.sk.tsukuba.ac.jp> <5224286E.4040409@g.nevcal.com>
 <87y57fbt2q.fsf@uwakimon.sk.tsukuba.ac.jp>
 <20130902214056.ADC342502EA@webabinitio.net>
 <87r4d6brcb.fsf@uwakimon.sk.tsukuba.ac.jp>
 <20130903140143.4EC862502EA@webabinitio.net>
Message-ID: <20130903152832.EB219250166@webabinitio.net>

On Tue, 03 Sep 2013 10:01:42 -0400, "R. David Murray" <rdmurray at bitdance.com> wrote:
> On Tue, 03 Sep 2013 10:56:36 +0900, "Stephen J. Turnbull" <stephen at xemacs.org> wrote:
> > R. David Murray writes:
> >  > I can understand the structure Glen found in Applemail:
> >  > a series of text/plain parts interspersed with image/jpg, with all parts
> >  > after the first being marked 'Contentent-Disposition: inline'.  Any MUA
> >  > that can display text and images *ought* to handle that correctly and
> >  > produce the expected result.  But that isn't what your structure above
> >  > would produce.  If you did:
> >  > 
> >  >     multipart/related
> >  >         multipart/alternative
> >  >             text/html
> >  >             text/plain
> >  >         image/png
> >  >         text/plain
> >  >         image/png
> >  >         text/plain
> >  > 
> >  > and only referred to the png parts in the text/html part and marked all
> >  > the parts as 'inline' (even though that is irrelevant in the text/html
> >  > related case), an MUA that *knew* about this technique *could* display it
> >  > "correctly", but an MUA that is just following the standards most
> >  > likely won't.
> > 
> > OK, I see that now.  It requires non-MIME information about the
> > treatment of the root entity by the implementation.  On the other
> > hand, it shouldn't *hurt*.  RFC 2387 explicitly specifies that at
> > least some parts of a contained multipart/related part should be able
> > to refer to entities related via the containing multipart/related.
> > Since it does not mention *any* restrictions on contained root
> > entities, I take it that it implicitly specifies that any contained
> > multipart may make such references.  But I suspect it's not
> > implemented by most MUAs.  I'll have to test.
> 
> OK, I see what you are driving at now.  Whether or not it works is
> dependent on whether or not typical MUAs handle a multipart/related with
> a text/plain root part by treating it as if it were a multipart/mixed

I meant "a text/plain root part *inside* a multipart/alternative", which
is what you said, I just didn't understand it at first :)  Although I
wonder how many GUI MUAs do the fallback to multipart/mixed with just a
normal text/plain root part, too.  I would expect a text-only MUA would,
since it has no other way to display a multipart/related...but a
graphical MUA might just assume that there will always be an html part
in a multipart/related.

> with inline or attachment sub-parts.  So yes, whether or not we should
> support and/or document this technique very much depends on whether or
> not typical MUAs do so.  I will, needless to say, be very interested in
> the results of your research :)
> 
> --David
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/rdmurray%40bitdance.com

From barry at python.org  Tue Sep  3 20:51:05 2013
From: barry at python.org (Barry Warsaw)
Date: Tue, 3 Sep 2013 14:51:05 -0400
Subject: [Python-Dev] Python 2.6 to end support with 2.6.9 in October 2013
Message-ID: <20130903145105.76944e79@anarchist>

Hello Pythonistas,

Python 2.6.9 is the last planned release of the 2.6.x series.  This will be a
security-only source-only release.  It is currently scheduled for October
2013, and after this Python 2.6 will have reached its end-of-life and the
branch will be retired.

http://www.python.org/dev/peps/pep-0361/

I would like to release 2.6.9rc1 on Monday, September 30th, and 2.6.9 final on
Monday, October 28th.  I've added both dates to the Python calendar.

Here are the list of candidates still to be fixed for 2.6.9:

- 18747 - Re-seed OpenSSL's PRNG after fork
- 16037 - httplib: header parsing is not delimited
- 16038 - ftplib: unlimited readline() from connection
- 16039 - imaplib: unlimited readline() from connection
- 16040 - nntplib: unlimited readline() from connection
- 16041 - poplib: unlimited readline() from connection
- 16042 - smtplib: unlimited readline() from connection
- 16043 - xmlrpc: gzip_decode has unlimited read()

These were the ones I previously had on my list, and I've now marked these all
as release blockers for 2.6.9... for now.

If you know of any others that I should be aware of, please let me know.

If you can contribute to 2.6.9 by reviewing, testing, developing, or
commenting, that would be greatly appreciated.  I will be spending some time
triaging these and any other issues that get identified as possible 2.6.9
candidates.

If you have any questions regarding 2.6.9, please contact me via mailing list
or IRC.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130903/d62efb81/attachment.sig>

From rymg19 at gmail.com  Tue Sep  3 21:53:53 2013
From: rymg19 at gmail.com (Ryan)
Date: Tue, 03 Sep 2013 14:53:53 -0500
Subject: [Python-Dev] Python 2.6 to end support with 2.6.9 in October
	2013
In-Reply-To: <20130903145105.76944e79@anarchist>
References: <20130903145105.76944e79@anarchist>
Message-ID: <ec8efc66-d6e7-4a4c-a77a-f6590f7fecea@email.android.com>

I'm still waiting on Python 2.7 for Android! Stuck on 2.6 for now...ugh!

Wonder if I can build it myself...

Barry Warsaw <barry at python.org> wrote:

>Hello Pythonistas,
>
>Python 2.6.9 is the last planned release of the 2.6.x series.  This
>will be a
>security-only source-only release.  It is currently scheduled for
>October
>2013, and after this Python 2.6 will have reached its end-of-life and
>the
>branch will be retired.
>
>http://www.python.org/dev/peps/pep-0361/
>
>I would like to release 2.6.9rc1 on Monday, September 30th, and 2.6.9
>final on
>Monday, October 28th.  I've added both dates to the Python calendar.
>
>Here are the list of candidates still to be fixed for 2.6.9:
>
>- 18747 - Re-seed OpenSSL's PRNG after fork
>- 16037 - httplib: header parsing is not delimited
>- 16038 - ftplib: unlimited readline() from connection
>- 16039 - imaplib: unlimited readline() from connection
>- 16040 - nntplib: unlimited readline() from connection
>- 16041 - poplib: unlimited readline() from connection
>- 16042 - smtplib: unlimited readline() from connection
>- 16043 - xmlrpc: gzip_decode has unlimited read()
>
>These were the ones I previously had on my list, and I've now marked
>these all
>as release blockers for 2.6.9... for now.
>
>If you know of any others that I should be aware of, please let me
>know.
>
>If you can contribute to 2.6.9 by reviewing, testing, developing, or
>commenting, that would be greatly appreciated.  I will be spending some
>time
>triaging these and any other issues that get identified as possible
>2.6.9
>candidates.
>
>If you have any questions regarding 2.6.9, please contact me via
>mailing list
>or IRC.
>
>Cheers,
>-Barry
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Python-Dev mailing list
>Python-Dev at python.org
>https://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe:
>https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com

-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130903/97f55af0/attachment.html>

From victor.stinner at gmail.com  Wed Sep  4 01:27:15 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 4 Sep 2013 01:27:15 +0200
Subject: [Python-Dev] RFC: PEP 454: Add a new tracemalloc module
Message-ID: <CAMpsgwZTOBdXy2Rcxs0itnPZYZgHU4X4tPNf5syoZrOJ5gYCFw@mail.gmail.com>

Hi,

Antoine Pitrou suggested me to write a PEP to discuss the API of the
new tracemalloc module that I proposed to add to Python 3.4. Here you
have.

If you prefer to read the HTML version:
http://www.python.org/dev/peps/pep-0454/

See also the documentation of the current implementation of the module.
http://hg.python.org/features/tracemalloc/file/tip/Doc/library/tracemalloc.rst

The documentaion contains examples and a short "tutorial".


PEP: 454
Title: Add a new tracemalloc module to trace Python memory allocations
Version: $Revision$
Last-Modified: $Date$
Author: Victor Stinner <victor.stinner at gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 3-September-2013
Python-Version: 3.4


Abstract
========

Add a new ``tracemalloc`` module to trace Python memory allocations.



Rationale
=========

Common debug tools tracing memory allocations read the C filename and
number.  Using such tool to analyze Python memory allocations does not
help because most memory allocations are done in the same C function,
``PyMem_Malloc()`` for example.

There are debug tools dedicated to the Python languages like ``Heapy``
and ``PySizer``. These projects analyze objects type and/or content.
These tools are useful when the most memory leak are instances of the
same type and this type in allocated only in a few functions. The
problem is when the object type is very common like ``str`` or
``tuple``, and it is hard to identify where these objects are allocated.

Finding reference cycles is also a difficult task. There are different
tools to draw a diagram of all references. These tools cannot be used
huge on large applications with thousands of objects because the diagram
is too huge to be analyzed manually.


Proposal
========

Using the PEP 445, it becomes easy to setup an hook on Python memory
allocators. The hook can inspect the current Python frame to get the
Python filename and line number.

This PEP proposes to add a new ``tracemalloc`` module. It is a debug
tool to trace memory allocations made by Python. The module provides the
following information:

* Statistics on Python memory allocations per Python filename and line
  number: size, number, and average size of allocations
* Compute differences between two snapshots of Python memory allocations
* Location of a Python memory allocation: size in bytes, Python filename
  and line number


Command line options
====================

The ``python -m tracemalloc`` command can be used to analyze and compare
snapshots. The command takes a list of snapshot filenames and has the
following options.

``-g``, ``--group-per-file``

    Group allocations per filename, instead of grouping per line number.

``-n NTRACES``, ``--number NTRACES``

    Number of traces displayed per top (default: 10).

``--first``

    Compare with the first snapshot, instead of comparing with the
    previous snapshot.

``--include PATTERN``

    Only include filenames matching pattern *PATTERN*. The option can be
    specified multiple times.

    See ``fnmatch.fnmatch()`` for the syntax of patterns.

``--exclude PATTERN``

    Exclude filenames matching pattern *PATTERN*. The option can be
    specified multiple times.

    See ``fnmatch.fnmatch()`` for the syntax of patterns.

``-S``, ``--hide-size``

    Hide the size of allocations.

``-C``, ``--hide-count``

    Hide the number of allocations.

``-A``, ``--hide-average``

    Hide the average size of allocations.

``-P PARTS``, ``--filename-parts=PARTS``

    Number of displayed filename parts (default: 3).

``--color``

    Force usage of colors even if ``sys.stdout`` is not a TTY device.

``--no-color``

    Disable colors if ``sys.stdout`` is a TTY device.


API
===

To trace the most Python memory allocations, the module should be
enabled as early as possible in your application by calling
``tracemalloc.enable()`` function, by setting the ``PYTHONTRACEMALLOC``
environment variable to ``1``, or  by using ``-X tracemalloc`` command
line option.


Functions
---------

``enable()`` function:

    Start tracing Python memory allocations.

``disable()`` function:

    Stop tracing Python memory allocations and stop the timer started by
    ``start_timer()``.

``is_enabled()`` function:

    Get the status of the module: ``True`` if it is enabled, ``False``
    otherwise.

``get_object_address(obj)`` function:

    Get the address of the memory block of the specified Python object.

``get_object_trace(obj)`` function:

    Get the trace of a Python object *obj* as a ``trace`` instance.

    Return ``None`` if the tracemalloc module did not save the location
    when the object was allocated, for example if the module was
    disabled.

``get_process_memory()`` function:

    Get the memory usage of the current process as a meminfo namedtuple
    with two attributes:

    * ``rss``: Resident Set Size in bytes
    * ``vms``: size of the virtual memory in bytes

    Return ``None`` if the platform is not supported.

    Use the ``psutil`` module if available.

``get_stats()`` function:

    Get statistics on Python memory allocations per Python filename and
    per Python line number.

    Return a dictionary
    ``{filename: str -> {line_number: int -> stats: line_stat}}``
    where *stats* in a ``line_stat`` instance. *filename* and
    *line_number* can be ``None``.

    Return an empty dictionary if the tracemalloc module is disabled.


``get_traces(obj)`` function:

   Get all traces of a Python memory allocations.
   Return a dictionary ``{pointer: int -> trace}`` where *trace*
   is a ``trace`` instance.

   Return an empty dictionary if the ``tracemalloc`` module is disabled.


``start_timer(delay: int, func: callable, args: tuple=(), kwargs:
dict={})`` function:

    Start a timer calling ``func(*args, **kwargs)`` every *delay*
    seconds.

    The timer is based on the Python memory allocator, it is not real
    time.  *func* is called after at least *delay* seconds, it is not
    called exactly after *delay* seconds if no Python memory allocation
    occurred.

    If ``start_timer()`` is called twice, previous parameters are
    replaced.  The timer has a resolution of 1 second.

    ``start_timer()`` is used by ``DisplayTop`` and ``TakeSnapshot`` to
    run regulary a task.

``stop_timer()`` function:

    Stop the timer started by ``start_timer()``.


trace class
-----------

``trace`` class:

    This class represents debug information of an allocated memory block.

``size`` attribute:

    Size in bytes of the memory block.

``filename`` attribute:

    Name of the Python script where the memory block was allocated,
    ``None`` if unknown.

``lineno`` attribute:

    Line number where the memory block was allocated, ``None`` if
    unknown.


line_stat class
----------------

``line_stat`` class:

    Statistics on Python memory allocations of a specific line number.

``size`` attribute:

    Total size in bytes of all memory blocks allocated on the line.

``count`` attribute:

    Number of memory blocks allocated on the line.


DisplayTop class
----------------

``DisplayTop(count: int=10, file=sys.stdout)`` class:

    Display the list of the *count* biggest memory allocations into
    *file*.

``display()`` method:

    Display the top once.

``start(delay: int)`` method:

    Start a task using ``tracemalloc`` timer to display the top every
    *delay* seconds.

``stop()`` method:

    Stop the task started by the ``DisplayTop.start()`` method

``color`` attribute:

    If ``True``, ``display()`` uses color. The default value is
    ``file.isatty()``.

``compare_with_previous`` attribute:

    If ``True`` (default value), ``display()`` compares with the
    previous snapshot. If ``False``, compare with the first snapshot.

``filename_parts`` attribute:

    Number of displayed filename parts (int, default: ``3``). Extra
    parts are replaced with ``"..."``.

``group_per_file`` attribute:

    If ``True``, group memory allocations per Python filename. If
    ``False`` (default value), group allocation per Python line number.

``show_average`` attribute:

      If ``True`` (default value), ``display()`` shows the average size
      of allocations.

``show_count`` attribute:

    If ``True`` (default value), ``display()`` shows the number of
    allocations.

``show_size`` attribute:

    If ``True`` (default value), ``display()`` shows the size of
    allocations.

``user_data_callback`` attribute:

    Optional callback collecting user data (callable, default:
    ``None``).  See ``Snapshot.create()``.


Snapshot class
--------------

``Snapshot()`` class:

    Snapshot of Python memory allocations.

    Use ``TakeSnapshot`` to take regulary snapshots.

``create(user_data_callback=None)`` method:

    Take a snapshot. If *user_data_callback* is specified, it must be a
    callable object returning a list of
    ``(title: str, format: str, value: int)``.
    *format* must be ``'size'``. The list must always have the same
    length and the same order to be able to compute differences between
    values.

    Example: ``[('Video memory', 'size', 234902)]``.

``filter_filenames(patterns: list, include: bool)`` method:

    Remove filenames not matching any pattern of *patterns* if *include*
    is ``True``, or remove filenames matching a pattern of *patterns* if
    *include* is ``False`` (exclude).

    See ``fnmatch.fnmatch()`` for the syntax of a pattern.

``load(filename)`` classmethod:

    Load a snapshot from a file.

``write(filename)`` method:

    Write the snapshot into a file.

``pid`` attribute:

    Identifier of the process which created the snapshot (int).

``process_memory`` attribute:

    Result of the ``get_process_memory()`` function, can be ``None``.

``stats`` attribute:

    Result of the ``get_stats()`` function (dict).

``timestamp`` attribute:

    Creation date and time of the snapshot, ``datetime.datetime``
    instance.

``user_data`` attribute:

    Optional list of user data, result of *user_data_callback* in
    ``Snapshot.create()`` (default: None).


TakeSnapshot class
------------------

``TakeSnapshot`` class:

    Task taking snapshots of Python memory allocations: write them into
    files. By default, snapshots are written in the current directory.

``start(delay: int)`` method:

    Start a task taking a snapshot every delay seconds.

``stop()`` method:

    Stop the task started by the ``TakeSnapshot.start()`` method.

``take_snapshot()`` method:

    Take a snapshot.

``filename_template`` attribute:

    Template (``str``) used to create a filename. The following
    variables can be used in the template:

    * ``$pid``: identifier of the current process
    * ``$timestamp``: current date and time
    * ``$counter``: counter starting at 1 and incremented at each snapshot

    The default pattern is ``'tracemalloc-$counter.pickle'``.

``user_data_callback`` attribute:

    Optional callback collecting user data (callable, default:
    ``None``).  See ``Snapshot.create()``.


Links
=====

Python issues:

* `#18874: Add a new tracemalloc module to trace Python
  memory allocations <http://bugs.python.org/issue18874>`_

Similar projects:

* `Meliae: Python Memory Usage Analyzer
  <https://pypi.python.org/pypi/meliae>`_
* `Guppy-PE: umbrella package combining Heapy and GSL
  <http://guppy-pe.sourceforge.net/>`_
* `PySizer <http://pysizer.8325.org/>`_: developed for Python 2.4
* `memory_profiler <https://pypi.python.org/pypi/memory_profiler>`_
* `pympler <http://code.google.com/p/pympler/>`_
* `Dozer <https://pypi.python.org/pypi/Dozer>`_: WSGI Middleware version of
  the CherryPy memory leak debugger
* `objgraph <http://mg.pov.lt/objgraph/>`_
* `caulk <https://github.com/smartfile/caulk/>`_

Copyright
=========

This document has been placed into the public domain.

From victor.stinner at gmail.com  Wed Sep  4 01:56:21 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 4 Sep 2013 01:56:21 +0200
Subject: [Python-Dev] RFC: PEP 454: Add a new tracemalloc module
In-Reply-To: <CAMpsgwZTOBdXy2Rcxs0itnPZYZgHU4X4tPNf5syoZrOJ5gYCFw@mail.gmail.com>
References: <CAMpsgwZTOBdXy2Rcxs0itnPZYZgHU4X4tPNf5syoZrOJ5gYCFw@mail.gmail.com>
Message-ID: <CAMpsgwagcq-vNr3W7mQ7kPx=1sq=tio6cDyGNbRiEc6=EQXHNA@mail.gmail.com>

> ``get_object_trace(obj)`` function:
>
>     Get the trace of a Python object *obj* as a ``trace`` instance.
>
>     Return ``None`` if the tracemalloc module did not save the location
>     when the object was allocated, for example if the module was
>     disabled.

This function and get_traces() can be reused by other debug tools like
Heapy and objgraph to add where objects were allocated.

> ``get_stats()`` function:
>
>     Get statistics on Python memory allocations per Python filename and
>     per Python line number.
>
>     Return a dictionary
>     ``{filename: str -> {line_number: int -> stats: line_stat}}``
>     where *stats* in a ``line_stat`` instance. *filename* and
>     *line_number* can be ``None``.
>
>     Return an empty dictionary if the tracemalloc module is disabled.
>
> ``get_traces(obj)`` function:
>
>    Get all traces of a Python memory allocations.
>    Return a dictionary ``{pointer: int -> trace}`` where *trace*
>    is a ``trace`` instance.
>
>    Return an empty dictionary if the ``tracemalloc`` module is disabled.

get_stats() can computed from get_traces(), example:
-----
import pprint, tracemalloc

traces = tracemalloc.get_traces()
stats = {}
for trace in traces.values():
    if trace.filename not in stats:
        stats[trace.filename] = line_stats = {}
    else:
        line_stats = stats[trace.filename]
    if trace.lineno not in line_stats:
        line_stats[trace.lineno] = line_stat = tracemalloc.line_stat((0, 0))
        size = trace.size
        count = 1
    else:
        line_stat = line_stats[trace.lineno]
        size = line_stat.size + trace.size
        count = line_stat.count + 1
    line_stats[trace.lineno] = tracemalloc.line_stat((size, count))

pprint.pprint(stats)
-----

The problem is the efficiency. At startup, Python already allocated
more than 20,000 memory blocks:

$ ./python -X tracemalloc -c 'import tracemalloc;
print(len(tracemalloc.get_traces()))'
21704

At the end of the Python test suite, Python allocated more than
500,000 memory blocks.

Storing all these traces in a snapshot eats a lot of memory, disk
space and uses CPU to build the statistics.

> ``start_timer(delay: int, func: callable, args: tuple=(), kwargs:
> dict={})`` function:
>
>     Start a timer calling ``func(*args, **kwargs)`` every *delay*
>     seconds. (...)
>
>     If ``start_timer()`` is called twice, previous parameters are
>     replaced.  The timer has a resolution of 1 second.
>
>     ``start_timer()`` is used by ``DisplayTop`` and ``TakeSnapshot`` to
>     run regulary a task.

So DisplayTop and TakeSnapshot cannot be used at the same time. It
would be convinient to be able to register more than one function.
What do you think?

> ``trace`` class:
>     This class represents debug information of an allocated memory block.
>
> ``size`` attribute:
>     Size in bytes of the memory block.
> ``filename`` attribute:
>     Name of the Python script where the memory block was allocated,
>     ``None`` if unknown.
> ``lineno`` attribute:
>     Line number where the memory block was allocated, ``None`` if
>     unknown.

I though twice and it would be posible to store more than 1 frame per
trace instance, to be able to rebuild a (partial) Python traceback.
The hook on the memory allocator has access to the chain of Python
frames. The API should be changed to support such enhancement.

> ``DisplayTop(count: int=10, file=sys.stdout)`` class:
>     Display the list of the *count* biggest memory allocations into
>     *file*.
> (...)
> ``group_per_file`` attribute:
>
>     If ``True``, group memory allocations per Python filename. If
>     ``False`` (default value), group allocation per Python line number.

This attribute is very important. We may add it to the constructor.

By the way, the self.stream attribute is not documented.

> Snapshot class
> --------------
>
> ``Snapshot()`` class:
>
>     Snapshot of Python memory allocations.
>
>     Use ``TakeSnapshot`` to take regulary snapshots.
>
> ``create(user_data_callback=None)`` method:
>
>     Take a snapshot. If *user_data_callback* is specified, it must be a
>     callable object returning a list of
>     ``(title: str, format: str, value: int)``.
>     *format* must be ``'size'``. The list must always have the same
>     length and the same order to be able to compute differences between
>     values.
>
>     Example: ``[('Video memory', 'size', 234902)]``.

(Oops, create() is a class method, not a method.)

Having to call a class method to build an instance of a class is
surprising. But I didn't find a way to implement the load() class
method otherwise.

The user_data_callback API can be improved. The "format must be size"
is not very convinient.

Victor

From stephen at xemacs.org  Wed Sep  4 03:30:28 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 04 Sep 2013 10:30:28 +0900
Subject: [Python-Dev] Completing the email6 API changes.
In-Reply-To: <20130903152832.EB219250166@webabinitio.net>
References: <20130831052136.42D4B2507F5@webabinitio.net>
 <87bo4echcr.fsf@uwakimon.sk.tsukuba.ac.jp>
 <20130901221040.DC76E250818@webabinitio.net>
 <5223C8E2.2000006@g.nevcal.com>
 <874na3diwj.fsf@uwakimon.sk.tsukuba.ac.jp>
 <5224286E.4040409@g.nevcal.com>
 <87y57fbt2q.fsf@uwakimon.sk.tsukuba.ac.jp>
 <20130902214056.ADC342502EA@webabinitio.net>
 <87r4d6brcb.fsf@uwakimon.sk.tsukuba.ac.jp>
 <20130903140143.4EC862502EA@webabinitio.net>
 <20130903152832.EB219250166@webabinitio.net>
Message-ID: <87eh95bcgb.fsf@uwakimon.sk.tsukuba.ac.jp>

R. David Murray writes:

 > I meant "a text/plain root part *inside* a multipart/alternative", which
 > is what you said, I just didn't understand it at first :)  Although I
 > wonder how many GUI MUAs do the fallback to multipart/mixed with just a
 > normal text/plain root part, too.  I would expect a text-only MUA would,
 > since it has no other way to display a multipart/related...but a
 > graphical MUA might just assume that there will always be an html part
 > in a multipart/related.

It's not really a problem with text vs. GUI, or an assumption of HMTL.
There are plenty of formats that have such links, and some which don't
have links, but rather assigned roles such as "Mac files" (with data
fork and resource fork) and digital signatures (though that turned out
to be worth designing a new multipart subtype).

The problem is that "multipart/related" says "pass all the part
entities to the handler appropriate to the root part entity, which
will process the links found in the root part entity".  If you
implement that in the natural way, you just pass the text/plain part
to the text/plain handler, which won't find any links for the simple
reason that it has no protocol for representing them.

This means that the kind of multipart/related handler I envision needs
to implement linking itself, rather than delegate them to the root
part handler.  This requires checking the type of the root part:

    # not intended to look like Email API
    def handle_multipart_related (part_list, root_part):
        if root_part.content_type in ['text/plain']:
            # just display the parts in order
            handle_multipart_mixed (part_list)
        else:
            # cid -> entities in internal representation
            entity_map = extract_entity_map(part_list)
            root_part.content_type.handle(root_part, entity_map)

From victor.stinner at gmail.com  Wed Sep  4 13:20:11 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 4 Sep 2013 13:20:11 +0200
Subject: [Python-Dev] RFC: PEP 454: Add a new tracemalloc module
In-Reply-To: <CAMpsgwagcq-vNr3W7mQ7kPx=1sq=tio6cDyGNbRiEc6=EQXHNA@mail.gmail.com>
References: <CAMpsgwZTOBdXy2Rcxs0itnPZYZgHU4X4tPNf5syoZrOJ5gYCFw@mail.gmail.com>
 <CAMpsgwagcq-vNr3W7mQ7kPx=1sq=tio6cDyGNbRiEc6=EQXHNA@mail.gmail.com>
Message-ID: <CAMpsgwbtfpYVXSTHiygYAxY+ktf9Tj9vW-J-Nv8HV-75q7oaUw@mail.gmail.com>

>> ``trace`` class:
>>     This class represents debug information of an allocated memory block.
>>
>> ``size`` attribute:
>>     Size in bytes of the memory block.
>> ``filename`` attribute:
>>     Name of the Python script where the memory block was allocated,
>>     ``None`` if unknown.
>> ``lineno`` attribute:
>>     Line number where the memory block was allocated, ``None`` if
>>     unknown.
>
> I though twice and it would be posible to store more than 1 frame per
> trace instance, to be able to rebuild a (partial) Python traceback.
> The hook on the memory allocator has access to the chain of Python
> frames. The API should be changed to support such enhancement.

Oh, it was much easier than expected to retrieve the traceback
(maximum 10 frames) instead of only the current frame.

I modified the trace class to replace filename and lineno with a new
frames attribute which is list of frames.

Script example:
---
import tracemalloc, linecache

def g():
    return object()

def f():
    return g()

tracemalloc.enable()
obj = f()
trace = tracemalloc.get_object_trace(obj)

print("Traceback (most recent first):")
for frame in trace.frames:
    print('  File "%s", line %s' % (frame.filename, frame.lineno))
    line = linecache.getline(frame.filename, frame.lineno)
    if line:
        print("    " + line.strip())
---

Output of the script:
---
Traceback (most recent first):
  File "x.py", line 4
    return object()
  File "x.py", line 7
    return g()
  File "x.py", line 10
    obj = f()
---


I updated the PEP 454 (add a new frame class, update trace class):

frame class
-----------

``frame`` class:

    Trace of a Python frame.

``filename`` attribute (``str``):

    Python filename, ``None`` if unknown.

``lineno`` attribute (``int``):

    Python line number, ``None`` if unknown.


trace class
-----------

``trace`` class:

    This class represents debug information of an allocated memory block.

``size`` attribute (``int``):

    Size in bytes of the memory block.

``frames`` attribute (``list``):

    Traceback where the memory block was allocated as a list of
    ``frame`` instances (most recent first).

    The list can be empty or incomplete if the tracemalloc module was
    unable to retrieve the full traceback.

    For efficiency, the traceback is truncated to 10 frames.

Victor

From alexander.belopolsky at gmail.com  Thu Sep  5 02:46:21 2013
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Wed, 4 Sep 2013 20:46:21 -0400
Subject: [Python-Dev] RFC: PEP 454: Add a new tracemalloc module
In-Reply-To: <CAMpsgwZTOBdXy2Rcxs0itnPZYZgHU4X4tPNf5syoZrOJ5gYCFw@mail.gmail.com>
References: <CAMpsgwZTOBdXy2Rcxs0itnPZYZgHU4X4tPNf5syoZrOJ5gYCFw@mail.gmail.com>
Message-ID: <CAP7h-xauzgw8UJqKJAb2WPMqPM9-_BvDhHgcj5RnKvj5yaMiYw@mail.gmail.com>

On Tue, Sep 3, 2013 at 7:27 PM, Victor Stinner <victor.stinner at gmail.com>wrote:

> API
> ===
>
> To trace the most Python memory allocations, the module should be
> enabled as early as possible in your application by calling
> ``tracemalloc.enable()`` function, by setting the ``PYTHONTRACEMALLOC``
> environment variable to ``1``, or  by using ``-X tracemalloc`` command
> line option.
>
>
> Functions
> ---------
>
> ``enable()`` function:
>
>     Start tracing Python memory allocations.
>
> ``disable()`` function:
>
>     Stop tracing Python memory allocations and stop the timer started by
>     ``start_timer()``.
>
> ``is_enabled()`` function:
>
>     Get the status of the module: ``True`` if it is enabled, ``False``
>     otherwise.
>

Please mention that this API is similar to that of faulthandler and add a
link to faulthandler docs.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130904/311e3a96/attachment.html>

From jcea at jcea.es  Thu Sep  5 19:31:59 2013
From: jcea at jcea.es (Jesus Cea)
Date: Thu, 05 Sep 2013 19:31:59 +0200
Subject: [Python-Dev] Offtopic: OpenID Providers
Message-ID: <5228C00F.7000209@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I just received an email from my OpenID provider, "myOpenID", saying
that they drop OpenID service next February. I wonder what other
OpenID providers are used by other python-dev fellows.

What are you using?. bugs.python.org admins could share some data?

I agree than OpenID is (quite) dead, but I rather prefer OpenID to use
user/pass. I have big hopes for Mozilla Persona, looking forward
Python infrastructure support :).

PS: I use "http://www.jcea.es/" as my OpenID identity, and I delegate
the actual service to "myOpenID". I can switch delegation trivially.

- -- 
Jes?s Cea Avi?n                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQCVAwUBUijAD5lgi5GaxT1NAQLH0wQAkKDORrAtfJFzzIHl3lHRp7GfxOzdqdNP
uiuW65l/pas+p9+B0G6qR6EE2AAL7YPozcNF5AkmuGmxkpyn/JyUYKJcUWmUotpj
V9Buz9jz3qpPuv7AlTnMbjBBQK4YTYenbdk2HgI41SVQHZHkU/+y4CL3Y1hWyJNo
C8CCWfR0VlA=
=YXIq
-----END PGP SIGNATURE-----

From phd at phdru.name  Thu Sep  5 20:12:31 2013
From: phd at phdru.name (Oleg Broytman)
Date: Thu, 5 Sep 2013 22:12:31 +0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <5228C00F.7000209@jcea.es>
References: <5228C00F.7000209@jcea.es>
Message-ID: <20130905181231.GA25306@iskra.aviel.ru>

On Thu, Sep 05, 2013 at 07:31:59PM +0200, Jesus Cea <jcea at jcea.es> wrote:
> I just received an email from my OpenID provider, "myOpenID", saying
> that they drop OpenID service next February. I wonder what other
> OpenID providers are used by other python-dev fellows.
> 
> What are you using?. bugs.python.org admins could share some data?
> 
> I agree than OpenID is (quite) dead, but I rather prefer OpenID to use
> user/pass. I have big hopes for Mozilla Persona, looking forward
> Python infrastructure support :).
> 
> PS: I use "http://www.jcea.es/" as my OpenID identity, and I delegate
> the actual service to "myOpenID". I can switch delegation trivially.

   I used to use myOpenID and became my own provider using poit[1].
These days I seldom use OpenID -- there are too few sites that allow
full-featured login with OpenID. The future lies in OAuth 2.0.

1. http://yangman.ca/poit/

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From donald at stufft.io  Thu Sep  5 20:16:29 2013
From: donald at stufft.io (Donald Stufft)
Date: Thu, 5 Sep 2013 14:16:29 -0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <20130905181231.GA25306@iskra.aviel.ru>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
Message-ID: <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>


On Sep 5, 2013, at 2:12 PM, Oleg Broytman <phd at phdru.name> wrote:

> On Thu, Sep 05, 2013 at 07:31:59PM +0200, Jesus Cea <jcea at jcea.es> wrote:
>> I just received an email from my OpenID provider, "myOpenID", saying
>> that they drop OpenID service next February. I wonder what other
>> OpenID providers are used by other python-dev fellows.
>> 
>> What are you using?. bugs.python.org admins could share some data?
>> 
>> I agree than OpenID is (quite) dead, but I rather prefer OpenID to use
>> user/pass. I have big hopes for Mozilla Persona, looking forward
>> Python infrastructure support :).
>> 
>> PS: I use "http://www.jcea.es/" as my OpenID identity, and I delegate
>> the actual service to "myOpenID". I can switch delegation trivially.
> 
>   I used to use myOpenID and became my own provider using poit[1].
> These days I seldom use OpenID -- there are too few sites that allow
> full-featured login with OpenID. The future lies in OAuth 2.0.

The Auth in OAuth stands for Authorization not Authentication.

> 
> 1. http://yangman.ca/poit/
> 
> Oleg.
> -- 
>     Oleg Broytman            http://phdru.name/            phd at phdru.name
>           Programmers don't die, they just GOSUB without RETURN.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130905/0f57ca25/attachment.sig>

From phd at phdru.name  Thu Sep  5 20:25:22 2013
From: phd at phdru.name (Oleg Broytman)
Date: Thu, 5 Sep 2013 22:25:22 +0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
Message-ID: <20130905182522.GA26090@iskra.aviel.ru>

On Thu, Sep 05, 2013 at 02:16:29PM -0400, Donald Stufft <donald at stufft.io> wrote:
> 
> On Sep 5, 2013, at 2:12 PM, Oleg Broytman <phd at phdru.name> wrote:
> >   I used to use myOpenID and became my own provider using poit[1].
> > These days I seldom use OpenID -- there are too few sites that allow
> > full-featured login with OpenID. The future lies in OAuth 2.0.
> 
> The Auth in OAuth stands for Authorization not Authentication.

   There is no authorization without authentication, so OAuth certainly
performs authentication: http://oauth.net/core/1.0a/#anchor9 ,
http://tools.ietf.org/html/rfc5849#section-3

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From barry at python.org  Thu Sep  5 20:30:43 2013
From: barry at python.org (Barry Warsaw)
Date: Thu, 5 Sep 2013 14:30:43 -0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <5228C00F.7000209@jcea.es>
References: <5228C00F.7000209@jcea.es>
Message-ID: <20130905143043.590a12b2@anarchist>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Sep 05, 2013, at 07:31 PM, Jesus Cea wrote:

>I just received an email from my OpenID provider, "myOpenID", saying
>that they drop OpenID service next February. I wonder what other
>OpenID providers are used by other python-dev fellows.
>
>What are you using?. bugs.python.org admins could share some data?

Launchpad.  It's not going anywhere.

>I agree than OpenID is (quite) dead, but I rather prefer OpenID to use
>user/pass. I have big hopes for Mozilla Persona, looking forward
>Python infrastructure support :).

We at the Mailman project like Persona a lot.  It'll be the primary way people
can log into Postorius (the new web ui).

- -Barry
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)

iQIcBAEBCAAGBQJSKM3TAAoJEBJutWOnSwa/jP8QAJ4BkV8285TexZq+9GuTwzFf
qx2OlUwtiv6lhDZq9uY0sXijyKl76N0/nTsSf5Fr8nDsn5KaETd14PFGfwbg6WVS
jMZcKz+3j7eq6zq/Qm47qFbcBb2moyxptkE96BMCqEw+DCnPo87q8DXOb2W67ROI
XDsRW2T/D8v5SvAqkmgOPlWsXWc1QzOpJCGkYcufzDeCjkFPXvck3mFI0TpizXnv
7L4IjrK9Qn6D1zNvKmovTkQOu3sk3C2FvPT2QG0b+DWkPtERE6o/xK2OTkhIzbBd
/YfLEkb60tJYtwKqfHei+n4pVreExT0DtCDFHh5bYbdcD23VuHfDZtaHCDzYEhK9
rjuY2odydxD/xH9F21enHqYuU76TBr8ceodYsP2FNlqO5XeOJwgAXhWdpBYwN+SU
G0B5k3wDb6VCDkqM1u7VGcA/yQLeGQ43s/klxxAxo08sA3url9HcIeuO5rnmnB5l
lGIehq/SJPVQXQevEJvL8Atm7uLnsaWEJg34+T04MyR+TLeUcjKsNrUZBS8bGOZu
agygKg3M/zi1k9LW/uxKR7IKLyBK+axYoEQS9yydqpgO1e8BQa6XlLQ89Sv9r38Y
y1qTFmGdRRQSItrlS1urMQ9Q8m7pJBSyfBywlsck07hpXlZLRwRm2S3DnMBrF6It
I7f28XXirrnBf/la/+6g
=xqI7
-----END PGP SIGNATURE-----

From donald at stufft.io  Thu Sep  5 20:35:16 2013
From: donald at stufft.io (Donald Stufft)
Date: Thu, 5 Sep 2013 14:35:16 -0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <20130905182522.GA26090@iskra.aviel.ru>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
Message-ID: <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>


On Sep 5, 2013, at 2:25 PM, Oleg Broytman <phd at phdru.name> wrote:

> On Thu, Sep 05, 2013 at 02:16:29PM -0400, Donald Stufft <donald at stufft.io> wrote:
>> 
>> On Sep 5, 2013, at 2:12 PM, Oleg Broytman <phd at phdru.name> wrote:
>>>  I used to use myOpenID and became my own provider using poit[1].
>>> These days I seldom use OpenID -- there are too few sites that allow
>>> full-featured login with OpenID. The future lies in OAuth 2.0.
>> 
>> The Auth in OAuth stands for Authorization not Authentication.
> 
>   There is no authorization without authentication, so OAuth certainly
> performs authentication: http://oauth.net/core/1.0a/#anchor9 ,
> http://tools.ietf.org/html/rfc5849#section-3

They are separate topics and authorization does not need to imply authentication,
it so happens that in many particular instances of OAuth you can estimate
authentication.

https://en.wikipedia.org/wiki/OAuth#OpenID_vs._pseudo-authentication_using_OAuth

Persona is the logical successor to OpenID.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130905/065d9b65/attachment.sig>

From a.badger at gmail.com  Thu Sep  5 20:33:47 2013
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Thu, 5 Sep 2013 11:33:47 -0700
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <20130905182522.GA26090@iskra.aviel.ru>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
Message-ID: <20130905183347.GO10747@unaka.lan>

On Thu, Sep 05, 2013 at 10:25:22PM +0400, Oleg Broytman wrote:
> On Thu, Sep 05, 2013 at 02:16:29PM -0400, Donald Stufft <donald at stufft.io> wrote:
> > 
> > On Sep 5, 2013, at 2:12 PM, Oleg Broytman <phd at phdru.name> wrote:
> > >   I used to use myOpenID and became my own provider using poit[1].
> > > These days I seldom use OpenID -- there are too few sites that allow
> > > full-featured login with OpenID. The future lies in OAuth 2.0.
> > 
> > The Auth in OAuth stands for Authorization not Authentication.
> 
>    There is no authorization without authentication, so OAuth certainly
> performs authentication: http://oauth.net/core/1.0a/#anchor9 ,
> http://tools.ietf.org/html/rfc5849#section-3
> 
Sortof.... The way OAuth looks to me, it's designed to prove that a given
client is authorized to perform an action.  It's not designed to prove that
the given client is a specific person.  In some cases, you really want to
know the latter and not merely the former.  So I think in these situations
Donald's separation of Authz and Authn makes sense.

-Toshio
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130905/d6c3e7f9/attachment.sig>

From phd at phdru.name  Thu Sep  5 20:43:37 2013
From: phd at phdru.name (Oleg Broytman)
Date: Thu, 5 Sep 2013 22:43:37 +0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
Message-ID: <20130905184337.GA26926@iskra.aviel.ru>

On Thu, Sep 05, 2013 at 02:35:16PM -0400, Donald Stufft <donald at stufft.io> wrote:
> Persona is the logical successor to OpenID.

   OpenID lived a short life and died a quiet death. I'm afraid Persona
wouldn't live even that much. Dead-born idea, in my so humble opinion.

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From eliben at gmail.com  Thu Sep  5 20:45:01 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Thu, 5 Sep 2013 11:45:01 -0700
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <5228C00F.7000209@jcea.es>
References: <5228C00F.7000209@jcea.es>
Message-ID: <CAF-Rda_aajDr71ZU=xJ8m1diBLuG_jr__TO6tR8kAMS625q7hg@mail.gmail.com>

On Thu, Sep 5, 2013 at 10:31 AM, Jesus Cea <jcea at jcea.es> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> I just received an email from my OpenID provider, "myOpenID", saying
> that they drop OpenID service next February. I wonder what other
> OpenID providers are used by other python-dev fellows.
>
> What are you using?. bugs.python.org admins could share some data?
>
> I agree than OpenID is (quite) dead, but I rather prefer OpenID to use
> user/pass. I have big hopes for Mozilla Persona, looking forward
> Python infrastructure support :).
>
> PS: I use "http://www.jcea.es/" as my OpenID identity, and I delegate
> the actual service to "myOpenID". I can switch delegation trivially.
>

http://bugs.python.org/?@action=openid_login&provider=Google

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130905/3d56e85c/attachment.html>

From donald at stufft.io  Thu Sep  5 20:50:44 2013
From: donald at stufft.io (Donald Stufft)
Date: Thu, 5 Sep 2013 14:50:44 -0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <20130905184337.GA26926@iskra.aviel.ru>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
Message-ID: <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>


On Sep 5, 2013, at 2:43 PM, Oleg Broytman <phd at phdru.name> wrote:

> On Thu, Sep 05, 2013 at 02:35:16PM -0400, Donald Stufft <donald at stufft.io> wrote:
>> Persona is the logical successor to OpenID.
> 
>   OpenID lived a short life and died a quiet death. I'm afraid Persona
> wouldn't live even that much. Dead-born idea, in my so humble opinion.

I don't think there's much evidence to support this. I'm seeing more sites support Persona
not less. It solves some of the major problems with OpenID.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130905/dedf861c/attachment.sig>

From barry at python.org  Thu Sep  5 20:53:43 2013
From: barry at python.org (Barry Warsaw)
Date: Thu, 5 Sep 2013 14:53:43 -0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <20130905183347.GO10747@unaka.lan>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <20130905183347.GO10747@unaka.lan>
Message-ID: <20130905145343.08d40db8@anarchist>

On Sep 05, 2013, at 11:33 AM, Toshio Kuratomi wrote:

>Sortof.... The way OAuth looks to me, it's designed to prove that a given
>client is authorized to perform an action.  It's not designed to prove that
>the given client is a specific person.  In some cases, you really want to
>know the latter and not merely the former.  So I think in these situations
>Donald's separation of Authz and Authn makes sense.

This probably isn't the only application of these technologies, but I've
always thought about OAuth as delegating authority to scripts and programs to
act on your behalf.  For example, you can write a script to interact with
Launchpad's REST API, but before you can use the script, you have to interact
with the web ui once (since your browser is trusted, presumably) to receive a
token which the script can then use to prove that it's acting on your behalf.
If at some point you stop trusting that script, you can revoke the token to
disable its access, without having to reset your password.

To me, OpenID is about logging into web sites using single-sign on.  For
example, once I've logged into Launchpad, I can essentially go anywhere that
accepts OpenID, type my OpenID and generally not have to log in again (things
like two-factor auth and such may change that interaction pattern).

Or to summarize to a rough approximation: OpenID is for logins, OAuth is for
scripts.

Persona seems to fit the OpenID use case.  You'd still want OAuth for
scripting.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130905/9a5db88e/attachment-0001.sig>

From skip at pobox.com  Thu Sep  5 21:07:11 2013
From: skip at pobox.com (Skip Montanaro)
Date: Thu, 5 Sep 2013 14:07:11 -0500
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
Message-ID: <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com>

>> On Thu, Sep 05, 2013 at 02:35:16PM -0400, Donald Stufft <donald at stufft.io> wrote:
>>> Persona is the logical successor to OpenID.
>>
>>   OpenID lived a short life and died a quiet death. I'm afraid Persona
>> wouldn't live even that much. Dead-born idea, in my so humble opinion.
>
> I don't think there's much evidence to support this. I'm seeing more sites support Persona
> not less. It solves some of the major problems with OpenID.

I was completely unaware of OpenID's demise.  Can someone point me
to/provide an explanation? I much prefer using OpenID to login to a
site than having to either come up with yet another username/password
which I will just forget, or using Facebook or similar (I don't really
trust them with my info).

Thx,

Skip

From solipsis at pitrou.net  Thu Sep  5 21:08:18 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 5 Sep 2013 21:08:18 +0200
Subject: [Python-Dev] Offtopic: OpenID Providers
References: <5228C00F.7000209@jcea.es>
Message-ID: <20130905210818.5fd499f8@fsol>

On Thu, 05 Sep 2013 19:31:59 +0200
Jesus Cea <jcea at jcea.es> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> I just received an email from my OpenID provider, "myOpenID", saying
> that they drop OpenID service next February. I wonder what other
> OpenID providers are used by other python-dev fellows.

I use a self-hosted SimpleID instance:

> 
> What are you using?. bugs.python.org admins could share some data?
> 
> I agree than OpenID is (quite) dead, but I rather prefer OpenID to use
> user/pass. I have big hopes for Mozilla Persona, looking forward
> Python infrastructure support :).
> 
> PS: I use "http://www.jcea.es/" as my OpenID identity, and I delegate
> the actual service to "myOpenID". I can switch delegation trivially.
> 
> - -- 
> Jes?s Cea Avi?n                         _/_/      _/_/_/        _/_/_/
> jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
> Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
> jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
> "Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
> "My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
> "El amor es poner tu felicidad en la felicidad de otro" - Leibniz
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
> 
> iQCVAwUBUijAD5lgi5GaxT1NAQLH0wQAkKDORrAtfJFzzIHl3lHRp7GfxOzdqdNP
> uiuW65l/pas+p9+B0G6qR6EE2AAL7YPozcNF5AkmuGmxkpyn/JyUYKJcUWmUotpj
> V9Buz9jz3qpPuv7AlTnMbjBBQK4YTYenbdk2HgI41SVQHZHkU/+y4CL3Y1hWyJNo
> C8CCWfR0VlA=
> =YXIq
> -----END PGP SIGNATURE-----




From solipsis at pitrou.net  Thu Sep  5 21:10:51 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 5 Sep 2013 21:10:51 +0200
Subject: [Python-Dev] Offtopic: OpenID Providers
References: <5228C00F.7000209@jcea.es>
Message-ID: <20130905211051.307f38ce@fsol>

On Thu, 05 Sep 2013 19:31:59 +0200
Jesus Cea <jcea at jcea.es> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> I just received an email from my OpenID provider, "myOpenID", saying
> that they drop OpenID service next February. I wonder what other
> OpenID providers are used by other python-dev fellows.
> 
> What are you using?. bugs.python.org admins could share some data?

I use a self-hosted SimpleID instance:
http://simpleid.koinic.net/

It works fine with python.org, except recent problems with PyPI for
which I haven't had an answer yet:
https://mail.python.org/pipermail/distutils-sig/2013-September/022583.html

(sorry for the previous message, keyboard mishap)

Regards

Antoine.



From a.badger at gmail.com  Thu Sep  5 21:15:30 2013
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Thu, 5 Sep 2013 12:15:30 -0700
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <20130905145343.08d40db8@anarchist>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <20130905183347.GO10747@unaka.lan>
 <20130905145343.08d40db8@anarchist>
Message-ID: <20130905191529.GP10747@unaka.lan>

On Thu, Sep 05, 2013 at 02:53:43PM -0400, Barry Warsaw wrote:
> 
> This probably isn't the only application of these technologies, but I've
> always thought about OAuth as delegating authority to scripts and programs to
> act on your behalf.  For example, you can write a script to interact with
> Launchpad's REST API, but before you can use the script, you have to interact
> with the web ui once (since your browser is trusted, presumably) to receive a
> token which the script can then use to prove that it's acting on your behalf.
> If at some point you stop trusting that script, you can revoke the token to
> disable its access, without having to reset your password.
> 
> To me, OpenID is about logging into web sites using single-sign on.  For
> example, once I've logged into Launchpad, I can essentially go anywhere that
> accepts OpenID, type my OpenID and generally not have to log in again (things
> like two-factor auth and such may change that interaction pattern).
> 
> Or to summarize to a rough approximation: OpenID is for logins, OAuth is for
> scripts.
> 
> Persona seems to fit the OpenID use case.  You'd still want OAuth for
> scripting.
> 
<nod>  However, in some cases, Persona/OpenID can make more sense for
scripts.  For instance, if you have a script that is primarily interactive
in nature, it may be better to have the user login via that script than to
have an OAuth token laying around on the filesystem all the time
(Contrariwise, if the script is primarily run from cron or similar, it's
better to have a token with limited permissions laying around on the
filesystem than your OpenID password ;-)

It's probably also useful to point out that OAuth (because it was developed
to let third party websites have limited permission to act on your behalf)
is more paranoid than strictly required for many scripts where that
"third-party" is a script that you've written running on a box that you
control.  If that's the main use case for your service, OAuth may not be
a good fit for your authz needs.

-Toshio
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130905/b8ea2fbb/attachment.sig>

From solipsis at pitrou.net  Thu Sep  5 21:07:33 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 5 Sep 2013 21:07:33 +0200
Subject: [Python-Dev] Offtopic: OpenID Providers
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
Message-ID: <20130905210733.29975b0a@fsol>

On Thu, 5 Sep 2013 14:50:44 -0400
Donald Stufft <donald at stufft.io> wrote:
> 
> On Sep 5, 2013, at 2:43 PM, Oleg Broytman <phd at phdru.name> wrote:
> 
> > On Thu, Sep 05, 2013 at 02:35:16PM -0400, Donald Stufft <donald at stufft.io> wrote:
> >> Persona is the logical successor to OpenID.
> > 
> >   OpenID lived a short life and died a quiet death. I'm afraid Persona
> > wouldn't live even that much. Dead-born idea, in my so humble opinion.
> 
> I don't think there's much evidence to support this. I'm seeing more sites support Persona
> not less.

Which sites exactly? I can login to BitBucket and *.python.org using
OpenID, not Persona.




From barry at python.org  Thu Sep  5 21:40:44 2013
From: barry at python.org (Barry Warsaw)
Date: Thu, 5 Sep 2013 15:40:44 -0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <20130905210733.29975b0a@fsol>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
 <20130905210733.29975b0a@fsol>
Message-ID: <20130905154044.44d93e67@anarchist>

On Sep 05, 2013, at 09:07 PM, Antoine Pitrou wrote:

>Which sites exactly? I can login to BitBucket and *.python.org using
>OpenID, not Persona.

I think Persona is just too new to see it around much yet.  Or maybe Mozilla
needs better PR.

-Barry

From skip at pobox.com  Thu Sep  5 21:52:29 2013
From: skip at pobox.com (Skip Montanaro)
Date: Thu, 5 Sep 2013 14:52:29 -0500
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <20130905154044.44d93e67@anarchist>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
 <20130905210733.29975b0a@fsol> <20130905154044.44d93e67@anarchist>
Message-ID: <CANc-5Uw6k5Susim4wAcPK3nUkV4ewOu+Y9c0iP7fSuSrfUPfNw@mail.gmail.com>

> I think Persona is just too new to see it around much yet.  Or maybe Mozilla
> needs better PR.

The Persona site touts: "Signing in using Persona requires only a
valid email address; allowing you to provide personal information on
as-needed basis, when and where you think it?s appropriate."

They clearly need a better example site. They chose something called
Voost. Sure enough, all I needed to enter was my Gmail address. That
got me signed in, but then Voost asked me for a bunch of other
personal information (name, gender, birthdate, etc), and wouldn't let
me go any farther without that. :-/

Skip

From ben+python at benfinney.id.au  Thu Sep  5 21:53:25 2013
From: ben+python at benfinney.id.au (Ben Finney)
Date: Fri, 06 Sep 2013 05:53:25 +1000
Subject: [Python-Dev] Offtopic: OpenID Providers
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
 <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com>
Message-ID: <7wk3ivyrii.fsf@benfinney.id.au>

Skip Montanaro <skip at pobox.com> writes:

> >> On Thu, Sep 05, 2013 at 02:35:16PM -0400, Donald Stufft <donald at stufft.io> wrote:
> >>> Persona is the logical successor to OpenID.
> >>
> >>   OpenID lived a short life and died a quiet death. I'm afraid Persona
> >> wouldn't live even that much. Dead-born idea, in my so humble opinion.
> >
> > I don't think there's much evidence to support this. I'm seeing more sites support Persona
> > not less. It solves some of the major problems with OpenID.
>
> I was completely unaware of OpenID's demise.

It has failed at its stated purpose, which was to obviate the need for
services to provide their own ad hoc systems and allow users to
consolidate their digital identities.

This is evident by lookig at how few sites have added OpenID login in
the past several years, and how many that once had it have dropped it.

If you're unaware of that, I can only surmise you haven't been trying to
log in with an OpenID to anything newer than about 2009.

> Can someone point me to/provide an explanation?

An explanation in terms of what? I can point you to punditry
<URL:http://www.25hoursaday.com/weblog/2011/01/30/LearningFromOurMistakesTheFailureOfOpenIDAtomPubAndXMLOnTheWeb.aspx>
and hand-wringing
<URL:http://geekyschmidt.com/2011/01/31/openid-death-greatly-exaggerated>.

My own take is that most people choose convenience and expedience over
security and freedom, hence Facebook and Twitter and Google have taken
over the online identity game instead of a federated identity system.

> I much prefer using OpenID to login to a site than having to either
> come up with yet another username/password which I will just forget,
> or using Facebook or similar (I don't really trust them with my info).

Agreed. Our preferences are not enough though.

-- 
 \      ?Anyone who believes exponential growth can go on forever in a |
  `\        finite world is either a madman or an economist.? ?Kenneth |
_o__)                                                         Boulding |
Ben Finney


From phd at phdru.name  Thu Sep  5 22:29:21 2013
From: phd at phdru.name (Oleg Broytman)
Date: Fri, 6 Sep 2013 00:29:21 +0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
Message-ID: <20130905202921.GA30331@iskra.aviel.ru>

On Thu, Sep 05, 2013 at 02:50:44PM -0400, Donald Stufft <donald at stufft.io> wrote:
> On Sep 5, 2013, at 2:43 PM, Oleg Broytman <phd at phdru.name> wrote:
> > On Thu, Sep 05, 2013 at 02:35:16PM -0400, Donald Stufft <donald at stufft.io> wrote:
> >> Persona is the logical successor to OpenID.
> > 
> >   OpenID lived a short life and died a quiet death. I'm afraid Persona
> > wouldn't live even that much. Dead-born idea, in my so humble opinion.
> 
> I don't think there's much evidence to support this. I'm seeing more sites support Persona
> not less. It solves some of the major problems with OpenID.

   I have seen exactly 0 (zero) sites that support Persona. Can you
point me?

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From tseaver at palladion.com  Thu Sep  5 22:30:31 2013
From: tseaver at palladion.com (Tres Seaver)
Date: Thu, 05 Sep 2013 16:30:31 -0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <CANc-5Uw6k5Susim4wAcPK3nUkV4ewOu+Y9c0iP7fSuSrfUPfNw@mail.gmail.com>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
 <20130905210733.29975b0a@fsol> <20130905154044.44d93e67@anarchist>
 <CANc-5Uw6k5Susim4wAcPK3nUkV4ewOu+Y9c0iP7fSuSrfUPfNw@mail.gmail.com>
Message-ID: <l0apkv$m40$1@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/05/2013 03:52 PM, Skip Montanaro wrote:
>> I think Persona is just too new to see it around much yet.  Or maybe
>> Mozilla needs better PR.
> 
> The Persona site touts: "Signing in using Persona requires only a 
> valid email address; allowing you to provide personal information on 
> as-needed basis, when and where you think it?s appropriate."
> 
> They clearly need a better example site. They chose something called 
> Voost. Sure enough, all I needed to enter was my Gmail address. That 
> got me signed in, but then Voost asked me for a bunch of other 
> personal information (name, gender, birthdate, etc), and wouldn't let 
> me go any farther without that. :-/

As sith OpenID, the key element to Persona is SSO:  you can authenticate
without needing to create / remember passwords for every site you visit.
 Whether a given site chooses to authroize an
authenticated-but-otherwise-unknown user to do anything meaningful is
logically distinct.

+1 for supporting Persona as an alternative to OpenID on all *.python.org
servers.



Tres.
- -- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlIo6ecACgkQ+gerLs4ltQ6gOwCgrIokRYnddNaNVIVPoY/M4d0k
kKcAni6hxXQE4T4QMij3bQHAJwBFX1uW
=9ZJP
-----END PGP SIGNATURE-----


From phd at phdru.name  Thu Sep  5 22:36:51 2013
From: phd at phdru.name (Oleg Broytman)
Date: Fri, 6 Sep 2013 00:36:51 +0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
 <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com>
Message-ID: <20130905203651.GB30331@iskra.aviel.ru>

On Thu, Sep 05, 2013 at 02:07:11PM -0500, Skip Montanaro <skip at pobox.com> wrote:
> >>   OpenID lived a short life and died a quiet death. I'm afraid Persona
> >> wouldn't live even that much. Dead-born idea, in my so humble opinion.
> >
> I was completely unaware of OpenID's demise.

   There was no demise. Because there was no take-off. OpenID was never
popular. I can remember a very limited set of major sites that allow
login using OpenID: SourceForge, LiveJournal, BitBucket. The first two
in the list allow limited login. BitBucket doesn't allow even that. They
only allow full-featured login if you have already created an account
and linked your OpenID URL with that account.
   You cannot login using OpenID to most interesting popular sites.
GMail? No. Twitter? No. Facebook? FriendFeed? identi.ca? No, no, no.
   Small uninteresting blogs? Yes, but who cares?

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From v+python at g.nevcal.com  Thu Sep  5 22:33:48 2013
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Thu, 05 Sep 2013 13:33:48 -0700
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <l0apkv$m40$1@ger.gmane.org>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
 <20130905210733.29975b0a@fsol> <20130905154044.44d93e67@anarchist>
 <CANc-5Uw6k5Susim4wAcPK3nUkV4ewOu+Y9c0iP7fSuSrfUPfNw@mail.gmail.com>
 <l0apkv$m40$1@ger.gmane.org>
Message-ID: <5228EAAC.2020800@g.nevcal.com>

On 9/5/2013 1:30 PM, Tres Seaver wrote:
> +1 for supporting Persona as an alternative to OpenID on all *.python.org
> servers.

Is there a Python implementation of Persona I can install on my web server?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130905/1e708f42/attachment-0001.html>

From solipsis at pitrou.net  Thu Sep  5 22:40:47 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 5 Sep 2013 22:40:47 +0200
Subject: [Python-Dev] Offtopic: OpenID Providers
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
 <20130905210733.29975b0a@fsol> <20130905154044.44d93e67@anarchist>
Message-ID: <20130905224047.1867fdee@fsol>

On Thu, 5 Sep 2013 15:40:44 -0400
Barry Warsaw <barry at python.org> wrote:

> On Sep 05, 2013, at 09:07 PM, Antoine Pitrou wrote:
> 
> >Which sites exactly? I can login to BitBucket and *.python.org using
> >OpenID, not Persona.
> 
> I think Persona is just too new to see it around much yet.  Or maybe Mozilla
> needs better PR.

Well, OpenID at least got some publicity since it appeared. Persona is
almost unknown at this point (though it was publicly launched two years
ago, according to Wikipedia).

Comparing the size of the respective Wikipedia pages actually tells
quite a bit:
http://en.wikipedia.org/wiki/OpenID
http://en.wikipedia.org/wiki/Mozilla_Persona

Regards

Antoine.



From fred at fdrake.net  Thu Sep  5 22:37:25 2013
From: fred at fdrake.net (Fred Drake)
Date: Thu, 5 Sep 2013 16:37:25 -0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <20130905202921.GA30331@iskra.aviel.ru>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
 <20130905202921.GA30331@iskra.aviel.ru>
Message-ID: <CAFT4OTFUsLBQFBW6nyFufq_3+eo68V80JwErYJ=KGVsevU0boQ@mail.gmail.com>

On Thu, Sep 5, 2013 at 4:29 PM, Oleg Broytman <phd at phdru.name> wrote:
> I have seen exactly 0 (zero) sites that support Persona. Can you
> point me?

We have an internal app that uses Persona, but we did that mostly to
play with it.

I've not run across any sites that use Persona in the wild, either.


  -Fred

-- 
Fred L. Drake, Jr.    <fred at fdrake.net>
"A storm broke loose in my mind."  --Albert Einstein

From benjamin at python.org  Thu Sep  5 22:45:57 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Thu, 5 Sep 2013 16:45:57 -0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <5228EAAC.2020800@g.nevcal.com>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
 <20130905210733.29975b0a@fsol> <20130905154044.44d93e67@anarchist>
 <CANc-5Uw6k5Susim4wAcPK3nUkV4ewOu+Y9c0iP7fSuSrfUPfNw@mail.gmail.com>
 <l0apkv$m40$1@ger.gmane.org> <5228EAAC.2020800@g.nevcal.com>
Message-ID: <CAPZV6o9DHy6LOWgP6zj+GsZjZyyf4K7SsN=qXmV-RArpaSMhuA@mail.gmail.com>

There's some sample Python code here:
https://developer.mozilla.org/en-US/docs/Mozilla/Persona/Quick_Setup

The API is so simple something generic like requests suffices.

2013/9/5 Glenn Linderman <v+python at g.nevcal.com>:
> On 9/5/2013 1:30 PM, Tres Seaver wrote:
>
> +1 for supporting Persona as an alternative to OpenID on all *.python.org
> servers.
>
>
> Is there a Python implementation of Persona I can install on my web server?
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org
>



-- 
Regards,
Benjamin

From skip at pobox.com  Thu Sep  5 22:42:17 2013
From: skip at pobox.com (Skip Montanaro)
Date: Thu, 5 Sep 2013 15:42:17 -0500
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <l0apkv$m40$1@ger.gmane.org>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
 <20130905210733.29975b0a@fsol> <20130905154044.44d93e67@anarchist>
 <CANc-5Uw6k5Susim4wAcPK3nUkV4ewOu+Y9c0iP7fSuSrfUPfNw@mail.gmail.com>
 <l0apkv$m40$1@ger.gmane.org>
Message-ID: <CANc-5UzpJcwsCa_0ScBQ=T43vM8ac1UVrn60vq6m=We7SmiQGw@mail.gmail.com>

> Whether a given site chooses to authroize an
> authenticated-but-otherwise-unknown user to do anything meaningful is
> logically distinct.

But the least they could have done was pick a demo site that didn't do
exactly what they contend you shouldn't need to do: cough up all sorts
of personal information to use their site.

Skip

From barry at python.org  Thu Sep  5 22:53:18 2013
From: barry at python.org (Barry Warsaw)
Date: Thu, 5 Sep 2013 16:53:18 -0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <20130905203651.GB30331@iskra.aviel.ru>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
 <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com>
 <20130905203651.GB30331@iskra.aviel.ru>
Message-ID: <20130905165318.2fa40bab@anarchist>

On Sep 06, 2013, at 12:36 AM, Oleg Broytman wrote:

>   You cannot login using OpenID to most interesting popular sites.
>GMail? No. Twitter? No. Facebook? FriendFeed? identi.ca? No, no, no.

I'd be surprised if you ever saw the big social networking sites support
OpenID or Persona.  They want to own that space themselves, so probably have
no business incentive to support 3rd party systems.

We're open source, and I think it benefits our mission to support open,
decentralized, and free systems like OpenID and Persona.

-Barry

From donald at stufft.io  Thu Sep  5 22:57:08 2013
From: donald at stufft.io (Donald Stufft)
Date: Thu, 5 Sep 2013 16:57:08 -0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <20130905165318.2fa40bab@anarchist>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
 <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com>
 <20130905203651.GB30331@iskra.aviel.ru> <20130905165318.2fa40bab@anarchist>
Message-ID: <00E21A32-DEC0-4CB0-A9AA-CDE3E2307D86@stufft.io>


On Sep 5, 2013, at 4:53 PM, Barry Warsaw <barry at python.org> wrote:

> On Sep 06, 2013, at 12:36 AM, Oleg Broytman wrote:
> 
>>  You cannot login using OpenID to most interesting popular sites.
>> GMail? No. Twitter? No. Facebook? FriendFeed? identi.ca? No, no, no.
> 
> I'd be surprised if you ever saw the big social networking sites support
> OpenID or Persona.  They want to own that space themselves, so probably have
> no business incentive to support 3rd party systems.

Not that it changes this statement at all but you wouldn't expect to see a Persona login
for gmail as persona solves the problem that people don't think of urls as personal
identifiers by replacing it with emails. So Gmail would be the Persona IdP

> 
> We're open source, and I think it benefits our mission to support open,
> decentralized, and free systems like OpenID and Persona.
> 
> -Barry
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130905/48e3cb8a/attachment.sig>

From tseaver at palladion.com  Thu Sep  5 22:58:19 2013
From: tseaver at palladion.com (Tres Seaver)
Date: Thu, 05 Sep 2013 16:58:19 -0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <20130905202921.GA30331@iskra.aviel.ru>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
 <20130905202921.GA30331@iskra.aviel.ru>
Message-ID: <l0ar93$758$1@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/05/2013 04:29 PM, Oleg Broytman wrote:
> On Thu, Sep 05, 2013 at 02:50:44PM -0400, Donald Stufft
> <donald at stufft.io> wrote:
>> On Sep 5, 2013, at 2:43 PM, Oleg Broytman <phd at phdru.name> wrote:
>>> On Thu, Sep 05, 2013 at 02:35:16PM -0400, Donald Stufft
>>> <donald at stufft.io> wrote:
>>>> Persona is the logical successor to OpenID.
>>> 
>>> OpenID lived a short life and died a quiet death. I'm afraid
>>> Persona wouldn't live even that much. Dead-born idea, in my so
>>> humble opinion.
>> 
>> I don't think there's much evidence to support this. I'm seeing more
>> sites support Persona not less. It solves some of the major problems
>> with OpenID.
> 
> I have seen exactly 0 (zero) sites that support Persona. Can you point
> me?


- From the "Mozilla Identity" blog:

- - https://webmaker.org/

- - http://bornthiswayfoundation.org/

- - http://firebase.com/

- - https://orionhub.org/

- - http://ting.com/

- - http://www.gnu.org/software/mailman/

- - http://discourse.org/

- - https://dailycred.com/



Tres.
- -- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlIo8GoACgkQ+gerLs4ltQ5OtACdEv/XvYKwGuFQESuLn+uBkNzm
UUAAn2YY22+YL+tS0WhE+95tIb0USmV7
=cB96
-----END PGP SIGNATURE-----


From dirkjan at ochtman.nl  Thu Sep  5 23:03:17 2013
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Thu, 5 Sep 2013 23:03:17 +0200
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <5228EAAC.2020800@g.nevcal.com>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
 <20130905210733.29975b0a@fsol> <20130905154044.44d93e67@anarchist>
 <CANc-5Uw6k5Susim4wAcPK3nUkV4ewOu+Y9c0iP7fSuSrfUPfNw@mail.gmail.com>
 <l0apkv$m40$1@ger.gmane.org> <5228EAAC.2020800@g.nevcal.com>
Message-ID: <CAKmKYaCeohRSmfjag3DBG7SAksh1pa2E6k8qQEesrHLx+C7=Ag@mail.gmail.com>

On Thu, Sep 5, 2013 at 10:33 PM, Glenn Linderman <v+python at g.nevcal.com> wrote:
> Is there a Python implementation of Persona I can install on my web server?

If you mean to use your web server as an Identity Provider, try this:

https://bitbucket.org/djc/persona-totp

It currently only implements TOTP-based authentication (i.e. no
passwords), but it should be easy to add a password or 2FA-mode if
you'd prefer that.

Cheers,

Dirkjan

From tseaver at palladion.com  Thu Sep  5 23:01:23 2013
From: tseaver at palladion.com (Tres Seaver)
Date: Thu, 05 Sep 2013 17:01:23 -0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <5228EAAC.2020800@g.nevcal.com>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
 <20130905210733.29975b0a@fsol> <20130905154044.44d93e67@anarchist>
 <CANc-5Uw6k5Susim4wAcPK3nUkV4ewOu+Y9c0iP7fSuSrfUPfNw@mail.gmail.com>
 <l0apkv$m40$1@ger.gmane.org> <5228EAAC.2020800@g.nevcal.com>
Message-ID: <l0ares$a70$1@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/05/2013 04:33 PM, Glenn Linderman wrote:
> On 9/5/2013 1:30 PM, Tres Seaver wrote:
>> +1 for supporting Persona as an alternative to OpenID on all
>> *.python.org servers.
> 
> Is there a Python implementation of Persona I can install on my web
> server?

- - https://readthedocs.org/projects/django-browserid/

- - https://pyramid_persona.readthedocs.org/en/latest/


Tres.
- -- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlIo8SMACgkQ+gerLs4ltQ57xgCfT2xVkJfMtuDjmed6jlhfsD8I
fxMAoNHT69tbi3iKtpkyEcSY0lwJosqc
=TTJ8
-----END PGP SIGNATURE-----


From dirkjan at ochtman.nl  Thu Sep  5 23:05:55 2013
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Thu, 5 Sep 2013 23:05:55 +0200
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <00E21A32-DEC0-4CB0-A9AA-CDE3E2307D86@stufft.io>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
 <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com>
 <20130905203651.GB30331@iskra.aviel.ru> <20130905165318.2fa40bab@anarchist>
 <00E21A32-DEC0-4CB0-A9AA-CDE3E2307D86@stufft.io>
Message-ID: <CAKmKYaDjssPK4F_=9vrC4687aP=BpjzYgbq1cygJ9fTVXXpAMg@mail.gmail.com>

On Thu, Sep 5, 2013 at 10:57 PM, Donald Stufft <donald at stufft.io> wrote:
> Not that it changes this statement at all but you wouldn't expect to see a Persona login
> for gmail as persona solves the problem that people don't think of urls as personal
> identifiers by replacing it with emails. So Gmail would be the Persona IdP

And actually you can already trivially login with your GMail account
on any Persona-based Relying Party (that is, a site that uses Persona
to authenticate you). This is because one of the nice parts of the
current implementation of Persona is that Mozilla has implemented
bridges that allow GMail and Yahoo addresses to be authenticated via
their respective OAuth implementations, such that you don't need to
setup an account at Mozilla's fallback IdP (which acts as an Identity
Provider for email addresses that don't currently have an IdP
available to them).

Cheers,

Dirkjan

From dirkjan at ochtman.nl  Thu Sep  5 23:08:13 2013
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Thu, 5 Sep 2013 23:08:13 +0200
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <l0apkv$m40$1@ger.gmane.org>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
 <20130905210733.29975b0a@fsol> <20130905154044.44d93e67@anarchist>
 <CANc-5Uw6k5Susim4wAcPK3nUkV4ewOu+Y9c0iP7fSuSrfUPfNw@mail.gmail.com>
 <l0apkv$m40$1@ger.gmane.org>
Message-ID: <CAKmKYaBVYE6HN2XDgHNT4b5gHuQ3W5GzL8mr5x_b8vqLtZs-gg@mail.gmail.com>

On Thu, Sep 5, 2013 at 10:30 PM, Tres Seaver <tseaver at palladion.com> wrote:
> +1 for supporting Persona as an alternative to OpenID on all *.python.org
> servers.

BTW, I'd be happy to assist with any Persona RP implementations for
python.org services. The MDN docs are pretty good, I got my first RP
going in just a few hours of looking at code (and you can probably do
better if you're more into frontend webdev stuff).

There's also ongoing work that will replace ReadTheDocs accounts with
Persona support.

Cheers,

Dirkjan

From phd at phdru.name  Thu Sep  5 23:09:24 2013
From: phd at phdru.name (Oleg Broytman)
Date: Fri, 6 Sep 2013 01:09:24 +0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <20130905165318.2fa40bab@anarchist>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
 <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com>
 <20130905203651.GB30331@iskra.aviel.ru>
 <20130905165318.2fa40bab@anarchist>
Message-ID: <20130905210924.GA30649@iskra.aviel.ru>

On Thu, Sep 05, 2013 at 04:53:18PM -0400, Barry Warsaw <barry at python.org> wrote:
> On Sep 06, 2013, at 12:36 AM, Oleg Broytman wrote:
> >   You cannot login using OpenID to most interesting popular sites.
> >GMail? No. Twitter? No. Facebook? FriendFeed? identi.ca? No, no, no.
> 
> I'd be surprised if you ever saw the big social networking sites support
> OpenID or Persona.  They want to own that space themselves, so probably have
> no business incentive to support 3rd party systems.

   But of course! And that IMO spells the end of the feature. Things
that aren't available for millions seldom are available for a few, and
if they are -- they are available for big price.

> We're open source, and I think it benefits our mission to support open,
> decentralized, and free systems like OpenID and Persona.

   But they also have disadvantages. Implementing such a major feature
is a significant burden to sysadmins and is an additional vein for
security breaches.
   That said, I don't mind if pydotorg would get such features. If FSF
pays salaries and admins are willing to work -- no objections from me.
But I am not going to use it. What gain if I can login to one site?
   I will change my mind when Google and GitHub start using them.

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From phd at phdru.name  Thu Sep  5 23:25:10 2013
From: phd at phdru.name (Oleg Broytman)
Date: Fri, 6 Sep 2013 01:25:10 +0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <l0ar93$758$1@ger.gmane.org>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
 <20130905202921.GA30331@iskra.aviel.ru>
 <l0ar93$758$1@ger.gmane.org>
Message-ID: <20130905212510.GB30649@iskra.aviel.ru>

On Thu, Sep 05, 2013 at 04:58:19PM -0400, Tres Seaver <tseaver at palladion.com> wrote:
> On 09/05/2013 04:29 PM, Oleg Broytman wrote:
> > I have seen exactly 0 (zero) sites that support Persona. Can you point
> > me?
> 
> - From the "Mozilla Identity" blog:
> 
> - - https://webmaker.org/
> - - http://bornthiswayfoundation.org/
> - - http://firebase.com/
> - - https://orionhub.org/
> - - http://ting.com/
> - - http://www.gnu.org/software/mailman/
> - - http://discourse.org/
> - - https://dailycred.com/

   Thank you! Never heard of these sites. Well, I saw WebMaker once, but
it's a Mozilla site, no wonder it supports Persona.

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From barry at python.org  Thu Sep  5 23:29:07 2013
From: barry at python.org (Barry Warsaw)
Date: Thu, 5 Sep 2013 17:29:07 -0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <20130905210924.GA30649@iskra.aviel.ru>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
 <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com>
 <20130905203651.GB30331@iskra.aviel.ru>
 <20130905165318.2fa40bab@anarchist>
 <20130905210924.GA30649@iskra.aviel.ru>
Message-ID: <20130905172907.67a08d29@anarchist>

On Sep 06, 2013, at 01:09 AM, Oleg Broytman wrote:

>I will change my mind when Google and GitHub start using them.

Neither Google nor GitHub are free or open.  Bitbucket and Facebook aren't
either.  I'm not saying they're bad services because of that of course, but I
don't want to have to rely on any of them to access python.org resources, and
I don't want my choice to be log into Facebook or manage a slew of passwords.

But I'm not volunteering to do the work, so I don't get to decide.  I'm just
stating that I think our principle should be that you *can* (not *must*) use
free and open services to access our resources.

-Barry

From victor.stinner at gmail.com  Thu Sep  5 23:30:03 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 5 Sep 2013 23:30:03 +0200
Subject: [Python-Dev] RFC: PEP 454: Add a new tracemalloc module
In-Reply-To: <CAP7h-xauzgw8UJqKJAb2WPMqPM9-_BvDhHgcj5RnKvj5yaMiYw@mail.gmail.com>
References: <CAMpsgwZTOBdXy2Rcxs0itnPZYZgHU4X4tPNf5syoZrOJ5gYCFw@mail.gmail.com>
 <CAP7h-xauzgw8UJqKJAb2WPMqPM9-_BvDhHgcj5RnKvj5yaMiYw@mail.gmail.com>
Message-ID: <CAMpsgwaXPgNdvtrajVX2AJ5+61JcSPA_vYPZbQYi=e+TGA-Jrw@mail.gmail.com>

2013/9/5 Alexander Belopolsky <alexander.belopolsky at gmail.com>:
> Please mention that this API is similar to that of faulthandler and add a
> link to faulthandler docs.

Done.

Victor

From victor.stinner at gmail.com  Thu Sep  5 23:30:53 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 5 Sep 2013 23:30:53 +0200
Subject: [Python-Dev] Add a new tracemalloc module to trace memory
	allocations
In-Reply-To: <CADiSq7dKojZ5f1bprhZZxu6DjovjWtwy-8o=-BYjYRvHZWqoFw@mail.gmail.com>
References: <CAMpsgwaa3pABNrMSw2S7-Og111u8DdXK6dYnWVJzS4TAKbFNEQ@mail.gmail.com>
 <CAMpsgwatDdFQg4q=b3s7W6BZ+WQmpo=KTeJZd7ESbuGG+RzaKA@mail.gmail.com>
 <CAGE7PNJ9DrKQ8Zb-E-F7QPRTP=t+S8StBvm3OQgvct8539gFPg@mail.gmail.com>
 <CAMpsgwZBeN7W+GK6hLSHp4VYXAYgWFQBTx_-K3J7KzOMDB3H=g@mail.gmail.com>
 <CADiSq7dKojZ5f1bprhZZxu6DjovjWtwy-8o=-BYjYRvHZWqoFw@mail.gmail.com>
Message-ID: <CAMpsgwa5YaeCXCRZrZcM=+V+0xSJhj246zJtP-MUqtaggfALYA@mail.gmail.com>

2013/9/1 Nick Coghlan <ncoghlan at gmail.com>:
> +1 from me for both tracemalloc and failmalloc in the same vein as
> faulthandler (and using similar API concepts to faulthandler).
>
> While I like the flat top level naming structure, we should clearly document
> these as implementation dependent runtime services. Other implementations
> may not provide them at all, and even if they do, many details will likely
> be different.
>
> The gc module may even fall into the same category.

I updated the PEP 454 to mention that tracemalloc has been written for CPython.

Victor

From victor.stinner at gmail.com  Thu Sep  5 23:37:52 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 5 Sep 2013 23:37:52 +0200
Subject: [Python-Dev] Add a new tracemalloc module to trace memory
	allocations
In-Reply-To: <CAGE7PNJ9DrKQ8Zb-E-F7QPRTP=t+S8StBvm3OQgvct8539gFPg@mail.gmail.com>
References: <CAMpsgwaa3pABNrMSw2S7-Og111u8DdXK6dYnWVJzS4TAKbFNEQ@mail.gmail.com>
 <CAMpsgwatDdFQg4q=b3s7W6BZ+WQmpo=KTeJZd7ESbuGG+RzaKA@mail.gmail.com>
 <CAGE7PNJ9DrKQ8Zb-E-F7QPRTP=t+S8StBvm3OQgvct8539gFPg@mail.gmail.com>
Message-ID: <CAMpsgwbGNUCtNjkR-+1cSnoBF32s9tRw-b4k=q-+0nY+HWE_4g@mail.gmail.com>

2013/8/31 Gregory P. Smith <greg at krypto.org>:
> Think of the possibilities, you could even setup a test runner to
> enable/disable before and after each test, test suite or test module to
> gather narrow statistics as to what code actually _caused_ the allocations
> rather than the ultimate individual file/line doing it.

It may be used with "-m test -R" to detect memory leaks, you need to
repeat a test.

> Taking that further: file and line information is great, but what if you
> extend the concept: could you allow for C API or even Python hooks to gather
> additional information at the time of each allocation or free? for example:
> Gathering the actual C and Python stack traces for correlation to figure out
> what call patterns lead allocations is powerful.

I modified the PEP 454 and the implementation. By default, tracemalloc
only stores 1 "frame" instance (filename and line number). But you can
now use tracemalloc.set_number_frame(10) to store 10 frames per memory
allocation. Don't store too much frames, remember that tracemalloc
traces all Python memory allocations! So when Python allocates 1 byte,
tracemalloc may need to allocate 1000 bytes to store the trace...

tracemalloc cannot get the C traceback. It was discussed in the design
of the PEP 445 (Add new APIs to customize Python memory allocators)
how to pass the C filename and line number. It was decided to not add
them to have a simpler API.

In general, a C function is binded to a Python function. Knowing the
Python function should be enough.

Victor

From phd at phdru.name  Thu Sep  5 23:56:22 2013
From: phd at phdru.name (Oleg Broytman)
Date: Fri, 6 Sep 2013 01:56:22 +0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <20130905172907.67a08d29@anarchist>
References: <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
 <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com>
 <20130905203651.GB30331@iskra.aviel.ru>
 <20130905165318.2fa40bab@anarchist>
 <20130905210924.GA30649@iskra.aviel.ru>
 <20130905172907.67a08d29@anarchist>
Message-ID: <20130905215622.GA31833@iskra.aviel.ru>

On Thu, Sep 05, 2013 at 05:29:07PM -0400, Barry Warsaw <barry at python.org> wrote:
> I don't want my choice to be log into Facebook or manage a slew of passwords.

   The last part is unavoidable. I regularly login to LiveJournal,
Twitter, SourceForge, BitBucket, Gitorious, GitHub and to hundreds of
other sites -- blogs, torrents, web shops. I already manage hundreds of
passwords. OpenID promised to save me from that and failed. Do you think
Persona would succeed in this regard (saved me from managing all those
passwords)? And if not -- what are the benefits? I already manage
hundreds of passwords -- two or three additional passwords for
bugs.python.org, wiki.python.org and so on don't make the situation
worse.
   IMO the very idea of single sign-on in the open web is meaningless.

> But I'm not volunteering to do the work, so I don't get to decide.  I'm just
> stating that I think our principle should be that you *can* (not *must*) use
> free and open services to access our resources.

   Well, I can only use services that are available, not those that are
promised. If python.org grows support for Persona -- who will be my
provider and for what price? I am not going to install and manage
additional software on my servers -- I don't want to be my own provider,
I have enough job already.

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From donald at stufft.io  Fri Sep  6 00:07:36 2013
From: donald at stufft.io (Donald Stufft)
Date: Thu, 5 Sep 2013 18:07:36 -0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <20130905215622.GA31833@iskra.aviel.ru>
References: <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
 <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com>
 <20130905203651.GB30331@iskra.aviel.ru> <20130905165318.2fa40bab@anarchist>
 <20130905210924.GA30649@iskra.aviel.ru> <20130905172907.67a08d29@anarchist>
 <20130905215622.GA31833@iskra.aviel.ru>
Message-ID: <637EAFBD-9870-48BD-9206-A8B67B4DBCD3@stufft.io>


On Sep 5, 2013, at 5:56 PM, Oleg Broytman <phd at phdru.name> wrote:

> On Thu, Sep 05, 2013 at 05:29:07PM -0400, Barry Warsaw <barry at python.org> wrote:
>> I don't want my choice to be log into Facebook or manage a slew of passwords.
> 
>   The last part is unavoidable. I regularly login to LiveJournal,
> Twitter, SourceForge, BitBucket, Gitorious, GitHub and to hundreds of
> other sites -- blogs, torrents, web shops. I already manage hundreds of
> passwords. OpenID promised to save me from that and failed. Do you think
> Persona would succeed in this regard (saved me from managing all those
> passwords)? And if not -- what are the benefits? I already manage
> hundreds of passwords -- two or three additional passwords for
> bugs.python.org, wiki.python.org and so on don't make the situation
> worse.
>   IMO the very idea of single sign-on in the open web is meaningless.
> 
>> But I'm not volunteering to do the work, so I don't get to decide.  I'm just
>> stating that I think our principle should be that you *can* (not *must*) use
>> free and open services to access our resources.
> 
>   Well, I can only use services that are available, not those that are
> promised. If python.org grows support for Persona -- who will be my
> provider and for what price? I am not going to install and manage
> additional software on my servers -- I don't want to be my own provider,
> I have enough job already.

Theoretically whoever runs the domain for your email address (since Persona
uses email as your identifier). In order to make it work as a stop gap they also
have more openid like idP's which they run a major one (that also offers a "bridge"
to make things like Gmail work on Persona without needing to register for anything
else).

> 
> Oleg.
> -- 
>     Oleg Broytman            http://phdru.name/            phd at phdru.name
>           Programmers don't die, they just GOSUB without RETURN.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130905/aadf01d0/attachment.sig>

From stephen at xemacs.org  Fri Sep  6 03:09:26 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 06 Sep 2013 10:09:26 +0900
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <20130905165318.2fa40bab@anarchist>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
 <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com>
 <20130905203651.GB30331@iskra.aviel.ru>
 <20130905165318.2fa40bab@anarchist>
Message-ID: <87hadyah89.fsf@uwakimon.sk.tsukuba.ac.jp>

Barry Warsaw writes:
 > On Sep 06, 2013, at 12:36 AM, Oleg Broytman wrote:

 > >   You cannot login using OpenID to most interesting popular sites.
 > >GMail? No. Twitter? No. Facebook? FriendFeed? identi.ca? No, no, no.
 >
 > I'd be surprised if you ever saw the big social networking sites support
 > OpenID or Persona.  They want to own that space themselves, so probably have
 > no business incentive to support 3rd party systems.

Quite the reverse, unfortunately.  That's why *those* sites *all*
appear on most sites that support OpenID.  They're not going to
delegate to each other until they are forced to.

 > We're open source, and I think it benefits our mission to support open,
 > decentralized, and free systems like OpenID and Persona.

Thus speaks an employee of yet another Provider-That-Won't-Accept-My-
Third-Party-Credentials.  Sorry, Barry, but you see the problem:
Unfortunately, we can't do it alone.  What needs to happen is there
needs to be a large network of sites that support login via O-D-F
systems like OpenID and Persona.  Too many of the sites I use (news
sources, GMail, etc) don't support them and my browser manages my
logins to most of them, so why bother learning OpenID, and then
setting it up site by site?

I'm not against it, but it's quixotic (and therefore valuable).

One reason that OpenID and Persona fail to achieve penetration is that
they overstate their mission.  A protocol that any email provider can
support is a protocol that provides authentication without
identification (imagine what havoc Dogbert could wreak with his own
Persona provider), and therefore cannot be used in authorization
(except trivially).  Think ident (port tcp/113).  And most general-
audience sites that want to provide high-quality "Web 2.0" service are
going to start by asking for your demographics.  It's probably at
least as effective as CAPTCHA for classifying mammals and 'bots, too!

The reason that the "big" providers can take advantage of these
protocols as providers without reciprocating as clients is that
identities on these sites are very valuable to at least 95% of people
who use them (that may or may not correspond to as much as 50% of the
accounts).  Losing your Facebook site for abuse of TOS is very costly:
you can't even contact your "circle" easily.  Nor do you want multiple
logins on one of these sites, because that will double the amount of
spam they send you.

Bottom line: A login via Facebook-provided OpenID means that the login
is unlikely to perform random mischief.

Of course, those issues are easy to deal with if you have even a bit
of Internet savvy.  So sites still have to worry about a deliberate
attack from a Facebook user, but a serious intruder has many ways to
get in the front door, so you need to lock up your Waterford crystal
and Noritake china anyway whether you support global ID logins or not.


From jcea at jcea.es  Fri Sep  6 03:43:53 2013
From: jcea at jcea.es (Jesus Cea)
Date: Fri, 06 Sep 2013 03:43:53 +0200
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <7wk3ivyrii.fsf@benfinney.id.au>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
 <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com>
 <7wk3ivyrii.fsf@benfinney.id.au>
Message-ID: <52293359.4080304@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/09/13 21:53, Ben Finney wrote:
> My own take is that most people choose convenience and expedience
> over security and freedom, hence Facebook and Twitter and Google
> have taken over the online identity game instead of a federated
> identity system.

That is one of the Persona improvements: If your email provider is not
supporting it, you can still use your email with "persona" (thru
Mozilla servers). If you provider support OAUTH authentication (let
say, Facebook, twitter, Google), You can use that identity to prove
your identity to Mozilla, and Mozilla to prove your email ID to any
Persona consumer. In the process, you get privacy (facebook doesn't
know where are you using authentication, beside Mozilla).

Being a Persona provider is easy, being a verifier is trivial.

An interesting property of Persona is that if its popularity grows, it
becomes decentralized "autom?gically".

Anyway, I was asking for alternative OpenID providers, not to open a
debate about single sign on methods :).

- -- 
Jes?s Cea Avi?n                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQCVAwUBUikzWZlgi5GaxT1NAQJMbgP/enbARAwUiDXzJ9PkBmvlAi/OrhXrrXwE
Vf1f6XHab+ERyJ6kj1V5yz6F0D0zxl6s7GL+abz7P0JEdGEGMfEJc+aK15HM2r3b
LAm9k5ofe+ysdJx0HEd/V6/viqAeK7medb5Hh3xNwxbY6qRe4VkXbAvc0KQuo7eR
1Uf005ibUT8=
=i0/D
-----END PGP SIGNATURE-----

From jcea at jcea.es  Fri Sep  6 03:51:30 2013
From: jcea at jcea.es (Jesus Cea)
Date: Fri, 06 Sep 2013 03:51:30 +0200
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <20130905203651.GB30331@iskra.aviel.ru>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
 <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com>
 <20130905203651.GB30331@iskra.aviel.ru>
Message-ID: <52293522.2060304@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/09/13 22:36, Oleg Broytman wrote:
> There was no demise. Because there was no take-off. OpenID was
> never popular. I can remember a very limited set of major sites
> that allow login using OpenID: SourceForge, LiveJournal, BitBucket.
> The first two

I remember a day where OpenID was the ONLY authentication method in
StackOverflow.

- -- 
Jes?s Cea Avi?n                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQCVAwUBUik1Iplgi5GaxT1NAQKHGQP/ayFi3hs5CWwlNCciFguNkUesQBfRaPFF
TxHrloNM2Uo9wB2dt2oAVhAQnGSTw3lIYoRMzhk5+tfx4Bn16QVPiDXnbotO5xqn
QkAH3ZE9q9/YTDDqPJPFXHP/7eoQwk9Ou4LrTJk97ofp0DM7JcfVm3W0Sys53wEQ
ddwoNkrIk/s=
=vFd+
-----END PGP SIGNATURE-----

From jcea at jcea.es  Fri Sep  6 03:52:35 2013
From: jcea at jcea.es (Jesus Cea)
Date: Fri, 06 Sep 2013 03:52:35 +0200
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <20130905215622.GA31833@iskra.aviel.ru>
References: <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
 <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com>
 <20130905203651.GB30331@iskra.aviel.ru> <20130905165318.2fa40bab@anarchist>
 <20130905210924.GA30649@iskra.aviel.ru> <20130905172907.67a08d29@anarchist>
 <20130905215622.GA31833@iskra.aviel.ru>
Message-ID: <52293563.5070206@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/09/13 23:56, Oleg Broytman wrote:

> Well, I can only use services that are available, not those that
> are promised. If python.org grows support for Persona -- who will
> be my provider and for what price? I am not going to install and
> manage additional software on my servers -- I don't want to be my
> own provider, I have enough job already.

If your email provider is not supporting Persona, the automatic
fallback is Mozilla, a non-profit, free web, I care about your
privacy, organization.

- -- 
Jes?s Cea Avi?n                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQCVAwUBUik1Y5lgi5GaxT1NAQLKjwP9EXXmMTVORDPOCK5ByBfrwveL71nG/1IJ
oTc0ZUjZPMp/JD2UBnibEUIhxJHtmAkTsuMTmNsSbzqx6F74n9zYokfMK4Y0iscC
2EpqDe7lcAzEjJXpIa93A/K8/fh3gaufWHAXND3Ynr7gMdlLkn9jRiXoVMHHX3Sm
lEIX/47kzlI=
=4+OP
-----END PGP SIGNATURE-----

From jcea at jcea.es  Fri Sep  6 03:54:01 2013
From: jcea at jcea.es (Jesus Cea)
Date: Fri, 06 Sep 2013 03:54:01 +0200
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <20130905202921.GA30331@iskra.aviel.ru>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
 <20130905202921.GA30331@iskra.aviel.ru>
Message-ID: <522935B9.9030208@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/09/13 22:29, Oleg Broytman wrote:

> I have seen exactly 0 (zero) sites that support Persona. Can you 
> point me?

"Python Espa?a" (Python Spain) association is going to provide Persona
Only login. Deployment in four weeks.

- -- 
Jes?s Cea Avi?n                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQCVAwUBUik1uZlgi5GaxT1NAQIA3wP/eH0HA+gxa/9c1S679lmC7GVoBqahWyWS
mmOnIg170MaozIxyjgLruReTKdnfhcFa/QxxWLpS7brGenOSA1UCXdOhRHRTpf5y
JMtQ/f1pmS3sdjEIal2F6Fm1Dt1i6YFZr813j9hdzVHgcx96PuxTx5UVO3LpAAkf
+edD8PBn3eM=
=y1Ri
-----END PGP SIGNATURE-----

From jcea at jcea.es  Fri Sep  6 03:56:55 2013
From: jcea at jcea.es (Jesus Cea)
Date: Fri, 06 Sep 2013 03:56:55 +0200
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <CAF-Rda_aajDr71ZU=xJ8m1diBLuG_jr__TO6tR8kAMS625q7hg@mail.gmail.com>
References: <5228C00F.7000209@jcea.es>
 <CAF-Rda_aajDr71ZU=xJ8m1diBLuG_jr__TO6tR8kAMS625q7hg@mail.gmail.com>
Message-ID: <52293667.8030609@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/09/13 20:45, Eli Bendersky wrote:
> PS: I use "http://www.jcea.es/" as my OpenID identity, and I
> delegate the actual service to "myOpenID". I can switch delegation
> trivially.
> 
> http://bugs.python.org/?@action=openid_login&provider=Google

Sorry, Google, Facebook, Twitter, etc., are not acceptable OpenID
providers for me. I should have made that point in my original email.
My excuses.

Any other suggestion?

- -- 
Jes?s Cea Avi?n                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQCVAwUBUik2Z5lgi5GaxT1NAQJEtQP+NdOaMbAB4LzgsUfoL1yelPcYA8wg2rnd
WjM7IEsfcs/3NFrmpg9XxjJqSwoxgqW8NqpcQLapGSUdqUxMrhYz3xUnXzSUmM75
IwxUWlTuVVQkscRWwrmIjqMWD20EI3KtmGBtCH5J1Tnmb1EeTH4+wrtpBcl3hBUi
Ph8+Afy+E6w=
=VkCq
-----END PGP SIGNATURE-----

From ericsnowcurrently at gmail.com  Fri Sep  6 07:26:31 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 5 Sep 2013 23:26:31 -0600
Subject: [Python-Dev] Pre-PEP: Redesigning extension modules
In-Reply-To: <kvctoj$qsv$1@ger.gmane.org>
References: <kv77o2$f96$1@ger.gmane.org>
 <CALeMXf5oxi0oX13zSfHwgVi1eXF_z_coY23MfBP-YOK8WVkWsg@mail.gmail.com>
 <kvctoj$qsv$1@ger.gmane.org>
Message-ID: <CALFfu7Bfvrp1yT0=X08P-f+CF8aLGifZzk8-Mv3YyzL69hUHnA@mail.gmail.com>

On Sat, Aug 24, 2013 at 7:07 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> PEP 3121 would no longer be necessary. Extension types can do all we need.
> No more special casing of modules, that was the idea.
>

One nice thing about PEP 3121 is the addition of md_state to module objects
to store internal module state.  Wouldn't we be better served by improving
the related API rather than abandoning it?  If md_state were the home for
all mutable internal state then load/reload could focus directly on just
md_state and md_dict and not worry about other internal state, since all
remaining state would be immutable (refcounts notwithstanding).  If the API
made this easier then we could leverage the strengths of PEP 3121 to make
loading safer and more independent.  Of course, we could certainly go the
other way and actively discourage mutable internal state...

This, coupled with the PEP 451-compatible API and with a proxying wrapper,
would go a long way to various "reloading" issues that extension modules
have.

On Sun, Aug 25, 2013 at 5:54 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> (regarding reloading into the existing module's namespace)
>
> I'm not sure this can be done in general. What if the module has threads
> running that access the global state? In that case, reinitialising the
> module object itself would almost certainly lead to a crash.
>
> And what if you do "from extmodule import some_function" in a Python
> module? Then reloading couldn't replace that reference, just as for normal
> Python modules. Meaning that you'd still have to keep both modules properly
> alive in order to prevent crashes due to lost global state of the imported
> function.
>
> The difference to Python modules here is that in Python code, you'll get
> some kind of exception if state is lost during a reload. In C code, you'll
> most likely get a crash.
>
> How would you even make sure global state is properly cleaned up? Would you
> call tp_clear() on the module object before re-running the init code? Or
> how else would you enable the init code to do the right thing during both
> the first run (where global state is uninitialised) and subsequent runs
> (where global state may hold valid state and owned Python references)?
>
> Even tp_clear() may not be enough, because it's only meant to clean up
> Python references, not C-level state. Basically, for reloading to be
> correct without changing the object reference, it would have to go all the
> way through tp_dealloc(), catch the object at the very end, right before it
> gets freed, and then re-initialise it.
>

Right.  It would probably require a separate
`PyImportInitializeState_<module>(PyObject *mod)` and/or some API that
helps make it easier to manage mutable internal module state (on md_state).


On Sun, Aug 25, 2013 at 6:36 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> PJ Eby, 25.08.2013 06:12:
> > My "Importing" package offers lazy imports by creating module objects
> > in sys.modules that are a subtype of ModuleType, and use a
> > __getattribute__ hook so that trying to use them fires off a reload()
> > of the module.
>
> I wonder if this wouldn't be an approach to fix the reloading problem in
> general. What if extension module loading, at least with the new scheme,
> didn't return the module object itself and put it into sys.modules but
> created a wrapper that redirects its __getattr__ and __setattr__ to the
> actual module object? That would have a tiny performance impact on
> attribute access, but I'd expect that to be negligible given that the usual
> reason for the extension module to exist is that it does non-trivial stuff
> in whatever its API provides. Reloading could then really create a
> completely new module object and replace the reference inside of the
> wrapper.
>
> That way, code that currently uses "from extmodule import xyz" would
> continue to see the original version of the module as of the time of its
> import, and code that just did "import extmodule" and then used attribute
> access at need would always see the current content of the module as it was
> last loaded. I think that, together with keeping module global state in the
> module object itself, would nicely fix both cases.
>

At first blush I like this.

-eric

p.s. Bear with me if I've missed something in the thread.  I'm slogging
through a backlog of email
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130905/20ddd112/attachment.html>

From solipsis at pitrou.net  Fri Sep  6 07:43:48 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 6 Sep 2013 07:43:48 +0200
Subject: [Python-Dev] Pre-PEP: Redesigning extension modules
References: <kv77o2$f96$1@ger.gmane.org>
 <CALeMXf5oxi0oX13zSfHwgVi1eXF_z_coY23MfBP-YOK8WVkWsg@mail.gmail.com>
 <kvctoj$qsv$1@ger.gmane.org>
 <CALFfu7Bfvrp1yT0=X08P-f+CF8aLGifZzk8-Mv3YyzL69hUHnA@mail.gmail.com>
Message-ID: <20130906074348.2779de48@fsol>

On Thu, 5 Sep 2013 23:26:31 -0600
Eric Snow <ericsnowcurrently at gmail.com> wrote:
> On Sat, Aug 24, 2013 at 7:07 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> 
> > PEP 3121 would no longer be necessary. Extension types can do all we need.
> > No more special casing of modules, that was the idea.
> >
> 
> One nice thing about PEP 3121 is the addition of md_state to module objects
> to store internal module state.  Wouldn't we be better served by improving
> the related API rather than abandoning it?

md_state isn't a PyObject and therefore its lifetime management is
quirky (as Py_buffer, same bad idea). So I'd be happy for it to
disappear from the next API.

> This, coupled with the PEP 451-compatible API and with a proxying wrapper,
> would go a long way to various "reloading" issues that extension modules
> have.

Proxying wrapper? We shouldn't need that kind of tricks.

Regards

Antoine.



From ericsnowcurrently at gmail.com  Fri Sep  6 07:47:25 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 5 Sep 2013 23:47:25 -0600
Subject: [Python-Dev] Pre-PEP: Redesigning extension modules
In-Reply-To: <CALeMXf5oxi0oX13zSfHwgVi1eXF_z_coY23MfBP-YOK8WVkWsg@mail.gmail.com>
References: <kv77o2$f96$1@ger.gmane.org>
 <CALeMXf5oxi0oX13zSfHwgVi1eXF_z_coY23MfBP-YOK8WVkWsg@mail.gmail.com>
Message-ID: <CALFfu7CTDWvTEmDRa5zvZFTOywjONaBjBNgkqCLBv3kr83qBVw@mail.gmail.com>

On Sat, Aug 24, 2013 at 10:12 PM, PJ Eby <pje at telecommunity.com> wrote:

> I haven't had a chance to address this on the import-sig discussion
> yet about ModuleSpec, but I would like to just mention that one
> property of the existing module system that I'm not sure either this
> proposal or the ModuleSpec proposal preserves is that it's possible to
> implement lazy importing of modules using standard reload() semantics.
>
> My "Importing" package offers lazy imports by creating module objects
> in sys.modules that are a subtype of ModuleType, and use a
> __getattribute__ hook so that trying to use them fires off a reload()
> of the module.  Because the dummy module doesn't have __file__ or
> anything else initialized, the import system searches for the module
> and then loads it, reusing the existing module object, even though
> it's actually only executing the module code for the first time.
>
> That the existing object be reused is important, because once the
> dummy is in sys.modules, it can also be imported by other modules, so
> references to it can abound everywhere, and we wish only for it to be
> loaded lazily, without needing to trace down and replace all instances
> of it.  This also preserves other invariants of the module system.
>
> Anyway, the reason I was asking why reloading is being handled as a
> special case in the ModuleSpec proposal -- and the reason I'm curious
> about certain provisions of this proposal -- is that making the
> assumption you can only reload something with the same
> spec/location/etc. it was originally loaded with, and/or that if you
> are reloading a module then you previously had a chance to do things
> to it, doesn't jibe with the way things work currently.


> That is to say, in the pure PEP 302 world, there is no special status
> for "reload" that is different from "load" -- the *only* thing that's
> different is that there is already a module object to use, and there
> is *no guarantee that it's a module object that was initialized by the
> loader now being invoked*.
>

In Python 3.3 (#13959) import.reload()  was updated to reuse a module's
__loader__, which  must now be set.  If __loader__ is not set, you get an
AttributeError.  If that's a problem we can create a tracker issue and
discuss there.

In Python 3.4 imp.reload() is just an alias to importlib.reload(), but it
works basically the same.

With ModuleSpec things won't work that differently.  If you reload such a
module as you described, it will look for __spec__ and call it's reload()
method.  If __spec__ is not set, you get an AttributeError.  It wouldn't be
that hard to build a spec from the module if need be and then use that.

-eric


> AFAICT both this proposal and the ModuleSpec one are making an invalid
> assumption per PEP 302, and aren't explicitly proposing to change the
> status quo: they just assume things that aren't actually assured by
> the prior specs or implementations.
>
> So, for example, this extension module proposal needs to cover what
> happens if an extension module is reloaded and the module object is
> not of the type or instance it's expecting.  Must it do its own
> checking?  Error handling?  Will some other portion of the import
> system be expected to handle it?
>
> For that matter, what happens (in either proposal) if you reload() a
> module which only has a __name__, and no other attributes?  I haven't
> tested with importlib, but with earlier Pythons this results in a
> standard module search being done by reload().  But the ModuleSpec
> proposal and this one seem to assume that a reload()-ed module must
> already be associated with a loader, location, and/or spec.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/ericsnowcurrently%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130905/3cad8555/attachment.html>

From chris at simplistix.co.uk  Fri Sep  6 07:50:07 2013
From: chris at simplistix.co.uk (Chris Withers)
Date: Fri, 06 Sep 2013 06:50:07 +0100
Subject: [Python-Dev] windows file closing race condition?
Message-ID: <52296D0F.2030400@simplistix.co.uk>

Hi All,

Continuous testing is a wonderful thing when it comes to finding weird 
edge case problems, like this one:

http://jenkins.simplistix.co.uk/job/testfixtures-tox/COMPONENTS=zc,PYTHON=3.3,label=windows/149/testReport/junit/testfixtures.tests.test_tempdirectory/TempDirectoryTests/test_check_all_tuple/

   File 
"C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\site-packages\testfixtures\tempdirectory.py", 
line 323, in __exit__
     self.cleanup()
   File 
"C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\site-packages\testfixtures\tempdirectory.py", 
line 78, in cleanup
     rmtree(self.path)
   File 
"C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\shutil.py", 
line 460, in rmtree
     return _rmtree_unsafe(path, onerror)
   File 
"C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\shutil.py", 
line 362, in _rmtree_unsafe
     _rmtree_unsafe(fullname, onerror)
   File 
"C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\shutil.py", 
line 371, in _rmtree_unsafe
     onerror(os.rmdir, path, sys.exc_info())
   File 
"C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\shutil.py", 
line 369, in _rmtree_unsafe
     os.rmdir(path)
OSError: [WinError 145] The directory is not empty: 
'c:\\users\\jenkins\\appdata\\local\\temp\\tmpkeg4d7\\a'

I'm 99% certain my code is correct here, the only place I open files for 
writing in that directory is here:

https://github.com/Simplistix/testfixtures/blob/master/testfixtures/tempdirectory.py#L275

So, from my perspective, I'm either looking at a bug in shutil.rmtree 
(which would be trying to delete a directory before deleting its content 
or failing to delete a file but ignoring an error) or the file object, 
when being used as a context manager, going through __exit__ without 
closing the file and releasing the handle.

This happens very infrequently, the OS is Windows 7 and the filesystem 
is NTFS, if that helps...

Any ideas?

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk

From ericsnowcurrently at gmail.com  Fri Sep  6 07:52:40 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 5 Sep 2013 23:52:40 -0600
Subject: [Python-Dev] Pre-PEP: Redesigning extension modules
In-Reply-To: <kvtfek$23p$1@ger.gmane.org>
References: <kv77o2$f96$1@ger.gmane.org> <20130823111822.49cba700@pitrou.net>
 <CADiSq7dhzpTWMw7gcBRrzZot7jaS=Aj=aukq0+jh_cK2+F_kGQ@mail.gmail.com>
 <CADiSq7dBruZiWhObnjKbcneji9+ziXmyRpg1+72a8YeMWbyO3A@mail.gmail.com>
 <kvabt8$jfc$1@ger.gmane.org>
 <CADiSq7c6OjFa8zo7sMB=k2zLSmaf1w-tAwhYKSQ26T3caLyj6A@mail.gmail.com>
 <kvaka3$sbu$1@ger.gmane.org>
 <CADiSq7dzQqyH9i6g9QYWp6_ZHHUD4+4npYo6of6mCNahQ+w8kw@mail.gmail.com>
 <kvcr9f$4gs$1@ger.gmane.org>
 <CADiSq7cfNH1M29W19BW88-BBQaWEpaMkE=s09wNyT5VGOrPB+g@mail.gmail.com>
 <kvtfek$23p$1@ger.gmane.org>
Message-ID: <CALFfu7CKZU1g8mrhZcpT-0a3coqghT23BpynmEM=+eE4m1ynfQ@mail.gmail.com>

On Sat, Aug 31, 2013 at 1:16 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> Nick Coghlan, 31.08.2013 18:49:
> > This is actually my primary motivation for trying to improve the
> > "can this be reloaded or not?" aspects of the loader API in PEP 451.
>
> I assume you mean that the extension module would be able to clearly signal
> that it can't be reloaded, right? I agree that that's helpful. If you're
> wrapping a C library, then the way that library is implemented might simply
> force you to prevent any attempts at reloading the wrapper module. But if
> reloading is possible at all, it would be even more helpful if we could
> make it really easy to properly support it.
>

 When loader.exec_module() gets called, it should raise ImportError if the
module does not support reloading.

-eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130905/2522477b/attachment.html>

From ericsnowcurrently at gmail.com  Fri Sep  6 07:57:56 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 5 Sep 2013 23:57:56 -0600
Subject: [Python-Dev] Pre-PEP: Redesigning extension modules
In-Reply-To: <20130902101642.04b69588@pitrou.net>
References: <kv77o2$f96$1@ger.gmane.org>
 <CALeMXf5oxi0oX13zSfHwgVi1eXF_z_coY23MfBP-YOK8WVkWsg@mail.gmail.com>
 <kvctoj$qsv$1@ger.gmane.org> <kvstdj$jo0$1@ger.gmane.org>
 <CADiSq7c5xtEcPBXGPWqQosfgGcb2n+2AnHiAWFoATeYfQTUcEQ@mail.gmail.com>
 <20130902101642.04b69588@pitrou.net>
Message-ID: <CALFfu7BZzRFWb0=_xfZHM6O=VnV5MhiNuej6f_sKNebBZ7KzHA@mail.gmail.com>

On Mon, Sep 2, 2013 at 2:16 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> I think the biggest challenge here is to propose an API that's simple
> and easy to use (i.e. that doesn't make extension module writing more
> complicated than it currently is).
>

+1


>
> The basic concept of putting custom module objects in sys.modules is
> sound, IMHO.
>
> As for "extension module as a wrapper", though, it shounds like the
> kind of complication I would personally prefer to stay away from. Also,
> it would make extension modules less like Python modules, rather than
> more.
>

It all depends on how useful it would be to be able to safely reload
extension modules from their files.

-eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130905/49715ee2/attachment.html>

From solipsis at pitrou.net  Fri Sep  6 08:10:20 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 6 Sep 2013 08:10:20 +0200
Subject: [Python-Dev] windows file closing race condition?
References: <52296D0F.2030400@simplistix.co.uk>
Message-ID: <20130906081020.76e37fba@fsol>

On Fri, 06 Sep 2013 06:50:07 +0100
Chris Withers <chris at simplistix.co.uk> wrote:
> 
> So, from my perspective, I'm either looking at a bug in shutil.rmtree 
> (which would be trying to delete a directory before deleting its content 
> or failing to delete a file but ignoring an error) or the file object, 
> when being used as a context manager, going through __exit__ without 
> closing the file and releasing the handle.

It seems someone forgot to remove the following snippet from
FileIO.__exit__:

    if random.random() > 0.0001:
        # Sometimes we leave the fd open, to annoy Chris
        os.close(self.fd)

> This happens very infrequently, the OS is Windows 7 and the filesystem 
> is NTFS, if that helps...

It should help indeed:
http://blogs.msdn.com/b/oldnewthing/archive/2012/09/07/10347136.aspx

:-)

Regards

Antoine.



From stefan_ml at behnel.de  Fri Sep  6 08:11:06 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 06 Sep 2013 08:11:06 +0200
Subject: [Python-Dev] Pre-PEP: Redesigning extension modules
In-Reply-To: <20130906074348.2779de48@fsol>
References: <kv77o2$f96$1@ger.gmane.org>
 <CALeMXf5oxi0oX13zSfHwgVi1eXF_z_coY23MfBP-YOK8WVkWsg@mail.gmail.com>
 <kvctoj$qsv$1@ger.gmane.org>
 <CALFfu7Bfvrp1yT0=X08P-f+CF8aLGifZzk8-Mv3YyzL69hUHnA@mail.gmail.com>
 <20130906074348.2779de48@fsol>
Message-ID: <l0brli$73j$1@ger.gmane.org>

Antoine Pitrou, 06.09.2013 07:43:
> Proxying wrapper? We shouldn't need that kind of tricks.

The advantage is that it's controlled by the loader, and transparent to the
module itself.

Stefan



From ncoghlan at gmail.com  Fri Sep  6 09:14:18 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 6 Sep 2013 17:14:18 +1000
Subject: [Python-Dev] windows file closing race condition?
In-Reply-To: <52296D0F.2030400@simplistix.co.uk>
References: <52296D0F.2030400@simplistix.co.uk>
Message-ID: <CADiSq7dYFHRSGgLMROXc+XNfTfbsstrkbwdfvzJGOXqdR7MwTA@mail.gmail.com>

On 6 September 2013 15:50, Chris Withers <chris at simplistix.co.uk> wrote:
> Hi All,
>
> Continuous testing is a wonderful thing when it comes to finding weird edge
> case problems, like this one:
>
> http://jenkins.simplistix.co.uk/job/testfixtures-tox/COMPONENTS=zc,PYTHON=3.3,label=windows/149/testReport/junit/testfixtures.tests.test_tempdirectory/TempDirectoryTests/test_check_all_tuple/
>
>   File
> "C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\site-packages\testfixtures\tempdirectory.py",
> line 323, in __exit__
>     self.cleanup()
>   File
> "C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\site-packages\testfixtures\tempdirectory.py",
> line 78, in cleanup
>     rmtree(self.path)
>   File
> "C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\shutil.py",
> line 460, in rmtree
>     return _rmtree_unsafe(path, onerror)
>   File
> "C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\shutil.py",
> line 362, in _rmtree_unsafe
>     _rmtree_unsafe(fullname, onerror)
>   File
> "C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\shutil.py",
> line 371, in _rmtree_unsafe
>     onerror(os.rmdir, path, sys.exc_info())
>   File
> "C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\shutil.py",
> line 369, in _rmtree_unsafe
>     os.rmdir(path)
> OSError: [WinError 145] The directory is not empty:
> 'c:\\users\\jenkins\\appdata\\local\\temp\\tmpkeg4d7\\a'
>
> I'm 99% certain my code is correct here, the only place I open files for
> writing in that directory is here:
>
> https://github.com/Simplistix/testfixtures/blob/master/testfixtures/tempdirectory.py#L275
>
> So, from my perspective, I'm either looking at a bug in shutil.rmtree (which
> would be trying to delete a directory before deleting its content or failing
> to delete a file but ignoring an error) or the file object, when being used
> as a context manager, going through __exit__ without closing the file and
> releasing the handle.
>
> This happens very infrequently, the OS is Windows 7 and the filesystem is
> NTFS, if that helps...
>
> Any ideas?

This feels a lot like an issue we were seeing on the Windows
buildbots, which we ended up working around in the test support
library: http://bugs.python.org/issue15496

That would be some awfully ugly code to upgrade from "hack in the test
support library" to "this is just how Python unlinks files on
Windows", though :P

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From chris at simplistix.co.uk  Fri Sep  6 09:48:50 2013
From: chris at simplistix.co.uk (Chris Withers)
Date: Fri, 06 Sep 2013 08:48:50 +0100
Subject: [Python-Dev] windows file closing race condition?
In-Reply-To: <20130906081020.76e37fba@fsol>
References: <52296D0F.2030400@simplistix.co.uk> <20130906081020.76e37fba@fsol>
Message-ID: <522988E2.20005@simplistix.co.uk>

On 06/09/2013 07:10, Antoine Pitrou wrote:
>> This happens very infrequently, the OS is Windows 7 and the filesystem
>> is NTFS, if that helps...
>
> It should help indeed:
> http://blogs.msdn.com/b/oldnewthing/archive/2012/09/07/10347136.aspx

The box in questions runs no AV software or indexing services...

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From stephen at xemacs.org  Fri Sep  6 09:28:21 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 06 Sep 2013 16:28:21 +0900
Subject: [Python-Dev] windows file closing race condition?
In-Reply-To: <20130906081020.76e37fba@fsol>
References: <52296D0F.2030400@simplistix.co.uk> <20130906081020.76e37fba@fsol>
Message-ID: <87a9jq9zoq.fsf@uwakimon.sk.tsukuba.ac.jp>

Antoine Pitrou writes:

 > http://blogs.msdn.com/b/oldnewthing/archive/2012/09/07/10347136.aspx

That was worth it just for the comment from Australia!

From mail at timgolden.me.uk  Fri Sep  6 09:58:06 2013
From: mail at timgolden.me.uk (Tim Golden)
Date: Fri, 06 Sep 2013 08:58:06 +0100
Subject: [Python-Dev] windows file closing race condition?
In-Reply-To: <CADiSq7dYFHRSGgLMROXc+XNfTfbsstrkbwdfvzJGOXqdR7MwTA@mail.gmail.com>
References: <52296D0F.2030400@simplistix.co.uk>
 <CADiSq7dYFHRSGgLMROXc+XNfTfbsstrkbwdfvzJGOXqdR7MwTA@mail.gmail.com>
Message-ID: <52298B0E.7090508@timgolden.me.uk>

On 06/09/2013 08:14, Nick Coghlan wrote:
> On 6 September 2013 15:50, Chris Withers <chris at simplistix.co.uk> wrote:
>> Continuous testing is a wonderful thing when it comes to finding weird edge
>> case problems, like this one:

[... snip ...]
>>     os.rmdir(path)
>> OSError: [WinError 145] The directory is not empty:
>> 'c:\\users\\jenkins\\appdata\\local\\temp\\tmpkeg4d7\\a'

> This feels a lot like an issue we were seeing on the Windows
> buildbots, which we ended up working around in the test support
> library: http://bugs.python.org/issue15496
> 
> That would be some awfully ugly code to upgrade from "hack in the test
> support library" to "this is just how Python unlinks files on
> Windows", though :P

I think that any kind of delay-retry loop falls squarely within the
remit of the calling application, not core Python. This isn't a problem
of Python's making: IIUC, you would see the same effect if you used any
other
language or simply deleted the folder from within Explorer. (I don't
know whether Explorer itself does anything canny under the covers to retry).

Obviously our test suite *is* a calling application, and so it makes
perfect sense to put some workaround in place.

The trouble with this class of problem, where a share-delete handle
allows operations to succeed and to fail later which would normally fail
early, is that the bad effect is at one remove from its cause. Here, by
the time the rmtree has reached the point of removing a parent
directory, it's long-since left behind the file which has a still-open
handle: the DeleteFile succeeded and the code moved on. You can't even
tell which file it was.

A related problem arises where the DeleteFile succeeds and an error
occurs when a subsequent CreateFile fails for the same filepath, again
because a share-delete handle is still open for a file at that path.
This is another one which hits our test suite because of an overuse of
one temp filename.

What should Python do? With some effort it could look for open file
handles against the file it's trying to delete, but what then? Wait
until they're all closed? That could leave it hanging. And even with a
timeout it would introduce a delay which might be unnecessary. A lot of
the time, no harm will come of the file existing a few seconds after the
DeleteFile has succeeded.

In short, I don't believe there's any mileage in introducing extra code
into Python's os or io modules. It falls on the shoulders of the calling
code to implement retry loops or whatever logic as needed.

TJG

From chris at simplistix.co.uk  Fri Sep  6 09:59:19 2013
From: chris at simplistix.co.uk (Chris Withers)
Date: Fri, 06 Sep 2013 08:59:19 +0100
Subject: [Python-Dev] windows file closing race condition?
In-Reply-To: <CADiSq7dYFHRSGgLMROXc+XNfTfbsstrkbwdfvzJGOXqdR7MwTA@mail.gmail.com>
References: <52296D0F.2030400@simplistix.co.uk>
 <CADiSq7dYFHRSGgLMROXc+XNfTfbsstrkbwdfvzJGOXqdR7MwTA@mail.gmail.com>
Message-ID: <52298B57.2020108@simplistix.co.uk>

On 06/09/2013 08:14, Nick Coghlan wrote:
> This feels a lot like an issue we were seeing on the Windows
> buildbots, which we ended up working around in the test support
> library: http://bugs.python.org/issue15496

Wow :'(

> That would be some awfully ugly code to upgrade from "hack in the test
> support library" to "this is just how Python unlinks files on
> Windows", though :P

Indeed...

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From solipsis at pitrou.net  Fri Sep  6 12:14:36 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 6 Sep 2013 12:14:36 +0200
Subject: [Python-Dev] windows file closing race condition?
References: <52296D0F.2030400@simplistix.co.uk>
 <CADiSq7dYFHRSGgLMROXc+XNfTfbsstrkbwdfvzJGOXqdR7MwTA@mail.gmail.com>
 <52298B0E.7090508@timgolden.me.uk>
Message-ID: <20130906121436.7db391f2@pitrou.net>

Le Fri, 06 Sep 2013 08:58:06 +0100,
Tim Golden <mail at timgolden.me.uk> a ?crit :
> 
> What should Python do?

Maybe using FILE_SHARE_DELETE could help?
http://bugs.python.org/issue15244

Regards

Antoine.



From mail at timgolden.me.uk  Fri Sep  6 12:23:39 2013
From: mail at timgolden.me.uk (Tim Golden)
Date: Fri, 06 Sep 2013 11:23:39 +0100
Subject: [Python-Dev] windows file closing race condition?
In-Reply-To: <20130906121436.7db391f2@pitrou.net>
References: <52296D0F.2030400@simplistix.co.uk>
 <CADiSq7dYFHRSGgLMROXc+XNfTfbsstrkbwdfvzJGOXqdR7MwTA@mail.gmail.com>
 <52298B0E.7090508@timgolden.me.uk> <20130906121436.7db391f2@pitrou.net>
Message-ID: <5229AD2B.5030206@timgolden.me.uk>

On 06/09/2013 11:14, Antoine Pitrou wrote:
> Le Fri, 06 Sep 2013 08:58:06 +0100,
> Tim Golden <mail at timgolden.me.uk> a ?crit :
>>
>> What should Python do?
> 
> Maybe using FILE_SHARE_DELETE could help?
> http://bugs.python.org/issue15244

I don't think so. It's the use of FILE_SHARE_DELETE (by other programs,
eg Virus Checkers) that typically causes the problem. IOW, the sequence is:

* [Some Other Prog] takes FILE_SHARE_DELETE handle, allowing other
programs to delete the file even while this handle is still open

* Python calls DeleteFile -- succeeds if the only open handles are
FILE_SHARE_DELETE; carries on

* File has apparently disappeared but still has open handles

* Any attempt to create a file of the same name or to delete a
containing directory fail because the file is still open, even though
it's successfully been deleted.

* (Some time later) [Some Other Prog] closes its handle and the file is
now completely gone


Unless I'm missing something, there's nothing Python can do to help here
apart from the sort of delay-retry dance which test.support uses and
which I'm advocating against as a core feature.


TJG

From shibturn at gmail.com  Fri Sep  6 13:50:34 2013
From: shibturn at gmail.com (Richard Oudkerk)
Date: Fri, 06 Sep 2013 12:50:34 +0100
Subject: [Python-Dev] windows file closing race condition?
In-Reply-To: <5229AD2B.5030206@timgolden.me.uk>
References: <52296D0F.2030400@simplistix.co.uk>
 <CADiSq7dYFHRSGgLMROXc+XNfTfbsstrkbwdfvzJGOXqdR7MwTA@mail.gmail.com>
 <52298B0E.7090508@timgolden.me.uk> <20130906121436.7db391f2@pitrou.net>
 <5229AD2B.5030206@timgolden.me.uk>
Message-ID: <l0cfia$dl$1@ger.gmane.org>

On 06/09/2013 11:23am, Tim Golden wrote:
> On 06/09/2013 11:14, Antoine Pitrou wrote:
>> Le Fri, 06 Sep 2013 08:58:06 +0100,
>> Tim Golden <mail at timgolden.me.uk> a ?crit :
>>>
>>> What should Python do?
>>
>> Maybe using FILE_SHARE_DELETE could help?
>> http://bugs.python.org/issue15244
>
> I don't think so. It's the use of FILE_SHARE_DELETE (by other programs,
> eg Virus Checkers) that typically causes the problem. IOW, the sequence is:
>
> * [Some Other Prog] takes FILE_SHARE_DELETE handle, allowing other
> programs to delete the file even while this handle is still open
>
> * Python calls DeleteFile -- succeeds if the only open handles are
> FILE_SHARE_DELETE; carries on
>
> * File has apparently disappeared but still has open handles
>
> * Any attempt to create a file of the same name or to delete a
> containing directory fail because the file is still open, even though
> it's successfully been deleted.
>
> * (Some time later) [Some Other Prog] closes its handle and the file is
> now completely gone
>
>
> Unless I'm missing something, there's nothing Python can do to help here
> apart from the sort of delay-retry dance which test.support uses and
> which I'm advocating against as a core feature.

Instead of deleting, the file could be moved to a temporary name in the 
root directory (or some other permanent directory on the same drive) and 
then deleted.  That would allow the directory to be closed even if a 
FILE_SHARE_DELETE handle is still open for the file.

-- 
Richard


From mail at timgolden.me.uk  Fri Sep  6 14:21:35 2013
From: mail at timgolden.me.uk (Tim Golden)
Date: Fri, 06 Sep 2013 13:21:35 +0100
Subject: [Python-Dev] windows file closing race condition?
In-Reply-To: <l0cfia$dl$1@ger.gmane.org>
References: <52296D0F.2030400@simplistix.co.uk>
 <CADiSq7dYFHRSGgLMROXc+XNfTfbsstrkbwdfvzJGOXqdR7MwTA@mail.gmail.com>
 <52298B0E.7090508@timgolden.me.uk> <20130906121436.7db391f2@pitrou.net>
 <5229AD2B.5030206@timgolden.me.uk> <l0cfia$dl$1@ger.gmane.org>
Message-ID: <5229C8CF.2050308@timgolden.me.uk>

On 06/09/2013 12:50, Richard Oudkerk wrote:
> On 06/09/2013 11:23am, Tim Golden wrote:
>> On 06/09/2013 11:14, Antoine Pitrou wrote:
>>> Le Fri, 06 Sep 2013 08:58:06 +0100,
>>> Tim Golden <mail at timgolden.me.uk> a ?crit :
>>>>
>>>> What should Python do?
>>>
>>> Maybe using FILE_SHARE_DELETE could help?
>>> http://bugs.python.org/issue15244
>>
>> I don't think so. It's the use of FILE_SHARE_DELETE (by other programs,
>> eg Virus Checkers) that typically causes the problem. IOW, the
>> sequence is:
>>
>> * [Some Other Prog] takes FILE_SHARE_DELETE handle, allowing other
>> programs to delete the file even while this handle is still open
>>
>> * Python calls DeleteFile -- succeeds if the only open handles are
>> FILE_SHARE_DELETE; carries on
>>
>> * File has apparently disappeared but still has open handles
>>
>> * Any attempt to create a file of the same name or to delete a
>> containing directory fail because the file is still open, even though
>> it's successfully been deleted.
>>
>> * (Some time later) [Some Other Prog] closes its handle and the file is
>> now completely gone
>>
>>
>> Unless I'm missing something, there's nothing Python can do to help here
>> apart from the sort of delay-retry dance which test.support uses and
>> which I'm advocating against as a core feature.
> 
> Instead of deleting, the file could be moved to a temporary name in the
> root directory (or some other permanent directory on the same drive) and
> then deleted.  That would allow the directory to be closed even if a
> FILE_SHARE_DELETE handle is still open for the file.
> 

True, but then you're into determining a temporary name somewhere on the
same volume if possible and avoiding collisions etc. Again, I don't
think this is something we need to be doing by default in core Python.

TJG

From p.f.moore at gmail.com  Fri Sep  6 14:55:17 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 6 Sep 2013 13:55:17 +0100
Subject: [Python-Dev] windows file closing race condition?
In-Reply-To: <5229C8CF.2050308@timgolden.me.uk>
References: <52296D0F.2030400@simplistix.co.uk>
 <CADiSq7dYFHRSGgLMROXc+XNfTfbsstrkbwdfvzJGOXqdR7MwTA@mail.gmail.com>
 <52298B0E.7090508@timgolden.me.uk>
 <20130906121436.7db391f2@pitrou.net>
 <5229AD2B.5030206@timgolden.me.uk> <l0cfia$dl$1@ger.gmane.org>
 <5229C8CF.2050308@timgolden.me.uk>
Message-ID: <CACac1F8QJeLf=KXo-Eq+LrKBJo_sNWKdpZpjEgsPtdxKU+buig@mail.gmail.com>

On 6 September 2013 13:21, Tim Golden <mail at timgolden.me.uk> wrote:
> True, but then you're into determining a temporary name somewhere on the
> same volume if possible and avoiding collisions etc. Again, I don't
> think this is something we need to be doing by default in core Python.

Agreed.

There simply is no solution that works in all cases. If you rename the
file to anywhere but the same directory, you potentially have
permission issues. And if you rename to the same directory, you still
can't delete the directory. It's a shame, because a helper to do this
would be useful for Unix users wanting to ensure that their code works
properly on Windows, but there really isn't an answer other than
knowing how things work (and crafting an appropriate solution for your
application), here...

Paul

From shibturn at gmail.com  Fri Sep  6 15:16:25 2013
From: shibturn at gmail.com (Richard Oudkerk)
Date: Fri, 06 Sep 2013 14:16:25 +0100
Subject: [Python-Dev] windows file closing race condition?
In-Reply-To: <CACac1F8QJeLf=KXo-Eq+LrKBJo_sNWKdpZpjEgsPtdxKU+buig@mail.gmail.com>
References: <52296D0F.2030400@simplistix.co.uk>
 <CADiSq7dYFHRSGgLMROXc+XNfTfbsstrkbwdfvzJGOXqdR7MwTA@mail.gmail.com>
 <52298B0E.7090508@timgolden.me.uk> <20130906121436.7db391f2@pitrou.net>
 <5229AD2B.5030206@timgolden.me.uk> <l0cfia$dl$1@ger.gmane.org>
 <5229C8CF.2050308@timgolden.me.uk>
 <CACac1F8QJeLf=KXo-Eq+LrKBJo_sNWKdpZpjEgsPtdxKU+buig@mail.gmail.com>
Message-ID: <l0ckja$qv2$1@ger.gmane.org>

On 06/09/2013 1:55pm, Paul Moore wrote:
> ... If you rename the
> file to anywhere but the same directory, you potentially have
> permission issues.

Using the root directory avoids permission issues -- users always have 
write access there.

-- 
Richard


From rdmurray at bitdance.com  Fri Sep  6 15:26:52 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Fri, 06 Sep 2013 09:26:52 -0400
Subject: [Python-Dev] windows file closing race condition?
In-Reply-To: <5229C8CF.2050308@timgolden.me.uk>
References: <52296D0F.2030400@simplistix.co.uk>
 <CADiSq7dYFHRSGgLMROXc+XNfTfbsstrkbwdfvzJGOXqdR7MwTA@mail.gmail.com>
 <52298B0E.7090508@timgolden.me.uk> <20130906121436.7db391f2@pitrou.net>
 <5229AD2B.5030206@timgolden.me.uk> <l0cfia$dl$1@ger.gmane.org>
 <5229C8CF.2050308@timgolden.me.uk>
Message-ID: <20130906132652.D566A2507F5@webabinitio.net>

On Fri, 06 Sep 2013 13:21:35 +0100, Tim Golden <mail at timgolden.me.uk> wrote:
> On 06/09/2013 12:50, Richard Oudkerk wrote:
> > On 06/09/2013 11:23am, Tim Golden wrote:
> >> On 06/09/2013 11:14, Antoine Pitrou wrote:
> >>> Le Fri, 06 Sep 2013 08:58:06 +0100,
> >>> Tim Golden <mail at timgolden.me.uk> a ??crit :
> >>>>
> >>>> What should Python do?
> >>>
> >>> Maybe using FILE_SHARE_DELETE could help?
> >>> http://bugs.python.org/issue15244
> >>
> >> I don't think so. It's the use of FILE_SHARE_DELETE (by other programs,
> >> eg Virus Checkers) that typically causes the problem. IOW, the
> >> sequence is:
> >>
> >> * [Some Other Prog] takes FILE_SHARE_DELETE handle, allowing other
> >> programs to delete the file even while this handle is still open
> >>
> >> * Python calls DeleteFile -- succeeds if the only open handles are
> >> FILE_SHARE_DELETE; carries on
> >>
> >> * File has apparently disappeared but still has open handles
> >>
> >> * Any attempt to create a file of the same name or to delete a
> >> containing directory fail because the file is still open, even though
> >> it's successfully been deleted.
> >>
> >> * (Some time later) [Some Other Prog] closes its handle and the file is
> >> now completely gone
> >>
> >>
> >> Unless I'm missing something, there's nothing Python can do to help here
> >> apart from the sort of delay-retry dance which test.support uses and
> >> which I'm advocating against as a core feature.
> > 
> > Instead of deleting, the file could be moved to a temporary name in the
> > root directory (or some other permanent directory on the same drive) and
> > then deleted.  That would allow the directory to be closed even if a
> > FILE_SHARE_DELETE handle is still open for the file.
> > 
> 
> True, but then you're into determining a temporary name somewhere on the
> same volume if possible and avoiding collisions etc. Again, I don't
> think this is something we need to be doing by default in core Python.

What about moving the test.support delete helper to somewhere in
unittest, since this will come up in pretty much every test suite that
runs on Windows?

--David

From ethan at stoneleaf.us  Fri Sep  6 15:32:46 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 06 Sep 2013 06:32:46 -0700
Subject: [Python-Dev] inspect and metaclasses
Message-ID: <5229D97E.9080103@stoneleaf.us>

Part of the fix for issue #18693  is to fix inspect to look in the metaclass for class attributes 
(http://bugs.python.org/issue18929).

In inspect.py in function get_mro() we can either add the metaclass unconditionally, or only if it is not 'type'.

If we add unconditionally, then help() adds the following:

       class A(builtins.object)
        |  Hello and goodbye
+      |
+      |  Method resolution order:
+      |      A
+      |      builtins.object
+      |      builtins.type
        |
        |  Methods defined here:

Do we want that, or should we just add the metaclass if it is not 'type'?

--
~Ethan~

From p.f.moore at gmail.com  Fri Sep  6 16:01:44 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 6 Sep 2013 15:01:44 +0100
Subject: [Python-Dev] windows file closing race condition?
In-Reply-To: <l0ckja$qv2$1@ger.gmane.org>
References: <52296D0F.2030400@simplistix.co.uk>
 <CADiSq7dYFHRSGgLMROXc+XNfTfbsstrkbwdfvzJGOXqdR7MwTA@mail.gmail.com>
 <52298B0E.7090508@timgolden.me.uk>
 <20130906121436.7db391f2@pitrou.net>
 <5229AD2B.5030206@timgolden.me.uk> <l0cfia$dl$1@ger.gmane.org>
 <5229C8CF.2050308@timgolden.me.uk>
 <CACac1F8QJeLf=KXo-Eq+LrKBJo_sNWKdpZpjEgsPtdxKU+buig@mail.gmail.com>
 <l0ckja$qv2$1@ger.gmane.org>
Message-ID: <CACac1F_VtkEYoqWy=vp2vMwfSoyQ3W5KsLcnwYbnx82jzwW7tw@mail.gmail.com>

On 6 September 2013 14:16, Richard Oudkerk <shibturn at gmail.com> wrote:
> On 06/09/2013 1:55pm, Paul Moore wrote:
>>
>> ... If you rename the
>>
>> file to anywhere but the same directory, you potentially have
>> permission issues.
>
>
> Using the root directory avoids permission issues -- users always have write
> access there.

I don't believe that's true. IIRC, some Windows versions protect the
root of the system drive from non-admin users. And I *know* I have had
problems with NTFS-formatted removable drives plugged into systems
they were not originally formatted on, where the owner of the drive no
longer exists. They are rare corner cases, for sure, but they do
happen and frankly are probably harder to explain when they come up
than saying "your virus checker is interfering". Everyone expects
their virus checker to screw up their system (cure worse than disease
syndrome :-))

Paul

From eliben at gmail.com  Fri Sep  6 16:14:11 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Fri, 6 Sep 2013 07:14:11 -0700
Subject: [Python-Dev] SEEK_* constants in io and os
In-Reply-To: <CAF-Rda_msMu--d-8Q8kPy8ReaJdNavKQ8keLrVQPOjMxpA41Yg@mail.gmail.com>
References: <CAF-Rda_2j=HTDJ5nEdKWxX1mCmbHLgUNe_zyDYXzGHE6-LUBmg@mail.gmail.com>
 <20130902102437.197c1708@pitrou.net>
 <CAF-Rda-_Z7QTQNf+jonLC1FqWj4pdLVk2h5r5ORTPaiUCwGauA@mail.gmail.com>
 <20130902154522.495b1697@pitrou.net> <20130902155109.7da869bd@pitrou.net>
 <CAF-Rda86kyZg+ZuJcw4jff8w6LgCYo__N=VDyR_GQqdEcQ6ePg@mail.gmail.com>
 <CAMpsgwYoR9p+F5Q9rXw-UbLKV2fFyEsZARR=VKXvjSLwGv8i6A@mail.gmail.com>
 <CAF-Rda_msMu--d-8Q8kPy8ReaJdNavKQ8keLrVQPOjMxpA41Yg@mail.gmail.com>
Message-ID: <CAF-Rda_NWd3FxvZRYnQJ7uoF1N3WK8G-_ZfKuQKDg+X=z=+veg@mail.gmail.com>

> On Mon, Sep 2, 2013 at 8:48 AM, Victor Stinner <victor.stinner at gmail.com>wrote:
>
>> 2013/9/2 Eli Bendersky <eliben at gmail.com>:
>> > Yes, now I see a 500 usec difference timed within the Python script.
>> When
>> > timing the whole execution of Python: (...)
>>
>> Can you please provide the list of imported modules by:
>> python -c 'import sys; print(sys.modules)'
>>
>> For python with default options and for python with -S (no site
>> module) options? And also with your patch?
>>
>> Python should be fast to write "hello world" (or "python -c pass),
>> it's a dummy but common benchmark (to compare Python to other VM /
>> other programming languages).
>>
>
> The sorted list for both default and -S (they're identical) is
> http://pastebin.com/4vzSMCu7 - there are 55 entries there, including
> things like itertools, heapq, functools and collections. With my patch,
> enum is also added (which makes sense since os is in the list). So the
> 0.5ms increase for the 34ms runtime kind-of makes sense.
>

This question is still kind-of open. I haven't received additional feedback
- only Antoine's and Victor's concerns wrt. runtime cost, which I believe I
addressed. Note that the runtime cost is globally one-time in the sense
that if additional modules (whether used at start-up or not) use enum, this
is a price only paid once.

So, is it worth it the extra 0.5ms of start-up time, or not?

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130906/692d8d35/attachment.html>

From ethan at stoneleaf.us  Fri Sep  6 16:23:46 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 06 Sep 2013 07:23:46 -0700
Subject: [Python-Dev] SEEK_* constants in io and os
In-Reply-To: <CAF-Rda_2j=HTDJ5nEdKWxX1mCmbHLgUNe_zyDYXzGHE6-LUBmg@mail.gmail.com>
References: <CAF-Rda_2j=HTDJ5nEdKWxX1mCmbHLgUNe_zyDYXzGHE6-LUBmg@mail.gmail.com>
Message-ID: <5229E572.5090509@stoneleaf.us>

On 09/01/2013 06:02 PM, Eli Bendersky wrote:
>
> os seems to import io in some functions; can this be done always? If yes, we can just define the constants once and
> os.SEEK_* will alias io.SEEK_*? The other way (io taking from os) is also a possibility (maybe the preferred one because
> io already refers to os.SEEK_HOLE/DATA, at least in the documentation).
>
> Any ideas and suggestions are welcome,

Ideally we should only define them once.

If these are values that could change per O/S then they should be in the os module.  Since they /can/ change per O/S 
(even if they don't currently), we should put them in os.

I'd say it's worth the extra 0.5ms startup time.

--
~Ethan~

From arigo at tunes.org  Fri Sep  6 16:47:15 2013
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 6 Sep 2013 16:47:15 +0200
Subject: [Python-Dev] inspect and metaclasses
In-Reply-To: <5229D97E.9080103@stoneleaf.us>
References: <5229D97E.9080103@stoneleaf.us>
Message-ID: <CAMSv6X1k30-PAWct_9tDTmwir4mDO0jAs-mbrvvBrJc5XE6VTQ@mail.gmail.com>

Hi Ethan,

Are you suggesting that inspect.get_mro(A) would return (A, object,
type)?  That seems very wrong to me.

If the goal is to fix `inspect.classify_class_attrs()`, then this
function only needs a specific fix, along the lines of looking in
`get_mro(A) + get_mro(type(A))`.  (A more minor issue is that the bug
report suggests `... + (type(A),)` only, but that's wrong: Python will
also look in all the base classes of type(A).)

"Fixing" inspect.get_mro() as suggested would break a lot of other usages of it.


A bient?t,

Armin.

From jcea at jcea.es  Fri Sep  6 17:08:23 2013
From: jcea at jcea.es (Jesus Cea)
Date: Fri, 06 Sep 2013 17:08:23 +0200
Subject: [Python-Dev] DTRACE support
Message-ID: <5229EFE7.8040702@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

As far as I know, Erlang, Ruby, PHP, Perl, etc., support Dtrace.
Python is embarrasingly missing from this list.

Some examples:

<http://crypt.codemancers.com/posts/2013-04-16-profile-ruby-apps-dtrace-part1/>
<http://www.phpdeveloper.org/news/18859>
<http://www.erlang.org/doc/apps/runtime_tools/DTRACE.html>

I have spend a very long time on a patch for Dtrace support in most
platforms with dtrace available. Currently working under Solaris and
derivatives, and MacOS X. Last time I checked, it would crash FreeBSD
because bugs in the dtrace port, but that was a long time ago.

I would like to push this to Python 3.4, and the window is going to be
closed soon, so I think this is the time to ask for opinions and
support here.

Does Python-Dev have any opinion or interest in this project?. Should
I push for it?

Some (not current) details: http://bugs.python.org/issue13405

DTrace is amazing: http://dtrace.org/blogs/

PS: I can't release this code as a PyPI project, because it mess with
the inner core of Python.

- -- 
Jes?s Cea Avi?n                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQCVAwUBUinv55lgi5GaxT1NAQIK7gP8DeaWxNVwNFa5PnfZe00Aay5l1rWzgj8d
CY0D3W+PAdPkBci9SYPmfv7ajXrQWo/ccANYIRaUdI/U9Zjq/od7eNemOFqyL7U6
BrQpAUMySI6tMlL+gYEfQ8O47SManvTqoyNvOFAz9mVJute8IxKsbCIK/jiRHDXz
vWyG7YrYN1A=
=4E7+
-----END PGP SIGNATURE-----

From jcea at jcea.es  Fri Sep  6 17:08:28 2013
From: jcea at jcea.es (Jesus Cea)
Date: Fri, 06 Sep 2013 17:08:28 +0200
Subject: [Python-Dev] Details of "import lock" in 3.3
Message-ID: <5229EFEC.1010503@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

With importlib and other recent changes, what are the current details
of "import lock"?. That is, the lock/locks held when Python code does
"import", specially in the case of multithreading. Is that documented
anywhere?

Thanks!

- -- 
Jes?s Cea Avi?n                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQCVAwUBUinv7Jlgi5GaxT1NAQLX7QP/SBNuy869xSKRVNlWmAiDLpZOR9IHm77K
fy+AG/aJDpk4BMcm+KNoHZyUtJJs2kndfz+OVTeU/mmlr+aXu/wA1yNB1HAjAwWO
4ZWyaN9WLrhv8xXomUlT8iqikc/9pEspgQbJAljUvMLVouixIdxga3IqtaT3FQZh
wkV9v04G4OA=
=P3PI
-----END PGP SIGNATURE-----

From ethan at stoneleaf.us  Fri Sep  6 16:51:06 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 06 Sep 2013 07:51:06 -0700
Subject: [Python-Dev] when to fix cross-version bugs?
Message-ID: <5229EBDA.1030902@stoneleaf.us>

I recently committed a fix for unicodeobject.c so that the %d, %i, and %u format specifiers always output values 
(otherwise, in subclasses, the str() was used instead).

Should this be fixed in 3.3 as well?

What guidelines determine when a bug is fixed in previous versions?

--
~Ethan~

From guido at python.org  Fri Sep  6 17:14:04 2013
From: guido at python.org (Guido van Rossum)
Date: Fri, 6 Sep 2013 08:14:04 -0700
Subject: [Python-Dev] DTRACE support
In-Reply-To: <5229EFE7.8040702@jcea.es>
References: <5229EFE7.8040702@jcea.es>
Message-ID: <CAP7+vJJsPVn_EHjfYeYZhK=nd3VfBZ3h_b6tTsShTt6gUSVNaw@mail.gmail.com>

I've heard good things about DTRACE but never used it myself.

Do I understand correctly that you have to build a separate Python
executable with it turned on?

I noticed one odd thing in the patch: apparently the dtrace module
only exists to have a flag that tells whether it is enabled. Can't
that flag be added to the sys module?

On Fri, Sep 6, 2013 at 8:08 AM, Jesus Cea <jcea at jcea.es> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> As far as I know, Erlang, Ruby, PHP, Perl, etc., support Dtrace.
> Python is embarrasingly missing from this list.
>
> Some examples:
>
> <http://crypt.codemancers.com/posts/2013-04-16-profile-ruby-apps-dtrace-part1/>
> <http://www.phpdeveloper.org/news/18859>
> <http://www.erlang.org/doc/apps/runtime_tools/DTRACE.html>
>
> I have spend a very long time on a patch for Dtrace support in most
> platforms with dtrace available. Currently working under Solaris and
> derivatives, and MacOS X. Last time I checked, it would crash FreeBSD
> because bugs in the dtrace port, but that was a long time ago.
>
> I would like to push this to Python 3.4, and the window is going to be
> closed soon, so I think this is the time to ask for opinions and
> support here.
>
> Does Python-Dev have any opinion or interest in this project?. Should
> I push for it?
>
> Some (not current) details: http://bugs.python.org/issue13405
>
> DTrace is amazing: http://dtrace.org/blogs/
>
> PS: I can't release this code as a PyPI project, because it mess with
> the inner core of Python.
>
> - --
> Jes?s Cea Avi?n                         _/_/      _/_/_/        _/_/_/
> jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
> Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
> jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
> "Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
> "My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
> "El amor es poner tu felicidad en la felicidad de otro" - Leibniz
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQCVAwUBUinv55lgi5GaxT1NAQIK7gP8DeaWxNVwNFa5PnfZe00Aay5l1rWzgj8d
> CY0D3W+PAdPkBci9SYPmfv7ajXrQWo/ccANYIRaUdI/U9Zjq/od7eNemOFqyL7U6
> BrQpAUMySI6tMlL+gYEfQ8O47SManvTqoyNvOFAz9mVJute8IxKsbCIK/jiRHDXz
> vWyG7YrYN1A=
> =4E7+
> -----END PGP SIGNATURE-----
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)

From A.J.Miller at bcs.org.uk  Fri Sep  6 11:54:45 2013
From: A.J.Miller at bcs.org.uk (Andrew Miller)
Date: Fri, 6 Sep 2013 10:54:45 +0100
Subject: [Python-Dev] unicodedata module is out of date
Message-ID: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com>

The unicodedata module only contains data up to Unicode 5.2 (October 2009),
so attempting to reference any character from a later version e.g:

unicodedata.lookup("TURKISH LIRA SIGN")

results in a KeyError.

Also, it seems to be limited to properties in the UnicodeData.txt file and
does not contain any data from the other files from the Unicode Character
Database (the perl library Unicode::UCD is far more complete).

Are there any plans to update this module to the latest Unicode version
(6.2, with 6.3 being released shortly), or is there another module that
provides more up to date information?

Thanks,

Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130906/48935204/attachment.html>

From skip at pobox.com  Fri Sep  6 17:18:49 2013
From: skip at pobox.com (Skip Montanaro)
Date: Fri, 6 Sep 2013 10:18:49 -0500
Subject: [Python-Dev] DTRACE support
In-Reply-To: <5229EFE7.8040702@jcea.es>
References: <5229EFE7.8040702@jcea.es>
Message-ID: <CANc-5UzS_jUBOmixhRjKz+ooEMFYjOEPvNkCTjzWfROCKH=nbw@mail.gmail.com>

> I have spend a very long time on a patch for Dtrace support in most
> platforms with dtrace available. Currently working under Solaris and
> derivatives, and MacOS X. Last time I checked, it would crash FreeBSD
> because bugs in the dtrace port, but that was a long time ago.

I looked at this several years ago. As I recall, the problem at the time was
that the Apple and Sun DTrace implementations were incompatible, or that
the probes they had inserted into their own /usr/bin/python instances were
incompatible. (Don't remember which off the top of my head.) Of course, the
DTrace folks at Apple and Sun weren't really interested in holding hands...

Skip

From jcea at jcea.es  Fri Sep  6 17:21:06 2013
From: jcea at jcea.es (Jesus Cea)
Date: Fri, 06 Sep 2013 17:21:06 +0200
Subject: [Python-Dev] DTRACE support
In-Reply-To: <CAP7+vJJsPVn_EHjfYeYZhK=nd3VfBZ3h_b6tTsShTt6gUSVNaw@mail.gmail.com>
References: <5229EFE7.8040702@jcea.es>
 <CAP7+vJJsPVn_EHjfYeYZhK=nd3VfBZ3h_b6tTsShTt6gUSVNaw@mail.gmail.com>
Message-ID: <5229F2E2.6090301@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/09/13 17:14, Guido van Rossum wrote:
> I've heard good things about DTRACE but never used it myself.
> 
> Do I understand correctly that you have to build a separate Python 
> executable with it turned on?

It is a patch you apply on stock Python and you get a new configure
option: "--with-dtrace". If you compile Python with that flag, you get
a python interpreter with dtrace probes on it.

> I noticed one odd thing in the patch: apparently the dtrace module 
> only exists to have a flag that tells whether it is enabled. Can't 
> that flag be added to the sys module?

My plan is to (in the future) use that module to create new probes on
the fly, like
<http://chrisa.github.io/blog/2011/12/04/libusdt-runtime-dtrace-providers/>,
and to export functions to communicate data to dtrace scripts, if running.

That is, this module is currently a shell I plan to fill.

- -- 
Jes?s Cea Avi?n                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQCVAwUBUiny4plgi5GaxT1NAQLngQP/eFoQ3rdPy12nmw8pPPO/+/RYjlhIi67+
T0n1xBJbxIvpFpIRufB+tkCfLWdb7YZKcf3Nb4V6pVsOPdbd9s6RcVWNA9Ds5mNx
ISrO644CAIqRF5XgeeS5uhsODL1DrZTEfX0QBhGsU6lcqyOzSfRX2t9hRBMh3f5y
2dPMHNznJhs=
=dV5g
-----END PGP SIGNATURE-----

From jcea at jcea.es  Fri Sep  6 17:24:08 2013
From: jcea at jcea.es (Jesus Cea)
Date: Fri, 06 Sep 2013 17:24:08 +0200
Subject: [Python-Dev] DTRACE support
In-Reply-To: <CANc-5UzS_jUBOmixhRjKz+ooEMFYjOEPvNkCTjzWfROCKH=nbw@mail.gmail.com>
References: <5229EFE7.8040702@jcea.es>
 <CANc-5UzS_jUBOmixhRjKz+ooEMFYjOEPvNkCTjzWfROCKH=nbw@mail.gmail.com>
Message-ID: <5229F398.3040802@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/09/13 17:18, Skip Montanaro wrote:
> I looked at this several years ago. As I recall, the problem at the
> time was that the Apple and Sun DTrace implementations were
> incompatible, or that the probes they had inserted into their own
> /usr/bin/python instances were incompatible. (Don't remember which
> off the top of my head.) Of course, the DTrace folks at Apple and
> Sun weren't really interested in holding hands...

Right. They use two different set of probes because Python doesn't
provide "official" probes :-).

If I recall correctly, they were basically identical, with an extra
parameter somewhere. I think my patch provides a superset.

Anyway, I don't think neither of them support Python 3. Time to step
in and liderate.

- -- 
Jes?s Cea Avi?n                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQCVAwUBUinzmJlgi5GaxT1NAQLxwQP/cTq90CCfZhiUVLsutP0dQk4Br62kvCyn
mmMV9E/ZPJEO3LYYQT+CSiIRnG5panqdZulMDAGBBvgIbJP/E8spuyoFZkspWtPM
fk5n2etHq8JMZs36mbxnhw++W3/jlZ4osdVWZZKoKdwVBRFz2YvpLa24q+Y06gU/
5YEXHuNDP/4=
=YFm2
-----END PGP SIGNATURE-----

From jcea at jcea.es  Fri Sep  6 17:28:56 2013
From: jcea at jcea.es (Jesus Cea)
Date: Fri, 06 Sep 2013 17:28:56 +0200
Subject: [Python-Dev] DTRACE support
In-Reply-To: <5229EFE7.8040702@jcea.es>
References: <5229EFE7.8040702@jcea.es>
Message-ID: <5229F4B8.7070200@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/09/13 17:08, Jesus Cea wrote:
> Does Python-Dev have any opinion or interest in this project?.
> Should I push for it?

I have using this code for ages on my Solaris machines:

Python 2.7.5 (dtrace-issue13405_2.7:f96ea83cd766, Aug 19 2013, 02:55:15)

Python 3.3.2 (dtrace-issue13405_3.3:23dafaf73d29, Aug 19 2013, 05:52:34)

Python 3.4.0a1+ (dtrace-issue13405:d7be45bb06a3, Aug 19 2013, 07:03:24)

- -- 
Jes?s Cea Avi?n                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQCVAwUBUin0uJlgi5GaxT1NAQJP/wQAmkxZ5UJk9L+wTZb94icd3vqUGdo+mgR6
CNs0/oyemd7fL2DSaHGycqDwVPqyqNVFE+aD/ziEmWOwmiFQebudX0Z7md4ZSF89
LU4ddBPMj1bI5E/EYtlqgZg4TqwbUP3XPy9IrCEeiiSKcIk2TJxYiZ12IlcFyXU+
ff3E+DyDkBY=
=svJD
-----END PGP SIGNATURE-----

From brian at python.org  Fri Sep  6 17:29:00 2013
From: brian at python.org (Brian Curtin)
Date: Fri, 6 Sep 2013 10:29:00 -0500
Subject: [Python-Dev] when to fix cross-version bugs?
In-Reply-To: <5229EBDA.1030902@stoneleaf.us>
References: <5229EBDA.1030902@stoneleaf.us>
Message-ID: <CAD+XWwpHdGLGFen+thhpmkaV9ArWB-V8pggBjsfjzFXmFGtPoA@mail.gmail.com>

On Fri, Sep 6, 2013 at 9:51 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
> I recently committed a fix for unicodeobject.c so that the %d, %i, and %u
> format specifiers always output values (otherwise, in subclasses, the str()
> was used instead).
>
> Should this be fixed in 3.3 as well?
>
> What guidelines determine when a bug is fixed in previous versions?

If it's a bug in that version and the version is accepting bug fixes,
i.e., not in security mode, go for it. This includes crossing the 2/3
boundary if applicable.

From cf.natali at gmail.com  Fri Sep  6 17:29:12 2013
From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=)
Date: Fri, 6 Sep 2013 17:29:12 +0200
Subject: [Python-Dev] DTRACE support
In-Reply-To: <5229EFE7.8040702@jcea.es>
References: <5229EFE7.8040702@jcea.es>
Message-ID: <CAH_1eM2gG2fVsXU88j1T3y8TeGJ+fnDu4BBR2aMVg2hw=37_pg@mail.gmail.com>

> As far as I know, Erlang, Ruby, PHP, Perl, etc., support Dtrace.
> Python is embarrasingly missing from this list.
>
> Some examples:
>
> <http://crypt.codemancers.com/posts/2013-04-16-profile-ruby-apps-dtrace-part1/>
> <http://www.phpdeveloper.org/news/18859>
> <http://www.erlang.org/doc/apps/runtime_tools/DTRACE.html>
>
> I have spend a very long time on a patch for Dtrace support in most
> platforms with dtrace available. Currently working under Solaris and
> derivatives, and MacOS X. Last time I checked, it would crash FreeBSD
> because bugs in the dtrace port, but that was a long time ago.
>
> I would like to push this to Python 3.4, and the window is going to be
> closed soon, so I think this is the time to ask for opinions and
> support here.
>
> Does Python-Dev have any opinion or interest in this project?. Should
> I push for it?

IMO, that's a large, intrusive patch, which distracts the reader from
the main code and logic.

Here's an extract from Modules/gcmodule.c:

static void
dtrace_gc_done(Py_ssize_t value)
{
PYTHON_GC_DONE((long) value);
/*
* Currently a USDT tail-call will not receive the correct arguments.
* Disable the tail call here.
*/
#if defined(__sparc)
asm("nop");
#endif
}

Also have a look at cevalc.c:
http://bugs.python.org/review/13405/diff/6152/Python/ceval.c


IMO it's not worth it (personally strace/gdb/valgrind are more than
enough for me, and we''re about to gain memory tracing with Victor's
tracemalloc).

cf

From rdmurray at bitdance.com  Fri Sep  6 17:31:53 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Fri, 06 Sep 2013 11:31:53 -0400
Subject: [Python-Dev] when to fix cross-version bugs?
In-Reply-To: <5229EBDA.1030902@stoneleaf.us>
References: <5229EBDA.1030902@stoneleaf.us>
Message-ID: <20130906153153.D5DDF250845@webabinitio.net>

On Fri, 06 Sep 2013 07:51:06 -0700, Ethan Furman <ethan at stoneleaf.us> wrote:
> I recently committed a fix for unicodeobject.c so that the %d, %i, and %u format specifiers always output values 
> (otherwise, in subclasses, the str() was used instead).
>
> Should this be fixed in 3.3 as well?
> 
> What guidelines determine when a bug is fixed in previous versions?

The basic guideline is: we try very hard not to break currently working
code in a maintenance release.  Making that decision very much depends on
the details of each individual case.

I'd say this one is borderline...it would probably be OK to backport it,
since programs depending on the str of number subclasses (that is what
we are talking about here, right?  Even though you say the fix is in
unicodeobject.c...) are likely to be rare, or already have a workaround
that won't get broken by the change, but by the same token it probably
doesn't have much positive impact if it does get backported, so is it
worth the (small) chance of breaking someone's code?

--David

From stefan_ml at behnel.de  Fri Sep  6 17:33:47 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 06 Sep 2013 17:33:47 +0200
Subject: [Python-Dev] unicodedata module is out of date
In-Reply-To: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com>
References: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com>
Message-ID: <l0cskj$tc1$1@ger.gmane.org>

Andrew Miller, 06.09.2013 11:54:
> The unicodedata module only contains data up to Unicode 5.2 (October 2009),
> so attempting to reference any character from a later version e.g:
> 
> unicodedata.lookup("TURKISH LIRA SIGN")
> 
> results in a KeyError.
> 
> Also, it seems to be limited to properties in the UnicodeData.txt file and
> does not contain any data from the other files from the Unicode Character
> Database (the perl library Unicode::UCD is far more complete).
> 
> Are there any plans to update this module to the latest Unicode version
> (6.2, with 6.3 being released shortly)

It's been updated to 6.2 almost a year ago, so Python 3.3 should have that.

I don't think 6.3 support will be added before Python 3.4, assuming it's
final by then. You should open a ticket so that it won't be forgotten
before the release.

http://bugs.python.org/

That being said, the module is (mostly) generated, so you might be able to
fix it up yourself if you need it sooner in a local installation.

Stefan



From rdmurray at bitdance.com  Fri Sep  6 17:35:06 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Fri, 06 Sep 2013 11:35:06 -0400
Subject: [Python-Dev] unicodedata module is out of date
In-Reply-To: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com>
References: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com>
Message-ID: <20130906153510.6CCB0250833@webabinitio.net>

On Fri, 06 Sep 2013 10:54:45 +0100, Andrew Miller <A.J.Miller at bcs.org.uk> wrote:
> Are there any plans to update this module to the latest Unicode version
> (6.2, with 6.3 being released shortly), or is there another module that
> provides more up to date information?

Python 3.4 currently has 6.2.  If 6.3 gets released before the first RC,
I'm guessing we will probably upgrade to it.

--David

From python at mrabarnett.plus.com  Fri Sep  6 17:38:03 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Fri, 06 Sep 2013 16:38:03 +0100
Subject: [Python-Dev] unicodedata module is out of date
In-Reply-To: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com>
References: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com>
Message-ID: <5229F6DB.40000@mrabarnett.plus.com>

On 06/09/2013 10:54, Andrew Miller wrote:
> The unicodedata module only contains data up to Unicode 5.2 (October
> 2009), so attempting to reference any character from a later version e.g:
>
> unicodedata.lookup("TURKISH LIRA SIGN")
>
> results in a KeyError.
>
> Also, it seems to be limited to properties in the UnicodeData.txt file
> and does not contain any data from the other files from the Unicode
> Character Database (the perl library Unicode::UCD is far more complete).
>
> Are there any plans to update this module to the latest Unicode version
> (6.2, with 6.3 being released shortly), or is there another module that
> provides more up to date information?
>
Which version of Python are you talking about? Python 3.3 uses Unicode 
version 6.1.

From ethan at stoneleaf.us  Fri Sep  6 17:14:09 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 06 Sep 2013 08:14:09 -0700
Subject: [Python-Dev] inspect and metaclasses
In-Reply-To: <CAMSv6X1k30-PAWct_9tDTmwir4mDO0jAs-mbrvvBrJc5XE6VTQ@mail.gmail.com>
References: <5229D97E.9080103@stoneleaf.us>
 <CAMSv6X1k30-PAWct_9tDTmwir4mDO0jAs-mbrvvBrJc5XE6VTQ@mail.gmail.com>
Message-ID: <5229F141.2050007@stoneleaf.us>

On 09/06/2013 07:47 AM, Armin Rigo wrote:
>
> Are you suggesting that inspect.getmro(A) would return (A, object,
> type)?  That seems very wrong to me.

Currently, `inspect.getmro(A)` returns `(A, object)`.

Considering that Python actually will look in A's metaclass to find a class attribute, I think returning `(A, object, 
type(A)` is appropriate:

=================================================================================
Python 3.4.0a1+ (default:61ca4732399b+, Sep  4 2013, 22:28:04)
[GCC 4.7.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
--> class Meta(type):
...   meta_attr = 42
...

--> class Class(metaclass=Meta):
...   cls_attr = 'Vitamin-soaked towel'
...   def __init__(self):
...     self.inst_attr = 'dolphins'
...

--> test = Class()

--> test.inst_attr
'dolphins'

--> test.cls_attr
'Vitamin-soaked towel'

--> test.meta_attr
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
AttributeError: 'Class' object has no attribute 'meta_attr'

--> Class.cls_attr
'Vitamin-soaked towel'

--> Class.meta_attr
42

--> import inspect

--> inspect.getmro(Class)  #  with patch in place
(<class '__main__.Class'>, <class 'object'>, <class '__main__.Meta'>)
=================================================================================

> If the goal is to fix `inspect.classify_class_attrs()`, then this
> function only needs a specific fix, along the lines of looking in
> `getmro(A) + getmro(type(A))`.  (A more minor issue is that the bug
> report suggests `... + (type(A),)` only, but that's wrong: Python will
> also look in all the base classes of type(A).)

Good point.  Will incorporate that into the final fix, whichever way it ends up.


> "Fixing" inspect.getmro() as suggested would break a lot of other usages of it.

Any examples?

--
~Ethan~

From rdmurray at bitdance.com  Fri Sep  6 17:39:01 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Fri, 06 Sep 2013 11:39:01 -0400
Subject: [Python-Dev] unicodedata module is out of date
In-Reply-To: <l0cskj$tc1$1@ger.gmane.org>
References: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com>
 <l0cskj$tc1$1@ger.gmane.org>
Message-ID: <20130906153901.9ED57250845@webabinitio.net>

On Fri, 06 Sep 2013 17:33:47 +0200, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Andrew Miller, 06.09.2013 11:54:
> > The unicodedata module only contains data up to Unicode 5.2 (October 2009),
> > so attempting to reference any character from a later version e.g:
> > 
> > unicodedata.lookup("TURKISH LIRA SIGN")
> > 
> > results in a KeyError.
> > 
> > Also, it seems to be limited to properties in the UnicodeData.txt file and
> > does not contain any data from the other files from the Unicode Character
> > Database (the perl library Unicode::UCD is far more complete).
> > 
> > Are there any plans to update this module to the latest Unicode version
> > (6.2, with 6.3 being released shortly)
> 
> It's been updated to 6.2 almost a year ago, so Python 3.3 should have that.

3.3 shipped with 6.1.

--David

From solipsis at pitrou.net  Fri Sep  6 17:41:11 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 6 Sep 2013 17:41:11 +0200
Subject: [Python-Dev] DTRACE support
References: <5229EFE7.8040702@jcea.es>
Message-ID: <20130906174111.446ea2c4@pitrou.net>

Le Fri, 06 Sep 2013 17:08:23 +0200,
Jesus Cea <jcea at jcea.es> a ?crit :
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> As far as I know, Erlang, Ruby, PHP, Perl, etc., support Dtrace.
> Python is embarrasingly missing from this list.
> 
> Some examples:
> 
> <http://crypt.codemancers.com/posts/2013-04-16-profile-ruby-apps-dtrace-part1/>
> <http://www.phpdeveloper.org/news/18859>
> <http://www.erlang.org/doc/apps/runtime_tools/DTRACE.html>
> 
> I have spend a very long time on a patch for Dtrace support in most
> platforms with dtrace available. Currently working under Solaris and
> derivatives, and MacOS X. Last time I checked, it would crash FreeBSD
> because bugs in the dtrace port, but that was a long time ago.
> 
> I would like to push this to Python 3.4, and the window is going to be
> closed soon, so I think this is the time to ask for opinions and
> support here.

You should start by addressing review comments:
http://bugs.python.org/issue13405#msg151751

Right now, I agree with Charles-Fran?ois: your patch is too intrusive.

Regards

Antoine.



From rdmurray at bitdance.com  Fri Sep  6 17:44:08 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Fri, 06 Sep 2013 11:44:08 -0400
Subject: [Python-Dev] inspect and metaclasses
In-Reply-To: <5229F141.2050007@stoneleaf.us>
References: <5229D97E.9080103@stoneleaf.us>
 <CAMSv6X1k30-PAWct_9tDTmwir4mDO0jAs-mbrvvBrJc5XE6VTQ@mail.gmail.com>
 <5229F141.2050007@stoneleaf.us>
Message-ID: <20130906154408.BFAAD250845@webabinitio.net>

On Fri, 06 Sep 2013 08:14:09 -0700, Ethan Furman <ethan at stoneleaf.us> wrote:
> On 09/06/2013 07:47 AM, Armin Rigo wrote:
> >
> > Are you suggesting that inspect.getmro(A) would return (A, object,
> > type)?  That seems very wrong to me.
> 
> Currently, `inspect.getmro(A)` returns `(A, object)`.

Which matches A.__mro__.  EOD, I think.

--David

From solipsis at pitrou.net  Fri Sep  6 17:43:13 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 6 Sep 2013 17:43:13 +0200
Subject: [Python-Dev] Details of "import lock" in 3.3
References: <5229EFEC.1010503@jcea.es>
Message-ID: <20130906174313.63e37ac3@pitrou.net>

Le Fri, 06 Sep 2013 17:08:28 +0200,
Jesus Cea <jcea at jcea.es> a ?crit :
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> With importlib and other recent changes, what are the current details
> of "import lock"?. That is, the lock/locks held when Python code does
> "import", specially in the case of multithreading. Is that documented
> anywhere?

Quick summary here:
http://docs.python.org/3/whatsnew/3.3.html#a-finer-grained-import-lock

Otherwise, I'm afraid the source code has the most information, e.g.:
http://hg.python.org/cpython/file/1d88d04aade2/Lib/importlib/_bootstrap.py#l124

Regards

Antoine.



From benjamin at python.org  Fri Sep  6 17:45:14 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Fri, 6 Sep 2013 11:45:14 -0400
Subject: [Python-Dev] unicodedata module is out of date
In-Reply-To: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com>
References: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com>
Message-ID: <CAPZV6o98M=fb1Re=R3b3RSBgqZdAF_Lo=NaxDTT1aq3nC5Bjhw@mail.gmail.com>

2013/9/6 Andrew Miller <A.J.Miller at bcs.org.uk>:
> The unicodedata module only contains data up to Unicode 5.2 (October 2009),
> so attempting to reference any character from a later version e.g:
>
> unicodedata.lookup("TURKISH LIRA SIGN")
>
> results in a KeyError.
>
> Also, it seems to be limited to properties in the UnicodeData.txt file and
> does not contain any data from the other files from the Unicode Character
> Database (the perl library Unicode::UCD is far more complete).
>
> Are there any plans to update this module to the latest Unicode version
> (6.2, with 6.3 being released shortly), or is there another module that
> provides more up to date information?

I usually keep the latest Python version up to date with the latest
Unicode version, so 3.4 will have Unicode 6.2.

-- 
Regards,
Benjamin

From jcea at jcea.es  Fri Sep  6 17:55:52 2013
From: jcea at jcea.es (Jesus Cea)
Date: Fri, 06 Sep 2013 17:55:52 +0200
Subject: [Python-Dev] DTRACE support
In-Reply-To: <CAH_1eM2gG2fVsXU88j1T3y8TeGJ+fnDu4BBR2aMVg2hw=37_pg@mail.gmail.com>
References: <5229EFE7.8040702@jcea.es>
 <CAH_1eM2gG2fVsXU88j1T3y8TeGJ+fnDu4BBR2aMVg2hw=37_pg@mail.gmail.com>
Message-ID: <5229FB08.20603@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/09/13 17:29, Charles-Fran?ois Natali wrote:
> IMO, that's a large, intrusive patch, which distracts the reader
> from the main code and logic.

Yes, the patch is intrusive. It must be, to get its goals. Could be
improved, nevertheless. Help and suggestions welcome.

I want to write a probe for the GIL, but I didn't because I know
adding the cost of an extra single machine code branch would be
anathema here :-) (lets do baby steps), but I would love to be able to
watch with detail all interaction between the GIL, threads, and OS
scheduling on my Solaris :). That is very valuable information to
have, for instance, to guide future improvements of the GIL. How can
you get that kind of information with any other tool?

> IMO it's not worth it (personally strace/gdb/valgrind are more
> than enough for me, and we''re about to gain memory tracing with
> Victor's tracemalloc).

The main value of DTrace is systemwide observability. You can see
something "strange" at kernel level and trace it to a particular line
of code in a random Python script. There is no other tool that can do
that. You have complete transversal observability of ALL the code
running in your computer, kernel or usermode, clean reports with
threads, etc.

Valgrind doesn't work on Solaris and *BSD support is unclean.

You can run a dtrace script in a long running python process if you
need it, after launching it, at runtime, and when the dtrace script is
done, the python program keeps running without any penalty. Just
poking around, for instance. I do it constantly for, for instance,
profiling covering both Python as any C code/libraries called from it.

Maybe the biggest objection would be that most python-devs are running
Linux, and you don't have dtrace support on linux unless you are
running Oracle distribution. But world is larger than linux, and there
are some efforts to port DTrace to Linux itself. DTrace is available
on Solaris and derivatives, MacOS X and FreeBSD.

- -- 
Jes?s Cea Avi?n                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQCVAwUBUin7CJlgi5GaxT1NAQKbfwP+PjOqf3pjlHHq78ggA8qyNkjPFXEoRVj9
9PKslZ7FiU+JWxzXY52k1GNspHhIox+PAcvdBL/gWU3rOiqjEm5fU8FX61Lh6FMj
bB+QRyZLpLfmANrLX43sBvwDbG9gTuq8FVUvqmtSme615vX2ygITwNZysQ7xPoD/
jXbeOAF0LZU=
=asEC
-----END PGP SIGNATURE-----

From A.J.Miller at bcs.org.uk  Fri Sep  6 17:29:32 2013
From: A.J.Miller at bcs.org.uk (Andrew Miller)
Date: Fri, 6 Sep 2013 16:29:32 +0100
Subject: [Python-Dev] Fwd: unicodedata module is out of date
In-Reply-To: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com>
References: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com>
Message-ID: <CAPMPPoTuTB6T-FBq4__DSNoZUtH5XY8moMJR9CgLS76mZ5xhoA@mail.gmail.com>

The unicodedata module only contains data up to Unicode 5.2 (October 2009),
so attempting to reference any character from a later version e.g:

unicodedata.lookup("TURKISH LIRA SIGN")

results in a KeyError.

Also, it seems to be limited to properties in the UnicodeData.txt file and
does not contain any data from the other files from the Unicode Character
Database (the perl library Unicode::UCD is far more complete).

Are there any plans to update this module to the latest Unicode version
(6.2, with 6.3 being released shortly), or is there another module that
provides more up to date information?

Thanks,

Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130906/ce3e7ce2/attachment.html>

From A.J.Miller at bcs.org.uk  Fri Sep  6 17:55:22 2013
From: A.J.Miller at bcs.org.uk (Andrew Miller)
Date: Fri, 6 Sep 2013 16:55:22 +0100
Subject: [Python-Dev] unicodedata module is out of date
In-Reply-To: <5229F6DB.40000@mrabarnett.plus.com>
References: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com>
 <5229F6DB.40000@mrabarnett.plus.com>
Message-ID: <CAPMPPoRMBsBbAasu8ewEwqJOJARLAsctJricjKdVYVkXdLy8Tg@mail.gmail.com>

I've just checked on Python 2.7.5 and Python 3.3.2 (Win32 versions).

In Python 3.3.2 unicodedata.unidata_version is set to '6.1.0'.

In Python 2.7.5 it is set to '5.2.0' so it looks as though this version is
no longer being updated.

Since my initial post I've downloaded the Python 2.7.5 source and have
found the makeunicodedata.py script which creates this module.

Are there plans to add the extra data from the other UCD files to this
module?  At the moment I am using a module from
https://gist.github.com/anonymous/2204527 to obtain the script of a
character but it would be nice if this was available from the standard
library.


On 6 September 2013 16:38, MRAB <python at mrabarnett.plus.com> wrote:

> On 06/09/2013 10:54, Andrew Miller wrote:
>
>> The unicodedata module only contains data up to Unicode 5.2 (October
>> 2009), so attempting to reference any character from a later version e.g:
>>
>> unicodedata.lookup("TURKISH LIRA SIGN")
>>
>> results in a KeyError.
>>
>> Also, it seems to be limited to properties in the UnicodeData.txt file
>> and does not contain any data from the other files from the Unicode
>> Character Database (the perl library Unicode::UCD is far more complete).
>>
>> Are there any plans to update this module to the latest Unicode version
>> (6.2, with 6.3 being released shortly), or is there another module that
>> provides more up to date information?
>>
>>  Which version of Python are you talking about? Python 3.3 uses Unicode
> version 6.1.
>
> ______________________________**_________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/**mailman/listinfo/python-dev<https://mail.python.org/mailman/listinfo/python-dev>
> Unsubscribe: https://mail.python.org/**mailman/options/python-dev/a.**
> j.miller%40bcs.org.uk<https://mail.python.org/mailman/options/python-dev/a.j.miller%40bcs.org.uk>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130906/f58de89a/attachment.html>

From jcea at jcea.es  Fri Sep  6 18:02:32 2013
From: jcea at jcea.es (Jesus Cea)
Date: Fri, 06 Sep 2013 18:02:32 +0200
Subject: [Python-Dev] Details of "import lock" in 3.3
In-Reply-To: <20130906174313.63e37ac3@pitrou.net>
References: <5229EFEC.1010503@jcea.es> <20130906174313.63e37ac3@pitrou.net>
Message-ID: <5229FC98.9080106@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/09/13 17:43, Antoine Pitrou wrote:
> Quick summary here: 
> http://docs.python.org/3/whatsnew/3.3.html#a-finer-grained-import-lock
>
>  Otherwise, I'm afraid the source code has the most information,
> e.g.: 
> http://hg.python.org/cpython/file/1d88d04aade2/Lib/importlib/_bootstrap.py#l124

Yes,
> 
I was depending of the global import lock in my code. I am
evaluating impact under Python 3.3.

Under Python 3.3, metafinder/loader MUST BE reentrant?.

Can they count on the fact that they could be called concurrently by
multiple threads but NOT for importing the SAME module?.

- -- 
Jes?s Cea Avi?n                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQCVAwUBUin8mJlgi5GaxT1NAQKQugP+N0VI/BhqKvcaUicu7YMR09VhFweoAE6h
c48ltOrhlWhrauYc9XYeoEPZ2ksl9/SDnMt83cE+qLunSBltkSbYv3j/0LXWyHwk
ZGjzh5vQGTgCu3/2iCkFWq/tSAsgUfk8YfbwPQK82L1gs7LbC5vt2jWlRwNQs0nl
2Sr1Nlbf7og=
=GL0o
-----END PGP SIGNATURE-----

From status at bugs.python.org  Fri Sep  6 18:07:41 2013
From: status at bugs.python.org (Python tracker)
Date: Fri,  6 Sep 2013 18:07:41 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20130906160741.06F3F56A17@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2013-08-30 - 2013-09-06)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    4195 (+13)
  closed 26523 (+47)
  total  30718 (+60)

Open issues with patches: 1915 


Issues opened (43)
==================

#16826: Don't check for PYTHONCASEOK if interpreter started with -E
http://bugs.python.org/issue16826  reopened by meador.inge

#18709: SSL module fails to handle NULL bytes inside subjectAltNames g
http://bugs.python.org/issue18709  reopened by neologix

#18886: BytesGenerator does not handle 'binary' CTE correctly
http://bugs.python.org/issue18886  opened by r.david.murray

#18887: test_multiprocessing.test_connection failure
http://bugs.python.org/issue18887  opened by neologix

#18890: Add a raw_data_manager content manager to the email package.
http://bugs.python.org/issue18890  opened by r.david.murray

#18891: Master patch for content manager addtion to email package.
http://bugs.python.org/issue18891  opened by r.david.murray

#18893: invalid exception handling in Lib/ctypes/macholib/dyld.py
http://bugs.python.org/issue18893  opened by scoder

#18894: In unittest.TestResult.failures remove deprecated fail* method
http://bugs.python.org/issue18894  opened by py.user

#18895: In unittest.TestResult.addError split the sentence
http://bugs.python.org/issue18895  opened by py.user

#18896: Remove namedtuple 255 arguments restriction
http://bugs.python.org/issue18896  opened by valorien

#18898: Apply the setobject optimizations to dictionaries
http://bugs.python.org/issue18898  opened by rhettinger

#18899: make pystone.py Py3 compatible in benchmark suite
http://bugs.python.org/issue18899  opened by scoder

#18900: Add the random.distrib module
http://bugs.python.org/issue18900  opened by serhiy.storchaka

#18902: Make ElementTree event handling more modular to allow custom t
http://bugs.python.org/issue18902  opened by eli.bendersky

#18903: IDLE file-completion is case-sensitive in Windows
http://bugs.python.org/issue18903  opened by anikom15

#18904: Unnecessary test in file descriptor inheritance test
http://bugs.python.org/issue18904  opened by vajrasky

#18905: pydoc -p 0 says the server is available at localhost:0
http://bugs.python.org/issue18905  opened by Wieland.Hoffmann

#18906: Create a way to always run tests in subprocesses within regrte
http://bugs.python.org/issue18906  opened by eli.bendersky

#18907: urllib2.open FTP open times out at 20 secs despite timeout par
http://bugs.python.org/issue18907  opened by nagle

#18908: Enum docs: sections leak out
http://bugs.python.org/issue18908  opened by elazar

#18910: IDLE: Unit test for textView.py
http://bugs.python.org/issue18910  opened by philwebster

#18911: minidom does not encode correctly when calling Document.writex
http://bugs.python.org/issue18911  opened by brianvanderburg2

#18913: ssl._ssl._test_decode_cert seems to leak memory with certain c
http://bugs.python.org/issue18913  opened by sYnfo

#18915: ssl.wrap_socket, pass in certfile and keyfile as PEM strings
http://bugs.python.org/issue18915  opened by mpb

#18916: Various out-of-date Lock text in 3.2+
http://bugs.python.org/issue18916  opened by tim.peters

#18917: python won't display greek characters in apache under windows
http://bugs.python.org/issue18917  opened by nickl1

#18918: help('FILES') finds no documentation
http://bugs.python.org/issue18918  opened by st.sempert at gmail.com

#18919: Unify audio modules tests
http://bugs.python.org/issue18919  opened by serhiy.storchaka

#18921: In imaplib, cached capabilities may be out of date after login
http://bugs.python.org/issue18921  opened by sjmurdoch

#18923: Use the new selectors module in the subprocess module
http://bugs.python.org/issue18923  opened by haypo

#18925: select.poll.modify is not documented
http://bugs.python.org/issue18925  opened by giampaolo.rodola

#18927: Lock.acquire() docs incorrect about negative timeout
http://bugs.python.org/issue18927  opened by tim.peters

#18929: inspect.classify_class_attrs ignores metaclass
http://bugs.python.org/issue18929  opened by ethan.furman

#18930: os.spawnXX functions terminates process if second argument is 
http://bugs.python.org/issue18930  opened by SSchukat

#18931: new selectors module should support devpoll on Solaris
http://bugs.python.org/issue18931  opened by giampaolo.rodola

#18932: selectors and modify()
http://bugs.python.org/issue18932  opened by giampaolo.rodola

#18934: multiprocessing: use selectors module
http://bugs.python.org/issue18934  opened by neologix

#18935: test_regrtest.test_timeout failure
http://bugs.python.org/issue18935  opened by neologix

#18936: getopt chokes on unicode option names
http://bugs.python.org/issue18936  opened by jason.coombs

#18937: add unittest assertion for logging
http://bugs.python.org/issue18937  opened by pitrou

#18943: argparse: default args in mutually exclusive groups
http://bugs.python.org/issue18943  opened by arigo

#18944: Minor mistake in test_set.py
http://bugs.python.org/issue18944  opened by arigo

#18945: Name collision handling in tempfile is not covered by tests
http://bugs.python.org/issue18945  opened by vlad



Most recent 15 issues with no replies (15)
==========================================

#18945: Name collision handling in tempfile is not covered by tests
http://bugs.python.org/issue18945

#18944: Minor mistake in test_set.py
http://bugs.python.org/issue18944

#18943: argparse: default args in mutually exclusive groups
http://bugs.python.org/issue18943

#18936: getopt chokes on unicode option names
http://bugs.python.org/issue18936

#18935: test_regrtest.test_timeout failure
http://bugs.python.org/issue18935

#18932: selectors and modify()
http://bugs.python.org/issue18932

#18930: os.spawnXX functions terminates process if second argument is 
http://bugs.python.org/issue18930

#18927: Lock.acquire() docs incorrect about negative timeout
http://bugs.python.org/issue18927

#18915: ssl.wrap_socket, pass in certfile and keyfile as PEM strings
http://bugs.python.org/issue18915

#18910: IDLE: Unit test for textView.py
http://bugs.python.org/issue18910

#18905: pydoc -p 0 says the server is available at localhost:0
http://bugs.python.org/issue18905

#18904: Unnecessary test in file descriptor inheritance test
http://bugs.python.org/issue18904

#18903: IDLE file-completion is case-sensitive in Windows
http://bugs.python.org/issue18903

#18895: In unittest.TestResult.addError split the sentence
http://bugs.python.org/issue18895

#18894: In unittest.TestResult.failures remove deprecated fail* method
http://bugs.python.org/issue18894



Most recent 15 issues waiting for review (15)
=============================================

#18944: Minor mistake in test_set.py
http://bugs.python.org/issue18944

#18935: test_regrtest.test_timeout failure
http://bugs.python.org/issue18935

#18934: multiprocessing: use selectors module
http://bugs.python.org/issue18934

#18931: new selectors module should support devpoll on Solaris
http://bugs.python.org/issue18931

#18919: Unify audio modules tests
http://bugs.python.org/issue18919

#18910: IDLE: Unit test for textView.py
http://bugs.python.org/issue18910

#18908: Enum docs: sections leak out
http://bugs.python.org/issue18908

#18905: pydoc -p 0 says the server is available at localhost:0
http://bugs.python.org/issue18905

#18904: Unnecessary test in file descriptor inheritance test
http://bugs.python.org/issue18904

#18899: make pystone.py Py3 compatible in benchmark suite
http://bugs.python.org/issue18899

#18898: Apply the setobject optimizations to dictionaries
http://bugs.python.org/issue18898

#18895: In unittest.TestResult.addError split the sentence
http://bugs.python.org/issue18895

#18894: In unittest.TestResult.failures remove deprecated fail* method
http://bugs.python.org/issue18894

#18893: invalid exception handling in Lib/ctypes/macholib/dyld.py
http://bugs.python.org/issue18893

#18891: Master patch for content manager addtion to email package.
http://bugs.python.org/issue18891



Top 10 most discussed issues (10)
=================================

#18808: Thread.join returns before PyThreadState is destroyed
http://bugs.python.org/issue18808  11 msgs

#18844: allow weights in random.choice
http://bugs.python.org/issue18844  11 msgs

#18843: Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at P
http://bugs.python.org/issue18843  10 msgs

#18693: help() not helpful with enum
http://bugs.python.org/issue18693   9 msgs

#18726: json functions have too many positional parameters
http://bugs.python.org/issue18726   9 msgs

#18913: ssl._ssl._test_decode_cert seems to leak memory with certain c
http://bugs.python.org/issue18913   8 msgs

#18831: importlib.import_module() bypasses builtins.__import__
http://bugs.python.org/issue18831   7 msgs

#18709: SSL module fails to handle NULL bytes inside subjectAltNames g
http://bugs.python.org/issue18709   6 msgs

#18840: Tutorial recommends pickle module without any warning of insec
http://bugs.python.org/issue18840   6 msgs

#18876: Problems with files opened in append mode with io module
http://bugs.python.org/issue18876   6 msgs



Issues closed (43)
==================

#12037: test_email failures under desktop Windows
http://bugs.python.org/issue12037  closed by terry.reedy

#15350: {urllib,urllib.parse}.urlencode.__doc__ is unclear
http://bugs.python.org/issue15350  closed by orsenthil

#16853: add a Selector to the select module
http://bugs.python.org/issue16853  closed by pitrou

#17224: can not open idle in python 2.7.3
http://bugs.python.org/issue17224  closed by terry.reedy

#17930: Search not needed in combinations_with_replacement
http://bugs.python.org/issue17930  closed by tim.peters

#18105: ElementTree writes invalid files when UTF-16 encoding is speci
http://bugs.python.org/issue18105  closed by eli.bendersky

#18418: Thread.isAlive() sometimes True after fork
http://bugs.python.org/issue18418  closed by neologix

#18489: IDLE Unit test for SearchEngine.py
http://bugs.python.org/issue18489  closed by terry.reedy

#18672: Fix format specifiers for debug output in _sre.c
http://bugs.python.org/issue18672  closed by serhiy.storchaka

#18720: Switch suitable constants in the socket module to IntEnum
http://bugs.python.org/issue18720  closed by python-dev

#18738: String formatting (% and str.format) issues with Enum
http://bugs.python.org/issue18738  closed by python-dev

#18745: Test enum in test_json is ignorant of infinity value
http://bugs.python.org/issue18745  closed by python-dev

#18750: '' % [1] doesn't fail
http://bugs.python.org/issue18750  closed by asvetlov

#18756: os.urandom() fails under high load
http://bugs.python.org/issue18756  closed by pitrou

#18780: SystemError when formatting int subclass
http://bugs.python.org/issue18780  closed by python-dev

#18826: reversed() requires a sequence - Could work on any iterator?
http://bugs.python.org/issue18826  closed by rhettinger

#18830: Remove duplicates from a result of getclasstree()
http://bugs.python.org/issue18830  closed by serhiy.storchaka

#18845: 2.7.5-r2: Fatal Python error: Segmentation fault
http://bugs.python.org/issue18845  closed by tim.peters

#18849: Failure to try another name for tempfile when directory with c
http://bugs.python.org/issue18849  closed by python-dev

#18850: xml.etree.ElementTree accepts control chars.
http://bugs.python.org/issue18850  closed by eli.bendersky

#18851: subprocess's Popen closes stdout/stderr filedescriptors used i
http://bugs.python.org/issue18851  closed by pitrou

#18870: eval() uses latin-1 to decode str
http://bugs.python.org/issue18870  closed by python-dev

#18878: Add support of the 'with' statement to sunau.open.
http://bugs.python.org/issue18878  closed by serhiy.storchaka

#18882: Add threading.main_thread() function
http://bugs.python.org/issue18882  closed by asvetlov

#18888: Add stdlib support for random sampling with replacement
http://bugs.python.org/issue18888  closed by mark.dickinson

#18889: test_sax: multiple failures on Windows desktop
http://bugs.python.org/issue18889  closed by tim.peters

#18892: sqlite3, valued records not persisted, default ones are
http://bugs.python.org/issue18892  closed by debewerker

#18897: Illegal instruction at Python-2.7.5/Modules/_sre.c:1173
http://bugs.python.org/issue18897  closed by pitrou

#18901: sunau.getparams should return a namedtuple
http://bugs.python.org/issue18901  closed by serhiy.storchaka

#18909: Segfaults on win-amd64 due to corrupt pointer to Tkapp_Interp
http://bugs.python.org/issue18909  closed by haypo

#18912: Intendation issue in example code in itertools.count documenta
http://bugs.python.org/issue18912  closed by python-dev

#18914: Confusing documentation in the urllib2 HOWTO
http://bugs.python.org/issue18914  closed by michael.foord

#18920: argparse module version action
http://bugs.python.org/issue18920  closed by eli.bendersky

#18922: Output versions of scripts to stdout
http://bugs.python.org/issue18922  closed by serhiy.storchaka

#18924: Enum members are easily replaced
http://bugs.python.org/issue18924  closed by python-dev

#18926: plistlib - str converted to bool
http://bugs.python.org/issue18926  closed by VertigoRay

#18928: Remove misleading documentation for random.shuffle
http://bugs.python.org/issue18928  closed by dbenbenn

#18933: Add link to source code in logging documentation
http://bugs.python.org/issue18933  closed by python-dev

#18938: Prepend Is Not A Word
http://bugs.python.org/issue18938  closed by benjamin.peterson

#18939: Venv docs regarding original python install
http://bugs.python.org/issue18939  closed by python-dev

#18940: TimedRotatingFileHandler and RotatingFileHandler fail to doRol
http://bugs.python.org/issue18940  closed by python-dev

#18941: RotatingFileHandler and TimedRotatingFileHandler do not respec
http://bugs.python.org/issue18941  closed by python-dev

#18942: _debugmallocstats() gibberish output on Windows
http://bugs.python.org/issue18942  closed by tim.peters

From jcea at jcea.es  Fri Sep  6 18:14:26 2013
From: jcea at jcea.es (Jesus Cea)
Date: Fri, 06 Sep 2013 18:14:26 +0200
Subject: [Python-Dev] DTRACE support
In-Reply-To: <20130906174111.446ea2c4@pitrou.net>
References: <5229EFE7.8040702@jcea.es> <20130906174111.446ea2c4@pitrou.net>
Message-ID: <5229FF62.9020702@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/09/13 17:41, Antoine Pitrou wrote:
> You should start by addressing review comments: 
> http://bugs.python.org/issue13405#msg151751

Antoine, my first step now is to poke Python-DEV about this subject.
If the consensus is "DON'T" I will probably maintain this patch by
myself. I saw big "BUTs" with this work in the bug tracker, so I
rather prefer to ask for actual interest before investing in iron out
patch implementation details.

If devs actually want it, then next step is to improve current work to
make it acceptable in mainline. There are quite a few ugly corners I
would love to file out, beside your (really valuable) feedback.

That is how I see it.

> Right now, I agree with Charles-Fran?ois: your patch is too
> intrusive.

It is intrusive. Yes. I think it must be, by its own nature. Probably
room for improvement and code transparency. But... are Python-DEVs
interested in the project?. That is the point :)

- -- 
Jes?s Cea Avi?n                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQCVAwUBUin/Yplgi5GaxT1NAQI6tAP+MaGbSm8j3xPBjMui9tePg2engB+G/ckh
TzVac8Gz5yhWhixOFLk3cXYySieB6ttjsAG1NZYKxIwyLSLrxcL8pTh4c8oHB8zu
/wYfIdXt6leH6HECXJquPLxLM39Z6KIOTt2QgKYSWy6ZHGzVaaeTFvrMhW0N/5rp
SCHSJQc7Vn4=
=65h4
-----END PGP SIGNATURE-----

From ethan at stoneleaf.us  Fri Sep  6 17:59:02 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 06 Sep 2013 08:59:02 -0700
Subject: [Python-Dev] inspect and metaclasses
In-Reply-To: <20130906154408.BFAAD250845@webabinitio.net>
References: <5229D97E.9080103@stoneleaf.us>
 <CAMSv6X1k30-PAWct_9tDTmwir4mDO0jAs-mbrvvBrJc5XE6VTQ@mail.gmail.com>
 <5229F141.2050007@stoneleaf.us> <20130906154408.BFAAD250845@webabinitio.net>
Message-ID: <5229FBC6.1020708@stoneleaf.us>

On 09/06/2013 08:44 AM, R. David Murray wrote:
> On Fri, 06 Sep 2013 08:14:09 -0700, Ethan Furman <ethan at stoneleaf.us> wrote:
>> On 09/06/2013 07:47 AM, Armin Rigo wrote:
>>>
>>> Are you suggesting that inspect.getmro(A) would return (A, object,
>>> type)?  That seems very wrong to me.
>>
>> Currently, `inspect.getmro(A)` returns `(A, object)`.
>
> Which matches A.__mro__.  EOD, I think.

I hope not, because currently this leaves a hole in the introspection of class attributes.

Is __mro__ aimed primarily at instances and not classes?  That seems to be how it works.  In which case, do we need 
another __mmro__ (or __cmro__ or ...) to handle the mro of classes themselves?

For the short term I can restrict the change to inspect.classify_class_attrs().

--
~Ethan~

From rdmurray at bitdance.com  Fri Sep  6 18:37:28 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Fri, 06 Sep 2013 12:37:28 -0400
Subject: [Python-Dev] inspect and metaclasses
In-Reply-To: <5229FBC6.1020708@stoneleaf.us>
References: <5229D97E.9080103@stoneleaf.us>
 <CAMSv6X1k30-PAWct_9tDTmwir4mDO0jAs-mbrvvBrJc5XE6VTQ@mail.gmail.com>
 <5229F141.2050007@stoneleaf.us> <20130906154408.BFAAD250845@webabinitio.net>
 <5229FBC6.1020708@stoneleaf.us>
Message-ID: <20130906163728.ACFAD2507F1@webabinitio.net>

On Fri, 06 Sep 2013 08:59:02 -0700, Ethan Furman <ethan at stoneleaf.us> wrote:
> On 09/06/2013 08:44 AM, R. David Murray wrote:
> > On Fri, 06 Sep 2013 08:14:09 -0700, Ethan Furman <ethan at stoneleaf.us> wrote:
> >> On 09/06/2013 07:47 AM, Armin Rigo wrote:
> >>>
> >>> Are you suggesting that inspect.getmro(A) would return (A, object,
> >>> type)?  That seems very wrong to me.
> >>
> >> Currently, `inspect.getmro(A)` returns `(A, object)`.
> >
> > Which matches A.__mro__.  EOD, I think.
> 
> I hope not, because currently this leaves a hole in the introspection of class attributes.
> 
> Is __mro__ aimed primarily at instances and not classes?  That seems to be how it works.  In which case, do we need 
> another __mmro__ (or __cmro__ or ...) to handle the mro of classes themselves?

Or maybe just a new inspect function?

> For the short term I can restrict the change to inspect.classify_class_attrs().

Sounds like the best course.

--David

From ethan at stoneleaf.us  Fri Sep  6 18:18:12 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 06 Sep 2013 09:18:12 -0700
Subject: [Python-Dev] when to fix cross-version bugs?
In-Reply-To: <5229EBDA.1030902@stoneleaf.us>
References: <5229EBDA.1030902@stoneleaf.us>
Message-ID: <522A0044.10302@stoneleaf.us>

On 09/06/2013 07:51 AM, Ethan Furman wrote:
>
> What guidelines determine when a bug is fixed in previous versions?

On 09/06/2013 08:29 AM, Brian Curtin wrote:
>
> If it's a bug in that version and the version is accepting bug fixes,
> i.e., not in security mode, go for it. This includes crossing the 2/3
> boundary if applicable.

On 09/06/2013 08:31 AM, R. David Murray wrote:
>
> The basic guideline is: we try very hard not to break currently working
> code in a maintenance release.  Making that decision very much depends on
> the details of each individual case.
>
> [so] it probably doesn't have much positive impact if it does get
> backported, so is it worth the (small) chance of breaking someone's code?

And they say never go to the elves for advice!  ;)

--
~Ethan~

From solipsis at pitrou.net  Fri Sep  6 19:02:47 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 6 Sep 2013 19:02:47 +0200
Subject: [Python-Dev] inspect and metaclasses
References: <5229D97E.9080103@stoneleaf.us>
 <CAMSv6X1k30-PAWct_9tDTmwir4mDO0jAs-mbrvvBrJc5XE6VTQ@mail.gmail.com>
 <5229F141.2050007@stoneleaf.us>
Message-ID: <20130906190247.61d5db99@fsol>

On Fri, 06 Sep 2013 08:14:09 -0700
Ethan Furman <ethan at stoneleaf.us> wrote:
> On 09/06/2013 07:47 AM, Armin Rigo wrote:
> >
> > Are you suggesting that inspect.getmro(A) would return (A, object,
> > type)?  That seems very wrong to me.
> 
> Currently, `inspect.getmro(A)` returns `(A, object)`.
> 
> Considering that Python actually will look in A's metaclass to find a class attribute, I think returning `(A, object, 
> type(A)` is appropriate:

No, I don't think it's appropriate at all. You are conflating two
things:
- the instance lookup of an attribute (on A, here)
- the class lookup when the instance lookup fails (on type, here)

Both lookups obey the same MRO rules, it just happens that the
second lookup uses a trivial MRO:

>>> type.__mro__
(<class 'type'>, <class 'object'>)


Regards

Antoine.



From solipsis at pitrou.net  Fri Sep  6 19:05:16 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 6 Sep 2013 19:05:16 +0200
Subject: [Python-Dev] DTRACE support
References: <5229EFE7.8040702@jcea.es> <20130906174111.446ea2c4@pitrou.net>
 <5229FF62.9020702@jcea.es>
Message-ID: <20130906190516.471028ba@fsol>

On Fri, 06 Sep 2013 18:14:26 +0200
Jesus Cea <jcea at jcea.es> wrote:
> 
> > Right now, I agree with Charles-Fran?ois: your patch is too
> > intrusive.
> 
> It is intrusive. Yes. I think it must be, by its own nature. Probably
> room for improvement and code transparency. But... are Python-DEVs
> interested in the project?. That is the point :)

Well, I'm not *personally* interested in anything that only addresses
Solaris, OS X and the like :)

And, no, it doesn't have to be *that* intrusive. Take a look at Dave
Malcolm's systemtap patch, which IIRC takes a much more sensible
approach.

Regards

Antoine.



From solipsis at pitrou.net  Fri Sep  6 19:06:21 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 6 Sep 2013 19:06:21 +0200
Subject: [Python-Dev] Details of "import lock" in 3.3
References: <5229EFEC.1010503@jcea.es> <20130906174313.63e37ac3@pitrou.net>
 <5229FC98.9080106@jcea.es>
Message-ID: <20130906190621.6b1b8a01@fsol>

On Fri, 06 Sep 2013 18:02:32 +0200
Jesus Cea <jcea at jcea.es> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 06/09/13 17:43, Antoine Pitrou wrote:
> > Quick summary here: 
> > http://docs.python.org/3/whatsnew/3.3.html#a-finer-grained-import-lock
> >
> >  Otherwise, I'm afraid the source code has the most information,
> > e.g.: 
> > http://hg.python.org/cpython/file/1d88d04aade2/Lib/importlib/_bootstrap.py#l124
> 
> Yes,
> > 
> I was depending of the global import lock in my code. I am
> evaluating impact under Python 3.3.
> 
> Under Python 3.3, metafinder/loader MUST BE reentrant?.

No, they shouldn't have to. Normally, the global import lock is still
held at this point. However, it is best to check the code if you want
to be sure.

Regards

Antoine.



From cf.natali at gmail.com  Fri Sep  6 19:12:46 2013
From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=)
Date: Fri, 6 Sep 2013 19:12:46 +0200
Subject: [Python-Dev] DTRACE support
In-Reply-To: <5229FB08.20603@jcea.es>
References: <5229EFE7.8040702@jcea.es>
 <CAH_1eM2gG2fVsXU88j1T3y8TeGJ+fnDu4BBR2aMVg2hw=37_pg@mail.gmail.com>
 <5229FB08.20603@jcea.es>
Message-ID: <CAH_1eM3ex5PpNY7icxH2zCewB4LZ8nbwKFYcOwmP3iQBuhicBw@mail.gmail.com>

> The main value of DTrace is systemwide observability. You can see
> something "strange" at kernel level and trace it to a particular line
> of code in a random Python script. There is no other tool that can do
> that. You have complete transversal observability of ALL the code
> running in your computer, kernel or usermode, clean reports with
> threads, etc.

Don't get me wrong, I'm not saying DTrace is useless.
I'm just saying that, as far as I'm concerned, I've never had any
trouble debugging/tunning a Python script with non-intrusive tools
(strace, gdb, valgrind, and oprofile for profiling). Of course, this
includes analysing bug reports.

> Maybe the biggest objection would be that most python-devs are running
> Linux, and you don't have dtrace support on linux unless you are
> running Oracle distribution. But world is larger than linux, and there
> are some efforts to port DTrace to Linux itself. DTrace is available
> on Solaris and derivatives, MacOS X and FreeBSD.

That's true, I might have a different opinion if I used Solaris. But
that's not the case, so te me, the cognitive overhead incurred by this
large patch isn't worth it.

So I'm -1, but that's a personal opinion :-)

cf

From dcallahan at mozilla.com  Fri Sep  6 19:22:49 2013
From: dcallahan at mozilla.com (Dan Callahan)
Date: Fri, 06 Sep 2013 12:22:49 -0500
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <5228C00F.7000209@jcea.es>
References: <5228C00F.7000209@jcea.es>
Message-ID: <l0d311$b7n$1@ger.gmane.org>

On 9/5/13 12:31 PM, Jesus Cea wrote:
> I have big hopes for Mozilla Persona, looking forward
> Python infrastructure support :).

Hi, I'm the project lead on Persona signin, and I spoke at PyCon earlier 
this year regarding why and how Mozilla is building Persona. If you'd 
like some more background, that video [0] is worth a look.

Let's pull this discussion up a level:

It sounds like many people (Jesus, Donald, Toshio, Barry, Tres, Dirkjan, 
etc.) are interested in seeing Persona on Python.org properties, and 
most of the objections coming from a place of "Persona hasn't gone 
viral, what if this is wasted effort?"

We can tackle that from two angles:

1. Dirkjan and I are willing to do the work to make this happen if 
someone from python-devel is willing to guide us through the contributor 
process for these systems.

2. There's a seamless migration path away from Persona if we fail: fall 
back to the pre-existing traditional email/password system using the 
same email addresses that Persona had previously been in charge of 
verifying.

So let's do this. The open web deserves better than just Google+, 
Facebook, or Passwords, and visible support from the Python community 
would be a huge step toward answering the chicken-and-egg objections 
raised in this thread.

At your service,
-Callahad

PS: Freeform OpenID has utterly failed as a user-empowering 
authentication system, and the protocol itself is rapidly being 
supplanted by vendor-specific OAuth[1] systems. If we want to ensure 
that "you *can* (not *must*) use free and open services to access our 
resources," then we must provide an option to use something akin to Persona.

[0]: http://pyvideo.org/video/1764

[1]: "Google's OpenID service is being replaced by Login with OAuth 
2.0." https://developers.google.com/accounts/docs/GettingStarted


From ethan at stoneleaf.us  Fri Sep  6 19:01:32 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 06 Sep 2013 10:01:32 -0700
Subject: [Python-Dev] inspect and metaclasses
In-Reply-To: <20130906163728.ACFAD2507F1@webabinitio.net>
References: <5229D97E.9080103@stoneleaf.us>
 <CAMSv6X1k30-PAWct_9tDTmwir4mDO0jAs-mbrvvBrJc5XE6VTQ@mail.gmail.com>
 <5229F141.2050007@stoneleaf.us> <20130906154408.BFAAD250845@webabinitio.net>
 <5229FBC6.1020708@stoneleaf.us> <20130906163728.ACFAD2507F1@webabinitio.net>
Message-ID: <522A0A6C.9040401@stoneleaf.us>

On 09/06/2013 09:37 AM, R. David Murray wrote:
> On Fri, 06 Sep 2013 08:59:02 -0700, Ethan Furman <ethan at stoneleaf.us> wrote:
>>
>> For the short term I can restrict the change to inspect.classify_class_attrs().
>
> Sounds like the best course.

There is one other function in inspect that calls getmro():

def getmembers(object, predicate=None):
     """Return all members of an object as (name, value) pairs sorted by name.
     Optionally, only return members that satisfy a given predicate."""
     if isclass(object):
         mro = (object,) + getmro(object)

Should I add `+ getmro(type(object))` here as well?

--
~Ethan~

From guido at python.org  Fri Sep  6 20:01:39 2013
From: guido at python.org (Guido van Rossum)
Date: Fri, 6 Sep 2013 11:01:39 -0700
Subject: [Python-Dev] DTRACE support
In-Reply-To: <CAH_1eM3ex5PpNY7icxH2zCewB4LZ8nbwKFYcOwmP3iQBuhicBw@mail.gmail.com>
References: <5229EFE7.8040702@jcea.es>
 <CAH_1eM2gG2fVsXU88j1T3y8TeGJ+fnDu4BBR2aMVg2hw=37_pg@mail.gmail.com>
 <5229FB08.20603@jcea.es>
 <CAH_1eM3ex5PpNY7icxH2zCewB4LZ8nbwKFYcOwmP3iQBuhicBw@mail.gmail.com>
Message-ID: <CAP7+vJKWgsTv_xUB2MZftzeeo0MRfP6oQmS45_9P_brYMFxb=g@mail.gmail.com>

I think there are a couple of issues.

- In order for the patch to be acceptable, we'd need someone to take
responsibility for owning and maintaining it for at least several
years. Jesus, are you willing to commit to this?

- I think it's not all that important whether any core developer would
want to use this for themselves; it's enough if it's useful for others
to have this, and it's clear that at least *some* users really like
this. Different people like different tools, and that's fine.

- There are good reasons (e.g. standardizing a set of probes) for
having a somewhat supported version around.

- Since this *has* to be done in the form of a patch plus a new
configure option, an externally-maintained patch is awkward at least,
and likely to be continually out of date -- this is not something that
you can just as easily put on PyPI as the recommended 3rd party
solution.


- So I'm personally inclined to say "yes, Jesus, if you are willing to
maintain this code (while it is in the core) for several more years,
please invest more of your time, and, assuming you can satisfy the
code reviewers' feedback, we want to accept this in the core,
hopefully in 3.4 if you get it in an acceptable state before 3.4 beta
1 is released."

On Fri, Sep 6, 2013 at 10:12 AM, Charles-Fran?ois Natali
<cf.natali at gmail.com> wrote:
>> The main value of DTrace is systemwide observability. You can see
>> something "strange" at kernel level and trace it to a particular line
>> of code in a random Python script. There is no other tool that can do
>> that. You have complete transversal observability of ALL the code
>> running in your computer, kernel or usermode, clean reports with
>> threads, etc.
>
> Don't get me wrong, I'm not saying DTrace is useless.
> I'm just saying that, as far as I'm concerned, I've never had any
> trouble debugging/tunning a Python script with non-intrusive tools
> (strace, gdb, valgrind, and oprofile for profiling). Of course, this
> includes analysing bug reports.
>
>> Maybe the biggest objection would be that most python-devs are running
>> Linux, and you don't have dtrace support on linux unless you are
>> running Oracle distribution. But world is larger than linux, and there
>> are some efforts to port DTrace to Linux itself. DTrace is available
>> on Solaris and derivatives, MacOS X and FreeBSD.
>
> That's true, I might have a different opinion if I used Solaris. But
> that's not the case, so te me, the cognitive overhead incurred by this
> large patch isn't worth it.
>
> So I'm -1, but that's a personal opinion :-)
>
> cf
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)

From tjreedy at udel.edu  Fri Sep  6 20:24:57 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 06 Sep 2013 14:24:57 -0400
Subject: [Python-Dev] unicodedata module is out of date
In-Reply-To: <CAPMPPoRMBsBbAasu8ewEwqJOJARLAsctJricjKdVYVkXdLy8Tg@mail.gmail.com>
References: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com>
 <5229F6DB.40000@mrabarnett.plus.com>
 <CAPMPPoRMBsBbAasu8ewEwqJOJARLAsctJricjKdVYVkXdLy8Tg@mail.gmail.com>
Message-ID: <l0d6lj$lnf$1@ger.gmane.org>

On 9/6/2013 11:55 AM, Andrew Miller wrote:
> I've just checked on Python 2.7.5 and Python 3.3.2 (Win32 versions).
>
> In Python 3.3.2 unicodedata.unidata_version is set to '6.1.0'.
>
> In Python 2.7.5 it is set to '5.2.0' so it looks as though this version
> is no longer being updated.

In general, new features do not go into bugfix releases (x.y.z, where z 
 >= 1). Updating the unidate_version add new features to the unicodedata 
module.

-- 
Terry Jan Reedy


From solipsis at pitrou.net  Fri Sep  6 20:33:34 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 6 Sep 2013 20:33:34 +0200
Subject: [Python-Dev] DTRACE support
References: <5229EFE7.8040702@jcea.es> <20130906174111.446ea2c4@pitrou.net>
 <5229FF62.9020702@jcea.es>
Message-ID: <20130906203334.1a99c99d@fsol>

On Fri, 06 Sep 2013 18:14:26 +0200
Jesus Cea <jcea at jcea.es> wrote:
> 
> It is intrusive. Yes. I think it must be, by its own nature. Probably
> room for improvement and code transparency. But... are Python-DEVs
> interested in the project?. That is the point :)

As a concrete data point:
- here are Dave's modifications to ceval.c for systemtap:
http://bugs.python.org/review/14776/diff/5177/Python/ceval.c
- here are your modifications to ceval.c for dtrace:
http://bugs.python.org/review/13405/diff/6151/Python/ceval.c

Regards

Antoine.



From rdmurray at bitdance.com  Fri Sep  6 20:34:54 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Fri, 06 Sep 2013 14:34:54 -0400
Subject: [Python-Dev] inspect and metaclasses
In-Reply-To: <522A0A6C.9040401@stoneleaf.us>
References: <5229D97E.9080103@stoneleaf.us>
 <CAMSv6X1k30-PAWct_9tDTmwir4mDO0jAs-mbrvvBrJc5XE6VTQ@mail.gmail.com>
 <5229F141.2050007@stoneleaf.us> <20130906154408.BFAAD250845@webabinitio.net>
 <5229FBC6.1020708@stoneleaf.us> <20130906163728.ACFAD2507F1@webabinitio.net>
 <522A0A6C.9040401@stoneleaf.us>
Message-ID: <20130906183455.28546250816@webabinitio.net>

On Fri, 06 Sep 2013 10:01:32 -0700, Ethan Furman <ethan at stoneleaf.us> wrote:
> On 09/06/2013 09:37 AM, R. David Murray wrote:
> > On Fri, 06 Sep 2013 08:59:02 -0700, Ethan Furman <ethan at stoneleaf.us> wrote:
> >>
> >> For the short term I can restrict the change to inspect.classify_class_attrs().
> >
> > Sounds like the best course.
> 
> There is one other function in inspect that calls getmro():
> 
> def getmembers(object, predicate=None):
>      """Return all members of an object as (name, value) pairs sorted by name.
>      Optionally, only return members that satisfy a given predicate."""
>      if isclass(object):
>          mro = (object,) + getmro(object)
> 
> Should I add `+ getmro(type(object))` here as well?

Not unless you want to void the warranty on the docs :)

    Note getmembers() does not return metaclass attributes when the
    argument is a class (this behavior is inherited from the dir()
    function).

--David

From donald at stufft.io  Fri Sep  6 20:53:00 2013
From: donald at stufft.io (Donald Stufft)
Date: Fri, 6 Sep 2013 14:53:00 -0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <l0d311$b7n$1@ger.gmane.org>
References: <5228C00F.7000209@jcea.es> <l0d311$b7n$1@ger.gmane.org>
Message-ID: <6DB0A7E3-FADD-4E35-A983-02A7C0D4CAE3@stufft.io>


On Sep 6, 2013, at 1:22 PM, Dan Callahan <dcallahan at mozilla.com> wrote:

> On 9/5/13 12:31 PM, Jesus Cea wrote:
>> I have big hopes for Mozilla Persona, looking forward
>> Python infrastructure support :).
> 
> Hi, I'm the project lead on Persona signin, and I spoke at PyCon earlier this year regarding why and how Mozilla is building Persona. If you'd like some more background, that video [0] is worth a look.
> 
> Let's pull this discussion up a level:
> 
> It sounds like many people (Jesus, Donald, Toshio, Barry, Tres, Dirkjan, etc.) are interested in seeing Persona on Python.org properties, and most of the objections coming from a place of "Persona hasn't gone viral, what if this is wasted effort?"
> 
> We can tackle that from two angles:
> 
> 1. Dirkjan and I are willing to do the work to make this happen if someone from python-devel is willing to guide us through the contributor process for these systems.

FWIW I'm a maintainer of PyPI and I do plan on enabling Persona there. Mostly blocked because I want to focus my PyPI efforts on the "next gen" code base instead.

> 
> 2. There's a seamless migration path away from Persona if we fail: fall back to the pre-existing traditional email/password system using the same email addresses that Persona had previously been in charge of verifying.
> 
> So let's do this. The open web deserves better than just Google+, Facebook, or Passwords, and visible support from the Python community would be a huge step toward answering the chicken-and-egg objections raised in this thread.
> 
> At your service,
> -Callahad
> 
> PS: Freeform OpenID has utterly failed as a user-empowering authentication system, and the protocol itself is rapidly being supplanted by vendor-specific OAuth[1] systems. If we want to ensure that "you *can* (not *must*) use free and open services to access our resources," then we must provide an option to use something akin to Persona.
> 
> [0]: http://pyvideo.org/video/1764
> 
> [1]: "Google's OpenID service is being replaced by Login with OAuth 2.0." https://developers.google.com/accounts/docs/GettingStarted
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130906/d60c5f28/attachment.sig>

From rdmurray at bitdance.com  Fri Sep  6 21:11:29 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Fri, 06 Sep 2013 15:11:29 -0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <6DB0A7E3-FADD-4E35-A983-02A7C0D4CAE3@stufft.io>
References: <5228C00F.7000209@jcea.es> <l0d311$b7n$1@ger.gmane.org>
 <6DB0A7E3-FADD-4E35-A983-02A7C0D4CAE3@stufft.io>
Message-ID: <20130906191130.B6511250816@webabinitio.net>

On Fri, 06 Sep 2013 14:53:00 -0400, Donald Stufft <donald at stufft.io> wrote:
> 
> On Sep 6, 2013, at 1:22 PM, Dan Callahan <dcallahan at mozilla.com> wrote:
> 
> > On 9/5/13 12:31 PM, Jesus Cea wrote:
> >> I have big hopes for Mozilla Persona, looking forward
> >> Python infrastructure support :).
> > 
> > Hi, I'm the project lead on Persona signin, and I spoke at PyCon
> > earlier this year regarding why and how Mozilla is building Persona.
> > If you'd like some more background, that video [0] is worth a look.
> > 
> > Let's pull this discussion up a level:
> > 
> > It sounds like many people (Jesus, Donald, Toshio, Barry, Tres,
> > Dirkjan, etc.) are interested in seeing Persona on Python.org
> > properties, and most of the objections coming from a place of
> > "Persona hasn't gone viral, what if this is wasted effort?"
> > 
> > We can tackle that from two angles:
> > 
> > 1. Dirkjan and I are willing to do the work to make this happen if
> > someone from python-devel is willing to guide us through the
> > contributor process for these systems.

Thanks.

I'm one of the people with admin access to the bug tracker (I haven't
done much maint lately, though, Ezio has done the most).  There is
information on setting up a replica of our production system here:

    https://wiki.python.org/moin/TrackerDevelopment

If you want to start hacking on a solution, the first step would be to
spin up a test setup.

If you propose a patch, either I or Ezio should be able to find the time
to review and apply it, if you also commit to maintaining it ;)

Tracker specific discussion happens on the tracker-discuss mailing list,
by the way (very low traffic).

> > 2. There's a seamless migration path away from Persona if we fail:
> > fall back to the pre-existing traditional email/password system
> > using the same email addresses that Persona had previously been in
> > charge of verifying.

Roundup uses database-derived numeric IDs.  An email is associated with
each account, but does not participate in authentication or authorization
after initial signup.  (Except for the email interface...but that is a
separate story and you shouldn't need to address that).

> > So let's do this. The open web deserves better than just Google+,
> > Facebook, or Passwords, and visible support from the Python
> > community would be a huge step toward answering the chicken-and-egg
> > objections raised in this thread.
> > 
> > At your service,
> > -Callahad
> > 
> > PS: Freeform OpenID has utterly failed as a user-empowering
> > authentication system, and the protocol itself is rapidly being
> > supplanted by vendor-specific OAuth[1] systems. If we want to ensure
> > that "you *can* (not *must*) use free and open services to access
> > our resources," then we must provide an option to use something akin
> > to Persona.

IMO, single signon is overrated.  Especially if one prefers not to make
it easy for various accounts to be automatically associated with one
another by various entities who shall remain nameless but have been in
the news a lot lately :)

--David

From donald at stufft.io  Fri Sep  6 21:17:12 2013
From: donald at stufft.io (Donald Stufft)
Date: Fri, 6 Sep 2013 15:17:12 -0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <20130906191130.B6511250816@webabinitio.net>
References: <5228C00F.7000209@jcea.es> <l0d311$b7n$1@ger.gmane.org>
 <6DB0A7E3-FADD-4E35-A983-02A7C0D4CAE3@stufft.io>
 <20130906191130.B6511250816@webabinitio.net>
Message-ID: <7087E6B8-1CD4-4002-AD3F-5578B045A3CB@stufft.io>


On Sep 6, 2013, at 3:11 PM, "R. David Murray" <rdmurray at bitdance.com> wrote:

> IMO, single signon is overrated.  Especially if one prefers not to make
> it easy for various accounts to be automatically associated with one
> another by various entities who shall remain nameless but have been in
> the news a lot lately :)

If I recall Persona doesn't leak this data like OpenID does, but perhaps Dan can speak to that better than I can.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130906/224cd0ea/attachment.sig>

From rdmurray at bitdance.com  Fri Sep  6 21:34:30 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Fri, 06 Sep 2013 15:34:30 -0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <7087E6B8-1CD4-4002-AD3F-5578B045A3CB@stufft.io>
References: <5228C00F.7000209@jcea.es> <l0d311$b7n$1@ger.gmane.org>
 <6DB0A7E3-FADD-4E35-A983-02A7C0D4CAE3@stufft.io>
 <20130906191130.B6511250816@webabinitio.net>
 <7087E6B8-1CD4-4002-AD3F-5578B045A3CB@stufft.io>
Message-ID: <20130906193430.AFA14250816@webabinitio.net>

On Fri, 06 Sep 2013 15:17:12 -0400, Donald Stufft <donald at stufft.io> wrote:
> On Sep 6, 2013, at 3:11 PM, "R. David Murray" <rdmurray at bitdance.com> wrote:
> 
> > IMO, single signon is overrated.  Especially if one prefers not to make
> > it easy for various accounts to be automatically associated with one
> > another by various entities who shall remain nameless but have been in
> > the news a lot lately :)
> 
> If I recall Persona doesn't leak this data like OpenID does, but
> perhaps Dan can speak to that better than I can.

Note that I said that single signon *itself* was overrated.  If you use
the same token to authenticate to multiple sites (and here the 'token'
is the email address) then your identities on those sites are ipso facto
associated with each other.  *If* that email address is also never
leaked (never displayed, even to other signed on users, all communication
with the site encrypted), then you only have to worry if the
sites exchange information about their accounts, or if the government
comes knocking on their doors....

Yes, I'm paranoid.  That doesn't mean they aren't listening.

That said, sometimes you *want* identities to be associated, so I'm not
saying Persona is a bad thing.  Just that single signon is overrated.

--David

From donald at stufft.io  Fri Sep  6 21:40:33 2013
From: donald at stufft.io (Donald Stufft)
Date: Fri, 6 Sep 2013 15:40:33 -0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <20130906193430.AFA14250816@webabinitio.net>
References: <5228C00F.7000209@jcea.es> <l0d311$b7n$1@ger.gmane.org>
 <6DB0A7E3-FADD-4E35-A983-02A7C0D4CAE3@stufft.io>
 <20130906191130.B6511250816@webabinitio.net>
 <7087E6B8-1CD4-4002-AD3F-5578B045A3CB@stufft.io>
 <20130906193430.AFA14250816@webabinitio.net>
Message-ID: <ADFAA327-A61E-42AB-8981-87645953FA95@stufft.io>


On Sep 6, 2013, at 3:34 PM, "R. David Murray" <rdmurray at bitdance.com> wrote:

> On Fri, 06 Sep 2013 15:17:12 -0400, Donald Stufft <donald at stufft.io> wrote:
>> On Sep 6, 2013, at 3:11 PM, "R. David Murray" <rdmurray at bitdance.com> wrote:
>> 
>>> IMO, single signon is overrated.  Especially if one prefers not to make
>>> it easy for various accounts to be automatically associated with one
>>> another by various entities who shall remain nameless but have been in
>>> the news a lot lately :)
>> 
>> If I recall Persona doesn't leak this data like OpenID does, but
>> perhaps Dan can speak to that better than I can.
> 
> Note that I said that single signon *itself* was overrated.  If you use
> the same token to authenticate to multiple sites (and here the 'token'
> is the email address) then your identities on those sites are ipso facto
> associated with each other.  *If* that email address is also never
> leaked (never displayed, even to other signed on users, all communication
> with the site encrypted), then you only have to worry if the
> sites exchange information about their accounts, or if the government
> comes knocking on their doors....
> 
> Yes, I'm paranoid.  That doesn't mean they aren't listening.
> 
> That said, sometimes you *want* identities to be associated, so I'm not
> saying Persona is a bad thing.  Just that single signon is overrated.

Well that's fine to have that opinion but I think you're under estimating just
how easy it is to link two disparate accounts especially if you have the
cooperation (willing or otherwise) of the site operators. I've personally
seen Google do some particularly amazing connections between accounts
that I don't believe using the same authentication token is going to make
that any easier or harder for them.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130906/af163ab9/attachment.sig>

From catch-all at masklinn.net  Fri Sep  6 21:50:38 2013
From: catch-all at masklinn.net (Xavier Morel)
Date: Fri, 6 Sep 2013 21:50:38 +0200
Subject: [Python-Dev] DTRACE support
In-Reply-To: <20130906190516.471028ba@fsol>
References: <5229EFE7.8040702@jcea.es> <20130906174111.446ea2c4@pitrou.net>
 <5229FF62.9020702@jcea.es> <20130906190516.471028ba@fsol>
Message-ID: <8E3B67D3-367C-440E-9268-87845E44BB15@masklinn.net>

On 2013-09-06, at 19:05 , Antoine Pitrou wrote:

> On Fri, 06 Sep 2013 18:14:26 +0200
> Jesus Cea <jcea at jcea.es> wrote:
>> 
>>> Right now, I agree with Charles-Fran?ois: your patch is too
>>> intrusive.
>> 
>> It is intrusive. Yes. I think it must be, by its own nature. Probably
>> room for improvement and code transparency. But... are Python-DEVs
>> interested in the project?. That is the point :)
> 
> Well, I'm not *personally* interested in anything that only addresses
> Solaris, OS X and the like :)

For what it's worth, there's also a linux port and oracle's distro has
dtrace support.

> And, no, it doesn't have to be *that* intrusive. Take a look at Dave
> Malcolm's systemtap patch, which IIRC takes a much more sensible
> approach.

Is there a possibility of compatibility there, using the same
placeholders for a --with-dtrace and --with-systemtap build?

Jesus seems to instrument more points than Dave, but the extra points
could just be defined to nothing in the systemtap implementation.

From jcea at jcea.es  Sat Sep  7 05:40:04 2013
From: jcea at jcea.es (Jesus Cea)
Date: Sat, 07 Sep 2013 05:40:04 +0200
Subject: [Python-Dev] DTRACE support
In-Reply-To: <20130906203334.1a99c99d@fsol>
References: <5229EFE7.8040702@jcea.es> <20130906174111.446ea2c4@pitrou.net>
 <5229FF62.9020702@jcea.es> <20130906203334.1a99c99d@fsol>
Message-ID: <522AA014.4060801@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/09/13 20:33, Antoine Pitrou wrote:
> On Fri, 06 Sep 2013 18:14:26 +0200 Jesus Cea <jcea at jcea.es> wrote:
>> 
>> It is intrusive. Yes. I think it must be, by its own nature.
>> Probably room for improvement and code transparency. But... are
>> Python-DEVs interested in the project?. That is the point :)
> 
> As a concrete data point: - here are Dave's modifications to
> ceval.c for systemtap: 
> http://bugs.python.org/review/14776/diff/5177/Python/ceval.c - here
> are your modifications to ceval.c for dtrace: 
> http://bugs.python.org/review/13405/diff/6151/Python/ceval.c

Unfair, because that code is not doing the same thing.

Most of the extra complexity is there to deal with DTRACE ability to
provide meaningful stackframes, with Python code instead of CPython
evaluation loop. This is kind of magical.

Look at this:

""
[root at stargate-host /]# ps -lAf|grep -i correo
 0 S   correo  1371     1   0  40 20        ? 277063        ?   Jul 29
?          90:43 /usr/local/bin/python /export/home/
 0 S     root 20397 20373   0  40 20        ?   1294        ? 05:18:16
pts/12      0:00 grep -i correo

[root at stargate-host /]# dtrace -n 'python1371:::function-entry
{jstack();}'

  5   3164 PyEval_EvalFrameEx:function-entry
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x89a
                [ /usr/local/lib/python2.7/logging/__init__.py:1332
(isEnabledFor) ]
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231
                [ /usr/local/lib/python2.7/logging/__init__.py:1202
(log) ]
              libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x59ae
                [ /home/correo/durus/connection.py:483 (shrink) ]
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231
                [ /home/correo/durus/connection.py:225 (shrink_cache) ]
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231
                [ /home/correo/durus/connection.py:303 (commit2) ]
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231
                [ /home/correo/durus/connection.py:261 (commit) ]
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231
                [ /export/home/correo/lmtp.py:191 (_monitor) ]
              libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x59ae
                [ /export/home/correo/lmtp.py:125 (process_vacation2) ]
              libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x59ae
                [ /export/home/correo/lmtp.py:138 (process_vacation) ]
              libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4
              libpython2.7.so.1.0`function_call+0x192
              libpython2.7.so.1.0`PyObject_Call+0x5c
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x2b94
                [ /usr/local/lib/python2.7/threading.py: ]
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231
              libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4
              libpython2.7.so.1.0`function_call+0xa4
              libpython2.7.so.1.0`PyObject_Call+0x5c
              libpython2.7.so.1.0`instancemethod_call+0xa1
              libpython2.7.so.1.0`PyObject_Call+0x5c
              libpython2.7.so.1.0`PyEval_CallObjectWithKeywords+0x58
              libpython2.7.so.1.0`t_bootstrap+0x52
              libc.so.1`_thr_setup+0x4e
              libc.so.1`_lwp_start

^C
"""

Lets say I want to know about what codepaths are hitting the OS "sync"
features, so I do this:

"""
[root at stargate-host /]# dtrace -n
'syscall::fdsync:entry/pid==1371/{jstack();}'
dtrace: description 'syscall::fdsync:entry' matched 1 probe

CPU     ID                    FUNCTION:NAME
  3  58344                     fdsync:entry
              libc.so.1`__fdsync+0x15
              libdb-6.0.so`__os_fsync+0x91
              libdb-6.0.so`__log_flush_int+0x685
              libdb-6.0.so`__log_flush+0x6a
              libdb-6.0.so`__log_flush_pp+0x121
              _pybsddb.so`DBEnv_log_flush+0x39
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6992
                [
/export/home/correo/durus-berkeleydbstorage/berkeleydb_storage.py:918
(flush) ]
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231
                [ /export/home/correo/lmtp.py:1005 (lmtp2) ]
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231
                [ /export/home/correo/lmtp.py:763 (lmtp) ]
              libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4
              libpython2.7.so.1.0`function_call+0x192
              libpython2.7.so.1.0`PyObject_Call+0x5c
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x2b94
                [ /export/home/correo/lmtp.py:214
(_ignore_socket_errors) ]
              libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4
              libpython2.7.so.1.0`function_call+0x192
              libpython2.7.so.1.0`PyObject_Call+0x5c
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x2b94
                [ /usr/local/lib/python2.7/threading.py:504 (run) ]
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231
                [ /usr/local/lib/python2.7/threading.py:551
(__bootstrap_inner) ]
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231
                [ /usr/local/lib/python2.7/threading.py:524
(__bootstrap) ]
              libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4
              libpython2.7.so.1.0`function_call+0xa4
              libpython2.7.so.1.0`PyObject_Call+0x5c
              libpython2.7.so.1.0`instancemethod_call+0xa1
              libpython2.7.so.1.0`PyObject_Call+0x5c
              libpython2.7.so.1.0`PyEval_CallObjectWithKeywords+0x58
              libpython2.7.so.1.0`t_bootstrap+0x52
              libc.so.1`_thr_setup+0x4e
              libc.so.1`_lwp_start
"""

I could even aggregate stacktraces and print them by frequency, cost,
whatever.

Be aware that to provide this information I am not even using the
probes I have integrated in Python. The program is in production and I
don't need to stop it to make those measurements. They are 100%
"live", not interrupting the service and without any kind of support
in that Python program.

I can even plug to TCP Dtrace providers in kernel and know what exact
line of Python code is sending those bytes that are being delayed
because the Nagle algorithm
<https://en.wikipedia.org/wiki/Nagle%27s_algorithm>. I actually digged
this five years ago. what I hunt it was! :-)

PS: And it is supporting Unicode :-), another big complexity in the code.

- -- 
Jes?s Cea Avi?n                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQCUAwUBUiqgFJlgi5GaxT1NAQLezQP2PRkvn6+9Zev6GlGu14hfYTpwAMie9gcn
tEieelaQniisLy22oj/MO2d2j78qlH4RoU3U/vzOTPru6uzt3OKy0UBivnBFqTcc
fD/dPUCcfkigZO4nBzcqjAP4lz93wcDkx1Y8RcL3ygxeI6fSe8JKdyZKyXvyRf/9
f+3MfWJVDA==
=29tQ
-----END PGP SIGNATURE-----

From python-dev at masklinn.net  Sat Sep  7 08:57:07 2013
From: python-dev at masklinn.net (Xavier Morel)
Date: Sat, 7 Sep 2013 08:57:07 +0200
Subject: [Python-Dev] DTRACE support
In-Reply-To: <522AA014.4060801@jcea.es>
References: <5229EFE7.8040702@jcea.es> <20130906174111.446ea2c4@pitrou.net>
 <5229FF62.9020702@jcea.es> <20130906203334.1a99c99d@fsol>
 <522AA014.4060801@jcea.es>
Message-ID: <557641DC-8E6B-450C-88A0-A3E2941678C7@masklinn.net>


On 2013-09-07, at 05:40 , Jesus Cea wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 06/09/13 20:33, Antoine Pitrou wrote:
>> On Fri, 06 Sep 2013 18:14:26 +0200 Jesus Cea <jcea at jcea.es> wrote:
>>> 
>>> It is intrusive. Yes. I think it must be, by its own nature.
>>> Probably room for improvement and code transparency. But... are
>>> Python-DEVs interested in the project?. That is the point :)
>> 
>> As a concrete data point: - here are Dave's modifications to
>> ceval.c for systemtap: 
>> http://bugs.python.org/review/14776/diff/5177/Python/ceval.c - here
>> are your modifications to ceval.c for dtrace: 
>> http://bugs.python.org/review/13405/diff/6151/Python/ceval.c
> 
> Unfair, because that code is not doing the same thing.
> 
> Most of the extra complexity is there to deal with DTRACE ability to
> provide meaningful stackframes, with Python code instead of CPython
> evaluation loop. This is kind of magical.

Antoine will correct me if I'm wrong, but I believe his point is less
about the complexity of dtrace provision and more about how much of it
lives in ceval.c: the systemtap provision also takes quite a bit of
code, but almost all of that code is extracted into a separate file and
only a pair of calls live in ceval.c

You patch, because it adds quite a bit of complexity to ceval.c, makes
reading it significantly more difficult (especially for people who don't
care for probe implementations). Dave's more or less doesn't change the
complexity of that file (people who don't care for probes don't have to
follow the macros to know what they do).

From solipsis at pitrou.net  Sat Sep  7 11:06:13 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 7 Sep 2013 11:06:13 +0200
Subject: [Python-Dev] DTRACE support
References: <5229EFE7.8040702@jcea.es> <20130906174111.446ea2c4@pitrou.net>
 <5229FF62.9020702@jcea.es> <20130906203334.1a99c99d@fsol>
 <522AA014.4060801@jcea.es>
 <557641DC-8E6B-450C-88A0-A3E2941678C7@masklinn.net>
Message-ID: <20130907110613.71a9a98d@fsol>

On Sat, 7 Sep 2013 08:57:07 +0200
Xavier Morel <python-dev at masklinn.net> wrote:
> 
> On 2013-09-07, at 05:40 , Jesus Cea wrote:
> 
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> > 
> > On 06/09/13 20:33, Antoine Pitrou wrote:
> >> On Fri, 06 Sep 2013 18:14:26 +0200 Jesus Cea <jcea at jcea.es> wrote:
> >>> 
> >>> It is intrusive. Yes. I think it must be, by its own nature.
> >>> Probably room for improvement and code transparency. But... are
> >>> Python-DEVs interested in the project?. That is the point :)
> >> 
> >> As a concrete data point: - here are Dave's modifications to
> >> ceval.c for systemtap: 
> >> http://bugs.python.org/review/14776/diff/5177/Python/ceval.c - here
> >> are your modifications to ceval.c for dtrace: 
> >> http://bugs.python.org/review/13405/diff/6151/Python/ceval.c
> > 
> > Unfair, because that code is not doing the same thing.
> > 
> > Most of the extra complexity is there to deal with DTRACE ability to
> > provide meaningful stackframes, with Python code instead of CPython
> > evaluation loop. This is kind of magical.
> 
> Antoine will correct me if I'm wrong, but I believe his point is less
> about the complexity of dtrace provision and more about how much of it
> lives in ceval.c: the systemtap provision also takes quite a bit of
> code, but almost all of that code is extracted into a separate file and
> only a pair of calls live in ceval.c

Yes, that's exactly my point. There can be an arbitrarily complex
ceval-dtrace.h for all I care :-)

Thanks for the examples, by the way.

Regards

Antoine.



From ncoghlan at gmail.com  Sat Sep  7 13:01:36 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 7 Sep 2013 21:01:36 +1000
Subject: [Python-Dev] when to fix cross-version bugs?
In-Reply-To: <522A0044.10302@stoneleaf.us>
References: <5229EBDA.1030902@stoneleaf.us>
	<522A0044.10302@stoneleaf.us>
Message-ID: <CADiSq7fy1to0quWH8v_2-RaC645PxJb4C32_u7yjPJkT7gMV=Q@mail.gmail.com>

On 7 Sep 2013 02:43, "Ethan Furman" <ethan at stoneleaf.us> wrote:
>
> On 09/06/2013 07:51 AM, Ethan Furman wrote:
>>
>>
>> What guidelines determine when a bug is fixed in previous versions?
>
>
> On 09/06/2013 08:29 AM, Brian Curtin wrote:
>>
>>
>> If it's a bug in that version and the version is accepting bug fixes,
>> i.e., not in security mode, go for it. This includes crossing the 2/3
>> boundary if applicable.
>
>
> On 09/06/2013 08:31 AM, R. David Murray wrote:
>>
>>
>> The basic guideline is: we try very hard not to break currently working
>> code in a maintenance release.  Making that decision very much depends on
>> the details of each individual case.
>>
>> [so] it probably doesn't have much positive impact if it does get
>>
>> backported, so is it worth the (small) chance of breaking someone's code?
>
>
> And they say never go to the elves for advice!  ;)

Fixes that risk breaking current doctests are rarely worth backporting, so
in this case, I'd say only fixing it in 3.4 is the better option.

Definitely the kind of borderline case worth asking about, though :)

Cheers,
Nick.

>
>
> --
> ~Ethan~
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130907/d8ca9aff/attachment.html>

From ncoghlan at gmail.com  Sat Sep  7 16:14:01 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 8 Sep 2013 00:14:01 +1000
Subject: [Python-Dev] DTRACE support
In-Reply-To: <20130907110613.71a9a98d@fsol>
References: <5229EFE7.8040702@jcea.es> <20130906174111.446ea2c4@pitrou.net>
 <5229FF62.9020702@jcea.es> <20130906203334.1a99c99d@fsol>
 <522AA014.4060801@jcea.es>
 <557641DC-8E6B-450C-88A0-A3E2941678C7@masklinn.net>
 <20130907110613.71a9a98d@fsol>
Message-ID: <CADiSq7epwhPEO+X16SuTOkH3wETVprbWx5h4f1g9miK9sNQhuQ@mail.gmail.com>

On 7 September 2013 19:06, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sat, 7 Sep 2013 08:57:07 +0200
> Xavier Morel <python-dev at masklinn.net> wrote:
>>
>> On 2013-09-07, at 05:40 , Jesus Cea wrote:
>>
>> > -----BEGIN PGP SIGNED MESSAGE-----
>> > Hash: SHA1
>> >
>> > On 06/09/13 20:33, Antoine Pitrou wrote:
>> >> On Fri, 06 Sep 2013 18:14:26 +0200 Jesus Cea <jcea at jcea.es> wrote:
>> >>>
>> >>> It is intrusive. Yes. I think it must be, by its own nature.
>> >>> Probably room for improvement and code transparency. But... are
>> >>> Python-DEVs interested in the project?. That is the point :)
>> >>
>> >> As a concrete data point: - here are Dave's modifications to
>> >> ceval.c for systemtap:
>> >> http://bugs.python.org/review/14776/diff/5177/Python/ceval.c - here
>> >> are your modifications to ceval.c for dtrace:
>> >> http://bugs.python.org/review/13405/diff/6151/Python/ceval.c
>> >
>> > Unfair, because that code is not doing the same thing.
>> >
>> > Most of the extra complexity is there to deal with DTRACE ability to
>> > provide meaningful stackframes, with Python code instead of CPython
>> > evaluation loop. This is kind of magical.
>>
>> Antoine will correct me if I'm wrong, but I believe his point is less
>> about the complexity of dtrace provision and more about how much of it
>> lives in ceval.c: the systemtap provision also takes quite a bit of
>> code, but almost all of that code is extracted into a separate file and
>> only a pair of calls live in ceval.c
>
> Yes, that's exactly my point. There can be an arbitrarily complex
> ceval-dtrace.h for all I care :-)
>
> Thanks for the examples, by the way.

Yep, complex stuff inline should be abstracted out so it doesn't
affect the code flow for those of us that don't care :)

Between Jesus, the Mac OS X using core devs and the Solaris and Mac OS
X buildbots it should be a maintainable addition.

However, it seems worthwhile to write up a short PEP (akin to the
tracemalloc PEP) to scope out the suggestion.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ericsnowcurrently at gmail.com  Sun Sep  8 00:22:36 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Sat, 7 Sep 2013 16:22:36 -0600
Subject: [Python-Dev] Pre-PEP: Redesigning extension modules
In-Reply-To: <CADiSq7c+R92r17c8g3ygW0T4p=VG8LiSf5XSGuOJsFHrOtOJDg@mail.gmail.com>
References: <kv77o2$f96$1@ger.gmane.org>
 <CALeMXf5oxi0oX13zSfHwgVi1eXF_z_coY23MfBP-YOK8WVkWsg@mail.gmail.com>
 <kvctoj$qsv$1@ger.gmane.org> <kvstdj$jo0$1@ger.gmane.org>
 <CADiSq7c5xtEcPBXGPWqQosfgGcb2n+2AnHiAWFoATeYfQTUcEQ@mail.gmail.com>
 <20130902101642.04b69588@pitrou.net>
 <CADiSq7c+R92r17c8g3ygW0T4p=VG8LiSf5XSGuOJsFHrOtOJDg@mail.gmail.com>
Message-ID: <CALFfu7C46gXvF37_=n2usyihCo1yUoRHS6QSx5OBChm9hb7dAQ@mail.gmail.com>

On Mon, Sep 2, 2013 at 7:02 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> The hook API I currently have in mind is a two step initialisation:
>
>     PyImport_PrepareNAME (optional)
>     PyImport_ExecNAME
>

Should we also look at an API change for the initfunc() of PyImport_Inittab
entries?  Currently the function takes a module name, which doesn't jive
with loader.exec_module() taking a module.  I noticed this while adding an
exec_module() to BuiltinImporter.  I suppose the same thing goes
for PyImport_ImportFrozenModuleObject().

-eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130907/73f9cab0/attachment.html>

From stefan_ml at behnel.de  Sun Sep  8 09:09:05 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 08 Sep 2013 09:09:05 +0200
Subject: [Python-Dev] Pre-PEP: Redesigning extension modules
In-Reply-To: <CALFfu7C46gXvF37_=n2usyihCo1yUoRHS6QSx5OBChm9hb7dAQ@mail.gmail.com>
References: <kv77o2$f96$1@ger.gmane.org>
 <CALeMXf5oxi0oX13zSfHwgVi1eXF_z_coY23MfBP-YOK8WVkWsg@mail.gmail.com>
 <kvctoj$qsv$1@ger.gmane.org> <kvstdj$jo0$1@ger.gmane.org>
 <CADiSq7c5xtEcPBXGPWqQosfgGcb2n+2AnHiAWFoATeYfQTUcEQ@mail.gmail.com>
 <20130902101642.04b69588@pitrou.net>
 <CADiSq7c+R92r17c8g3ygW0T4p=VG8LiSf5XSGuOJsFHrOtOJDg@mail.gmail.com>
 <CALFfu7C46gXvF37_=n2usyihCo1yUoRHS6QSx5OBChm9hb7dAQ@mail.gmail.com>
Message-ID: <l0h7q9$nm2$1@ger.gmane.org>

Eric Snow, 08.09.2013 00:22:
> On Mon, Sep 2, 2013 at 7:02 AM, Nick Coghlan wrote:
> 
>> The hook API I currently have in mind is a two step initialisation:
>>
>>     PyImport_PrepareNAME (optional)
>>     PyImport_ExecNAME
>>
> 
> Should we also look at an API change for the initfunc() of PyImport_Inittab
> entries?  Currently the function takes a module name, which doesn't jive
> with loader.exec_module() taking a module.  I noticed this while adding an
> exec_module() to BuiltinImporter.  I suppose the same thing goes
> for PyImport_ImportFrozenModuleObject().

Is it still the case that the inittab mechanism only works for the
embedding case? It would be nice to have a declarative mechanism for
registering a set of modules from a running module init function.

Stefan



From ethan at stoneleaf.us  Sun Sep  8 09:42:20 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 08 Sep 2013 00:42:20 -0700
Subject: [Python-Dev] metaclasses, classes, instances,
	and proper nomenclature
Message-ID: <522C2A5C.4030708@stoneleaf.us>

I've run across two different ways to think about this:

   1) the type of the first argument

   2) where the method/attribute lives

Since attributes don't take a first argument they default to 2:  an instance attribute lives in the instance, a class 
attribute lives in the class, and a metaclass attribute lives in the metaclass.

Methods, on the other hand, do take a first argument:  an instance method takes itself, a class method takes the class, 
and a metaclass method takes the metaclass.

Going by option 1 above there is only one way to get an instance method, and only one way to get a metaclass method -- 
calling with the instance (either directly or indirectly via the class), or calling a metaclass method that has been 
marked as a @classmethod.

Therein lies my confusion.

class Meta(type):

     @classmethod
     def meta_method(mcls):
         print("I'm a metaclass method!")

     def cls_method1(cls):
         print("I'm a class method!  Aren't I?")

class Class(metaclass=Meta):

     @classmethod
     def cls_method2(cls):
         print("I'm a class method for sure!")

     def instance_method(self):
         print("And I'm a regular ol' instance method")


So, is Meta.cls_method1 a class method?  On the one hand, it takes the class as it's first parameter, on the other hand 
it lives in the metaclass.  And on the third hand you can't get to it from the instance Class().


If you're wondering why this is posted to PyDev, the related question is this:  What is the proper role of a metaclass? 
  Should it basically fiddle with the class creation process and then get out of the way?  The case in point is, of 
course, Enum.  Currently it has a custom __getattr__, but it lives in the metaclass, EnumMeta.  Now this is handy, 
because it means that Color.red.blue raises an AttributeError, where if __getattr__ lived in Enum itself that would 
work.  It also has the __members__ attribute living in EnumMeta, which means it's not accessible from Color.red.  In 
other words, EnumMeta is not getting out the way, it is still very much involved.  On the one hand, that's cool; on the 
other hand, I hand to think hard to figure out why Color.red.blue was not getting routed through EnumMeta's __getattr__, 
but was instead getting routed through object.__getattr__.

--
~Ethan~

From ncoghlan at gmail.com  Sun Sep  8 12:06:17 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 8 Sep 2013 20:06:17 +1000
Subject: [Python-Dev] metaclasses, classes, instances,
	and proper nomenclature
In-Reply-To: <522C2A5C.4030708@stoneleaf.us>
References: <522C2A5C.4030708@stoneleaf.us>
Message-ID: <CADiSq7eHZg_dgi3+n=v6452iQ+AwzN1kvr=P6XnBSDem+NdqBA@mail.gmail.com>

On 8 Sep 2013 18:38, "Ethan Furman" <ethan at stoneleaf.us> wrote:
>
> I've run across two different ways to think about this:
>
>   1) the type of the first argument
>
>   2) where the method/attribute lives
>
> Since attributes don't take a first argument they default to 2:  an
instance attribute lives in the instance, a class attribute lives in the
class, and a metaclass attribute lives in the metaclass.
>
> Methods, on the other hand, do take a first argument:  an instance method
takes itself, a class method takes the class, and a metaclass method takes
the metaclass.

No, there's no such thing as a "metaclass method".

Metaclass instance methods are equivalent to hidden class methods - they
don't appear in dir() and can't be accessed through instances of the class.

It's only class methods on the metaclass that receive that rather than the
class object (__new__ is technically a static method, but still accepts the
metaclass as the first argument).

> Going by option 1 above there is only one way to get an instance method,
and only one way to get a metaclass method -- calling with the instance
(either directly or indirectly via the class), or calling a metaclass
method that has been marked as a @classmethod.
>
> Therein lies my confusion.
>
> class Meta(type):
>
>     @classmethod
>     def meta_method(mcls):
>         print("I'm a metaclass method!")
>
>     def cls_method1(cls):
>         print("I'm a class method!  Aren't I?")
>
> class Class(metaclass=Meta):
>
>     @classmethod
>     def cls_method2(cls):
>         print("I'm a class method for sure!")
>
>     def instance_method(self):
>         print("And I'm a regular ol' instance method")
>
>
> So, is Meta.cls_method1 a class method?  On the one hand, it takes the
class as it's first parameter, on the other hand it lives in the metaclass.
 And on the third hand you can't get to it from the instance Class().

It's a hidden class method.

> If you're wondering why this is posted to PyDev, the related question is
this:  What is the proper role of a metaclass?  Should it basically fiddle
with the class creation process and then get out of the way?  The case in
point is, of course, Enum.  Currently it has a custom __getattr__, but it
lives in the metaclass, EnumMeta.  Now this is handy, because it means that
Color.red.blue raises an AttributeError, where if __getattr__ lived in Enum
itself that would work.  It also has the __members__ attribute living in
EnumMeta, which means it's not accessible from Color.red.  In other words,
EnumMeta is not getting out the way, it is still very much involved.  On
the one hand, that's cool; on the other hand, I hand to think hard to
figure out why Color.red.blue was not getting routed through EnumMeta's
__getattr__, but was instead getting routed through object.__getattr__.

This is exactly how a metaclass is intended to be used - to affect the
behaviour of the class without affecting the behaviour of instances.

And yes, introspection does get a little interesting when a non-trivial
metaclass is in play :)

Cheers,
Nick.

>
> --
> ~Ethan~
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130908/ee9f1ed2/attachment.html>

From solipsis at pitrou.net  Sun Sep  8 13:23:37 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 8 Sep 2013 13:23:37 +0200
Subject: [Python-Dev] cpython: Issue #18904: test_socket: add
 inheritance tests using fcntl and FD_CLOEXEC
References: <3cXns46D3sz7LjR@mail.python.org>
Message-ID: <20130908132337.69634518@fsol>

On Sun,  8 Sep 2013 11:54:00 +0200 (CEST)
victor.stinner <python-checkins at python.org> wrote:
> http://hg.python.org/cpython/rev/b7f6f6f59e91
> changeset:   85619:b7f6f6f59e91
> user:        Victor Stinner <victor.stinner at gmail.com>
> date:        Sun Sep 08 11:53:09 2013 +0200
> summary:
>   Issue #18904: test_socket: add inheritance tests using fcntl and FD_CLOEXEC
> 
[...]
>  
> +    if fcntl:
> +        def test_get_inheritable_cloexec(self):

The right way to do this would be to skip the test if fcntl doesn't
exist.

Regards

Antoine.



From steve at pearwood.info  Sun Sep  8 14:37:06 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 8 Sep 2013 22:37:06 +1000
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <52215BDF.2090109@pearwood.info>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
Message-ID: <20130908123706.GG26319@ando>

<bump>

I'd like to get some attention for this please.


On Sat, Aug 31, 2013 at 12:58:39PM +1000, Steven D'Aprano wrote:
> Hi all,
> 
> 
> I think that PEP 450 is now ready for a PEP dictator. There have been a 
> number of code reviews, and feedback has been taken into account. The test 
> suite passes. I'm not aware of any unanswered issues with the code. At 
> least two people other than myself think that the implementation is ready 
> for a dictator, and nobody has objected.
> 
> There is still on-going work on speeding up the implementation for the 
> statistics.sum function, but that will not effect the interface or the 
> substantially change the test suite.
> 
> http://bugs.python.org/issue18606
> http://www.python.org/dev/peps/pep-0450/
> 
> 
> 
> 
> -- 
> Steven
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/steve%40pearwood.info
> 

From victor.stinner at gmail.com  Sun Sep  8 15:38:01 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sun, 8 Sep 2013 15:38:01 +0200
Subject: [Python-Dev] cpython: Issue #18904: test_socket: add
 inheritance tests using fcntl and FD_CLOEXEC
In-Reply-To: <20130908132337.69634518@fsol>
References: <3cXns46D3sz7LjR@mail.python.org> <20130908132337.69634518@fsol>
Message-ID: <CAMpsgwa4tdO+jOmH5b=gCDD7fZxgqD6NCWd5jCXtncp1NbJ6YA@mail.gmail.com>

2013/9/8 Antoine Pitrou <solipsis at pitrou.net>:
> On Sun,  8 Sep 2013 11:54:00 +0200 (CEST)
> victor.stinner <python-checkins at python.org> wrote:
>> http://hg.python.org/cpython/rev/b7f6f6f59e91
>> changeset:   85619:b7f6f6f59e91
>> user:        Victor Stinner <victor.stinner at gmail.com>
>> date:        Sun Sep 08 11:53:09 2013 +0200
>> summary:
>>   Issue #18904: test_socket: add inheritance tests using fcntl and FD_CLOEXEC
>>
> [...]
>>
>> +    if fcntl:
>> +        def test_get_inheritable_cloexec(self):
>
> The right way to do this would be to skip the test if fcntl doesn't
> exist.

Ok.

First, I added two methods to get and set the FD_CLOXEC flag. Later, I
inlined these methods because it makes the code more readable.

I modified the tests to use @unittest.skipIf decorator:

New changeset aea58e1cae75 by Victor Stinner in branch 'default':
Issue #18904: test_os and test_socket use unittest.skipIf() to check if fcntl
http://hg.python.org/cpython/rev/aea58e1cae75

Victor

From victor.stinner at gmail.com  Sun Sep  8 16:03:43 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sun, 8 Sep 2013 16:03:43 +0200
Subject: [Python-Dev] RFC: PEP 454: Add a new tracemalloc module
In-Reply-To: <CAMpsgwZTOBdXy2Rcxs0itnPZYZgHU4X4tPNf5syoZrOJ5gYCFw@mail.gmail.com>
References: <CAMpsgwZTOBdXy2Rcxs0itnPZYZgHU4X4tPNf5syoZrOJ5gYCFw@mail.gmail.com>
Message-ID: <CAMpsgwZ3SoU-FuarRT_73yJ_VfFVoCBQ7phzcRQ+g_oFaYQCfA@mail.gmail.com>

2013/9/4 Victor Stinner <victor.stinner at gmail.com>:
> http://www.python.org/dev/peps/pep-0454/
>
> PEP: 454
> Title: Add a new tracemalloc module to trace Python memory allocations
> Version: $Revision$
> Last-Modified: $Date$
> Author: Victor Stinner <victor.stinner at gmail.com>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 3-September-2013
> Python-Version: 3.4

I added a function get_tracemalloc_size() to see how much memory is
used by the tracemalloc module itself. Result on the Python test
suite:

* 1 frame: +52% (+%68%)
  Python=34 MiB; _tracemalloc=18 MiB, tracemalloc.py=5 MiB
* 10 frames: +155% (+170%)
  Python=34 MiB, _tracemalloc=53 MiB, tracemalloc.py=5 MiB
* 100 frames: +1273% (+1283%)
  Python=30 MiB, _tracemalloc=382 MiB, tracemalloc.py=6 MiB

On a small application and a computer with GB of memory, it may not
matter. In a big application on an embedded device, it can be a
blocker point to use tracemalloc. So I added filters (on the filename
and line number) directly in the C module:


``add_filter(include: bool, filename: str, lineno: int=None)`` function:

    Add a filter. If *include* is ``True``, only trace memory blocks
    allocated in a file with a name matching *filename*. If
    *include* is ``False``, don't trace memory blocks allocated in a
    file with a name matching *filename*.

    The match is done using *filename* as a prefix. For example,
    ``'/usr/bin/'`` only matchs files the ``/usr/bin`` directories. The
    ``.pyc`` and ``.pyo`` suffixes are automatically replaced with
    ``.py`` when matching the filename.

    *lineno* is a line number. If *lineno* is ``None`` or lesser than
    ``1``, it matches any line number.

``clear_filters()`` function:

    Reset the filter list.

``get_filters()`` function:

    Get the filters as list of
    ``(include: bool, filename: str, lineno: int)`` tuples.

    If *lineno* is ``None``, a filter matchs any line number.

   By default, the filename of the Python tracemalloc module
   (``tracemalloc.py``) is excluded.


Right now, the match is done using a PyUnicode_Tailmatch(). It is not
convinient. I will see if it is possible to implement the joker
character "*" matching any string, so the API would be closer to
Snapshot.filter_filenames() (which uses fnmatch.fnmatch).

Victor

From rymg19 at gmail.com  Sun Sep  8 15:52:32 2013
From: rymg19 at gmail.com (Ryan)
Date: Sun, 08 Sep 2013 08:52:32 -0500
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <20130908123706.GG26319@ando>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
Message-ID: <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>

...what's a PEP dictator?

Steven D'Aprano <steve at pearwood.info> wrote:

><bump>
>
>I'd like to get some attention for this please.
>
>
>On Sat, Aug 31, 2013 at 12:58:39PM +1000, Steven D'Aprano wrote:
>> Hi all,
>> 
>> 
>> I think that PEP 450 is now ready for a PEP dictator. There have been
>a 
>> number of code reviews, and feedback has been taken into account. The
>test 
>> suite passes. I'm not aware of any unanswered issues with the code.
>At 
>> least two people other than myself think that the implementation is
>ready 
>> for a dictator, and nobody has objected.
>> 
>> There is still on-going work on speeding up the implementation for
>the 
>> statistics.sum function, but that will not effect the interface or
>the 
>> substantially change the test suite.
>> 
>> http://bugs.python.org/issue18606
>> http://www.python.org/dev/peps/pep-0450/
>> 
>> 
>> 
>> 
>> -- 
>> Steven
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: 
>>
>http://mail.python.org/mailman/options/python-dev/steve%40pearwood.info
>> 
>_______________________________________________
>Python-Dev mailing list
>Python-Dev at python.org
>https://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe:
>https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com

-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130908/118e06a9/attachment.html>

From solipsis at pitrou.net  Sun Sep  8 17:20:11 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 8 Sep 2013 17:20:11 +0200
Subject: [Python-Dev] PEP 450 adding statistics module
References: <520C2E23.40405@pearwood.info>
	<52215BDF.2090109@pearwood.info>
Message-ID: <20130908172011.276c19af@fsol>

On Sat, 31 Aug 2013 12:58:39 +1000
Steven D'Aprano <steve at pearwood.info> wrote:

> Hi all,
> 
> 
> I think that PEP 450 is now ready for a PEP dictator.

Perhaps Mark would like to apply?

Regards

Antoine.



From ethan at stoneleaf.us  Sun Sep  8 18:06:29 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 08 Sep 2013 09:06:29 -0700
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
Message-ID: <522CA085.6000601@stoneleaf.us>

On 09/08/2013 06:52 AM, Ryan wrote:
>
> ...what's a PEP dictator?

The person tasked with deciding on the fate of an individual PEP.

--
~Ethan~

From guido at python.org  Sun Sep  8 19:32:34 2013
From: guido at python.org (Guido van Rossum)
Date: Sun, 8 Sep 2013 10:32:34 -0700
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <522CA085.6000601@stoneleaf.us>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
Message-ID: <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>

Going over the open issues:

- Parallel arrays or arrays of tuples? I think the API should require
an array of tuples. It is trivial to zip up parallel arrays to the
required format, while if you have an array of tuples, extracting the
parallel arrays is slightly more cumbersome. Also for manipulating of
the raw data, an array of tuples makes it easier to do insertions or
removals without worrying about losing the correspondence between the
arrays.

- Requiring concrete sequences as opposed to iterators sounds fine.
I'm guessing that good algorithms for doing certain calculations in a
single pass, assuming the full input doesn't fit in memory, are quite
different from good algorithms for doing the same calculations without
having that worry. (Just like you can't expect to use the same code to
do a good job of sorting in-memory and on-disk data.)

- Postponing some algorithms to Python 3.5  sounds fine.

On Sun, Sep 8, 2013 at 9:06 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
> On 09/08/2013 06:52 AM, Ryan wrote:
>>
>>
>> ...what's a PEP dictator?
>
>
> The person tasked with deciding on the fate of an individual PEP.
>
> --
> ~Ethan~
> _______________________________________________
>
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)

From guido at python.org  Sun Sep  8 19:25:22 2013
From: guido at python.org (Guido van Rossum)
Date: Sun, 8 Sep 2013 10:25:22 -0700
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <20130908123706.GG26319@ando>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
Message-ID: <CAP7+vJKR=nK1OvHUzYoV-P6aP8pOrsAt+WA6aE+2xvKSLcaJ4A@mail.gmail.com>

Steven, I'd like to just approve the PEP, given the amount of
discussion that's happened already (though I didn't follow much of
it). I quickly glanced through the PEP and didn't find anything I'd
personally object to, but then I found your section of open issues,
and I realized that you don't actually specify the proposed API in the
PEP itself. It's highly unusual to approve a PEP that doesn't contain
a specification. What did I miss?

On Sun, Sep 8, 2013 at 5:37 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> <bump>
>
> I'd like to get some attention for this please.
>
>
> On Sat, Aug 31, 2013 at 12:58:39PM +1000, Steven D'Aprano wrote:
>> Hi all,
>>
>>
>> I think that PEP 450 is now ready for a PEP dictator. There have been a
>> number of code reviews, and feedback has been taken into account. The test
>> suite passes. I'm not aware of any unanswered issues with the code. At
>> least two people other than myself think that the implementation is ready
>> for a dictator, and nobody has objected.
>>
>> There is still on-going work on speeding up the implementation for the
>> statistics.sum function, but that will not effect the interface or the
>> substantially change the test suite.
>>
>> http://bugs.python.org/issue18606
>> http://www.python.org/dev/peps/pep-0450/
>>
>>
>>
>>
>> --
>> Steven
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> http://mail.python.org/mailman/options/python-dev/steve%40pearwood.info
>>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)

From alexander.belopolsky at gmail.com  Sun Sep  8 19:45:32 2013
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Sun, 8 Sep 2013 21:45:32 +0400
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
Message-ID: <CAP7h-xZ+_vmZXWU5yQXH1zKxygVdWGwdsioahQJR_gjzH3Cmpw@mail.gmail.com>

On Sun, Sep 8, 2013 at 9:32 PM, Guido van Rossum <guido at python.org> wrote:

> - Parallel arrays or arrays of tuples? I think the API should require
> an array of tuples. It is trivial to zip up parallel arrays to the
> required format, while if you have an array of tuples, extracting the
> parallel arrays is slightly more cumbersome.
>

I agree with your conclusion but not with the rationale: converting between
array of tuples and parallel arrays is trivial both ways:

>>> at = [(1,2,3), (4,5,6), (7,8,9), (10, 11, 12)]
>>> zip(*at)
[(1, 4, 7, 10), (2, 5, 8, 11), (3, 6, 9, 12)]
>>> zip(*_)
[(1, 2, 3), (4, 5, 6), (7, 8, 9), (10, 11, 12)]

(Note that zip(*x) is basically transpose(x).)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130908/8d85efc9/attachment.html>

From guido at python.org  Sun Sep  8 19:51:57 2013
From: guido at python.org (Guido van Rossum)
Date: Sun, 8 Sep 2013 10:51:57 -0700
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
Message-ID: <CAP7+vJJ6QxcUwcpdt3Mq6r07Q2dGUQt-VWGdsYGLv4wrRW5dGg@mail.gmail.com>

Never mind, I found the patch and the issue. I really think that the
*PEP* is ready for inclusion after the open issues are changed into
something like Discussion or Future Work, and after adding a more
prominent link to the issue with the patch. Then the *patch* can be
reviewed some more until it is ready -- it looks very close already.

On Sun, Sep 8, 2013 at 10:32 AM, Guido van Rossum <guido at python.org> wrote:
> Going over the open issues:
>
> - Parallel arrays or arrays of tuples? I think the API should require
> an array of tuples. It is trivial to zip up parallel arrays to the
> required format, while if you have an array of tuples, extracting the
> parallel arrays is slightly more cumbersome. Also for manipulating of
> the raw data, an array of tuples makes it easier to do insertions or
> removals without worrying about losing the correspondence between the
> arrays.
>
> - Requiring concrete sequences as opposed to iterators sounds fine.
> I'm guessing that good algorithms for doing certain calculations in a
> single pass, assuming the full input doesn't fit in memory, are quite
> different from good algorithms for doing the same calculations without
> having that worry. (Just like you can't expect to use the same code to
> do a good job of sorting in-memory and on-disk data.)
>
> - Postponing some algorithms to Python 3.5  sounds fine.
>
> On Sun, Sep 8, 2013 at 9:06 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
>> On 09/08/2013 06:52 AM, Ryan wrote:
>>>
>>>
>>> ...what's a PEP dictator?
>>
>>
>> The person tasked with deciding on the fate of an individual PEP.
>>
>> --
>> ~Ethan~
>> _______________________________________________
>>
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
>
> --
> --Guido van Rossum (python.org/~guido)



-- 
--Guido van Rossum (python.org/~guido)

From guido at python.org  Sun Sep  8 19:52:28 2013
From: guido at python.org (Guido van Rossum)
Date: Sun, 8 Sep 2013 10:52:28 -0700
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <CAP7h-xZ+_vmZXWU5yQXH1zKxygVdWGwdsioahQJR_gjzH3Cmpw@mail.gmail.com>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
 <CAP7h-xZ+_vmZXWU5yQXH1zKxygVdWGwdsioahQJR_gjzH3Cmpw@mail.gmail.com>
Message-ID: <CAP7+vJLCFgd_ndhqTrTccO-tO7SxN-bdVQNM4Fj-CysjVHA8iw@mail.gmail.com>

Well, to me zip(*x) is unnatural, and it's inefficient when the arrays are long.

On Sun, Sep 8, 2013 at 10:45 AM, Alexander Belopolsky
<alexander.belopolsky at gmail.com> wrote:
>
> On Sun, Sep 8, 2013 at 9:32 PM, Guido van Rossum <guido at python.org> wrote:
>>
>> - Parallel arrays or arrays of tuples? I think the API should require
>> an array of tuples. It is trivial to zip up parallel arrays to the
>> required format, while if you have an array of tuples, extracting the
>> parallel arrays is slightly more cumbersome.
>
>
> I agree with your conclusion but not with the rationale: converting between
> array of tuples and parallel arrays is trivial both ways:
>
>>>> at = [(1,2,3), (4,5,6), (7,8,9), (10, 11, 12)]
>>>> zip(*at)
> [(1, 4, 7, 10), (2, 5, 8, 11), (3, 6, 9, 12)]
>>>> zip(*_)
> [(1, 2, 3), (4, 5, 6), (7, 8, 9), (10, 11, 12)]
>
> (Note that zip(*x) is basically transpose(x).)



-- 
--Guido van Rossum (python.org/~guido)

From steve at pearwood.info  Sun Sep  8 21:19:54 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 9 Sep 2013 05:19:54 +1000
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <CAP7+vJKR=nK1OvHUzYoV-P6aP8pOrsAt+WA6aE+2xvKSLcaJ4A@mail.gmail.com>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <CAP7+vJKR=nK1OvHUzYoV-P6aP8pOrsAt+WA6aE+2xvKSLcaJ4A@mail.gmail.com>
Message-ID: <20130908191954.GJ26319@ando>

On Sun, Sep 08, 2013 at 10:25:22AM -0700, Guido van Rossum wrote:
> Steven, I'd like to just approve the PEP, given the amount of
> discussion that's happened already (though I didn't follow much of
> it). I quickly glanced through the PEP and didn't find anything I'd
> personally object to, but then I found your section of open issues,
> and I realized that you don't actually specify the proposed API in the
> PEP itself. It's highly unusual to approve a PEP that doesn't contain
> a specification. What did I miss?

You didn't miss anything, but I may have.

Should the PEP go through each public function in the module (there are 
only 11)? That may be a little repetitive, since most have the same, or 
almost the same, signatures. Or is it acceptable to just include an 
overview? I've come up with this:


API

    The initial version of the library will provide univariate (single
    variable) statistics functions.  The general API will be based on a
    functional model ``function(data, ...) -> result``, where ``data``
    is a mandatory iterable of (usually) numeric data.

    The author expects that lists will be the most common data type used,
    but any iterable type should be acceptable.  Where necessary, functions
    may convert to lists internally.  Where possible, functions are
    expected to conserve the type of the data values, for example, the mean
    of a list of Decimals should be a Decimal rather than float.


    Calculating the mean, median and mode

        The ``mean``, ``median`` and ``mode`` functions take a single
        mandatory argument and return the appropriate statistic, e.g.:

        >>> mean([1, 2, 3])
        2.0

        ``mode`` is the sole exception to the rule that the data argument
        must be numeric.  It will also accept an iterable of nominal data,
        such as strings.


    Calculating variance and standard deviation

        In order to be similar to scientific calculators, the statistics
        module will include separate functions for population and sample
        variance and standard deviation.  All four functions have similar
        signatures, with a single mandatory argument, an iterable of
        numeric data, e.g.:

        >>> variance([1, 2, 2, 2, 3])
        0.5

        All four functions also accept a second, optional, argument, the
        mean of the data.  This is modelled on a similar API provided by
        the GNU Scientific Library[18].  There are three use-cases for
        using this argument, in no particular order:

            1)  The value of the mean is known *a priori*.

            2)  You have already calculated the mean, and wish to avoid
                calculating it again.

            3)  You wish to (ab)use the variance functions to calculate
                the second moment about some given point other than the
                mean.

        In each case, it is the caller's responsibility to ensure that
        given argument is meaningful.




Is this satisfactory or do I need to go into more detail?


-- 
Steven

From p.f.moore at gmail.com  Sun Sep  8 22:14:39 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 8 Sep 2013 21:14:39 +0100
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <20130908191954.GJ26319@ando>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <CAP7+vJKR=nK1OvHUzYoV-P6aP8pOrsAt+WA6aE+2xvKSLcaJ4A@mail.gmail.com>
 <20130908191954.GJ26319@ando>
Message-ID: <CACac1F8mrqM9aEXkbtzDjbVzF49ofrwuZjaBLgVbKd9myQx2eQ@mail.gmail.com>

On 8 September 2013 20:19, Steven D'Aprano <steve at pearwood.info> wrote:
[...]
> Is this satisfactory or do I need to go into more detail?

It describes only 7 functions, and yet you state there are 11. I'd
suggest you add a 1-line summary of each function, something like:

mean - calculate the (arithmetic) mean of the data
median - calculate the median value of the data
etc.

Paul

From oscar.j.benjamin at gmail.com  Sun Sep  8 22:48:26 2013
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Sun, 8 Sep 2013 21:48:26 +0100
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
Message-ID: <CAHVvXxRAi_HHFOD2dY8UGsXpbgd-LyFU9rsg9x6Uk_=VuKG4hA@mail.gmail.com>

On 8 September 2013 18:32, Guido van Rossum <guido at python.org> wrote:
> Going over the open issues:
>
> - Parallel arrays or arrays of tuples? I think the API should require
> an array of tuples. It is trivial to zip up parallel arrays to the
> required format, while if you have an array of tuples, extracting the
> parallel arrays is slightly more cumbersome. Also for manipulating of
> the raw data, an array of tuples makes it easier to do insertions or
> removals without worrying about losing the correspondence between the
> arrays.

For something like this, where there are multiple obvious formats for
the input data, I think it's reasonable to just request whatever is
convenient for the implementation. Otherwise you're asking at least
some of your users to convert data from one format to another just so
that you can convert it back again. In any real problem you'll likely
have more than two variables, so you'll be writing some code to
prepare the data for the function anyway.

The most obvious alternative that isn't explicitly mentioned in the
PEP is to accept either:

def correlation(x, y=None):
    if y is None:
        xs = []
        ys = []
        for x, y in x:
            xs.append(x)
            ys.append(y)
    else:
        xs = list(x)
        ys = list(y)
        assert len(xs) == len(ys)
    # In reality a helper function does the above.
    # Now compute stuff

This avoids any unnecessary conversions and is as convenient as
possible for all users at the expense of having a slightly more
complicated API.


Oscar

From janzert at janzert.com  Sun Sep  8 22:52:05 2013
From: janzert at janzert.com (Janzert)
Date: Sun, 08 Sep 2013 16:52:05 -0400
Subject: [Python-Dev] RFC: PEP 454: Add a new tracemalloc module
In-Reply-To: <CAMpsgwZTOBdXy2Rcxs0itnPZYZgHU4X4tPNf5syoZrOJ5gYCFw@mail.gmail.com>
References: <CAMpsgwZTOBdXy2Rcxs0itnPZYZgHU4X4tPNf5syoZrOJ5gYCFw@mail.gmail.com>
Message-ID: <l0io1m$cqd$1@ger.gmane.org>

It seems like most of this could live on PyPi for a while so the API can 
get hashed out in use? If that's not the case is it because the PEP 445 
API isn't rich enough?

Janzert


From guido at python.org  Sun Sep  8 23:41:35 2013
From: guido at python.org (Guido van Rossum)
Date: Sun, 8 Sep 2013 14:41:35 -0700
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <CAHVvXxRAi_HHFOD2dY8UGsXpbgd-LyFU9rsg9x6Uk_=VuKG4hA@mail.gmail.com>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
 <CAHVvXxRAi_HHFOD2dY8UGsXpbgd-LyFU9rsg9x6Uk_=VuKG4hA@mail.gmail.com>
Message-ID: <CAP7+vJK0HRT4p0MEqraCr8EY1MN29E6CnkVcxvOks_s1rMka_w@mail.gmail.com>

On Sun, Sep 8, 2013 at 1:48 PM, Oscar Benjamin
<oscar.j.benjamin at gmail.com> wrote:
> On 8 September 2013 18:32, Guido van Rossum <guido at python.org> wrote:
>> Going over the open issues:
>>
>> - Parallel arrays or arrays of tuples? I think the API should require
>> an array of tuples. It is trivial to zip up parallel arrays to the
>> required format, while if you have an array of tuples, extracting the
>> parallel arrays is slightly more cumbersome. Also for manipulating of
>> the raw data, an array of tuples makes it easier to do insertions or
>> removals without worrying about losing the correspondence between the
>> arrays.
>
> For something like this, where there are multiple obvious formats for
> the input data, I think it's reasonable to just request whatever is
> convenient for the implementation.

Not really. The implementation may change, or its needs may not be
obvious to the caller. I would say the right thing to do is request
something easy to remember, which often means consistent. In general,
Python APIs definitely skew towards lists of tuples rather than
parallel arrays, and for good reasons -- that way you benefit most
from built-in operations like slices and insert/append.

> Otherwise you're asking at least
> some of your users to convert data from one format to another just so
> that you can convert it back again. In any real problem you'll likely
> have more than two variables, so you'll be writing some code to
> prepare the data for the function anyway.

Yeah, so you might as well prepare it in the form that the API expects.

> The most obvious alternative that isn't explicitly mentioned in the
> PEP is to accept either:
>
> def correlation(x, y=None):
>     if y is None:
>         xs = []
>         ys = []
>         for x, y in x:
>             xs.append(x)
>             ys.append(y)
>     else:
>         xs = list(x)
>         ys = list(y)
>         assert len(xs) == len(ys)
>     # In reality a helper function does the above.
>     # Now compute stuff
>
> This avoids any unnecessary conversions and is as convenient as
> possible for all users at the expense of having a slightly more
> complicated API.

I don't think this is really more convenient -- it is more to learn,
and can cause surprises (e.g. when a user is only familiar with one
format and then sees an example using the other format, they may be
unable to understand the example).

The one argument I *haven't* heard yet which *might* sway me would be
something along the line "every other statistics package that users
might be familiar with does it this way" or "all the statistics
textbooks do it this way". (Because, frankly, when it comes to
statistics I'm a rank amateur and I really want Steven's new module to
educate me as much as help me compute specific statistical functions.)

-- 
--Guido van Rossum (python.org/~guido)

From victor.stinner at gmail.com  Mon Sep  9 01:05:11 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 9 Sep 2013 01:05:11 +0200
Subject: [Python-Dev] RFC: PEP 454: Add a new tracemalloc module
In-Reply-To: <l0io1m$cqd$1@ger.gmane.org>
References: <CAMpsgwZTOBdXy2Rcxs0itnPZYZgHU4X4tPNf5syoZrOJ5gYCFw@mail.gmail.com>
 <l0io1m$cqd$1@ger.gmane.org>
Message-ID: <CAMpsgwYUZm00eaunv6cxe1rGMzYYe9JA-eBgwBW988y_7S-Lmg@mail.gmail.com>

2013/9/8 Janzert <janzert at janzert.com>:
> It seems like most of this could live on PyPi for a while so the API can get
> hashed out in use?

The pytracemalloc is available on PyPI since 6 months. The only
feedback I had was something trying to compile it on Windows (which is
complex because of the dependency to glib, I don't think that it
succeed to install it on Windows). I guess that I didn't get more
feedback because it requires to patch and recompile Python, which is
not trivial.

I expect more feedback on python-dev with a working implementation (on
hg.python.org) and a PEP.

The version available on PyPI works and should be enough for most use
cases to be able to identify a memory leak.

Gregory P. Smith asked me if it would be possible to get more frames
(filename and line number) of the Python traceback, instead of just
the one frame (the last frame). I implemented it, but now I have new
issues (memory usage of the tracemalloc module itself), so I'm working
on filters directly implemented in the C module (_tracemalloc). It was
already possible to filter traces from a snapshot read from the disk.

I still have some tasks in my TODO list to finish the API and the
implementation. When I will be done, I will post post a new version of
the PEP on python-dev.

> If that's not the case is it because the PEP 445 API
> isn't rich enough?

The PEP 445 API is only designed to allow to develop new tools like
failmalloc or tracemalloc, without adding overhead if such debug tool
is not used.

The tracemalloc module reads the current Python traceback (filename
and line number) which is "not directly" accessible from
PyMem_Malloc(). I hope that existing tools like Heapy and Melia will
benefit directly from tracemalloc instead of having to develop their
own memory allocator hooks to get the same information (Python
traceback).

Victor

From steve at pearwood.info  Mon Sep  9 03:59:48 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 9 Sep 2013 11:59:48 +1000
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <CAP7+vJK0HRT4p0MEqraCr8EY1MN29E6CnkVcxvOks_s1rMka_w@mail.gmail.com>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
 <CAHVvXxRAi_HHFOD2dY8UGsXpbgd-LyFU9rsg9x6Uk_=VuKG4hA@mail.gmail.com>
 <CAP7+vJK0HRT4p0MEqraCr8EY1MN29E6CnkVcxvOks_s1rMka_w@mail.gmail.com>
Message-ID: <20130909015948.GK26319@ando>

On Sun, Sep 08, 2013 at 02:41:35PM -0700, Guido van Rossum wrote:
> On Sun, Sep 8, 2013 at 1:48 PM, Oscar Benjamin
> <oscar.j.benjamin at gmail.com> wrote:

> > The most obvious alternative that isn't explicitly mentioned in the
> > PEP is to accept either:
> >
> > def correlation(x, y=None):
> >     if y is None:
> >         xs = []
> >         ys = []
> >         for x, y in x:
> >             xs.append(x)
> >             ys.append(y)
> >     else:
> >         xs = list(x)
> >         ys = list(y)
> >         assert len(xs) == len(ys)
> >     # In reality a helper function does the above.
> >     # Now compute stuff
> >
> > This avoids any unnecessary conversions and is as convenient as
> > possible for all users at the expense of having a slightly more
> > complicated API.

The PEP does mention that, as "some combination of the above".

The PEP also mentions that the decision of what API to use for 
multivariate stats is deferred until 3.5, so there's plenty of time for 
people to bike-shed this :-)

 
> I don't think this is really more convenient -- it is more to learn,
> and can cause surprises (e.g. when a user is only familiar with one
> format and then sees an example using the other format, they may be
> unable to understand the example).
> 
> The one argument I *haven't* heard yet which *might* sway me would be
> something along the line "every other statistics package that users
> might be familiar with does it this way" or "all the statistics
> textbooks do it this way". (Because, frankly, when it comes to
> statistics I'm a rank amateur and I really want Steven's new module to
> educate me as much as help me compute specific statistical functions.)

I don't think that there is one common API for multivariate stats 
packages. It partially depends on whether the package is aimed at basic 
use or advanced use. I haven't done a systematic comparison of the most 
common, but here are a few examples:

- The Casio Classpad graphing calculator has a spreadsheet-like 
interface, which I consider equivalent to func(xdata, ydata).

- The HP-48G series of calculators uses a fixed global variable holding 
a matrix, and a second global variable specifying which columns to use.

- The R "cor" (correlation coefficient) function takes either a pair of 
vectors (lists), and calculates a single value, or a matrix, in which 
case it calculates the correlation matrix.

- numpy.corrcoeff takes one or two array arguments, and a third argument 
specifying whether to treat rows or columns as variables, and like R 
returns either a single value or the correlation matrix.

- Minitab expects two seperate vector arguments, and returns the 
correlation coefficient between them.

- If I'm reading the below page correctly, the SAS corr procedure 
takes anything up to 27 arguments.

http://support.sas.com/documentation/cdl/en/procstat/63104/HTML/default/procstat_corr_sect004.htm

I don't suggest we follow that API :-)


Quite frankly, I consider the majority of stats APIs to be confusing 
with a steep learning curve.



-- 
Steven

From stephen at xemacs.org  Mon Sep  9 04:57:50 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Mon, 09 Sep 2013 11:57:50 +0900
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <CAP7+vJK0HRT4p0MEqraCr8EY1MN29E6CnkVcxvOks_s1rMka_w@mail.gmail.com>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
 <CAHVvXxRAi_HHFOD2dY8UGsXpbgd-LyFU9rsg9x6Uk_=VuKG4hA@mail.gmail.com>
 <CAP7+vJK0HRT4p0MEqraCr8EY1MN29E6CnkVcxvOks_s1rMka_w@mail.gmail.com>
Message-ID: <87ppsi8zwx.fsf@uwakimon.sk.tsukuba.ac.jp>

Guido van Rossum writes:
 > On Sun, Sep 8, 2013 at 1:48 PM, Oscar Benjamin
 > <oscar.j.benjamin at gmail.com> wrote:
 > > On 8 September 2013 18:32, Guido van Rossum <guido at python.org> wrote:
 > >> Going over the open issues:
 > >>
 > >> - Parallel arrays or arrays of tuples? I think the API should require
 > >> an array of tuples. It is trivial to zip up parallel arrays to the
 > >> required format, while if you have an array of tuples, extracting the
 > >> parallel arrays is slightly more cumbersome.
 > >>
 > >> Also for manipulating of the raw data, an array of tuples makes
 > >> it easier to do insertions or removals without worrying about
 > >> losing the correspondence between the arrays.

I don't necessarily find this persuasive.  It's more common when
working with existing databases that you add variables than add
observations.  This is going to require attention to the
correspondence in any case.  Observations aren't added, and they're
"removed" temporarily for statistics on subsets by slicing.  If you
use the same slice for all variables, you're not going to make a
mistake.

 > Not really. The implementation may change, or its needs may not be
 > obvious to the caller. I would say the right thing to do is request
 > something easy to remember, which often means consistent. In general,
 > Python APIs definitely skew towards lists of tuples rather than
 > parallel arrays, and for good reasons -- that way you benefit most
 > from built-in operations like slices and insert/append.

However, it's common in economic statistics to have a rectangular
array, and extract both certain rows (tuples of observations on
variables) and certain columns (variables).  For example you might
have data on populations of American states from 1900 to 2012, and
extract the data on New England states from 1946 to 2012 for analysis.

 > The one argument I *haven't* heard yet which *might* sway me would be
 > something along the line "every other statistics package that users
 > might be familiar with does it this way" or "all the statistics
 > textbooks do it this way". (Because, frankly, when it comes to
 > statistics I'm a rank amateur and I really want Steven's new module to
 > educate me as much as help me compute specific statistical functions.)

In economic statistics, most software traditionally inputs variables
in column-major order (ie, parallel arrays).  That said, most software
nowadays allows input as spreadsheet tables.  You pays your money and
you takes your choice.

I think the example above of state population data shows that rows and
columns are pretty symmetric here.  Many databases will have "too many"
of both, and you'll want to "slice" both to get the sample and
variables relevant to your analysis.

This is all just for consideration; I am quite familiar with economic
statistics and software, but not so much for that used in sociology,
psychology, and medical applications.  In the end, I think it's best
to leave it up to Steven's judgment as to what is convenient for him
to maintain.

From greg.ewing at canterbury.ac.nz  Mon Sep  9 02:26:05 2013
From: greg.ewing at canterbury.ac.nz (Greg)
Date: Mon, 09 Sep 2013 12:26:05 +1200
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <CAP7+vJLCFgd_ndhqTrTccO-tO7SxN-bdVQNM4Fj-CysjVHA8iw@mail.gmail.com>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
 <CAP7h-xZ+_vmZXWU5yQXH1zKxygVdWGwdsioahQJR_gjzH3Cmpw@mail.gmail.com>
 <CAP7+vJLCFgd_ndhqTrTccO-tO7SxN-bdVQNM4Fj-CysjVHA8iw@mail.gmail.com>
Message-ID: <522D159D.90200@canterbury.ac.nz>

On 9/09/2013 5:52 a.m., Guido van Rossum wrote:
> Well, to me zip(*x) is unnatural, and it's inefficient when the arrays are long.

Would it be worth having a transpose() function in the stdlib
somewhere, that returns a view instead of copying the data?

-- 
Greg


From guido at python.org  Mon Sep  9 05:16:19 2013
From: guido at python.org (Guido van Rossum)
Date: Sun, 8 Sep 2013 20:16:19 -0700
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <87ppsi8zwx.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
 <CAHVvXxRAi_HHFOD2dY8UGsXpbgd-LyFU9rsg9x6Uk_=VuKG4hA@mail.gmail.com>
 <CAP7+vJK0HRT4p0MEqraCr8EY1MN29E6CnkVcxvOks_s1rMka_w@mail.gmail.com>
 <87ppsi8zwx.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CAP7+vJL2NZPUm0mO_moVYdkvL4YAKkS-CWgTkg9goLyLtCSNxA@mail.gmail.com>

Yeah, so this and Steven's review of various other APIs suggests that the
field of statistics hasn't really reached the object-oriented age (or
perhaps the OO view isn't suitable for the field), and people really think
of their data as a matrix of some sort. We should respect that. Now, if
this was NumPy, it would *still* make sense to require a single argument,
to be interpreted in the usual fashion. So I'm using that as a kind of
leverage to still recommend taking a list of pairs instead of a pair of
lists. Also, it's quite likely that at least *some* of the users of the new
statistics module will be more familiar with OO programming (e.g. the
Python DB API , PEP 249) than they are with other statistics packages.


On Sun, Sep 8, 2013 at 7:57 PM, Stephen J. Turnbull <stephen at xemacs.org>wrote:

> Guido van Rossum writes:
>  > On Sun, Sep 8, 2013 at 1:48 PM, Oscar Benjamin
>  > <oscar.j.benjamin at gmail.com> wrote:
>  > > On 8 September 2013 18:32, Guido van Rossum <guido at python.org> wrote:
>  > >> Going over the open issues:
>  > >>
>  > >> - Parallel arrays or arrays of tuples? I think the API should require
>  > >> an array of tuples. It is trivial to zip up parallel arrays to the
>  > >> required format, while if you have an array of tuples, extracting the
>  > >> parallel arrays is slightly more cumbersome.
>  > >>
>  > >> Also for manipulating of the raw data, an array of tuples makes
>  > >> it easier to do insertions or removals without worrying about
>  > >> losing the correspondence between the arrays.
>
> I don't necessarily find this persuasive.  It's more common when
> working with existing databases that you add variables than add
> observations.  This is going to require attention to the
> correspondence in any case.  Observations aren't added, and they're
> "removed" temporarily for statistics on subsets by slicing.  If you
> use the same slice for all variables, you're not going to make a
> mistake.
>
>  > Not really. The implementation may change, or its needs may not be
>  > obvious to the caller. I would say the right thing to do is request
>  > something easy to remember, which often means consistent. In general,
>  > Python APIs definitely skew towards lists of tuples rather than
>  > parallel arrays, and for good reasons -- that way you benefit most
>  > from built-in operations like slices and insert/append.
>
> However, it's common in economic statistics to have a rectangular
> array, and extract both certain rows (tuples of observations on
> variables) and certain columns (variables).  For example you might
> have data on populations of American states from 1900 to 2012, and
> extract the data on New England states from 1946 to 2012 for analysis.
>
>  > The one argument I *haven't* heard yet which *might* sway me would be
>  > something along the line "every other statistics package that users
>  > might be familiar with does it this way" or "all the statistics
>  > textbooks do it this way". (Because, frankly, when it comes to
>  > statistics I'm a rank amateur and I really want Steven's new module to
>  > educate me as much as help me compute specific statistical functions.)
>
> In economic statistics, most software traditionally inputs variables
> in column-major order (ie, parallel arrays).  That said, most software
> nowadays allows input as spreadsheet tables.  You pays your money and
> you takes your choice.
>
> I think the example above of state population data shows that rows and
> columns are pretty symmetric here.  Many databases will have "too many"
> of both, and you'll want to "slice" both to get the sample and
> variables relevant to your analysis.
>
> This is all just for consideration; I am quite familiar with economic
> statistics and software, but not so much for that used in sociology,
> psychology, and medical applications.  In the end, I think it's best
> to leave it up to Steven's judgment as to what is convenient for him
> to maintain.
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130908/054b9fc5/attachment.html>

From guido at python.org  Mon Sep  9 05:18:46 2013
From: guido at python.org (Guido van Rossum)
Date: Sun, 8 Sep 2013 20:18:46 -0700
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <522D159D.90200@canterbury.ac.nz>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
 <CAP7h-xZ+_vmZXWU5yQXH1zKxygVdWGwdsioahQJR_gjzH3Cmpw@mail.gmail.com>
 <CAP7+vJLCFgd_ndhqTrTccO-tO7SxN-bdVQNM4Fj-CysjVHA8iw@mail.gmail.com>
 <522D159D.90200@canterbury.ac.nz>
Message-ID: <CAP7+vJ+L0g+C5fszy7eub5cLn0By+Qe+afe9-6-CfRpKpeBu+g@mail.gmail.com>

On Sun, Sep 8, 2013 at 5:26 PM, Greg <greg.ewing at canterbury.ac.nz> wrote:

> On 9/09/2013 5:52 a.m., Guido van Rossum wrote:
>
>> Well, to me zip(*x) is unnatural, and it's inefficient when the arrays
>> are long.
>>
>
> Would it be worth having a transpose() function in the stdlib
> somewhere, that returns a view instead of copying the data?


I'd be hesitant to add just that one function, given that there's hardly
any support for multi-dimensional arrays in the stdlib. (NumPy of course
has a transpose(), and that's where it arguably belongs.)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130908/022d1b16/attachment.html>

From steve at pearwood.info  Mon Sep  9 05:27:22 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 9 Sep 2013 13:27:22 +1000
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <CACac1F8mrqM9aEXkbtzDjbVzF49ofrwuZjaBLgVbKd9myQx2eQ@mail.gmail.com>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <CAP7+vJKR=nK1OvHUzYoV-P6aP8pOrsAt+WA6aE+2xvKSLcaJ4A@mail.gmail.com>
 <20130908191954.GJ26319@ando>
 <CACac1F8mrqM9aEXkbtzDjbVzF49ofrwuZjaBLgVbKd9myQx2eQ@mail.gmail.com>
Message-ID: <20130909032722.GL26319@ando>

On Sun, Sep 08, 2013 at 09:14:39PM +0100, Paul Moore wrote:
> On 8 September 2013 20:19, Steven D'Aprano <steve at pearwood.info> wrote:
> [...]
> > Is this satisfactory or do I need to go into more detail?
> 
> It describes only 7 functions, and yet you state there are 11. I'd
> suggest you add a 1-line summary of each function, something like:
> 
> mean - calculate the (arithmetic) mean of the data
> median - calculate the median value of the data
> etc.

Thanks Paul, will do.

I think PEP 1 needs to be a bit clearer about this part of the process. 
For instance, if I had a module with 100 functions and methods, would I 
need to document all of them in the PEP? I expect not, but then I didn't 
expect I needed to document all 11 either :-)



-- 
Steven

From steve at pearwood.info  Mon Sep  9 05:43:59 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 9 Sep 2013 13:43:59 +1000
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <522D159D.90200@canterbury.ac.nz>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
 <CAP7h-xZ+_vmZXWU5yQXH1zKxygVdWGwdsioahQJR_gjzH3Cmpw@mail.gmail.com>
 <CAP7+vJLCFgd_ndhqTrTccO-tO7SxN-bdVQNM4Fj-CysjVHA8iw@mail.gmail.com>
 <522D159D.90200@canterbury.ac.nz>
Message-ID: <20130909034359.GM26319@ando>

On Mon, Sep 09, 2013 at 12:26:05PM +1200, Greg wrote:
> On 9/09/2013 5:52 a.m., Guido van Rossum wrote:
> >Well, to me zip(*x) is unnatural, and it's inefficient when the arrays are 
> >long.
> 
> Would it be worth having a transpose() function in the stdlib
> somewhere, that returns a view instead of copying the data?

I've intentionally left out multivariate statistics from the initial 
version of statistics.py so there will be plenty of time to get feedback 
from users before deciding on an API before 3.5.

If there was a transpose function in the std lib, the obvious place 
would be the statistics module itself. There is precedent: R includes a 
transpose function, and presumably the creators of R expect it to be 
used frequently because they've given it a single-letter name.

http://stat.ethz.ch/R-manual/R-devel/library/base/html/t.html


-- 
Steven

From tjreedy at udel.edu  Mon Sep  9 09:06:24 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 09 Sep 2013 03:06:24 -0400
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <CAP7+vJK0HRT4p0MEqraCr8EY1MN29E6CnkVcxvOks_s1rMka_w@mail.gmail.com>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
 <CAHVvXxRAi_HHFOD2dY8UGsXpbgd-LyFU9rsg9x6Uk_=VuKG4hA@mail.gmail.com>
 <CAP7+vJK0HRT4p0MEqraCr8EY1MN29E6CnkVcxvOks_s1rMka_w@mail.gmail.com>
Message-ID: <l0js19$nuf$1@ger.gmane.org>

On 9/8/2013 5:41 PM, Guido van Rossum wrote:
> On Sun, Sep 8, 2013 at 1:48 PM, Oscar Benjamin
> <oscar.j.benjamin at gmail.com> wrote:
>> On 8 September 2013 18:32, Guido van Rossum <guido at python.org> wrote:
>>> Going over the open issues:
>>>
>>> - Parallel arrays or arrays of tuples? I think the API should require
>>> an array of tuples. It is trivial to zip up parallel arrays to the
>>> required format, while if you have an array of tuples, extracting the
>>> parallel arrays is slightly more cumbersome. Also for manipulating of
>>> the raw data, an array of tuples makes it easier to do insertions or
>>> removals without worrying about losing the correspondence between the
>>> arrays.
>>
>> For something like this, where there are multiple obvious formats for
>> the input data, I think it's reasonable to just request whatever is
>> convenient for the implementation.
>
> Not really. The implementation may change, or its needs may not be
> obvious to the caller. I would say the right thing to do is request
> something easy to remember, which often means consistent. In general,
> Python APIs definitely skew towards lists of tuples rather than
> parallel arrays, and for good reasons -- that way you benefit most
> from built-in operations like slices and insert/append.

This question has been discussed in the statistical software community 
for decades, going back to when storage was on magnetic tape, where 
contiguity was even more important than cache locality. In my experience 
with multiple packages, the most common format for input is tables where 
rows represent cases, samples, or whatever, which translates as lists of 
records (or tuples), just as with relational databases. Columns then 
represent a 'variable'. So I think we should go with that.

Some packages might transpose the data internally, but that is an 
internal matter. The tradeoff is that storing by cases makes adding a 
new case easier, while storing by variables makes adding a new variable 
easier.

-- 
Terry Jan Reedy


From tjreedy at udel.edu  Mon Sep  9 09:30:15 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 09 Sep 2013 03:30:15 -0400
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <87ppsi8zwx.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
 <CAHVvXxRAi_HHFOD2dY8UGsXpbgd-LyFU9rsg9x6Uk_=VuKG4hA@mail.gmail.com>
 <CAP7+vJK0HRT4p0MEqraCr8EY1MN29E6CnkVcxvOks_s1rMka_w@mail.gmail.com>
 <87ppsi8zwx.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <l0jte0$7bi$1@ger.gmane.org>

On 9/8/2013 10:57 PM, Stephen J. Turnbull wrote:

> I don't necessarily find this persuasive.  It's more common when
> working with existing databases that you add variables than add
> observations.

My experience with general scientific research is the opposite. One 
decides on the variables to measure and then adds rows (records) of data 
as you measure each experimental or observational subject. New 
calculated variables may be added (and often are) after the data 
collection is complete (at least for the moment).

Time series analysis is a distinct and specialized subfield of 
statistics. The corresponding data collections is often different: one 
may start with a fixed set of subjects (50 US states for instance) and 
add 'variables' (population in year X) indefinitely. Much economic 
statistics is in this category.

A third category is interaction analysis, where the data form a true 
matrix where both rows and columns represent subjects and entries 
represent interaction (how many times John emailed Joe, for instance).

-- 
Terry Jan Reedy


From paul at colomiets.name  Mon Sep  9 09:51:31 2013
From: paul at colomiets.name (Paul Colomiets)
Date: Mon, 9 Sep 2013 10:51:31 +0300
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
Message-ID: <CAA0gF6rFbwwU2cK_gEcApaWdb9GOeSy6gWMytma+Dt_wkX7cDQ@mail.gmail.com>

Hi Guido,

On Sun, Sep 8, 2013 at 8:32 PM, Guido van Rossum <guido at python.org> wrote:
> Going over the open issues:
>
> - Parallel arrays or arrays of tuples? I think the API should require
> an array of tuples. It is trivial to zip up parallel arrays to the
> required format, while if you have an array of tuples, extracting the
> parallel arrays is slightly more cumbersome. Also for manipulating of
> the raw data, an array of tuples makes it easier to do insertions or
> removals without worrying about losing the correspondence between the
> arrays.

I think there is a big reason to use parallel arrays that might be
overlooked. You can feed an array.array('f') to the function, which
may save a lot of memory.

--
Paul

From storchaka at gmail.com  Mon Sep  9 10:02:05 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 09 Sep 2013 11:02:05 +0300
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <CAP7+vJLCFgd_ndhqTrTccO-tO7SxN-bdVQNM4Fj-CysjVHA8iw@mail.gmail.com>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
 <CAP7h-xZ+_vmZXWU5yQXH1zKxygVdWGwdsioahQJR_gjzH3Cmpw@mail.gmail.com>
 <CAP7+vJLCFgd_ndhqTrTccO-tO7SxN-bdVQNM4Fj-CysjVHA8iw@mail.gmail.com>
Message-ID: <l0jv92$oqm$1@ger.gmane.org>

08.09.13 20:52, Guido van Rossum ???????(??):
> Well, to me zip(*x) is unnatural, and it's inefficient when the arrays are long.

Perhaps we need zip.from_iterable()?



From oscar.j.benjamin at gmail.com  Mon Sep  9 10:36:23 2013
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Mon, 9 Sep 2013 09:36:23 +0100
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <l0jv92$oqm$1@ger.gmane.org>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
 <CAP7h-xZ+_vmZXWU5yQXH1zKxygVdWGwdsioahQJR_gjzH3Cmpw@mail.gmail.com>
 <CAP7+vJLCFgd_ndhqTrTccO-tO7SxN-bdVQNM4Fj-CysjVHA8iw@mail.gmail.com>
 <l0jv92$oqm$1@ger.gmane.org>
Message-ID: <CAHVvXxSnF1fAP6dNxMvdB9_NqPMMhXQu+70tdDJ6BEieMZOU=g@mail.gmail.com>

On 9 September 2013 09:02, Serhiy Storchaka <storchaka at gmail.com> wrote:
> 08.09.13 20:52, Guido van Rossum ???????(??):
>
>> Well, to me zip(*x) is unnatural, and it's inefficient when the arrays are
>> long.
>
> Perhaps we need zip.from_iterable()?

I would prefer it if chain.from_iterable were named something like
flatten (as it is in the itertools recipes). Similarly transpose is a
better name than zip.from_iterable.


Oscar

From oscar.j.benjamin at gmail.com  Mon Sep  9 11:00:24 2013
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Mon, 9 Sep 2013 10:00:24 +0100
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <CAP7+vJL2NZPUm0mO_moVYdkvL4YAKkS-CWgTkg9goLyLtCSNxA@mail.gmail.com>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
 <CAHVvXxRAi_HHFOD2dY8UGsXpbgd-LyFU9rsg9x6Uk_=VuKG4hA@mail.gmail.com>
 <CAP7+vJK0HRT4p0MEqraCr8EY1MN29E6CnkVcxvOks_s1rMka_w@mail.gmail.com>
 <87ppsi8zwx.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAP7+vJL2NZPUm0mO_moVYdkvL4YAKkS-CWgTkg9goLyLtCSNxA@mail.gmail.com>
Message-ID: <CAHVvXxTyn2hcUL0KPrwcVwD1mfqmN4ypj=UiUziRHhALkaQPJA@mail.gmail.com>

On 9 September 2013 04:16, Guido van Rossum <guido at python.org> wrote:
>
> Yeah, so this and Steven's review of various other APIs suggests that the
> field of statistics hasn't really reached the object-oriented age (or
> perhaps the OO view isn't suitable for the field), and people really think
> of their data as a matrix of some sort. We should respect that. Now, if this
> was NumPy, it would *still* make sense to require a single argument, to be
> interpreted in the usual fashion. So I'm using that as a kind of leverage to
> still recommend taking a list of pairs instead of a pair of lists. Also,
> it's quite likely that at least *some* of the users of the new statistics
> module will be more familiar with OO programming (e.g. the Python DB API ,
> PEP 249) than they are with other statistics packages.

I'm not sure if I understand what you mean by this. Numpy has built
everything on top of a core ndarray class whose methods make the
issues about multivariate stats APIs trivial. The transpose of an
array A is simply the attribute A.T which is both convenient and cheap
since it's just an alternate view on the underlying buffer.

Also numpy provides record arrays that enable you to use names instead
of numeric indices:

>>> import numpy as np
>>> dt = np.dtype([('Year', int), ('Arizona', float), ('Dakota', float)])
>>> a = np.array([(2001, 123., 456.), (2002, 234., 345), (2003, 345., 567)], dt)
>>> a
array([(2001, 123.0, 456.0), (2002, 234.0, 345.0), (2003, 345.0, 567.0)],
      dtype=[('Year', '<i4'), ('Arizona', '<f8'), ('Dakota', '<f8')])
>>> a['Year']
array([2001, 2002, 2003])
>>> a['Arizona']
array([ 123.,  234.,  345.])
>>> np.corrcoef(a['Arizona'], a['Dakota'])
array([[ 1. ,  0.5],
       [ 0.5,  1. ]])
>>> included = a[a['Year'] > 2001]
>>> included
array([(2002, 234.0, 345.0), (2003, 345.0, 567.0)],
      dtype=[('Year', '<i4'), ('Arizona', '<f8'), ('Dakota', '<f8')])
>>> np.corrcoef(included['Arizona'], included['Dakota'])
array([[ 1.,  1.],
       [ 1.,  1.]])

So perhaps the statistics module could have a similar NameTupleArray
type that can be easily loaded and saved from a csv file and makes it
easy to put your data in whatever form is required.


Oscar

From stuart at stuartbishop.net  Mon Sep  9 12:38:50 2013
From: stuart at stuartbishop.net (Stuart Bishop)
Date: Mon, 9 Sep 2013 17:38:50 +0700
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <52293667.8030609@jcea.es>
References: <5228C00F.7000209@jcea.es>
 <CAF-Rda_aajDr71ZU=xJ8m1diBLuG_jr__TO6tR8kAMS625q7hg@mail.gmail.com>
 <52293667.8030609@jcea.es>
Message-ID: <CADmi=6PFXSB2226W4U7vbrjepwjjbCBuj8ypmpgE0JARV4OMYA@mail.gmail.com>

On Fri, Sep 6, 2013 at 8:56 AM, Jesus Cea <jcea at jcea.es> wrote:

> Sorry, Google, Facebook, Twitter, etc., are not acceptable OpenID
> providers for me. I should have made that point in my original email.
> My excuses.
>
> Any other suggestion?

As Barry mentioned earlier, launchpad.net. Look for the 'lp' icon on
pypi, bugs.python.org etc.


-- 
Stuart Bishop <stuart at stuartbishop.net>
http://www.stuartbishop.net/

From skip at pobox.com  Mon Sep  9 12:44:43 2013
From: skip at pobox.com (Skip Montanaro)
Date: Mon, 9 Sep 2013 05:44:43 -0500
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <87ppsi8zwx.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
 <CAHVvXxRAi_HHFOD2dY8UGsXpbgd-LyFU9rsg9x6Uk_=VuKG4hA@mail.gmail.com>
 <CAP7+vJK0HRT4p0MEqraCr8EY1MN29E6CnkVcxvOks_s1rMka_w@mail.gmail.com>
 <87ppsi8zwx.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CANc-5UxZ5M7MXxCPYY6Tfn0rpjRUT+yjyOWnGgFO8ROSyWfrHQ@mail.gmail.com>

> However, it's common in economic statistics to have a rectangular
> array, and extract both certain rows (tuples of observations on
> variables) and certain columns (variables).  For example you might
> have data on populations of American states from 1900 to 2012, and
> extract the data on New England states from 1946 to 2012 for analysis.

When Steven first brought up this PEP on comp.lang.python, my main concern
was basically, "we have SciPy, why do we need this?" Steven's response, which
I have come to accept, is that there are uses for basic statistics for
which SciPy's
stats module would be overkill.

However, once you start slicing your data structure along more than one axis, I
think you very quickly will find that you need numpy arrays for performance
reasons, at which point you might as go "all the way" and install SciPy. I don't
think slicing along multiple dimensions should be a significant concern for this
package.

Alternatively, I thought there was discussion a long time ago about
getting numpy's
(or even further back, numeric's?) array type into the core. Python
has an array type
which I don't think gets a lot of use (or love). Might it be
worthwhile to make sure the
PEP 450 package works with that? Then extend it to multiple dimensions? Or just
bite the bullet and get numpy's array type into the Python core once
and for all?

Sort of Tulip for arrays...

Skip

From martin at v.loewis.de  Mon Sep  9 13:43:08 2013
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 09 Sep 2013 13:43:08 +0200
Subject: [Python-Dev] unicodedata module is out of date
In-Reply-To: <CAPMPPoRMBsBbAasu8ewEwqJOJARLAsctJricjKdVYVkXdLy8Tg@mail.gmail.com>
References: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com>
 <5229F6DB.40000@mrabarnett.plus.com>
 <CAPMPPoRMBsBbAasu8ewEwqJOJARLAsctJricjKdVYVkXdLy8Tg@mail.gmail.com>
Message-ID: <522DB44C.8050802@v.loewis.de>

Am 06.09.13 17:55, schrieb Andrew Miller:
> Are there plans to add the extra data from the other UCD files to this
> module?  At the moment I am using a module from
> https://gist.github.com/anonymous/2204527 to obtain the script of a
> character but it would be nice if this was available from the standard
> library.

Well, it is available, and new versions of the UCD are added to new
Python releases. Please consider Python 2 as dead wrt. Unicode support.

Regards,
Martin


From martin at v.loewis.de  Mon Sep  9 13:41:21 2013
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 09 Sep 2013 13:41:21 +0200
Subject: [Python-Dev] unicodedata module is out of date
In-Reply-To: <l0d6lj$lnf$1@ger.gmane.org>
References: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com>
 <5229F6DB.40000@mrabarnett.plus.com>
 <CAPMPPoRMBsBbAasu8ewEwqJOJARLAsctJricjKdVYVkXdLy8Tg@mail.gmail.com>
 <l0d6lj$lnf$1@ger.gmane.org>
Message-ID: <522DB3E1.30105@v.loewis.de>

Am 06.09.13 20:24, schrieb Terry Reedy:
>> In Python 2.7.5 it is set to '5.2.0' so it looks as though this version
>> is no longer being updated.
> 
> In general, new features do not go into bugfix releases (x.y.z, where z
>>= 1). Updating the unidate_version add new features to the unicodedata
> module.
> 

One might argue that an update of the UCD data is within the scope of
2.7, since it's just data, not code that is being changed.

I'd argue against that, since this specific change has a chance of
breaking existing tests that people might have.

Of course, it is up the the release manager of 2.7 to decide on that
if such a change would be proposed.

Regards,
Martin


From martin at v.loewis.de  Mon Sep  9 13:38:09 2013
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 09 Sep 2013 13:38:09 +0200
Subject: [Python-Dev] DTRACE support
In-Reply-To: <CAP7+vJJsPVn_EHjfYeYZhK=nd3VfBZ3h_b6tTsShTt6gUSVNaw@mail.gmail.com>
References: <5229EFE7.8040702@jcea.es>
 <CAP7+vJJsPVn_EHjfYeYZhK=nd3VfBZ3h_b6tTsShTt6gUSVNaw@mail.gmail.com>
Message-ID: <522DB321.1080905@v.loewis.de>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Am 06.09.13 17:14, schrieb Guido van Rossum:
> I've heard good things about DTRACE but never used it myself.
> 
> Do I understand correctly that you have to build a separate Python 
> executable with it turned on?

My understanding is that you would normally have dtrace support built
on systems that support it. The claim of the dtrace authors is that
there is virtually no overhead in having trace points integrated that
are not active; it achieves that by patching trace instructions over
the NOPs that have been put there during compile time.

Regards,
Martin

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.18 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlItsyEACgkQavBT8H2dyNLhNgCfXfgXakq9wgvNGADrYNMvvAJu
05YAn0SxHrOCgoW8k9wjW+ajj+0jtpg9
=E/GT
-----END PGP SIGNATURE-----

From larry at hastings.org  Mon Sep  9 14:02:27 2013
From: larry at hastings.org (Larry Hastings)
Date: Mon, 09 Sep 2013 21:02:27 +0900
Subject: [Python-Dev] [RELEASED] Python 3.4.0a2
Message-ID: <522DB8D3.1030803@hastings.org>


On behalf of the Python development team, I'm chuffed to announce the
second alpha release of Python 3.4.

This is a preview release, and its use is not recommended for
production settings.

Python 3.4 includes a range of improvements of the 3.x series, including
hundreds of small improvements and bug fixes.  Major new features and
changes in the 3.4 release series so far include:

* PEP 435, a standardized "enum" module
* PEP 442, improved semantics for object finalization
* PEP 443, adding single-dispatch generic functions to the standard library
* PEP 445, a new C API for implementing custom memory allocators
* PEP 446, changing file descriptors to not be inherited by default
            in subprocesses
* PEP 447, a new magic method for metaclasses (``__typelookup__``)
* PEP 448, making automatic sequence unpacking more general


To download Python 3.4.0a2 visit:

     http://www.python.org/download/releases/3.4.0/


Please consider trying Python 3.4.0a2 with your code and reporting any
issues you notice to:

      http://bugs.python.org/


Enjoy!

--
Larry Hastings, Release Manager
larry at hastings.org
(on behalf of the entire python-dev team and 3.4's contributors)

From steve at pearwood.info  Mon Sep  9 14:07:58 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 9 Sep 2013 22:07:58 +1000
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <CANc-5UxZ5M7MXxCPYY6Tfn0rpjRUT+yjyOWnGgFO8ROSyWfrHQ@mail.gmail.com>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
 <CAHVvXxRAi_HHFOD2dY8UGsXpbgd-LyFU9rsg9x6Uk_=VuKG4hA@mail.gmail.com>
 <CAP7+vJK0HRT4p0MEqraCr8EY1MN29E6CnkVcxvOks_s1rMka_w@mail.gmail.com>
 <87ppsi8zwx.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CANc-5UxZ5M7MXxCPYY6Tfn0rpjRUT+yjyOWnGgFO8ROSyWfrHQ@mail.gmail.com>
Message-ID: <20130909120758.GO26319@ando>

On Mon, Sep 09, 2013 at 05:44:43AM -0500, Skip Montanaro wrote:
> > However, it's common in economic statistics to have a rectangular
> > array, and extract both certain rows (tuples of observations on
> > variables) and certain columns (variables).  For example you might
> > have data on populations of American states from 1900 to 2012, and
> > extract the data on New England states from 1946 to 2012 for analysis.
> 
> When Steven first brought up this PEP on comp.lang.python, my main concern
> was basically, "we have SciPy, why do we need this?" Steven's response, which
> I have come to accept, is that there are uses for basic statistics for
> which SciPy's
> stats module would be overkill.
> 
> However, once you start slicing your data structure along more than one axis, I
> think you very quickly will find that you need numpy arrays for performance
> reasons, at which point you might as go "all the way" and install SciPy. I don't
> think slicing along multiple dimensions should be a significant concern for this
> package.

I agree. I'm not interested in trying to compete with numpy in areas 
where numpy is best. That's a fight any pure-Python module is going to 
lose :-)


> Alternatively, I thought there was discussion a long time ago about
> getting numpy's
> (or even further back, numeric's?) array type into the core. Python
> has an array type
> which I don't think gets a lot of use (or love). Might it be
> worthwhile to make sure the
> PEP 450 package works with that? Then extend it to multiple dimensions? Or just
> bite the bullet and get numpy's array type into the Python core once
> and for all?

I haven't tested PEP 450 statistics with numpy array, but any sequence 
type ought to work. While I haven't done extensive testing on the 
array.array type, basic testing shows that it works as expected:

py> import array
py> import statistics
py> data = array.array('f', range(1, 101))
py> statistics.mean(data)
50.5
py> statistics.variance(data)
841.6666666666666



-- 
Steven

From brett at python.org  Mon Sep  9 14:16:06 2013
From: brett at python.org (Brett Cannon)
Date: Mon, 9 Sep 2013 08:16:06 -0400
Subject: [Python-Dev] [python-committers] [RELEASED] Python 3.4.0a2
In-Reply-To: <522DB8D3.1030803@hastings.org>
References: <522DB8D3.1030803@hastings.org>
Message-ID: <CAP1=2W4HNkPodjskLXtT=RvaC0DXhtmQbggRJYgQX_4VG-axXw@mail.gmail.com>

On Mon, Sep 9, 2013 at 8:02 AM, Larry Hastings <larry at hastings.org> wrote:

>
> On behalf of the Python development team, I'm chuffed to announce the
> second alpha release of Python 3.4.
>
> This is a preview release, and its use is not recommended for
> production settings.
>
> Python 3.4 includes a range of improvements of the 3.x series, including
> hundreds of small improvements and bug fixes.  Major new features and
> changes in the 3.4 release series so far include:
>
> * PEP 435, a standardized "enum" module
> * PEP 442, improved semantics for object finalization
> * PEP 443, adding single-dispatch generic functions to the standard library
> * PEP 445, a new C API for implementing custom memory allocators
> * PEP 446, changing file descriptors to not be inherited by default
>            in subprocesses
> * PEP 447, a new magic method for metaclasses (``__typelookup__``)
> * PEP 448, making automatic sequence unpacking more general
>

Those last two PEPs are still in draft form and not accepted nor have any
committed code yet.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130909/4ccde394/attachment.html>

From ncoghlan at gmail.com  Mon Sep  9 13:56:05 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 9 Sep 2013 21:56:05 +1000
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <CANc-5UxZ5M7MXxCPYY6Tfn0rpjRUT+yjyOWnGgFO8ROSyWfrHQ@mail.gmail.com>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
 <CAHVvXxRAi_HHFOD2dY8UGsXpbgd-LyFU9rsg9x6Uk_=VuKG4hA@mail.gmail.com>
 <CAP7+vJK0HRT4p0MEqraCr8EY1MN29E6CnkVcxvOks_s1rMka_w@mail.gmail.com>
 <87ppsi8zwx.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CANc-5UxZ5M7MXxCPYY6Tfn0rpjRUT+yjyOWnGgFO8ROSyWfrHQ@mail.gmail.com>
Message-ID: <CADiSq7fXsYCy125GyX9HenHkzXMs_2HZu4Smrc+q5_J-tWyv1A@mail.gmail.com>

On 9 Sep 2013 20:46, "Skip Montanaro" <skip at pobox.com> wrote:
>
> > However, it's common in economic statistics to have a rectangular
> > array, and extract both certain rows (tuples of observations on
> > variables) and certain columns (variables).  For example you might
> > have data on populations of American states from 1900 to 2012, and
> > extract the data on New England states from 1946 to 2012 for analysis.
>
> When Steven first brought up this PEP on comp.lang.python, my main concern
> was basically, "we have SciPy, why do we need this?" Steven's response,
which
> I have come to accept, is that there are uses for basic statistics for
> which SciPy's
> stats module would be overkill.
>
> However, once you start slicing your data structure along more than one
axis, I
> think you very quickly will find that you need numpy arrays for
performance
> reasons, at which point you might as go "all the way" and install SciPy.
I don't
> think slicing along multiple dimensions should be a significant concern
for this
> package.
>
> Alternatively, I thought there was discussion a long time ago about
> getting numpy's
> (or even further back, numeric's?) array type into the core. Python
> has an array type
> which I don't think gets a lot of use (or love). Might it be
> worthwhile to make sure the
> PEP 450 package works with that? Then extend it to multiple dimensions?
Or just
> bite the bullet and get numpy's array type into the Python core once
> and for all?
>
> Sort of Tulip for arrays...

Aka memoryview :)

Stefan Krah already fixed most of the multidimensional support issues in
3.3 (including the "cast" method to reinterpret the contents in a different
format). The main missing API elements are multidimensional slicing and the
ability to export them from types defined in Python.

Cheers,
Nick.

>
> Skip
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130909/72a66d88/attachment.html>

From victor.stinner at gmail.com  Mon Sep  9 14:30:50 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 9 Sep 2013 14:30:50 +0200
Subject: [Python-Dev] [python-committers] [RELEASED] Python 3.4.0a2
In-Reply-To: <522DB8D3.1030803@hastings.org>
References: <522DB8D3.1030803@hastings.org>
Message-ID: <CAMpsgwZ3Ltsc4dKRnN5zBVvcSF4WFX4YQRcG9oaq_iMvx0NjvQ@mail.gmail.com>

2013/9/9 Larry Hastings <larry at hastings.org>:
> Python 3.4 includes a range of improvements of the 3.x series, including
> hundreds of small improvements and bug fixes.  Major new features and
> changes in the 3.4 release series so far include:
>
> * PEP 446, changing file descriptors to not be inherited by default
>            in subprocesses

The title of the PEP is "Make newly created file descriptors
non-inheritable". It has an impact on all functions creating files and
sockets not only the subprocess module.

You can also add a link to the nice What?s New In Python 3.4 document:
http://docs.python.org/dev/whatsnew/3.4.html

Victor

From oscar.j.benjamin at gmail.com  Mon Sep  9 14:57:50 2013
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Mon, 9 Sep 2013 13:57:50 +0100
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <CADiSq7fXsYCy125GyX9HenHkzXMs_2HZu4Smrc+q5_J-tWyv1A@mail.gmail.com>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
 <CAHVvXxRAi_HHFOD2dY8UGsXpbgd-LyFU9rsg9x6Uk_=VuKG4hA@mail.gmail.com>
 <CAP7+vJK0HRT4p0MEqraCr8EY1MN29E6CnkVcxvOks_s1rMka_w@mail.gmail.com>
 <87ppsi8zwx.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CANc-5UxZ5M7MXxCPYY6Tfn0rpjRUT+yjyOWnGgFO8ROSyWfrHQ@mail.gmail.com>
 <CADiSq7fXsYCy125GyX9HenHkzXMs_2HZu4Smrc+q5_J-tWyv1A@mail.gmail.com>
Message-ID: <CAHVvXxTy83-=GJvq3ud6Vee2pF5FuR5n_C3Ac_+aCCjgDwZOkw@mail.gmail.com>

On 9 September 2013 12:56, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> Alternatively, I thought there was discussion a long time ago about
>> getting numpy's
>> (or even further back, numeric's?) array type into the core. Python
>> has an array type
>> which I don't think gets a lot of use (or love). Might it be
>> worthwhile to make sure the
>> PEP 450 package works with that? Then extend it to multiple dimensions? Or
>> just
>> bite the bullet and get numpy's array type into the Python core once
>> and for all?
>>
>> Sort of Tulip for arrays...
>
> Aka memoryview :)
>
> Stefan Krah already fixed most of the multidimensional support issues in 3.3
> (including the "cast" method to reinterpret the contents in a different
> format). The main missing API elements are multidimensional slicing and the
> ability to export them from types defined in Python.

Being very familiar with numpy's ndarrays and not so much with
memoryviews this prompted me to go and have a look at them.

How exactly are you supposed to create a multidimensional array using
memoryviews? The best I could come up with was something like:

$ py -3.3
Python 3.3.2 (v3.3.2:d047928ae3f6, May 16 2013, 00:03:43) [MSC v.1600
32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import array
>>> v = memoryview(array.array('b', [1, 2, 3, 4])).cast('b', (2, 2))
>>> v.shape
(2, 2)

However I don't seem to be able to access the elements:

>>> v[0, 1]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: memoryview: invalid slice key
>>> v[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NotImplementedError: multi-dimensional sub-views are not implemented
>>> v[0:1]
<memory at 0x00DF8B70>
>>> v[0:1].shape
(1, 2)
>>> v[0:1][0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NotImplementedError: multi-dimensional sub-views are not implemented
>>> v[1]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NotImplementedError: multi-dimensional sub-views are not implemented
>>> v[:, 1]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: memoryview: invalid slice key
>>> v[1, :]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: memoryview: invalid slice key

And the .cast method bails if you try to use a more useful type code:

>>> v = memoryview(array.array('q', [1, 2, 3, 4])).cast('q', (2, 2))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: memoryview: cannot cast between two non-byte formats


Oscar

From ncoghlan at gmail.com  Mon Sep  9 16:15:25 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 10 Sep 2013 00:15:25 +1000
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <CAHVvXxTy83-=GJvq3ud6Vee2pF5FuR5n_C3Ac_+aCCjgDwZOkw@mail.gmail.com>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
 <CAHVvXxRAi_HHFOD2dY8UGsXpbgd-LyFU9rsg9x6Uk_=VuKG4hA@mail.gmail.com>
 <CAP7+vJK0HRT4p0MEqraCr8EY1MN29E6CnkVcxvOks_s1rMka_w@mail.gmail.com>
 <87ppsi8zwx.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CANc-5UxZ5M7MXxCPYY6Tfn0rpjRUT+yjyOWnGgFO8ROSyWfrHQ@mail.gmail.com>
 <CADiSq7fXsYCy125GyX9HenHkzXMs_2HZu4Smrc+q5_J-tWyv1A@mail.gmail.com>
 <CAHVvXxTy83-=GJvq3ud6Vee2pF5FuR5n_C3Ac_+aCCjgDwZOkw@mail.gmail.com>
Message-ID: <CADiSq7czgXzgTFUr-AANWGUx_umRggwKbOvGR3ixTMv2odhs_w@mail.gmail.com>

On 9 Sep 2013 22:58, "Oscar Benjamin" <oscar.j.benjamin at gmail.com> wrote:
>
> On 9 September 2013 12:56, Nick Coghlan <ncoghlan at gmail.com> wrote:
> >> Alternatively, I thought there was discussion a long time ago about
> >> getting numpy's
> >> (or even further back, numeric's?) array type into the core. Python
> >> has an array type
> >> which I don't think gets a lot of use (or love). Might it be
> >> worthwhile to make sure the
> >> PEP 450 package works with that? Then extend it to multiple
dimensions? Or
> >> just
> >> bite the bullet and get numpy's array type into the Python core once
> >> and for all?
> >>
> >> Sort of Tulip for arrays...
> >
> > Aka memoryview :)
> >
> > Stefan Krah already fixed most of the multidimensional support issues
in 3.3
> > (including the "cast" method to reinterpret the contents in a different
> > format). The main missing API elements are multidimensional slicing and
the
> > ability to export them from types defined in Python.
>
> Being very familiar with numpy's ndarrays and not so much with
> memoryviews this prompted me to go and have a look at them.
>
> How exactly are you supposed to create a multidimensional array using
> memoryviews? The best I could come up with was something like:
>
> $ py -3.3
> Python 3.3.2 (v3.3.2:d047928ae3f6, May 16 2013, 00:03:43) [MSC v.1600
> 32 bit (Intel)] on win32
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import array
> >>> v = memoryview(array.array('b', [1, 2, 3, 4])).cast('b', (2, 2))
> >>> v.shape
> (2, 2)
>
> However I don't seem to be able to access the elements:
>
> >>> v[0, 1]
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> TypeError: memoryview: invalid slice key
> >>> v[0]
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> NotImplementedError: multi-dimensional sub-views are not implemented
> >>> v[0:1]
> <memory at 0x00DF8B70>
> >>> v[0:1].shape
> (1, 2)
> >>> v[0:1][0]
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> NotImplementedError: multi-dimensional sub-views are not implemented
> >>> v[1]
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> NotImplementedError: multi-dimensional sub-views are not implemented
> >>> v[:, 1]
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> TypeError: memoryview: invalid slice key
> >>> v[1, :]
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> TypeError: memoryview: invalid slice key
>
> And the .cast method bails if you try to use a more useful type code:
>
> >>> v = memoryview(array.array('q', [1, 2, 3, 4])).cast('q', (2, 2))
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> TypeError: memoryview: cannot cast between two non-byte formats

Oops, forgot the type casting restrictions, too.

My main point was that PEP 3118 is already intended as the tulip equivalent
for multi-dimensional arrays, and memoryview is the stdlib API for that.
It's just incomplete, since most serious multi-dimensional use cases
involve skipping memoryview and go straight to NumPy or one of the image
libraries.

As far as I am aware, there's no opposition to fixing the multi-dimensional
support in memoryview *per se*, just the usual concerns about
maintainability and a question of volunteers with the time to actually
resolve the relevant open issues on the bug tracker. The fairly extensive
3.3 changes focused on fixing stuff that was previously outright broken,
but several limitations remain, often because the best API wasn't clear, or
because it reached the point where "just use NumPy" seemed like a
justifiable answer.

Cheers,
Nick.

>
>
> Oscar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/54651c2a/attachment.html>

From ncoghlan at gmail.com  Mon Sep  9 16:20:54 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 10 Sep 2013 00:20:54 +1000
Subject: [Python-Dev] [Python-checkins] cpython: Post-3.4.0a2-release
	fixups.
In-Reply-To: <3cYSwh0rHdz7Lkm@mail.python.org>
References: <3cYSwh0rHdz7Lkm@mail.python.org>
Message-ID: <CADiSq7fVsbLCks=oK4UxgJNnzngTmEYLmOW_HEj_OXHMfJwUow@mail.gmail.com>

On 9 Sep 2013 22:15, "larry.hastings" <python-checkins at python.org> wrote:
>
> http://hg.python.org/cpython/rev/6b211a0c8042
> changeset:   85645:6b211a0c8042
> user:        Larry Hastings <larry at hastings.org>
> date:        Mon Sep 09 21:08:52 2013 +0900
> summary:
>   Post-3.4.0a2-release fixups.
>
> files:
>   Include/patchlevel.h |   2 +-
>   Misc/NEWS            |  14 +++++++++++++-
>   2 files changed, 14 insertions(+), 2 deletions(-)
>
>
> diff --git a/Include/patchlevel.h b/Include/patchlevel.h
> --- a/Include/patchlevel.h
> +++ b/Include/patchlevel.h
> @@ -23,7 +23,7 @@
>  #define PY_RELEASE_SERIAL      2
>
>  /* Version as a string */
> -#define PY_VERSION             "3.4.0a2"
> +#define PY_VERSION             "3.4.0a2+"
>  /*--end constants--*/
>
>  /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2.
> diff --git a/Misc/NEWS b/Misc/NEWS
> --- a/Misc/NEWS
> +++ b/Misc/NEWS
> @@ -2,10 +2,22 @@
>  Python News
>  +++++++++++
>
> +What's New in Python 3.4.0 Alpha 3?
> +===================================
> +
> +Projected Release date: 2013-09-29
> +
> +Core and Builtins
> +-----------------
> +
> +Library
> +-------
> +
> +

I had already pushed alpha 3 entries in NEWS, so something seems to have
gone wrong here.

Perhaps, if RMs are preparing the release out of tree, we could get the
NEWS file headings on default updated immediately after the last included
commit?

Cheers,
Nick.

>  What's New in Python 3.4.0 Alpha 2?
>  ===================================
>
> -Projected Release date: 2013-09-08
> +Release date: 2013-09-09
>
>  Core and Builtins
>  -----------------
>
> --
> Repository URL: http://hg.python.org/cpython
>
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> https://mail.python.org/mailman/listinfo/python-checkins
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/02037892/attachment.html>

From ethan at stoneleaf.us  Mon Sep  9 16:30:12 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 09 Sep 2013 07:30:12 -0700
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
 <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
Message-ID: <522DDB74.5080609@stoneleaf.us>

On 07/30/2013 11:17 PM, Ronald Oussoren wrote:
>
> And something I forgot to ask: is anyone willing to be the BDFL-Delegate for
> PEP 447?

*Bump*.

It would be nice if this could make into 3.4.

--
~Ethan~

From rdmurray at bitdance.com  Mon Sep  9 16:54:22 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Mon, 09 Sep 2013 10:54:22 -0400
Subject: [Python-Dev] [python-committers] [RELEASED] Python 3.4.0a2
In-Reply-To: <20130909144551.76b84640@pitrou.net>
References: <522DB8D3.1030803@hastings.org>
 <CAMpsgwZ3Ltsc4dKRnN5zBVvcSF4WFX4YQRcG9oaq_iMvx0NjvQ@mail.gmail.com>
 <20130909144551.76b84640@pitrou.net>
Message-ID: <20130909145422.B5689250166@webabinitio.net>

On Mon, 09 Sep 2013 14:45:51 +0200, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Le Mon, 9 Sep 2013 14:30:50 +0200,
> Victor Stinner <victor.stinner at gmail.com> a ??crit :
> > 2013/9/9 Larry Hastings <larry at hastings.org>:
> > > Python 3.4 includes a range of improvements of the 3.x series,
> > > including hundreds of small improvements and bug fixes.  Major new
> > > features and changes in the 3.4 release series so far include:
> > >
> > > * PEP 446, changing file descriptors to not be inherited by default
> > >            in subprocesses
> > 
> > The title of the PEP is "Make newly created file descriptors
> > non-inheritable". It has an impact on all functions creating files and
> > sockets not only the subprocess module.
> 
> I don't think Larry's description is wrong. "Non-inheritable" is a
> shorthand for "non-inheritable in subprocesses" with "subprocesses"
> taken in the general sense (i.e. not only created with the subprocess
> module).

Not wrong, but definitely confusing.  It is worth clarifying *somehow*
that this does not apply only to the subprocess module, which is what
a naive (or fast) reader will assume.

--David

From jcea at jcea.es  Mon Sep  9 17:11:21 2013
From: jcea at jcea.es (Jesus Cea)
Date: Mon, 09 Sep 2013 17:11:21 +0200
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <20130906193430.AFA14250816@webabinitio.net>
References: <5228C00F.7000209@jcea.es> <l0d311$b7n$1@ger.gmane.org>
 <6DB0A7E3-FADD-4E35-A983-02A7C0D4CAE3@stufft.io>
 <20130906191130.B6511250816@webabinitio.net>
 <7087E6B8-1CD4-4002-AD3F-5578B045A3CB@stufft.io>
 <20130906193430.AFA14250816@webabinitio.net>
Message-ID: <522DE519.7080909@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/09/13 21:34, R. David Murray wrote:
> Note that I said that single signon *itself* was overrated.  If you
> use the same token to authenticate to multiple sites (and here the
> 'token' is the email address) then your identities on those sites
> are ipso facto associated with each other.  *If* that email address
> is also never leaked (never displayed, even to other signed on
> users, all communication with the site encrypted), then you only
> have to worry if the sites exchange information about their
> accounts, or if the government comes knocking on their doors....
> 
> Yes, I'm paranoid.  That doesn't mean they aren't listening.

Being paranoid is good. Fix for this is actually trivial: Use
different emails for different "personalities".

If you are doing things you really NEED to hide, virtual machines and
TOR is the way to go.

- -- 
Jes?s Cea Avi?n                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQCVAwUBUi3lGZlgi5GaxT1NAQLykAQAi8WDuWmEAAX7bP1glDT8iLrMRpKlu+Vh
WndX9ObB/os2D9RZkL7DZB01EDMRvfjGlWFm3gQV0CbM9smkgGkhJNLuxYzA2fpK
PQlbO4KUKCoQG7qm413TA1g0xxOGzG2n9g2kJisFBCNu2Y2PXroUhm6p41CbTd89
ovwZLLUcPZU=
=EQu4
-----END PGP SIGNATURE-----

From guido at python.org  Mon Sep  9 17:27:41 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 9 Sep 2013 08:27:41 -0700
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <522DDB74.5080609@stoneleaf.us>
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
 <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us>
Message-ID: <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>

Let's just accept this PEP. It looks like a nice addition to the metaclass
machinery and I don't think we'll get much more useful feedback by waiting.


On Mon, Sep 9, 2013 at 7:30 AM, Ethan Furman <ethan at stoneleaf.us> wrote:

> On 07/30/2013 11:17 PM, Ronald Oussoren wrote:
>
>>
>> And something I forgot to ask: is anyone willing to be the BDFL-Delegate
>> for
>> PEP 447?
>>
>
> *Bump*.
>
> It would be nice if this could make into 3.4.
>
> --
> ~Ethan~
>
> ______________________________**_________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/**mailman/listinfo/python-dev<https://mail.python.org/mailman/listinfo/python-dev>
> Unsubscribe: https://mail.python.org/**mailman/options/python-dev/**
> guido%40python.org<https://mail.python.org/mailman/options/python-dev/guido%40python.org>
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130909/e78eb7d7/attachment.html>

From benjamin at python.org  Mon Sep  9 17:32:20 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Mon, 9 Sep 2013 11:32:20 -0400
Subject: [Python-Dev] [Python-checkins] cpython: Post-3.4.0a2-release
	fixups.
In-Reply-To: <CADiSq7fVsbLCks=oK4UxgJNnzngTmEYLmOW_HEj_OXHMfJwUow@mail.gmail.com>
References: <3cYSwh0rHdz7Lkm@mail.python.org>
 <CADiSq7fVsbLCks=oK4UxgJNnzngTmEYLmOW_HEj_OXHMfJwUow@mail.gmail.com>
Message-ID: <CAPZV6o8jzs_nUkgtJL0u6vxoGgX=3JihF5FXUovnLGK96m1o0Q@mail.gmail.com>

Well, it's important for the release manager to make sure what the
script is doing is sane. :)

2013/9/9 Nick Coghlan <ncoghlan at gmail.com>:
>
> On 9 Sep 2013 22:15, "larry.hastings" <python-checkins at python.org> wrote:
>>
>> http://hg.python.org/cpython/rev/6b211a0c8042
>> changeset:   85645:6b211a0c8042
>> user:        Larry Hastings <larry at hastings.org>
>> date:        Mon Sep 09 21:08:52 2013 +0900
>> summary:
>>   Post-3.4.0a2-release fixups.
>>
>> files:
>>   Include/patchlevel.h |   2 +-
>>   Misc/NEWS            |  14 +++++++++++++-
>>   2 files changed, 14 insertions(+), 2 deletions(-)
>>
>>
>> diff --git a/Include/patchlevel.h b/Include/patchlevel.h
>> --- a/Include/patchlevel.h
>> +++ b/Include/patchlevel.h
>> @@ -23,7 +23,7 @@
>>  #define PY_RELEASE_SERIAL      2
>>
>>  /* Version as a string */
>> -#define PY_VERSION             "3.4.0a2"
>> +#define PY_VERSION             "3.4.0a2+"
>>  /*--end constants--*/
>>
>>  /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2.
>> diff --git a/Misc/NEWS b/Misc/NEWS
>> --- a/Misc/NEWS
>> +++ b/Misc/NEWS
>> @@ -2,10 +2,22 @@
>>  Python News
>>  +++++++++++
>>
>> +What's New in Python 3.4.0 Alpha 3?
>> +===================================
>> +
>> +Projected Release date: 2013-09-29
>> +
>> +Core and Builtins
>> +-----------------
>> +
>> +Library
>> +-------
>> +
>> +
>
> I had already pushed alpha 3 entries in NEWS, so something seems to have
> gone wrong here.
>
> Perhaps, if RMs are preparing the release out of tree, we could get the NEWS
> file headings on default updated immediately after the last included commit?
>
> Cheers,
> Nick.
>
>>  What's New in Python 3.4.0 Alpha 2?
>>  ===================================
>>
>> -Projected Release date: 2013-09-08
>> +Release date: 2013-09-09
>>
>>  Core and Builtins
>>  -----------------
>>
>> --
>> Repository URL: http://hg.python.org/cpython
>>
>> _______________________________________________
>> Python-checkins mailing list
>> Python-checkins at python.org
>> https://mail.python.org/mailman/listinfo/python-checkins
>>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org
>



-- 
Regards,
Benjamin

From rdmurray at bitdance.com  Mon Sep  9 17:36:32 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Mon, 09 Sep 2013 11:36:32 -0400
Subject: [Python-Dev] Offtopic: paranoia
In-Reply-To: <522DE519.7080909@jcea.es>
References: <5228C00F.7000209@jcea.es> <l0d311$b7n$1@ger.gmane.org>
 <6DB0A7E3-FADD-4E35-A983-02A7C0D4CAE3@stufft.io>
 <20130906191130.B6511250816@webabinitio.net>
 <7087E6B8-1CD4-4002-AD3F-5578B045A3CB@stufft.io>
 <20130906193430.AFA14250816@webabinitio.net> <522DE519.7080909@jcea.es>
Message-ID: <20130909153632.EDACD250166@webabinitio.net>

On Mon, 09 Sep 2013 17:11:21 +0200, Jesus Cea <jcea at jcea.es> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 06/09/13 21:34, R. David Murray wrote:
> > Note that I said that single signon *itself* was overrated.  If you
> > use the same token to authenticate to multiple sites (and here the
> > 'token' is the email address) then your identities on those sites
> > are ipso facto associated with each other.  *If* that email address
> > is also never leaked (never displayed, even to other signed on
> > users, all communication with the site encrypted), then you only
> > have to worry if the sites exchange information about their
> > accounts, or if the government comes knocking on their doors....
> > 
> > Yes, I'm paranoid.  That doesn't mean they aren't listening.
> 
> Being paranoid is good. Fix for this is actually trivial: Use
> different emails for different "personalities".

Yes, that's exactly my point.

> If you are doing things you really NEED to hide, virtual machines and
> TOR is the way to go.

Well, it would helpful if a lot more people started routing traffic
through TOR even when they didn't NEED to.  I plan to start doing so soon.

--David

From mark at hotpy.org  Mon Sep  9 17:43:31 2013
From: mark at hotpy.org (Mark Shannon)
Date: Mon, 09 Sep 2013 16:43:31 +0100
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
 <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
Message-ID: <522DECA3.2030709@hotpy.org>

I would like time to investigate this further, but at the moment I think it will either make attribute lookup poorly defined or slow.

Of the top of my head, the problem as a I see it is basically this:
Currently, type.__getattribute__() is a fixed point in the lookup of attributes.
The proposal means that a fixed point is not reached until the cls parameter of type.__getattribute__() is either object or type,
otherwise type.__getattribute__() and type.__locallookup__ must bounce back and forth.

This will slow down *every* attribute lookup for what is a fairly obscure use case.

Cheers,
Mark.

On 09/09/13 16:27, Guido van Rossum wrote:
> Let's just accept this PEP. It looks like a nice addition to the metaclass machinery and I don't think we'll get much more useful feedback by waiting.
>
>
> On Mon, Sep 9, 2013 at 7:30 AM, Ethan Furman <ethan at stoneleaf.us <mailto:ethan at stoneleaf.us>> wrote:
>
>     On 07/30/2013 11:17 PM, Ronald Oussoren wrote:
>
>
>         And something I forgot to ask: is anyone willing to be the BDFL-Delegate for
>         PEP 447?
>
>
>     *Bump*.
>
>     It would be nice if this could make into 3.4.
>
>     --
>     ~Ethan~
>
>     _________________________________________________
>     Python-Dev mailing list
>     Python-Dev at python.org <mailto:Python-Dev at python.org>
>     https://mail.python.org/__mailman/listinfo/python-dev <https://mail.python.org/mailman/listinfo/python-dev>
>     Unsubscribe: https://mail.python.org/__mailman/options/python-dev/__guido%40python.org <https://mail.python.org/mailman/options/python-dev/guido%40python.org>
>
>
>
>
> --
> --Guido van Rossum (python.org/~guido <http://python.org/~guido>)
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/mark%40hotpy.org
>

From guido at python.org  Mon Sep  9 18:03:23 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 9 Sep 2013 09:03:23 -0700
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <522DECA3.2030709@hotpy.org>
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
 <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
 <522DECA3.2030709@hotpy.org>
Message-ID: <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com>

OK, how much time do you need?

--Guido van Rossum (sent from Android phone)
On Sep 9, 2013 8:44 AM, "Mark Shannon" <mark at hotpy.org> wrote:

> I would like time to investigate this further, but at the moment I think
> it will either make attribute lookup poorly defined or slow.
>
> Of the top of my head, the problem as a I see it is basically this:
> Currently, type.__getattribute__() is a fixed point in the lookup of
> attributes.
> The proposal means that a fixed point is not reached until the cls
> parameter of type.__getattribute__() is either object or type,
> otherwise type.__getattribute__() and type.__locallookup__ must bounce
> back and forth.
>
> This will slow down *every* attribute lookup for what is a fairly obscure
> use case.
>
> Cheers,
> Mark.
>
> On 09/09/13 16:27, Guido van Rossum wrote:
>
>> Let's just accept this PEP. It looks like a nice addition to the
>> metaclass machinery and I don't think we'll get much more useful feedback
>> by waiting.
>>
>>
>> On Mon, Sep 9, 2013 at 7:30 AM, Ethan Furman <ethan at stoneleaf.us <mailto:
>> ethan at stoneleaf.us>> wrote:
>>
>>     On 07/30/2013 11:17 PM, Ronald Oussoren wrote:
>>
>>
>>         And something I forgot to ask: is anyone willing to be the
>> BDFL-Delegate for
>>         PEP 447?
>>
>>
>>     *Bump*.
>>
>>     It would be nice if this could make into 3.4.
>>
>>     --
>>     ~Ethan~
>>
>>     ______________________________**___________________
>>     Python-Dev mailing list
>>     Python-Dev at python.org <mailto:Python-Dev at python.org>
>>     https://mail.python.org/__**mailman/listinfo/python-dev<https://mail.python.org/__mailman/listinfo/python-dev><
>> https://mail.python.org/**mailman/listinfo/python-dev<https://mail.python.org/mailman/listinfo/python-dev>
>> >
>>     Unsubscribe: https://mail.python.org/__**
>> mailman/options/python-dev/__**guido%40python.org<https://mail.python.org/__mailman/options/python-dev/__guido%40python.org><
>> https://mail.python.org/**mailman/options/python-dev/**guido%40python.org<https://mail.python.org/mailman/options/python-dev/guido%40python.org>
>> >
>>
>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido <http://python.org/~guido>)
>>
>>
>> ______________________________**_________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/**mailman/listinfo/python-dev<https://mail.python.org/mailman/listinfo/python-dev>
>> Unsubscribe: https://mail.python.org/**mailman/options/python-dev/**
>> mark%40hotpy.org<https://mail.python.org/mailman/options/python-dev/mark%40hotpy.org>
>>
>>  ______________________________**_________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/**mailman/listinfo/python-dev<https://mail.python.org/mailman/listinfo/python-dev>
> Unsubscribe: https://mail.python.org/**mailman/options/python-dev/**
> guido%40python.org<https://mail.python.org/mailman/options/python-dev/guido%40python.org>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130909/22254236/attachment-0001.html>

From mark at hotpy.org  Mon Sep  9 18:06:28 2013
From: mark at hotpy.org (Mark Shannon)
Date: Mon, 09 Sep 2013 17:06:28 +0100
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com>
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
 <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
 <522DECA3.2030709@hotpy.org>
 <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com>
Message-ID: <522DF204.7040109@hotpy.org>

I'll look into it this evening.

On 09/09/13 17:03, Guido van Rossum wrote:
> OK, how much time do you need?
>
> --Guido van Rossum (sent from Android phone)
>
> On Sep 9, 2013 8:44 AM, "Mark Shannon" <mark at hotpy.org <mailto:mark at hotpy.org>> wrote:
>
>     I would like time to investigate this further, but at the moment I think it will either make attribute lookup poorly defined or slow.
>
>     Of the top of my head, the problem as a I see it is basically this:
>     Currently, type.__getattribute__() is a fixed point in the lookup of attributes.
>     The proposal means that a fixed point is not reached until the cls parameter of type.__getattribute__() is either object or type,
>     otherwise type.__getattribute__() and type.__locallookup__ must bounce back and forth.
>
>     This will slow down *every* attribute lookup for what is a fairly obscure use case.
>
>     Cheers,
>     Mark.
>
>     On 09/09/13 16:27, Guido van Rossum wrote:
>
>         Let's just accept this PEP. It looks like a nice addition to the metaclass machinery and I don't think we'll get much more useful feedback by waiting.
>
>
>         On Mon, Sep 9, 2013 at 7:30 AM, Ethan Furman <ethan at stoneleaf.us <mailto:ethan at stoneleaf.us> <mailto:ethan at stoneleaf.us <mailto:ethan at stoneleaf.us>>> wrote:
>
>              On 07/30/2013 11:17 PM, Ronald Oussoren wrote:
>
>
>                  And something I forgot to ask: is anyone willing to be the BDFL-Delegate for
>                  PEP 447?
>
>
>              *Bump*.
>
>              It would be nice if this could make into 3.4.
>
>              --
>              ~Ethan~
>
>              ___________________________________________________
>              Python-Dev mailing list
>         Python-Dev at python.org <mailto:Python-Dev at python.org> <mailto:Python-Dev at python.org <mailto:Python-Dev at python.org>>
>         https://mail.python.org/____mailman/listinfo/python-dev <https://mail.python.org/__mailman/listinfo/python-dev> <https://mail.python.org/__mailman/listinfo/python-dev
>         <https://mail.python.org/mailman/listinfo/python-dev>>
>              Unsubscribe: https://mail.python.org/____mailman/options/python-dev/____guido%40python.org <https://mail.python.org/__mailman/options/python-dev/__guido%40python.org>
>         <https://mail.python.org/__mailman/options/python-dev/__guido%40python.org <https://mail.python.org/mailman/options/python-dev/guido%40python.org>>
>
>
>
>
>         --
>         --Guido van Rossum (python.org/~guido <http://python.org/~guido> <http://python.org/~guido>)
>
>
>         _________________________________________________
>         Python-Dev mailing list
>         Python-Dev at python.org <mailto:Python-Dev at python.org>
>         https://mail.python.org/__mailman/listinfo/python-dev <https://mail.python.org/mailman/listinfo/python-dev>
>         Unsubscribe: https://mail.python.org/__mailman/options/python-dev/__mark%40hotpy.org <https://mail.python.org/mailman/options/python-dev/mark%40hotpy.org>
>
>     _________________________________________________
>     Python-Dev mailing list
>     Python-Dev at python.org <mailto:Python-Dev at python.org>
>     https://mail.python.org/__mailman/listinfo/python-dev <https://mail.python.org/mailman/listinfo/python-dev>
>     Unsubscribe: https://mail.python.org/__mailman/options/python-dev/__guido%40python.org <https://mail.python.org/mailman/options/python-dev/guido%40python.org>
>

From ncoghlan at gmail.com  Mon Sep  9 18:12:27 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 10 Sep 2013 02:12:27 +1000
Subject: [Python-Dev] [Python-checkins] cpython: Post-3.4.0a2-release
	fixups.
In-Reply-To: <CAPZV6o8jzs_nUkgtJL0u6vxoGgX=3JihF5FXUovnLGK96m1o0Q@mail.gmail.com>
References: <3cYSwh0rHdz7Lkm@mail.python.org>
 <CADiSq7fVsbLCks=oK4UxgJNnzngTmEYLmOW_HEj_OXHMfJwUow@mail.gmail.com>
 <CAPZV6o8jzs_nUkgtJL0u6vxoGgX=3JihF5FXUovnLGK96m1o0Q@mail.gmail.com>
Message-ID: <CADiSq7f_fSk8kUZuRp9zX7yCfiQyHKT5GJnFve1TT_WY7Rar8A@mail.gmail.com>

On 10 Sep 2013 01:32, "Benjamin Peterson" <benjamin at python.org> wrote:
>
> Well, it's important for the release manager to make sure what the
> script is doing is sane. :)

Sure, preparing out of tree is fine and sensible. But we should either
freeze the tree or update the NEWS headers immediately, otherwise we're
going to have updates going into the wrong section.

Cheers,
Nick.

>
> 2013/9/9 Nick Coghlan <ncoghlan at gmail.com>:
> >
> > On 9 Sep 2013 22:15, "larry.hastings" <python-checkins at python.org>
wrote:
> >>
> >> http://hg.python.org/cpython/rev/6b211a0c8042
> >> changeset:   85645:6b211a0c8042
> >> user:        Larry Hastings <larry at hastings.org>
> >> date:        Mon Sep 09 21:08:52 2013 +0900
> >> summary:
> >>   Post-3.4.0a2-release fixups.
> >>
> >> files:
> >>   Include/patchlevel.h |   2 +-
> >>   Misc/NEWS            |  14 +++++++++++++-
> >>   2 files changed, 14 insertions(+), 2 deletions(-)
> >>
> >>
> >> diff --git a/Include/patchlevel.h b/Include/patchlevel.h
> >> --- a/Include/patchlevel.h
> >> +++ b/Include/patchlevel.h
> >> @@ -23,7 +23,7 @@
> >>  #define PY_RELEASE_SERIAL      2
> >>
> >>  /* Version as a string */
> >> -#define PY_VERSION             "3.4.0a2"
> >> +#define PY_VERSION             "3.4.0a2+"
> >>  /*--end constants--*/
> >>
> >>  /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2.
> >> diff --git a/Misc/NEWS b/Misc/NEWS
> >> --- a/Misc/NEWS
> >> +++ b/Misc/NEWS
> >> @@ -2,10 +2,22 @@
> >>  Python News
> >>  +++++++++++
> >>
> >> +What's New in Python 3.4.0 Alpha 3?
> >> +===================================
> >> +
> >> +Projected Release date: 2013-09-29
> >> +
> >> +Core and Builtins
> >> +-----------------
> >> +
> >> +Library
> >> +-------
> >> +
> >> +
> >
> > I had already pushed alpha 3 entries in NEWS, so something seems to have
> > gone wrong here.
> >
> > Perhaps, if RMs are preparing the release out of tree, we could get the
NEWS
> > file headings on default updated immediately after the last included
commit?
> >
> > Cheers,
> > Nick.
> >
> >>  What's New in Python 3.4.0 Alpha 2?
> >>  ===================================
> >>
> >> -Projected Release date: 2013-09-08
> >> +Release date: 2013-09-09
> >>
> >>  Core and Builtins
> >>  -----------------
> >>
> >> --
> >> Repository URL: http://hg.python.org/cpython
> >>
> >> _______________________________________________
> >> Python-checkins mailing list
> >> Python-checkins at python.org
> >> https://mail.python.org/mailman/listinfo/python-checkins
> >>
> >
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/benjamin%40python.org
> >
>
>
>
> --
> Regards,
> Benjamin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/8ae74a09/attachment.html>

From ethan at stoneleaf.us  Mon Sep  9 18:05:57 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 09 Sep 2013 09:05:57 -0700
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <522DECA3.2030709@hotpy.org>
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
 <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
 <522DECA3.2030709@hotpy.org>
Message-ID: <522DF1E5.5050309@stoneleaf.us>

On 09/09/2013 08:43 AM, Mark Shannon wrote:
> I would like time to investigate this further, but at the moment I think it will either make attribute lookup poorly
> defined or slow.
>
> Of the top of my head, the problem as a I see it is basically this:
> Currently, type.__getattribute__() is a fixed point in the lookup of attributes.
> The proposal means that a fixed point is not reached until the cls parameter of type.__getattribute__() is either object
> or type,
> otherwise type.__getattribute__() and type.__locallookup__ must bounce back and forth.
>
> This will slow down *every* attribute lookup for what is a fairly obscure use case.

Looks like there's a patch we can try at http://bugs.python.org/issue18181.

Here are Ronald's last timings:

	-------------------------------------------------------------------------------
	PYBENCH 2.1
	-------------------------------------------------------------------------------
	* using CPython 3.4.0a0 (default, Jul 29 2013, 13:01:34) [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)]
	* disabled garbage collection
	* system check interval set to maximum: 2147483647
	* using timer: time.perf_counter
	* timer: resolution=1e-09, implementation=clock_gettime(CLOCK_MONOTONIC)

	-------------------------------------------------------------------------------
	Benchmark: pep447.pybench
	-------------------------------------------------------------------------------

	    Rounds: 10
	    Warp:   10
	    Timer:  time.perf_counter

	    Machine Details:
	       Platform ID:    Linux-2.6.32-358.114.1.openstack.el6.x86_64-x86_64-with-centos-6.4-Final
	       Processor:      x86_64

	    Python:
	       Implementation: CPython
	       Executable:     /tmp/default-pep447/bin/python3
	       Version:        3.4.0a0
	       Compiler:       GCC 4.4.7 20120313 (Red Hat 4.4.7-3)
	       Bits:           64bit
	       Build:          Jul 29 2013 14:09:12 (#default)
	       Unicode:        UCS4


	-------------------------------------------------------------------------------
	Comparing with: default.pybench
	-------------------------------------------------------------------------------

	    Rounds: 10
	    Warp:   10
	    Timer:  time.perf_counter

	    Machine Details:
	       Platform ID:    Linux-2.6.32-358.114.1.openstack.el6.x86_64-x86_64-with-centos-6.4-Final
	       Processor:      x86_64

	    Python:
	       Implementation: CPython
	       Executable:     /tmp/default/bin/python3
	       Version:        3.4.0a0
	       Compiler:       GCC 4.4.7 20120313 (Red Hat 4.4.7-3)
	       Bits:           64bit
	       Build:          Jul 29 2013 13:01:34 (#default)
	       Unicode:        UCS4


	Test                             minimum run-time        average  run-time
					 this    other   diff    this    other   diff
	-------------------------------------------------------------------------------
		  BuiltinFunctionCalls:    45ms    44ms   +1.3%    45ms    44ms   +1.3%
		   BuiltinMethodLookup:    26ms    27ms   -2.4%    27ms    27ms   -2.2%
			 CompareFloats:    33ms    34ms   -0.7%    33ms    34ms   -1.1%
		 CompareFloatsIntegers:    66ms    67ms   -0.9%    66ms    67ms   -0.8%
		       CompareIntegers:    51ms    50ms   +0.9%    51ms    50ms   +0.8%
		CompareInternedStrings:    34ms    33ms   +0.4%    34ms    34ms   -0.4%
			  CompareLongs:    29ms    29ms   -0.1%    29ms    29ms   -0.0%
			CompareStrings:    43ms    44ms   -1.8%    44ms    44ms   -1.8%
	    ComplexPythonFunctionCalls:    44ms    42ms   +3.9%    44ms    42ms   +4.1%
			 ConcatStrings:    33ms    33ms   -0.4%    33ms    33ms   -1.0%
		       CreateInstances:    47ms    48ms   -2.9%    47ms    49ms   -3.4%
		    CreateNewInstances:    35ms    36ms   -2.5%    36ms    36ms   -2.5%
	       CreateStringsWithConcat:    69ms    70ms   -0.7%    69ms    70ms   -0.9%
			  DictCreation:    52ms    50ms   +3.1%    52ms    50ms   +3.0%
		     DictWithFloatKeys:    40ms    44ms  -10.1%    43ms    45ms   -5.8%
		   DictWithIntegerKeys:    32ms    36ms  -11.2%    35ms    37ms   -4.6%
		    DictWithStringKeys:    29ms    34ms  -15.7%    35ms    40ms  -11.0%
			      ForLoops:    30ms    29ms   +2.2%    30ms    29ms   +2.2%
			    IfThenElse:    38ms    41ms   -6.7%    38ms    41ms   -6.9%
			   ListSlicing:    36ms    36ms   -0.7%    36ms    37ms   -1.3%
			NestedForLoops:    43ms    45ms   -3.1%    43ms    45ms   -3.2%
	      NestedListComprehensions:    39ms    40ms   -1.7%    39ms    40ms   -2.1%
		  NormalClassAttribute:    86ms    82ms   +5.1%    86ms    82ms   +5.0%
	       NormalInstanceAttribute:    42ms    42ms   +0.3%    42ms    42ms   +0.0%
		   PythonFunctionCalls:    39ms    38ms   +3.5%    39ms    38ms   +2.8%
		     PythonMethodCalls:    51ms    49ms   +3.0%    51ms    50ms   +2.8%
			     Recursion:    67ms    68ms   -1.4%    67ms    68ms   -1.4%
			  SecondImport:    41ms    36ms  +12.5%    41ms    36ms  +12.6%
		   SecondPackageImport:    45ms    40ms  +13.1%    45ms    40ms  +13.2%
		 SecondSubmoduleImport:    92ms    95ms   -2.4%    95ms    98ms   -3.6%
	       SimpleComplexArithmetic:    28ms    28ms   -0.1%    28ms    28ms   -0.2%
		SimpleDictManipulation:    57ms    57ms   -1.0%    57ms    58ms   -1.0%
		 SimpleFloatArithmetic:    29ms    28ms   +4.7%    29ms    28ms   +4.9%
	      SimpleIntFloatArithmetic:    37ms    41ms   -8.5%    37ms    41ms   -8.7%
	       SimpleIntegerArithmetic:    37ms    41ms   -9.4%    37ms    42ms  -10.2%
	      SimpleListComprehensions:    33ms    33ms   -1.9%    33ms    34ms   -2.9%
		SimpleListManipulation:    28ms    30ms   -4.3%    29ms    30ms   -4.1%
		  SimpleLongArithmetic:    26ms    26ms   +0.5%    26ms    26ms   +0.5%
			    SmallLists:    40ms    40ms   +0.1%    40ms    40ms   +0.1%
			   SmallTuples:    46ms    47ms   -2.4%    46ms    48ms   -3.0%
		 SpecialClassAttribute:   126ms   120ms   +4.7%   126ms   121ms   +4.4%
	      SpecialInstanceAttribute:    42ms    42ms   +0.6%    42ms    42ms   +0.8%
			StringMappings:    94ms    91ms   +3.9%    94ms    91ms   +3.8%
		      StringPredicates:    48ms    49ms   -1.7%    48ms    49ms   -2.1%
			 StringSlicing:    45ms    45ms   +1.4%    46ms    45ms   +1.5%
			     TryExcept:    23ms    22ms   +4.9%    23ms    22ms   +4.8%
			    TryFinally:    32ms    32ms   -0.1%    32ms    32ms   +0.1%
			TryRaiseExcept:    17ms    17ms   +0.9%    17ms    17ms   +0.5%
			  TupleSlicing:    49ms    48ms   +1.1%    49ms    49ms   +1.0%
			   WithFinally:    48ms    47ms   +2.3%    48ms    47ms   +2.4%
		       WithRaiseExcept:    45ms    44ms   +0.8%    45ms    45ms   +0.5%
	-------------------------------------------------------------------------------
	Totals:                          2284ms  2287ms   -0.1%  2306ms  2308ms   -0.1%

	(this=pep447.pybench, other=default.pybench)

From a.badger at gmail.com  Mon Sep  9 19:39:11 2013
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Mon, 9 Sep 2013 10:39:11 -0700
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <87hadyah89.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru>
 <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
 <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com>
 <20130905203651.GB30331@iskra.aviel.ru>
 <20130905165318.2fa40bab@anarchist>
 <87hadyah89.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CABVPEKpCgeAH5uLThNCMNLLN6quViRWu2orC_386q09sPiMDyg@mail.gmail.com>

On Thu, Sep 5, 2013 at 6:09 PM, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> Barry Warsaw writes:

>  > We're open source, and I think it benefits our mission to support open,
>  > decentralized, and free systems like OpenID and Persona.
>
> Thus speaks an employee of yet another Provider-That-Won't-Accept-My-
> Third-Party-Credentials.  Sorry, Barry, but you see the problem:
> Unfortunately, we can't do it alone.  What needs to happen is there
> needs to be a large network of sites that support login via O-D-F
> systems like OpenID and Persona.  Too many of the sites I use (news
> sources, GMail, etc) don't support them and my browser manages my
> logins to most of them, so why bother learning OpenID, and then
> setting it up site by site?
>
[snipped lots of observations that I generally agree with]

There's been a lot of negativity towards OpenID in this thread -- I'd
like to say that in Fedora Infrastructure we've found OpenID to be
very very good -- but not at addressing the problem that most people
are after here.  As you've observed being an OpenID provider is a
relatively easy to swallow proposition; accepting OpenID from third
parties is another thing entirely.  As you've also observed, this has
to do with trust.  A site can trust their own account system and
practices and issue OpenID based on those.  It is much riskier for the
site to trust someone else's account system and practices when
deciding whether a user is actually the owner of the account that they
claim.

So OpenID fails as a truly generic SSO method across sites on the
internet... what have we found it good for then?  SSO within our site.
 More and more apps support OpenID out of the box.  Many web
frameworks have modules for the code you write to authenticate against
an OpenID server.  A site configures these apps and modules to only
trust the site's OpenID service and then deploys them with less custom
code.  Sites also get a choice about how much risk they consider
compromised accounts to a particular application.  If they run a web
forum and a build system for instance, they might constrain the build
system to only their OpenID service but allow the forum to allow
OpenID from other providers. And finally, having an openid service
lets their users sign into more trusting sites like python.org
properties (unlike say, LDAP) :-)

-Toshio

From phd at phdru.name  Mon Sep  9 19:46:58 2013
From: phd at phdru.name (Oleg Broytman)
Date: Mon, 9 Sep 2013 21:46:58 +0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <CABVPEKpCgeAH5uLThNCMNLLN6quViRWu2orC_386q09sPiMDyg@mail.gmail.com>
References: <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io>
 <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
 <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com>
 <20130905203651.GB30331@iskra.aviel.ru>
 <20130905165318.2fa40bab@anarchist>
 <87hadyah89.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CABVPEKpCgeAH5uLThNCMNLLN6quViRWu2orC_386q09sPiMDyg@mail.gmail.com>
Message-ID: <20130909174658.GA8795@iskra.aviel.ru>

On Mon, Sep 09, 2013 at 10:39:11AM -0700, Toshio Kuratomi <a.badger at gmail.com> wrote:
> So OpenID fails as a truly generic SSO method across sites on the
> internet... what have we found it good for then?  SSO within our site.

   I.e., OpenID could be good for core developers (using @python.org
email adresses as IDs) but not for general public to login to pydotorg
sites, right?

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From solipsis at pitrou.net  Mon Sep  9 19:49:59 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 9 Sep 2013 19:49:59 +0200
Subject: [Python-Dev] PEP 447: add type.__locallookup__
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
 <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
 <522DECA3.2030709@hotpy.org> <522DF1E5.5050309@stoneleaf.us>
Message-ID: <20130909194959.1e8115ef@fsol>

On Mon, 09 Sep 2013 09:05:57 -0700
Ethan Furman <ethan at stoneleaf.us> wrote:
> On 09/09/2013 08:43 AM, Mark Shannon wrote:
> > I would like time to investigate this further, but at the moment I think it will either make attribute lookup poorly
> > defined or slow.
> >
> > Of the top of my head, the problem as a I see it is basically this:
> > Currently, type.__getattribute__() is a fixed point in the lookup of attributes.
> > The proposal means that a fixed point is not reached until the cls parameter of type.__getattribute__() is either object
> > or type,
> > otherwise type.__getattribute__() and type.__locallookup__ must bounce back and forth.
> >
> > This will slow down *every* attribute lookup for what is a fairly obscure use case.
> 
> Looks like there's a patch we can try at http://bugs.python.org/issue18181.
> 
> Here are Ronald's last timings:

Thanks but can you run a benchmark that actually exercises the feature?
(I don't know enough about it to know what that would be, but I suppose
it has to do with lookups on classes, rather than on instances)

Regards

Antoine.



From phd at phdru.name  Mon Sep  9 20:08:00 2013
From: phd at phdru.name (Oleg Broytman)
Date: Mon, 9 Sep 2013 22:08:00 +0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <20130909174658.GA8795@iskra.aviel.ru>
References: <20130905182522.GA26090@iskra.aviel.ru>
 <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io>
 <20130905184337.GA26926@iskra.aviel.ru>
 <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io>
 <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com>
 <20130905203651.GB30331@iskra.aviel.ru>
 <20130905165318.2fa40bab@anarchist>
 <87hadyah89.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CABVPEKpCgeAH5uLThNCMNLLN6quViRWu2orC_386q09sPiMDyg@mail.gmail.com>
 <20130909174658.GA8795@iskra.aviel.ru>
Message-ID: <20130909180800.GB8795@iskra.aviel.ru>

On Mon, Sep 09, 2013 at 09:46:58PM +0400, Oleg Broytman <phd at phdru.name> wrote:
> On Mon, Sep 09, 2013 at 10:39:11AM -0700, Toshio Kuratomi <a.badger at gmail.com> wrote:
> > So OpenID fails as a truly generic SSO method across sites on the
> > internet... what have we found it good for then?  SSO within our site.
> 
>    I.e., OpenID could be good for core developers (using @python.org
> email adresses as IDs) but not for general public to login to pydotorg
> sites, right?

   Oops, completely messed OpenID URLs and Persona emails.

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From zuo at chopin.edu.pl  Mon Sep  9 20:23:40 2013
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Mon, 09 Sep 2013 20:23:40 +0200
Subject: [Python-Dev]
 =?utf-8?q?PEP_447=3A_add_type=2E=5F=5Flocallookup=5F?=
 =?utf-8?q?=5F?=
In-Reply-To: <522DF204.7040109@hotpy.org>
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
 <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
 <522DECA3.2030709@hotpy.org>
 <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com>
 <522DF204.7040109@hotpy.org>
Message-ID: <0752376899dd7436f6115bfc03594574@chopin.edu.pl>

Is '__locallookup__' a really good name? In Python, *local* -- 
especially in context of *lookups* -- usually associates with locals() 
i.e. a namespace of a function/method execution frame or a namespace of 
a class, during *definition* of that class... So '__locallookup__' can 
be confusing.

Why not just '__getclassattribute__' or '__classlookup__', or 
'__classattribute__'...?

Cheers.
*j


From nad at acm.org  Mon Sep  9 21:29:27 2013
From: nad at acm.org (Ned Deily)
Date: Mon, 09 Sep 2013 12:29:27 -0700
Subject: [Python-Dev] [RELEASED] Python 3.4.0a2
References: <522DB8D3.1030803@hastings.org>
Message-ID: <nad-2495B7.12292709092013@news.gmane.org>

In article <522DB8D3.1030803 at hastings.org>,
 Larry Hastings <larry at hastings.org> wrote:

> On behalf of the Python development team, I'm chuffed to announce the
> second alpha release of Python 3.4.

Yay!  3.4.0a2 also contains a new batteries-included feature for OS X 
users.  The python.org 64-bit/32-bit installer now includes its own 
private version of Tcl/Tk 8.5.14 so it is finally no longer necessary to 
install a third-party version of Tcl/Tk 8.5 to workaround the 
problematic system versions shipped in OS X 10.6+.  More improvements to 
come.

-- 
 Ned Deily,
 nad at acm.org


From victor.stinner at gmail.com  Mon Sep  9 21:51:46 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 9 Sep 2013 21:51:46 +0200
Subject: [Python-Dev] [python-committers] [RELEASED] Python 3.4.0a2
In-Reply-To: <20130909144551.76b84640@pitrou.net>
References: <522DB8D3.1030803@hastings.org>
 <CAMpsgwZ3Ltsc4dKRnN5zBVvcSF4WFX4YQRcG9oaq_iMvx0NjvQ@mail.gmail.com>
 <20130909144551.76b84640@pitrou.net>
Message-ID: <CAMpsgwZFuAKe+VKCXaXf6aU1wz_3+p3s=0-QMLEwgBEBLgVf0g@mail.gmail.com>

2013/9/9 Antoine Pitrou <solipsis at pitrou.net>:
> Le Mon, 9 Sep 2013 14:30:50 +0200,
> Victor Stinner <victor.stinner at gmail.com> a ?crit :
>> 2013/9/9 Larry Hastings <larry at hastings.org>:
>> > Python 3.4 includes a range of improvements of the 3.x series,
>> > including hundreds of small improvements and bug fixes.  Major new
>> > features and changes in the 3.4 release series so far include:
>> >
>> > * PEP 446, changing file descriptors to not be inherited by default
>> >            in subprocesses
>>
>> The title of the PEP is "Make newly created file descriptors
>> non-inheritable". It has an impact on all functions creating files and
>> sockets not only the subprocess module.
>
> I don't think Larry's description is wrong. "Non-inheritable" is a
> shorthand for "non-inheritable in subprocesses" with "subprocesses"
> taken in the general sense (i.e. not only created with the subprocess
> module).

Oh, I misunderstood "in subprocesses", I read "in the subprocess module".

The definition of FD inheritance is tricky. For example, on UNIX
"non-inheritable" file descriptors are still inherited at fork :-)

I hope that the documentation is explicit enough:
http://docs.python.org/dev/library/os.html#inheritance-of-file-descriptors

Victor

From yselivanov.ml at gmail.com  Mon Sep  9 22:01:47 2013
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Mon, 9 Sep 2013 16:01:47 -0400
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <0752376899dd7436f6115bfc03594574@chopin.edu.pl>
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
 <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
 <522DECA3.2030709@hotpy.org>
 <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com>
 <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl>
Message-ID: <FA8F15D0-95C8-43D1-ACB6-0499CBE10F95@gmail.com>

Yes, I don't like the 'local' prefix too.  How about '__dictlookup__'?
It's just more self-describing.

Yury

On 2013-09-09, at 2:23 PM, Jan Kaliszewski <zuo at chopin.edu.pl> wrote:

> Is '__locallookup__' a really good name? In Python, *local* -- especially in context of *lookups* -- usually associates with locals() i.e. a namespace of a function/method execution frame or a namespace of a class, during *definition* of that class... So '__locallookup__' can be confusing.
> 
> Why not just '__getclassattribute__' or '__classlookup__', or '__classattribute__'...?
> 
> Cheers.
> *j
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com


From benjamin at python.org  Mon Sep  9 22:16:33 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Mon, 9 Sep 2013 16:16:33 -0400
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <522DDB74.5080609@stoneleaf.us>
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
 <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us>
Message-ID: <CAPZV6o8oKepeSgHq13CUHXY8q36tmnAJ5dqEeR2QFO0Q6M81Dg@mail.gmail.com>

Since the main problem is super(), maybe we can just add a __super__
method to get a custom super implementation?

2013/9/9 Ethan Furman <ethan at stoneleaf.us>:
> On 07/30/2013 11:17 PM, Ronald Oussoren wrote:
>>
>>
>> And something I forgot to ask: is anyone willing to be the BDFL-Delegate
>> for
>> PEP 447?
>
>
> *Bump*.
>
> It would be nice if this could make into 3.4.
>
> --
> ~Ethan~
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org



-- 
Regards,
Benjamin

From mark at hotpy.org  Mon Sep  9 23:18:38 2013
From: mark at hotpy.org (Mark Shannon)
Date: Mon, 09 Sep 2013 22:18:38 +0100
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <522DDB74.5080609@stoneleaf.us>
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
 <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us>
Message-ID: <522E3B2E.6050605@hotpy.org>

On 09/09/13 15:30, Ethan Furman wrote:
> On 07/30/2013 11:17 PM, Ronald Oussoren wrote:
>>
>> And something I forgot to ask: is anyone willing to be the
>> BDFL-Delegate for
>> PEP 447?
>
> *Bump*.
>
> It would be nice if this could make into 3.4.
>

IMO, there are some issues that need to be addressed before PEP 447 
should be accepted.

1. Is there even a problem at all, or is this just a bug in super?
Why doesn't super() respect the __getattribute__ method of the superclass?

2. Is this the best way to solve the problem (if there is a problem)?
Would a __super__ special method be sufficient and less intrusive.

3. Are the proposed semantics OK?
I think they are, but very low level changes such as these can have 
unforeseen consequences. For example, PEP 3135 and issue 12370.

4. What is the performance impact. pybench really doesn't count as a 
benchmark.

5. Other implementations. What do the Jython/IronPython/PyPy developers 
think?


Cheers,
Mark.

p.s.
Apologies for top-posting earlier

From benjamin at python.org  Mon Sep  9 23:25:14 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Mon, 9 Sep 2013 17:25:14 -0400
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <522E3B2E.6050605@hotpy.org>
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
 <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us> <522E3B2E.6050605@hotpy.org>
Message-ID: <CAPZV6o903HwDnbyOaWm5nTV8xYBHxDvnG0NGkeVOhwrSTBJnig@mail.gmail.com>

2013/9/9 Mark Shannon <mark at hotpy.org>:
> On 09/09/13 15:30, Ethan Furman wrote:
>>
>> On 07/30/2013 11:17 PM, Ronald Oussoren wrote:
>>>
>>>
>>> And something I forgot to ask: is anyone willing to be the
>>> BDFL-Delegate for
>>> PEP 447?
>>
>>
>> *Bump*.
>>
>> It would be nice if this could make into 3.4.
>>
>
> IMO, there are some issues that need to be addressed before PEP 447 should
> be accepted.
>
> 1. Is there even a problem at all, or is this just a bug in super?
> Why doesn't super() respect the __getattribute__ method of the superclass?

You want to be looking things up on the class, not an instance.

-- 
Regards,
Benjamin

From mark at hotpy.org  Tue Sep 10 00:36:15 2013
From: mark at hotpy.org (Mark Shannon)
Date: Mon, 09 Sep 2013 23:36:15 +0100
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <CAPZV6o903HwDnbyOaWm5nTV8xYBHxDvnG0NGkeVOhwrSTBJnig@mail.gmail.com>
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
 <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us> <522E3B2E.6050605@hotpy.org>
 <CAPZV6o903HwDnbyOaWm5nTV8xYBHxDvnG0NGkeVOhwrSTBJnig@mail.gmail.com>
Message-ID: <522E4D5F.4050505@hotpy.org>

On 09/09/13 22:25, Benjamin Peterson wrote:
> 2013/9/9 Mark Shannon <mark at hotpy.org>:
>> On 09/09/13 15:30, Ethan Furman wrote:
>>>
>>> On 07/30/2013 11:17 PM, Ronald Oussoren wrote:
>>>>
>>>>
>>>> And something I forgot to ask: is anyone willing to be the
>>>> BDFL-Delegate for
>>>> PEP 447?
>>>
>>>
>>> *Bump*.
>>>
>>> It would be nice if this could make into 3.4.
>>>
>>
>> IMO, there are some issues that need to be addressed before PEP 447 should
>> be accepted.
>>
>> 1. Is there even a problem at all, or is this just a bug in super?
>> Why doesn't super() respect the __getattribute__ method of the superclass?
>
> You want to be looking things up on the class, not an instance.
>
Sorry, I meant 'type of the superclass' rather than 'superclass'.

I was suggesting that super().m should be 
type(type(self).__mro__[1]).__getattribute__('m')
rather than type(self).__mro__[1].__dict__['m']
(ignoring descriptor __get__ calls)

Unfortunately this brings its own problems, due to __getattribute__ 
doing its own traversal of the mro.
So, scratch point 1.

Cheers,
Mark.




From rymg19 at gmail.com  Tue Sep 10 02:28:15 2013
From: rymg19 at gmail.com (Ryan)
Date: Mon, 09 Sep 2013 19:28:15 -0500
Subject: [Python-Dev] [RELEASED] Python 3.4.0a2
In-Reply-To: <nad-2495B7.12292709092013@news.gmane.org>
References: <522DB8D3.1030803@hastings.org>
 <nad-2495B7.12292709092013@news.gmane.org>
Message-ID: <610dd75a-ada8-4ae1-9810-7320cbeda9ce@email.android.com>

HALLELUJAH!!!

Ned Deily <nad at acm.org> wrote:

>In article <522DB8D3.1030803 at hastings.org>,
> Larry Hastings <larry at hastings.org> wrote:
>
>> On behalf of the Python development team, I'm chuffed to announce the
>> second alpha release of Python 3.4.
>
>Yay!  3.4.0a2 also contains a new batteries-included feature for OS X 
>users.  The python.org 64-bit/32-bit installer now includes its own 
>private version of Tcl/Tk 8.5.14 so it is finally no longer necessary
>to 
>install a third-party version of Tcl/Tk 8.5 to workaround the 
>problematic system versions shipped in OS X 10.6+.  More improvements
>to 
>come.
>
>-- 
> Ned Deily,
> nad at acm.org
>
>_______________________________________________
>Python-Dev mailing list
>Python-Dev at python.org
>https://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe:
>https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com

-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130909/44236030/attachment.html>

From stefan_ml at behnel.de  Tue Sep 10 07:48:51 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 10 Sep 2013 07:48:51 +0200
Subject: [Python-Dev] [RELEASED] Python 3.4.0a2
In-Reply-To: <nad-2495B7.12292709092013@news.gmane.org>
References: <522DB8D3.1030803@hastings.org>
 <nad-2495B7.12292709092013@news.gmane.org>
Message-ID: <l0mbrr$ofo$1@ger.gmane.org>

Ned Deily, 09.09.2013 21:29:
> 3.4.0a2 also contains a new batteries-included feature for OS X 
> users.  The python.org 64-bit/32-bit installer now includes its own 
> private version of Tcl/Tk 8.5.14 so it is finally no longer necessary to 
> install a third-party version of Tcl/Tk 8.5 to workaround the 
> problematic system versions shipped in OS X 10.6+.  More improvements to 
> come.

With MS Windows on steep decline, I'm so happy we have at least MacOS-X
rising to keep us busy.

Stefan



From solipsis at pitrou.net  Tue Sep 10 08:08:46 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 10 Sep 2013 08:08:46 +0200
Subject: [Python-Dev] cpython (2.7): Clarify mmap.close method behavior.
 Addresses issue #18815
References: <3cYw7T6lKwz7LjZ@mail.python.org>
Message-ID: <20130910080846.578093a7@fsol>

On Tue, 10 Sep 2013 07:40:21 +0200 (CEST)
senthil.kumaran <python-checkins at python.org> wrote:
> http://hg.python.org/cpython/rev/443d12b61e5b
> changeset:   85653:443d12b61e5b
> branch:      2.7
> parent:      85639:740bd510a888
> user:        Senthil Kumaran <senthil at uthcode.com>
> date:        Mon Sep 09 22:38:58 2013 -0700
> summary:
>   Clarify mmap.close method  behavior.  Addresses issue  #18815
> Patch contributed by Anoop Thomas Mathew.
> 
> files:
>   Doc/library/mmap.rst |  5 +++--
>   1 files changed, 3 insertions(+), 2 deletions(-)
> 
> 
> diff --git a/Doc/library/mmap.rst b/Doc/library/mmap.rst
> --- a/Doc/library/mmap.rst
> +++ b/Doc/library/mmap.rst
> @@ -152,8 +152,9 @@
>  
>     .. method:: close()
>  
> -      Close the file.  Subsequent calls to other methods of the object will
> -      result in an exception being raised.
> +      Closes the mmap. Subsequent calls to other methods of the object will
> +      result in a ValueError exception being raised. This will not close
> +      the open file.

I think this could be improved.  Anonymous mmap objects don't have an
"open file". Also, it would be clearer if it read "the underlying file"
rather than "the underlying file object" rather than "the open file",
because an mmap object is a file as well (it has read(), write(),
seek(), etc.).

Regards

Antoine.



From v+python at g.nevcal.com  Tue Sep 10 09:21:28 2013
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Tue, 10 Sep 2013 00:21:28 -0700
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <l0d311$b7n$1@ger.gmane.org>
References: <5228C00F.7000209@jcea.es> <l0d311$b7n$1@ger.gmane.org>
Message-ID: <522EC878.6080409@g.nevcal.com>

On 9/6/2013 10:22 AM, Dan Callahan wrote:
> On 9/5/13 12:31 PM, Jesus Cea wrote:
>> I have big hopes for Mozilla Persona, looking forward
>> Python infrastructure support :).
>
> Hi, I'm the project lead on Persona signin, and I spoke at PyCon 
> earlier this year regarding why and how Mozilla is building Persona. 
> If you'd like some more background, that video [0] is worth a look.
>
> Let's pull this discussion up a level:
>
> It sounds like many people (Jesus, Donald, Toshio, Barry, Tres, 
> Dirkjan, etc.) are interested in seeing Persona on Python.org 
> properties, and most of the objections coming from a place of "Persona 
> hasn't gone viral, what if this is wasted effort?"

OK, let's pull this discussion down a level: testing it out.

So I tried to login to the crossword.thetimes.co.uk -- I used an email 
address persona had never seen, it asked me for a password, and sent me 
a confirmation message, containing a link that I clicked on.

However, as I was reading clues and filling in blanks, I got a popup 
that said "login failure [object Object]". And crossword told me it was 
saving locally, and to login to save to the server. And the Log in 
button stayed displayed, rather than a Log out button, which I assume it 
might get replaced with if I ever get successfully logged in.

Firefox 23.0.1, Windows 7 64-bit with autoupdates. Need any other info? 
Write me privately if you want the email address I used (not the one I 
use here), or the password. I used new ones, so I can share for testing, 
and then discard them and use different ones "for real". If the system 
actually works. Hey, the video demos looked great...

Glenn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/04d2b36a/attachment.html>

From solipsis at pitrou.net  Tue Sep 10 10:10:42 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 10 Sep 2013 10:10:42 +0200
Subject: [Python-Dev] Offtopic: OpenID Providers
References: <5228C00F.7000209@jcea.es> <l0d311$b7n$1@ger.gmane.org>
 <522EC878.6080409@g.nevcal.com>
Message-ID: <20130910101042.7c2b66c7@pitrou.net>


Ok, can this discussion go off python-dev, please? This has been
terribly off-topic for a long time (arguably from the beginning,
actually).

Thank you

Antoine.



Le Tue, 10 Sep 2013 00:21:28 -0700,
Glenn Linderman <v+python at g.nevcal.com> a ?crit :
> On 9/6/2013 10:22 AM, Dan Callahan wrote:
> > On 9/5/13 12:31 PM, Jesus Cea wrote:
> >> I have big hopes for Mozilla Persona, looking forward
> >> Python infrastructure support :).
> >
> > Hi, I'm the project lead on Persona signin, and I spoke at PyCon 
> > earlier this year regarding why and how Mozilla is building
> > Persona. If you'd like some more background, that video [0] is
> > worth a look.
> >
> > Let's pull this discussion up a level:
> >
> > It sounds like many people (Jesus, Donald, Toshio, Barry, Tres, 
> > Dirkjan, etc.) are interested in seeing Persona on Python.org 
> > properties, and most of the objections coming from a place of
> > "Persona hasn't gone viral, what if this is wasted effort?"
> 
> OK, let's pull this discussion down a level: testing it out.
> 
> So I tried to login to the crossword.thetimes.co.uk -- I used an
> email address persona had never seen, it asked me for a password, and
> sent me a confirmation message, containing a link that I clicked on.
> 
> However, as I was reading clues and filling in blanks, I got a popup 
> that said "login failure [object Object]". And crossword told me it
> was saving locally, and to login to save to the server. And the Log
> in button stayed displayed, rather than a Log out button, which I
> assume it might get replaced with if I ever get successfully logged
> in.
> 
> Firefox 23.0.1, Windows 7 64-bit with autoupdates. Need any other
> info? Write me privately if you want the email address I used (not
> the one I use here), or the password. I used new ones, so I can share
> for testing, and then discard them and use different ones "for real".
> If the system actually works. Hey, the video demos looked great...
> 
> Glenn
> 



From solipsis at pitrou.net  Tue Sep 10 11:28:32 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 10 Sep 2013 11:28:32 +0200
Subject: [Python-Dev] Add a "transformdict" to collections
Message-ID: <20130910112832.75ed6c8b@pitrou.net>


Hello,

In http://bugs.python.org/issue18986 I proposed adding a new mapping
type to the collections module.

The original use case is quite common in network programming and
elsewhere (Eric Snow on the tracker mentioned an application with stock
symbols). You want to have an associative container which matches keys
case-insensitively but also preserves the original casing (e.g. for
presentation). It is a commonly reimplemented container.

It is also an instance of a more general pattern: match keys after
applying a derivation (or coercion) function, but at the same time keep
track of the original key. Note that the derivation function needn't be
(and generally won't be) bijective, otherwise it's too simple.

Therefore I propose adding the general pattern. Simple example:

   >>> d = transformdict(str.lower)
   >>> d['Foo'] = 5
   >>> d['foo']
   5
   >>> d['FOO']
   5
   >>> list(d)
   ['Foo']

(case-insensitive but case-preserving, as the best filesystems are ;-))


On the tracker issue, it seems everyone agreed on the principle. There
is some bikeshedding left to do, though. So here are the reasonable
naming proposals so far:

- transformkeydict
- coercekeydict
- transformdict
- coercedict

I have a sweet spot for "transformdict" myself.

Regards

Antoine.



From skip at pobox.com  Tue Sep 10 11:42:46 2013
From: skip at pobox.com (Skip Montanaro)
Date: Tue, 10 Sep 2013 04:42:46 -0500
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130910112832.75ed6c8b@pitrou.net>
References: <20130910112832.75ed6c8b@pitrou.net>
Message-ID: <CANc-5Uzgd2nwbxN39ykzUv-1YZzg1BEuvVZObQrnrdWd8ofTbA@mail.gmail.com>

> (case-insensitive but case-preserving, as the best filesystems are ;-))

> I have a sweet spot for "transformdict" myself.

Antoine,

"Transform" does not remind me of "case-insensitive but
case-preserving". If this is important enough to put into the
collections module (I'm skeptical), shouldn't that behavior be more
strongly hinted at in the name? Lot's of things are transformations.
You're interested in a very specific one.

Skip

From solipsis at pitrou.net  Tue Sep 10 12:00:12 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 10 Sep 2013 12:00:12 +0200
Subject: [Python-Dev] Add a "transformdict" to collections
References: <20130910112832.75ed6c8b@pitrou.net>
 <CANc-5Uzgd2nwbxN39ykzUv-1YZzg1BEuvVZObQrnrdWd8ofTbA@mail.gmail.com>
Message-ID: <20130910120012.6628c98b@pitrou.net>

Le Tue, 10 Sep 2013 04:42:46 -0500,
Skip Montanaro <skip at pobox.com> a ?crit :
> > (case-insensitive but case-preserving, as the best filesystems
> > are ;-))
> 
> > I have a sweet spot for "transformdict" myself.
> 
> Antoine,
> 
> "Transform" does not remind me of "case-insensitive but
> case-preserving".

That was only describing the example, not the proposal.
(and I hoped it would be obvious that that sentence was a joke ;-))

Regards

Antoine.



From victor.stinner at gmail.com  Tue Sep 10 12:04:51 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 10 Sep 2013 12:04:51 +0200
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130910112832.75ed6c8b@pitrou.net>
References: <20130910112832.75ed6c8b@pitrou.net>
Message-ID: <CAMpsgwbbCOx9tGBkz-Pj5hwJaNP+Dp2UPfov75PksD72+5FVfg@mail.gmail.com>

2013/9/10 Antoine Pitrou <solipsis at pitrou.net>:
> In http://bugs.python.org/issue18986 I proposed adding a new mapping
> type to the collections module.
>
> The original use case is quite common in network programming and
> elsewhere (Eric Snow on the tracker mentioned an application with stock
> symbols). You want to have an associative container which matches keys
> case-insensitively but also preserves the original casing (e.g. for
> presentation). It is a commonly reimplemented container.

If it is commonly reimplemented, what is the most common name? :-)

The http.client and email.message modules convert headers to lower
case, but keep the original case.

> - transformkeydict

Do you know a use case where values need also to be transformed? If
not, I prefer the transformdict name.

> - coercekeydict
> - coercedict

I only read "coerce" in old Python documentation, not in other
languages. I prefer the more common and generic term "tranform".

Victor

From oscar.j.benjamin at gmail.com  Tue Sep 10 12:34:54 2013
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Tue, 10 Sep 2013 11:34:54 +0100
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130910112832.75ed6c8b@pitrou.net>
References: <20130910112832.75ed6c8b@pitrou.net>
Message-ID: <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>

On 10 September 2013 10:28, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On the tracker issue, it seems everyone agreed on the principle. There
> is some bikeshedding left to do, though. So here are the reasonable
> naming proposals so far:
>
> - transformkeydict
> - coercekeydict
> - transformdict
> - coercedict
>
> I have a sweet spot for "transformdict" myself.

Could you not use a name with some sense of purpose like approxmatchdict?


Oscar

From dirkjan at ochtman.nl  Tue Sep 10 12:46:33 2013
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Tue, 10 Sep 2013 12:46:33 +0200
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130910112832.75ed6c8b@pitrou.net>
References: <20130910112832.75ed6c8b@pitrou.net>
Message-ID: <CAKmKYaAYRNNkqOLdi7m72h6_ja+eBeca-fnNa85LLj2ZzXGbfA@mail.gmail.com>

On Tue, Sep 10, 2013 at 11:28 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On the tracker issue, it seems everyone agreed on the principle. There
> is some bikeshedding left to do, though. So here are the reasonable
> naming proposals so far:
>
> - transformkeydict
> - coercekeydict
> - transformdict
> - coercedict
>
> I have a sweet spot for "transformdict" myself.

Given OrderedDict, it seems like this should definitely be
TransformDict rather than transformdict.

Cheers,

Dirkjan

From martin at v.loewis.de  Tue Sep 10 13:56:56 2013
From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 10 Sep 2013 13:56:56 +0200
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <5228C00F.7000209@jcea.es>
References: <5228C00F.7000209@jcea.es>
Message-ID: <522F0908.9030303@v.loewis.de>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Am 05.09.13 19:31, schrieb Jesus Cea:
> What are you using?. bugs.python.org admins could share some data?

Most users use one of the large services:

https://www.google.com 3326
https://login.launchpad.net 335
https://*.myopenid.com 253
https://launchpad.net 23
https://*.id.fedoraproject.org 11

The remaining ones are mostly private URLs. Of those, a majority
again delegates to providers, namely (ignoring providers with less
than 3 users)

www.startssl.com 3
www.clavid.com 3
openid.stackexchange.com 4
openid.claimid.com 4
login.launchpad.net 7
openid.yandex.ru 8
www.google.com 14
pip.verisignlabs.com 20
www.myopenid.com 41

Regards,
Martin
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.18 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlIvCQgACgkQavBT8H2dyNIX/gCeL9As+Z/aRAggbP+bjwUxFtGo
wZYAn2YkEEf/iv+bPGDBEnV+UWZmrf9M
=B5pE
-----END PGP SIGNATURE-----

From ncoghlan at gmail.com  Tue Sep 10 14:00:37 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 10 Sep 2013 22:00:37 +1000
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
Message-ID: <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>

Is this just syntactic sugar for recursive lookup of a transformed version
in __missing__? Or a way of supplying a custom "key" function to a
dictionary?

Any such proposal should consider how it composes with other dict variants
like defaultdict, OrderedDict and counter.

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/eb002faa/attachment.html>

From solipsis at pitrou.net  Tue Sep 10 14:22:58 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 10 Sep 2013 14:22:58 +0200
Subject: [Python-Dev] Add a "transformdict" to collections
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
 <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>
Message-ID: <20130910142258.02c148dd@pitrou.net>

Le Tue, 10 Sep 2013 22:00:37 +1000,
Nick Coghlan <ncoghlan at gmail.com> a ?crit :
> Is this just syntactic sugar for recursive lookup of a transformed
> version in __missing__?

Nope. For one, it doesn't use __missing__ at all. I think
using __missing__ would be... missing the point, because it wouldn't
working properly if you have e.g. X != Y such that transform(X) == Y
and transform(Y) == X.

(a simple example being transform = str.swapcase :-))

> Or a way of supplying a custom "key" function
> to a dictionary?

Probably, although I'm not entirely sure what you mean by that :-)

> Any such proposal should consider how it composes with other dict
> variants like defaultdict, OrderedDict and counter.

Well, one sticking point here is that those variants don't compose with
each other already :-)
But, yes, I may make another proposal with composition in mind.

Regards

Antoine.



From p.f.moore at gmail.com  Tue Sep 10 14:24:29 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 10 Sep 2013 13:24:29 +0100
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
 <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>
Message-ID: <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com>

On 10 September 2013 13:00, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Is this just syntactic sugar for recursive lookup of a transformed version
> in __missing__? Or a way of supplying a custom "key" function to a
> dictionary?

Not quite, because the dict should preserve the originally entered key.

>>> td['FOO'] = 42
>>> td['bar'] = 1
>>> td['foo']
42
>>> td['BAR']
1
>>> list(td.keys())
['FOO', 'bar']

This actually prompts the question, what should the following produce:

>>> td['FOO'] = 42
>>> td['foo'] = 32
>>> list(td.keys())

['FOO'] or ['foo']? Both answers are justifiable. Both are possibly
even useful depending on context...

Paul

From p.f.moore at gmail.com  Tue Sep 10 14:25:51 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 10 Sep 2013 13:25:51 +0100
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com>
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
 <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>
 <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com>
Message-ID: <CACac1F9-OhKM-m+Vh1ZMz2a=NXRXNYMtA-FQCmOihxBdL_T34Q@mail.gmail.com>

On 10 September 2013 13:24, Paul Moore <p.f.moore at gmail.com> wrote:
> On 10 September 2013 13:00, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> Is this just syntactic sugar for recursive lookup of a transformed version
>> in __missing__? Or a way of supplying a custom "key" function to a
>> dictionary?
>
> Not quite, because the dict should preserve the originally entered key.
>
>>>> td['FOO'] = 42
>>>> td['bar'] = 1
>>>> td['foo']
> 42
>>>> td['BAR']
> 1
>>>> list(td.keys())
> ['FOO', 'bar']
>
> This actually prompts the question, what should the following produce:
>
>>>> td['FOO'] = 42
>>>> td['foo'] = 32
>>>> list(td.keys())
>
> ['FOO'] or ['foo']? Both answers are justifiable. Both are possibly
> even useful depending on context...

I'm using the case where the transform is str.lower here - sorry I
wasn't explicit.
Paul

From solipsis at pitrou.net  Tue Sep 10 14:35:58 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 10 Sep 2013 14:35:58 +0200
Subject: [Python-Dev] Add a "transformdict" to collections
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
 <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>
 <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com>
Message-ID: <20130910143558.5677b55a@pitrou.net>

Le Tue, 10 Sep 2013 13:24:29 +0100,
Paul Moore <p.f.moore at gmail.com> a ?crit :
> On 10 September 2013 13:00, Nick Coghlan <ncoghlan at gmail.com> wrote:
> > Is this just syntactic sugar for recursive lookup of a transformed
> > version in __missing__? Or a way of supplying a custom "key"
> > function to a dictionary?
> 
> Not quite, because the dict should preserve the originally entered
> key.
> 
> >>> td['FOO'] = 42
> >>> td['bar'] = 1
> >>> td['foo']
> 42
> >>> td['BAR']
> 1
> >>> list(td.keys())
> ['FOO', 'bar']
> 
> This actually prompts the question, what should the following produce:
> 
> >>> td['FOO'] = 42
> >>> td['foo'] = 32
> >>> list(td.keys())
> 
> ['FOO'] or ['foo']? Both answers are justifiable. Both are possibly
> even useful depending on context...

I think it would be best to leave it as an implementation detail,
because whichever is easiest to implement depends on the exact
implementation choices (e.g. C vs. Python).

(my prototypes right now conserve the last __setitem__ key, not the
first)

Regards

Antoine.



From martin at v.loewis.de  Tue Sep 10 14:44:01 2013
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Tue, 10 Sep 2013 14:44:01 +0200
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130910143558.5677b55a@pitrou.net>
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
 <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>
 <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com>
 <20130910143558.5677b55a@pitrou.net>
Message-ID: <522F1411.8010109@v.loewis.de>

Am 10.09.13 14:35, schrieb Antoine Pitrou:
>> ['FOO'] or ['foo']? Both answers are justifiable. Both are possibly
>> even useful depending on context...
> 
> I think it would be best to leave it as an implementation detail,
> because whichever is easiest to implement depends on the exact
> implementation choices (e.g. C vs. Python).

I think this is the point where the datatype is *not* clearly
straight-forward, and thus deserves a PEP.

I think it would be a flaw to have this detail implementation-defined.
This would be like saying that it is implementation-defined which
of A,B,C is returned from "A and B and C" if all are true.

Regards,
Martin


From hrvoje.niksic at avl.com  Tue Sep 10 15:09:56 2013
From: hrvoje.niksic at avl.com (Hrvoje Niksic)
Date: Tue, 10 Sep 2013 15:09:56 +0200
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com>
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
 <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>
 <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com>
Message-ID: <522F1A24.5030808@avl.com>

On 09/10/2013 02:24 PM, Paul Moore wrote:
>>>> td['FOO'] = 42
>>>> td['foo'] = 32
>>>> list(td.keys())
>
> ['FOO'] or ['foo']? Both answers are justifiable.

Note that the same question can be reasonably asked for dict itself:

 >>> d = {}
 >>> d[1.0] = 'foo'
 >>> d[1] = 'bar'
 >>> d
{1.0: 'bar'}

So, dict.__setitem__ only replaces the value, leaving the original key 
in place. transformdict should probably do the same, returning 'FOO' in 
your example.


From shibturn at gmail.com  Tue Sep 10 15:42:39 2013
From: shibturn at gmail.com (Richard Oudkerk)
Date: Tue, 10 Sep 2013 14:42:39 +0100
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130910112832.75ed6c8b@pitrou.net>
References: <20130910112832.75ed6c8b@pitrou.net>
Message-ID: <522F21CF.4060609@gmail.com>

On 10/09/2013 10:28am, Antoine Pitrou wrote:
> Therefore I propose adding the general pattern. Simple example:
>
>     >>> d = transformdict(str.lower)
>     >>> d['Foo'] = 5
>     >>> d['foo']
>     5
>     >>> d['FOO']
>     5
>     >>> list(d)
>     ['Foo']

I guess another example is creating an "identity dict" (see 
http://code.activestate.com/lists/python-ideas/7161/) by doing

     d = transformdict(id)

-- 
Richard


From barry at python.org  Tue Sep 10 15:49:28 2013
From: barry at python.org (Barry Warsaw)
Date: Tue, 10 Sep 2013 09:49:28 -0400
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <CAMpsgwbbCOx9tGBkz-Pj5hwJaNP+Dp2UPfov75PksD72+5FVfg@mail.gmail.com>
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAMpsgwbbCOx9tGBkz-Pj5hwJaNP+Dp2UPfov75PksD72+5FVfg@mail.gmail.com>
Message-ID: <20130910094928.1cad3f53@anarchist>

On Sep 10, 2013, at 12:04 PM, Victor Stinner wrote:

>The http.client and email.message modules convert headers to lower
>case, but keep the original case.

As RDM pointed out on the tracker, email headers aren't a great use case for
this because they aren't really dictionaries.  They're lists with some
dict-like syntax.

-Barry

From solipsis at pitrou.net  Tue Sep 10 15:57:40 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 10 Sep 2013 15:57:40 +0200
Subject: [Python-Dev] Add a "transformdict" to collections
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAMpsgwbbCOx9tGBkz-Pj5hwJaNP+Dp2UPfov75PksD72+5FVfg@mail.gmail.com>
 <20130910094928.1cad3f53@anarchist>
Message-ID: <20130910155740.1aeab97a@pitrou.net>

Le Tue, 10 Sep 2013 09:49:28 -0400,
Barry Warsaw <barry at python.org> a ?crit :
> On Sep 10, 2013, at 12:04 PM, Victor Stinner wrote:
> 
> >The http.client and email.message modules convert headers to lower
> >case, but keep the original case.
> 
> As RDM pointed out on the tracker, email headers aren't a great use
> case for this because they aren't really dictionaries.  They're lists
> with some dict-like syntax.

defaultdict(list)?




From barry at python.org  Tue Sep 10 16:06:00 2013
From: barry at python.org (Barry Warsaw)
Date: Tue, 10 Sep 2013 10:06:00 -0400
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130910155740.1aeab97a@pitrou.net>
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAMpsgwbbCOx9tGBkz-Pj5hwJaNP+Dp2UPfov75PksD72+5FVfg@mail.gmail.com>
 <20130910094928.1cad3f53@anarchist>
 <20130910155740.1aeab97a@pitrou.net>
Message-ID: <20130910100600.0078b29a@anarchist>

On Sep 10, 2013, at 03:57 PM, Antoine Pitrou wrote:

>Le Tue, 10 Sep 2013 09:49:28 -0400,
>Barry Warsaw <barry at python.org> a ?crit :
>> On Sep 10, 2013, at 12:04 PM, Victor Stinner wrote:
>> 
>> >The http.client and email.message modules convert headers to lower
>> >case, but keep the original case.
>> 
>> As RDM pointed out on the tracker, email headers aren't a great use
>> case for this because they aren't really dictionaries.  They're lists
>> with some dict-like syntax.
>
>defaultdict(list)?

Not really.  Email headers support duplicates, although the dict-syntax will
only return the first such header.  Alternative syntax like .get_all() gives
you all of them.

But anyway, don't let this stop you!  I like your robots-in-disguise[1] dicts
no matter what you call them, and even if email can't use them.

-Barry

[1] http://www.youtube.com/watch?v=59QeKkjishs

From arigo at tunes.org  Tue Sep 10 16:09:16 2013
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 10 Sep 2013 16:09:16 +0200
Subject: [Python-Dev] PEP 447: add type.__locallookup__
Message-ID: <CAMSv6X205ONcjSZx75pyCErc7E6UeXAJ51BU-1hgEQKueNX1iw@mail.gmail.com>

Hi Mark,

On Mon, Sep 9, 2013 at 11:18 PM, Mark Shannon <mark at hotpy.org> wrote:
> 5. Other implementations. What do the Jython/IronPython/PyPy developers
> think?

Thanks for asking :-)  I'm fine with staying out of language design
issues like this one, and I believe it's the general concensus in
PyPy.  Whatever gets decided will probably be easy to port to PyPy and
have no measurable performance impact: the extra checks on the type,
if any, are all constant-folded by the JIT.


A bient?t,

Armin.

From arigo at tunes.org  Tue Sep 10 16:15:56 2013
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 10 Sep 2013 16:15:56 +0200
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <522F21CF.4060609@gmail.com>
References: <20130910112832.75ed6c8b@pitrou.net> <522F21CF.4060609@gmail.com>
Message-ID: <CAMSv6X2MmJwg2E-kom7VF2_+U3btqRQfdEj1ioxg3AZAi5T61g@mail.gmail.com>

Hi Richard,

On Tue, Sep 10, 2013 at 3:42 PM, Richard Oudkerk <shibturn at gmail.com> wrote:
> I guess another example is creating an "identity dict" (see
> http://code.activestate.com/lists/python-ideas/7161/) by doing
>
>     d = transformdict(id)

This is bogus, because only the id will be stored, and the original
key object will be forgotten (and its id probably reused).


A bient?t,

Armin.

From eliben at gmail.com  Tue Sep 10 16:18:41 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Tue, 10 Sep 2013 07:18:41 -0700
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130910142258.02c148dd@pitrou.net>
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
 <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>
 <20130910142258.02c148dd@pitrou.net>
Message-ID: <CAF-Rda9j0T0O=SmpOn0FD4CzrRi5qtkLHfqsadTQ=8q5MWpseQ@mail.gmail.com>

On Tue, Sep 10, 2013 at 5:22 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> Le Tue, 10 Sep 2013 22:00:37 +1000,
> Nick Coghlan <ncoghlan at gmail.com> a ?crit :
> > Is this just syntactic sugar for recursive lookup of a transformed
> > version in __missing__?
>
> Nope. For one, it doesn't use __missing__ at all. I think
> using __missing__ would be... missing the point, because it wouldn't
> working properly if you have e.g. X != Y such that transform(X) == Y
> and transform(Y) == X.
>
> (a simple example being transform = str.swapcase :-))
>
> > Or a way of supplying a custom "key" function
> > to a dictionary?
>
> Probably, although I'm not entirely sure what you mean by that :-)
>
> > Any such proposal should consider how it composes with other dict
> > variants like defaultdict, OrderedDict and counter.
>
> Well, one sticking point here is that those variants don't compose with
> each other already :-)
> But, yes, I may make another proposal with composition in mind.
>

Does anyone have an idea how to make the existing variants more composable?
It would be nice to get a better understanding of this before we add
another variant. Conceptually, composability makes a lot of sense (what if
we want this transformdict but also in insertion order...)

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/eb84e9b3/attachment.html>

From solipsis at pitrou.net  Tue Sep 10 16:37:19 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 10 Sep 2013 16:37:19 +0200
Subject: [Python-Dev] Add a "transformdict" to collections
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
 <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>
 <20130910142258.02c148dd@pitrou.net>
 <CAF-Rda9j0T0O=SmpOn0FD4CzrRi5qtkLHfqsadTQ=8q5MWpseQ@mail.gmail.com>
Message-ID: <20130910163719.0c07be69@pitrou.net>

Le Tue, 10 Sep 2013 07:18:41 -0700,
Eli Bendersky <eliben at gmail.com> a ?crit :
> On Tue, Sep 10, 2013 at 5:22 AM, Antoine Pitrou <solipsis at pitrou.net>
> wrote:
> 
> > Le Tue, 10 Sep 2013 22:00:37 +1000,
> > Nick Coghlan <ncoghlan at gmail.com> a ?crit :
> > > Is this just syntactic sugar for recursive lookup of a transformed
> > > version in __missing__?
> >
> > Nope. For one, it doesn't use __missing__ at all. I think
> > using __missing__ would be... missing the point, because it wouldn't
> > working properly if you have e.g. X != Y such that transform(X) == Y
> > and transform(Y) == X.
> >
> > (a simple example being transform = str.swapcase :-))
> >
> > > Or a way of supplying a custom "key" function
> > > to a dictionary?
> >
> > Probably, although I'm not entirely sure what you mean by that :-)
> >
> > > Any such proposal should consider how it composes with other dict
> > > variants like defaultdict, OrderedDict and counter.
> >
> > Well, one sticking point here is that those variants don't compose
> > with each other already :-)
> > But, yes, I may make another proposal with composition in mind.
> >
> 
> Does anyone have an idea how to make the existing variants more
> composable? It would be nice to get a better understanding of this
> before we add another variant. Conceptually, composability makes a
> lot of sense (what if we want this transformdict but also in
> insertion order...)

AFAICT, the main problem with composability is that the implementation
is less simple and it removes many opportunities for optimization. It
may or may not be a problem, depending on your use cases.

(I'm playing a bit with a TransformMap which is a composable version of
transformdict, and it's clear the implementation is less amenable to
optimization tweaks and shortcuts: in fact, I have to *add* some logic
to e.g. cater with defaultdict)

Regards

Antoine.



From solipsis at pitrou.net  Tue Sep 10 16:37:57 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 10 Sep 2013 16:37:57 +0200
Subject: [Python-Dev] Add a "transformdict" to collections
References: <20130910112832.75ed6c8b@pitrou.net> <522F21CF.4060609@gmail.com>
 <CAMSv6X2MmJwg2E-kom7VF2_+U3btqRQfdEj1ioxg3AZAi5T61g@mail.gmail.com>
Message-ID: <20130910163757.30288824@pitrou.net>

Le Tue, 10 Sep 2013 16:15:56 +0200,
Armin Rigo <arigo at tunes.org> a ?crit :
> Hi Richard,
> 
> On Tue, Sep 10, 2013 at 3:42 PM, Richard Oudkerk <shibturn at gmail.com>
> wrote:
> > I guess another example is creating an "identity dict" (see
> > http://code.activestate.com/lists/python-ideas/7161/) by doing
> >
> >     d = transformdict(id)
> 
> This is bogus, because only the id will be stored, and the original
> key object will be forgotten (and its id probably reused).

No, the original key is kept as well.

Regards

Antoine.



From ethan at stoneleaf.us  Tue Sep 10 16:00:17 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 10 Sep 2013 07:00:17 -0700
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <522F1A24.5030808@avl.com>
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
 <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>
 <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com>
 <522F1A24.5030808@avl.com>
Message-ID: <522F25F1.4030104@stoneleaf.us>

On 09/10/2013 06:09 AM, Hrvoje Niksic wrote:
> On 09/10/2013 02:24 PM, Paul Moore wrote:
>>>>> td['FOO'] = 42
>>>>> td['foo'] = 32
>>>>> list(td.keys())
>>
>> ['FOO'] or ['foo']? Both answers are justifiable.
>
> Note that the same question can be reasonably asked for dict itself:
>
>>>> d = {}
>>>> d[1.0] = 'foo'
>>>> d[1] = 'bar'
>>>> d
> {1.0: 'bar'}
>
> So, dict.__setitem__ only replaces the value, leaving the original key in place. transformdict should probably do the
> same, returning 'FOO' in your example.

+1

From solipsis at pitrou.net  Tue Sep 10 16:54:38 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 10 Sep 2013 16:54:38 +0200
Subject: [Python-Dev] Add a "transformdict" to collections
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
 <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>
 <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com>
 <522F1A24.5030808@avl.com>
Message-ID: <20130910165438.297d71bc@pitrou.net>

Le Tue, 10 Sep 2013 15:09:56 +0200,
Hrvoje Niksic <hrvoje.niksic at avl.com> a ?crit :
> On 09/10/2013 02:24 PM, Paul Moore wrote:
> >>>> td['FOO'] = 42
> >>>> td['foo'] = 32
> >>>> list(td.keys())
> >
> > ['FOO'] or ['foo']? Both answers are justifiable.
> 
> Note that the same question can be reasonably asked for dict itself:
> 
>  >>> d = {}
>  >>> d[1.0] = 'foo'
>  >>> d[1] = 'bar'
>  >>> d
> {1.0: 'bar'}
> 
> So, dict.__setitem__ only replaces the value, leaving the original
> key in place. transformdict should probably do the same, returning
> 'FOO' in your example.

It's not that obvious. It's not common to rely on that aspect of dict
semantics, because you will usually lookup using the exact same type,
not a compatible one. I would expect very little code, if any, to rely
on this.

(also, I would intuitively expect the latest key to be held, not the
first one, since that's what happens for values.)

Regards

Antoine.



From corbellini.andrea at gmail.com  Tue Sep 10 11:37:01 2013
From: corbellini.andrea at gmail.com (Andrea Corbellini)
Date: Tue, 10 Sep 2013 11:37:01 +0200
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130910112832.75ed6c8b@pitrou.net>
References: <20130910112832.75ed6c8b@pitrou.net>
Message-ID: <CAD8LpNACA_XS3aOW7N86XT+TKVTRJWg0o_2vK7XLe3gXi3997Q@mail.gmail.com>

On Tue, Sep 10, 2013 at 11:28 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Therefore I propose adding the general pattern. Simple example:
>
>    >>> d = transformdict(str.lower)
>    >>> d['Foo'] = 5
>    >>> d['foo']
>    5
>    >>> d['FOO']
>    5
>    >>> list(d)
>    ['Foo']
>
> (case-insensitive but case-preserving, as the best filesystems are ;-))

Just a nitpick: if this example (or a similar one) gets into the
documentation, it would be better to replace str.lower() with
str.casefold().

From shibturn at gmail.com  Tue Sep 10 15:42:39 2013
From: shibturn at gmail.com (Richard Oudkerk)
Date: Tue, 10 Sep 2013 14:42:39 +0100
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130910112832.75ed6c8b@pitrou.net>
References: <20130910112832.75ed6c8b@pitrou.net>
Message-ID: <522F21CF.4060609@gmail.com>

On 10/09/2013 10:28am, Antoine Pitrou wrote:
> Therefore I propose adding the general pattern. Simple example:
>
>     >>> d = transformdict(str.lower)
>     >>> d['Foo'] = 5
>     >>> d['foo']
>     5
>     >>> d['FOO']
>     5
>     >>> list(d)
>     ['Foo']

I guess another example is creating an "identity dict" (see 
http://code.activestate.com/lists/python-ideas/7161/) by doing

     d = transformdict(id)

-- 
Richard

From shibturn at gmail.com  Tue Sep 10 16:40:55 2013
From: shibturn at gmail.com (Richard Oudkerk)
Date: Tue, 10 Sep 2013 15:40:55 +0100
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <CAMSv6X2MmJwg2E-kom7VF2_+U3btqRQfdEj1ioxg3AZAi5T61g@mail.gmail.com>
References: <20130910112832.75ed6c8b@pitrou.net> <522F21CF.4060609@gmail.com>
 <CAMSv6X2MmJwg2E-kom7VF2_+U3btqRQfdEj1ioxg3AZAi5T61g@mail.gmail.com>
Message-ID: <522F2F77.9020805@gmail.com>

On 10/09/2013 3:15pm, Armin Rigo wrote:
> Hi Richard,
>
> On Tue, Sep 10, 2013 at 3:42 PM, Richard Oudkerk <shibturn at gmail.com> wrote:
>> I guess another example is creating an "identity dict" (see
>> http://code.activestate.com/lists/python-ideas/7161/) by doing
>>
>>      d = transformdict(id)
>
> This is bogus, because only the id will be stored, and the original
> key object will be forgotten (and its id probably reused).

Seems to work for me:

 >>> import collections
 >>> d = collections.transformdict(id)
 >>> L = [1,2,3]
 >>> d[L] = None
 >>> L in d
True
 >>> [1,2,3] in d
False
 >>> print(d[L])
None
 >>> d._data
{41444136: ([1, 2, 3], None)}
 >>> list(d)
[[1, 2, 3]]

However __repr__() is broken:

 >>> d
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "C:\Repos\cpython-dirty\lib\collections\abc.py", line 444, in 
__repr__
     return '{0.__class__.__name__}({0._mapping!r})'.format(self)
   File "C:\Repos\cpython-dirty\lib\collections\__init__.py", line 944, 
in __repr__
     self._transform, repr(dict(self)))
TypeError: unhashable type: 'list'

--
Richard

From zhangpeipei812 at outlook.com  Tue Sep 10 12:58:32 2013
From: zhangpeipei812 at outlook.com (=?gb2312?B?1cXF5cXl?=)
Date: Tue, 10 Sep 2013 18:58:32 +0800
Subject: [Python-Dev] IOCP (I/O Completion Port) in Python ?
Message-ID: <BLU402-EAS15259F82C4E6C2CEBFC83789D380@phx.gbl>

Hello:
	I wondering why there is no standard IOCP module in Python ?
As I know: Python3 have support epoll in linux and kqueue in freebsd.
Is there a plan to add IOCP module for Windows ?

Regards!
peipei

From guido at python.org  Tue Sep 10 17:08:20 2013
From: guido at python.org (Guido van Rossum)
Date: Tue, 10 Sep 2013 08:08:20 -0700
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <522EC878.6080409@g.nevcal.com>
References: <5228C00F.7000209@jcea.es> <l0d311$b7n$1@ger.gmane.org>
 <522EC878.6080409@g.nevcal.com>
Message-ID: <CAP7+vJJ0oG+QFZQKc4fLvOHwiu315WVoozv45dBdOqv59fw=Gw@mail.gmail.com>

Why do several posts in this thread have an Unsubscribe link that tries to
unsubscribe me from the list? (I saw one by Glen, and another one by Donald
Stufft.)

(Come to think of it, what's the point of having an Unbub link in ever
message that goes out?)


On Tue, Sep 10, 2013 at 12:21 AM, Glenn Linderman <v+python at g.nevcal.com>wrote:

>  On 9/6/2013 10:22 AM, Dan Callahan wrote:
>
> On 9/5/13 12:31 PM, Jesus Cea wrote:
>
> I have big hopes for Mozilla Persona, looking forward
> Python infrastructure support :).
>
>
> Hi, I'm the project lead on Persona signin, and I spoke at PyCon earlier
> this year regarding why and how Mozilla is building Persona. If you'd like
> some more background, that video [0] is worth a look.
>
> Let's pull this discussion up a level:
>
> It sounds like many people (Jesus, Donald, Toshio, Barry, Tres, Dirkjan,
> etc.) are interested in seeing Persona on Python.org properties, and most
> of the objections coming from a place of "Persona hasn't gone viral, what
> if this is wasted effort?"
>
>
> OK, let's pull this discussion down a level: testing it out.
>
> So I tried to login to the crossword.thetimes.co.uk -- I used an email
> address persona had never seen, it asked me for a password, and sent me a
> confirmation message, containing a link that I clicked on.
>
> However, as I was reading clues and filling in blanks, I got a popup that
> said "login failure [object Object]". And crossword told me it was saving
> locally, and to login to save to the server. And the Log in button stayed
> displayed, rather than a Log out button, which I assume it might get
> replaced with if I ever get successfully logged in.
>
> Firefox 23.0.1, Windows 7 64-bit with autoupdates. Need any other info?
> Write me privately if you want the email address I used (not the one I use
> here), or the password. I used new ones, so I can share for testing, and
> then discard them and use different ones "for real". If the system actually
> works. Hey, the video demos looked great...
>
> Glenn
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/f2b0491e/attachment-0001.html>

From guido at python.org  Tue Sep 10 17:11:05 2013
From: guido at python.org (Guido van Rossum)
Date: Tue, 10 Sep 2013 08:11:05 -0700
Subject: [Python-Dev] IOCP (I/O Completion Port) in Python ?
In-Reply-To: <BLU402-EAS15259F82C4E6C2CEBFC83789D380@phx.gbl>
References: <BLU402-EAS15259F82C4E6C2CEBFC83789D380@phx.gbl>
Message-ID: <CAP7+vJLrMFY5kZjBR4hR5nZo+pWapTeWfjKKewBhWqJO9BLzrQ@mail.gmail.com>

One will come with Tulip (http://code.google.com/p/tulip/), assuming we can
get PEP 3156 accepted at least as "experimental" and the core Tulip code in
the stdlib. At the moment the IOCP code there is pretty specialized for use
with Tulip -- but if you want to help extracting the IOCP code from there
for inclusion into the stdlib that would be great!


On Tue, Sep 10, 2013 at 3:58 AM, ??? <zhangpeipei812 at outlook.com> wrote:

> Hello:
>         I wondering why there is no standard IOCP module in Python ?
> As I know: Python3 have support epoll in linux and kqueue in freebsd.
> Is there a plan to add IOCP module for Windows ?
>
> Regards!
> peipei
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/15ddf52a/attachment.html>

From v+python at g.nevcal.com  Tue Sep 10 17:29:52 2013
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Tue, 10 Sep 2013 08:29:52 -0700
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <CAP7+vJJ0oG+QFZQKc4fLvOHwiu315WVoozv45dBdOqv59fw=Gw@mail.gmail.com>
References: <5228C00F.7000209@jcea.es> <l0d311$b7n$1@ger.gmane.org>
 <522EC878.6080409@g.nevcal.com>
 <CAP7+vJJ0oG+QFZQKc4fLvOHwiu315WVoozv45dBdOqv59fw=Gw@mail.gmail.com>
Message-ID: <522F3AF0.8020300@g.nevcal.com>

On 9/10/2013 8:08 AM, Guido van Rossum wrote:
> Why do several posts in this thread have an Unsubscribe link that 
> tries to unsubscribe me from the list? (I saw one by Glen, and another 
> one by Donald Stufft.)

Seems to be in all of them. Probably added by the mailing list software.

Why don't you always see it?

Possibly a combination of being directly listed as a recipient, so that 
you get a copy not sent through the mailing list, together with an email 
client that suppresses the duplicates (if the mailing list software 
doesn't change the Message-Id:). But since heretofore you haven't 
participated in this thread, if you don't see the Unsub link in all of 
this thread, I doubt this is the explanation. You message to which I'm 
replying is the first in this thread that came directly to me and which 
doesn't have the link, because you addressed me directly.  Use 
Reply-List rather than Reply-All if your MUA supports that.

> (Come to think of it, what's the point of having an Unbub link in ever 
> message that goes out?)

To avoid people that accidentally subscribe from asking how to 
unsubscribe, perhaps.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/059114e6/attachment.html>

From barry at python.org  Tue Sep 10 17:41:34 2013
From: barry at python.org (Barry Warsaw)
Date: Tue, 10 Sep 2013 11:41:34 -0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <CAP7+vJJ0oG+QFZQKc4fLvOHwiu315WVoozv45dBdOqv59fw=Gw@mail.gmail.com>
References: <5228C00F.7000209@jcea.es> <l0d311$b7n$1@ger.gmane.org>
 <522EC878.6080409@g.nevcal.com>
 <CAP7+vJJ0oG+QFZQKc4fLvOHwiu315WVoozv45dBdOqv59fw=Gw@mail.gmail.com>
Message-ID: <20130910114134.1015365d@anarchist>

On Sep 10, 2013, at 08:08 AM, Guido van Rossum wrote:

>Why do several posts in this thread have an Unsubscribe link that tries to
>unsubscribe me from the list? (I saw one by Glen, and another one by Donald
>Stufft.)

This is way off topic, but I suspect your original response didn't trim your
little unsub footer and they didn't trim it from their responses.

Looking at my list copy of *this* message, I see my own unsub footer, but my
MUA automatically trims it in my response buffer.  OTOH, I always try to trim
my responses anyway, which I think is good netiquette, and which is easy for
me with Emacs as my edit/composer of messages.

>(Come to think of it, what's the point of having an Unbub link in ever
>message that goes out?)

As a general Mailman feature, it decreases the likelihood that laypeople who
want off a mailing list will do bad things to unsub because they can't figure
out how to do it, like complain back to the list or the admins, spam block, or
worse, report the list as spam.  All of which and more we've seen in the wild.
Adding the footer with the unsub block doesn't eliminate this, because some
people are lazy, stressed, mean, or frustrated, but it does reduce the
incidences.

Now, whether it's appropriate for a highly technical list like python-dev is
up for discussion (but maybe not here?).  While messages are personalized on
this list (meaning, you get a unique copy of it for bounce tracking, and yes
unsub personalization), those unsub stanzas cannot currently be disabled on a
per-user basis.

-Barry

From ethan at stoneleaf.us  Tue Sep 10 16:59:22 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 10 Sep 2013 07:59:22 -0700
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130910165438.297d71bc@pitrou.net>
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
 <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>
 <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com>
 <522F1A24.5030808@avl.com> <20130910165438.297d71bc@pitrou.net>
Message-ID: <522F33CA.6000909@stoneleaf.us>

On 09/10/2013 07:54 AM, Antoine Pitrou wrote:
> Le Tue, 10 Sep 2013 15:09:56 +0200,
> Hrvoje Niksic <hrvoje.niksic at avl.com> a ?crit :
>> On 09/10/2013 02:24 PM, Paul Moore wrote:
>>>>>> td['FOO'] = 42
>>>>>> td['foo'] = 32
>>>>>> list(td.keys())
>>>
>>> ['FOO'] or ['foo']? Both answers are justifiable.
>>
>> Note that the same question can be reasonably asked for dict itself:
>>
>>   >>> d = {}
>>   >>> d[1.0] = 'foo'
>>   >>> d[1] = 'bar'
>>   >>> d
>> {1.0: 'bar'}
>>
>> So, dict.__setitem__ only replaces the value, leaving the original
>> key in place. transformdict should probably do the same, returning
>> 'FOO' in your example.
>
> It's not that obvious. It's not common to rely on that aspect of dict
> semantics, because you will usually lookup using the exact same type,
> not a compatible one. I would expect very little code, if any, to rely
> on this.
>
> (also, I would intuitively expect the latest key to be held, not the
> first one, since that's what happens for values.)

Whether there is a bunch of code that relies on this behavior is irrelevant.  That behavior is already in place, and 
having another dict that is just the opposite will only lead to more confusion, not less.

--
~Ethan~

From rymg19 at gmail.com  Tue Sep 10 17:26:00 2013
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Tue, 10 Sep 2013 10:26:00 -0500
Subject: [Python-Dev] [RELEASED] Python 3.4.0a2
In-Reply-To: <l0mbrr$ofo$1@ger.gmane.org>
References: <522DB8D3.1030803@hastings.org>
 <nad-2495B7.12292709092013@news.gmane.org>
 <l0mbrr$ofo$1@ger.gmane.org>
Message-ID: <CAO41-mPnC-Rdk=WANFk9dqaWufLeoH-GZM5nP-Q1oc6xgUtecA@mail.gmail.com>

MS Windows is on a steep decline? I mean, I know Windows 8 isn't the most
popular thing on the planet, but...Windows itself? If anything would be
rising, I'd still prefer it to be either Linux or Haiku.


On Tue, Sep 10, 2013 at 12:48 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> Ned Deily, 09.09.2013 21:29:
> > 3.4.0a2 also contains a new batteries-included feature for OS X
> > users.  The python.org 64-bit/32-bit installer now includes its own
> > private version of Tcl/Tk 8.5.14 so it is finally no longer necessary to
> > install a third-party version of Tcl/Tk 8.5 to workaround the
> > problematic system versions shipped in OS X 10.6+.  More improvements to
> > come.
>
> With MS Windows on steep decline, I'm so happy we have at least MacOS-X
> rising to keep us busy.
>
> Stefan
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com
>



-- 
Ryan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/6753146e/attachment.html>

From donald.stufft at gmail.com  Tue Sep 10 17:28:25 2013
From: donald.stufft at gmail.com (Donald Stufft)
Date: Tue, 10 Sep 2013 11:28:25 -0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <CAP7+vJJ0oG+QFZQKc4fLvOHwiu315WVoozv45dBdOqv59fw=Gw@mail.gmail.com>
References: <5228C00F.7000209@jcea.es> <l0d311$b7n$1@ger.gmane.org>
 <522EC878.6080409@g.nevcal.com>
 <CAP7+vJJ0oG+QFZQKc4fLvOHwiu315WVoozv45dBdOqv59fw=Gw@mail.gmail.com>
Message-ID: <EF440AAC-D5D9-4A31-966A-8C093A345F10@gmail.com>


On Sep 10, 2013, at 11:08 AM, Guido van Rossum <guido at python.org> wrote:

> Why do several posts in this thread have an Unsubscribe link that tries to unsubscribe me from the list? (I saw one by Glen, and another one by Donald Stufft.)
> 
> (Come to think of it, what's the point of having an Unbub link in ever message that goes out?)

All posts should have that. It's possible your mail client is collapsing it in some cases though because it looks like quoted text.

> 
> 
> On Tue, Sep 10, 2013 at 12:21 AM, Glenn Linderman <v+python at g.nevcal.com> wrote:
> On 9/6/2013 10:22 AM, Dan Callahan wrote:
>> On 9/5/13 12:31 PM, Jesus Cea wrote: 
>>> I have big hopes for Mozilla Persona, looking forward 
>>> Python infrastructure support :). 
>> 
>> Hi, I'm the project lead on Persona signin, and I spoke at PyCon earlier this year regarding why and how Mozilla is building Persona. If you'd like some more background, that video [0] is worth a look. 
>> 
>> Let's pull this discussion up a level: 
>> 
>> It sounds like many people (Jesus, Donald, Toshio, Barry, Tres, Dirkjan, etc.) are interested in seeing Persona on Python.org properties, and most of the objections coming from a place of "Persona hasn't gone viral, what if this is wasted effort?"
> 
> OK, let's pull this discussion down a level: testing it out.
> 
> So I tried to login to the crossword.thetimes.co.uk -- I used an email address persona had never seen, it asked me for a password, and sent me a confirmation message, containing a link that I clicked on.
> 
> However, as I was reading clues and filling in blanks, I got a popup that said "login failure [object Object]". And crossword told me it was saving locally, and to login to save to the server. And the Log in button stayed displayed, rather than a Log out button, which I assume it might get replaced with if I ever get successfully logged in.
> 
> Firefox 23.0.1, Windows 7 64-bit with autoupdates. Need any other info? Write me privately if you want the email address I used (not the one I use here), or the password. I used new ones, so I can share for testing, and then discard them and use different ones "for real". If the system actually works. Hey, the video demos looked great...
> 
> Glenn
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
> 
> 
> 
> 
> -- 
> --Guido van Rossum (python.org/~guido)


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/a650beeb/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/a650beeb/attachment.sig>

From janzert at janzert.com  Tue Sep 10 17:54:20 2013
From: janzert at janzert.com (Janzert)
Date: Tue, 10 Sep 2013 11:54:20 -0400
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130910165438.297d71bc@pitrou.net>
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
 <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>
 <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com>
 <522F1A24.5030808@avl.com> <20130910165438.297d71bc@pitrou.net>
Message-ID: <l0nfbc$tp1$1@ger.gmane.org>

On 9/10/2013 10:54 AM, Antoine Pitrou wrote:
> (also, I would intuitively expect the latest key to be held, not the
> first one, since that's what happens for values.)
>
> Regards
>
> Antoine.
>

I intuitively expected, and I think most often would want, the first key 
to be held. The reason being that I would except most often the initial 
insertion to be the canonical form.

Whichever way is adopted, I think even if you say it's implementation 
specific, people will end up relying on it in their code.

Janzert


From phd at phdru.name  Tue Sep 10 18:03:35 2013
From: phd at phdru.name (Oleg Broytman)
Date: Tue, 10 Sep 2013 20:03:35 +0400
Subject: [Python-Dev] Offtopic: OpenID Providers
In-Reply-To: <EF440AAC-D5D9-4A31-966A-8C093A345F10@gmail.com>
References: <5228C00F.7000209@jcea.es> <l0d311$b7n$1@ger.gmane.org>
 <522EC878.6080409@g.nevcal.com>
 <CAP7+vJJ0oG+QFZQKc4fLvOHwiu315WVoozv45dBdOqv59fw=Gw@mail.gmail.com>
 <EF440AAC-D5D9-4A31-966A-8C093A345F10@gmail.com>
Message-ID: <20130910160335.GA3093@iskra.aviel.ru>

On Tue, Sep 10, 2013 at 11:28:25AM -0400, Donald Stufft <donald.stufft at gmail.com> wrote:
> On Sep 10, 2013, at 11:08 AM, Guido van Rossum <guido at python.org> wrote:
> > Why do several posts in this thread have an Unsubscribe link that tries to unsubscribe me from the list? (I saw one by Glen, and another one by Donald Stufft.)
> > 
> > (Come to think of it, what's the point of having an Unbub link in ever message that goes out?)
> 
> All posts should have that. It's possible your mail client is collapsing it in some cases though because it looks like quoted text.

   When a mail message is all plain/text Mailman simply inline the
stance into the text. When a message is a complex tree of MIME parts
(like the message I'm replying to) Mailman append the stance to the end
of the message as another MIME part. Some MUAs could ignore that part --
they neither show the part nor include it in replies. Mine (mutt) shows
it and includes in replies anyway, I always trim it myself.

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From v+python at g.nevcal.com  Tue Sep 10 17:58:43 2013
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Tue, 10 Sep 2013 08:58:43 -0700
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <l0nfbc$tp1$1@ger.gmane.org>
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
 <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>
 <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com>
 <522F1A24.5030808@avl.com> <20130910165438.297d71bc@pitrou.net>
 <l0nfbc$tp1$1@ger.gmane.org>
Message-ID: <522F41B3.9020608@g.nevcal.com>

On 9/10/2013 8:54 AM, Janzert wrote:
>
> I intuitively expected, and I think most often would want, the first 
> key to be held. 

My intuition matches yours, but my thoughts are that it should be 
changeable by specific request.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/239efdb8/attachment.html>

From eric at trueblade.com  Tue Sep 10 18:12:43 2013
From: eric at trueblade.com (Eric V. Smith)
Date: Tue, 10 Sep 2013 12:12:43 -0400
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <l0nfbc$tp1$1@ger.gmane.org>
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
 <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>
 <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com>
 <522F1A24.5030808@avl.com> <20130910165438.297d71bc@pitrou.net>
 <l0nfbc$tp1$1@ger.gmane.org>
Message-ID: <522F44FB.8040408@trueblade.com>

On 09/10/2013 11:54 AM, Janzert wrote:
> On 9/10/2013 10:54 AM, Antoine Pitrou wrote:
>> (also, I would intuitively expect the latest key to be held, not the
>> first one, since that's what happens for values.)
>>
>> Regards
>>
>> Antoine.
>>
> 
> I intuitively expected, and I think most often would want, the first key
> to be held. The reason being that I would except most often the initial
> insertion to be the canonical form.

My stock ticker example (but from a real problem I had a few days ago)
would want to have this example: I want the first key, because I'm
pre-seeding the dict with a variety of keys that are of the canonical form.

> Whichever way is adopted, I think even if you say it's implementation
> specific, people will end up relying on it in their code.

Agreed.

Eric.


From nigel at nigelsmall.com  Tue Sep 10 17:21:18 2013
From: nigel at nigelsmall.com (Nigel Small)
Date: Tue, 10 Sep 2013 16:21:18 +0100
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <522F2F77.9020805@gmail.com>
References: <20130910112832.75ed6c8b@pitrou.net> <522F21CF.4060609@gmail.com>
 <CAMSv6X2MmJwg2E-kom7VF2_+U3btqRQfdEj1ioxg3AZAi5T61g@mail.gmail.com>
 <522F2F77.9020805@gmail.com>
Message-ID: <CAFaXQDo1ifjvNrzohbsDV_m38fbpKfd5G2Q0yZfbJp9HB6FzPQ@mail.gmail.com>

Could a more generic variant of this class work? In the same way that
`sorted` can accept a comparison function, similar could be done for a
dictionary-like class:

d = transformdict(key=str.lower)

Strictly speaking, this would provide case-insensitive but not
case-preserving behaviour. For any given use case, though, a function could
instead be supplied to "normalise" the key (upper, lower, title case, etc)
in a way that fits that case. I can't think of many real cases where
multiple types of capitalisation would be useful within the same dictionary.

Nigel



On 10 September 2013 15:40, Richard Oudkerk <shibturn at gmail.com> wrote:

> On 10/09/2013 3:15pm, Armin Rigo wrote:
>
>> Hi Richard,
>>
>> On Tue, Sep 10, 2013 at 3:42 PM, Richard Oudkerk <shibturn at gmail.com>
>> wrote:
>>
>>> I guess another example is creating an "identity dict" (see
>>> http://code.activestate.com/**lists/python-ideas/7161/<http://code.activestate.com/lists/python-ideas/7161/>)
>>> by doing
>>>
>>>      d = transformdict(id)
>>>
>>
>> This is bogus, because only the id will be stored, and the original
>> key object will be forgotten (and its id probably reused).
>>
>
> Seems to work for me:
>
> >>> import collections
> >>> d = collections.transformdict(id)
> >>> L = [1,2,3]
> >>> d[L] = None
> >>> L in d
> True
> >>> [1,2,3] in d
> False
> >>> print(d[L])
> None
> >>> d._data
> {41444136: ([1, 2, 3], None)}
> >>> list(d)
> [[1, 2, 3]]
>
> However __repr__() is broken:
>
> >>> d
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "C:\Repos\cpython-dirty\lib\**collections\abc.py", line 444, in
> __repr__
>     return '{0.__class__.__name__}({0._**mapping!r})'.format(self)
>   File "C:\Repos\cpython-dirty\lib\**collections\__init__.py", line 944,
> in __repr__
>     self._transform, repr(dict(self)))
> TypeError: unhashable type: 'list'
>
> --
> Richard
>
> ______________________________**_________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/**mailman/listinfo/python-dev<https://mail.python.org/mailman/listinfo/python-dev>
> Unsubscribe: https://mail.python.org/**mailman/options/python-dev/**
> nigel%40nigelsmall.com<https://mail.python.org/mailman/options/python-dev/nigel%40nigelsmall.com>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/5107ee06/attachment.html>

From shibturn at gmail.com  Tue Sep 10 19:09:30 2013
From: shibturn at gmail.com (Richard Oudkerk)
Date: Tue, 10 Sep 2013 18:09:30 +0100
Subject: [Python-Dev] IOCP (I/O Completion Port) in Python ?
In-Reply-To: <BLU402-EAS15259F82C4E6C2CEBFC83789D380@phx.gbl>
References: <BLU402-EAS15259F82C4E6C2CEBFC83789D380@phx.gbl>
Message-ID: <522F524A.6000800@gmail.com>

On 10/09/2013 11:58am, ??? wrote:
> Hello:
> 	I wondering why there is no standard IOCP module in Python ?
> As I know: Python3 have support epoll in linux and kqueue in freebsd.
> Is there a plan to add IOCP module for Windows ?

_winapi does have some undocumented support for IOCP (used only by 
multiprocessing), but the APIs are not very nice and should not be 
considered stable.

(Also _winapi.ReadFile(), _winapi.WriteFile() only support pipes or 
sockets, not disk-based files where you also have to control file-position.)

-- 
Richard


From shibturn at gmail.com  Tue Sep 10 19:09:30 2013
From: shibturn at gmail.com (Richard Oudkerk)
Date: Tue, 10 Sep 2013 18:09:30 +0100
Subject: [Python-Dev] IOCP (I/O Completion Port) in Python ?
In-Reply-To: <BLU402-EAS15259F82C4E6C2CEBFC83789D380@phx.gbl>
References: <BLU402-EAS15259F82C4E6C2CEBFC83789D380@phx.gbl>
Message-ID: <522F524A.6000800@gmail.com>

On 10/09/2013 11:58am, ??? wrote:
> Hello:
> 	I wondering why there is no standard IOCP module in Python ?
> As I know: Python3 have support epoll in linux and kqueue in freebsd.
> Is there a plan to add IOCP module for Windows ?

_winapi does have some undocumented support for IOCP (used only by 
multiprocessing), but the APIs are not very nice and should not be 
considered stable.

(Also _winapi.ReadFile(), _winapi.WriteFile() only support pipes or 
sockets, not disk-based files where you also have to control file-position.)

-- 
Richard

From rdmurray at bitdance.com  Tue Sep 10 20:28:00 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Tue, 10 Sep 2013 14:28:00 -0400
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <CAFaXQDo1ifjvNrzohbsDV_m38fbpKfd5G2Q0yZfbJp9HB6FzPQ@mail.gmail.com>
References: <20130910112832.75ed6c8b@pitrou.net> <522F21CF.4060609@gmail.com>
 <CAMSv6X2MmJwg2E-kom7VF2_+U3btqRQfdEj1ioxg3AZAi5T61g@mail.gmail.com>
 <522F2F77.9020805@gmail.com>
 <CAFaXQDo1ifjvNrzohbsDV_m38fbpKfd5G2Q0yZfbJp9HB6FzPQ@mail.gmail.com>
Message-ID: <20130910182800.6A0112507A2@webabinitio.net>

On Tue, 10 Sep 2013 16:21:18 +0100, Nigel Small <nigel at nigelsmall.com> wrote:
> Could a more generic variant of this class work? In the same way that
> `sorted` can accept a comparison function, similar could be done for a
> dictionary-like class:
> 
> d = transformdict(key=str.lower)
> 
> Strictly speaking, this would provide case-insensitive but not
> case-preserving behaviour. For any given use case, though, a function could
> instead be supplied to "normalise" the key (upper, lower, title case, etc)
> in a way that fits that case. I can't think of many real cases where
> multiple types of capitalisation would be useful within the same dictionary.

I can:

  MIME-Version
  Message-ID
  Content-Type

For network protocols you really do want to *preserve* the case.

--David

From solipsis at pitrou.net  Tue Sep 10 20:31:40 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 10 Sep 2013 20:31:40 +0200
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <522F1411.8010109@v.loewis.de>
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
 <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>
 <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com>
 <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de>
Message-ID: <20130910203140.1e8c1008@fsol>

On Tue, 10 Sep 2013 14:44:01 +0200
"Martin v. L?wis" <martin at v.loewis.de> wrote:
> Am 10.09.13 14:35, schrieb Antoine Pitrou:
> >> ['FOO'] or ['foo']? Both answers are justifiable. Both are possibly
> >> even useful depending on context...
> > 
> > I think it would be best to leave it as an implementation detail,
> > because whichever is easiest to implement depends on the exact
> > implementation choices (e.g. C vs. Python).
> 
> I think this is the point where the datatype is *not* clearly
> straight-forward, and thus deserves a PEP.

Not saying you're necessary wrong, but a few data points:
- defaultdict was added without a PEP:
  http://bugs.python.org/issue1433928
- Counter was added without a PEP:
  http://bugs.python.org/issue1696199
- ChainMap was added without a PEP:
  http://bugs.python.org/issue11089
  http://bugs.python.org/issue11297
- namedtuple was added without a PEP:
  https://mail.python.org/pipermail/python-dev/2007-February/071196.html
  (no tracker reference for its addition)

OrderedDict, however, came with a PEP:
http://www.python.org/dev/peps/pep-0372/

> I think it would be a flaw to have this detail implementation-defined.
> This would be like saying that it is implementation-defined which
> of A,B,C is returned from "A and B and C" if all are true.

Ok, it seems everyone (except me :-)) agrees that it should return the
first key value, so that's how it will be.

Regards

Antoine.

From p.f.moore at gmail.com  Tue Sep 10 21:08:56 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 10 Sep 2013 20:08:56 +0100
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130910203140.1e8c1008@fsol>
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
 <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>
 <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com>
 <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de>
 <20130910203140.1e8c1008@fsol>
Message-ID: <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com>

On 10 September 2013 19:31, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> I think it would be a flaw to have this detail implementation-defined.
>> This would be like saying that it is implementation-defined which
>> of A,B,C is returned from "A and B and C" if all are true.
>
> Ok, it seems everyone (except me :-)) agrees that it should return the
> first key value, so that's how it will be.

If you retain the first key value, it's easy enough for the
application to implement "retain the last" semantics:

try:
    del d[k]
finally:
    d[k] = v

If you provide "retain the last", I can't see any obvious way of
implementing "retain the first" in application code without in effect
reimplementing the class.

Paul

From python at mrabarnett.plus.com  Tue Sep 10 21:59:56 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 10 Sep 2013 20:59:56 +0100
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com>
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
 <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>
 <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com>
 <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de>
 <20130910203140.1e8c1008@fsol>
 <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com>
Message-ID: <522F7A3C.1040701@mrabarnett.plus.com>

On 10/09/2013 20:08, Paul Moore wrote:
> On 10 September 2013 19:31, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>> I think it would be a flaw to have this detail implementation-defined.
>>> This would be like saying that it is implementation-defined which
>>> of A,B,C is returned from "A and B and C" if all are true.
>>
>> Ok, it seems everyone (except me :-)) agrees that it should return the
>> first key value, so that's how it will be.
>
> If you retain the first key value, it's easy enough for the
> application to implement "retain the last" semantics:
>
> try:
>      del d[k]
> finally:
>      d[k] = v
>
That would raise a KeyError is the key was missing. A better way is:

d.pop(k, None)
d[k] = v

> If you provide "retain the last", I can't see any obvious way of
> implementing "retain the first" in application code without in effect
> reimplementing the class.
>
"Retain the first" does feel more natural to me.

From lukas.lueg at gmail.com  Tue Sep 10 22:36:18 2013
From: lukas.lueg at gmail.com (Lukas Lueg)
Date: Tue, 10 Sep 2013 22:36:18 +0200
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <522F7A3C.1040701@mrabarnett.plus.com>
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
 <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>
 <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com>
 <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de>
 <20130910203140.1e8c1008@fsol>
 <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com>
 <522F7A3C.1040701@mrabarnett.plus.com>
Message-ID: <CAJF-kYn6aX4cx7z4ERGdXTnuZcj0tcSmt6F6jBgu_qG_ZLZxpw@mail.gmail.com>

Should'nt the key'ing behaviour be controlled by the type of the key
instead of the type of the container?


2013/9/10 MRAB <python at mrabarnett.plus.com>

> On 10/09/2013 20:08, Paul Moore wrote:
>
>> On 10 September 2013 19:31, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>
>>> I think it would be a flaw to have this detail implementation-defined.
>>>> This would be like saying that it is implementation-defined which
>>>> of A,B,C is returned from "A and B and C" if all are true.
>>>>
>>>
>>> Ok, it seems everyone (except me :-)) agrees that it should return the
>>> first key value, so that's how it will be.
>>>
>>
>> If you retain the first key value, it's easy enough for the
>> application to implement "retain the last" semantics:
>>
>> try:
>>      del d[k]
>> finally:
>>      d[k] = v
>>
>>  That would raise a KeyError is the key was missing. A better way is:
>
> d.pop(k, None)
>
> d[k] = v
>
>  If you provide "retain the last", I can't see any obvious way of
>> implementing "retain the first" in application code without in effect
>> reimplementing the class.
>>
>>  "Retain the first" does feel more natural to me.
>
> ______________________________**_________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/**mailman/listinfo/python-dev<https://mail.python.org/mailman/listinfo/python-dev>
> Unsubscribe: https://mail.python.org/**mailman/options/python-dev/**
> lukas.lueg%40gmail.com<https://mail.python.org/mailman/options/python-dev/lukas.lueg%40gmail.com>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/7810d8bf/attachment.html>

From jsbueno at python.org.br  Tue Sep 10 22:38:26 2013
From: jsbueno at python.org.br (Joao S. O. Bueno)
Date: Tue, 10 Sep 2013 17:38:26 -0300
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com>
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
 <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>
 <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com>
 <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de>
 <20130910203140.1e8c1008@fsol>
 <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com>
Message-ID: <CAH0mxTQBPy4ARJDXYNCCyGs5Oqjb0puL5gtL38FpjhZVcRy6LQ@mail.gmail.com>

On 10 September 2013 16:08, Paul Moore <p.f.moore at gmail.com> wrote:
> If you provide "retain the last", I can't see any obvious way of
> implementing "retain the first" in application code without in effect
> reimplementing the class.

Which reminds one - this class should obviously have a method for
retrivieng the original key value, given a matching key -

d.canonical('foo') -> 'Foo'

From solipsis at pitrou.net  Tue Sep 10 23:06:07 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 10 Sep 2013 23:06:07 +0200
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <CAH0mxTQBPy4ARJDXYNCCyGs5Oqjb0puL5gtL38FpjhZVcRy6LQ@mail.gmail.com>
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
 <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>
 <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com>
 <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de>
 <20130910203140.1e8c1008@fsol>
 <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com>
 <CAH0mxTQBPy4ARJDXYNCCyGs5Oqjb0puL5gtL38FpjhZVcRy6LQ@mail.gmail.com>
Message-ID: <20130910230607.701bba16@fsol>

On Tue, 10 Sep 2013 17:38:26 -0300
"Joao S. O. Bueno" <jsbueno at python.org.br> wrote:
> On 10 September 2013 16:08, Paul Moore <p.f.moore at gmail.com> wrote:
> > If you provide "retain the last", I can't see any obvious way of
> > implementing "retain the first" in application code without in effect
> > reimplementing the class.
> 
> Which reminds one - this class should obviously have a method for
> retrivieng the original key value, given a matching key -
> 
> d.canonical('foo') -> 'Foo'

I don't know. Is there any use case?
(sure, it is trivially implemented)

Regards

Antoine.

From p.f.moore at gmail.com  Tue Sep 10 23:12:51 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 10 Sep 2013 22:12:51 +0100
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <522F7A3C.1040701@mrabarnett.plus.com>
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
 <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>
 <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com>
 <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de>
 <20130910203140.1e8c1008@fsol>
 <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com>
 <522F7A3C.1040701@mrabarnett.plus.com>
Message-ID: <CACac1F9UgGGOMxfmV2E4H89UH8eYqvLOccDZ7t=9FRz+pOwayQ@mail.gmail.com>

On 10 September 2013 20:59, MRAB <python at mrabarnett.plus.com> wrote:
>> try:
>>      del d[k]
>> finally:
>>      d[k] = v
>>
> That would raise a KeyError is the key was missing. A better way is:
>
> d.pop(k, None)

Sorry, I was thinking of try...except: pass. But pop is indeed better.
Teach me to code stuff on the fly without testing :-)
Paul

From ethan at stoneleaf.us  Tue Sep 10 23:17:24 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 10 Sep 2013 14:17:24 -0700
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <CAJF-kYn6aX4cx7z4ERGdXTnuZcj0tcSmt6F6jBgu_qG_ZLZxpw@mail.gmail.com>
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
 <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>
 <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com>
 <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de>
 <20130910203140.1e8c1008@fsol>
 <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com>
 <522F7A3C.1040701@mrabarnett.plus.com>
 <CAJF-kYn6aX4cx7z4ERGdXTnuZcj0tcSmt6F6jBgu_qG_ZLZxpw@mail.gmail.com>
Message-ID: <522F8C64.1080005@stoneleaf.us>

On 09/10/2013 01:36 PM, Lukas Lueg wrote:
>
> Should'nt the key'ing behaviour be controlled by the type of the key instead of the type of the container?

That would be up to the key function.

--
~Ethan~

From jsbueno at python.org.br  Tue Sep 10 23:44:20 2013
From: jsbueno at python.org.br (Joao S. O. Bueno)
Date: Tue, 10 Sep 2013 18:44:20 -0300
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130910230607.701bba16@fsol>
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
 <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>
 <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com>
 <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de>
 <20130910203140.1e8c1008@fsol>
 <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com>
 <CAH0mxTQBPy4ARJDXYNCCyGs5Oqjb0puL5gtL38FpjhZVcRy6LQ@mail.gmail.com>
 <20130910230607.701bba16@fsol>
Message-ID: <CAH0mxTSTzEkhgXuuzZG-XMn+SV6rqEQNGpp9Xyf96zRBSBh4+A@mail.gmail.com>

On 10 September 2013 18:06, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Tue, 10 Sep 2013 17:38:26 -0300
> "Joao S. O. Bueno" <jsbueno at python.org.br> wrote:
>> On 10 September 2013 16:08, Paul Moore <p.f.moore at gmail.com> wrote:
>> > If you provide "retain the last", I can't see any obvious way of
>> > implementing "retain the first" in application code without in effect
>> > reimplementing the class.
>>
>> Which reminds one - this class should obviously have a method for
>> retrivieng the original key value, given a matching key -
>>
>> d.canonical('foo') -> 'Foo'
>
> I don't know. Is there any use case?
> (sure, it is trivially implemented)


Well, I'd expect it to simply be there. I had not thought of
other usecases for the transformdict itself -

but if I would use it and would need the original key
without such a method it would not be trivial to get it.

For example, in latim languages it is common to want
accented letters to match their unaccented counterparts
- pick my own first name "Jo?o" - if I'd use a transform to strip
the diactriticals, and have an user input "joao" - it would match,
as intended - but I would not be able to retrieve the accented version
without re-implementing the class behavior.

  js
 -><-



>
> Regards
>
> Antoine.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/jsbueno%40python.org.br

From solipsis at pitrou.net  Tue Sep 10 23:46:59 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 10 Sep 2013 23:46:59 +0200
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <CAH0mxTSTzEkhgXuuzZG-XMn+SV6rqEQNGpp9Xyf96zRBSBh4+A@mail.gmail.com>
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
 <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>
 <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com>
 <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de>
 <20130910203140.1e8c1008@fsol>
 <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com>
 <CAH0mxTQBPy4ARJDXYNCCyGs5Oqjb0puL5gtL38FpjhZVcRy6LQ@mail.gmail.com>
 <20130910230607.701bba16@fsol>
 <CAH0mxTSTzEkhgXuuzZG-XMn+SV6rqEQNGpp9Xyf96zRBSBh4+A@mail.gmail.com>
Message-ID: <20130910234659.39284dab@fsol>

On Tue, 10 Sep 2013 18:44:20 -0300
"Joao S. O. Bueno" <jsbueno at python.org.br> wrote:
> On 10 September 2013 18:06, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > On Tue, 10 Sep 2013 17:38:26 -0300
> > "Joao S. O. Bueno" <jsbueno at python.org.br> wrote:
> >> On 10 September 2013 16:08, Paul Moore <p.f.moore at gmail.com> wrote:
> >> > If you provide "retain the last", I can't see any obvious way of
> >> > implementing "retain the first" in application code without in effect
> >> > reimplementing the class.
> >>
> >> Which reminds one - this class should obviously have a method for
> >> retrivieng the original key value, given a matching key -
> >>
> >> d.canonical('foo') -> 'Foo'
> >
> > I don't know. Is there any use case?
> > (sure, it is trivially implemented)
> 
> 
> Well, I'd expect it to simply be there. I had not thought of
> other usecases for the transformdict itself -

Well, it is not here for dict, set, etc.

> For example, in latim languages it is common to want
> accented letters to match their unaccented counterparts
> - pick my own first name "Jo?o" - if I'd use a transform to strip
> the diactriticals, and have an user input "joao" - it would match,
> as intended - but I would not be able to retrieve the accented version
> without re-implementing the class behavior.

Interesting example, thanks.

Regards

Antoine.

From python at mrabarnett.plus.com  Wed Sep 11 00:12:56 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 10 Sep 2013 23:12:56 +0100
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130910234659.39284dab@fsol>
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
 <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>
 <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com>
 <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de>
 <20130910203140.1e8c1008@fsol>
 <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com>
 <CAH0mxTQBPy4ARJDXYNCCyGs5Oqjb0puL5gtL38FpjhZVcRy6LQ@mail.gmail.com>
 <20130910230607.701bba16@fsol>
 <CAH0mxTSTzEkhgXuuzZG-XMn+SV6rqEQNGpp9Xyf96zRBSBh4+A@mail.gmail.com>
 <20130910234659.39284dab@fsol>
Message-ID: <522F9968.5030108@mrabarnett.plus.com>

On 10/09/2013 22:46, Antoine Pitrou wrote:
> On Tue, 10 Sep 2013 18:44:20 -0300
> "Joao S. O. Bueno" <jsbueno at python.org.br> wrote:
>> On 10 September 2013 18:06, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> > On Tue, 10 Sep 2013 17:38:26 -0300
>> > "Joao S. O. Bueno" <jsbueno at python.org.br> wrote:
>> >> On 10 September 2013 16:08, Paul Moore <p.f.moore at gmail.com> wrote:
>> >> > If you provide "retain the last", I can't see any obvious way of
>> >> > implementing "retain the first" in application code without in effect
>> >> > reimplementing the class.
>> >>
>> >> Which reminds one - this class should obviously have a method for
>> >> retrivieng the original key value, given a matching key -
>> >>
>> >> d.canonical('foo') -> 'Foo'
>> >
>> > I don't know. Is there any use case?
>> > (sure, it is trivially implemented)
>>
>> Well, I'd expect it to simply be there. I had not thought of
>> other usecases for the transformdict itself -
>
I had the same thought.

> Well, it is not here for dict, set, etc.
>
In those cases the key in the dict == the key you're looking for.

>> For example, in latim languages it is common to want
>> accented letters to match their unaccented counterparts
>> - pick my own first name "Jo?o" - if I'd use a transform to strip
>> the diactriticals, and have an user input "joao" - it would match,
>> as intended - but I would not be able to retrieve the accented version
>> without re-implementing the class behavior.
>
> Interesting example, thanks.
>


From ethan at stoneleaf.us  Wed Sep 11 00:18:05 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 10 Sep 2013 15:18:05 -0700
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <522F9968.5030108@mrabarnett.plus.com>
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
 <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>
 <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com>
 <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de>
 <20130910203140.1e8c1008@fsol>
 <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com>
 <CAH0mxTQBPy4ARJDXYNCCyGs5Oqjb0puL5gtL38FpjhZVcRy6LQ@mail.gmail.com>
 <20130910230607.701bba16@fsol>
 <CAH0mxTSTzEkhgXuuzZG-XMn+SV6rqEQNGpp9Xyf96zRBSBh4+A@mail.gmail.com>
 <20130910234659.39284dab@fsol> <522F9968.5030108@mrabarnett.plus.com>
Message-ID: <522F9A9D.9010008@stoneleaf.us>

On 09/10/2013 03:12 PM, MRAB wrote:
> On 10/09/2013 22:46, Antoine Pitrou wrote:
>> On Tue, 10 Sep 2013 18:44:20 -0300
>> "Joao S. O. Bueno" <jsbueno at python.org.br> wrote:
>>> On 10 September 2013 18:06, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>> > On Tue, 10 Sep 2013 17:38:26 -0300
>>> > "Joao S. O. Bueno" <jsbueno at python.org.br> wrote:
>>> >> On 10 September 2013 16:08, Paul Moore <p.f.moore at gmail.com> wrote:
>>> >> > If you provide "retain the last", I can't see any obvious way of
>>> >> > implementing "retain the first" in application code without in effect
>>> >> > reimplementing the class.
>>> >>
>>> >> Which reminds one - this class should obviously have a method for
>>> >> retrivieng the original key value, given a matching key -
>>> >>
>>> >> d.canonical('foo') -> 'Foo'
>>> >
>>> > I don't know. Is there any use case?
>>> > (sure, it is trivially implemented)
>>>
>>> Well, I'd expect it to simply be there. I had not thought of
>>> other usecases for the transformdict itself -
>>
> I had the same thought.
>
>> Well, it is not here for dict, set, etc.
>>
> In those cases the key in the dict == the key you're looking for.

With the exception of numbers, of course (float vs int vs Decimal, etc.).

But if that distinction was useful for someone they could use this new TransformDict.  :)

--
~Ethan~

From eric at trueblade.com  Wed Sep 11 02:26:55 2013
From: eric at trueblade.com (Eric V. Smith)
Date: Tue, 10 Sep 2013 20:26:55 -0400
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <522F9A9D.9010008@stoneleaf.us>
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
 <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>
 <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com>
 <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de>
 <20130910203140.1e8c1008@fsol>
 <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com>
 <CAH0mxTQBPy4ARJDXYNCCyGs5Oqjb0puL5gtL38FpjhZVcRy6LQ@mail.gmail.com>
 <20130910230607.701bba16@fsol>
 <CAH0mxTSTzEkhgXuuzZG-XMn+SV6rqEQNGpp9Xyf96zRBSBh4+A@mail.gmail.com>
 <20130910234659.39284dab@fsol> <522F9968.5030108@mrabarnett.plus.com>
 <522F9A9D.9010008@stoneleaf.us>
Message-ID: <522FB8CF.5020400@trueblade.com>

On 9/10/2013 6:18 PM, Ethan Furman wrote:
> On 09/10/2013 03:12 PM, MRAB wrote:
>> On 10/09/2013 22:46, Antoine Pitrou wrote:
>>> On Tue, 10 Sep 2013 18:44:20 -0300
>>> "Joao S. O. Bueno" <jsbueno at python.org.br> wrote:
>>>> On 10 September 2013 18:06, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>>> > On Tue, 10 Sep 2013 17:38:26 -0300
>>>> > "Joao S. O. Bueno" <jsbueno at python.org.br> wrote:
>>>> >> On 10 September 2013 16:08, Paul Moore <p.f.moore at gmail.com> wrote:
>>>> >> > If you provide "retain the last", I can't see any obvious way of
>>>> >> > implementing "retain the first" in application code without in
>>>> effect
>>>> >> > reimplementing the class.
>>>> >>
>>>> >> Which reminds one - this class should obviously have a method for
>>>> >> retrivieng the original key value, given a matching key -
>>>> >>
>>>> >> d.canonical('foo') -> 'Foo'
>>>> >
>>>> > I don't know. Is there any use case?
>>>> > (sure, it is trivially implemented)
>>>>
>>>> Well, I'd expect it to simply be there. I had not thought of
>>>> other usecases for the transformdict itself -
>>>
>> I had the same thought.
>>
>>> Well, it is not here for dict, set, etc.
>>>
>> In those cases the key in the dict == the key you're looking for.
> 
> With the exception of numbers, of course (float vs int vs Decimal, etc.).

They'd still be ==, wouldn't they?

--
Eric.

From v+python at g.nevcal.com  Wed Sep 11 03:15:33 2013
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Tue, 10 Sep 2013 18:15:33 -0700
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130910234659.39284dab@fsol>
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
 <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>
 <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com>
 <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de>
 <20130910203140.1e8c1008@fsol>
 <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com>
 <CAH0mxTQBPy4ARJDXYNCCyGs5Oqjb0puL5gtL38FpjhZVcRy6LQ@mail.gmail.com>
 <20130910230607.701bba16@fsol>
 <CAH0mxTSTzEkhgXuuzZG-XMn+SV6rqEQNGpp9Xyf96zRBSBh4+A@mail.gmail.com>
 <20130910234659.39284dab@fsol>
Message-ID: <522FC435.20405@g.nevcal.com>

On 9/10/2013 2:46 PM, Antoine Pitrou wrote:
>>>> > >>Which reminds one - this class should obviously have a method for
>>>> > >>retrivieng the original key value, given a matching key -
>>>> > >>
>>>> > >>d.canonical('foo') -> 'Foo'
>>> > >
>>> > >I don't know. Is there any use case?
>>> > >(sure, it is trivially implemented)
>> >
>> >
>> >Well, I'd expect it to simply be there. I had not thought of
>> >other usecases for the transformdict itself -
> Well, it is not here for dict, set, etc.

But they don't change the keys (although numbers have different 
representations on occasion).

One use of transformdict might be to allow use of non-hashable items as 
keys, by extracting an actual key from the internals of the non-hashable 
item. The key may be sufficiently unique to enable use of the dict 
structure for lookups, but it would certainly be handy to obtain the 
actual item again. Without a canonical lookup feature, one would be 
forced to also include the key as part of the value, or some such hack.

I also thought Jo?o's example was a very practical reason to have the 
canonical lookup feature, by some name or another.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/5dde4a29/attachment.html>

From ethan at stoneleaf.us  Wed Sep 11 03:19:20 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 10 Sep 2013 18:19:20 -0700
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <522FB8CF.5020400@trueblade.com>
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
 <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>
 <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com>
 <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de>
 <20130910203140.1e8c1008@fsol>
 <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com>
 <CAH0mxTQBPy4ARJDXYNCCyGs5Oqjb0puL5gtL38FpjhZVcRy6LQ@mail.gmail.com>
 <20130910230607.701bba16@fsol>
 <CAH0mxTSTzEkhgXuuzZG-XMn+SV6rqEQNGpp9Xyf96zRBSBh4+A@mail.gmail.com>
 <20130910234659.39284dab@fsol> <522F9968.5030108@mrabarnett.plus.com>
 <522F9A9D.9010008@stoneleaf.us> <522FB8CF.5020400@trueblade.com>
Message-ID: <522FC518.6060604@stoneleaf.us>

On 09/10/2013 05:26 PM, Eric V. Smith wrote:
> On 9/10/2013 6:18 PM, Ethan Furman wrote:
>> On 09/10/2013 03:12 PM, MRAB wrote:
>>> On 10/09/2013 22:46, Antoine Pitrou wrote:
>>>> On Tue, 10 Sep 2013 18:44:20 -0300
>>>> "Joao S. O. Bueno" <jsbueno at python.org.br> wrote:
>>>>> On 10 September 2013 18:06, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>>>>> On Tue, 10 Sep 2013 17:38:26 -0300
>>>>>> "Joao S. O. Bueno" <jsbueno at python.org.br> wrote:
>>>>>>> On 10 September 2013 16:08, Paul Moore <p.f.moore at gmail.com> wrote:
>>>>>>>> If you provide "retain the last", I can't see any obvious way of
>>>>>>>> implementing "retain the first" in application code without in
>>>>> effect
>>>>>>>> reimplementing the class.
>>>>>>>
>>>>>>> Which reminds one - this class should obviously have a method for
>>>>>>> retrivieng the original key value, given a matching key -
>>>>>>>
>>>>>>> d.canonical('foo') -> 'Foo'
>>>>>>
>>>>>> I don't know. Is there any use case?
>>>>>> (sure, it is trivially implemented)
>>>>>
>>>>> Well, I'd expect it to simply be there. I had not thought of
>>>>> other usecases for the transformdict itself -
>>>>
>>> I had the same thought.
>>>
>>>> Well, it is not here for dict, set, etc.
>>>>
>>> In those cases the key in the dict == the key you're looking for.
>>
>> With the exception of numbers, of course (float vs int vs Decimal, etc.).
>
> They'd still be ==, wouldn't they?

Yes, but for presentation purposes not identical.

--
~Ethan~

From jsbueno at python.org.br  Wed Sep 11 04:25:03 2013
From: jsbueno at python.org.br (Joao S. O. Bueno)
Date: Tue, 10 Sep 2013 23:25:03 -0300
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130910234659.39284dab@fsol>
References: <20130910112832.75ed6c8b@pitrou.net>
 <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com>
 <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com>
 <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com>
 <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de>
 <20130910203140.1e8c1008@fsol>
 <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com>
 <CAH0mxTQBPy4ARJDXYNCCyGs5Oqjb0puL5gtL38FpjhZVcRy6LQ@mail.gmail.com>
 <20130910230607.701bba16@fsol>
 <CAH0mxTSTzEkhgXuuzZG-XMn+SV6rqEQNGpp9Xyf96zRBSBh4+A@mail.gmail.com>
 <20130910234659.39284dab@fsol>
Message-ID: <CAH0mxTQ6aF5HKdpKtSmfYNLiEJ3ZX-CTEfTRjcsVZVcWKPJUMg@mail.gmail.com>

On 10 September 2013 18:46, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Tue, 10 Sep 2013 18:44:20 -0300
> "Joao S. O. Bueno" <jsbueno at python.org.br> wrote:
>> On 10 September 2013 18:06, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> > On Tue, 10 Sep 2013 17:38:26 -0300
>> > "Joao S. O. Bueno" <jsbueno at python.org.br> wrote:
>> >> On 10 September 2013 16:08, Paul Moore <p.f.moore at gmail.com> wrote:
>> >> > If you provide "retain the last", I can't see any obvious way of
>> >> > implementing "retain the first" in application code without in effect
>> >> > reimplementing the class.
>> >>
>> >> Which reminds one - this class should obviously have a method for
>> >> retrivieng the original key value, given a matching key -
>> >>
>> >> d.canonical('foo') -> 'Foo'
>> >
>> > I don't know. Is there any use case?
>> > (sure, it is trivially implemented)
>>
>>
>> Well, I'd expect it to simply be there. I had not thought of
>> other usecases for the transformdict itself -
>
> Well, it is not here for dict, set, etc.

For the simple motive that once you retrieve or find that an
element is contained in one of these classes, you already
have the canonical key. :-)

>
>> For example, in latim languages it is common to want
>> accented letters to match their unaccented counterparts
>> - pick my own first name "Jo?o" - if I'd use a transform to strip
>> the diactriticals, and have an user input "joao" - it would match,
>> as intended - but I would not be able to retrieve the accented version
>> without re-implementing the class behavior.
>
> Interesting example, thanks.
>
> Regards
>
> Antoine.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/jsbueno%40python.org.br

From raymond.hettinger at gmail.com  Wed Sep 11 04:40:36 2013
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Tue, 10 Sep 2013 21:40:36 -0500
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130910112832.75ed6c8b@pitrou.net>
References: <20130910112832.75ed6c8b@pitrou.net>
Message-ID: <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>


On Sep 10, 2013, at 4:28 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> In http://bugs.python.org/issue18986 I proposed adding a new mapping
> type to the collections module.

I would *really* like for this to start outside the standard library.
It needs to mature with user feedback before being dumped
in the collections module (which was never intended to be a
giant pile of every collection a person could think of).  

Adding yet more dictionary variants is an example of
way-too-many-ways-to-do-it.   


Raymond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/edc1a2fa/attachment-0001.html>

From solipsis at pitrou.net  Wed Sep 11 10:27:32 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 11 Sep 2013 10:27:32 +0200
Subject: [Python-Dev] Add a "transformdict" to collections
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
Message-ID: <20130911102732.2cb268ca@pitrou.net>

Le Tue, 10 Sep 2013 21:40:36 -0500,
Raymond Hettinger <raymond.hettinger at gmail.com> a ?crit :
> 
> On Sep 10, 2013, at 4:28 AM, Antoine Pitrou <solipsis at pitrou.net>
> wrote:
> 
> > In http://bugs.python.org/issue18986 I proposed adding a new mapping
> > type to the collections module.
> 
> I would *really* like for this to start outside the standard library.

From a quick search:

- case-insensitive dicts (use cases and implementation attempts):
http://twistedmatrix.com/documents/current/api/twisted.python.util.InsensitiveDict.html
https://mail.python.org/pipermail/python-list/2013-May/647243.html
https://mail.python.org/pipermail/python-list/2005-April/296208.html
https://mail.python.org/pipermail/python-list/2004-June/241748.html
http://bugs.python.org/msg197376
http://stackoverflow.com/a/2082169
http://stackoverflow.com/a/3296782
http://code.activestate.com/recipes/66315-case-insensitive-dictionary/
https://gist.github.com/babakness/3901174
http://www.wikier.org/blog/key-insensitive-dictionary-in-python
http://en.sharejs.com/python/14534
http://www.voidspace.org.uk/python/archive.shtml#caseless

- identity dicts:
https://mail.python.org/pipermail/python-ideas/2010-May/007235.html
http://www.gossamer-threads.com/lists/python/python/209527
Python's own pickle module:
http://hg.python.org/cpython/file/0e70bf1f32a3/Lib/pickle.py#l234

> It needs to mature with user feedback before being dumped
> in the collections module (which was never intended to be a
> giant pile of every collection a person could think of).

Well, thanks for the reminder, I was *indeed* going to dump all the
collections I could think of in the collections module :-)
(that would have been embarassing!)

Seriously, I'm curious: what needs to mature, according to you? The
proposed collection is a plain MutableMapping with the single addition
of transforming the key on lookup. The use cases are well-defined and
well-known. If you have any concrete questions or concerns then please
offer them here.

> Adding yet more dictionary variants is an example of
> way-too-many-ways-to-do-it.

So what is your proposal for what is definitely (see examples above,
and this thread's and the tracker's responses) a very common need? Keep
letting people write suboptimal, incomplete, buggy versions of the same
thing?

Thanks

Antoine.



From storchaka at gmail.com  Wed Sep 11 11:38:13 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 11 Sep 2013 12:38:13 +0300
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130910112832.75ed6c8b@pitrou.net>
References: <20130910112832.75ed6c8b@pitrou.net>
Message-ID: <l0pdl9$sc2$1@ger.gmane.org>

There is a question about specifying the transform function.

There are three ways to do this:

1. Positional argument of the constructor.

d = TransformDict(str.casefold, Foo=5)

2. Subclassing.

class CaseInsensitiveDict(TransformDict):
     def transform(self, key):
         return key.casefold()
d = CaseInsensitiveDict(Foo=5)

3. Type generator.

d = TransformDict(str.casefold)(Foo=5)

Second method looks a little simpler to me from implementation side 
(What if you call __init__ for already initialized object? What if you 
used the object before calling __init__?).

Third method allows you to customize other aspects of dict behavior 
(combine OrderedDict, defaultdict,..).

First method looks less cumbersome from user side at first look. But how 
many different transform functions you use? Aren't they deserve named 
classes?



From solipsis at pitrou.net  Wed Sep 11 11:52:59 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 11 Sep 2013 11:52:59 +0200
Subject: [Python-Dev] Add a "transformdict" to collections
References: <20130910112832.75ed6c8b@pitrou.net> <l0pdl9$sc2$1@ger.gmane.org>
Message-ID: <20130911115259.57be1a82@pitrou.net>

Le Wed, 11 Sep 2013 12:38:13 +0300,
Serhiy Storchaka <storchaka at gmail.com> a ?crit :
> 2. Subclassing.
> 
> class CaseInsensitiveDict(TransformDict):
>      def transform(self, key):
>          return key.casefold()
> d = CaseInsensitiveDict(Foo=5)

I thought about this first, and then I remembered that python-dev isn't
generally very keen on subclassing-based APIs :-)

> 3. Type generator.
> 
> d = TransformDict(str.casefold)(Foo=5)
> 
[...]
> 
> Third method allows you to customize other aspects of dict behavior 
> (combine OrderedDict, defaultdict,..).

Well, no, it's not that easy. Especially since OrderedDict and
defaultdict weren't written with combination in mind.

Regards

Antoine.



From storchaka at gmail.com  Wed Sep 11 12:09:36 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 11 Sep 2013 13:09:36 +0300
Subject: [Python-Dev] Need testing audio files
Message-ID: <l0pfg4$i7l$1@ger.gmane.org>

I work on enhancement of audio modules testing [1], and I need free (in 
both senses) small sample audio files in different formats. We already 
have audiotest.au (mono, and sunau has a bug in processing multichannel 
files [2]) and Sine-1000Hz-300ms.aif, but this is not enough. I have 
generated a pack of files like Sine-1000Hz-300ms.aif by Python, it is 
enough for regression testing but only if current implementation is correct.

I found some collections of sample files at [3], but I'm not sure about 
copyright, and perhaps they are a little too big.

In ideal it should be one high-quality (float64?) multichannel (5+1?) 
but short master file and it's lower-quality copies made by third-party 
tools. In ideal the content should be related to Python.

[1] http://bugs.python.org/issue18919
[2] http://bugs.python.org/issue18950
[3] http://www-mmsp.ece.mcgill.ca/documents/AudioFormats/


From storchaka at gmail.com  Wed Sep 11 12:37:53 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 11 Sep 2013 13:37:53 +0300
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130911115259.57be1a82@pitrou.net>
References: <20130910112832.75ed6c8b@pitrou.net> <l0pdl9$sc2$1@ger.gmane.org>
 <20130911115259.57be1a82@pitrou.net>
Message-ID: <l0pi51$hkq$1@ger.gmane.org>

11.09.13 12:52, Antoine Pitrou ???????(??):
> Le Wed, 11 Sep 2013 12:38:13 +0300,
> Serhiy Storchaka <storchaka at gmail.com> a ?crit :
>> 2. Subclassing.
>>
>> class CaseInsensitiveDict(TransformDict):
>>       def transform(self, key):
>>           return key.casefold()
>> d = CaseInsensitiveDict(Foo=5)
>
> I thought about this first, and then I remembered that python-dev isn't
> generally very keen on subclassing-based APIs :-)

Why? This way looks most natural to me.

>> 3. Type generator.
>>
>> d = TransformDict(str.casefold)(Foo=5)
>>
> [...]
>>
>> Third method allows you to customize other aspects of dict behavior
>> (combine OrderedDict, defaultdict,..).
>
> Well, no, it's not that easy. Especially since OrderedDict and
> defaultdict weren't written with combination in mind.

They can be rewritten.

Actually the defaultdict is just simple wrapper around existing 
functionality of the __missing__ method. We can add the __transform__ 
method directly in the dict class. I think it will significant (2-3x) 
decrease a size of needed C code (no need in defining new type with 
constructors/destructors/etc, only lookup_transform).


From solipsis at pitrou.net  Wed Sep 11 13:07:15 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 11 Sep 2013 13:07:15 +0200
Subject: [Python-Dev] Add a "transformdict" to collections
References: <20130910112832.75ed6c8b@pitrou.net> <l0pdl9$sc2$1@ger.gmane.org>
 <20130911115259.57be1a82@pitrou.net> <l0pi51$hkq$1@ger.gmane.org>
Message-ID: <20130911130715.2f834a21@pitrou.net>

Le Wed, 11 Sep 2013 13:37:53 +0300,
Serhiy Storchaka <storchaka at gmail.com> a ?crit :
> 
> Actually the defaultdict is just simple wrapper around existing 
> functionality of the __missing__ method. We can add the __transform__ 
> method directly in the dict class. I think it will significant (2-3x) 
> decrease a size of needed C code (no need in defining new type with 
> constructors/destructors/etc, only lookup_transform).

Right now I am not proposing any C implementation of TransformDict.
That could be added, but it's not what I'm pushing for here :-)

But I don't think the "type generator" API would be easier to implement
in C, anyway.

Regards

Antoine.



From skip at pobox.com  Wed Sep 11 13:08:25 2013
From: skip at pobox.com (Skip Montanaro)
Date: Wed, 11 Sep 2013 06:08:25 -0500
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130911102732.2cb268ca@pitrou.net>
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
Message-ID: <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>

> Seriously, I'm curious: what needs to mature, according to you?

In my mind, its availability on PyPI along with demonstrated use in
the wild (plus corresponding votes to demonstrate that people use/like
it) would help.  That you can find several implementations at this
doesn't mean it's necessarily worth adding to the std lib. Once in, it
is very difficult to evict something that is later deemed not to have
belonged in the std lib, so I think some extra scrutiny is worthwhile.

Is there some obvious advantage to having a single API for this
available to all Python applications? Are the semantics well-defined
(do all the implementations you cited offer basically the same
semantics)? The discussion so far here suggest that the best semantics
might not be completely settled.

(I still don't care for the name. "Transform" != "case folding" in my
mind. A quick scan of your links suggests most people think something
like "cidict" or "CaseInsensitiveDict" would be more descriptive.)

Skip

From storchaka at gmail.com  Wed Sep 11 13:23:21 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 11 Sep 2013 14:23:21 +0300
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130911130715.2f834a21@pitrou.net>
References: <20130910112832.75ed6c8b@pitrou.net> <l0pdl9$sc2$1@ger.gmane.org>
 <20130911115259.57be1a82@pitrou.net> <l0pi51$hkq$1@ger.gmane.org>
 <20130911130715.2f834a21@pitrou.net>
Message-ID: <l0pjq9$423$1@ger.gmane.org>

11.09.13 14:07, Antoine Pitrou ???????(??):
> But I don't think the "type generator" API would be easier to implement
> in C, anyway.

No, I mean subclassing approach.



From rdmurray at bitdance.com  Wed Sep 11 13:57:06 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Wed, 11 Sep 2013 07:57:06 -0400
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
 <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
Message-ID: <20130911115706.C2D98250925@webabinitio.net>

On Wed, 11 Sep 2013 06:08:25 -0500, Skip Montanaro <skip at pobox.com> wrote:
> > Seriously, I'm curious: what needs to mature, according to you?
> 
> In my mind, its availability on PyPI along with demonstrated use in
> the wild (plus corresponding votes to demonstrate that people use/like
> it) would help.  That you can find several implementations at this
> doesn't mean it's necessarily worth adding to the std lib. Once in, it
> is very difficult to evict something that is later deemed not to have
> belonged in the std lib, so I think some extra scrutiny is worthwhile.

The problem is that it is hard to get traction on pypi for a "small"
feature like this.  Most people will just implement a special purpose
version that solves their specific need (possibly in a somewhat broken
fashion) rather than add yet another dependency.  Our approach in cases
like this has (I think) been to learn from the existing implementations
and build our own based on that (isn't that what happened with
OrderedDict?)

> Is there some obvious advantage to having a single API for this
> available to all Python applications? Are the semantics well-defined
> (do all the implementations you cited offer basically the same
> semantics)? The discussion so far here suggest that the best semantics
> might not be completely settled.

I think the only question was what happens to the key on assignment,
and that has been settled.  Well, I guess there's also a question of how
you create one, but that's not a question a pypi version would be any
help in resolving:  we'd still bikeshed it upon addition to the stdlib.
In fact, I would not be surprised if *all* of the bikeshedding we are
doing now (except this bit :) would take place if an existing pypi module
for this were proposed for addition.

I think the bigger question is composition of different dict types,
and that is again something that isn't going to be solved by a pypi
module.

> (I still don't care for the name. "Transform" != "case folding" in my
> mind. A quick scan of your links suggests most people think something
> like "cidict" or "CaseInsensitiveDict" would be more descriptive.)

Except it is wider than that: the transform function can be anything,
not just case folding.

I suggested surjectiondict or ontodict, but Antoine didn't like those :)
(I had to look up the terms...it's been a long time since I studied
math.)

--David

From solipsis at pitrou.net  Wed Sep 11 14:45:53 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 11 Sep 2013 14:45:53 +0200
Subject: [Python-Dev] Need testing audio files
References: <l0pfg4$i7l$1@ger.gmane.org>
Message-ID: <20130911144553.17909e76@pitrou.net>

Le Wed, 11 Sep 2013 13:09:36 +0300,
Serhiy Storchaka <storchaka at gmail.com> a ?crit :
> I work on enhancement of audio modules testing [1], and I need free
> (in both senses) small sample audio files in different formats. We
> already have audiotest.au (mono, and sunau has a bug in processing
> multichannel files [2]) and Sine-1000Hz-300ms.aif, but this is not
> enough. I have generated a pack of files like Sine-1000Hz-300ms.aif
> by Python, it is enough for regression testing but only if current
> implementation is correct.

If you want to edit, shorten, convert sounds between different formats,
you can try Audacity, a free sound editor:
http://audacity.sourceforge.net/

Regards

Antoine.



From victor.stinner at gmail.com  Wed Sep 11 14:46:06 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 11 Sep 2013 14:46:06 +0200
Subject: [Python-Dev] Need testing audio files
In-Reply-To: <l0pfg4$i7l$1@ger.gmane.org>
References: <l0pfg4$i7l$1@ger.gmane.org>
Message-ID: <CAMpsgwbpOLM8ghr91YYA+uVDGwqNjQdjoP70c0QNqBg4FEVxXQ@mail.gmail.com>

Use your microphone, say "python" and save the file in your favorite
file format. Try for example Audacity. I suppose that you don't need
specific audio content and you don't need a huge file.

Victor

2013/9/11 Serhiy Storchaka <storchaka at gmail.com>:
> I work on enhancement of audio modules testing [1], and I need free (in both
> senses) small sample audio files in different formats. We already have
> audiotest.au (mono, and sunau has a bug in processing multichannel files
> [2]) and Sine-1000Hz-300ms.aif, but this is not enough. I have generated a
> pack of files like Sine-1000Hz-300ms.aif by Python, it is enough for
> regression testing but only if current implementation is correct.
>
> I found some collections of sample files at [3], but I'm not sure about
> copyright, and perhaps they are a little too big.
>
> In ideal it should be one high-quality (float64?) multichannel (5+1?) but
> short master file and it's lower-quality copies made by third-party tools.
> In ideal the content should be related to Python.
>
> [1] http://bugs.python.org/issue18919
> [2] http://bugs.python.org/issue18950
> [3] http://www-mmsp.ece.mcgill.ca/documents/AudioFormats/
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com

From ncoghlan at gmail.com  Wed Sep 11 14:47:12 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 11 Sep 2013 22:47:12 +1000
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130911115706.C2D98250925@webabinitio.net>
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
 <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
 <20130911115706.C2D98250925@webabinitio.net>
Message-ID: <CADiSq7cjF6RGdrobJh1=f0etG3rVXARRUSCOJXzina0u1rkJTQ@mail.gmail.com>

On 11 September 2013 21:57, R. David Murray <rdmurray at bitdance.com> wrote:
> Except it is wider than that: the transform function can be anything,
> not just case folding.
>
> I suggested surjectiondict or ontodict, but Antoine didn't like those :)
> (I had to look up the terms...it's been a long time since I studied
> math.)

I'll join the chorus requesting that this live on PyPI for a while first.

I think this is a case similar to what happened with
contextlib.ExitStack: I'm not sure if anyone actually *used*
contextlib2 for anything significant (if they did, they didn't tell me
about it), but just going through the process of properly documenting,
publishing and testing it as a relatively independent project forced
*me* to think through the design. Add in a couple of interesting
conversation with Brandon Rhodes and others about the API design, and
the end result was a *vast* improvement over what originally went up
on PyPI.

The barriers to making use of PyPI libraries are also falling with
time, so a solid implementation may still see adoption, even if it's
in the form of copy-and-paste programming rather than actual
dependencies.

I think there are also additional API decisions to be made beyond just
the one about how to declare the mapping function, related to how to
get the mapped key values *out*, as well as how to find out whether or
not two potential key values map to the same actual key.

As my preferred bikeshed colour, I'm going to suggest "MappedKeyDict"
(using a similar naming style to OrderedDict), since the purpose of
the container is to map the domain of the supplied keys to a different
range before doing the value lookup.

Suggested additional methods:

    md.map_key(key)  # Applies the mapping function to the supplied key
    md.mapped_keys()  # Like keys(), but with the key mapping function applied
    md.mapped_items()  # Like items(), but with the key mapping function applied

Another (more dubious) possible method:

    md.same_key(key1, key2)  # "md.map_key(key1) == md.map_key(key2)"

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Wed Sep 11 15:04:23 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 11 Sep 2013 15:04:23 +0200
Subject: [Python-Dev] Add a "transformdict" to collections
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
 <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
 <20130911115706.C2D98250925@webabinitio.net>
 <CADiSq7cjF6RGdrobJh1=f0etG3rVXARRUSCOJXzina0u1rkJTQ@mail.gmail.com>
Message-ID: <20130911150423.62f77bf2@pitrou.net>

Le Wed, 11 Sep 2013 22:47:12 +1000,
Nick Coghlan <ncoghlan at gmail.com> a ?crit :
> 
> I'll join the chorus requesting that this live on PyPI for a while
> first.
> 
> I think this is a case similar to what happened with
> contextlib.ExitStack: I'm not sure if anyone actually *used*
> contextlib2 for anything significant (if they did, they didn't tell me
> about it), but just going through the process of properly documenting,
> publishing and testing it as a relatively independent project forced
> *me* to think through the design.

ExitStack was quite a new thing, API-wise. The proposal here is to
generalize something which already exists in various forms all over the
Internet, and respecting a well-known API, MutableMapping.

There are not many possible APIs to create case-insensitive dicts, or
identity dicts. The only API contention until now has been about
whether the first or last key would be retained (which settled on
the former), and Serhiy's three instantiation schemes.

The additional methods you suggest may be nice to have, but they are
not necessary and (deliberately) not part of the original proposal. That
is, they can be grafted later on.

> As my preferred bikeshed colour, I'm going to suggest "MappedKeyDict"
> (using a similar naming style to OrderedDict),

Ha, another colour! Thanks :-)

Regards

Antoine.



From steve at pearwood.info  Wed Sep 11 15:40:32 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 11 Sep 2013 23:40:32 +1000
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
 <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
Message-ID: <20130911134032.GA16820@ando>

On Wed, Sep 11, 2013 at 06:08:25AM -0500, Skip Montanaro wrote:

> (I still don't care for the name. "Transform" != "case folding" in my
> mind. A quick scan of your links suggests most people think something
> like "cidict" or "CaseInsensitiveDict" would be more descriptive.)

But the proposal is not for a case-insensitive dict. It is more general
than that, with case-insensitivity just one specific use-case for such
a transformative dict. Arguably the most natural, or at least obvious,
such transformation, but there are others.

I have code that does something like this:

MAPPING = {'spam': 23, 'ham': 42, 'eggs': 17}
result = MAPPING[key.strip()]
# later...
answer = MAPPING[key]  # Oops, forgot to strip! This is broken.

Using Antoine's proposal:

MAPPING = TransformDict(str.strip)
MAPPING.update({'spam': 23, 'ham': 42, 'eggs': 17})
result = MAPPING[key]
# later...
answer = MAPPING[key]  # Fine now.

so that the mapping handles stripping the keys, not the caller.

This isn't just about strings, and certainly not just 
case-insensitivity.



-- 
Steven

From stephen at xemacs.org  Wed Sep 11 15:50:08 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 11 Sep 2013 22:50:08 +0900
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130911150423.62f77bf2@pitrou.net>
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
 <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
 <20130911115706.C2D98250925@webabinitio.net>
 <CADiSq7cjF6RGdrobJh1=f0etG3rVXARRUSCOJXzina0u1rkJTQ@mail.gmail.com>
 <20130911150423.62f77bf2@pitrou.net>
Message-ID: <87txhr79in.fsf@uwakimon.sk.tsukuba.ac.jp>

Antoine Pitrou writes:

 > ExitStack was quite a new thing, API-wise. The proposal here is to
 > generalize something which already exists in various forms all over the
 > Internet, and respecting a well-known API, MutableMapping.

What Nick said was "I was too close to the design, and time and a very
few good comments greatly improved the product."  That's worth
considering as generic advice rather than for the specifics of the
case he faced compared to yours.

Which modules in the stdlib would benefit from rewriting using
"transformdict"?  How about on PyPI?

From brett at python.org  Wed Sep 11 15:54:44 2013
From: brett at python.org (Brett Cannon)
Date: Wed, 11 Sep 2013 09:54:44 -0400
Subject: [Python-Dev] Need testing audio files
In-Reply-To: <CAMpsgwbpOLM8ghr91YYA+uVDGwqNjQdjoP70c0QNqBg4FEVxXQ@mail.gmail.com>
References: <l0pfg4$i7l$1@ger.gmane.org>
 <CAMpsgwbpOLM8ghr91YYA+uVDGwqNjQdjoP70c0QNqBg4FEVxXQ@mail.gmail.com>
Message-ID: <CAP1=2W4a=Mzt=DVj4qeBZG8gxQ8iaJ7RUPhRb6-YvfTC2TbYfA@mail.gmail.com>

On Wed, Sep 11, 2013 at 8:46 AM, Victor Stinner <victor.stinner at gmail.com>wrote:

> Use your microphone, say "python" and save the file in your favorite
> file format. Try for example Audacity. I suppose that you don't need
> specific audio content and you don't need a huge file.
>

And if you don't want your voice in the test suite forever you can see if
you can convince Guido to say "Python" into a microphone.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130911/236cf5f2/attachment-0001.html>

From steve at pearwood.info  Wed Sep 11 15:56:01 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 11 Sep 2013 23:56:01 +1000
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <CADiSq7cjF6RGdrobJh1=f0etG3rVXARRUSCOJXzina0u1rkJTQ@mail.gmail.com>
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
 <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
 <20130911115706.C2D98250925@webabinitio.net>
 <CADiSq7cjF6RGdrobJh1=f0etG3rVXARRUSCOJXzina0u1rkJTQ@mail.gmail.com>
Message-ID: <20130911135601.GB16820@ando>

On Wed, Sep 11, 2013 at 10:47:12PM +1000, Nick Coghlan wrote:

> I'll join the chorus requesting that this live on PyPI for a while first.

Another alternative would be ActiveState's Python recipes:

http://code.activestate.com/recipes/langs/python

which is a lot lighter weight than creating a package on PyPI.

I believe that ChainMap, namedtuple, groupby, lru_cache and even 
math.fsum started life on ActiveState.


-- 
Steven

From victor.stinner at gmail.com  Wed Sep 11 15:58:35 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 11 Sep 2013 15:58:35 +0200
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130911134032.GA16820@ando>
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
 <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
 <20130911134032.GA16820@ando>
Message-ID: <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com>

2013/9/11 Steven D'Aprano <steve at pearwood.info>:
> But the proposal is not for a case-insensitive dict. It is more general
> than that, with case-insensitivity just one specific use-case for such
> a transformative dict. Arguably the most natural, or at least obvious,
> such transformation, but there are others.
>
> I have code that does something like this:
>
> MAPPING = {'spam': 23, 'ham': 42, 'eggs': 17}
> result = MAPPING[key.strip()]
> # later...
> answer = MAPPING[key]  # Oops, forgot to strip! This is broken.

For this use case, you should not keep the key unchanged, but it's
better to store the stripped key (so MAPPING.keys() gives you the
expected result). The transformdict type proposed by Antoine cannot be
used for this use case.

The os.environ mapping uses a subprocess of MutableMapping which
accepts 4 functions: encoder/decoder for the key and encoder/decoder
for the value. Such type is even more generic. transformdict cannot
replace os._Environ.

(Or do you prefer to store the ' eggs' key?)

Victor

From barry at python.org  Wed Sep 11 16:10:53 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 11 Sep 2013 10:10:53 -0400
Subject: [Python-Dev] Need testing audio files
In-Reply-To: <l0pfg4$i7l$1@ger.gmane.org>
References: <l0pfg4$i7l$1@ger.gmane.org>
Message-ID: <20130911101053.71deed3b@anarchist>

On Sep 11, 2013, at 01:09 PM, Serhiy Storchaka wrote:

>In ideal it should be one high-quality (float64?) multichannel (5+1?) but
>short master file and it's lower-quality copies made by third-party tools. In
>ideal the content should be related to Python.

I have some pro-audio recording capabilities and would be happy to generate
some copyright-donated clips for Python.  Please contact me off-list if
needed.

-Barry

From phd at phdru.name  Wed Sep 11 16:10:22 2013
From: phd at phdru.name (Oleg Broytman)
Date: Wed, 11 Sep 2013 18:10:22 +0400
Subject: [Python-Dev] Need testing audio files
In-Reply-To: <CAP1=2W4a=Mzt=DVj4qeBZG8gxQ8iaJ7RUPhRb6-YvfTC2TbYfA@mail.gmail.com>
References: <l0pfg4$i7l$1@ger.gmane.org>
 <CAMpsgwbpOLM8ghr91YYA+uVDGwqNjQdjoP70c0QNqBg4FEVxXQ@mail.gmail.com>
 <CAP1=2W4a=Mzt=DVj4qeBZG8gxQ8iaJ7RUPhRb6-YvfTC2TbYfA@mail.gmail.com>
Message-ID: <20130911141022.GA30682@iskra.aviel.ru>

On Wed, Sep 11, 2013 at 09:54:44AM -0400, Brett Cannon <brett at python.org> wrote:
> On Wed, Sep 11, 2013 at 8:46 AM, Victor Stinner <victor.stinner at gmail.com>wrote:
> And if you don't want your voice in the test suite forever you can see if
> you can convince Guido to say "Python" into a microphone.

   Wouldn't his name be enough? http://www.python.org/~guido/guido.au

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From ethan at stoneleaf.us  Wed Sep 11 16:04:17 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 11 Sep 2013 07:04:17 -0700
Subject: [Python-Dev] importance of dir
Message-ID: <52307861.6010102@stoneleaf.us>

http://docs.python.org/3/library/functions.html#dir:
>
> Note:  Because dir() is supplied primarily as a convenience for
>  use at an interactive prompt [...]

I suspect this comment is out of date, as there are two functions in the inspect module that rely on dir(), which also 
means that help indirectly relies on dir().  And we also now have the __dir__ method that we can use to customize what 
gets returned.

> [...] more than it tries to supply a rigorously or consistently
> defined set of names, [...]

If this is still true, should inspect be relying on dir()?

--
~Ethan~

From solipsis at pitrou.net  Wed Sep 11 16:32:23 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 11 Sep 2013 16:32:23 +0200
Subject: [Python-Dev] Add a "transformdict" to collections
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
 <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
 <20130911115706.C2D98250925@webabinitio.net>
 <CADiSq7cjF6RGdrobJh1=f0etG3rVXARRUSCOJXzina0u1rkJTQ@mail.gmail.com>
 <20130911150423.62f77bf2@pitrou.net>
 <87txhr79in.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <20130911163223.1334c00d@pitrou.net>

Le Wed, 11 Sep 2013 22:50:08 +0900,
"Stephen J. Turnbull" <stephen at xemacs.org> a ?crit :
> Antoine Pitrou writes:
> 
>  > ExitStack was quite a new thing, API-wise. The proposal here is to
>  > generalize something which already exists in various forms all
>  > over the Internet, and respecting a well-known API, MutableMapping.
> 
> What Nick said was "I was too close to the design, and time and a very
> few good comments greatly improved the product."  That's worth
> considering as generic advice rather than for the specifics of the
> case he faced compared to yours.

"Time and comments" is why I'm posting a discussion thread here and
waiting for responses. There are currently 34517 packages on PyPI. A new
34518th one won't magically get a lot of attention :-)

(for example, I got very few comments when posting pathlib on PyPI -
despite making several actual releases there -, compared to what I got
from the PEP 428 discussion on python-ideas)

Regards

Antoine.



From ethan at stoneleaf.us  Wed Sep 11 16:34:32 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 11 Sep 2013 07:34:32 -0700
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <CADiSq7cjF6RGdrobJh1=f0etG3rVXARRUSCOJXzina0u1rkJTQ@mail.gmail.com>
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
 <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
 <20130911115706.C2D98250925@webabinitio.net>
 <CADiSq7cjF6RGdrobJh1=f0etG3rVXARRUSCOJXzina0u1rkJTQ@mail.gmail.com>
Message-ID: <52307F78.7050303@stoneleaf.us>

On 09/11/2013 05:47 AM, Nick Coghlan wrote:
> On 11 September 2013 21:57, R. David Murray <rdmurray at bitdance.com> wrote:
>> Except it is wider than that: the transform function can be anything,
>> not just case folding.
>>
>> I suggested surjectiondict or ontodict, but Antoine didn't like those :)
>> (I had to look up the terms...it's been a long time since I studied
>> math.)
>
> I'll join the chorus requesting that this live on PyPI for a while first.

Personally, I would not add a PyPI dependency for a single object like transformdict.  If we are that nervous about it 
we can make it provisional, but I don't see that as necessary.


> I think this is a case similar to what happened with
> contextlib.ExitStack: I'm not sure if anyone actually *used*
> contextlib2 for anything significant (if they did, they didn't tell me
> about it), but just going through the process of properly documenting,
> publishing and testing it as a relatively independent project forced
> *me* to think through the design.

I had the opposite experience.  Going through PyDev to get Enum was *far* more educational, informative, and useful than 
creating an independent package.  Plus, transformdict is not that new or different from existing dicts, and most 
decisions (which key to keep?  how to initialize?) can be answered by simply being consistent with what we already have.

--
~Ethan~

From ethan at stoneleaf.us  Wed Sep 11 16:17:29 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 11 Sep 2013 07:17:29 -0700
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <l0pdl9$sc2$1@ger.gmane.org>
References: <20130910112832.75ed6c8b@pitrou.net> <l0pdl9$sc2$1@ger.gmane.org>
Message-ID: <52307B79.1080002@stoneleaf.us>

On 09/11/2013 02:38 AM, Serhiy Storchaka wrote:
> There is a question about specifying the transform function.
>
> There are three ways to do this:
>
> 1. Positional argument of the constructor.
>
> d = TransformDict(str.casefold, Foo=5)

This method follows the precedent of defaultdict:

--> from collections import defaultdict
--> l = defaultdict(list, foo=['ham','eggs'])
--> l
defaultdict(<class 'list'>, {'foo': ['ham', 'eggs']})

Without a compelling reason to change, we should keep it consistent.

--
~Ethan~

From ethan at stoneleaf.us  Wed Sep 11 16:27:44 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 11 Sep 2013 07:27:44 -0700
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com>
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
 <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
 <20130911134032.GA16820@ando>
 <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com>
Message-ID: <52307DE0.9040208@stoneleaf.us>

On 09/11/2013 06:58 AM, Victor Stinner wrote:
> 2013/9/11 Steven D'Aprano <steve at pearwood.info>:
>> But the proposal is not for a case-insensitive dict. It is more general
>> than that, with case-insensitivity just one specific use-case for such
>> a transformative dict. Arguably the most natural, or at least obvious,
>> such transformation, but there are others.
>>
>> I have code that does something like this:
>>
>> MAPPING = {'spam': 23, 'ham': 42, 'eggs': 17}
>> result = MAPPING[key.strip()]
>> # later...
>> answer = MAPPING[key]  # Oops, forgot to strip! This is broken.
>
> For this use case, you should not keep the key unchanged, but it's
> better to store the stripped key (so MAPPING.keys() gives you the
> expected result).

He isn't keeping the key unchanged (notice no white space in MAPPING), he's merely providing a function that will 
automatically strip the whitespace from key lookups.

> The transformdict type proposed by Antoine cannot be
> used for this use case.

Yes, it can.

--> from collections import transformdict
--> MAPPING = transformdict(str.strip)
--> MAPPING.update({'spam': 23, 'ham': 42, 'eggs': 17})
--> MAPPING
transformdict(<method 'strip' of 'str' objects>, {'ham': 42, 'spam': 23, 'eggs': 17})
--> MAPPING[' eggs ']
17

--
~Ethan~

From skip at pobox.com  Wed Sep 11 17:17:37 2013
From: skip at pobox.com (Skip Montanaro)
Date: Wed, 11 Sep 2013 10:17:37 -0500
Subject: [Python-Dev] importance of dir
In-Reply-To: <52307861.6010102@stoneleaf.us>
References: <52307861.6010102@stoneleaf.us>
Message-ID: <CANc-5UzZbCjQwY=LUK+3a_4U_aLO4xfbXOxWJUYSt+yifzyVMw@mail.gmail.com>

>> Note:  Because dir() is supplied primarily as a convenience for
>>  use at an interactive prompt [...]

This was always my interpretation of its intent. In fact, I use a
customized dir() for my own needs which would probably break inspect
(elides _-prefixed functions by default, notes modules or packages
within other packages which haven't been imported yet). I never
realized that module used it.

Skip

From ethan at stoneleaf.us  Wed Sep 11 16:48:56 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 11 Sep 2013 07:48:56 -0700
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com>
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
 <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
 <20130911134032.GA16820@ando>
 <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com>
Message-ID: <523082D8.2040607@stoneleaf.us>

On 09/11/2013 06:58 AM, Victor Stinner wrote:
>
> The os.environ mapping uses a subclass of MutableMapping which
> accepts 4 functions: encoder/decoder for the key and encoder/decoder
> for the value. Such type is even more generic. transformdict cannot
> replace os._Environ.

True, it's more generic -- but is it used more often?  Or often enough to have those four functions be part of 
transformdict instead of just the one encodekey function?  (I mean the idea of os._Environ, not that specific 
implementation.)

Personally, I wouldn't mind having all four; for one thing, the name 'transformdict' would then be entirely appropriate.  ;)

--
~Ethan~

From solipsis at pitrou.net  Wed Sep 11 17:48:09 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 11 Sep 2013 17:48:09 +0200
Subject: [Python-Dev] Add a "transformdict" to collections
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
 <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
 <20130911134032.GA16820@ando>
 <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com>
 <523082D8.2040607@stoneleaf.us>
Message-ID: <20130911174809.2c12a7d0@pitrou.net>

Le Wed, 11 Sep 2013 07:48:56 -0700,
Ethan Furman <ethan at stoneleaf.us> a ?crit :
> On 09/11/2013 06:58 AM, Victor Stinner wrote:
> >
> > The os.environ mapping uses a subclass of MutableMapping which
> > accepts 4 functions: encoder/decoder for the key and encoder/decoder
> > for the value. Such type is even more generic. transformdict cannot
> > replace os._Environ.
> 
> True, it's more generic -- but is it used more often?  Or often
> enough to have those four functions be part of transformdict instead
> of just the one encodekey function?  (I mean the idea of os._Environ,
> not that specific implementation.)
> 
> Personally, I wouldn't mind having all four; for one thing, the name
> 'transformdict' would then be entirely appropriate.  ;)

The key decoder function is quite useless since the original key is
retained. As for the value encoder/decoder, I don't really see the
point: just store whichever value you want to retrieve later. The main
point of transformdict is that the *lookup* is done using a different
key than the user-provided key.

Regards

Antoine.



From victor.stinner at gmail.com  Wed Sep 11 17:49:25 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 11 Sep 2013 17:49:25 +0200
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <52307DE0.9040208@stoneleaf.us>
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
 <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
 <20130911134032.GA16820@ando>
 <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com>
 <52307DE0.9040208@stoneleaf.us>
Message-ID: <CAMpsgwb1iB7fMKjYX5T3HR0E+3rmt4H055z2cjyJ-fMEjoEe6Q@mail.gmail.com>

2013/9/11 Ethan Furman <ethan at stoneleaf.us>:
> He isn't keeping the key unchanged (notice no white space in MAPPING), he's
> merely providing a function that will automatically strip the whitespace
> from key lookups.

transformdict keeps the key unchanged, see the first message:

   >>> d = transformdict(str.lower)
   >>> d['Foo'] = 5
   >>> d['foo']
   5
   >>> d['FOO']
   5
   >>> list(d)
   ['Foo']

'Foo' is stored as 'Foo', not as 'foo'. So for stripped keys:

d=transformdict(str.strip); d['   abc   ']; print(list(d))

should print "['   abc   ']", not "['abc']".

Is it the expected result?

Victor

From ethan at stoneleaf.us  Wed Sep 11 18:03:18 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 11 Sep 2013 09:03:18 -0700
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <CAMpsgwb1iB7fMKjYX5T3HR0E+3rmt4H055z2cjyJ-fMEjoEe6Q@mail.gmail.com>
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
 <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
 <20130911134032.GA16820@ando>
 <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com>
 <52307DE0.9040208@stoneleaf.us>
 <CAMpsgwb1iB7fMKjYX5T3HR0E+3rmt4H055z2cjyJ-fMEjoEe6Q@mail.gmail.com>
Message-ID: <52309446.6070809@stoneleaf.us>

On 09/11/2013 08:49 AM, Victor Stinner wrote:
> 2013/9/11 Ethan Furman <ethan at stoneleaf.us>:
>> He isn't keeping the key unchanged (notice no white space in MAPPING), he's
>> merely providing a function that will automatically strip the whitespace
>> from key lookups.
>
> transformdict keeps the key unchanged, see the first message:
>
>     >>> d = transformdict(str.lower)
>     >>> d['Foo'] = 5
>     >>> d['foo']
>     5
>     >>> d['FOO']
>     5
>     >>> list(d)
>     ['Foo']
>
> 'Foo' is stored as 'Foo', not as 'foo'. So for stripped keys:
>
> d=transformdict(str.strip); d['   abc   ']; print(list(d))
>
> should print "['   abc   ']", not "['abc']".

And indeed it does:

Python 3.4.0a1+ (default:833246d42825+, Aug 31 2013, 14:17:59)
[GCC 4.7.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
--> from collections import transformdict
--> d=transformdict(str.strip); d['   abc   '] = 42; print(list(d))
['   abc   ']

> Is it the expected result?

Yup!  :)

--
~Ethan~

From ethan at stoneleaf.us  Wed Sep 11 18:04:25 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 11 Sep 2013 09:04:25 -0700
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130911174809.2c12a7d0@pitrou.net>
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
 <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
 <20130911134032.GA16820@ando>
 <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com>
 <523082D8.2040607@stoneleaf.us> <20130911174809.2c12a7d0@pitrou.net>
Message-ID: <52309489.6000008@stoneleaf.us>

On 09/11/2013 08:48 AM, Antoine Pitrou wrote:
> Le Wed, 11 Sep 2013 07:48:56 -0700,
> Ethan Furman <ethan at stoneleaf.us> a ?crit :
>>
>> Personally, I wouldn't mind having all four; for one thing, the name
>> 'transformdict' would then be entirely appropriate.  ;)
>
> The key decoder function is quite useless since the original key is
> retained. As for the value encoder/decoder, I don't really see the
> point: just store whichever value you want to retrieve later. The main
> point of transformdict is that the *lookup* is done using a different
> key than the user-provided key.

Very good points.  I am convinced.

--
~Ethan~

From storchaka at gmail.com  Wed Sep 11 17:37:24 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 11 Sep 2013 18:37:24 +0300
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <87txhr79in.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
 <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
 <20130911115706.C2D98250925@webabinitio.net>
 <CADiSq7cjF6RGdrobJh1=f0etG3rVXARRUSCOJXzina0u1rkJTQ@mail.gmail.com>
 <20130911150423.62f77bf2@pitrou.net>
 <87txhr79in.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <l0q655$ac0$1@ger.gmane.org>

11.09.13 16:50, Stephen J. Turnbull ???????(??):
> Which modules in the stdlib would benefit from rewriting using
> "transformdict"?  How about on PyPI?

At least _threading_local, cProfile, doctest, json, and perhaps future 
implementations of __sizeof__ for some classes would benefit from 
rewriting using IdentityDict. Unfortunately copy and pickle expose their 
replacements of IdentityDict to public and can't change them.

I don't known anything about general TransformDict use cases.


From storchaka at gmail.com  Wed Sep 11 18:34:35 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 11 Sep 2013 19:34:35 +0300
Subject: [Python-Dev] Need testing audio files
In-Reply-To: <20130911101053.71deed3b@anarchist>
References: <l0pfg4$i7l$1@ger.gmane.org> <20130911101053.71deed3b@anarchist>
Message-ID: <l0q667$ac0$5@ger.gmane.org>

11.09.13 17:10, Barry Warsaw ???????(??):
> I have some pro-audio recording capabilities and would be happy to generate
> some copyright-donated clips for Python.  Please contact me off-list if
> needed.

Thank you.



From storchaka at gmail.com  Wed Sep 11 18:00:43 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 11 Sep 2013 19:00:43 +0300
Subject: [Python-Dev] Need testing audio files
In-Reply-To: <20130911144553.17909e76@pitrou.net>
References: <l0pfg4$i7l$1@ger.gmane.org> <20130911144553.17909e76@pitrou.net>
Message-ID: <l0q65f$ac0$4@ger.gmane.org>

11.09.13 15:45, Antoine Pitrou ???????(??):
> If you want to edit, shorten, convert sounds between different formats,
> you can try Audacity, a free sound editor:
> http://audacity.sourceforge.net/

Yes, Audacity is great.



From storchaka at gmail.com  Wed Sep 11 17:45:25 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 11 Sep 2013 18:45:25 +0300
Subject: [Python-Dev] Need testing audio files
In-Reply-To: <CAMpsgwbpOLM8ghr91YYA+uVDGwqNjQdjoP70c0QNqBg4FEVxXQ@mail.gmail.com>
References: <l0pfg4$i7l$1@ger.gmane.org>
 <CAMpsgwbpOLM8ghr91YYA+uVDGwqNjQdjoP70c0QNqBg4FEVxXQ@mail.gmail.com>
Message-ID: <l0q658$ac0$2@ger.gmane.org>

11.09.13 15:46, Victor Stinner ???????(??):
> Use your microphone, say "python" and save the file in your favorite
> file format. Try for example Audacity. I suppose that you don't need
> specific audio content and you don't need a huge file.

My voice is even more ugly than my English. I don't want perpetuate it 
in Python test suite.



From storchaka at gmail.com  Wed Sep 11 17:47:18 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 11 Sep 2013 18:47:18 +0300
Subject: [Python-Dev] Need testing audio files
In-Reply-To: <20130911141022.GA30682@iskra.aviel.ru>
References: <l0pfg4$i7l$1@ger.gmane.org>
 <CAMpsgwbpOLM8ghr91YYA+uVDGwqNjQdjoP70c0QNqBg4FEVxXQ@mail.gmail.com>
 <CAP1=2W4a=Mzt=DVj4qeBZG8gxQ8iaJ7RUPhRb6-YvfTC2TbYfA@mail.gmail.com>
 <20130911141022.GA30682@iskra.aviel.ru>
Message-ID: <l0q65b$ac0$3@ger.gmane.org>

11.09.13 17:10, Oleg Broytman ???????(??):
>     Wouldn't his name be enough? http://www.python.org/~guido/guido.au

It is Lib/test/audiotest.au. 1-channel and 8-bit. No, it wouldn't enough.



From martin at v.loewis.de  Wed Sep 11 19:31:56 2013
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Wed, 11 Sep 2013 19:31:56 +0200
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130911150423.62f77bf2@pitrou.net>
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
 <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
 <20130911115706.C2D98250925@webabinitio.net>
 <CADiSq7cjF6RGdrobJh1=f0etG3rVXARRUSCOJXzina0u1rkJTQ@mail.gmail.com>
 <20130911150423.62f77bf2@pitrou.net>
Message-ID: <5230A90C.3090102@v.loewis.de>

Am 11.09.13 15:04, schrieb Antoine Pitrou:
> There are not many possible APIs to create case-insensitive dicts, or
> identity dicts. 

That is certainly not true. Most obviously, you have the choice of a
specialized case-mapping dict, or a generalized type that can be used
for case mapping also. Does any of the referenced use cases actually
use the API that you are proposing (i.e. with a key transformation
function)?

FWIW, I can think of yet another API for this: write a Proxy class

class TransformedKey: # untested
  def __init__(self, original):
    self.original = original
    self.transformed = self.transform(original)
  def __hash__(self):
    return hash(self.transformed)
  def __eq__(self, other):
    return self.transformed == other.transformed
  def transform(self, key):
     raise NotImplementedError

If there is then disciplined use in the form

  d[TransformedKey(key)] == value

then you can use the existing dictionary type. Notice that this can do
both case-insensitive and identity dicts (plus there are multiple
choices of getting the transform function into it, as well as
variations of the __eq__ implementation).

There is evidence in this thread that people "grasp" case-insensitive
more easily than the generalized API. For the record, I initially typed
two responses into the tracker which I found to be incorrect before
posting them, so I ended up posting neither. The "transformdict" type
is not at all "natural", even though it may be useful.

If you really want to "push" this API into 3.4, I think you will need
to write a PEP, and find a PEP dictator who is willing to approve it.
As you seem to dislike the idea of writing a PEP, I suggest to follow
the idea of publishing it on PyPI now, and then proposing it for
inclusion into 3.5.

Regards,
Martin


From solipsis at pitrou.net  Wed Sep 11 19:49:35 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 11 Sep 2013 19:49:35 +0200
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <5230A90C.3090102@v.loewis.de>
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
 <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
 <20130911115706.C2D98250925@webabinitio.net>
 <CADiSq7cjF6RGdrobJh1=f0etG3rVXARRUSCOJXzina0u1rkJTQ@mail.gmail.com>
 <20130911150423.62f77bf2@pitrou.net> <5230A90C.3090102@v.loewis.de>
Message-ID: <20130911194935.1b116266@fsol>

On Wed, 11 Sep 2013 19:31:56 +0200
"Martin v. L?wis" <martin at v.loewis.de> wrote:
> Am 11.09.13 15:04, schrieb Antoine Pitrou:
> > There are not many possible APIs to create case-insensitive dicts, or
> > identity dicts. 
> 
> That is certainly not true. Most obviously, you have the choice of a
> specialized case-mapping dict, or a generalized type that can be used
> for case mapping also. Does any of the referenced use cases actually
> use the API that you are proposing (i.e. with a key transformation
> function)?

Well, when you have a specialized need, you don't implement a generic
version (*), so I don't think anybody implemented it ;-). The point of a
generic API is to help replace all specialized implementations at once,
rather than a small subset of them.

(*) even the Twisted people don't seem to go that far

> FWIW, I can think of yet another API for this: write a Proxy class
> 
> class TransformedKey: # untested
>   def __init__(self, original):
>     self.original = original
>     self.transformed = self.transform(original)
>   def __hash__(self):
>     return hash(self.transformed)
>   def __eq__(self, other):
>     return self.transformed == other.transformed
>   def transform(self, key):
>      raise NotImplementedError
> 
> If there is then disciplined use in the form
> 
>   d[TransformedKey(key)] == value

The problem is "disciplined use" here. This means any consumer of your
API has to know about that quirk, so it's an inferior solution.

(TransformedKey could be an internal detail of the
implementation, as with weak dictionaries; but as a public API it
doesn't seem very desirable)

> The "transformdict" type
> is not at all "natural", even though it may be useful.

By that token, nothing is "natural" ;-) defaultdict isn't natural, yet
it went in without a PEP.

However, to all persons who already had that need and responded to the
proposal, the generic version *does* seem natural enough, AFAICT. Only
people who've never had such a need seem to fail to "grasp" it (and I
only counted one such person, but my memory may be failing me).

You know, I'm not against having "only" a case-insensitive dict. But
that would be less useful all the while not being significantly easier
to use, so I'm not really seeing the point.

> If you really want to "push" this API into 3.4, I think you will need
> to write a PEP, and find a PEP dictator who is willing to approve it.
> As you seem to dislike the idea of writing a PEP, I suggest to follow
> the idea of publishing it on PyPI now, and then proposing it for
> inclusion into 3.5.

What I dislike is the idea of doing additional work because some
barriers are imposed ;-). PEP or PyPI are on a similar scale here. At
least a PEP would help record the various discussion details, so I'd
favour that over the PyPI path.

Regards

Antoine.

From ethan at stoneleaf.us  Wed Sep 11 20:07:55 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 11 Sep 2013 11:07:55 -0700
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130911194935.1b116266@fsol>
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
 <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
 <20130911115706.C2D98250925@webabinitio.net>
 <CADiSq7cjF6RGdrobJh1=f0etG3rVXARRUSCOJXzina0u1rkJTQ@mail.gmail.com>
 <20130911150423.62f77bf2@pitrou.net> <5230A90C.3090102@v.loewis.de>
 <20130911194935.1b116266@fsol>
Message-ID: <5230B17B.3000607@stoneleaf.us>

On 09/11/2013 10:49 AM, Antoine Pitrou wrote:
>
> What I dislike is the idea of doing additional work because some
> barriers are imposed ;-). PEP or PyPI are on a similar scale here. At
> least a PEP would help record the various discussion details, so I'd
> favour that over the PyPI path.

I would think a good summary on the bug tracker would preserve the detail just fine.

--
~Ethan~

From tjreedy at udel.edu  Wed Sep 11 22:04:45 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 11 Sep 2013 16:04:45 -0400
Subject: [Python-Dev] Need testing audio files
In-Reply-To: <20130911101053.71deed3b@anarchist>
References: <l0pfg4$i7l$1@ger.gmane.org> <20130911101053.71deed3b@anarchist>
Message-ID: <l0qicn$vs2$1@ger.gmane.org>

On 9/11/2013 10:10 AM, Barry Warsaw wrote:
> On Sep 11, 2013, at 01:09 PM, Serhiy Storchaka wrote:
>
>> In ideal it should be one high-quality (float64?) multichannel (5+1?) but
>> short master file and it's lower-quality copies made by third-party tools. In
>> ideal the content should be related to Python.

How about "Python is a great programming language." Long enough?

> I have some pro-audio recording capabilities and would be happy to generate
> some copyright-donated clips for Python.  Please contact me off-list if
> needed.


-- 
Terry Jan Reedy


From timothy.c.delaney at gmail.com  Wed Sep 11 23:39:31 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Thu, 12 Sep 2013 07:39:31 +1000
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <52309446.6070809@stoneleaf.us>
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
 <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
 <20130911134032.GA16820@ando>
 <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com>
 <52307DE0.9040208@stoneleaf.us>
 <CAMpsgwb1iB7fMKjYX5T3HR0E+3rmt4H055z2cjyJ-fMEjoEe6Q@mail.gmail.com>
 <52309446.6070809@stoneleaf.us>
Message-ID: <CAN8CLg=MzvW77F9XKCa3g+26Ufby6EwGyO5yTQbw+--A0iQhYA@mail.gmail.com>

On 12 September 2013 02:03, Ethan Furman <ethan at stoneleaf.us> wrote:

> On 09/11/2013 08:49 AM, Victor Stinner wrote:
>
>> 2013/9/11 Ethan Furman <ethan at stoneleaf.us>:
>>
>>> He isn't keeping the key unchanged (notice no white space in MAPPING),
>>> he's
>>> merely providing a function that will automatically strip the whitespace
>>> from key lookups.
>>>
>>
>> transformdict keeps the key unchanged, see the first message:
>>
>>     >>> d = transformdict(str.lower)
>>     >>> d['Foo'] = 5
>>     >>> d['foo']
>>     5
>>     >>> d['FOO']
>>     5
>>     >>> list(d)
>>     ['Foo']
>>
>
That seems backwards to me. I would think that retrieving the keys from the
dict would return the transformed keys (I'd call them canonical keys). That
way there's no question about which key is stored - it's *always* the
transformed key.

In fact, I think this might get more traction if it were referred to as a
canonicalising dictionary (bikeshedding, I know).

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130912/33424550/attachment.html>

From ethan at stoneleaf.us  Wed Sep 11 23:56:22 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 11 Sep 2013 14:56:22 -0700
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <CAN8CLg=MzvW77F9XKCa3g+26Ufby6EwGyO5yTQbw+--A0iQhYA@mail.gmail.com>
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
 <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
 <20130911134032.GA16820@ando>
 <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com>
 <52307DE0.9040208@stoneleaf.us>
 <CAMpsgwb1iB7fMKjYX5T3HR0E+3rmt4H055z2cjyJ-fMEjoEe6Q@mail.gmail.com>
 <52309446.6070809@stoneleaf.us>
 <CAN8CLg=MzvW77F9XKCa3g+26Ufby6EwGyO5yTQbw+--A0iQhYA@mail.gmail.com>
Message-ID: <5230E706.2070103@stoneleaf.us>

On 09/11/2013 02:39 PM, Tim Delaney wrote:
> On 12 September 2013 02:03, Ethan Furman <ethan at stoneleaf.us <mailto:ethan at stoneleaf.us>> wrote:
>
>     On 09/11/2013 08:49 AM, Victor Stinner wrote:
>
>         2013/9/11 Ethan Furman <ethan at stoneleaf.us <mailto:ethan at stoneleaf.us>>:
>
>             He isn't keeping the key unchanged (notice no white space in MAPPING), he's
>             merely providing a function that will automatically strip the whitespace
>             from key lookups.
>
>
>         transformdict keeps the key unchanged, see the first message:
>
>              >>> d = transformdict(str.lower)
>              >>> d['Foo'] = 5
>              >>> d['foo']
>              5
>              >>> d['FOO']
>              5
>              >>> list(d)
>              ['Foo']
>
> That seems backwards to me. I would think that retrieving the keys from the dict would return the transformed keys (I'd
> call them canonical keys). That way there's no question about which key is stored - it's *always* the transformed key.

At this point there is still no question: it's the first version of the key seen.  For a stupid example:

--> d = transformdict(str.lower)
--> d['ThePyramid'] = 'Game Show'
--> d['AtOnce'] = now()
--> for k, v in d.items():
...     print(k, v)

Imagine writing a function to get that capitalization right.


> In fact, I think this might get more traction if it were referred to as a canonicalising dictionary (bikeshedding, I know).

Whoa, that's way harder to spell!  ;)  Drop the 'ising', though, and I'm in.

--
~Ethan~

From ben+python at benfinney.id.au  Thu Sep 12 01:12:36 2013
From: ben+python at benfinney.id.au (Ben Finney)
Date: Thu, 12 Sep 2013 09:12:36 +1000
Subject: [Python-Dev] Need testing audio files
References: <l0pfg4$i7l$1@ger.gmane.org>
Message-ID: <7w7gen2brv.fsf@benfinney.id.au>

Serhiy Storchaka <storchaka at gmail.com> writes:

> I work on enhancement of audio modules testing [1], and I need free
> (in both senses) small sample audio files in different formats.

The Internet Archive <URL:https://archive.org/> has a wide selection of
free-software media, many of which have free license terms that would be
suitable for inclusion in Python.

I haven't used it, but <URL:http://www.pdsounds.org/> appears to be a
collection of sounds in the public domain.

-- 
 \            ?The whole area of [treating source code as intellectual |
  `\    property] is almost assuring a customer that you are not going |
_o__)               to do any innovation in the future.? ?Gary Barnett |
Ben Finney


From solipsis at pitrou.net  Thu Sep 12 08:14:23 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 12 Sep 2013 08:14:23 +0200
Subject: [Python-Dev] cpython (merge 3.3 -> default): merge from 3.3
References: <3cb8QF2vy7z7Ljh@mail.python.org>
Message-ID: <20130912081423.4607ea82@fsol>

On Thu, 12 Sep 2013 07:57:25 +0200 (CEST)
senthil.kumaran <python-checkins at python.org> wrote:
>  
> +<<<<<<< local
>          Optional argument random is a 0-argument function returning a
>          random float in [0.0, 1.0); if it is the default None, the
>          standard random.random will be used.
> +=======
> +        Optional arg random is a 0-argument function returning a random
> +        float in [0.0, 1.0); by default, the standard random.random.
> +
> +        Do not supply the 'int' argument.
> +>>>>>>> other

Can you fix this?

Thanks

Antoine.



From ethan at stoneleaf.us  Thu Sep 12 16:08:47 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Thu, 12 Sep 2013 07:08:47 -0700
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <CAN8CLg=MzvW77F9XKCa3g+26Ufby6EwGyO5yTQbw+--A0iQhYA@mail.gmail.com>
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
 <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
 <20130911134032.GA16820@ando>
 <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com>
 <52307DE0.9040208@stoneleaf.us>
 <CAMpsgwb1iB7fMKjYX5T3HR0E+3rmt4H055z2cjyJ-fMEjoEe6Q@mail.gmail.com>
 <52309446.6070809@stoneleaf.us>
 <CAN8CLg=MzvW77F9XKCa3g+26Ufby6EwGyO5yTQbw+--A0iQhYA@mail.gmail.com>
Message-ID: <5231CAEF.1080500@stoneleaf.us>

On 09/11/2013 02:39 PM, Tim Delaney wrote:
>
> I would think that retrieving the keys from the dict would return the transformed keys (I'd
> call them canonical keys).

The more I think about this the more I agree.  A canonicaldict with a key function that simply stored the transformed 
key and it's value would seem to be a lot simpler:

   - no need to store a separate "presentation" key
   - no confusion about which of the first key/last key seen is stored
   - no mistakes with the "first" key not being added before real data
     and getting the presentation key wrong

Further, in order to store the non-canonical keys a separate list must be kept of the keys to preseed the canonicaldict; 
if we store the canonical keys a separate list must be kept for presentation purposes -- so worst case scenario we're 
keeping the same amount of information and best-case scenario the presentation of the keys doesn't matter and we just 
saved ourselves an extra data structure.

--
~Ethan~

From ronaldoussoren at mac.com  Thu Sep 12 16:42:39 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Thu, 12 Sep 2013 16:42:39 +0200
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <0752376899dd7436f6115bfc03594574@chopin.edu.pl>
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
 <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
 <522DECA3.2030709@hotpy.org>
 <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com>
 <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl>
Message-ID: <8F0FD1DE-A8ED-40D7-878A-9A005ABDF70C@mac.com>


On 9 Sep, 2013, at 20:23, Jan Kaliszewski <zuo at chopin.edu.pl> wrote:

> Is '__locallookup__' a really good name? In Python, *local* -- especially in context of *lookups* -- usually associates with locals() i.e. a namespace of a function/method execution frame or a namespace of a class, during *definition* of that class... So '__locallookup__' can be confusing.
> 
> Why not just '__getclassattribute__' or '__classlookup__', or '__classattribute__'...?

I don't particularly like __locallookup__ either, but haven't found a better name yet.  "__lookup_in_class__" was the best alternative I could come up with, and that feels different than other special methods.  The name in the PEP is more or less derived from _PyType_Lookup, with "local" meaning "only in this class, don't recurse in the rest of the MRO".  


Ronald


From solipsis at pitrou.net  Thu Sep 12 16:43:00 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 12 Sep 2013 16:43:00 +0200
Subject: [Python-Dev] Add a "transformdict" to collections
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
 <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
 <20130911134032.GA16820@ando>
 <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com>
 <52307DE0.9040208@stoneleaf.us>
 <CAMpsgwb1iB7fMKjYX5T3HR0E+3rmt4H055z2cjyJ-fMEjoEe6Q@mail.gmail.com>
 <52309446.6070809@stoneleaf.us>
 <CAN8CLg=MzvW77F9XKCa3g+26Ufby6EwGyO5yTQbw+--A0iQhYA@mail.gmail.com>
 <5231CAEF.1080500@stoneleaf.us>
Message-ID: <20130912164300.5cd27a1a@pitrou.net>

Le Thu, 12 Sep 2013 07:08:47 -0700,
Ethan Furman <ethan at stoneleaf.us> a ?crit :
> On 09/11/2013 02:39 PM, Tim Delaney wrote:
> >
> > I would think that retrieving the keys from the dict would return
> > the transformed keys (I'd call them canonical keys).
> 
> The more I think about this the more I agree.  A canonicaldict with a
> key function that simply stored the transformed key and it's value
> would seem to be a lot simpler:

And it wouldn't solve the use cases. What's the point?

> Further, in order to store the non-canonical keys a separate list
> must be kept of the keys to preseed the canonicaldict;

Yeah, so this is totally silly. What you're basically saying is "we
don't need TransformDict since people can re-implement it themselves".

Regards

Antoine.



From duda.piotr at gmail.com  Thu Sep 12 16:51:26 2013
From: duda.piotr at gmail.com (Piotr Duda)
Date: Thu, 12 Sep 2013 16:51:26 +0200
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <5231CAEF.1080500@stoneleaf.us>
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
 <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
 <20130911134032.GA16820@ando>
 <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com>
 <52307DE0.9040208@stoneleaf.us>
 <CAMpsgwb1iB7fMKjYX5T3HR0E+3rmt4H055z2cjyJ-fMEjoEe6Q@mail.gmail.com>
 <52309446.6070809@stoneleaf.us>
 <CAN8CLg=MzvW77F9XKCa3g+26Ufby6EwGyO5yTQbw+--A0iQhYA@mail.gmail.com>
 <5231CAEF.1080500@stoneleaf.us>
Message-ID: <CAJ1Wxn2GOodq_USD_L3ret73cfq7WEPtaGXftuQprOUVQ+FzMA@mail.gmail.com>

2013/9/12 Ethan Furman <ethan at stoneleaf.us>:
> On 09/11/2013 02:39 PM, Tim Delaney wrote:
>>
>>
>> I would think that retrieving the keys from the dict would return the
>> transformed keys (I'd
>> call them canonical keys).
>
>
> The more I think about this the more I agree.  A canonicaldict with a key
> function that simply stored the transformed key and it's value would seem to
> be a lot simpler:
>
>   - no need to store a separate "presentation" key
>   - no confusion about which of the first key/last key seen is stored
>   - no mistakes with the "first" key not being added before real data
>     and getting the presentation key wrong

If original keys aren't stored then

d = TransformDict(transfunc, ...)
for k in d:
  dosomething(k, d[k])

will break if transfunc(transfunc(x)) != transfunc(x)



-- 
????????
??????

From ronaldoussoren at mac.com  Thu Sep 12 16:25:56 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Thu, 12 Sep 2013 16:25:56 +0200
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <522DECA3.2030709@hotpy.org>
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
 <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
 <522DECA3.2030709@hotpy.org>
Message-ID: <D280AEE7-B847-45B6-9F6B-359384971CBA@mac.com>


On 9 Sep, 2013, at 17:43, Mark Shannon <mark at hotpy.org> wrote:

> I would like time to investigate this further, but at the moment I think it will either make attribute lookup poorly defined or slow.
> 
> Of the top of my head, the problem as a I see it is basically this:
> Currently, type.__getattribute__() is a fixed point in the lookup of attributes.
> The proposal means that a fixed point is not reached until the cls parameter of type.__getattribute__() is either object or type,
> otherwise type.__getattribute__() and type.__locallookup__ must bounce back and forth.
> 
> This will slow down *every* attribute lookup for what is a fairly obscure use case.

I did a benchmark run (see the pep for details) and that seems to indicate that the performance impact is very small, possibly because the patch keeps the attribute lookup cache used by _PyType_Lookup.

Anyway, I'm glad that there is now some real discussion on the proposal. Not unsurprisingly I'd love to have this, or something simular, in 3.4.  I had hoped to repost the PEP a while back with more information on how the API would affect PyObjC (code removal, performance impact), but haven't had time to move forward on that front :-(

Ronald


> 
> Cheers,
> Mark.
> 
> On 09/09/13 16:27, Guido van Rossum wrote:
>> Let's just accept this PEP. It looks like a nice addition to the metaclass machinery and I don't think we'll get much more useful feedback by waiting.
>> 
>> 
>> On Mon, Sep 9, 2013 at 7:30 AM, Ethan Furman <ethan at stoneleaf.us <mailto:ethan at stoneleaf.us>> wrote:
>> 
>>    On 07/30/2013 11:17 PM, Ronald Oussoren wrote:
>> 
>> 
>>        And something I forgot to ask: is anyone willing to be the BDFL-Delegate for
>>        PEP 447?
>> 
>> 
>>    *Bump*.
>> 
>>    It would be nice if this could make into 3.4.
>> 
>>    --
>>    ~Ethan~
>> 
>>    _________________________________________________
>>    Python-Dev mailing list
>>    Python-Dev at python.org <mailto:Python-Dev at python.org>
>>    https://mail.python.org/__mailman/listinfo/python-dev <https://mail.python.org/mailman/listinfo/python-dev>
>>    Unsubscribe: https://mail.python.org/__mailman/options/python-dev/__guido%40python.org <https://mail.python.org/mailman/options/python-dev/guido%40python.org>
>> 
>> 
>> 
>> 
>> --
>> --Guido van Rossum (python.org/~guido <http://python.org/~guido>)
>> 
>> 
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/mark%40hotpy.org
>> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com


From ethan at stoneleaf.us  Thu Sep 12 17:05:44 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Thu, 12 Sep 2013 08:05:44 -0700
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130912164300.5cd27a1a@pitrou.net>
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
 <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
 <20130911134032.GA16820@ando>
 <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com>
 <52307DE0.9040208@stoneleaf.us>
 <CAMpsgwb1iB7fMKjYX5T3HR0E+3rmt4H055z2cjyJ-fMEjoEe6Q@mail.gmail.com>
 <52309446.6070809@stoneleaf.us>
 <CAN8CLg=MzvW77F9XKCa3g+26Ufby6EwGyO5yTQbw+--A0iQhYA@mail.gmail.com>
 <5231CAEF.1080500@stoneleaf.us> <20130912164300.5cd27a1a@pitrou.net>
Message-ID: <5231D848.1090709@stoneleaf.us>

On 09/12/2013 07:43 AM, Antoine Pitrou wrote:
>
> Yeah, so this is totally silly. What you're basically saying is "we
> don't need TransformDict since people can re-implement it themselves".

No, what I'm saying is that the "case-preserving" aspect of transformdict is silly.  The main point of transformdict is 
to enable, for example, 'IBM', 'Ibm', and 'ibm' to all match up as the same key.  But why?  Because you don't trust the 
user data.  And if you don't trust the user data you have to add the correct version of the key yourself before you ever 
process that data, which means you already have the correct version stored somewhere.

--
~Ethan~

From rdmurray at bitdance.com  Thu Sep 12 17:30:49 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Thu, 12 Sep 2013 11:30:49 -0400
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <8F0FD1DE-A8ED-40D7-878A-9A005ABDF70C@mac.com>
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
 <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
 <522DECA3.2030709@hotpy.org>
 <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com>
 <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl>
 <8F0FD1DE-A8ED-40D7-878A-9A005ABDF70C@mac.com>
Message-ID: <20130912153050.3CBC0250843@webabinitio.net>

On Thu, 12 Sep 2013 16:42:39 +0200, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
> 
> On 9 Sep, 2013, at 20:23, Jan Kaliszewski <zuo at chopin.edu.pl> wrote:
> 
> > Is '__locallookup__' a really good name? In Python, *local* --
> > especially in context of *lookups* -- usually associates with
> > locals() i.e. a namespace of a function/method execution frame or a
> > namespace of a class, during *definition* of that class... So
> > '__locallookup__' can be confusing.
> > 
> > Why not just '__getclassattribute__' or '__classlookup__', or
> > '__classattribute__'...?
> 
> I don't particularly like __locallookup__ either, but haven't found a
> better name yet.  "__lookup_in_class__" was the best alternative I
> could come up with, and that feels different than other special
> methods.  The name in the PEP is more or less derived from
> _PyType_Lookup, with "local" meaning "only in this class, don't
> recurse in the rest of the MRO".  

Why is __getclassattribute__ worse than __locallookup__?

--David

From ronaldoussoren at mac.com  Thu Sep 12 16:33:12 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Thu, 12 Sep 2013 16:33:12 +0200
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <522E3B2E.6050605@hotpy.org>
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
 <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us>
 <522E3B2E.6050605@hotpy.org>
Message-ID: <B4E5C214-4D51-4C0C-A906-01DC717B69DB@mac.com>


On 9 Sep, 2013, at 23:18, Mark Shannon <mark at hotpy.org> wrote:

> On 09/09/13 15:30, Ethan Furman wrote:
>> On 07/30/2013 11:17 PM, Ronald Oussoren wrote:
>>> 
>>> And something I forgot to ask: is anyone willing to be the
>>> BDFL-Delegate for
>>> PEP 447?
>> 
>> *Bump*.
>> 
>> It would be nice if this could make into 3.4.
>> 
> 
> IMO, there are some issues that need to be addressed before PEP 447 should be accepted.
> 
> 1. Is there even a problem at all, or is this just a bug in super?
> Why doesn't super() respect the __getattribute__ method of the superclass?

Because __getattribute__ looks in the instance __dict__ before walking the MRO, while super does not.

> 
> 2. Is this the best way to solve the problem (if there is a problem)?
> Would a __super__ special method be sufficient and less intrusive.

One reason for the __locallookup__ method is to make normal and super attribute lookup more simular, adding a __super__ special method would lead to code duplication: both __getattribute__ and __super__ would either contain simular code, or would call out to a shared method anyway.

> 
> 3. Are the proposed semantics OK?
> I think they are, but very low level changes such as these can have unforeseen consequences. For example, PEP 3135 and issue 12370.
> 
> 4. What is the performance impact. pybench really doesn't count as a benchmark.

What kind of benchmark would you like to see?  BTW. I ran more than pybench, I also ran the part of the performance benchmark that worked on py3k at the time.

Ronald

From solipsis at pitrou.net  Thu Sep 12 17:40:31 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 12 Sep 2013 17:40:31 +0200
Subject: [Python-Dev] Add a "transformdict" to collections
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
 <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
 <20130911134032.GA16820@ando>
 <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com>
 <52307DE0.9040208@stoneleaf.us>
 <CAMpsgwb1iB7fMKjYX5T3HR0E+3rmt4H055z2cjyJ-fMEjoEe6Q@mail.gmail.com>
 <52309446.6070809@stoneleaf.us>
 <CAN8CLg=MzvW77F9XKCa3g+26Ufby6EwGyO5yTQbw+--A0iQhYA@mail.gmail.com>
 <5231CAEF.1080500@stoneleaf.us>
 <20130912164300.5cd27a1a@pitrou.net>
 <5231D848.1090709@stoneleaf.us>
Message-ID: <20130912174031.5971338f@pitrou.net>

Le Thu, 12 Sep 2013 08:05:44 -0700,
Ethan Furman <ethan at stoneleaf.us> a ?crit :
> On 09/12/2013 07:43 AM, Antoine Pitrou wrote:
> >
> > Yeah, so this is totally silly. What you're basically saying is "we
> > don't need TransformDict since people can re-implement it
> > themselves".
> 
> No, what I'm saying is that the "case-preserving" aspect of
> transformdict is silly.  The main point of transformdict is to
> enable, for example, 'IBM', 'Ibm', and 'ibm' to all match up as the
> same key.  But why?  Because you don't trust the user data.  And if
> you don't trust the user data you have to add the correct version of
> the key yourself before you ever process that data, which means you
> already have the correct version stored somewhere.

That's assuming there is an a priori "correct" version. But there might
not be any. Keeping the original key is important for different reasons
depending on the use case:

- for case-insensitive dicts, you want to keep the original key for
  presentation, logging and debugging purposes (*)

- for identity dicts, the original key is mandatory because the id()
  value in itself is completely useless, it's just used for matching

(*) For a well-known example of such behaviour, think about Windows
filesystems.

Regards

Antoine.



From rdmurray at bitdance.com  Thu Sep 12 17:46:11 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Thu, 12 Sep 2013 11:46:11 -0400
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <5231D848.1090709@stoneleaf.us>
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
 <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
 <20130911134032.GA16820@ando>
 <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com>
 <52307DE0.9040208@stoneleaf.us>
 <CAMpsgwb1iB7fMKjYX5T3HR0E+3rmt4H055z2cjyJ-fMEjoEe6Q@mail.gmail.com>
 <52309446.6070809@stoneleaf.us>
 <CAN8CLg=MzvW77F9XKCa3g+26Ufby6EwGyO5yTQbw+--A0iQhYA@mail.gmail.com>
 <5231CAEF.1080500@stoneleaf.us> <20130912164300.5cd27a1a@pitrou.net>
 <5231D848.1090709@stoneleaf.us>
Message-ID: <20130912154611.CEDD4250843@webabinitio.net>

On Thu, 12 Sep 2013 08:05:44 -0700, Ethan Furman <ethan at stoneleaf.us> wrote:
> On 09/12/2013 07:43 AM, Antoine Pitrou wrote:
> >
> > Yeah, so this is totally silly. What you're basically saying is "we
> > don't need TransformDict since people can re-implement it themselves".
> 
> No, what I'm saying is that the "case-preserving" aspect of
> transformdict is silly.  The main point of transformdict is to enable,
> for example, 'IBM', 'Ibm', and 'ibm' to all match up as the same key.
> But why?  Because you don't trust the user data.  And if you don't
> trust the user data you have to add the correct version of the key
> yourself before you ever process that data, which means you already
> have the correct version stored somewhere.

No, in the *(original use case* (and many other use cases, such as the
identity dict) we *do* trust the user data.  It is more trustworthy than
the key forms used to look up the data inside the program.  That's the
whole point.  (Don't be distracted by Eric's particular use case.)

But even for the general case...well, let me repost here what I said on
the tracker:

In our view, this data structure is for cases where the original
key is the most important piece of information (about the keys).
The transformation in the lookup process is entirely in the service of
looking up the value paired with that original key when there is more
than one possible representation of that key.  It is the original key
that is critical when re-serializing the data or otherwise making use
of the keys for anything other than lookup.  So this is about making
the data structure succinctly model the problem domain, which is what
OO is supposed to be good at :)

--David

From ethan at stoneleaf.us  Thu Sep 12 17:48:22 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Thu, 12 Sep 2013 08:48:22 -0700
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130912174031.5971338f@pitrou.net>
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
 <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
 <20130911134032.GA16820@ando>
 <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com>
 <52307DE0.9040208@stoneleaf.us>
 <CAMpsgwb1iB7fMKjYX5T3HR0E+3rmt4H055z2cjyJ-fMEjoEe6Q@mail.gmail.com>
 <52309446.6070809@stoneleaf.us>
 <CAN8CLg=MzvW77F9XKCa3g+26Ufby6EwGyO5yTQbw+--A0iQhYA@mail.gmail.com>
 <5231CAEF.1080500@stoneleaf.us> <20130912164300.5cd27a1a@pitrou.net>
 <5231D848.1090709@stoneleaf.us> <20130912174031.5971338f@pitrou.net>
Message-ID: <5231E246.2020507@stoneleaf.us>

On 09/12/2013 08:40 AM, Antoine Pitrou wrote:
> Le Thu, 12 Sep 2013 08:05:44 -0700,
> Ethan Furman <ethan at stoneleaf.us> a ?crit :
>> On 09/12/2013 07:43 AM, Antoine Pitrou wrote:
>>>
>>> Yeah, so this is totally silly. What you're basically saying is "we
>>> don't need TransformDict since people can re-implement it
>>> themselves".
>>
>> No, what I'm saying is that the "case-preserving" aspect of
>> transformdict is silly.  The main point of transformdict is to
>> enable, for example, 'IBM', 'Ibm', and 'ibm' to all match up as the
>> same key.  But why?  Because you don't trust the user data.  And if
>> you don't trust the user data you have to add the correct version of
>> the key yourself before you ever process that data, which means you
>> already have the correct version stored somewhere.
>
> That's assuming there is an a priori "correct" version. But there might
> not be any. Keeping the original key is important for different reasons
> depending on the use case:
>
> - for case-insensitive dicts, you want to keep the original key for
>    presentation, logging and debugging purposes (*)
>
> - for identity dicts, the original key is mandatory because the id()
>    value in itself is completely useless, it's just used for matching
>
> (*) For a well-known example of such behaviour, think about Windows
> filesystems.

Ah, I see.  Thank you for explaining.

--
~Ethan~

From ronaldoussoren at mac.com  Thu Sep 12 20:12:18 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Thu, 12 Sep 2013 20:12:18 +0200
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <20130912153050.3CBC0250843@webabinitio.net>
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
 <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
 <522DECA3.2030709@hotpy.org>
 <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com>
 <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl>
 <8F0FD1DE-A8ED-40D7-878A-9A005ABDF70C@mac.com>
 <20130912153050.3CBC0250843@webabinitio.net>
Message-ID: <35B9C3A0-3039-49CA-90D6-319890611618@mac.com>



> On 12 sep. 2013, at 17:30, "R. David Murray" <rdmurray at bitdance.com> wrote:
> 
>> On Thu, 12 Sep 2013 16:42:39 +0200, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
>> 
>>> On 9 Sep, 2013, at 20:23, Jan Kaliszewski <zuo at chopin.edu.pl> wrote:
>>> 
>>> Is '__locallookup__' a really good name? In Python, *local* --
>>> especially in context of *lookups* -- usually associates with
>>> locals() i.e. a namespace of a function/method execution frame or a
>>> namespace of a class, during *definition* of that class... So
>>> '__locallookup__' can be confusing.
>>> 
>>> Why not just '__getclassattribute__' or '__classlookup__', or
>>> '__classattribute__'...?
>> 
>> I don't particularly like __locallookup__ either, but haven't found a
>> better name yet.  "__lookup_in_class__" was the best alternative I
>> could come up with, and that feels different than other special
>> methods.  The name in the PEP is more or less derived from
>> _PyType_Lookup, with "local" meaning "only in this class, don't
>> recurse in the rest of the MRO".  
> 
> Why is __getclassattribute__ worse than __locallookup__?

Getclassattribute feels like it is related to classmethod, or fetches an attribute of a class. The method does however fetch a value from the class that is transformed to the actual attribute value through the descriptor protocol. 

Ronald
> 
> --David
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com

From v+python at g.nevcal.com  Thu Sep 12 21:15:14 2013
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Thu, 12 Sep 2013 12:15:14 -0700
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130912174031.5971338f@pitrou.net>
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
 <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
 <20130911134032.GA16820@ando>
 <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com>
 <52307DE0.9040208@stoneleaf.us>
 <CAMpsgwb1iB7fMKjYX5T3HR0E+3rmt4H055z2cjyJ-fMEjoEe6Q@mail.gmail.com>
 <52309446.6070809@stoneleaf.us>
 <CAN8CLg=MzvW77F9XKCa3g+26Ufby6EwGyO5yTQbw+--A0iQhYA@mail.gmail.com>
 <5231CAEF.1080500@stoneleaf.us> <20130912164300.5cd27a1a@pitrou.net>
 <5231D848.1090709@stoneleaf.us> <20130912174031.5971338f@pitrou.net>
Message-ID: <523212C2.6040705@g.nevcal.com>

On 9/12/2013 8:40 AM, Antoine Pitrou wrote:
> Le Thu, 12 Sep 2013 08:05:44 -0700,
> Ethan Furman <ethan at stoneleaf.us> a ?crit :
>> On 09/12/2013 07:43 AM, Antoine Pitrou wrote:
>>> Yeah, so this is totally silly. What you're basically saying is "we
>>> don't need TransformDict since people can re-implement it
>>> themselves".
>> No, what I'm saying is that the "case-preserving" aspect of
>> transformdict is silly.  The main point of transformdict is to
>> enable, for example, 'IBM', 'Ibm', and 'ibm' to all match up as the
>> same key.  But why?  Because you don't trust the user data.  And if
>> you don't trust the user data you have to add the correct version of
>> the key yourself before you ever process that data, which means you
>> already have the correct version stored somewhere.
> That's assuming there is an a priori "correct" version. But there might
> not be any. Keeping the original key is important for different reasons
> depending on the use case:
>
> - for case-insensitive dicts, you want to keep the original key for
>    presentation, logging and debugging purposes (*)
>
> - for identity dicts, the original key is mandatory because the id()
>    value in itself is completely useless, it's just used for matching

  - for dict with non-hashable key, but a transform function that can 
derive a hashable key from it, the presentation key value may be much 
more complex than the actual key value.
>
> (*) For a well-known example of such behaviour, think about Windows
> filesystems.
>
> Regards
>
> Antoine.
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130912/89fcb0a4/attachment.html>

From xdegaye at gmail.com  Thu Sep 12 22:20:04 2013
From: xdegaye at gmail.com (Xavier de Gaye)
Date: Thu, 12 Sep 2013 22:20:04 +0200
Subject: [Python-Dev] f_lineno accessors
Message-ID: <523221F4.8020406@gmail.com>

In issues 7238 [1], 16482 [2], 17697 [3] and 17277 [4], the line
number may be incorrect when the global trace function has been
removed but not the frame f_trace function.
A simple test (see below) in issue 17288 [5] crashes the interpreter
when setting f_lineno in a generator from a return trace function.
All those issues are few months old. There is a patch at issue 17277
[4] that fixes the first 4 issues. There is also a patch for issue
17288 [5].


######   Setting f_lineno in a generator   ######
$ cat jump.py
def gen():
     for i in range(1):
         yield i
     lineno = 4

for i in gen():
     pass

$ python -m pdb jump.py
 > /tmp/jump.py(1)<module>()
-> def gen():
(Pdb) import sys; print(sys.version)
3.4.0a1+ (default:975d1e180689, Sep  6 2013, 09:26:12)
[GCC 4.3.2]
(Pdb) break 3
Breakpoint 1 at /tmp/jump.py:3
(Pdb) continue
 > /tmp/jump.py(3)gen()
-> yield i
(Pdb) step
--Return--
 > /tmp/jump.py(3)gen()->0
-> yield i
(Pdb) jump 2
 > /tmp/jump.py(2)gen()->0
-> for i in range(1):
(Pdb) continue
Segmentation fault

[1] http://bugs.python.org/issue7238
[2] http://bugs.python.org/issue16482
[3] http://bugs.python.org/issue17697
[4] http://bugs.python.org/issue17277
[5] http://bugs.python.org/issue17288

Xavier

From timothy.c.delaney at gmail.com  Thu Sep 12 23:29:30 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Fri, 13 Sep 2013 07:29:30 +1000
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <20130912174031.5971338f@pitrou.net>
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
 <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
 <20130911134032.GA16820@ando>
 <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com>
 <52307DE0.9040208@stoneleaf.us>
 <CAMpsgwb1iB7fMKjYX5T3HR0E+3rmt4H055z2cjyJ-fMEjoEe6Q@mail.gmail.com>
 <52309446.6070809@stoneleaf.us>
 <CAN8CLg=MzvW77F9XKCa3g+26Ufby6EwGyO5yTQbw+--A0iQhYA@mail.gmail.com>
 <5231CAEF.1080500@stoneleaf.us>
 <20130912164300.5cd27a1a@pitrou.net>
 <5231D848.1090709@stoneleaf.us>
 <20130912174031.5971338f@pitrou.net>
Message-ID: <CAN8CLgk3eG_uf7M-f0AuQX0XXq4t2B7JOjS2DL40NC94QioabA@mail.gmail.com>

On 13 September 2013 01:40, Antoine Pitrou <solipsis at pitrou.net> wrote:

> Le Thu, 12 Sep 2013 08:05:44 -0700,
> Ethan Furman <ethan at stoneleaf.us> a ?crit :
> > On 09/12/2013 07:43 AM, Antoine Pitrou wrote:
> > >
> > > Yeah, so this is totally silly. What you're basically saying is "we
> > > don't need TransformDict since people can re-implement it
> > > themselves".
> >
> > No, what I'm saying is that the "case-preserving" aspect of
> > transformdict is silly.  The main point of transformdict is to
> > enable, for example, 'IBM', 'Ibm', and 'ibm' to all match up as the
> > same key.  But why?  Because you don't trust the user data.  And if
> > you don't trust the user data you have to add the correct version of
> > the key yourself before you ever process that data, which means you
> > already have the correct version stored somewhere.
>
> That's assuming there is an a priori "correct" version. But there might
> not be any. Keeping the original key is important for different reasons
> depending on the use case:
>
> - for case-insensitive dicts, you want to keep the original key for
>   presentation, logging and debugging purposes (*)
>
> - for identity dicts, the original key is mandatory because the id()
>   value in itself is completely useless, it's just used for matching
>
> (*) For a well-known example of such behaviour, think about Windows
> filesystems.
>

In this case though, there are two pieces of information:

1. A canonical key (which may or may not equal the original key);

2. The original key.

It seems to me then that TransformDict is a specialised case of
CanonicalDict, where the canonical key is defined to be the first key
inserted. It would in fact be possible (though inefficient) to implement
that using a canonicalising callable that maintained state - something like
(untested):

class OriginalKeys:
    def __init__(self)::
        self.keys = CanonicalDict(str.lower)

    def __call__(self, key):
        return self.keys.setdefault(key, key)

class OriginalKeyDict(CanonicalDict):
    def __init__(self)::
        super().__init__(OriginalKeys())

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130913/ab28b609/attachment.html>

From timothy.c.delaney at gmail.com  Thu Sep 12 23:34:15 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Fri, 13 Sep 2013 07:34:15 +1000
Subject: [Python-Dev] Add a "transformdict" to collections
In-Reply-To: <CAN8CLgk3eG_uf7M-f0AuQX0XXq4t2B7JOjS2DL40NC94QioabA@mail.gmail.com>
References: <20130910112832.75ed6c8b@pitrou.net>
 <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com>
 <20130911102732.2cb268ca@pitrou.net>
 <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com>
 <20130911134032.GA16820@ando>
 <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com>
 <52307DE0.9040208@stoneleaf.us>
 <CAMpsgwb1iB7fMKjYX5T3HR0E+3rmt4H055z2cjyJ-fMEjoEe6Q@mail.gmail.com>
 <52309446.6070809@stoneleaf.us>
 <CAN8CLg=MzvW77F9XKCa3g+26Ufby6EwGyO5yTQbw+--A0iQhYA@mail.gmail.com>
 <5231CAEF.1080500@stoneleaf.us>
 <20130912164300.5cd27a1a@pitrou.net>
 <5231D848.1090709@stoneleaf.us>
 <20130912174031.5971338f@pitrou.net>
 <CAN8CLgk3eG_uf7M-f0AuQX0XXq4t2B7JOjS2DL40NC94QioabA@mail.gmail.com>
Message-ID: <CAN8CLg=L8+URU-JB0-hA+yk4U9M+U4dVgkeDR7t-Fes=5BfHEg@mail.gmail.com>

On 13 September 2013 07:29, Tim Delaney <timothy.c.delaney at gmail.com> wrote:

>
> In this case though, there are two pieces of information:
>
> 1. A canonical key (which may or may not equal the original key);
>
> 2. The original key.
>
> It seems to me then that TransformDict is a specialised case of
> CanonicalDict, where the canonical key is defined to be the first key
> inserted. It would in fact be possible (though inefficient) to implement
> that using a canonicalising callable that maintained state - something like
> (untested):
>
> class OriginalKeys:
>     def __init__(self)::
>         self.keys = CanonicalDict(str.lower)
>
>     def __call__(self, key):
>         return self.keys.setdefault(key, key)
>
> class OriginalKeyDict(CanonicalDict):
>     def __init__(self)::
>         super().__init__(OriginalKeys())
>

Bah - got myself mixed up with original key and case preserving there ...
try this:

class OriginalKeys:
    def __init__(self, func)::
        self.keys = CanonicalDict(func)

    def __call__(self, key):
        return self.keys.setdefault(key, key)

class OriginalKeyDict(CanonicalDict):
    def __init__(self, func)::
        super().__init__(OriginalKeys(func))

class IdentityDict(OriginalKeyDict):
    def __init__(self):
        super().__init__(id)

class CasePreservingDict(OriginalKeyDict):
    def __init__(self):
        super().__init__(str.lower)

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130913/ede12be5/attachment.html>

From steve at pearwood.info  Fri Sep 13 01:08:21 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 13 Sep 2013 09:08:21 +1000
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <8F0FD1DE-A8ED-40D7-878A-9A005ABDF70C@mac.com>
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
 <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
 <522DECA3.2030709@hotpy.org>
 <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com>
 <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl>
 <8F0FD1DE-A8ED-40D7-878A-9A005ABDF70C@mac.com>
Message-ID: <20130912230820.GC16820@ando>

On Thu, Sep 12, 2013 at 04:42:39PM +0200, Ronald Oussoren wrote:

> I don't particularly like __locallookup__ either, but haven't found a 
> better name yet.  "__lookup_in_class__" was the best alternative I 
> could come up with, and that feels different than other special 
> methods.  The name in the PEP is more or less derived from 
> _PyType_Lookup, with "local" meaning "only in this class, don't 
> recurse in the rest of the MRO".

How about __typelookup__ ? Surely that's the obvious name to derive from 
_PyType_Lookup :-)


-- 
Steven

From Steve.Dower at microsoft.com  Fri Sep 13 05:04:49 2013
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Fri, 13 Sep 2013 03:04:49 +0000
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <20130912230820.GC16820@ando>
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
 <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
 <522DECA3.2030709@hotpy.org>
 <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com>
 <522DF204.7040109@hotpy.org>
 <0752376899dd7436f6115bfc03594574@chopin.edu.pl>
 <8F0FD1DE-A8ED-40D7-878A-9A005ABDF70C@mac.com>,<20130912230820.GC16820@ando>
Message-ID: <51c253d972c94acbace9e46fc43b922a@BLUPR03MB199.namprd03.prod.outlook.com>

What about __getlocalattribute__ or __getattributenorecurse__? Long, but this isn't going to be used often.

Putting "type" or "class" in the name would be misleading. It's an instance method (that is most useful when implemented on a metaclass).

(Apologies for the top post.)

Sent from my Windows Phone
________________________________
From: Steven D'Aprano<mailto:steve at pearwood.info>
Sent: ?9/?12/?2013 16:09
To: python-dev at python.org<mailto:python-dev at python.org>
Subject: Re: [Python-Dev] PEP 447: add type.__locallookup__

On Thu, Sep 12, 2013 at 04:42:39PM +0200, Ronald Oussoren wrote:

> I don't particularly like __locallookup__ either, but haven't found a
> better name yet.  "__lookup_in_class__" was the best alternative I
> could come up with, and that feels different than other special
> methods.  The name in the PEP is more or less derived from
> _PyType_Lookup, with "local" meaning "only in this class, don't
> recurse in the rest of the MRO".

How about __typelookup__ ? Surely that's the obvious name to derive from
_PyType_Lookup :-)


--
Steven
_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130913/a7be4a71/attachment.html>

From steve at pearwood.info  Fri Sep 13 05:59:17 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 13 Sep 2013 13:59:17 +1000
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <51c253d972c94acbace9e46fc43b922a@BLUPR03MB199.namprd03.prod.outlook.com>
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
 <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
 <522DECA3.2030709@hotpy.org>
 <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com>
 <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl>
 <51c253d972c94acbace9e46fc43b922a@BLUPR03MB199.namprd03.prod.outlook.com>
Message-ID: <20130913035917.GE16820@ando>

On Fri, Sep 13, 2013 at 03:04:49AM +0000, Steve Dower wrote:

> What about __getlocalattribute__ or __getattributenorecurse__? Long, 
> but this isn't going to be used often.

This has nothing to do with locals, nor does it have anything to do with 
recursion, so both those names are misleading.


> Putting "type" or "class" in the name would be misleading. It's an 
> instance method (that is most useful when implemented on a metaclass).

Regardless of whether it is an instance method or not, by default it 
performs the lookup on the type. Hence the C function _PyType_Lookup and 
hence my suggestion __typelookup__.

But I think that __typelookup__ does describe quite well what the method 
does. It looks up on the type. The PEP is fairly clear on how this is 
supposed to work, e.g. the default type.__<whatever>__ method will look 
up in the class/type dict. PEP 447 includes an example of how you might 
implement this in Python:

class MetaType(type):
    def __locallookup__(cls, name):
        try:
            return cls.__dict__[name]
        except KeyError:
            raise AttributeError(name) from None


"local lookup" doesn't even come close to describing what the method 
does or why you would use it. It suggests something to do with locals, 
which is not the case. Neither does __getattributenorecurse__, which 
suggests looking up an attribute on an object without following the 
inheritance hierarchy, e.g. looking in the instance __dict__ but not the 
class __dict__. So the complete opposite of what it actually does.


-- 
Steven

From Steve.Dower at microsoft.com  Fri Sep 13 06:26:06 2013
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Fri, 13 Sep 2013 04:26:06 +0000
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <20130913035917.GE16820@ando>
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
 <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
 <522DECA3.2030709@hotpy.org>
 <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com>
 <522DF204.7040109@hotpy.org>
 <0752376899dd7436f6115bfc03594574@chopin.edu.pl>
 <51c253d972c94acbace9e46fc43b922a@BLUPR03MB199.namprd03.prod.outlook.com>,
 <20130913035917.GE16820@ando>
Message-ID: <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com>

Last I checked, looking up in the instance dict us exactly what it does. Even the example you posted is doing that. And the only difference from __getattribute__ is that it throws instead of following the MRO, which is intended to allow base classes (via super, and another call to this method) to dynamically respond to a getattr without the cooperation of subclasses.

Consider class A, which knows it has a method F, but will not create it until the first __getattribute__ call. Now class B derives from A, and someone calls super(B).F(obj). Currently, super only looks in __dict__ for F, which will fail to invoke A.__getattribute__. Because super is used to provide MRO traversal, it can't rely on B.__getattribute__ to perform the traversal, so it currently has no choice.

The PEP was originally adding a special class method to provide a __getattribute__ equivalent that would not traverse the MRO, so that super could use it and people can create dynamic classes that can act as base classes. I pointed out that this could be an instance method (thereby avoid automatic-classmethod magic) and implemented on a metaclass for the class behavior. Unless the PEP has changed recently, this is still an instance method that will look up members defined directly on the type and not on base classes.

There may still be valid questions to answer (such as, should overrides if this method on base classes be inherited), but whether it is a type/class method is no longer one of those.


Cheers,
Steve

Sent from my Windows Phone
________________________________
From: Steven D'Aprano<mailto:steve at pearwood.info>
Sent: ?9/?12/?2013 21:00
To: python-dev at python.org<mailto:python-dev at python.org>
Subject: Re: [Python-Dev] PEP 447: add type.__locallookup__

On Fri, Sep 13, 2013 at 03:04:49AM +0000, Steve Dower wrote:

> What about __getlocalattribute__ or __getattributenorecurse__? Long,
> but this isn't going to be used often.

This has nothing to do with locals, nor does it have anything to do with
recursion, so both those names are misleading.


> Putting "type" or "class" in the name would be misleading. It's an
> instance method (that is most useful when implemented on a metaclass).

Regardless of whether it is an instance method or not, by default it
performs the lookup on the type. Hence the C function _PyType_Lookup and
hence my suggestion __typelookup__.

But I think that __typelookup__ does describe quite well what the method
does. It looks up on the type. The PEP is fairly clear on how this is
supposed to work, e.g. the default type.__<whatever>__ method will look
up in the class/type dict. PEP 447 includes an example of how you might
implement this in Python:

class MetaType(type):
    def __locallookup__(cls, name):
        try:
            return cls.__dict__[name]
        except KeyError:
            raise AttributeError(name) from None


"local lookup" doesn't even come close to describing what the method
does or why you would use it. It suggests something to do with locals,
which is not the case. Neither does __getattributenorecurse__, which
suggests looking up an attribute on an object without following the
inheritance hierarchy, e.g. looking in the instance __dict__ but not the
class __dict__. So the complete opposite of what it actually does.


--
Steven
_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130913/fec6fd3c/attachment-0001.html>

From steve at pearwood.info  Fri Sep 13 08:52:38 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 13 Sep 2013 16:52:38 +1000
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com>
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
 <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
 <522DECA3.2030709@hotpy.org>
 <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com>
 <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl>
 <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com>
Message-ID: <20130913065236.GF16820@ando>

On Fri, Sep 13, 2013 at 04:26:06AM +0000, Steve Dower wrote:

> Last I checked, looking up in the instance dict us exactly what it 
> does. Even the example you posted is doing that.

The example from the PEP shows:

    return cls.__dict__[name]

not "self.__dict__[name]". It is true that "the instance" in this case 
refers to it being an instance of the metaclass, but that instance is, 
in fact, a class/type. That's why we normally call it "cls" in a 
metaclass method rather than "self".

I was reacting to your statement that [quote]Putting "type" or "class" 
in the name would be misleading[end quote]. I don't believe it is 
misleading. "type lookup" is exactly what it does: it does a lookup on a 
type. Take your example below:

> Consider class A, which knows it has a method F, but will not create 
> it until the first __getattribute__ call. Now class B derives from A, 
> and someone calls super(B).F(obj). 

[Aside: I'm not sure why you are using an unbound super object super(B) 
instead of the more usual super(B, obj).F(). Have I missed something?]

As I understand it, that ends up calling type(B).__<whatever>__(B, 'F'). 
So the objects being used are:

- the metaclass type(B);
- the class B

but not the instance self = B(), even though F is a regular instance 
method on A that ends up seeing self as the first argument.

Given this, I believe that "lookup on the type" is exactly what the 
method does, whether you interpret "the type" as the metaclass (the 
__<whatever>__ method is called on the metaclass) or the class B (which 
ends up as the first argument to the <whatever> method).


By the way, I think the PEP should have a more complex example. The 
SillyObject example is nice and easy to understand, but it doesn't 
really help with the motivating use-case "dynamic classes that can grow 
new methods on demand". Ronald, if you're reading this, can you add such 
an example please? Also, there's a typo in the SillyObject M method 
("fourtytwo" should not have a U in it).


[...]
> There may still be valid questions to answer (such as, should 
> overrides if this method on base classes be inherited), but whether it 
> is a type/class method is no longer one of those.

I don't believe that anyone is arguing that it should be a class method. 
I'm certainly not. But even if somebody is, that doesn't have anything 
to do with the name. We write dict.fromkeys(), not dict.typefromkeys().



-- 
Steven

From ncoghlan at gmail.com  Fri Sep 13 12:42:46 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 13 Sep 2013 20:42:46 +1000
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <20130913065236.GF16820@ando>
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
 <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
 <522DECA3.2030709@hotpy.org>
 <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com>
 <522DF204.7040109@hotpy.org>
 <0752376899dd7436f6115bfc03594574@chopin.edu.pl>
 <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com>
 <20130913065236.GF16820@ando>
Message-ID: <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com>

Perhaps "__getdescriptor__" would work as the method name? Yes, it can
technically return a non-descriptor, but the *primary* purpose is to
customise the retrieval of objects that will be checked to see if they're
descriptors. It *won't* be invoked when looking for ordinary attributes in
an instance dict, but *will* be invoked when looking on the class object.

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130913/ef922aa2/attachment.html>

From steve at pearwood.info  Fri Sep 13 14:23:28 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 13 Sep 2013 22:23:28 +1000
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com>
References: <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
 <522DECA3.2030709@hotpy.org>
 <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com>
 <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl>
 <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com>
 <20130913065236.GF16820@ando>
 <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com>
Message-ID: <20130913122328.GI16820@ando>

On Fri, Sep 13, 2013 at 08:42:46PM +1000, Nick Coghlan wrote:
> Perhaps "__getdescriptor__" would work as the method name? Yes, it can
> technically return a non-descriptor,

So technically that name is, um, what's the term... oh yes, "a lie". 

:-)


> but the *primary* purpose is to
> customise the retrieval of objects that will be checked to see if they're
> descriptors. 

If that's the case, the PEP should make that clear.

[Aside: the PEP states that the method shouldn't invoke descriptors. 
What's the reason for that? If I take the statement literally, doesn't 
it mean that the method mustn't use any other methods at all? Surely 
that can't be what is intended, but I'm not sure what is intended.]


> It *won't* be invoked when looking for ordinary attributes in
> an instance dict, but *will* be invoked when looking on the class object.

Just to be clear, if I have:

instance = MyClass()
x = instance.name

and "name" is found in instance.__dict__, then this special method will 
not be invoked. But if "name" is not found in the instance dict, then 
"name" will be looked up on the class object MyClass, which may invoke 
this special method. Am I correct?



-- 
Steven


From eric at trueblade.com  Fri Sep 13 16:49:05 2013
From: eric at trueblade.com (Eric V. Smith)
Date: Fri, 13 Sep 2013 10:49:05 -0400
Subject: [Python-Dev] [Python-checkins] cpython (merge 3.3 -> default):
 Fix http.server's request handling case on trailing '/'.
In-Reply-To: <3cbpGS1zZSz7Lk4@mail.python.org>
References: <3cbpGS1zZSz7Lk4@mail.python.org>
Message-ID: <523325E1.70608@trueblade.com>

On 9/13/2013 3:22 AM, senthil.kumaran wrote:
> http://hg.python.org/cpython/rev/b85c9d2a5227
> changeset:   85672:b85c9d2a5227
> parent:      85668:66ec8431032d
> parent:      85671:1fcccbbe15e2
> user:        Senthil Kumaran <senthil at uthcode.com>
> date:        Fri Sep 13 00:22:45 2013 -0700
> summary:
>   Fix http.server's request handling case on trailing '/'.
> 
> Patch contributed by Vajrasky Kok. Addresses Issue #17324

> +        trailing_slash = True if path.rstrip().endswith('/') else False

Wouldn't this be better just as:
          trailing_slash = path.rstrip().endswith('/')

-- 
Eric.

From status at bugs.python.org  Fri Sep 13 18:07:33 2013
From: status at bugs.python.org (Python tracker)
Date: Fri, 13 Sep 2013 18:07:33 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20130913160733.5F38B56A5E@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2013-09-06 - 2013-09-13)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    4212 (+17)
  closed 26571 (+48)
  total  30783 (+65)

Open issues with patches: 1935 


Issues opened (46)
==================

#18939: Venv docs regarding original python install
http://bugs.python.org/issue18939  reopened by gwideman

#18948: deliberately crashing tests should prevent core dumps
http://bugs.python.org/issue18948  opened by pitrou

#18950: Miscellaneous fixes for the sunau module
http://bugs.python.org/issue18950  opened by serhiy.storchaka

#18951: In unittest.TestCase.assertRegex change "re" and "regex" to "r
http://bugs.python.org/issue18951  opened by py.user

#18955: Confusing documentation in Lib/importlib/util.py
http://bugs.python.org/issue18955  opened by vajrasky

#18956: Document useful functions in ???pydoc??? module
http://bugs.python.org/issue18956  opened by bignose

#18958: Exception('No JSON object could be decoded') when parsing a va
http://bugs.python.org/issue18958  opened by Gallaecio

#18959: Create a "Superseded modules" section in standard library ToC
http://bugs.python.org/issue18959  opened by ncoghlan

#18960: First line can be executed twice
http://bugs.python.org/issue18960  opened by serhiy.storchaka

#18961: Non-UTF8 encoding line
http://bugs.python.org/issue18961  opened by serhiy.storchaka

#18965: 2to3 can produce illegal bytes literals
http://bugs.python.org/issue18965  opened by serhiy.storchaka

#18966: Threads within multiprocessing Process terminate early
http://bugs.python.org/issue18966  opened by pietvo

#18967: Find a less conflict prone approach to Misc/NEWS
http://bugs.python.org/issue18967  opened by ncoghlan

#18968: Find a way to detect incorrectly skipped tests
http://bugs.python.org/issue18968  opened by ncoghlan

#18969: test suite: enable faulthandler timeout in assert_python
http://bugs.python.org/issue18969  opened by neologix

#18970: run_setup() behavior differs from cli invocation of setup.py
http://bugs.python.org/issue18970  opened by l

#18971: Use argparse in the profile/cProfile modules
http://bugs.python.org/issue18971  opened by serhiy.storchaka

#18972: Use argparse in email example scripts
http://bugs.python.org/issue18972  opened by serhiy.storchaka

#18973: Use argparse in the calendar module
http://bugs.python.org/issue18973  opened by serhiy.storchaka

#18974: Use argparse in the diff script
http://bugs.python.org/issue18974  opened by serhiy.storchaka

#18975: timeit: Use thousands separators and print number of loops per
http://bugs.python.org/issue18975  opened by jstasiak

#18976: distutils/command/build_ext passes wrong linker flags
http://bugs.python.org/issue18976  opened by Benedikt.Morbach

#18977: The -t option has no effect in for uu command-line
http://bugs.python.org/issue18977  opened by serhiy.storchaka

#18978: Allow urllib.request.Request subclasses to override method
http://bugs.python.org/issue18978  opened by jason.coombs

#18979: Use argparse in the uu module
http://bugs.python.org/issue18979  opened by serhiy.storchaka

#18981: Typo in the ctypes tests
http://bugs.python.org/issue18981  opened by Anoop.Thomas.Mathew

#18982: Add tests for CLI of the calendar module
http://bugs.python.org/issue18982  opened by serhiy.storchaka

#18983: Specify time unit for timeit CLI
http://bugs.python.org/issue18983  opened by serhiy.storchaka

#18985: Improve the documentation in fcntl module
http://bugs.python.org/issue18985  opened by vajrasky

#18986: Add a case-insensitive case-preserving dict
http://bugs.python.org/issue18986  opened by pitrou

#18987: distutils.utils.get_platform() for 32-bit Python on	a	64-bit m
http://bugs.python.org/issue18987  opened by sferencik

#18989: reuse of enum names in class creation inconsistent
http://bugs.python.org/issue18989  opened by ethan.furman

#18990: Remove unnecessary API inconsistency from ElementTree.XMLPullP
http://bugs.python.org/issue18990  opened by scoder

#18993: There is an overshadowed and invalid test in testmock.py
http://bugs.python.org/issue18993  opened by vajrasky

#18994: Inside fcntl module, we does not check the return code of all_
http://bugs.python.org/issue18994  opened by vajrasky

#18995: Enum does not work with reversed
http://bugs.python.org/issue18995  opened by vajrasky

#18996: unittest: more helpful truncating long strings
http://bugs.python.org/issue18996  opened by serhiy.storchaka

#18998: iter() not working in ElementTree
http://bugs.python.org/issue18998  opened by Kronuz

#18999: Robustness issues in multiprocessing.{get,set}_start_method
http://bugs.python.org/issue18999  opened by larsmans

#19001: test_gdb fails on Fedora buildbot
http://bugs.python.org/issue19001  opened by pitrou

#19003: email.generator.BytesGenerator corrupts data by changing line 
http://bugs.python.org/issue19003  opened by Alexander.Kruppa

#19005: PyIter_Next crashes if passed a non-iterator
http://bugs.python.org/issue19005  opened by abacabadabacaba

#19006: UnitTest docs should have a single list of assertions
http://bugs.python.org/issue19006  opened by roysmith

#19007: precise time.time() under Windows 8
http://bugs.python.org/issue19007  opened by pitrou

#19009: Enhance HTTPResponse.readline() performance
http://bugs.python.org/issue19009  opened by kristjan.jonsson

#19010: Make XMLPullParser in ElementTree inherit from XMLParser
http://bugs.python.org/issue19010  opened by scoder



Most recent 15 issues with no replies (15)
==========================================

#19010: Make XMLPullParser in ElementTree inherit from XMLParser
http://bugs.python.org/issue19010

#19005: PyIter_Next crashes if passed a non-iterator
http://bugs.python.org/issue19005

#19003: email.generator.BytesGenerator corrupts data by changing line 
http://bugs.python.org/issue19003

#19001: test_gdb fails on Fedora buildbot
http://bugs.python.org/issue19001

#18996: unittest: more helpful truncating long strings
http://bugs.python.org/issue18996

#18994: Inside fcntl module, we does not check the return code of all_
http://bugs.python.org/issue18994

#18985: Improve the documentation in fcntl module
http://bugs.python.org/issue18985

#18982: Add tests for CLI of the calendar module
http://bugs.python.org/issue18982

#18981: Typo in the ctypes tests
http://bugs.python.org/issue18981

#18979: Use argparse in the uu module
http://bugs.python.org/issue18979

#18977: The -t option has no effect in for uu command-line
http://bugs.python.org/issue18977

#18976: distutils/command/build_ext passes wrong linker flags
http://bugs.python.org/issue18976

#18974: Use argparse in the diff script
http://bugs.python.org/issue18974

#18972: Use argparse in email example scripts
http://bugs.python.org/issue18972

#18971: Use argparse in the profile/cProfile modules
http://bugs.python.org/issue18971



Most recent 15 issues waiting for review (15)
=============================================

#19009: Enhance HTTPResponse.readline() performance
http://bugs.python.org/issue19009

#18999: Robustness issues in multiprocessing.{get,set}_start_method
http://bugs.python.org/issue18999

#18996: unittest: more helpful truncating long strings
http://bugs.python.org/issue18996

#18995: Enum does not work with reversed
http://bugs.python.org/issue18995

#18994: Inside fcntl module, we does not check the return code of all_
http://bugs.python.org/issue18994

#18993: There is an overshadowed and invalid test in testmock.py
http://bugs.python.org/issue18993

#18990: Remove unnecessary API inconsistency from ElementTree.XMLPullP
http://bugs.python.org/issue18990

#18989: reuse of enum names in class creation inconsistent
http://bugs.python.org/issue18989

#18986: Add a case-insensitive case-preserving dict
http://bugs.python.org/issue18986

#18985: Improve the documentation in fcntl module
http://bugs.python.org/issue18985

#18982: Add tests for CLI of the calendar module
http://bugs.python.org/issue18982

#18981: Typo in the ctypes tests
http://bugs.python.org/issue18981

#18979: Use argparse in the uu module
http://bugs.python.org/issue18979

#18978: Allow urllib.request.Request subclasses to override method
http://bugs.python.org/issue18978

#18977: The -t option has no effect in for uu command-line
http://bugs.python.org/issue18977



Top 10 most discussed issues (10)
=================================

#18986: Add a case-insensitive case-preserving dict
http://bugs.python.org/issue18986  46 msgs

#18943: argparse: default args in mutually exclusive groups
http://bugs.python.org/issue18943  12 msgs

#18999: Robustness issues in multiprocessing.{get,set}_start_method
http://bugs.python.org/issue18999  12 msgs

#17797: Visual C++ 11.0 reports fileno(stdin) == 0 for non-console pro
http://bugs.python.org/issue17797  10 msgs

#18967: Find a less conflict prone approach to Misc/NEWS
http://bugs.python.org/issue18967  10 msgs

#18975: timeit: Use thousands separators and print number of loops per
http://bugs.python.org/issue18975  10 msgs

#18987: distutils.utils.get_platform() for 32-bit Python on	a	64-bit m
http://bugs.python.org/issue18987  10 msgs

#5815: locale.getdefaultlocale() missing corner case
http://bugs.python.org/issue5815   9 msgs

#18829: csv produces confusing error message when passed a non-string 
http://bugs.python.org/issue18829   8 msgs

#1565525: tracebacks eat up memory by holding references to locals and g
http://bugs.python.org/issue1565525   8 msgs



Issues closed (46)
==================

#14927: add "Do not supply 'int' argument" to random.shuffle docstring
http://bugs.python.org/issue14927  closed by orsenthil

#14971: (unittest) loadTestsFromName does not work on method with a de
http://bugs.python.org/issue14971  closed by python-dev

#17324: SimpleHTTPServer serves files even if the URL has a trailing s
http://bugs.python.org/issue17324  closed by orsenthil

#18033: Example for Profile Module shows incorrect method
http://bugs.python.org/issue18033  closed by orsenthil

#18206: license url in site.py should always use X.Y.Z form of version
http://bugs.python.org/issue18206  closed by orsenthil

#18301: In itertools.chain.from_iterable() there is no cls argument
http://bugs.python.org/issue18301  closed by rhettinger

#18438: Obsolete url in comment inside decimal module
http://bugs.python.org/issue18438  closed by orsenthil

#18553: os.isatty() is not Unix only
http://bugs.python.org/issue18553  closed by orsenthil

#18623: Factor out the _SuppressCoreFiles context manager
http://bugs.python.org/issue18623  closed by pitrou

#18752: Make chain.from_iterable an alias for a new chain_iterable.
http://bugs.python.org/issue18752  closed by rhettinger

#18784: minor uuid.py loading optimization
http://bugs.python.org/issue18784  closed by serhiy.storchaka

#18800: Document Fraction's numerator and denominator properties
http://bugs.python.org/issue18800  closed by orsenthil

#18808: Thread.join returns before PyThreadState is destroyed
http://bugs.python.org/issue18808  closed by pitrou

#18815: DOCUMENTATION: "mmap .close()" doesn't close the underlying fi
http://bugs.python.org/issue18815  closed by orsenthil

#18818: Empty PYTHONIOENCODING is not the same as nonexistent
http://bugs.python.org/issue18818  closed by serhiy.storchaka

#18821: Add .lastitem attribute to takewhile instances
http://bugs.python.org/issue18821  closed by rhettinger

#18852: site.py does not handle readline.__doc__ being None
http://bugs.python.org/issue18852  closed by r.david.murray

#18894: In unittest.TestResult.failures remove deprecated fail* method
http://bugs.python.org/issue18894  closed by ezio.melotti

#18895: In unittest.TestResult.addError split the sentence
http://bugs.python.org/issue18895  closed by ezio.melotti

#18904: Unnecessary test in file descriptor inheritance test
http://bugs.python.org/issue18904  closed by haypo

#18908: Enum docs: sections leak out
http://bugs.python.org/issue18908  closed by python-dev

#18915: ssl.wrap_socket, pass in certfile and keyfile as PEM strings
http://bugs.python.org/issue18915  closed by asvetlov

#18917: python won't display greek characters in apache under windows
http://bugs.python.org/issue18917  closed by nickl1

#18934: multiprocessing: use selectors module
http://bugs.python.org/issue18934  closed by neologix

#18935: test_regrtest.test_timeout failure
http://bugs.python.org/issue18935  closed by neologix

#18944: Minor mistake in test_set.py
http://bugs.python.org/issue18944  closed by tim.peters

#18946: HTMLParser should ignore errors when parsing text in <script> 
http://bugs.python.org/issue18946  closed by ezio.melotti

#18947: Bug in module netaddr
http://bugs.python.org/issue18947  closed by r.david.murray

#18949: codeop possible flow error
http://bugs.python.org/issue18949  closed by serhiy.storchaka

#18952: Fix test.support.open_urlresource (support package regression)
http://bugs.python.org/issue18952  closed by python-dev

#18953: Typo in NEWS about fixed format specifiers for Py_ssize_t in d
http://bugs.python.org/issue18953  closed by python-dev

#18954: Grammatically incorrect sentences in Python/fileutils.c
http://bugs.python.org/issue18954  closed by python-dev

#18957: PYTHONFAULTHANDLER enables faulthandler when the variable is e
http://bugs.python.org/issue18957  closed by python-dev

#18962: Add special case for single iterator in heapq.merge function
http://bugs.python.org/issue18962  closed by rhettinger

#18963: test_selectors test_above_fd_setsize cases fail on OS X due to
http://bugs.python.org/issue18963  closed by neologix

#18964: test_tcl fails when _tkinter linked with Tcl 8.4
http://bugs.python.org/issue18964  closed by python-dev

#18980: Docs: enum - ReST format
http://bugs.python.org/issue18980  closed by python-dev

#18984: Remove .stopped Event from Thread internals
http://bugs.python.org/issue18984  closed by tim.peters

#18988: Tab key doesn't work sometimes.
http://bugs.python.org/issue18988  closed by serhiy.storchaka

#18991: Overriding OrderedDict.__getitem__() doesn't work
http://bugs.python.org/issue18991  closed by alex

#18992: test_sax fails on Windows under 3.4.0a2
http://bugs.python.org/issue18992  closed by loewis

#18997: Crash when using pickle and ElementTree
http://bugs.python.org/issue18997  closed by eli.bendersky

#19000: Alt + <Num> Does Not Work in IDLE.
http://bugs.python.org/issue19000  closed by Howitzer21

#19002: ``dir`` function does not work correctly with classes.
http://bugs.python.org/issue19002  closed by r.david.murray

#19004: datetime: Read in isoformat() output
http://bugs.python.org/issue19004  closed by r.david.murray

#19008: tkinter: UnboundLocalError: local variable 'sys' referenced be
http://bugs.python.org/issue19008  closed by Arfrever

From Steve.Dower at microsoft.com  Fri Sep 13 18:19:19 2013
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Fri, 13 Sep 2013 16:19:19 +0000
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <20130913065236.GF16820@ando>
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
 <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
 <522DECA3.2030709@hotpy.org>
 <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com>
 <522DF204.7040109@hotpy.org>
 <0752376899dd7436f6115bfc03594574@chopin.edu.pl>
 <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com>
 <20130913065236.GF16820@ando>
Message-ID: <f482c1d7f04f49d99616f3ca3c3b82da@BLUPR03MB199.namprd03.prod.outlook.com>

From: Steven D'Aprano
> On Fri, Sep 13, 2013 at 04:26:06AM +0000, Steve Dower wrote:
> 
>> Last I checked, looking up in the instance dict us exactly what it
>> does. Even the example you posted is doing that.
> 
> The example from the PEP shows:
> 
> return cls.__dict__[name]
> 
> not "self.__dict__[name]". It is true that "the instance" in this case refers to
> it being an instance of the metaclass, but that instance is, in fact, a
> class/type. That's why we normally call it "cls" in a metaclass method rather
> than "self".

Right, but where's the difference between these two?

class A:
    def __tdb__(self, name):
        if name == 'some_attribute_on_my_instance_of_A':
            return its_value
        try:
            return self.__dict__[name]
        except KeyError:
            raise AttributeError(name)

class MetaB:
    def __tdb__(cls, name):
        if name == 'some_attribute_on_my_class_B':
            return its_value
        try:
            return cls.__dict__[name]
        except KeyError:
            raise AttributeError(name)

(Yes, either of these could be written with __getattribute__, but that function cannot be called by super().)

As I see it, there are two (correct) ways to interpret what this method is for, which influences what it should be called.

1. It directly replaces obj.__dict__[name] everywhere that is done, including internally in the interpreter.
2. It is the same as __getattribute__ without the final call to object.__getattribute__

I guess it's also correct to view it as a special helper for super(), but it is more generally applicable than that.

[...]

> By the way, I think the PEP should have a more complex example. The SillyObject
> example is nice and easy to understand, but it doesn't really help with the
> motivating use-case "dynamic classes that can grow new methods on demand".
> Ronald, if you're reading this, can you add such an example please? Also,
> there's a typo in the SillyObject M method ("fourtytwo" should not have a U in
> it).

Agreed. No harm in more examples.

Here's a quick example of code that does not behave correctly at present.

class A:
    def __getattribute__(self, name):
        if name == 'foo':
            return 'A.foo'
        return object.__getattribute__(self, name)

class B(A):
    def get_foo(self):
        return super().foo

>>> B().get_foo()  # skips A.__getattribute__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in get_foo
AttributeError: 'super' object has no attribute 'foo'
>>> B().foo
A.foo

After changing to use the __tbd__ method:

class A:
    def __getattribute__(self, name):
        '''Not strictly necessary for this example, but I would expect
        that most types overriding __tbd__ also want to override
        __getattribute__ to use it. Or maybe object.__getattribute__
        should be changed to use __tbd__ too...?'''
        try:
            return self.__tbd__(name)
        except AttributeError:
            return object.__getattribute__(self, name)
    
    def __tbd__(self, name):        # CHANGED
        if name == 'foo':
            return 'A.foo'
        try:
            return self.__dict__[name]
        except KeyError:
            raise AttributeError(name)  # CHANGED

class B(A):
    def get_foo(self):
        return super().foo

>>> B().get_foo()  # does not skip A.__tbd__
A.foo              # hopefully this is the result :)
>>> B().foo
A.foo

A full example of where this may realistically be needed is longer and certainly involves metaclasses, but fundamentally it's just the same as __getattribute__ with slightly different semantics.

Cheers,
Steve

From solipsis at pitrou.net  Fri Sep 13 20:40:58 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 13 Sep 2013 20:40:58 +0200
Subject: [Python-Dev] PEP 455: TransformDict
Message-ID: <20130913204058.518f43d3@fsol>


Hello,

Following the python-dev discussion, I've written a PEP to recap the
proposal and the various arguments. It's inlined below, and it will
probably appear soon at http://www.python.org/dev/peps/pep-0455/, too.

Regards

Antoine.


PEP: 455
Title: Adding a key-transforming dictionary to collections
Version: $Revision$
Last-Modified: $Date$
Author: Antoine Pitrou <solipsis at pitrou.net>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 13-Sep-2013
Python-Version: 3.4
Post-History:


Abstract
========

This PEP proposes a new data structure for the ``collections`` module,
called "TransformDict" in this PEP.  This structure is a mutable mapping
which transforms the key using a given function when doing a lookup, but
retains the original key when reading.


Rationale
=========

Numerous specialized versions of this pattern exist.  The most common
is a case-insensitive case-preserving dict, i.e. a dict-like container
which matches keys in a case-insensitive fashion but retains the
original casing.  It is a very common need in network programming, as
many protocols feature some arrays of "key / value" properties in their
messages, where the keys are textual strings whose casing isn't
relevant.

Another common request is an identity dict, where keys are matched
according to their respective id()s instead of normal matching.

Both are instances of a more general pattern, where a given
transformation function is applied to keys when looking them up: that
function being ``str.lower`` in the former example and the built-in
``id`` function in the latter.

(it can be said that the pattern *projects* keys from the user-visible
set onto the internal lookup set, hence this PEP's title)


Semantics
=========

TransformDict is a ``MutableMapping`` implementation: it faithfully
implements the well-known API of mutable mappings, as ``dict`` itself
and other dict-like classes in the standard library.  Therefore, this
PEP won't rehash the semantics of most TransformDict methods.

The transformation function needn't be bijective, it can be strictly
surjective as in the case-insensitive example::

   >>> d = TransformDict(str.lower)
   >>> d['SomeKey'] = 5
   >>> d['somekey']
   5
   >>> d['SOMEKEY']
   5

TransformDict retains the first key used when creating an entry::

   >>> d = TransformDict(str.lower)
   >>> d['SomeKey'] = 1
   >>> d['somekey'] = 2
   >>> list(d.items())
   [('SomeKey', 2)]

The original keys needn't be hashable, as long as the transformation
function returns a hashable one::

   >>> d = TransformDict(id)
   >>> l = [None]
   >>> d[l] = 5
   >>> l in d
   True

Constructor
-----------

As shown in the example aboves, creating a TransformDict requires
passing the key transformation function as the first argument (much
like creating a ``defaultdict`` requires passing the factory function
as first argument).

The constructor also takes other optional arguments which can be used
to initialize the TransformDict with certain key-value pairs.  Those
optional arguments are the same as in the ``dict`` and ``defaultdict``
constructors::

   >>> d = TransformDict(str.lower, [('Foo': 1)], Bar=2)
   >>> sorted(d.items())
   [('Bar', 2), ('Foo', 1)]


Alternative proposals and questions
===================================

Retaining the last original key
-------------------------------

Most python-dev respondents found retaining the first user-supplied key
more intuitive than retaining the last.  Also, it matches the dict
object's own behaviour when using different but equal keys::

   >>> d = {}
   >>> d[1] = 'hello'
   >>> d[1.0] = 'world'
   >>> d
   {1: 'world'}

Furthermore, explicitly retaining the last key in a first-key-retaining
scheme is still possible using the following approach::

   d.pop(key, None)
   d[key] = value

while the converse (retaining the first key in a last-key-retaining
scheme) doesn't look possible without rewriting part of the container's
code.

Using an encoder / decoder pair
-------------------------------

Using a function pair isn't necessary, since the original key is
retained by the container.  Moreover, an encoder / decoder pair would
require the transformation to be bijective, which prevents important
use cases like case-insensitive matching.

Providing a transformation function for values
----------------------------------------------

Dictionary values are not used for lookup, their semantics are totally
irrelevant to the container's operation.  Therefore, there is no point
in having both an "original" and a "transformed" value: the transformed
value wouldn't be used for anything.

Providing a specialized container, not generic
----------------------------------------------

It was asked why we would provide the generic TransformDict construct
rather than a specialized case-insensitive dict variant.  The answer
is that it's nearly as cheap (code-wise and performance-wise) to provide
the generic construct, and it can fill more use cases.


Implementation
==============

A patch for the collections module is tracked on the bug tracker at
http://bugs.python.org/issue18986.


Existing work
=============

Case-insensitive dicts are a popular request:

*
  http://twistedmatrix.com/documents/current/api/twisted.python.util.InsensitiveDict.html
* https://mail.python.org/pipermail/python-list/2013-May/647243.html
* https://mail.python.org/pipermail/python-list/2005-April/296208.html
* https://mail.python.org/pipermail/python-list/2004-June/241748.html
* http://bugs.python.org/msg197376
* http://stackoverflow.com/a/2082169
* http://stackoverflow.com/a/3296782
* http://code.activestate.com/recipes/66315-case-insensitive-dictionary/
* https://gist.github.com/babakness/3901174
* http://www.wikier.org/blog/key-insensitive-dictionary-in-python
* http://en.sharejs.com/python/14534
* http://www.voidspace.org.uk/python/archive.shtml#caseless

Identity dicts have been requested too:

* https://mail.python.org/pipermail/python-ideas/2010-May/007235.html
* http://www.gossamer-threads.com/lists/python/python/209527

Python's own pickle module uses identity lookups for object
memoization:
http://hg.python.org/cpython/file/0e70bf1f32a3/Lib/pickle.py#l234


Copyright
=========

This document has been placed in the public domain.


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:



From storchaka at gmail.com  Fri Sep 13 21:31:02 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Fri, 13 Sep 2013 22:31:02 +0300
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <20130913204058.518f43d3@fsol>
References: <20130913204058.518f43d3@fsol>
Message-ID: <l0vp4p$3lj$1@ger.gmane.org>

13.09.13 21:40, Antoine Pitrou ???????(??):
> Both are instances of a more general pattern, where a given
> transformation function is applied to keys when looking them up: that
> function being ``str.lower`` in the former example and the built-in
> ``id`` function in the latter.

Please use str.casefold in examples.

>     >>> d = TransformDict(str.lower, [('Foo': 1)], Bar=2)

{'Foo': 1} or [('Foo', 1)].

> Providing a specialized container, not generic
> ----------------------------------------------
>
> It was asked why we would provide the generic TransformDict construct
> rather than a specialized case-insensitive dict variant.  The answer
> is that it's nearly as cheap (code-wise and performance-wise) to provide
> the generic construct, and it can fill more use cases.

Except lightweight IdentityDict which can be implemented more efficient 
than TransformDict(id). It doesn't need in calling the transform 
function, computing the hash of transformed key, comparing keys. But 
perhaps in many cases TransformDict(id) is enough.

> Python's own pickle module uses identity lookups for object
> memoization:
> http://hg.python.org/cpython/file/0e70bf1f32a3/Lib/pickle.py#l234

Also copy, json, cProfile, doctest and _threading_local.



From ethan at stoneleaf.us  Fri Sep 13 20:49:28 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 13 Sep 2013 11:49:28 -0700
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <20130913204058.518f43d3@fsol>
References: <20130913204058.518f43d3@fsol>
Message-ID: <52335E38.5010309@stoneleaf.us>

+1  :)


From solipsis at pitrou.net  Fri Sep 13 21:37:25 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 13 Sep 2013 21:37:25 +0200
Subject: [Python-Dev] PEP 455: TransformDict
References: <20130913204058.518f43d3@fsol>
	<l0vp4p$3lj$1@ger.gmane.org>
Message-ID: <20130913213725.19dd8817@fsol>

On Fri, 13 Sep 2013 22:31:02 +0300
Serhiy Storchaka <storchaka at gmail.com> wrote:
> 13.09.13 21:40, Antoine Pitrou ???????(??):
> > Both are instances of a more general pattern, where a given
> > transformation function is applied to keys when looking them up: that
> > function being ``str.lower`` in the former example and the built-in
> > ``id`` function in the latter.
> 
> Please use str.casefold in examples.
> 
> >     >>> d = TransformDict(str.lower, [('Foo': 1)], Bar=2)
> 
> {'Foo': 1} or [('Foo', 1)].

Ok, thanks.

> > Providing a specialized container, not generic
> > ----------------------------------------------
> >
> > It was asked why we would provide the generic TransformDict construct
> > rather than a specialized case-insensitive dict variant.  The answer
> > is that it's nearly as cheap (code-wise and performance-wise) to provide
> > the generic construct, and it can fill more use cases.
> 
> Except lightweight IdentityDict which can be implemented more efficient 
> than TransformDict(id). It doesn't need in calling the transform 
> function, computing the hash of transformed key, comparing keys.
> But 
> perhaps in many cases TransformDict(id) is enough.

That's true. But it's only important if TransformDict is the
bottleneck. I doubt the memoizing dictionary is a bottleneck in
e.g. the pure Python implementation of pickle or json.

> > Python's own pickle module uses identity lookups for object
> > memoization:
> > http://hg.python.org/cpython/file/0e70bf1f32a3/Lib/pickle.py#l234
> 
> Also copy, json, cProfile, doctest and _threading_local.

Thanks, will add them.

Regards

Antoine.



From jsbueno at python.org.br  Fri Sep 13 21:54:01 2013
From: jsbueno at python.org.br (Joao S. O. Bueno)
Date: Fri, 13 Sep 2013 16:54:01 -0300
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <20130913213725.19dd8817@fsol>
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
Message-ID: <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>

I see the PEP does not contemplate a way to retrieve the original key
- like we've talked about somewhere along the thread.

On 13 September 2013 16:37, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Fri, 13 Sep 2013 22:31:02 +0300
> Serhiy Storchaka <storchaka at gmail.com> wrote:
>> 13.09.13 21:40, Antoine Pitrou ???????(??):
>> > Both are instances of a more general pattern, where a given
>> > transformation function is applied to keys when looking them up: that
>> > function being ``str.lower`` in the former example and the built-in
>> > ``id`` function in the latter.
>>
>> Please use str.casefold in examples.
>>
>> >     >>> d = TransformDict(str.lower, [('Foo': 1)], Bar=2)
>>
>> {'Foo': 1} or [('Foo', 1)].
>
> Ok, thanks.
>
>> > Providing a specialized container, not generic
>> > ----------------------------------------------
>> >
>> > It was asked why we would provide the generic TransformDict construct
>> > rather than a specialized case-insensitive dict variant.  The answer
>> > is that it's nearly as cheap (code-wise and performance-wise) to provide
>> > the generic construct, and it can fill more use cases.
>>
>> Except lightweight IdentityDict which can be implemented more efficient
>> than TransformDict(id). It doesn't need in calling the transform
>> function, computing the hash of transformed key, comparing keys.
>> But
>> perhaps in many cases TransformDict(id) is enough.
>
> That's true. But it's only important if TransformDict is the
> bottleneck. I doubt the memoizing dictionary is a bottleneck in
> e.g. the pure Python implementation of pickle or json.
>
>> > Python's own pickle module uses identity lookups for object
>> > memoization:
>> > http://hg.python.org/cpython/file/0e70bf1f32a3/Lib/pickle.py#l234
>>
>> Also copy, json, cProfile, doctest and _threading_local.
>
> Thanks, will add them.
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/jsbueno%40python.org.br

From solipsis at pitrou.net  Fri Sep 13 22:02:36 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 13 Sep 2013 22:02:36 +0200
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
Message-ID: <20130913220236.1f27b73f@fsol>

On Fri, 13 Sep 2013 16:54:01 -0300
"Joao S. O. Bueno" <jsbueno at python.org.br> wrote:
> I see the PEP does not contemplate a way to retrieve the original key
> - like we've talked about somewhere along the thread.

Indeed. If that's important I can add it. I was hoping to keep very
close to the MutableMapping API, to make the PEP as few
controversial as possible.

Regards

Antoine.

From storchaka at gmail.com  Fri Sep 13 22:06:05 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Fri, 13 Sep 2013 23:06:05 +0300
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <20130913213725.19dd8817@fsol>
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
Message-ID: <l0vr6i$aso$1@ger.gmane.org>

13.09.13 22:37, Antoine Pitrou ???????(??):
> That's true. But it's only important if TransformDict is the
> bottleneck. I doubt the memoizing dictionary is a bottleneck in
> e.g. the pure Python implementation of pickle or json.

It can be used in the C implementation.



From rdmurray at bitdance.com  Fri Sep 13 22:09:10 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Fri, 13 Sep 2013 16:09:10 -0400
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <20130913204058.518f43d3@fsol>
References: <20130913204058.518f43d3@fsol>
Message-ID: <20130913200910.697972502EA@webabinitio.net>

On Fri, 13 Sep 2013 20:40:58 +0200, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Rationale
> =========
> 
> Numerous specialized versions of this pattern exist.  The most common
> is a case-insensitive case-preserving dict, i.e. a dict-like container
> which matches keys in a case-insensitive fashion but retains the
> original casing.  It is a very common need in network programming, as
> many protocols feature some arrays of "key / value" properties in their
> messages, where the keys are textual strings whose casing isn't
> relevant.

This motivation would be stronger if the last phrase was something like
"where the keys are textual strings whose case is specified to be ignored
on receipt but by either specification or custom is to be preserved
or non-trivially canonicalized when retransmitted."

> (it can be said that the pattern *projects* keys from the user-visible
> set onto the internal lookup set, hence this PEP's title)

Not clear what "projects" has to do with the PEP title.

> Constructor
> -----------
> 
> As shown in the example aboves, creating a TransformDict requires
> passing the key transformation function as the first argument (much
> like creating a ``defaultdict`` requires passing the factory function
> as first argument).
> 
> The constructor also takes other optional arguments which can be used
> to initialize the TransformDict with certain key-value pairs.  Those
> optional arguments are the same as in the ``dict`` and ``defaultdict``
> constructors::
> 
>    >>> d = TransformDict(str.lower, [('Foo': 1)], Bar=2)
>    >>> sorted(d.items())
>    [('Bar', 2), ('Foo', 1)]
> 
> 
> Alternative proposals and questions
> ===================================

You don't mention the alternate constructor discussion, or the
rationale for following the defaultdict pattern (ie: following the
established pattern :)

--David

From storchaka at gmail.com  Fri Sep 13 22:16:10 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Fri, 13 Sep 2013 23:16:10 +0300
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <20130913204058.518f43d3@fsol>
References: <20130913204058.518f43d3@fsol>
Message-ID: <l0vrpf$cso$1@ger.gmane.org>

13.09.13 21:40, Antoine Pitrou ???????(??):
> Alternative proposals and questions
> ===================================

Yet one alternative proposal is to add the dict.__transform__() method. 
Actually this not contradict TransformDict, but generalize it. 
TransformDict will be just handly interface to __transform__() as 
defaultdict to __missing__(). It provides only constructor, repr and 
pickling.

My second patch for http://bugs.python.org/issue18986 shows that as 
implementation of dict.__transform__(), so implementation of 
TransformDict using dict.__transform__() are simple enough.


From storchaka at gmail.com  Fri Sep 13 22:18:39 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Fri, 13 Sep 2013 23:18:39 +0300
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <20130913220236.1f27b73f@fsol>
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol>
Message-ID: <l0vrtv$cso$2@ger.gmane.org>

13.09.13 23:02, Antoine Pitrou ???????(??):
> On Fri, 13 Sep 2013 16:54:01 -0300
> "Joao S. O. Bueno" <jsbueno at python.org.br> wrote:
>> I see the PEP does not contemplate a way to retrieve the original key
>> - like we've talked about somewhere along the thread.
>
> Indeed. If that's important I can add it. I was hoping to keep very
> close to the MutableMapping API, to make the PEP as few
> controversial as possible.

I think that's important. As OrderectDict has additional methods besides 
the MutableMapping API, so TransformDict should provide useful 
specialized methods.


From solipsis at pitrou.net  Fri Sep 13 22:21:40 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 13 Sep 2013 22:21:40 +0200
Subject: [Python-Dev] PEP 455: TransformDict
References: <20130913204058.518f43d3@fsol>
	<l0vrpf$cso$1@ger.gmane.org>
Message-ID: <20130913222140.1aaa5742@fsol>

On Fri, 13 Sep 2013 23:16:10 +0300
Serhiy Storchaka <storchaka at gmail.com> wrote:
> 13.09.13 21:40, Antoine Pitrou ???????(??):
> > Alternative proposals and questions
> > ===================================
> 
> Yet one alternative proposal is to add the dict.__transform__() method. 
> Actually this not contradict TransformDict, but generalize it. 
> TransformDict will be just handly interface to __transform__() as 
> defaultdict to __missing__(). It provides only constructor, repr and 
> pickling.

Is it an alternative proposal or is it compatible with the PEP?
The PEP specifies the API, not the implementation.

Regards

Antoine.



From solipsis at pitrou.net  Fri Sep 13 22:22:51 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 13 Sep 2013 22:22:51 +0200
Subject: [Python-Dev] PEP 455: TransformDict
References: <20130913204058.518f43d3@fsol>
 <20130913200910.697972502EA@webabinitio.net>
Message-ID: <20130913222251.292d60ca@fsol>

On Fri, 13 Sep 2013 16:09:10 -0400
"R. David Murray" <rdmurray at bitdance.com> wrote:
> On Fri, 13 Sep 2013 20:40:58 +0200, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > Rationale
> > =========
> > 
> > Numerous specialized versions of this pattern exist.  The most common
> > is a case-insensitive case-preserving dict, i.e. a dict-like container
> > which matches keys in a case-insensitive fashion but retains the
> > original casing.  It is a very common need in network programming, as
> > many protocols feature some arrays of "key / value" properties in their
> > messages, where the keys are textual strings whose casing isn't
> > relevant.
> 
> This motivation would be stronger if the last phrase was something like
> "where the keys are textual strings whose case is specified to be ignored
> on receipt but by either specification or custom is to be preserved
> or non-trivially canonicalized when retransmitted."

Thanks, will add.

> > (it can be said that the pattern *projects* keys from the user-visible
> > set onto the internal lookup set, hence this PEP's title)
> 
> Not clear what "projects" has to do with the PEP title.

The PEP was originally titled "a key-projecting dictionary" and I've
changed it as it was too obscure.  I've now also removed the obsolete
part of the sentence above :-)

> > Alternative proposals and questions
> > ===================================
> 
> You don't mention the alternate constructor discussion, or the
> rationale for following the defaultdict pattern (ie: following the
> established pattern :)

Ok, will do that too.

Regards

Antoine.



From storchaka at gmail.com  Fri Sep 13 23:05:27 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 14 Sep 2013 00:05:27 +0300
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <20130913222140.1aaa5742@fsol>
References: <20130913204058.518f43d3@fsol> <l0vrpf$cso$1@ger.gmane.org>
 <20130913222140.1aaa5742@fsol>
Message-ID: <l0vuls$lp9$1@ger.gmane.org>

13.09.13 23:21, Antoine Pitrou ???????(??):
> On Fri, 13 Sep 2013 23:16:10 +0300
> Serhiy Storchaka <storchaka at gmail.com> wrote:
>> 13.09.13 21:40, Antoine Pitrou ???????(??):
>>> Alternative proposals and questions
>>> ===================================
>>
>> Yet one alternative proposal is to add the dict.__transform__() method.
>> Actually this not contradict TransformDict, but generalize it.
>> TransformDict will be just handly interface to __transform__() as
>> defaultdict to __missing__(). It provides only constructor, repr and
>> pickling.
>
> Is it an alternative proposal or is it compatible with the PEP?
> The PEP specifies the API, not the implementation.

Both. On one side, with this proposition TransformDict itself doesn't 
deserve PEP. It will be trivial and obvious thing. Not very important. 
On other side, TransformDict can be implemented even without 
dict.__transform__(), on pure Python.



From solipsis at pitrou.net  Fri Sep 13 23:20:25 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 13 Sep 2013 23:20:25 +0200
Subject: [Python-Dev] PEP 455: TransformDict
References: <20130913204058.518f43d3@fsol> <l0vrpf$cso$1@ger.gmane.org>
 <20130913222140.1aaa5742@fsol> <l0vuls$lp9$1@ger.gmane.org>
Message-ID: <20130913232025.781fe9a8@fsol>

On Sat, 14 Sep 2013 00:05:27 +0300
Serhiy Storchaka <storchaka at gmail.com> wrote:
> 13.09.13 23:21, Antoine Pitrou ???????(??):
> > On Fri, 13 Sep 2013 23:16:10 +0300
> > Serhiy Storchaka <storchaka at gmail.com> wrote:
> >> 13.09.13 21:40, Antoine Pitrou ???????(??):
> >>> Alternative proposals and questions
> >>> ===================================
> >>
> >> Yet one alternative proposal is to add the dict.__transform__() method.
> >> Actually this not contradict TransformDict, but generalize it.
> >> TransformDict will be just handly interface to __transform__() as
> >> defaultdict to __missing__(). It provides only constructor, repr and
> >> pickling.
> >
> > Is it an alternative proposal or is it compatible with the PEP?
> > The PEP specifies the API, not the implementation.
> 
> Both. On one side, with this proposition TransformDict itself doesn't 
> deserve PEP. It will be trivial and obvious thing.

Well, TransformDict would still be the user-visible API, not
__transform__; like defaultdict is the user-visible API, not
__missing__.

Regards

Antoine.



From rymg19 at gmail.com  Fri Sep 13 23:24:22 2013
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Fri, 13 Sep 2013 16:24:22 -0500
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <20130913204058.518f43d3@fsol>
References: <20130913204058.518f43d3@fsol>
Message-ID: <CAO41-mOx8bGJ+LqnExxwSb8FrBN6UoKaL_gsYRvTy-JRQ5jwCg@mail.gmail.com>

You know, I can think up several use cases for the case-insensitive idea.

Not sure if I missed something, but there should be a function to retrieve
the original key from the modified. i.e.:

>>> mydict['SomeStr'] = 5
>>> mydict.get_key('somestr')
'SomeStr'
>>>


On Fri, Sep 13, 2013 at 1:40 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

>
> Hello,
>
> Following the python-dev discussion, I've written a PEP to recap the
> proposal and the various arguments. It's inlined below, and it will
> probably appear soon at http://www.python.org/dev/peps/pep-0455/, too.
>
> Regards
>
> Antoine.
>
>
> PEP: 455
> Title: Adding a key-transforming dictionary to collections
> Version: $Revision$
> Last-Modified: $Date$
> Author: Antoine Pitrou <solipsis at pitrou.net>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 13-Sep-2013
> Python-Version: 3.4
> Post-History:
>
>
> Abstract
> ========
>
> This PEP proposes a new data structure for the ``collections`` module,
> called "TransformDict" in this PEP.  This structure is a mutable mapping
> which transforms the key using a given function when doing a lookup, but
> retains the original key when reading.
>
>
> Rationale
> =========
>
> Numerous specialized versions of this pattern exist.  The most common
> is a case-insensitive case-preserving dict, i.e. a dict-like container
> which matches keys in a case-insensitive fashion but retains the
> original casing.  It is a very common need in network programming, as
> many protocols feature some arrays of "key / value" properties in their
> messages, where the keys are textual strings whose casing isn't
> relevant.
>
> Another common request is an identity dict, where keys are matched
> according to their respective id()s instead of normal matching.
>
> Both are instances of a more general pattern, where a given
> transformation function is applied to keys when looking them up: that
> function being ``str.lower`` in the former example and the built-in
> ``id`` function in the latter.
>
> (it can be said that the pattern *projects* keys from the user-visible
> set onto the internal lookup set, hence this PEP's title)
>
>
> Semantics
> =========
>
> TransformDict is a ``MutableMapping`` implementation: it faithfully
> implements the well-known API of mutable mappings, as ``dict`` itself
> and other dict-like classes in the standard library.  Therefore, this
> PEP won't rehash the semantics of most TransformDict methods.
>
> The transformation function needn't be bijective, it can be strictly
> surjective as in the case-insensitive example::
>
>    >>> d = TransformDict(str.lower)
>    >>> d['SomeKey'] = 5
>    >>> d['somekey']
>    5
>    >>> d['SOMEKEY']
>    5
>
> TransformDict retains the first key used when creating an entry::
>
>    >>> d = TransformDict(str.lower)
>    >>> d['SomeKey'] = 1
>    >>> d['somekey'] = 2
>    >>> list(d.items())
>    [('SomeKey', 2)]
>
> The original keys needn't be hashable, as long as the transformation
> function returns a hashable one::
>
>    >>> d = TransformDict(id)
>    >>> l = [None]
>    >>> d[l] = 5
>    >>> l in d
>    True
>
> Constructor
> -----------
>
> As shown in the example aboves, creating a TransformDict requires
> passing the key transformation function as the first argument (much
> like creating a ``defaultdict`` requires passing the factory function
> as first argument).
>
> The constructor also takes other optional arguments which can be used
> to initialize the TransformDict with certain key-value pairs.  Those
> optional arguments are the same as in the ``dict`` and ``defaultdict``
> constructors::
>
>    >>> d = TransformDict(str.lower, [('Foo': 1)], Bar=2)
>    >>> sorted(d.items())
>    [('Bar', 2), ('Foo', 1)]
>
>
> Alternative proposals and questions
> ===================================
>
> Retaining the last original key
> -------------------------------
>
> Most python-dev respondents found retaining the first user-supplied key
> more intuitive than retaining the last.  Also, it matches the dict
> object's own behaviour when using different but equal keys::
>
>    >>> d = {}
>    >>> d[1] = 'hello'
>    >>> d[1.0] = 'world'
>    >>> d
>    {1: 'world'}
>
> Furthermore, explicitly retaining the last key in a first-key-retaining
> scheme is still possible using the following approach::
>
>    d.pop(key, None)
>    d[key] = value
>
> while the converse (retaining the first key in a last-key-retaining
> scheme) doesn't look possible without rewriting part of the container's
> code.
>
> Using an encoder / decoder pair
> -------------------------------
>
> Using a function pair isn't necessary, since the original key is
> retained by the container.  Moreover, an encoder / decoder pair would
> require the transformation to be bijective, which prevents important
> use cases like case-insensitive matching.
>
> Providing a transformation function for values
> ----------------------------------------------
>
> Dictionary values are not used for lookup, their semantics are totally
> irrelevant to the container's operation.  Therefore, there is no point
> in having both an "original" and a "transformed" value: the transformed
> value wouldn't be used for anything.
>
> Providing a specialized container, not generic
> ----------------------------------------------
>
> It was asked why we would provide the generic TransformDict construct
> rather than a specialized case-insensitive dict variant.  The answer
> is that it's nearly as cheap (code-wise and performance-wise) to provide
> the generic construct, and it can fill more use cases.
>
>
> Implementation
> ==============
>
> A patch for the collections module is tracked on the bug tracker at
> http://bugs.python.org/issue18986.
>
>
> Existing work
> =============
>
> Case-insensitive dicts are a popular request:
>
> *
>
> http://twistedmatrix.com/documents/current/api/twisted.python.util.InsensitiveDict.html
> * https://mail.python.org/pipermail/python-list/2013-May/647243.html
> * https://mail.python.org/pipermail/python-list/2005-April/296208.html
> * https://mail.python.org/pipermail/python-list/2004-June/241748.html
> * http://bugs.python.org/msg197376
> * http://stackoverflow.com/a/2082169
> * http://stackoverflow.com/a/3296782
> * http://code.activestate.com/recipes/66315-case-insensitive-dictionary/
> * https://gist.github.com/babakness/3901174
> * http://www.wikier.org/blog/key-insensitive-dictionary-in-python
> * http://en.sharejs.com/python/14534
> * http://www.voidspace.org.uk/python/archive.shtml#caseless
>
> Identity dicts have been requested too:
>
> * https://mail.python.org/pipermail/python-ideas/2010-May/007235.html
> * http://www.gossamer-threads.com/lists/python/python/209527
>
> Python's own pickle module uses identity lookups for object
> memoization:
> http://hg.python.org/cpython/file/0e70bf1f32a3/Lib/pickle.py#l234
>
>
> Copyright
> =========
>
> This document has been placed in the public domain.
>
>
> ..
>    Local Variables:
>    mode: indented-text
>    indent-tabs-mode: nil
>    sentence-end-double-space: t
>    fill-column: 70
>    coding: utf-8
>    End:
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com
>



-- 
Ryan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130913/6bda162d/attachment-0001.html>

From solipsis at pitrou.net  Fri Sep 13 23:45:16 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 13 Sep 2013 23:45:16 +0200
Subject: [Python-Dev] PEP 455: TransformDict
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
Message-ID: <20130913234516.10661214@fsol>

On Fri, 13 Sep 2013 23:18:39 +0300
Serhiy Storchaka <storchaka at gmail.com> wrote:
> 13.09.13 23:02, Antoine Pitrou ???????(??):
> > On Fri, 13 Sep 2013 16:54:01 -0300
> > "Joao S. O. Bueno" <jsbueno at python.org.br> wrote:
> >> I see the PEP does not contemplate a way to retrieve the original key
> >> - like we've talked about somewhere along the thread.
> >
> > Indeed. If that's important I can add it. I was hoping to keep very
> > close to the MutableMapping API, to make the PEP as few
> > controversial as possible.
> 
> I think that's important. As OrderectDict has additional methods besides 
> the MutableMapping API, so TransformDict should provide useful 
> specialized methods.

Ok, I have a better (IMO) proposal:

    >>> d = TransformDict(str.casefold, {'Foo': 1})
    >>> d.getitem('foo')
    ('Foo', 1)
    >>> d.getitem('bar')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    KeyError: 'bar'

Regards

Antoine.



From ethan at stoneleaf.us  Sat Sep 14 00:32:55 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 13 Sep 2013 15:32:55 -0700
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <20130913234516.10661214@fsol>
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol>
Message-ID: <52339297.3070203@stoneleaf.us>

On 09/13/2013 02:45 PM, Antoine Pitrou wrote:
> Serhiy Storchaka wrote:
>>
>> I think that's important. As OrderectDict has additional methods besides
>> the MutableMapping API, so TransformDict should provide useful
>> specialized methods.
>
> Ok, I have a better (IMO) proposal:
>
>      >>> d = TransformDict(str.casefold, {'Foo': 1})
>      >>> d.getitem('foo')
>      ('Foo', 1)
>      >>> d.getitem('bar')
>      Traceback (most recent call last):
>        File "<stdin>", line 1, in <module>
>      KeyError: 'bar'

-1

When I want the canonical key, I'm not going to care about the value; and if I do, it's easy enough to get.

--
~Ethan~

From steve at pearwood.info  Sat Sep 14 02:49:52 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 14 Sep 2013 10:49:52 +1000
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <20130913234516.10661214@fsol>
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol>
Message-ID: <20130914004952.GJ16820@ando>

On Fri, Sep 13, 2013 at 11:45:16PM +0200, Antoine Pitrou wrote:

> Ok, I have a better (IMO) proposal:
> 
>     >>> d = TransformDict(str.casefold, {'Foo': 1})
>     >>> d.getitem('foo')
>     ('Foo', 1)
>     >>> d.getitem('bar')
>     Traceback (most recent call last):
>       File "<stdin>", line 1, in <module>
>     KeyError: 'bar'


Is it more common to want both the canonical key and value at the same 
time, or to just want the canonical key? My gut feeling is that I'm 
likely to have code like this:


d = TransformDict(...)
for key in data:
    key = d.get_canonical(key)
    value = d[key]
    print("{}: {}".format(key, value))


in which case having a single call to return both will be great:

for key in data:
    key, value = d.getitem(key)
    print("{}: {}".format(key, value))


but I'm really not sure. Maybe Ethan is right.

I think these sorts of associated questions are why some people 
(Raymond, Nick) want to see the proposal live outside of the standard 
library for a while first. The general idea is great, but we're going to 
bike shed the API without having much idea of how it will actually be 
used.

So, my suggestion is this:

- Let's add __transform__ to dicts for 3.4, similar to __missing__, and 
see how people end up using it in the real world.

- Then, in 3.5, we can make a much better informed decision about the 
best API for a TransformedDict front-end to __transform__.


+1 on __transform__ method on dicts.

+0 on TransformedDict in 3.4

+1 on waiting for 3.5 based on experience on using __transform__.



-- 
Steven

From steve at pearwood.info  Sat Sep 14 02:59:21 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 14 Sep 2013 10:59:21 +1000
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <CAP7+vJJ6QxcUwcpdt3Mq6r07Q2dGUQt-VWGdsYGLv4wrRW5dGg@mail.gmail.com>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
 <CAP7+vJJ6QxcUwcpdt3Mq6r07Q2dGUQt-VWGdsYGLv4wrRW5dGg@mail.gmail.com>
Message-ID: <20130914005921.GK16820@ando>

On Sun, Sep 08, 2013 at 10:51:57AM -0700, Guido van Rossum wrote:
> Never mind, I found the patch and the issue. I really think that the
> *PEP* is ready for inclusion after the open issues are changed into
> something like Discussion or Future Work, and after adding a more
> prominent link to the issue with the patch. Then the *patch* can be
> reviewed some more until it is ready -- it looks very close already.

I've updated the PEP as requested. Is there anything further that needs 
to be done to have it approved?

http://www.python.org/dev/peps/pep-0450/



-- 
Steven

From solipsis at pitrou.net  Sat Sep 14 03:02:52 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 14 Sep 2013 03:02:52 +0200
Subject: [Python-Dev] PEP 455: TransformDict
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando>
Message-ID: <20130914030252.1b9f4378@fsol>

On Sat, 14 Sep 2013 10:49:52 +1000
Steven D'Aprano <steve at pearwood.info> wrote:
> 
> Is it more common to want both the canonical key and value at the same 
> time, or to just want the canonical key? My gut feeling is that I'm 
> likely to have code like this:
> 
> d = TransformDict(...)
> for key in data:
>     key = d.get_canonical(key)
>     value = d[key]
>     print("{}: {}".format(key, value))
> 
> in which case having a single call to return both will be great:
> 
> for key in data:
>     key, value = d.getitem(key)
>     print("{}: {}".format(key, value))
> 
> but I'm really not sure. Maybe Ethan is right.

I don't think it really matters. From getitem() it's trivial to extract
the key alone.

> I think these sorts of associated questions are why some people 
> (Raymond, Nick) want to see the proposal live outside of the standard 
> library for a while first. The general idea is great, but we're going to 
> bike shed the API without having much idea of how it will actually be 
> used.

Even after it's used, it will still be bikeshedded: such is how
proposals are generally discussed. There really isn't very much to
decide here, and whether we only return a key or a (key, value) pair is
almost cosmetic: both APIs are reasonably convenient.

> So, my suggestion is this:
> 
> - Let's add __transform__ to dicts for 3.4, similar to __missing__, and 
> see how people end up using it in the real world.

Well, __missing__ isn't used very much, I think. People use defaultdict.
(note that I'm not even proposing __transform__ right now :-))

> +1 on __transform__ method on dicts.
> 
> +0 on TransformedDict in 3.4
> 
> +1 on waiting for 3.5 based on experience on using __transform__.

Ok, thanks.

Regards

Antoine.



From solipsis at pitrou.net  Sat Sep 14 03:03:34 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 14 Sep 2013 03:03:34 +0200
Subject: [Python-Dev] PEP 455: TransformDict
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol> <52339297.3070203@stoneleaf.us>
Message-ID: <20130914030334.344cec78@fsol>

On Fri, 13 Sep 2013 15:32:55 -0700
Ethan Furman <ethan at stoneleaf.us> wrote:
> On 09/13/2013 02:45 PM, Antoine Pitrou wrote:
> > Serhiy Storchaka wrote:
> >>
> >> I think that's important. As OrderectDict has additional methods besides
> >> the MutableMapping API, so TransformDict should provide useful
> >> specialized methods.
> >
> > Ok, I have a better (IMO) proposal:
> >
> >      >>> d = TransformDict(str.casefold, {'Foo': 1})
> >      >>> d.getitem('foo')
> >      ('Foo', 1)
> >      >>> d.getitem('bar')
> >      Traceback (most recent call last):
> >        File "<stdin>", line 1, in <module>
> >      KeyError: 'bar'
> 
> -1
> 
> When I want the canonical key, I'm not going to care about the value; and if I do, it's easy enough to get.

But if you don't care about the value, the key is enough to get ;-)




From python at mrabarnett.plus.com  Sat Sep 14 03:25:42 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Sat, 14 Sep 2013 02:25:42 +0100
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <20130914004952.GJ16820@ando>
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando>
Message-ID: <5233BB16.8050201@mrabarnett.plus.com>

On 14/09/2013 01:49, Steven D'Aprano wrote:
> On Fri, Sep 13, 2013 at 11:45:16PM +0200, Antoine Pitrou wrote:
>
>> Ok, I have a better (IMO) proposal:
>>
>>     >>> d = TransformDict(str.casefold, {'Foo': 1})
>>     >>> d.getitem('foo')
>>     ('Foo', 1)
>>     >>> d.getitem('bar')
>>     Traceback (most recent call last):
>>       File "<stdin>", line 1, in <module>
>>     KeyError: 'bar'
>
>
> Is it more common to want both the canonical key and value at the same
> time, or to just want the canonical key? My gut feeling is that I'm
> likely to have code like this:
>
>
> d = TransformDict(...)
> for key in data:
>      key = d.get_canonical(key)
>      value = d[key]
>      print("{}: {}".format(key, value))
>
I think must be missing something. I thought that iterating over the
dict would yield the original keys, so if you wanted the original key
and value you would write:

for key, value in data.items():
     print("{}: {}".format(key, value))

and if you wanted the transformed key you would apply the transform
function to the key.

>
> in which case having a single call to return both will be great:
>
> for key in data:
>      key, value = d.getitem(key)
>      print("{}: {}".format(key, value))
>
>
> but I'm really not sure. Maybe Ethan is right.
>
> I think these sorts of associated questions are why some people
> (Raymond, Nick) want to see the proposal live outside of the standard
> library for a while first. The general idea is great, but we're going to
> bike shed the API without having much idea of how it will actually be
> used.
>
> So, my suggestion is this:
>
> - Let's add __transform__ to dicts for 3.4, similar to __missing__, and
> see how people end up using it in the real world.
>
> - Then, in 3.5, we can make a much better informed decision about the
> best API for a TransformedDict front-end to __transform__.
>
>
> +1 on __transform__ method on dicts.
>
> +0 on TransformedDict in 3.4
>
> +1 on waiting for 3.5 based on experience on using __transform__.
>


From ethan at stoneleaf.us  Sat Sep 14 03:00:18 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 13 Sep 2013 18:00:18 -0700
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <20130914004952.GJ16820@ando>
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando>
Message-ID: <5233B522.7070104@stoneleaf.us>

On 09/13/2013 05:49 PM, Steven D'Aprano wrote:
>
> +1 on __transform__ method on dicts.

What would __transform__ do?  Just canonicalize the key, or also save the original key?  How much front-end work will 
each user have to do to actually use this new magic method to good effect?

Personally, if there's a bunch of push-back against just adding TransformDict directly, why don't we make it 
provisional?  I thought that was what provisional was for (meaning: we're going to add it, PyPI is not really 
appropriate, there may be some API changes).

--
~Ethan~

From ethan at stoneleaf.us  Sat Sep 14 03:40:06 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 13 Sep 2013 18:40:06 -0700
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <5233BB16.8050201@mrabarnett.plus.com>
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando>
 <5233BB16.8050201@mrabarnett.plus.com>
Message-ID: <5233BE76.2040206@stoneleaf.us>

On 09/13/2013 06:25 PM, MRAB wrote:
> On 14/09/2013 01:49, Steven D'Aprano wrote:
>>
>> Is it more common to want both the canonical key and value at the same
>> time, or to just want the canonical key? My gut feeling is that I'm
>> likely to have code like this:
>>
>>
>> d = TransformDict(...)
>> for key in data:
>>      key = d.get_canonical(key)
>>      value = d[key]
>>      print("{}: {}".format(key, value))
>>
> I think I must be missing something. I thought that iterating over the
> dict would yield the original keys, so if you wanted the original key
> and value you would write:
>
> for key, value in data.items():
>      print("{}: {}".format(key, value))

Well, that's certainly how I would do it.  ;)


> and if you wanted the transformed key you would apply the transform
> function to the key.

Indeed.  The question is:  how?  It is entirely possible that your function has a TransformDict alone, and no memory of 
the transform function used to create the dict...

If the key transform function were saved directly on the TransformDict instance as, say, .transform_key, then problem 
solved.

--
~Ethan~

From python at mrabarnett.plus.com  Sat Sep 14 04:39:18 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Sat, 14 Sep 2013 03:39:18 +0100
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <5233BE76.2040206@stoneleaf.us>
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando>
 <5233BB16.8050201@mrabarnett.plus.com> <5233BE76.2040206@stoneleaf.us>
Message-ID: <5233CC56.8070608@mrabarnett.plus.com>

On 14/09/2013 02:40, Ethan Furman wrote:
> On 09/13/2013 06:25 PM, MRAB wrote:
>> On 14/09/2013 01:49, Steven D'Aprano wrote:
>>>
>>> Is it more common to want both the canonical key and value at the same
>>> time, or to just want the canonical key? My gut feeling is that I'm
>>> likely to have code like this:
>>>
>>>
>>> d = TransformDict(...)
>>> for key in data:
>>>      key = d.get_canonical(key)
>>>      value = d[key]
>>>      print("{}: {}".format(key, value))
>>>
>> I think I must be missing something. I thought that iterating over the
>> dict would yield the original keys, so if you wanted the original key
>> and value you would write:
>>
>> for key, value in data.items():
>>      print("{}: {}".format(key, value))
>
> Well, that's certainly how I would do it.  ;)
>
>
>> and if you wanted the transformed key you would apply the transform
>> function to the key.
>
> Indeed.  The question is:  how?  It is entirely possible that your function has a TransformDict alone, and no memory of
> the transform function used to create the dict...
>
> If the key transform function were saved directly on the TransformDict instance as, say, .transform_key, then problem
> solved.
>
defaultdict has .default_factory, so having something like
.transform_key would have the added advantage of consistency.


From steve at pearwood.info  Sat Sep 14 04:44:52 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 14 Sep 2013 12:44:52 +1000
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <5233B522.7070104@stoneleaf.us>
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando>
 <5233B522.7070104@stoneleaf.us>
Message-ID: <20130914024452.GL16820@ando>

On Fri, Sep 13, 2013 at 06:00:18PM -0700, Ethan Furman wrote:

> Personally, if there's a bunch of push-back against just adding 
> TransformDict directly, why don't we make it provisional?  I thought that 
> was what provisional was for (meaning: we're going to add it, PyPI is not 
> really appropriate, there may be some API changes).

Not according to PEP 411. It implies that only modules/packages can be 
provisional, not individual functions, and states that "most packages" 
are expected to be provisional. So either PEP 411 doesn't apply to 
TransformDict at all, or it applies by default. The PEP doesn't say.

http://www.python.org/dev/peps/pep-0411/


Everything below the line is about PEP 411, not TransformDict. If you 
don't care about PEP 411, you can stop reading now.


==========================


Personally, I think it's a poor PEP. It doesn't document opposition to 
the idea, and if I recall the discussion at the time correctly, there 
was plenty of opposition.

- Since people cannot rely on provisional features still being available 
in the future, if they care the slightest about forward compatibility, 
they dare not use them.

- If people do use them, and we care about backwards compatibility, we 
dare not remove provisional packages without going through the same 
deprecation process as for ordinary packages.

- It relies on people reading the documentation and noticing that a 
package is marked "provisional". Like that's going to happen.

None of these arguments are documented in the PEP, let alone refuted. 
Even if the decision to approve the PEP ends up being vindicated, I 
think it was poor form for the PEP to ignore arguments against.

I don't think that "provisional" helps end users at all. If anything, it 
hurts them -- it means more uncertainty and doubt. Packages may languish 
in provisional status indefinitely. Speaking as an end user, I used to 
know that once a feature hit the std lib, it was stable and wouldn't be 
removed without going through a lengthy period of deprecation (except 
under truly remarkable circumstances). But for provisional packages, 
that promise doesn't apply, and although PEP 411 says that packages 
won't be gratuitously removed, I have to assume that any provisional 
package in version 3.x could be gone without notice or warning in 3.x+1. 
I don't see how that helps me. It just means that for some indefinite 
period, there's a package in the std lib doing exactly what I want that 
I dare not use in production software because there are no stability 
guarantees.

The PEP has a lengthy section that claims to be about how it will help 
end users. It actually isn't -- it is about how inclusion in the 
standard library helps end users. Not surprisingly, there's nothing 
about why reserving the right to rip out a package without warning is 
good for end uers, since it isn't.

End of rant.


-- 
Steven

From steve at pearwood.info  Sat Sep 14 05:18:12 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 14 Sep 2013 13:18:12 +1000
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <5233BB16.8050201@mrabarnett.plus.com>
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando>
 <5233BB16.8050201@mrabarnett.plus.com>
Message-ID: <20130914031812.GM16820@ando>

On Sat, Sep 14, 2013 at 02:25:42AM +0100, MRAB wrote:
> On 14/09/2013 01:49, Steven D'Aprano wrote:

> >d = TransformDict(...)
> >for key in data:
> >     key = d.get_canonical(key)
> >     value = d[key]
> >     print("{}: {}".format(key, value))
> >
> I think must be missing something. I thought that iterating over the
> dict would yield the original keys, so if you wanted the original key
> and value you would write:
> 
> for key, value in data.items():
>     print("{}: {}".format(key, value))

You're missing that I'm not iterating over the entire dict, just some 
subset ("data") that I got from elsewhere. In real code, of course I 
would have to handle missing keys. My gut feeling is that normally I 
would want both the canonical key and the value, not just any valid key:

# not this
for key in data:
    value = d[key]
    print("{}: {}".format(key, value))

# but this
for key in data:
    key, value = d.getitem(key)
    print("{}: {}".format(key, value))

but that's just a gut feeling, and I daresay that it would depend on the 
application.


> and if you wanted the transformed key you would apply the transform
> function to the key.

The caller may not know or care what the transformation function is. The 
caller may only care about one or two things:

- does this key match a key in the dict?

- what is the canonical version of this key?

For example, in a case-preserving file system, the canonical version of 
a file name is whatever the user first typed when they created the file:

create file "SoMe FiLe"

so the canonical version is, "SoMe FiLe". But any of these may be 
expected to match:

some file
SOME FILE
Some File
some%20file

etc. It is *not* simply the case that the canonical version is "some 
file". Certainly that's not how case-preserving file systems work.



-- 
Steven

From steve at pearwood.info  Sat Sep 14 05:25:47 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 14 Sep 2013 13:25:47 +1000
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <5233B522.7070104@stoneleaf.us>
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando>
 <5233B522.7070104@stoneleaf.us>
Message-ID: <20130914032546.GN16820@ando>

On Fri, Sep 13, 2013 at 06:00:18PM -0700, Ethan Furman wrote:
> On 09/13/2013 05:49 PM, Steven D'Aprano wrote:
> >
> >+1 on __transform__ method on dicts.
> 
> What would __transform__ do?  Just canonicalize the key, or also save the 
> original key?  How much front-end work will each user have to do to 
> actually use this new magic method to good effect?

Sorry, I missed replying to this part.

See Serhiy's post:

https://mail.python.org/pipermail/python-dev/2013-September/128633.html

and suggested patch here:

http://bugs.python.org/issue18986




-- 
Steven

From steve at pearwood.info  Sat Sep 14 05:33:51 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 14 Sep 2013 13:33:51 +1000
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <5233BE76.2040206@stoneleaf.us>
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando>
 <5233BB16.8050201@mrabarnett.plus.com> <5233BE76.2040206@stoneleaf.us>
Message-ID: <20130914033351.GO16820@ando>

On Fri, Sep 13, 2013 at 06:40:06PM -0700, Ethan Furman wrote:

> >and if you wanted the transformed key you would apply the transform
> >function to the key.
> 
> Indeed.  The question is:  how?  It is entirely possible that your function 
> has a TransformDict alone, and no memory of the transform function used to 
> create the dict...
> 
> If the key transform function were saved directly on the TransformDict 
> instance as, say, .transform_key, then problem solved.

While I think it is good and proper to have the transformation function 
available, if you're manually applying the transformation function then 
IMO chances are good that you've missed the whole point of the 
transformdict.

Think of defaultdict. The purpose of defaultdict is to avoid needing to 
check whether the key is missing or not. If I suggested that the way 
to use a defaultdict was to write code like this:

value = d[key] if key in d else d.default_factory()

I'd be missing the point. Likewise, the point of transformdict is to 
avoid needing to care about the transformation function 99% of the 
time.



-- 
Steven

From ethan at stoneleaf.us  Sat Sep 14 06:45:48 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 13 Sep 2013 21:45:48 -0700
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <20130914033351.GO16820@ando>
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando>
 <5233BB16.8050201@mrabarnett.plus.com> <5233BE76.2040206@stoneleaf.us>
 <20130914033351.GO16820@ando>
Message-ID: <5233E9FC.10509@stoneleaf.us>

On 09/13/2013 08:33 PM, Steven D'Aprano wrote:
>
> Likewise, the point of transformdict is to avoid needing to care
> about the transformation function 99% of the time.

No one's arguing against that point.  It's the 1% of the time that being able to get the canonical name back is a Good 
Thing, and the best way to do that is to have the transform_key function easily available.

--
~Ethan~

From jsbueno at python.org.br  Sat Sep 14 06:53:20 2013
From: jsbueno at python.org.br (Joao S. O. Bueno)
Date: Sat, 14 Sep 2013 01:53:20 -0300
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <5233BE76.2040206@stoneleaf.us>
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando>
 <5233BB16.8050201@mrabarnett.plus.com>
 <5233BE76.2040206@stoneleaf.us>
Message-ID: <CAH0mxTQOkNm8zMEXD-jtLxnoru-ik7pRJ92sRe7dD463-Eh3gg@mail.gmail.com>

On 13 September 2013 22:40, Ethan Furman <ethan at stoneleaf.us> wrote:
> On 09/13/2013 06:25 PM, MRAB wrote:
>>
>> On 14/09/2013 01:49, Steven D'Aprano wrote:
>>>
>>>
>>> Is it more common to want both the canonical key and value at the same
>>> time, or to just want the canonical key? My gut feeling is that I'm
>>> likely to have code like this:
>>>
>>>
>>> d = TransformDict(...)
>>> for key in data:
>>>      key = d.get_canonical(key)
>>>      value = d[key]
>>>      print("{}: {}".format(key, value))
>>>
>> I think I must be missing something. I thought that iterating over the
>>
>> dict would yield the original keys, so if you wanted the original key
>> and value you would write:
>>
>> for key, value in data.items():
>>      print("{}: {}".format(key, value))
>
>
> Well, that's certainly how I would do it.  ;)

I hope you are aware that this pattern does not help when one  wants
 _one_ canonical key having a non-canonical one, besides having to linearly
walk through all keys and check the "__transform__"
to each one.

I mean - given no function to retrieve the canonical key,
one would have to resort to:

my_key = data.__transform__(given_key)
for key, value in data.items():
    if data.__transform__(key) == my_key:
        ....


WHich would defeat not only the purpose of a Transform dict, but the purpose
of a dict alltogether.

(of course, the one obvious way to do it would be to have the original key
stored along the value - which is just a bit less silly than the example above)


OTOH, having a `get_canonical_key` method or similar seens trivial enough-
if the only provided method retrieves the value as well, it would not
be that bad.




>
>
>
>> and if you wanted the transformed key you would apply the transform
>> function to the key.
>
>
> Indeed.  The question is:  how?  It is entirely possible that your function
> has a TransformDict alone, and no memory of the transform function used to
> create the dict...
>
> If the key transform function were saved directly on the TransformDict
> instance as, say, .transform_key, then problem solved.
>



> --
> ~Ethan~
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/jsbueno%40python.org.br

From g.brandl at gmx.net  Sat Sep 14 07:21:00 2013
From: g.brandl at gmx.net (Georg Brandl)
Date: Sat, 14 Sep 2013 07:21:00 +0200
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <20130913234516.10661214@fsol>
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol>
Message-ID: <l10rme$75n$1@ger.gmane.org>

On 09/13/2013 11:45 PM, Antoine Pitrou wrote:
> On Fri, 13 Sep 2013 23:18:39 +0300
> Serhiy Storchaka <storchaka at gmail.com> wrote:
>> 13.09.13 23:02, Antoine Pitrou ???????(??):
>> > On Fri, 13 Sep 2013 16:54:01 -0300
>> > "Joao S. O. Bueno" <jsbueno at python.org.br> wrote:
>> >> I see the PEP does not contemplate a way to retrieve the original key
>> >> - like we've talked about somewhere along the thread.
>> >
>> > Indeed. If that's important I can add it. I was hoping to keep very
>> > close to the MutableMapping API, to make the PEP as few
>> > controversial as possible.
>> 
>> I think that's important. As OrderectDict has additional methods besides 
>> the MutableMapping API, so TransformDict should provide useful 
>> specialized methods.
> 
> Ok, I have a better (IMO) proposal:
> 
>     >>> d = TransformDict(str.casefold, {'Foo': 1})
>     >>> d.getitem('foo')
>     ('Foo', 1)
>     >>> d.getitem('bar')
>     Traceback (most recent call last):
>       File "<stdin>", line 1, in <module>
>     KeyError: 'bar'

Except that getitem is very close to __getitem__, which will be confusing.
(Although it would be the correct name, and __getitem__ is the wrong one).

cheers,
Georg


From g.brandl at gmx.net  Sat Sep 14 07:24:27 2013
From: g.brandl at gmx.net (Georg Brandl)
Date: Sat, 14 Sep 2013 07:24:27 +0200
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <20130913200910.697972502EA@webabinitio.net>
References: <20130913204058.518f43d3@fsol>
 <20130913200910.697972502EA@webabinitio.net>
Message-ID: <l10rst$899$1@ger.gmane.org>

On 09/13/2013 10:09 PM, R. David Murray wrote:
> On Fri, 13 Sep 2013 20:40:58 +0200, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> Rationale
>> =========
>> 
>> Numerous specialized versions of this pattern exist.  The most common
>> is a case-insensitive case-preserving dict, i.e. a dict-like container
>> which matches keys in a case-insensitive fashion but retains the
>> original casing.  It is a very common need in network programming, as
>> many protocols feature some arrays of "key / value" properties in their
>> messages, where the keys are textual strings whose casing isn't
>> relevant.
> 
> This motivation would be stronger if the last phrase was something like
> "where the keys are textual strings whose case is specified to be ignored
> on receipt but by either specification or custom is to be preserved
> or non-trivially canonicalized when retransmitted."
> 
>> (it can be said that the pattern *projects* keys from the user-visible
>> set onto the internal lookup set, hence this PEP's title)
> 
> Not clear what "projects" has to do with the PEP title.

Clearly it should be called ProjectionDict!

cheers,
Georg



From g.brandl at gmx.net  Sat Sep 14 07:31:01 2013
From: g.brandl at gmx.net (Georg Brandl)
Date: Sat, 14 Sep 2013 07:31:01 +0200
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <20130913204058.518f43d3@fsol>
References: <20130913204058.518f43d3@fsol>
Message-ID: <l10s98$9mk$1@ger.gmane.org>

On 09/13/2013 08:40 PM, Antoine Pitrou wrote:
> 
> Hello,
> 
> Following the python-dev discussion, I've written a PEP to recap the
> proposal and the various arguments. It's inlined below, and it will
> probably appear soon at http://www.python.org/dev/peps/pep-0455/, too.
> 

Looks good to me.

Georg


From larry at hastings.org  Sat Sep 14 07:33:56 2013
From: larry at hastings.org (Larry Hastings)
Date: Sat, 14 Sep 2013 14:33:56 +0900
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <20130913204058.518f43d3@fsol>
References: <20130913204058.518f43d3@fsol>
Message-ID: <5233F544.7030604@hastings.org>

On 09/14/2013 03:40 AM, Antoine Pitrou wrote:
> Hello,
>
> Following the python-dev discussion, I've written a PEP to recap the
> proposal and the various arguments. It's inlined below, and it will
> probably appear soon at http://www.python.org/dev/peps/pep-0455/, too.

Whenever I read a discussion about the dict, I always wonder whether the 
same thing applies to a set.  Have you considered the utility of a 
TransformSet?  Or is it YAGNI?

Cheers,


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130914/c20f74ce/attachment.html>

From ethan at stoneleaf.us  Sat Sep 14 06:47:25 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 13 Sep 2013 21:47:25 -0700
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <20130914031812.GM16820@ando>
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando>
 <5233BB16.8050201@mrabarnett.plus.com> <20130914031812.GM16820@ando>
Message-ID: <5233EA5D.6060500@stoneleaf.us>

On 09/13/2013 08:18 PM, Steven D'Aprano wrote:
>
> You're missing that I'm not iterating over the entire dict, just some
> subset ("data") that I got from elsewhere.

Ah, okay.  Between you and Antoine I am convinced that .getitem() is a good thing.  So have that and .transform_key and 
we're golden!  :)

--
~Ethan~

From ncoghlan at gmail.com  Sat Sep 14 08:30:54 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 14 Sep 2013 16:30:54 +1000
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <20130913122328.GI16820@ando>
References: <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
 <522DECA3.2030709@hotpy.org>
 <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com>
 <522DF204.7040109@hotpy.org>
 <0752376899dd7436f6115bfc03594574@chopin.edu.pl>
 <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com>
 <20130913065236.GF16820@ando>
 <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com>
 <20130913122328.GI16820@ando>
Message-ID: <CADiSq7cdSEHryP+LMppRdabTuUG-GzMd56ub-oWpFNKPHNzsHA@mail.gmail.com>

On 13 September 2013 22:23, Steven D'Aprano <steve at pearwood.info> wrote:
> On Fri, Sep 13, 2013 at 08:42:46PM +1000, Nick Coghlan wrote:
>> Perhaps "__getdescriptor__" would work as the method name? Yes, it can
>> technically return a non-descriptor,
>
> So technically that name is, um, what's the term... oh yes, "a lie".
>
> :-)

In this case, "__getdescriptor__" means "I am looking for a
descriptor, please don't invoke the descriptor methods or traverse the
MRO, just return the raw object", not "you *must* give me a
descriptor".

The name suggestion comes from the fact that name bindings on the
instance and names bindings on the class are *different*, in that only
the latter participate in the descriptor protocol:

>>> class A:
...     def m(self): pass
...
>>> A.m
<function A.m at 0x7fb51ad63320>
>>> a = A()
>>> a.f = A.m
>>> a.m
<bound method A.m of <__main__.A object at 0x7fb51ad60550>>
>>> a.f
<function A.m at 0x7fb51ad63320>

It's the same function object underneath, so what's going on?

The trick is that only *types* get to play the descriptor game, where
the interpreter looks for __get__, __set__ and __delete__ on the
returned object and invokes them with found. Ordinary instances don't
have that behaviour - instead, they go through type(self) to look for
descriptors, and anything they find in the instance dictionary is
returned unaltered.

This difference in how attribute lookups are handled is actually the
most fundamental difference between normal class instances and classes
themselves (which are instances of metaclasses).

>> but the *primary* purpose is to
>> customise the retrieval of objects that will be checked to see if they're
>> descriptors.
>
> If that's the case, the PEP should make that clear.

Technically, that's what "Currently object.__getattribute__ and
super.__getattribute__ peek in the __dict__ of classes on the MRO for
a class when looking for an attribute." means.

However, I agree the current wording only conveys that to the handful
of people that already know exactly when in the attribute lookup
sequence that step occurs, which is a rather niche audience :)

It's also why I like __getdescriptor__ as a name - it's based on *why*
we're doing the lookup, rather than *how* we expect it to be done.

> [Aside: the PEP states that the method shouldn't invoke descriptors.
> What's the reason for that? If I take the statement literally, doesn't
> it mean that the method mustn't use any other methods at all? Surely
> that can't be what is intended, but I'm not sure what is intended.]

It means it shouldn't invoke __get__, __set__ or __delete__ on the
returned object, since that's the responsibility of the caller.

For example, there are times when a descriptor will be retrieved to
check for the presence of a __set__ or __delete__ method, but never
actually have its methods invoked because it is shadowed in the
instance dictionary:

>>> class Shadowable:
...     def __get__(self, *args):
...         print("Shadowable.__get__ called!")
...
>>> class Enforced:
...     def __get__(self, *args):
...         print("Enforced.__get__ called!")
...     def __set__(self, *args):
...         print("Enforced.__set__ called!")
...
>>> class Example:
...     s = Shadowable()
...     e = Enforced()
...     def __getattribute__(self, attr):
...         print("Retrieving {} from class".format(attr))
...         return super().__getattribute__(attr)
...
>>> x = Example()
>>> x.s
Retrieving s from class
Shadowable.__get__ called!
>>> x.s = 1
>>> x.s
Retrieving s from class
1

This is the key line: we retrieved 's' from the class, but *didn't*
invoke the __get__ method because it was shadowed in the instance
dictionary.

It works this way because *if* the descriptor defines __set__ or
__delete__, then Python will *ignore* the instance variable:

>>> x.e
Retrieving e from class
Enforced.__get__ called!
>>> x.__dict__["e"] = 1
Retrieving __dict__ from class
>>> x.e
Retrieving e from class
Enforced.__get__ called!

So my proposed name is based on the idea that what Ronald is after
with the PEP is a hook that *only* gets invoked when the interpreter
is doing this hunt for descriptors, but *not* for ordinary attribute
lookups.

>> It *won't* be invoked when looking for ordinary attributes in
>> an instance dict, but *will* be invoked when looking on the class object.
>
> Just to be clear, if I have:
>
> instance = MyClass()
> x = instance.name
>
> and "name" is found in instance.__dict__, then this special method will
> not be invoked. But if "name" is not found in the instance dict, then
> "name" will be looked up on the class object MyClass, which may invoke
> this special method. Am I correct?

Well, that's *my* proposal. While re-reading the current PEP, I
realised my suggested change actually goes quite a bit further than
just proposing a different name: unlike the current PEP, my advice is
that the new hook should NOT be invoked for instance attribute lookups
and should *not* replace looking directly into the class dict.
Instead, it would be the descriptor lookup counterpart to __getattr__:
whereas __getattr__ only fires for lookups on instances of the class,
__getdescriptor__ would only fire for lookups on the class itself (so,
almost exactly equivalent to defining __getattr__ on the metaclass,
but without needing a custom metaclass). A class that wanted to affect
both would be able to define __getdescriptor__ for the fallback lookup
on the class and then call it from __getattr__.

This also suggests to me that __getdescriptor__ should be an implicit
static method like __new__ (it would also be possible to make it an
implicit classmethod, but an implicit staticmethod would be more
consistent with the way __new__ works, so if we're going to have
implicit magic, it may as well be *consistent* implicit magic):

>>> class C:
...     def __new__(cls):
...         print(cls)
...         return super().__new__(cls)
...
>>> C()
<class '__main__.C'>
<__main__.C object at 0x7fb51ad60c50>
>>> class D(C): pass
...
>>> D()
<class '__main__.D'>
<__main__.D object at 0x7fb51ad60c90>
>>> C.__new__
<function C.__new__ at 0x7fb51ad638c0>
>>> C().__new__
<class '__main__.C'>
<function C.__new__ at 0x7fb51ad638c0>

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ethan at stoneleaf.us  Sat Sep 14 06:59:11 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 13 Sep 2013 21:59:11 -0700
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <CAH0mxTQOkNm8zMEXD-jtLxnoru-ik7pRJ92sRe7dD463-Eh3gg@mail.gmail.com>
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando>
 <5233BB16.8050201@mrabarnett.plus.com> <5233BE76.2040206@stoneleaf.us>
 <CAH0mxTQOkNm8zMEXD-jtLxnoru-ik7pRJ92sRe7dD463-Eh3gg@mail.gmail.com>
Message-ID: <5233ED1F.9020808@stoneleaf.us>

On 09/13/2013 09:53 PM, Joao S. O. Bueno wrote:
> On 13 September 2013 22:40, Ethan Furman <ethan at stoneleaf.us> wrote:
>> On 09/13/2013 06:25 PM, MRAB wrote:
>>>
>>> On 14/09/2013 01:49, Steven D'Aprano wrote:
>>>>
>>>>
>>>> Is it more common to want both the canonical key and value at the same
>>>> time, or to just want the canonical key? My gut feeling is that I'm
>>>> likely to have code like this:
>>>>
>>>>
>>>> d = TransformDict(...)
>>>> for key in data:
>>>>       key = d.get_canonical(key)
>>>>       value = d[key]
>>>>       print("{}: {}".format(key, value))
>>>>
>>> I think I must be missing something. I thought that iterating over the
>>>
>>> dict would yield the original keys, so if you wanted the original key
>>> and value you would write:
>>>
>>> for key, value in data.items():
>>>       print("{}: {}".format(key, value))
>>
>>
>> Well, that's certainly how I would do it.  ;)
>
> I hope you are aware that this pattern does not help when one  wants
>   _one_ canonical key having a non-canonical one [...]

True, but I was thinking Steve was talking about printing the entire dict, in which case that is, indeed, how I would do it.


> I mean - given no function to retrieve the canonical key,
> one would have to resort to:
>
> my_key = data.__transform__(given_key)
> for key, value in data.items():
>      if data.__transform__(key) == my_key:
>          ....

Which is exactly why I, and others, would like to have the transform function easily available.  Besides being able to 
use it to get a canonical key, one could use it to get the function itself.  Yay, introspection!

--
~Ethan~

From stephen at xemacs.org  Sat Sep 14 08:59:33 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 14 Sep 2013 15:59:33 +0900
Subject: [Python-Dev] "Provisional" considered harmful? [was: PEP 455:
	TransformDict]
In-Reply-To: <20130914024452.GL16820@ando>
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando>
 <5233B522.7070104@stoneleaf.us> <20130914024452.GL16820@ando>
Message-ID: <87fvt77usq.fsf@uwakimon.sk.tsukuba.ac.jp>

As will become evident, I disagree with Steven at almost every point.
However, I think his point about users not reading documentation is
well-taken.  Due to hyperlinking, users are very likely to skip past
module docstrings.  I suggest two (perhaps informal) additions to the
documentation policy in PEP 411:

    (1) Provisional packages should be explicitly described as such in
        release announcements.  I think this is done already, cf.
        "Improvements in email" in the Release announcement and What's
        New for Python 3.3.0.[1]  However the policy should be explicit
        and the word "provisional" should be emphasized and linked to
        PEP 411.

    (2) Individual APIs in those packages should be documented as
        provisional.

Steven D'Aprano writes:

 > - Since people cannot rely on provisional features still being available 
 > in the future, if they care the slightest about forward compatibility, 
 > they dare not use them.

You exaggerate to the extent that you harm your argument.  People make
tradeoffs.  Some will choose the extreme you describe here, others
will take more risk.  Your real argument is that the benefits are
small because you expect that provisional stdlib packages will not get
much if any additional use over PyPI packages.  Can't know without
trying ... thus, the PEP.  Similar considerations apply to your other
arguments.

 > None of these arguments are documented in the PEP, let alone refuted.

They're not real arguments, as stated.  If you rephrase them as actual
arguments, they turn out to merely be the "half-empty" phrasing of the
"half-full" arguments used in favor.  Can't know without trying.

 > I don't think that "provisional" helps end users at all. If anything, it 
 > hurts them -- it means more uncertainty and doubt.

True, it *increases* the uncertainty *inside* the stdlib.  On the flip
side, it decreases uncertainty *overall* by announcing that the core
developers *want* this feature in stdlib *now*, and probably the API
won't change *at all*, and very probably the API won't change "much".

 > But for provisional packages, that promise doesn't apply, and
 > although PEP 411 says that packages won't be gratuitously removed,
 > I have to assume that any provisional package in version 3.x could
 > be gone without notice or warning in 3.x+1.

It's unlikely it will be "gone", it will end up on PyPI.

 > I don't see how that helps me.

Since you evidently demand absolute certainty, you're right, it
doesn't help you with respect to any given provisional package.  I'm
more flexible, and I greatly value having the core developers evaluate
the modules on PyPI (or the modules that have only seen the
flourescent lights of Google Labs up to now) for me, and putting their
Good Codesmithing Stamp of Approval on some of them more quickly.

And if in fact the policy works in the sense that more code gets into
the stdlib (non-provisionally) faster than without the PEP, in the
long run *you* benefit even with if you adopt a policy of using only
non-provisional APIs.

 > The PEP has a lengthy section that claims to be about how it will
 > help end users. It actually isn't -- it is about how inclusion in
 > the standard library helps end users.

Right.  And this PEP is about how to get good code into the stdlib
faster.  It's a means to that intermediate end, and transitively a
means to the real end of helping end users.

 > Not surprisingly, there's nothing about why reserving the right to
 > rip out a package without warning is good for end uers, since it
 > isn't.

What it really comes down to is you're saying you fear you will
frequently disagree with the judgment of python-dev with regard to the
options provided by "provisional".  You want them bound by hard and
fast rules.  That's fine; everybody has their own requirements.
Nevertheless, it's logically possible that as long as you pay
attention to "provisional" markings this PEP will be a win for you.

I agree with the PEP proponents that "logically possible" can be
strengthened to "probably true", but of course YMMV.




Footnotes: 
[1]  http://www.python.org/download/releases/3.3.0/
     http://docs.python.org/3.3/whatsnew/3.3.html


From solipsis at pitrou.net  Sat Sep 14 12:27:06 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 14 Sep 2013 12:27:06 +0200
Subject: [Python-Dev] PEP 455: TransformDict
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando>
 <5233BB16.8050201@mrabarnett.plus.com>
 <5233BE76.2040206@stoneleaf.us>
 <CAH0mxTQOkNm8zMEXD-jtLxnoru-ik7pRJ92sRe7dD463-Eh3gg@mail.gmail.com>
 <5233ED1F.9020808@stoneleaf.us>
Message-ID: <20130914122706.6c1efbc7@fsol>

On Fri, 13 Sep 2013 21:59:11 -0700
Ethan Furman <ethan at stoneleaf.us> wrote:
> 
> > I mean - given no function to retrieve the canonical key,
> > one would have to resort to:
> >
> > my_key = data.__transform__(given_key)
> > for key, value in data.items():
> >      if data.__transform__(key) == my_key:
> >          ....
> 
> Which is exactly why I, and others, would like to have the transform function easily available.  Besides being able to 
> use it to get a canonical key, one could use it to get the function itself.  Yay, introspection!

Well, no, you misunderstand :) The transform function takes an
original key (perhaps "canonical") and returns the transformed key, it
can't do the reverse which is what getitem() does.  i.e.:

>>> d = TransformDict(str.lower)
>>> d['Foo'] = 5
>>> d._transform_func('Foo')
'foo'
>>> d.getitem('foo')
[('Foo', 5)]


What getitem() does is make the surjection bijective by restricting its
input domain to the set of stored keys.

Regards

Antoine.



From solipsis at pitrou.net  Sat Sep 14 12:30:46 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 14 Sep 2013 12:30:46 +0200
Subject: [Python-Dev] PEP 455: TransformDict
References: <20130913204058.518f43d3@fsol>
	<5233F544.7030604@hastings.org>
Message-ID: <20130914123046.4f053e6a@fsol>

On Sat, 14 Sep 2013 14:33:56 +0900
Larry Hastings <larry at hastings.org> wrote:
> On 09/14/2013 03:40 AM, Antoine Pitrou wrote:
> > Hello,
> >
> > Following the python-dev discussion, I've written a PEP to recap the
> > proposal and the various arguments. It's inlined below, and it will
> > probably appear soon at http://www.python.org/dev/peps/pep-0455/, too.
> 
> Whenever I read a discussion about the dict, I always wonder whether the 
> same thing applies to a set.  Have you considered the utility of a 
> TransformSet?  Or is it YAGNI?

Well, a TransformSet is like a normal dict, you just need to call the
transformation function yourself when inserting the keys.
i.e.:

  d = TransformSet(str.lower)
  d.add('Foo')

is the same conceptually as:

  d = {}
  d['Foo'.lower()] = 'Foo'
  d['foo']  # gets the original key

Regards

Antoine.



From ncoghlan at gmail.com  Sat Sep 14 18:25:46 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 15 Sep 2013 02:25:46 +1000
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <20130914024452.GL16820@ando>
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando>
 <5233B522.7070104@stoneleaf.us> <20130914024452.GL16820@ando>
Message-ID: <CADiSq7euQnhBdQAMJkv-c6-t+scevGNUA0Qg9ammZvthAnU6fA@mail.gmail.com>

On 14 September 2013 12:44, Steven D'Aprano <steve at pearwood.info> wrote:
> On Fri, Sep 13, 2013 at 06:00:18PM -0700, Ethan Furman wrote:
>
>> Personally, if there's a bunch of push-back against just adding
>> TransformDict directly, why don't we make it provisional?  I thought that
>> was what provisional was for (meaning: we're going to add it, PyPI is not
>> really appropriate, there may be some API changes).
>
> Not according to PEP 411. It implies that only modules/packages can be
> provisional, not individual functions, and states that "most packages"
> are expected to be provisional. So either PEP 411 doesn't apply to
> TransformDict at all, or it applies by default. The PEP doesn't say.
>
> http://www.python.org/dev/peps/pep-0411/
>
>
> Everything below the line is about PEP 411, not TransformDict. If you
> don't care about PEP 411, you can stop reading now.
>
>
> ==========================
>
>
> Personally, I think it's a poor PEP. It doesn't document opposition to
> the idea, and if I recall the discussion at the time correctly, there
> was plenty of opposition.
>
> - Since people cannot rely on provisional features still being available
> in the future, if they care the slightest about forward compatibility,
> they dare not use them.

Correct, that's exactly what they should do.

> - If people do use them, and we care about backwards compatibility, we
> dare not remove provisional packages without going through the same
> deprecation process as for ordinary packages.


> - It relies on people reading the documentation and noticing that a
> package is marked "provisional". Like that's going to happen.
>
> None of these arguments are documented in the PEP, let alone refuted.
> Even if the decision to approve the PEP ends up being vindicated, I
> think it was poor form for the PEP to ignore arguments against.
>
> I don't think that "provisional" helps end users at all. If anything, it
> hurts them -- it means more uncertainty and doubt. Packages may languish
> in provisional status indefinitely. Speaking as an end user, I used to
> know that once a feature hit the std lib, it was stable and wouldn't be
> removed without going through a lengthy period of deprecation (except
> under truly remarkable circumstances). But for provisional packages,
> that promise doesn't apply, and although PEP 411 says that packages
> won't be gratuitously removed, I have to assume that any provisional
> package in version 3.x could be gone without notice or warning in 3.x+1.
> I don't see how that helps me. It just means that for some indefinite
> period, there's a package in the std lib doing exactly what I want that
> I dare not use in production software because there are no stability
> guarantees.

Correct, it doesn't help you until it graduates from provisional
status. Less risk averse users may choose to use it sooner (since any
removed modules would almost certainly find a home on PyPI in fairly
short order)

> The PEP has a lengthy section that claims to be about how it will help
> end users. It actually isn't -- it is about how inclusion in the
> standard library helps end users. Not surprisingly, there's nothing
> about why reserving the right to rip out a package without warning is
> good for end uers, since it isn't.

"We expect that most of the API of most provisional packages will be
unchanged at graduation. Withdrawals are expected to be rare."

Cheers,
Nick.


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat Sep 14 18:27:32 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 15 Sep 2013 02:27:32 +1000
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <20130914024452.GL16820@ando>
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando>
 <5233B522.7070104@stoneleaf.us> <20130914024452.GL16820@ando>
Message-ID: <CADiSq7f22svN0UGpBATM-RyKQ3EV6nnE=QOtpdVd4pBdS+_QCw@mail.gmail.com>

On 14 September 2013 12:44, Steven D'Aprano <steve at pearwood.info> wrote:
> On Fri, Sep 13, 2013 at 06:00:18PM -0700, Ethan Furman wrote:
>
>> Personally, if there's a bunch of push-back against just adding
>> TransformDict directly, why don't we make it provisional?  I thought that
>> was what provisional was for (meaning: we're going to add it, PyPI is not
>> really appropriate, there may be some API changes).
>
> Not according to PEP 411. It implies that only modules/packages can be
> provisional, not individual functions, and states that "most packages"
> are expected to be provisional. So either PEP 411 doesn't apply to
> TransformDict at all, or it applies by default. The PEP doesn't say.
>
> http://www.python.org/dev/peps/pep-0411/
>
>
> Everything below the line is about PEP 411, not TransformDict. If you
> don't care about PEP 411, you can stop reading now.
>
>
> ==========================
>
>
> Personally, I think it's a poor PEP. It doesn't document opposition to
> the idea, and if I recall the discussion at the time correctly, there
> was plenty of opposition.

Oops, meant to reply to this part, too.

PEP 411 is an Informational PEP, not a standards track PEP. It's there
to describe the policy, not to make the case for *having* the policy.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From python at mrabarnett.plus.com  Sat Sep 14 19:16:29 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Sat, 14 Sep 2013 18:16:29 +0100
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <5233EA5D.6060500@stoneleaf.us>
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando>
 <5233BB16.8050201@mrabarnett.plus.com> <20130914031812.GM16820@ando>
 <5233EA5D.6060500@stoneleaf.us>
Message-ID: <523499ED.4030301@mrabarnett.plus.com>

On 14/09/2013 05:47, Ethan Furman wrote:
> On 09/13/2013 08:18 PM, Steven D'Aprano wrote:
>>
>> You're missing that I'm not iterating over the entire dict, just some
>> subset ("data") that I got from elsewhere.
>
> Ah, okay.  Between you and Antoine I am convinced that .getitem() is a good thing.  So have that and .transform_key and
> we're golden!  :)
>
+1



From ethan at stoneleaf.us  Sat Sep 14 18:43:13 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 14 Sep 2013 09:43:13 -0700
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <20130914122706.6c1efbc7@fsol>
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando>
 <5233BB16.8050201@mrabarnett.plus.com> <5233BE76.2040206@stoneleaf.us>
 <CAH0mxTQOkNm8zMEXD-jtLxnoru-ik7pRJ92sRe7dD463-Eh3gg@mail.gmail.com>
 <5233ED1F.9020808@stoneleaf.us> <20130914122706.6c1efbc7@fsol>
Message-ID: <52349221.10808@stoneleaf.us>

On 09/14/2013 03:27 AM, Antoine Pitrou wrote:
> On Fri, 13 Sep 2013 21:59:11 -0700
> Ethan Furman <ethan at stoneleaf.us> wrote:
>>
>>> I mean - given no function to retrieve the canonical key,
>>> one would have to resort to:
>>>
>>> my_key = data.__transform__(given_key)
>>> for key, value in data.items():
>>>       if data.__transform__(key) == my_key:
>>>           ....
>>
>> Which is exactly why I, and others, would like to have the transform function easily available.  Besides being able to
>> use it to get a canonical key, one could use it to get the function itself.  Yay, introspection!
>
> Well, no, you misunderstand :) The transform function takes an
> original key (perhaps "canonical") and returns the transformed key, it
> can't do the reverse which is what getitem() does.  i.e.:

Argh, of course you are right.

Still, I think it would be useful to expose the transform function.  Any good reason not to?

--
~Ethan~

From solipsis at pitrou.net  Sat Sep 14 19:41:31 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 14 Sep 2013 19:41:31 +0200
Subject: [Python-Dev] PEP 455: TransformDict
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando>
 <5233BB16.8050201@mrabarnett.plus.com>
 <5233BE76.2040206@stoneleaf.us>
 <CAH0mxTQOkNm8zMEXD-jtLxnoru-ik7pRJ92sRe7dD463-Eh3gg@mail.gmail.com>
 <5233ED1F.9020808@stoneleaf.us> <20130914122706.6c1efbc7@fsol>
 <52349221.10808@stoneleaf.us>
Message-ID: <20130914194131.232668c3@fsol>

On Sat, 14 Sep 2013 09:43:13 -0700
Ethan Furman <ethan at stoneleaf.us> wrote:
> On 09/14/2013 03:27 AM, Antoine Pitrou wrote:
> > On Fri, 13 Sep 2013 21:59:11 -0700
> > Ethan Furman <ethan at stoneleaf.us> wrote:
> >>
> >>> I mean - given no function to retrieve the canonical key,
> >>> one would have to resort to:
> >>>
> >>> my_key = data.__transform__(given_key)
> >>> for key, value in data.items():
> >>>       if data.__transform__(key) == my_key:
> >>>           ....
> >>
> >> Which is exactly why I, and others, would like to have the transform function easily available.  Besides being able to
> >> use it to get a canonical key, one could use it to get the function itself.  Yay, introspection!
> >
> > Well, no, you misunderstand :) The transform function takes an
> > original key (perhaps "canonical") and returns the transformed key, it
> > can't do the reverse which is what getitem() does.  i.e.:
> 
> Argh, of course you are right.
> 
> Still, I think it would be useful to expose the transform function.
> Any good reason not to?

No good reason. What's the name? transform_func?

Regards

Antoine.



From ethan at stoneleaf.us  Sat Sep 14 20:11:24 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 14 Sep 2013 11:11:24 -0700
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <20130914194131.232668c3@fsol>
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando>
 <5233BB16.8050201@mrabarnett.plus.com> <5233BE76.2040206@stoneleaf.us>
 <CAH0mxTQOkNm8zMEXD-jtLxnoru-ik7pRJ92sRe7dD463-Eh3gg@mail.gmail.com>
 <5233ED1F.9020808@stoneleaf.us> <20130914122706.6c1efbc7@fsol>
 <52349221.10808@stoneleaf.us> <20130914194131.232668c3@fsol>
Message-ID: <5234A6CC.1010900@stoneleaf.us>

On 09/14/2013 10:41 AM, Antoine Pitrou wrote:
> On Sat, 14 Sep 2013 09:43:13 -0700 Ethan Furman wrote:
>>
>> Still, I think it would be useful to expose the transform function.
>> Any good reason not to?
>
> No good reason. What's the name? transform_func?

I had originally thought transform_key, but transform_func is also good, and possibly less confusing.

--
~Ethan~

From storchaka at gmail.com  Sat Sep 14 21:07:50 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 14 Sep 2013 22:07:50 +0300
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <20130914194131.232668c3@fsol>
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando>
 <5233BB16.8050201@mrabarnett.plus.com> <5233BE76.2040206@stoneleaf.us>
 <CAH0mxTQOkNm8zMEXD-jtLxnoru-ik7pRJ92sRe7dD463-Eh3gg@mail.gmail.com>
 <5233ED1F.9020808@stoneleaf.us> <20130914122706.6c1efbc7@fsol>
 <52349221.10808@stoneleaf.us> <20130914194131.232668c3@fsol>
Message-ID: <l12c5a$m72$1@ger.gmane.org>

14.09.13 20:41, Antoine Pitrou ???????(??):
> On Sat, 14 Sep 2013 09:43:13 -0700
> Ethan Furman <ethan at stoneleaf.us> wrote:
>> Still, I think it would be useful to expose the transform function.
>> Any good reason not to?
>
> No good reason. What's the name? transform_func?

There is one reason -- serialization. For example pickle saves the 
transform function as an argument for TransformDict constructor. Repr 
exposes the transform function too (in evaluable representation). Other 
serializers need the transform function too. My implementations expose 
it as public property (transform), your -- as private attribute 
(_transform).

Or perhaps I misunderstood you?



From solipsis at pitrou.net  Sat Sep 14 21:12:25 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 14 Sep 2013 21:12:25 +0200
Subject: [Python-Dev] PEP 455: TransformDict
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando>
 <5233BB16.8050201@mrabarnett.plus.com>
 <5233BE76.2040206@stoneleaf.us>
 <CAH0mxTQOkNm8zMEXD-jtLxnoru-ik7pRJ92sRe7dD463-Eh3gg@mail.gmail.com>
 <5233ED1F.9020808@stoneleaf.us> <20130914122706.6c1efbc7@fsol>
 <52349221.10808@stoneleaf.us> <20130914194131.232668c3@fsol>
 <l12c5a$m72$1@ger.gmane.org>
Message-ID: <20130914211225.2d99fcb9@fsol>

On Sat, 14 Sep 2013 22:07:50 +0300
Serhiy Storchaka <storchaka at gmail.com> wrote:
> 14.09.13 20:41, Antoine Pitrou ???????(??):
> > On Sat, 14 Sep 2013 09:43:13 -0700
> > Ethan Furman <ethan at stoneleaf.us> wrote:
> >> Still, I think it would be useful to expose the transform function.
> >> Any good reason not to?
> >
> > No good reason. What's the name? transform_func?
> 
> There is one reason -- serialization. For example pickle saves the 
> transform function as an argument for TransformDict constructor. Repr 
> exposes the transform function too (in evaluable representation). Other 
> serializers need the transform function too. My implementations expose 
> it as public property (transform), your -- as private attribute 
> (_transform).
> 
> Or perhaps I misunderstood you?

"No good reason" not to expose it. It was a double negation, sorry ;-)

Regards

Antoine.



From tjreedy at udel.edu  Sat Sep 14 20:14:10 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 14 Sep 2013 14:14:10 -0400
Subject: [Python-Dev] [Python-checkins] cpython: Issue #18937: Add an
 assertLogs() context manager to unittest.TestCase to
In-Reply-To: <3cch2p3mLCz7LjT@mail.python.org>
References: <3cch2p3mLCz7LjT@mail.python.org>
Message-ID: <5234A772.4000800@udel.edu>

On 9/14/2013 1:45 PM, antoine.pitrou wrote:
> http://hg.python.org/cpython/rev/4f5815747f58
> changeset:   85701:4f5815747f58
> user:        Antoine Pitrou <solipsis at pitrou.net>
> date:        Sat Sep 14 19:45:47 2013 +0200
> summary:
>    Issue #18937: Add an assertLogs() context manager to unittest.TestCase to ensure that a block of code emits a message using the logging module.
>
> files:
>    Doc/library/unittest.rst       |   41 +++++++
>    Lib/unittest/case.py           |  109 +++++++++++++++++++-
>    Lib/unittest/test/test_case.py |   96 ++++++++++++++++++
>    Misc/NEWS                      |    2 +
>    4 files changed, 242 insertions(+), 6 deletions(-)
>
>
> diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst
> --- a/Doc/library/unittest.rst
> +++ b/Doc/library/unittest.rst
> @@ -1031,6 +1031,47 @@
>         .. versionchanged:: 3.3
>            Added the *msg* keyword argument when used as a context manager.

assertWarnsRegex

> +   .. method:: assertLogs(logger=None, level=None)
> +
> +      A context manager to test that at least one message is logged on
> +      the *logger* or one of its children, with at least the given
> +      *level*.
> +
> +      If given, *logger* should be a :class:`logging.Logger` object or a
> +      :class:`str` giving the name of a logger.  The default is the root
> +      logger, which will catch all messages.
> +
> +      If given, *level* should be either a numeric logging level or
> +      its string equivalent (for example either ``"ERROR"`` or
> +      :attr:`logging.ERROR`).  The default is :attr:`logging.INFO`.
> +
> +      The test passes if at least one message emitted inside the ``with``
> +      block matches the *logger* and *level* conditions, otherwise it fails.
> +
> +      The object returned by the context manager is a recording helper
> +      which keeps tracks of the matching log messages.  It has two
> +      attributes:
> +
> +      .. attribute:: records
> +
> +         A list of :class:`logging.LogRecord` objects of the matching
> +         log messages.
> +
> +      .. attribute:: output
> +
> +         A list of :class:`str` objects with the formatted output of
> +         matching messages.
> +
> +      Example::
> +
> +         with self.assertLogs('foo', level='INFO') as cm:
> +            logging.getLogger('foo').info('first message')
> +            logging.getLogger('foo.bar').error('second message')
> +         self.assertEqual(cm.output, ['INFO:foo:first message',
> +                                      'ERROR:foo.bar:second message'])
> +
> +      .. versionadded:: 3.4
> +
>
>      There are also other methods used to perform more specific checks, such as:

The new method should be added to the table of raises/warns methods that 
starts this group. The table intro sentence

"It is also possible to check that exceptions and warnings are raised 
using the following methods:"

should be modified to something like

"It is also possible to check the production of exception, warning, and 
log messages using the following methods:"

Terry



From guido at python.org  Sat Sep 14 21:31:36 2013
From: guido at python.org (Guido van Rossum)
Date: Sat, 14 Sep 2013 12:31:36 -0700
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <20130913204058.518f43d3@fsol>
References: <20130913204058.518f43d3@fsol>
Message-ID: <CAP7+vJJzzCOLz0WCX23=aZFDP4-_OWum1weV_-KjZs3=2CMC5A@mail.gmail.com>

On Fri, Sep 13, 2013 at 11:40 AM, Antoine Pitrou <solipsis at pitrou.net>wrote:

> Following the python-dev discussion, I've written a PEP to recap the
> proposal and the various arguments. It's inlined below, and it will
> probably appear soon at http://www.python.org/dev/peps/pep-0455/, too.
>

Thanks, Antoine!

Raymond Hettinger has volunteered to be the PEP dictator (is that the word
we use?) for this PEP.
-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130914/1dcc7fa9/attachment-0001.html>

From raymond.hettinger at gmail.com  Sat Sep 14 21:44:17 2013
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sat, 14 Sep 2013 12:44:17 -0700
Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4
Message-ID: <70C99F87-E9A5-4838-A1E9-4739FBF2E6A5@gmail.com>

I was exercising the alpha two release of 3.4 and noticed that
it is still being built under GCC 4.2.1.  

Is there any reason we have to use an old compiler?

I would like to see it built under the latest version of Clang
(like the other tools on the Mac) or under GCC 4.8.1.
I've better using the latter for many months.  It works great
and produces *much* better code (in terms of readability
for those of us who look at assembly output and in terms
of speed (approx 20-25% better for cpu bound code)).

I'm wondering whether is some issue that forces us to use
GCC 4.2.1 or whether that is just an artifact of continuing
to do what we've always been doing.


Raymond

From nad at acm.org  Sat Sep 14 22:32:04 2013
From: nad at acm.org (Ned Deily)
Date: Sat, 14 Sep 2013 13:32:04 -0700
Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4
References: <70C99F87-E9A5-4838-A1E9-4739FBF2E6A5@gmail.com>
Message-ID: <nad-EC5D57.13320314092013@news.gmane.org>

In article <70C99F87-E9A5-4838-A1E9-4739FBF2E6A5 at gmail.com>,
 Raymond Hettinger <raymond.hettinger at gmail.com> wrote:
> I was exercising the alpha two release of 3.4 and noticed that
> it is still being built under GCC 4.2.1.  
> 
> Is there any reason we have to use an old compiler?

Yes, kinda.  It's because the 64-bit/32-bit installer we supply is 
designed to run on multiple versions of OS X: 10.6 (Snow Leopard), 10.7, 
10.8, and the upcoming 10.9.  The safest way to ensure that the bits 
produced (executables, shared libs, and .so bundles) are linked with 
compatible versions of system shared libraries and frameworks is to 
build on the lowest supported system, 10.6, or - carefully - with a 10.6 
SDK on a newer system.  The standard compiler in the Apple Developer 
Tools for 10.6 is gcc 4.2.  Clang was very immature at that point.  The 
most recent Developer Tools for 10.8 and 10.7 systems, Xcode 4.6.x, have 
a mature clang but do not provide a 10.6 SDK.  Even with using an SDK, 
it's still possible to end up inadvertently linking with the wrong 
versions of system libraries.  We have been burned by that in the past.

> I would like to see it built under the latest version of Clang
> (like the other tools on the Mac) or under GCC 4.8.1.
> I've better using the latter for many months.  It works great
> and produces *much* better code (in terms of readability
> for those of us who look at assembly output and in terms
> of speed (approx 20-25% better for cpu bound code)).
> 
> I'm wondering whether is some issue that forces us to use
> GCC 4.2.1 or whether that is just an artifact of continuing
> to do what we've always been doing.

I'd like to move to Apple's latest clang, as well.  (Apple is dropping 
gcc totally from the next release of Xcode.)  There are a couple of 
approaches that may work and still reliably support multiple versions of 
OS X.  Investigating this is on my list of things to do prior to the 
next alpha.  I've opened a bug tracker issue for this (Issue19019).

-- 
 Ned Deily,
 nad at acm.org


From solipsis at pitrou.net  Sat Sep 14 22:36:09 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 14 Sep 2013 22:36:09 +0200
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <CAP7+vJJzzCOLz0WCX23=aZFDP4-_OWum1weV_-KjZs3=2CMC5A@mail.gmail.com>
References: <20130913204058.518f43d3@fsol>
 <CAP7+vJJzzCOLz0WCX23=aZFDP4-_OWum1weV_-KjZs3=2CMC5A@mail.gmail.com>
Message-ID: <20130914223609.4deceb6a@fsol>


On Sat, 14 Sep 2013 12:31:36 -0700
Guido van Rossum <guido at python.org> wrote:
> On Fri, Sep 13, 2013 at 11:40 AM, Antoine Pitrou <solipsis at pitrou.net>wrote:
> 
> > Following the python-dev discussion, I've written a PEP to recap the
> > proposal and the various arguments. It's inlined below, and it will
> > probably appear soon at http://www.python.org/dev/peps/pep-0455/, too.
> >
> 
> Thanks, Antoine!
> 
> Raymond Hettinger has volunteered to be the PEP dictator (is that the word
> we use?) for this PEP.

The discussion has settled down for the most part, and there is a
consensus amongst participants on the desireability of the feature and
its characteristics.  The implementation is straightforward pure Python
(Serhiy's C proposal should probably be a separate enhancement request
on the tracker).  I think the proposal will soon be ready for a
pronouncement - unless other concrete questions and suggestions are
recorded.

Thanks

Antoine.

From raymond.hettinger at gmail.com  Sat Sep 14 23:29:37 2013
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sat, 14 Sep 2013 14:29:37 -0700
Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4
In-Reply-To: <nad-EC5D57.13320314092013@news.gmane.org>
References: <70C99F87-E9A5-4838-A1E9-4739FBF2E6A5@gmail.com>
 <nad-EC5D57.13320314092013@news.gmane.org>
Message-ID: <B3293155-E4D5-4389-A555-C31BC49CE539@gmail.com>


On Sep 14, 2013, at 1:32 PM, Ned Deily <nad at acm.org> wrote:
>  The 
> most recent Developer Tools for 10.8 and 10.7 systems, Xcode 4.6.x, have 
> a mature clang but do not provide a 10.6 SDK.  Even with using an SDK, 
> it's still possible to end up inadvertently linking with the wrong 
> versions of system libraries.  We have been burned by that in the past.

I think we should offer a separate Mac build just for 10.6
(much like we do for the 32-bit PPC option for 10.5).

Otherwise, were trapped in an old world and using the same
reasoning as companies that still mandate Internet Explorer 6;
forgoing the benefits of newer browsers just to make sure
a few pieces of legacy code can be supported.

> 
> I'd like to move to Apple's latest clang, as well.  (Apple is dropping 
> gcc totally from the next release of Xcode.)  

That is an excellent reason for us to move as well.
Otherwise, we risk getting out of sync with the rest of the ecosystem.

> I've opened a bug tracker issue for this (Issue19019).

Thank you.

One thing that can be done is to take advantage of the 3.4 alphas
and betas to experiment with having both a legacy version and
fresh build with better tools.   That would let us find out early whether
the worries and fears are real and will manifest themselves in practice.

I've been building under GCC 4.8.1 for months and have encountered
no problems at all with the popular third-party modules or with linking
to existing SO files.


Raymond

From storchaka at gmail.com  Sat Sep 14 23:55:35 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sun, 15 Sep 2013 00:55:35 +0300
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <20130914194131.232668c3@fsol>
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando>
 <5233BB16.8050201@mrabarnett.plus.com> <5233BE76.2040206@stoneleaf.us>
 <CAH0mxTQOkNm8zMEXD-jtLxnoru-ik7pRJ92sRe7dD463-Eh3gg@mail.gmail.com>
 <5233ED1F.9020808@stoneleaf.us> <20130914122706.6c1efbc7@fsol>
 <52349221.10808@stoneleaf.us> <20130914194131.232668c3@fsol>
Message-ID: <l12m0b$9f6$1@ger.gmane.org>

14.09.13 20:41, Antoine Pitrou ???????(??):
> No good reason. What's the name? transform_func?

transform_func looks... truncated. Why not transform_function or trans_func?


From solipsis at pitrou.net  Sat Sep 14 23:58:44 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 14 Sep 2013 23:58:44 +0200
Subject: [Python-Dev] PEP 455: TransformDict
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando>
 <5233BB16.8050201@mrabarnett.plus.com>
 <5233BE76.2040206@stoneleaf.us>
 <CAH0mxTQOkNm8zMEXD-jtLxnoru-ik7pRJ92sRe7dD463-Eh3gg@mail.gmail.com>
 <5233ED1F.9020808@stoneleaf.us> <20130914122706.6c1efbc7@fsol>
 <52349221.10808@stoneleaf.us> <20130914194131.232668c3@fsol>
 <l12m0b$9f6$1@ger.gmane.org>
Message-ID: <20130914235844.08eacd96@fsol>

On Sun, 15 Sep 2013 00:55:35 +0300
Serhiy Storchaka <storchaka at gmail.com> wrote:
> 14.09.13 20:41, Antoine Pitrou ???????(??):
> > No good reason. What's the name? transform_func?
> 
> transform_func looks... truncated. Why not transform_function or trans_func?

The stdlib commonly uses "func" rather than "function". For example
"functools", where partial has an attribute named "func".
I suppose it's like "int" vs. "integer".

Regards

Antoine.



From eliben at gmail.com  Sun Sep 15 00:31:38 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Sat, 14 Sep 2013 15:31:38 -0700
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <l12m0b$9f6$1@ger.gmane.org>
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando>
 <5233BB16.8050201@mrabarnett.plus.com> <5233BE76.2040206@stoneleaf.us>
 <CAH0mxTQOkNm8zMEXD-jtLxnoru-ik7pRJ92sRe7dD463-Eh3gg@mail.gmail.com>
 <5233ED1F.9020808@stoneleaf.us> <20130914122706.6c1efbc7@fsol>
 <52349221.10808@stoneleaf.us> <20130914194131.232668c3@fsol>
 <l12m0b$9f6$1@ger.gmane.org>
Message-ID: <CAF-Rda_efmsrrU3+fws7orO6WpndE4XXS397HRb+OeXprGfY+A@mail.gmail.com>

On Sat, Sep 14, 2013 at 2:55 PM, Serhiy Storchaka <storchaka at gmail.com>wrote:

> 14.09.13 20:41, Antoine Pitrou ???????(??):
>
>> No good reason. What's the name? transform_func?
>>
>
> transform_func looks... truncated. Why not transform_function or
> trans_func?


transform_?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130914/ffcb6223/attachment.html>

From mark at hotpy.org  Sun Sep 15 00:54:58 2013
From: mark at hotpy.org (Mark Shannon)
Date: Sat, 14 Sep 2013 23:54:58 +0100
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <CAF-Rda_efmsrrU3+fws7orO6WpndE4XXS397HRb+OeXprGfY+A@mail.gmail.com>
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando>
 <5233BB16.8050201@mrabarnett.plus.com> <5233BE76.2040206@stoneleaf.us>
 <CAH0mxTQOkNm8zMEXD-jtLxnoru-ik7pRJ92sRe7dD463-Eh3gg@mail.gmail.com>
 <5233ED1F.9020808@stoneleaf.us> <20130914122706.6c1efbc7@fsol>
 <52349221.10808@stoneleaf.us> <20130914194131.232668c3@fsol>
 <l12m0b$9f6$1@ger.gmane.org>
 <CAF-Rda_efmsrrU3+fws7orO6WpndE4XXS397HRb+OeXprGfY+A@mail.gmail.com>
Message-ID: <5234E942.7090801@hotpy.org>

On 14/09/13 23:31, Eli Bendersky wrote:
>
>
>
> On Sat, Sep 14, 2013 at 2:55 PM, Serhiy Storchaka <storchaka at gmail.com
> <mailto:storchaka at gmail.com>> wrote:
>
>     14.09.13 20:41, Antoine Pitrou ???????(??):
>
>         No good reason. What's the name? transform_func?
>
>
>     transform_func looks... truncated. Why not transform_function or
>     trans_func?
>
>
> transform_?
>
Shouldn't that be transform_?? (PEP 3117)


From rymg19 at gmail.com  Sun Sep 15 00:56:06 2013
From: rymg19 at gmail.com (Ryan)
Date: Sat, 14 Sep 2013 17:56:06 -0500
Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4
In-Reply-To: <B3293155-E4D5-4389-A555-C31BC49CE539@gmail.com>
References: <70C99F87-E9A5-4838-A1E9-4739FBF2E6A5@gmail.com>
 <nad-EC5D57.13320314092013@news.gmane.org>
 <B3293155-E4D5-4389-A555-C31BC49CE539@gmail.com>
Message-ID: <534e7dd6-e7e2-49cd-b719-8a070852b623@email.android.com>

+1. A 10.6-only build makes sense.

If you aren't having problems with GCC 4.8, then Clang shouldn't give any trouble. Honestly, I still think Clang should be a compiler option in Windows distutils...

Raymond Hettinger <raymond.hettinger at gmail.com> wrote:

>
>On Sep 14, 2013, at 1:32 PM, Ned Deily <nad at acm.org> wrote:
>>  The 
>> most recent Developer Tools for 10.8 and 10.7 systems, Xcode 4.6.x,
>have 
>> a mature clang but do not provide a 10.6 SDK.  Even with using an
>SDK, 
>> it's still possible to end up inadvertently linking with the wrong 
>> versions of system libraries.  We have been burned by that in the
>past.
>
>I think we should offer a separate Mac build just for 10.6
>(much like we do for the 32-bit PPC option for 10.5).
>
>Otherwise, were trapped in an old world and using the same
>reasoning as companies that still mandate Internet Explorer 6;
>forgoing the benefits of newer browsers just to make sure
>a few pieces of legacy code can be supported.
>
>> 
>> I'd like to move to Apple's latest clang, as well.  (Apple is
>dropping 
>> gcc totally from the next release of Xcode.)  
>
>That is an excellent reason for us to move as well.
>Otherwise, we risk getting out of sync with the rest of the ecosystem.
>
>> I've opened a bug tracker issue for this (Issue19019).
>
>Thank you.
>
>One thing that can be done is to take advantage of the 3.4 alphas
>and betas to experiment with having both a legacy version and
>fresh build with better tools.   That would let us find out early
>whether
>the worries and fears are real and will manifest themselves in
>practice.
>
>I've been building under GCC 4.8.1 for months and have encountered
>no problems at all with the popular third-party modules or with linking
>to existing SO files.
>
>
>Raymond
>_______________________________________________
>Python-Dev mailing list
>Python-Dev at python.org
>https://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe:
>https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com

-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130914/2bdb2b26/attachment.html>

From storchaka at gmail.com  Sun Sep 15 02:32:05 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sun, 15 Sep 2013 03:32:05 +0300
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <20130914235844.08eacd96@fsol>
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando>
 <5233BB16.8050201@mrabarnett.plus.com> <5233BE76.2040206@stoneleaf.us>
 <CAH0mxTQOkNm8zMEXD-jtLxnoru-ik7pRJ92sRe7dD463-Eh3gg@mail.gmail.com>
 <5233ED1F.9020808@stoneleaf.us> <20130914122706.6c1efbc7@fsol>
 <52349221.10808@stoneleaf.us> <20130914194131.232668c3@fsol>
 <l12m0b$9f6$1@ger.gmane.org> <20130914235844.08eacd96@fsol>
Message-ID: <l12v58$gi7$1@ger.gmane.org>

15.09.13 00:58, Antoine Pitrou ???????(??):
> On Sun, 15 Sep 2013 00:55:35 +0300
> Serhiy Storchaka <storchaka at gmail.com> wrote:
>> 14.09.13 20:41, Antoine Pitrou ???????(??):
>>> No good reason. What's the name? transform_func?
>>
>> transform_func looks... truncated. Why not transform_function or trans_func?
>
> The stdlib commonly uses "func" rather than "function". For example
> "functools", where partial has an attribute named "func".
> I suppose it's like "int" vs. "integer".

The same functools has decorating_function() with the user_function 
argument.

Also find_function, search_function, create_function, isfunction, 
isgeneratorfunction, from_function, has_function, copy_function, 
pickle_function, register_function, emit_function, and print_function. 
This is not counting tests, IDLE and private names.


From ethan at stoneleaf.us  Sun Sep 15 02:57:18 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 14 Sep 2013 17:57:18 -0700
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <l12v58$gi7$1@ger.gmane.org>
References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org>
 <20130913213725.19dd8817@fsol>
 <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com>
 <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org>
 <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando>
 <5233BB16.8050201@mrabarnett.plus.com> <5233BE76.2040206@stoneleaf.us>
 <CAH0mxTQOkNm8zMEXD-jtLxnoru-ik7pRJ92sRe7dD463-Eh3gg@mail.gmail.com>
 <5233ED1F.9020808@stoneleaf.us> <20130914122706.6c1efbc7@fsol>
 <52349221.10808@stoneleaf.us> <20130914194131.232668c3@fsol>
 <l12m0b$9f6$1@ger.gmane.org> <20130914235844.08eacd96@fsol>
 <l12v58$gi7$1@ger.gmane.org>
Message-ID: <523505EE.1060306@stoneleaf.us>

On 09/14/2013 05:32 PM, Serhiy Storchaka wrote:
> 15.09.13 00:58, Antoine Pitrou ???????(??):
>> On Sun, 15 Sep 2013 00:55:35 +0300
>> Serhiy Storchaka <storchaka at gmail.com> wrote:
>>> 14.09.13 20:41, Antoine Pitrou ???????(??):
>>>> No good reason. What's the name? transform_func?
>>>
>>> transform_func looks... truncated. Why not transform_function or trans_func?
>>
>> The stdlib commonly uses "func" rather than "function". For example
>> "functools", where partial has an attribute named "func".
>> I suppose it's like "int" vs. "integer".
>
> The same functools has decorating_function() with the user_function argument.
>
> Also find_function, search_function, create_function, isfunction, isgeneratorfunction, from_function, has_function,
> copy_function, pickle_function, register_function, emit_function, and print_function. This is not counting tests, IDLE
> and private names.

The name of the parameter in __init__ is 'transform'.  We could just call it that.

--
~Ethan~

From larry at hastings.org  Sun Sep 15 06:56:26 2013
From: larry at hastings.org (Larry Hastings)
Date: Sun, 15 Sep 2013 13:56:26 +0900
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <20130914123046.4f053e6a@fsol>
References: <20130913204058.518f43d3@fsol> <5233F544.7030604@hastings.org>
 <20130914123046.4f053e6a@fsol>
Message-ID: <52353DFA.9060804@hastings.org>

On 09/14/2013 07:30 PM, Antoine Pitrou wrote:
> On Sat, 14 Sep 2013 14:33:56 +0900
> Larry Hastings <larry at hastings.org> wrote:
>> Whenever I read a discussion about the dict, I always wonder whether the
>> same thing applies to a set.  Have you considered the utility of a
>> TransformSet?  Or is it YAGNI?
> Well, a TransformSet is like a normal dict, you just need to call the
> transformation function yourself when inserting the keys.

s/normal dict/normal set/

But, then, a TransformDict is like a normal dict, you just need to call 
the transformation function yourself when inserting the keys. Yet a 
TransformDict is a useful enough concept that it is being proposed for 
addition; I was wondering if a TransformSet would be useful, too.  But I 
suppose we should take baby steps here.


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130915/0b8d4b83/attachment-0001.html>

From g.brandl at gmx.net  Sun Sep 15 10:34:04 2013
From: g.brandl at gmx.net (Georg Brandl)
Date: Sun, 15 Sep 2013 10:34:04 +0200
Subject: [Python-Dev] cpython: Issue #18571: Implementation of the PEP
 446: file descriptors and file handles
In-Reply-To: <3cPmKh2nMPz7LlL@mail.python.org>
References: <3cPmKh2nMPz7LlL@mail.python.org>
Message-ID: <l13rcd$23b$1@ger.gmane.org>

On 08/28/2013 01:20 AM, victor.stinner wrote:
> http://hg.python.org/cpython/rev/ef889c3d5dc6
> changeset:   85420:ef889c3d5dc6
> user:        Victor Stinner <victor.stinner at gmail.com>
> date:        Wed Aug 28 00:53:59 2013 +0200
> summary:
>   Issue #18571: Implementation of the PEP 446: file descriptors and file handles
> are now created non-inheritable; add functions os.get/set_inheritable(),
> os.get/set_handle_inheritable() and socket.socket.get/set_inheritable().

> +.. _fd_inheritance:
> +
> +Inheritance of File Descriptors
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +A file descriptor has a inheritable flag which indicates if the file descriptor
> +can be inherited or not in child processes. Since Python 3.4, file descriptors
> +created by Python are non-inheritable by default.
> +
> +On UNIX, non-inheritable file descriptors are closed in child processes at the
> +execution of a new program, other file descriptors are inherited.
> +
> +On Windows, non-inheritable handles and file descriptors are closed in child
> +processes, except standard streams (file descriptors 0, 1 and 2: stdin, stdout
> +and stderr) which are always inherited. Using :func:`os.spawn*` functions,
> +all inheritable handles and all inheritable file descriptors are inherited.
> +Using the :mod:`subprocess` module, all file descriptors except standard
> +streams are closed, inheritable handles are only inherited if the *close_fds*
> +parameter is ``False``.
> +
> +.. versionadded:: 3.4
> +
> +.. function:: get_inheritable(fd)
> +
> +   Get the `inheritable flag <fd_inheritance>`_ of the specified file
> +   descriptor. Return a :class:`bool`.
> +
> +.. function:: set_inheritable(fd, inheritable)
> +
> +   Set the `inheritable flag <fd_inheritance>`_ of the specified file descriptor.
> +
> +.. function:: get_handle_inheritable(handle)
> +
> +   Get the `inheritable flag <fd_inheritance>`_ of the specified handle. Return a :class:`bool`.
> +
> +   Availability: Windows.
> +
> +.. function:: set_handle_inheritable(handle, inheritable)
> +
> +   Set the `inheritable flag <fd_inheritance>`_ of the specified handle.
> +
> +   Availability: Windows.
> +

"Handle" is used nowhere else in the os module documentation.  Do you think it
would make sense to have a brief paragraph on what are possible handles under
Windows (and that file descriptors aren't handles)?

Georg


From solipsis at pitrou.net  Sun Sep 15 13:23:29 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 15 Sep 2013 13:23:29 +0200
Subject: [Python-Dev] PEP 455: TransformDict
References: <20130913204058.518f43d3@fsol> <5233F544.7030604@hastings.org>
 <20130914123046.4f053e6a@fsol> <52353DFA.9060804@hastings.org>
Message-ID: <20130915132329.5ea6adf6@fsol>

On Sun, 15 Sep 2013 13:56:26 +0900
Larry Hastings <larry at hastings.org> wrote:
> On 09/14/2013 07:30 PM, Antoine Pitrou wrote:
> > On Sat, 14 Sep 2013 14:33:56 +0900
> > Larry Hastings <larry at hastings.org> wrote:
> >> Whenever I read a discussion about the dict, I always wonder whether the
> >> same thing applies to a set.  Have you considered the utility of a
> >> TransformSet?  Or is it YAGNI?
> > Well, a TransformSet is like a normal dict, you just need to call the
> > transformation function yourself when inserting the keys.
> 
> s/normal dict/normal set/

No, read my example.




From storchaka at gmail.com  Sun Sep 15 15:14:24 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sun, 15 Sep 2013 16:14:24 +0300
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <20130915132329.5ea6adf6@fsol>
References: <20130913204058.518f43d3@fsol> <5233F544.7030604@hastings.org>
 <20130914123046.4f053e6a@fsol> <52353DFA.9060804@hastings.org>
 <20130915132329.5ea6adf6@fsol>
Message-ID: <l14bqq$cdk$1@ger.gmane.org>

15.09.13 14:23, Antoine Pitrou ???????(??):
> On Sun, 15 Sep 2013 13:56:26 +0900
> Larry Hastings <larry at hastings.org> wrote:
>> On 09/14/2013 07:30 PM, Antoine Pitrou wrote:
>>> On Sat, 14 Sep 2013 14:33:56 +0900
>>> Larry Hastings <larry at hastings.org> wrote:
>>>> Whenever I read a discussion about the dict, I always wonder whether the
>>>> same thing applies to a set.  Have you considered the utility of a
>>>> TransformSet?  Or is it YAGNI?
>>> Well, a TransformSet is like a normal dict, you just need to call the
>>> transformation function yourself when inserting the keys.
>>
>> s/normal dict/normal set/
>
> No, read my example.

A TransformDict is like a normal dict, you just need to call the 
transformation function yourself when inserting the keys and insert a 
pair of (orig_key, value), or is like two normal dicts, you just need to 
call the transformation function yourself when inserting the keys and 
update parallel mapping from transformed keys to original keys.


From solipsis at pitrou.net  Sun Sep 15 15:57:07 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 15 Sep 2013 15:57:07 +0200
Subject: [Python-Dev] PEP 455: TransformDict
References: <20130913204058.518f43d3@fsol> <5233F544.7030604@hastings.org>
 <20130914123046.4f053e6a@fsol> <52353DFA.9060804@hastings.org>
 <20130915132329.5ea6adf6@fsol> <l14bqq$cdk$1@ger.gmane.org>
Message-ID: <20130915155707.6ee9790d@fsol>

On Sun, 15 Sep 2013 16:14:24 +0300
Serhiy Storchaka <storchaka at gmail.com> wrote:
> 15.09.13 14:23, Antoine Pitrou ???????(??):
> > On Sun, 15 Sep 2013 13:56:26 +0900
> > Larry Hastings <larry at hastings.org> wrote:
> >> On 09/14/2013 07:30 PM, Antoine Pitrou wrote:
> >>> On Sat, 14 Sep 2013 14:33:56 +0900
> >>> Larry Hastings <larry at hastings.org> wrote:
> >>>> Whenever I read a discussion about the dict, I always wonder whether the
> >>>> same thing applies to a set.  Have you considered the utility of a
> >>>> TransformSet?  Or is it YAGNI?
> >>> Well, a TransformSet is like a normal dict, you just need to call the
> >>> transformation function yourself when inserting the keys.
> >>
> >> s/normal dict/normal set/
> >
> > No, read my example.
> 
> A TransformDict is like a normal dict, you just need to call the 
> transformation function yourself when inserting the keys and insert a 
> pair of (orig_key, value), or is like two normal dicts, you just need to 
> call the transformation function yourself when inserting the keys and 
> update parallel mapping from transformed keys to original keys.

I don't really care. What's the point in the end? TransformDict is
non-trivial to implement, while the so-called "TransformSet" is just a
dict with a different API.




From victor.stinner at gmail.com  Sun Sep 15 16:04:45 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sun, 15 Sep 2013 16:04:45 +0200
Subject: [Python-Dev] cpython: Issue #18571: Implementation of the PEP
 446: file descriptors and file handles
In-Reply-To: <l13rcd$23b$1@ger.gmane.org>
References: <3cPmKh2nMPz7LlL@mail.python.org>
	<l13rcd$23b$1@ger.gmane.org>
Message-ID: <CAMpsgwb+W_fOtjvPjHcGT-VgwJ34ZEHvMRiFEybSkQ18pnvtnw@mail.gmail.com>

Yes, but I'm not interested to write such doc.

Victor
Le 15 sept. 2013 10:34, "Georg Brandl" <g.brandl at gmx.net> a ?crit :

> On 08/28/2013 01:20 AM, victor.stinner wrote:
> > http://hg.python.org/cpython/rev/ef889c3d5dc6
> > changeset:   85420:ef889c3d5dc6
> > user:        Victor Stinner <victor.stinner at gmail.com>
> > date:        Wed Aug 28 00:53:59 2013 +0200
> > summary:
> >   Issue #18571: Implementation of the PEP 446: file descriptors and file
> handles
> > are now created non-inheritable; add functions os.get/set_inheritable(),
> > os.get/set_handle_inheritable() and socket.socket.get/set_inheritable().
>
> > +.. _fd_inheritance:
> > +
> > +Inheritance of File Descriptors
> > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > +
> > +A file descriptor has a inheritable flag which indicates if the file
> descriptor
> > +can be inherited or not in child processes. Since Python 3.4, file
> descriptors
> > +created by Python are non-inheritable by default.
> > +
> > +On UNIX, non-inheritable file descriptors are closed in child processes
> at the
> > +execution of a new program, other file descriptors are inherited.
> > +
> > +On Windows, non-inheritable handles and file descriptors are closed in
> child
> > +processes, except standard streams (file descriptors 0, 1 and 2: stdin,
> stdout
> > +and stderr) which are always inherited. Using :func:`os.spawn*`
> functions,
> > +all inheritable handles and all inheritable file descriptors are
> inherited.
> > +Using the :mod:`subprocess` module, all file descriptors except standard
> > +streams are closed, inheritable handles are only inherited if the
> *close_fds*
> > +parameter is ``False``.
> > +
> > +.. versionadded:: 3.4
> > +
> > +.. function:: get_inheritable(fd)
> > +
> > +   Get the `inheritable flag <fd_inheritance>`_ of the specified file
> > +   descriptor. Return a :class:`bool`.
> > +
> > +.. function:: set_inheritable(fd, inheritable)
> > +
> > +   Set the `inheritable flag <fd_inheritance>`_ of the specified file
> descriptor.
> > +
> > +.. function:: get_handle_inheritable(handle)
> > +
> > +   Get the `inheritable flag <fd_inheritance>`_ of the specified
> handle. Return a :class:`bool`.
> > +
> > +   Availability: Windows.
> > +
> > +.. function:: set_handle_inheritable(handle, inheritable)
> > +
> > +   Set the `inheritable flag <fd_inheritance>`_ of the specified handle.
> > +
> > +   Availability: Windows.
> > +
>
> "Handle" is used nowhere else in the os module documentation.  Do you
> think it
> would make sense to have a brief paragraph on what are possible handles
> under
> Windows (and that file descriptors aren't handles)?
>
> Georg
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130915/62cb1131/attachment.html>

From ethan at stoneleaf.us  Sun Sep 15 15:46:08 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 15 Sep 2013 06:46:08 -0700
Subject: [Python-Dev] PEP 428:  Pathlib
Message-ID: <5235BA20.8010105@stoneleaf.us>

I see PEP 428 is both targeted at 3.4 and still in draft status.

What remains to be done to ask for pronouncement?

--
~Ethan~

From ethan at stoneleaf.us  Sun Sep 15 16:19:32 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 15 Sep 2013 07:19:32 -0700
Subject: [Python-Dev] PEP 422: Simpler customisation of class creation
Message-ID: <5235C1F4.5070804@stoneleaf.us>

Three questions:


Would the new __initclass__ function be usable with actual full-blown metaclasses?


The PEP says this:
>
> If present on the created object, this new hook will be called by the class creation machinery after the __class__ reference has been initialised.

Given that statement, and this example:

     class A:
         def __initclass__(cls):
             # do stuff

     class B(A):
         pass

am I correct that A's __initclass__ will not run when B is created?


Where are we at with being ready for pronouncement?


--
~Ethan~

From ncoghlan at gmail.com  Sun Sep 15 16:40:48 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 16 Sep 2013 00:40:48 +1000
Subject: [Python-Dev] PEP 422: Simpler customisation of class creation
In-Reply-To: <5235C1F4.5070804@stoneleaf.us>
References: <5235C1F4.5070804@stoneleaf.us>
Message-ID: <CADiSq7f6kSYHdZqPuUsOSk8PR3bOf+TQxWQ8augBJjbx1PrDnA@mail.gmail.com>

On 16 September 2013 00:19, Ethan Furman <ethan at stoneleaf.us> wrote:
> Three questions:
>
>
> Would the new __initclass__ function be usable with actual full-blown
> metaclasses?

Sure (as long as the metaclass in question called it at the
appropriate time, which those inheriting from type would do
automatically)).

> The PEP says this:
>> If present on the created object, this new hook will be called by the
>> class creation machinery after the __class__ reference has been initialised.
>
>
> Given that statement, and this example:
>
>     class A:
>         def __initclass__(cls):
>             # do stuff
>
>     class B(A):
>         pass
>
> am I correct that A's __initclass__ will not run when B is created?

No, that would be pointless (since that's the way class decorators
already behave). "present on the created object" means "hasattr(cls,
'__initclass__') returns True.

> Where are we at with being ready for pronouncement?

The PEP needs to be updated to incorporate the feedback from the last
time I proposed it for inclusion. That's not going to happen for 3.4,
I just forgot to mark it as Deferred - fixed now.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From storchaka at gmail.com  Sun Sep 15 16:45:00 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sun, 15 Sep 2013 17:45:00 +0300
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <20130915155707.6ee9790d@fsol>
References: <20130913204058.518f43d3@fsol> <5233F544.7030604@hastings.org>
 <20130914123046.4f053e6a@fsol> <52353DFA.9060804@hastings.org>
 <20130915132329.5ea6adf6@fsol> <l14bqq$cdk$1@ger.gmane.org>
 <20130915155707.6ee9790d@fsol>
Message-ID: <l14h4d$5lq$1@ger.gmane.org>

15.09.13 16:57, Antoine Pitrou ???????(??):
> I don't really care. What's the point in the end? TransformDict is
> non-trivial to implement, while the so-called "TransformSet" is just a
> dict with a different API.

I don't see a difference. To me TransformDict is just a dict (or two).



From solipsis at pitrou.net  Sun Sep 15 16:52:07 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 15 Sep 2013 16:52:07 +0200
Subject: [Python-Dev] PEP 455: TransformDict
References: <20130913204058.518f43d3@fsol> <5233F544.7030604@hastings.org>
 <20130914123046.4f053e6a@fsol> <52353DFA.9060804@hastings.org>
 <20130915132329.5ea6adf6@fsol> <l14bqq$cdk$1@ger.gmane.org>
 <20130915155707.6ee9790d@fsol> <l14h4d$5lq$1@ger.gmane.org>
Message-ID: <20130915165207.6e4733bb@fsol>

On Sun, 15 Sep 2013 17:45:00 +0300
Serhiy Storchaka <storchaka at gmail.com> wrote:
> 15.09.13 16:57, Antoine Pitrou ???????(??):
> > I don't really care. What's the point in the end? TransformDict is
> > non-trivial to implement, while the so-called "TransformSet" is just a
> > dict with a different API.
> 
> I don't see a difference. To me TransformDict is just a dict (or two).

Well, feel free to propose a TransformSet PEP, but please do so in a
separate discussion thread.

Regards

Antoine.



From ncoghlan at gmail.com  Sun Sep 15 16:55:27 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 16 Sep 2013 00:55:27 +1000
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <l14h4d$5lq$1@ger.gmane.org>
References: <20130913204058.518f43d3@fsol> <5233F544.7030604@hastings.org>
 <20130914123046.4f053e6a@fsol> <52353DFA.9060804@hastings.org>
 <20130915132329.5ea6adf6@fsol> <l14bqq$cdk$1@ger.gmane.org>
 <20130915155707.6ee9790d@fsol> <l14h4d$5lq$1@ger.gmane.org>
Message-ID: <CADiSq7dayhKrabumYv6WbJrHfv_cL4p49uNXsK-oQ77iQc_F8w@mail.gmail.com>

On 16 September 2013 00:45, Serhiy Storchaka <storchaka at gmail.com> wrote:
> 15.09.13 16:57, Antoine Pitrou ???????(??):
>
>> I don't really care. What's the point in the end? TransformDict is
>> non-trivial to implement, while the so-called "TransformSet" is just a
>> dict with a different API.
>
>
> I don't see a difference. To me TransformDict is just a dict (or two).

I think it's best to hold off a TransformSet, since you can trivially
emulate it by setting the values in a TransformDict to None (that's
how Python itself lived without a set type for so long: people just
used dicts with all the values set to None).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From vitaly.murashev at gmail.com  Sun Sep 15 22:37:22 2013
From: vitaly.murashev at gmail.com (Vitaly Murashev)
Date: Mon, 16 Sep 2013 00:37:22 +0400
Subject: [Python-Dev] Relative path in co_filename for zipped modules
In-Reply-To: <CALPW6hzs1NLihojpbwtJE1YKCr6ggHawda8wusbWvNEM=5P2PA@mail.gmail.com>
References: <CALPW6hzs1NLihojpbwtJE1YKCr6ggHawda8wusbWvNEM=5P2PA@mail.gmail.com>
Message-ID: <CALPW6hwGHwB5Y=aVMmrbmin960awQadAH3fF8Zh2M3n5KE3_ww@mail.gmail.com>

This bug has been reported as http://bugs.python.org/issue18307 over a
month ago
?nd I've found out the root cause of this issue so far. Unit-test and patch
are suggested.
Without patch test fails, with patch - passed.



On Wed, Jun 26, 2013 at 12:11 AM, Vitaly Murashev <vitaly.murashev at gmail.com
> wrote:

> Dear Python developers community,
>
> Recently I found out that it not possible to debug python code if it
> is a part of zip-module.
> Python version being used is 3.3.0
>
> Well known GUI debuggers like Eclipse+PyDev or PyCharm are unable to
> start debugging and give the following warning:
> ---
> pydev debugger: CRITICAL WARNING: This version of python seems to be
> incorrectly compiled (internal generated filenames are not absolute)
> pydev debugger: The debugger may still function, but it will work
> slower and may miss breakpoints.
> ---
> So I started my own investigation of this issue and results are the
> following.
>
> At first I took traditional python debugger 'pdb' to analyze how it
> behaves during debugging of zipped module.
> 'pdb' showed me some backtaces and filename part for stack entries
> looks malformed.
> I expected something like
> 'full-path-to-zip-dir/my_zipped_module.zip/subdir/test_module.py'
> but realy it looks like 'full-path-to-current-dir/subdir/test_module.py'
>
> Source code in pdb.py and bdb.py (which one are a part of python
> stdlib) gave me the answer why it happens.
>
> The root cause are inside Bdb.format_stack_entry() + Bdb.canonic()
>
> Please take a look at the following line inside 'format_stack_entry'
> method:
>
>     filename = self.canonic(frame.f_code.co_filename)
>
> For zipped module variable 'frame.f_code.co_filename' holds _relative_
> file path started from the root of zip archive like
> 'subdir/test_module.py'
> And as result Bdb.canonic() method gives what we have -
> 'full-path-to-current-dir/subdir/test_module.py'
> ---
> So my final question.
> Could anyone confirm that it is a bug in python core subsystem which
> one is responsible for loading zipped modules,
> or something is wrong with my zipped module ?
>
> If it is a bug could you please submit it into official python bugtracker ?
> I never did it before and afraid to do something wrong.
>
> Thanks,
> Vitaly Murashev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130916/5a2114c0/attachment.html>

From techtonik at gmail.com  Mon Sep 16 08:28:28 2013
From: techtonik at gmail.com (anatoly techtonik)
Date: Mon, 16 Sep 2013 09:28:28 +0300
Subject: [Python-Dev] TransformDict (PEP 455) Naming
Message-ID: <CAPkN8xJzJK2o=5VQ=1X2gvm7u1FiPciDsA3LU5r_bRQkszWXPw@mail.gmail.com>

Does anybody know if http://vote.python.org is already operational?

I decided to start a separate thread for TransformDict name, because I
want to change it.
Current implementation of PEP 455 only touches dictionary keys and it
is more narrow than the name suggests. I'd reserve TransformDict name
for something that is used to transform some other data. For my data
transformation theory I have an idea of mapping with annotated fields
that is used to change the names of some source data structure to
target data structure, converting types and applying custom rules on
the way. This is a different, but more intuitive application of such
name.

Name for the PEP 455 should be less abstract, more specialized. Which
transform? What is transformed? Where the transformation is taking
place? How it is going to be used? Why do you need a generalized
structure?

In PEP 455 I see two use cases only.
1. Case-insensitive keys.
  dictik()  - dictionary with insensitive keys
2. Identity dict.
  dictik()  - dictionary with implicit keys

The generic name for collections entry can be VagueKeysDict() or
DynamicKeysDict() or DynKeyDict(). The name should tell what's going
on with the type to people completely unfamiliar with concept.
OrderedDict() is a good name - dict with ordered entries, NamedTuple()
is so-so - tuple that has a name, but also ok. TransformDict() is just
too generic and doesn't tell anything - something is transformed, or
dict somehow transformed, or dict for transform. There should be a
better name.

--
anatoly t.

From solipsis at pitrou.net  Mon Sep 16 10:15:08 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 16 Sep 2013 10:15:08 +0200
Subject: [Python-Dev] PEP 428:  Pathlib
References: <5235BA20.8010105@stoneleaf.us>
Message-ID: <20130916101508.135926de@pitrou.net>

Le Sun, 15 Sep 2013 06:46:08 -0700,
Ethan Furman <ethan at stoneleaf.us> a ?crit :
> I see PEP 428 is both targeted at 3.4 and still in draft status.
> 
> What remains to be done to ask for pronouncement?

I think I have a couple of items left to integrate in the PEP.
Mostly it needs me to take a bit of time and finalize the PEP, and
then have a PEP delegate (or Guido) pronounce on it.

That's unless someone else wants to add it, of course. I don't mind
suggestions, as long as they don't deter from the short-/middle-term
goal of getting the PEP approved :)

Note that the pathlib API has to be provisional. While "OO path
objects" have been a common desire for a long time, it is a much less
well-paved avenue than e.g. TransformDict :)

Regards

Antoine.



From guido at python.org  Mon Sep 16 17:20:30 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 16 Sep 2013 08:20:30 -0700
Subject: [Python-Dev] PEP 428: Pathlib
In-Reply-To: <20130916101508.135926de@pitrou.net>
References: <5235BA20.8010105@stoneleaf.us>
 <20130916101508.135926de@pitrou.net>
Message-ID: <CAP7+vJK1EXcCoXh8vH4wNDM8-Fp==MtexJi4XXEZAxEN5+XpUQ@mail.gmail.com>

I hope there is a volunteer for delegate.

--Guido van Rossum (sent from Android phone)
On Sep 16, 2013 1:17 AM, "Antoine Pitrou" <solipsis at pitrou.net> wrote:

> Le Sun, 15 Sep 2013 06:46:08 -0700,
> Ethan Furman <ethan at stoneleaf.us> a ?crit :
> > I see PEP 428 is both targeted at 3.4 and still in draft status.
> >
> > What remains to be done to ask for pronouncement?
>
> I think I have a couple of items left to integrate in the PEP.
> Mostly it needs me to take a bit of time and finalize the PEP, and
> then have a PEP delegate (or Guido) pronounce on it.
>
> That's unless someone else wants to add it, of course. I don't mind
> suggestions, as long as they don't deter from the short-/middle-term
> goal of getting the PEP approved :)
>
> Note that the pathlib API has to be provisional. While "OO path
> objects" have been a common desire for a long time, it is a much less
> well-paved avenue than e.g. TransformDict :)
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130916/0db3cc12/attachment.html>

From guido at python.org  Mon Sep 16 17:42:12 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 16 Sep 2013 08:42:12 -0700
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <20130914005921.GK16820@ando>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
 <CAP7+vJJ6QxcUwcpdt3Mq6r07Q2dGUQt-VWGdsYGLv4wrRW5dGg@mail.gmail.com>
 <20130914005921.GK16820@ando>
Message-ID: <CAP7+vJL81VR=kdO+2rCz3Di3GVNSwhkRCXrw2HAL=7ywT42rvA@mail.gmail.com>

I'm ready to accept this PEP. Because I haven't read this entire thread
(and 60 messages about random diversions is really too much to try and
catch up on) I'll give people 24 hours to remind me of outstanding
rejections.

I also haven't reviewed the code in any detail, but I believe the code
review is going well, so I'm not concerned that the PEP would have to
revised based on that alone.


On Fri, Sep 13, 2013 at 5:59 PM, Steven D'Aprano <steve at pearwood.info>wrote:

> On Sun, Sep 08, 2013 at 10:51:57AM -0700, Guido van Rossum wrote:
> > Never mind, I found the patch and the issue. I really think that the
> > *PEP* is ready for inclusion after the open issues are changed into
> > something like Discussion or Future Work, and after adding a more
> > prominent link to the issue with the patch. Then the *patch* can be
> > reviewed some more until it is ready -- it looks very close already.
>
> I've updated the PEP as requested. Is there anything further that needs
> to be done to have it approved?
>
> http://www.python.org/dev/peps/pep-0450/
>
>
>
> --
> Steven
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130916/6a3a9728/attachment.html>

From oscar.j.benjamin at gmail.com  Mon Sep 16 18:45:45 2013
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Mon, 16 Sep 2013 17:45:45 +0100
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <CAP7+vJL81VR=kdO+2rCz3Di3GVNSwhkRCXrw2HAL=7ywT42rvA@mail.gmail.com>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
 <CAP7+vJJ6QxcUwcpdt3Mq6r07Q2dGUQt-VWGdsYGLv4wrRW5dGg@mail.gmail.com>
 <20130914005921.GK16820@ando>
 <CAP7+vJL81VR=kdO+2rCz3Di3GVNSwhkRCXrw2HAL=7ywT42rvA@mail.gmail.com>
Message-ID: <CAHVvXxSGBAV9GgEB2fznmA=wuAfGZBBkREgVaPJR5xiir5LmtA@mail.gmail.com>

On 16 September 2013 16:42, Guido van Rossum <guido at python.org> wrote:
> I'm ready to accept this PEP. Because I haven't read this entire thread (and
> 60 messages about random diversions is really too much to try and catch up
> on) I'll give people 24 hours to remind me of outstanding rejections.
>
> I also haven't reviewed the code in any detail, but I believe the code
> review is going well, so I'm not concerned that the PEP would have to
> revised based on that alone.

I think Steven has addressed all of the issues raised. Briefly from memory:

1) There was concern about having an additional sum function. Steven
has pointed out that neither of sum/fsum is accurate for all stdlib
numeric types as is the intention for the statistics module. It is not
possible to modify either of sum/fsum in a backward compatible way
that would make them suitable here.

2) The initial names for the median functions were median.low
median.high etc. This naming scheme was considered non-standard by
some and has been redesigned as median_low, median_high etc. (there
was also discussion about the method used to attach the names to the
median function but this became irrelevant after the rename).

3) The mode function also provided an algorithm for estimating the
mode of a continuous probability distribution from a sample. It was
suggested that there is no uniquely good way of doing this and that it
is not commonly needed. This was removed and the API for mode() was
simplified (it now returns a unique mode or raises an error).

4) Some of the functions (e.g. variance) used different algorithms
(and produced different results) when given an iterator instead of a
collection. These are now changed to always use the same algorithm and
build a collection internally if necessary.

5) It was suggested that it should also be possible to compute the
mean of e.g. timedelta objects but it was pointed out that they can be
converted to numbers with the timedelta.total_seconds() method.

6) I raised an issue about the way the sum function behaved for
decimals but this was changed in a subsequent patch presenting a new
sum function that isn't susceptible to accumulated rounding errors
with Decimals.


Oscar

From v+python at g.nevcal.com  Mon Sep 16 17:31:58 2013
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Mon, 16 Sep 2013 08:31:58 -0700
Subject: [Python-Dev] TransformDict (PEP 455) Naming
In-Reply-To: <CAPkN8xJzJK2o=5VQ=1X2gvm7u1FiPciDsA3LU5r_bRQkszWXPw@mail.gmail.com>
References: <CAPkN8xJzJK2o=5VQ=1X2gvm7u1FiPciDsA3LU5r_bRQkszWXPw@mail.gmail.com>
Message-ID: <5237246E.6060400@g.nevcal.com>

On 9/15/2013 11:28 PM, anatoly techtonik wrote:
> Does anybody know if http://vote.python.org is already operational?
>
> I decided to start a separate thread for TransformDict name, because I
> want to change it.
> Current implementation of PEP 455 only touches dictionary keys and it
> is more narrow than the name suggests. I'd reserve TransformDict name
> for something that is used to transform some other data. For my data
> transformation theory I have an idea of mapping with annotated fields
> that is used to change the names of some source data structure to
> target data structure, converting types and applying custom rules on
> the way. This is a different, but more intuitive application of such
> name.

The multitude of data transformations that are possible are certainly 
broader than the scope of TransformDict. However, such transformations 
have little to do with the operation of a dict ... the key 
characteristic of a dict is accessing data by key value, and the idea of 
transformation for a dict is easily understood to be a transformation of 
that access pattern, rather than a rich transformation of the data.

Rich data transformations may be useful, and if possible to abstract a 
large number of useful data transformations into an API that would 
become popular, it would seem that such transformations would want to be 
applied not only to dict, but also to list and other data structures. It 
would be more of an object-to-object mapping, independent of the 
container that might hold the object during part of its lifetime.

Hence, it seems unlikely to me that "dict" would be part of the name or 
requirements for such rich data transformations, leaving TransformDict 
available to be used for exactly what PEP 455 proposes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130916/2e316906/attachment.html>

From cf.natali at gmail.com  Mon Sep 16 19:06:37 2013
From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=)
Date: Mon, 16 Sep 2013 19:06:37 +0200
Subject: [Python-Dev] PEP 428: Pathlib
In-Reply-To: <20130916101508.135926de@pitrou.net>
References: <5235BA20.8010105@stoneleaf.us>
 <20130916101508.135926de@pitrou.net>
Message-ID: <CAH_1eM1uo0W29xy-pJPOpe-Orfc830M3BUd3coBO1dDdHSxekg@mail.gmail.com>

2013/9/16 Antoine Pitrou <solipsis at pitrou.net>:
> Le Sun, 15 Sep 2013 06:46:08 -0700,
> Ethan Furman <ethan at stoneleaf.us> a ?crit :
>> I see PEP 428 is both targeted at 3.4 and still in draft status.
>>
>> What remains to be done to ask for pronouncement?
>
> I think I have a couple of items left to integrate in the PEP.
> Mostly it needs me to take a bit of time and finalize the PEP, and
> then have a PEP delegate (or Guido) pronounce on it.

IIRC, during the last discussion round, we were still debating between
implicit stat() result caching - which requires an explicit restat()
method - vs a mapping between the stat() method and a stat() syscall.

What was the conclusion?

From rowen at uw.edu  Mon Sep 16 21:31:10 2013
From: rowen at uw.edu (Russell E. Owen)
Date: Mon, 16 Sep 2013 12:31:10 -0700
Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4
References: <70C99F87-E9A5-4838-A1E9-4739FBF2E6A5@gmail.com>
 <nad-EC5D57.13320314092013@news.gmane.org>
 <B3293155-E4D5-4389-A555-C31BC49CE539@gmail.com>
Message-ID: <rowen-8E335B.12310916092013@news.gmane.org>

In article <B3293155-E4D5-4389-A555-C31BC49CE539 at gmail.com>,
 Raymond Hettinger <raymond.hettinger at gmail.com> wrote:

> On Sep 14, 2013, at 1:32 PM, Ned Deily <nad at acm.org> wrote:
> >  The 
> > most recent Developer Tools for 10.8 and 10.7 systems, Xcode 4.6.x, have 
> > a mature clang but do not provide a 10.6 SDK.  Even with using an SDK, 
> > it's still possible to end up inadvertently linking with the wrong 
> > versions of system libraries.  We have been burned by that in the past.
> 
> I think we should offer a separate Mac build just for 10.6
> (much like we do for the 32-bit PPC option for 10.5).

If Apple drops support for gcc in 10.9 I guess we have to go this route, 
but please be careful. Every time you add a new version of python for 
MacOS X it means that folks providing binary installers (e.g. for numpy) 
have to provide another binary, and folks using those installers have 
another chance of picking the wrong one.

If you do make a 10.6-only installer, what is the minimum version of 
MacOS X the modern compiler would support? 10.7 gives a more measured 
upgrade path, but 10.8 gives a better compiler.

-- Russell


From solipsis at pitrou.net  Mon Sep 16 21:45:36 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 16 Sep 2013 21:45:36 +0200
Subject: [Python-Dev] PEP 428: Pathlib -> stat caching
In-Reply-To: <CAH_1eM1uo0W29xy-pJPOpe-Orfc830M3BUd3coBO1dDdHSxekg@mail.gmail.com>
References: <5235BA20.8010105@stoneleaf.us>
 <20130916101508.135926de@pitrou.net>
 <CAH_1eM1uo0W29xy-pJPOpe-Orfc830M3BUd3coBO1dDdHSxekg@mail.gmail.com>
Message-ID: <20130916214536.55907f19@fsol>

On Mon, 16 Sep 2013 19:06:37 +0200
Charles-Fran?ois Natali <cf.natali at gmail.com> wrote:
> 2013/9/16 Antoine Pitrou <solipsis at pitrou.net>:
> > Le Sun, 15 Sep 2013 06:46:08 -0700,
> > Ethan Furman <ethan at stoneleaf.us> a ?crit :
> >> I see PEP 428 is both targeted at 3.4 and still in draft status.
> >>
> >> What remains to be done to ask for pronouncement?
> >
> > I think I have a couple of items left to integrate in the PEP.
> > Mostly it needs me to take a bit of time and finalize the PEP, and
> > then have a PEP delegate (or Guido) pronounce on it.
> 
> IIRC, during the last discussion round, we were still debating between
> implicit stat() result caching - which requires an explicit restat()
> method - vs a mapping between the stat() method and a stat() syscall.
> 
> What was the conclusion?

No definite conclusion. You and Nick liked the idea of a rich stat
object (returned by os.stat()) with is_dir() methods and the like:
https://mail.python.org/pipermail/python-dev/2013-May/125809.html

However, nothing was done about that since then ;-)

There was also the scandir() proposal to return rich objects with
optional stat-like fields, but similarly it didn't get a conclusion:
https://mail.python.org/pipermail/python-dev/2013-May/126119.html

So I would like to propose the following API change:

- Path.stat() (and stat-accessing methods such as get_mtime()...)
  returns an uncached stat object by default

- Path.cache_stat() can be called to return the stat() *and* cache it
  for future use, such that any future call to stat(), cache_stat() or
  a stat-accessing function reuses that cached stat

In other words, only if you use cache_stat() at least once is the
stat() value cached and reused by the Path object.
(also, it's a per-Path decision)

Regards

Antoine.

From brett at python.org  Mon Sep 16 21:48:54 2013
From: brett at python.org (Brett Cannon)
Date: Mon, 16 Sep 2013 15:48:54 -0400
Subject: [Python-Dev] PEP 428: Pathlib -> stat caching
In-Reply-To: <20130916214536.55907f19@fsol>
References: <5235BA20.8010105@stoneleaf.us>
 <20130916101508.135926de@pitrou.net>
 <CAH_1eM1uo0W29xy-pJPOpe-Orfc830M3BUd3coBO1dDdHSxekg@mail.gmail.com>
 <20130916214536.55907f19@fsol>
Message-ID: <CAP1=2W5yq_t2iMiuptkW5w7x7kiowVCFXM4qQUEHug1QVAx6ug@mail.gmail.com>

On Mon, Sep 16, 2013 at 3:45 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Mon, 16 Sep 2013 19:06:37 +0200
> Charles-Fran?ois Natali <cf.natali at gmail.com> wrote:
> > 2013/9/16 Antoine Pitrou <solipsis at pitrou.net>:
> > > Le Sun, 15 Sep 2013 06:46:08 -0700,
> > > Ethan Furman <ethan at stoneleaf.us> a ?crit :
> > >> I see PEP 428 is both targeted at 3.4 and still in draft status.
> > >>
> > >> What remains to be done to ask for pronouncement?
> > >
> > > I think I have a couple of items left to integrate in the PEP.
> > > Mostly it needs me to take a bit of time and finalize the PEP, and
> > > then have a PEP delegate (or Guido) pronounce on it.
> >
> > IIRC, during the last discussion round, we were still debating between
> > implicit stat() result caching - which requires an explicit restat()
> > method - vs a mapping between the stat() method and a stat() syscall.
> >
> > What was the conclusion?
>
> No definite conclusion. You and Nick liked the idea of a rich stat
> object (returned by os.stat()) with is_dir() methods and the like:
> https://mail.python.org/pipermail/python-dev/2013-May/125809.html
>
> However, nothing was done about that since then ;-)
>
> There was also the scandir() proposal to return rich objects with
> optional stat-like fields, but similarly it didn't get a conclusion:
> https://mail.python.org/pipermail/python-dev/2013-May/126119.html
>
> So I would like to propose the following API change:
>
> - Path.stat() (and stat-accessing methods such as get_mtime()...)
>   returns an uncached stat object by default
>
> - Path.cache_stat() can be called to return the stat() *and* cache it
>   for future use, such that any future call to stat(), cache_stat() or
>   a stat-accessing function reuses that cached stat
>
> In other words, only if you use cache_stat() at least once is the
> stat() value cached and reused by the Path object.
> (also, it's a per-Path decision)
>

Any reason why stat() can't get a keyword-only cached=True argument
instead? Or have stat() never cache() but stat_cache() always so that
people can choose if they want fresh or cached based on API and not whether
some library happened to make a decision for them?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130916/7d9608fb/attachment.html>

From victor.stinner at gmail.com  Mon Sep 16 22:00:53 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 16 Sep 2013 22:00:53 +0200
Subject: [Python-Dev] PEP 428: Pathlib -> stat caching
In-Reply-To: <CAP1=2W5yq_t2iMiuptkW5w7x7kiowVCFXM4qQUEHug1QVAx6ug@mail.gmail.com>
References: <5235BA20.8010105@stoneleaf.us>
 <20130916101508.135926de@pitrou.net>
 <CAH_1eM1uo0W29xy-pJPOpe-Orfc830M3BUd3coBO1dDdHSxekg@mail.gmail.com>
 <20130916214536.55907f19@fsol>
 <CAP1=2W5yq_t2iMiuptkW5w7x7kiowVCFXM4qQUEHug1QVAx6ug@mail.gmail.com>
Message-ID: <CAMpsgwZbyegAMONaXJH=Mi46SpW1X-_YTgzKS+CWThq5qccf9Q@mail.gmail.com>

2013/9/16 Brett Cannon <brett at python.org>:
> Any reason why stat() can't get a keyword-only cached=True argument instead?
> Or have stat() never cache() but stat_cache() always so that people can
> choose if they want fresh or cached based on API and not whether some
> library happened to make a decision for them?

I also prefer a single function, but only if the default is
cached=False. Caching by default can be surprising and unexpected.

Victor

From janssen at parc.com  Mon Sep 16 21:56:42 2013
From: janssen at parc.com (Bill Janssen)
Date: Mon, 16 Sep 2013 12:56:42 -0700
Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4
In-Reply-To: <rowen-8E335B.12310916092013@news.gmane.org>
References: <70C99F87-E9A5-4838-A1E9-4739FBF2E6A5@gmail.com>
 <nad-EC5D57.13320314092013@news.gmane.org>
 <B3293155-E4D5-4389-A555-C31BC49CE539@gmail.com>
 <rowen-8E335B.12310916092013@news.gmane.org>
Message-ID: <51403.1379361402@parc.com>

Russell E. Owen <rowen at uw.edu> wrote:

> In article <B3293155-E4D5-4389-A555-C31BC49CE539 at gmail.com>,
>  Raymond Hettinger <raymond.hettinger at gmail.com> wrote:
> 
> > On Sep 14, 2013, at 1:32 PM, Ned Deily <nad at acm.org> wrote:
> > >  The 
> > > most recent Developer Tools for 10.8 and 10.7 systems, Xcode 4.6.x, have 
> > > a mature clang but do not provide a 10.6 SDK.  Even with using an SDK, 
> > > it's still possible to end up inadvertently linking with the wrong 
> > > versions of system libraries.  We have been burned by that in the past.
> > 
> > I think we should offer a separate Mac build just for 10.6
> > (much like we do for the 32-bit PPC option for 10.5).
> 
> If Apple drops support for gcc in 10.9 I guess we have to go this route, 

Could go the Sage route -- Sage first checks for an up-to-date version
of gcc, and downloads it and builds it for its own use if necessary.

Bill

> but please be careful. Every time you add a new version of python for 
> MacOS X it means that folks providing binary installers (e.g. for numpy) 
> have to provide another binary, and folks using those installers have 
> another chance of picking the wrong one.
> 
> If you do make a 10.6-only installer, what is the minimum version of 
> MacOS X the modern compiler would support? 10.7 gives a more measured 
> upgrade path, but 10.8 gives a better compiler.
> 
> -- Russell
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/bill%40janssen.org

From solipsis at pitrou.net  Mon Sep 16 22:05:46 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 16 Sep 2013 22:05:46 +0200
Subject: [Python-Dev] PEP 428: Pathlib -> stat caching
In-Reply-To: <CAP1=2W5yq_t2iMiuptkW5w7x7kiowVCFXM4qQUEHug1QVAx6ug@mail.gmail.com>
References: <5235BA20.8010105@stoneleaf.us>
 <20130916101508.135926de@pitrou.net>
 <CAH_1eM1uo0W29xy-pJPOpe-Orfc830M3BUd3coBO1dDdHSxekg@mail.gmail.com>
 <20130916214536.55907f19@fsol>
 <CAP1=2W5yq_t2iMiuptkW5w7x7kiowVCFXM4qQUEHug1QVAx6ug@mail.gmail.com>
Message-ID: <20130916220546.341e369f@fsol>

On Mon, 16 Sep 2013 15:48:54 -0400
Brett Cannon <brett at python.org> wrote:
> >
> > So I would like to propose the following API change:
> >
> > - Path.stat() (and stat-accessing methods such as get_mtime()...)
> >   returns an uncached stat object by default
> >
> > - Path.cache_stat() can be called to return the stat() *and* cache it
> >   for future use, such that any future call to stat(), cache_stat() or
> >   a stat-accessing function reuses that cached stat
> >
> > In other words, only if you use cache_stat() at least once is the
> > stat() value cached and reused by the Path object.
> > (also, it's a per-Path decision)
> >
> 
> Any reason why stat() can't get a keyword-only cached=True argument
> instead? Or have stat() never cache() but stat_cache() always so that
> people can choose if they want fresh or cached based on API and not whether
> some library happened to make a decision for them?

1. Because you also want the helper functions (get_mtime(), etc.) to
cache the value too. It's not only about stat().

2. Because of the reverse use case where you want a library to reuse a
cached value despite the library not using an explicit caching call.

Basically, the rationale is:

1. Caching should be opt-in, which is what this new API achieves.

2. Once you have asked for caching, most always you also want the
subsequent accesses to be cached.

I realize there should be a third method clear_cache(), though ;-)

Regards

Antoine.

From rdmurray at bitdance.com  Mon Sep 16 22:14:43 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Mon, 16 Sep 2013 16:14:43 -0400
Subject: [Python-Dev] PEP 428: Pathlib -> stat caching
In-Reply-To: <CAP1=2W5yq_t2iMiuptkW5w7x7kiowVCFXM4qQUEHug1QVAx6ug@mail.gmail.com>
References: <5235BA20.8010105@stoneleaf.us>
 <20130916101508.135926de@pitrou.net>
 <CAH_1eM1uo0W29xy-pJPOpe-Orfc830M3BUd3coBO1dDdHSxekg@mail.gmail.com>
 <20130916214536.55907f19@fsol>
 <CAP1=2W5yq_t2iMiuptkW5w7x7kiowVCFXM4qQUEHug1QVAx6ug@mail.gmail.com>
Message-ID: <20130916201444.34AD7250512@webabinitio.net>

On Mon, 16 Sep 2013 15:48:54 -0400, Brett Cannon <brett at python.org> wrote:
> On Mon, Sep 16, 2013 at 3:45 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > So I would like to propose the following API change:
> >
> > - Path.stat() (and stat-accessing methods such as get_mtime()...)
> >   returns an uncached stat object by default
> >
> > - Path.cache_stat() can be called to return the stat() *and* cache it
> >   for future use, such that any future call to stat(), cache_stat() or
> >   a stat-accessing function reuses that cached stat
> >
> > In other words, only if you use cache_stat() at least once is the
> > stat() value cached and reused by the Path object.
> > (also, it's a per-Path decision)
> >
> 
> Any reason why stat() can't get a keyword-only cached=True argument
> instead? Or have stat() never cache() but stat_cache() always so that
> people can choose if they want fresh or cached based on API and not whether
> some library happened to make a decision for them?

Well, we tend to avoid single boolean arguments in favor of differently
named functions.

But here is an alternate API:  expose the state by having a 'cache_stat'
attribute of the Path that is 'False' by default but can be set 'True'.
It could also (or only?) be set via an optional constructor argument.

--David

From rymg19 at gmail.com  Mon Sep 16 22:28:24 2013
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Mon, 16 Sep 2013 15:28:24 -0500
Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4
In-Reply-To: <51403.1379361402@parc.com>
References: <70C99F87-E9A5-4838-A1E9-4739FBF2E6A5@gmail.com>
 <nad-EC5D57.13320314092013@news.gmane.org>
 <B3293155-E4D5-4389-A555-C31BC49CE539@gmail.com>
 <rowen-8E335B.12310916092013@news.gmane.org> <51403.1379361402@parc.com>
Message-ID: <CAO41-mMUQD=K7WbSPcHSZk6jh_K3zv=Aa-YAGL2dRrwckCqcOg@mail.gmail.com>

Meh...I hate it when tools download stuff without me noticing.

Honestly, a separate 10.6 build would work well. Plus, if a new Clang
versions includes some awesome feature that could make Python builds
better, you'd be able to take advantage of it better.


On Mon, Sep 16, 2013 at 2:56 PM, Bill Janssen <janssen at parc.com> wrote:

> Russell E. Owen <rowen at uw.edu> wrote:
>
> > In article <B3293155-E4D5-4389-A555-C31BC49CE539 at gmail.com>,
> >  Raymond Hettinger <raymond.hettinger at gmail.com> wrote:
> >
> > > On Sep 14, 2013, at 1:32 PM, Ned Deily <nad at acm.org> wrote:
> > > >  The
> > > > most recent Developer Tools for 10.8 and 10.7 systems, Xcode 4.6.x,
> have
> > > > a mature clang but do not provide a 10.6 SDK.  Even with using an
> SDK,
> > > > it's still possible to end up inadvertently linking with the wrong
> > > > versions of system libraries.  We have been burned by that in the
> past.
> > >
> > > I think we should offer a separate Mac build just for 10.6
> > > (much like we do for the 32-bit PPC option for 10.5).
> >
> > If Apple drops support for gcc in 10.9 I guess we have to go this route,
>
> Could go the Sage route -- Sage first checks for an up-to-date version
> of gcc, and downloads it and builds it for its own use if necessary.
>
> Bill
>
> > but please be careful. Every time you add a new version of python for
> > MacOS X it means that folks providing binary installers (e.g. for numpy)
> > have to provide another binary, and folks using those installers have
> > another chance of picking the wrong one.
> >
> > If you do make a 10.6-only installer, what is the minimum version of
> > MacOS X the modern compiler would support? 10.7 gives a more measured
> > upgrade path, but 10.8 gives a better compiler.
> >
> > -- Russell
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/bill%40janssen.org
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com
>



-- 
Ryan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130916/f2b2a677/attachment.html>

From solipsis at pitrou.net  Mon Sep 16 22:43:31 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 16 Sep 2013 22:43:31 +0200
Subject: [Python-Dev] PEP 428: Pathlib -> stat caching
References: <5235BA20.8010105@stoneleaf.us>
 <20130916101508.135926de@pitrou.net>
 <CAH_1eM1uo0W29xy-pJPOpe-Orfc830M3BUd3coBO1dDdHSxekg@mail.gmail.com>
 <20130916214536.55907f19@fsol>
 <CAP1=2W5yq_t2iMiuptkW5w7x7kiowVCFXM4qQUEHug1QVAx6ug@mail.gmail.com>
 <20130916201444.34AD7250512@webabinitio.net>
Message-ID: <20130916224331.4a035d6c@fsol>

On Mon, 16 Sep 2013 16:14:43 -0400
"R. David Murray" <rdmurray at bitdance.com> wrote:
> On Mon, 16 Sep 2013 15:48:54 -0400, Brett Cannon <brett at python.org> wrote:
> > On Mon, Sep 16, 2013 at 3:45 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > > So I would like to propose the following API change:
> > >
> > > - Path.stat() (and stat-accessing methods such as get_mtime()...)
> > >   returns an uncached stat object by default
> > >
> > > - Path.cache_stat() can be called to return the stat() *and* cache it
> > >   for future use, such that any future call to stat(), cache_stat() or
> > >   a stat-accessing function reuses that cached stat
> > >
> > > In other words, only if you use cache_stat() at least once is the
> > > stat() value cached and reused by the Path object.
> > > (also, it's a per-Path decision)
> > >
> > 
> > Any reason why stat() can't get a keyword-only cached=True argument
> > instead? Or have stat() never cache() but stat_cache() always so that
> > people can choose if they want fresh or cached based on API and not whether
> > some library happened to make a decision for them?
> 
> Well, we tend to avoid single boolean arguments in favor of differently
> named functions.
> 
> But here is an alternate API:  expose the state by having a 'cache_stat'
> attribute of the Path that is 'False' by default but can be set 'True'.

Thanks for the suggestion, that's a possibility too.

> It could also (or only?) be set via an optional constructor argument.

That's impractical if you get the Path object from a library call.

Regards

Antoine.



From tjreedy at udel.edu  Mon Sep 16 22:50:41 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 16 Sep 2013 16:50:41 -0400
Subject: [Python-Dev] PEP 428: Pathlib -> stat caching
In-Reply-To: <20130916201444.34AD7250512@webabinitio.net>
References: <5235BA20.8010105@stoneleaf.us>
 <20130916101508.135926de@pitrou.net>
 <CAH_1eM1uo0W29xy-pJPOpe-Orfc830M3BUd3coBO1dDdHSxekg@mail.gmail.com>
 <20130916214536.55907f19@fsol>
 <CAP1=2W5yq_t2iMiuptkW5w7x7kiowVCFXM4qQUEHug1QVAx6ug@mail.gmail.com>
 <20130916201444.34AD7250512@webabinitio.net>
Message-ID: <l17qus$l1f$1@ger.gmane.org>

On 9/16/2013 4:14 PM, R. David Murray wrote:

> Well, we tend to avoid single boolean arguments in favor of differently
> named functions.

The stdlib has lots of boolean arguments. My impression is that they are 
to be avoided when they would change the return type or otherwise do 
something disjointly different. I do not think this would apply here.

-- 
Terry Jan Reedy


From ncoghlan at gmail.com  Tue Sep 17 01:15:26 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 17 Sep 2013 09:15:26 +1000
Subject: [Python-Dev] PEP 428: Pathlib -> stat caching
In-Reply-To: <20130916224331.4a035d6c@fsol>
References: <5235BA20.8010105@stoneleaf.us>
 <20130916101508.135926de@pitrou.net>
 <CAH_1eM1uo0W29xy-pJPOpe-Orfc830M3BUd3coBO1dDdHSxekg@mail.gmail.com>
 <20130916214536.55907f19@fsol>
 <CAP1=2W5yq_t2iMiuptkW5w7x7kiowVCFXM4qQUEHug1QVAx6ug@mail.gmail.com>
 <20130916201444.34AD7250512@webabinitio.net>
 <20130916224331.4a035d6c@fsol>
Message-ID: <CADiSq7eESmFN7_LOhdgoS0xDwDLZ40OJ5yvqog5yjf+=+-59-Q@mail.gmail.com>

On 17 Sep 2013 06:45, "Antoine Pitrou" <solipsis at pitrou.net> wrote:
>
> On Mon, 16 Sep 2013 16:14:43 -0400
> "R. David Murray" <rdmurray at bitdance.com> wrote:
> > On Mon, 16 Sep 2013 15:48:54 -0400, Brett Cannon <brett at python.org>
wrote:
> > > On Mon, Sep 16, 2013 at 3:45 PM, Antoine Pitrou <solipsis at pitrou.net>
wrote:
> > > > So I would like to propose the following API change:
> > > >
> > > > - Path.stat() (and stat-accessing methods such as get_mtime()...)
> > > >   returns an uncached stat object by default
> > > >
> > > > - Path.cache_stat() can be called to return the stat() *and* cache
it
> > > >   for future use, such that any future call to stat(), cache_stat()
or
> > > >   a stat-accessing function reuses that cached stat
> > > >
> > > > In other words, only if you use cache_stat() at least once is the
> > > > stat() value cached and reused by the Path object.
> > > > (also, it's a per-Path decision)
> > > >
> > >
> > > Any reason why stat() can't get a keyword-only cached=True argument
> > > instead? Or have stat() never cache() but stat_cache() always so that
> > > people can choose if they want fresh or cached based on API and not
whether
> > > some library happened to make a decision for them?
> >
> > Well, we tend to avoid single boolean arguments in favor of differently
> > named functions.
> >
> > But here is an alternate API:  expose the state by having a 'cache_stat'
> > attribute of the Path that is 'False' by default but can be set 'True'.
>
> Thanks for the suggestion, that's a possibility too.
>
> > It could also (or only?) be set via an optional constructor argument.
>
> That's impractical if you get the Path object from a library call.

Given that this is a behavioural state change, I think asking for a
possibly *new* path with caching enabled in that case would be a good way
to go. If we treat path objects as effectively immutable (aside from the
optional internal stat cache), then checking in __new__ if a passed in path
object already has the appropriate caching status and returning it directly
if so, but otherwise creating a new path object with the cache setting
changed would avoid having libraries potentially alter the behaviour of
applications' path objects and vice-versa.

In effect, the unique "identity" of a path would be a triple representing
the type, the filesystem path and whether or not it cached stat results
internally. If you wanted to change any of those, you would have to create
a new object.

Cheers,
Nick.

>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130917/1bf099a6/attachment.html>

From victor.stinner at gmail.com  Tue Sep 17 01:47:24 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 17 Sep 2013 01:47:24 +0200
Subject: [Python-Dev] PEP 454: add a new tracemalloc module (second round)
Message-ID: <CAMpsgwaEuVQYt1HfLUyvN2Bjgj=116gxWR39kAQgKJPtbSVDwA@mail.gmail.com>

Hi,

Thanks to the early remarks on the PEP 454, I redesigned and enhanced
the API of the new tracemalloc module. Changes:

* it is now possibility to record more than 1 frame per memory allocation
* add filters on filename and line number
* new GroupedStats and StatsDiff class to generate and compare statistics
* cumulative statistics
* display the traceback of a memory block
* almost all the code has unit tests
* better documentation

HTML version of the documentation:
http://www.haypocalc.com/tmp/tracemalloc/library/tracemalloc.html

The documentation contains output examples and a short tutorial
("Usage"), but also the documentation of the command line, which are
not included in the PEP.

HTML version of the PEP:
http://www.python.org/dev/peps/pep-0454/

Issue tracking the implementation:
http://bugs.python.org/issue18874


PEP: 454
Title: Add a new tracemalloc module to trace Python memory allocations
Version: $Revision$
Last-Modified: $Date$
Author: Victor Stinner <victor.stinner at gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 3-September-2013
Python-Version: 3.4


Abstract
========

Add a new ``tracemalloc`` module to trace memory blocks allocated by Python.



Rationale
=========

Common debug tools tracing memory allocations read the C filename and
number.  Using such tool to analyze Python memory allocations does not
help because most memory block are allocated in the same C function,
in ``PyMem_Malloc()`` for example.

There are debug tools dedicated to the Python language like ``Heapy``
and ``PySizer``. These projects analyze objects type and/or content.
These tools are useful when most memory leaks are instances of the
same type and this type is only instancied in a few functions. The
problem is when the object type is very common like ``str`` or
``tuple``, and it is hard to identify where these objects are
instancied.

Finding reference cycles is also a difficult problem. There are
different tools to draw a diagram of all references. These tools cannot
be used on large applications with thousands of objects because the
diagram is too huge to be analyzed manually.


Proposal
========

Using the PEP 445, it becomes easy to setup an hook on Python memory
allocators. The hook can inspect the current Python frame to get the
Python filename and line number.

This PEP proposes to add a new ``tracemalloc`` module. It is a debug
tool to trace memory allocations made by Python. The module provides the
following information:

* Compute the differences between two snapshots to detect memory leaks
* Statistics on allocated memory blocks per filename and per line number:
  total size, number and average size of allocated memory blocks
* For each allocated memory block: its size and the traceback where the block
  was allocated

The API of the tracemalloc module is similar to the API of the
faulthandler module: ``enable()``, ``disable()`` and ``is_enabled()``
functions, an environment variable (``PYTHONFAULTHANDLER`` and
``PYTHONTRACEMALLOC``), a ``-X`` command line option (``-X
faulthandler`` and ``-X tracemalloc``). See the
`documentation of the faulthandler module
<http://docs.python.org/dev/library/faulthandler.html>`_.

The tracemalloc module has been written for CPython. Other
implementations of Python may not provide it.


API
===

To trace most memory blocks allocated by Python, the module should be
enabled as early as possible by calling ``tracemalloc.enable()``
function, by setting the ``PYTHONTRACEMALLOC`` environment variable to
``1``, or by using ``-X tracemalloc`` command line option.

By default, the ``Trace.traceback`` attribute only stores one ``Frame``
instance per allocated memory block. Use ``set_traceback_limit()`` to
store more frames.


Functions
---------

``add_filter(filter)`` function:

    Add a new filter on Python memory allocations, *filter* is a
    ``Filter`` instance.

    All inclusive filters are applied at once, a memory allocation is
    only ignored if no inclusive filter match its trace. A memory
    allocation is ignored if at least one exclusive filter matchs its
    trace.

    The new filter is not applied on already collected traces. Use
    ``clear_traces()`` to ensure that all traces match the new filter.


``add_include_filter(filename: str, lineno: int=None, traceback:
bool=False)`` function:

    Add an inclusive filter: helper for ``add_filter()`` creating a
    ``Filter`` instance with ``include`` attribute set to ``True``.

    Example: ``tracemalloc.add_include_filter(tracemalloc.__file__)``
    only includes memory blocks allocated by the ``tracemalloc`` module.


``add_exclude_filter(filename: str, lineno: int=None, traceback:
bool=False)`` function:

    Add an exclusive filter: helper for ``add_filter()`` creating a
    ``Filter`` instance with ``include`` attribute set to ``False``.

    Example: ``tracemalloc.add_exclude_filter(tracemalloc.__file__)``
    ignores memory blocks allocated by the ``tracemalloc`` module.


``clear_filters()`` function:

    Reset the filter list.


``clear_traces()`` function:

    Clear all traces and statistics on Python memory allocations, and
    reset the ``get_traced_memory()`` counter.


``disable()`` function:

    Stop tracing Python memory allocations and stop the timer started by
    ``start_timer()``.

    See also ``enable()`` and ``is_enabled()`` functions.


``enable()`` function:

    Start tracing Python memory allocations.

    See also ``disable()`` and ``is_enabled()`` functions.


``get_filters()`` function:

    Get the filters on Python memory allocations as list of ``Filter``
    instances.


``get_traceback_limit()`` function:

    Get the maximum number of ``Frame`` instances stored in the
    ``traceback`` attribute of a ``Trace`` instance.

    Use ``set_traceback_limit()`` to change the limit.


``get_object_address(obj)`` function:

    Get the address of the memory block of the specified Python object.


``get_object_trace(obj)`` function:

    Get the trace of a Python object *obj* as a ``Trace`` instance.

    The function only returns the trace of the memory block directly
    holding to object. The ``size`` attribute of the trace is smaller
    than the total size of the object if the object is composed of more
    than one memory block.

    Return ``None`` if the ``tracemalloc`` module did not trace the
    allocation of the object.

    See also ``gc.get_referrers()`` and ``sys.getsizeof()`` functions.


``get_process_memory()`` function:

    Get the memory usage of the current process as a meminfo namedtuple
    with two attributes:

    * ``rss``: Resident Set Size in bytes
    * ``vms``: size of the virtual memory in bytes

    Return ``None`` if the platform is not supported.


``get_stats()`` function:

    Get statistics on traced Python memory blocks as a dictionary
    ``{filename (str): {line_number (int): stats}}`` where *stats* in a
    ``TraceStats`` instance, *filename* and *line_number* can be
    ``None``.

    Return an empty dictionary if the ``tracemalloc`` module is
    disabled.


``get_traced_memory()`` function:

    Get the total size of all traced memory blocks allocated by Python.


``get_tracemalloc_size()`` function:

    Get the memory usage in bytes of the ``tracemalloc`` module.


``get_traces(obj)`` function:

    Get all traces of Python memory allocations as a dictionary
    ``{address (int): trace}`` where *trace* is a ``Trace`` instance.

    Return an empty dictionary if the ``tracemalloc`` module is
    disabled.


``is_enabled()`` function:

    ``True`` if the ``tracemalloc`` module is tracing Python memory
    allocations, ``False`` otherwise.

    See also ``enable()`` and ``disable()`` functions.


``start_timer(delay: int, func: callable, args: tuple=(), kwargs:
dict={})`` function:

    Start a timer calling ``func(*args, **kwargs)`` every *delay*
    seconds.  Enable the ``tracemalloc`` module if it is disabled. The
    timer is based on the Python memory allocator, it is not real time.
    *func* is called after at least *delay* seconds, it is not called
    exactly after *delay* seconds if no Python memory allocation
    occurred. The timer has a resolution of 1 second.

    If the ``start_timer()`` function is called twice, previous
    parameters are replaced. Call the ``stop_timer()`` function to stop
    the timer.

    The ``DisplayTopTask.start()`` and ``TakeSnapshot.start()`` methods
    use the ``start_timer()`` function to run regulary a task.


``set_traceback_limit(limit: int)`` function:

    Set the maximum number of ``Frame`` instances stored in the
    ``traceback`` attribute of a ``Trace`` instance. Clear all traces
    and statistics on Python memory allocations if the ``tracemalloc``
    module is enabled,

    Storing the traceback of each memory allocation has an important
    overhead on the memory usage. Example with the Python test suite:
    tracing all memory allocations increases the memory usage by
    ``+50%`` when storing only 1 frame and ``+150%`` when storing 10
    frames. Use ``get_tracemalloc_size()`` to measure the overhead and
    ``add_filter()`` to select which memory allocations are traced.

    Use ``get_traceback_limit()`` to get the current limit.


``stop_timer()`` function:

    Stop the timer started by ``start_timer()``.


DisplayTop class
----------------

``DisplayTop()`` class:

   Display the top of allocated memory blocks.

``display_snapshot(snapshot, count=10, group_by="filename_lineno",
cumulative=False, file=None)`` method:

    Display a snapshot of memory blocks allocated by Python, *snapshot*
    is a ``Snapshot`` instance.

``display_top_diff(top_diff, count=10, file=None)`` method:

    Display differences between two ``GroupedStats`` instances,
    *top_diff* is a ``StatsDiff`` instance.

``display_top_stats(top_stats, count=10, file=None)`` method:

    Display the top of allocated memory blocks grouped by the
    ``group_by`` attribute of *top_stats*, *top_stats* is a
    ``GroupedStats`` instance.

``color`` attribute:

    If ``True``, always use colors. If ``False``, never use colors. The
    default value is ``None``: use colors if the *file* parameter is a
    TTY device.

``compare_with_previous`` attribute:

    If ``True`` (default value), compare with the previous snapshot. If
    ``False``, compare with the first snapshot.

``filename_parts`` attribute:

    Number of displayed filename parts (int, default: ``3``). Extra
    parts are replaced with ``'...'``.

``show_average`` attribute:

    If ``True`` (default value), display the average size of memory blocks.

``show_count`` attribute:

    If ``True`` (default value), display the number of allocated memory
    blocks.

``show_size`` attribute:

    If ``True`` (default value), display the size of memory blocks.


DisplayTopTask class
--------------------

``DisplayTopTask(count=10, group_by="filename_lineno",
cumulative=False, file=sys.stdout, user_data_callback=None)`` class:

    Task taking temporary snapshots and displaying the top *count*
    memory allocations grouped by *group_by*.

    Call the ``start()`` method to start the task.

``display()`` method:

    Take a snapshot and display the top *count* biggest allocated memory
    blocks grouped by *group_by* using the ``display_top`` attribute.

    Return the snapshot, a ``Snapshot`` instance.

``start(delay: int)`` method:

    Start a task using the ``start_timer()`` function calling the
    ``display()`` method every *delay* seconds.

``stop()`` method:

    Stop the task started by the ``start()`` method using the
    ``stop_timer()`` function.

``count`` attribute:

    Maximum number of displayed memory blocks.

``cumulative`` attribute:

    If ``True``, cumulate size and count of memory blocks of all frames
    of each ``Trace`` instance, not only the most recent frame. The
    default value is ``False``.

    The option is ignored if the traceback limit is ``1``, see the
    ``get_traceback_limit()`` function.

``display_top`` attribute:

    Instance of ``DisplayTop``.

``file`` attribute:

    The top is written into *file*.

``group_by`` attribute:

    Determine how memory allocations are grouped: see
    ``Snapshot.top_by`` for the available values.

``user_data_callback`` attribute:

    Optional callback collecting user data (callable, default:
    ``None``).  See ``Snapshot.create()``.


Filter class
------------

``Filter(include: bool, pattern: str, lineno: int=None, traceback:
bool=False)`` class:

    Filter to select which memory allocations are traced. Filters can be
    used to reduce the memory usage of the ``tracemalloc`` module, which
    can be read using ``get_tracemalloc_size()``.

``match_trace(trace)`` method:

    Return ``True`` if the ``Trace`` instance must be kept according to
    the filter, ``False`` otherwise.

``match(filename: str, lineno: int)`` method:

    Return ``True`` if the filename and line number must be kept
    according to the filter, ``False`` otherwise.

``match_filename(filename: str)`` method:

    Return ``True`` if the filename must be kept according to the
    filter, ``False`` otherwise.

``match_lineno(lineno: int)`` method:

    Return ``True`` if the line number must be kept according to the
    filter, ``False`` otherwise.

``include`` attribute:

    If *include* is ``True``, only trace memory blocks allocated in a
    file with a name matching filename ``pattern`` at line number
    ``lineno``. If *include* is ``False``, ignore memory blocks
    allocated in a file with a name matching filename :attr`pattern` at
    line number ``lineno``.

``pattern`` attribute:

    The filename *pattern* can contain one or many ``*`` joker
    characters which match any substring, including an empty string. The
    ``.pyc`` and ``.pyo`` suffixes are replaced with ``.py``. On
    Windows, the comparison is case insensitive and the alternative
    separator ``/`` is replaced with the standard separator ``\``.

``lineno`` attribute:

    Line number (``int``). If is is ``None`` or lesser than ``1``, it
    matches any line number.

``traceback`` attribute:

    If *traceback* is ``True``, all frames of the ``traceback``
    attribute of ``Trace`` instances are checked. If *traceback* is
    ``False``, only the most recent frame is checked.

    This attribute only has an effect on the ``match_trace()`` method
    and only if the traceback limit is greater than ``1``. See the
    ``get_traceback_limit()`` function.


Frame class
-----------

``Frame`` class:

    Trace of a Python frame, used by ``Trace.traceback`` attribute.

``filename`` attribute:

    Python filename, ``None`` if unknown.

``lineno`` attribute:

    Python line number, ``None`` if unknown.


GroupedStats class
------------------

``GroupedStats(stats: dict, group_by: str, cumulative=False,
timestamp=None, process_memory=None, tracemalloc_size=None)`` class:

    Top of allocated memory blocks grouped by on *group_by* as a
    dictionary.

    The ``Snapshot.top_by()`` method creates a ``GroupedStats`` instance.

``compare_to(old_stats: GroupedStats=None)`` method:

    Compare to an older ``GroupedStats`` instance.  Return a
    ``StatsDiff`` instance.

``cumulative`` attribute:

    If ``True``, cumulate size and count of memory blocks of all frames
    of ``Trace``, not only the most recent frame.

``group_by`` attribute:

    Determine how memory allocations were grouped. The type of ``stats``
    keys depends on *group_by*:

    =====================  ========================  ==============
    group_by               description               key type
    =====================  ========================  ==============
    ``'filename'``         filename                  ``str``
    ``'filename_lineno'``  filename and line number  ``(str, str)``
    ``'address'``          memory block address      ``int``
    =====================  ========================  ==============

    See the *group_by* parameter of the ``Snapshot.top_by()`` method.

``stats`` attribute:

    Dictionary ``{key: stats}`` where the *key* type depends on the
    ``group_by`` attribute and *stats* type is ``TraceStats``.

``process_memory`` attribute:

    Result of the ``get_process_memory()`` function, can be ``None``.

``timestamp`` attribute:

    Creation date and time of the snapshot, ``datetime.datetime``
    instance.

``tracemalloc_size`` attribute:

    The memory usage in bytes of the ``tracemalloc`` module, result of
    the ``get_tracemalloc_size()`` function.


Snapshot class
--------------

``Snapshot`` class:

    Snapshot of memory blocks allocated by Python.

    Use ``TakeSnapshot`` to take regulary snapshots.

``apply_filters(filters)`` method:

    Apply a list filters on the ``traces`` and ``stats`` dictionaries,
    *filters* is a list of ``Filter`` instances.

``create(\*, with_traces=False, with_stats=True,
user_data_callback=None)`` classmethod:

    Take a snapshot of traces and/or statistics of allocated memory
    blocks.

    If *with_traces* is ``True``, ``get_traces()`` is called and its
    result is stored in the ``traces`` attribute. This attribute
    contains more information than ``stats`` and uses more memory and
    more disk space. If *with_traces* is ``False``, ``traces`` is set to
    ``None``.

    If *with_stats* is ``True``, ``get_stats()`` is called and its
    result is stored in the ``Snapshot.stats`` attribute. If
    *with_stats* is ``False``, ``Snapshot.stats`` is set to ``None``.

    *with_traces* and *with_stats* cannot be ``False`` at the same time.

    *user_data_callback* is an optional callable object. Its result
    should be serializable by the ``pickle`` module, or
    ``Snapshot.write()`` would fail.  If *user_data_callback* is set, it
    is called and the result is stored in the ``Snapshot.user_data``
    attribute. Otherwise, ``Snapshot.user_data`` is set to ``None``.

    The ``tracemalloc`` module must be enabled to take a snapshot. See
    the ``enable()`` function.

``load(filename)`` classmethod:

    Load a snapshot from a file.

``top_by(group_by: str, cumulative: bool=False)`` method:

    Compute top statistics grouped by *group_by* as a ``GroupedStats``
    instance:

    =====================  ========================  ==============
    group_by               description               key type
    =====================  ========================  ==============
    ``'filename'``         filename                  ``str``
    ``'filename_lineno'``  filename and line number  ``(str, str)``
    ``'address'``          memory block address      ``int``
    =====================  ========================  ==============

    If *cumulative* is ``True``, cumulate size and count of memory
    blocks of all frames of each ``Trace`` instance, not only the most
    recent frame. The *cumulative* parameter is ignored if *group_by* is
    ``'address'`` or if the traceback limit is ``1``. See the
    ``traceback_limit`` attribute.

``write(filename)`` method:

    Write the snapshot into a file.

``pid`` attribute:

    Identifier of the process which created the snapshot, result of
    ``os.getpid()``.

``process_memory`` attribute:

    Memory usage of the current process, result of the
    ``get_process_memory()`` function. It can be ``None``.

``stats`` attribute:

    Statistics on traced Python memory, result of the ``get_stats()``
    function, if ``create()`` was called with *with_stats* equals to
    ``True``, ``None`` otherwise.

``tracemalloc_size`` attribute:

    The memory usage in bytes of the ``tracemalloc`` module, result of
    the ``get_tracemalloc_size()`` function.

``traceback_limit`` attribute:

    The maximum number of frames stored in the ``traceback`` attribute
    of a ``Trace``, result of the ``get_traceback_limit()`` function.

``traces`` attribute:

    Traces of Python memory allocations, result of the ``get_traces()``
    function, if ``create()`` was called with *with_traces* equals to
    ``True``, ``None`` otherwise.

    The ``traceback`` attribute of each ``Trace`` instance is limited to
    ``traceback_limit`` frames.

``timestamp`` attribute:

    Creation date and time of the snapshot, ``datetime.datetime``
    instance.

``user_data`` attribute:

    Result of *user_data_callback* called in ``Snapshot.create()``
    (default: ``None``).


StatsDiff class
---------------

``StatsDiff(differences, old_stats, new_stats)`` class:

    Differences between two ``GroupedStats`` instances. By default, the
    ``differences`` list is unsorted: call ``sort()`` to sort it.

    The ``GroupedStats.compare_to()`` method creates a ``StatsDiff``
    instance.

``sort()`` method:

    Sort the ``differences`` list from the biggest allocation to the
    smallest.  Sort by *size_diff*, *size*, *count_diff*, *count* and
    then by *key*.

``differences`` attribute:

    Differences between ``old_stats`` and ``new_stats`` as a list of
    ``(size_diff, size, count_diff, count, key)`` tuples. *size_diff*,
    *size*, *count_diff* and *count* are ``int``. The key type depends
    on the ``group_by`` attribute of ``new_stats``:

    =====================  ========================  ==============
    group_by               description               key type
    =====================  ========================  ==============
    ``'filename'``         filename                  ``str``
    ``'filename_lineno'``  filename and line number  ``(str, str)``
    ``'address'``          memory block address      ``int``
    =====================  ========================  ==============

    See the ``group_by`` attribute of the ``GroupedStats`` class.

``old_stats`` attribute:

    Old ``GroupedStats`` instance, can be ``None``.

``new_stats`` attribute:

    New ``GroupedStats`` instance.


Trace class
-----------

``Trace`` class:

    Debug information of a memory block allocated by Python.

``size`` attribute:

    Size in bytes of the memory block.

``traceback`` attribute:

    Traceback where the memory block was allocated as a list of
    ``Frame`` instances, most recent first.

    The list can be empty or incomplete if the ``tracemalloc`` module
    was unable to retrieve the full traceback.

    The traceback is limited to ``get_traceback_limit()`` frames. Use
    ``set_traceback_limit()`` to store more frames.


TraceStats class
----------------

``TraceStats`` class:

    Statistics on Python memory allocations.

``size`` attribute:

    Total size in bytes of allocated memory blocks.

``count`` attribute:

    Number of allocated memory blocks.


Links
=====

tracemalloc:

* `#18874: Add a new tracemalloc module to trace Python
  memory allocations <http://bugs.python.org/issue18874>`_
* `pytracemalloc on PyPI
  <https://pypi.python.org/pypi/pytracemalloc>`_

Similar projects:

* `Meliae: Python Memory Usage Analyzer
  <https://pypi.python.org/pypi/meliae>`_
* `Guppy-PE: umbrella package combining Heapy and GSL
  <http://guppy-pe.sourceforge.net/>`_
* `PySizer <http://pysizer.8325.org/>`_: developed for Python 2.4
* `memory_profiler <https://pypi.python.org/pypi/memory_profiler>`_
* `pympler <http://code.google.com/p/pympler/>`_
* `Dozer <https://pypi.python.org/pypi/Dozer>`_: WSGI Middleware version of
  the CherryPy memory leak debugger
* `objgraph <http://mg.pov.lt/objgraph/>`_
* `caulk <https://github.com/smartfile/caulk/>`_

Copyright
=========

This document has been placed into the public domain.


Victor

From steve at pearwood.info  Tue Sep 17 01:59:13 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 17 Sep 2013 09:59:13 +1000
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <CAP7+vJL81VR=kdO+2rCz3Di3GVNSwhkRCXrw2HAL=7ywT42rvA@mail.gmail.com>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
 <CAP7+vJJ6QxcUwcpdt3Mq6r07Q2dGUQt-VWGdsYGLv4wrRW5dGg@mail.gmail.com>
 <20130914005921.GK16820@ando>
 <CAP7+vJL81VR=kdO+2rCz3Di3GVNSwhkRCXrw2HAL=7ywT42rvA@mail.gmail.com>
Message-ID: <20130916235912.GG19939@ando>

On Mon, Sep 16, 2013 at 08:42:12AM -0700, Guido van Rossum wrote:
> I'm ready to accept this PEP. Because I haven't read this entire thread
> (and 60 messages about random diversions is really too much to try and
> catch up on) I'll give people 24 hours to remind me of outstanding
> rejections.
> 
> I also haven't reviewed the code in any detail, but I believe the code
> review is going well, so I'm not concerned that the PEP would have to
> revised based on that alone.

There are a couple of outstanding issues that I am aware of, but I don't 
believe that either of these affect acceptance/rejection of the PEP. 
Please correct me if I am wrong.

1) Implementation details of the statistics.sum function. Oscar is 
giving me a lot of very valuable assistance speeding up the 
implementation of sum.

2) The current implementation has extensive docstrings, but will also 
need a separate statistics.rst file.


I don't recall any other outstanding issues, if I have forgotten any, 
please remind me.




> On Fri, Sep 13, 2013 at 5:59 PM, Steven D'Aprano <steve at pearwood.info>wrote:
> 
> > On Sun, Sep 08, 2013 at 10:51:57AM -0700, Guido van Rossum wrote:
> > > Never mind, I found the patch and the issue. I really think that the
> > > *PEP* is ready for inclusion after the open issues are changed into
> > > something like Discussion or Future Work, and after adding a more
> > > prominent link to the issue with the patch. Then the *patch* can be
> > > reviewed some more until it is ready -- it looks very close already.
> >
> > I've updated the PEP as requested. Is there anything further that needs
> > to be done to have it approved?
> >
> > http://www.python.org/dev/peps/pep-0450/
> >
> >
> >
> > --
> > Steven
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/guido%40python.org
> >
> 
> 
> 
> -- 
> --Guido van Rossum (python.org/~guido)



-- 
Steven

From guido at python.org  Tue Sep 17 02:06:55 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 16 Sep 2013 17:06:55 -0700
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <20130916235912.GG19939@ando>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
 <CAP7+vJJ6QxcUwcpdt3Mq6r07Q2dGUQt-VWGdsYGLv4wrRW5dGg@mail.gmail.com>
 <20130914005921.GK16820@ando>
 <CAP7+vJL81VR=kdO+2rCz3Di3GVNSwhkRCXrw2HAL=7ywT42rvA@mail.gmail.com>
 <20130916235912.GG19939@ando>
Message-ID: <CAP7+vJKPDCZd4ZQn6mSkiAx7=wjv7WmmP3HjbOj-FLGUY-yNRg@mail.gmail.com>

On Mon, Sep 16, 2013 at 4:59 PM, Steven D'Aprano <steve at pearwood.info>wrote:

> On Mon, Sep 16, 2013 at 08:42:12AM -0700, Guido van Rossum wrote:
> > I'm ready to accept this PEP. Because I haven't read this entire thread
> > (and 60 messages about random diversions is really too much to try and
> > catch up on) I'll give people 24 hours to remind me of outstanding
> > rejections.
> >
> > I also haven't reviewed the code in any detail, but I believe the code
> > review is going well, so I'm not concerned that the PEP would have to
> > revised based on that alone.
>
> There are a couple of outstanding issues that I am aware of, but I don't
> believe that either of these affect acceptance/rejection of the PEP.
> Please correct me if I am wrong.
>
> 1) Implementation details of the statistics.sum function. Oscar is
> giving me a lot of very valuable assistance speeding up the
> implementation of sum.
>
> 2) The current implementation has extensive docstrings, but will also
> need a separate statistics.rst file.
>
>
> I don't recall any other outstanding issues, if I have forgotten any,
> please remind me.
>

Those certainly don't stand in the way of the PEP's acceptance (but they do
block the commit of the code :-).

The issues that Oscar listed also all seem resolved (though they would make
a nice addition to the "Discussion" section in the PEP).

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130916/0fa44e6a/attachment.html>

From stephen at xemacs.org  Tue Sep 17 02:34:39 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Tue, 17 Sep 2013 09:34:39 +0900
Subject: [Python-Dev] PEP 428: Pathlib -> stat caching
In-Reply-To: <l17qus$l1f$1@ger.gmane.org>
References: <5235BA20.8010105@stoneleaf.us>
 <20130916101508.135926de@pitrou.net>
 <CAH_1eM1uo0W29xy-pJPOpe-Orfc830M3BUd3coBO1dDdHSxekg@mail.gmail.com>
 <20130916214536.55907f19@fsol>
 <CAP1=2W5yq_t2iMiuptkW5w7x7kiowVCFXM4qQUEHug1QVAx6ug@mail.gmail.com>
 <20130916201444.34AD7250512@webabinitio.net>
 <l17qus$l1f$1@ger.gmane.org>
Message-ID: <87eh8o5lr4.fsf@uwakimon.sk.tsukuba.ac.jp>

Terry Reedy writes:
 > On 9/16/2013 4:14 PM, R. David Murray wrote:
 > 
 > > Well, we tend to avoid single boolean arguments in favor of differently
 > > named functions.
 > 
 > The stdlib has lots of boolean arguments. My impression is that they are 
 > to be avoided when they would change the return type or otherwise do 
 > something disjointly different. I do not think this would apply here.

I remember reading that the criterion is whether the argument is most
often given a literal value.  Then "stat_cache()" is preferable to
"stat(cache=True)".  OTOH, "stat(cache=want_cache)" is better than

    if want_cache:
        result = stat_cache()
    else:
        result = stat()

or "result = stat_cache() if want_cache else stat()".


From victor.stinner at gmail.com  Tue Sep 17 12:36:03 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 17 Sep 2013 12:36:03 +0200
Subject: [Python-Dev] PEP 454: add a new tracemalloc module (second
	round)
In-Reply-To: <CAMpsgwaEuVQYt1HfLUyvN2Bjgj=116gxWR39kAQgKJPtbSVDwA@mail.gmail.com>
References: <CAMpsgwaEuVQYt1HfLUyvN2Bjgj=116gxWR39kAQgKJPtbSVDwA@mail.gmail.com>
Message-ID: <CAMpsgwYL6KpL+R+FquooS+pEu_e9H-m1t40rb6DqOJmww9RerA@mail.gmail.com>

2013/9/17 Victor Stinner <victor.stinner at gmail.com>:
> Issue tracking the implementation:
> http://bugs.python.org/issue18874

If you want to test the implementation, you can try the following repository:
http://hg.python.org/features/tracemalloc

Or try the patch attached on the issue #18874 on the Python default
version. Compile Python and use "-X tracemalloc" command line option
to enable the module at startup. Then you can play with
tracemalloc.DisplayTopTask.display() and
tracemalloc.TakeSnapshot.take_snapshot().

To create Python snapshots easily, modify Lib/test/regrtest.py to use
the following block:

take = tracemalloc.TakeSnapshot()
# tracemalloc.set_traceback_limit(15); take.with_traces = True
take.filename_template = "/tmp/tracemalloc-$pid-$counter.pickle"
take.start(10)

And then start:

./python -X tracemalloc -m test

"take.with_traces = True" is slower but required if you want to use
cumulative views, show the traceback, or group traces by address. Use
a lower traceback limit if the test suite is too slow.

When you get a snapshot file, you can analyze it using:

./python -m tracemalloc /tmp/tracemalloc-1564-0001.pickle

Add --help to see all options. I like the --traceback option.

> ``get_filters()`` function:
>
>     Get the filters on Python memory allocations as list of ``Filter``
>     instances.

I hesitate to add a Filters class which would contain a list of
filters. The logic to check if list of filters matchs is non-trivial.
You have to split inclusive and exclusive filters and take care of
empty list of inclusive/exclusive filters. See the code of
Snapshot.apply_filters() for example.

> ``get_object_trace(obj)`` function:
>
>     Get the trace of a Python object *obj* as a ``Trace`` instance.
>
>     The function only returns the trace of the memory block directly
>     holding to object. The ``size`` attribute of the trace is smaller
>     than the total size of the object if the object is composed of more
>     than one memory block.
>
>     Return ``None`` if the ``tracemalloc`` module did not trace the
>     allocation of the object.
>
>     See also ``gc.get_referrers()`` and ``sys.getsizeof()`` functions.

The function can be see of a lie because it does not count all bytes
of a object (as explained in the doc above). The function should maybe
be renamed to "get_trace(address)" to avoid the confusion.

> DisplayTop class
> ----------------

Oh, I forgot to document the new "previous_top_stats" attribute. It is
used to compare two snapshots.

> DisplayTopTask class
> --------------------
>
> ``start(delay: int)`` method:
>
>     Start a task using the ``start_timer()`` function calling the
>     ``display()`` method every *delay* seconds.

I should probably repeat here that only one timer can used at the same
time. So only one DisplayTopTask or one TakeSnapshot instance can be
used at the same time.

It's a design choice to keep start_timer() simple, there is no need
for a complex scheduler for such simple debug tool.

You *can* run the two tasks at the same time by writing your own function:

def mytask(top_task, snapshot_task):
    top_task.display()
    snapshot_task.take_snapshot()

tracemalloc.start_timer(10, mytask, top_task, snapshot_task)

> Snapshot class
> --------------
>
> ``create(\*, with_traces=False, with_stats=True,
> user_data_callback=None)`` classmethod:

It's the only function using keyword-only parameters. I don't know
it's a good practice and should be used on other methods, or if it
should be avoided?

>     *user_data_callback* is an optional callable object. Its result
>     should be serializable by the ``pickle`` module, or
>     ``Snapshot.write()`` would fail.  If *user_data_callback* is set, it
>     is called and the result is stored in the ``Snapshot.user_data``
>     attribute. Otherwise, ``Snapshot.user_data`` is set to ``None``.

The idea is to attach arbitrary data to a snapshot. Examples:

* size of Python caches: cache of linecache and re modules
* size of the internal Unicode intern dict
* gc.get_stats()
* gc.get_count()
* len(gc.get_objects())
* ("tracemalloc_size" should maybe moved to the user_data)

I hesitate to use a dictionary for user_data. The problem is to decice how to
display such data in DisplayTop. For example, gc.get_count() is a number
whereas tracemalloc_size is size is bytes (should be formatted using kB, MB,
etc. suffixes).


What do you think?

Victor

From ncoghlan at gmail.com  Tue Sep 17 16:46:01 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 18 Sep 2013 00:46:01 +1000
Subject: [Python-Dev] PEP 453: Explicit bootstrapping of pip
Message-ID: <CADiSq7f2yabBwu6_esKtvcs7tnFeifM7qdZCzO5ZAuevxsjo+w@mail.gmail.com>

After a couple of rounds of review on distutils-sig, and with Martin
agreeing to serve as BDFL-Delegate, it's time for the pip
bootstrapping proposal to run the gauntlet of python-dev :)

The last round of review showed that there were a few things we were
assuming people knew (based on the many, many discussions around this
topic on distutils-sig, starting even before Richard wrote PEP 439),
so I hope I've managed to cover those better in this version. It also
goes into more details on the proposed module API for getpip and the
associated API updates in venv.

There are still a couple of open questions related to the Windows
installers, and there may still be unasked questions affecting the Mac
OS X installers.

HTML version: http://www.python.org/dev/peps/pep-0453/

For those that reviewed the previous versions on distutils-sig, the
latest diff is here: http://hg.python.org/peps/rev/df9e4c301415

Cheers,
Nick.

==============================
PEP: 453
Title: Explicit bootstrapping of pip in Python installations
Version: $Revision$
Last-Modified: $Date$
Author: Donald Stufft <donald at stufft.io>,
        Nick Coghlan <ncoghlan at gmail.com>
BDFL-Delegate: Martin von L?wis
Status: Draft
Type: Process
Content-Type: text/x-rst
Created: 10-Aug-2013
Post-History: 30-Aug-2013, 15-Sep-2013, 18-Sep-2013


Abstract
========

This PEP proposes that the `pip`_ package manager be made available by
default when installing CPython and when creating virtual environments
using the standard library's ``venv`` module (including via the
``pyvenv`` command line utility).

To clearly demarcate development responsibilities, and to avoid
inadvertently downgrading ``pip`` when updating CPython, the proposed
mechanism to achieve this is to include an explicit `pip`_ bootstrapping
mechanism in the standard library that is invoked automatically by the
CPython installers provided on python.org.

The PEP also strongly recommends that CPython redistributors and other Python
implementations ensure that ``pip`` is available by default, or
at the very least, explicitly document the fact that it is not included.


Proposal
========

This PEP proposes the inclusion of a ``getpip`` bootstrapping module in
Python 3.4, as well as in the next maintenance releases of Python 3.3 and
2.7.

This PEP does *not* propose making pip (or any dependencies) part of the
standard library. Instead, pip will be a bundled application provided
along with CPython for the convenience of Python users, but subject to
its own development life cycle and able to be upgraded independently of
the core interpreter and standard library.


Rationale
=========

Currently, on systems without a platform package manager and repository,
installing a third-party Python package into a freshly installed Python
requires first identifying an appropriate package manager and then
installing it.

Even on systems that *do* have a platform package manager, it is unlikely to
include every package that is available on the Python Package Index, and
even when a desired third-party package is available, the correct name in
the platform package manager may not be clear.

This means that, to work effectively with the Python Package Index
ecosystem, users must know which package manager to install, where to get
it, and how to install it. The effect of this is that third-party Python
projects are currently required to choose from a variety of undesirable
alternatives:

* assume the user already has a suitable cross-platform package manager
  installed
* duplicate the instructions and tell their users how to install the
  package manager
* completely forgo the use of dependencies to ease installation concerns
  for their users

All of these available options have significant drawbacks.

If a project simply assumes a user already has the tooling then beginning
users may get a confusing error message when the installation command
doesn't work. Some operating systems may ease this pain by providing a
global hook that looks for commands that don't exist and suggest an OS
package they can install to make the command work, but that only works
on Linux systems with platform package managers. No such assistance is
availabe for Windows and Mac OS X users. The challenges of dealing with
this problem are a regular feature of feedback the core Python developers
receive from professional educators and others introducing new users to
Python.

If a project chooses to duplicate the installation instructions and tell
their users how to install the package manager before telling them how to
install their own project then whenever these instructions need updates
they need updating by every project that has duplicated them. This is
particular problematic when there are multiple competing installation
tools available, and different projects recommend different tools.

This specific problem can be partially alleviated by strongly promoting
``pip`` as the default installer and recommending that other projects
reference `pip's own bootstrapping instructions
<http://www.pip-installer.org/en/latest/installing.html>`__ rather than
duplicating them. However the user experience created by this approach
still isn't good (especially on Windows, where downloading and running
the ``get-pip.py`` bootstrap script with the default OS configuration is
significantly more painful than downloading and running a binary executable
or installer). The situation becomes even more complicated when multiple
Python versions are involved (for example, parallel installations of
Python 2 and Python 3), since that makes it harder to create and maintain
good platform specific ``pip`` installers independently of the CPython
installers.

The projects that have decided to forgo dependencies altogether are forced
to either duplicate the efforts of other projects by inventing their own
solutions to problems or are required to simply include the other projects
in their own source trees. Both of these options present their own problems
either in duplicating maintenance work across the ecosystem or potentially
leaving users vulnerable to security issues because the included code or
duplicated efforts are not automatically updated when upstream releases a new
version.

By providing a cross-platform package manager by default it will be easier
for users trying to install these third-party packages as well as easier
for the people distributing them as they should now be able to safely assume
that most users will have the appropriate installation tools available.
This is expected to become more important in the future as the Wheel_
package format (deliberately) does not have a built in "installer" in the
form of ``setup.py`` so users wishing to install from a wheel file will want
an installer even in the simplest cases.

Reducing the burden of actually installing a third-party package should
also decrease the pressure to add every useful module to the standard
library. This will allow additions to the standard library to focus more
on why Python should have a particular tool out of the box instead of
using the general difficulty of installing third-party packages as
justification for inclusion.

Providing a standard installation system also helps with bootstrapping
alternate build and installer systems, such as ``setuptools``,
``zc.buildout`` and the ``hashdist``/``conda`` combination that is aimed
specifically at the scientific community. So long as
``pip install <tool>`` works, then a standard Python-specific installer
provides a reasonably secure, cross platform mechanism to get access to
these utilities.


Why pip?
--------

``pip`` has been chosen as the preferred default installer, as it
addresses several design and user experience issues with its predecessor
``easy_install`` (these issues can't readily be fixed in ``easy_install``
itself due to backwards compatibility concerns). ``pip`` is also well suited
to working within the bounds of a single Python runtime installation
(including associated virtual environments), which is a desirable feature
for a tool bundled with CPython.

Other tools like ``zc.buildout`` and ``conda`` are more ambitious in their
aims (and hence substantially better than ``pip`` at handling external
binary dependencies), so it makes sense for the Python ecosystem to treat
them more like platform package managers to interoperate with rather than
as the default cross-platform installation tool. This relationship is
similar to that between ``pip`` and platform package management systems
like ``apt`` and ``yum`` (which are also designed to handle arbitrary
binary dependencies).


Explicit bootstrapping mechanism
================================

An additional module called ``getpip`` will be added to the standard library
whose purpose is to install pip and any of its dependencies into the
appropriate location (most commonly site-packages). It will expose a single
callable named ``bootstrap()`` as well as offer direct execution via
``python -m getpip``. Options for installing it such as index server,
installation location (``--user``, ``--root``, etc) will also be available
to enable different installation schemes.

It is believed that users will want the most recent versions available to be
installed so that they can take advantage of the new advances in packaging.
Since any particular version of Python has a much longer staying power than
a version of pip in order to satisfy a user's desire to have the most recent
version the bootstrap will (by default) contact PyPI, find the latest
version, download it, and then install it. This process is security
sensitive, difficult to get right, and evolves along with the rest of
packaging.

Instead of attempting to maintain a "mini pip" for the sole purpose of
installing pip, the ``getpip`` module will, as an implementation detail,
include a private copy of pip and its dependencies which will be used to
discover and install pip from PyPI. It is important to stress that this
private copy of pip is *only* an implementation detail and it should *not*
be relied on or assumed to exist.

Not all users will have network access to PyPI whenever they run the
bootstrap. In order to ensure that these users will still be able to
bootstrap pip the bootstrap will fallback to simply installing the included
copy of pip. The pip ``--no-download`` command line option will be supported
to force installation of the bundled version, without even attempting to
contact PyPI.

This presents a balance between giving users the latest version of pip,
saving them from needing to immediately upgrade pip after bootstrapping it,
and allowing the bootstrap to work offline in situations where users might
already have packages downloaded that they wish to install.


Proposed CLI
------------

The proposed CLI is based on a subset of the existing ``pip install``
options::

    Usage:
      python -m getpip [options]

    Download Options:
      --no-download           Install the bundled version, don't
attempt to download
      -i, --index-url <url>   Base URL of Python Package Index
(default https://pypi.python.org/simple/).
      --proxy <proxy>         Specify a proxy in the form
[user:passwd@]proxy.server:port.
      --timeout <sec>         Set the socket timeout (default 15 seconds).
      --cert <path>           Path to alternate CA bundle.

    Installation Options:
      -U, --upgrade           Upgrade pip and dependencies, even if
already installed
      --user                  Install using the user scheme.
      --root <dir>            Install everything relative to this
alternate root directory.

Additional options (such as verbosity and logging options) may also
be supported.


Proposed module API
-------------------

The proposed ``getpip`` module API is a single ``bootstrap`` function with
parameter names derived directly from the proposed CLI::

    def bootstrap(download=True, upgrade=False, root=None, user=False,
                  index_url=None, cert=None, proxy=None, timeout=15):
        """Bootstrap pip into the current Python installation (or the given
           root directory)"""

The only changes are to replace the ``--no-download`` opt-out option with
the True-by-default ``download`` option and to replace the hyphen in
``index-url`` with an underscore to create a legal Python identifier.


Invocation from the CPython installers
--------------------------------------

The CPython Windows and Mac OS X installers will each gain two new options:

* Install pip (the default Python package management utility)?
* Upgrade pip to the latest version (requires network access)?

Both options will be checked by default, with the option to upgrade pip
being available for selection only if the option to install pip is checked.

If both options are checked, then the installer will invoke the following
command with the just installed Python::

    python -m getpip --upgrade

If only the "Install pip" option is checked, then the following command will
be invoked::

    python -m getpip --upgrade --no-download

This ensures that, by default, installing or updating CPython will ensure
that either the latest available version of PyPI is installed (directly from
PyPI if permitted, otherwise whichever is more recent out of an already
installed version and the private copy inside ``getpip``)


Installing from source
----------------------

While the prebuilt binary installers will be updated to run
``python -m getpip`` by default, no such change will be made to the
``make install`` and ``make altinstall`` commands of the source distribution.

``getpip`` itself will still be installed normally (as it is a regular
part of the standard library), only the implicit installation of pip and
its dependencies will be skipped.

Keeping the pip bootstrapping as a separate step for ``make``-based
installations should minimize the changes CPython redistributors need to
make to their build processes. Avoiding the layer of indirection through
``make`` for the ``getpip`` invocation also ensures those installing from
a custom source build can easily force an offline installation of pip,
install it from a private index server, or skip installing pip entirely.


Changes to virtual environments
-------------------------------

Python 3.3 included a standard library approach to virtual Python environments
through the ``venv`` module. Since it's release it has become clear that very
few users have been willing to use this feature directly, in part due to the
lack of an installer present by default inside of the virtual environment.
They have instead opted to continue using the ``virtualenv`` package which
*does* include pip installed by default.

To make the ``venv`` more useful to users it will be modified to issue the
pip bootstrap by default inside of the new environment while creating it. This
will allow people the same convenience inside of the virtual environment as
this PEP provides outside of it as well as bringing the ``venv`` module closer
to feature parity with the external ``virtualenv`` package, making it a more
suitable replacement. To handle cases where a user does not wish to have pip
bootstrapped into their virtual environment a ``--without-pip`` option will be
added. The ``--no-download`` option will also be supported, to force the
use of the bundled ``pip`` rather than retrieving the latest version from
PyPI.

The ``venv.EnvBuilder`` and ``venv.create`` APIs will be updated to accept
two new parameters: ``with_pip`` (defaulting to ``False``) and
``bootstrap_options`` (accepting a dictionary of keyword arguments to
pass to ``getpip.bootstrap`` if ``with_pip`` is set,  defaulting to
``None``).

This particular change will be made only for Python 3.4 and later versions.
The third-party ``virtualenv`` project will still be needed to obtain a
consistent cross-version experience in Python 3.3 and 2.7.


Documentation
-------------

The "Installing Python Modules" section of the standard library
documentation will be updated to recommend the use of the bootstrapped
`pip` installer. It will give a brief description of the most common
commands and options, but delegate to the externally maintained ``pip``
documentation for the full details.

The existing content of the module installation guide will be retained,
but under a new "Invoking distutils directly" subsection.


Bundling CA certificates with CPython
-------------------------------------

The reference ``getpip`` implementation includes the ``pip`` CA
bundle along with the rest of pip. This means CPython effectively includes
a CA bundle that is used solely for ``getpip``.

This is considered desirable, as it ensures that ``pip`` will behave the
same across all supported versions of Python, even those prior to Python
3.4 that cannot access the system certificate store on Windows.


Automatic installation of setuptools
------------------------------------

``pip`` currently depends on ``setuptools`` to handle metadata generation
during the build process, along with some other features. While work is
ongoing to reduce or eliminate this dependency, it is not clear if that
work will be complete for pip 1.5 (which is the version likely to be current
when Python 3.4.0 is released).

This PEP proposes that, if pip still requires it as a dependency,
``getpip`` will include a private copy of ``setuptools`` (in addition
to the private copy of ``pip``). In normal operation, ``python -m getpip``
will then download and install the latest version of ``setuptools`` from
PyPI (as a dependency of ``pip``), while ``python -m getpip --no-download``
will install the private copy.

However, this behaviour is officially considered an implementation
detail. Other projects which explicitly require ``setuptools`` must still
provide an appropriate dependency declaration, rather than assuming
``setuptools`` will always be installed alongside ``pip``.

Once pip is able to run ``pip install --upgrade pip`` without needing
``setuptools`` installed first, then the private copy of ``setuptools``
will be removed from ``getpip``.


Updating the bundled pip
------------------------

In order to keep up with evolutions in packaging as well as providing users
who are using the offline installation method with as recent version a
possible the ``getpip`` module will be regularly updated to the latest
versions of everything it bootstraps.

After each new pip release, and again during the preparation for any
release of Python (including feature releases), a script, provided as part
of this PEP, will be run to ensure the private copies stored in the CPython
source repository have been updated to the latest versions.


Updating the getpip module API and CLI
--------------------------------------

Future security updates for pip and PyPI (for example, automatic
verification of package signatures) may also provide desirable security
enhancements for the ``getpip`` bootstrapping mechanism.

It is desirable that these features be made available in standard library
maintenance releases, not just new feature releases.

Accordingly, a slight relaxation of the usual "no new features in
maintenance releases" rule is proposed for the ``getpip`` module. This
relaxation also indirectly affects the new ``bootstrap_options`` parameter
in the ``venv`` module APIs.

Specifically, new security related flags will be permitted, with the
following restrictions:

- for compatibility with third-party usage of ``getpip`` module (for
  example, with a private index server), any such flag must be *off* by
  default in maintenance releases. It *should* be switched on by
  default in the next feature release.
- the CPython installers and the ``pyvenv`` CLI in the affected maintenance
  release should explicitly opt-in to the enhanced security features when
  automatically bootstrapping ``pip``

This means that maintenance releases of the CPython installers will
benefit from security enhancements by default, while avoiding breaking
customised usage of the bootstrap mechanism.


Feature addition in maintenance releases
========================================

Adding a new module to the standard library in Python 2.7 and 3.3
maintenance releases breaks the usual policy of "no new features in
maintenance releases".

It is being proposed in this case as the current bootstrapping issues for
the third-party Python package ecosystem greatly affects the experience of
new users, especially on Python 2 where many Python 3 standard library
improvements are available as backports on PyPI, but are not included in
the Python 2 standard library.

By updating Python 2.7, 3.3 and 3.4 to easily bootstrap the PyPI ecosystem,
this change should aid the vast majority of current Python users, rather
than only those with the freedom to adopt Python 3.4 as soon as it is
released.

This is also a matter of starting as we mean to continue: as noted above,
``getpip`` will have a limited permanent exemption from the "no new
features in maintenance releases" restriction, as it will include (and
rely on) upgraded private copies of ``pip`` and ``setuptools`` even in
maintenance releases, and may offer new security related options itself.


Open Question: Uninstallation
=============================

No changes are currently proposed to the uninstallation process. The
bootstrapped pip will be installed the same way as any other pip
installed packages, and will be handled in the same way as any other
post-install additions to the Python environment.

At least on Windows, that means the bootstrapped files will be
left behind after uninstallation, since those files won't be associated
with the Python MSI installer.

.. note::

   Perhaps the installer should be updated to clobber everything in
   site-packages and the Scripts directory when uninstalled (treating them
   as "data directories" from Python's point of view), but I would prefer
   not to make this PEP conditional on that change.


Open Question: Script Execution on Windows
==========================================

While the Windows installer was updated in Python 3.3 to optionally
make ``python`` available on the PATH, no such change was made to
include the Scripts directory. This PEP proposes that this installer option
be changed to also add the Scripts directory to PATH (either always, or
else as a checked by default suboption).

Without this change, the most reliable way to invoke pip on Windows (without
tinkering manually with PATH) is actually ``py -m pip`` (or ``py -3 -m pip``
to select the Python 3 version if both Python 2 and 3 are installed)
rather than simply calling ``pip``.

Adding the scripts directory to the system PATH would mean that ``pip``
works reliably in the "only one Python installation on the system PATH"
case, with ``py -m pip`` needed only to select a non-default version in
the parallel installation case (and outside a virtual environment).

While the script invocations on recent versions of Python will run through
the Python launcher for Windows, this shouldn't cause any issues, as long
as the Python files in the Scripts directory correctly specify a Python version
in their shebang line or have an adjacent Windows executable (as
``easy_install`` and ``pip`` do).


Recommendations for Downstream Distributors
===========================================

A common source of Python installations are through downstream distributors
such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX
package managers [#homebrew]_, or Python-specific tools [#conda]_. In order
to provide a consistent, user-friendly experience to all users of Python
regardless of how they attained Python this PEP recommends and asks that
downstream distributors:

* Ensure that whenever Python is installed pip is also installed.

  * This may take the form of separate packages with dependencies on each
    other so that installing the Python package installs the pip package
    and installing the pip package installs the Python package.

* Do not remove the bundled copy of pip.

  * This is required for offline installation of pip into a virtual
    environment by the ``venv`` module.
  * This is similar to the existing ``virtualenv`` package for which many
    downstream distributors have already made exception to the common
    "debundling" policy.
  * This does mean that if ``pip`` needs to be updated due to a security
    issue, so does the bundled version in the ``getpip`` bootstrap module
  * However, altering the bundled version of pip to remove the embedded
    CA certificate bundle and rely the system CA bundle instead is a
    reasonable change.

* Migrate build systems to utilize `pip`_ and `Wheel`_ instead of directly
  using ``setup.py``.

  * This will ensure that downstream packages can more easily utilize the
    new metadata formats which may not have a ``setup.py``.

* Ensure that all features of this PEP continue to work with any modifications
  made.

  * Online installation of the latest version of pip into a global or virtual
    python environment using ``python -m getpip``.
  * Offline installation of the bundled version of pip into a global or virtual
    python environment using ``python -m getpip``.
  * ``pip install --upgrade pip`` in a global installation should not affect
    any already created virtual environments.
  * ``pip install --upgrade pip`` in a virtual environment should not affect
    the global installation.

In the event that a Python redistributor chooses *not* to follow these
recommendations, we request that they explicitly document this fact and
provide their users with suitable guidance on translating upstream ``pip``
based installation instructions into something appropriate for the platform.

Other Python implementations are also encouraged to follow these guidelines
where applicable.


Policies & Governance
=====================

The maintainers of the bootstrapped software and the CPython core team will
work together in order to address the needs of both. The bootstrapped
software will still remain external to CPython and this PEP does not
include CPython subsuming the development responsibilities or design
decisions of the bootstrapped software. This PEP aims to decrease the
burden on end users wanting to use third-party packages and the
decisions inside it are pragmatic ones that represent the trust that the
Python community has already placed in the Python Packaging Authority as
the authors and maintainers of ``pip``, ``setuptools``, PyPI, ``virtualenv``
and other related projects.


Backwards Compatibility
-----------------------

Except for security enhancements (as noted above), the public API of the
``getpip`` module itself will fall under the typical backwards compatibility
policy of Python for its standard library. The externally developed software
that this PEP bundles does not.

Most importantly, this means that the bootstrapped version of pip may gain
new features in CPython maintenance releases, and pip continues to operate on
its own 6 month release cycle rather than CPython's 18-24 month cycle.


Security Releases
-----------------

Any security update that affects the ``getpip`` module will be shared prior to
release with the Python Security Response Team (security at python.org). The
PSRT will then decide if the reported issue warrants a security release of
CPython.


Appendix: Rejected Proposals
============================


Implicit bootstrap
------------------

`PEP439`_, the predecessor for this PEP, proposes its own solution. Its
solution involves shipping a fake ``pip`` command that when executed would
implicitly bootstrap and install pip if it does not already exist. This has
been rejected because it is too "magical". It hides from the end user when
exactly the pip command will be installed or that it is being installed at
all. It also does not provide any recommendations or considerations towards
downstream packagers who wish to manage the globally installed pip through
the mechanisms typical for their system.

The implicit bootstrap mechanism also ran into possible permissions issues,
if a user inadvertently attempted to bootstrap pip without write access to
the appropriate installation directories.


Including pip directly in the standard library
----------------------------------------------

Similar to this PEP is the proposal of just including pip in the standard
library. This would ensure that Python always includes pip and fixes all of the
end user facing problems with not having pip present by default. This has been
rejected because we've learned through the inclusion and history of
``distutils`` in the standard library that losing the ability to update the
packaging tools independently can leave the tooling in a state of constant
limbo. Making it unable to ever reasonably evolve in a timeframe that actually
affects users as any new features will not be available to the general
population for *years*.

Allowing the packaging tools to progress separately from the Python release
and adoption schedules allows the improvements to be used by *all* members
of the Python community and not just those able to live on the bleeding edge
of Python releases.

There have also been issues in the past with the "dual maintenance" problem
if a project continues to be maintained externally while *also* having a
fork maintained in the standard library. Since external maintenance of
``pip`` will always be needed to support earlier Python versions, the
proposed bootstrapping mechanism will becoming the explicit responsibility
of the CPython core developers (assisted by the pip developers), while
pip issues reported to the CPython tracker will be migrated to the pip
issue tracker. There will no doubt still be some user confusion over which
tracker to use, but hopefully less than has been seen historically when
including complete public copies of third-party projects in the standard
library.

Finally, the approach described in this PEP avoids some technical issues
related to handle CPython maintenance updates when pip has been independently
updated to a more recent version. The proposed pip-based bootstrapping
mechanism handles that automatically, since pip and the system installer
never get into a fight about who owns the pip installation (it is always
managed through pip, either directly, or indirectly via the getpip bootstrap
module).


Defaulting to --user installation
---------------------------------

Some consideration was given to bootstrapping pip into the per-user
site-packages directory by default. However, this behaviour would be
surprising (as it differs from the default behaviour of pip itself)
and is also not currently considered reliable (there are some edge cases
which are not handled correctly when pip is installed into the user
site-packages directory rather than the system site-packages).


.. _Wheel: http://www.python.org/dev/peps/pep-0427/
.. _pip: http://www.pip-installer.org
.. _setuptools: https://pypi.python.org/pypi/setuptools
.. _PEP439: http://www.python.org/dev/peps/pep-0439/


References
==========

.. [#ubuntu] `Ubuntu <http://www.ubuntu.com/>`
.. [#debian] `Debian <http://www.debian.org>`
.. [#fedora] `Fedora <https://fedoraproject.org/>`
.. [#homebrew] `Homebrew  <http://brew.sh/>`
.. [#conda] `Conda <http://www.continuum.io/blog/conda>`


Copyright
=========

This document has been placed in the public domain.


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From tjreedy at udel.edu  Tue Sep 17 17:37:48 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 17 Sep 2013 11:37:48 -0400
Subject: [Python-Dev] License() release list is imcomplete; intentional?
Message-ID: <l19t08$uj0$1@ger.gmane.org>

On 2.7, >>> license() return a text that includes a complete list of 
releases from 1.6 to 2.7 and stops there
     Release         Derived     Year        Owner       GPL-
                     from                                compatible? (1)

     0.9.0 thru 1.2              1991-1995   CWI         yes
     1.3 thru 1.5.2  1.2         1995-1999   CNRI        yes
     1.6             1.5.2       2000        CNRI        no
     2.0             1.6         2000        BeOpen.com  no
...
     2.6.5           2.6.4       2010        PSF         yes
     2.7             2.6         2010        PSF         yes

Was it intentional to stop with 2.7 and not continue with 2.7.1, etc?

On 3.3.2, the 2.x list ends with 2.6.5 and never mentions 2.7. Intentional?
It then jumps back to 3.0 and ends with the 'previous' release, 3.3.1. 
Should 3.3.2 be included in the 3.3.2 list?

...
     2.6.4           2.6.3       2009        PSF         yes
     2.6.5           2.6.4       2010        PSF         yes
     3.0             2.6         2008        PSF         yes
     3.0.1           3.0         2009        PSF         yes
...
     3.2.4           3.2.3       2013        PSF         yes
     3.3.0           3.2         2012        PSF         yes
     3.3.1           3.3.0       2013        PSF         yes

-- 
Terry Jan Reedy


From solipsis at pitrou.net  Tue Sep 17 17:47:36 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 17 Sep 2013 17:47:36 +0200
Subject: [Python-Dev] License() release list is imcomplete; intentional?
References: <l19t08$uj0$1@ger.gmane.org>
Message-ID: <20130917174736.38068e19@pitrou.net>

Le Tue, 17 Sep 2013 11:37:48 -0400,
Terry Reedy <tjreedy at udel.edu> a ?crit :

> On 2.7, >>> license() return a text that includes a complete list of 
> releases from 1.6 to 2.7 and stops there
>      Release         Derived     Year        Owner       GPL-
>                      from                                compatible?
> (1)
> 
>      0.9.0 thru 1.2              1991-1995   CWI         yes
>      1.3 thru 1.5.2  1.2         1995-1999   CNRI        yes
>      1.6             1.5.2       2000        CNRI        no
>      2.0             1.6         2000        BeOpen.com  no
> ...
>      2.6.5           2.6.4       2010        PSF         yes
>      2.7             2.6         2010        PSF         yes
> 
> Was it intentional to stop with 2.7 and not continue with 2.7.1, etc?
> 
> On 3.3.2, the 2.x list ends with 2.6.5 and never mentions 2.7.
> Intentional? It then jumps back to 3.0 and ends with the 'previous'
> release, 3.3.1. Should 3.3.2 be included in the 3.3.2 list?
> 
> ...
>      2.6.4           2.6.3       2009        PSF         yes
>      2.6.5           2.6.4       2010        PSF         yes
>      3.0             2.6         2008        PSF         yes
>      3.0.1           3.0         2009        PSF         yes
> ...
>      3.2.4           3.2.3       2013        PSF         yes
>      3.3.0           3.2         2012        PSF         yes
>      3.3.1           3.3.0       2013        PSF         yes

I don't really understand why the releases should be manually listed.
Is it some kind of defensive coding?

Regards

Antoine.






From python at mrabarnett.plus.com  Tue Sep 17 17:48:44 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 17 Sep 2013 16:48:44 +0100
Subject: [Python-Dev] License() release list is imcomplete; intentional?
In-Reply-To: <l19t08$uj0$1@ger.gmane.org>
References: <l19t08$uj0$1@ger.gmane.org>
Message-ID: <523879DC.5070402@mrabarnett.plus.com>

On 17/09/2013 16:37, Terry Reedy wrote:
> On 2.7, >>> license() return a text that includes a complete list of
> releases from 1.6 to 2.7 and stops there
>       Release         Derived     Year        Owner       GPL-
>                       from                                compatible? (1)
>
>       0.9.0 thru 1.2              1991-1995   CWI         yes
>       1.3 thru 1.5.2  1.2         1995-1999   CNRI        yes
>       1.6             1.5.2       2000        CNRI        no
>       2.0             1.6         2000        BeOpen.com  no
> ...
>       2.6.5           2.6.4       2010        PSF         yes
>       2.7             2.6         2010        PSF         yes
>
> Was it intentional to stop with 2.7 and not continue with 2.7.1, etc?
>
> On 3.3.2, the 2.x list ends with 2.6.5 and never mentions 2.7. Intentional?
> It then jumps back to 3.0 and ends with the 'previous' release, 3.3.1.
> Should 3.3.2 be included in the 3.3.2 list?
>
> ...
>       2.6.4           2.6.3       2009        PSF         yes
>       2.6.5           2.6.4       2010        PSF         yes
>       3.0             2.6         2008        PSF         yes
>       3.0.1           3.0         2009        PSF         yes
> ...
>       3.2.4           3.2.3       2013        PSF         yes
>       3.3.0           3.2         2012        PSF         yes
>       3.3.1           3.3.0       2013        PSF         yes
>
To me it looks like Python 2.7 isn't mentioned in the licence for
Python 3 because Python 3 is derived from Python 2.6 and Python 2.7 is
on a different branch, so it isn't relevant.

From rdmurray at bitdance.com  Tue Sep 17 18:25:42 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Tue, 17 Sep 2013 12:25:42 -0400
Subject: [Python-Dev] PEP 453: Explicit bootstrapping of pip
In-Reply-To: <CADiSq7f2yabBwu6_esKtvcs7tnFeifM7qdZCzO5ZAuevxsjo+w@mail.gmail.com>
References: <CADiSq7f2yabBwu6_esKtvcs7tnFeifM7qdZCzO5ZAuevxsjo+w@mail.gmail.com>
Message-ID: <20130917162543.2AC0D250909@webabinitio.net>

This PEP looks great to me.  Thanks to everyone involved.

On Wed, 18 Sep 2013 00:46:01 +1000, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Reducing the burden of actually installing a third-party package should
> also decrease the pressure to add every useful module to the standard
> library. This will allow additions to the standard library to focus more
> on why Python should have a particular tool out of the box instead of
> using the general difficulty of installing third-party packages as
> justification for inclusion.

I don't think this PEP changes the inclusion calculus, because I don't
think we've given any real weight to that in stdlib inclusion decisions.
I think the decision making is already focused on why Python should have
a particular tool out of the box.

The only context I recall where "installation difficulty" has been
considered relevant is the issue of corporate users restricted from
installing 3rd party packages...which will not be affected at all by
this PEP.

--David

From donald at stufft.io  Tue Sep 17 18:38:31 2013
From: donald at stufft.io (Donald Stufft)
Date: Tue, 17 Sep 2013 12:38:31 -0400
Subject: [Python-Dev] PEP 453: Explicit bootstrapping of pip
In-Reply-To: <20130917162543.2AC0D250909@webabinitio.net>
References: <CADiSq7f2yabBwu6_esKtvcs7tnFeifM7qdZCzO5ZAuevxsjo+w@mail.gmail.com>
 <20130917162543.2AC0D250909@webabinitio.net>
Message-ID: <877B56B4-16B2-4694-8CEF-469B0C16EDB7@stufft.io>


On Sep 17, 2013, at 12:25 PM, "R. David Murray" <rdmurray at bitdance.com> wrote:

> This PEP looks great to me.  Thanks to everyone involved.
> 
> On Wed, 18 Sep 2013 00:46:01 +1000, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> Reducing the burden of actually installing a third-party package should
>> also decrease the pressure to add every useful module to the standard
>> library. This will allow additions to the standard library to focus more
>> on why Python should have a particular tool out of the box instead of
>> using the general difficulty of installing third-party packages as
>> justification for inclusion.
> 
> I don't think this PEP changes the inclusion calculus, because I don't
> think we've given any real weight to that in stdlib inclusion decisions.
> I think the decision making is already focused on why Python should have
> a particular tool out of the box.

When I originally wrote this statement I wasn't trying to say that modules
were included because of the lack of an easy installer, but instead that
a not insignificant number of people had wanted to include stuff and some
of their reasoning that I had seen in the past included the difficulty in
bootstrapping the installer.

So it decreases the *pressure* from others to include stuff for connivence
not that it necessarily changes how python-dev itself weighs the decision
to do so (but it might for some people?).

> 
> The only context I recall where "installation difficulty" has been
> considered relevant is the issue of corporate users restricted from
> installing 3rd party packages...which will not be affected at all by
> this PEP.
> 
> --David
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130917/b13f1454/attachment.sig>

From skip at pobox.com  Tue Sep 17 18:42:22 2013
From: skip at pobox.com (Skip Montanaro)
Date: Tue, 17 Sep 2013 11:42:22 -0500
Subject: [Python-Dev] License() release list is imcomplete; intentional?
In-Reply-To: <20130917174736.38068e19@pitrou.net>
References: <l19t08$uj0$1@ger.gmane.org> <20130917174736.38068e19@pitrou.net>
Message-ID: <CANc-5UzmYe_CwZR+ns5ftK5u74V-N+HB2JZ2VjTByYKXAJ62+Q@mail.gmail.com>

> I don't really understand why the releases should be manually listed.
> Is it some kind of defensive coding?

I think it's to give people who care about such things all the
information they need to make informed decisions. As I recall, the 1.6
series was problematic, because it wasn't actually open source. That's
why 2.0 is a derivative of 1.5.2. I doubt there are other licensing
weirdities lurking, but better to give people everything they might
need, especially if it's not difficult. At this point I think the hard
work is done. Release managers or their minions just need to remember
to tack on the next line.

Skip

From guido at python.org  Tue Sep 17 18:47:48 2013
From: guido at python.org (Guido van Rossum)
Date: Tue, 17 Sep 2013 09:47:48 -0700
Subject: [Python-Dev] License() release list is imcomplete; intentional?
In-Reply-To: <20130917174736.38068e19@pitrou.net>
References: <l19t08$uj0$1@ger.gmane.org> <20130917174736.38068e19@pitrou.net>
Message-ID: <CAP7+vJJ7L8KG0zQ-gOzbMAV3KcLu3bnZC3i0tAezy1UNBvsNMA@mail.gmail.com>

On Tue, Sep 17, 2013 at 8:47 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> Le Tue, 17 Sep 2013 11:37:48 -0400,
> Terry Reedy <tjreedy at udel.edu> a ?crit :
>
> > On 2.7, >>> license() return a text that includes a complete list of
> > releases from 1.6 to 2.7 and stops there
> >      Release         Derived     Year        Owner       GPL-
> >                      from                                compatible?
> > (1)
> >
> >      0.9.0 thru 1.2              1991-1995   CWI         yes
> >      1.3 thru 1.5.2  1.2         1995-1999   CNRI        yes
> >      1.6             1.5.2       2000        CNRI        no
> >      2.0             1.6         2000        BeOpen.com  no
> > ...
> >      2.6.5           2.6.4       2010        PSF         yes
> >      2.7             2.6         2010        PSF         yes
> >
> > Was it intentional to stop with 2.7 and not continue with 2.7.1, etc?
> >
> > On 3.3.2, the 2.x list ends with 2.6.5 and never mentions 2.7.
> > Intentional? It then jumps back to 3.0 and ends with the 'previous'
> > release, 3.3.1. Should 3.3.2 be included in the 3.3.2 list?
> >
> > ...
> >      2.6.4           2.6.3       2009        PSF         yes
> >      2.6.5           2.6.4       2010        PSF         yes
> >      3.0             2.6         2008        PSF         yes
> >      3.0.1           3.0         2009        PSF         yes
> > ...
> >      3.2.4           3.2.3       2013        PSF         yes
> >      3.3.0           3.2         2012        PSF         yes
> >      3.3.1           3.3.0       2013        PSF         yes
>
> I don't really understand why the releases should be manually listed.
> Is it some kind of defensive coding?
>

Worse, it's superstition based on myth.

IIRC this table was added when a few core Python developers including
myself left CNRI in 2000. We had a bit of an argument about the license
(not too much though -- in the end things came out alright). Some lawyer at
CNRI thought it was a good idea to record a release history like this with
the license, as a defense against whatever claims of ownership to the code
someone else might suddenly come up with. Since all I wanted was to get out
of there while causing them minimal upset, I told them I'd comply. But
that's over 13 years ago now, and I'm not sure if it ever made sense (the
internet is a different place than CNRI's lawyers envisioned). Only the top
10 of so lines of the table are in the least interesting (note that it
describes a graph). I propose that we truncate the table and add a note
saying that all following releases are owned by the PSF, GPL-compatible,
and derived from previous PSF-owned and GPL-compatible releases. That
should do until the PSF goes out of business (which I hope will never
happen -- this is one reason why I wish the conferences were run by a
separate entity, to avoid a conference bankruptcy from risking Python's
continued open-source status).

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130917/1a267881/attachment.html>

From solipsis at pitrou.net  Tue Sep 17 19:39:26 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 17 Sep 2013 19:39:26 +0200
Subject: [Python-Dev] License() release list is imcomplete; intentional?
In-Reply-To: <CAP7+vJJ7L8KG0zQ-gOzbMAV3KcLu3bnZC3i0tAezy1UNBvsNMA@mail.gmail.com>
References: <l19t08$uj0$1@ger.gmane.org> <20130917174736.38068e19@pitrou.net>
 <CAP7+vJJ7L8KG0zQ-gOzbMAV3KcLu3bnZC3i0tAezy1UNBvsNMA@mail.gmail.com>
Message-ID: <20130917193926.0d180351@fsol>

On Tue, 17 Sep 2013 09:47:48 -0700
Guido van Rossum <guido at python.org> wrote:
> 
> IIRC this table was added when a few core Python developers including
> myself left CNRI in 2000. We had a bit of an argument about the license
> (not too much though -- in the end things came out alright). Some lawyer at
> CNRI thought it was a good idea to record a release history like this with
> the license, as a defense against whatever claims of ownership to the code
> someone else might suddenly come up with. Since all I wanted was to get out
> of there while causing them minimal upset, I told them I'd comply. But
> that's over 13 years ago now, and I'm not sure if it ever made sense (the
> internet is a different place than CNRI's lawyers envisioned). Only the top
> 10 of so lines of the table are in the least interesting (note that it
> describes a graph). I propose that we truncate the table and add a note
> saying that all following releases are owned by the PSF, GPL-compatible,
> and derived from previous PSF-owned and GPL-compatible releases. That
> should do until the PSF goes out of business (which I hope will never
> happen -- this is one reason why I wish the conferences were run by a
> separate entity, to avoid a conference bankruptcy from risking Python's
> continued open-source status).

The PSF isn't technically the copyright holder, so would that pose a
significant threat?
(at worse the PSF could relicense Python based on the copyright
agreements, but Python would still be distributable under the original
license)

Regards

Antoine.

From guido at python.org  Tue Sep 17 19:51:28 2013
From: guido at python.org (Guido van Rossum)
Date: Tue, 17 Sep 2013 10:51:28 -0700
Subject: [Python-Dev] License() release list is imcomplete; intentional?
In-Reply-To: <20130917193926.0d180351@fsol>
References: <l19t08$uj0$1@ger.gmane.org> <20130917174736.38068e19@pitrou.net>
 <CAP7+vJJ7L8KG0zQ-gOzbMAV3KcLu3bnZC3i0tAezy1UNBvsNMA@mail.gmail.com>
 <20130917193926.0d180351@fsol>
Message-ID: <CAP7+vJKeTDEZa4kFy18j6vO++6PB-swxf+KNL0pcBj-Erqqzcg@mail.gmail.com>

On Tue, Sep 17, 2013 at 10:39 AM, Antoine Pitrou <solipsis at pitrou.net>wrote:

> On Tue, 17 Sep 2013 09:47:48 -0700
> Guido van Rossum <guido at python.org> wrote:
> >
> > IIRC this table was added when a few core Python developers including
> > myself left CNRI in 2000. We had a bit of an argument about the license
> > (not too much though -- in the end things came out alright). Some lawyer
> at
> > CNRI thought it was a good idea to record a release history like this
> with
> > the license, as a defense against whatever claims of ownership to the
> code
> > someone else might suddenly come up with. Since all I wanted was to get
> out
> > of there while causing them minimal upset, I told them I'd comply. But
> > that's over 13 years ago now, and I'm not sure if it ever made sense (the
> > internet is a different place than CNRI's lawyers envisioned). Only the
> top
> > 10 of so lines of the table are in the least interesting (note that it
> > describes a graph). I propose that we truncate the table and add a note
> > saying that all following releases are owned by the PSF, GPL-compatible,
> > and derived from previous PSF-owned and GPL-compatible releases. That
> > should do until the PSF goes out of business (which I hope will never
> > happen -- this is one reason why I wish the conferences were run by a
> > separate entity, to avoid a conference bankruptcy from risking Python's
> > continued open-source status).
>
> The PSF isn't technically the copyright holder, so would that pose a
> significant threat?
>

I have no idea. If you want a real answer talk to a lawyer.


> (at worse the PSF could relicense Python based on the copyright
> agreements, but Python would still be distributable under the original
> license)
>

But only the PSF has the list of original contributors and their licenses.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130917/c5f0dfd7/attachment.html>

From rdmurray at bitdance.com  Tue Sep 17 20:52:33 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Tue, 17 Sep 2013 14:52:33 -0400
Subject: [Python-Dev] PEP 453: Explicit bootstrapping of pip
In-Reply-To: <877B56B4-16B2-4694-8CEF-469B0C16EDB7@stufft.io>
References: <CADiSq7f2yabBwu6_esKtvcs7tnFeifM7qdZCzO5ZAuevxsjo+w@mail.gmail.com>
 <20130917162543.2AC0D250909@webabinitio.net>
 <877B56B4-16B2-4694-8CEF-469B0C16EDB7@stufft.io>
Message-ID: <20130917185233.8191E250512@webabinitio.net>

On Tue, 17 Sep 2013 12:38:31 -0400, Donald Stufft <donald at stufft.io> wrote:
> On Sep 17, 2013, at 12:25 PM, "R. David Murray" <rdmurray at bitdance.com> wrote:
> > I don't think this PEP changes the inclusion calculus, because I don't
> > think we've given any real weight to that in stdlib inclusion decisions.
> > I think the decision making is already focused on why Python should have
> > a particular tool out of the box.
> 
> When I originally wrote this statement I wasn't trying to say that modules
> were included because of the lack of an easy installer, but instead that
> a not insignificant number of people had wanted to include stuff and some
> of their reasoning that I had seen in the past included the difficulty in
> bootstrapping the installer.
> 
> So it decreases the *pressure* from others to include stuff for connivence
> not that it necessarily changes how python-dev itself weighs the decision
> to do so (but it might for some people?).

I see.  Yes, that makes sense.

--David

From v+python at g.nevcal.com  Tue Sep 17 20:45:33 2013
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Tue, 17 Sep 2013 11:45:33 -0700
Subject: [Python-Dev] License() release list is imcomplete; intentional?
In-Reply-To: <CAP7+vJKeTDEZa4kFy18j6vO++6PB-swxf+KNL0pcBj-Erqqzcg@mail.gmail.com>
References: <l19t08$uj0$1@ger.gmane.org> <20130917174736.38068e19@pitrou.net>
 <CAP7+vJJ7L8KG0zQ-gOzbMAV3KcLu3bnZC3i0tAezy1UNBvsNMA@mail.gmail.com>
 <20130917193926.0d180351@fsol>
 <CAP7+vJKeTDEZa4kFy18j6vO++6PB-swxf+KNL0pcBj-Erqqzcg@mail.gmail.com>
Message-ID: <5238A34D.6030408@g.nevcal.com>

On 9/17/2013 10:51 AM, Guido van Rossum wrote:
> But only the PSF has the list of original contributors and their 
> licenses. 

So can that list be made public, and available in multiple archives?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130917/6f12820e/attachment.html>

From brett at python.org  Tue Sep 17 21:39:25 2013
From: brett at python.org (Brett Cannon)
Date: Tue, 17 Sep 2013 15:39:25 -0400
Subject: [Python-Dev] License() release list is imcomplete; intentional?
In-Reply-To: <5238A34D.6030408@g.nevcal.com>
References: <l19t08$uj0$1@ger.gmane.org> <20130917174736.38068e19@pitrou.net>
 <CAP7+vJJ7L8KG0zQ-gOzbMAV3KcLu3bnZC3i0tAezy1UNBvsNMA@mail.gmail.com>
 <20130917193926.0d180351@fsol>
 <CAP7+vJKeTDEZa4kFy18j6vO++6PB-swxf+KNL0pcBj-Erqqzcg@mail.gmail.com>
 <5238A34D.6030408@g.nevcal.com>
Message-ID: <CAP1=2W4D0_iwq1zhq06Z4PKrNAfEfXVGKoHoB4GRNdR2g0JTfA@mail.gmail.com>

On Tue, Sep 17, 2013 at 2:45 PM, Glenn Linderman <v+python at g.nevcal.com>wrote:

>  On 9/17/2013 10:51 AM, Guido van Rossum wrote:
>
> But only the PSF has the list of original contributors and their licenses.
>
>
> So can that list be made public, and available in multiple archives?
>

Technically it already is public based on the * next to your name on
bugs.python.org. As for a concise list that's more human readable I'm sure
it could if someone chose to put the effort in to write the code to make
the list.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130917/881cfe71/attachment-0001.html>

From tjreedy at udel.edu  Tue Sep 17 21:40:58 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 17 Sep 2013 15:40:58 -0400
Subject: [Python-Dev] License() release list is imcomplete; intentional?
In-Reply-To: <523879DC.5070402@mrabarnett.plus.com>
References: <l19t08$uj0$1@ger.gmane.org> <523879DC.5070402@mrabarnett.plus.com>
Message-ID: <l1ab86$jt7$1@ger.gmane.org>

On 9/17/2013 11:48 AM, MRAB wrote:
> On 17/09/2013 16:37, Terry Reedy wrote:
>> On 2.7, >>> license() return a text that includes a complete list of
>> releases from 1.6 to 2.7 and stops there
>>       Release         Derived     Year        Owner       GPL-
>>                       from                                compatible? (1)
>>
>>       0.9.0 thru 1.2              1991-1995   CWI         yes
>>       1.3 thru 1.5.2  1.2         1995-1999   CNRI        yes
>>       1.6             1.5.2       2000        CNRI        no
>>       2.0             1.6         2000        BeOpen.com  no
>> ...
>>       2.6.5           2.6.4       2010        PSF         yes
>>       2.7             2.6         2010        PSF         yes
>>
>> Was it intentional to stop with 2.7 and not continue with 2.7.1, etc?
>>
>> On 3.3.2, the 2.x list ends with 2.6.5 and never mentions 2.7.
>> Intentional?
>> It then jumps back to 3.0 and ends with the 'previous' release, 3.3.1.
>> Should 3.3.2 be included in the 3.3.2 list?
>>
>> ...
>>       2.6.4           2.6.3       2009        PSF         yes
>>       2.6.5           2.6.4       2010        PSF         yes
>>       3.0             2.6         2008        PSF         yes
>>       3.0.1           3.0         2009        PSF         yes
>> ...
>>       3.2.4           3.2.3       2013        PSF         yes
>>       3.3.0           3.2         2012        PSF         yes
>>       3.3.1           3.3.0       2013        PSF         yes
>>
> To me it looks like Python 2.7 isn't mentioned in the licence for
> Python 3 because Python 3 is derived from Python 2.6 and Python 2.7 is
> on a different branch, so it isn't relevant.

I thought of that, but if that were the reason, nothing after 2.6(.0) 
should be listed. I think we should follow Guido's idea of truncating 
after n lines, summarizing the rest, and freezing the file until there 
is a significant change.

-- 
Terry Jan Reedy


From rdmurray at bitdance.com  Tue Sep 17 22:46:12 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Tue, 17 Sep 2013 16:46:12 -0400
Subject: [Python-Dev] License() release list is imcomplete; intentional?
In-Reply-To: <CAP1=2W4D0_iwq1zhq06Z4PKrNAfEfXVGKoHoB4GRNdR2g0JTfA@mail.gmail.com>
References: <l19t08$uj0$1@ger.gmane.org> <20130917174736.38068e19@pitrou.net>
 <CAP7+vJJ7L8KG0zQ-gOzbMAV3KcLu3bnZC3i0tAezy1UNBvsNMA@mail.gmail.com>
 <20130917193926.0d180351@fsol>
 <CAP7+vJKeTDEZa4kFy18j6vO++6PB-swxf+KNL0pcBj-Erqqzcg@mail.gmail.com>
 <5238A34D.6030408@g.nevcal.com>
 <CAP1=2W4D0_iwq1zhq06Z4PKrNAfEfXVGKoHoB4GRNdR2g0JTfA@mail.gmail.com>
Message-ID: <20130917204613.376D8250818@webabinitio.net>

On Tue, 17 Sep 2013 15:39:25 -0400, Brett Cannon <brett at python.org> wrote:
> On Tue, Sep 17, 2013 at 2:45 PM, Glenn Linderman <v+python at g.nevcal.com>wrote:
> >  On 9/17/2013 10:51 AM, Guido van Rossum wrote:
> >
> > But only the PSF has the list of original contributors and their licenses.
> >
> > So can that list be made public, and available in multiple archives?
> >
> 
> Technically it already is public based on the * next to your name on
> bugs.python.org. As for a concise list that's more human readable I'm sure
> it could if someone chose to put the effort in to write the code to make
> the list.

The list is public in that sense (but probably not complete).  However,
whatever is considered to be the legal info, including the license the
contributor chose, is not in the tracker.  That information is in the
hands of the PSF secretary, and to my (outsider, and therefore possibly
incorrect :) understanding most of it is currently in the form of physical
paper in a filing cabinet.

I don't know if there is a project to digitize the archives or not,
but it would seem like a sensible thing to do, for backup reasons if
nothing else (perhaps there are already physical backups, though).

--David

From guido at python.org  Tue Sep 17 23:21:16 2013
From: guido at python.org (Guido van Rossum)
Date: Tue, 17 Sep 2013 14:21:16 -0700
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <CAP7+vJKPDCZd4ZQn6mSkiAx7=wjv7WmmP3HjbOj-FLGUY-yNRg@mail.gmail.com>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
 <CAP7+vJJ6QxcUwcpdt3Mq6r07Q2dGUQt-VWGdsYGLv4wrRW5dGg@mail.gmail.com>
 <20130914005921.GK16820@ando>
 <CAP7+vJL81VR=kdO+2rCz3Di3GVNSwhkRCXrw2HAL=7ywT42rvA@mail.gmail.com>
 <20130916235912.GG19939@ando>
 <CAP7+vJKPDCZd4ZQn6mSkiAx7=wjv7WmmP3HjbOj-FLGUY-yNRg@mail.gmail.com>
Message-ID: <CAP7+vJ+rtwSmKXG4Pwq7jD7Xegdka6dGhGVKW8Lf3PtWLCCpXA@mail.gmail.com>

Congrats, I've accepted the PEP. Nice work! Please work with the reviewers
on the issue on the code.

(Steven or Oscar, if either of you could work Oscar's list of resolved
issues into a patch for the PEP I'll happily update it, just mail it to
peps at python.org.)


On Mon, Sep 16, 2013 at 5:06 PM, Guido van Rossum <guido at python.org> wrote:

> On Mon, Sep 16, 2013 at 4:59 PM, Steven D'Aprano <steve at pearwood.info>wrote:
>
>> On Mon, Sep 16, 2013 at 08:42:12AM -0700, Guido van Rossum wrote:
>> > I'm ready to accept this PEP. Because I haven't read this entire thread
>> > (and 60 messages about random diversions is really too much to try and
>> > catch up on) I'll give people 24 hours to remind me of outstanding
>> > rejections.
>> >
>> > I also haven't reviewed the code in any detail, but I believe the code
>> > review is going well, so I'm not concerned that the PEP would have to
>> > revised based on that alone.
>>
>> There are a couple of outstanding issues that I am aware of, but I don't
>> believe that either of these affect acceptance/rejection of the PEP.
>> Please correct me if I am wrong.
>>
>> 1) Implementation details of the statistics.sum function. Oscar is
>> giving me a lot of very valuable assistance speeding up the
>> implementation of sum.
>>
>> 2) The current implementation has extensive docstrings, but will also
>> need a separate statistics.rst file.
>>
>>
>> I don't recall any other outstanding issues, if I have forgotten any,
>> please remind me.
>>
>
> Those certainly don't stand in the way of the PEP's acceptance (but they
> do block the commit of the code :-).
>
> The issues that Oscar listed also all seem resolved (though they would
> make a nice addition to the "Discussion" section in the PEP).
>
> --
> --Guido van Rossum (python.org/~guido)
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130917/4b35d7ec/attachment.html>

From steve at pearwood.info  Wed Sep 18 00:05:55 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 18 Sep 2013 08:05:55 +1000
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <CAP7+vJ+rtwSmKXG4Pwq7jD7Xegdka6dGhGVKW8Lf3PtWLCCpXA@mail.gmail.com>
References: <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
 <CAP7+vJJ6QxcUwcpdt3Mq6r07Q2dGUQt-VWGdsYGLv4wrRW5dGg@mail.gmail.com>
 <20130914005921.GK16820@ando>
 <CAP7+vJL81VR=kdO+2rCz3Di3GVNSwhkRCXrw2HAL=7ywT42rvA@mail.gmail.com>
 <20130916235912.GG19939@ando>
 <CAP7+vJKPDCZd4ZQn6mSkiAx7=wjv7WmmP3HjbOj-FLGUY-yNRg@mail.gmail.com>
 <CAP7+vJ+rtwSmKXG4Pwq7jD7Xegdka6dGhGVKW8Lf3PtWLCCpXA@mail.gmail.com>
Message-ID: <20130917220554.GH19939@ando>

On Tue, Sep 17, 2013 at 02:21:16PM -0700, Guido van Rossum wrote:
> Congrats, I've accepted the PEP. Nice work! Please work with the reviewers
> on the issue on the code.

Thank you, and thanks to everyone who contributed with the discussion.

> (Steven or Oscar, if either of you could work Oscar's list of resolved
> issues into a patch for the PEP I'll happily update it, just mail it to
> peps at python.org.)

I will do this.



-- 
Steven

From ethan at stoneleaf.us  Tue Sep 17 23:40:21 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 17 Sep 2013 14:40:21 -0700
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <CAP7+vJ+rtwSmKXG4Pwq7jD7Xegdka6dGhGVKW8Lf3PtWLCCpXA@mail.gmail.com>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
 <CAP7+vJJ6QxcUwcpdt3Mq6r07Q2dGUQt-VWGdsYGLv4wrRW5dGg@mail.gmail.com>
 <20130914005921.GK16820@ando>
 <CAP7+vJL81VR=kdO+2rCz3Di3GVNSwhkRCXrw2HAL=7ywT42rvA@mail.gmail.com>
 <20130916235912.GG19939@ando>
 <CAP7+vJKPDCZd4ZQn6mSkiAx7=wjv7WmmP3HjbOj-FLGUY-yNRg@mail.gmail.com>
 <CAP7+vJ+rtwSmKXG4Pwq7jD7Xegdka6dGhGVKW8Lf3PtWLCCpXA@mail.gmail.com>
Message-ID: <5238CC45.2070104@stoneleaf.us>

On 09/17/2013 02:21 PM, Guido van Rossum wrote:
> Congrats, I've accepted the PEP. Nice work! Please work with the reviewers on the issue on the code.

Congratulations, Stephen!

--
~Ethan~

From victor.stinner at gmail.com  Wed Sep 18 01:17:21 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 18 Sep 2013 01:17:21 +0200
Subject: [Python-Dev] benchmark
Message-ID: <CAMpsgwa=Tq3OtbQVn6ieO3fp77tHX-ywoNqBXa_sY9gxhOMWDw@mail.gmail.com>


-------------- next part --------------
A non-text attachment was scrubbed...
Name: benchmark.patch
Type: application/octet-stream
Size: 36048 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130918/871cef31/attachment-0001.obj>

From ncoghlan at gmail.com  Wed Sep 18 01:44:24 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 18 Sep 2013 09:44:24 +1000
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <5238CC45.2070104@stoneleaf.us>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
 <CAP7+vJJ6QxcUwcpdt3Mq6r07Q2dGUQt-VWGdsYGLv4wrRW5dGg@mail.gmail.com>
 <20130914005921.GK16820@ando>
 <CAP7+vJL81VR=kdO+2rCz3Di3GVNSwhkRCXrw2HAL=7ywT42rvA@mail.gmail.com>
 <20130916235912.GG19939@ando>
 <CAP7+vJKPDCZd4ZQn6mSkiAx7=wjv7WmmP3HjbOj-FLGUY-yNRg@mail.gmail.com>
 <CAP7+vJ+rtwSmKXG4Pwq7jD7Xegdka6dGhGVKW8Lf3PtWLCCpXA@mail.gmail.com>
 <5238CC45.2070104@stoneleaf.us>
Message-ID: <CADiSq7e9a5GzUQoEwJ+MGnXVdkGNd5044xx4YN4_z6bWmFRHnw@mail.gmail.com>

On 18 Sep 2013 08:36, "Ethan Furman" <ethan at stoneleaf.us> wrote:
>
> On 09/17/2013 02:21 PM, Guido van Rossum wrote:
>>
>> Congrats, I've accepted the PEP. Nice work! Please work with the
reviewers on the issue on the code.
>
>
> Congratulations, Stephen!

Yay!

Cheers,
Nick.

>
> --
> ~Ethan~
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130918/b03541b0/attachment.html>

From pjenvey at underboss.org  Wed Sep 18 03:10:53 2013
From: pjenvey at underboss.org (Philip Jenvey)
Date: Tue, 17 Sep 2013 18:10:53 -0700
Subject: [Python-Dev] PEP 428: Pathlib -> stat caching
In-Reply-To: <20130916220546.341e369f@fsol>
References: <5235BA20.8010105@stoneleaf.us>
 <20130916101508.135926de@pitrou.net>
 <CAH_1eM1uo0W29xy-pJPOpe-Orfc830M3BUd3coBO1dDdHSxekg@mail.gmail.com>
 <20130916214536.55907f19@fsol>
 <CAP1=2W5yq_t2iMiuptkW5w7x7kiowVCFXM4qQUEHug1QVAx6ug@mail.gmail.com>
 <20130916220546.341e369f@fsol>
Message-ID: <BEA6DBE1-BB54-44A7-9E83-CD9AA8CB533A@underboss.org>


On Sep 16, 2013, at 1:05 PM, Antoine Pitrou wrote:

> On Mon, 16 Sep 2013 15:48:54 -0400
> Brett Cannon <brett at python.org> wrote:
>>> 
>>> So I would like to propose the following API change:
>>> 
>>> - Path.stat() (and stat-accessing methods such as get_mtime()...)
>>>  returns an uncached stat object by default
>>> 
>>> - Path.cache_stat() can be called to return the stat() *and* cache it
>>>  for future use, such that any future call to stat(), cache_stat() or
>>>  a stat-accessing function reuses that cached stat
>>> 
>>> In other words, only if you use cache_stat() at least once is the
>>> stat() value cached and reused by the Path object.
>>> (also, it's a per-Path decision)
>>> 
>> 
>> Any reason why stat() can't get a keyword-only cached=True argument
>> instead? Or have stat() never cache() but stat_cache() always so that
>> people can choose if they want fresh or cached based on API and not whether
>> some library happened to make a decision for them?
> 
> 1. Because you also want the helper functions (get_mtime(), etc.) to
> cache the value too. It's not only about stat().

With the proposed rich stat object the convenience methods living on Path wouldn't result in much added convenience:

p.is_dir() vs p.stat().is_dir()

Why not move these methods from Path to a rich stat obj and not cache stat results at all? It's easy enough for users to cache them themselves and much more explicit.

--
Philip Jenvey


From ncoghlan at gmail.com  Wed Sep 18 03:32:14 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 18 Sep 2013 11:32:14 +1000
Subject: [Python-Dev] PEP 428: Pathlib -> stat caching
In-Reply-To: <BEA6DBE1-BB54-44A7-9E83-CD9AA8CB533A@underboss.org>
References: <5235BA20.8010105@stoneleaf.us>
 <20130916101508.135926de@pitrou.net>
 <CAH_1eM1uo0W29xy-pJPOpe-Orfc830M3BUd3coBO1dDdHSxekg@mail.gmail.com>
 <20130916214536.55907f19@fsol>
 <CAP1=2W5yq_t2iMiuptkW5w7x7kiowVCFXM4qQUEHug1QVAx6ug@mail.gmail.com>
 <20130916220546.341e369f@fsol>
 <BEA6DBE1-BB54-44A7-9E83-CD9AA8CB533A@underboss.org>
Message-ID: <CADiSq7fuXZMFjaEJKyFd9O1O=z+hzkesAFtxrHVa5m-f2WbOeA@mail.gmail.com>

On 18 September 2013 11:10, Philip Jenvey <pjenvey at underboss.org> wrote:
>
> On Sep 16, 2013, at 1:05 PM, Antoine Pitrou wrote:
>
>> On Mon, 16 Sep 2013 15:48:54 -0400
>> Brett Cannon <brett at python.org> wrote:
>>>>
>>>> So I would like to propose the following API change:
>>>>
>>>> - Path.stat() (and stat-accessing methods such as get_mtime()...)
>>>>  returns an uncached stat object by default
>>>>
>>>> - Path.cache_stat() can be called to return the stat() *and* cache it
>>>>  for future use, such that any future call to stat(), cache_stat() or
>>>>  a stat-accessing function reuses that cached stat
>>>>
>>>> In other words, only if you use cache_stat() at least once is the
>>>> stat() value cached and reused by the Path object.
>>>> (also, it's a per-Path decision)
>>>>
>>>
>>> Any reason why stat() can't get a keyword-only cached=True argument
>>> instead? Or have stat() never cache() but stat_cache() always so that
>>> people can choose if they want fresh or cached based on API and not whether
>>> some library happened to make a decision for them?
>>
>> 1. Because you also want the helper functions (get_mtime(), etc.) to
>> cache the value too. It's not only about stat().
>
> With the proposed rich stat object the convenience methods living on Path wouldn't result in much added convenience:
>
> p.is_dir() vs p.stat().is_dir()
>
> Why not move these methods from Path to a rich stat obj and not cache stat results at all? It's easy enough for users to cache them themselves and much more explicit.

Because that doesn't help iterator based os.walk inspired APIs like
walkdir, which would benefit greatly from a path type with implicit
caching, but would have to complicate their APIs significantly to pass
around separate stat objects.

Rewriting walkdir to depend on pathlib has been on my todo list for a
while, as it solves a potentially serious walkdir performance problem
where chained iterators have to make repeated stat calls to answer
questions that were already asked by earlier iterators in the
pipeline.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From oscar.j.benjamin at gmail.com  Wed Sep 18 04:32:20 2013
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Wed, 18 Sep 2013 03:32:20 +0100
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <CAP7+vJ+rtwSmKXG4Pwq7jD7Xegdka6dGhGVKW8Lf3PtWLCCpXA@mail.gmail.com>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
 <CAP7+vJJ6QxcUwcpdt3Mq6r07Q2dGUQt-VWGdsYGLv4wrRW5dGg@mail.gmail.com>
 <20130914005921.GK16820@ando>
 <CAP7+vJL81VR=kdO+2rCz3Di3GVNSwhkRCXrw2HAL=7ywT42rvA@mail.gmail.com>
 <20130916235912.GG19939@ando>
 <CAP7+vJKPDCZd4ZQn6mSkiAx7=wjv7WmmP3HjbOj-FLGUY-yNRg@mail.gmail.com>
 <CAP7+vJ+rtwSmKXG4Pwq7jD7Xegdka6dGhGVKW8Lf3PtWLCCpXA@mail.gmail.com>
Message-ID: <CAHVvXxQC7eKnLUvx8vfBsRu7Yhut2XMkUrkTnUi_SPNPqVy0Ew@mail.gmail.com>

On 17 September 2013 22:21, Guido van Rossum <guido at python.org> wrote:
> Congrats, I've accepted the PEP. Nice work! Please work with the reviewers
> on the issue on the code.

Good work, Steven!

From greg at krypto.org  Wed Sep 18 08:43:57 2013
From: greg at krypto.org (Gregory P. Smith)
Date: Tue, 17 Sep 2013 23:43:57 -0700
Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4
In-Reply-To: <CAO41-mMUQD=K7WbSPcHSZk6jh_K3zv=Aa-YAGL2dRrwckCqcOg@mail.gmail.com>
References: <70C99F87-E9A5-4838-A1E9-4739FBF2E6A5@gmail.com>
 <nad-EC5D57.13320314092013@news.gmane.org>
 <B3293155-E4D5-4389-A555-C31BC49CE539@gmail.com>
 <rowen-8E335B.12310916092013@news.gmane.org> <51403.1379361402@parc.com>
 <CAO41-mMUQD=K7WbSPcHSZk6jh_K3zv=Aa-YAGL2dRrwckCqcOg@mail.gmail.com>
Message-ID: <CAGE7PNJYjxhu2YkLyiUcrm8PdhYUmMLOGTYzc1WXm3K2+tqgVQ@mail.gmail.com>

Just drop support for 10.6 with Python 3.4. Problem solved. People on that
old of a version of the OS can build their own Python 3.4 or do the right
thing and upgrade or just install Linux.

This isn't Windows. Compiler tool chains are freely available for the
legacy platform. We don't need to maintain such a long legacy support tail
there ourselves.

-gps


On Mon, Sep 16, 2013 at 1:28 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:

> Meh...I hate it when tools download stuff without me noticing.
>
> Honestly, a separate 10.6 build would work well. Plus, if a new Clang
> versions includes some awesome feature that could make Python builds
> better, you'd be able to take advantage of it better.
>
>
> On Mon, Sep 16, 2013 at 2:56 PM, Bill Janssen <janssen at parc.com> wrote:
>
>> Russell E. Owen <rowen at uw.edu> wrote:
>>
>> > In article <B3293155-E4D5-4389-A555-C31BC49CE539 at gmail.com>,
>> >  Raymond Hettinger <raymond.hettinger at gmail.com> wrote:
>> >
>> > > On Sep 14, 2013, at 1:32 PM, Ned Deily <nad at acm.org> wrote:
>> > > >  The
>> > > > most recent Developer Tools for 10.8 and 10.7 systems, Xcode 4.6.x,
>> have
>> > > > a mature clang but do not provide a 10.6 SDK.  Even with using an
>> SDK,
>> > > > it's still possible to end up inadvertently linking with the wrong
>> > > > versions of system libraries.  We have been burned by that in the
>> past.
>> > >
>> > > I think we should offer a separate Mac build just for 10.6
>> > > (much like we do for the 32-bit PPC option for 10.5).
>> >
>> > If Apple drops support for gcc in 10.9 I guess we have to go this route,
>>
>> Could go the Sage route -- Sage first checks for an up-to-date version
>> of gcc, and downloads it and builds it for its own use if necessary.
>>
>> Bill
>>
>> > but please be careful. Every time you add a new version of python for
>> > MacOS X it means that folks providing binary installers (e.g. for numpy)
>> > have to provide another binary, and folks using those installers have
>> > another chance of picking the wrong one.
>> >
>> > If you do make a 10.6-only installer, what is the minimum version of
>> > MacOS X the modern compiler would support? 10.7 gives a more measured
>> > upgrade path, but 10.8 gives a better compiler.
>> >
>> > -- Russell
>> >
>> > _______________________________________________
>> > Python-Dev mailing list
>> > Python-Dev at python.org
>> > https://mail.python.org/mailman/listinfo/python-dev
>> > Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/bill%40janssen.org
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com
>>
>
>
>
> --
> Ryan
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130917/bc7b8df1/attachment.html>

From victor.stinner at gmail.com  Wed Sep 18 10:14:17 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 18 Sep 2013 10:14:17 +0200
Subject: [Python-Dev] benchmark
In-Reply-To: <CAMpsgwa=Tq3OtbQVn6ieO3fp77tHX-ywoNqBXa_sY9gxhOMWDw@mail.gmail.com>
References: <CAMpsgwa=Tq3OtbQVn6ieO3fp77tHX-ywoNqBXa_sY9gxhOMWDw@mail.gmail.com>
Message-ID: <CAMpsgwY7-GhgEfC32nWAX+wyQftCN9ABTM6qESPzQ8iC10DZew@mail.gmail.com>

Oops, I wanted to send the patch to myself, not to the python-dev
mailing list, sorry :-)

(It's a set of patch to try to optimize the tracemalloc module.)

Victor

2013/9/18 Victor Stinner <victor.stinner at gmail.com>:
>

From solipsis at pitrou.net  Wed Sep 18 10:26:06 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 18 Sep 2013 10:26:06 +0200
Subject: [Python-Dev] PEP 450 adding statistics module
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
 <CAP7+vJJ6QxcUwcpdt3Mq6r07Q2dGUQt-VWGdsYGLv4wrRW5dGg@mail.gmail.com>
 <20130914005921.GK16820@ando>
 <CAP7+vJL81VR=kdO+2rCz3Di3GVNSwhkRCXrw2HAL=7ywT42rvA@mail.gmail.com>
 <20130916235912.GG19939@ando>
 <CAP7+vJKPDCZd4ZQn6mSkiAx7=wjv7WmmP3HjbOj-FLGUY-yNRg@mail.gmail.com>
 <CAP7+vJ+rtwSmKXG4Pwq7jD7Xegdka6dGhGVKW8Lf3PtWLCCpXA@mail.gmail.com>
 <5238CC45.2070104@stoneleaf.us>
Message-ID: <20130918102606.45e0f175@pitrou.net>

Le Tue, 17 Sep 2013 14:40:21 -0700,
Ethan Furman <ethan at stoneleaf.us> a ?crit :
> On 09/17/2013 02:21 PM, Guido van Rossum wrote:
> > Congrats, I've accepted the PEP. Nice work! Please work with the
> > reviewers on the issue on the code.
> 
> Congratulations, Stephen!

Or Steven even ;)



From solipsis at pitrou.net  Wed Sep 18 10:26:43 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 18 Sep 2013 10:26:43 +0200
Subject: [Python-Dev] PEP 428: Pathlib -> stat caching
References: <5235BA20.8010105@stoneleaf.us>
 <20130916101508.135926de@pitrou.net>
 <CAH_1eM1uo0W29xy-pJPOpe-Orfc830M3BUd3coBO1dDdHSxekg@mail.gmail.com>
 <20130916214536.55907f19@fsol>
 <CAP1=2W5yq_t2iMiuptkW5w7x7kiowVCFXM4qQUEHug1QVAx6ug@mail.gmail.com>
 <20130916220546.341e369f@fsol>
 <BEA6DBE1-BB54-44A7-9E83-CD9AA8CB533A@underboss.org>
Message-ID: <20130918102643.7be29a61@pitrou.net>

Le Tue, 17 Sep 2013 18:10:53 -0700,
Philip Jenvey <pjenvey at underboss.org> a ?crit :
> 
> On Sep 16, 2013, at 1:05 PM, Antoine Pitrou wrote:
> 
> > On Mon, 16 Sep 2013 15:48:54 -0400
> > Brett Cannon <brett at python.org> wrote:
> >>> 
> >>> So I would like to propose the following API change:
> >>> 
> >>> - Path.stat() (and stat-accessing methods such as get_mtime()...)
> >>>  returns an uncached stat object by default
> >>> 
> >>> - Path.cache_stat() can be called to return the stat() *and*
> >>> cache it for future use, such that any future call to stat(),
> >>> cache_stat() or a stat-accessing function reuses that cached stat
> >>> 
> >>> In other words, only if you use cache_stat() at least once is the
> >>> stat() value cached and reused by the Path object.
> >>> (also, it's a per-Path decision)
> >>> 
> >> 
> >> Any reason why stat() can't get a keyword-only cached=True argument
> >> instead? Or have stat() never cache() but stat_cache() always so
> >> that people can choose if they want fresh or cached based on API
> >> and not whether some library happened to make a decision for them?
> > 
> > 1. Because you also want the helper functions (get_mtime(), etc.) to
> > cache the value too. It's not only about stat().
> 
> With the proposed rich stat object the convenience methods living on
> Path wouldn't result in much added convenience:
> 
> p.is_dir() vs p.stat().is_dir()

One reason is that the proposed rich stat object doesn't exist yet :-)

Regards

Antoine.



From g.brandl at gmx.net  Wed Sep 18 10:54:28 2013
From: g.brandl at gmx.net (Georg Brandl)
Date: Wed, 18 Sep 2013 10:54:28 +0200
Subject: [Python-Dev] License() release list is imcomplete; intentional?
In-Reply-To: <l19t08$uj0$1@ger.gmane.org>
References: <l19t08$uj0$1@ger.gmane.org>
Message-ID: <l1bpmn$j54$1@ger.gmane.org>

On 09/17/2013 05:37 PM, Terry Reedy wrote:
> On 2.7, >>> license() return a text that includes a complete list of 
> releases from 1.6 to 2.7 and stops there
>      Release         Derived     Year        Owner       GPL-
>                      from                                compatible? (1)
> 
>      0.9.0 thru 1.2              1991-1995   CWI         yes
>      1.3 thru 1.5.2  1.2         1995-1999   CNRI        yes
>      1.6             1.5.2       2000        CNRI        no
>      2.0             1.6         2000        BeOpen.com  no
> ...
>      2.6.5           2.6.4       2010        PSF         yes
>      2.7             2.6         2010        PSF         yes
> 
> Was it intentional to stop with 2.7 and not continue with 2.7.1, etc?
> 
> On 3.3.2, the 2.x list ends with 2.6.5 and never mentions 2.7. Intentional?
> It then jumps back to 3.0 and ends with the 'previous' release, 3.3.1. 
> Should 3.3.2 be included in the 3.3.2 list?
> 
> ...
>      2.6.4           2.6.3       2009        PSF         yes
>      2.6.5           2.6.4       2010        PSF         yes
>      3.0             2.6         2008        PSF         yes
>      3.0.1           3.0         2009        PSF         yes
> ...
>      3.2.4           3.2.3       2013        PSF         yes
>      3.3.0           3.2         2012        PSF         yes
>      3.3.1           3.3.0       2013        PSF         yes

Since there are 3 versions of this table, it's unavoidable that one of them
gets out of sync :)

* LICENSE
* Doc/license.rst
* the versions for each release on the website

So I wholeheartedly support truncating before 2.2 (which is the first
release where all micro versions are PSF owned and GPL compatible) and
just saying "2.2 onwards" (or whatever is the best way to say it).

cheers,
Georg


From ethan at stoneleaf.us  Wed Sep 18 11:55:10 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 18 Sep 2013 02:55:10 -0700
Subject: [Python-Dev] PEP 450 adding statistics module
In-Reply-To: <20130918102606.45e0f175@pitrou.net>
References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info>
 <20130908123706.GG26319@ando>
 <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com>
 <522CA085.6000601@stoneleaf.us>
 <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com>
 <CAP7+vJJ6QxcUwcpdt3Mq6r07Q2dGUQt-VWGdsYGLv4wrRW5dGg@mail.gmail.com>
 <20130914005921.GK16820@ando>
 <CAP7+vJL81VR=kdO+2rCz3Di3GVNSwhkRCXrw2HAL=7ywT42rvA@mail.gmail.com>
 <20130916235912.GG19939@ando>
 <CAP7+vJKPDCZd4ZQn6mSkiAx7=wjv7WmmP3HjbOj-FLGUY-yNRg@mail.gmail.com>
 <CAP7+vJ+rtwSmKXG4Pwq7jD7Xegdka6dGhGVKW8Lf3PtWLCCpXA@mail.gmail.com>
 <5238CC45.2070104@stoneleaf.us> <20130918102606.45e0f175@pitrou.net>
Message-ID: <5239787E.8080408@stoneleaf.us>

On 09/18/2013 01:26 AM, Antoine Pitrou wrote:
> Le Tue, 17 Sep 2013 14:40:21 -0700,
> Ethan Furman <ethan at stoneleaf.us> a ?crit :
>> On 09/17/2013 02:21 PM, Guido van Rossum wrote:
>>> Congrats, I've accepted the PEP. Nice work! Please work with the
>>> reviewers on the issue on the code.
>>
>> Congratulations, Stephen!
>
> Or Steven even ;)

*sigh*  I used to be a good speller, too!

--
~Ethan~

From martin at v.loewis.de  Wed Sep 18 14:54:32 2013
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 18 Sep 2013 14:54:32 +0200
Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4
In-Reply-To: <CAGE7PNJYjxhu2YkLyiUcrm8PdhYUmMLOGTYzc1WXm3K2+tqgVQ@mail.gmail.com>
References: <70C99F87-E9A5-4838-A1E9-4739FBF2E6A5@gmail.com>
 <nad-EC5D57.13320314092013@news.gmane.org>
 <B3293155-E4D5-4389-A555-C31BC49CE539@gmail.com>
 <rowen-8E335B.12310916092013@news.gmane.org> <51403.1379361402@parc.com>
 <CAO41-mMUQD=K7WbSPcHSZk6jh_K3zv=Aa-YAGL2dRrwckCqcOg@mail.gmail.com>
 <CAGE7PNJYjxhu2YkLyiUcrm8PdhYUmMLOGTYzc1WXm3K2+tqgVQ@mail.gmail.com>
Message-ID: <5239A288.2050403@v.loewis.de>

Am 18.09.13 08:43, schrieb Gregory P. Smith:
> Just drop support for 10.6 with Python 3.4. Problem solved. People on
> that old of a version of the OS can build their own Python 3.4 or do the
> right thing and upgrade or just install Linux.
> 
> This isn't Windows. Compiler tool chains are freely available for the
> legacy platform. We don't need to maintain such a long legacy support
> tail there ourselves.

I don't mind such a decision in principle, but also in principle, I'd
prefer if there was a pre-set policy to decide this question, documented
in PEP 11.

Here a piece of OSX release history:
- 10.5:       October 2007
  * 10.5.8:   August 2009
- 10.6:       August 2009
  * 10.6.8:   July 2011
- 10.7:       July 2011
  * 10.7.5:   July 2012
- 10.8:       July 2012

So possible policy that would now exclude 10.6 for binary installers
would be:

- only the two latest feature releases are supported
- only feature releases younger than 3 years are supported

Note that a separate policy should be added to decide whether support
for older versions is actively removed from source code (or equally
bug fixes for old versions are not accepted anymore).

Regards,
Martin


From martin at v.loewis.de  Wed Sep 18 15:03:19 2013
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 18 Sep 2013 15:03:19 +0200
Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4
In-Reply-To: <534e7dd6-e7e2-49cd-b719-8a070852b623@email.android.com>
References: <70C99F87-E9A5-4838-A1E9-4739FBF2E6A5@gmail.com>
 <nad-EC5D57.13320314092013@news.gmane.org>
 <B3293155-E4D5-4389-A555-C31BC49CE539@gmail.com>
 <534e7dd6-e7e2-49cd-b719-8a070852b623@email.android.com>
Message-ID: <5239A497.4070204@v.loewis.de>

Am 15.09.13 00:56, schrieb Ryan:
> +1. A 10.6-only build makes sense.

I'd like to support Russell's point: this could put a burden on everyone
releasing extension modules to also provide two binary releases, which
e.g. would then mess up downloads from PyPI. So -1.

+0 on dropping 10.6 support from the binary installers as proposed
by Greg.

Regards,
Martin


From solipsis at pitrou.net  Wed Sep 18 15:36:44 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 18 Sep 2013 15:36:44 +0200
Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4
References: <70C99F87-E9A5-4838-A1E9-4739FBF2E6A5@gmail.com>
 <nad-EC5D57.13320314092013@news.gmane.org>
 <B3293155-E4D5-4389-A555-C31BC49CE539@gmail.com>
 <rowen-8E335B.12310916092013@news.gmane.org>
 <51403.1379361402@parc.com>
 <CAO41-mMUQD=K7WbSPcHSZk6jh_K3zv=Aa-YAGL2dRrwckCqcOg@mail.gmail.com>
 <CAGE7PNJYjxhu2YkLyiUcrm8PdhYUmMLOGTYzc1WXm3K2+tqgVQ@mail.gmail.com>
 <5239A288.2050403@v.loewis.de>
Message-ID: <20130918153644.73eae75f@pitrou.net>

Le Wed, 18 Sep 2013 14:54:32 +0200,
"Martin v. L?wis" <martin at v.loewis.de> a ?crit :
> Am 18.09.13 08:43, schrieb Gregory P. Smith:
> > Just drop support for 10.6 with Python 3.4. Problem solved. People
> > on that old of a version of the OS can build their own Python 3.4
> > or do the right thing and upgrade or just install Linux.
> > 
> > This isn't Windows. Compiler tool chains are freely available for
> > the legacy platform. We don't need to maintain such a long legacy
> > support tail there ourselves.
> 
> I don't mind such a decision in principle, but also in principle, I'd
> prefer if there was a pre-set policy to decide this question,
> documented in PEP 11.
> 
> Here a piece of OSX release history:
> - 10.5:       October 2007
>   * 10.5.8:   August 2009
> - 10.6:       August 2009
>   * 10.6.8:   July 2011
> - 10.7:       July 2011
>   * 10.7.5:   July 2012
> - 10.8:       July 2012

This means that any Mac shipped pre-July 2011 has 10.6 or lower?
I know Mac users are fashion victims, but 2011 doesn't sound that old
to me ;-)
(it's even younger than Barry!)

Regards

Antoine.



From ronaldoussoren at mac.com  Wed Sep 18 15:41:56 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Wed, 18 Sep 2013 13:41:56 +0000 (GMT)
Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4
In-Reply-To: <20130918153644.73eae75f@pitrou.net>
Message-ID: <2f235f3a-af73-48b0-9d9f-26047f7175ad@me.com>



On Sep 18, 2013, at 03:36 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

Le Wed, 18 Sep 2013 14:54:32 +0200,
"Martin v. L?wis" <martin at v.loewis.de> a ?crit :
Am 18.09.13 08:43, schrieb Gregory P. Smith:
> Just drop support for 10.6 with Python 3.4. Problem solved. People
> on that old of a version of the OS can build their own Python 3.4
> or do the right thing and upgrade or just install Linux.
>
> This isn't Windows. Compiler tool chains are freely available for
> the legacy platform. We don't need to maintain such a long legacy
> support tail there ourselves.
I don't mind such a decision in principle, but also in principle, I'd
prefer if there was a pre-set policy to decide this question,
documented in PEP 11.
Here a piece of OSX release history:
- 10.5: October 2007
* 10.5.8: August 2009
- 10.6: August 2009
* 10.6.8: July 2011
- 10.7: July 2011
* 10.7.5: July 2012
- 10.8: July 2012

This means that any Mac shipped pre-July 2011 has 10.6 or lower?
I know Mac users are fashion victims, but 2011 doesn't sound that old
to me ;-)
(it's even younger than Barry!)
?
That's correct, and IMHO it is too early to drop support for 10.6 and doubly so when it is only done to use a newer compiler for the binary build.

Still-using-10.5-in-production-ly yours,

? ? Ronald


Regards

Antoine.


_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130918/ef1a6b42/attachment.html>

From rowen at uw.edu  Wed Sep 18 15:58:01 2013
From: rowen at uw.edu (Russell Owen)
Date: Wed, 18 Sep 2013 06:58:01 -0700
Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4
In-Reply-To: <5239A288.2050403@v.loewis.de>
References: <70C99F87-E9A5-4838-A1E9-4739FBF2E6A5@gmail.com>
 <nad-EC5D57.13320314092013@news.gmane.org>
 <B3293155-E4D5-4389-A555-C31BC49CE539@gmail.com>
 <rowen-8E335B.12310916092013@news.gmane.org> <51403.1379361402@parc.com>
 <CAO41-mMUQD=K7WbSPcHSZk6jh_K3zv=Aa-YAGL2dRrwckCqcOg@mail.gmail.com>
 <CAGE7PNJYjxhu2YkLyiUcrm8PdhYUmMLOGTYzc1WXm3K2+tqgVQ@mail.gmail.com>
 <5239A288.2050403@v.loewis.de>
Message-ID: <9829E582-9A68-4679-8087-31D42069F20D@uw.edu>

I agree that a policy is a good idea, and I suggest it be primarily based on age, since we cannot assume Apple will release new versions of the OS on a given timeline.

I personally think too early to drop support for MacOS X 10.6 and am on the edge about 10.5.

-- Russell

On Sep 18, 2013, at 5:54 AM, "Martin v. L?wis" <martin at v.loewis.de> wrote:

> Am 18.09.13 08:43, schrieb Gregory P. Smith:
>> Just drop support for 10.6 with Python 3.4. Problem solved. People on
>> that old of a version of the OS can build their own Python 3.4 or do the
>> right thing and upgrade or just install Linux.
>> 
>> This isn't Windows. Compiler tool chains are freely available for the
>> legacy platform. We don't need to maintain such a long legacy support
>> tail there ourselves.
> 
> I don't mind such a decision in principle, but also in principle, I'd
> prefer if there was a pre-set policy to decide this question, documented
> in PEP 11.
> 
> Here a piece of OSX release history:
> - 10.5:       October 2007
>  * 10.5.8:   August 2009
> - 10.6:       August 2009
>  * 10.6.8:   July 2011
> - 10.7:       July 2011
>  * 10.7.5:   July 2012
> - 10.8:       July 2012
> 
> So possible policy that would now exclude 10.6 for binary installers
> would be:
> 
> - only the two latest feature releases are supported
> - only feature releases younger than 3 years are supported
> 
> Note that a separate policy should be added to decide whether support
> for older versions is actively removed from source code (or equally
> bug fixes for old versions are not accepted anymore).
> 
> Regards,
> Martin
> 


From ronaldoussoren at mac.com  Wed Sep 18 15:38:48 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Wed, 18 Sep 2013 13:38:48 +0000 (GMT)
Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4
In-Reply-To: <5239A497.4070204@v.loewis.de>
Message-ID: <8279a538-028f-45ae-a762-25302c52e370@me.com>



On Sep 18, 2013, at 03:03 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:

Am 15.09.13 00:56, schrieb Ryan:
+1. A 10.6-only build makes sense.

I'd like to support Russell's point: this could put a burden on everyone
releasing extension modules to also provide two binary releases, which
e.g. would then mess up downloads from PyPI. So -1.

+0 on dropping 10.6 support from the binary installers as proposed
by Greg.
?

Dropping support for 10.6 isn't really necessary, just build (carefully) on a later release of OSX (preferably the last one to pick up the most recent developer tools). ??

I'll try to provide a more detailed response tonight, but the basic idea is to use OSX 10.8, the current Xcode release and the OSX 10.8 SDK to do the build. This is a supported configuration for building binaries that target older OSX releases, but some care is needed with the configure script for 3th-party libraries: those might pick up features that aren't available in 10.6.?

Ronald



Regards,
Martin

_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130918/e50ddff8/attachment.html>

From guido at python.org  Wed Sep 18 16:57:22 2013
From: guido at python.org (Guido van Rossum)
Date: Wed, 18 Sep 2013 07:57:22 -0700
Subject: [Python-Dev] License() release list is imcomplete; intentional?
In-Reply-To: <l1bpmn$j54$1@ger.gmane.org>
References: <l19t08$uj0$1@ger.gmane.org> <l1bpmn$j54$1@ger.gmane.org>
Message-ID: <CAP7+vJJLZ93sEs3nR6NHy7m8A_14e54qOO-8iA9p1=N77t7BhA@mail.gmail.com>

OK. Let's do it.


On Wed, Sep 18, 2013 at 1:54 AM, Georg Brandl <g.brandl at gmx.net> wrote:

> On 09/17/2013 05:37 PM, Terry Reedy wrote:
> > On 2.7, >>> license() return a text that includes a complete list of
> > releases from 1.6 to 2.7 and stops there
> >      Release         Derived     Year        Owner       GPL-
> >                      from                                compatible? (1)
> >
> >      0.9.0 thru 1.2              1991-1995   CWI         yes
> >      1.3 thru 1.5.2  1.2         1995-1999   CNRI        yes
> >      1.6             1.5.2       2000        CNRI        no
> >      2.0             1.6         2000        BeOpen.com  no
> > ...
> >      2.6.5           2.6.4       2010        PSF         yes
> >      2.7             2.6         2010        PSF         yes
> >
> > Was it intentional to stop with 2.7 and not continue with 2.7.1, etc?
> >
> > On 3.3.2, the 2.x list ends with 2.6.5 and never mentions 2.7.
> Intentional?
> > It then jumps back to 3.0 and ends with the 'previous' release, 3.3.1.
> > Should 3.3.2 be included in the 3.3.2 list?
> >
> > ...
> >      2.6.4           2.6.3       2009        PSF         yes
> >      2.6.5           2.6.4       2010        PSF         yes
> >      3.0             2.6         2008        PSF         yes
> >      3.0.1           3.0         2009        PSF         yes
> > ...
> >      3.2.4           3.2.3       2013        PSF         yes
> >      3.3.0           3.2         2012        PSF         yes
> >      3.3.1           3.3.0       2013        PSF         yes
>
> Since there are 3 versions of this table, it's unavoidable that one of them
> gets out of sync :)
>
> * LICENSE
> * Doc/license.rst
> * the versions for each release on the website
>
> So I wholeheartedly support truncating before 2.2 (which is the first
> release where all micro versions are PSF owned and GPL compatible) and
> just saying "2.2 onwards" (or whatever is the best way to say it).
>
> cheers,
> Georg
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130918/5466a52e/attachment.html>

From skip at pobox.com  Wed Sep 18 17:04:45 2013
From: skip at pobox.com (Skip Montanaro)
Date: Wed, 18 Sep 2013 10:04:45 -0500
Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4
In-Reply-To: <8279a538-028f-45ae-a762-25302c52e370@me.com>
References: <5239A497.4070204@v.loewis.de>
 <8279a538-028f-45ae-a762-25302c52e370@me.com>
Message-ID: <CANc-5UwCtaFGJ40PVkKzE+PtroRxz=yQ0coQU5irVoALH5Okog@mail.gmail.com>

As a MacBook Pro user running Snow Leopard/10.6.8, I would find the
lack of a binary release problematic, were it not for the fact that I
routinely build from source (and don't do anything Mac-specific with
Python). For those not familiar with the platform, it's perhaps worth
noting that upgrade is out of my hands. Apple dropped support for my
"ancient" MBP (2006?), so 10.6.8 is as up-to-date as I can get. I
guess that's what you expect from a company whose main revenues these
days come from cell phones and tablets, and addition of pastels to
their color pallet warrants announcement in a TV commercial.

I suspect other Mac users stuck on Snow Leopard who are not Python
developers would rue the lack of binary installers more than me.

Skip

From janssen at parc.com  Wed Sep 18 18:31:09 2013
From: janssen at parc.com (Bill Janssen)
Date: Wed, 18 Sep 2013 09:31:09 -0700
Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4
In-Reply-To: <9829E582-9A68-4679-8087-31D42069F20D@uw.edu>
References: <70C99F87-E9A5-4838-A1E9-4739FBF2E6A5@gmail.com>
 <nad-EC5D57.13320314092013@news.gmane.org>
 <B3293155-E4D5-4389-A555-C31BC49CE539@gmail.com>
 <rowen-8E335B.12310916092013@news.gmane.org> <51403.1379361402@parc.com>
 <CAO41-mMUQD=K7WbSPcHSZk6jh_K3zv=Aa-YAGL2dRrwckCqcOg@mail.gmail.com>
 <CAGE7PNJYjxhu2YkLyiUcrm8PdhYUmMLOGTYzc1WXm3K2+tqgVQ@mail.gmail.com>
 <5239A288.2050403@v.loewis.de> <9829E582-9A68-4679-8087-31D42069F20D@uw.edu>
Message-ID: <71841.1379521869@parc.com>

Russell Owen <rowen at uw.edu> wrote:

> I agree that a policy is a good idea, and I suggest it be primarily based on age, since we cannot assume Apple will release new versions of the OS on a given timeline.
> 
> I personally think too early to drop support for MacOS X 10.6 and am on the edge about 10.5.

I run one 10.5 machine, three 10.6 machines, and 5+ 10.7+ machines right
now.  I have a perfectly fine and fast Mac Pro which can't even be
upgraded past 10.7.  So using Apple's OS release numbers as the gating
factor might be a problem.

I'd be happy to drop pre-built binary installers for 10.6, but that's
different from dropping all support for it.

Bill

> -- Russell
> 
> On Sep 18, 2013, at 5:54 AM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> 
> > Am 18.09.13 08:43, schrieb Gregory P. Smith:
> >> Just drop support for 10.6 with Python 3.4. Problem solved. People on
> >> that old of a version of the OS can build their own Python 3.4 or do the
> >> right thing and upgrade or just install Linux.
> >> 
> >> This isn't Windows. Compiler tool chains are freely available for the
> >> legacy platform. We don't need to maintain such a long legacy support
> >> tail there ourselves.
> > 
> > I don't mind such a decision in principle, but also in principle, I'd
> > prefer if there was a pre-set policy to decide this question, documented
> > in PEP 11.
> > 
> > Here a piece of OSX release history:
> > - 10.5:       October 2007
> >  * 10.5.8:   August 2009
> > - 10.6:       August 2009
> >  * 10.6.8:   July 2011
> > - 10.7:       July 2011
> >  * 10.7.5:   July 2012
> > - 10.8:       July 2012
> > 
> > So possible policy that would now exclude 10.6 for binary installers
> > would be:
> > 
> > - only the two latest feature releases are supported
> > - only feature releases younger than 3 years are supported
> > 
> > Note that a separate policy should be added to decide whether support
> > for older versions is actively removed from source code (or equally
> > bug fixes for old versions are not accepted anymore).
> > 
> > Regards,
> > Martin
> > 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/bill%40janssen.org

From stephen at xemacs.org  Wed Sep 18 18:35:03 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Thu, 19 Sep 2013 01:35:03 +0900
Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4
In-Reply-To: <CANc-5UwCtaFGJ40PVkKzE+PtroRxz=yQ0coQU5irVoALH5Okog@mail.gmail.com>
References: <5239A497.4070204@v.loewis.de>
 <8279a538-028f-45ae-a762-25302c52e370@me.com>
 <CANc-5UwCtaFGJ40PVkKzE+PtroRxz=yQ0coQU5irVoALH5Okog@mail.gmail.com>
Message-ID: <87zjra3x6w.fsf@uwakimon.sk.tsukuba.ac.jp>

Skip Montanaro writes:

 > I suspect other Mac users stuck on Snow Leopard who are not Python
 > developers would rue the lack of binary installers more than me.

Perhaps, but if Python current as of now isn't good enough for someone
for the foreseeable future, aren't they going to want up-to-date
libraries to backup the Python modules that use them, not to mention
security fixes and the like?  That's why I get my Python (for Snow
Leopard) from MacPorts.  I wouldn't hesitate to build it myself, but I
get timely upgrades from MacPorts so there's no need to bother.  I'd
be far less likely to upgrade quickly (for bugfix releases) if I had
to make a special trip to python.org for installers.

The MacPorts build could break at any time, I suppose, but at least
for the moment there are alternatives to building it yourself.




From skip at pobox.com  Wed Sep 18 18:44:12 2013
From: skip at pobox.com (Skip Montanaro)
Date: Wed, 18 Sep 2013 11:44:12 -0500
Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4
In-Reply-To: <87zjra3x6w.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <5239A497.4070204@v.loewis.de>
 <8279a538-028f-45ae-a762-25302c52e370@me.com>
 <CANc-5UwCtaFGJ40PVkKzE+PtroRxz=yQ0coQU5irVoALH5Okog@mail.gmail.com>
 <87zjra3x6w.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CANc-5UzNGSEdAHOHno7Tx_Z_zjD5WwbLuZ-icdonhzEiCwVNpw@mail.gmail.com>

> That's why I get my Python (for Snow Leopard) from MacPorts.

Unless things have changed, that probably doesn't support Mac-specific
stuff, does it?

I was thinking more of non-developer users who are likely to need/want
Mac-specific interfaces for tools which are written in Python. That
might just shift the burden to the developers of such tools. My guess
is that there is a domino effect. Apple stops supporting Snow Leopard,
which, all other things considered, forces third-party developers who
use Python in their products to stop supporting it, which then
pressures their users to buy new hardware so they can run newer
versions of those developers' tools.  This is good for Apple, but
creates headaches and monetary costs for developers and users
downstream.

S

From nad at acm.org  Wed Sep 18 19:09:55 2013
From: nad at acm.org (Ned Deily)
Date: Wed, 18 Sep 2013 10:09:55 -0700
Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4
References: <20130918153644.73eae75f@pitrou.net>
 <2f235f3a-af73-48b0-9d9f-26047f7175ad@me.com>
Message-ID: <nad-87D0F5.10095518092013@news.gmane.org>

Thank you all for your comments so far on this subject.  I have noted 
two separate issues raised here: one, how to build the Pythons provided 
by binary installers to get optimum performance (i.e. use more recent 
compilers); and, two, what OS X releases should we support with binary 
installers.  As I noted earlier, I've opened Issue19019 and I will 
update it with concrete proposals when we've complete the necessary 
testing and fixing of various build configurations, including of the 
sort Ronald and I outlined.  If you are interested in the details of 
this, please move that discussion to the bug tracker.

As to point two, I will put a stake in the ground here and declare that 
we will continue to support 10.6 with 3.4 batteries-included installers.  
For various reasons, 10.6 remains surprisingly popular (at a recent 
Python hackathon meetup in San Francisco, every person I helped who had 
a Mac was running 10.6) and it is not that old even by Apple standards.  
There are tradeoffs in how best to provide that support.  Among those 
tradeoffs are the impacts to those who provide binary packages for 
extension modules and third-party libraries, as Russell notes.  Again, 
after investigating and testing the nitty-gritty details, if it seems 
that a change in how we provide installers is warranted, we can discuss 
that on Issue19019 and report back here prior to any final decision.  
Also, at that time, it would be appropriate to consider a policy for 
long-term support of OS X releases; it's a bit premature to do so now.

-- 
 Ned Deily,
 nad at acm.org


From stephen at xemacs.org  Wed Sep 18 19:20:13 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Thu, 19 Sep 2013 02:20:13 +0900
Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4
In-Reply-To: <CANc-5UzNGSEdAHOHno7Tx_Z_zjD5WwbLuZ-icdonhzEiCwVNpw@mail.gmail.com>
References: <5239A497.4070204@v.loewis.de>
 <8279a538-028f-45ae-a762-25302c52e370@me.com>
 <CANc-5UwCtaFGJ40PVkKzE+PtroRxz=yQ0coQU5irVoALH5Okog@mail.gmail.com>
 <87zjra3x6w.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CANc-5UzNGSEdAHOHno7Tx_Z_zjD5WwbLuZ-icdonhzEiCwVNpw@mail.gmail.com>
Message-ID: <87wqme3v3m.fsf@uwakimon.sk.tsukuba.ac.jp>

Skip Montanaro writes:

 > > That's why I get my Python (for Snow Leopard) from MacPorts.
 > 
 > Unless things have changed, that probably doesn't support Mac-specific
 > stuff, does it?

You mean in the Python port or in general?  MacPorts supports
Mac-specific APIs in a number of ports where upstream does.

In Python, I don't know.  I would assume that anything python.org
supports is supported by MacPorts, though.




From nad at acm.org  Wed Sep 18 20:11:43 2013
From: nad at acm.org (Ned Deily)
Date: Wed, 18 Sep 2013 11:11:43 -0700
Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4
References: <5239A497.4070204@v.loewis.de>
 <8279a538-028f-45ae-a762-25302c52e370@me.com>
 <CANc-5UwCtaFGJ40PVkKzE+PtroRxz=yQ0coQU5irVoALH5Okog@mail.gmail.com>
 <87zjra3x6w.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CANc-5UzNGSEdAHOHno7Tx_Z_zjD5WwbLuZ-icdonhzEiCwVNpw@mail.gmail.com>
 <87wqme3v3m.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <nad-97155F.11114318092013@news.gmane.org>

In article <87wqme3v3m.fsf at uwakimon.sk.tsukuba.ac.jp>,
 "Stephen J. Turnbull" <stephen at xemacs.org> wrote:
> Skip Montanaro writes:
>  > > That's why I get my Python (for Snow Leopard) from MacPorts.
>  > Unless things have changed, that probably doesn't support Mac-specific
>  > stuff, does it?
> You mean in the Python port or in general?  MacPorts supports
> Mac-specific APIs in a number of ports where upstream does.
> 
> In Python, I don't know.  I would assume that anything python.org
> supports is supported by MacPorts, though.

Yes, there are no significant differences in feature support between a 
python.org Python and a MacPorts-supplied Python or a Homebrew-supplied 
Python or a Fink-supplied Python.  If you are doing any sort of a 
hard-core Python development work that requires the use of third-party C 
libraries et al, you are probably better off using a complete solution, 
including Python, from one of the above big-three package managers for 
OS X.  If you need to support multiple versions of OS X, IME, MacPorts 
does the best job of supporting older versions (I have up-to-date 
MacPorts installations on OS X 10.4 through 10.9).

But the primary mission for python.org installers is somewhat different: 
provide up-to-date, easy-to-install, batteries-included Pythons for 
users who are not necessarily experienced with Python itself.  People 
who need development environments with significant third-party libraries 
not included with Python or OS X itself are often better served by 
either investing in the time to set up and become familiar with using 
one of the general open source distributors (MacPorts, Homebrew, et al) 
or by using a specialized Python distribution (e.g. for scientific 
users).

-- 
 Ned Deily,
 nad at acm.org


From g.brandl at gmx.net  Thu Sep 19 03:01:10 2013
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 19 Sep 2013 03:01:10 +0200
Subject: [Python-Dev] License() release list is imcomplete; intentional?
In-Reply-To: <CAP7+vJJLZ93sEs3nR6NHy7m8A_14e54qOO-8iA9p1=N77t7BhA@mail.gmail.com>
References: <l19t08$uj0$1@ger.gmane.org> <l1bpmn$j54$1@ger.gmane.org>
 <CAP7+vJJLZ93sEs3nR6NHy7m8A_14e54qOO-8iA9p1=N77t7BhA@mail.gmail.com>
Message-ID: <l1dib8$oii$1@ger.gmane.org>

Now tracked at http://bugs.python.org/issue19043

Georg

On 09/18/2013 04:57 PM, Guido van Rossum wrote:
> OK. Let's do it.
> 
> 
> On Wed, Sep 18, 2013 at 1:54 AM, Georg Brandl <g.brandl at gmx.net
> <mailto:g.brandl at gmx.net>> wrote:
> 
>     On 09/17/2013 05:37 PM, Terry Reedy wrote:
>     > On 2.7, >>> license() return a text that includes a complete list of
>     > releases from 1.6 to 2.7 and stops there
>     >      Release         Derived     Year        Owner       GPL-
>     >                      from                                compatible? (1)
>     >
>     >      0.9.0 thru 1.2              1991-1995   CWI         yes
>     >      1.3 thru 1.5.2  1.2         1995-1999   CNRI        yes
>     >      1.6             1.5.2       2000        CNRI        no
>     >      2.0             1.6         2000        BeOpen.com  no
>     > ...
>     >      2.6.5           2.6.4       2010        PSF         yes
>     >      2.7             2.6         2010        PSF         yes
>     >
>     > Was it intentional to stop with 2.7 and not continue with 2.7.1, etc?
>     >
>     > On 3.3.2, the 2.x list ends with 2.6.5 and never mentions 2.7. Intentional?
>     > It then jumps back to 3.0 and ends with the 'previous' release, 3.3.1.
>     > Should 3.3.2 be included in the 3.3.2 list?
>     >
>     > ...
>     >      2.6.4           2.6.3       2009        PSF         yes
>     >      2.6.5           2.6.4       2010        PSF         yes
>     >      3.0             2.6         2008        PSF         yes
>     >      3.0.1           3.0         2009        PSF         yes
>     > ...
>     >      3.2.4           3.2.3       2013        PSF         yes
>     >      3.3.0           3.2         2012        PSF         yes
>     >      3.3.1           3.3.0       2013        PSF         yes
> 
>     Since there are 3 versions of this table, it's unavoidable that one of them
>     gets out of sync :)
> 
>     * LICENSE
>     * Doc/license.rst
>     * the versions for each release on the website
> 
>     So I wholeheartedly support truncating before 2.2 (which is the first
>     release where all micro versions are PSF owned and GPL compatible) and
>     just saying "2.2 onwards" (or whatever is the best way to say it).
> 
>     cheers,
>     Georg
> 
>     _______________________________________________
>     Python-Dev mailing list
>     Python-Dev at python.org <mailto:Python-Dev at python.org>
>     https://mail.python.org/mailman/listinfo/python-dev
>     Unsubscribe:
>     https://mail.python.org/mailman/options/python-dev/guido%40python.org
> 
> 
> 
> 
> -- 
> --Guido van Rossum (python.org/~guido <http://python.org/~guido>)
> 
> 



From barry at python.org  Thu Sep 19 03:29:50 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 18 Sep 2013 21:29:50 -0400
Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4
In-Reply-To: <CANc-5UwCtaFGJ40PVkKzE+PtroRxz=yQ0coQU5irVoALH5Okog@mail.gmail.com>
References: <5239A497.4070204@v.loewis.de>
 <8279a538-028f-45ae-a762-25302c52e370@me.com>
 <CANc-5UwCtaFGJ40PVkKzE+PtroRxz=yQ0coQU5irVoALH5Okog@mail.gmail.com>
Message-ID: <20130918212950.0abfd469@limelight.wooz.org>

On Sep 18, 2013, at 10:04 AM, Skip Montanaro wrote:

>As a MacBook Pro user running Snow Leopard/10.6.8, I would find the
>lack of a binary release problematic, were it not for the fact that I
>routinely build from source (and don't do anything Mac-specific with
>Python). For those not familiar with the platform, it's perhaps worth
>noting that upgrade is out of my hands. Apple dropped support for my
>"ancient" MBP (2006?), so 10.6.8 is as up-to-date as I can get. I
>guess that's what you expect from a company whose main revenues these
>days come from cell phones and tablets, and addition of pastels to
>their color pallet warrants announcement in a TV commercial.

I'm in a similar boat with my old MBP, and I am not upgrading my somewhat
newer one because of compatibility with other software.  But OTOH, I have
never installed the binary releases, I just build them from MacPorts, so as
long as that still works, I'm happy.  (I have a 8mo Air running the latest
Lion and will likely update that when Mavericks is out.)

-Barry

From barry at python.org  Thu Sep 19 03:38:48 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 18 Sep 2013 21:38:48 -0400
Subject: [Python-Dev] [Python-checkins] cpython (2.6): #14984: only
 import pwd on POSIX.
In-Reply-To: <3cg1WQ2NDkz7LnP@mail.python.org>
References: <3cg1WQ2NDkz7LnP@mail.python.org>
Message-ID: <20130918213848.6e943835@limelight.wooz.org>

On Sep 18, 2013, at 03:00 PM, r.david.murray wrote:

>http://hg.python.org/cpython/rev/fb3ad8a749c8
>changeset:   85749:fb3ad8a749c8
>branch:      2.6
>parent:      85735:1b673e0fd8f3
>user:        R David Murray <rdmurray at bitdance.com>
>date:        Wed Sep 18 08:49:25 2013 -0400
>summary:
>  #14984: only import pwd on POSIX.
>
>files:
>  Lib/netrc.py |  4 +++-
>  1 files changed, 3 insertions(+), 1 deletions(-)
>
>
>diff --git a/Lib/netrc.py b/Lib/netrc.py
>--- a/Lib/netrc.py
>+++ b/Lib/netrc.py
>@@ -2,7 +2,9 @@
> 
> # Module and documentation by Eric S. Raymond, 21 Dec 1998
> 
>-import os, stat, shlex, pwd
>+import os, stat, shlex
>+if os.name == 'posix':
>+    import pwd

Would it make more sense to do:

try:
    import pwd
except ImportError:
    pwd = None

?

From rdmurray at bitdance.com  Thu Sep 19 06:09:30 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Thu, 19 Sep 2013 00:09:30 -0400
Subject: [Python-Dev] [Python-checkins] cpython (2.6): #14984: only
	import pwd on POSIX.
In-Reply-To: <20130918213848.6e943835@limelight.wooz.org>
References: <3cg1WQ2NDkz7LnP@mail.python.org>
 <20130918213848.6e943835@limelight.wooz.org>
Message-ID: <20130919040931.B21C3250925@webabinitio.net>

On Wed, 18 Sep 2013 21:38:48 -0400, Barry Warsaw <barry at python.org> wrote:
> On Sep 18, 2013, at 03:00 PM, r.david.murray wrote:
> 
> >http://hg.python.org/cpython/rev/fb3ad8a749c8
> >changeset:   85749:fb3ad8a749c8
> >branch:      2.6
> >parent:      85735:1b673e0fd8f3
> >user:        R David Murray <rdmurray at bitdance.com>
> >date:        Wed Sep 18 08:49:25 2013 -0400
> >summary:
> >  #14984: only import pwd on POSIX.
> >
> >files:
> >  Lib/netrc.py |  4 +++-
> >  1 files changed, 3 insertions(+), 1 deletions(-)
> >
> >
> >diff --git a/Lib/netrc.py b/Lib/netrc.py
> >--- a/Lib/netrc.py
> >+++ b/Lib/netrc.py
> >@@ -2,7 +2,9 @@
> > 
> > # Module and documentation by Eric S. Raymond, 21 Dec 1998
> > 
> >-import os, stat, shlex, pwd
> >+import os, stat, shlex
> >+if os.name == 'posix':
> >+    import pwd
> 
> Would it make more sense to do:
> 
> try:
>     import pwd
> except ImportError:
>     pwd = None

I don't think so.  The code that uses pwd is protected by an 'if os.name
== 'posix' as well.  I think that's clearer than setting pwd to none
and guarding that section by 'if pwd'.  And in 3.4 I just moved the
import into the if block that uses it.

--David

From greg at krypto.org  Thu Sep 19 06:43:11 2013
From: greg at krypto.org (Gregory P. Smith)
Date: Wed, 18 Sep 2013 21:43:11 -0700
Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4
In-Reply-To: <8279a538-028f-45ae-a762-25302c52e370@me.com>
References: <5239A497.4070204@v.loewis.de>
 <8279a538-028f-45ae-a762-25302c52e370@me.com>
Message-ID: <CAGE7PN+QZmoAn=e5oxgAupzpwdmwaa7OktOJymS7s+HRy9vPDg@mail.gmail.com>

On Wed, Sep 18, 2013 at 6:38 AM, Ronald Oussoren <ronaldoussoren at mac.com>wrote:

>
>
> On Sep 18, 2013, at 03:03 PM, "Martin v. L?wis" <martin at v.loewis.de>
> wrote:
>
> Am 15.09.13 00:56, schrieb Ryan:
>
> +1. A 10.6-only build makes sense.
>
>
> I'd like to support Russell's point: this could put a burden on everyone
> releasing extension modules to also provide two binary releases, which
> e.g. would then mess up downloads from PyPI. So -1.
>
> +0 on dropping 10.6 support from the binary installers as proposed
> by Greg.
>
>
>
> Dropping support for 10.6 isn't really necessary, just build (carefully)
> on a later release of OSX (preferably the last one to pick up the most
> recent developer tools).
>
> I'll try to provide a more detailed response tonight, but the basic idea
> is to use OSX 10.8, the current Xcode release and the OSX 10.8 SDK to do
> the build. This is a supported configuration for building binaries that
> target older OSX releases, but some care is needed with the configure
> script for 3th-party libraries: those might pick up features that aren't
> available in 10.6.
>
>
Cool, that sounds like we should _really_ be doing. Best of both worlds and
it'll give a much faster to build to everyone. Good luck!

-gps
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130918/bce4c59b/attachment.html>

From ronaldoussoren at mac.com  Thu Sep 19 10:35:22 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Thu, 19 Sep 2013 10:35:22 +0200
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com>
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
 <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
 <522DECA3.2030709@hotpy.org>
 <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com>
 <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl>
 <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com>
 <20130913065236.GF16820@ando>
 <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com>
Message-ID: <2251401A-0CCB-42EE-8030-B21DAB416116@mac.com>


On 13 Sep, 2013, at 12:42, Nick Coghlan <ncoghlan at gmail.com> wrote:

> Perhaps "__getdescriptor__" would work as the method name? Yes, it can technically return a non-descriptor, but the *primary* purpose is to customise the retrieval of objects that will be checked to see if they're descriptors. It *won't* be invoked when looking for ordinary attributes in an instance dict, but *will* be invoked when looking on the class object.

__getdescriptor__ would work. The name is not 100% accurate, but a lot clearer than the other alternatives I've seen.

Ronald

P.S. Sorry about the slow response, its hard to find enough time to seriously participate in the discussion.


From ronaldoussoren at mac.com  Thu Sep 19 10:50:48 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Thu, 19 Sep 2013 10:50:48 +0200
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <20130913122328.GI16820@ando>
References: <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
 <522DECA3.2030709@hotpy.org>
 <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com>
 <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl>
 <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com>
 <20130913065236.GF16820@ando>
 <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com>
 <20130913122328.GI16820@ando>
Message-ID: <61609457-2C6A-4860-9AD5-B6C7618A7D13@mac.com>


On 13 Sep, 2013, at 14:23, Steven D'Aprano <steve at pearwood.info> wrote:

> On Fri, Sep 13, 2013 at 08:42:46PM +1000, Nick Coghlan wrote:
>> Perhaps "__getdescriptor__" would work as the method name? Yes, it can
>> technically return a non-descriptor,
> 
> So technically that name is, um, what's the term... oh yes, "a lie". 
> 
> :-)
> 
> 
>> but the *primary* purpose is to
>> customise the retrieval of objects that will be checked to see if they're
>> descriptors. 
> 
> If that's the case, the PEP should make that clear.
> 
> [Aside: the PEP states that the method shouldn't invoke descriptors. 
> What's the reason for that? If I take the statement literally, doesn't 
> it mean that the method mustn't use any other methods at all? Surely 
> that can't be what is intended, but I'm not sure what is intended.]

What it tries to say is that even if cls.__dict__[name] is a descriptor the
__locallookup__ method should not call its __get__ method. That was more relevant
when __locallookup__ had access to the object for which method resolution was performed.

I'm pretty sure this muddies the water even further, but don't have a clearer
description at the moment.

> 
> 
>> It *won't* be invoked when looking for ordinary attributes in
>> an instance dict, but *will* be invoked when looking on the class object.
> 
> Just to be clear, if I have:
> 
> instance = MyClass()
> x = instance.name
> 
> and "name" is found in instance.__dict__, then this special method will 
> not be invoked. But if "name" is not found in the instance dict, then 
> "name" will be looked up on the class object MyClass, which may invoke 
> this special method. Am I correct?

No.  This special method will always be called when "instance.__getattribute__" eventually
calls PyObject_GenericGetAttribute (otherwise all bets are off).

PyObject_GenericGetAttribute always looks for a candicate descriptor in the class (or
one of the other classes on the MRO). When a candidate descriptor is found, and this is 
a data descriptor the descriptor is used instead of the value from instance.__dict__ (otherwise
the value from instance.__dict__ is used).

Ronald


From ronaldoussoren at mac.com  Thu Sep 19 11:00:25 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Thu, 19 Sep 2013 11:00:25 +0200
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <f482c1d7f04f49d99616f3ca3c3b82da@BLUPR03MB199.namprd03.prod.outlook.com>
References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com>
 <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
 <522DECA3.2030709@hotpy.org>
 <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com>
 <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl>
 <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com>
 <20130913065236.GF16820@ando>
 <f482c1d7f04f49d99616f3ca3c3b82da@BLUPR03MB199.namprd03.prod.outlook.com>
Message-ID: <8AF283AC-3C8E-4D14-BCD8-4DD9002E56C6@mac.com>


On 13 Sep, 2013, at 18:19, Steve Dower <Steve.Dower at microsoft.com> wrote:

> From: Steven D'Aprano
>> On Fri, Sep 13, 2013 at 04:26:06AM +0000, Steve Dower wrote:
>> 
>>> Last I checked, looking up in the instance dict us exactly what it
>>> does. Even the example you posted is doing that.
>> 
>> The example from the PEP shows:
>> 
>> return cls.__dict__[name]
>> 
>> not "self.__dict__[name]". It is true that "the instance" in this case refers to
>> it being an instance of the metaclass, but that instance is, in fact, a
>> class/type. That's why we normally call it "cls" in a metaclass method rather
>> than "self".
> 
> Right, but where's the difference between these two?
> 
> class A:
>    def __tdb__(self, name):
>        if name == 'some_attribute_on_my_instance_of_A':
>            return its_value
>        try:
>            return self.__dict__[name]
>        except KeyError:
>            raise AttributeError(name)
> 
> class MetaB:
>    def __tdb__(cls, name):
>        if name == 'some_attribute_on_my_class_B':
>            return its_value
>        try:
>            return cls.__dict__[name]
>        except KeyError:
>            raise AttributeError(name)
> 
> (Yes, either of these could be written with __getattribute__, but that function cannot be called by super().)
> 
> As I see it, there are two (correct) ways to interpret what this method is for, which influences what it should be called.
> 
> 1. It directly replaces obj.__dict__[name] everywhere that is done, including internally in the interpreter.
> 2. It is the same as __getattribute__ without the final call to object.__getattribute__
> 
> I guess it's also correct to view it as a special helper for super(), but it is more generally applicable than that.

It directly replaces cls.__dict__[name] for object.__getattribute__ and super.__getattribute__.  

The primary reason for writing a proposal is that __getattribute__ can be used to override attribute lookup on an instance, but there is way to override how super() looks up an attribute.  Using the method for both super().__getattribute__ and object.__getattribute__ results in a cleaner model than just having a new hook for super().__getattribute__.


> 
> [...]
> 
>> By the way, I think the PEP should have a more complex example. The SillyObject
>> example is nice and easy to understand, but it doesn't really help with the
>> motivating use-case "dynamic classes that can grow new methods on demand".
>> Ronald, if you're reading this, can you add such an example please? Also,
>> there's a typo in the SillyObject M method ("fourtytwo" should not have a U in
>> it).
> 
> Agreed. No harm in more examples.

[...]

> 
> A full example of where this may realistically be needed is longer and certainly involves metaclasses, but fundamentally it's just the same as __getattribute__ with slightly different semantics.

PyObjC will be a truly realistic example, but that involves loads of fairly complex C code and likely won't help to explain anything beyond (hopefully) showing that this proposal can lead to significant code removal in some situations.


Ronald


From ronaldoussoren at mac.com  Thu Sep 19 11:32:17 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Thu, 19 Sep 2013 11:32:17 +0200
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <CADiSq7cdSEHryP+LMppRdabTuUG-GzMd56ub-oWpFNKPHNzsHA@mail.gmail.com>
References: <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
 <522DECA3.2030709@hotpy.org>
 <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com>
 <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl>
 <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com>
 <20130913065236.GF16820@ando>
 <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com>
 <20130913122328.GI16820@ando>
 <CADiSq7cdSEHryP+LMppRdabTuUG-GzMd56ub-oWpFNKPHNzsHA@mail.gmail.com>
Message-ID: <9BBAFB63-68C5-441B-8ED4-43604241AA42@mac.com>


On 14 Sep, 2013, at 8:30, Nick Coghlan <ncoghlan at gmail.com> wrote:

[... interesting text that I'll respond to later ...]

> 
> So my proposed name is based on the idea that what Ronald is after
> with the PEP is a hook that *only* gets invoked when the interpreter
> is doing this hunt for descriptors, but *not* for ordinary attribute
> lookups.

I'll describe the usecase that led to my proposal. I'm amongst other the primary
author (and sole maintainer) of PyObjC, which is a bridge between the Python
and Objective-C class/object models.  PyObjC defines a proxy class for every
(Objective-C) class in the Objective-C runtime and when an Objective-C object
is made available to Python code the PyObjC bridge creates a proxy object that 
is an instance of this proxy object.  This is simular what wxWidgets or PyQt do, 
but everything is done at runtime by introspecting the Objective-C runtime.

The first time a method is called the bridge looks for an Objective-C selector 
with the same name and adds that to the class dictionary. This works fine for normal
method lookups, by overriding __getattribute__, but causes problems with super:
super happily ignores __getattribute__ and peeks in the class __dict__ which may
not yet contain the name we're looking for and that can result in incorrect results
(both incorrect AttributeErrors and totally incorrect results when the name is
not yet present in the parent class' __dict__ but is in the grandparent's __dict__).

The current release of PyObjC "solves" this problem by providing a subclass of
super (objc.super) that should be used instead of the builtin one and that works,
although the implementation of this class is a gross hack.

Users's of PyObjC are currently oblivious of the problem because I sneak in objc.super
as a module's globals()['super'] when it imports PyObjC in the currently prefered 
way (which is using *-imports: "from Cocoa import *").  I'm currently migrating to
deprecating *-imports because those are bad for the usual reasons, but also because
framework binding modules are huge and using plain imports makes it possible to 
delay, and often even avoid, the cost of loading stuff from the framework binding
modules.  That switch will however make the problem of using __builtin__.super extremely
visible for PyObjC users.

That, and the implementation hack of objc.super, is why I started looking for a way
to cleanly provide a way to hook into the attribute resolution for super(), which ended
up as PEP 447.

Note that the description is slightly simplified from what PyObjC reall does, Objective-C
can (and does) have class and instance methods with the same name, because of that
all PyObjC classes have a meta class of the same name and that makes everything
even more complicated, but that should not be important for the previous description.

PyObjC also enables implementing Objective-C classes in Python, but that's also
not relevant for this discussion.

Ronald


From p.f.moore at gmail.com  Thu Sep 19 12:00:01 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 19 Sep 2013 11:00:01 +0100
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <9BBAFB63-68C5-441B-8ED4-43604241AA42@mac.com>
References: <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
 <522DECA3.2030709@hotpy.org>
 <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com>
 <522DF204.7040109@hotpy.org>
 <0752376899dd7436f6115bfc03594574@chopin.edu.pl>
 <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com>
 <20130913065236.GF16820@ando>
 <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com>
 <20130913122328.GI16820@ando>
 <CADiSq7cdSEHryP+LMppRdabTuUG-GzMd56ub-oWpFNKPHNzsHA@mail.gmail.com>
 <9BBAFB63-68C5-441B-8ED4-43604241AA42@mac.com>
Message-ID: <CACac1F_gT=mDEV3_VwvhjvS2H=JmNodWdP1vqMcEopWKMrPPHA@mail.gmail.com>

On 19 September 2013 10:32, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
> The first time a method is called the bridge looks for an Objective-C selector
> with the same name and adds that to the class dictionary. This works fine for normal
> method lookups, by overriding __getattribute__, but causes problems with super:
> super happily ignores __getattribute__ and peeks in the class __dict__ which may
> not yet contain the name we're looking for and that can result in incorrect results
> (both incorrect AttributeErrors and totally incorrect results when the name is
> not yet present in the parent class' __dict__ but is in the grandparent's __dict__).

As an alternative approach, could you use a custom dict subclass as
the class __dict__, and catch the peeking in the class __dict__ that
way? Or is this one of those places where only a real dict will do?

Paul

From ronaldoussoren at mac.com  Thu Sep 19 12:04:48 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Thu, 19 Sep 2013 12:04:48 +0200
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <CACac1F_gT=mDEV3_VwvhjvS2H=JmNodWdP1vqMcEopWKMrPPHA@mail.gmail.com>
References: <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
 <522DECA3.2030709@hotpy.org>
 <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com>
 <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl>
 <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com>
 <20130913065236.GF16820@ando>
 <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com>
 <20130913122328.GI16820@ando>
 <CADiSq7cdSEHryP+LMppRdabTuUG-GzMd56ub-oWpFNKPHNzsHA@mail.gmail.com>
 <9BBAFB63-68C5-441B-8ED4-43604241AA42@mac.com>
 <CACac1F_gT=mDEV3_VwvhjvS2H=JmNodWdP1vqMcEopWKMrPPHA@mail.gmail.com>
Message-ID: <C1DAADB8-E611-4359-B0F4-7914E9579368@mac.com>


On 19 Sep, 2013, at 12:00, Paul Moore <p.f.moore at gmail.com> wrote:

> On 19 September 2013 10:32, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
>> The first time a method is called the bridge looks for an Objective-C selector
>> with the same name and adds that to the class dictionary. This works fine for normal
>> method lookups, by overriding __getattribute__, but causes problems with super:
>> super happily ignores __getattribute__ and peeks in the class __dict__ which may
>> not yet contain the name we're looking for and that can result in incorrect results
>> (both incorrect AttributeErrors and totally incorrect results when the name is
>> not yet present in the parent class' __dict__ but is in the grandparent's __dict__).
> 
> As an alternative approach, could you use a custom dict subclass as
> the class __dict__, and catch the peeking in the class __dict__ that
> way? Or is this one of those places where only a real dict will do?

The C code uses PyDict_GetItem and AFAIK that doesn't look for a __getitem__ 
implementation in a subclass.

Ronald

> 
> Paul


From ncoghlan at gmail.com  Thu Sep 19 12:12:33 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 19 Sep 2013 20:12:33 +1000
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <CACac1F_gT=mDEV3_VwvhjvS2H=JmNodWdP1vqMcEopWKMrPPHA@mail.gmail.com>
References: <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
 <522DECA3.2030709@hotpy.org>
 <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com>
 <522DF204.7040109@hotpy.org>
 <0752376899dd7436f6115bfc03594574@chopin.edu.pl>
 <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com>
 <20130913065236.GF16820@ando>
 <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com>
 <20130913122328.GI16820@ando>
 <CADiSq7cdSEHryP+LMppRdabTuUG-GzMd56ub-oWpFNKPHNzsHA@mail.gmail.com>
 <9BBAFB63-68C5-441B-8ED4-43604241AA42@mac.com>
 <CACac1F_gT=mDEV3_VwvhjvS2H=JmNodWdP1vqMcEopWKMrPPHA@mail.gmail.com>
Message-ID: <CADiSq7eQ457=Vva-_3xbpi8o2EtS9ea8Q3XG=oHe0xBBAtzywQ@mail.gmail.com>

On 19 Sep 2013 20:00, "Paul Moore" <p.f.moore at gmail.com> wrote:
>
> On 19 September 2013 10:32, Ronald Oussoren <ronaldoussoren at mac.com>
wrote:
> > The first time a method is called the bridge looks for an Objective-C
selector
> > with the same name and adds that to the class dictionary. This works
fine for normal
> > method lookups, by overriding __getattribute__, but causes problems
with super:
> > super happily ignores __getattribute__ and peeks in the class __dict__
which may
> > not yet contain the name we're looking for and that can result in
incorrect results
> > (both incorrect AttributeErrors and totally incorrect results when the
name is
> > not yet present in the parent class' __dict__ but is in the
grandparent's __dict__).
>
> As an alternative approach, could you use a custom dict subclass as
> the class __dict__, and catch the peeking in the class __dict__ that
> way? Or is this one of those places where only a real dict will do?

Even Python 3 doesn't let you control the *runtime* type of the class dict,
only the type used during evaluation of the class body.

I've played with changing that - it makes for a rather special interpreter
experience :)

Cheers,
Nick.

>
> Paul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/05d27e0a/attachment.html>

From ronaldoussoren at mac.com  Thu Sep 19 12:32:38 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Thu, 19 Sep 2013 12:32:38 +0200
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <CADiSq7eQ457=Vva-_3xbpi8o2EtS9ea8Q3XG=oHe0xBBAtzywQ@mail.gmail.com>
References: <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
 <522DECA3.2030709@hotpy.org>
 <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com>
 <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl>
 <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com>
 <20130913065236.GF16820@ando>
 <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com>
 <20130913122328.GI16820@ando>
 <CADiSq7cdSEHryP+LMppRdabTuUG-GzMd56ub-oWpFNKPHNzsHA@mail.gmail.com>
 <9BBAFB63-68C5-441B-8ED4-43604241AA42@mac.com>
 <CACac1F_gT=mDEV3_VwvhjvS2H=JmNodWdP1vqMcEopWKMrPPHA@mail.gmail.com>
 <CADiSq7eQ457=Vva-_3xbpi8o2EtS9ea8Q3XG=oHe0xBBAtzywQ@mail.gmail.com>
Message-ID: <F77A9A16-183E-4927-A96A-948F7494CD1D@mac.com>


On 19 Sep, 2013, at 12:12, Nick Coghlan <ncoghlan at gmail.com> wrote:

> 
> On 19 Sep 2013 20:00, "Paul Moore" <p.f.moore at gmail.com> wrote:
> >
> > On 19 September 2013 10:32, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
> > > The first time a method is called the bridge looks for an Objective-C selector
> > > with the same name and adds that to the class dictionary. This works fine for normal
> > > method lookups, by overriding __getattribute__, but causes problems with super:
> > > super happily ignores __getattribute__ and peeks in the class __dict__ which may
> > > not yet contain the name we're looking for and that can result in incorrect results
> > > (both incorrect AttributeErrors and totally incorrect results when the name is
> > > not yet present in the parent class' __dict__ but is in the grandparent's __dict__).
> >
> > As an alternative approach, could you use a custom dict subclass as
> > the class __dict__, and catch the peeking in the class __dict__ that
> > way? Or is this one of those places where only a real dict will do?
> 
> Even Python 3 doesn't let you control the *runtime* type of the class dict, only the type used during evaluation of the class body.

Changing the class dict type from C is easy enough, but as you wrote below doing this gives you an interesting experience. Changing PyDict_* to call the subclass implementation of the corresponding slot would be easy enough, but that changes the behavior of a core CPython API and I wouldn't look forward to auditing the CPython code base to check if such a change is safe (let alone all other extensions). Some code will use the PyDict_* API instead of the abstract API because the former is faster, but at least some callers for PyDict_GetItem will do this to explicitly get the base class implementation.

Ronald


> 
> I've played with changing that - it makes for a rather special interpreter experience :)
> 
> Cheers,
> Nick.
> 
> >
> > Paul


From ronaldoussoren at mac.com  Thu Sep 19 11:49:10 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Thu, 19 Sep 2013 11:49:10 +0200
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <CADiSq7cdSEHryP+LMppRdabTuUG-GzMd56ub-oWpFNKPHNzsHA@mail.gmail.com>
References: <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
 <522DECA3.2030709@hotpy.org>
 <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com>
 <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl>
 <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com>
 <20130913065236.GF16820@ando>
 <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com>
 <20130913122328.GI16820@ando>
 <CADiSq7cdSEHryP+LMppRdabTuUG-GzMd56ub-oWpFNKPHNzsHA@mail.gmail.com>
Message-ID: <4CE957E3-ECE7-4DD1-AB62-2EA7776B9F85@mac.com>


On 14 Sep, 2013, at 8:30, Nick Coghlan <ncoghlan at gmail.com> wrote:

> 
> 
>>> but the *primary* purpose is to
>>> customise the retrieval of objects that will be checked to see if they're
>>> descriptors.
>> 
>> If that's the case, the PEP should make that clear.
> 
> Technically, that's what "Currently object.__getattribute__ and
> super.__getattribute__ peek in the __dict__ of classes on the MRO for
> a class when looking for an attribute." means.
> 
> However, I agree the current wording only conveys that to the handful
> of people that already know exactly when in the attribute lookup
> sequence that step occurs, which is a rather niche audience :)

I've been fooling around with this long enough that I forgot that not
everyone knows this :-).

I guess I'd better include a clearer and more complete description 
of the current attribute resolution protocol and how my proposal affects 
that.  A nice readable Python implementation of that protocol would be nice
to have regardless of the fate of this PEP.

Ronald


From donald at stufft.io  Thu Sep 19 15:27:24 2013
From: donald at stufft.io (Donald Stufft)
Date: Thu, 19 Sep 2013 09:27:24 -0400
Subject: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in
	Python installations
Message-ID: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io>

We've updated PEP453 based on some of the early feedback we've gotten from -dev and Martin.

Major changes:

* Removal of the option to fetch pip from PyPI in order not to modify the trust model of the Python installers
* Consequently rename the model from ``getpip`` to ``extractpip``


Also available online at: http://www.python.org/dev/peps/pep-0453/


Abstract
========

This PEP proposes that the `pip`_ package manager be made available by
default when installing CPython and when creating virtual environments
using the standard library's ``venv`` module via the ``pyvenv`` command line
utility).

To clearly demarcate development responsibilities, and to avoid
inadvertently downgrading ``pip`` when updating CPython, the proposed
mechanism to achieve this is to include an explicit `pip`_ bootstrapping
mechanism in the standard library that is invoked automatically by the
CPython installers provided on python.org.

The PEP also strongly recommends that CPython redistributors and other Python
implementations ensure that ``pip`` is available by default, or
at the very least, explicitly document the fact that it is not included.


Proposal
========

This PEP proposes the inclusion of an ``extractpip`` bootstrapping module in
Python 3.4, as well as in the next maintenance releases of Python 3.3 and
2.7.

This PEP does *not* propose making pip (or any dependencies) directly
available as part of the standard library. Instead, pip will be a
bundled application provided along with CPython for the convenience
of Python users, but subject to its own development life cycle and able
to be upgraded independently of the core interpreter and standard library.


Rationale
=========

Currently, on systems without a platform package manager and repository,
installing a third-party Python package into a freshly installed Python
requires first identifying an appropriate package manager and then
installing it.

Even on systems that *do* have a platform package manager, it is unlikely to
include every package that is available on the Python Package Index, and
even when a desired third-party package is available, the correct name in
the platform package manager may not be clear.

This means that, to work effectively with the Python Package Index
ecosystem, users must know which package manager to install, where to get
it, and how to install it. The effect of this is that third-party Python
projects are currently required to choose from a variety of undesirable
alternatives:

* Assume the user already has a suitable cross-platform package manager
  installed.
* Duplicate the instructions and tell their users how to install the
  package manager.
* Completely forgo the use of dependencies to ease installation concerns
  for their users.

All of these available options have significant drawbacks.

If a project simply assumes a user already has the tooling then beginning
users may get a confusing error message when the installation command
doesn't work. Some operating systems may ease this pain by providing a
global hook that looks for commands that don't exist and suggest an OS
package they can install to make the command work, but that only works
on systems with platform package managers (such as major Linux
distributions). No such assistance is available for Windows and
Mac OS X users. The challenges of dealing with this problem are a
regular feature of feedback the core Python developers receive from
professional educators and others introducing new users to Python.

If a project chooses to duplicate the installation instructions and tell
their users how to install the package manager before telling them how to
install their own project then whenever these instructions need updates
they need updating by every project that has duplicated them. This is
particular problematic when there are multiple competing installation
tools available, and different projects recommend different tools.

This specific problem can be partially alleviated by strongly promoting
``pip`` as the default installer and recommending that other projects
reference `pip's own bootstrapping instructions
<http://www.pip-installer.org/en/latest/installing.html>`__ rather than
duplicating them. However the user experience created by this approach
still isn't good (especially on Windows, where downloading and running
the ``get-pip.py`` bootstrap script with the default OS configuration is
significantly more painful than downloading and running a binary executable
or installer). The situation becomes even more complicated when multiple
Python versions are involved (for example, parallel installations of
Python 2 and Python 3), since that makes it harder to create and maintain
good platform specific ``pip`` installers independently of the CPython
installers.

The projects that have decided to forgo dependencies altogether are forced
to either duplicate the efforts of other projects by inventing their own
solutions to problems or are required to simply include the other projects
in their own source trees. Both of these options present their own problems
either in duplicating maintenance work across the ecosystem or potentially
leaving users vulnerable to security issues because the included code or
duplicated efforts are not automatically updated when upstream releases a new
version.

By providing a cross-platform package manager by default it will be easier
for users trying to install these third-party packages as well as easier
for the people distributing them as they should now be able to safely assume
that most users will have the appropriate installation tools available.
This is expected to become more important in the future as the Wheel_
package format (deliberately) does not have a built in "installer" in the
form of ``setup.py`` so users wishing to install from a wheel file will want
an installer even in the simplest cases.

Reducing the burden of actually installing a third-party package should
also decrease the pressure to add every useful module to the standard
library. This will allow additions to the standard library to focus more
on why Python should have a particular tool out of the box, and why it
is reasonable for that package to adopt the standard library's 18-24 month
feature release cycle, instead of using the general difficulty of installing
third-party packages as justification for inclusion.

Providing a standard installation system also helps with bootstrapping
alternate build and installer systems, such as ``setuptools``,
``zc.buildout`` and the ``hashdist``/``conda`` combination that is aimed
specifically at the scientific community. So long as
``pip install <tool>`` works, then a standard Python-specific installer
provides a reasonably secure, cross platform mechanism to get access to
these utilities.


Why pip?
--------

``pip`` has been chosen as the preferred default installer, as it
addresses several design and user experience issues with its predecessor
``easy_install`` (these issues can't readily be fixed in ``easy_install``
itself due to backwards compatibility concerns). ``pip`` is also well suited
to working within the bounds of a single Python runtime installation
(including associated virtual environments), which is a desirable feature
for a tool bundled with CPython.

Other tools like ``zc.buildout`` and ``conda`` are more ambitious in their
aims (and hence substantially better than ``pip`` at handling external
binary dependencies), so it makes sense for the Python ecosystem to treat
them more like platform package managers to inter operate with rather than
as the default cross-platform installation tool. This relationship is
similar to that between ``pip`` and platform package management systems
like ``apt`` and ``yum`` (which are also designed to handle arbitrary
binary dependencies).


Explicit bootstrapping mechanism
================================

An additional module called ``extractpip`` will be added to the standard
library whose purpose is to install pip and any of its dependencies into the
appropriate location (most commonly site-packages). It will expose a
callable named ``bootstrap()`` as well as offer direct execution via
``python -m extractpip``.

The bootstrap will *not* contact PyPI, but instead rely on a private copy
of pip stored inside the standard library. Accordingly, only options
related to the installation location will be supported (``--user``,
``--root``, etc).

It is considered desirable that users be strongly encouraged to use the
latest available version of ``pip``, in order to take advantage of the
ongoing efforts to improve the security of the PyPI based ecosystem, as
well as benefiting from the efforts to improve the speed, reliability and
flexibility of that ecosystem.

In order to satisfy this goal of providing the most recent version of
``pip`` by default, the private copy of ``pip`` will be updated in CPython
maintenance releases, which should align well with the 6-month cycle used
for new ``pip`` releases.


Security considerations
-----------------------

The design in this PEP has been deliberately chosen to avoid making any
significant changes to the trust model of the CPython installers for end
users that do not subsequently make use of ``pip``.

The installers will contain all the components of a fully functioning
version of Python, including the ``pip`` installer. The installation
process will *not* require network access, and will *not* rely on
trusting the security of the network connection established between
``pip`` and the Python package index.

Only users that choose to use ``pip`` directly will need to pay
attention to any PyPI related security considerations.


Implementation strategy
-----------------------

To ensure there is no need for network access when installing Python or
creating virtual environments, the ``extractpip`` module will, as an
implementation detail, include a complete private copy of pip and its
dependencies which will be used to extract pip and install it into the target
environment. It is important to stress that this private copy of pip is
*only* an implementation detail and it should *not* be relied on or
assumed to exist beyond the public capabilities exposed through the
``extractpip`` module (and indirectly through ``venv``).

There is not yet a reference ``extractpip`` implementation. The existing
``get-pip.py`` bootstrap script demonstrates an earlier variation of the
general concept, but the standard library version would take advantage of
the improved distribution capabilities offered by the CPython installers
to include private copies of ``pip`` and ``setuptools`` as wheel files
(rather than as embedded base64 encoded data), and would not try to
contact PyPI (instead installing directly from the private wheel files.

Rather than including separate code to handle the bootstrapping, the
``extractpip`` module will manipulate sys.path appropriately to allow
the wheel files to be used to install themselves, either into the current
Python installation or into a virtual environment (as determined by the
options passed to the bootstrap command).


Proposed CLI
------------

The proposed CLI is based on a subset of the existing ``pip install``
options::

    Usage:
      python -m extractpip [options]

    General Options:
      -h, --help          Show help.
      -v, --verbose       Give more output. Option is additive, and can be used up to 3 times.
      -V, --version       Show the pip version that would be extracted and exit.
      -q, --quiet         Give less output.

    Installation Options:
      -U, --upgrade       Upgrade pip and dependencies, even if already installed
      --user              Install using the user scheme.
      --root <dir>        Install everything relative to this alternate root directory.

In most cases, end users won't need to use this CLI directly, as ``pip``
should have been installed automatically when installing Python or when
creating a virtual environment.

Users that want to retrieve the latest version from PyPI, or otherwise
needing more flexibility, should invoke the extracted ``pip``
appropriately.


Proposed module API
-------------------

The proposed ``extractpip`` module API consists of the following two
functions::

    def version():
        """
        Returns a string specifying the bundled version of pip.
        """

    def bootstrap(root=None, upgrade=False, user=False, verbosity=0):
        """
        Bootstrap pip into the current Python installation (or the given root
        directory).
        """


Invocation from the CPython installers
--------------------------------------

The CPython Windows and Mac OS X installers will each gain a new option:

* Install pip (the default Python package management utility)?

This option will be checked by default.

If the option is checked, then the installer will invoke the following
command with the just installed Python::

    python -m extractpip --upgrade

This ensures that, by default, installing or updating CPython will ensure
that the installed version of pip is at least as recent as the one included
with that version of CPython.


Installing from source
----------------------

While the prebuilt binary installers will be updated to run
``python -m extractpip`` by default, no such change will be made to the
``make install`` and ``make altinstall`` commands of the source
distribution.

``extractpip`` itself (including the private copy of ``pip`` and its
dependencies) will still be installed normally (as it is a regular
part of the standard library), only the implicit installation of pip and
its dependencies will be skipped.

Keeping the pip bootstrapping as a separate step for ``make``-based
installations should minimize the changes CPython redistributors need to
make to their build processes. Avoiding the layer of indirection through
``make`` for the ``extractpip`` invocation avoids any challenges
associated with determining where to install the extracted ``pip``.


Changes to virtual environments
-------------------------------

Python 3.3 included a standard library approach to virtual Python environments
through the ``venv`` module. Since its release it has become clear that very
few users have been willing to use this feature directly, in part due to the
lack of an installer present by default inside of the virtual environment.
They have instead opted to continue using the ``virtualenv`` package which
*does* include pip installed by default.

To make the ``venv`` more useful to users it will be modified to issue the
pip bootstrap by default inside of the new environment while creating it. This
will allow people the same convenience inside of the virtual environment as
this PEP provides outside of it as well as bringing the ``venv`` module closer
to feature parity with the external ``virtualenv`` package, making it a more
suitable replacement.

To handle cases where a user does not wish to have pip bootstrapped into
their virtual environment a ``--without-pip`` option will be
added.

The ``venv.EnvBuilder`` and ``venv.create`` APIs will be updated to accept
one new parameter: ``with_pip`` (defaulting to ``False``).

The new default for the module API is chosen for backwards compatibility
with the current behaviour (as it is assumed that most invocation of the
``venv`` module happens through third part tools that likely will not
want ``pip`` installed without explicitly requesting it), while the
default for the command line interface is chosen to try to ensure ``pip``
is available in most virtual environments without additional action on the
part of the end user.

This particular change will be made only for Python 3.4 and later versions.
The third-party ``virtualenv`` project will still be needed to obtain a
consistent cross-version experience in Python 3.3 and 2.7.


Documentation
-------------

The "Installing Python Modules" section of the standard library
documentation will be updated to recommend the use of the bootstrapped
`pip` installer. It will give a brief description of the most common
commands and options, but delegate to the externally maintained ``pip``
documentation for the full details.

The existing content of the module installation guide will be retained,
but under a new "Invoking distutils directly" subsection.


Bundling CA certificates with CPython
-------------------------------------

The ``extractpip`` implementation will include the ``pip`` CA bundle along
with the rest of ``pip``. This means CPython effectively includes
a CA bundle that is used solely by ``pip`` after it has been extracted.

This is considered preferable to relying solely on the system
certificate stores, as it ensures that ``pip`` will behave the same
across all supported versions of Python, even those prior to Python 3.4
that cannot access the system certificate store on Windows.


Automatic installation of setuptools
------------------------------------

``pip`` currently depends on ``setuptools`` to handle metadata generation
during the build process, along with some other features. While work is
ongoing to reduce or eliminate this dependency, it is not clear if that
work will be complete for pip 1.5 (which is the version likely to be current
when Python 3.4.0 is released).

This PEP proposes that, if pip still requires it as a dependency,
``extractpip`` will include a private copy of ``setuptools`` (in addition
to the private copy of ``extractpip``). ``python -m extractpip`` will then
install the private copy in addition to installing ``pip`` itself.

However, this behavior is officially considered an implementation
detail. Other projects which explicitly require ``setuptools`` must still
provide an appropriate dependency declaration, rather than assuming
``setuptools`` will always be installed alongside ``pip``.

Once pip is able to run ``pip install --upgrade pip`` without needing
``setuptools`` installed first, then the private copy of ``setuptools``
will be removed from ``extractpip`` in subsequent CPython releases.


Updating the private copy of pip
--------------------------------

In order to keep up with evolutions in packaging as well as providing users
with as recent version a possible the ``extractpip`` module will be
regularly updated to the latest versions of everything it bootstraps.

After each new ``pip`` release, and again during the preparation for any
release of Python (including feature releases), a script, provided as part
of this PEP, will be run to ensure the private copies stored in the CPython
source repository have been updated to the latest versions.


Updating the extractpip module API and CLI
------------------------------------------

Like ``venv`` and ``pyvenv``, the ``extractpip`` module API and CLI
will be governed by the normal rules for the standard library: no
new features are permitted in maintenance releases.

However, the embedded components may be updated as noted above, so
the extracted ``pip`` may offer additional functionality in maintenance
releases.


Feature addition in maintenance releases
========================================

Adding a new module to the standard library in Python 2.7 and 3.3
maintenance releases breaks the usual policy of "no new features in
maintenance releases".

It is being proposed in this case as the current bootstrapping issues for
the third-party Python package ecosystem greatly affects the experience of
new users, especially on Python 2 where many Python 3 standard library
improvements are available as backports on PyPI, but are not included in
the Python 2 standard library.

By updating Python 2.7, 3.3 and 3.4 to easily bootstrap the PyPI ecosystem,
this change should aid the vast majority of current Python users, rather
than only those with the freedom to adopt Python 3.4 as soon as it is
released.


Open Question: Uninstallation
=============================

No changes are currently proposed to the uninstallation process. The
bootstrapped pip will be installed the same way as any other pip
installed packages, and will be handled in the same way as any other
post-install additions to the Python environment.

At least on Windows, that means the bootstrapped files will be
left behind after uninstallation, since those files won't be associated
with the Python MSI installer.

.. note::

   Perhaps the installer should be updated to clobber everything in
   site-packages and the Scripts directory when uninstalled (treating them
   as "data directories" from Python's point of view), but I would prefer
   not to make this PEP conditional on that change.


Open Question: Script Execution on Windows
==========================================

.. note::

   Perhaps this question should be separated out from the PEP?

While the Windows installer was updated in Python 3.3 to optionally
make ``python`` available on the PATH, no such change was made to
include the Scripts directory. Independently of this PEP, a proposal has
also been made to rename the ``Tools\Scripts`` subdirectory to ``bin`` in
order to improve consistency with the typical script installation directory
names on \*nix systems.

Accordingly, in addition to adding the option to extract and install ``pip``
during installation, this PEP proposes that the Windows installer (and
``sysconfig``) in Python 3.4 and later be updated to:

- install scripts to PythonXY\bin rather than PythonXY\Tools\Scripts
- add PythonXY\bin to the Windows PATH (in addition to PythonXY) when the
  PATH modification option is enabled during installation

For Python 2.7 and 3.3, it is proposed that the only change be the one
to bootstrap ``pip`` by default.

This means that, for Python 3.3, the most reliable way to invoke pip on
Windows (without tinkering manually with PATH) will actually be
``py -m pip`` (or ``py -3 -m pip`` to select the Python 3 version if both
Python 2 and 3 are installed) rather than simply calling ``pip``.

For Python 2.7 and 3.2, the most reliable mechanism will be to install the
standalone Python launcher for Windows and then use ``py -m pip`` as noted
above.

Adding the scripts directory to the system PATH would mean that ``pip``
works reliably in the "only one Python installation on the system PATH"
case, with ``py -m pip``, ``pipX``, or ``pipX.Y`` needed only to select a
non-default version in the parallel installation case (and outside a virtual
environment). This change should also make the ``pyvenv`` command substantially
easier to invoke on Windows, along with all scripts installed by ``pip``,
``easy_install`` and similar tools.

While the script invocations on recent versions of Python will run through
the Python launcher for Windows, this shouldn't cause any issues, as long
as the Python files in the Scripts directory correctly specify a Python version
in their shebang line or have an adjacent Windows executable (as
``easy_install`` and ``pip`` do).


Recommendations for Downstream Distributors
===========================================

A common source of Python installations are through downstream distributors
such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX
package managers [#homebrew]_, or Python-specific tools [#conda]_. In order
to provide a consistent, user-friendly experience to all users of Python
regardless of how they attained Python this PEP recommends and asks that
downstream distributors:

* Ensure that whenever Python is installed pip is also installed.

  * This may take the form of separate packages with dependencies on each
    other so that installing the Python package installs the pip package
    and installing the pip package installs the Python package.

* Do not remove the bundled copy of pip.

  * This is required for installation of pip into a virtual environment by the
    ``venv`` module.
  * This is similar to the existing ``virtualenv`` package for which many
    downstream distributors have already made exception to the common
    "debundling" policy.
  * This does mean that if ``pip`` needs to be updated due to a security
    issue, so does the bundled version in the ``extractpip`` bootstrap module
  * However, altering the bundled version of pip to remove the embedded
    CA certificate bundle and rely on the system CA bundle instead is a
    reasonable change.

* Migrate build systems to utilize `pip`_ and `Wheel`_ instead of directly
  using ``setup.py``.

  * This will ensure that downstream packages can more easily utilize the
    new metadata formats which may not have a ``setup.py``.

* Ensure that all features of this PEP continue to work with any modifications
  made to the redistributed version of Python.

  * Checking the version of pip that will be bootstrapped using
    ``python -m extractpip --version`` or ``extractpip.version()``.
  * Installation of the bundled version of pip into a global or virtual
    python environment using ``python -m extractpip`` or
    ``extractpip.bootstrap()``.
  * ``pip install --upgrade pip`` in a global installation should not affect
    any already created virtual environments (but is permitted to affect
    future virtual environments, even though it will not do so in the
    normal case).
  * ``pip install --upgrade pip`` in a virtual environment should not affect
    the global installation.

In the event that a Python redistributor chooses *not* to follow these
recommendations, we request that they explicitly document this fact and
provide their users with suitable guidance on translating upstream ``pip``
based installation instructions into something appropriate for the platform.

Other Python implementations are also encouraged to follow these guidelines
where applicable.


Policies & Governance
=====================

The maintainers of the bootstrapped software and the CPython core team will
work together in order to address the needs of both. The bootstrapped
software will still remain external to CPython and this PEP does not
include CPython subsuming the development responsibilities or design
decisions of the bootstrapped software. This PEP aims to decrease the
burden on end users wanting to use third-party packages and the
decisions inside it are pragmatic ones that represent the trust that the
Python community has already placed in the Python Packaging Authority as
the authors and maintainers of ``pip``, ``setuptools``, PyPI, ``virtualenv``
and other related projects.


Backwards Compatibility
-----------------------

The public API and CLI of the ``extractpip`` module itself will fall under
the typical backwards compatibility policy of Python for its standard
library. The externally developed software that this PEP bundles does not.

Most importantly, this means that the bootstrapped version of pip may gain
new features in CPython maintenance releases, and pip continues to operate on
its own 6 month release cycle rather than CPython's 18-24 month cycle.


Security Releases
-----------------

Any security update that affects the ``extractpip`` module will be shared
prior to release with the Python Security Response Team
(security at python.org). The PSRT will then decide if the reported issue
warrants a security release of CPython with an updated private copy of
``pip``.


Appendix: Rejected Proposals
============================


Automatically contacting PyPI when bootstrapping pip
----------------------------------------------------

Earlier versions of this PEP called the bootstrapping module ``getpip`` and
defaulted to downloading and installing ``pip`` from PyPI, with the private
copy used only as a fallback option or when explicitly requested.

This resulted in several complex edge cases, along with difficulties in
defining a clean API and CLI for the bootstrap module. It also significantly
altered the default trust model for the binary installers published on
python.org, as end users would need to explicitly *opt-out* of trusting
the security of the PyPI ecosystem (rather than opting in to it by
explicitly invoking ``pip`` following installation).

As a result, the PEP was simplified to the current design, where the
bootstrapping *always* uses the private copy of ``pip``. Contacting PyPI
is now always an explicit separate step, with direct access to the full
pip interface.


Implicit bootstrap
------------------

`PEP439`_, the predecessor for this PEP, proposes its own solution. Its
solution involves shipping a fake ``pip`` command that when executed would
implicitly bootstrap and install pip if it does not already exist. This has
been rejected because it is too "magical". It hides from the end user when
exactly the pip command will be installed or that it is being installed at
all. It also does not provide any recommendations or considerations towards
downstream packagers who wish to manage the globally installed pip through
the mechanisms typical for their system.

The implicit bootstrap mechanism also ran into possible permissions issues,
if a user inadvertently attempted to bootstrap pip without write access to
the appropriate installation directories.


Including pip directly in the standard library
----------------------------------------------

Similar to this PEP is the proposal of just including pip in the standard
library. This would ensure that Python always includes pip and fixes all of the
end user facing problems with not having pip present by default. This has been
rejected because we've learned, through the inclusion and history of
``distutils`` in the standard library, that losing the ability to update the
packaging tools independently can leave the tooling in a state of constant
limbo. Making it unable to ever reasonably evolve in a time frame that actually
affects users as any new features will not be available to the general
population for *years*.

Allowing the packaging tools to progress separately from the Python release
and adoption schedules allows the improvements to be used by *all* members
of the Python community and not just those able to live on the bleeding edge
of Python releases.

There have also been issues in the past with the "dual maintenance" problem
if a project continues to be maintained externally while *also* having a
fork maintained in the standard library. Since external maintenance of
``pip`` will always be needed to support earlier Python versions, the
proposed bootstrapping mechanism will becoming the explicit responsibility
of the CPython core developers (assisted by the pip developers), while
pip issues reported to the CPython tracker will be migrated to the pip
issue tracker. There will no doubt still be some user confusion over which
tracker to use, but hopefully less than has been seen historically when
including complete public copies of third-party projects in the standard
library.

Finally, the approach described in this PEP avoids some technical issues
related to handle CPython maintenance updates when pip has been independently
updated to a more recent version. The proposed pip-based bootstrapping
mechanism handles that automatically, since pip and the system installer
never get into a fight about who owns the pip installation (it is always
managed through pip, either directly, or indirectly via the getpip bootstrap
module).


Defaulting to --user installation
---------------------------------

Some consideration was given to bootstrapping pip into the per-user
site-packages directory by default. However, this behavior would be
surprising (as it differs from the default behavior of pip itself)
and is also not currently considered reliable (there are some edge cases
which are not handled correctly when pip is installed into the user
site-packages directory rather than the system site-packages).


.. _Wheel: http://www.python.org/dev/peps/pep-0427/
.. _pip: http://www.pip-installer.org
.. _setuptools: https://pypi.python.org/pypi/setuptools
.. _PEP439: http://www.python.org/dev/peps/pep-0439/


References
==========

.. [#ubuntu] `Ubuntu <http://www.ubuntu.com/>`
.. [#debian] `Debian <http://www.debian.org>`
.. [#fedora] `Fedora <https://fedoraproject.org/>`
.. [#homebrew] `Homebrew  <http://brew.sh/>`
.. [#conda] `Conda <http://www.continuum.io/blog/conda>`


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/c6314883/attachment-0001.sig>

From donald at stufft.io  Thu Sep 19 15:30:21 2013
From: donald at stufft.io (Donald Stufft)
Date: Thu, 19 Sep 2013 09:30:21 -0400
Subject: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in
	Python installations
In-Reply-To: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io>
References: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io>
Message-ID: <4D249E79-D01A-407E-99B8-38E0311A71DF@stufft.io>


On Sep 19, 2013, at 9:27 AM, Donald Stufft <donald at stufft.io> wrote:

> We've updated PEP453 based on some of the early feedback we've gotten from -dev and Martin.
> 
> Major changes:
> 
> * Removal of the option to fetch pip from PyPI in order not to modify the trust model of the Python installers
> * Consequently rename the model from ``getpip`` to ``extractpip``
> 
> 
> Also available online at: http://www.python.org/dev/peps/pep-0453/
> 
> [snip]

One thing I don't like is the name ``extractpip``, maybe ``installpip`` would be better?

Naming things is hard.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/fbdfda0d/attachment.sig>

From ncoghlan at gmail.com  Thu Sep 19 15:40:25 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 19 Sep 2013 23:40:25 +1000
Subject: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in
 Python installations
In-Reply-To: <4D249E79-D01A-407E-99B8-38E0311A71DF@stufft.io>
References: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io>
 <4D249E79-D01A-407E-99B8-38E0311A71DF@stufft.io>
Message-ID: <CADiSq7fvEif0jdVgQwv+g3tN0+DGfNXshGjHppa7eOctffssiQ@mail.gmail.com>

On 19 Sep 2013 23:30, "Donald Stufft" <donald at stufft.io> wrote:
>
>
> On Sep 19, 2013, at 9:27 AM, Donald Stufft <donald at stufft.io> wrote:
>
> > We've updated PEP453 based on some of the early feedback we've gotten
from -dev and Martin.
> >
> > Major changes:
> >
> > * Removal of the option to fetch pip from PyPI in order not to modify
the trust model of the Python installers
> > * Consequently rename the model from ``getpip`` to ``extractpip``
> >
> >
> > Also available online at: http://www.python.org/dev/peps/pep-0453/
> >
> > [snip]
>
> One thing I don't like is the name ``extractpip``, maybe ``installpip``
would be better?

I figured that was too close to "pip install", not to mention the actual
phrase "install pip". There are also multiple ways to install pip - the new
bootstrap will just be the simplest.

"extractpip" is a bit clunky, but it does exactly what it says on the tin
and should be a hidden implementation detail the vast majority of the time.

I also believe that choice of name may help avoid the problem of users
expecting a global pip update to implicitly affect future virtual
environments.

> Naming things is hard.

Yup :)

Cheers,
Nick.

>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
DCFA
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/5887c9c5/attachment.html>

From p.f.moore at gmail.com  Thu Sep 19 15:43:09 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 19 Sep 2013 14:43:09 +0100
Subject: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in
 Python installations
In-Reply-To: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io>
References: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io>
Message-ID: <CACac1F8J7H8YZ5HjQDVydqu5PJJ7AF2d+qMKoqMg=Tqumzrf5w@mail.gmail.com>

On 19 September 2013 14:27, Donald Stufft <donald at stufft.io> wrote:
> Major changes:
>
> * Removal of the option to fetch pip from PyPI in order not to modify the trust model of the Python installers
> * Consequently rename the model from ``getpip`` to ``extractpip``

If extractpip (I agree, I don't like the name, installpip is better)
only ever unpacks the bundled pip, and it's always run, why bother?
Why not just bundle pip directly into site-packages? The extra step
seems to add little or no value.

Paul

From donald at stufft.io  Thu Sep 19 15:43:46 2013
From: donald at stufft.io (Donald Stufft)
Date: Thu, 19 Sep 2013 09:43:46 -0400
Subject: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in
	Python installations
In-Reply-To: <20130919133601.GA6420@helios.localdomain>
References: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io>
 <20130919133601.GA6420@helios.localdomain>
Message-ID: <76C8A128-90AC-453F-8E46-81A0D29D8D56@stufft.io>


On Sep 19, 2013, at 9:36 AM, Paul Tagliamonte <paultag at debian.org> wrote:

> On Thu, Sep 19, 2013 at 09:27:24AM -0400, Donald Stufft wrote:
>> Rationale
>> =========
>> 
>> Currently, on systems without a platform package manager and repository,
>> installing a third-party Python package into a freshly installed Python
>> requires first identifying an appropriate package manager and then
>> installing it.
> 
> Howdy Donald,
> 
> Thanks for helping make Python better.
> 
> However, speaking as a Debian-folk (but not for the Debian-folk, I'll
> let barry take care of that), many users of Python software don't even
> know what pip is (or even that Python exists) -- as such, I find it very
> unlikely that a development tool would be shiped as part of the Python
> distribution in Debian. I don't see this changing, even with this pep.
> python-pip is still installable, but I don't see it pulled in by
> default.

That is obviously Debian's right. Hopefully if Debian does *not* pull it in by
default they'll be the odd man out and documentation can be updated to say
that on Debian based systems additional steps to get the "standard" Python
install must be taken. I believe Fedora has +1'd this so if this gets accepted
there will likely be that additional difference.

> 
> 
>> Even on systems that *do* have a platform package manager, it is unlikely to
>> include every package that is available on the Python Package Index, and
> 
> Yes. This is true. However, it's not Debian's job to include all of
> pypi. Pypi allows anyone to upload, and we have quite a bit of attention
> to concerns such as providing source for everything (less of a concern
> for Python, but pickles can still contain data not in the prefered form
> of modification), and proper free software licensing.
> 
> We also have a concern about stability; so we manage all the package set
> together to allow non-technical end-users to install stuff and not worry
> about breaking their system. It's not always the case where you can
> upgrade a library without breaking API.

This statement isn't meant to imply that it's Debian's job to include all of PyPI,
it merely calls out the fact that people often do wish to install things that
are not available in an OS repository while they are available on PyPI.

> 
> 
> I'm trimming the rest, since I don't want to get dragged into
> side-conversations about pip as a package manager.
> 
> 
> Cheers,
>  Paul
> 
> 
> -- 
> .''`.  Paul Tagliamonte <paultag at debian.org>
> : :'  : Proud Debian Developer
> `. `'`  4096R / 8F04 9AD8 2C92 066C 7352  D28A 7B58 5B30 807C 2A87
> `-     http://people.debian.org/~paultag


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/15eae484/attachment.sig>

From ncoghlan at gmail.com  Thu Sep 19 15:48:49 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 19 Sep 2013 23:48:49 +1000
Subject: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in
 Python installations
In-Reply-To: <20130919133601.GA6420@helios.localdomain>
References: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io>
 <20130919133601.GA6420@helios.localdomain>
Message-ID: <CADiSq7eG3ExCmpWUswKoR9Ya9=+if5ToUoQKd7o3yVWpLOjzaQ@mail.gmail.com>

On 19 Sep 2013 23:37, "Paul Tagliamonte" <paultag at debian.org> wrote:
>
> On Thu, Sep 19, 2013 at 09:27:24AM -0400, Donald Stufft wrote:
> > Rationale
> > =========
> >
> > Currently, on systems without a platform package manager and repository,
> > installing a third-party Python package into a freshly installed Python
> > requires first identifying an appropriate package manager and then
> > installing it.
>
> Howdy Donald,
>
> Thanks for helping make Python better.
>
> However, speaking as a Debian-folk (but not for the Debian-folk, I'll
> let barry take care of that), many users of Python software don't even
> know what pip is (or even that Python exists) -- as such, I find it very
> unlikely that a development tool would be shiped as part of the Python
> distribution in Debian. I don't see this changing, even with this pep.
> python-pip is still installable, but I don't see it pulled in by
> default.
>
>
> > Even on systems that *do* have a platform package manager, it is
unlikely to
> > include every package that is available on the Python Package Index, and
>
> Yes. This is true. However, it's not Debian's job to include all of
> pypi. Pypi allows anyone to upload, and we have quite a bit of attention
> to concerns such as providing source for everything (less of a concern
> for Python, but pickles can still contain data not in the prefered form
> of modification), and proper free software licensing.
>
> We also have a concern about stability; so we manage all the package set
> together to allow non-technical end-users to install stuff and not worry
> about breaking their system. It's not always the case where you can
> upgrade a library without breaking API.
>
>
> I'm trimming the rest, since I don't want to get dragged into
> side-conversations about pip as a package manager.

That's OK, especially if running "pip" recommends installing "python-pip"
and "pyvenv" and "extractpip" are also separated out so that running
"pyvenv" recommends installing a suitable package that depends on both
Python and "python-pip".

We can add that to the PEP as an alternative approach for redistributors to
take that still offers a reasonable end-user experience.

Cheers,
Nick.

>
>
> Cheers,
>   Paul
>
>
> --
>  .''`.  Paul Tagliamonte <paultag at debian.org>
> : :'  : Proud Debian Developer
> `. `'`  4096R / 8F04 9AD8 2C92 066C 7352  D28A 7B58 5B30 807C 2A87
>  `-     http://people.debian.org/~paultag
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/354cad7b/attachment.html>

From solipsis at pitrou.net  Thu Sep 19 15:50:44 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 19 Sep 2013 15:50:44 +0200
Subject: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in
 Python installations
References: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io>
Message-ID: <20130919155044.41d992f6@pitrou.net>

Le Thu, 19 Sep 2013 09:27:24 -0400,
Donald Stufft <donald at stufft.io> a ?crit :
> We've updated PEP453 based on some of the early feedback we've gotten
> from -dev and Martin.
> 
> Major changes:
> 
> * Removal of the option to fetch pip from PyPI in order not to modify
> the trust model of the Python installers
> * Consequently rename the model from ``getpip`` to ``extractpip``

"ensurepip" ?

What happens if there is already a higher pip version installed?
I suppose "extractpip" doesn't do anything in that case?

(sorry, perhaps it's mentioned in the PEP and I haven't seen it.
The PEP has become so long that I've only skimmed through it.)

Regards

Antoine.



From donald at stufft.io  Thu Sep 19 15:52:01 2013
From: donald at stufft.io (Donald Stufft)
Date: Thu, 19 Sep 2013 09:52:01 -0400
Subject: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in
	Python installations
In-Reply-To: <CACac1F8J7H8YZ5HjQDVydqu5PJJ7AF2d+qMKoqMg=Tqumzrf5w@mail.gmail.com>
References: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io>
 <CACac1F8J7H8YZ5HjQDVydqu5PJJ7AF2d+qMKoqMg=Tqumzrf5w@mail.gmail.com>
Message-ID: <C1F08515-8078-4259-906A-8D1B8A6477F4@stufft.io>


On Sep 19, 2013, at 9:43 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 19 September 2013 14:27, Donald Stufft <donald at stufft.io> wrote:
>> Major changes:
>> 
>> * Removal of the option to fetch pip from PyPI in order not to modify the trust model of the Python installers
>> * Consequently rename the model from ``getpip`` to ``extractpip``
> 
> If extractpip (I agree, I don't like the name, installpip is better)
> only ever unpacks the bundled pip, and it's always run, why bother?
> Why not just bundle pip directly into site-packages? The extra step
> seems to add little or no value.
> 
> Paul

Well it's not always run (the PEP has it as an option in the
installers that is checked by default) but even if it were always
installed:

- Nick has stated something about making it clear in the OSs
  installer DB which files are owned by Python and which are
  owned by pip
- Upgrading becomes harder, instead of simply using pip's own
  mechanism the installer needs to take care not to clobber a user
  installed pip that is even newer than the bundled version.
- "Fixing" a broken environment. If someone accidentally uninstalls
   pip this provides a simple way to reinstall it that doesn't require
   the old standby of getpip.py

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/b8d08673/attachment.sig>

From donald at stufft.io  Thu Sep 19 15:54:08 2013
From: donald at stufft.io (Donald Stufft)
Date: Thu, 19 Sep 2013 09:54:08 -0400
Subject: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in
	Python installations
In-Reply-To: <20130919155044.41d992f6@pitrou.net>
References: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io>
 <20130919155044.41d992f6@pitrou.net>
Message-ID: <E5558EB5-DFF3-4566-8EB4-42A5980667A8@stufft.io>


On Sep 19, 2013, at 9:50 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> Le Thu, 19 Sep 2013 09:27:24 -0400,
> Donald Stufft <donald at stufft.io> a ?crit :
>> We've updated PEP453 based on some of the early feedback we've gotten
>> from -dev and Martin.
>> 
>> Major changes:
>> 
>> * Removal of the option to fetch pip from PyPI in order not to modify
>> the trust model of the Python installers
>> * Consequently rename the model from ``getpip`` to ``extractpip``
> 
> "ensurepip" ?
> 
> What happens if there is already a higher pip version installed?
> I suppose "extractpip" doesn't do anything in that case?
> 
> (sorry, perhaps it's mentioned in the PEP and I haven't seen it.
> The PEP has become so long that I've only skimmed through it.)

We could explicitly call this out but it's implied it will do nothing
because it's going to just call out to to the privately installed pip
to actually install itself, and pip itself won't "upgrade" to an older
version.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/154ea63d/attachment.sig>

From donald at stufft.io  Thu Sep 19 15:57:02 2013
From: donald at stufft.io (Donald Stufft)
Date: Thu, 19 Sep 2013 09:57:02 -0400
Subject: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in
	Python installations
In-Reply-To: <20130919135025.GA7393@helios.localdomain>
References: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io>
 <20130919133601.GA6420@helios.localdomain>
 <CADiSq7eG3ExCmpWUswKoR9Ya9=+if5ToUoQKd7o3yVWpLOjzaQ@mail.gmail.com>
 <20130919135025.GA7393@helios.localdomain>
Message-ID: <1E689A33-EA02-4967-BE7B-FD99BEE1BB40@stufft.io>


On Sep 19, 2013, at 9:50 AM, Paul Tagliamonte <paultag at debian.org> wrote:

> On Thu, Sep 19, 2013 at 11:48:49PM +1000, Nick Coghlan wrote:
>> That's OK, especially if running "pip" recommends installing "python-pip" and
>> "pyvenv" and "extractpip" are also separated out so that running "pyvenv"
>> recommends installing a suitable package that depends on both Python and
>> "python-pip".
> 
> Sounds sane to me!
> 
>> We can add that to the PEP as an alternative approach for redistributors to
>> take that still offers a reasonable end-user experience.
> 
> That'd really be great, and I'm sure it'd cut off the small bit of
> concern some Debian-folk have about this stuff :)

This also sounds reasonable to me. Really the PEP is about reducing user friction,
especially for new users, so the ideal method is to ensure it's always installed
but it'd be totally OK to do what Nick suggested as that still at leasts lets pypi
packages to simply document installing as ``pip install <package>`` and if it's not
installed by default on Debian they'll get a good message telling them what they
need to do.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/7fe7863e/attachment-0001.sig>

From ncoghlan at gmail.com  Thu Sep 19 15:58:58 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 19 Sep 2013 23:58:58 +1000
Subject: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in
 Python installations
In-Reply-To: <CACac1F8J7H8YZ5HjQDVydqu5PJJ7AF2d+qMKoqMg=Tqumzrf5w@mail.gmail.com>
References: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io>
 <CACac1F8J7H8YZ5HjQDVydqu5PJJ7AF2d+qMKoqMg=Tqumzrf5w@mail.gmail.com>
Message-ID: <CADiSq7eko=mSqo3MtziHyU8dmR-zjQoCVmsCM33Lz2PWq3AzXg@mail.gmail.com>

On 19 Sep 2013 23:43, "Paul Moore" <p.f.moore at gmail.com> wrote:
>
> On 19 September 2013 14:27, Donald Stufft <donald at stufft.io> wrote:
> > Major changes:
> >
> > * Removal of the option to fetch pip from PyPI in order not to modify
the trust model of the Python installers
> > * Consequently rename the model from ``getpip`` to ``extractpip``
>
> If extractpip (I agree, I don't like the name, installpip is better)
> only ever unpacks the bundled pip, and it's always run, why bother?
> Why not just bundle pip directly into site-packages? The extra step
> seems to add little or no value.

It's not always run - it's opt-out for the Windows and Mac OS X installers
and not invoked implicitly at all if installing from source. The bootstrap
also needs to be available for use by "venv".

There's also the significant fact that python-dev still has a blanket ban
on externally maintained standard library modules (for good reasons) and
the clear separation between the bootstrap module and pip itself is what is
allowing this proposal to adhere to that guideline.

Cheers,
Nick.

>
> Paul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/4106fdeb/attachment.html>

From ncoghlan at gmail.com  Thu Sep 19 16:05:26 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 20 Sep 2013 00:05:26 +1000
Subject: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in
 Python installations
In-Reply-To: <20130919155044.41d992f6@pitrou.net>
References: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io>
 <20130919155044.41d992f6@pitrou.net>
Message-ID: <CADiSq7e9gnzttq--i_T-+YqEASHucs1jGERb-AKUF=y9aENUMg@mail.gmail.com>

On 19 Sep 2013 23:53, "Antoine Pitrou" <solipsis at pitrou.net> wrote:
>
> Le Thu, 19 Sep 2013 09:27:24 -0400,
> Donald Stufft <donald at stufft.io> a ?crit :
> > We've updated PEP453 based on some of the early feedback we've gotten
> > from -dev and Martin.
> >
> > Major changes:
> >
> > * Removal of the option to fetch pip from PyPI in order not to modify
> > the trust model of the Python installers
> > * Consequently rename the model from ``getpip`` to ``extractpip``
>
> "ensurepip" ?

Oh, I like it - describes exactly why the module exists :)

>
> What happens if there is already a higher pip version installed?
> I suppose "extractpip" doesn't do anything in that case?
>
> (sorry, perhaps it's mentioned in the PEP and I haven't seen it.
> The PEP has become so long that I've only skimmed through it.)

As Donald said, the fact nothing happens in that case is currently only
implied by the fact this is the way pip's "--upgrade" option works.

We should make that explicit in the PEP.

Cheers,
Nick.

>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130920/1e593cca/attachment.html>

From donald at stufft.io  Thu Sep 19 16:15:45 2013
From: donald at stufft.io (Donald Stufft)
Date: Thu, 19 Sep 2013 10:15:45 -0400
Subject: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in
	Python installations
In-Reply-To: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io>
References: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io>
Message-ID: <28F9A6A3-4758-45F7-8D6F-EEBEB2C73512@stufft.io>

I've updated the PEP to include the solution for Debian that Nick
mentioned and to clarify the behavior in the upgrade case.

Online: http://hg.python.org/peps/rev/6c1b658bc16c


diff -r 2899ba3bcef1 pep-0453.txt
--- a/pep-0453.txt	Thu Sep 19 09:23:03 2013 -0400
+++ b/pep-0453.txt	Thu Sep 19 10:12:59 2013 -0400
@@ -294,7 +294,9 @@

 This ensures that, by default, installing or updating CPython will ensure
Update PEP453 to address Debian's concerns and Antoine's
 that the installed version of pip is at least as recent as the one included
-with that version of CPython.
+with that version of CPython. If a newer version of pip has already been
+installed then ``python -m extractpip --upgrade`` will simply return without
+doing anything.


 Installing from source
@@ -532,6 +534,12 @@
   * This may take the form of separate packages with dependencies on each
     other so that installing the Python package installs the pip package
     and installing the pip package installs the Python package.
+  * Another reasonable way to implement this is to package pip separately but
+    ensure that there is some sort of global hook that will recommend
+    installing the separate pip package when a user executes ``pip`` without
+    it being installed. Systems that choose this option should ensure that
+    the ``pyvenv`` command still installs pip into the virtual environment
+    by default.

 * Do not remove the bundled copy of pip.


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/4b41c680/attachment.sig>

From paultag at debian.org  Thu Sep 19 15:36:01 2013
From: paultag at debian.org (Paul Tagliamonte)
Date: Thu, 19 Sep 2013 09:36:01 -0400
Subject: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in
 Python installations
In-Reply-To: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io>
References: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io>
Message-ID: <20130919133601.GA6420@helios.localdomain>

On Thu, Sep 19, 2013 at 09:27:24AM -0400, Donald Stufft wrote:
> Rationale
> =========
> 
> Currently, on systems without a platform package manager and repository,
> installing a third-party Python package into a freshly installed Python
> requires first identifying an appropriate package manager and then
> installing it.

Howdy Donald,

Thanks for helping make Python better.

However, speaking as a Debian-folk (but not for the Debian-folk, I'll
let barry take care of that), many users of Python software don't even
know what pip is (or even that Python exists) -- as such, I find it very
unlikely that a development tool would be shiped as part of the Python
distribution in Debian. I don't see this changing, even with this pep.
python-pip is still installable, but I don't see it pulled in by
default.


> Even on systems that *do* have a platform package manager, it is unlikely to
> include every package that is available on the Python Package Index, and

Yes. This is true. However, it's not Debian's job to include all of
pypi. Pypi allows anyone to upload, and we have quite a bit of attention
to concerns such as providing source for everything (less of a concern
for Python, but pickles can still contain data not in the prefered form
of modification), and proper free software licensing.

We also have a concern about stability; so we manage all the package set
together to allow non-technical end-users to install stuff and not worry
about breaking their system. It's not always the case where you can
upgrade a library without breaking API.


I'm trimming the rest, since I don't want to get dragged into
side-conversations about pip as a package manager.


Cheers,
  Paul


-- 
 .''`.  Paul Tagliamonte <paultag at debian.org>
: :'  : Proud Debian Developer
`. `'`  4096R / 8F04 9AD8 2C92 066C 7352  D28A 7B58 5B30 807C 2A87
 `-     http://people.debian.org/~paultag
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/21d80dd8/attachment.sig>

From paultag at debian.org  Thu Sep 19 15:50:25 2013
From: paultag at debian.org (Paul Tagliamonte)
Date: Thu, 19 Sep 2013 09:50:25 -0400
Subject: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in
 Python installations
In-Reply-To: <CADiSq7eG3ExCmpWUswKoR9Ya9=+if5ToUoQKd7o3yVWpLOjzaQ@mail.gmail.com>
References: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io>
 <20130919133601.GA6420@helios.localdomain>
 <CADiSq7eG3ExCmpWUswKoR9Ya9=+if5ToUoQKd7o3yVWpLOjzaQ@mail.gmail.com>
Message-ID: <20130919135025.GA7393@helios.localdomain>

On Thu, Sep 19, 2013 at 11:48:49PM +1000, Nick Coghlan wrote:
> That's OK, especially if running "pip" recommends installing "python-pip" and
> "pyvenv" and "extractpip" are also separated out so that running "pyvenv"
> recommends installing a suitable package that depends on both Python and
> "python-pip".

Sounds sane to me!

> We can add that to the PEP as an alternative approach for redistributors to
> take that still offers a reasonable end-user experience.

That'd really be great, and I'm sure it'd cut off the small bit of
concern some Debian-folk have about this stuff :)


Cheers,
  Paul

-- 
 .''`.  Paul Tagliamonte <paultag at debian.org>
: :'  : Proud Debian Developer
`. `'`  4096R / 8F04 9AD8 2C92 066C 7352  D28A 7B58 5B30 807C 2A87
 `-     http://people.debian.org/~paultag
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/bd096e17/attachment.sig>

From ericsnowcurrently at gmail.com  Thu Sep 19 22:21:40 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 19 Sep 2013 14:21:40 -0600
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <CADiSq7eQ457=Vva-_3xbpi8o2EtS9ea8Q3XG=oHe0xBBAtzywQ@mail.gmail.com>
References: <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
 <522DECA3.2030709@hotpy.org>
 <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com>
 <522DF204.7040109@hotpy.org>
 <0752376899dd7436f6115bfc03594574@chopin.edu.pl>
 <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com>
 <20130913065236.GF16820@ando>
 <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com>
 <20130913122328.GI16820@ando>
 <CADiSq7cdSEHryP+LMppRdabTuUG-GzMd56ub-oWpFNKPHNzsHA@mail.gmail.com>
 <9BBAFB63-68C5-441B-8ED4-43604241AA42@mac.com>
 <CACac1F_gT=mDEV3_VwvhjvS2H=JmNodWdP1vqMcEopWKMrPPHA@mail.gmail.com>
 <CADiSq7eQ457=Vva-_3xbpi8o2EtS9ea8Q3XG=oHe0xBBAtzywQ@mail.gmail.com>
Message-ID: <CALFfu7D2U0oG4LjRh0wLKPdaxAJv31-azR8LEG=+=2HgkNKzWA@mail.gmail.com>

On Thu, Sep 19, 2013 at 4:12 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 19 Sep 2013 20:00, "Paul Moore" <p.f.moore at gmail.com> wrote:
> >
> > On 19 September 2013 10:32, Ronald Oussoren <ronaldoussoren at mac.com>
> wrote:
> > > The first time a method is called the bridge looks for an Objective-C
> selector
> > > with the same name and adds that to the class dictionary. This works
> fine for normal
> > > method lookups, by overriding __getattribute__, but causes problems
> with super:
> > > super happily ignores __getattribute__ and peeks in the class __dict__
> which may
> > > not yet contain the name we're looking for and that can result in
> incorrect results
> > > (both incorrect AttributeErrors and totally incorrect results when the
> name is
> > > not yet present in the parent class' __dict__ but is in the
> grandparent's __dict__).
> >
> > As an alternative approach, could you use a custom dict subclass as
> > the class __dict__, and catch the peeking in the class __dict__ that
> > way? Or is this one of those places where only a real dict will do?
>
> Even Python 3 doesn't let you control the *runtime* type of the class
> dict, only the type used during evaluation of the class body.
>
> I've played with changing that - it makes for a rather special interpreter
> experience :)
>
Same here. :)  The PyDict_* API is not your friend for that.  It's why I
gave up on using a C OrderedDict for tp_dict (opting for a
__definition_order__ attribute on classes instead).

-eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/bc4c279e/attachment.html>

From ericsnowcurrently at gmail.com  Thu Sep 19 22:23:12 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 19 Sep 2013 14:23:12 -0600
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <C1DAADB8-E611-4359-B0F4-7914E9579368@mac.com>
References: <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
 <522DECA3.2030709@hotpy.org>
 <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com>
 <522DF204.7040109@hotpy.org>
 <0752376899dd7436f6115bfc03594574@chopin.edu.pl>
 <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com>
 <20130913065236.GF16820@ando>
 <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com>
 <20130913122328.GI16820@ando>
 <CADiSq7cdSEHryP+LMppRdabTuUG-GzMd56ub-oWpFNKPHNzsHA@mail.gmail.com>
 <9BBAFB63-68C5-441B-8ED4-43604241AA42@mac.com>
 <CACac1F_gT=mDEV3_VwvhjvS2H=JmNodWdP1vqMcEopWKMrPPHA@mail.gmail.com>
 <C1DAADB8-E611-4359-B0F4-7914E9579368@mac.com>
Message-ID: <CALFfu7CTHJohah-detWO=WoDe6jhLBCXcDgez2xgPLshZ-i13g@mail.gmail.com>

On Thu, Sep 19, 2013 at 4:04 AM, Ronald Oussoren <ronaldoussoren at mac.com>wrote:

> The C code uses PyDict_GetItem and AFAIK that doesn't look for a
> __getitem__
> implementation in a subclass.
>

Yeah, the PyDict_* API is definitely not subclass friendly. :(

-eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/19f7f5c4/attachment.html>

From ericsnowcurrently at gmail.com  Thu Sep 19 22:25:20 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 19 Sep 2013 14:25:20 -0600
Subject: [Python-Dev] PEP 447: add type.__locallookup__
In-Reply-To: <4CE957E3-ECE7-4DD1-AB62-2EA7776B9F85@mac.com>
References: <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com>
 <522DDB74.5080609@stoneleaf.us>
 <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com>
 <522DECA3.2030709@hotpy.org>
 <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com>
 <522DF204.7040109@hotpy.org>
 <0752376899dd7436f6115bfc03594574@chopin.edu.pl>
 <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com>
 <20130913065236.GF16820@ando>
 <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com>
 <20130913122328.GI16820@ando>
 <CADiSq7cdSEHryP+LMppRdabTuUG-GzMd56ub-oWpFNKPHNzsHA@mail.gmail.com>
 <4CE957E3-ECE7-4DD1-AB62-2EA7776B9F85@mac.com>
Message-ID: <CALFfu7Bm5gvY+xDrDwpKwhfwhhSUSRYQE=T=pPP_Yo0XB7SioQ@mail.gmail.com>

On Thu, Sep 19, 2013 at 3:49 AM, Ronald Oussoren <ronaldoussoren at mac.com>wrote:

> On 14 Sep, 2013, at 8:30, Nick Coghlan <ncoghlan at gmail.com> wrote:
> > However, I agree the current wording only conveys that to the handful
> > of people that already know exactly when in the attribute lookup
> > sequence that step occurs, which is a rather niche audience :)
>
> I've been fooling around with this long enough that I forgot that not
> everyone knows this :-).
>
> I guess I'd better include a clearer and more complete description
> of the current attribute resolution protocol and how my proposal affects
> that.  A nice readable Python implementation of that protocol would be nice
> to have regardless of the fate of this PEP.
>

+1

-eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/f22ef310/attachment.html>

From benhgift at gmail.com  Thu Sep 19 22:54:08 2013
From: benhgift at gmail.com (Ben Gift)
Date: Thu, 19 Sep 2013 13:54:08 -0700
Subject: [Python-Dev] Use an empty def as a lambda
Message-ID: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com>

I think the lambda keyword is difficult to understand for many people. It
would be more pythonic to use an empty def call instead.

For instance this:

    words.sort(key = lambda x: x[2])

could look like this:

    words.sort(key = def (x): x[2])

It's obvious and explicit that we're creating an unnamed, anonymous
function this way.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/0b985608/attachment-0001.html>

From joe.pinsonault at gmail.com  Thu Sep 19 23:00:07 2013
From: joe.pinsonault at gmail.com (Joe Pinsonault)
Date: Thu, 19 Sep 2013 14:00:07 -0700
Subject: [Python-Dev] Use an empty def as a lambda
In-Reply-To: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com>
References: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com>
Message-ID: <CALFC9k-QaxOGBoCKK6Lvx+b5d2rxfaA8qendbhE22+jE=4EoCg@mail.gmail.com>

I think it's a great idea personally. It's explicit and obvious. "lamda" is
too computer sciencey
On Sep 19, 2013 1:55 PM, "Ben Gift" <benhgift at gmail.com> wrote:

> I think the lambda keyword is difficult to understand for many people. It
> would be more pythonic to use an empty def call instead.
>
> For instance this:
>
>     words.sort(key = lambda x: x[2])
>
> could look like this:
>
>     words.sort(key = def (x): x[2])
>
> It's obvious and explicit that we're creating an unnamed, anonymous
> function this way.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/joe.pinsonault%40gmail.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/88a252e1/attachment.html>

From ncoghlan at gmail.com  Thu Sep 19 23:17:58 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 20 Sep 2013 07:17:58 +1000
Subject: [Python-Dev] Use an empty def as a lambda
In-Reply-To: <CALFC9k-QaxOGBoCKK6Lvx+b5d2rxfaA8qendbhE22+jE=4EoCg@mail.gmail.com>
References: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com>
 <CALFC9k-QaxOGBoCKK6Lvx+b5d2rxfaA8qendbhE22+jE=4EoCg@mail.gmail.com>
Message-ID: <CADiSq7dkj6aP-909ifBmt3jWxSwD4i=eM==T+vQK7y1TnLW2TA@mail.gmail.com>

On 20 Sep 2013 07:04, "Joe Pinsonault" <joe.pinsonault at gmail.com> wrote:
>
> I think it's a great idea personally. It's explicit and obvious. "lamda"
is too computer sciencey

This suggestion has been made many times, occasionally with the associated
"must be contained in parentheses when used as an expression" caveat that
is needed to avoid making the language grammar ambiguous at the statement
level.

It mainly runs afoul of two problems:

- reusing the same keyword would make the additional syntactic restrictions
of the expression form even more confusing.
- Guido doesn't particularly like the notion of functions-as-expressions in
the first place (which is why lambda was on thin ice when Python 3 was
being designed), so doesn't actually mind the fact that people avoid using
them because they don't like the keyword.

Cheers,
Nick.

>
> On Sep 19, 2013 1:55 PM, "Ben Gift" <benhgift at gmail.com> wrote:
>>
>> I think the lambda keyword is difficult to understand for many people.
It would be more pythonic to use an empty def call instead.
>>
>> For instance this:
>>
>>     words.sort(key = lambda x: x[2])
>>
>> could look like this:
>>
>>     words.sort(key = def (x): x[2])
>>
>> It's obvious and explicit that we're creating an unnamed, anonymous
function this way.
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/joe.pinsonault%40gmail.com
>>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130920/290c0ca3/attachment.html>

From catch-all at masklinn.net  Thu Sep 19 23:21:11 2013
From: catch-all at masklinn.net (Xavier Morel)
Date: Thu, 19 Sep 2013 23:21:11 +0200
Subject: [Python-Dev] Use an empty def as a lambda
In-Reply-To: <CADiSq7dkj6aP-909ifBmt3jWxSwD4i=eM==T+vQK7y1TnLW2TA@mail.gmail.com>
References: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com>
 <CALFC9k-QaxOGBoCKK6Lvx+b5d2rxfaA8qendbhE22+jE=4EoCg@mail.gmail.com>
 <CADiSq7dkj6aP-909ifBmt3jWxSwD4i=eM==T+vQK7y1TnLW2TA@mail.gmail.com>
Message-ID: <B0854DED-7AC6-41E7-B409-E32C0ADC2C03@masklinn.net>

On 2013-09-19, at 23:17 , Nick Coghlan wrote:

> On 20 Sep 2013 07:04, "Joe Pinsonault" <joe.pinsonault at gmail.com> wrote:
>> 
>> I think it's a great idea personally. It's explicit and obvious. "lamda"
> is too computer sciencey
> 
> This suggestion has been made many times, occasionally with the associated
> "must be contained in parentheses when used as an expression" caveat that
> is needed to avoid making the language grammar ambiguous at the statement
> level.

Examples of some of these times:

https://wiki.python.org/moin/AlternateLambdaSyntax
https://mail.python.org/pipermail/python-dev/2006-February/060415.html
https://mail.python.org/pipermail/python-dev/2006-February/thread.html#60415

Unless significant new insight is developed or Guido has picked the
functional bug at dropbox, merely suggesting a name change from lambda
to def (which has already been suggested in the past) probably isn't
going to cut it.

From alexander.belopolsky at gmail.com  Fri Sep 20 00:19:47 2013
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Thu, 19 Sep 2013 18:19:47 -0400
Subject: [Python-Dev] Use an empty def as a lambda
In-Reply-To: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com>
References: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com>
Message-ID: <CAP7h-xbomsduOfwDMKBGk4bJfgRQBu7g16Db-eMmRhKJFK3usA@mail.gmail.com>

On Thu, Sep 19, 2013 at 4:54 PM, Ben Gift <benhgift at gmail.com> wrote:

> It would be more pythonic to use an empty def call instead.


No, it won't.  Python draws a very strong distinction between expressions
and statements.  This line has been blurred somewhat with the advent of
 comprehensions and the if-else expression, but it would still require more
benefit than three characters in a keyword saving to allow def use in both
statements and expressions.

The following, for example, does not look pythonic at all:

*def* transform(*seq*, *func*=*def*(*x*):*x*):
      ...

(Note that I attempted to emulate syntax highlighting to make my point.)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/0ee8e0fd/attachment.html>

From rymg19 at gmail.com  Fri Sep 20 00:24:22 2013
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Thu, 19 Sep 2013 17:24:22 -0500
Subject: [Python-Dev] Use an empty def as a lambda
In-Reply-To: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com>
References: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com>
Message-ID: <CAO41-mPj9jigQVBNaWUeLmJPxYkdiyAdV9dbBjDFWWxKq=_zmg@mail.gmail.com>

Nice idea, BUT...

Not sure how a parser addition that supports it would go. Imagine this: if
you did a one-line function:

def test(x): print(x)

Python could interpret it two ways:

`def` `name` `lparen` `name` `rparen` `colon`...

OR, it could see it as a lambda-like thingamajig and throw a syntax error.
And, if someone accidentally wrote:

def (x): print(x)

Python should throw a syntax error. But it won't. And it'll take the person
a tad bit to realize he forgot the function name. Whoops.

And, it just would be odd in general.


On Thu, Sep 19, 2013 at 3:54 PM, Ben Gift <benhgift at gmail.com> wrote:

> I think the lambda keyword is difficult to understand for many people. It
> would be more pythonic to use an empty def call instead.
>
> For instance this:
>
>     words.sort(key = lambda x: x[2])
>
> could look like this:
>
>     words.sort(key = def (x): x[2])
>
> It's obvious and explicit that we're creating an unnamed, anonymous
> function this way.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com
>
>


-- 
Ryan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/c3382176/attachment.html>

From steve at pearwood.info  Fri Sep 20 02:27:07 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 20 Sep 2013 10:27:07 +1000
Subject: [Python-Dev] Use an empty def as a lambda
In-Reply-To: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com>
References: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com>
Message-ID: <20130920002707.GN19939@ando>

On Thu, Sep 19, 2013 at 01:54:08PM -0700, Ben Gift wrote:
> I think the lambda keyword is difficult to understand for many people. It
> would be more pythonic to use an empty def call instead.

Hi Ben, and welcome! Is this your first post? I'm afraid I don't 
recognise your name.

I think this discussion belongs on the python-ideas mailing list rather 
than here. Proposals for changes to syntax and functionality are 
normally expected to gather feedback on python-ideas before coming to 
python-dev for final approval or rejection.

https://mail.python.org/mailman/listinfo/python-ideas

http://docs.python.org/devguide/communication.html



-- 
Steven

From jcea at jcea.es  Fri Sep 20 13:19:24 2013
From: jcea at jcea.es (Jesus Cea)
Date: Fri, 20 Sep 2013 13:19:24 +0200
Subject: [Python-Dev] sys.intern should work on bytes
Message-ID: <523C2F3C.7020605@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

"""
sys.intern(b'12121212')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: must be str, not bytes
"""

I wonder why.

- -- 
Jes?s Cea Avi?n                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQCVAwUBUjwvPJlgi5GaxT1NAQIINQP/ZmEyPBSapa52yZRhQf8QSVSBm5tXpWrC
k9MbcvsK/5K6ArRogkulk1GSu1EJPPvuHMAXX8EenNFBXPvDRm0mOxrHkcYw5IKx
Ml2ORENm+EEM/ziUJMRtY4aqD5Jp6pXSSl9UmP5OQMDJfuabSrVqs7X2409OOhUj
BXeg9HvURxo=
=78M+
-----END PGP SIGNATURE-----

From victor.stinner at gmail.com  Fri Sep 20 14:04:11 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 20 Sep 2013 14:04:11 +0200
Subject: [Python-Dev] sys.intern should work on bytes
In-Reply-To: <523C2F3C.7020605@jcea.es>
References: <523C2F3C.7020605@jcea.es>
Message-ID: <CAMpsgwammTPe_2UAM+AdqxmXP8c6hnqPiOXwMUKLvyTjFnQL6g@mail.gmail.com>

2013/9/20 Jesus Cea <jcea at jcea.es>:
> """
> sys.intern(b'12121212')
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> TypeError: must be str, not bytes
> """
>
> I wonder why.

Intern strings optimize dictionary lookup. In Python 3, most
dictionaries use str keys (ex: __dict__ of classes).

What would you be the use case of interned bytes objets?

Victor

From solipsis at pitrou.net  Fri Sep 20 14:15:51 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 20 Sep 2013 14:15:51 +0200
Subject: [Python-Dev] sys.intern should work on bytes
References: <523C2F3C.7020605@jcea.es>
Message-ID: <20130920141551.0ac2c769@pitrou.net>

Le Fri, 20 Sep 2013 13:19:24 +0200,
Jesus Cea <jcea at jcea.es> a ?crit :
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> """
> sys.intern(b'12121212')
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> TypeError: must be str, not bytes
> """
> 
> I wonder why.

From http://docs.python.org/3.3/library/sys.html#sys.intern

"""sys.intern(string)

    Enter string in the table of ?interned? strings and return the
    interned string [...]"""


In Python 3 context, "string" means "str".

Regards

Antoine.



From jcea at jcea.es  Fri Sep 20 15:14:37 2013
From: jcea at jcea.es (Jesus Cea)
Date: Fri, 20 Sep 2013 15:14:37 +0200
Subject: [Python-Dev] sys.intern should work on bytes
In-Reply-To: <CAMpsgwammTPe_2UAM+AdqxmXP8c6hnqPiOXwMUKLvyTjFnQL6g@mail.gmail.com>
References: <523C2F3C.7020605@jcea.es>
 <CAMpsgwammTPe_2UAM+AdqxmXP8c6hnqPiOXwMUKLvyTjFnQL6g@mail.gmail.com>
Message-ID: <523C4A3D.90901@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 20/09/13 14:04, Victor Stinner wrote:
> What would you be the use case of interned bytes objets?

Performance and memory. Pickle sizes (my particular issue now).

- -- 
Jes?s Cea Avi?n                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQCVAwUBUjxKPZlgi5GaxT1NAQJ6FwQAim2H+OGpRD75KplahNhnKIfm9AUqVnHg
CaLakWhADdHBYlit+DxRQsxtv5C7gyhhqMryydyvx97z33VaI2p1RGOOcK/lWdNw
ETcetqJo8UswS2PSthJ0e5snOUsIeVJRomhJ48n8sJfIadCxAk6ozdMR75pHP5Y3
lJoUuUgdcJU=
=vbuK
-----END PGP SIGNATURE-----

From solipsis at pitrou.net  Fri Sep 20 15:31:12 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 20 Sep 2013 15:31:12 +0200
Subject: [Python-Dev] sys.intern should work on bytes
References: <523C2F3C.7020605@jcea.es>
 <CAMpsgwammTPe_2UAM+AdqxmXP8c6hnqPiOXwMUKLvyTjFnQL6g@mail.gmail.com>
 <523C4A3D.90901@jcea.es>
Message-ID: <20130920153112.5af58729@pitrou.net>

Le Fri, 20 Sep 2013 15:14:37 +0200,
Jesus Cea <jcea at jcea.es> a ?crit :
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 20/09/13 14:04, Victor Stinner wrote:
> > What would you be the use case of interned bytes objets?
> 
> Performance and memory. Pickle sizes (my particular issue now).

sys.intern is an internal interpreter optimization and should be
orthogonal to pickling. If pickle can't detect already-seen bytes
object, then you may file an improvement request on the bug tracker.

Regards

Antoine.



From jcea at jcea.es  Fri Sep 20 15:33:05 2013
From: jcea at jcea.es (Jesus Cea)
Date: Fri, 20 Sep 2013 15:33:05 +0200
Subject: [Python-Dev] sys.intern should work on bytes
In-Reply-To: <20130920141551.0ac2c769@pitrou.net>
References: <523C2F3C.7020605@jcea.es> <20130920141551.0ac2c769@pitrou.net>
Message-ID: <523C4E91.4050905@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 20/09/13 14:15, Antoine Pitrou wrote:
> From http://docs.python.org/3.3/library/sys.html#sys.intern
> 
> """sys.intern(string)
> 
> Enter string in the table of ?interned? strings and return the 
> interned string [...]"""
> 
> 
> In Python 3 context, "string" means "str".

I read that, Antoine. In fact I read the manual, I thought it was a
mistake carried over from 2.x documentation, I tried it just in case
before reporting the "documentation mistake", and I was surprised it
was actually true :-).

I know that intern is used for performance reasons internally to the
interpreter. But I am thinking about memory usage optimizations. For
instance, I have a pickle that is 14MB in size, when "interning" the
strings on it (there are a lot of redundancy), the new size is only
3MB and it loads faster. I can do it because most data in the pickle
are strings, I could NOT do it if I used bytes.

I could do a manual "intern" for hashable objects by hand using an
"object:object" dictionary (that would work for integers too), but I
wonder if extending builtin "sys.intern" would be something to consider.

Anyway, this pattern is easy enough:

Instead of

  object = sys.intern(object)

I could do

  interned = dict()
  ...
  object = interned.setdefault(object, object)

- -- 
Jes?s Cea Avi?n                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQCVAwUBUjxOkZlgi5GaxT1NAQIOVgQAhN36yRAAQP1YWbDsXGSamgZnhEULTloB
penRZYTYz/Ir/VM9l27GoXS7ThGrucAkkYZoJqXnUr2vyP0hq6rsfp+N5lzl61Nf
mDJBCtAczzKNdYqQSgMQ+Ugk43KnbEFFX7SB9Y5IkYroWCeWq7+5y6KX3ZKBspXG
lmXotLgpvW0=
=/RNw
-----END PGP SIGNATURE-----

From benjamin at python.org  Fri Sep 20 15:33:35 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Fri, 20 Sep 2013 09:33:35 -0400
Subject: [Python-Dev] sys.intern should work on bytes
In-Reply-To: <523C4A3D.90901@jcea.es>
References: <523C2F3C.7020605@jcea.es>
 <CAMpsgwammTPe_2UAM+AdqxmXP8c6hnqPiOXwMUKLvyTjFnQL6g@mail.gmail.com>
 <523C4A3D.90901@jcea.es>
Message-ID: <CAPZV6o_z6gyiGSEwP2i7uDagjxdn8i1p-cUWEHN-JAUXR=+zFw@mail.gmail.com>

Well, the pickler should memoize bytes objects if you have lots of the
same one in a pickle...

2013/9/20 Jesus Cea <jcea at jcea.es>:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 20/09/13 14:04, Victor Stinner wrote:
>> What would you be the use case of interned bytes objets?
>
> Performance and memory. Pickle sizes (my particular issue now).
>
> - --
> Jes?s Cea Avi?n                         _/_/      _/_/_/        _/_/_/
> jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
> Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
> jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
> "Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
> "My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
> "El amor es poner tu felicidad en la felicidad de otro" - Leibniz
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQCVAwUBUjxKPZlgi5GaxT1NAQJ6FwQAim2H+OGpRD75KplahNhnKIfm9AUqVnHg
> CaLakWhADdHBYlit+DxRQsxtv5C7gyhhqMryydyvx97z33VaI2p1RGOOcK/lWdNw
> ETcetqJo8UswS2PSthJ0e5snOUsIeVJRomhJ48n8sJfIadCxAk6ozdMR75pHP5Y3
> lJoUuUgdcJU=
> =vbuK
> -----END PGP SIGNATURE-----
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/benjamin%40python.org



-- 
Regards,
Benjamin

From jcea at jcea.es  Fri Sep 20 15:36:45 2013
From: jcea at jcea.es (Jesus Cea)
Date: Fri, 20 Sep 2013 15:36:45 +0200
Subject: [Python-Dev] sys.intern should work on bytes
In-Reply-To: <20130920153112.5af58729@pitrou.net>
References: <523C2F3C.7020605@jcea.es>
 <CAMpsgwammTPe_2UAM+AdqxmXP8c6hnqPiOXwMUKLvyTjFnQL6g@mail.gmail.com>
 <523C4A3D.90901@jcea.es> <20130920153112.5af58729@pitrou.net>
Message-ID: <523C4F6D.8060602@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 20/09/13 15:31, Antoine Pitrou wrote:
> sys.intern is an internal interpreter optimization and should be 
> orthogonal to pickling. If pickle can't detect already-seen bytes 
> object, then you may file an improvement request on the bug
> tracker.

Understood. Thanks for the clarification.

Pickle manage correctly "same object" references, but not "different
objects but equivalent". That is the issue. But for most uses this is
not a problem, and implementing this redundance removal looks like a
performance cost that few users would benefice from, but everybody
pays. Pickle is already slow enough now :).

- -- 
Jes?s Cea Avi?n                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQCVAwUBUjxPbZlgi5GaxT1NAQIyYQQAplpwDnz2/0bNTF7KN7V0PQnXZQknEnvL
0VACm298Y386hs8bJQFlRlTOzfhguulaaEwdqLaBPkXMKA7LBaVRHM1v5o+Xb40X
7DwSKglkbt6HgX7/nDMX4qk9Kxb2ZEVz+XOozPta2NRoZmaz8y7Xyqc/+4+UTzHH
XsNJ2H5yVEo=
=GPtw
-----END PGP SIGNATURE-----

From solipsis at pitrou.net  Fri Sep 20 15:42:17 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 20 Sep 2013 15:42:17 +0200
Subject: [Python-Dev] sys.intern should work on bytes
References: <523C2F3C.7020605@jcea.es>
 <CAMpsgwammTPe_2UAM+AdqxmXP8c6hnqPiOXwMUKLvyTjFnQL6g@mail.gmail.com>
 <523C4A3D.90901@jcea.es> <20130920153112.5af58729@pitrou.net>
 <523C4F6D.8060602@jcea.es>
Message-ID: <20130920154217.7f564312@pitrou.net>

Le Fri, 20 Sep 2013 15:36:45 +0200,
Jesus Cea <jcea at jcea.es> a ?crit :
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 20/09/13 15:31, Antoine Pitrou wrote:
> > sys.intern is an internal interpreter optimization and should be 
> > orthogonal to pickling. If pickle can't detect already-seen bytes 
> > object, then you may file an improvement request on the bug
> > tracker.
> 
> Understood. Thanks for the clarification.
> 
> Pickle manage correctly "same object" references, but not "different
> objects but equivalent". That is the issue.

Ah, well, in that case the issue is not in pickle, it's in your
code.  pickle doesn't try to guess if "equal" is really functionally
equivalent to "identical".

Regards

Antoine.



From solipsis at pitrou.net  Fri Sep 20 15:44:15 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 20 Sep 2013 15:44:15 +0200
Subject: [Python-Dev] sys.intern should work on bytes
References: <523C2F3C.7020605@jcea.es> <20130920141551.0ac2c769@pitrou.net>
 <523C4E91.4050905@jcea.es>
Message-ID: <20130920154415.5e6cfa09@pitrou.net>

Le Fri, 20 Sep 2013 15:33:05 +0200,
Jesus Cea <jcea at jcea.es> a ?crit :
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 20/09/13 14:15, Antoine Pitrou wrote:
> > From http://docs.python.org/3.3/library/sys.html#sys.intern
> > 
> > """sys.intern(string)
> > 
> > Enter string in the table of ?interned? strings and return the 
> > interned string [...]"""
> > 
> > 
> > In Python 3 context, "string" means "str".
> 
> I read that, Antoine. In fact I read the manual, I thought it was a
> mistake carried over from 2.x documentation, I tried it just in case
> before reporting the "documentation mistake", and I was surprised it
> was actually true :-).
> 
> I know that intern is used for performance reasons internally to the
> interpreter. But I am thinking about memory usage optimizations. For
> instance, I have a pickle that is 14MB in size, when "interning" the
> strings on it (there are a lot of redundancy), the new size is only
> 3MB and it loads faster. I can do it because most data in the pickle
> are strings, I could NOT do it if I used bytes.
> 
> I could do a manual "intern" for hashable objects by hand using an
> "object:object" dictionary (that would work for integers too), but I
> wonder if extending builtin "sys.intern" would be something to
> consider.
> 
> Anyway, this pattern is easy enough:
> 
> Instead of
> 
>   object = sys.intern(object)
> 
> I could do
> 
>   interned = dict()
>   ...
>   object = interned.setdefault(object, object)

Yes. The main difference is that sys.intern() will remove the interned
strings when every external reference vanishes. It requires either
weakref'ability (which both str and bytes lack) or special cooperation
from the object destructor (which is why sys.intern() is restricted to
str instead of working with arbitrary objects).

Regards

Antoine.



From jcea at jcea.es  Fri Sep 20 15:46:41 2013
From: jcea at jcea.es (Jesus Cea)
Date: Fri, 20 Sep 2013 15:46:41 +0200
Subject: [Python-Dev] dict.setdefault(object,
 object) instead of "sys.intern()" (was Re: sys.intern should work
 on bytes)
In-Reply-To: <CAPZV6o_z6gyiGSEwP2i7uDagjxdn8i1p-cUWEHN-JAUXR=+zFw@mail.gmail.com>
References: <523C2F3C.7020605@jcea.es>
 <CAMpsgwammTPe_2UAM+AdqxmXP8c6hnqPiOXwMUKLvyTjFnQL6g@mail.gmail.com>
 <523C4A3D.90901@jcea.es>
 <CAPZV6o_z6gyiGSEwP2i7uDagjxdn8i1p-cUWEHN-JAUXR=+zFw@mail.gmail.com>
Message-ID: <523C51C1.9090901@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 20/09/13 15:33, Benjamin Peterson wrote:
> Well, the pickler should memoize bytes objects if you have lots of
> the same one in a pickle...

Only if they are the very same object. Not diferent bytes objects with
the same value. Pickle doesn't do "a==b" but "id(a)==id(b)".

Yes, I know that "a==b" would break mutable objects. It is just an
example.

I don't want to pursue that path. Performance of pickle is already
appallingly slow.

In my project, I will do the redundancy removal on my own way, as
explained in ither message on this thread.

Example:

* Original pickle: 14416284 bytes

* Pickle with "interned" strings: 3004880 bytes
(quite an improvement, but this is particular to my case, I have a lot
of string duplications here. The pickle also loads a bit faster)

* Pickle including an extra dictionary of "interned" strings, created
using the "interned.setdefault(object,object)" pattern: 5126587 bytes.
Sniff.

Could I do this more compactly?.


- -- 
Jes?s Cea Avi?n                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQCVAwUBUjxRwZlgi5GaxT1NAQKW8wP/dhVa/v3RZbOKvOtogpHGs5nZyjhtChwn
lFK1Lr1wl/+6IgCjgu9axkrRM0LLRaBN91HW+e9AkAM9XSFBQp6qAAqjJpI/jLDp
xRLW9fMRHpD21m1tG9zxziz4ACCLNNDnlsyY9l7oHHbMzaAX6Gbigyml3hEbj0uK
G5hk4VhyKEY=
=m/3T
-----END PGP SIGNATURE-----

From jcea at jcea.es  Fri Sep 20 15:54:48 2013
From: jcea at jcea.es (Jesus Cea)
Date: Fri, 20 Sep 2013 15:54:48 +0200
Subject: [Python-Dev] sys.intern should work on bytes
In-Reply-To: <20130920154415.5e6cfa09@pitrou.net>
References: <523C2F3C.7020605@jcea.es> <20130920141551.0ac2c769@pitrou.net>
 <523C4E91.4050905@jcea.es> <20130920154415.5e6cfa09@pitrou.net>
Message-ID: <523C53A8.4070106@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 20/09/13 15:44, Antoine Pitrou wrote:

> Yes. The main difference is that sys.intern() will remove the
> interned strings when every external reference vanishes. It
> requires either weakref'ability (which both str and bytes lack) or
> special cooperation from the object destructor (which is why
> sys.intern() is restricted to str instead of working with arbitrary
> objects).

Great comment. Thanks.

Why str/bytes doesn't support weakrefs, beside memory use?

- -- 
Jes?s Cea Avi?n                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQCVAwUBUjxTqJlgi5GaxT1NAQIbvwP/fs7e5MJwF0pCa0NObTx0xN8CFQIX9/jt
VkQ1Q7lPLSuRlZMC2B+0xfp9QoAsD6N3xqSXwG+T9uf7w6nZ9y3keI06kAdSn/Cz
D8EyxoeuNk2aGq0VIzMO260mgs9Gr+3DtWAROcWCMG+8sr5uekJ/LAhI04/xMkqZ
zr7aOy1xDgk=
=hzGP
-----END PGP SIGNATURE-----

From benjamin at python.org  Fri Sep 20 15:57:26 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Fri, 20 Sep 2013 09:57:26 -0400
Subject: [Python-Dev] sys.intern should work on bytes
In-Reply-To: <523C53A8.4070106@jcea.es>
References: <523C2F3C.7020605@jcea.es> <20130920141551.0ac2c769@pitrou.net>
 <523C4E91.4050905@jcea.es> <20130920154415.5e6cfa09@pitrou.net>
 <523C53A8.4070106@jcea.es>
Message-ID: <CAPZV6o_z3svzeznqB0KK8sk+pzXh8o121cKBPjTEA04DSVJ0MQ@mail.gmail.com>

2013/9/20 Jesus Cea <jcea at jcea.es>:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 20/09/13 15:44, Antoine Pitrou wrote:
>
>> Yes. The main difference is that sys.intern() will remove the
>> interned strings when every external reference vanishes. It
>> requires either weakref'ability (which both str and bytes lack) or
>> special cooperation from the object destructor (which is why
>> sys.intern() is restricted to str instead of working with arbitrary
>> objects).
>
> Great comment. Thanks.
>
> Why str/bytes doesn't support weakrefs, beside memory use?

Is increased memory use for every str/bytes object not a good enough reason?

-- 
Regards,
Benjamin

From status at bugs.python.org  Fri Sep 20 18:07:41 2013
From: status at bugs.python.org (Python tracker)
Date: Fri, 20 Sep 2013 18:07:41 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20130920160741.ECE98568E3@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2013-09-13 - 2013-09-20)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    4228 (+16)
  closed 26602 (+31)
  total  30830 (+47)

Open issues with patches: 1949 


Issues opened (37)
==================

#12085: subprocess.Popen.__del__ raises AttributeError if __init__ was
http://bugs.python.org/issue12085  reopened by serhiy.storchaka

#14927: add "Do not supply 'int' argument" to random.shuffle docstring
http://bugs.python.org/issue14927  reopened by orsenthil

#19011: Enum should have a __getattr__ that makes all the instances av
http://bugs.python.org/issue19011  opened by lambacck

#19012: urllib2: bad proxy configuration throws "getaddrinfo" error
http://bugs.python.org/issue19012  opened by SjoerdOptLand

#19014: Allow memoryview.cast() for empty views
http://bugs.python.org/issue19014  opened by serhiy.storchaka

#19016: autospecced namedtuples should be truthy by default
http://bugs.python.org/issue19016  opened by skrisman

#19017: selectors: towards uniform EBADF handling
http://bugs.python.org/issue19017  opened by neologix

#19019: Investigate using Apple clang for building OS X installers
http://bugs.python.org/issue19019  opened by ned.deily

#19020: Regression: Windows-tkinter-idle, unicode, and 0xxx filename
http://bugs.python.org/issue19020  opened by terry.reedy

#19021: AttributeError in Popen.__del__
http://bugs.python.org/issue19021  opened by serhiy.storchaka

#19022: Improve handling of type.__abstractmethods__ descriptor
http://bugs.python.org/issue19022  opened by ncoghlan

#19023: ctypes docs: Unimplemented and undocumented features
http://bugs.python.org/issue19023  opened by syeberman

#19024: Document asterisk (*), splat or star operator
http://bugs.python.org/issue19024  opened by techtonik

#19025: Deleting attribute of Enum gives misleading error message
http://bugs.python.org/issue19025  opened by vajrasky

#19027: undefined symbol: _PyParser_Grammar
http://bugs.python.org/issue19027  opened by arigo

#19028: tkinter.tkapp.merge() fails on non-strings
http://bugs.python.org/issue19028  opened by serhiy.storchaka

#19030: inspect.getmembers and inspect.classify_class_attrs mishandle 
http://bugs.python.org/issue19030  opened by ethan.furman

#19031: Make help() enum aware
http://bugs.python.org/issue19031  opened by ethan.furman

#19034: More useful repr for Tcl_Obj
http://bugs.python.org/issue19034  opened by serhiy.storchaka

#19035: tokenize.generate_tokens treat '\f' symbol as the end of file 
http://bugs.python.org/issue19035  opened by Alexey.Umnov

#19036: setlocale fails due to locale.h being wrapped up in LANGINFO c
http://bugs.python.org/issue19036  opened by alanh

#19040: Problems with overriding Enum.__new__
http://bugs.python.org/issue19040  opened by Drekin

#19042: Idle: add option to autosave 'Untitled' edit window
http://bugs.python.org/issue19042  opened by terry.reedy

#19043: Remove detailed listing of all versions from LICENSE, Doc/lice
http://bugs.python.org/issue19043  opened by georg.brandl

#19044: getaddrinfo raises near-useless exception
http://bugs.python.org/issue19044  opened by Nikratio

#19045: Make on Solaris 11 x64 with OracleStudio12.3 failed
http://bugs.python.org/issue19045  opened by podlipnik

#19046: SystemError: ..\Objects\weakrefobject.c:903: bad argument to i
http://bugs.python.org/issue19046  opened by mherrmann.at

#19047: Assorted weakref docs improvements
http://bugs.python.org/issue19047  opened by ncoghlan

#19048: itertools.tee doesn't have a __sizeof__ method
http://bugs.python.org/issue19048  opened by pitrou

#19049: itertools.tee uses int for indices
http://bugs.python.org/issue19049  opened by pitrou

#19050: crash while writing to a closed file descriptor
http://bugs.python.org/issue19050  opened by damiro

#19051: Unify buffered readers
http://bugs.python.org/issue19051  opened by serhiy.storchaka

#19053: read1() from zipfile returns empty data
http://bugs.python.org/issue19053  opened by serhiy.storchaka

#19054: Descriptors howto
http://bugs.python.org/issue19054  opened by marco.buttu

#19055: Regular expressions: * does not match as many repetitions as p
http://bugs.python.org/issue19055  opened by Jason.Stumpf

#19056: Windows 7, script exec not working without explicit cal of pyt
http://bugs.python.org/issue19056  opened by tsunwell

#19057: Sometimes urllib2 raises URLError when trying POST with httpS
http://bugs.python.org/issue19057  opened by mrDoctorWho0..



Most recent 15 issues with no replies (15)
==========================================

#19056: Windows 7, script exec not working without explicit cal of pyt
http://bugs.python.org/issue19056

#19054: Descriptors howto
http://bugs.python.org/issue19054

#19040: Problems with overriding Enum.__new__
http://bugs.python.org/issue19040

#19034: More useful repr for Tcl_Obj
http://bugs.python.org/issue19034

#19031: Make help() enum aware
http://bugs.python.org/issue19031

#19028: tkinter.tkapp.merge() fails on non-strings
http://bugs.python.org/issue19028

#19027: undefined symbol: _PyParser_Grammar
http://bugs.python.org/issue19027

#19022: Improve handling of type.__abstractmethods__ descriptor
http://bugs.python.org/issue19022

#19010: Make XMLPullParser in ElementTree inherit from XMLParser
http://bugs.python.org/issue19010

#19005: PyIter_Next crashes if passed a non-iterator
http://bugs.python.org/issue19005

#19003: email.generator.BytesGenerator corrupts data by changing line 
http://bugs.python.org/issue19003

#18982: Add tests for CLI of the calendar module
http://bugs.python.org/issue18982

#18979: Use argparse in the uu module
http://bugs.python.org/issue18979

#18977: The -t option has no effect in for uu command-line
http://bugs.python.org/issue18977

#18976: distutils/command/build_ext passes wrong linker flags
http://bugs.python.org/issue18976



Most recent 15 issues waiting for review (15)
=============================================

#19054: Descriptors howto
http://bugs.python.org/issue19054

#19053: read1() from zipfile returns empty data
http://bugs.python.org/issue19053

#19051: Unify buffered readers
http://bugs.python.org/issue19051

#19049: itertools.tee uses int for indices
http://bugs.python.org/issue19049

#19048: itertools.tee doesn't have a __sizeof__ method
http://bugs.python.org/issue19048

#19043: Remove detailed listing of all versions from LICENSE, Doc/lice
http://bugs.python.org/issue19043

#19036: setlocale fails due to locale.h being wrapped up in LANGINFO c
http://bugs.python.org/issue19036

#19034: More useful repr for Tcl_Obj
http://bugs.python.org/issue19034

#19030: inspect.getmembers and inspect.classify_class_attrs mishandle 
http://bugs.python.org/issue19030

#19028: tkinter.tkapp.merge() fails on non-strings
http://bugs.python.org/issue19028

#19025: Deleting attribute of Enum gives misleading error message
http://bugs.python.org/issue19025

#19023: ctypes docs: Unimplemented and undocumented features
http://bugs.python.org/issue19023

#19021: AttributeError in Popen.__del__
http://bugs.python.org/issue19021

#19020: Regression: Windows-tkinter-idle, unicode, and 0xxx filename
http://bugs.python.org/issue19020

#19016: autospecced namedtuples should be truthy by default
http://bugs.python.org/issue19016



Top 10 most discussed issues (10)
=================================

#19048: itertools.tee doesn't have a __sizeof__ method
http://bugs.python.org/issue19048  28 msgs

#18986: Add a case-insensitive case-preserving dict
http://bugs.python.org/issue18986  20 msgs

#18967: Find a less conflict prone approach to Misc/NEWS
http://bugs.python.org/issue18967  18 msgs

#11798: Test cases not garbage collected after run
http://bugs.python.org/issue11798  15 msgs

#19030: inspect.getmembers and inspect.classify_class_attrs mishandle 
http://bugs.python.org/issue19030  14 msgs

#16042: smtplib: unlimited readline() from connection
http://bugs.python.org/issue16042  10 msgs

#19036: setlocale fails due to locale.h being wrapped up in LANGINFO c
http://bugs.python.org/issue19036  10 msgs

#19049: itertools.tee uses int for indices
http://bugs.python.org/issue19049   8 msgs

#14927: add "Do not supply 'int' argument" to random.shuffle docstring
http://bugs.python.org/issue14927   7 msgs

#18990: Return root element from ElementTree.XMLPullParser.close() to 
http://bugs.python.org/issue18990   7 msgs



Issues closed (30)
==================

#13758: compile() should not encode 'filename' (at least on Windows)
http://bugs.python.org/issue13758  closed by haypo

#14984: netrc module allows read of non-secured .netrc file
http://bugs.python.org/issue14984  closed by r.david.murray

#16201: socket.gethostbyname incorrectly parses ip
http://bugs.python.org/issue16201  closed by neologix

#17734: Failure when running test_builtin after test_genexps
http://bugs.python.org/issue17734  closed by serhiy.storchaka

#17764: Support http.server passing bind address via commend line argu
http://bugs.python.org/issue17764  closed by orsenthil

#18230: test_builtin fails/hangs when run after test_getopt
http://bugs.python.org/issue18230  closed by serhiy.storchaka

#18526: Add resource management/guarding to unittest
http://bugs.python.org/issue18526  closed by zach.ware

#18824: Adding LogRecord attribute "traceback"
http://bugs.python.org/issue18824  closed by vinay.sajip

#18856: Added test coverage for calendar print functions
http://bugs.python.org/issue18856  closed by ezio.melotti

#18929: inspect.classify_class_attrs ignores metaclass
http://bugs.python.org/issue18929  closed by python-dev

#18937: add unittest assertion for logging
http://bugs.python.org/issue18937  closed by pitrou

#18945: Name collision handling in tempfile is not covered by tests
http://bugs.python.org/issue18945  closed by python-dev

#18951: In unittest.TestCase.assertRegex change "re" and "regex" to "r
http://bugs.python.org/issue18951  closed by ezio.melotti

#18955: Confusing documentation in Lib/importlib/util.py
http://bugs.python.org/issue18955  closed by brett.cannon

#18981: Typo in the ctypes tests
http://bugs.python.org/issue18981  closed by ezio.melotti

#18989: reuse of enum names in class creation inconsistent
http://bugs.python.org/issue18989  closed by python-dev

#18995: Enum does not work with reversed
http://bugs.python.org/issue18995  closed by ethan.furman

#18998: iter() not working in ElementTree
http://bugs.python.org/issue18998  closed by eli.bendersky

#19013: unittest's own test suite is not CLI-friendly
http://bugs.python.org/issue19013  closed by pitrou

#19015: Too long command returns 32512
http://bugs.python.org/issue19015  closed by Sworddragon

#19018: Heapq.merge suppreses IndexError from user generator
http://bugs.python.org/issue19018  closed by rhettinger

#19026: OrderedDict should not accept dict as parameter
http://bugs.python.org/issue19026  closed by rhettinger

#19029: tix.py uses StringType
http://bugs.python.org/issue19029  closed by serhiy.storchaka

#19032: __reduce_ex__ on lock object
http://bugs.python.org/issue19032  closed by cool-RR

#19033: Python 3 won't go on PC-BSD 9.1
http://bugs.python.org/issue19033  closed by ned.deily

#19037: mailbox module modifies maildir file times after moving from t
http://bugs.python.org/issue19037  closed by r.david.murray

#19038: Fix sort order in Misc/ACKS.
http://bugs.python.org/issue19038  closed by georg.brandl

#19039: sphinx search, result sorting
http://bugs.python.org/issue19039  closed by georg.brandl

#19041: requested (DLL) module could not be found
http://bugs.python.org/issue19041  closed by r.david.murray

#19052: Python's CGIHTTPServer does not handle Expect: 100-continue gr
http://bugs.python.org/issue19052  closed by Joseph.Warren

From pje at telecommunity.com  Fri Sep 20 18:50:44 2013
From: pje at telecommunity.com (PJ Eby)
Date: Fri, 20 Sep 2013 12:50:44 -0400
Subject: [Python-Dev] sys.intern should work on bytes
In-Reply-To: <523C53A8.4070106@jcea.es>
References: <523C2F3C.7020605@jcea.es> <20130920141551.0ac2c769@pitrou.net>
 <523C4E91.4050905@jcea.es> <20130920154415.5e6cfa09@pitrou.net>
 <523C53A8.4070106@jcea.es>
Message-ID: <CALeMXf7Qo94xYsP_8cufhp=OR0x4QyWu-gnzn=fq1k7NF2QEuw@mail.gmail.com>

On Fri, Sep 20, 2013 at 9:54 AM, Jesus Cea <jcea at jcea.es> wrote:
> Why str/bytes doesn't support weakrefs, beside memory use?

The typical use case for weakrefs is to break reference cycles, but
str and bytes can't *be* part of a reference cycle, so outside of
interning-like use cases, there's no need for weakref support there.

From anikom15 at gmail.com  Sat Sep 21 03:27:16 2013
From: anikom15 at gmail.com (=?iso-8859-1?Q?Westley_Mart=EDnez?=)
Date: Fri, 20 Sep 2013 18:27:16 -0700
Subject: [Python-Dev] Use an empty def as a lambda
In-Reply-To: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com>
References: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com>
Message-ID: <001e01ceb669$b6a1c740$23e555c0$@gmail.com>

'def' is no more ambiguous than 'lambda', and is in fact more ambiguous,
for 'def' doesn't lend itself to anything other than the word define,
whilst 'lambda' can only mean lambda function.  Calling def explicit is
silly.  It only makes sense because def arbitrarily means a function in
Python (I'm proposing def become func or proc in Python 4000).

To call lambda too 'computer sciencey' is equally ridiculous, for pro-
gramming is a key spawn of computer science.  A programmer needs to have
some knowledge of computer science to program, just like a physicist
needs knowledge of calculus to understand mechanics.

> -----Original Message-----
> From: Python-Dev [mailto:python-dev-bounces+anikom15=gmail.com at python.org] On
> Behalf Of Ben Gift
> Sent: Thursday, September 19, 2013 1:54 PM
> To: python-dev at python.org
> Subject: [Python-Dev] Use an empty def as a lambda
> 
> I think the lambda keyword is difficult to understand for many people. It
> would be more pythonic to use an empty def call instead.
> 
> 
> For instance this:
> 
> 
>     words.sort(key = lambda x: x[2])
> 
> 
> could look like this:
> 
>     words.sort(key = def (x): x[2])
> 
> 
> It's obvious and explicit that we're creating an unnamed, anonymous function
> this way.



From regebro at gmail.com  Sat Sep 21 07:42:19 2013
From: regebro at gmail.com (Lennart Regebro)
Date: Sat, 21 Sep 2013 07:42:19 +0200
Subject: [Python-Dev] Use an empty def as a lambda
In-Reply-To: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com>
References: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com>
Message-ID: <CAL0kPAV8DYq5LwoBtuK8ELZyN6CxLymD_o9UK=fU7uLGpF92Zg@mail.gmail.com>

On Thu, Sep 19, 2013 at 10:54 PM, Ben Gift <benhgift at gmail.com> wrote:
> I think the lambda keyword is difficult to understand for many people. It
> would be more pythonic to use an empty def call instead.

I agree, but that ship has sailed, at least until the time when Python
2 is dead. I don't want these kinds of syntax changes for a *looooong*
time now, personally.

//Lennart

From stefan_ml at behnel.de  Sat Sep 21 11:15:28 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 21 Sep 2013 11:15:28 +0200
Subject: [Python-Dev] dict.setdefault(object,
 object) instead of "sys.intern()" (was Re: sys.intern should work
 on bytes)
In-Reply-To: <523C51C1.9090901@jcea.es>
References: <523C2F3C.7020605@jcea.es>
 <CAMpsgwammTPe_2UAM+AdqxmXP8c6hnqPiOXwMUKLvyTjFnQL6g@mail.gmail.com>
 <523C4A3D.90901@jcea.es>
 <CAPZV6o_z6gyiGSEwP2i7uDagjxdn8i1p-cUWEHN-JAUXR=+zFw@mail.gmail.com>
 <523C51C1.9090901@jcea.es>
Message-ID: <l1jo38$a9q$1@ger.gmane.org>

Jesus Cea, 20.09.2013 15:46:
> On 20/09/13 15:33, Benjamin Peterson wrote:
>> Well, the pickler should memoize bytes objects if you have lots of
>> the same one in a pickle...
> 
> Only if they are the very same object. Not diferent bytes objects with
> the same value. Pickle doesn't do "a==b" but "id(a)==id(b)".
> 
> Yes, I know that "a==b" would break mutable objects. It is just an
> example.
> 
> I don't want to pursue that path. Performance of pickle is already
> appallingly slow.
> 
> In my project, I will do the redundancy removal on my own way, as
> explained in ither message on this thread.
> 
> Example:
> 
> * Original pickle: 14416284 bytes
> 
> * Pickle with "interned" strings: 3004880 bytes
> (quite an improvement, but this is particular to my case, I have a lot
> of string duplications here. The pickle also loads a bit faster)
> 
> * Pickle including an extra dictionary of "interned" strings, created
> using the "interned.setdefault(object,object)" pattern: 5126587 bytes.
> Sniff.
> 
> Could I do this more compactly?.

ISTM that what you are looking for is a compression-like pattern that
efficiently encodes repeated literals (i.e. constants of safe types) in the
pickle. That could be achieved by extending the pickle protocol to include
backreferences to earlier objects, I guess (I'm not all that familiar with
the internals of the pickle format). Any of the well known compression
algorithms that are capable of handling streaming data would apply here.
Assuming you don't want to simply send the pickle output through gzip &
friends, that is...

It also seems to me that python-dev isn't the right place to discuss this.
python-ideas seems more appropriate for now.

Stefan


From rymg19 at gmail.com  Sat Sep 21 19:55:20 2013
From: rymg19 at gmail.com (Ryan)
Date: Sat, 21 Sep 2013 12:55:20 -0500
Subject: [Python-Dev] Use an empty def as a lambda
In-Reply-To: <001e01ceb669$b6a1c740$23e555c0$@gmail.com>
References: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com>
 <001e01ceb669$b6a1c740$23e555c0$@gmail.com>
Message-ID: <5e5e1efe-5560-41ea-ba84-e474baa25985@email.android.com>

Change def to func? That's the worst idea I've heard yet. Def is already there; why break all existing code just for a word?

"Westley Mart?nez" <anikom15 at gmail.com> wrote:

>'def' is no more ambiguous than 'lambda', and is in fact more
>ambiguous,
>for 'def' doesn't lend itself to anything other than the word define,
>whilst 'lambda' can only mean lambda function.  Calling def explicit is
>silly.  It only makes sense because def arbitrarily means a function in
>Python (I'm proposing def become func or proc in Python 4000).
>
>To call lambda too 'computer sciencey' is equally ridiculous, for pro-
>gramming is a key spawn of computer science.  A programmer needs to
>have
>some knowledge of computer science to program, just like a physicist
>needs knowledge of calculus to understand mechanics.
>
>> -----Original Message-----
>> From: Python-Dev
>[mailto:python-dev-bounces+anikom15=gmail.com at python.org] On
>> Behalf Of Ben Gift
>> Sent: Thursday, September 19, 2013 1:54 PM
>> To: python-dev at python.org
>> Subject: [Python-Dev] Use an empty def as a lambda
>> 
>> I think the lambda keyword is difficult to understand for many
>people. It
>> would be more pythonic to use an empty def call instead.
>> 
>> 
>> For instance this:
>> 
>> 
>>     words.sort(key = lambda x: x[2])
>> 
>> 
>> could look like this:
>> 
>>     words.sort(key = def (x): x[2])
>> 
>> 
>> It's obvious and explicit that we're creating an unnamed, anonymous
>function
>> this way.
>
>
>_______________________________________________
>Python-Dev mailing list
>Python-Dev at python.org
>https://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe:
>https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com

-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130921/e254e254/attachment.html>

From rdmurray at bitdance.com  Sat Sep 21 21:00:39 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Sat, 21 Sep 2013 15:00:39 -0400
Subject: [Python-Dev] Use an empty def as a lambda
In-Reply-To: <5e5e1efe-5560-41ea-ba84-e474baa25985@email.android.com>
References: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com>
 <001e01ceb669$b6a1c740$23e555c0$@gmail.com>
 <5e5e1efe-5560-41ea-ba84-e474baa25985@email.android.com>
Message-ID: <20130921190040.6B5BB250955@webabinitio.net>

On Sat, 21 Sep 2013 12:55:20 -0500, Ryan <rymg19 at gmail.com> wrote:
> "Westley Mart??nez" <anikom15 at gmail.com> wrote:
> >'def' is no more ambiguous than 'lambda', and is in fact more
> >ambiguous,
> >for 'def' doesn't lend itself to anything other than the word define,
> >whilst 'lambda' can only mean lambda function.  Calling def explicit is
> >silly.  It only makes sense because def arbitrarily means a function in
> >Python (I'm proposing def become func or proc in Python 4000).
>
> Change def to func? That's the worst idea I've heard yet. Def is
> already there; why break all existing code just for a word?

Note that whatever happens with Python4, breaking of backward
compatibility (by anything other than the deletion of cruft) is very
unlikely to be part of it.

--David

From anikom15 at gmail.com  Sat Sep 21 21:22:14 2013
From: anikom15 at gmail.com (=?UTF-8?Q?Westley_Mart=C3=ADnez?=)
Date: Sat, 21 Sep 2013 12:22:14 -0700
Subject: [Python-Dev] Use an empty def as a lambda
In-Reply-To: <5e5e1efe-5560-41ea-ba84-e474baa25985@email.android.com>
References: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com>
 <001e01ceb669$b6a1c740$23e555c0$@gmail.com>
 <5e5e1efe-5560-41ea-ba84-e474baa25985@email.android.com>
Message-ID: <000f01ceb6ff$e1e9e180$a5bda480$@gmail.com>

When I say Python 4000, I don't actually mean Python 4.x, I mean the
next backwards-incompatible revision to Python (if it comes to pass).
My reasoning is that we use class to make classes, lambda to make
lambda functions, and def to make--well not defs--functions, which
doesn't really make sense to me.

Changing lambda to def would break compatibility as well and restrict
the grammar further.  In addition, it provides little foreseeable
benefit since the term def has no real meaning outside of Python,
whilst lambda has a clearly defined meaning.  Had the original C pro-
grammers chosen def in lieu of typedef or #def in lieu of #define, I
doubt that def would have been chosen as the keyword for functions in
Python.

> -----Original Message-----
> From: Ryan [mailto:rymg19 at gmail.com]
> Sent: Saturday, September 21, 2013 10:55 AM
> To: Westley Mart?nez; 'Ben Gift'; python-dev at python.org
> Subject: Re: [Python-Dev] Use an empty def as a lambda
> 
> Change def to func? That's the worst idea I've heard yet. Def is already
> there; why break all existing code just for a word?
> 
> 
> "Westley Mart?nez" <anikom15 at gmail.com> wrote:
> 
> 	'def' is no more ambiguous than 'lambda', and is in fact more ambiguous,
> 	for 'def' doesn't lend itself to anything other than the word define,
> 	whilst 'lambda' can only mean lambda function.  Calling def explicit is
> 	silly.  It only makes sense because def arbitrarily means a function in
> 	Python (I'm proposing def become func or proc in Python 4000).
> 
> 	To call lambda too 'computer sciencey' is equally ridiculous, for pro-
> 	gramming is a key spawn of computer science.  A programmer needs to have
> 	some knowledge of computer science to program, just like a physicist
> 	needs knowledge of calculus to understand mechanics.
> 
> 
> 		-----Original Message-----
> 		From: Python-Dev [mailto:python-dev-
> bounces+anikom15=gmail.com at python.org] On
> 		Behalf Of Ben Gift
> 		Sent: Thursday, September 19, 2013 1:54 PM
> 		To: python-dev at python.org
> 		Subject: [Python-Dev] Use an empty def as a lambda
> 
> 		I think the lambda keyword is difficult to understand for many
> people. It
> 		would be more pythonic to use an empty def call instead.
> 
> 
> 		For instance this:
> 
> 
> 		words.sort(key = lambda x: x[2])
> 
> 
> 		could look like this:
> 
> 		words.sort(key = def (x): x[2])
> 
> 
> 		It's obvious and explicit that we're creating an unnamed,
> anonymous function
> 		this way.
> 
> 
> 
> ________________________________
> 
> 
> 	Python-Dev mailing list
> 	Python-Dev at python.org
> 	https://mail.python.org/mailman/listinfo/python-dev
> 	Unsubscribe: https://mail.python.org/mailman/options/python-
> dev/rymg19%40gmail.com
> 
> 
> --
> Sent from my Android phone with K-9 Mail. Please excuse my brevity.


From ben+python at benfinney.id.au  Sat Sep 21 21:56:02 2013
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sun, 22 Sep 2013 05:56:02 +1000
Subject: [Python-Dev] Use an empty def as a lambda
References: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com>
 <001e01ceb669$b6a1c740$23e555c0$@gmail.com>
 <5e5e1efe-5560-41ea-ba84-e474baa25985@email.android.com>
 <000f01ceb6ff$e1e9e180$a5bda480$@gmail.com>
Message-ID: <7wk3iauey5.fsf@benfinney.id.au>

Westley Mart?nez <anikom15 at gmail.com> writes:

> My reasoning is that we use class to make classes, lambda to make
> lambda functions, and def to make--well not defs--functions, which
> doesn't really make sense to me.

Your reasoning is flawed. There is no such thing in Python as a ?lambda
function?.

Python has functions. It doesn't matter whether you use a ?lambda? or
?def? statement to create it, there's no resulting difference in the
type of the object. It is a function.

So: you make a class with a ?class? statement; you make a function using
either a ?def? statement or a ?lambda? expression. There is no third
type of object being discussed here.

-- 
 \         ?Faith may be defined briefly as an illogical belief in the |
  `\                  occurrence of the improbable.? ?Henry L. Mencken |
_o__)                                                                  |
Ben Finney


From tjreedy at udel.edu  Sat Sep 21 23:16:41 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 21 Sep 2013 17:16:41 -0400
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message
Message-ID: <l1l2bm$1bp$1@ger.gmane.org>

When an AttributeError is raised in a __del__ method, it is caught and 
ignored, except that it is not completely ignored but is replaced by a 
warning message sent to stderr. Example:
 >>> class C():
	def __del__(self): raise AttributeError
	
 >>> c=C()
 >>> del c
Exception AttributeError: AttributeError() in <bound method C.__del__ of 
<__main__.C object at 0x000000000351A198>> ignored

The AttributeError is ignored, I presume, because one may be expected if 
__new__ runs but __init__ does not, because of an exception. 
AttributeErrors can also be expected during exit cleanup.

The message seems not to be a Warning. So it seems that is cannot be 
suppressed. The OP of http://bugs.python.org/issue12085 encountered this 
message with an unnecessary use of subprocesses.Popen. (There is better 
code that avoids the problem.)  He considered the message a problem as 
it interfered with his output. He suggested the fix of changing 
'self.attr' to "getattr(self. 'attr', False)".

Viktor Stinner applied this fix. I believe it should be reverted for 
three reasons:

1. It results in TypeError on exit when gettattr is set to None before 
__delete__ is called.  http://bugs.python.org/issue19021.  Even is that 
is fixed with an alternate patch (creating a 'backup' class attribute), 
I think it would still be wrong.

2. If we do not agree with the OP that putting output on stderr is bad, 
then no fix is needed and there should be none.

3. If we do agree that the unconditional output is a problem, then there 
should be a general fix. It is, in a sense, just happenstance that the 
OP encountered this with Popen. Some possibilities:

a. Decide the the message is not needed; just silently ignore the 
__del__ AttributeError. I am not sure what its intended use is.

b. Make the message a Warning that can be blocked, perhaps a 
RuntimeWarning. Does this warning message predate the warnings module, 
new in 2.1?

c. If the __del__ AttributeError occurs while processing the __init__ 
error, just chain it normally. But this would, I presume interfere with 
catching the __init__ exception.

d. Add a missing attribute in cleanup message to end of the __init__ 
exception message.

-- 
Terry Jan Reedy


From rdmurray at bitdance.com  Sun Sep 22 00:15:24 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Sat, 21 Sep 2013 18:15:24 -0400
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <l1l2bm$1bp$1@ger.gmane.org>
References: <l1l2bm$1bp$1@ger.gmane.org>
Message-ID: <20130921221524.55D4225007F@webabinitio.net>

On Sat, 21 Sep 2013 17:16:41 -0400, Terry Reedy <tjreedy at udel.edu> wrote:
> When an AttributeError is raised in a __del__ method, it is caught and 
> ignored, except that it is not completely ignored but is replaced by a 
> warning message sent to stderr. Example:
>  >>> class C():
> 	def __del__(self): raise AttributeError
> 	
>  >>> c=C()
>  >>> del c
> Exception AttributeError: AttributeError() in <bound method C.__del__ of 
> <__main__.C object at 0x000000000351A198>> ignored

This is a replacement for a traceback.  In later Python versions, the
full traceback is printed.  In the general case it represents a bug in
the code that should be fixed.  Most such errors arise from the vagaries
of module finalization (such as your issue 19021), but not all of them
do: the rest represent real bugs in __del__ methods (which are executed
asynchronously in the general case).

So the question is, is the bug in the user code, or the stdlib code?
>From the issue, it sounds like it could be considered either (or most
likely, both).

--David

From tjreedy at udel.edu  Sun Sep 22 01:55:05 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 21 Sep 2013 19:55:05 -0400
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <20130921221524.55D4225007F@webabinitio.net>
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net>
Message-ID: <l1lbkl$k0g$1@ger.gmane.org>

On 9/21/2013 6:15 PM, R. David Murray wrote:
> On Sat, 21 Sep 2013 17:16:41 -0400, Terry Reedy <tjreedy at udel.edu> wrote:
>> When an AttributeError is raised in a __del__ method, it is caught and
>> ignored, except that it is not completely ignored but is replaced by a
>> warning message sent to stderr. Example:
>>   >>> class C():
>> 	def __del__(self): raise AttributeError
>> 	
>>   >>> c=C()
>>   >>> del c
>> Exception AttributeError: AttributeError() in <bound method C.__del__ of
>> <__main__.C object at 0x000000000351A198>> ignored
>
> This is a replacement for a traceback.  In later Python versions, the
> full traceback is printed.

The above is 3.3.2. In 3.4.0a2, the traceback of the ignored exception 
is indeed printed.

Exception ignored in: <bound method C.__del__ of <__main__.C object at 
0x00000000039946D8>>
Traceback (most recent call last):
   File "<pyshell#0>", line 2, in __del__
AttributeError:

> In the general case it represents a bug in
> the code that should be fixed.  Most such errors arise from the vagaries
> of module finalization (such as your issue 19021),

Lets call that a buglet ;-). Not really harmful, but annoying. Accepting 
that even such buglets 'should' be fixed in the stdllib, so that the 
message does not appear, is there any reason *not* to make it a 
RuntimeWarning so that users who care about clean output can filter it 
out while waiting for us to fix it?

This would be a separate issue from #12085.

 > but not all of them
> do: the rest represent real bugs in __del__ methods (which are executed
> asynchronously in the general case).

Which is why the message should be printed, so the developer can decide.

-- 
Terry Jan Reedy


From anikom15 at gmail.com  Sun Sep 22 03:49:39 2013
From: anikom15 at gmail.com (=?UTF-8?Q?Westley_Mart=C3=ADnez?=)
Date: Sat, 21 Sep 2013 18:49:39 -0700
Subject: [Python-Dev] Use an empty def as a lambda
In-Reply-To: <7wk3iauey5.fsf@benfinney.id.au>
References: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com>
 <001e01ceb669$b6a1c740$23e555c0$@gmail.com>
 <5e5e1efe-5560-41ea-ba84-e474baa25985@email.android.com>
 <000f01ceb6ff$e1e9e180$a5bda480$@gmail.com> <7wk3iauey5.fsf@benfinney.id.au>
Message-ID: <004701ceb736$014a0bf0$03de23d0$@gmail.com>

> -----Original Message-----
> From: Python-Dev [mailto:python-dev-bounces+anikom15=gmail.com at python.org] On
> Behalf Of Ben Finney
> Sent: Saturday, September 21, 2013 12:56 PM
> To: python-dev at python.org
> Subject: Re: [Python-Dev] Use an empty def as a lambda
> 
> Westley Mart?nez <anikom15 at gmail.com> writes:
> 
> > My reasoning is that we use class to make classes, lambda to make
> > lambda functions, and def to make--well not defs--functions, which
> > doesn't really make sense to me.
> 
> Your reasoning is flawed. There is no such thing in Python as a ?lambda
> function?.
> 
> Python has functions. It doesn't matter whether you use a ?lambda? or
> ?def? statement to create it, there's no resulting difference in the
> type of the object. It is a function.
> 
> So: you make a class with a ?class? statement; you make a function using
> either a ?def? statement or a ?lambda? expression. There is no third
> type of object being discussed here.

This is true.  The final object the Python creates is, to my knowledge,
the same regardless of whether it uses define or lambda.  But it's
irrelevant if it's the same or not, since one could argue that every-
thing is going to end up as a series of 0s and 1s.  In that sense a
function is nothing more than a glorified integer.  Lambda function is
just a term, like method or procedure instead function, although it is
certainly more specific in the sense that it is anonymous.  In my list I
was citing general concepts, not specific Python objects.

class is obvious, lambda is obvious, def is not as obvious, though
certainly more obvious than C-style declarations.  If lambda should
change I think anon or expr would be more suitable, or even func (so
long as def not become func).


From guido at python.org  Sun Sep 22 04:30:19 2013
From: guido at python.org (Guido van Rossum)
Date: Sat, 21 Sep 2013 19:30:19 -0700
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <l1lbkl$k0g$1@ger.gmane.org>
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net>
 <l1lbkl$k0g$1@ger.gmane.org>
Message-ID: <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>

Exceptions in __del__ point to bugs (sometimes in the stdlib) that should
be fixed, period. The only reason they do not result in exceptions that are
properly bubbled up and catchable is because __del__ is called from a
DECREF macro which has no return value. Also, IMO writing to stderr is fair
game -- reporting errors is what it is for.


As to making them warnings, I don't know that the warnings machinery is
easily adaptable for this purpose. Warnings can be suppressed but they can
also be turned into full exceptions; the latter doesn't really apply here
(see previous paragraph). But I would not object if someone found a way to
do this, though I'd prefer the default behavior to remain what it is in 3.4
(print a full traceback).


On Sat, Sep 21, 2013 at 4:55 PM, Terry Reedy <tjreedy at udel.edu> wrote:

> On 9/21/2013 6:15 PM, R. David Murray wrote:
>
>> On Sat, 21 Sep 2013 17:16:41 -0400, Terry Reedy <tjreedy at udel.edu> wrote:
>>
>>> When an AttributeError is raised in a __del__ method, it is caught and
>>> ignored, except that it is not completely ignored but is replaced by a
>>> warning message sent to stderr. Example:
>>>   >>> class C():
>>>         def __del__(self): raise AttributeError
>>>
>>>   >>> c=C()
>>>   >>> del c
>>> Exception AttributeError: AttributeError() in <bound method C.__del__ of
>>> <__main__.C object at 0x000000000351A198>> ignored
>>>
>>
>> This is a replacement for a traceback.  In later Python versions, the
>> full traceback is printed.
>>
>
> The above is 3.3.2. In 3.4.0a2, the traceback of the ignored exception is
> indeed printed.
>
> Exception ignored in: <bound method C.__del__ of <__main__.C object at
> 0x00000000039946D8>>
> Traceback (most recent call last):
>   File "<pyshell#0>", line 2, in __del__
> AttributeError:
>
>
>  In the general case it represents a bug in
>> the code that should be fixed.  Most such errors arise from the vagaries
>> of module finalization (such as your issue 19021),
>>
>
> Lets call that a buglet ;-). Not really harmful, but annoying. Accepting
> that even such buglets 'should' be fixed in the stdllib, so that the
> message does not appear, is there any reason *not* to make it a
> RuntimeWarning so that users who care about clean output can filter it out
> while waiting for us to fix it?
>
> This would be a separate issue from #12085.
>
>
> > but not all of them
>
>> do: the rest represent real bugs in __del__ methods (which are executed
>> asynchronously in the general case).
>>
>
> Which is why the message should be printed, so the developer can decide.
>
> --
> Terry Jan Reedy
>
>
> ______________________________**_________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/**mailman/listinfo/python-dev<https://mail.python.org/mailman/listinfo/python-dev>
> Unsubscribe: https://mail.python.org/**mailman/options/python-dev/**
> guido%40python.org<https://mail.python.org/mailman/options/python-dev/guido%40python.org>
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130921/7e693193/attachment-0001.html>

From steve at pearwood.info  Sun Sep 22 04:49:16 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 22 Sep 2013 12:49:16 +1000
Subject: [Python-Dev] Use an empty def as a lambda
In-Reply-To: <004701ceb736$014a0bf0$03de23d0$@gmail.com>
References: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com>
 <001e01ceb669$b6a1c740$23e555c0$@gmail.com>
 <5e5e1efe-5560-41ea-ba84-e474baa25985@email.android.com>
 <000f01ceb6ff$e1e9e180$a5bda480$@gmail.com> <7wk3iauey5.fsf@benfinney.id.au>
 <004701ceb736$014a0bf0$03de23d0$@gmail.com>
Message-ID: <20130922024916.GC19939@ando>

Guys, this thread is not Python-Dev territory. It should have gone to
Python-Ideas. I repeat what I posted two days ago:

    Proposals for changes to syntax and functionality are normally 
    expected to gather feedback on python-ideas before coming to 
    python-dev for final approval or rejection.

    https://mail.python.org/mailman/listinfo/python-ideas

    http://docs.python.org/devguide/communication.html


Please drop the discussion here.


-- 
Steven

From steve at pearwood.info  Sun Sep 22 05:13:04 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 22 Sep 2013 13:13:04 +1000
Subject: [Python-Dev] Best practice for documentation for std lib
Message-ID: <20130922031304.GE19939@ando>

Hi all,

I have a question about how I should manage documentation for the 
statistics module for Python 3.4. At the moment, I have extensive 
docstrings in the module itself. I don't believe anyone has flagged that 
as "too much information" in a code review, so I'm going to assume that 
large docstrings will be acceptable.

However, I have been asked to ensure that there is a separate 
statistics.rst file for documentation.

I don't want to have to maintain the documentation in two places, both 
in the .py module and in .rst file. Can anyone give me some pointers as 
to best practice in this situation? Is there a "How To Write Docs For 
The Standard Library" document somewhere? Perhaps I missed it, but my 
searches found nothing useful. I have read this:

http://docs.python.org/devguide/documenting.html

but it didn't shed any light on my situation.


Thanks in advance,



-- 
Steve

From benjamin at python.org  Sun Sep 22 05:25:10 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Sat, 21 Sep 2013 23:25:10 -0400
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <20130922031304.GE19939@ando>
References: <20130922031304.GE19939@ando>
Message-ID: <CAPZV6o8HnVfc-wFdPjHZtbh_s4X7MU-Kqi0QewUR2ar_n3Gogg@mail.gmail.com>

There's not really much to do but maintain them separately. Truncate
the docstrings if it makes life easier.

2013/9/21 Steven D'Aprano <steve at pearwood.info>:
> Hi all,
>
> I have a question about how I should manage documentation for the
> statistics module for Python 3.4. At the moment, I have extensive
> docstrings in the module itself. I don't believe anyone has flagged that
> as "too much information" in a code review, so I'm going to assume that
> large docstrings will be acceptable.
>
> However, I have been asked to ensure that there is a separate
> statistics.rst file for documentation.
>
> I don't want to have to maintain the documentation in two places, both
> in the .py module and in .rst file. Can anyone give me some pointers as
> to best practice in this situation? Is there a "How To Write Docs For
> The Standard Library" document somewhere? Perhaps I missed it, but my
> searches found nothing useful. I have read this:
>
> http://docs.python.org/devguide/documenting.html
>
> but it didn't shed any light on my situation.
>
>
> Thanks in advance,
>
>
>
> --
> Steve
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/benjamin%40python.org



-- 
Regards,
Benjamin

From victor.stinner at gmail.com  Sun Sep 22 09:58:42 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sun, 22 Sep 2013 09:58:42 +0200
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <20130922031304.GE19939@ando>
References: <20130922031304.GE19939@ando>
Message-ID: <CAMpsgwYaGPf=rw=istdUnmMUEDcvANzsON3i5G3z9vpqScopeA@mail.gmail.com>

I don't like having to browse the source code to read the documentation. I
prefer short docstring and longer reST documentation. In ReST, you get a
table of content and nice links to functions, modules, etc.

When I read doc, I like having two levels: tutorial listing most important
functions and parameters with examples, and a full API doc.

Using reST, you can also use tables, notes, footnotes, etc.

Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/3017cf02/attachment.html>

From solipsis at pitrou.net  Sun Sep 22 10:19:36 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 22 Sep 2013 10:19:36 +0200
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net>
 <l1lbkl$k0g$1@ger.gmane.org>
Message-ID: <20130922101936.53b9e68b@fsol>

On Sat, 21 Sep 2013 19:55:05 -0400
Terry Reedy <tjreedy at udel.edu> wrote:
> 
> > In the general case it represents a bug in
> > the code that should be fixed.  Most such errors arise from the vagaries
> > of module finalization (such as your issue 19021),
> 
> Lets call that a buglet ;-). Not really harmful, but annoying.

It's not a buglet. A buggy __del__ method is as much a bug as any other
buggy method. __del__ issues can lead to resources not being properly
finalized and released to the system.

> Accepting 
> that even such buglets 'should' be fixed in the stdllib, so that the 
> message does not appear, is there any reason *not* to make it a 
> RuntimeWarning so that users who care about clean output can filter it 
> out while waiting for us to fix it?

Yes, the reason is that these are real exceptions, not warnings. The
reason the exceptions are not propagated is that they can occur at any
point (finalization is pretty much asynchronous, it can occur from
unrelated places), so propagating them to whatever code happens to
execute at the time would be a huge instability factor.

Making them warnings means they could be silenced depending on the
current warning settings.

Regards

Antoine.



From solipsis at pitrou.net  Sun Sep 22 10:20:46 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 22 Sep 2013 10:20:46 +0200
Subject: [Python-Dev] Best practice for documentation for std lib
References: <20130922031304.GE19939@ando>
Message-ID: <20130922102046.764abea5@fsol>

On Sun, 22 Sep 2013 13:13:04 +1000
Steven D'Aprano <steve at pearwood.info> wrote:
> Hi all,
> 
> I have a question about how I should manage documentation for the 
> statistics module for Python 3.4. At the moment, I have extensive 
> docstrings in the module itself. I don't believe anyone has flagged that 
> as "too much information" in a code review, so I'm going to assume that 
> large docstrings will be acceptable.

Related question: do the extensive docstrings make "help(stats)"
painful to browse through?

Regards

Antoine.



From g.brandl at gmx.net  Sun Sep 22 10:48:33 2013
From: g.brandl at gmx.net (Georg Brandl)
Date: Sun, 22 Sep 2013 10:48:33 +0200
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <20130922101936.53b9e68b@fsol>
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org>
 <20130922101936.53b9e68b@fsol>
Message-ID: <l1marj$uta$1@ger.gmane.org>

On 09/22/2013 10:19 AM, Antoine Pitrou wrote:
> On Sat, 21 Sep 2013 19:55:05 -0400
> Terry Reedy <tjreedy at udel.edu> wrote:
>> 
>> > In the general case it represents a bug in
>> > the code that should be fixed.  Most such errors arise from the vagaries
>> > of module finalization (such as your issue 19021),
>> 
>> Lets call that a buglet ;-). Not really harmful, but annoying.
> 
> It's not a buglet. A buggy __del__ method is as much a bug as any other
> buggy method. __del__ issues can lead to resources not being properly
> finalized and released to the system.
> 
>> Accepting 
>> that even such buglets 'should' be fixed in the stdllib, so that the 
>> message does not appear, is there any reason *not* to make it a 
>> RuntimeWarning so that users who care about clean output can filter it 
>> out while waiting for us to fix it?
> 
> Yes, the reason is that these are real exceptions, not warnings. The
> reason the exceptions are not propagated is that they can occur at any
> point (finalization is pretty much asynchronous, it can occur from
> unrelated places), so propagating them to whatever code happens to
> execute at the time would be a huge instability factor.
> 
> Making them warnings means they could be silenced depending on the
> current warning settings.

Which is not too bad, right?  (Assuming it's not silent by default.)

However, I'm not sure invoking the whole complicated warning filtering
and emitting code is what we want to do in a destructor...

Georg


From g.brandl at gmx.net  Sun Sep 22 10:54:42 2013
From: g.brandl at gmx.net (Georg Brandl)
Date: Sun, 22 Sep 2013 10:54:42 +0200
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <20130922031304.GE19939@ando>
References: <20130922031304.GE19939@ando>
Message-ID: <l1mb75$3vk$1@ger.gmane.org>

On 09/22/2013 05:13 AM, Steven D'Aprano wrote:
> Hi all,
> 
> I have a question about how I should manage documentation for the 
> statistics module for Python 3.4. At the moment, I have extensive 
> docstrings in the module itself. I don't believe anyone has flagged that 
> as "too much information" in a code review, so I'm going to assume that 
> large docstrings will be acceptable.
> 
> However, I have been asked to ensure that there is a separate 
> statistics.rst file for documentation.
> 
> I don't want to have to maintain the documentation in two places, both 
> in the .py module and in .rst file. Can anyone give me some pointers as 
> to best practice in this situation? Is there a "How To Write Docs For 
> The Standard Library" document somewhere? Perhaps I missed it, but my 
> searches found nothing useful. I have read this:
> 
> http://docs.python.org/devguide/documenting.html
> 
> but it didn't shed any light on my situation.

This is the "How To Write etc." document.

If your docstrings really are complete, and marked up correctly, the new
module *could* be the first to use Sphinx autodoc for the stdlib documentation.
However, some core devs (including me) have stated opposition in the past.

The reason I myself have never really wanted to do this for the stdlib,
apart from most modules needing complete rewrites of their docstrings, is
that when documenting a standard module, you have to go through a few hoops
to make sure the build process actually imports the correct module to document,
and not the one of the Python version used to build the docs (which can
be different).

I don't really buy the argument "but then there is no complete documentation
set in the checkout" -- who reads that in preference to docs.python.org?
And if anybody does want plain-text docs, they are already available for build
and for download anyway (reST processed by Sphinx to remove non-plain markup).

cheers,
Georg


From solipsis at pitrou.net  Sun Sep 22 10:55:37 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 22 Sep 2013 10:55:37 +0200
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net>
 <l1lbkl$k0g$1@ger.gmane.org> <20130922101936.53b9e68b@fsol>
 <l1marj$uta$1@ger.gmane.org>
Message-ID: <20130922105537.1ec46f3f@fsol>

On Sun, 22 Sep 2013 10:48:33 +0200
Georg Brandl <g.brandl at gmx.net> wrote:

> On 09/22/2013 10:19 AM, Antoine Pitrou wrote:
> > On Sat, 21 Sep 2013 19:55:05 -0400
> > Terry Reedy <tjreedy at udel.edu> wrote:
> >> 
> >> > In the general case it represents a bug in
> >> > the code that should be fixed.  Most such errors arise from the vagaries
> >> > of module finalization (such as your issue 19021),
> >> 
> >> Lets call that a buglet ;-). Not really harmful, but annoying.
> > 
> > It's not a buglet. A buggy __del__ method is as much a bug as any other
> > buggy method. __del__ issues can lead to resources not being properly
> > finalized and released to the system.
> > 
> >> Accepting 
> >> that even such buglets 'should' be fixed in the stdllib, so that the 
> >> message does not appear, is there any reason *not* to make it a 
> >> RuntimeWarning so that users who care about clean output can filter it 
> >> out while waiting for us to fix it?
> > 
> > Yes, the reason is that these are real exceptions, not warnings. The
> > reason the exceptions are not propagated is that they can occur at any
> > point (finalization is pretty much asynchronous, it can occur from
> > unrelated places), so propagating them to whatever code happens to
> > execute at the time would be a huge instability factor.
> > 
> > Making them warnings means they could be silenced depending on the
> > current warning settings.
> 
> Which is not too bad, right?  (Assuming it's not silent by default.)

The default is to print a given warning message only once, which isn't
the normal semantics of exceptions.

> However, I'm not sure invoking the whole complicated warning filtering
> and emitting code is what we want to do in a destructor...

Good point, too :-)

Regards

Antoine.



From ncoghlan at gmail.com  Sun Sep 22 12:16:00 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 22 Sep 2013 20:16:00 +1000
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <l1mb75$3vk$1@ger.gmane.org>
References: <20130922031304.GE19939@ando>
	<l1mb75$3vk$1@ger.gmane.org>
Message-ID: <CADiSq7cs5bTkzRREL5FWiqXA7Z+a6MKm9oyk=vu9ku9ivsmVDw@mail.gmail.com>

On 22 September 2013 18:54, Georg Brandl <g.brandl at gmx.net> wrote:
> I don't really buy the argument "but then there is no complete documentation
> set in the checkout" -- who reads that in preference to docs.python.org?
> And if anybody does want plain-text docs, they are already available for build
> and for download anyway (reST processed by Sphinx to remove non-plain markup).

This argument only applies to projects which have source and docs in
separate checkouts, which doesn't apply to CPython :)

As others have noted, the preferred approach is indeed to maintain the
prose docs independently of the docstrings. The latter are often
trimmed to just be a quick reminder of the details of how the function
works for those that already know, while the prose docs go into more
depth and have more examples.

It's a bit of a pain, and we do occasionally get bug reports where the
docstrings get out of date, but it's the least bad of the currently
available options.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From python-dev at masklinn.net  Sun Sep 22 13:05:14 2013
From: python-dev at masklinn.net (Xavier Morel)
Date: Sun, 22 Sep 2013 13:05:14 +0200
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <CADiSq7cs5bTkzRREL5FWiqXA7Z+a6MKm9oyk=vu9ku9ivsmVDw@mail.gmail.com>
References: <20130922031304.GE19939@ando> <l1mb75$3vk$1@ger.gmane.org>
 <CADiSq7cs5bTkzRREL5FWiqXA7Z+a6MKm9oyk=vu9ku9ivsmVDw@mail.gmail.com>
Message-ID: <4DF8E630-28E5-4635-BFBD-1F03481FD9E3@masklinn.net>

On 2013-09-22, at 12:16 , Nick Coghlan wrote:
> 
> It's a bit of a pain, and we do occasionally get bug reports where the
> docstrings get out of date, but it's the least bad of the currently
> available options.

Is it really less bad than allowing limited fine-grained use of autodoc?
Not necessarily class-level and definitely not module-level, but
function- and method-level autodoc could allow avoiding duplication and
make it clearer that the prose and docstrings are getting out of sync.

From g.brandl at gmx.net  Sun Sep 22 13:18:00 2013
From: g.brandl at gmx.net (Georg Brandl)
Date: Sun, 22 Sep 2013 13:18:00 +0200
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <CADiSq7cs5bTkzRREL5FWiqXA7Z+a6MKm9oyk=vu9ku9ivsmVDw@mail.gmail.com>
References: <20130922031304.GE19939@ando> <l1mb75$3vk$1@ger.gmane.org>
 <CADiSq7cs5bTkzRREL5FWiqXA7Z+a6MKm9oyk=vu9ku9ivsmVDw@mail.gmail.com>
Message-ID: <l1mjjq$e7e$1@ger.gmane.org>

On 09/22/2013 12:16 PM, Nick Coghlan wrote:
> On 22 September 2013 18:54, Georg Brandl <g.brandl at gmx.net> wrote:
>> I don't really buy the argument "but then there is no complete documentation
>> set in the checkout" -- who reads that in preference to docs.python.org?
>> And if anybody does want plain-text docs, they are already available for build
>> and for download anyway (reST processed by Sphinx to remove non-plain markup).
> 
> This argument only applies to projects which have source and docs in
> separate checkouts, which doesn't apply to CPython :)
> 
> As others have noted, the preferred approach is indeed to maintain the
> prose docs independently of the docstrings. The latter are often
> trimmed to just be a quick reminder of the details of how the function
> works for those that already know, while the prose docs go into more
> depth and have more examples.

That is the convincing argument, yes.

It could be argued that autodoc allows adding content in the rst file itself,
such as

.. autofunction:: mean

   <docstring would go here in the output>

   Examples::

       >>> mean(life)
       "Try and be nice to people, avoid eating fat, read a good book ..."

But that means that when touching the docstring, one always has to look in
the rst file as well to check if the added content still makes sense.

cheers,
Georg


From steve at pearwood.info  Sun Sep 22 14:53:23 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 22 Sep 2013 22:53:23 +1000
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <20130922102046.764abea5@fsol>
References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol>
Message-ID: <20130922125323.GI19939@ando>

On Sun, Sep 22, 2013 at 10:20:46AM +0200, Antoine Pitrou wrote:
> On Sun, 22 Sep 2013 13:13:04 +1000
> Steven D'Aprano <steve at pearwood.info> wrote:
> > Hi all,
> > 
> > I have a question about how I should manage documentation for the 
> > statistics module for Python 3.4. At the moment, I have extensive 
> > docstrings in the module itself. I don't believe anyone has flagged that 
> > as "too much information" in a code review, so I'm going to assume that 
> > large docstrings will be acceptable.
> 
> Related question: do the extensive docstrings make "help(stats)"
> painful to browse through?

Not to me. I can page through help(statistics) with 18 presses of the 
space bar, versus 20 for random or 45 for unittest. (29 lines per page.)

Admittedly statistics has fewer functions/classes than random, but I 
find that fewer, larger pieces of documentation are easier to read than 
lots of tiny one-line mentions that don't actually tell you anything. 
E.g. from unittest:

     |  Methods defined here:
     |
     |  __init__(self, stream, descriptions, verbosity)
     |
     |  addError(self, test, err)
     |
     |  addExpectedFailure(self, test, err)
     |
     |  addFailure(self, test, err)
     |
     |  addSkip(self, test, reason)

and so on for nearly a page. unittest is also packed with many, many 
one-line methods listed as deprecated.

I admit I'm a bit of a stats and maths junkie, I read stats text books 
for fun. So perhaps I'm not the best person to judge how much 
information is too much information. Comments to the tracker please:

http://bugs.python.org/issue18606



-- 
Steven

From eliben at gmail.com  Sun Sep 22 14:54:57 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Sun, 22 Sep 2013 05:54:57 -0700
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <20130922031304.GE19939@ando>
References: <20130922031304.GE19939@ando>
Message-ID: <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>

On Sat, Sep 21, 2013 at 8:13 PM, Steven D'Aprano <steve at pearwood.info>wrote:

> Hi all,
>
> I have a question about how I should manage documentation for the
> statistics module for Python 3.4. At the moment, I have extensive
> docstrings in the module itself. I don't believe anyone has flagged that
> as "too much information" in a code review, so I'm going to assume that
> large docstrings will be acceptable.
>
> However, I have been asked to ensure that there is a separate
> statistics.rst file for documentation.
>
> I don't want to have to maintain the documentation in two places, both
> in the .py module and in .rst file. Can anyone give me some pointers as
> to best practice in this situation? Is there a "How To Write Docs For
> The Standard Library" document somewhere? Perhaps I missed it, but my
> searches found nothing useful. I have read this:
>
> http://docs.python.org/devguide/documenting.html
>
> but it didn't shed any light on my situation.
>

IMHO the right way to think about it is that the .rst files are by far the
more important documentation. Sometimes we forget that most Python
programmers are people who won't go into the source to read docstrings.
Moreover, the nice web layout, table of contents, index, and link-ability
of .rst is very important - I also prefer to read it as opposed to going
through docstrings. I only go to docstrings/code when I didn't find
something in the .rst docs, at this point also usually opening a bug to fix
that.

So whatever you do for statistics, full .rst docs must be there. It's the
docstrings that are "optional" here. The best solution may be auto
generating, for sure. But a module without .rst documented is unacceptable.

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/8d9cbea9/attachment.html>

From g.brandl at gmx.net  Sun Sep 22 15:06:40 2013
From: g.brandl at gmx.net (Georg Brandl)
Date: Sun, 22 Sep 2013 15:06:40 +0200
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
References: <20130922031304.GE19939@ando>
 <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
Message-ID: <l1mpvi$b1r$1@ger.gmane.org>

On 09/22/2013 02:54 PM, Eli Bendersky wrote:
> 
> 
> 
> On Sat, Sep 21, 2013 at 8:13 PM, Steven D'Aprano <steve at pearwood.info
> <mailto:steve at pearwood.info>> wrote:
> 
>     Hi all,
> 
>     I have a question about how I should manage documentation for the
>     statistics module for Python 3.4. At the moment, I have extensive
>     docstrings in the module itself. I don't believe anyone has flagged that
>     as "too much information" in a code review, so I'm going to assume that
>     large docstrings will be acceptable.
> 
>     However, I have been asked to ensure that there is a separate
>     statistics.rst file for documentation.
> 
>     I don't want to have to maintain the documentation in two places, both
>     in the .py module and in .rst file. Can anyone give me some pointers as
>     to best practice in this situation? Is there a "How To Write Docs For
>     The Standard Library" document somewhere? Perhaps I missed it, but my
>     searches found nothing useful. I have read this:
> 
>     http://docs.python.org/devguide/documenting.html
> 
>     but it didn't shed any light on my situation.
> 
> 
> IMHO the right way to think about it is that the .rst files are by far the more
> important documentation. Sometimes we forget that most Python programmers are
> people who won't go into the source to read docstrings. Moreover, the nice web
> layout, table of contents, index, and link-ability of .rst is very important - I
> also prefer to read it as opposed to going through docstrings.

Note -- using autodoc gives you this.

> I only go to docstrings/code when I didn't find something in the .rst docs, at 
> this point also usually opening a bug to fix that.
> So whatever you do for statistics, full .rst docs must be there.

I guess you don't mean .rst here; you mean .html (or .pdf, etc), whatever the
toolchain outputs.

cheers,
Georg


From p.f.moore at gmail.com  Sun Sep 22 15:46:02 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 22 Sep 2013 14:46:02 +0100
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
References: <20130922031304.GE19939@ando>
 <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
Message-ID: <CACac1F-xcmXNGuBzd9wqFteitqhhJG+RNi+rh4sM6DE4PbyjpQ@mail.gmail.com>

On 22 September 2013 13:54, Eli Bendersky <eliben at gmail.com> wrote:
> IMHO the right way to think about it is that the .rst files are by far the
> more important documentation. Sometimes we forget that most Python
> programmers are people who won't go into the source to read docstrings.

While I agree entirely with this, it's worth noting that with the
increasing popularity of IPython (even outside of the scientific
community), more and more people are becoming used to using docstrings
as a first means of finding out about a function's behaviour.

Paul

From eliben at gmail.com  Sun Sep 22 16:16:57 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Sun, 22 Sep 2013 07:16:57 -0700
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <l1mpvi$b1r$1@ger.gmane.org>
References: <20130922031304.GE19939@ando>
 <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
 <l1mpvi$b1r$1@ger.gmane.org>
Message-ID: <CAF-Rda_W_M3nJBbzMZxXv8oTYVmOTW5FuJxO996ESZBozS_53Q@mail.gmail.com>

On Sun, Sep 22, 2013 at 6:06 AM, Georg Brandl <g.brandl at gmx.net> wrote:

> On 09/22/2013 02:54 PM, Eli Bendersky wrote:
> >
> >
> >
> > On Sat, Sep 21, 2013 at 8:13 PM, Steven D'Aprano <steve at pearwood.info
> > <mailto:steve at pearwood.info>> wrote:
> >
> >     Hi all,
> >
> >     I have a question about how I should manage documentation for the
> >     statistics module for Python 3.4. At the moment, I have extensive
> >     docstrings in the module itself. I don't believe anyone has flagged
> that
> >     as "too much information" in a code review, so I'm going to assume
> that
> >     large docstrings will be acceptable.
> >
> >     However, I have been asked to ensure that there is a separate
> >     statistics.rst file for documentation.
> >
> >     I don't want to have to maintain the documentation in two places,
> both
> >     in the .py module and in .rst file. Can anyone give me some pointers
> as
> >     to best practice in this situation? Is there a "How To Write Docs For
> >     The Standard Library" document somewhere? Perhaps I missed it, but my
> >     searches found nothing useful. I have read this:
> >
> >     http://docs.python.org/devguide/documenting.html
> >
> >     but it didn't shed any light on my situation.
> >
> >
> > IMHO the right way to think about it is that the .rst files are by far
> the more
> > important documentation. Sometimes we forget that most Python
> programmers are
> > people who won't go into the source to read docstrings. Moreover, the
> nice web
> > layout, table of contents, index, and link-ability of .rst is very
> important - I
> > also prefer to read it as opposed to going through docstrings.
>
> Note -- using autodoc gives you this.
>
> > I only go to docstrings/code when I didn't find something in the .rst
> docs, at
> > this point also usually opening a bug to fix that.
> > So whatever you do for statistics, full .rst docs must be there.
>
> I guess you don't mean .rst here; you mean .html (or .pdf, etc), whatever
> the
> toolchain outputs.
>

Yes, by .rst docs I mean the final HTML output, not the .rst files
themselves :-)

 Paul Moore:
>While I agree entirely with this, it's worth noting that with the
increasing popularity of IPython (even outside of the scientific
community), more and more people are becoming used to using docstrings
as a first means of finding out about a function's behaviour.
--

That's a good point. I would still posit that HTML documentation gets by
far the most use, but docstrings are definitely important too. One more
point in favor of either:

1. Maintaining both, as tiresome as it is (we try to do this, not always
successfully, for all stdlib modules).
2. autodoc

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/2791ac89/attachment.html>

From ncoghlan at gmail.com  Sun Sep 22 16:25:52 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 23 Sep 2013 00:25:52 +1000
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <CAF-Rda_W_M3nJBbzMZxXv8oTYVmOTW5FuJxO996ESZBozS_53Q@mail.gmail.com>
References: <20130922031304.GE19939@ando>
 <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
 <l1mpvi$b1r$1@ger.gmane.org>
 <CAF-Rda_W_M3nJBbzMZxXv8oTYVmOTW5FuJxO996ESZBozS_53Q@mail.gmail.com>
Message-ID: <CADiSq7eG_pg6WUHSUr_zjw4aUuv5fXE=4xXKhzsNjjFcdL7oEw@mail.gmail.com>

On 23 September 2013 00:16, Eli Bendersky <eliben at gmail.com> wrote:
> That's a good point. I would still posit that HTML documentation gets by far
> the most use, but docstrings are definitely important too. One more point in
> favor of either:
>
> 1. Maintaining both, as tiresome as it is (we try to do this, not always
> successfully, for all stdlib modules).
> 2. autodoc

FWIW, I've generally found *tactical* use of autodoc (i.e. function
and method level usage for cases where the docstrings and prose docs
*were* the same) to be quite effective. Then if there later proved to
value in splitting them for a given case, that's what I would do. It
isn't an all-or-nothing decision.

As Georg noted, we'd have to do some fancy footwork to make sure
autodoc didn't pick up the wrong module versions for the standard
library docs, though.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From brett at python.org  Sun Sep 22 16:34:12 2013
From: brett at python.org (Brett Cannon)
Date: Sun, 22 Sep 2013 10:34:12 -0400
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <20130922125323.GI19939@ando>
References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol>
 <20130922125323.GI19939@ando>
Message-ID: <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com>

On Sun, Sep 22, 2013 at 8:53 AM, Steven D'Aprano <steve at pearwood.info>wrote:

> On Sun, Sep 22, 2013 at 10:20:46AM +0200, Antoine Pitrou wrote:
> > On Sun, 22 Sep 2013 13:13:04 +1000
> > Steven D'Aprano <steve at pearwood.info> wrote:
> > > Hi all,
> > >
> > > I have a question about how I should manage documentation for the
> > > statistics module for Python 3.4. At the moment, I have extensive
> > > docstrings in the module itself. I don't believe anyone has flagged
> that
> > > as "too much information" in a code review, so I'm going to assume that
> > > large docstrings will be acceptable.
> >
> > Related question: do the extensive docstrings make "help(stats)"
> > painful to browse through?
>
> Not to me. I can page through help(statistics) with 18 presses of the
> space bar, versus 20 for random or 45 for unittest. (29 lines per page.)
>
> Admittedly statistics has fewer functions/classes than random, but I
> find that fewer, larger pieces of documentation are easier to read than
> lots of tiny one-line mentions that don't actually tell you anything.
> E.g. from unittest:
>
>      |  Methods defined here:
>      |
>      |  __init__(self, stream, descriptions, verbosity)
>      |
>      |  addError(self, test, err)
>      |
>      |  addExpectedFailure(self, test, err)
>      |
>      |  addFailure(self, test, err)
>      |
>      |  addSkip(self, test, reason)
>
> and so on for nearly a page. unittest is also packed with many, many
> one-line methods listed as deprecated.
>
> I admit I'm a bit of a stats and maths junkie, I read stats text books
> for fun. So perhaps I'm not the best person to judge how much
> information is too much information. Comments to the tracker please:
>
> http://bugs.python.org/issue18606


The rule of thumb I go by is the docstring should be enough to answer the
question "what args does this thing take and what does it do in general to
know it's the function I want and another one in the same module?" quickly
and succinctly; i.e. just enough so that help() reminds you about details
for a module you are already familiar with that might come up while at the
interpreter prompt. Everything else -- in-depth discussion of the
algorithms, extra examples, why you want to use this function, etc. -- all
go in the .rst docs.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/669ee320/attachment-0001.html>

From brewdog.org at masklinn.net  Sun Sep 22 09:05:14 2013
From: brewdog.org at masklinn.net (Xavier Morel)
Date: Sun, 22 Sep 2013 09:05:14 +0200
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <CAPZV6o8HnVfc-wFdPjHZtbh_s4X7MU-Kqi0QewUR2ar_n3Gogg@mail.gmail.com>
References: <20130922031304.GE19939@ando>
 <CAPZV6o8HnVfc-wFdPjHZtbh_s4X7MU-Kqi0QewUR2ar_n3Gogg@mail.gmail.com>
Message-ID: <E6A3A834-3AEF-4A7F-9A03-E3E5CA1A988F@masklinn.net>

> On 22 Sep 2013, at 05:25, Benjamin Peterson <benjamin at python.org> wrote:
> 
> There's not really much to do but maintain them separately. Truncate
> the docstrings if it makes life easier.

Autodoc could be enabled and allowed in a limited manner.

From barry at python.org  Sun Sep 22 17:49:33 2013
From: barry at python.org (Barry Warsaw)
Date: Sun, 22 Sep 2013 11:49:33 -0400
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com>
References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol>
 <20130922125323.GI19939@ando>
 <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com>
Message-ID: <20130922114933.7c8ecde7@anarchist>

On Sep 22, 2013, at 10:34 AM, Brett Cannon wrote:

>The rule of thumb I go by is the docstring should be enough to answer the
>question "what args does this thing take and what does it do in general to
>know it's the function I want and another one in the same module?" quickly
>and succinctly; i.e. just enough so that help() reminds you about details
>for a module you are already familiar with that might come up while at the
>interpreter prompt. Everything else -- in-depth discussion of the
>algorithms, extra examples, why you want to use this function, etc. -- all
>go in the .rst docs.

That's exactly my own rule of thumb too, so +1.

-Barry

From eliben at gmail.com  Sun Sep 22 18:37:15 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Sun, 22 Sep 2013 09:37:15 -0700
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <20130922114933.7c8ecde7@anarchist>
References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol>
 <20130922125323.GI19939@ando>
 <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com>
 <20130922114933.7c8ecde7@anarchist>
Message-ID: <CAF-Rda_NnM-9Nt8uhc5b3k9MQwY0qVthki=fZdnPWvufoiXm=Q@mail.gmail.com>

On Sun, Sep 22, 2013 at 8:49 AM, Barry Warsaw <barry at python.org> wrote:

> On Sep 22, 2013, at 10:34 AM, Brett Cannon wrote:
>
> >The rule of thumb I go by is the docstring should be enough to answer the
> >question "what args does this thing take and what does it do in general to
> >know it's the function I want and another one in the same module?" quickly
> >and succinctly; i.e. just enough so that help() reminds you about details
> >for a module you are already familiar with that might come up while at the
> >interpreter prompt. Everything else -- in-depth discussion of the
> >algorithms, extra examples, why you want to use this function, etc. -- all
> >go in the .rst docs.
>
> That's exactly my own rule of thumb too, so +1.
>

It makes a lot of sense to me too. That said, I don't see a reason why we
can't auto-generate the referenc-y stuff from docstrings into the .rst
automatically, i.e.:

module.rst:

Algorithms, bla bla bla, examples bla bla bla

<ref:module.py Foo.bar>  # << Auto generator pastes function signature and
docstring here

More algorithms, more bla bla, examples.

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/05930883/attachment.html>

From stephen at xemacs.org  Sun Sep 22 18:53:10 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Mon, 23 Sep 2013 01:53:10 +0900
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
References: <20130922031304.GE19939@ando>
 <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
Message-ID: <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp>

Eli Bendersky writes:

 > IMHO the right way to think about it is that the .rst files are by
 > far the more important documentation.  Sometimes we forget that
 > most Python programmers are people who won't go into the source

Why "source"?  The whole point of docstrings is that they are *not*
comments found only in the source, but available at run time. In fact,
programmers who also use environments like Lisp or R (not to forget
Idle) will reach for "help(mean)", and that works fine for Steven,
because he provides such nice docstrings.

Some people prefer to write separate manuals, and some modules
*should* be documented that way because their internal complexity or
whatever.  That's true, but I would hope authors who prefer "literate
programming" (or the poor man's lit prog that is writing only
docstrings) are encouraged to do so when appropriate.

Of course, like any other contribution, since that style is *not*
currently supported by python-dev, they'd be asked to step up and
support it themselves -- if a user reports the docs won't build, they
need to address that like they would a build bug in the code.

From guido at python.org  Sun Sep 22 19:07:28 2013
From: guido at python.org (Guido van Rossum)
Date: Sun, 22 Sep 2013 10:07:28 -0700
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <20130922031304.GE19939@ando>
 <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
 <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com>

On Sun, Sep 22, 2013 at 9:53 AM, Stephen J. Turnbull <stephen at xemacs.org>wrote:

> Eli Bendersky writes:
>
>  > IMHO the right way to think about it is that the .rst files are by
>  > far the more important documentation.  Sometimes we forget that
>  > most Python programmers are people who won't go into the source
>
> Why "source"?  The whole point of docstrings is that they are *not*
> comments found only in the source, but available at run time. In fact,
> programmers who also use environments like Lisp or R (not to forget
> Idle) will reach for "help(mean)", and that works fine for Steven,
> because he provides such nice docstrings.
>
> Some people prefer to write separate manuals, and some modules
> *should* be documented that way because their internal complexity or
> whatever.  That's true, but I would hope authors who prefer "literate
> programming" (or the poor man's lit prog that is writing only
> docstrings) are encouraged to do so when appropriate.
>
> Of course, like any other contribution, since that style is *not*
> currently supported by python-dev, they'd be asked to step up and
> support it themselves -- if a user reports the docs won't build, they
> need to address that like they would a build bug in the code.


Authors writing 3rd party packages can do what they want.

But for the stdlib it's been settled for ages: docstrings should be concise
(but not cryptic(*)), longer documentation go into the separately written
text for docs.python.org.

(*) This is too concise to my taste:
$ ls -?
ls: illegal option -- ?
usage: ls [-ABCFGHLOPRSTUWabcdefghiklmnopqrstuwx1] [file ...]
$


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/a257de78/attachment.html>

From eliben at gmail.com  Sun Sep 22 19:25:08 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Sun, 22 Sep 2013 10:25:08 -0700
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com>
References: <20130922031304.GE19939@ando>
 <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
 <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com>
Message-ID: <CAF-Rda-qJ3fZQTpVO_x31WhWOaf7Q4+0n7w-sFZgK8tXWLL0zw@mail.gmail.com>

On Sun, Sep 22, 2013 at 10:07 AM, Guido van Rossum <guido at python.org> wrote:

> On Sun, Sep 22, 2013 at 9:53 AM, Stephen J. Turnbull <stephen at xemacs.org>wrote:
>
>> Eli Bendersky writes:
>>
>>  > IMHO the right way to think about it is that the .rst files are by
>>  > far the more important documentation.  Sometimes we forget that
>>  > most Python programmers are people who won't go into the source
>>
>> Why "source"?  The whole point of docstrings is that they are *not*
>> comments found only in the source, but available at run time. In fact,
>> programmers who also use environments like Lisp or R (not to forget
>> Idle) will reach for "help(mean)", and that works fine for Steven,
>> because he provides such nice docstrings.
>>
>> Some people prefer to write separate manuals, and some modules
>> *should* be documented that way because their internal complexity or
>> whatever.  That's true, but I would hope authors who prefer "literate
>> programming" (or the poor man's lit prog that is writing only
>> docstrings) are encouraged to do so when appropriate.
>>
>> Of course, like any other contribution, since that style is *not*
>> currently supported by python-dev, they'd be asked to step up and
>> support it themselves -- if a user reports the docs won't build, they
>> need to address that like they would a build bug in the code.
>
>
> Authors writing 3rd party packages can do what they want.
>
> But for the stdlib it's been settled for ages: docstrings should be
> concise (but not cryptic(*)), longer documentation go into the separately
> written text for docs.python.org.
>

I think there's a general agreement in this thread that we don't intend to
change the status quo. Both .rst docs and docstrings are important. The
remaining question is - can we use some tool to generates parts of the
former from the latter and thus avoid duplication and rot?

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/047b7764/attachment.html>

From ethan at stoneleaf.us  Sun Sep 22 18:32:31 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 22 Sep 2013 09:32:31 -0700
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <20130922114933.7c8ecde7@anarchist>
References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol>
 <20130922125323.GI19939@ando>
 <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com>
 <20130922114933.7c8ecde7@anarchist>
Message-ID: <523F1B9F.90101@stoneleaf.us>

On 09/22/2013 08:49 AM, Barry Warsaw wrote:
> On Sep 22, 2013, at 10:34 AM, Brett Cannon wrote:
>
>> The rule of thumb I go by is the docstring should be enough to answer the
>> question "what args does this thing take and what does it do in general to
>> know it's the function I want and another one in the same module?" quickly
>> and succinctly; i.e. just enough so that help() reminds you about details
>> for a module you are already familiar with that might come up while at the
>> interpreter prompt. Everything else -- in-depth discussion of the
>> algorithms, extra examples, why you want to use this function, etc. -- all
>> go in the .rst docs.
>
> That's exactly my own rule of thumb too, so +1.

Another +1.  So it that three rules or three thumbs?  ;)

--
~Ethan~

From tjreedy at udel.edu  Sun Sep 22 19:35:05 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 22 Sep 2013 13:35:05 -0400
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
Message-ID: <l1n9o6$ap3$1@ger.gmane.org>

On 9/21/2013 10:30 PM, Guido van Rossum wrote:
> Exceptions in __del__ point to bugs (sometimes in the stdlib) that
> should be fixed, period. The only reason they do not result in
> exceptions that are properly bubbled up and catchable is because __del__
> is called from a DECREF macro which has no return value.

That is clear enough. What fooled me is the word 'ignored', in both the 
doc and message. How about 'skipped' (for technical reasons)?

> Also, IMO
> writing to stderr is fair game -- reporting errors is what it is for.

So developers who really want to control all screen output should 
redirect or capture it somehow.

> As to making them warnings, I don't know that the warnings machinery is
> easily adaptable for this purpose. Warnings can be suppressed but they
> can also be turned into full exceptions; the latter doesn't really apply
> here (see previous paragraph). But I would not object if someone found a
> way to do this, though I'd prefer the default behavior to remain what it
> is in 3.4 (print a full traceback).

Antoine and Georg think it a dubious idea, so I will not pursue it. 
Developers who encounter messages from the stdlib can report and wait 
for a fix.

-- 
Terry Jan Reedy


From tjreedy at udel.edu  Sun Sep 22 19:45:25 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 22 Sep 2013 13:45:25 -0400
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <CADiSq7eG_pg6WUHSUr_zjw4aUuv5fXE=4xXKhzsNjjFcdL7oEw@mail.gmail.com>
References: <20130922031304.GE19939@ando>
 <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
 <l1mpvi$b1r$1@ger.gmane.org>
 <CAF-Rda_W_M3nJBbzMZxXv8oTYVmOTW5FuJxO996ESZBozS_53Q@mail.gmail.com>
 <CADiSq7eG_pg6WUHSUr_zjw4aUuv5fXE=4xXKhzsNjjFcdL7oEw@mail.gmail.com>
Message-ID: <l1nabi$h23$1@ger.gmane.org>

On 9/22/2013 10:25 AM, Nick Coghlan wrote:

> As Georg noted, we'd have to do some fancy footwork to make sure
> autodoc didn't pick up the wrong module versions for the standard
> library docs, though.

Is that a one-time nuisance, a per-module nuisance, a per-autodoc-use 
nuisance, or a per-build nuisance?

-- 
Terry Jan Reedy


From anikom15 at gmail.com  Sun Sep 22 20:02:17 2013
From: anikom15 at gmail.com (=?UTF-8?Q?Westley_Mart=C3=ADnez?=)
Date: Sun, 22 Sep 2013 11:02:17 -0700
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com>
References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol>
 <20130922125323.GI19939@ando>
 <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com>
Message-ID: <003301ceb7bd$e1a59430$a4f0bc90$@gmail.com>

Since help() is usually the first thing I use to remind myself of an
API, it's imperative that the doc strings are up to date and make sense
to end users.

So, autogeneration of doc strings would be good for someone like me who
hardly uses the html docs.

> -----Original Message-----
> From: Python-Dev [mailto:python-dev-bounces+anikom15=gmail.com at python.org] On
> Behalf Of Brett Cannon
> Sent: Sunday, September 22, 2013 7:34 AM
> To: Steven D'Aprano
> Cc: python-dev
> Subject: Re: [Python-Dev] Best practice for documentation for std lib
> 
> 
> 
> 
> On Sun, Sep 22, 2013 at 8:53 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> 
> 
> 	On Sun, Sep 22, 2013 at 10:20:46AM +0200, Antoine Pitrou wrote:
> 	> On Sun, 22 Sep 2013 13:13:04 +1000
> 	> Steven D'Aprano <steve at pearwood.info> wrote:
> 	> > Hi all,
> 	> >
> 	> > I have a question about how I should manage documentation for the
> 	> > statistics module for Python 3.4. At the moment, I have extensive
> 	> > docstrings in the module itself. I don't believe anyone has flagged
> that
> 	> > as "too much information" in a code review, so I'm going to assume
> that
> 	> > large docstrings will be acceptable.
> 	>
> 	> Related question: do the extensive docstrings make "help(stats)"
> 	> painful to browse through?
> 
> 
> 	Not to me. I can page through help(statistics) with 18 presses of the
> 	space bar, versus 20 for random or 45 for unittest. (29 lines per page.)
> 
> 	Admittedly statistics has fewer functions/classes than random, but I
> 	find that fewer, larger pieces of documentation are easier to read than
> 	lots of tiny one-line mentions that don't actually tell you anything.
> 	E.g. from unittest:
> 
> 	     |  Methods defined here:
> 	     |
> 	     |  __init__(self, stream, descriptions, verbosity)
> 	     |
> 	     |  addError(self, test, err)
> 	     |
> 	     |  addExpectedFailure(self, test, err)
> 	     |
> 	     |  addFailure(self, test, err)
> 	     |
> 	     |  addSkip(self, test, reason)
> 
> 	and so on for nearly a page. unittest is also packed with many, many
> 	one-line methods listed as deprecated.
> 
> 	I admit I'm a bit of a stats and maths junkie, I read stats text books
> 	for fun. So perhaps I'm not the best person to judge how much
> 	information is too much information. Comments to the tracker please:
> 
> 	http://bugs.python.org/issue18606
> 
> 
> The rule of thumb I go by is the docstring should be enough to answer the
> question "what args does this thing take and what does it do in general to
> know it's the function I want and another one in the same module?" quickly and
> succinctly; i.e. just enough so that help() reminds you about details for a
> module you are already familiar with that might come up while at the
> interpreter prompt. Everything else -- in-depth discussion of the algorithms,
> extra examples, why you want to use this function, etc. -- all go in the .rst
> docs.


From guido at python.org  Sun Sep 22 20:34:56 2013
From: guido at python.org (Guido van Rossum)
Date: Sun, 22 Sep 2013 11:34:56 -0700
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <003301ceb7bd$e1a59430$a4f0bc90$@gmail.com>
References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol>
 <20130922125323.GI19939@ando>
 <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com>
 <003301ceb7bd$e1a59430$a4f0bc90$@gmail.com>
Message-ID: <CAP7+vJLKVNWEDT62U4pJ7FGARthhVqb74hnLWznBag353s-Obw@mail.gmail.com>

On Sun, Sep 22, 2013 at 11:02 AM, Westley Mart?nez <anikom15 at gmail.com>wrote:

> Since help() is usually the first thing I use to remind myself of an
> API, it's imperative that the doc strings are up to date and make sense
> to end users.
>
> So, autogeneration of doc strings would be good for someone like me who
> hardly uses the html docs.
>

You seem to misunderstand the use of "autogeneration". It refers to
generating the .rst docs from the docstrings in the source. And FWIW, I'm
against that practice.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/1425e7e9/attachment-0001.html>

From guido at python.org  Sun Sep 22 20:40:41 2013
From: guido at python.org (Guido van Rossum)
Date: Sun, 22 Sep 2013 11:40:41 -0700
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <CAF-Rda-qJ3fZQTpVO_x31WhWOaf7Q4+0n7w-sFZgK8tXWLL0zw@mail.gmail.com>
References: <20130922031304.GE19939@ando>
 <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
 <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com>
 <CAF-Rda-qJ3fZQTpVO_x31WhWOaf7Q4+0n7w-sFZgK8tXWLL0zw@mail.gmail.com>
Message-ID: <CAP7+vJLmLZOYmMeoKqoqrAGqVF=Z41iYBRiis5buAMyaON+_+A@mail.gmail.com>

On Sun, Sep 22, 2013 at 10:25 AM, Eli Bendersky <eliben at gmail.com> wrote:

>
> I think there's a general agreement in this thread that we don't intend to
> change the status quo. Both .rst docs and docstrings are important. The
> remaining question is - can we use some tool to generates parts of the
> former from the latter and thus avoid duplication and rot?
>

I don't think that duplication is much of an issue. Natural language
understanding is not at the level yet where you can generate a meaningful
summary from a longer text fully automatically (let alone vice versa :-) so
I think having to write both a concise docstring and a longer more detailed
description for the Doc tree is not a waste of effort at all.

As for rot, it's just as likely that rot occurs as a *result* of
autogeneration. Having to edit/patch the source code in order to improve
the documentation most likely adds an extra barrier towards improving the
docs.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/85f5f82b/attachment.html>

From guido at python.org  Sun Sep 22 20:41:29 2013
From: guido at python.org (Guido van Rossum)
Date: Sun, 22 Sep 2013 11:41:29 -0700
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <l1n9o6$ap3$1@ger.gmane.org>
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net>
 <l1lbkl$k0g$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
 <l1n9o6$ap3$1@ger.gmane.org>
Message-ID: <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>

On Sun, Sep 22, 2013 at 10:35 AM, Terry Reedy <tjreedy at udel.edu> wrote:

> On 9/21/2013 10:30 PM, Guido van Rossum wrote:
>
>> Exceptions in __del__ point to bugs (sometimes in the stdlib) that
>> should be fixed, period. The only reason they do not result in
>> exceptions that are properly bubbled up and catchable is because __del__
>> is called from a DECREF macro which has no return value.
>>
>
> That is clear enough. What fooled me is the word 'ignored', in both the
> doc and message. How about 'skipped' (for technical reasons)?


That's a good point, although I'm not sure 'skipped' is better. Maybe use a
more neutral verb like 'occurred'?

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/5efd7650/attachment.html>

From eliben at gmail.com  Sun Sep 22 20:44:43 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Sun, 22 Sep 2013 11:44:43 -0700
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <CAP7+vJLmLZOYmMeoKqoqrAGqVF=Z41iYBRiis5buAMyaON+_+A@mail.gmail.com>
References: <20130922031304.GE19939@ando>
 <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
 <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com>
 <CAF-Rda-qJ3fZQTpVO_x31WhWOaf7Q4+0n7w-sFZgK8tXWLL0zw@mail.gmail.com>
 <CAP7+vJLmLZOYmMeoKqoqrAGqVF=Z41iYBRiis5buAMyaON+_+A@mail.gmail.com>
Message-ID: <CAF-Rda8H2hi5V=HakbGsD2WzH0HQ6hLuxkQYZ3ZC+MzDfb1hVw@mail.gmail.com>

On Sun, Sep 22, 2013 at 11:40 AM, Guido van Rossum <guido at python.org> wrote:

> On Sun, Sep 22, 2013 at 10:25 AM, Eli Bendersky <eliben at gmail.com> wrote:
>
>>
>> I think there's a general agreement in this thread that we don't intend
>> to change the status quo. Both .rst docs and docstrings are important. The
>> remaining question is - can we use some tool to generates parts of the
>> former from the latter and thus avoid duplication and rot?
>>
>
> I don't think that duplication is much of an issue. Natural language
> understanding is not at the level yet where you can generate a meaningful
> summary from a longer text fully automatically (let alone vice versa :-) so
> I think having to write both a concise docstring and a longer more detailed
> description for the Doc tree is not a waste of effort at all.
>

I don't think the proposal is to generate summaries from a longer text.
AFAIU, the proposal is to embed parts of the concise docstring into the
more verbose .rst documentation.

I write .rst docs quite a lot, and as such I do notice the annoying amount
of duplication between docstrings and .rst docs. Without doubt, all the
free-flow text and examples in .rst have to be written manually and are
very important. But for dry method references, it's certainly interesting
to explore the idea of writing them once in the code and then having Sphinx
automatically insert the relevant parts into the .rst before generating
HTML from it. For end users (those who read the docs online) the result is
indistinguishable from what we have now. For us devs it means writing the
same text only once and maintaining it in a single place.

Think about the new statistics module as a guinea pig. Steven will have a
whole lot of copy-pasting to do :-)


>
> As for rot, it's just as likely that rot occurs as a *result* of
> autogeneration. Having to edit/patch the source code in order to improve
> the documentation most likely adds an extra barrier towards improving the
> docs.
>

This is a valid concern, but perhaps one that can be addressed separately?
(i.e. lowering that barrier of entry).

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/98d4d2df/attachment.html>

From anikom15 at gmail.com  Sun Sep 22 21:24:04 2013
From: anikom15 at gmail.com (=?iso-8859-1?Q?Westley_Mart=EDnez?=)
Date: Sun, 22 Sep 2013 12:24:04 -0700
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <CAP7+vJLKVNWEDT62U4pJ7FGARthhVqb74hnLWznBag353s-Obw@mail.gmail.com>
References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol>
 <20130922125323.GI19939@ando>
 <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com>
 <003301ceb7bd$e1a59430$a4f0bc90$@gmail.com>
 <CAP7+vJLKVNWEDT62U4pJ7FGARthhVqb74hnLWznBag353s-Obw@mail.gmail.com>
Message-ID: <005001ceb7c9$4dc308e0$e9491aa0$@gmail.com>

> From: gvanrossum at gmail.com [mailto:gvanrossum at gmail.com] On Behalf Of Guido
> van Rossum
> Sent: Sunday, September 22, 2013 11:35 AM
> 
> You seem to misunderstand the use of "autogeneration". It refers to generating
> the .rst docs from the docstrings in the source. And FWIW, I'm against that
> practice. 

Oh I see.  Well in that case, the docstrings can still become outdated,
and so then the .rst docs will be outdated, too.  It doesn't seem to
offer much benefit since you still have to keep both updated, plus you
have an extra tool that must be maintained.


From tjreedy at udel.edu  Sun Sep 22 22:37:08 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 22 Sep 2013 16:37:08 -0400
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
 <l1n9o6$ap3$1@ger.gmane.org>
 <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
Message-ID: <l1nkdh$lh7$1@ger.gmane.org>

On 9/22/2013 2:41 PM, Guido van Rossum wrote:
> On Sun, Sep 22, 2013 at 10:35 AM, Terry Reedy <tjreedy at udel.edu
> <mailto:tjreedy at udel.edu>> wrote:
>
>     On 9/21/2013 10:30 PM, Guido van Rossum wrote:
>
>         Exceptions in __del__ point to bugs (sometimes in the stdlib) that
>         should be fixed, period. The only reason they do not result in
>         exceptions that are properly bubbled up and catchable is because
>         __del__
>         is called from a DECREF macro which has no return value.
>
>
>     That is clear enough. What fooled me is the word 'ignored', in both
>     the doc and message. How about 'skipped' (for technical reasons)?
>
>
> That's a good point, although I'm not sure 'skipped' is better.

Only slightly ;-). The problem with both words is that they try to say 
two things. What happened, and what Python did about it.

> Maybe use a more neutral verb like 'occurred'?

"Exception occurred in ..." is even better at say what happened.

I think we should then add an explict statement as to what Python did, 
and hint at what the user should do, something like
"Although caught internally, it still indicates a problem."

Otherwise, when no other output follows, as in
...
 > del c
Exception ocurred in: <bound method C.__del__ ...
Traceback (most recent call last):
   File "<pyshell#0>", line 2, in __del__
AttributeError:
 >>>
It may not be completely obvious to a non-expert that the traceback is 
not a 'real' traceback from an exception that was allowed to propagate, 
and that it did not stop execution and cannot be caught.

-- 
Terry Jan Reedy


From python-dev at masklinn.net  Sun Sep 22 23:41:39 2013
From: python-dev at masklinn.net (Xavier Morel)
Date: Sun, 22 Sep 2013 23:41:39 +0200
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <005001ceb7c9$4dc308e0$e9491aa0$@gmail.com>
References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol>
 <20130922125323.GI19939@ando>
 <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com>
 <003301ceb7bd$e1a59430$a4f0bc90$@gmail.com>
 <CAP7+vJLKVNWEDT62U4pJ7FGARthhVqb74hnLWznBag353s-Obw@mail.gmail.com>
 <005001ceb7c9$4dc308e0$e9491aa0$@gmail.com>
Message-ID: <06E35F19-F893-415B-B127-77D1588A565A@masklinn.net>


On 2013-09-22, at 21:24 , Westley Mart?nez wrote:

>> From: gvanrossum at gmail.com [mailto:gvanrossum at gmail.com] On Behalf Of Guido
>> van Rossum
>> Sent: Sunday, September 22, 2013 11:35 AM
>> 
>> You seem to misunderstand the use of "autogeneration". It refers to generating
>> the .rst docs from the docstrings in the source. And FWIW, I'm against that
>> practice. 
> 
> Oh I see.  Well in that case, the docstrings can still become outdated,
> and so then the .rst docs will be outdated, too.

The points here are that there's a single source of truth (so we can't
have conflicting docstring and rst documentation), and documentation
becoming outdated can be noticed from both docstring and published
documentation.

>  It doesn't seem to
> offer much benefit since you still have to keep both updated, plus you
> have an extra tool that must be maintained.

There is no extra tool, autodoc is part of the standard Sphinx
distribution: http://sphinx-doc.org/ext/autodoc.html

From anikom15 at gmail.com  Sun Sep 22 23:49:56 2013
From: anikom15 at gmail.com (=?iso-8859-1?Q?Westley_Mart=EDnez?=)
Date: Sun, 22 Sep 2013 14:49:56 -0700
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <06E35F19-F893-415B-B127-77D1588A565A@masklinn.net>
References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol>
 <20130922125323.GI19939@ando>
 <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com>
 <003301ceb7bd$e1a59430$a4f0bc90$@gmail.com>
 <CAP7+vJLKVNWEDT62U4pJ7FGARthhVqb74hnLWznBag353s-Obw@mail.gmail.com>
 <005001ceb7c9$4dc308e0$e9491aa0$@gmail.com>
 <06E35F19-F893-415B-B127-77D1588A565A@masklinn.net>
Message-ID: <000001ceb7dd$aedfa1b0$0c9ee510$@gmail.com>

Well, I'm wholly confused now, so I'll leave this discussion to the
devs.

> -----Original Message-----
> From: Python-Dev [mailto:python-dev-bounces+anikom15=gmail.com at python.org] On
> Behalf Of Xavier Morel
> Sent: Sunday, September 22, 2013 2:42 PM
> To: python-dev
> Subject: Re: [Python-Dev] Best practice for documentation for std lib
> 
> 
> On 2013-09-22, at 21:24 , Westley Mart?nez wrote:
> 
> >> From: gvanrossum at gmail.com [mailto:gvanrossum at gmail.com] On Behalf Of Guido
> >> van Rossum
> >> Sent: Sunday, September 22, 2013 11:35 AM
> >>
> >> You seem to misunderstand the use of "autogeneration". It refers to
> generating
> >> the .rst docs from the docstrings in the source. And FWIW, I'm against that
> >> practice.
> >
> > Oh I see.  Well in that case, the docstrings can still become outdated,
> > and so then the .rst docs will be outdated, too.
> 
> The points here are that there's a single source of truth (so we can't
> have conflicting docstring and rst documentation), and documentation
> becoming outdated can be noticed from both docstring and published
> documentation.
> 
> >  It doesn't seem to
> > offer much benefit since you still have to keep both updated, plus you
> > have an extra tool that must be maintained.
> 
> There is no extra tool, autodoc is part of the standard Sphinx
> distribution: http://sphinx-doc.org/ext/autodoc.html
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-
> dev/anikom15%40gmail.com


From guido at python.org  Mon Sep 23 00:03:10 2013
From: guido at python.org (Guido van Rossum)
Date: Sun, 22 Sep 2013 15:03:10 -0700
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <CAF-Rda8H2hi5V=HakbGsD2WzH0HQ6hLuxkQYZ3ZC+MzDfb1hVw@mail.gmail.com>
References: <20130922031304.GE19939@ando>
 <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
 <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com>
 <CAF-Rda-qJ3fZQTpVO_x31WhWOaf7Q4+0n7w-sFZgK8tXWLL0zw@mail.gmail.com>
 <CAP7+vJLmLZOYmMeoKqoqrAGqVF=Z41iYBRiis5buAMyaON+_+A@mail.gmail.com>
 <CAF-Rda8H2hi5V=HakbGsD2WzH0HQ6hLuxkQYZ3ZC+MzDfb1hVw@mail.gmail.com>
Message-ID: <CAP7+vJ+oVnq07Lfh_R1nqaqtK-EnxDaqPeZVTQMQTzfcEdqagg@mail.gmail.com>

On Sun, Sep 22, 2013 at 11:44 AM, Eli Bendersky <eliben at gmail.com> wrote:

>
> On Sun, Sep 22, 2013 at 11:40 AM, Guido van Rossum <guido at python.org>wrote:
>
>> On Sun, Sep 22, 2013 at 10:25 AM, Eli Bendersky <eliben at gmail.com> wrote:
>>
>>>
>>> I think there's a general agreement in this thread that we don't intend
>>> to change the status quo. Both .rst docs and docstrings are important. The
>>> remaining question is - can we use some tool to generates parts of the
>>> former from the latter and thus avoid duplication and rot?
>>>
>>
>> I don't think that duplication is much of an issue. Natural language
>> understanding is not at the level yet where you can generate a meaningful
>> summary from a longer text fully automatically (let alone vice versa :-) so
>> I think having to write both a concise docstring and a longer more detailed
>> description for the Doc tree is not a waste of effort at all.
>>
>
> I don't think the proposal is to generate summaries from a longer text.
>

I know, I was being facetious. :-)


> AFAIU, the proposal is to embed parts of the concise docstring into the
> more verbose .rst documentation.
>

I still think that's a bad idea. Someone editing the docstring may
introduce a terminology change or some other style/grammar/flow change that
makes perfectly sense by itself but doesn't when taken in the context of
the larger .rst doc (e.g. it could introducing duplication or cause
dissonance between the two parts). Also, someone who wants to improve the
docs would have to figure out how to edit the source code if they wanted to
change some part of the docs.


> I write .rst docs quite a lot, and as such I do notice the annoying amount
> of duplication between docstrings and .rst docs. Without doubt, all the
> free-flow text and examples in .rst have to be written manually and are
> very important. But for dry method references, it's certainly interesting
> to explore the idea of writing them once in the code and then having Sphinx
> automatically insert the relevant parts into the .rst before generating
> HTML from it. For end users (those who read the docs online) the result is
> indistinguishable from what we have now. For us devs it means writing the
> same text only once and maintaining it in a single place.
>

You seem to have caught the DRY bug. But it doesn't always make sense to
factor things so that everything is said in exactly one place. (For an
example of abstraction gone wild, ironically, see docutils and sphinx. I
challenge you to find out the actual code that translates e.g.
:meth:`foobar` into <a href="(what goes here?)">foobar</a>. :-)


> Think about the new statistics module as a guinea pig. Steven will have a
> whole lot of copy-pasting to do :-)
>

The docstrings should probably be limited to the amount of text that's
currently devoted to each function in the PEP. What's currently in the
docstrings should go into the .rst docs.


> > As for rot, it's just as likely that rot occurs as a *result* of
> autogeneration.
>
> Having to edit/patch the source code in order to improve the
> > documentation most likely adds an extra barrier towards improving the
> docs.
>
> This is a valid concern, but perhaps one that can be addressed separately?
> (i.e. lowering that barrier of entry).
>

I can't see that there's any way to interpret the change you propose
(changing things so that in order to improve the HTML docs you might need
to edit both .rst and .py files) as lowering the barrier to entry. Please
give it up.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/5f9f495d/attachment-0001.html>

From guido at python.org  Mon Sep 23 01:19:21 2013
From: guido at python.org (Guido van Rossum)
Date: Sun, 22 Sep 2013 16:19:21 -0700
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <06E35F19-F893-415B-B127-77D1588A565A@masklinn.net>
References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol>
 <20130922125323.GI19939@ando>
 <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com>
 <003301ceb7bd$e1a59430$a4f0bc90$@gmail.com>
 <CAP7+vJLKVNWEDT62U4pJ7FGARthhVqb74hnLWznBag353s-Obw@mail.gmail.com>
 <005001ceb7c9$4dc308e0$e9491aa0$@gmail.com>
 <06E35F19-F893-415B-B127-77D1588A565A@masklinn.net>
Message-ID: <CAP7+vJLRn-HAhNwBGLOmdPZA7YBq5JCs=H19FVeard6Y8yqYNQ@mail.gmail.com>

On Sun, Sep 22, 2013 at 2:41 PM, Xavier Morel <python-dev at masklinn.net>wrote:

> The points here are that there's a single source of truth (so we can't
> have conflicting docstring and rst documentation), and documentation
> becoming outdated can be noticed from both docstring and published
> documentation.
>

Another case of DRY madness.

It seems too many programmers see documentation as unpleasant red tape they
want to cut through as quickly as possible, instead of an opportunity to
communicate with their *users*. To the contrary: users should be the most
important people in the world if you're writing code that's worth
documenting at all.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/e5b2ccc3/attachment.html>

From steve at pearwood.info  Mon Sep 23 02:07:44 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 23 Sep 2013 10:07:44 +1000
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
References: <20130922031304.GE19939@ando>
 <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
Message-ID: <20130923000744.GM19939@ando>

On Sun, Sep 22, 2013 at 05:54:57AM -0700, Eli Bendersky wrote:

> IMHO the right way to think about it is that the .rst files are by far the
> more important documentation. Sometimes we forget that most Python
> programmers are people who won't go into the source to read docstrings.

Docstrings are never more than one command away in the interactive 
interpreter:

help(some.function)


If you're going to the source to read docstrings, you're doing it wrong
:-)

I always go for interactive help first, and the web docs second. The 
only reason I go to the source is to read the source *code*, not the 
docstrings.



-- 
Steven

From guido at python.org  Mon Sep 23 02:30:43 2013
From: guido at python.org (Guido van Rossum)
Date: Sun, 22 Sep 2013 17:30:43 -0700
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <20130923000744.GM19939@ando>
References: <20130922031304.GE19939@ando>
 <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
 <20130923000744.GM19939@ando>
Message-ID: <CAP7+vJJAfVCT5TPnaPTZTGTFKWqpay0_41H1UpB8v3F8=G6W7w@mail.gmail.com>

On Sun, Sep 22, 2013 at 5:07 PM, Steven D'Aprano <steve at pearwood.info>wrote:

> If you're going to the source to read docstrings, you're doing it wrong
> :-)
>

Be that as it may, when I'm reading the source I'm grateful for docstrings.
*And* I'm grateful when they are short.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/cce93e8f/attachment.html>

From steve at pearwood.info  Mon Sep 23 02:31:42 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 23 Sep 2013 10:31:42 +1000
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com>
References: <20130922031304.GE19939@ando>
 <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
 <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com>
Message-ID: <20130923003142.GO19939@ando>

On Sun, Sep 22, 2013 at 10:07:28AM -0700, Guido van Rossum wrote:

> Authors writing 3rd party packages can do what they want.
> 
> But for the stdlib it's been settled for ages: docstrings should be concise
> (but not cryptic(*)), longer documentation go into the separately written
> text for docs.python.org.

It is the second half that I'm not sure about. How long is *too long*
for a doc string? My own preference is to err on the side of too much 
rather than too little, since I live by help() and only fall back on the 
web documentation when I really must.

Rather than keep talking in generalities, I'll come up with a first 
draft rst document over the next week or so, put it on the tracker, and 
wait for some concrete feedback on that.

What's the policy about linking to external content from the web docs?



-- 
Steven

From ethan at stoneleaf.us  Mon Sep 23 01:52:36 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 22 Sep 2013 16:52:36 -0700
Subject: [Python-Dev] Enum Eccentricities
Message-ID: <523F82C4.4060202@stoneleaf.us>

http://bugs.python.org/issue19011

So, as anyone who has worked with Enum is probably aware by now, Enum's are a strange duck -- you might even call them 
platypuses.

For example, Enum members are instances of the class, but they are defined inside the class structure, and new ones 
cannot be created afterwards; Enum classes also support iteration and containment checks.

What I'm looking for feedback on is the question is #19011: should an Enum member be considered a class level attribute?

On the one hand, they are defined inside the class, and they are accessed via dot notation (EnumClass.member).

On the other hand, inside the class they look like 3 and 5 and 'up' and 'east', and they don't live in the class dictionary.

Also, if we change Enum so that members do act more like class attributes, then things like Color.red.blue.green.blue 
will result in Color.blue, and that seems stranger to me than having class instances be available on the class without 
be full-fledged class-attributes.

Thoughts?  Opinions?  Pearls of wisdom?

--
~Ethan~

From ethan at stoneleaf.us  Mon Sep 23 03:16:58 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 22 Sep 2013 18:16:58 -0700
Subject: [Python-Dev] PEP 455: TransformDict
In-Reply-To: <CAP7+vJJzzCOLz0WCX23=aZFDP4-_OWum1weV_-KjZs3=2CMC5A@mail.gmail.com>
References: <20130913204058.518f43d3@fsol>
 <CAP7+vJJzzCOLz0WCX23=aZFDP4-_OWum1weV_-KjZs3=2CMC5A@mail.gmail.com>
Message-ID: <523F968A.6090907@stoneleaf.us>

On 09/14/2013 12:31 PM, Guido van Rossum wrote:
> On Fri, Sep 13, 2013 at 11:40 AM, Antoine Pitrou wrote:
>>
>> Following the python-dev discussion, I've written a PEP to recap the
>> proposal and the various arguments. It's inlined below, and it will
>> probably appear soon at http://www.python.org/dev/peps/pep-0455/, too.
>
> Thanks, Antoine!
>
> Raymond Hettinger has volunteered to be the PEP dictator (is that the word we use?) for this PEP.

Are we close to asking for pronouncement?  I haven't seen any chatter for a few days.

--
~Ethan~

From guido at python.org  Mon Sep 23 04:04:04 2013
From: guido at python.org (Guido van Rossum)
Date: Sun, 22 Sep 2013 19:04:04 -0700
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <20130923003142.GO19939@ando>
References: <20130922031304.GE19939@ando>
 <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
 <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com>
 <20130923003142.GO19939@ando>
Message-ID: <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com>

On Sun, Sep 22, 2013 at 5:31 PM, Steven D'Aprano <steve at pearwood.info>wrote:

> On Sun, Sep 22, 2013 at 10:07:28AM -0700, Guido van Rossum wrote:
>
> > Authors writing 3rd party packages can do what they want.
> >
> > But for the stdlib it's been settled for ages: docstrings should be
> concise
> > (but not cryptic(*)), longer documentation go into the separately written
> > text for docs.python.org.
>
> It is the second half that I'm not sure about. How long is *too long*
> for a doc string? My own preference is to err on the side of too much
> rather than too little, since I live by help() and only fall back on the
> web documentation when I really must.
>

I guess preferences differ. I like reading the source, and often I find
overly long docstrings distracting.

If I had the final word, I'd recommend using the current docstrings as the
.rst contents and replacing the docstrings with the 1-2 line function
descriptions from the PEP, e.g.:

            * median(data) -> median (middle value) of data, taking the
              average of the two middle values when there are an even
              number of values.

But omitting the signature proper, and replacing "->" with "Returns" or
"Returns the". E.g.

    def median(data):
        """Returns the median (middle value) of data, taking the
           average of the two middle values when there are an even
           number of values.
        """
        ...

I'd personally also rewrite them all so that the first line of the
docstring is a phrase that can stand by itself, as I describe in PEP 8, but
this is used only sporadically in the stdlib.


> Rather than keep talking in generalities, I'll come up with a first
> draft rst document over the next week or so, put it on the tracker, and
> wait for some concrete feedback on that.
>
> What's the policy about linking to external content from the web docs?
>

If it's for background information or an introduction to the theory that's
fine. If it's as a substitute for proper documentation of an API I'd frown
upon it. Someone in a spaceship on its way to Mars  with a copy of
docs.python.org should have no problems using the library, as long as they
are familiar with the basic theory such as might be explained on Wikipedia
(of which the spaceship would of course also have a copy :-).

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/79218312/attachment.html>

From tjreedy at udel.edu  Mon Sep 23 04:25:57 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 22 Sep 2013 22:25:57 -0400
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com>
References: <20130922031304.GE19939@ando>
 <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
 <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com>
 <20130923003142.GO19939@ando>
 <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com>
Message-ID: <l1o8ri$mjv$1@ger.gmane.org>

On 9/22/2013 10:04 PM, Guido van Rossum wrote:

> If I had the final word, I'd recommend using the current docstrings as
> the .rst contents and replacing the docstrings with the 1-2 line
> function descriptions from the PEP, e.g.:
>
>              * median(data) -> median (middle value) of data, taking the
>                average of the two middle values when there are an even
>                number of values.
>
> But omitting the signature proper, and replacing "->" with "Returns" or
> "Returns the". E.g.
>
>      def median(data):
>          """Returns the median (middle value) of data, taking the
>             average of the two middle values when there are an even
>             number of values.
>          """
>          ...
>
> I'd personally also rewrite them all so that the first line of the
> docstring is a phrase that can stand by itself, as I describe in PEP 8,
> but this is used only sporadically in the stdlib.

I am gradually changing Idle docstrings, though only Idle developers 
will ever see them. Writing a 60 char summary forces a clear 
understanding of the function. Applied to the above.

       def median(data):
           """Return the median (middle value) of data.

           Use the average of the two middle values when
           there are an even number of values.
           """
('Return' rather than 'Returns' is the current convention.)

-- 
Terry Jan Reedy


From guido at python.org  Mon Sep 23 04:37:49 2013
From: guido at python.org (Guido van Rossum)
Date: Sun, 22 Sep 2013 19:37:49 -0700
Subject: [Python-Dev] Enum Eccentricities
In-Reply-To: <523F82C4.4060202@stoneleaf.us>
References: <523F82C4.4060202@stoneleaf.us>
Message-ID: <CAP7+vJLRtx3nQyvxN68M8dAPoe4evKOVXyQ19WEgp5Cd8EHFTw@mail.gmail.com>

On Sun, Sep 22, 2013 at 4:52 PM, Ethan Furman <ethan at stoneleaf.us> wrote:

> http://bugs.python.org/**issue19011 <http://bugs.python.org/issue19011>
>
> So, as anyone who has worked with Enum is probably aware by now, Enum's
> are a strange duck -- you might even call them platypuses.
>
> For example, Enum members are instances of the class, but they are defined
> inside the class structure, and new ones cannot be created afterwards; Enum
> classes also support iteration and containment checks.
>
> What I'm looking for feedback on is the question is #19011: should an Enum
> member be considered a class level attribute?
>
> On the one hand, they are defined inside the class, and they are accessed
> via dot notation (EnumClass.member).
>
> On the other hand, inside the class they look like 3 and 5 and 'up' and
> 'east', and they don't live in the class dictionary.
>
> Also, if we change Enum so that members do act more like class attributes,
> then things like Color.red.blue.green.blue will result in Color.blue, and
> that seems stranger to me than having class instances be available on the
> class without be full-fledged class-attributes.
>
> Thoughts?  Opinions?  Pearls of wisdom?
>

I wouldn't lose much sleep over this. Classes can override __getattribute__
so that instance variables appear to exist even though they are not in the
instance's __dict__. It's the same for metaclasses.

As for attributes appearing different inside the class than when accessed
as an attribute, that's not unusual -- descriptors can legitimately do
that. It was more common in Python 2, where this applied to all unbound
methods, but even in Python 3 it applies to static and class methods.

I would draw the line at being able to access members as attributes of
other members. Making Color.red.blue be a spelling for Color.blue feels
like an abomination. If you can make dir(Color) return the strings 'red',
'blue' etc. in addition to other class attributes without making
Color.red.blue work, go for it. If you can't, that's fine too. Users should
(obviously) steer clear from relying on either behavior.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/80bf98a6/attachment.html>

From guido at python.org  Mon Sep 23 04:44:09 2013
From: guido at python.org (Guido van Rossum)
Date: Sun, 22 Sep 2013 19:44:09 -0700
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <l1o8ri$mjv$1@ger.gmane.org>
References: <20130922031304.GE19939@ando>
 <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
 <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com>
 <20130923003142.GO19939@ando>
 <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com>
 <l1o8ri$mjv$1@ger.gmane.org>
Message-ID: <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com>

On Sun, Sep 22, 2013 at 7:25 PM, Terry Reedy <tjreedy at udel.edu> wrote:

> On 9/22/2013 10:04 PM, Guido van Rossum wrote:
>
>  If I had the final word, I'd recommend using the current docstrings as
>> the .rst contents and replacing the docstrings with the 1-2 line
>> function descriptions from the PEP, e.g.:
>>
>>              * median(data) -> median (middle value) of data, taking the
>>                average of the two middle values when there are an even
>>                number of values.
>>
>> But omitting the signature proper, and replacing "->" with "Returns" or
>> "Returns the". E.g.
>>
>>      def median(data):
>>          """Returns the median (middle value) of data, taking the
>>             average of the two middle values when there are an even
>>             number of values.
>>          """
>>          ...
>>
>> I'd personally also rewrite them all so that the first line of the
>> docstring is a phrase that can stand by itself, as I describe in PEP 8,
>> but this is used only sporadically in the stdlib.
>>
>
> I am gradually changing Idle docstrings, though only Idle developers will
> ever see them. Writing a 60 char summary forces a clear understanding of
> the function. Applied to the above.
>
>       def median(data):
>           """Return the median (middle value) of data.
>
>           Use the average of the two middle values when
>
>           there are an even number of values.
>           """
>

Glad you like it. I still do, too, but I've given up hope to convince all
core developers to stick to this style. :-(


>  ('Return' rather than 'Returns' is the current convention.)
>

That's actually a religious argument which in the stdlib takes no strict
position -- a quick grep shows that both are used, although 'Return' is
more frequent by a 5-to-1 margin. IIRC in the Java world you *have* to use
'Returns', but I don't buy the argument from nit-picky grammarians that
leads to this rule. (It's something about the documentation not being a
command. But English is more flexible than that.)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/8eb6438d/attachment-0001.html>

From greg.ewing at canterbury.ac.nz  Mon Sep 23 02:53:38 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 23 Sep 2013 12:53:38 +1200
Subject: [Python-Dev] Enum Eccentricities
In-Reply-To: <523F82C4.4060202@stoneleaf.us>
References: <523F82C4.4060202@stoneleaf.us>
Message-ID: <523F9112.8050506@canterbury.ac.nz>

Ethan Furman wrote:
> Also, if we change Enum so that members do act more like class 
> attributes, then things like Color.red.blue.green.blue will result in 
> Color.blue,

I thought we already decided it was worth making that
not happen?

-- 
Greg

From ncoghlan at gmail.com  Mon Sep 23 05:24:56 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 23 Sep 2013 13:24:56 +1000
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <l1nkdh$lh7$1@ger.gmane.org>
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net>
 <l1lbkl$k0g$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
 <l1n9o6$ap3$1@ger.gmane.org>
 <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
 <l1nkdh$lh7$1@ger.gmane.org>
Message-ID: <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>

On 23 Sep 2013 06:38, "Terry Reedy" <tjreedy at udel.edu> wrote:
>
> On 9/22/2013 2:41 PM, Guido van Rossum wrote:
>>
>> On Sun, Sep 22, 2013 at 10:35 AM, Terry Reedy <tjreedy at udel.edu
>> <mailto:tjreedy at udel.edu>> wrote:
>>
>>     On 9/21/2013 10:30 PM, Guido van Rossum wrote:
>>
>>         Exceptions in __del__ point to bugs (sometimes in the stdlib)
that
>>         should be fixed, period. The only reason they do not result in
>>         exceptions that are properly bubbled up and catchable is because
>>         __del__
>>         is called from a DECREF macro which has no return value.
>>
>>
>>     That is clear enough. What fooled me is the word 'ignored', in both
>>     the doc and message. How about 'skipped' (for technical reasons)?
>>
>>
>> That's a good point, although I'm not sure 'skipped' is better.
>
>
> Only slightly ;-). The problem with both words is that they try to say
two things. What happened, and what Python did about it.
>
>
>> Maybe use a more neutral verb like 'occurred'?
>
>
> "Exception occurred in ..." is even better at say what happened.
>
> I think we should then add an explict statement as to what Python did,
and hint at what the user should do, something like
> "Although caught internally, it still indicates a problem."

Brevity is still a virtue. The relevant C API function is called
"PyErr_WriteUnraisable", so just starting the message as something like
"Unraisable exception suppressed in..." might work.

Cheers,
Nick.

>
> Otherwise, when no other output follows, as in
> ...
> > del c
> Exception ocurred in: <bound method C.__del__ ...
>
> Traceback (most recent call last):
>   File "<pyshell#0>", line 2, in __del__
> AttributeError:
> >>>
> It may not be completely obvious to a non-expert that the traceback is
not a 'real' traceback from an exception that was allowed to propagate, and
that it did not stop execution and cannot be caught.
>
>
> --
> Terry Jan Reedy
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130923/651f5019/attachment.html>

From z at etiol.net  Mon Sep 23 04:41:39 2013
From: z at etiol.net (Zero Piraeus)
Date: Sun, 22 Sep 2013 23:41:39 -0300
Subject: [Python-Dev] Enum Eccentricities
In-Reply-To: <523F82C4.4060202@stoneleaf.us>
References: <523F82C4.4060202@stoneleaf.us>
Message-ID: <20130923024139.GA26561@piedra>

:

On Sun, Sep 22, 2013 at 04:52:36PM -0700, Ethan Furman wrote:
> So, as anyone who has worked with Enum is probably aware by now,
> Enum's are a strange duck -- you might even call them platypuses.

Yes :-)

> What I'm looking for feedback on is the question is #19011: should an
> Enum member be considered a class level attribute?

I may be misunderstanding the use case given in the issue, but it seems
to me that having to use

    Color.red.__class__.blue

(what is being complained about in the issue), while not exactly pretty,
makes a lot more semantic sense than

    Color.red.blue

... which is just bizarre.

Enum members aren't class attributes, even if the way they're defined
makes them look as though they are. Allowing this is just asking for
more confusion on the part of anyone using them IMHO.

 -[]z.

-- 
Zero Piraeus: omnia omnibus
http://etiol.net/pubkey.asc

From guido at python.org  Mon Sep 23 06:29:13 2013
From: guido at python.org (Guido van Rossum)
Date: Sun, 22 Sep 2013 21:29:13 -0700
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net>
 <l1lbkl$k0g$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
 <l1n9o6$ap3$1@ger.gmane.org>
 <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
 <l1nkdh$lh7$1@ger.gmane.org>
 <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>
Message-ID: <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com>

On Sunday, September 22, 2013, Nick Coghlan wrote:
>
> Brevity is still a virtue. The relevant C API function is called
> "PyErr_WriteUnraisable", so just starting the message as something like
> "Unraisable exception suppressed in..." might work.
>

Somehow "unraisable" sounds too technical, and "suppressed" is hardly right
given that it is printed...

Let's leave well enough alone.

--Guido


-- 
--Guido van Rossum (on iPad)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/270f4e67/attachment.html>

From ethan at stoneleaf.us  Mon Sep 23 06:06:40 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 22 Sep 2013 21:06:40 -0700
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
 <l1n9o6$ap3$1@ger.gmane.org>
 <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
 <l1nkdh$lh7$1@ger.gmane.org>
 <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>
Message-ID: <523FBE50.4060404@stoneleaf.us>

On 09/22/2013 08:24 PM, Nick Coghlan wrote:
>
> On 23 Sep 2013 06:38, "Terry Reedy" <tjreedy at udel.edu <mailto:tjreedy at udel.edu>> wrote:
>>
>> On 9/22/2013 2:41 PM, Guido van Rossum wrote:
>>>
>>> On Sun, Sep 22, 2013 at 10:35 AM, Terry Reedy <tjreedy at udel.edu <mailto:tjreedy at udel.edu>
>>> <mailto:tjreedy at udel.edu <mailto:tjreedy at udel.edu>>> wrote:
>>>
>>>     On 9/21/2013 10:30 PM, Guido van Rossum wrote:
>>>
>>>         Exceptions in __del__ point to bugs (sometimes in the stdlib) that
>>>         should be fixed, period. The only reason they do not result in
>>>         exceptions that are properly bubbled up and catchable is because
>>>         __del__
>>>         is called from a DECREF macro which has no return value.
>>>
>>>
>>>     That is clear enough. What fooled me is the word 'ignored', in both
>>>     the doc and message. How about 'skipped' (for technical reasons)?
>>>
>>>
>>> That's a good point, although I'm not sure 'skipped' is better.
>>
>>
>> Only slightly ;-). The problem with both words is that they try to say two things. What happened, and what Python did about it.
>>
>>
>>> Maybe use a more neutral verb like 'occurred'?
>>
>>
>> "Exception occurred in ..." is even better at say what happened.
>>
>> I think we should then add an explict statement as to what Python did, and hint at what the user should do, something like
>> "Although caught internally, it still indicates a problem."
>
> Brevity is still a virtue. The relevant C API function is called "PyErr_WriteUnraisable", so just starting the message
> as something like "Unraisable exception suppressed in..." might work.

I like that!  +1

--
~Ethan~

From guido at python.org  Mon Sep 23 06:25:23 2013
From: guido at python.org (Guido van Rossum)
Date: Sun, 22 Sep 2013 21:25:23 -0700
Subject: [Python-Dev] Enum Eccentricities
In-Reply-To: <20130923024139.GA26561@piedra>
References: <523F82C4.4060202@stoneleaf.us>
	<20130923024139.GA26561@piedra>
Message-ID: <CAP7+vJJgbzS6jG-yzFKEUNiEHnTzH_7yotv9sCXh-M=hZ85vDg@mail.gmail.com>

On Sunday, September 22, 2013, Zero Piraeus wrote:

> I may be misunderstanding the use case given in the issue, but it seems
> to me that having to use
>
>     Color.red.__class__.blue
>
> (what is being complained about in the issue), while not exactly pretty,
> makes a lot more semantic sense than
>
>     Color.red.blue
>
> ... which is just bizarre.


Right.


> Enum members aren't class attributes, even if the way they're defined
> makes them look as though they are. Allowing this is just asking for
> more confusion on the part of anyone using them IMHO.
>

Depends on how you define "class sttribute".

--Guido


-- 
--Guido van Rossum (on iPad)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/2f06f791/attachment.html>

From v+python at g.nevcal.com  Mon Sep 23 07:28:05 2013
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Sun, 22 Sep 2013 22:28:05 -0700
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com>
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
 <l1n9o6$ap3$1@ger.gmane.org>
 <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
 <l1nkdh$lh7$1@ger.gmane.org>
 <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>
 <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com>
Message-ID: <523FD165.1000300@g.nevcal.com>

On 9/22/2013 9:29 PM, Guido van Rossum wrote:
> On Sunday, September 22, 2013, Nick Coghlan wrote:
>
>     Brevity is still a virtue. The relevant C API function is called
>     "PyErr_WriteUnraisable", so just starting the message as something
>     like "Unraisable exception suppressed in..." might work.
>
>
> Somehow "unraisable" sounds too technical, and "suppressed" is hardly 
> right given that it is printed...
>
> Let's leave well enough alone.

"printable-only exception in..."
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/04ed4af6/attachment.html>

From turnbull at sk.tsukuba.ac.jp  Mon Sep 23 07:34:54 2013
From: turnbull at sk.tsukuba.ac.jp (Stephen J. Turnbull)
Date: Mon, 23 Sep 2013 14:34:54 +0900
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com>
References: <20130922031304.GE19939@ando>
 <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
 <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com>
 <20130923003142.GO19939@ando>
 <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com>
Message-ID: <87d2o02j9d.fsf@uwakimon.sk.tsukuba.ac.jp>

Guido van Rossum writes:
 > On Sun, Sep 22, 2013 at 5:31 PM, Steven D'Aprano <steve at pearwood.info> wrote:

 >> My own preference is to err on the side of too much rather than
 >> too little, since I live by help() and only fall back on the web
 >> documentation when I really must.

 > I guess preferences differ.

Indeed.  Abstractly I agree with Steven: there are some modules such
as statistics where it would be nice for users to have long docstrings
that might even be the full source for the web docs if the module
author so prefers.

But given that Python has an existing policy of concise docstrings
(which indeed is useful when browsing code) and a separate manual, I
withdraw my support for changing practice.  (Sorry, Steven.)


From greg.ewing at canterbury.ac.nz  Mon Sep 23 08:11:27 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 23 Sep 2013 18:11:27 +1200
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute
	error	message
In-Reply-To: <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com>
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
 <l1n9o6$ap3$1@ger.gmane.org>
 <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
 <l1nkdh$lh7$1@ger.gmane.org>
 <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>
 <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com>
Message-ID: <523FDB8F.1080207@canterbury.ac.nz>

Guido van Rossum wrote:
> Somehow "unraisable" sounds too technical,

It's not even really accurate. It's been raised, it just
can't be propagated any further. But "unpropagatable
exception" would be a bit of a mouthful.

-- 
Greg


From larry at hastings.org  Mon Sep 23 09:57:01 2013
From: larry at hastings.org (Larry Hastings)
Date: Mon, 23 Sep 2013 08:57:01 +0100
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com>
References: <20130922031304.GE19939@ando>
 <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
 <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com>
 <20130923003142.GO19939@ando>
 <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com>
 <l1o8ri$mjv$1@ger.gmane.org>
 <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com>
Message-ID: <523FF44D.1000303@hastings.org>

On 09/23/2013 03:44 AM, Guido van Rossum wrote:
> On Sun, Sep 22, 2013 at 7:25 PM, Terry Reedy <tjreedy at udel.edu 
> <mailto:tjreedy at udel.edu>> wrote:
>
>     I am gradually changing Idle docstrings, though only Idle
>     developers will ever see them. Writing a 60 char summary forces a
>     clear understanding of the function.
>
>
> Glad you like it. I still do, too, but I've given up hope to convince 
> all core developers to stick to this style. :-(

... Argument Clinic to the rescue?  Since the last time this subject 
came up, Clinic has started enforcing a summary line in docstrings. I 
didn't realize it had to be 60 columns though, should I add that? Is the 
entire docstring supposed to be 60 columns max?

p.s. status update: AC is done enough to be worth considering checking 
in.  Now I have to finish the PEP and write the documentation.


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130923/772a7caa/attachment.html>

From hrvoje.niksic at avl.com  Mon Sep 23 10:03:12 2013
From: hrvoje.niksic at avl.com (Hrvoje Niksic)
Date: Mon, 23 Sep 2013 10:03:12 +0200
Subject: [Python-Dev] sys.intern should work on bytes
In-Reply-To: <CALeMXf7Qo94xYsP_8cufhp=OR0x4QyWu-gnzn=fq1k7NF2QEuw@mail.gmail.com>
References: <523C2F3C.7020605@jcea.es> <20130920141551.0ac2c769@pitrou.net>
 <523C4E91.4050905@jcea.es> <20130920154415.5e6cfa09@pitrou.net>
 <523C53A8.4070106@jcea.es>
 <CALeMXf7Qo94xYsP_8cufhp=OR0x4QyWu-gnzn=fq1k7NF2QEuw@mail.gmail.com>
Message-ID: <523FF5C0.7080701@avl.com>

On 09/20/2013 06:50 PM, PJ Eby wrote:
> On Fri, Sep 20, 2013 at 9:54 AM, Jesus Cea <jcea at jcea.es> wrote:
>> Why str/bytes doesn't support weakrefs, beside memory use?
>
> The typical use case for weakrefs is to break reference cycles,

Another typical use case, and the prime reason why languages without 
reference counting tend to introduce weak references, is managing object 
caches with automatic disposal of otherwise unused items. Such a cache 
is rarely necessary for primitive objects, so Python's choice to spare 
memory for weakrefs is quite justified.

However, if one wanted to implement their own sys.intern(), inability to 
refer to strings would become a problem. This is one reason why 
sys.intern() directly fiddles with reference counts instead of reusing 
the weakref machinery. (The other of course being that intern predates 
weakrefs by many years.)


From ncoghlan at gmail.com  Mon Sep 23 10:17:51 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 23 Sep 2013 18:17:51 +1000
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <523FDB8F.1080207@canterbury.ac.nz>
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net>
 <l1lbkl$k0g$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
 <l1n9o6$ap3$1@ger.gmane.org>
 <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
 <l1nkdh$lh7$1@ger.gmane.org>
 <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>
 <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com>
 <523FDB8F.1080207@canterbury.ac.nz>
Message-ID: <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com>

On 23 September 2013 16:11, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Guido van Rossum wrote:
>>
>> Somehow "unraisable" sounds too technical,
>
>
> It's not even really accurate. It's been raised, it just
> can't be propagated any further. But "unpropagatable
> exception" would be a bit of a mouthful.

I felt I needed a reminder of the context where this string gets
printed, so I created this toy example:

>>> class C:
...     def __del__(self):
...         raise RuntimeError("Going away now")
...
>>> x = C()
>>> del x
Exception ignored in: <bound method C.__del__ of <__main__.C object at
0x7f98b8b61538>>
Traceback (most recent call last):
  File "<stdin>", line 3, in __del__
RuntimeError: Going away now

Here's what I suggest changing that error to:

>>> del x
Unraisable exception suppressed when calling <bound method C.__del__
of <__main__.C object at 0x7f98b8b61538>>
Traceback (most recent call last):
  File "<stdin>", line 3, in __del__
RuntimeError: Going away now

First and foremost, "Unraisable exception suppressed" should be quite
search engine friendly. The only current hits for that are on archived
versions of this thread, whereas the existing "Exception ignored"
sends you down a rabbit hole of Stack Overflow answers related to
broken pipe handling. If we create a new FAQ entry for "What does
'Exception ignored' or 'Unraisable exception suppressed' mean?" when
making the change for 3.4, then that will definitely help 3.4 users
and may eventually help users of other versions as well :)

Secondly, it concisely explains exactly what is going on, so
developers that look it up once should be able to remember the meaning
in the future:

- the given exception was suppressed (*what* happened)
- it was suppressed because Python couldn't raise it any further
(*why* it happened)

I think it's OK for the terms to be a little technical and arcane
because problems relating to broken finalizers and failures during
interpreter startup and shutdown *are* a little arcane.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Mon Sep 23 10:45:00 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 23 Sep 2013 10:45:00 +0200
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net>
 <l1lbkl$k0g$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
 <l1n9o6$ap3$1@ger.gmane.org>
 <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
 <l1nkdh$lh7$1@ger.gmane.org>
 <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>
 <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com>
 <523FDB8F.1080207@canterbury.ac.nz>
 <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com>
Message-ID: <20130923104500.5645fdbb@pitrou.net>

Le Mon, 23 Sep 2013 18:17:51 +1000,
Nick Coghlan <ncoghlan at gmail.com> a ?crit :
> 
> Here's what I suggest changing that error to:
> 
> >>> del x
> Unraisable exception suppressed when calling <bound method C.__del__
> of <__main__.C object at 0x7f98b8b61538>>
> Traceback (most recent call last):
>   File "<stdin>", line 3, in __del__
> RuntimeError: Going away now

Why not simply "Exception automatically caught in <bound method
C.__del__> [...]" ?

Regards

Antoine.



From ncoghlan at gmail.com  Mon Sep 23 10:51:04 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 23 Sep 2013 18:51:04 +1000
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <20130923104500.5645fdbb@pitrou.net>
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net>
 <l1lbkl$k0g$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
 <l1n9o6$ap3$1@ger.gmane.org>
 <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
 <l1nkdh$lh7$1@ger.gmane.org>
 <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>
 <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com>
 <523FDB8F.1080207@canterbury.ac.nz>
 <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com>
 <20130923104500.5645fdbb@pitrou.net>
Message-ID: <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com>

On 23 September 2013 18:45, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Le Mon, 23 Sep 2013 18:17:51 +1000,
> Nick Coghlan <ncoghlan at gmail.com> a ?crit :
>>
>> Here's what I suggest changing that error to:
>>
>> >>> del x
>> Unraisable exception suppressed when calling <bound method C.__del__
>> of <__main__.C object at 0x7f98b8b61538>>
>> Traceback (most recent call last):
>>   File "<stdin>", line 3, in __del__
>> RuntimeError: Going away now
>
> Why not simply "Exception automatically caught in <bound method
> C.__del__> [...]" ?

It only answers the "what" (i.e. the exception was automatically
caught), without addressing the "why" (i.e. because there wasn't
anything else useful the interpreter could do with it)

"Suppressed" also says more than "caught" does, since "caught and
reraised the exception" makes sense, while "suppressed and reraised
the exception" contradicts itself.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Mon Sep 23 13:15:09 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 23 Sep 2013 21:15:09 +1000
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement?
Message-ID: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>

With the last round of updates, I believe PEP 453 is ready for
Martin's pronouncement.

HTML: http://www.python.org/dev/peps/pep-0453/
Major diffs: http://hg.python.org/peps/rev/b2993450b32a

Many of the updates are just clarifications in response to questions,
but the actual significant changes are:

- the proposed module name is now "ensurepip" (thanks Antoine for that
suggestion)
- distros are granted considerable latitude to ensure the distro pip
package is used with the system Python installation, so long as they
ensure virtual environments continue to work as expected
- the two open questions regarding uninstallation and tweaking the
directory layout on Windows are decided in favour of the approaches
that were already documented in the PEP (since we hadn't received any
objections to them in earlier discussions)

Regards,
Nick.

=====================
PEP: 453
Title: Explicit bootstrapping of pip in Python installations
Version: $Revision$
Last-Modified: $Date$
Author: Donald Stufft <donald at stufft.io>,
        Nick Coghlan <ncoghlan at gmail.com>
BDFL-Delegate: Martin von L?wis
Status: Draft
Type: Process
Content-Type: text/x-rst
Created: 10-Aug-2013
Post-History: 30-Aug-2013, 15-Sep-2013, 18-Sep-2013, 19-Sep-2013, 23-Sep-2013


Abstract
========

This PEP proposes that the `pip`_ package manager be made available by
default when installing CPython and when creating virtual environments
using the standard library's ``venv`` module via the ``pyvenv`` command line
utility).

To clearly demarcate development responsibilities, and to avoid
inadvertently downgrading ``pip`` when updating CPython, the proposed
mechanism to achieve this is to include an explicit `pip`_ bootstrapping
mechanism in the standard library that is invoked automatically by the
CPython installers provided on python.org.

The PEP also strongly recommends that CPython redistributors and other Python
implementations ensure that ``pip`` is available by default, or
at the very least, explicitly document the fact that it is not included.


Proposal
========

This PEP proposes the inclusion of an ``ensurepip`` bootstrapping module in
Python 3.4, as well as in the next maintenance releases of Python 3.3 and
2.7.

This PEP does *not* propose making pip (or any dependencies) directly
available as part of the standard library. Instead, pip will be a
bundled application provided along with CPython for the convenience
of Python users, but subject to its own development life cycle and able
to be upgraded independently of the core interpreter and standard library.


Rationale
=========

Currently, on systems without a platform package manager and repository,
installing a third-party Python package into a freshly installed Python
requires first identifying an appropriate package manager and then
installing it.

Even on systems that *do* have a platform package manager, it is unlikely to
include every package that is available on the Python Package Index, and
even when a desired third-party package is available, the correct name in
the platform package manager may not be clear.

This means that, to work effectively with the Python Package Index
ecosystem, users must know which package manager to install, where to get
it, and how to install it. The effect of this is that third-party Python
projects are currently required to choose from a variety of undesirable
alternatives:

* Assume the user already has a suitable cross-platform package manager
  installed.
* Duplicate the instructions and tell their users how to install the
  package manager.
* Completely forgo the use of dependencies to ease installation concerns
  for their users.

All of these available options have significant drawbacks.

If a project simply assumes a user already has the tooling then beginning
users may get a confusing error message when the installation command
doesn't work. Some operating systems may ease this pain by providing a
global hook that looks for commands that don't exist and suggest an OS
package they can install to make the command work, but that only works
on systems with platform package managers (such as major Linux
distributions). No such assistance is available for Windows and
Mac OS X users. The challenges of dealing with this problem are a
regular feature of feedback the core Python developers receive from
professional educators and others introducing new users to Python.

If a project chooses to duplicate the installation instructions and tell
their users how to install the package manager before telling them how to
install their own project then whenever these instructions need updates
they need updating by every project that has duplicated them. This is
particular problematic when there are multiple competing installation
tools available, and different projects recommend different tools.

This specific problem can be partially alleviated by strongly promoting
``pip`` as the default installer and recommending that other projects
reference `pip's own bootstrapping instructions
<http://www.pip-installer.org/en/latest/installing.html>`__ rather than
duplicating them. However the user experience created by this approach
still isn't good (especially on Windows, where downloading and running
the ``get-pip.py`` bootstrap script with the default OS configuration is
significantly more painful than downloading and running a binary executable
or installer). The situation becomes even more complicated when multiple
Python versions are involved (for example, parallel installations of
Python 2 and Python 3), since that makes it harder to create and maintain
good platform specific ``pip`` installers independently of the CPython
installers.

The projects that have decided to forgo dependencies altogether are forced
to either duplicate the efforts of other projects by inventing their own
solutions to problems or are required to simply include the other projects
in their own source trees. Both of these options present their own problems
either in duplicating maintenance work across the ecosystem or potentially
leaving users vulnerable to security issues because the included code or
duplicated efforts are not automatically updated when upstream releases a new
version.

By providing a cross-platform package manager by default it will be easier
for users trying to install these third-party packages as well as easier
for the people distributing them as they should now be able to safely assume
that most users will have the appropriate installation tools available.
This is expected to become more important in the future as the Wheel_
package format (deliberately) does not have a built in "installer" in the
form of ``setup.py`` so users wishing to install from a wheel file will want
an installer even in the simplest cases.

Reducing the burden of actually installing a third-party package should
also decrease the pressure to add every useful module to the standard
library. This will allow additions to the standard library to focus more
on why Python should have a particular tool out of the box, and why it
is reasonable for that package to adopt the standard library's 18-24 month
feature release cycle, instead of using the general difficulty of installing
third-party packages as justification for inclusion.

Providing a standard installation system also helps with bootstrapping
alternate build and installer systems, such as ``setuptools``,
``zc.buildout`` and the ``hashdist``/``conda`` combination that is aimed
specifically at the scientific community. So long as
``pip install <tool>`` works, then a standard Python-specific installer
provides a reasonably secure, cross platform mechanism to get access to
these utilities.


Why pip?
--------

``pip`` has been chosen as the preferred default installer, as it
addresses several design and user experience issues with its predecessor
``easy_install`` (these issues can't readily be fixed in ``easy_install``
itself due to backwards compatibility concerns). ``pip`` is also well suited
to working within the bounds of a single Python runtime installation
(including associated virtual environments), which is a desirable feature
for a tool bundled with CPython.

Other tools like ``zc.buildout`` and ``conda`` are more ambitious in their
aims (and hence substantially better than ``pip`` at handling external
binary dependencies), so it makes sense for the Python ecosystem to treat
them more like platform package managers to inter operate with rather than
as the default cross-platform installation tool. This relationship is
similar to that between ``pip`` and platform package management systems
like ``apt`` and ``yum`` (which are also designed to handle arbitrary
binary dependencies).


Explicit bootstrapping mechanism
================================

An additional module called ``ensurepip`` will be added to the standard
library whose purpose is to install pip and any of its dependencies into the
appropriate location (most commonly site-packages). It will expose a
callable named ``bootstrap()`` as well as offer direct execution via
``python -m ensurepip``.

The bootstrap will *not* contact PyPI, but instead rely on a private copy
of pip stored inside the standard library. Accordingly, only options
related to the installation location will be supported (``--user``,
``--root``, etc).

It is considered desirable that users be strongly encouraged to use the
latest available version of ``pip``, in order to take advantage of the
ongoing efforts to improve the security of the PyPI based ecosystem, as
well as benefiting from the efforts to improve the speed, reliability and
flexibility of that ecosystem.

In order to satisfy this goal of providing the most recent version of
``pip`` by default, the private copy of ``pip`` will be updated in CPython
maintenance releases, which should align well with the 6-month cycle used
for new ``pip`` releases.


Security considerations
-----------------------

The design in this PEP has been deliberately chosen to avoid making any
significant changes to the trust model of the CPython installers for end
users that do not subsequently make use of ``pip``.

The installers will contain all the components of a fully functioning
version of Python, including the ``pip`` installer. The installation
process will *not* require network access, and will *not* rely on
trusting the security of the network connection established between
``pip`` and the Python package index.

Only users that choose to use ``pip`` directly will need to pay
attention to any PyPI related security considerations.


Implementation strategy
-----------------------

To ensure there is no need for network access when installing Python or
creating virtual environments, the ``ensurepip`` module will, as an
implementation detail, include a complete private copy of pip and its
dependencies which will be used to extract pip and install it into the target
environment. It is important to stress that this private copy of pip is
*only* an implementation detail and it should *not* be relied on or
assumed to exist beyond the public capabilities exposed through the
``ensurepip`` module (and indirectly through ``venv``).

There is not yet a reference ``ensurepip`` implementation. The existing
``get-pip.py`` bootstrap script demonstrates an earlier variation of the
general concept, but the standard library version would take advantage of
the improved distribution capabilities offered by the CPython installers
to include private copies of ``pip`` and ``setuptools`` as wheel files
(rather than as embedded base64 encoded data), and would not try to
contact PyPI (instead installing directly from the private wheel files.

Rather than including separate code to handle the bootstrapping, the
``ensurepip`` module will manipulate sys.path appropriately to allow
the wheel files to be used to install themselves, either into the current
Python installation or into a virtual environment (as determined by the
options passed to the bootstrap command).

It is proposed that the implementation be carried out in five separate
steps (all steps after the first are independent of each other and can be
carried out in any order):

* the first step would add the ``ensurepip`` module and the private copies
  of the most recently released versions of pip and setuptools, and update
  the "Installing Python Modules" documentation. This change
  would be applied to Python 2.7, 3.3 and 3.4.
* the Windows installer would be updated to offer the new ``pip``
  installation option for Python 2.7.6, 3.3.3 and 3.4.0.
* the Mac OS X installer would be updated to offer the new ``pip``
  installation option for Python 2.7.6, 3.3.3 and 3.4.0.
* the ``venv`` module and ``pyvenv`` command would be updated to make use
  of ``ensurepip`` in Python 3.4+
* the PATH handling and ``sysconfig`` directory layout on Windows would be
  updated for Python 3.4+


Proposed CLI
------------

The proposed CLI is based on a subset of the existing ``pip install``
options::

    Usage:
      python -m ensurepip [options]

    General Options:
      -h, --help          Show help.
      -v, --verbose       Give more output. Option is additive, and
can be used up to 3 times.
      -V, --version       Show the pip version that would be extracted and exit.
      -q, --quiet         Give less output.

    Installation Options:
      -U, --upgrade       Upgrade pip and dependencies, even if
already installed
      --user              Install using the user scheme.
      --root <dir>        Install everything relative to this
alternate root directory.

In most cases, end users won't need to use this CLI directly, as ``pip``
should have been installed automatically when installing Python or when
creating a virtual environment.

Users that want to retrieve the latest version from PyPI, or otherwise
need more flexibility, should invoke the extracted ``pip`` appropriately.


Proposed module API
-------------------

The proposed ``ensurepip`` module API consists of the following two
functions::

    def version():
        """
        Returns a string specifying the bundled version of pip.
        """

    def bootstrap(root=None, upgrade=False, user=False, verbosity=0):
        """
        Bootstrap pip into the current Python installation (or the given root
        directory).
        """


Invocation from the CPython installers
--------------------------------------

The CPython Windows and Mac OS X installers will each gain a new option:

* Install pip (the default Python package management utility)?

This option will be checked by default.

If the option is checked, then the installer will invoke the following
command with the just installed Python::

    python -m ensurepip --upgrade

This ensures that, by default, installing or updating CPython will ensure
that the installed version of pip is at least as recent as the one included
with that version of CPython. If a newer version of pip has already been
installed then ``python -m ensurepip --upgrade`` will simply return without
doing anything.


Installing from source
----------------------

While the prebuilt binary installers will be updated to run
``python -m ensurepip`` by default, no such change will be made to the
``make install`` and ``make altinstall`` commands of the source
distribution.

``ensurepip`` itself (including the private copy of ``pip`` and its
dependencies) will still be installed normally (as it is a regular
part of the standard library), only the implicit installation of pip and
its dependencies will be skipped.

Keeping the pip bootstrapping as a separate step for ``make``-based
installations should minimize the changes CPython redistributors need to
make to their build processes. Avoiding the layer of indirection through
``make`` for the ``ensurepip`` invocation avoids any challenges
associated with determining where to install the extracted ``pip``.


Changes to virtual environments
-------------------------------

Python 3.3 included a standard library approach to virtual Python environments
through the ``venv`` module. Since its release it has become clear that very
few users have been willing to use this feature directly, in part due to the
lack of an installer present by default inside of the virtual environment.
They have instead opted to continue using the ``virtualenv`` package which
*does* include pip installed by default.

To make the ``venv`` more useful to users it will be modified to issue the
pip bootstrap by default inside of the new environment while creating it. This
will allow people the same convenience inside of the virtual environment as
this PEP provides outside of it as well as bringing the ``venv`` module closer
to feature parity with the external ``virtualenv`` package, making it a more
suitable replacement.

To handle cases where a user does not wish to have pip bootstrapped into
their virtual environment a ``--without-pip`` option will be
added.

The ``venv.EnvBuilder`` and ``venv.create`` APIs will be updated to accept
one new parameter: ``with_pip`` (defaulting to ``False``).

The new default for the module API is chosen for backwards compatibility
with the current behaviour (as it is assumed that most invocation of the
``venv`` module happens through third part tools that likely will not
want ``pip`` installed without explicitly requesting it), while the
default for the command line interface is chosen to try to ensure ``pip``
is available in most virtual environments without additional action on the
part of the end user.

This particular change will be made only for Python 3.4 and later versions.
The third-party ``virtualenv`` project will still be needed to obtain a
consistent cross-version experience in Python 3.3 and 2.7.


Documentation
-------------

The "Installing Python Modules" section of the standard library
documentation will be updated to recommend the use of the bootstrapped
`pip` installer. It will give a brief description of the most common
commands and options, but delegate to the externally maintained ``pip``
documentation for the full details.

The existing content of the module installation guide will be retained,
but under a new "Invoking distutils directly" subsection.


Bundling CA certificates with CPython
-------------------------------------

The ``ensurepip`` implementation will include the ``pip`` CA bundle along
with the rest of ``pip``. This means CPython effectively includes
a CA bundle that is used solely by ``pip`` after it has been extracted.

This is considered preferable to relying solely on the system
certificate stores, as it ensures that ``pip`` will behave the same
across all supported versions of Python, even those prior to Python 3.4
that cannot access the system certificate store on Windows.


Automatic installation of setuptools
------------------------------------

``pip`` currently depends on ``setuptools`` to handle metadata generation
during the build process, along with some other features. While work is
ongoing to reduce or eliminate this dependency, it is not clear if that
work will be complete for pip 1.5 (which is the version likely to be current
when Python 3.4.0 is released).

This PEP proposes that, if pip still requires it as a dependency,
``ensurepip`` will include a private copy of ``setuptools`` (in addition
to the private copy of ``ensurepip``). ``python -m ensurepip`` will then
install the private copy in addition to installing ``pip`` itself.

However, this behavior is officially considered an implementation
detail. Other projects which explicitly require ``setuptools`` must still
provide an appropriate dependency declaration, rather than assuming
``setuptools`` will always be installed alongside ``pip``.

Once pip is able to run ``pip install --upgrade pip`` without needing
``setuptools`` installed first, then the private copy of ``setuptools``
will be removed from ``ensurepip`` in subsequent CPython releases.


Updating the private copy of pip
--------------------------------

In order to keep up with evolutions in packaging as well as providing users
with as recent version a possible the ``ensurepip`` module will be
regularly updated to the latest versions of everything it bootstraps.

After each new ``pip`` release, and again during the preparation for any
release of Python (including feature releases), a script, provided as part
of this PEP, will be run to ensure the private copies stored in the CPython
source repository have been updated to the latest versions.


Updating the ensurepip module API and CLI
-----------------------------------------

Like ``venv`` and ``pyvenv``, the ``ensurepip`` module API and CLI
will be governed by the normal rules for the standard library: no
new features are permitted in maintenance releases.

However, the embedded components may be updated as noted above, so
the extracted ``pip`` may offer additional functionality in maintenance
releases.


Feature addition in maintenance releases
========================================

Adding a new module to the standard library in Python 2.7 and 3.3
maintenance releases breaks the usual policy of "no new features in
maintenance releases".

It is being proposed in this case as the current bootstrapping issues for
the third-party Python package ecosystem greatly affects the experience of
new users, especially on Python 2 where many Python 3 standard library
improvements are available as backports on PyPI, but are not included in
the Python 2 standard library.

By updating Python 2.7, 3.3 and 3.4 to easily bootstrap the PyPI ecosystem,
this change should aid the vast majority of current Python users, rather
than only those with the freedom to adopt Python 3.4 as soon as it is
released.


Uninstallation
==============

No changes are proposed to the uninstallation process by this PEP. The
bootstrapped pip will be installed the same way as any other pip
installed packages, and will be handled in the same way as any other
post-install additions to the Python environment.

At least on Windows, that means the bootstrapped files will be
left behind after uninstallation, since those files won't be associated
with the Python MSI installer.

While the case can be made for the CPython installers clearing out these
directories automatically, changing that behaviour is considered outside
the scope of this PEP.


Script Execution on Windows
===========================

While the Windows installer was updated in Python 3.3 to optionally
make ``python`` available on the PATH, no such change was made to
include the Scripts directory. Independently of this PEP, a proposal has
also been made to rename the ``Tools\Scripts`` subdirectory to ``bin`` in
order to improve consistency with the typical script installation directory
names on \*nix systems.

Accordingly, in addition to adding the option to extract and install ``pip``
during installation, this PEP proposes that the Windows installer (and
``sysconfig``) in Python 3.4 and later be updated to:

- install scripts to PythonXY\bin rather than PythonXY\Tools\Scripts
- add PythonXY\bin to the Windows PATH (in addition to PythonXY) when the
  PATH modification option is enabled during installation

For Python 2.7 and 3.3, it is proposed that the only change be the one
to bootstrap ``pip`` by default.

This means that, for Python 3.3, the most reliable way to invoke pip on
Windows (without tinkering manually with PATH) will actually be
``py -m pip`` (or ``py -3 -m pip`` to select the Python 3 version if both
Python 2 and 3 are installed) rather than simply calling ``pip``.

For Python 2.7 and 3.2, the most reliable mechanism will be to install the
standalone Python launcher for Windows and then use ``py -m pip`` as noted
above.

Adding the scripts directory to the system PATH would mean that ``pip``
works reliably in the "only one Python installation on the system PATH"
case, with ``py -m pip``, ``pipX``, or ``pipX.Y`` needed only to select a
non-default version in the parallel installation case (and outside a virtual
environment). This change should also make the ``pyvenv`` command substantially
easier to invoke on Windows, along with all scripts installed by ``pip``,
``easy_install`` and similar tools.

While the script invocations on recent versions of Python will run through
the Python launcher for Windows, this shouldn't cause any issues, as long
as the Python files in the Scripts directory correctly specify a Python version
in their shebang line or have an adjacent Windows executable (as
``easy_install`` and ``pip`` do).


Recommendations for Downstream Distributors
===========================================

A common source of Python installations are through downstream distributors
such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX
package managers [#homebrew]_, or Python-specific tools [#conda]_. In order
to provide a consistent, user-friendly experience to all users of Python
regardless of how they attained Python this PEP recommends and asks that
downstream distributors:

* Ensure that whenever Python is installed pip is also installed.

  * This may take the form of separate packages with dependencies on each
    other so that installing the Python package installs the pip package
    and installing the pip package installs the Python package.
  * Another reasonable way to implement this is to package pip separately but
    ensure that there is some sort of global hook that will recommend
    installing the separate pip package when a user executes ``pip`` without
    it being installed. Systems that choose this option should ensure that
    the ``pyvenv`` command still installs pip into the virtual environment
    by default, but may modify the ``ensurepip`` module in the system Python
    installation to redirect to the platform provided mechanism when
    installing ``pip`` globally.

* Do not remove the bundled copy of pip.

  * This is required for installation of pip into a virtual environment by the
    ``venv`` module.
  * This is similar to the existing ``virtualenv`` package for which many
    downstream distributors have already made exception to the common
    "debundling" policy.
  * This does mean that if ``pip`` needs to be updated due to a security
    issue, so does the private copy in the ``ensurepip`` bootstrap module
  * However, altering the private copy of pip to remove the embedded
    CA certificate bundle and rely on the system CA bundle instead is a
    reasonable change.

* Migrate build systems to utilize `pip`_ and `Wheel`_ instead of directly
  using ``setup.py``.

  * This will ensure that downstream packages can more easily utilize the
    new metadata formats which may not have a ``setup.py``.

* Ensure that all features of this PEP continue to work with any modifications
  made to the redistributed version of Python.

  * Checking the version of pip that will be bootstrapped using
    ``python -m ensurepip --version`` or ``ensurepip.version()``.
  * Installation of pip into a global or virtual python environment using
    ``python -m ensurepip`` or ``ensurepip.bootstrap()``.
  * ``pip install --upgrade pip`` in a global installation should not affect
    any already created virtual environments (but is permitted to affect
    future virtual environments, even though it will not do so when using
    the upstream version of ``ensurepip``).
  * ``pip install --upgrade pip`` in a virtual environment should not affect
    the global installation.

In the event that a Python redistributor chooses *not* to follow these
recommendations, we request that they explicitly document this fact and
provide their users with suitable guidance on translating upstream ``pip``
based installation instructions into something appropriate for the platform.

Other Python implementations are also encouraged to follow these guidelines
where applicable.


Policies & Governance
=====================

The maintainers of the bootstrapped software and the CPython core team will
work together in order to address the needs of both. The bootstrapped
software will still remain external to CPython and this PEP does not
include CPython subsuming the development responsibilities or design
decisions of the bootstrapped software. This PEP aims to decrease the
burden on end users wanting to use third-party packages and the
decisions inside it are pragmatic ones that represent the trust that the
Python community has already placed in the Python Packaging Authority as
the authors and maintainers of ``pip``, ``setuptools``, PyPI, ``virtualenv``
and other related projects.


Backwards Compatibility
-----------------------

The public API and CLI of the ``ensurepip`` module itself will fall under
the typical backwards compatibility policy of Python for its standard
library. The externally developed software that this PEP bundles does not.

Most importantly, this means that the bootstrapped version of pip may gain
new features in CPython maintenance releases, and pip continues to operate on
its own 6 month release cycle rather than CPython's 18-24 month cycle.


Security Releases
-----------------

Any security update that affects the ``ensurepip`` module will be shared
prior to release with the Python Security Response Team
(security at python.org). The PSRT will then decide if the reported issue
warrants a security release of CPython with an updated private copy of
``pip``.


Appendix: Rejected Proposals
============================


Automatically contacting PyPI when bootstrapping pip
----------------------------------------------------

Earlier versions of this PEP called the bootstrapping module ``getpip`` and
defaulted to downloading and installing ``pip`` from PyPI, with the private
copy used only as a fallback option or when explicitly requested.

This resulted in several complex edge cases, along with difficulties in
defining a clean API and CLI for the bootstrap module. It also significantly
altered the default trust model for the binary installers published on
python.org, as end users would need to explicitly *opt-out* of trusting
the security of the PyPI ecosystem (rather than opting in to it by
explicitly invoking ``pip`` following installation).

As a result, the PEP was simplified to the current design, where the
bootstrapping *always* uses the private copy of ``pip``. Contacting PyPI
is now always an explicit separate step, with direct access to the full
pip interface.


Implicit bootstrap
------------------

`PEP439`_, the predecessor for this PEP, proposes its own solution. Its
solution involves shipping a fake ``pip`` command that when executed would
implicitly bootstrap and install pip if it does not already exist. This has
been rejected because it is too "magical". It hides from the end user when
exactly the pip command will be installed or that it is being installed at
all. It also does not provide any recommendations or considerations towards
downstream packagers who wish to manage the globally installed pip through
the mechanisms typical for their system.

The implicit bootstrap mechanism also ran into possible permissions issues,
if a user inadvertently attempted to bootstrap pip without write access to
the appropriate installation directories.


Including pip directly in the standard library
----------------------------------------------

Similar to this PEP is the proposal of just including pip in the standard
library. This would ensure that Python always includes pip and fixes all of the
end user facing problems with not having pip present by default. This has been
rejected because we've learned, through the inclusion and history of
``distutils`` in the standard library, that losing the ability to update the
packaging tools independently can leave the tooling in a state of constant
limbo. Making it unable to ever reasonably evolve in a time frame that actually
affects users as any new features will not be available to the general
population for *years*.

Allowing the packaging tools to progress separately from the Python release
and adoption schedules allows the improvements to be used by *all* members
of the Python community and not just those able to live on the bleeding edge
of Python releases.

There have also been issues in the past with the "dual maintenance" problem
if a project continues to be maintained externally while *also* having a
fork maintained in the standard library. Since external maintenance of
``pip`` will always be needed to support earlier Python versions, the
proposed bootstrapping mechanism will becoming the explicit responsibility
of the CPython core developers (assisted by the pip developers), while
pip issues reported to the CPython tracker will be migrated to the pip
issue tracker. There will no doubt still be some user confusion over which
tracker to use, but hopefully less than has been seen historically when
including complete public copies of third-party projects in the standard
library.

The approach described in this PEP also avoids some technical issues
related to handling CPython maintenance updates when pip has been
independently updated to a more recent version. The proposed pip-based
bootstrapping mechanism handles that automatically, since pip and the
system installer never get into a fight about who owns the pip
installation (it is always managed through pip, either directly, or
indirectly via the ``ensurepip`` bootstrap module).

Finally, the separate bootstrapping step means it also easy to avoid
installing ``pip`` at all if end users so desire. This is often the case
if integrators are using system packages to handle installation of
components written in multiple languages using a common set of tools.


Defaulting to --user installation
---------------------------------

Some consideration was given to bootstrapping pip into the per-user
site-packages directory by default. However, this behavior would be
surprising (as it differs from the default behavior of pip itself)
and is also not currently considered reliable (there are some edge cases
which are not handled correctly when pip is installed into the user
site-packages directory rather than the system site-packages).


.. _Wheel: http://www.python.org/dev/peps/pep-0427/
.. _pip: http://www.pip-installer.org
.. _setuptools: https://pypi.python.org/pypi/setuptools
.. _PEP439: http://www.python.org/dev/peps/pep-0439/


References
==========

.. [1] Discussion thread 1 (distutils-sig)
   (https://mail.python.org/pipermail/distutils-sig/2013-August/022529.html)

.. [2] Discussion thread 2 (distutils-sig)
   (https://mail.python.org/pipermail/distutils-sig/2013-September/022702.html)

.. [3] Discussion thread 3 (python-dev)
   (https://mail.python.org/pipermail/python-dev/2013-September/128723.html)

.. [4] Discussion thread 4 (python-dev)
   (https://mail.python.org/pipermail/python-dev/2013-September/128780.html)

.. [#ubuntu] `Ubuntu <http://www.ubuntu.com/>`
.. [#debian] `Debian <http://www.debian.org>`
.. [#fedora] `Fedora <https://fedoraproject.org/>`
.. [#homebrew] `Homebrew  <http://brew.sh/>`
.. [#conda] `Conda <http://www.continuum.io/blog/conda>`

Copyright
=========

This document has been placed in the public domain.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From walter at livinglogic.de  Mon Sep 23 13:27:47 2013
From: walter at livinglogic.de (=?UTF-8?B?V2FsdGVyIETDtnJ3YWxk?=)
Date: Mon, 23 Sep 2013 13:27:47 +0200
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com>
References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol>
 <20130922125323.GI19939@ando>
 <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com>
Message-ID: <524025B3.7020305@livinglogic.de>

On 22.09.13 16:34, Brett Cannon wrote:

> The rule of thumb I go by is the docstring should be enough to answer
> the question "what args does this thing take and what does it do in
> general to know it's the function I want and another one in the same
> module?" quickly and succinctly; i.e. just enough so that help() reminds
> you about details for a module you are already familiar with that might
> come up while at the interpreter prompt. Everything else -- in-depth
> discussion of the algorithms, extra examples, why you want to use this
> function, etc. -- all go in the .rst docs.

It would be great if the docstring contained a link to the online 
documentation.

Servus,
    Walter


From fred at fdrake.net  Mon Sep 23 15:38:06 2013
From: fred at fdrake.net (Fred Drake)
Date: Mon, 23 Sep 2013 09:38:06 -0400
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <524025B3.7020305@livinglogic.de>
References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol>
 <20130922125323.GI19939@ando>
 <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com>
 <524025B3.7020305@livinglogic.de>
Message-ID: <CAFT4OTGV=xf24Sufd=C_Ej5Mqg7KcOHv0ABWRSD=+dg8HkARFg@mail.gmail.com>

On Mon, Sep 23, 2013 at 7:27 AM, Walter D?rwald <walter at livinglogic.de> wrote:
> It would be great if the docstring contained a link to the online
> documentation.

The docstring itself, or the presentation generated by help() ?


  -Fred

-- 
Fred L. Drake, Jr.    <fred at fdrake.net>
"A storm broke loose in my mind."  --Albert Einstein

From chris at kateandchris.net  Mon Sep 23 15:45:46 2013
From: chris at kateandchris.net (Chris Lambacher)
Date: Mon, 23 Sep 2013 09:45:46 -0400
Subject: [Python-Dev] Enum Eccentricities
In-Reply-To: <20130923024139.GA26561@piedra>
References: <523F82C4.4060202@stoneleaf.us>
	<20130923024139.GA26561@piedra>
Message-ID: <CAAXXHgKPNsXH1dgv2GeqhVC9+s1YaziCYXq0Aah-e=a3mSLAmg@mail.gmail.com>

On Sun, Sep 22, 2013 at 10:41 PM, Zero Piraeus <z at etiol.net> wrote:

> I may be misunderstanding the use case given in the issue,
>

To clarify the use case, since it is my bug, this is so that the enum
values are always available for comparison. The exact use case is in Django
templates where a value comes from the database. If you want to compare you
either have to use __class__ which I would say is a code smell, or you have
to provide the Enum class. The code we are using for this is open source
http://github.com/tindie/django-tidyenum. An example of how this will be
used in practice is:

    {% if object.state == object.state.completed %}
      some html
    {% endif %}


> but it seems
> to me that having to use
>
>     Color.red.__class__.blue
>
> (what is being complained about in the issue), while not exactly pretty,
> makes a lot more semantic sense than
>
>     Color.red.blue
>
> ... which is just bizarre.
>

Any more bizarre than any other class that has properties of it's own type?

      a = 0
      a.real.numerator.real.numerator

It is only weird because we are not used to seeing classes where the
properties are instances of the class. I am not a core developer, but I
have been programming in Python for more than 10 years, and I (and the
other similarly experienced developers I work with) found that not having
access to the class level properties was weird (yes I am aware that they
are not actually class level properties but everywhere else Enum works hard
to make it look like they are. See for instance the __dir__ method:
http://hg.python.org/cpython/file/ed011b0d7daf/Lib/enum.py#l242).

-Chris

-- 
Christopher Lambacher
chris at kateandchris.net
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130923/91305d45/attachment.html>

From walter at livinglogic.de  Mon Sep 23 15:51:27 2013
From: walter at livinglogic.de (=?UTF-8?B?V2FsdGVyIETDtnJ3YWxk?=)
Date: Mon, 23 Sep 2013 15:51:27 +0200
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <CAFT4OTGV=xf24Sufd=C_Ej5Mqg7KcOHv0ABWRSD=+dg8HkARFg@mail.gmail.com>
References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol>
 <20130922125323.GI19939@ando>
 <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com>
 <524025B3.7020305@livinglogic.de>
 <CAFT4OTGV=xf24Sufd=C_Ej5Mqg7KcOHv0ABWRSD=+dg8HkARFg@mail.gmail.com>
Message-ID: <5240475F.4030407@livinglogic.de>

On 23.09.13 15:38, Fred Drake wrote:

> On Mon, Sep 23, 2013 at 7:27 AM, Walter D?rwald <walter at livinglogic.de> wrote:
>> It would be great if the docstring contained a link to the online
>> documentation.
>
> The docstring itself, or the presentation generated by help() ?

The presentation generated by help(), or the output of IPython's foo? or 
foo?? syntax.

Servus,
    Walter


From solipsis at pitrou.net  Mon Sep 23 16:07:45 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 23 Sep 2013 16:07:45 +0200
Subject: [Python-Dev] Best practice for documentation for std lib
References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol>
 <20130922125323.GI19939@ando>
 <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com>
 <524025B3.7020305@livinglogic.de>
 <CAFT4OTGV=xf24Sufd=C_Ej5Mqg7KcOHv0ABWRSD=+dg8HkARFg@mail.gmail.com>
 <5240475F.4030407@livinglogic.de>
Message-ID: <20130923160745.12e73f1f@pitrou.net>

Le Mon, 23 Sep 2013 15:51:27 +0200,
Walter D?rwald <walter at livinglogic.de> a ?crit :
> On 23.09.13 15:38, Fred Drake wrote:
> 
> > On Mon, Sep 23, 2013 at 7:27 AM, Walter D?rwald
> > <walter at livinglogic.de> wrote:
> >> It would be great if the docstring contained a link to the online
> >> documentation.
> >
> > The docstring itself, or the presentation generated by help() ?
> 
> The presentation generated by help(), or the output of IPython's foo?
> or foo?? syntax.

Perhaps objects could gain some kind of optional __docurl__ attribute,
to avoid cluttering __doc__ with external links.
Not sure how that could be auto-generated, though (we certainly don't
want to maintain doc links manually).

Regards

Antoine.



From rdmurray at bitdance.com  Mon Sep 23 16:43:10 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Mon, 23 Sep 2013 10:43:10 -0400
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <CAP7+vJLRn-HAhNwBGLOmdPZA7YBq5JCs=H19FVeard6Y8yqYNQ@mail.gmail.com>
References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol>
 <20130922125323.GI19939@ando>
 <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com>
 <003301ceb7bd$e1a59430$a4f0bc90$@gmail.com>
 <CAP7+vJLKVNWEDT62U4pJ7FGARthhVqb74hnLWznBag353s-Obw@mail.gmail.com>
 <005001ceb7c9$4dc308e0$e9491aa0$@gmail.com>
 <06E35F19-F893-415B-B127-77D1588A565A@masklinn.net>
 <CAP7+vJLRn-HAhNwBGLOmdPZA7YBq5JCs=H19FVeard6Y8yqYNQ@mail.gmail.com>
Message-ID: <20130923144311.23751250959@webabinitio.net>

On Sun, 22 Sep 2013 16:19:21 -0700, Guido van Rossum <guido at python.org> wrote:
> On Sun, Sep 22, 2013 at 2:41 PM, Xavier Morel <python-dev at masklinn.net>wrote:
> 
> > The points here are that there's a single source of truth (so we can't
> > have conflicting docstring and rst documentation), and documentation
> > becoming outdated can be noticed from both docstring and published
> > documentation.

Another thing that hasn't been mentioned about docstrings vs rst docs,
is that even when the text is identical, it generally isn't.  By that I
mean the rst docs have ReST markup, but the docstrings don't.  So using
autodoc doesn't just mean adding autodoc to our doc building toolchain,
it *also* means adding support in pydoc for turning ReST into plain text.
And that still leaves the markup in the docstring, where it will be very
distracting while actually reading the source code.  Which, unlike other
commenters, I spend more time doing that than I do using help (and that
doesn't apply to just the stdlib for me).
 
> >
> Another case of DRY madness.
> 
> It seems too many programmers see documentation as unpleasant red tape they
> want to cut through as quickly as possible, instead of an opportunity to
> communicate with their *users*. To the contrary: users should be the most
> important people in the world if you're writing code that's worth
> documenting at all.

I won't pretend that I find having to edit two places when updating
fundamental documentation pleasant, but...

I find that as often as not I want to word things *differently* in
the docstring vs the rst docs for the same function.  Sometimes the
difference is subtle; most often it is an omission of detail along the
lines Guido suggests.  But when writing the rst version, it generally
isn't that I'm simply continuing on after the first paragraph; instead,
the organization of even that starting text is different, because
I'm aware that the reader has a different mindset (quick help vs
reference documentation) when reading the two texts.[*]

As with the question of NEWS items versus checkin messages, the intended
audience, or in this case the mindset of the intended audience, is
*different*, and so the text should very often be different as well[**].

--David

[*] I posit that this is even *more* true for those who say they only
use help and only fall back to the full docs when the docstrings
aren't enough.

[**] I also wonder if long function docstrings have a negative impact on
usability in IDEs that pop up windows with docstring information in them.

From guido at python.org  Mon Sep 23 16:53:00 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 23 Sep 2013 07:53:00 -0700
Subject: [Python-Dev] Enum Eccentricities
In-Reply-To: <CAAXXHgKPNsXH1dgv2GeqhVC9+s1YaziCYXq0Aah-e=a3mSLAmg@mail.gmail.com>
References: <523F82C4.4060202@stoneleaf.us> <20130923024139.GA26561@piedra>
 <CAAXXHgKPNsXH1dgv2GeqhVC9+s1YaziCYXq0Aah-e=a3mSLAmg@mail.gmail.com>
Message-ID: <CAP7+vJKisYvdOug-fmR_SZEET7icHGjEx2pFYYHXyZHrhjWhzw@mail.gmail.com>

On Mon, Sep 23, 2013 at 6:45 AM, Chris Lambacher <chris at kateandchris.net>wrote:

>
> On Sun, Sep 22, 2013 at 10:41 PM, Zero Piraeus <z at etiol.net> wrote:
>
>> I may be misunderstanding the use case given in the issue,
>>
>
> To clarify the use case, since it is my bug, this is so that the enum
> values are always available for comparison. The exact use case is in Django
> templates where a value comes from the database. If you want to compare you
> either have to use __class__ which I would say is a code smell, or you have
> to provide the Enum class. The code we are using for this is open source
> http://github.com/tindie/django-tidyenum. An example of how this will be
> used in practice is:
>
>     {% if object.state == object.state.completed %}
>       some html
>     {% endif %}
>

Now I see your use case. But I disagree that the best solution is to allow
accessing the enum values as attributes on object.state -- I would
recommend using __class__ to make it clear to the reader what's going on,
or to add the right Enum subclass to your template parameters. Tbe
expression you show in your example here will just baffle most readers who
haven't thought deeply about the issue.

(How would you compare a value that is a timedelta with a known timedelta?
You'd have to import the datetime module or use __class__, right?)


>  but it seems
>> to me that having to use
>>
>>     Color.red.__class__.blue
>>
>> (what is being complained about in the issue), while not exactly pretty,
>> makes a lot more semantic sense than
>>
>>     Color.red.blue
>>
>> ... which is just bizarre.
>>
>
> Any more bizarre than any other class that has properties of it's own type?
>

Yes, because you rarely if ever see them accessed that way.


>       a = 0
>       a.real.numerator.real.numerator
>

But that's different! Each of the attributes here (.real, .numerator) is
defined as an instance attribute.


>
> It is only weird because we are not used to seeing classes where the
> properties are instances of the class. I am not a core developer, but I
> have been programming in Python for more than 10 years, and I (and the
> other similarly experienced developers I work with) found that not having
> access to the class level properties was weird (yes I am aware that they
> are not actually class level properties but everywhere else Enum works hard
> to make it look like they are. See for instance the __dir__ method:
> http://hg.python.org/cpython/file/ed011b0d7daf/Lib/enum.py#l242).
>

Sorry to burst your bubble, but there is no rule that because you can
access something on the class you should be able to access it on the
instance. Try asking an instance for its class's __mro__ or __bases__.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130923/f6dc14c3/attachment.html>

From guido at python.org  Mon Sep 23 16:56:00 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 23 Sep 2013 07:56:00 -0700
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <523FF44D.1000303@hastings.org>
References: <20130922031304.GE19939@ando>
 <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
 <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com>
 <20130923003142.GO19939@ando>
 <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com>
 <l1o8ri$mjv$1@ger.gmane.org>
 <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com>
 <523FF44D.1000303@hastings.org>
Message-ID: <CAP7+vJLcGYXfS5iSjmCqh4BKFmatjWoGdZi5DednagfcvcUf1w@mail.gmail.com>

On Mon, Sep 23, 2013 at 12:57 AM, Larry Hastings <larry at hastings.org> wrote:

>  On 09/23/2013 03:44 AM, Guido van Rossum wrote:
>
>  On Sun, Sep 22, 2013 at 7:25 PM, Terry Reedy <tjreedy at udel.edu> wrote:
>
>> I am gradually changing Idle docstrings, though only Idle developers will
>> ever see them. Writing a 60 char summary forces a clear understanding of
>> the function.
>
>
>  Glad you like it. I still do, too, but I've given up hope to convince
> all core developers to stick to this style. :-(
>
>
> ... Argument Clinic to the rescue?  Since the last time this subject came
> up, Clinic has started enforcing a summary line in docstrings.  I didn't
> realize it had to be 60 columns though, should I add that?  Is the entire
> docstring supposed to be 60 columns max?
>

I think 60 is just a guideline. In stdlib .py source code I want it not to
extend beyond the 79th column (see recent PEP 8 argument). For a typical
class, where the docstring is indented 4 spaces, that leaves 72 characters
for the summary line (including a final period!). For a method it's 68 due
to the extra indent.

FWIW I also prefer having the summary line on the same line as the opening
""":

def foo():
    """This is the summary.

    This is the rest.
    """


> p.s. status update: AC is done enough to be worth considering checking
> in.  Now I have to finish the PEP and write the documentation.
>

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130923/8acbe2fd/attachment.html>

From guido at python.org  Mon Sep 23 16:56:47 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 23 Sep 2013 07:56:47 -0700
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <524025B3.7020305@livinglogic.de>
References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol>
 <20130922125323.GI19939@ando>
 <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com>
 <524025B3.7020305@livinglogic.de>
Message-ID: <CAP7+vJLDo9KkdzzQfnJe_bTzunAP6+gBznY1Dx5pvxBqva0YjQ@mail.gmail.com>

On Mon, Sep 23, 2013 at 4:27 AM, Walter D?rwald <walter at livinglogic.de>wrote:

> It would be great if the docstring contained a link to the online
> documentation.
>

That would have to be a feature of help(), not hardcoded in each docstring.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130923/abcc8955/attachment.html>

From steve at pearwood.info  Mon Sep 23 17:16:23 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 24 Sep 2013 01:16:23 +1000
Subject: [Python-Dev] Enum Eccentricities
In-Reply-To: <CAP7+vJKisYvdOug-fmR_SZEET7icHGjEx2pFYYHXyZHrhjWhzw@mail.gmail.com>
References: <523F82C4.4060202@stoneleaf.us> <20130923024139.GA26561@piedra>
 <CAAXXHgKPNsXH1dgv2GeqhVC9+s1YaziCYXq0Aah-e=a3mSLAmg@mail.gmail.com>
 <CAP7+vJKisYvdOug-fmR_SZEET7icHGjEx2pFYYHXyZHrhjWhzw@mail.gmail.com>
Message-ID: <20130923151623.GB7989@ando>

On Mon, Sep 23, 2013 at 07:53:00AM -0700, Guido van Rossum wrote:

> there is no rule that because you can
> access something on the class you should be able to access it on the
> instance. Try asking an instance for its class's __mro__ or __bases__.

It might not be a rule, but it's certainly the norm. I reckon that class 
attributes that aren't accessible from the instance are significantly 
more surprising than Color.red.blue.

I know I'm in a minority here, but Color.red.blue seems obvious and 
straightforward to me. The fact that it doesn't work surprises me. Given 
that 

instance = Color.red
assert isinstance(instance, Color)  # well of course it is
assert hasattr(Color, "blue")

I would expect instance.blue to work, and I'm completely at a loss as to 
how Enum has managed to prevent it.



-- 
Steven

From solipsis at pitrou.net  Mon Sep 23 17:22:45 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 23 Sep 2013 17:22:45 +0200
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com>
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net>
 <l1lbkl$k0g$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
 <l1n9o6$ap3$1@ger.gmane.org>
 <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
 <l1nkdh$lh7$1@ger.gmane.org>
 <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>
 <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com>
 <523FDB8F.1080207@canterbury.ac.nz>
 <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com>
 <20130923104500.5645fdbb@pitrou.net>
 <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com>
Message-ID: <20130923172245.0cbaec81@fsol>

On Mon, 23 Sep 2013 18:51:04 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 23 September 2013 18:45, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > Le Mon, 23 Sep 2013 18:17:51 +1000,
> > Nick Coghlan <ncoghlan at gmail.com> a ?crit :
> >>
> >> Here's what I suggest changing that error to:
> >>
> >> >>> del x
> >> Unraisable exception suppressed when calling <bound method C.__del__
> >> of <__main__.C object at 0x7f98b8b61538>>
> >> Traceback (most recent call last):
> >>   File "<stdin>", line 3, in __del__
> >> RuntimeError: Going away now
> >
> > Why not simply "Exception automatically caught in <bound method
> > C.__del__> [...]" ?
> 
> It only answers the "what" (i.e. the exception was automatically
> caught), without addressing the "why" (i.e. because there wasn't
> anything else useful the interpreter could do with it)

Yes, but I agree with Greg that "unraisable" is wrong. After all, it
was raised, and it can even be caught by the programmer (inside
__del__).

Regards

Antoine.

From solipsis at pitrou.net  Mon Sep 23 17:23:20 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 23 Sep 2013 17:23:20 +0200
Subject: [Python-Dev] Best practice for documentation for std lib
References: <20130922031304.GE19939@ando>
 <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
 <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com>
 <20130923003142.GO19939@ando>
 <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com>
 <l1o8ri$mjv$1@ger.gmane.org>
 <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com>
 <523FF44D.1000303@hastings.org>
Message-ID: <20130923172320.5da170cb@fsol>

On Mon, 23 Sep 2013 08:57:01 +0100
Larry Hastings <larry at hastings.org> wrote:
> On 09/23/2013 03:44 AM, Guido van Rossum wrote:
> > On Sun, Sep 22, 2013 at 7:25 PM, Terry Reedy <tjreedy at udel.edu 
> > <mailto:tjreedy at udel.edu>> wrote:
> >
> >     I am gradually changing Idle docstrings, though only Idle
> >     developers will ever see them. Writing a 60 char summary forces a
> >     clear understanding of the function.
> >
> >
> > Glad you like it. I still do, too, but I've given up hope to convince 
> > all core developers to stick to this style. :-(
> 
> ... Argument Clinic to the rescue?  Since the last time this subject 
> came up, Clinic has started enforcing a summary line in docstrings. I 
> didn't realize it had to be 60 columns though, should I add that? Is the 
> entire docstring supposed to be 60 columns max?
> 
> p.s. status update: AC is done enough to be worth considering checking 
> in.  Now I have to finish the PEP and write the documentation.

I hope it can make it in for 3.4 :)

cheers

Antoine.



From skip at pobox.com  Mon Sep 23 17:18:22 2013
From: skip at pobox.com (Skip Montanaro)
Date: Mon, 23 Sep 2013 10:18:22 -0500
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <CAP7+vJLDo9KkdzzQfnJe_bTzunAP6+gBznY1Dx5pvxBqva0YjQ@mail.gmail.com>
References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol>
 <20130922125323.GI19939@ando>
 <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com>
 <524025B3.7020305@livinglogic.de>
 <CAP7+vJLDo9KkdzzQfnJe_bTzunAP6+gBznY1Dx5pvxBqva0YjQ@mail.gmail.com>
Message-ID: <CANc-5UzYUtta=4Q3c7n-LXdcYBnWrh5VYdMwSNPJ=5a0HuVfig@mail.gmail.com>

>> It would be great if the docstring contained a link to the online
>> documentation.
>
> That would have to be a feature of help(), not hardcoded in each docstring.

That *is* a feature of the help function:

Help on built-in module sys:

>>> help(sys)
NAME
    sys

FILE
    (built-in)

MODULE DOCS
    http://docs.python.org/library/sys
...

(pydoc too, though I'm 99.9% sure they use the same underlying
facility Ping originally implemented.)

Skip

From d.s.seljebotn at astro.uio.no  Mon Sep 23 17:17:31 2013
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Mon, 23 Sep 2013 17:17:31 +0200
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <20130923144311.23751250959@webabinitio.net>
References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol>
 <20130922125323.GI19939@ando>
 <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com>
 <003301ceb7bd$e1a59430$a4f0bc90$@gmail.com>
 <CAP7+vJLKVNWEDT62U4pJ7FGARthhVqb74hnLWznBag353s-Obw@mail.gmail.com>
 <005001ceb7c9$4dc308e0$e9491aa0$@gmail.com>
 <06E35F19-F893-415B-B127-77D1588A565A@masklinn.net>
 <CAP7+vJLRn-HAhNwBGLOmdPZA7YBq5JCs=H19FVeard6Y8yqYNQ@mail.gmail.com>
 <20130923144311.23751250959@webabinitio.net>
Message-ID: <52405B8B.4090704@astro.uio.no>

On 09/23/2013 04:43 PM, R. David Murray wrote:
> On Sun, 22 Sep 2013 16:19:21 -0700, Guido van Rossum <guido at python.org> wrote:
>> On Sun, Sep 22, 2013 at 2:41 PM, Xavier Morel <python-dev at masklinn.net>wrote:
>>
>>> The points here are that there's a single source of truth (so we can't
>>> have conflicting docstring and rst documentation), and documentation
>>> becoming outdated can be noticed from both docstring and published
>>> documentation.
>
> Another thing that hasn't been mentioned about docstrings vs rst docs,
> is that even when the text is identical, it generally isn't.  By that I
> mean the rst docs have ReST markup, but the docstrings don't.  So using

FYI, the scientific Python community have their own standard (the NumPy 
docstring standard), with light RST markup in docstrings:

https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt

It's a fairly popular standard in the scientific sub-culture.

I think there's something about scientific codes which tips the scales 
away from Guido's position of short docstrings being better; when 
working with computation, you pretty much need the documentation, 
formulas, references to paper it is based on and so on in order to read 
and understand the code in the first place, so interspersing makes more 
sense than it may in the stdlib. Seeing code with references "x", 
"alpha", "beta", "gamma" without their definition is pretty useless :-)

Dag Sverre

From guido at python.org  Mon Sep 23 17:49:28 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 23 Sep 2013 08:49:28 -0700
Subject: [Python-Dev] Enum Eccentricities
In-Reply-To: <20130923151623.GB7989@ando>
References: <523F82C4.4060202@stoneleaf.us> <20130923024139.GA26561@piedra>
 <CAAXXHgKPNsXH1dgv2GeqhVC9+s1YaziCYXq0Aah-e=a3mSLAmg@mail.gmail.com>
 <CAP7+vJKisYvdOug-fmR_SZEET7icHGjEx2pFYYHXyZHrhjWhzw@mail.gmail.com>
 <20130923151623.GB7989@ando>
Message-ID: <CAP7+vJK3chF=TUvjJBD=gcTtAQ4eEcJ5jgkOSpADJUuLGnKqkQ@mail.gmail.com>

On Mon, Sep 23, 2013 at 8:16 AM, Steven D'Aprano <steve at pearwood.info>wrote:

> On Mon, Sep 23, 2013 at 07:53:00AM -0700, Guido van Rossum wrote:
>
> > there is no rule that because you can
> > access something on the class you should be able to access it on the
> > instance. Try asking an instance for its class's __mro__ or __bases__.
>
> It might not be a rule, but it's certainly the norm. I reckon that class
> attributes that aren't accessible from the instance are significantly
> more surprising than Color.red.blue.
>

Whether this feels right (to me) depends on whether the class attribute was
*meant* to be used as an instance attribute as well, or not. For methods,
that's an obvious yes. For class attributes used as defaults for instance
variables, yes again. For class attributes that are meant to be per-class
state rather than per-instance state, I would use the class name or
__class__ to make it clear to the (human) reader that we're using a class
variable. This also avoids accidentally overriding it with an instance
variable (although that's not an issue for Enum).


> I know I'm in a minority here, but Color.red.blue seems obvious and
> straightforward to me. The fact that it doesn't work surprises me. Given
> that
>
> instance = Color.red
> assert isinstance(instance, Color)  # well of course it is
> assert hasattr(Color, "blue")
>
> I would expect instance.blue to work, and I'm completely at a loss as to
> how Enum has managed to prevent it.
>

Through the magic of metaclasses. :-)

The only attributes that an enum instance should have are things like .name
and .value, and the methods implementing various standard protocols like
__repr__ and __eq__.

The whole point of Enum is that it involves a fair bit of magic to offer a
robust way to define Enums that "feel right" as Enums. You shouldn't think
of it as a slightly odd class. You should think of it as a different data
type altogether that happens to be implemented using metaclass tecnology.
But many things you "know" about classes don't apply to Enums; this is just
one of many.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130923/734eabf0/attachment.html>

From ethan at stoneleaf.us  Mon Sep 23 18:12:27 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 23 Sep 2013 09:12:27 -0700
Subject: [Python-Dev] Enum Eccentricities
In-Reply-To: <20130923151623.GB7989@ando>
References: <523F82C4.4060202@stoneleaf.us> <20130923024139.GA26561@piedra>
 <CAAXXHgKPNsXH1dgv2GeqhVC9+s1YaziCYXq0Aah-e=a3mSLAmg@mail.gmail.com>
 <CAP7+vJKisYvdOug-fmR_SZEET7icHGjEx2pFYYHXyZHrhjWhzw@mail.gmail.com>
 <20130923151623.GB7989@ando>
Message-ID: <5240686B.3040504@stoneleaf.us>

On 09/23/2013 08:16 AM, Steven D'Aprano wrote:
>
> I would expect instance.blue to work, and I'm completely at a loss as to
> how Enum has managed to prevent it.

[A peek behind the curtains...]

Currently, Enum members do not live in the class dict.  So when, for example, blue is searched for in red, it is not 
found in the instance dict, and it is not found in the class dict.  As you know, __getattr__ will then be invoked -- but 
the Enum class does not have its own __getattr__, nor its own __getattribute__, and so we get an AttributeError.

Well, you may ask, if blue does not live in the class dict, how does Color.blue work?  I'm glad you asked.  ;)

Color is of type EnumMeta, and EnumMeta /does/ have __getattr__, so a failed /class/ lookup will invoke the metaclass 
__getattr__, which will search in the right place, find, and return, Color.blue.

--
~Ethan~

From rdmurray at bitdance.com  Mon Sep 23 18:23:55 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Mon, 23 Sep 2013 12:23:55 -0400
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <20130923172245.0cbaec81@fsol>
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
 <l1n9o6$ap3$1@ger.gmane.org>
 <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
 <l1nkdh$lh7$1@ger.gmane.org>
 <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>
 <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com>
 <523FDB8F.1080207@canterbury.ac.nz>
 <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com>
 <20130923104500.5645fdbb@pitrou.net>
 <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com>
 <20130923172245.0cbaec81@fsol>
Message-ID: <20130923162356.2135F25095A@webabinitio.net>

On Mon, 23 Sep 2013 17:22:45 +0200, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Mon, 23 Sep 2013 18:51:04 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
> > On 23 September 2013 18:45, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > > Le Mon, 23 Sep 2013 18:17:51 +1000,
> > > Nick Coghlan <ncoghlan at gmail.com> a ??crit :
> > >>
> > >> Here's what I suggest changing that error to:
> > >>
> > >> >>> del x
> > >> Unraisable exception suppressed when calling <bound method C.__del__
> > >> of <__main__.C object at 0x7f98b8b61538>>
> > >> Traceback (most recent call last):
> > >>   File "<stdin>", line 3, in __del__
> > >> RuntimeError: Going away now
> > >
> > > Why not simply "Exception automatically caught in <bound method
> > > C.__del__> [...]" ?
> > 
> > It only answers the "what" (i.e. the exception was automatically
> > caught), without addressing the "why" (i.e. because there wasn't
> > anything else useful the interpreter could do with it)
> 
> Yes, but I agree with Greg that "unraisable" is wrong. After all, it
> was raised, and it can even be caught by the programmer (inside
> __del__).

Would it work to say "Asynchronous exception suppressed..."?  It's
not-entirely-precise, but it's less imprecise than "unraisable".

--David

From z at etiol.net  Mon Sep 23 17:17:35 2013
From: z at etiol.net (Zero Piraeus)
Date: Mon, 23 Sep 2013 12:17:35 -0300
Subject: [Python-Dev] Enum Eccentricities
In-Reply-To: <CAAXXHgKPNsXH1dgv2GeqhVC9+s1YaziCYXq0Aah-e=a3mSLAmg@mail.gmail.com>
References: <523F82C4.4060202@stoneleaf.us> <20130923024139.GA26561@piedra>
 <CAAXXHgKPNsXH1dgv2GeqhVC9+s1YaziCYXq0Aah-e=a3mSLAmg@mail.gmail.com>
Message-ID: <20130923151735.GA3364@piedra>

On Mon, Sep 23, 2013 at 09:45:46AM -0400, Chris Lambacher wrote:
> [...] The exact use case is in Django templates where a value comes
> from the database. If you want to compare you either have to use
> __class__ which I would say is a code smell, or you have
> to provide the Enum class.

I'm having a hard time seeing why the latter is problematic, I must
admit (certainly no more so than the "Alice in Wonderland" side effects
previously described).

> [...] An example of how this will be used in practice is:
> 
>     {% if object.state == object.state.completed %}
>       some html
>     {% endif %}

The names used slightly obscure the weirdness of it, but what you're
really saying there is:

    if my_state == my_state.another_state

... which feels more like a code smell to me than

     {% if object.state == State.completed %}
       some html
     {% endif %}

That's quite intelligible, and doesn't require anyone to know that an
Enum member's siblings can, in your proposal, be accessed directly via
dot notation (an unintuitive state of affairs, to me at least).

 -[]z.

-- 
Zero Piraeus: omnia omnibus
http://etiol.net/pubkey.asc

From guido at python.org  Mon Sep 23 18:50:49 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 23 Sep 2013 09:50:49 -0700
Subject: [Python-Dev] Enum Eccentricities
In-Reply-To: <20130923151735.GA3364@piedra>
References: <523F82C4.4060202@stoneleaf.us> <20130923024139.GA26561@piedra>
 <CAAXXHgKPNsXH1dgv2GeqhVC9+s1YaziCYXq0Aah-e=a3mSLAmg@mail.gmail.com>
 <20130923151735.GA3364@piedra>
Message-ID: <CAP7+vJJbci1FND=AEqjxu6HK0hNjtJ3Ud_Dp_Sks_tKpn5XRaA@mail.gmail.com>

On Mon, Sep 23, 2013 at 8:17 AM, Zero Piraeus <z at etiol.net> wrote:

> On Mon, Sep 23, 2013 at 09:45:46AM -0400, Chris Lambacher wrote:
> > [...] The exact use case is in Django templates where a value comes
> > from the database. If you want to compare you either have to use
> > __class__ which I would say is a code smell, or you have
> > to provide the Enum class.
>
> I'm having a hard time seeing why the latter is problematic, I must
> admit (certainly no more so than the "Alice in Wonderland" side effects
> previously described).
>
> > [...] An example of how this will be used in practice is:
> >
> >     {% if object.state == object.state.completed %}
> >       some html
> >     {% endif %}
>
> The names used slightly obscure the weirdness of it, but what you're
> really saying there is:
>
>     if my_state == my_state.another_state
>
> ... which feels more like a code smell to me than
>
>      {% if object.state == State.completed %}
>        some html
>      {% endif %}
>
> That's quite intelligible, and doesn't require anyone to know that an
> Enum member's siblings can, in your proposal, be accessed directly via
> dot notation (an unintuitive state of affairs, to me at least).
>

Right. The OP is just concerned that (because these are Django templates)
he will have to pass in the 'State' class as a separate template parameter
for this to work. But to me that's a problem with Django, and not something
for which the Enum type should bend over backwards to cover up for. Given
that it's a Djano weakness, IMO the __class__ solution is reasonable
enough, although in theory it would allow having object.state be something
of the wrong class that happens to have a 'completed' attribute -- that
would be a bug of a different color, though. :-)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130923/5695d625/attachment.html>

From stephen at xemacs.org  Mon Sep 23 19:46:30 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Tue, 24 Sep 2013 02:46:30 +0900
Subject: [Python-Dev] Enum Eccentricities
In-Reply-To: <CAP7+vJJbci1FND=AEqjxu6HK0hNjtJ3Ud_Dp_Sks_tKpn5XRaA@mail.gmail.com>
References: <523F82C4.4060202@stoneleaf.us> <20130923024139.GA26561@piedra>
 <CAAXXHgKPNsXH1dgv2GeqhVC9+s1YaziCYXq0Aah-e=a3mSLAmg@mail.gmail.com>
 <20130923151735.GA3364@piedra>
 <CAP7+vJJbci1FND=AEqjxu6HK0hNjtJ3Ud_Dp_Sks_tKpn5XRaA@mail.gmail.com>
Message-ID: <8761tr2zyh.fsf@uwakimon.sk.tsukuba.ac.jp>

Guido van Rossum writes:
 > On Mon, Sep 23, 2013 at 8:17 AM, Zero Piraeus <z at etiol.net> wrote:
 >> On Mon, Sep 23, 2013 at 09:45:46AM -0400, Chris Lambacher wrote:

 >>> [...] An example of how this will be used in practice is:>
 >>> ? ? {% if object.state == object.state.completed %}
 >>> ? ? ? some html
 >>> ? ? {% endif %}

[Zero Piraeus suggests instead:]

 >> ? ??{% if object.state == State.completed %}
 >>? ? ? ?some html
 >>? ? ?{% endif %}
 >>
 >> That's quite intelligible, and doesn't require anyone to know that
 >> an Enum member's siblings can, in your proposal, be accessed directly
 >> via dot notation (an unintuitive state of affairs, to me at least).

 > Right. The OP is just concerned that (because these are Django
 > templates) he will have to pass in the 'State' class as a separate
 > template parameter for this to work.

Given your earlier description of what makes sense for class
attributes, an alternative solution might be to put State-valued class
attributes (constants) on DjangoObject (the object's class), like
DjangoObject.completed_state = State.completed, and so on.  Then you
write "{% if object.state == object.completed_state %}".

IIUC, you wouldn't have a problem with that?  It still doesn't feel
quite right, but given the limitations of a template language, it
might grow on me.

Another alternative would be to have attributes like 'completed' be
*boolean* properties computed from a State-valued attribute, and write
just "{% if object.completed %}".  This actually feels good to me
given it's a templating language.

But I don't know if either of those is reasonable in the context.

From tjreedy at udel.edu  Mon Sep 23 19:52:28 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 23 Sep 2013 13:52:28 -0400
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <CAP7+vJLcGYXfS5iSjmCqh4BKFmatjWoGdZi5DednagfcvcUf1w@mail.gmail.com>
References: <20130922031304.GE19939@ando>
 <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
 <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com>
 <20130923003142.GO19939@ando>
 <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com>
 <l1o8ri$mjv$1@ger.gmane.org>
 <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com>
 <523FF44D.1000303@hastings.org>
 <CAP7+vJLcGYXfS5iSjmCqh4BKFmatjWoGdZi5DednagfcvcUf1w@mail.gmail.com>
Message-ID: <l1pv4k$ls7$1@ger.gmane.org>

On 9/23/2013 10:56 AM, Guido van Rossum wrote:

> I think 60 is just a guideline. In stdlib .py source code I want it not
> to extend beyond the 79th column (see recent PEP 8 argument). For a

PEP 8 says "Limit all lines to a maximum of 79 characters.

For flowing long blocks of text with fewer structural restrictions 
(docstrings or comments), the line length should be limited to 72 
characters."

I do not understand 72 instead of 79, but to me, that means 72 chars 
total, including indents, triple quote and period, which means the 
cursor not past 73. If that is not what you mean, please clarify.

> typical class, where the docstring is indented 4 spaces, that leaves 72
> characters for the summary line (including a final period!). For a
> method it's 68 due to the extra indent.

72 - 2 indents - """ - . = 60, which is a bit skimpy ;-). Why not the 
full 79 at least?

-- 
Terry Jan Reedy


From guido at python.org  Mon Sep 23 19:53:34 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 23 Sep 2013 10:53:34 -0700
Subject: [Python-Dev] Enum Eccentricities
In-Reply-To: <8761tr2zyh.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <523F82C4.4060202@stoneleaf.us> <20130923024139.GA26561@piedra>
 <CAAXXHgKPNsXH1dgv2GeqhVC9+s1YaziCYXq0Aah-e=a3mSLAmg@mail.gmail.com>
 <20130923151735.GA3364@piedra>
 <CAP7+vJJbci1FND=AEqjxu6HK0hNjtJ3Ud_Dp_Sks_tKpn5XRaA@mail.gmail.com>
 <8761tr2zyh.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CAP7+vJLbU0=_zXtimTMbTUpKSrCHk=SboAHXspvws8v1f-Xssw@mail.gmail.com>

I am quickly losing interest in this -- I was only jumping in because I
feared there was support for making Color.red.blue "work". I don't think I
hve to worry about that any more -- what's best for Django templates is up
to the template author.


On Mon, Sep 23, 2013 at 10:46 AM, Stephen J. Turnbull <stephen at xemacs.org>wrote:

> Guido van Rossum writes:
>  > On Mon, Sep 23, 2013 at 8:17 AM, Zero Piraeus <z at etiol.net> wrote:
>  >> On Mon, Sep 23, 2013 at 09:45:46AM -0400, Chris Lambacher wrote:
>
>  >>> [...] An example of how this will be used in practice is:>
>  >>>     {% if object.state == object.state.completed %}
>  >>>       some html
>  >>>     {% endif %}
>
> [Zero Piraeus suggests instead:]
>
>  >>     {% if object.state == State.completed %}
>  >>       some html
>  >>     {% endif %}
>  >>
>  >> That's quite intelligible, and doesn't require anyone to know that
>  >> an Enum member's siblings can, in your proposal, be accessed directly
>  >> via dot notation (an unintuitive state of affairs, to me at least).
>
>  > Right. The OP is just concerned that (because these are Django
>  > templates) he will have to pass in the 'State' class as a separate
>  > template parameter for this to work.
>
> Given your earlier description of what makes sense for class
> attributes, an alternative solution might be to put State-valued class
> attributes (constants) on DjangoObject (the object's class), like
> DjangoObject.completed_state = State.completed, and so on.  Then you
> write "{% if object.state == object.completed_state %}".
>
> IIUC, you wouldn't have a problem with that?  It still doesn't feel
> quite right, but given the limitations of a template language, it
> might grow on me.
>
> Another alternative would be to have attributes like 'completed' be
> *boolean* properties computed from a State-valued attribute, and write
> just "{% if object.completed %}".  This actually feels good to me
> given it's a templating language.
>
> But I don't know if either of those is reasonable in the context.
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130923/d855f562/attachment.html>

From guido at python.org  Mon Sep 23 19:59:06 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 23 Sep 2013 10:59:06 -0700
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <l1pv4k$ls7$1@ger.gmane.org>
References: <20130922031304.GE19939@ando>
 <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
 <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com>
 <20130923003142.GO19939@ando>
 <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com>
 <l1o8ri$mjv$1@ger.gmane.org>
 <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com>
 <523FF44D.1000303@hastings.org>
 <CAP7+vJLcGYXfS5iSjmCqh4BKFmatjWoGdZi5DednagfcvcUf1w@mail.gmail.com>
 <l1pv4k$ls7$1@ger.gmane.org>
Message-ID: <CAP7+vJ+wJX78hA7OzD48k-uk=DKiTks8vs5WHF-E-frQ6eMRFw@mail.gmail.com>

On Mon, Sep 23, 2013 at 10:52 AM, Terry Reedy <tjreedy at udel.edu> wrote:

> On 9/23/2013 10:56 AM, Guido van Rossum wrote:
>
>  I think 60 is just a guideline. In stdlib .py source code I want it not
>> to extend beyond the 79th column (see recent PEP 8 argument). For a
>>
>
> PEP 8 says "Limit all lines to a maximum of 79 characters.
>
> For flowing long blocks of text with fewer structural restrictions
> (docstrings or comments), the line length should be limited to 72
> characters."
>
> I do not understand 72 instead of 79, but to me, that means 72 chars
> total, including indents, triple quote and period, which means the cursor
> not past 73. If that is not what you mean, please clarify.


This is partly based on Emacs defaults (when I hit ESC-q, it reflows a
paragraph using the 72 char limit), partly on old punched card conventions
(columns 73-80 were reserved for sequence numbers), partly just to limit
the line length to of *columns* of text to something a bit narrower than
the full line width (i.e. 80 chars) to improve readability.

But the PEP specifically qualifies this with "for flowing long blocks of
text" which is meant to apply only to blocks of reflowable text that don't
fit on a line.

If it fits on a line, you can go to 79.


> typical class, where the docstring is indented 4 spaces, that leaves 72
> characters for the summary line (including a final period!). For a
> method it's 68 due to the extra indent.
>

72 - 2 indents - """ - . = 60, which is a bit skimpy ;-). Why not the full
> 79 at least?
>

For the summary line, that's fine (and actually how I interpret the PEP).

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130923/22e2c0be/attachment-0001.html>

From tjreedy at udel.edu  Mon Sep 23 20:08:26 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 23 Sep 2013 14:08:26 -0400
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <CANc-5UzYUtta=4Q3c7n-LXdcYBnWrh5VYdMwSNPJ=5a0HuVfig@mail.gmail.com>
References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol>
 <20130922125323.GI19939@ando>
 <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com>
 <524025B3.7020305@livinglogic.de>
 <CAP7+vJLDo9KkdzzQfnJe_bTzunAP6+gBznY1Dx5pvxBqva0YjQ@mail.gmail.com>
 <CANc-5UzYUtta=4Q3c7n-LXdcYBnWrh5VYdMwSNPJ=5a0HuVfig@mail.gmail.com>
Message-ID: <l1q02i$16h$1@ger.gmane.org>

On 9/23/2013 11:18 AM, Skip Montanaro wrote:
>>> It would be great if the docstring contained a link to the online
>>> documentation.
>>
>> That would have to be a feature of help(), not hardcoded in each docstring.
>
> That *is* a feature of the help function:
>
> Help on built-in module sys:
>
>>>> help(sys)
> NAME
>      sys
>
> FILE
>      (built-in)
>
> MODULE DOCS
>      http://docs.python.org/library/sys

On 3.3.2 and 3.4.0:

Help on built-in module sys:

NAME
     sys

MODULE REFERENCE
     http://docs.python.org/3.3/library/sys
...
FILE
       (built-in)

On Windows, one will not see the top of the help(sys) output unless one 
either uses Idle or increases the number of lines in the console window 
from 300.

-- 
Terry Jan Reedy


From tjreedy at udel.edu  Mon Sep 23 20:27:25 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 23 Sep 2013 14:27:25 -0400
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <20130923162356.2135F25095A@webabinitio.net>
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
 <l1n9o6$ap3$1@ger.gmane.org>
 <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
 <l1nkdh$lh7$1@ger.gmane.org>
 <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>
 <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com>
 <523FDB8F.1080207@canterbury.ac.nz>
 <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com>
 <20130923104500.5645fdbb@pitrou.net>
 <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com>
 <20130923172245.0cbaec81@fsol> <20130923162356.2135F25095A@webabinitio.net>
Message-ID: <l1q164$fgi$1@ger.gmane.org>

On 9/23/2013 12:23 PM, R. David Murray wrote:
> On Mon, 23 Sep 2013 17:22:45 +0200, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> On Mon, 23 Sep 2013 18:51:04 +1000
>> Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> On 23 September 2013 18:45, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>>> Le Mon, 23 Sep 2013 18:17:51 +1000,
>>>> Nick Coghlan <ncoghlan at gmail.com> a ?crit :
>>>>>
>>>>> Here's what I suggest changing that error to:
>>>>>
>>>>>>>> del x
>>>>> Unraisable exception suppressed when calling <bound method C.__del__
>>>>> of <__main__.C object at 0x7f98b8b61538>>
>>>>> Traceback (most recent call last):
>>>>>    File "<stdin>", line 3, in __del__
>>>>> RuntimeError: Going away now
>>>>
>>>> Why not simply "Exception automatically caught in <bound method
>>>> C.__del__> [...]" ?
>>>
>>> It only answers the "what" (i.e. the exception was automatically
>>> caught), without addressing the "why" (i.e. because there wasn't
>>> anything else useful the interpreter could do with it)
>>
>> Yes, but I agree with Greg that "unraisable" is wrong. After all, it
>> was raised, and it can even be caught by the programmer (inside
>> __del__).
>
> Would it work to say "Asynchronous exception suppressed..."?  It's
> not-entirely-precise,

As in the example above ('del x').

> but it's less imprecise than "unraisable".

How 'troublesome'? That is always accurate, as proven by this thread.

We really need an "Understanding Exceptions" HOWTO, and I expect we will 
get one. So I agree with Nick that something easily searched would be good.

-- 
Terry Jan Reedy



From tjreedy at udel.edu  Mon Sep 23 21:01:56 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 23 Sep 2013 15:01:56 -0400
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com>
References: <20130922031304.GE19939@ando>
 <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
 <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com>
 <20130923003142.GO19939@ando>
 <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com>
 <l1o8ri$mjv$1@ger.gmane.org>
 <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com>
Message-ID: <l1q36s$99l$1@ger.gmane.org>

On 9/22/2013 10:44 PM, Guido van Rossum wrote:

> Glad you like it. I still do, too, but I've given up hope to convince
> all core developers to stick to this style. :-(

 >[me]  ('Return' rather than 'Returns' is the current convention.)

> That's actually a religious argument which in the stdlib takes no strict
> position -- a quick grep shows that both are used, although 'Return' is
> more frequent by a 5-to-1 margin.

In the .rst docs, 'Return' versus 'Returns', exact uppercase word match, 
is a little over 3 to 1. I am sure I have seen 'Return' and similiar 
directive forms ('Print', 'Store', 'Compare', etc) recommended as 
current doc style, as prefered by the current doc crew.

 > IIRC in the Java world you *have* to
> use 'Returns', but I don't buy the argument from nit-picky grammarians
> that leads to this rule. (It's something about the documentation not
> being a command. But English is more flexible than that.)

My take is that 'Returns' describes to the programmer what the function 
(interpreter) does, while 'Return' says what the programmer says to the 
interpreter when using the function. I strongly prefer the directive 
form. Why? For one thing, *because* it is different from normal 
descriptive text, such as the first sentence of this paragraph. For 
another, the descriptive form seems addressed to me as code reader while 
the directive form seems addressed to me as code writer. For me, the 
latter seems more energizing.

-- 
Terry Jan Reedy


From python at mrabarnett.plus.com  Mon Sep 23 21:11:35 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Mon, 23 Sep 2013 20:11:35 +0100
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <l1q36s$99l$1@ger.gmane.org>
References: <20130922031304.GE19939@ando>
 <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
 <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com>
 <20130923003142.GO19939@ando>
 <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com>
 <l1o8ri$mjv$1@ger.gmane.org>
 <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com>
 <l1q36s$99l$1@ger.gmane.org>
Message-ID: <52409267.20206@mrabarnett.plus.com>

On 23/09/2013 20:01, Terry Reedy wrote:
> On 9/22/2013 10:44 PM, Guido van Rossum wrote:
>
>> Glad you like it. I still do, too, but I've given up hope to convince
>> all core developers to stick to this style. :-(
>
>   >[me]  ('Return' rather than 'Returns' is the current convention.)
>
>> That's actually a religious argument which in the stdlib takes no strict
>> position -- a quick grep shows that both are used, although 'Return' is
>> more frequent by a 5-to-1 margin.
>
> In the .rst docs, 'Return' versus 'Returns', exact uppercase word match,
> is a little over 3 to 1. I am sure I have seen 'Return' and similiar
> directive forms ('Print', 'Store', 'Compare', etc) recommended as
> current doc style, as prefered by the current doc crew.
>
>   > IIRC in the Java world you *have* to
>> use 'Returns', but I don't buy the argument from nit-picky grammarians
>> that leads to this rule. (It's something about the documentation not
>> being a command. But English is more flexible than that.)
>
> My take is that 'Returns' describes to the programmer what the function
> (interpreter) does, while 'Return' says what the programmer says to the
> interpreter when using the function. I strongly prefer the directive
> form. Why? For one thing, *because* it is different from normal
> descriptive text, such as the first sentence of this paragraph. For
> another, the descriptive form seems addressed to me as code reader while
> the directive form seems addressed to me as code writer. For me, the
> latter seems more energizing.
>
<pedantic>I think you mean "imperative" vs "indicative".</pedantic>

From alexander.belopolsky at gmail.com  Mon Sep 23 21:18:01 2013
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Mon, 23 Sep 2013 15:18:01 -0400
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <l1q36s$99l$1@ger.gmane.org>
References: <20130922031304.GE19939@ando>
 <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
 <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com>
 <20130923003142.GO19939@ando>
 <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com>
 <l1o8ri$mjv$1@ger.gmane.org>
 <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com>
 <l1q36s$99l$1@ger.gmane.org>
Message-ID: <CAP7h-xZ-E_3--JrQGE+htXZStagmBjHmozW6v6jEbiXUicQ8fQ@mail.gmail.com>

On Mon, Sep 23, 2013 at 3:01 PM, Terry Reedy <tjreedy at udel.edu> wrote:

> > IIRC in the Java world you *have* to
>
>> use 'Returns', but I don't buy the argument from nit-picky grammarians
>> that leads to this rule. (It's something about the documentation not
>> being a command. But English is more flexible than that.)
>>
>
> My take is that 'Returns' describes to the programmer what the function
> (interpreter) does, while 'Return' says what the programmer says to the
> interpreter when using the function. I strongly prefer the directive form.


+1

I don't think "Returns bar." is a valid English sentence because it lacks
subject.  I would not mind

def foo():
      """returns bar"""

which I would read as "Function foo() returns bar," but in this case
"returns" should be in lower case.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130923/3264fc0a/attachment.html>

From fred at fdrake.net  Mon Sep 23 21:37:54 2013
From: fred at fdrake.net (Fred Drake)
Date: Mon, 23 Sep 2013 15:37:54 -0400
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <l1q36s$99l$1@ger.gmane.org>
References: <20130922031304.GE19939@ando>
 <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
 <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com>
 <20130923003142.GO19939@ando>
 <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com>
 <l1o8ri$mjv$1@ger.gmane.org>
 <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com>
 <l1q36s$99l$1@ger.gmane.org>
Message-ID: <CAFT4OTFEpNcYAT0Z9ZkSbYodESNpoQeQVorAQWo83E9Rv2kihA@mail.gmail.com>

On Mon, Sep 23, 2013 at 3:01 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> 'Return' versus 'Returns', exact uppercase word match, is a little over 3 to
> 1. I am sure I have seen 'Return' and similiar directive forms ('Print',
> 'Store', 'Compare', etc) recommended as current doc style, as prefered by
> the current doc crew.

I don't know about the current crew, but this dates way back to
Guido's initial LaTeX documentation, and Guido's strong preference on
this.


  -Fred

-- 
Fred L. Drake, Jr.    <fred at fdrake.net>
"A storm broke loose in my mind."  --Albert Einstein

From ncoghlan at gmail.com  Mon Sep 23 23:19:14 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 24 Sep 2013 07:19:14 +1000
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <20130923172245.0cbaec81@fsol>
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net>
 <l1lbkl$k0g$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
 <l1n9o6$ap3$1@ger.gmane.org>
 <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
 <l1nkdh$lh7$1@ger.gmane.org>
 <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>
 <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com>
 <523FDB8F.1080207@canterbury.ac.nz>
 <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com>
 <20130923104500.5645fdbb@pitrou.net>
 <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com>
 <20130923172245.0cbaec81@fsol>
Message-ID: <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com>

On 24 Sep 2013 01:24, "Antoine Pitrou" <solipsis at pitrou.net> wrote:
>
> On Mon, 23 Sep 2013 18:51:04 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
> > On 23 September 2013 18:45, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > > Le Mon, 23 Sep 2013 18:17:51 +1000,
> > > Nick Coghlan <ncoghlan at gmail.com> a ?crit :
> > >>
> > >> Here's what I suggest changing that error to:
> > >>
> > >> >>> del x
> > >> Unraisable exception suppressed when calling <bound method C.__del__
> > >> of <__main__.C object at 0x7f98b8b61538>>
> > >> Traceback (most recent call last):
> > >>   File "<stdin>", line 3, in __del__
> > >> RuntimeError: Going away now
> > >
> > > Why not simply "Exception automatically caught in <bound method
> > > C.__del__> [...]" ?
> >
> > It only answers the "what" (i.e. the exception was automatically
> > caught), without addressing the "why" (i.e. because there wasn't
> > anything else useful the interpreter could do with it)
>
> Yes, but I agree with Greg that "unraisable" is wrong. After all, it
> was raised, and it can even be caught by the programmer (inside
> __del__).

The word doesn't literally mean the exception itself was unraisable. It
means it was raised, we caught it and we're writing it to stderr because we
*can't raise it again*.

The relevant C API function is just called "PyErr_WriteUnraisable", not
"PyErr_WriteUnraisableButThatIsTechnicallyWrongSinceItWasAlreadyRaisedAndWeJustCaughtItAndAreNowReportingItToStdErr".

Cheers,
Nick.

>
> Regards
>
> Antoine.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130924/3d93cb57/attachment.html>

From solipsis at pitrou.net  Mon Sep 23 23:35:07 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 23 Sep 2013 23:35:07 +0200
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com>
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net>
 <l1lbkl$k0g$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
 <l1n9o6$ap3$1@ger.gmane.org>
 <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
 <l1nkdh$lh7$1@ger.gmane.org>
 <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>
 <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com>
 <523FDB8F.1080207@canterbury.ac.nz>
 <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com>
 <20130923104500.5645fdbb@pitrou.net>
 <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com>
 <20130923172245.0cbaec81@fsol>
 <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com>
Message-ID: <20130923233507.4e5ebb9f@fsol>

On Tue, 24 Sep 2013 07:19:14 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 24 Sep 2013 01:24, "Antoine Pitrou" <solipsis at pitrou.net> wrote:
> >
> > On Mon, 23 Sep 2013 18:51:04 +1000
> > Nick Coghlan <ncoghlan at gmail.com> wrote:
> > > On 23 September 2013 18:45, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > > > Le Mon, 23 Sep 2013 18:17:51 +1000,
> > > > Nick Coghlan <ncoghlan at gmail.com> a ?crit :
> > > >>
> > > >> Here's what I suggest changing that error to:
> > > >>
> > > >> >>> del x
> > > >> Unraisable exception suppressed when calling <bound method C.__del__
> > > >> of <__main__.C object at 0x7f98b8b61538>>
> > > >> Traceback (most recent call last):
> > > >>   File "<stdin>", line 3, in __del__
> > > >> RuntimeError: Going away now
> > > >
> > > > Why not simply "Exception automatically caught in <bound method
> > > > C.__del__> [...]" ?
> > >
> > > It only answers the "what" (i.e. the exception was automatically
> > > caught), without addressing the "why" (i.e. because there wasn't
> > > anything else useful the interpreter could do with it)
> >
> > Yes, but I agree with Greg that "unraisable" is wrong. After all, it
> > was raised, and it can even be caught by the programmer (inside
> > __del__).
> 
> The word doesn't literally mean the exception itself was unraisable. It
> means it was raised, we caught it and we're writing it to stderr because we
> *can't raise it again*.

But that's because you already know what it's supposed to convey. The
average user doesn't, and only sees "unraisable".

> The relevant C API function is just called "PyErr_WriteUnraisable", not
> "PyErr_WriteUnraisableButThatIsTechnicallyWrongSinceItWasAlreadyRaisedAndWeJustCaughtItAndAreNowReportingItToStdErr".

"PyErr_WriteUnraisable" is right at the point where it used: I can't
raise that exception at this point, therefore I call
PyErr_WriteUnraisable. However, from the point of view of the user
reading the traceback, the exception *was* raised.

Regards

Antoine.

From ethan at stoneleaf.us  Mon Sep 23 23:26:17 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 23 Sep 2013 14:26:17 -0700
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com>
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
 <l1n9o6$ap3$1@ger.gmane.org>
 <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
 <l1nkdh$lh7$1@ger.gmane.org>
 <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>
 <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com>
 <523FDB8F.1080207@canterbury.ac.nz>
 <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com>
 <20130923104500.5645fdbb@pitrou.net>
 <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com>
 <20130923172245.0cbaec81@fsol>
 <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com>
Message-ID: <5240B1F9.7020602@stoneleaf.us>

On 09/23/2013 02:19 PM, Nick Coghlan wrote:
>
> The relevant C API function is just called "PyErr_WriteUnraisable", not
> "PyErr_WriteUnraisableButThatIsTechnicallyWrongSinceItWasAlreadyRaisedAndWeJustCaughtItAndAreNowReportingItToStdErr".

Wow.  How many legs does that HumptyCamel have, anyway?  ;)

--
~Ethan~

From greg.ewing at canterbury.ac.nz  Tue Sep 24 00:07:45 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 24 Sep 2013 10:07:45 +1200
Subject: [Python-Dev] Enum Eccentricities
In-Reply-To: <20130923151623.GB7989@ando>
References: <523F82C4.4060202@stoneleaf.us> <20130923024139.GA26561@piedra>
 <CAAXXHgKPNsXH1dgv2GeqhVC9+s1YaziCYXq0Aah-e=a3mSLAmg@mail.gmail.com>
 <CAP7+vJKisYvdOug-fmR_SZEET7icHGjEx2pFYYHXyZHrhjWhzw@mail.gmail.com>
 <20130923151623.GB7989@ando>
Message-ID: <5240BBB1.5010900@canterbury.ac.nz>

Steven D'Aprano wrote:
> It might not be a rule, but it's certainly the norm. I reckon that class 
> attributes that aren't accessible from the instance are significantly 
> more surprising than Color.red.blue.

There are really two different kinds of things that we
refer to as "class attributes". One is things that really
are attributes of the class itself, and the other is
things that are meant to serve as default or shared
instance attributes.

The confusion comes in because we use the same terminology
for both, and because Python doesn't provide any straightforward
way of creating user-defined class-only attributes, so
shared attributes tend to get abused for that purpose.
Then when someone comes along and creates a true
class-only attribute, people get all surprised and
complain about it. They shouldn't, IMO.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Tue Sep 24 00:11:54 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 24 Sep 2013 10:11:54 +1200
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute
	error	message
In-Reply-To: <20130923172245.0cbaec81@fsol>
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
 <l1n9o6$ap3$1@ger.gmane.org>
 <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
 <l1nkdh$lh7$1@ger.gmane.org>
 <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>
 <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com>
 <523FDB8F.1080207@canterbury.ac.nz>
 <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com>
 <20130923104500.5645fdbb@pitrou.net>
 <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com>
 <20130923172245.0cbaec81@fsol>
Message-ID: <5240BCAA.5030604@canterbury.ac.nz>

Antoine Pitrou wrote:
> Yes, but I agree with Greg that "unraisable" is wrong. After all, it
> was raised, and it can even be caught by the programmer (inside
> __del__).

How about something like "Uncaught exception in __del__
method ignored"? It explains fairly clearly what has
happened, and also indicates what do do about it --
catch it in the __del__ method.

-- 
Greg

From ethan at stoneleaf.us  Mon Sep 23 23:38:48 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 23 Sep 2013 14:38:48 -0700
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <20130923233507.4e5ebb9f@fsol>
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
 <l1n9o6$ap3$1@ger.gmane.org>
 <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
 <l1nkdh$lh7$1@ger.gmane.org>
 <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>
 <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com>
 <523FDB8F.1080207@canterbury.ac.nz>
 <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com>
 <20130923104500.5645fdbb@pitrou.net>
 <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com>
 <20130923172245.0cbaec81@fsol>
 <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com>
 <20130923233507.4e5ebb9f@fsol>
Message-ID: <5240B4E8.6020701@stoneleaf.us>

On 09/23/2013 02:35 PM, Antoine Pitrou wrote:
> On Tue, 24 Sep 2013 07:19:14 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
>> On 24 Sep 2013 01:24, "Antoine Pitrou" <solipsis at pitrou.net> wrote:
>>>
>>> On Mon, 23 Sep 2013 18:51:04 +1000
>>> Nick Coghlan <ncoghlan at gmail.com> wrote:
>>>> On 23 September 2013 18:45, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>>>> Le Mon, 23 Sep 2013 18:17:51 +1000,
>>>>> Nick Coghlan <ncoghlan at gmail.com> a ?crit :
>>>>>>
>>>>>> Here's what I suggest changing that error to:
>>>>>>
>>>>>>>>> del x
>>>>>> Unraisable exception suppressed when calling <bound method C.__del__
>>>>>> of <__main__.C object at 0x7f98b8b61538>>
>>>>>> Traceback (most recent call last):
>>>>>>    File "<stdin>", line 3, in __del__
>>>>>> RuntimeError: Going away now
>>>>>
>>>>> Why not simply "Exception automatically caught in <bound method
>>>>> C.__del__> [...]" ?
>>>>
>>>> It only answers the "what" (i.e. the exception was automatically
>>>> caught), without addressing the "why" (i.e. because there wasn't
>>>> anything else useful the interpreter could do with it)
>>>
>>> Yes, but I agree with Greg that "unraisable" is wrong. After all, it
>>> was raised, and it can even be caught by the programmer (inside
>>> __del__).
>>
>> The word doesn't literally mean the exception itself was unraisable. It
>> means it was raised, we caught it and we're writing it to stderr because we
>> *can't raise it again*.
>
> But that's because you already know what it's supposed to convey. The
> average user doesn't, and only sees "unraisable".

All the more reason to have text in the error message that is easily searchable.

--
~Ethan~

From solipsis at pitrou.net  Tue Sep 24 00:29:55 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 24 Sep 2013 00:29:55 +0200
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net>
 <l1lbkl$k0g$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
 <l1n9o6$ap3$1@ger.gmane.org>
 <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
 <l1nkdh$lh7$1@ger.gmane.org>
 <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>
 <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com>
 <523FDB8F.1080207@canterbury.ac.nz>
 <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com>
 <20130923104500.5645fdbb@pitrou.net>
 <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com>
 <20130923172245.0cbaec81@fsol>
 <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com>
 <20130923233507.4e5ebb9f@fsol> <5240B4E8.6020701@stoneleaf.us>
Message-ID: <20130924002955.49a99f9f@fsol>

On Mon, 23 Sep 2013 14:38:48 -0700
Ethan Furman <ethan at stoneleaf.us> wrote:
> > But that's because you already know what it's supposed to convey. The
> > average user doesn't, and only sees "unraisable".
> 
> All the more reason to have text in the error message that is easily searchable.

Then I propose "CA6yuaYV6dygPfJRxUrhtg". It should be easily
searchable ;-)

Seriously, "easily searchable" is a rather weak criterion for an
error message. It should be easily understandable and non-misleading
first.

Regards

Antoine.



From python at mrabarnett.plus.com  Tue Sep 24 01:04:28 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 24 Sep 2013 00:04:28 +0100
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com>
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
 <l1n9o6$ap3$1@ger.gmane.org>
 <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
 <l1nkdh$lh7$1@ger.gmane.org>
 <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>
 <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com>
 <523FDB8F.1080207@canterbury.ac.nz>
 <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com>
 <20130923104500.5645fdbb@pitrou.net>
 <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com>
 <20130923172245.0cbaec81@fsol>
 <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com>
Message-ID: <5240C8FC.8070608@mrabarnett.plus.com>

On 23/09/2013 22:19, Nick Coghlan wrote:
>
> On 24 Sep 2013 01:24, "Antoine Pitrou" <solipsis at pitrou.net
> <mailto:solipsis at pitrou.net>> wrote:
>  >
>  > On Mon, 23 Sep 2013 18:51:04 +1000
>  > Nick Coghlan <ncoghlan at gmail.com <mailto:ncoghlan at gmail.com>> wrote:
>  > > On 23 September 2013 18:45, Antoine Pitrou <solipsis at pitrou.net
> <mailto:solipsis at pitrou.net>> wrote:
>  > > > Le Mon, 23 Sep 2013 18:17:51 +1000,
>  > > > Nick Coghlan <ncoghlan at gmail.com <mailto:ncoghlan at gmail.com>> a
> ?crit :
>  > > >>
>  > > >> Here's what I suggest changing that error to:
>  > > >>
>  > > >> >>> del x
>  > > >> Unraisable exception suppressed when calling <bound method C.__del__
>  > > >> of <__main__.C object at 0x7f98b8b61538>>
>  > > >> Traceback (most recent call last):
>  > > >>   File "<stdin>", line 3, in __del__
>  > > >> RuntimeError: Going away now
>  > > >
>  > > > Why not simply "Exception automatically caught in <bound method
>  > > > C.__del__> [...]" ?
>  > >
>  > > It only answers the "what" (i.e. the exception was automatically
>  > > caught), without addressing the "why" (i.e. because there wasn't
>  > > anything else useful the interpreter could do with it)
>  >
>  > Yes, but I agree with Greg that "unraisable" is wrong. After all, it
>  > was raised, and it can even be caught by the programmer (inside
>  > __del__).
>
> The word doesn't literally mean the exception itself was unraisable. It
> means it was raised, we caught it and we're writing it to stderr because
> we *can't raise it again*.
>
Ah, you mean "unreraisable". :-)

> The relevant C API function is just called "PyErr_WriteUnraisable", not
> "PyErr_WriteUnraisableButThatIsTechnicallyWrongSinceItWasAlreadyRaisedAndWeJustCaughtItAndAreNowReportingItToStdErr".
>


From ncoghlan at gmail.com  Tue Sep 24 01:25:10 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 24 Sep 2013 09:25:10 +1000
Subject: [Python-Dev] Enum Eccentricities
In-Reply-To: <5240BBB1.5010900@canterbury.ac.nz>
References: <523F82C4.4060202@stoneleaf.us> <20130923024139.GA26561@piedra>
 <CAAXXHgKPNsXH1dgv2GeqhVC9+s1YaziCYXq0Aah-e=a3mSLAmg@mail.gmail.com>
 <CAP7+vJKisYvdOug-fmR_SZEET7icHGjEx2pFYYHXyZHrhjWhzw@mail.gmail.com>
 <20130923151623.GB7989@ando> <5240BBB1.5010900@canterbury.ac.nz>
Message-ID: <CADiSq7evT-spFX+cHAQB_K2jH=5YM-uFOYGFGsPvOaEQYaYxzQ@mail.gmail.com>

On 24 Sep 2013 08:09, "Greg Ewing" <greg.ewing at canterbury.ac.nz> wrote:
>
> Steven D'Aprano wrote:
>>
>> It might not be a rule, but it's certainly the norm. I reckon that class
attributes that aren't accessible from the instance are significantly more
surprising than Color.red.blue.
>
>
> There are really two different kinds of things that we
> refer to as "class attributes". One is things that really
> are attributes of the class itself, and the other is
> things that are meant to serve as default or shared
> instance attributes.
>
> The confusion comes in because we use the same terminology
> for both, and because Python doesn't provide any straightforward
> way of creating user-defined class-only attributes,

Using @property in a metaclass definition isn't *that* complicated :)

(says the guy who helps maintain the type system)

> so
> shared attributes tend to get abused for that purpose.
> Then when someone comes along and creates a true
> class-only attribute, people get all surprised and
> complain about it.

One of the interesting aspects of adding Enum has been the subtle
descriptor handling bugs it has uncovered in the inspect module :)

> They shouldn't, IMO.

There's a helper for Enum's descriptors that will probably be exposed
through the types module in the next 3.4 alpha (tentative name is
types.DynamicClassAttribute). It's the inverse, though - it throws
AttributeError when looked up on the *class* in order to trigger
__getattr__ on the metaclass.

Cheers,
Nick.

>
> --
> Greg
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130924/155de6aa/attachment.html>

From nad at acm.org  Tue Sep 24 01:34:54 2013
From: nad at acm.org (Ned Deily)
Date: Mon, 23 Sep 2013 16:34:54 -0700
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
Message-ID: <nad-F39A9C.16345423092013@news.gmane.org>

In general, I think this is a very important usability feature and I
am in favor of the general approach.  Good work, all!  I do have some
comments, primarily about items that are not currently addressed.

> ``ensurepip`` itself (including the private copy of ``pip`` and its
> dependencies) will still be installed normally (as it is a regular
> part of the standard library), only the implicit installation of pip and
> its dependencies will be skipped.

The PEP should address the license implications of this since the PEP will
add the source of pip and setuptools to all new Python releases, both
source and binary installers releases.  For Python itself, we go to some
trouble to ensure that all contributions are contributed under a suitable
license.  What steps are needed, initially and on-going, to ensure that
these third-party items to be pulled in at each release are compatible
with the license of Python itself?  Who will do this: release managers?
Are updates needed in the internal and external documentation for releases
(README, LICENSE)?  I took a quick look at the current source releases for
both pip and setuptools.  I noted a reference to the LGPL for pip's CA
certs bundle; I didn't see a license file for setuptools.

> This PEP proposes that, if pip still requires it as a dependency,
> ``ensurepip`` will include a private copy of ``setuptools`` (in addition
> to the private copy of ``ensurepip``). ``python -m ensurepip`` will then
> install the private copy in addition to installing ``pip`` itself.
> 
> However, this behavior is officially considered an implementation
> detail. Other projects which explicitly require ``setuptools`` must still
> provide an appropriate dependency declaration, rather than assuming
> ``setuptools`` will always be installed alongside ``pip``.
> 
> Once pip is able to run ``pip install --upgrade pip`` without needing
> ``setuptools`` installed first, then the private copy of ``setuptools``
> will be removed from ``ensurepip`` in subsequent CPython releases.

The PEP does not sufficiently address the issue of the setuptools-provided
easy_install command.  I think it is important to do that for several
reasons:

1. The PEP should make clear whether the easy_install scripts are or
are not installed, globally or in a venv, as a side effect of running
ensurepip, at least while pip has the implicit private dependency
on setuptools.  By default, the scripts are installed so ensurepip
would need to take action to prevent this if that is not desired.

2. Despite the current strong preference for using pip as a command
line installer, the fact is that much third-party documentation still refers
to and recommends using easy_install for their projects.  That is not
going to change overnight.  The PEP should identify some steps needed to
facilitate the migration, like perhaps adding something to the "Installing
Python Modules" documentation explicitly deprecating easy_install use
and/or giving pip equivalents for the most common easy_install
scenarios (or linking to other documentation that does so).

3. A specific issue for OS X users is that, as part of OS X, Apple ships
versions of Python 2 and various third-party packages, including
setuptools (older, pre-reunification versions, at that) but not pip.
That means that OS X ships with easy_install and easy_install-2.n
commands in /usr/bin, including 2.7 in recent OS X releases.  This has
been an ongoing source of confusion for OS X users who install a newer
version of Python.  Depending on the distributor and/or source build
configure options, the new version of Python will be installed to a
different path, like /usr/local/bin.  Which instance is used
is generally managed by direct PATH environment manipulations (or by
launcher programs similar to the Windows py launcher).  If setuptools
is not installed using the new instance, users invoking the easy_install
command will by default end up with distributions being installed to a
system Python rather than to the new Python.  This leads to the
unfortunately common scenario of:

    sudo easy_install blah # --> installs to system Python 2.7
    python -c 'import blah' # --> fails because using newer Python 2.7

This may very well be the most common current usability problem today
with python.org (and other third-party) Pythons on OS X.

With this PEP, we have an opportunity to fix this problem in one of
two ways.  As long as pip requires setuptools and we do not inhibit
the installation of the easy_install scripts is to the same directory
as the python command itself, the new easy_install scripts will
shadow the Apple-supplied ones just as the python commands do.  If it
is decided to not install the easy_install scripts now or in the
future (e.g. when pip no longer depends on setuptools),
ensurepip could install dummy easy_install scripts (if there
are none already installed there) that merely emit a message to the user
to encourage migration to pip and suggest, if needed, to use pip to install
setuptools to replace the placeholder easy_install.  Either approach
would be a huge usability improvement over the situation today.

(The actual implementation details for either approach are slightly more
complicated for OS X framework builds like the python.org installers
as the actual scripts directory is a ``bin`` directory within the
framework itself with optional symlinks from ``/usr/local/bin`` to
the fw bin directory for specific commands.  The installer will now need
to also install symlinks in ``/usr/local/bin`` for ``pip`` and, as
proposed above, for ``easy_install``.)

> A common source of Python installations are through downstream distributors
> such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX
> package managers [#homebrew]_, or Python-specific tools [#conda]_. In order

If you are going to call out Homebrew, you should include the other major
OS X package managers, MacPorts and Fink.

> .. [#homebrew] `Homebrew  <http://brew.sh/>`

 .. [#macports] `MacPorts  <http://macports.org>`
 .. [#fink] `Fink <http://finkproject.org>`

One final comment is that the PEP does not go into any detail on how it
will be implemented.  As it stands, there is a fair amount of one-time
work, including implementing ensurepip, changes to the Windows installer,
changes to the OS X installer, documentation changes, all with testing and
backporting to 2.7.x and 3.3.x.  Then there are the on-going process changes
for all future releases, impacting all release team members, which will need
to be documented in PEP 101.  Do we have an understanding of who is do the
big pieces and by when?  For the record, I am willing to do the extra
one-time and ongoing work for the OS X installers but it would be helpful
to know how we are going to get it all done in time for upcoming releases.

-- 
 Ned Deily,
 nad at acm.org


From donald at stufft.io  Tue Sep 24 01:39:53 2013
From: donald at stufft.io (Donald Stufft)
Date: Mon, 23 Sep 2013 19:39:53 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <nad-F39A9C.16345423092013@news.gmane.org>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <nad-F39A9C.16345423092013@news.gmane.org>
Message-ID: <C9FCE644-3132-4B99-B4CF-1927D7B565EE@stufft.io>


On Sep 23, 2013, at 7:34 PM, Ned Deily <nad at acm.org> wrote:

> One final comment is that the PEP does not go into any detail on how it
> will be implemented.  As it stands, there is a fair amount of one-time
> work, including implementing ensurepip, changes to the Windows installer,
> changes to the OS X installer, documentation changes, all with testing and
> backporting to 2.7.x and 3.3.x.  Then there are the on-going process changes
> for all future releases, impacting all release team members, which will need
> to be documented in PEP 101.  Do we have an understanding of who is do the
> big pieces and by when?  For the record, I am willing to do the extra
> one-time and ongoing work for the OS X installers but it would be helpful
> to know how we are going to get it all done in time for upcoming releases.

I'm in a meeting right now so I only skimmed this email, but I wanted to mention
here that I've been assuming I would be writing the python portions of this. I don't
know how to do the OSX/Window installer pieces and I was hoping that the
relevant people would be at least willing to help me figure it out or would implement
the parts that need to be done inside the installers (mostly running the command).

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130923/16ddab36/attachment.sig>

From donald at stufft.io  Tue Sep 24 02:12:01 2013
From: donald at stufft.io (Donald Stufft)
Date: Mon, 23 Sep 2013 20:12:01 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <nad-F39A9C.16345423092013@news.gmane.org>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <nad-F39A9C.16345423092013@news.gmane.org>
Message-ID: <44F4E1F8-5533-45A7-810E-B9C13530E9DD@stufft.io>


On Sep 23, 2013, at 7:34 PM, Ned Deily <nad at acm.org> wrote:

> In general, I think this is a very important usability feature and I
> am in favor of the general approach.  Good work, all!  I do have some
> comments, primarily about items that are not currently addressed.
> 
>> ``ensurepip`` itself (including the private copy of ``pip`` and its
>> dependencies) will still be installed normally (as it is a regular
>> part of the standard library), only the implicit installation of pip and
>> its dependencies will be skipped.
> 
> The PEP should address the license implications of this since the PEP will
> add the source of pip and setuptools to all new Python releases, both
> source and binary installers releases.  For Python itself, we go to some
> trouble to ensure that all contributions are contributed under a suitable
> license.  What steps are needed, initially and on-going, to ensure that
> these third-party items to be pulled in at each release are compatible
> with the license of Python itself?  Who will do this: release managers?
> Are updates needed in the internal and external documentation for releases
> (README, LICENSE)?  I took a quick look at the current source releases for
> both pip and setuptools.  I noted a reference to the LGPL for pip's CA
> certs bundle; I didn't see a license file for setuptools.

As far as I know the certificate bundle is licensed under either GPL, MPL,
or LGPL. However there is debate about wether a CA bundle *can* be
licensed at all (see https://github.com/pypa/pip/pull/971).

Pip also includes some code taken from other libraries however everything
is licensed as either 1, 2, or 3 clause BSD (besides the aforementioned
certificates). I can't think of us being ok with adding a copyleft license
such as GPL so code we bring in will likely be restricted to things like BSD.

> 
>> This PEP proposes that, if pip still requires it as a dependency,
>> ``ensurepip`` will include a private copy of ``setuptools`` (in addition
>> to the private copy of ``ensurepip``). ``python -m ensurepip`` will then
>> install the private copy in addition to installing ``pip`` itself.
>> 
>> However, this behavior is officially considered an implementation
>> detail. Other projects which explicitly require ``setuptools`` must still
>> provide an appropriate dependency declaration, rather than assuming
>> ``setuptools`` will always be installed alongside ``pip``.
>> 
>> Once pip is able to run ``pip install --upgrade pip`` without needing
>> ``setuptools`` installed first, then the private copy of ``setuptools``
>> will be removed from ``ensurepip`` in subsequent CPython releases.
> 
> The PEP does not sufficiently address the issue of the setuptools-provided
> easy_install command.  I think it is important to do that for several
> reasons:
> 
> 1. The PEP should make clear whether the easy_install scripts are or
> are not installed, globally or in a venv, as a side effect of running
> ensurepip, at least while pip has the implicit private dependency
> on setuptools.  By default, the scripts are installed so ensurepip
> would need to take action to prevent this if that is not desired.
> 
> 2. Despite the current strong preference for using pip as a command
> line installer, the fact is that much third-party documentation still refers
> to and recommends using easy_install for their projects.  That is not
> going to change overnight.  The PEP should identify some steps needed to
> facilitate the migration, like perhaps adding something to the "Installing
> Python Modules" documentation explicitly deprecating easy_install use
> and/or giving pip equivalents for the most common easy_install
> scenarios (or linking to other documentation that does so).
> 
> 3. A specific issue for OS X users is that, as part of OS X, Apple ships
> versions of Python 2 and various third-party packages, including
> setuptools (older, pre-reunification versions, at that) but not pip.
> That means that OS X ships with easy_install and easy_install-2.n
> commands in /usr/bin, including 2.7 in recent OS X releases.  This has
> been an ongoing source of confusion for OS X users who install a newer
> version of Python.  Depending on the distributor and/or source build
> configure options, the new version of Python will be installed to a
> different path, like /usr/local/bin.  Which instance is used
> is generally managed by direct PATH environment manipulations (or by
> launcher programs similar to the Windows py launcher).  If setuptools
> is not installed using the new instance, users invoking the easy_install
> command will by default end up with distributions being installed to a
> system Python rather than to the new Python.  This leads to the
> unfortunately common scenario of:
> 
>    sudo easy_install blah # --> installs to system Python 2.7
>    python -c 'import blah' # --> fails because using newer Python 2.7
> 
> This may very well be the most common current usability problem today
> with python.org (and other third-party) Pythons on OS X.
> 
> With this PEP, we have an opportunity to fix this problem in one of
> two ways.  As long as pip requires setuptools and we do not inhibit
> the installation of the easy_install scripts is to the same directory
> as the python command itself, the new easy_install scripts will
> shadow the Apple-supplied ones just as the python commands do.  If it
> is decided to not install the easy_install scripts now or in the
> future (e.g. when pip no longer depends on setuptools),
> ensurepip could install dummy easy_install scripts (if there
> are none already installed there) that merely emit a message to the user
> to encourage migration to pip and suggest, if needed, to use pip to install
> setuptools to replace the placeholder easy_install.  Either approach
> would be a huge usability improvement over the situation today.
> 
> (The actual implementation details for either approach are slightly more
> complicated for OS X framework builds like the python.org installers
> as the actual scripts directory is a ``bin`` directory within the
> framework itself with optional symlinks from ``/usr/local/bin`` to
> the fw bin directory for specific commands.  The installer will now need
> to also install symlinks in ``/usr/local/bin`` for ``pip`` and, as
> proposed above, for ``easy_install``.)

So this is an interesting question. On one hand I would say we shouldn't be
installing easy_install as that feels very user facing to me and the fact that
setuptools is being installed is an implementation detail. On the other hand
if we put in stub commands that just simply direct the user to use pip then
people who *want* to use easy_install (perhaps for Egg support) won't be
able too (unless perhaps something is released on PyPI that restores the
easy_install commands?)


> 
>> A common source of Python installations are through downstream distributors
>> such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX
>> package managers [#homebrew]_, or Python-specific tools [#conda]_. In order
> 
> If you are going to call out Homebrew, you should include the other major
> OS X package managers, MacPorts and Fink.
> 
>> .. [#homebrew] `Homebrew  <http://brew.sh/>`
> 
> .. [#macports] `MacPorts  <http://macports.org>`
> .. [#fink] `Fink <http://finkproject.org>`

I can add them, I added homebrew because it's what I use :) Didn't really
think to hard about being super inclusive.

> 
> One final comment is that the PEP does not go into any detail on how it
> will be implemented.  As it stands, there is a fair amount of one-time
> work, including implementing ensurepip, changes to the Windows installer,
> changes to the OS X installer, documentation changes, all with testing and
> backporting to 2.7.x and 3.3.x.  Then there are the on-going process changes
> for all future releases, impacting all release team members, which will need
> to be documented in PEP 101.  Do we have an understanding of who is do the
> big pieces and by when?  For the record, I am willing to do the extra
> one-time and ongoing work for the OS X installers but it would be helpful
> to know how we are going to get it all done in time for upcoming releases.

To further expand upon my original answer, I'm volunteering to do the initial
work on the ensurepip library, the scripts that will automated the ongoing
maintenance work, and the back porting of both of the above. I can also attempt
to do the OSX Installer and Windows installer but I have zero idea how the
installers actually function so it's probably better for someone else to do that.

Since it sounds like you're willing to do the work for the OSX installer that
saves me from having to flail around trying to figure out how to do that part,
so hopefully MvL or someone can do the same with the Windows installer.

I'm not sure what needs done outside of the up front work, do I just propose
changes to PEP 101? Do I make a whole new PEP? Is there more than
just updating PEP 101?


> 
> -- 
> Ned Deily,
> nad at acm.org
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130923/2f69b4a1/attachment-0001.sig>

From donald at stufft.io  Tue Sep 24 02:35:11 2013
From: donald at stufft.io (Donald Stufft)
Date: Mon, 23 Sep 2013 20:35:11 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <44F4E1F8-5533-45A7-810E-B9C13530E9DD@stufft.io>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <nad-F39A9C.16345423092013@news.gmane.org>
 <44F4E1F8-5533-45A7-810E-B9C13530E9DD@stufft.io>
Message-ID: <A897426D-A70B-4AD8-B775-E456207E24F0@stufft.io>


On Sep 23, 2013, at 8:12 PM, Donald Stufft <donald at stufft.io> wrote:

>> 
>> 
>>> A common source of Python installations are through downstream distributors
>>> such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX
>>> package managers [#homebrew]_, or Python-specific tools [#conda]_. In order
>> 
>> If you are going to call out Homebrew, you should include the other major
>> OS X package managers, MacPorts and Fink.
>> 
>>> .. [#homebrew] `Homebrew  <http://brew.sh/>`
>> 
>> .. [#macports] `MacPorts  <http://macports.org>`
>> .. [#fink] `Fink <http://finkproject.org>`
> 
> I can add them, I added homebrew because it's what I use :) Didn't really
> think to hard about being super inclusive.

Updated this in PEP453.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130923/0a9d3e5d/attachment.sig>

From stephen at xemacs.org  Tue Sep 24 02:50:15 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Tue, 24 Sep 2013 09:50:15 +0900
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute
	error	message
In-Reply-To: <5240C8FC.8070608@mrabarnett.plus.com>
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net>
 <l1lbkl$k0g$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
 <l1n9o6$ap3$1@ger.gmane.org>
 <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
 <l1nkdh$lh7$1@ger.gmane.org>
 <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>
 <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com>
 <523FDB8F.1080207@canterbury.ac.nz>
 <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com>
 <20130923104500.5645fdbb@pitrou.net>
 <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com>
 <20130923172245.0cbaec81@fsol>
 <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com>
 <5240C8FC.8070608@mrabarnett.plus.com>
Message-ID: <87zjr311rs.fsf@uwakimon.sk.tsukuba.ac.jp>

MRAB writes:

 > > The word doesn't literally mean the exception itself was unraisable. It
 > > means it was raised, we caught it and we're writing it to stderr because
 > > we *can't raise it again*.

 > Ah, you mean "unreraisable". :-)

+1

Ugly as sin, but satisfies all other criteria (except for Antoine's "easily
understandable" which simply can't be satisfied).

From steve at pearwood.info  Tue Sep 24 03:31:56 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 24 Sep 2013 11:31:56 +1000
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <CAP7h-xZ-E_3--JrQGE+htXZStagmBjHmozW6v6jEbiXUicQ8fQ@mail.gmail.com>
References: <20130922031304.GE19939@ando>
 <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
 <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com>
 <20130923003142.GO19939@ando>
 <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com>
 <l1o8ri$mjv$1@ger.gmane.org>
 <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com>
 <l1q36s$99l$1@ger.gmane.org>
 <CAP7h-xZ-E_3--JrQGE+htXZStagmBjHmozW6v6jEbiXUicQ8fQ@mail.gmail.com>
Message-ID: <20130924013155.GD7989@ando>

This is getting off-topic, if you're not interested in English grammar 
you should stop reading.

On Mon, Sep 23, 2013 at 03:18:01PM -0400, Alexander Belopolsky wrote:

> I don't think "Returns bar." is a valid English sentence because it lacks
> subject. 

Subjectless sentences are unusual in English, but you do see them. 
Sentences consisting of only an interjection are subjectless: 

"Ouch!" "Hear hear!" "Rubbish!" "Oh dear!"

Imperative sentences often have no explicit subject:

"Close the door." "Put that light out!" "Follow me."

Conversational English (especially spoken English) often displays the 
phenomenon called "Conversational Deletion", where the beginning of 
sentences are eroded away, dropping (e.g.) possessives, articles, and 
subject nouns.

"Hope this helps." "See you next week." "No need to get upset!"

In the example given, "Returns bar", I would identify this as an example 
of conversational deletion. The full sentence would be "This function 
returns bar". Personally, I don't mind such a conversational style, 
although many people consider it too informal for written English, even 
docstrings :-)


> I would not mind
> 
> def foo():
>       """returns bar"""
> 
> which I would read as "Function foo() returns bar," but in this case
> "returns" should be in lower case.

I certainly don't like that. Sentences, even eroded sentences, start 
with capital letters in English. Unless you are the poet e.e. cummings, 
capital letters are non-negotiable.


-- 
Steven

From alexander.belopolsky at gmail.com  Tue Sep 24 03:59:55 2013
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Mon, 23 Sep 2013 21:59:55 -0400
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <20130924013155.GD7989@ando>
References: <20130922031304.GE19939@ando>
 <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
 <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com>
 <20130923003142.GO19939@ando>
 <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com>
 <l1o8ri$mjv$1@ger.gmane.org>
 <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com>
 <l1q36s$99l$1@ger.gmane.org>
 <CAP7h-xZ-E_3--JrQGE+htXZStagmBjHmozW6v6jEbiXUicQ8fQ@mail.gmail.com>
 <20130924013155.GD7989@ando>
Message-ID: <CAP7h-xaDEMaCj9e=nX5e2j+Qw8GGMEwVGPLMY7OBoqcKJShRVg@mail.gmail.com>

Given near-consensus on the "Return ..." form, the following discussion is
of academic interest only.

On Mon, Sep 23, 2013 at 9:31 PM, Steven D'Aprano <steve at pearwood.info>wrote:

> > I would not mind
> >
> > def foo():
> >       """returns bar"""
> >
> > which I would read as "Function foo() returns bar," but in this case
> > "returns" should be in lower case.
>
> I certainly don't like that. Sentences, even eroded sentences, start
> with capital letters in English. Unless you are the poet e.e. cummings,
> capital letters are non-negotiable.


I still believe "Returns bar." is a non-sentence even if "Return bar!" is.
 While absence of subject noun in itself may not be an issue, but combined
with a singular third-person verb form, it makes "Returns bar."
grammatically incorrect.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130923/5de51848/attachment.html>

From nad at acm.org  Tue Sep 24 06:32:37 2013
From: nad at acm.org (Ned Deily)
Date: Mon, 23 Sep 2013 21:32:37 -0700
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <nad-F39A9C.16345423092013@news.gmane.org>
 <44F4E1F8-5533-45A7-810E-B9C13530E9DD@stufft.io>
Message-ID: <nad-AC1D3B.21323723092013@news.gmane.org>

In article <44F4E1F8-5533-45A7-810E-B9C13530E9DD at stufft.io>,
 Donald Stufft <donald at stufft.io> wrote:
> On Sep 23, 2013, at 7:34 PM, Ned Deily <nad at acm.org> wrote:
> > [... license implications ...]
> 
> As far as I know the certificate bundle is licensed under either GPL, MPL,
> or LGPL. However there is debate about wether a CA bundle *can* be
> licensed at all (see https://github.com/pypa/pip/pull/971).
> 
> Pip also includes some code taken from other libraries however everything
> is licensed as either 1, 2, or 3 clause BSD (besides the aforementioned
> certificates). I can't think of us being ok with adding a copyleft license
> such as GPL so code we bring in will likely be restricted to things like BSD.

IANAL, but I think it would be good if, at least, the setuptools license were 
clearer and the LGPL reference for the cert bundle was changed.  It *might* be 
a good idea to get an opinion from Van Lindberg.  But I'm happy to defer to 
Martin's judgement on this.

>  > [... easy_install on OS X ...]
> 
> So this is an interesting question. On one hand I would say we shouldn't be
> installing easy_install as that feels very user facing to me and the fact 
> that
> setuptools is being installed is an implementation detail. On the other hand
> if we put in stub commands that just simply direct the user to use pip then
> people who *want* to use easy_install (perhaps for Egg support) won't be
> able too (unless perhaps something is released on PyPI that restores the
> easy_install commands?)

I was thinking that, in the latter case, those users who really want to use 
easy_install could be told to just use pip to install a PyPI version of 
setuptools which would replace the stub commands with the real things - 
assuming that works.

> > [... implementation plan ...]
> 
> To further expand upon my original answer, I'm volunteering to do the initial
> work on the ensurepip library, the scripts that will automated the ongoing
> maintenance work, and the back porting of both of the above. I can also 
> attempt
> to do the OSX Installer and Windows installer but I have zero idea how the
> installers actually function so it's probably better for someone else to do 
> that.
> 
> Since it sounds like you're willing to do the work for the OSX installer that
> saves me from having to flail around trying to figure out how to do that 
> part,
> so hopefully MvL or someone can do the same with the Windows installer.
> 
> I'm not sure what needs done outside of the up front work, do I just propose
> changes to PEP 101? Do I make a whole new PEP? Is there more than
> just updating PEP 101?

I think the thing to do is get review buy-in from the release managers for the 
affected active releases (2.7.x = Benjamin, 3.3.x = Georg, 3.4.x = Larry) and 
let them worry about updating PEP 101.  The key point is that this PEP is 
implicitly adding some new responsibilities for the release team; I think they 
just need to be explicit.

-- 
 Ned Deily,
 nad at acm.org


From donald at stufft.io  Tue Sep 24 06:35:05 2013
From: donald at stufft.io (Donald Stufft)
Date: Tue, 24 Sep 2013 00:35:05 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <nad-AC1D3B.21323723092013@news.gmane.org>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <nad-F39A9C.16345423092013@news.gmane.org>
 <44F4E1F8-5533-45A7-810E-B9C13530E9DD@stufft.io>
 <nad-AC1D3B.21323723092013@news.gmane.org>
Message-ID: <F6BE0C28-86D2-4F25-9F11-93AB0FC3C067@stufft.io>


On Sep 24, 2013, at 12:32 AM, Ned Deily <nad at acm.org> wrote:

> IANAL, but I think it would be good if, at least, the setuptools license were 
> clearer and the LGPL reference for the cert bundle was changed.  It *might* be 
> a good idea to get an opinion from Van Lindberg.  But I'm happy to defer to 
> Martin's judgement on this.

After your concern was raised I went ahead and emailed VanL.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130924/5ca2f486/attachment.sig>

From ncoghlan at gmail.com  Tue Sep 24 09:25:10 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 24 Sep 2013 17:25:10 +1000
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <20130924002955.49a99f9f@fsol>
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net>
 <l1lbkl$k0g$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
 <l1n9o6$ap3$1@ger.gmane.org>
 <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
 <l1nkdh$lh7$1@ger.gmane.org>
 <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>
 <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com>
 <523FDB8F.1080207@canterbury.ac.nz>
 <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com>
 <20130923104500.5645fdbb@pitrou.net>
 <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com>
 <20130923172245.0cbaec81@fsol>
 <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com>
 <20130923233507.4e5ebb9f@fsol> <5240B4E8.6020701@stoneleaf.us>
 <20130924002955.49a99f9f@fsol>
Message-ID: <CADiSq7euer--7omtAdA=gWi_BOV2YWE-TgwEsbdRZA7xveQUdw@mail.gmail.com>

On 24 September 2013 08:29, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Mon, 23 Sep 2013 14:38:48 -0700
> Ethan Furman <ethan at stoneleaf.us> wrote:
>> > But that's because you already know what it's supposed to convey. The
>> > average user doesn't, and only sees "unraisable".
>>
>> All the more reason to have text in the error message that is easily searchable.
>
> Then I propose "CA6yuaYV6dygPfJRxUrhtg". It should be easily
> searchable ;-)
>
> Seriously, "easily searchable" is a rather weak criterion for an
> error message. It should be easily understandable and non-misleading
> first.

You are setting the bar unreasonably high for an error message that
has to convey a complex concept in as few words as possible. There is
*NO* wording that can concisely express the concepts involved without
resorting to jargon, because the concepts behind it are *complex and
unintuitive*. The current wording is flat out wrong, because the
exception isn't being ignored, it's being printed to stderr. If it was
genuinely being ignored, people wouldn't complain about it.

Jargon that can be easily looked up with a search engine is greatly
superior to a message that is simply wrong, as the former provides a
gateway to understanding, just like coming across a word you don't
understand when reading a novel. Preferring the status quo because
you're holding out a forlorn hope for a concise wording that explains:

- there are places where exceptions may occur but the interpreter
can't reraise them
- this is one of those cases, so we're printing it to stderr instead

*without users needing to do any research*.

A verbose wording is no good either, as that degenerates into a wall
of text that people will *still* have to extract pieces from to plug
into a search engine to figure out what they mean.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Tue Sep 24 09:29:59 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 24 Sep 2013 17:29:59 +1000
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <87zjr311rs.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net>
 <l1lbkl$k0g$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
 <l1n9o6$ap3$1@ger.gmane.org>
 <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
 <l1nkdh$lh7$1@ger.gmane.org>
 <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>
 <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com>
 <523FDB8F.1080207@canterbury.ac.nz>
 <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com>
 <20130923104500.5645fdbb@pitrou.net>
 <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com>
 <20130923172245.0cbaec81@fsol>
 <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com>
 <5240C8FC.8070608@mrabarnett.plus.com>
 <87zjr311rs.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CADiSq7erOnT=PkjN3L__3teOZDjQtV3XU4sEOTTVs0dwsqcp5Q@mail.gmail.com>

On 24 September 2013 10:50, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> MRAB writes:
>
>  > > The word doesn't literally mean the exception itself was unraisable. It
>  > > means it was raised, we caught it and we're writing it to stderr because
>  > > we *can't raise it again*.
>
>  > Ah, you mean "unreraisable". :-)
>
> +1
>
> Ugly as sin, but satisfies all other criteria (except for Antoine's "easily
> understandable" which simply can't be satisfied).

If you're drawing a distinction between the first time an exception
hits the eval loop and the second and subsequent times, then neither
"Unraisable" nor "Unreraisable" is 100% correct. I just think it's a
meaningless distinction, which is why I favour "Unraisable" - at the
point in time where that message is displayed, that exception cannot
be raised any further, whether it was created directly from C or was
received from an underlying call back into Python code..

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Tue Sep 24 09:32:15 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 24 Sep 2013 17:32:15 +1000
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <CADiSq7euer--7omtAdA=gWi_BOV2YWE-TgwEsbdRZA7xveQUdw@mail.gmail.com>
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net>
 <l1lbkl$k0g$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
 <l1n9o6$ap3$1@ger.gmane.org>
 <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
 <l1nkdh$lh7$1@ger.gmane.org>
 <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>
 <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com>
 <523FDB8F.1080207@canterbury.ac.nz>
 <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com>
 <20130923104500.5645fdbb@pitrou.net>
 <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com>
 <20130923172245.0cbaec81@fsol>
 <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com>
 <20130923233507.4e5ebb9f@fsol> <5240B4E8.6020701@stoneleaf.us>
 <20130924002955.49a99f9f@fsol>
 <CADiSq7euer--7omtAdA=gWi_BOV2YWE-TgwEsbdRZA7xveQUdw@mail.gmail.com>
Message-ID: <CADiSq7ehgCz7K=9BJbhNp5J4mimmZPoGOoU6_E+ogKipN3dQkw@mail.gmail.com>

On 24 September 2013 17:25, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Preferring the status quo because
> you're holding out a forlorn hope for a concise wording that explains:
>
> - there are places where exceptions may occur but the interpreter
> can't reraise them
> - this is one of those cases, so we're printing it to stderr instead
>
> *without users needing to do any research*.

Oops, reworded the second part of that sentence without fixing the
first part. It should have said:

==============
It doesn't make sense to prefer the status quo because you're holding
out a forlorn hope for
a concise wording that explains:

- there are places where exceptions may occur but the interpreter
can't reraise them
- this is one of those cases, so we're printing it to stderr instead

*without users needing to do any research*.
==============

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Tue Sep 24 09:34:17 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 24 Sep 2013 09:34:17 +0200
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <CADiSq7euer--7omtAdA=gWi_BOV2YWE-TgwEsbdRZA7xveQUdw@mail.gmail.com>
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net>
 <l1lbkl$k0g$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
 <l1n9o6$ap3$1@ger.gmane.org>
 <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
 <l1nkdh$lh7$1@ger.gmane.org>
 <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>
 <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com>
 <523FDB8F.1080207@canterbury.ac.nz>
 <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com>
 <20130923104500.5645fdbb@pitrou.net>
 <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com>
 <20130923172245.0cbaec81@fsol>
 <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com>
 <20130923233507.4e5ebb9f@fsol> <5240B4E8.6020701@stoneleaf.us>
 <20130924002955.49a99f9f@fsol>
 <CADiSq7euer--7omtAdA=gWi_BOV2YWE-TgwEsbdRZA7xveQUdw@mail.gmail.com>
Message-ID: <20130924093417.34abbc42@fsol>

On Tue, 24 Sep 2013 17:25:10 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> You are setting the bar unreasonably high for an error message that
> has to convey a complex concept in as few words as possible. There is
> *NO* wording that can concisely express the concepts involved without
> resorting to jargon, because the concepts behind it are *complex and
> unintuitive*. The current wording is flat out wrong, because the
> exception isn't being ignored, it's being printed to stderr. If it was
> genuinely being ignored, people wouldn't complain about it.
> 
> Jargon that can be easily looked up with a search engine is greatly
> superior to a message that is simply wrong, as the former provides a
> gateway to understanding, just like coming across a word you don't
> understand when reading a novel.

"Unraisable" is not a word I don't understand, it's a word that I
understand and which conveys the wrong meaning.

If you want something that people won't understand, you can use
something like "asynchronous exception".

> Preferring the status quo because
> you're holding out a forlorn hope for a concise wording that explains:

I've proposed other options.

Regards

Antoine.

From ncoghlan at gmail.com  Tue Sep 24 09:52:32 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 24 Sep 2013 17:52:32 +1000
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <nad-F39A9C.16345423092013@news.gmane.org>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <nad-F39A9C.16345423092013@news.gmane.org>
Message-ID: <CADiSq7cuAYtpwqh7Z2R7n9ButRt7-4PP9_oFrk69feVnWOCiSA@mail.gmail.com>

On 24 September 2013 09:34, Ned Deily <nad at acm.org> wrote:
> In general, I think this is a very important usability feature and I
> am in favor of the general approach.  Good work, all!  I do have some
> comments, primarily about items that are not currently addressed.

Your reply and Barry's suggest that Betteridge's law [1] applies to
email subject lines, too ;)

As far as easy_install goes, my current plan was actually to tackle
that on the upstream side. If pip still depends on setuptools by the
time of the Python 3.4 release, then it will depend on the *real*
setuptools, easy_install and all. From my perspective, one golden rule
of this integration is that we do *not* mess with the contents of the
wheel files for pip and its dependencies - they're pristine upstream
releases. This is mostly for technical reasons, but it also draws a
sharp line of demarcation for any "aggregation or derivation?"
questions, too.

If pip has been updated by the time of its inclusion to depend on a
cut down setuptools derivative that omits easy_install (or pip has
switched to its own internal replacements instead), so much the
better, but I consider that to be essentially independent of the
CPython bundling situation, since it isn't something we have direct
control over, and I consider the slight downside of potentially
installing easy_install alongside pip to be dwarfed by the benefits of
installing pip.

As far as I am aware, the licensing on setuptools is currently limited
to the "ZPL or PSF" declaration in the distribution metadata.

Cheers,
Nick.

[1] https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines



-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Tue Sep 24 10:06:15 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 24 Sep 2013 18:06:15 +1000
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <20130924093417.34abbc42@fsol>
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net>
 <l1lbkl$k0g$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
 <l1n9o6$ap3$1@ger.gmane.org>
 <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
 <l1nkdh$lh7$1@ger.gmane.org>
 <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>
 <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com>
 <523FDB8F.1080207@canterbury.ac.nz>
 <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com>
 <20130923104500.5645fdbb@pitrou.net>
 <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com>
 <20130923172245.0cbaec81@fsol>
 <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com>
 <20130923233507.4e5ebb9f@fsol> <5240B4E8.6020701@stoneleaf.us>
 <20130924002955.49a99f9f@fsol>
 <CADiSq7euer--7omtAdA=gWi_BOV2YWE-TgwEsbdRZA7xveQUdw@mail.gmail.com>
 <20130924093417.34abbc42@fsol>
Message-ID: <CADiSq7dQ=YcEiTea3GrrQJ64ypBDCH4sN+fHJirPL4hjkep5FA@mail.gmail.com>

On 24 September 2013 17:34, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Tue, 24 Sep 2013 17:25:10 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> You are setting the bar unreasonably high for an error message that
>> has to convey a complex concept in as few words as possible. There is
>> *NO* wording that can concisely express the concepts involved without
>> resorting to jargon, because the concepts behind it are *complex and
>> unintuitive*. The current wording is flat out wrong, because the
>> exception isn't being ignored, it's being printed to stderr. If it was
>> genuinely being ignored, people wouldn't complain about it.
>>
>> Jargon that can be easily looked up with a search engine is greatly
>> superior to a message that is simply wrong, as the former provides a
>> gateway to understanding, just like coming across a word you don't
>> understand when reading a novel.
>
> "Unraisable" is not a word I don't understand, it's a word that I
> understand and which conveys the wrong meaning.

How is it wrong? At the point where the interpreter says "This
exception is now unraisable", what, precisely, is it saying that is
wrong?

It isn't saying "this has never been raised". It is saying, "where it
is currently being processed, this exception cannot be raised".

> If you want something that people won't understand, you can use
> something like "asynchronous exception".

Asynchronous exception is *even more* wrong, because that's the
terminology used for an exception injected into the current thread by
a different thread.

>> Preferring the status quo because
>> you're holding out a forlorn hope for a concise wording that explains:
>
> I've proposed other options.

"Automatically caught" says nothing about why the exception is being
printed to stderr instead of propagating normally. Exceptions are
automatically caught by any matching except clause all the time, but
most of those don't result in errors printed to stderr.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Tue Sep 24 10:16:11 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 24 Sep 2013 10:16:11 +0200
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <CADiSq7dQ=YcEiTea3GrrQJ64ypBDCH4sN+fHJirPL4hjkep5FA@mail.gmail.com>
References: <l1l2bm$1bp$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
 <l1n9o6$ap3$1@ger.gmane.org>
 <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
 <l1nkdh$lh7$1@ger.gmane.org>
 <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>
 <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com>
 <523FDB8F.1080207@canterbury.ac.nz>
 <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com>
 <20130923104500.5645fdbb@pitrou.net>
 <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com>
 <20130923172245.0cbaec81@fsol>
 <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com>
 <20130923233507.4e5ebb9f@fsol> <5240B4E8.6020701@stoneleaf.us>
 <20130924002955.49a99f9f@fsol>
 <CADiSq7euer--7omtAdA=gWi_BOV2YWE-TgwEsbdRZA7xveQUdw@mail.gmail.com>
 <20130924093417.34abbc42@fsol>
 <CADiSq7dQ=YcEiTea3GrrQJ64ypBDCH4sN+fHJirPL4hjkep5FA@mail.gmail.com>
Message-ID: <20130924101611.5526f552@fsol>

On Tue, 24 Sep 2013 18:06:15 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> How is it wrong? At the point where the interpreter says "This
> exception is now unraisable", what, precisely, is it saying that is
> wrong?
> It isn't saying "this has never been raised". It is saying, "where it
> is currently being processed, this exception cannot be raised".

Well, it is saying it. If it's conceptually unraisable, it can't be
raised. I know your point is that it is only unraisable *now*, but
that's not the intuitive interpretation.

> >> Preferring the status quo because
> >> you're holding out a forlorn hope for a concise wording that explains:
> >
> > I've proposed other options.
> 
> "Automatically caught" says nothing about why the exception is being
> printed to stderr instead of propagating normally. Exceptions are
> automatically caught by any matching except clause all the time, but
> most of those don't result in errors printed to stderr.

I would say they are manually caught by except clauses (user code ==
manual intervention), and automatically caught when "silenced" or
"unraised" by the interpreter.

Regards

Antoine.

From python at mrabarnett.plus.com  Tue Sep 24 11:58:54 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 24 Sep 2013 10:58:54 +0100
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <CADiSq7dQ=YcEiTea3GrrQJ64ypBDCH4sN+fHJirPL4hjkep5FA@mail.gmail.com>
References: <l1l2bm$1bp$1@ger.gmane.org> <l1lbkl$k0g$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
 <l1n9o6$ap3$1@ger.gmane.org>
 <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
 <l1nkdh$lh7$1@ger.gmane.org>
 <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>
 <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com>
 <523FDB8F.1080207@canterbury.ac.nz>
 <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com>
 <20130923104500.5645fdbb@pitrou.net>
 <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com>
 <20130923172245.0cbaec81@fsol>
 <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com>
 <20130923233507.4e5ebb9f@fsol> <5240B4E8.6020701@stoneleaf.us>
 <20130924002955.49a99f9f@fsol>
 <CADiSq7euer--7omtAdA=gWi_BOV2YWE-TgwEsbdRZA7xveQUdw@mail.gmail.com>
 <20130924093417.34abbc42@fsol>
 <CADiSq7dQ=YcEiTea3GrrQJ64ypBDCH4sN+fHJirPL4hjkep5FA@mail.gmail.com>
Message-ID: <5241625E.7030701@mrabarnett.plus.com>

On 24/09/2013 09:06, Nick Coghlan wrote:
> On 24 September 2013 17:34, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> On Tue, 24 Sep 2013 17:25:10 +1000
>> Nick Coghlan <ncoghlan at gmail.com> wrote:
>>>
>>> You are setting the bar unreasonably high for an error message that
>>> has to convey a complex concept in as few words as possible. There is
>>> *NO* wording that can concisely express the concepts involved without
>>> resorting to jargon, because the concepts behind it are *complex and
>>> unintuitive*. The current wording is flat out wrong, because the
>>> exception isn't being ignored, it's being printed to stderr. If it was
>>> genuinely being ignored, people wouldn't complain about it.
>>>
>>> Jargon that can be easily looked up with a search engine is greatly
>>> superior to a message that is simply wrong, as the former provides a
>>> gateway to understanding, just like coming across a word you don't
>>> understand when reading a novel.
>>
>> "Unraisable" is not a word I don't understand, it's a word that I
>> understand and which conveys the wrong meaning.
>
> How is it wrong? At the point where the interpreter says "This
> exception is now unraisable", what, precisely, is it saying that is
> wrong?
>
> It isn't saying "this has never been raised". It is saying, "where it
> is currently being processed, this exception cannot be raised".
>
>> If you want something that people won't understand, you can use
>> something like "asynchronous exception".
>
> Asynchronous exception is *even more* wrong, because that's the
> terminology used for an exception injected into the current thread by
> a different thread.
>
>>> Preferring the status quo because
>>> you're holding out a forlorn hope for a concise wording that explains:
>>
>> I've proposed other options.
>
> "Automatically caught" says nothing about why the exception is being
> printed to stderr instead of propagating normally. Exceptions are
> automatically caught by any matching except clause all the time, but
> most of those don't result in errors printed to stderr.
>
Why not just say something like "Cannot propagate exception..."; it's
simpler than "Unpropagatable exception...".

From walter at livinglogic.de  Tue Sep 24 12:21:45 2013
From: walter at livinglogic.de (=?UTF-8?B?V2FsdGVyIETDtnJ3YWxk?=)
Date: Tue, 24 Sep 2013 12:21:45 +0200
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <CANc-5UzYUtta=4Q3c7n-LXdcYBnWrh5VYdMwSNPJ=5a0HuVfig@mail.gmail.com>
References: <20130922031304.GE19939@ando>	<20130922102046.764abea5@fsol>	<20130922125323.GI19939@ando>	<CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com>	<524025B3.7020305@livinglogic.de>	<CAP7+vJLDo9KkdzzQfnJe_bTzunAP6+gBznY1Dx5pvxBqva0YjQ@mail.gmail.com>
 <CANc-5UzYUtta=4Q3c7n-LXdcYBnWrh5VYdMwSNPJ=5a0HuVfig@mail.gmail.com>
Message-ID: <524167B9.1090004@livinglogic.de>

On 23.09.13 17:18, Skip Montanaro wrote:

>>> It would be great if the docstring contained a link to the online
>>> documentation.
>>
>> That would have to be a feature of help(), not hardcoded in each docstring.
>
> That *is* a feature of the help function:
>
> Help on built-in module sys:
>
>>>> help(sys)
> NAME
>      sys
>
> FILE
>      (built-in)
>
> MODULE DOCS
>      http://docs.python.org/library/sys
> ...
>
> (pydoc too, though I'm 99.9% sure they use the same underlying
> facility Ping originally implemented.)

Hmm, but it doesn't work for functions:

 >>> import sys
 >>> help(sys.settracee)

Help on built-in function settrace in module sys:

settrace(...)
     settrace(function)

     Set the global debug tracing function.  It will be called on each
     function call.  See the debugger chapter in the library manual.

Servus,
    Walter


From ncoghlan at gmail.com  Tue Sep 24 14:51:28 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 24 Sep 2013 22:51:28 +1000
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <5241625E.7030701@mrabarnett.plus.com>
References: <l1l2bm$1bp$1@ger.gmane.org> <l1lbkl$k0g$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
 <l1n9o6$ap3$1@ger.gmane.org>
 <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
 <l1nkdh$lh7$1@ger.gmane.org>
 <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>
 <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com>
 <523FDB8F.1080207@canterbury.ac.nz>
 <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com>
 <20130923104500.5645fdbb@pitrou.net>
 <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com>
 <20130923172245.0cbaec81@fsol>
 <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com>
 <20130923233507.4e5ebb9f@fsol> <5240B4E8.6020701@stoneleaf.us>
 <20130924002955.49a99f9f@fsol>
 <CADiSq7euer--7omtAdA=gWi_BOV2YWE-TgwEsbdRZA7xveQUdw@mail.gmail.com>
 <20130924093417.34abbc42@fsol>
 <CADiSq7dQ=YcEiTea3GrrQJ64ypBDCH4sN+fHJirPL4hjkep5FA@mail.gmail.com>
 <5241625E.7030701@mrabarnett.plus.com>
Message-ID: <CADiSq7eiAkDBe9MwodRD3NDuGn935goYo7OKPu7M_aNe1buvHw@mail.gmail.com>

On 24 Sep 2013 20:06, "MRAB" <python at mrabarnett.plus.com> wrote:
>
> On 24/09/2013 09:06, Nick Coghlan wrote:
>>
>> On 24 September 2013 17:34, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>>
>>> On Tue, 24 Sep 2013 17:25:10 +1000
>>> Nick Coghlan <ncoghlan at gmail.com> wrote:
>>>>
>>>>
>>>> You are setting the bar unreasonably high for an error message that
>>>> has to convey a complex concept in as few words as possible. There is
>>>> *NO* wording that can concisely express the concepts involved without
>>>> resorting to jargon, because the concepts behind it are *complex and
>>>> unintuitive*. The current wording is flat out wrong, because the
>>>> exception isn't being ignored, it's being printed to stderr. If it was
>>>> genuinely being ignored, people wouldn't complain about it.
>>>>
>>>> Jargon that can be easily looked up with a search engine is greatly
>>>> superior to a message that is simply wrong, as the former provides a
>>>> gateway to understanding, just like coming across a word you don't
>>>> understand when reading a novel.
>>>
>>>
>>> "Unraisable" is not a word I don't understand, it's a word that I
>>> understand and which conveys the wrong meaning.
>>
>>
>> How is it wrong? At the point where the interpreter says "This
>> exception is now unraisable", what, precisely, is it saying that is
>> wrong?
>>
>> It isn't saying "this has never been raised". It is saying, "where it
>> is currently being processed, this exception cannot be raised".
>>
>>> If you want something that people won't understand, you can use
>>> something like "asynchronous exception".
>>
>>
>> Asynchronous exception is *even more* wrong, because that's the
>> terminology used for an exception injected into the current thread by
>> a different thread.
>>
>>>> Preferring the status quo because
>>>> you're holding out a forlorn hope for a concise wording that explains:
>>>
>>>
>>> I've proposed other options.
>>
>>
>> "Automatically caught" says nothing about why the exception is being
>> printed to stderr instead of propagating normally. Exceptions are
>> automatically caught by any matching except clause all the time, but
>> most of those don't result in errors printed to stderr.
>>
> Why not just say something like "Cannot propagate exception..."; it's
> simpler than "Unpropagatable exception...".

That would definitely be an improvement on the status quo and avoids
Antoine's concern about an adjective being interpreted as an inherent
property of the exception rather than the circumstances where the exception
was encountered.

Cheers,
Nick.

>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130924/124db00f/attachment.html>

From zuo at chopin.edu.pl  Tue Sep 24 15:53:06 2013
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Tue, 24 Sep 2013 15:53:06 +0200
Subject: [Python-Dev]
 =?utf-8?q?Revert_=2312085_fix_for_=5F=5Fdel=5F=5F_at?=
 =?utf-8?q?tribute_error_message?=
In-Reply-To: <20130924101611.5526f552@fsol>
References: <l1l2bm$1bp$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
 <l1n9o6$ap3$1@ger.gmane.org>
 <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
 <l1nkdh$lh7$1@ger.gmane.org>
 <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>
 <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com>
 <523FDB8F.1080207@canterbury.ac.nz>
 <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com>
 <20130923104500.5645fdbb@pitrou.net>
 <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com>
 <20130923172245.0cbaec81@fsol>
 <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com>
 <20130923233507.4e5ebb9f@fsol> <5240B4E8.6020701@stoneleaf.us>
 <20130924002955.49a99f9f@fsol>
 <CADiSq7euer--7omtAdA=gWi_BOV2YWE-TgwEsbdRZA7xveQUdw@mail.gmail.com>
 <20130924093417.34abbc42@fsol>
 <CADiSq7dQ=YcEiTea3GrrQJ64ypBDCH4sN+fHJirPL4hjkep5FA@mail.gmail.com>
 <20130924101611.5526f552@fsol>
Message-ID: <3e055c69fc6126dc0e4d0a27b9ada354@chopin.edu.pl>

24.09.2013 10:16, Antoine Pitrou wrote:

> On Tue, 24 Sep 2013 18:06:15 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> How is it wrong? At the point where the interpreter says "This
>> exception is now unraisable", what, precisely, is it saying that is
>> wrong?
>> It isn't saying "this has never been raised". It is saying, "where 
>> it
>> is currently being processed, this exception cannot be raised".
>
> Well, it is saying it. If it's conceptually unraisable, it can't be
> raised. I know your point is that it is only unraisable *now*, but
> that's not the intuitive interpretation.

And what about:

     Exception not propagated from <bound method C.__del__
     of <__main__.C object at 0x7f98b8b61538>>
     ...

Or:
     Exception that cannot be propagated from <bound method C.__del__
     of <__main__.C object at 0x7f98b8b61538>>
     ...

Cheers.
*j


From v+python at g.nevcal.com  Tue Sep 24 18:33:21 2013
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Tue, 24 Sep 2013 09:33:21 -0700
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <CADiSq7eiAkDBe9MwodRD3NDuGn935goYo7OKPu7M_aNe1buvHw@mail.gmail.com>
References: <l1l2bm$1bp$1@ger.gmane.org> <l1n9o6$ap3$1@ger.gmane.org>
 <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
 <l1nkdh$lh7$1@ger.gmane.org>
 <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>
 <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com>
 <523FDB8F.1080207@canterbury.ac.nz>
 <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com>
 <20130923104500.5645fdbb@pitrou.net>
 <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com>
 <20130923172245.0cbaec81@fsol>
 <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com>
 <20130923233507.4e5ebb9f@fsol> <5240B4E8.6020701@stoneleaf.us>
 <20130924002955.49a99f9f@fsol>
 <CADiSq7euer--7omtAdA=gWi_BOV2YWE-TgwEsbdRZA7xveQUdw@mail.gmail.com>
 <20130924093417.34abbc42@fsol>
 <CADiSq7dQ=YcEiTea3GrrQJ64ypBDCH4sN+fHJirPL4hjkep5FA@mail.gmail.com>
 <5241625E.7030701@mrabarnett.plus.com>
 <CADiSq7eiAkDBe9MwodRD3NDuGn935goYo7OKPu7M_aNe1buvHw@mail.gmail.com>
Message-ID: <5241BED1.7010905@g.nevcal.com>

On 9/24/2013 5:51 AM, Nick Coghlan wrote:
>
> > Why not just say something like "Cannot propagate exception..."; it's
> > simpler than "Unpropagatable exception...".
>
> That would definitely be an improvement on the status quo and avoids 
> Antoine's concern about an adjective being interpreted as an inherent 
> property of the exception rather than the circumstances where the 
> exception was encountered.
>
First one I've heard that accurately and unambiguously and briefly 
describes the issue.
+1
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130924/9e724b78/attachment.html>

From g.brandl at gmx.net  Wed Sep 25 08:22:27 2013
From: g.brandl at gmx.net (Georg Brandl)
Date: Wed, 25 Sep 2013 08:22:27 +0200
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <5240BCAA.5030604@canterbury.ac.nz>
References: <l1l2bm$1bp$1@ger.gmane.org>
 <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org>
 <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com>
 <l1n9o6$ap3$1@ger.gmane.org>
 <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com>
 <l1nkdh$lh7$1@ger.gmane.org>
 <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com>
 <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com>
 <523FDB8F.1080207@canterbury.ac.nz>
 <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com>
 <20130923104500.5645fdbb@pitrou.net>
 <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com>
 <20130923172245.0cbaec81@fsol> <5240BCAA.5030604@canterbury.ac.nz>
Message-ID: <l1tvd9$6rg$1@ger.gmane.org>

Am 24.09.2013 00:11, schrieb Greg Ewing:
> Antoine Pitrou wrote:
>> Yes, but I agree with Greg that "unraisable" is wrong. After all, it
>> was raised, and it can even be caught by the programmer (inside
>> __del__).
> 
> How about something like "Uncaught exception in __del__
> method ignored"? It explains fairly clearly what has
> happened, and also indicates what do do about it --
> catch it in the __del__ method.

"Exception in __del__ caught and not propagated:"

Georg


From g.brandl at gmx.net  Wed Sep 25 08:30:37 2013
From: g.brandl at gmx.net (Georg Brandl)
Date: Wed, 25 Sep 2013 08:30:37 +0200
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <CAP7+vJ+oVnq07Lfh_R1nqaqtK-EnxDaqPeZVTQMQTzfcEdqagg@mail.gmail.com>
References: <20130922031304.GE19939@ando>
 <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
 <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com>
 <CAF-Rda-qJ3fZQTpVO_x31WhWOaf7Q4+0n7w-sFZgK8tXWLL0zw@mail.gmail.com>
 <CAP7+vJLmLZOYmMeoKqoqrAGqVF=Z41iYBRiis5buAMyaON+_+A@mail.gmail.com>
 <CAF-Rda8H2hi5V=HakbGsD2WzH0HQ6hLuxkQYZ3ZC+MzDfb1hVw@mail.gmail.com>
 <CAP7+vJ+oVnq07Lfh_R1nqaqtK-EnxDaqPeZVTQMQTzfcEdqagg@mail.gmail.com>
Message-ID: <l1tvsj$jmv$1@ger.gmane.org>

Am 23.09.2013 00:03, schrieb Guido van Rossum:

>     AFAIU, the proposal is to embed parts of the concise docstring into the more
>     verbose .rst documentation.
> 
> 
> I still think that's a bad idea. Someone editing the docstring may introduce a
> terminology change or some other style/grammar/flow change that makes perfectly
> sense by itself but doesn't when taken in the context of the larger .rst doc
> (e.g. it could introducing duplication or cause dissonance between the two
> parts).

Yes, this style is better suited for smaller documentations where there
isn't that much more info in the .rst than in the docstring -- the .rst giving
maybe an introduction and then just a logical order in which docstrings are
pulled.

Since we have the policy (now reconfirmed) of small docstrings and more
verbose prose on docs.python.org, this would not be feasible there.

> Also, someone who wants to improve the docs would have to figure out how
> to edit the source code if they wanted to change some part of the docs.

I agree that is another good point.  And should the "suggest a change on
docs.python.org" feature ever be shipped, it will be much harder, if not
impossible, to generate a patch and let developers submit it automatically.

>     I write .rst docs quite a lot, and as such I do notice the annoying amount
>     of duplication between docstrings and .rst docs. Without doubt, all the
>     free-flow text and examples in .rst have to be written manually and are very
>     important. But for dry method references, it's certainly interesting to
>     explore the idea of writing them once in the code and then having Sphinx
>     automatically insert the relevant parts into the .rst before generating HTML
>     from it. For end users (those who read the docs online) the result is
>     indistinguishable from what we have now. For us devs it means writing the
>     same text only once and maintaining it in a single place.
> 
> 
> You seem to have caught the DRY bug. But it doesn't always make sense to factor
> things so that everything is said in exactly one place. (For an example of
> abstraction gone wild, ironically, see docutils and sphinx. I challenge you to
> find out the actual code that translates e.g. :meth:`foobar` into <a href="(what
> goes here?)">foobar</a>. :-)

PSA: I can only recommend to everyone not to take up Guido's dare... (and I'm
not terribly proud of that).

cheers,
Georg



From eliben at gmail.com  Wed Sep 25 15:15:19 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Wed, 25 Sep 2013 06:15:19 -0700
Subject: [Python-Dev] astgen.py vs. ASDL for ASTs
Message-ID: <CAF-Rda_bRpq_aGtiUSyODiykB=C0eA4HWGCBmnHK0bZuTxtU3w@mail.gmail.com>

Hello,

Some history (as best as I could collect it) followed by a question:

Before Python 2.5, the ast stdlib module was auto generated by a script
named astgen.py from a textual AST definition in Tools/compiler/ast.txt.

Since 2.5 (http://www.python.org/dev/peps/pep-0339/) ASTs are part of the
normal compilation flow by the Python compiler itself, and the ast module
uses the same ASTs generated from Parser/Python.asdl

My question is, when the switch was made in 2.5 - why didn't the existing
AST-generating code was used and the path moved to ASDL instead? What
advantages does ASDL have over the previous approach? One reason I could
think of is that ASDL nodes are typed and that's maybe better for the
generated C code to handle.

[My interest here is personal. One of my projects (pycparser) uses a
astgen.py-like approach, and in a new project I'm considering the options
again and remembered ASDL. ADSL's documentation it extremely scarce online
- seems like CPython is one of its only somewhat-visible users these days]

Thanks in advance,
Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/c16b75c0/attachment.html>

From benjamin at python.org  Wed Sep 25 15:28:08 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Wed, 25 Sep 2013 09:28:08 -0400
Subject: [Python-Dev] astgen.py vs. ASDL for ASTs
In-Reply-To: <CAF-Rda_bRpq_aGtiUSyODiykB=C0eA4HWGCBmnHK0bZuTxtU3w@mail.gmail.com>
References: <CAF-Rda_bRpq_aGtiUSyODiykB=C0eA4HWGCBmnHK0bZuTxtU3w@mail.gmail.com>
Message-ID: <CAPZV6o-CUGTB0cK4cXPamQACWcWZRQ57eP=HVOD1_dNKXm7qRQ@mail.gmail.com>

2013/9/25 Eli Bendersky <eliben at gmail.com>:
> Hello,
>
> Some history (as best as I could collect it) followed by a question:
>
> Before Python 2.5, the ast stdlib module was auto generated by a script
> named astgen.py from a textual AST definition in Tools/compiler/ast.txt.

Sorry, what Python 2.4 ast stdlib module? Any module named "ast" (or
"_ast") didn't exist until 2.5.


-- 
Regards,
Benjamin

From brett at python.org  Wed Sep 25 15:35:02 2013
From: brett at python.org (Brett Cannon)
Date: Wed, 25 Sep 2013 09:35:02 -0400
Subject: [Python-Dev] astgen.py vs. ASDL for ASTs
In-Reply-To: <CAF-Rda_bRpq_aGtiUSyODiykB=C0eA4HWGCBmnHK0bZuTxtU3w@mail.gmail.com>
References: <CAF-Rda_bRpq_aGtiUSyODiykB=C0eA4HWGCBmnHK0bZuTxtU3w@mail.gmail.com>
Message-ID: <CAP1=2W6E+9a-R+CHP2O_ev42SCQnRrBngR7i5PV9jrPNUYT4hw@mail.gmail.com>

cc'ing Jeremy Hylton who made the decision to use Zephyr.


On Wed, Sep 25, 2013 at 9:15 AM, Eli Bendersky <eliben at gmail.com> wrote:

> Hello,
>
> Some history (as best as I could collect it) followed by a question:
>
> Before Python 2.5, the ast stdlib module was auto generated by a script
> named astgen.py from a textual AST definition in Tools/compiler/ast.txt.
>
> Since 2.5 (http://www.python.org/dev/peps/pep-0339/) ASTs are part of the
> normal compilation flow by the Python compiler itself, and the ast module
> uses the same ASTs generated from Parser/Python.asdl
>
> My question is, when the switch was made in 2.5 - why didn't the existing
> AST-generating code was used and the path moved to ASDL instead? What
> advantages does ASDL have over the previous approach? One reason I could
> think of is that ASDL nodes are typed and that's maybe better for the
> generated C code to handle.
>
> [My interest here is personal. One of my projects (pycparser) uses a
> astgen.py-like approach, and in a new project I'm considering the options
> again and remembered ASDL. ADSL's documentation it extremely scarce online
> - seems like CPython is one of its only somewhat-visible users these days]
>
> Thanks in advance,
> Eli
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/17b719c5/attachment.html>

From brett at python.org  Wed Sep 25 15:35:22 2013
From: brett at python.org (Brett Cannon)
Date: Wed, 25 Sep 2013 09:35:22 -0400
Subject: [Python-Dev] astgen.py vs. ASDL for ASTs
In-Reply-To: <CAPZV6o-CUGTB0cK4cXPamQACWcWZRQ57eP=HVOD1_dNKXm7qRQ@mail.gmail.com>
References: <CAF-Rda_bRpq_aGtiUSyODiykB=C0eA4HWGCBmnHK0bZuTxtU3w@mail.gmail.com>
 <CAPZV6o-CUGTB0cK4cXPamQACWcWZRQ57eP=HVOD1_dNKXm7qRQ@mail.gmail.com>
Message-ID: <CAP1=2W42p+D-bdV8H3m+KGKjDLbe5_Fc=7r-cpceg6-YuyRrmQ@mail.gmail.com>

On Wed, Sep 25, 2013 at 9:28 AM, Benjamin Peterson <benjamin at python.org>wrote:

> 2013/9/25 Eli Bendersky <eliben at gmail.com>:
> > Hello,
> >
> > Some history (as best as I could collect it) followed by a question:
> >
> > Before Python 2.5, the ast stdlib module was auto generated by a script
> > named astgen.py from a textual AST definition in Tools/compiler/ast.txt.
>
> Sorry, what Python 2.4 ast stdlib module? Any module named "ast" (or
> "_ast") didn't exist until 2.5.
>

I think Eli is talking about the compiler package's own AST.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/da14ed8d/attachment.html>

From eliben at gmail.com  Wed Sep 25 15:35:53 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Wed, 25 Sep 2013 06:35:53 -0700
Subject: [Python-Dev] astgen.py vs. ASDL for ASTs
In-Reply-To: <CAPZV6o-CUGTB0cK4cXPamQACWcWZRQ57eP=HVOD1_dNKXm7qRQ@mail.gmail.com>
References: <CAF-Rda_bRpq_aGtiUSyODiykB=C0eA4HWGCBmnHK0bZuTxtU3w@mail.gmail.com>
 <CAPZV6o-CUGTB0cK4cXPamQACWcWZRQ57eP=HVOD1_dNKXm7qRQ@mail.gmail.com>
Message-ID: <CAF-Rda-Ro0RqPc1Jwx-85iYFuJuu3cZFbfr=9U4JHfjS3-d-8Q@mail.gmail.com>

On Wed, Sep 25, 2013 at 6:28 AM, Benjamin Peterson <benjamin at python.org>wrote:

> 2013/9/25 Eli Bendersky <eliben at gmail.com>:
> > Hello,
> >
> > Some history (as best as I could collect it) followed by a question:
> >
> > Before Python 2.5, the ast stdlib module was auto generated by a script
> > named astgen.py from a textual AST definition in Tools/compiler/ast.txt.
>
> Sorry, what Python 2.4 ast stdlib module? Any module named "ast" (or
> "_ast") didn't exist until 2.5.
>

Lib/compiler/ast.py (that's in 2.4)

Sorry, I should have been more precise
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/2a547895/attachment.html>

From eliben at gmail.com  Wed Sep 25 15:36:51 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Wed, 25 Sep 2013 06:36:51 -0700
Subject: [Python-Dev] astgen.py vs. ASDL for ASTs
In-Reply-To: <CAF-Rda-Ro0RqPc1Jwx-85iYFuJuu3cZFbfr=9U4JHfjS3-d-8Q@mail.gmail.com>
References: <CAF-Rda_bRpq_aGtiUSyODiykB=C0eA4HWGCBmnHK0bZuTxtU3w@mail.gmail.com>
 <CAPZV6o-CUGTB0cK4cXPamQACWcWZRQ57eP=HVOD1_dNKXm7qRQ@mail.gmail.com>
 <CAF-Rda-Ro0RqPc1Jwx-85iYFuJuu3cZFbfr=9U4JHfjS3-d-8Q@mail.gmail.com>
Message-ID: <CAF-Rda9jxiS90PcRj9uW66okRSNb+gO7QFxra462HZtFVydU5A@mail.gmail.com>

On Wed, Sep 25, 2013 at 6:35 AM, Eli Bendersky <eliben at gmail.com> wrote:

>
>
>
> On Wed, Sep 25, 2013 at 6:28 AM, Benjamin Peterson <benjamin at python.org>wrote:
>
>> 2013/9/25 Eli Bendersky <eliben at gmail.com>:
>> > Hello,
>> >
>> > Some history (as best as I could collect it) followed by a question:
>> >
>> > Before Python 2.5, the ast stdlib module was auto generated by a script
>> > named astgen.py from a textual AST definition in Tools/compiler/ast.txt.
>>
>> Sorry, what Python 2.4 ast stdlib module? Any module named "ast" (or
>> "_ast") didn't exist until 2.5.
>>
>
> Lib/compiler/ast.py (that's in 2.4)
>
> Sorry, I should have been more precise
>

http://docs.python.org/release/2.4/lib/module-compiler.ast.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/7d5d27a9/attachment.html>

From rajalokan at gmail.com  Wed Sep 25 10:07:28 2013
From: rajalokan at gmail.com (alok bhan)
Date: Wed, 25 Sep 2013 13:37:28 +0530
Subject: [Python-Dev] Installing python27-devel on Centos5.8 running
	python2.4
Message-ID: <CAL9ehhpSNL3RfCgka-F=gMo4bMen7pgrYJcujO+gFOLT4VK6jw@mail.gmail.com>

Hi All

I need help in installing python development packages (python27-devel) on
Centos5.8. Upgrading hardware is not an option as of now.

I've installed python2.7 in the same machine by
$ wget --no-check-certificate
http://www.python.org/ftp/python/2.7.3/Python-2.7.3.tgz
$ ./configure --prefix=/usr/local/Python2.7
$ make altinstall

Now I need help in installing python-devel package for 2.7.

$ sudo yum info python2.7-devel gives no info. Whereas the package is there
for python26-devel.

Tried many suggestions after googling but still no luck. Any help or
pointer for exact steps will be great.

Thanks
Alok Bhan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/b75d965f/attachment.html>

From brett at python.org  Wed Sep 25 18:17:12 2013
From: brett at python.org (Brett Cannon)
Date: Wed, 25 Sep 2013 12:17:12 -0400
Subject: [Python-Dev] Installing python27-devel on Centos5.8 running
	python2.4
In-Reply-To: <CAL9ehhpSNL3RfCgka-F=gMo4bMen7pgrYJcujO+gFOLT4VK6jw@mail.gmail.com>
References: <CAL9ehhpSNL3RfCgka-F=gMo4bMen7pgrYJcujO+gFOLT4VK6jw@mail.gmail.com>
Message-ID: <CAP1=2W6ja8FnoMnCm5U1CjJOnLUx-waJx=Ezz4jFw3U6VphA=A@mail.gmail.com>

This list is for the developing *of *Python, not it's use. You can try
asking on python-list for some help.


On Wed, Sep 25, 2013 at 4:07 AM, alok bhan <rajalokan at gmail.com> wrote:

> Hi All
>
> I need help in installing python development packages (python27-devel) on
> Centos5.8. Upgrading hardware is not an option as of now.
>
> I've installed python2.7 in the same machine by
> $ wget --no-check-certificate
> http://www.python.org/ftp/python/2.7.3/Python-2.7.3.tgz
> $ ./configure --prefix=/usr/local/Python2.7
>  $ make altinstall
>
> Now I need help in installing python-devel package for 2.7.
>
> $ sudo yum info python2.7-devel gives no info. Whereas the package is
> there for python26-devel.
>
> Tried many suggestions after googling but still no luck. Any help or
> pointer for exact steps will be great.
>
> Thanks
> Alok Bhan
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/196a0926/attachment.html>

From barry at python.org  Wed Sep 25 22:50:22 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 25 Sep 2013 16:50:22 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
Message-ID: <20130925165022.3d67e290@anarchist>

On Sep 23, 2013, at 09:15 PM, Nick Coghlan wrote:

>With the last round of updates, I believe PEP 453 is ready for
>Martin's pronouncement.

I want to raise an objection to PEP's proposal to add this as a new feature to
Python 2.7 and 3.3.  I understand the rationale as stated here:

http://www.python.org/dev/peps/pep-0453/#id38

but I still object. ;)

I think we've learned it's generally pretty risky to add new features in point
releases, and this is a fairly major (and would be the first?) violation of
the principal of Python 2.7's conservative maintenance.

At the very least, I'd like to know why there's no other alternative.  For
example, you could provide a PyPI package with this functionality, and let the
binary packagers adopt it into their binary packages, for platforms that care.

So for example, Linux systems which start from the source tarball could opt
out (by doing nothing special) since there are other ways to achieve a similar
benefit.  For OS X and Windows users getting binary downloads from
www.python.org, I think it would be acceptable if the .dmg or .msi included
this external package.

Why does it have to be added to the source tree for stable releases?

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/369258fa/attachment.sig>

From solipsis at pitrou.net  Wed Sep 25 23:04:20 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 25 Sep 2013 23:04:20 +0200
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
Message-ID: <20130925230420.4dac278e@fsol>

On Wed, 25 Sep 2013 16:50:22 -0400
Barry Warsaw <barry at python.org> wrote:

> On Sep 23, 2013, at 09:15 PM, Nick Coghlan wrote:
> 
> >With the last round of updates, I believe PEP 453 is ready for
> >Martin's pronouncement.
> 
> I want to raise an objection to PEP's proposal to add this as a new feature to
> Python 2.7 and 3.3.  I understand the rationale as stated here:
> 
> http://www.python.org/dev/peps/pep-0453/#id38
> 
> but I still object. ;)
> 
> I think we've learned it's generally pretty risky to add new features in point
> releases, and this is a fairly major (and would be the first?) violation of
> the principal of Python 2.7's conservative maintenance.
> 
> At the very least, I'd like to know why there's no other alternative.  For
> example, you could provide a PyPI package with this functionality, and let the
> binary packagers adopt it into their binary packages, for platforms that care.
> 
> So for example, Linux systems which start from the source tarball could opt
> out (by doing nothing special) since there are other ways to achieve a similar
> benefit.  For OS X and Windows users getting binary downloads from
> www.python.org, I think it would be acceptable if the .dmg or .msi included
> this external package.

If it's bundled in the official binary installers, it makes little
sense not to add to the source tree, IMHO. First because most people
get one of the installers rather than the source tree (on
http://www.python.org/webstats/, MSI builds win by ten to one). Second
because the few people who get the source tree are experienced users
who will get bitten less by hypothetical quirks or oddities.
Third because otherwise we're probably making things harder for our
beloved packagers ;-)

Regards

Antoine.



From ncoghlan at gmail.com  Wed Sep 25 23:14:18 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 26 Sep 2013 07:14:18 +1000
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <20130925165022.3d67e290@anarchist>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
Message-ID: <CADiSq7dxDZE_bRSdJnVfd_t3bVnO=irw2MoV8SOofXE-gta8KA@mail.gmail.com>

On 26 Sep 2013 06:53, "Barry Warsaw" <barry at python.org> wrote:
>
> On Sep 23, 2013, at 09:15 PM, Nick Coghlan wrote:
>
> >With the last round of updates, I believe PEP 453 is ready for
> >Martin's pronouncement.
>
> I want to raise an objection to PEP's proposal to add this as a new
feature to
> Python 2.7 and 3.3.  I understand the rationale as stated here:
>
> http://www.python.org/dev/peps/pep-0453/#id38
>
> but I still object. ;)
>
> I think we've learned it's generally pretty risky to add new features in
point
> releases, and this is a fairly major (and would be the first?) violation
of
> the principal of Python 2.7's conservative maintenance.
>
> At the very least, I'd like to know why there's no other alternative.  For
> example, you could provide a PyPI package with this functionality, and
let the
> binary packagers adopt it into their binary packages, for platforms that
care.
>
> So for example, Linux systems which start from the source tarball could
opt
> out (by doing nothing special) since there are other ways to achieve a
similar
> benefit.  For OS X and Windows users getting binary downloads from
> www.python.org, I think it would be acceptable if the .dmg or .msi
included
> this external package.
>
> Why does it have to be added to the source tree for stable releases?

If it can get into the installers another way, it doesn't, really. It only
needs to be in the source tree for 3.4+.

Alternatively, we could be explicit that the "don't remove it" guideline
only applies in 3.4+, where it will be needed by pyvenv.

That way, people doing custom builds from source can easily bootstrap pip
and we don't need to come up with an alternate solution for getting them
into the installers.

Cheers,
Nick.

>
> Cheers,
> -Barry
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130926/7bd25867/attachment.html>

From donald at stufft.io  Wed Sep 25 23:33:24 2013
From: donald at stufft.io (Donald Stufft)
Date: Wed, 25 Sep 2013 17:33:24 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <20130925165022.3d67e290@anarchist>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
Message-ID: <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>


On Sep 25, 2013, at 4:50 PM, Barry Warsaw <barry at python.org> wrote:

> Why does it have to be added to the source tree for stable releases?

I think it should be placed in the source tree for the stable releases. The
reasoning is that 2.7 is going to stick around for a long time. Immediately
this won't be ubiquitous but as time goes on you'll be able to be ensured
that a ``python -m ensure pip`` exists so that in situations where you don't
have pip you'll be able to install it.

While not directly relevant to the change I do think this is something users
support. I've received a fair but of feedback as I was writing the original
draft of the PEP and then throughout the process when me and Nick
were working on it. Almost all of it was positive (some of it extremely so)
a fair bit of them pointed out the backport to 2.7 as something they were
*really* wanting.

An early draft of this did not have the backport to 2.7 and when I showed
*that* version around to get feedback people were less enthusiastic
about it and generally viewed it as "nice but worthless to me for N years".

What users want isn't rationale in and of itself but I think it's an important
data point, especially given how long 2.7.LASTEVER is going to be
relevant to end users.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/54b62fb6/attachment.sig>

From barry at python.org  Wed Sep 25 23:51:29 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 25 Sep 2013 17:51:29 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
 pronouncement?
In-Reply-To: <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
Message-ID: <20130925175129.1edc4348@anarchist>

On Sep 25, 2013, at 05:33 PM, Donald Stufft wrote:

>I think it should be placed in the source tree for the stable releases. The
>reasoning is that 2.7 is going to stick around for a long time. Immediately
>this won't be ubiquitous but as time goes on you'll be able to be ensured
>that a ``python -m ensure pip`` exists so that in situations where you don't
>have pip you'll be able to install it.
>
>While not directly relevant to the change I do think this is something users
>support. I've received a fair but of feedback as I was writing the original
>draft of the PEP and then throughout the process when me and Nick
>were working on it. Almost all of it was positive (some of it extremely so)
>a fair bit of them pointed out the backport to 2.7 as something they were
>*really* wanting.
>
>An early draft of this did not have the backport to 2.7 and when I showed
>*that* version around to get feedback people were less enthusiastic
>about it and generally viewed it as "nice but worthless to me for N years".

Yeah, I get all this, but it's essentially the same reasoning that lead to
Python 2.2.1's addition of True and False.  The same flaw occurs though
because you cannot guarantee that invocation of `python -m ensurepip` will
work unless you micro version check (LBYL) or prepare to catch resulting
errors.  This essentially means that if you want to write portable Python 2.7
code and scripts, you have to be very cautious, or what works on one box won't
necessarily work on another, even though they both have "Python 2.7".

It also means that anybody who's documenting about this great new feature has
to warn people that, well, maybe it won't work on your machine even though you
have Python 2.7 because you don't have a new enough Python 2.7.

It means that not all Python 2.7's are alike in a rather fundamental and
highly visible way.

I personally think that's a recipe for more problems than it solves, and if I
was the RM for 2.7 I would not allow it.  But I'm not.

>What users want isn't rationale in and of itself but I think it's an important
>data point, especially given how long 2.7.LASTEVER is going to be
>relevant to end users.

Users want what users want.  It's our job to make the best technical decisions
based on all the facts.  I understand that Python 2.7 will be around for a
long time, and that it would be very convenient to do this.  Why is this not
opening up the door to more new features being added in future Python 2.7
point releases (or any other stable release)?

At least I think the burden should be very high, and the PEP should do a
better job of explaining why *no* other option will accomplish the goal.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/91c80684/attachment.sig>

From barry at python.org  Wed Sep 25 23:53:48 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 25 Sep 2013 17:53:48 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
 pronouncement?
In-Reply-To: <CADiSq7dxDZE_bRSdJnVfd_t3bVnO=irw2MoV8SOofXE-gta8KA@mail.gmail.com>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <CADiSq7dxDZE_bRSdJnVfd_t3bVnO=irw2MoV8SOofXE-gta8KA@mail.gmail.com>
Message-ID: <20130925175348.20f2e263@anarchist>

On Sep 26, 2013, at 07:14 AM, Nick Coghlan wrote:

>On 26 Sep 2013 06:53, "Barry Warsaw" <barry at python.org> wrote:
>>
>> Why does it have to be added to the source tree for stable releases?
>
>If it can get into the installers another way, it doesn't, really. It only
>needs to be in the source tree for 3.4+.

Sure, new features can of course go in the development release. :)

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/7ca753d7/attachment.sig>

From donald at stufft.io  Thu Sep 26 00:15:18 2013
From: donald at stufft.io (Donald Stufft)
Date: Wed, 25 Sep 2013 18:15:18 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <20130925175129.1edc4348@anarchist>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
Message-ID: <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>


On Sep 25, 2013, at 5:51 PM, Barry Warsaw <barry at python.org> wrote:

> On Sep 25, 2013, at 05:33 PM, Donald Stufft wrote:
> 
>> I think it should be placed in the source tree for the stable releases. The
>> reasoning is that 2.7 is going to stick around for a long time. Immediately
>> this won't be ubiquitous but as time goes on you'll be able to be ensured
>> that a ``python -m ensure pip`` exists so that in situations where you don't
>> have pip you'll be able to install it.
>> 
>> While not directly relevant to the change I do think this is something users
>> support. I've received a fair but of feedback as I was writing the original
>> draft of the PEP and then throughout the process when me and Nick
>> were working on it. Almost all of it was positive (some of it extremely so)
>> a fair bit of them pointed out the backport to 2.7 as something they were
>> *really* wanting.
>> 
>> An early draft of this did not have the backport to 2.7 and when I showed
>> *that* version around to get feedback people were less enthusiastic
>> about it and generally viewed it as "nice but worthless to me for N years".
> 
> Yeah, I get all this, but it's essentially the same reasoning that lead to
> Python 2.2.1's addition of True and False.  The same flaw occurs though
> because you cannot guarantee that invocation of `python -m ensurepip` will
> work unless you micro version check (LBYL) or prepare to catch resulting
> errors.  This essentially means that if you want to write portable Python 2.7
> code and scripts, you have to be very cautious, or what works on one box won't
> necessarily work on another, even though they both have "Python 2.7".
> 
> It also means that anybody who's documenting about this great new feature has
> to warn people that, well, maybe it won't work on your machine even though you
> have Python 2.7 because you don't have a new enough Python 2.7.
> 
> It means that not all Python 2.7's are alike in a rather fundamental and
> highly visible way.
> 
> I personally think that's a recipe for more problems than it solves, and if I
> was the RM for 2.7 I would not allow it.  But I'm not.

Well I don't think many scripts will be executing ensurepip, maybe i'm wrong,
but yes it does mean that not all Python 2.7's are alike. Most likely though
at some point 2.7.XXX is going to be near standard as the 2.7 hold outs
stay on it and time progresses. It more or less shifts the "time until projects
can assume people have pip available or easily installable" from "until
3.4+ is ubiquitous and 2.7 is gone (Years?) to "until Python 2.7.6 is ubiquitous"
which will be a much shorter time.

> 
>> What users want isn't rationale in and of itself but I think it's an important
>> data point, especially given how long 2.7.LASTEVER is going to be
>> relevant to end users.
> 
> Users want what users want.  It's our job to make the best technical decisions
> based on all the facts.  I understand that Python 2.7 will be around for a
> long time, and that it would be very convenient to do this.  Why is this not
> opening up the door to more new features being added in future Python 2.7
> point releases (or any other stable release)?
> 

Other than the fact that it in itself is an exception to the rule this actually strengthens
that because it makes it easier for people to install new things into Python 2.7
outside of the release cycle. So things like the new enum module can more easily
be backported and used as an installable than as part of the language. The obvious
exemption to this is language level features, but as this itself isn't a language level
feature i'm not sure how relevant that is. 

> At least I think the burden should be very high, and the PEP should do a
> better job of explaining why *no* other option will accomplish the goal.

This is totally fair and we can expand on it more for sure. 

> 
> Cheers,
> -Barry
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/8cc17b8c/attachment.sig>

From ncoghlan at gmail.com  Thu Sep 26 00:51:31 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 26 Sep 2013 08:51:31 +1000
Subject: [Python-Dev] [Python-checkins] cpython: Close #19030:
 improvements to inspect and Enum.
In-Reply-To: <3clLrQ2f4wz7LjP@mail.python.org>
References: <3clLrQ2f4wz7LjP@mail.python.org>
Message-ID: <CADiSq7eBoK_i0qtQqt7NQu7RNM5FHYdfz__mJq+Kb+Y3iH3e5w@mail.gmail.com>

Hmm, I think I just noticed a subtle bug in the new version that I missed
previously - I believe it may now misclassify a function in an instance
dict as a method.

Would need to try it out to be sure.

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130926/e56e5739/attachment.html>

From ncoghlan at gmail.com  Thu Sep 26 01:01:12 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 26 Sep 2013 09:01:12 +1000
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <20130925175129.1edc4348@anarchist>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
Message-ID: <CADiSq7fVrruk1zcE-ZwHUOyUqOPhTBnn2jcp9rKQ8dS6QeL+Zw@mail.gmail.com>

On 26 Sep 2013 07:54, "Barry Warsaw" <barry at python.org> wrote:
>
> On Sep 25, 2013, at 05:33 PM, Donald Stufft wrote:
>
> >I think it should be placed in the source tree for the stable releases.
The
> >reasoning is that 2.7 is going to stick around for a long time.
Immediately
> >this won't be ubiquitous but as time goes on you'll be able to be ensured
> >that a ``python -m ensure pip`` exists so that in situations where you
don't
> >have pip you'll be able to install it.
> >
> >While not directly relevant to the change I do think this is something
users
> >support. I've received a fair but of feedback as I was writing the
original
> >draft of the PEP and then throughout the process when me and Nick
> >were working on it. Almost all of it was positive (some of it extremely
so)
> >a fair bit of them pointed out the backport to 2.7 as something they were
> >*really* wanting.
> >
> >An early draft of this did not have the backport to 2.7 and when I showed
> >*that* version around to get feedback people were less enthusiastic
> >about it and generally viewed it as "nice but worthless to me for N
years".
>
> Yeah, I get all this, but it's essentially the same reasoning that lead to
> Python 2.2.1's addition of True and False.  The same flaw occurs though
> because you cannot guarantee that invocation of `python -m ensurepip` will
> work unless you micro version check (LBYL) or prepare to catch resulting
> errors.  This essentially means that if you want to write portable Python
2.7
> code and scripts, you have to be very cautious, or what works on one box
won't
> necessarily work on another, even though they both have "Python 2.7".

There's a bit of a difference between adding new builtins and adding a new
utility module, though.

The only regular invocations of ensurepip in 2.7 and 3.3 should be from our
installers and people doing a custom build from source.

Would a leading underscore in the module name make you more comfortable
with the idea? It's really intended mostly as a hidden implementation
detail of the installers and pyvenv anyway, so calling it "_ensurepip"
would help make that explicit while still letting people invoke it directly
if absolutely necessary.

Cheers,
Nick.

>
> It also means that anybody who's documenting about this great new feature
has
> to warn people that, well, maybe it won't work on your machine even
though you
> have Python 2.7 because you don't have a new enough Python 2.7.
>
> It means that not all Python 2.7's are alike in a rather fundamental and
> highly visible way.
>
> I personally think that's a recipe for more problems than it solves, and
if I
> was the RM for 2.7 I would not allow it.  But I'm not.
>
> >What users want isn't rationale in and of itself but I think it's an
important
> >data point, especially given how long 2.7.LASTEVER is going to be
> >relevant to end users.
>
> Users want what users want.  It's our job to make the best technical
decisions
> based on all the facts.  I understand that Python 2.7 will be around for a
> long time, and that it would be very convenient to do this.  Why is this
not
> opening up the door to more new features being added in future Python 2.7
> point releases (or any other stable release)?
>
> At least I think the burden should be very high, and the PEP should do a
> better job of explaining why *no* other option will accomplish the goal.
>
> Cheers,
> -Barry
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130926/b22663ba/attachment-0001.html>

From barry at python.org  Thu Sep 26 01:04:10 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 25 Sep 2013 19:04:10 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
 pronouncement?
In-Reply-To: <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
Message-ID: <20130925190410.0a5729f9@anarchist>

On Sep 25, 2013, at 06:15 PM, Donald Stufft wrote:

>Well I don't think many scripts will be executing ensurepip, maybe i'm wrong,
>but yes it does mean that not all Python 2.7's are alike. Most likely though
>at some point 2.7.XXX is going to be near standard as the 2.7 hold outs stay
>on it and time progresses.

I think it's too optimistic to hope that we'll ever see significant
convergence across the Python 2.7 ecosystem.

For example, I'd be really surprised if you'd see this show up in a stable
release of Debian or Ubuntu (the only ones I can speak somewhat
authoritatively about), even if we wanted to cherrypick updates in newer
Python 2.7 point releases.  Linux distros tend to be even more anal about
avoiding new features in stable releases, for the same really good reasons,
IMHO :).

I think for the platforms you're targeting (OS X, Windows), you're going to
see a wide variety for a very long time.  It's depressing how infrequently
people upgrade *anything*.

Another reason to oppose this is what I've heard quite often from people
regarding Python 2.7.  I've been told that many folks are actually really
happy with using 2.7 precisely because it extremely stable.  They don't have
to worry about their stuff breaking or incompatibilities cropping up, because
it *doesn't* change.  Python 2.7 is like a long-term maintenance release, with
guaranteed multi-year stability, and I think while that was unintended, it's a
*good* thing.  This PEP proposes to break that, and I'm loathe to give up that
good reputation for this particular feature.

>It more or less shifts the "time until projects can assume people have pip
>available or easily installable" from "until 3.4+ is ubiquitous and 2.7 is
>gone (Years?) to "until Python 2.7.6 is ubiquitous" which will be a much
>shorter time.

Yeah, maybe, but I'm pretty skeptical about the enthusiasm for upgrades. ;)

>> Users want what users want.  It's our job to make the best technical
>> decisions based on all the facts.  I understand that Python 2.7 will be
>> around for a long time, and that it would be very convenient to do this.
>> Why is this not opening up the door to more new features being added in
>> future Python 2.7 point releases (or any other stable release)?
>> 
>
>Other than the fact that it in itself is an exception to the rule this
>actually strengthens that because it makes it easier for people to install
>new things into Python 2.7 outside of the release cycle. So things like the
>new enum module can more easily be backported and used as an installable than
>as part of the language. The obvious exemption to this is language level
>features, but as this itself isn't a language level feature i'm not sure how
>relevant that is.

So, why shouldn't we add enum to the Python 2.7 stdlib?  Or any other new
feature?  Just be aware that if this PEP is accepted for Python 2.7, the bar
will be set much lower for other Really Useful Features That Make Users Lives
Better.

>> At least I think the burden should be very high, and the PEP should do a
>> better job of explaining why *no* other option will accomplish the goal.
>
>This is totally fair and we can expand on it more for sure. 

Cool.  Maybe in the course of that discussion, a better alternative that
doesn't add a new feature to Python 2.7 will present itself.  I really hope
so.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/05772c7d/attachment.sig>

From donald at stufft.io  Thu Sep 26 01:08:32 2013
From: donald at stufft.io (Donald Stufft)
Date: Wed, 25 Sep 2013 19:08:32 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <20130925190410.0a5729f9@anarchist>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
Message-ID: <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>


On Sep 25, 2013, at 7:04 PM, Barry Warsaw <barry at python.org> wrote:

> So, why shouldn't we add enum to the Python 2.7 stdlib?  Or any other new
> feature?  Just be aware that if this PEP is accepted for Python 2.7, the bar
> will be set much lower for other Really Useful Features That Make Users Lives
> Better.

Because with PEP453 you can just ``pip install enum34`` it :)

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/53fd5d4a/attachment.sig>

From barry at python.org  Thu Sep 26 01:08:54 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 25 Sep 2013 19:08:54 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
 pronouncement?
In-Reply-To: <CADiSq7fVrruk1zcE-ZwHUOyUqOPhTBnn2jcp9rKQ8dS6QeL+Zw@mail.gmail.com>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <CADiSq7fVrruk1zcE-ZwHUOyUqOPhTBnn2jcp9rKQ8dS6QeL+Zw@mail.gmail.com>
Message-ID: <20130925190854.6ac76200@anarchist>

On Sep 26, 2013, at 09:01 AM, Nick Coghlan wrote:

>Would a leading underscore in the module name make you more comfortable
>with the idea? It's really intended mostly as a hidden implementation
>detail of the installers and pyvenv anyway, so calling it "_ensurepip"
>would help make that explicit while still letting people invoke it directly
>if absolutely necessary.

If it was also undocumented, and we can guarantee that it will have no impact
outside of the module itself, then maybe.  I'm not sure it would be enough for
me to accept it if I were the RM, but I'm not (thankfully. :).

I still think it's optimistic to pin any hopes on its widespread availability
in future 2.7 point releases on people's actual systems.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/6c283445/attachment.sig>

From barry at python.org  Thu Sep 26 01:15:05 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 25 Sep 2013 19:15:05 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
 <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>
Message-ID: <20130925191505.59fb9c0f@anarchist>

On Sep 25, 2013, at 07:08 PM, Donald Stufft wrote:

>On Sep 25, 2013, at 7:04 PM, Barry Warsaw <barry at python.org> wrote:
>
>> So, why shouldn't we add enum to the Python 2.7 stdlib?

>Because with PEP453 you can just ``pip install enum34`` it :)

Of course, pip messing with global Python state on a package managed system
like most Linux distros, is a whole 'nuther happy fun ball of killer
beeswax. :)

mixing-metaphors-ly y'rs,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/cd35739a/attachment.sig>

From donald at stufft.io  Thu Sep 26 01:15:41 2013
From: donald at stufft.io (Donald Stufft)
Date: Wed, 25 Sep 2013 19:15:41 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <20130925190410.0a5729f9@anarchist>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
Message-ID: <D995BB4B-F2AA-40B7-BBF3-3DDF36CCB5B9@stufft.io>


On Sep 25, 2013, at 7:04 PM, Barry Warsaw <barry at python.org> wrote:

> Another reason to oppose this is what I've heard quite often from people
> regarding Python 2.7.  I've been told that many folks are actually really
> happy with using 2.7 precisely because it extremely stable.  They don't have
> to worry about their stuff breaking or incompatibilities cropping up, because
> it *doesn't* change.  Python 2.7 is like a long-term maintenance release, with
> guaranteed multi-year stability, and I think while that was unintended, it's a
> *good* thing.  This PEP proposes to break that, and I'm loathe to give up that
> good reputation for this particular feature.

Maybe I'm just naive but can you expand on how adding a module that nothing
else in the system imports (besides the installers; but you weren't arguing against
adding this to the installers) would break someones use any other module? If
they don't import it (which the vast bulk of people won't directly, nor at all during
the operation of their applications) how does it's existence on the file system
risk a breakage to their system?

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/bb9e739e/attachment.sig>

From donald at stufft.io  Thu Sep 26 01:18:56 2013
From: donald at stufft.io (Donald Stufft)
Date: Wed, 25 Sep 2013 19:18:56 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <20130925191505.59fb9c0f@anarchist>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
 <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>
 <20130925191505.59fb9c0f@anarchist>
Message-ID: <DA69848B-8532-4DB8-B489-CCFF5D7E6373@stufft.io>


On Sep 25, 2013, at 7:15 PM, Barry Warsaw <barry at python.org> wrote:

> Of course, pip messing with global Python state on a package managed system
> like most Linux distros, is a whole 'nuther happy fun ball of killer
> beeswax. :)
> 
> mixing-metaphors-ly y'rs,
> -Barry

<virtualenv invocation not shown>

(For reals a pip and apt-get playing nicely is on my stack of PEPs to do)

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/331eef01/attachment.sig>

From cs at zip.com.au  Thu Sep 26 01:05:20 2013
From: cs at zip.com.au (Cameron Simpson)
Date: Thu, 26 Sep 2013 09:05:20 +1000
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <5241BED1.7010905@g.nevcal.com>
 <l1tvd9$6rg$1@ger.gmane.org>
References: <5241BED1.7010905@g.nevcal.com>
 <l1tvd9$6rg$1@ger.gmane.org>
Message-ID: <20130925230520.GA94785@cskk.homeip.net>

[ I've replied to two messages here: Georg's and Glenn's (supporting MRAB's).
  - Cameron
]

On 25Sep2013 08:22, Georg Brandl <g.brandl at gmx.net> wrote:
| Am 24.09.2013 00:11, schrieb Greg Ewing:
| > How about something like "Uncaught exception in __del__
| > method ignored"? It explains fairly clearly what has
| > happened, and also indicates what do do about it --
| > catch it in the __del__ method.
| 
| "Exception in __del__ caught and not propagated:"
| Georg


On 24Sep2013 09:33, Glenn Linderman <v+python at g.nevcal.com> wrote:
| [MRAB]:
| >> Why not just say something like "Cannot propagate exception..."; it's
| >> simpler than "Unpropagatable exception...". [...]
|
| First one I've heard that accurately and unambiguously and briefly
| describes the issue.
| +1

I'm strongly in favour of Georg's one ("Exception in __del__ caught and not propagated").

Why?

It says simply and clearly what has happened.
It denotes the relevant context (__del__) in which it happened.
The reader can then decide to find out why that decision may have been made.

Why not MRAB's? ("Cannot propagate exception...")

While better than "Unpropagatable exception" and "unraisable" and
"unreraisable", it has the same flaw that I think underlies Antoine's
concerns: it suggests the reason there's an error printed instead
of further propagation is a _property of the exception_.
It doesn't say it outright, but as an outsider that is definitely
what I would at first infer.

So: a small +0.1 for "Cannot propagate exception..."

And: a big +2 for "Exception in __del__ caught and not propagated:".

Cheers,
-- 
Cameron Simpson <cs at zip.com.au>

Let the bullet of your thought achieve the true and final path of its
trajectory before it reaches the ear of the listener, lest it plow out the
other side of his head. - Henry David Thoreau

From python at mrabarnett.plus.com  Thu Sep 26 02:17:14 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Thu, 26 Sep 2013 01:17:14 +0100
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <20130925230520.GA94785@cskk.homeip.net>
References: <5241BED1.7010905@g.nevcal.com> <l1tvd9$6rg$1@ger.gmane.org>
 <20130925230520.GA94785@cskk.homeip.net>
Message-ID: <52437D0A.7080501@mrabarnett.plus.com>

On 26/09/2013 00:05, Cameron Simpson wrote:
> [ I've replied to two messages here: Georg's and Glenn's (supporting MRAB's).
>    - Cameron
> ]
>
> On 25Sep2013 08:22, Georg Brandl <g.brandl at gmx.net> wrote:
> | Am 24.09.2013 00:11, schrieb Greg Ewing:
> | > How about something like "Uncaught exception in __del__
> | > method ignored"? It explains fairly clearly what has
> | > happened, and also indicates what do do about it --
> | > catch it in the __del__ method.
> |
> | "Exception in __del__ caught and not propagated:"
> | Georg
>
>
> On 24Sep2013 09:33, Glenn Linderman <v+python at g.nevcal.com> wrote:
> | [MRAB]:
> | >> Why not just say something like "Cannot propagate exception..."; it's
> | >> simpler than "Unpropagatable exception...". [...]
> |
> | First one I've heard that accurately and unambiguously and briefly
> | describes the issue.
> | +1
>
> I'm strongly in favour of Georg's one ("Exception in __del__ caught and not propagated").
>
> Why?
>
> It says simply and clearly what has happened.
> It denotes the relevant context (__del__) in which it happened.
> The reader can then decide to find out why that decision may have been made.
>
> Why not MRAB's? ("Cannot propagate exception...")
>
> While better than "Unpropagatable exception" and "unraisable" and
> "unreraisable", it has the same flaw that I think underlies Antoine's
> concerns: it suggests the reason there's an error printed instead
> of further propagation is a _property of the exception_.
> It doesn't say it outright, but as an outsider that is definitely
> what I would at first infer.
>
> So: a small +0.1 for "Cannot propagate exception..."
>
> And: a big +2 for "Exception in __del__ caught and not propagated:".
>
Well, my suggestion was more about eliminating the adjective+noun form; 
it seemed it just wasn't
possible to cram the desired meaning into a single word, so it was 
better not to try!


From v+python at g.nevcal.com  Thu Sep 26 03:46:39 2013
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Wed, 25 Sep 2013 18:46:39 -0700
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <52437D0A.7080501@mrabarnett.plus.com>
References: <5241BED1.7010905@g.nevcal.com> <l1tvd9$6rg$1@ger.gmane.org>
 <20130925230520.GA94785@cskk.homeip.net>
 <52437D0A.7080501@mrabarnett.plus.com>
Message-ID: <524391FF.4010209@g.nevcal.com>

On 9/25/2013 5:17 PM, MRAB wrote:
> On 26/09/2013 00:05, Cameron Simpson wrote:
>> [ I've replied to two messages here: Georg's and Glenn's (supporting 
>> MRAB's).
>>    - Cameron
>> ]
>>
>> On 25Sep2013 08:22, Georg Brandl <g.brandl at gmx.net> wrote:
>> | Am 24.09.2013 00:11, schrieb Greg Ewing:
>> | > How about something like "Uncaught exception in __del__
>> | > method ignored"? It explains fairly clearly what has
>> | > happened, and also indicates what do do about it --
>> | > catch it in the __del__ method.
>> |
>> | "Exception in __del__ caught and not propagated:"
>> | Georg
>>
>>
>> On 24Sep2013 09:33, Glenn Linderman <v+python at g.nevcal.com> wrote:
>> | [MRAB]:
>> | >> Why not just say something like "Cannot propagate exception..."; 
>> it's
>> | >> simpler than "Unpropagatable exception...". [...]
>> |
>> | First one I've heard that accurately and unambiguously and briefly
>> | describes the issue.
>> | +1
>>
>> I'm strongly in favour of Georg's one ("Exception in __del__ caught 
>> and not propagated").
>>
>> Why?
>>
>> It says simply and clearly what has happened.
>> It denotes the relevant context (__del__) in which it happened.
>> The reader can then decide to find out why that decision may have 
>> been made.
>>
>> Why not MRAB's? ("Cannot propagate exception...")
>>
>> While better than "Unpropagatable exception" and "unraisable" and
>> "unreraisable", it has the same flaw that I think underlies Antoine's
>> concerns: it suggests the reason there's an error printed instead
>> of further propagation is a _property of the exception_.
>> It doesn't say it outright, but as an outsider that is definitely
>> what I would at first infer.
>>
>> So: a small +0.1 for "Cannot propagate exception..."
>>
>> And: a big +2 for "Exception in __del__ caught and not propagated:".
>>
> Well, my suggestion was more about eliminating the adjective+noun 
> form; it seemed it just wasn't
> possible to cram the desired meaning into a single word, so it was 
> better not to try!

Georg hadn't suggested his yet when I approved MRAB's, but your 
reasoning is sound, Cameron, unless there is some reason that the 
message can't or shouldn't be as long as what Georg suggested.

In between in length and clarity might be:

__del__ cannot propagate exception ....

All of these are less confusing that the adj+noun form, that MRAB was 
the first to get away from.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/8c6dff61/attachment-0001.html>

From stephen at xemacs.org  Thu Sep 26 03:52:25 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Thu, 26 Sep 2013 10:52:25 +0900
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready
	for	pronouncement?
In-Reply-To: <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
 <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>
Message-ID: <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp>

Donald Stufft writes:

 > On Sep 25, 2013, at 7:04 PM, Barry Warsaw <barry at python.org> wrote:
 > 
 > > So, why shouldn't we add enum to the Python 2.7 stdlib?  Or any
 > > other new feature?  Just be aware that if this PEP is accepted
 > > for Python 2.7, the bar will be set much lower for other Really
 > > Useful Features That Make Users Lives Better.
 > 
 > Because with PEP453 you can just ``pip install enum34`` it :)

Uh, I've been doing that for years (well, not with ``enum34``, to be
sure).  And not just with 2.7, 2.6 and 2.5 too.  Who needs PEP 453?
Not me (ironically, I'll probably have a PEP 453 Python within 48
hours of its public release).

OTOH, as Barry points out, the target population doesn't upgrade very
often.  What gets them to upgrade (or install in parallel) is the
installation of a package that requires an upgraded Python.  But that
typically means a bump from x.y to x.z, and that will never happen to
2.7.  So they'll see a failure of some kind anyway if the ensurepip
feature is used!  That is going to blowback on distributors, which is
going to make PEP 453 real popular with them.  Not.




From ncoghlan at gmail.com  Thu Sep 26 06:30:18 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 26 Sep 2013 14:30:18 +1000
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <20130925190854.6ac76200@anarchist>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <CADiSq7fVrruk1zcE-ZwHUOyUqOPhTBnn2jcp9rKQ8dS6QeL+Zw@mail.gmail.com>
 <20130925190854.6ac76200@anarchist>
Message-ID: <CADiSq7cMMkTjArg=0=rmDypHMo5aACYNQX=ct18N9jzru2=JQQ@mail.gmail.com>

On 26 September 2013 09:08, Barry Warsaw <barry at python.org> wrote:
> On Sep 26, 2013, at 09:01 AM, Nick Coghlan wrote:
>
>>Would a leading underscore in the module name make you more comfortable
>>with the idea? It's really intended mostly as a hidden implementation
>>detail of the installers and pyvenv anyway, so calling it "_ensurepip"
>>would help make that explicit while still letting people invoke it directly
>>if absolutely necessary.
>
> If it was also undocumented, and we can guarantee that it will have no impact
> outside of the module itself, then maybe.  I'm not sure it would be enough for
> me to accept it if I were the RM, but I'm not (thankfully. :).
>
> I still think it's optimistic to pin any hopes on its widespread availability
> in future 2.7 point releases on people's actual systems.

The recommended user experience for 2.7 users on Windows and Mac OS X
will be "Install the latest Python 2.7 from python.org". This feature
is *not* aimed at existing users *at all*. For Python 2.7, the
benefits are aimed almost entirely at beginners installing Python for
the first time, regardless of platform. There's a *side effect* for
existing users, where for Windows and Mac OS X users, the pip
bootstrapping instructions become "install the latest maintenance
release" rather than downloading and running the bootstrap script from
pip-installer.org.

The presence or absence of _ensurepip should be completely irrelevant
to end users, as it will be there in 2.7 solely to make the new
feature of the installer work.

Your alternative proposal is to instead come up with a *new* mechanism
to support the feature, specifically for 2.7. That's increasing the
chance of breakage, not reducing it, *because the _ensurepip tests
would no longer be executed by the Python 2.7 buildbots*.

That said, there are changes that I think are definitely worth making
due to the concerns you raise:

- the module name should be "_ensurepip" in all versions
- the PEP should explicitly state that the "don't remove _ensurepip
and it's wheel files" caveat for redistributors applies only in 3.4+
(where removing it will break pyvenv)

The benefits are most obvious in the case of 2.7, so both Donald and I
are also fine with skipping making any changes to Python 3.3. The kind
of environment that could make it difficult to upgrade from Python
3.3. to 3.4 is unlikely to condone the installation and use of pip
anyway.

There is one other change which is potentially reasonable, and that's
changing the default state of the "Install pip (the default Python
package management utility)?" checkbox. I believe it should be checked
by default in all versions, but if Benjamin decides that it should be
unchecked by default in 2.7, I'd be willing to go along with that.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Thu Sep 26 06:39:40 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 26 Sep 2013 14:39:40 +1000
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
 <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>
 <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com>

On 26 September 2013 11:52, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> Donald Stufft writes:
>
>  > On Sep 25, 2013, at 7:04 PM, Barry Warsaw <barry at python.org> wrote:
>  >
>  > > So, why shouldn't we add enum to the Python 2.7 stdlib?  Or any
>  > > other new feature?  Just be aware that if this PEP is accepted
>  > > for Python 2.7, the bar will be set much lower for other Really
>  > > Useful Features That Make Users Lives Better.
>  >
>  > Because with PEP453 you can just ``pip install enum34`` it :)
>
> Uh, I've been doing that for years (well, not with ``enum34``, to be
> sure).  And not just with 2.7, 2.6 and 2.5 too.  Who needs PEP 453?
> Not me (ironically, I'll probably have a PEP 453 Python within 48
> hours of its public release).

New users on Windows and Mac OS X. I've heard many more complaints
from folks running tutorials about the pip bootstrapping process than
I ever have from the community at large about the GIL :P

Linux and other *nix platforms are affected primarily due to the fact
that we also want pyvenv to start making pip available by default in
3.4+, so we need the PEP to account for making that possible while
still playing nice with the Linux distro vendors.

> OTOH, as Barry points out, the target population doesn't upgrade very
> often.

If somebody already has Python installed, they're not really part of
the target population, except in the sense that the pip bootstrap
instruction become "upgrade Python to the latest maintenance release"
rather than "got to this website, download and run this bootstrap
script". (Keeping those instructions consistent for 3.3 users as well
would be the advantage of updating 3.3 in addition to 2.7 and 3.4)

> What gets them to upgrade (or install in parallel) is the
> installation of a package that requires an upgraded Python.  But that
> typically means a bump from x.y to x.z, and that will never happen to
> 2.7.  So they'll see a failure of some kind anyway if the ensurepip
> feature is used!  That is going to blowback on distributors, which is
> going to make PEP 453 real popular with them.  Not.

I'm not sure what usage model you're assuming for _ensurepip, but it
appears to be wrong. End users should be able to just run pip, and
either have it work, or else get a message from the OS vendor telling
them to install the appropriate system package. If it doesn't work,
then the advice will be to upgrade to the latest Python maintenance
release.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Thu Sep 26 06:59:17 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 26 Sep 2013 14:59:17 +1000
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <20130925230520.GA94785@cskk.homeip.net>
References: <5241BED1.7010905@g.nevcal.com> <l1tvd9$6rg$1@ger.gmane.org>
 <20130925230520.GA94785@cskk.homeip.net>
Message-ID: <CADiSq7fywLacQywujMma+HTov948FXgmtJ51gRon_iU4Fc2nxg@mail.gmail.com>

On 26 September 2013 09:05, Cameron Simpson <cs at zip.com.au> wrote:
> On 24Sep2013 09:33, Glenn Linderman <v+python at g.nevcal.com> wrote:
> | [MRAB]:
> | >> Why not just say something like "Cannot propagate exception..."; it's
> | >> simpler than "Unpropagatable exception...". [...]
> |
> | First one I've heard that accurately and unambiguously and briefly
> | describes the issue.
> | +1
>
> I'm strongly in favour of Georg's one ("Exception in __del__ caught and not propagated").
>
> Why?
>
> It says simply and clearly what has happened.
> It denotes the relevant context (__del__) in which it happened.
> The reader can then decide to find out why that decision may have been made.

Such a change is highly unlikely to happen, as it would require
changing every location where we call PyErr_WriteUnraisable.

> Why not MRAB's? ("Cannot propagate exception...")
>
> While better than "Unpropagatable exception" and "unraisable" and
> "unreraisable", it has the same flaw that I think underlies Antoine's
> concerns: it suggests the reason there's an error printed instead
> of further propagation is a _property of the exception_.
> It doesn't say it outright, but as an outsider that is definitely
> what I would at first infer.

It's substantially better than the status quo, though, and it
shouldn't require any changes outside PyErr_WriteUnraisable.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Thu Sep 26 06:54:49 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 26 Sep 2013 14:54:49 +1000
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <CADiSq7cMMkTjArg=0=rmDypHMo5aACYNQX=ct18N9jzru2=JQQ@mail.gmail.com>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <CADiSq7fVrruk1zcE-ZwHUOyUqOPhTBnn2jcp9rKQ8dS6QeL+Zw@mail.gmail.com>
 <20130925190854.6ac76200@anarchist>
 <CADiSq7cMMkTjArg=0=rmDypHMo5aACYNQX=ct18N9jzru2=JQQ@mail.gmail.com>
Message-ID: <CADiSq7cmSqnWCL0-s-g5dc1AgRRkfhd0VjYVSDj47bWQh537NA@mail.gmail.com>

On 26 September 2013 14:30, Nick Coghlan <ncoghlan at gmail.com> wrote:
> That said, there are changes that I think are definitely worth making
> due to the concerns you raise:
>
> - the module name should be "_ensurepip" in all versions
> - the PEP should explicitly state that the "don't remove _ensurepip
> and it's wheel files" caveat for redistributors applies only in 3.4+
> (where removing it will break pyvenv)

Donald pointed out it makes more sense to continue with the idea of a
properly documented public "ensurepip" module in 3.4+, and have the
"_ensurepip" version as an implementation detail of the 2.7 and 3.3
installers that is included in the stdlib primarily so it can be
covered by the existing buildbot fleet.

Redistributors will be free to remove "_ensurepip" from 2.7 and 3.3,
but the PEP will continue to advise strongly against removing
"ensurepip" from 3.4+ (since it will be a documented feature and a
dependency of pyvenv)

> The benefits are most obvious in the case of 2.7, so both Donald and I
> are also fine with skipping making any changes to Python 3.3. The kind
> of environment that could make it difficult to upgrade from Python
> 3.3. to 3.4 is unlikely to condone the installation and use of pip
> anyway.

In replying to Stephen I realised the benefit of making a similar
change in the 3.3 installer is that it means the preferred pip
bootstrapping instructions for both 2.7 and 3.3 can be to update to
the latest maintenance release of CPython, rather than needing to tell
3.3 users to consider upgrading to 3.4 when that release contains
potentially breaking changes to the way file descriptors are handled.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From arigo at tunes.org  Thu Sep 26 07:42:54 2013
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 26 Sep 2013 07:42:54 +0200
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <CADiSq7fywLacQywujMma+HTov948FXgmtJ51gRon_iU4Fc2nxg@mail.gmail.com>
References: <5241BED1.7010905@g.nevcal.com> <l1tvd9$6rg$1@ger.gmane.org>
 <20130925230520.GA94785@cskk.homeip.net>
 <CADiSq7fywLacQywujMma+HTov948FXgmtJ51gRon_iU4Fc2nxg@mail.gmail.com>
Message-ID: <CAMSv6X1tQGPhLs0rnvNwV12z9czPkLWZx8dk9kmtYQ_RNwnY_Q@mail.gmail.com>

Hi Nick,

On Thu, Sep 26, 2013 at 6:59 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> I'm strongly in favour of Georg's one ("Exception in __del__ caught and not propagated").
>
> Such a change is highly unlikely to happen, as it would require
> changing every location where we call PyErr_WriteUnraisable.

Er, why?  It seems to me it's a matter of changing these three lines
in PyErr_WriteUnraisable():

-           PyFile_WriteString("Exception ignored in: ", f);
+          PyFile_WriteString("Exception in ", f);
            PyFile_WriteObject(obj, f, 0);
-           PyFile_WriteString("\n", f);
+          PyFile_WriteString(" caught and not propagated:\n", f);

I don't see what makes this technically different from the other
solution, "Cannot propagate exception..."


A bient?t,

Armin.

From ncoghlan at gmail.com  Thu Sep 26 08:22:34 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 26 Sep 2013 16:22:34 +1000
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <CAMSv6X1tQGPhLs0rnvNwV12z9czPkLWZx8dk9kmtYQ_RNwnY_Q@mail.gmail.com>
References: <5241BED1.7010905@g.nevcal.com> <l1tvd9$6rg$1@ger.gmane.org>
 <20130925230520.GA94785@cskk.homeip.net>
 <CADiSq7fywLacQywujMma+HTov948FXgmtJ51gRon_iU4Fc2nxg@mail.gmail.com>
 <CAMSv6X1tQGPhLs0rnvNwV12z9czPkLWZx8dk9kmtYQ_RNwnY_Q@mail.gmail.com>
Message-ID: <CADiSq7eiZku6BXEd3DtOGg4BRPNP3-QvtNNQbq0=SRf7eCJKYA@mail.gmail.com>

On 26 September 2013 15:42, Armin Rigo <arigo at tunes.org> wrote:
> Hi Nick,
>
> On Thu, Sep 26, 2013 at 6:59 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> I'm strongly in favour of Georg's one ("Exception in __del__ caught and not propagated").
>>
>> Such a change is highly unlikely to happen, as it would require
>> changing every location where we call PyErr_WriteUnraisable.
>
> Er, why?  It seems to me it's a matter of changing these three lines
> in PyErr_WriteUnraisable():
>
> -           PyFile_WriteString("Exception ignored in: ", f);
> +          PyFile_WriteString("Exception in ", f);
>             PyFile_WriteObject(obj, f, 0);
> -           PyFile_WriteString("\n", f);
> +          PyFile_WriteString(" caught and not propagated:\n", f);
>
> I don't see what makes this technically different from the other
> solution, "Cannot propagate exception..."

Sure, that's doable, but it dumps the full repr of "obj" in the middle
of the sentence. The thing that's not practical is the neat and tidy
wording Georg proposed, because the thing passed as "obj" is actually
an arbitrary Python object that may have a messy repr (like a bound
method, which is what gets passed in the __del__ case), so there's
definite merit in keeping that repr at the *end* of the header line.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Thu Sep 26 08:35:36 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 26 Sep 2013 08:35:36 +0200
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
 <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>
 <20130925191505.59fb9c0f@anarchist>
Message-ID: <20130926083536.04344210@fsol>

On Wed, 25 Sep 2013 19:15:05 -0400
Barry Warsaw <barry at python.org> wrote:
> On Sep 25, 2013, at 07:08 PM, Donald Stufft wrote:
> 
> >On Sep 25, 2013, at 7:04 PM, Barry Warsaw <barry at python.org> wrote:
> >
> >> So, why shouldn't we add enum to the Python 2.7 stdlib?
> 
> >Because with PEP453 you can just ``pip install enum34`` it :)
> 
> Of course, pip messing with global Python state on a package managed system
> like most Linux distros, is a whole 'nuther happy fun ball of killer
> beeswax. :)

pip install --user enum34 :-)



From g.brandl at gmx.net  Thu Sep 26 08:53:14 2013
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 26 Sep 2013 08:53:14 +0200
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <CADiSq7eiZku6BXEd3DtOGg4BRPNP3-QvtNNQbq0=SRf7eCJKYA@mail.gmail.com>
References: <5241BED1.7010905@g.nevcal.com> <l1tvd9$6rg$1@ger.gmane.org>
 <20130925230520.GA94785@cskk.homeip.net>
 <CADiSq7fywLacQywujMma+HTov948FXgmtJ51gRon_iU4Fc2nxg@mail.gmail.com>
 <CAMSv6X1tQGPhLs0rnvNwV12z9czPkLWZx8dk9kmtYQ_RNwnY_Q@mail.gmail.com>
 <CADiSq7eiZku6BXEd3DtOGg4BRPNP3-QvtNNQbq0=SRf7eCJKYA@mail.gmail.com>
Message-ID: <l20liu$gla$1@ger.gmane.org>

Am 26.09.2013 08:22, schrieb Nick Coghlan:
> On 26 September 2013 15:42, Armin Rigo <arigo at tunes.org> wrote:
>> Hi Nick,
>>
>> On Thu, Sep 26, 2013 at 6:59 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>>> I'm strongly in favour of Georg's one ("Exception in __del__ caught and not propagated").
>>>
>>> Such a change is highly unlikely to happen, as it would require
>>> changing every location where we call PyErr_WriteUnraisable.
>>
>> Er, why?  It seems to me it's a matter of changing these three lines
>> in PyErr_WriteUnraisable():
>>
>> -           PyFile_WriteString("Exception ignored in: ", f);
>> +          PyFile_WriteString("Exception in ", f);
>>             PyFile_WriteObject(obj, f, 0);
>> -           PyFile_WriteString("\n", f);
>> +          PyFile_WriteString(" caught and not propagated:\n", f);
>>
>> I don't see what makes this technically different from the other
>> solution, "Cannot propagate exception..."
> 
> Sure, that's doable, but it dumps the full repr of "obj" in the middle
> of the sentence. The thing that's not practical is the neat and tidy
> wording Georg proposed, because the thing passed as "obj" is actually
> an arbitrary Python object that may have a messy repr (like a bound
> method, which is what gets passed in the __del__ case), so there's
> definite merit in keeping that repr at the *end* of the header line.

Then this should be fine, I guess?

Exception caught and not propagated in: <....>

Georg


From ncoghlan at gmail.com  Thu Sep 26 09:17:02 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 26 Sep 2013 17:17:02 +1000
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <l20liu$gla$1@ger.gmane.org>
References: <5241BED1.7010905@g.nevcal.com> <l1tvd9$6rg$1@ger.gmane.org>
 <20130925230520.GA94785@cskk.homeip.net>
 <CADiSq7fywLacQywujMma+HTov948FXgmtJ51gRon_iU4Fc2nxg@mail.gmail.com>
 <CAMSv6X1tQGPhLs0rnvNwV12z9czPkLWZx8dk9kmtYQ_RNwnY_Q@mail.gmail.com>
 <CADiSq7eiZku6BXEd3DtOGg4BRPNP3-QvtNNQbq0=SRf7eCJKYA@mail.gmail.com>
 <l20liu$gla$1@ger.gmane.org>
Message-ID: <CADiSq7fBvyQjNJp4u6AVaQRWkJfYFTBMX=Nb8fjSv=8JBG0H5w@mail.gmail.com>

On 26 September 2013 16:53, Georg Brandl <g.brandl at gmx.net> wrote:
>> Sure, that's doable, but it dumps the full repr of "obj" in the middle
>> of the sentence. The thing that's not practical is the neat and tidy
>> wording Georg proposed, because the thing passed as "obj" is actually
>> an arbitrary Python object that may have a messy repr (like a bound
>> method, which is what gets passed in the __del__ case), so there's
>> definite merit in keeping that repr at the *end* of the header line.
>
> Then this should be fine, I guess?
>
> Exception caught and not propagated in: <....>

Sure. I still prefer something like "Could not propagate exception
from:" or "Caller could not propagate exception from <repr>" that
better indicates we're suppressing it because it's infeasible to raise
it rather than just because we feel like it, but any of them would
offer a decent improvement over the status quo.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From eliben at gmail.com  Thu Sep 26 15:52:20 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Thu, 26 Sep 2013 06:52:20 -0700
Subject: [Python-Dev] asdl.py and Python-ast.[hc]
Message-ID: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com>

Hi All,

Earlier this morning I had a slight tackle with a couple of the 3.4 bots
(sorry everyone!). I fixed some problems in asdl.py -
http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with'
statement. Some bots don't have Python 2.6+ and couldn't bootstrap
Python-ast.h/c

Two questions:

* Should I always check-in Python-ast.h and Python-ast.c when I touch asdl*
? The generated files are unchanged, it's only the timestamp that changed.
* Can we, in theory, use new Pythons for asdl* code, because Python-ast.*
are, in fact, checked in so they don't have to be rebuilt by the bots or
users?

Thanks in advance,
Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130926/407f3bb2/attachment.html>

From brett at python.org  Thu Sep 26 16:00:15 2013
From: brett at python.org (Brett Cannon)
Date: Thu, 26 Sep 2013 10:00:15 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <D995BB4B-F2AA-40B7-BBF3-3DDF36CCB5B9@stufft.io>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
 <D995BB4B-F2AA-40B7-BBF3-3DDF36CCB5B9@stufft.io>
Message-ID: <CAP1=2W7bukJGw8BWgk389MkKbo-DHjRsbv+eZAS2FFH2+nguFg@mail.gmail.com>

On Wed, Sep 25, 2013 at 7:15 PM, Donald Stufft <donald at stufft.io> wrote:

>
> On Sep 25, 2013, at 7:04 PM, Barry Warsaw <barry at python.org> wrote:
>
> > Another reason to oppose this is what I've heard quite often from people
> > regarding Python 2.7.  I've been told that many folks are actually really
> > happy with using 2.7 precisely because it extremely stable.  They don't
> have
> > to worry about their stuff breaking or incompatibilities cropping up,
> because
> > it *doesn't* change.  Python 2.7 is like a long-term maintenance
> release, with
> > guaranteed multi-year stability, and I think while that was unintended,
> it's a
> > *good* thing.  This PEP proposes to break that, and I'm loathe to give
> up that
> > good reputation for this particular feature.
>
> Maybe I'm just naive but can you expand on how adding a module that nothing
> else in the system imports (besides the installers; but you weren't
> arguing against
> adding this to the installers) would break someones use any other module?
> If
> they don't import it (which the vast bulk of people won't directly, nor at
> all during
> the operation of their applications) how does it's existence on the file
> system
> risk a breakage to their system?


I think Barry's worry is precisely the fact that users do silly things, so
having a new module suddenly show up in the stdlib can be a problem if
people start using the module. We could conceivably try and not expose the
module on sys.path somehow so that it can't be directly imported by anyone
who doesn't have any business using it unless they jump through some hoops
(e.g. stick it in Tools for Python 2.7 so the installers can get access but
it won't end up in the stdlib install).

Otherwise if Barry is worrying about a command-line tool being installed
for some users of Python 2.7 vs. not then I don't know what the worry is
because that happens on OS upgrades as well, let alone people who just
happen to have done the install themselves vs. not. People are already told
to install pip anyway, so if instructions point to ensurepip on PyPI as the
canonical way to install pip then whether it's installed or not is fine as
it will be mostly a no-op for 2.7.6 users and an actual install for others.
But by including it then it does simplify teaching scenarios and long tail
helps get pip out faster and more easily.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130926/589b77d4/attachment.html>

From solipsis at pitrou.net  Thu Sep 26 16:10:46 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 26 Sep 2013 16:10:46 +0200
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <CADiSq7fVrruk1zcE-ZwHUOyUqOPhTBnn2jcp9rKQ8dS6QeL+Zw@mail.gmail.com>
 <20130925190854.6ac76200@anarchist>
 <CADiSq7cMMkTjArg=0=rmDypHMo5aACYNQX=ct18N9jzru2=JQQ@mail.gmail.com>
 <CADiSq7cmSqnWCL0-s-g5dc1AgRRkfhd0VjYVSDj47bWQh537NA@mail.gmail.com>
Message-ID: <20130926161046.17a6d4f6@pitrou.net>

Le Thu, 26 Sep 2013 14:54:49 +1000,
Nick Coghlan <ncoghlan at gmail.com> a ?crit :
> On 26 September 2013 14:30, Nick Coghlan <ncoghlan at gmail.com> wrote:
> > That said, there are changes that I think are definitely worth
> > making due to the concerns you raise:
> >
> > - the module name should be "_ensurepip" in all versions
> > - the PEP should explicitly state that the "don't remove _ensurepip
> > and it's wheel files" caveat for redistributors applies only in 3.4+
> > (where removing it will break pyvenv)
> 
> Donald pointed out it makes more sense to continue with the idea of a
> properly documented public "ensurepip" module in 3.4+, and have the
> "_ensurepip" version as an implementation detail of the 2.7 and 3.3
> installers that is included in the stdlib primarily so it can be
> covered by the existing buildbot fleet.

Hmm, but what is the point of "_ensurepip" exactly? Are people supposed
to type "python -m _ensurepip"?

With all due respect, Barry's argument looks rather paranoid to me.
I would suggest a clear choice:
- either having "ensurepip" in 2.7 is useful and we endorse it as a
  public module (not something hidden somewhere) - which I personally
  think is reasonable
- or it's not useful and we don't introduce it at all

A middleground doesn't make sense here, except in a broken "design by
committee" sense.

Regards

Antoine.



From brett at python.org  Thu Sep 26 16:18:30 2013
From: brett at python.org (Brett Cannon)
Date: Thu, 26 Sep 2013 10:18:30 -0400
Subject: [Python-Dev] asdl.py and Python-ast.[hc]
In-Reply-To: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com>
References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com>
Message-ID: <CAP1=2W5XB+iWi0Dv4Tzb1CMmn=25GBoaLZLsXMY1UfQG_ifF0w@mail.gmail.com>

On Thu, Sep 26, 2013 at 9:52 AM, Eli Bendersky <eliben at gmail.com> wrote:

> Hi All,
>
> Earlier this morning I had a slight tackle with a couple of the 3.4 bots
> (sorry everyone!). I fixed some problems in asdl.py -
> http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with'
> statement. Some bots don't have Python 2.6+ and couldn't bootstrap
> Python-ast.h/c
>
> Two questions:
>
> * Should I always check-in Python-ast.h and Python-ast.c when I touch
> asdl* ? The generated files are unchanged, it's only the timestamp that
> changed.
>

If Python-ast.* are checked in then yes.


> * Can we, in theory, use new Pythons for asdl* code, because Python-ast.*
> are, in fact, checked in so they don't have to be rebuilt by the bots or
> users?
>

I don't see why not. the touch extension for hg is there specifically for
these files to prevent having to regenerate them.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130926/6ed35b8c/attachment.html>

From donald at stufft.io  Thu Sep 26 16:22:55 2013
From: donald at stufft.io (Donald Stufft)
Date: Thu, 26 Sep 2013 10:22:55 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <20130926161046.17a6d4f6@pitrou.net>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <CADiSq7fVrruk1zcE-ZwHUOyUqOPhTBnn2jcp9rKQ8dS6QeL+Zw@mail.gmail.com>
 <20130925190854.6ac76200@anarchist>
 <CADiSq7cMMkTjArg=0=rmDypHMo5aACYNQX=ct18N9jzru2=JQQ@mail.gmail.com>
 <CADiSq7cmSqnWCL0-s-g5dc1AgRRkfhd0VjYVSDj47bWQh537NA@mail.gmail.com>
 <20130926161046.17a6d4f6@pitrou.net>
Message-ID: <97A4ED42-8961-4B01-BE20-DF684FC0930E@stufft.io>

Ideally people won't be typing either of them because it'll be installed automatically. They might in some cases (accidentally uninstalled pip?)

I agree that it seems there is paranoia going on here and that the risk is low and making it just be a special cased new feature is ok. However the point of the underscore prefix on 2.7 and 3.3 is to more effectively communicate that on these pythons you shouldn't rely on that module existing. I think that's worse situation then just making it ensurepip everywhere but better than 2.7 not getting it at all or Barry's suggestion. 

> On Sep 26, 2013, at 10:10 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> 
> Le Thu, 26 Sep 2013 14:54:49 +1000,
> Nick Coghlan <ncoghlan at gmail.com> a ?crit :
>> On 26 September 2013 14:30, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> That said, there are changes that I think are definitely worth
>>> making due to the concerns you raise:
>>> 
>>> - the module name should be "_ensurepip" in all versions
>>> - the PEP should explicitly state that the "don't remove _ensurepip
>>> and it's wheel files" caveat for redistributors applies only in 3.4+
>>> (where removing it will break pyvenv)
>> 
>> Donald pointed out it makes more sense to continue with the idea of a
>> properly documented public "ensurepip" module in 3.4+, and have the
>> "_ensurepip" version as an implementation detail of the 2.7 and 3.3
>> installers that is included in the stdlib primarily so it can be
>> covered by the existing buildbot fleet.
> 
> Hmm, but what is the point of "_ensurepip" exactly? Are people supposed
> to type "python -m _ensurepip"?
> 
> With all due respect, Barry's argument looks rather paranoid to me.
> I would suggest a clear choice:
> - either having "ensurepip" in 2.7 is useful and we endorse it as a
>  public module (not something hidden somewhere) - which I personally
>  think is reasonable
> - or it's not useful and we don't introduce it at all
> 
> A middleground doesn't make sense here, except in a broken "design by
> committee" sense.
> 
> Regards
> 
> Antoine.
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

From eliben at gmail.com  Thu Sep 26 16:27:05 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Thu, 26 Sep 2013 07:27:05 -0700
Subject: [Python-Dev] asdl.py and Python-ast.[hc]
In-Reply-To: <CAP1=2W5XB+iWi0Dv4Tzb1CMmn=25GBoaLZLsXMY1UfQG_ifF0w@mail.gmail.com>
References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com>
 <CAP1=2W5XB+iWi0Dv4Tzb1CMmn=25GBoaLZLsXMY1UfQG_ifF0w@mail.gmail.com>
Message-ID: <CAF-Rda_Wq1Be1K01Afow3sLM-cw6Dx56Y9uxwogtb7O-vLN9fQ@mail.gmail.com>

On Thu, Sep 26, 2013 at 7:18 AM, Brett Cannon <brett at python.org> wrote:

>
>
>
> On Thu, Sep 26, 2013 at 9:52 AM, Eli Bendersky <eliben at gmail.com> wrote:
>
>> Hi All,
>>
>> Earlier this morning I had a slight tackle with a couple of the 3.4 bots
>> (sorry everyone!). I fixed some problems in asdl.py -
>> http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with'
>> statement. Some bots don't have Python 2.6+ and couldn't bootstrap
>> Python-ast.h/c
>>
>> Two questions:
>>
>> * Should I always check-in Python-ast.h and Python-ast.c when I touch
>> asdl* ? The generated files are unchanged, it's only the timestamp that
>> changed.
>>
>
> If Python-ast.* are checked in then yes.
>

Hmm, how do I do that? hg doesn't accept them for commit since their
contents are not changed.


>
>
>> * Can we, in theory, use new Pythons for asdl* code, because Python-ast.*
>> are, in fact, checked in so they don't have to be rebuilt by the bots or
>> users?
>>
>
> I don't see why not. the touch extension for hg is there specifically for
> these files to prevent having to regenerate them.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130926/2d229fc4/attachment.html>

From benjamin at python.org  Thu Sep 26 16:28:46 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Thu, 26 Sep 2013 10:28:46 -0400
Subject: [Python-Dev] asdl.py and Python-ast.[hc]
In-Reply-To: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com>
References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com>
Message-ID: <CAPZV6o_y-8f0TJgEC=YKg+ZkV6vEzhet0U3rqV9DXo-PBaw=3g@mail.gmail.com>

2013/9/26 Eli Bendersky <eliben at gmail.com>:
> Hi All,
>
> Earlier this morning I had a slight tackle with a couple of the 3.4 bots
> (sorry everyone!). I fixed some problems in asdl.py -
> http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with'
> statement. Some bots don't have Python 2.6+ and couldn't bootstrap
> Python-ast.h/c
>
> Two questions:
>
> * Should I always check-in Python-ast.h and Python-ast.c when I touch asdl*
> ? The generated files are unchanged, it's only the timestamp that changed.
> * Can we, in theory, use new Pythons for asdl* code, because Python-ast.*
> are, in fact, checked in so they don't have to be rebuilt by the bots or
> users?

We should have the buildbots run "make touch", so they don't need to
run asdl_c.py.

-- 
Regards,
Benjamin

From solipsis at pitrou.net  Thu Sep 26 16:28:42 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 26 Sep 2013 16:28:42 +0200
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <CADiSq7fVrruk1zcE-ZwHUOyUqOPhTBnn2jcp9rKQ8dS6QeL+Zw@mail.gmail.com>
 <20130925190854.6ac76200@anarchist>
 <CADiSq7cMMkTjArg=0=rmDypHMo5aACYNQX=ct18N9jzru2=JQQ@mail.gmail.com>
 <CADiSq7cmSqnWCL0-s-g5dc1AgRRkfhd0VjYVSDj47bWQh537NA@mail.gmail.com>
 <20130926161046.17a6d4f6@pitrou.net>
 <97A4ED42-8961-4B01-BE20-DF684FC0930E@stufft.io>
Message-ID: <20130926162842.58b74a5c@pitrou.net>

Le Thu, 26 Sep 2013 10:22:55 -0400,
Donald Stufft <donald at stufft.io> a ?crit :
> Ideally people won't be typing either of them because it'll be
> installed automatically. They might in some cases (accidentally
> uninstalled pip?)

Installing from source perhaps.

> I agree that it seems there is paranoia going on here and that the
> risk is low and making it just be a special cased new feature is ok.
> However the point of the underscore prefix on 2.7 and 3.3 is to more
> effectively communicate that on these pythons you shouldn't rely on
> that module existing.

That's a fair argument, although it also makes the thing a bit ugly.

Regards

Antoine.



From g.brandl at gmx.net  Thu Sep 26 18:07:39 2013
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 26 Sep 2013 18:07:39 +0200
Subject: [Python-Dev] cpython: Don't use fancy new Python features like
 'with' - some bots don't have them
In-Reply-To: <3cly4S54LQz7Lk4@mail.python.org>
References: <3cly4S54LQz7Lk4@mail.python.org>
Message-ID: <52445BCB.10501@gmx.net>

Am 26.09.2013 15:42, schrieb eli.bendersky:
> http://hg.python.org/cpython/rev/931d95e9067f
> changeset:   85801:931d95e9067f
> user:        Eli Bendersky <eliben at gmail.com>
> date:        Thu Sep 26 06:41:36 2013 -0700
> summary:
>   Don't use fancy new Python features like 'with' - some bots don't have them
> and can't bootstrap the parser.
> 
> files:
>   Parser/asdl.py |  5 ++++-
>   1 files changed, 4 insertions(+), 1 deletions(-)
> 
> 
> diff --git a/Parser/asdl.py b/Parser/asdl.py
> --- a/Parser/asdl.py
> +++ b/Parser/asdl.py
> @@ -398,8 +398,11 @@
>      scanner = ASDLScanner()
>      parser = ASDLParser()
>  
> -    with open(file) as f:
> +    try:
> +        f = open(file)
>          buf = f.read()
> +    finally:
> +        f.close()
>      tokens = scanner.tokenize(buf)
>      try:
>          return parser.parse(tokens)

The open call needs to go outside the try-finally.

cheers,
Georg

From eliben at gmail.com  Thu Sep 26 18:36:07 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Thu, 26 Sep 2013 09:36:07 -0700
Subject: [Python-Dev] cpython: Don't use fancy new Python features like
 'with' - some bots don't have them
In-Reply-To: <52445BCB.10501@gmx.net>
References: <3cly4S54LQz7Lk4@mail.python.org> <52445BCB.10501@gmx.net>
Message-ID: <CAF-Rda8WQmQqsRe_MaeBzzVb9mEAkvhue7dAtbOujPsuML2Z5w@mail.gmail.com>

On Thu, Sep 26, 2013 at 9:07 AM, Georg Brandl <g.brandl at gmx.net> wrote:

> Am 26.09.2013 15:42, schrieb eli.bendersky:
> > http://hg.python.org/cpython/rev/931d95e9067f
> > changeset:   85801:931d95e9067f
> > user:        Eli Bendersky <eliben at gmail.com>
> > date:        Thu Sep 26 06:41:36 2013 -0700
> > summary:
> >   Don't use fancy new Python features like 'with' - some bots don't have
> them
> > and can't bootstrap the parser.
> >
> > files:
> >   Parser/asdl.py |  5 ++++-
> >   1 files changed, 4 insertions(+), 1 deletions(-)
> >
> >
> > diff --git a/Parser/asdl.py b/Parser/asdl.py
> > --- a/Parser/asdl.py
> > +++ b/Parser/asdl.py
> > @@ -398,8 +398,11 @@
> >      scanner = ASDLScanner()
> >      parser = ASDLParser()
> >
> > -    with open(file) as f:
> > +    try:
> > +        f = open(file)
> >          buf = f.read()
> > +    finally:
> > +        f.close()
> >      tokens = scanner.tokenize(buf)
> >      try:
> >          return parser.parse(tokens)
>
> The open call needs to go outside the try-finally.\
>

Done, thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130926/721d6715/attachment-0001.html>

From eliben at gmail.com  Thu Sep 26 18:37:03 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Thu, 26 Sep 2013 09:37:03 -0700
Subject: [Python-Dev] asdl.py and Python-ast.[hc]
In-Reply-To: <CAPZV6o_y-8f0TJgEC=YKg+ZkV6vEzhet0U3rqV9DXo-PBaw=3g@mail.gmail.com>
References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com>
 <CAPZV6o_y-8f0TJgEC=YKg+ZkV6vEzhet0U3rqV9DXo-PBaw=3g@mail.gmail.com>
Message-ID: <CAF-Rda_vthC4GdvJARvs75TYZTUNPLWbF2a4Uvm=7_V67bZvXg@mail.gmail.com>

On Thu, Sep 26, 2013 at 7:28 AM, Benjamin Peterson <benjamin at python.org>wrote:

> 2013/9/26 Eli Bendersky <eliben at gmail.com>:
> > Hi All,
> >
> > Earlier this morning I had a slight tackle with a couple of the 3.4 bots
> > (sorry everyone!). I fixed some problems in asdl.py -
> > http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with'
> > statement. Some bots don't have Python 2.6+ and couldn't bootstrap
> > Python-ast.h/c
> >
> > Two questions:
> >
> > * Should I always check-in Python-ast.h and Python-ast.c when I touch
> asdl*
> > ? The generated files are unchanged, it's only the timestamp that
> changed.
> > * Can we, in theory, use new Pythons for asdl* code, because Python-ast.*
> > are, in fact, checked in so they don't have to be rebuilt by the bots or
> > users?
>
> We should have the buildbots run "make touch", so they don't need to
> run asdl_c.py.
>

How should we go about doing this? I don't think we have it documented in
the devguide how to tweak the buildbot scripts?

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130926/2c4cee6e/attachment.html>

From donald at stufft.io  Thu Sep 26 18:40:39 2013
From: donald at stufft.io (Donald Stufft)
Date: Thu, 26 Sep 2013 12:40:39 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <20130926162842.58b74a5c@pitrou.net>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <CADiSq7fVrruk1zcE-ZwHUOyUqOPhTBnn2jcp9rKQ8dS6QeL+Zw@mail.gmail.com>
 <20130925190854.6ac76200@anarchist>
 <CADiSq7cMMkTjArg=0=rmDypHMo5aACYNQX=ct18N9jzru2=JQQ@mail.gmail.com>
 <CADiSq7cmSqnWCL0-s-g5dc1AgRRkfhd0VjYVSDj47bWQh537NA@mail.gmail.com>
 <20130926161046.17a6d4f6@pitrou.net>
 <97A4ED42-8961-4B01-BE20-DF684FC0930E@stufft.io>
 <20130926162842.58b74a5c@pitrou.net>
Message-ID: <1201A938-D911-4541-8B92-1B9B7000FE6A@stufft.io>


On Sep 26, 2013, at 10:28 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> Le Thu, 26 Sep 2013 10:22:55 -0400,
> Donald Stufft <donald at stufft.io> a ?crit :
>> Ideally people won't be typing either of them because it'll be
>> installed automatically. They might in some cases (accidentally
>> uninstalled pip?)
> 
> Installing from source perhaps.

Ah yea, installing from source is the time when they'd execute it,
that slipped my mind.

> 
>> I agree that it seems there is paranoia going on here and that the
>> risk is low and making it just be a special cased new feature is ok.
>> However the point of the underscore prefix on 2.7 and 3.3 is to more
>> effectively communicate that on these pythons you shouldn't rely on
>> that module existing.
> 
> That's a fair argument, although it also makes the thing a bit ugly.

Yea It's not my preferred thing to do and I think that calling it just ensurepip
is fine, however I think if folks are worried about it, it could offer a compromise.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130926/8728ea81/attachment.sig>

From benjamin at python.org  Thu Sep 26 20:43:48 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Thu, 26 Sep 2013 14:43:48 -0400
Subject: [Python-Dev] asdl.py and Python-ast.[hc]
In-Reply-To: <CAF-Rda_vthC4GdvJARvs75TYZTUNPLWbF2a4Uvm=7_V67bZvXg@mail.gmail.com>
References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com>
 <CAPZV6o_y-8f0TJgEC=YKg+ZkV6vEzhet0U3rqV9DXo-PBaw=3g@mail.gmail.com>
 <CAF-Rda_vthC4GdvJARvs75TYZTUNPLWbF2a4Uvm=7_V67bZvXg@mail.gmail.com>
Message-ID: <CAPZV6o9kt3qONZK=_y2pJCou0iBPrqpQeQ9vhVYkyof+3caxrA@mail.gmail.com>

2013/9/26 Eli Bendersky <eliben at gmail.com>:
>
>
>
> On Thu, Sep 26, 2013 at 7:28 AM, Benjamin Peterson <benjamin at python.org>
> wrote:
>>
>> 2013/9/26 Eli Bendersky <eliben at gmail.com>:
>> > Hi All,
>> >
>> > Earlier this morning I had a slight tackle with a couple of the 3.4 bots
>> > (sorry everyone!). I fixed some problems in asdl.py -
>> > http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with'
>> > statement. Some bots don't have Python 2.6+ and couldn't bootstrap
>> > Python-ast.h/c
>> >
>> > Two questions:
>> >
>> > * Should I always check-in Python-ast.h and Python-ast.c when I touch
>> > asdl*
>> > ? The generated files are unchanged, it's only the timestamp that
>> > changed.
>> > * Can we, in theory, use new Pythons for asdl* code, because
>> > Python-ast.*
>> > are, in fact, checked in so they don't have to be rebuilt by the bots or
>> > users?
>>
>> We should have the buildbots run "make touch", so they don't need to
>> run asdl_c.py.
>
>
> How should we go about doing this?

Complain to Antoine I suppose. :)


-- 
Regards,
Benjamin

From eliben at gmail.com  Thu Sep 26 20:46:07 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Thu, 26 Sep 2013 11:46:07 -0700
Subject: [Python-Dev] asdl.py and Python-ast.[hc]
In-Reply-To: <CAPZV6o9kt3qONZK=_y2pJCou0iBPrqpQeQ9vhVYkyof+3caxrA@mail.gmail.com>
References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com>
 <CAPZV6o_y-8f0TJgEC=YKg+ZkV6vEzhet0U3rqV9DXo-PBaw=3g@mail.gmail.com>
 <CAF-Rda_vthC4GdvJARvs75TYZTUNPLWbF2a4Uvm=7_V67bZvXg@mail.gmail.com>
 <CAPZV6o9kt3qONZK=_y2pJCou0iBPrqpQeQ9vhVYkyof+3caxrA@mail.gmail.com>
Message-ID: <CAF-Rda-d1_0So4JYCw9Zo6EuShRVoEkx1=x2J_ZRru2XpGMBkQ@mail.gmail.com>

> >> 2013/9/26 Eli Bendersky <eliben at gmail.com>:
> >> > Hi All,
> >> >
> >> > Earlier this morning I had a slight tackle with a couple of the 3.4
> bots
> >> > (sorry everyone!). I fixed some problems in asdl.py -
> >> > http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with'
> >> > statement. Some bots don't have Python 2.6+ and couldn't bootstrap
> >> > Python-ast.h/c
> >> >
> >> > Two questions:
> >> >
> >> > * Should I always check-in Python-ast.h and Python-ast.c when I touch
> >> > asdl*
> >> > ? The generated files are unchanged, it's only the timestamp that
> >> > changed.
> >> > * Can we, in theory, use new Pythons for asdl* code, because
> >> > Python-ast.*
> >> > are, in fact, checked in so they don't have to be rebuilt by the bots
> or
> >> > users?
> >>
> >> We should have the buildbots run "make touch", so they don't need to
> >> run asdl_c.py.
> >
> >
> > How should we go about doing this?
>
> Complain to Antoine I suppose. :)
>
[+ Antoine]
Done :)

While we're at it, it seems that .hgtouch is wrong:

Include/ast.h: Parser/Python.asdl Parser/asdl.py Parser/asdl_c.py
Python/Python-ast.c: Include/ast.h

The file Include/ast.h is not, in fact, auto-generated. But
Include/Python-ast.h *is*, and it does not appear in this file.

Typo? [+Ezio]

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130926/6c4cb3a2/attachment.html>

From tjreedy at udel.edu  Thu Sep 26 21:51:25 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 26 Sep 2013 15:51:25 -0400
Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error
	message
In-Reply-To: <CADiSq7fBvyQjNJp4u6AVaQRWkJfYFTBMX=Nb8fjSv=8JBG0H5w@mail.gmail.com>
References: <5241BED1.7010905@g.nevcal.com> <l1tvd9$6rg$1@ger.gmane.org>
 <20130925230520.GA94785@cskk.homeip.net>
 <CADiSq7fywLacQywujMma+HTov948FXgmtJ51gRon_iU4Fc2nxg@mail.gmail.com>
 <CAMSv6X1tQGPhLs0rnvNwV12z9czPkLWZx8dk9kmtYQ_RNwnY_Q@mail.gmail.com>
 <CADiSq7eiZku6BXEd3DtOGg4BRPNP3-QvtNNQbq0=SRf7eCJKYA@mail.gmail.com>
 <l20liu$gla$1@ger.gmane.org>
 <CADiSq7fBvyQjNJp4u6AVaQRWkJfYFTBMX=Nb8fjSv=8JBG0H5w@mail.gmail.com>
Message-ID: <l2237l$u8h$1@ger.gmane.org>

On 9/26/2013 3:17 AM, Nick Coghlan wrote:
> On 26 September 2013 16:53, Georg Brandl <g.brandl at gmx.net> wrote:
>>> Sure, that's doable, but it dumps the full repr of "obj" in the middle
>>> of the sentence. The thing that's not practical is the neat and tidy
>>> wording Georg proposed, because the thing passed as "obj" is actually
>>> an arbitrary Python object that may have a messy repr (like a bound
>>> method, which is what gets passed in the __del__ case), so there's
>>> definite merit in keeping that repr at the *end* of the header line.
>>
>> Then this should be fine, I guess?
>>
>> Exception caught and not propagated in: <....>
>
> Sure. I still prefer something like "Could not propagate exception
> from:" or "Caller could not propagate exception from <repr>" that
> better indicates we're suppressing it because it's infeasible to raise
> it rather than just because we feel like it, but any of them would
> offer a decent improvement over the status quo.

With the full traceback printed, with the line where the exception 
originated, I do not think that the representation of the object is 
needed. It was a substitute for the traceback.

-- 
Terry Jan Reedy


From solipsis at pitrou.net  Thu Sep 26 22:45:22 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 26 Sep 2013 22:45:22 +0200
Subject: [Python-Dev] asdl.py and Python-ast.[hc]
In-Reply-To: <CAF-Rda-d1_0So4JYCw9Zo6EuShRVoEkx1=x2J_ZRru2XpGMBkQ@mail.gmail.com>
References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com>
 <CAPZV6o_y-8f0TJgEC=YKg+ZkV6vEzhet0U3rqV9DXo-PBaw=3g@mail.gmail.com>
 <CAF-Rda_vthC4GdvJARvs75TYZTUNPLWbF2a4Uvm=7_V67bZvXg@mail.gmail.com>
 <CAPZV6o9kt3qONZK=_y2pJCou0iBPrqpQeQ9vhVYkyof+3caxrA@mail.gmail.com>
 <CAF-Rda-d1_0So4JYCw9Zo6EuShRVoEkx1=x2J_ZRru2XpGMBkQ@mail.gmail.com>
Message-ID: <20130926224522.58719924@fsol>

On Thu, 26 Sep 2013 11:46:07 -0700
Eli Bendersky <eliben at gmail.com> wrote:

> > >> 2013/9/26 Eli Bendersky <eliben at gmail.com>:
> > >> > Hi All,
> > >> >
> > >> > Earlier this morning I had a slight tackle with a couple of the 3.4
> > bots
> > >> > (sorry everyone!). I fixed some problems in asdl.py -
> > >> > http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with'
> > >> > statement. Some bots don't have Python 2.6+ and couldn't bootstrap
> > >> > Python-ast.h/c
> > >> >
> > >> > Two questions:
> > >> >
> > >> > * Should I always check-in Python-ast.h and Python-ast.c when I touch
> > >> > asdl*
> > >> > ? The generated files are unchanged, it's only the timestamp that
> > >> > changed.
> > >> > * Can we, in theory, use new Pythons for asdl* code, because
> > >> > Python-ast.*
> > >> > are, in fact, checked in so they don't have to be rebuilt by the bots
> > or
> > >> > users?
> > >>
> > >> We should have the buildbots run "make touch", so they don't need to
> > >> run asdl_c.py.
> > >
> > >
> > > How should we go about doing this?
> >
> > Complain to Antoine I suppose. :)
> >
> [+ Antoine]
> Done :)

"make touch"? Wow, I'm learning new stuff everyday :-)

(what point is there in adding things to Makefile if nobody knows about
them?)

cheers

Antoine.

From eliben at gmail.com  Thu Sep 26 22:49:58 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Thu, 26 Sep 2013 13:49:58 -0700
Subject: [Python-Dev] asdl.py and Python-ast.[hc]
In-Reply-To: <20130926224522.58719924@fsol>
References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com>
 <CAPZV6o_y-8f0TJgEC=YKg+ZkV6vEzhet0U3rqV9DXo-PBaw=3g@mail.gmail.com>
 <CAF-Rda_vthC4GdvJARvs75TYZTUNPLWbF2a4Uvm=7_V67bZvXg@mail.gmail.com>
 <CAPZV6o9kt3qONZK=_y2pJCou0iBPrqpQeQ9vhVYkyof+3caxrA@mail.gmail.com>
 <CAF-Rda-d1_0So4JYCw9Zo6EuShRVoEkx1=x2J_ZRru2XpGMBkQ@mail.gmail.com>
 <20130926224522.58719924@fsol>
Message-ID: <CAF-Rda-mfUSkWwyZYaqW2_2xJSemaP8vUDytSDY98NpwDXsAMw@mail.gmail.com>

On Thu, Sep 26, 2013 at 1:45 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Thu, 26 Sep 2013 11:46:07 -0700
> Eli Bendersky <eliben at gmail.com> wrote:
>
> > > >> 2013/9/26 Eli Bendersky <eliben at gmail.com>:
> > > >> > Hi All,
> > > >> >
> > > >> > Earlier this morning I had a slight tackle with a couple of the
> 3.4
> > > bots
> > > >> > (sorry everyone!). I fixed some problems in asdl.py -
> > > >> > http://hg.python.org/cpython/rev/21d46e3ae60c - and used the
> 'with'
> > > >> > statement. Some bots don't have Python 2.6+ and couldn't bootstrap
> > > >> > Python-ast.h/c
> > > >> >
> > > >> > Two questions:
> > > >> >
> > > >> > * Should I always check-in Python-ast.h and Python-ast.c when I
> touch
> > > >> > asdl*
> > > >> > ? The generated files are unchanged, it's only the timestamp that
> > > >> > changed.
> > > >> > * Can we, in theory, use new Pythons for asdl* code, because
> > > >> > Python-ast.*
> > > >> > are, in fact, checked in so they don't have to be rebuilt by the
> bots
> > > or
> > > >> > users?
> > > >>
> > > >> We should have the buildbots run "make touch", so they don't need to
> > > >> run asdl_c.py.
> > > >
> > > >
> > > > How should we go about doing this?
> > >
> > > Complain to Antoine I suppose. :)
> > >
> > [+ Antoine]
> > Done :)
>
> "make touch"? Wow, I'm learning new stuff everyday :-)
>
> (what point is there in adding things to Makefile if nobody knows about
> them?)
>

The devguide would be a good place to mention this.
http://bugs.python.org/issue15964 tracked the task for a while, but I'm not
sure what the status is.

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130926/cfe9adcb/attachment.html>

From solipsis at pitrou.net  Thu Sep 26 23:00:33 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 26 Sep 2013 23:00:33 +0200
Subject: [Python-Dev] asdl.py and Python-ast.[hc]
References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com>
 <CAPZV6o_y-8f0TJgEC=YKg+ZkV6vEzhet0U3rqV9DXo-PBaw=3g@mail.gmail.com>
 <CAF-Rda_vthC4GdvJARvs75TYZTUNPLWbF2a4Uvm=7_V67bZvXg@mail.gmail.com>
 <CAPZV6o9kt3qONZK=_y2pJCou0iBPrqpQeQ9vhVYkyof+3caxrA@mail.gmail.com>
Message-ID: <20130926230033.7bd7c584@fsol>

On Thu, 26 Sep 2013 14:43:48 -0400
Benjamin Peterson <benjamin at python.org> wrote:
> 2013/9/26 Eli Bendersky <eliben at gmail.com>:
> >
> >
> >
> > On Thu, Sep 26, 2013 at 7:28 AM, Benjamin Peterson <benjamin at python.org>
> > wrote:
> >>
> >> 2013/9/26 Eli Bendersky <eliben at gmail.com>:
> >> > Hi All,
> >> >
> >> > Earlier this morning I had a slight tackle with a couple of the 3.4 bots
> >> > (sorry everyone!). I fixed some problems in asdl.py -
> >> > http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with'
> >> > statement. Some bots don't have Python 2.6+ and couldn't bootstrap
> >> > Python-ast.h/c
> >> >
> >> > Two questions:
> >> >
> >> > * Should I always check-in Python-ast.h and Python-ast.c when I touch
> >> > asdl*
> >> > ? The generated files are unchanged, it's only the timestamp that
> >> > changed.
> >> > * Can we, in theory, use new Pythons for asdl* code, because
> >> > Python-ast.*
> >> > are, in fact, checked in so they don't have to be rebuilt by the bots or
> >> > users?
> >>
> >> We should have the buildbots run "make touch", so they don't need to
> >> run asdl_c.py.
> >
> >
> > How should we go about doing this?
> 
> Complain to Antoine I suppose. :)

Here you are:
http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/28/steps/compile/logs/stdio

Of course, when it's using "-jN" there may be a race condition :-)

Regards

Antoine.



From eliben at gmail.com  Thu Sep 26 23:19:59 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Thu, 26 Sep 2013 14:19:59 -0700
Subject: [Python-Dev] asdl.py and Python-ast.[hc]
In-Reply-To: <20130926230033.7bd7c584@fsol>
References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com>
 <CAPZV6o_y-8f0TJgEC=YKg+ZkV6vEzhet0U3rqV9DXo-PBaw=3g@mail.gmail.com>
 <CAF-Rda_vthC4GdvJARvs75TYZTUNPLWbF2a4Uvm=7_V67bZvXg@mail.gmail.com>
 <CAPZV6o9kt3qONZK=_y2pJCou0iBPrqpQeQ9vhVYkyof+3caxrA@mail.gmail.com>
 <20130926230033.7bd7c584@fsol>
Message-ID: <CAF-Rda-x35WUuXns-WEuKvKkr=D0=KajyZr2KrhMFn-JNwMVpQ@mail.gmail.com>

On Thu, Sep 26, 2013 at 2:00 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Thu, 26 Sep 2013 14:43:48 -0400
> Benjamin Peterson <benjamin at python.org> wrote:
> > 2013/9/26 Eli Bendersky <eliben at gmail.com>:
> > >
> > >
> > >
> > > On Thu, Sep 26, 2013 at 7:28 AM, Benjamin Peterson <
> benjamin at python.org>
> > > wrote:
> > >>
> > >> 2013/9/26 Eli Bendersky <eliben at gmail.com>:
> > >> > Hi All,
> > >> >
> > >> > Earlier this morning I had a slight tackle with a couple of the 3.4
> bots
> > >> > (sorry everyone!). I fixed some problems in asdl.py -
> > >> > http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with'
> > >> > statement. Some bots don't have Python 2.6+ and couldn't bootstrap
> > >> > Python-ast.h/c
> > >> >
> > >> > Two questions:
> > >> >
> > >> > * Should I always check-in Python-ast.h and Python-ast.c when I
> touch
> > >> > asdl*
> > >> > ? The generated files are unchanged, it's only the timestamp that
> > >> > changed.
> > >> > * Can we, in theory, use new Pythons for asdl* code, because
> > >> > Python-ast.*
> > >> > are, in fact, checked in so they don't have to be rebuilt by the
> bots or
> > >> > users?
> > >>
> > >> We should have the buildbots run "make touch", so they don't need to
> > >> run asdl_c.py.
> > >
> > >
> > > How should we go about doing this?
> >
> > Complain to Antoine I suppose. :)
>
> Here you are:
>
> http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/28/steps/compile/logs/stdio
>
> Of course, when it's using "-jN" there may be a race condition :-)
>

Right, which is probably even worse than before because now it's
non-deterministic (first build will fail, subsequent will succeed).

Is there a way to split it to:

$ make touch
$ make -jN all


?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130926/412467a4/attachment-0001.html>

From ncoghlan at gmail.com  Thu Sep 26 23:20:49 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 27 Sep 2013 07:20:49 +1000
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <20130926161046.17a6d4f6@pitrou.net>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <CADiSq7fVrruk1zcE-ZwHUOyUqOPhTBnn2jcp9rKQ8dS6QeL+Zw@mail.gmail.com>
 <20130925190854.6ac76200@anarchist>
 <CADiSq7cMMkTjArg=0=rmDypHMo5aACYNQX=ct18N9jzru2=JQQ@mail.gmail.com>
 <CADiSq7cmSqnWCL0-s-g5dc1AgRRkfhd0VjYVSDj47bWQh537NA@mail.gmail.com>
 <20130926161046.17a6d4f6@pitrou.net>
Message-ID: <CADiSq7c2BF1+p5GoVQ_Ftnqtk0dfPrzXXurg2uyuKW7gTQLtYg@mail.gmail.com>

On 27 Sep 2013 00:12, "Antoine Pitrou" <solipsis at pitrou.net> wrote:
>
> Le Thu, 26 Sep 2013 14:54:49 +1000,
> Nick Coghlan <ncoghlan at gmail.com> a ?crit :
> > On 26 September 2013 14:30, Nick Coghlan <ncoghlan at gmail.com> wrote:
> > > That said, there are changes that I think are definitely worth
> > > making due to the concerns you raise:
> > >
> > > - the module name should be "_ensurepip" in all versions
> > > - the PEP should explicitly state that the "don't remove _ensurepip
> > > and it's wheel files" caveat for redistributors applies only in 3.4+
> > > (where removing it will break pyvenv)
> >
> > Donald pointed out it makes more sense to continue with the idea of a
> > properly documented public "ensurepip" module in 3.4+, and have the
> > "_ensurepip" version as an implementation detail of the 2.7 and 3.3
> > installers that is included in the stdlib primarily so it can be
> > covered by the existing buildbot fleet.
>
> Hmm, but what is the point of "_ensurepip" exactly? Are people supposed
> to type "python -m _ensurepip"?

>
> With all due respect, Barry's argument looks rather paranoid to me.
> I would suggest a clear choice:
> - either having "ensurepip" in 2.7 is useful and we endorse it as a
>   public module (not something hidden somewhere) - which I personally
>   think is reasonable
> - or it's not useful and we don't introduce it at all
>
> A middleground doesn't make sense here, except in a broken "design by
> committee" sense.

The *only* reason to have _ensurepip in the older versions is to make the
"install pip" option in the installers work. By contrast, in 3.4+ it will
be integrated into pyvenv and a fully supported API.

The two levels make sense to me, especially since pyvenv in 3.3 *won't* be
updated to invoke it automatically. 2.7 and 3.3 users will still need to
install and use virtualenv to get pip automatically installed in virtual
environments.

Cheers,
Nick.

>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/85418481/attachment.html>

From solipsis at pitrou.net  Thu Sep 26 23:23:42 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 26 Sep 2013 23:23:42 +0200
Subject: [Python-Dev] asdl.py and Python-ast.[hc]
In-Reply-To: <CAF-Rda-x35WUuXns-WEuKvKkr=D0=KajyZr2KrhMFn-JNwMVpQ@mail.gmail.com>
References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com>
 <CAPZV6o_y-8f0TJgEC=YKg+ZkV6vEzhet0U3rqV9DXo-PBaw=3g@mail.gmail.com>
 <CAF-Rda_vthC4GdvJARvs75TYZTUNPLWbF2a4Uvm=7_V67bZvXg@mail.gmail.com>
 <CAPZV6o9kt3qONZK=_y2pJCou0iBPrqpQeQ9vhVYkyof+3caxrA@mail.gmail.com>
 <20130926230033.7bd7c584@fsol>
 <CAF-Rda-x35WUuXns-WEuKvKkr=D0=KajyZr2KrhMFn-JNwMVpQ@mail.gmail.com>
Message-ID: <20130926232342.2da80669@fsol>

On Thu, 26 Sep 2013 14:19:59 -0700
Eli Bendersky <eliben at gmail.com> wrote:
> On Thu, Sep 26, 2013 at 2:00 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> 
> > On Thu, 26 Sep 2013 14:43:48 -0400
> > Benjamin Peterson <benjamin at python.org> wrote:
> > > 2013/9/26 Eli Bendersky <eliben at gmail.com>:
> > > >
> > > >
> > > >
> > > > On Thu, Sep 26, 2013 at 7:28 AM, Benjamin Peterson <
> > benjamin at python.org>
> > > > wrote:
> > > >>
> > > >> 2013/9/26 Eli Bendersky <eliben at gmail.com>:
> > > >> > Hi All,
> > > >> >
> > > >> > Earlier this morning I had a slight tackle with a couple of the 3.4
> > bots
> > > >> > (sorry everyone!). I fixed some problems in asdl.py -
> > > >> > http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with'
> > > >> > statement. Some bots don't have Python 2.6+ and couldn't bootstrap
> > > >> > Python-ast.h/c
> > > >> >
> > > >> > Two questions:
> > > >> >
> > > >> > * Should I always check-in Python-ast.h and Python-ast.c when I
> > touch
> > > >> > asdl*
> > > >> > ? The generated files are unchanged, it's only the timestamp that
> > > >> > changed.
> > > >> > * Can we, in theory, use new Pythons for asdl* code, because
> > > >> > Python-ast.*
> > > >> > are, in fact, checked in so they don't have to be rebuilt by the
> > bots or
> > > >> > users?
> > > >>
> > > >> We should have the buildbots run "make touch", so they don't need to
> > > >> run asdl_c.py.
> > > >
> > > >
> > > > How should we go about doing this?
> > >
> > > Complain to Antoine I suppose. :)
> >
> > Here you are:
> >
> > http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/28/steps/compile/logs/stdio
> >
> > Of course, when it's using "-jN" there may be a race condition :-)
> >
> 
> Right, which is probably even worse than before because now it's
> non-deterministic (first build will fail, subsequent will succeed).

Only if "N" in "-jN" is big enough, I suspect.

> Is there a way to split it to:
> 
> $ make touch
> $ make -jN all

Probably but... I'm really too lazy to do this myself. Buildbot isn't
very pleasant. Perhaps someone else wants to do it?

Regards

Antoine.

From solipsis at pitrou.net  Fri Sep 27 14:09:50 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 27 Sep 2013 14:09:50 +0200
Subject: [Python-Dev] asdl.py and Python-ast.[hc]
References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com>
 <CAPZV6o_y-8f0TJgEC=YKg+ZkV6vEzhet0U3rqV9DXo-PBaw=3g@mail.gmail.com>
Message-ID: <20130927140950.25c11567@pitrou.net>

Le Thu, 26 Sep 2013 10:28:46 -0400,
Benjamin Peterson <benjamin at python.org> a ?crit :
> 2013/9/26 Eli Bendersky <eliben at gmail.com>:
> > Hi All,
> >
> > Earlier this morning I had a slight tackle with a couple of the 3.4
> > bots (sorry everyone!). I fixed some problems in asdl.py -
> > http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with'
> > statement. Some bots don't have Python 2.6+ and couldn't bootstrap
> > Python-ast.h/c
> >
> > Two questions:
> >
> > * Should I always check-in Python-ast.h and Python-ast.c when I
> > touch asdl* ? The generated files are unchanged, it's only the
> > timestamp that changed.
> > * Can we, in theory, use new Pythons for asdl* code, because
> > Python-ast.* are, in fact, checked in so they don't have to be
> > rebuilt by the bots or users?
> 
> We should have the buildbots run "make touch", so they don't need to
> run asdl_c.py.

So, you're gonna like this:

hg --config extensions.touch=Tools/hg/hgtouch.py touch -v
Tools/hg/hgtouch.py:21: Warning: 'with' will become a reserved keyword in Python 2.6
*** failed to import extension touch from Tools/hg/hgtouch.py: invalid syntax (hgtouch.py, line 21)
hg: unknown command 'touch'

http://buildbot.python.org/all/buildslaves/bolen-ubuntu

(that buildbot runs Ubuntu 8.04 LTS, for the record)

Regards

Antoine.



From eliben at gmail.com  Fri Sep 27 15:46:44 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Fri, 27 Sep 2013 06:46:44 -0700
Subject: [Python-Dev] asdl.py and Python-ast.[hc]
In-Reply-To: <20130927140950.25c11567@pitrou.net>
References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com>
 <CAPZV6o_y-8f0TJgEC=YKg+ZkV6vEzhet0U3rqV9DXo-PBaw=3g@mail.gmail.com>
 <20130927140950.25c11567@pitrou.net>
Message-ID: <CAF-Rda-dcwFA__pa2qA-EMMih5f2Xmr5EvXL0p=WWvtJi2YGyg@mail.gmail.com>

On Fri, Sep 27, 2013 at 5:09 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> Le Thu, 26 Sep 2013 10:28:46 -0400,
> Benjamin Peterson <benjamin at python.org> a ?crit :
> > 2013/9/26 Eli Bendersky <eliben at gmail.com>:
> > > Hi All,
> > >
> > > Earlier this morning I had a slight tackle with a couple of the 3.4
> > > bots (sorry everyone!). I fixed some problems in asdl.py -
> > > http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with'
> > > statement. Some bots don't have Python 2.6+ and couldn't bootstrap
> > > Python-ast.h/c
> > >
> > > Two questions:
> > >
> > > * Should I always check-in Python-ast.h and Python-ast.c when I
> > > touch asdl* ? The generated files are unchanged, it's only the
> > > timestamp that changed.
> > > * Can we, in theory, use new Pythons for asdl* code, because
> > > Python-ast.* are, in fact, checked in so they don't have to be
> > > rebuilt by the bots or users?
> >
> > We should have the buildbots run "make touch", so they don't need to
> > run asdl_c.py.
>
> So, you're gonna like this:
>
> hg --config extensions.touch=Tools/hg/hgtouch.py touch -v
> Tools/hg/hgtouch.py:21: Warning: 'with' will become a reserved keyword in
> Python 2.6
> *** failed to import extension touch from Tools/hg/hgtouch.py: invalid
> syntax (hgtouch.py, line 21)
> hg: unknown command 'touch'
>
> http://buildbot.python.org/all/buildslaves/bolen-ubuntu
>
> (that buildbot runs Ubuntu 8.04 LTS, for the record)
>

We have to define a clear boundary between what needs to be compatible to
old Pythons and what doesn't. As far as auto-generated files are concerned,
once we set up the bots properly and fix the .hgtouch bugs, there's no
reason for anyone to need an old Python for anything except the hgtouch
extension itself. But the latter has to be back-ported to work on older
Pythons (I suppose from 2.4, or is there reason to go earlier?)

FWIW Antoine, I'll be happy to help with the buildbot setup, but you'll
have to tell me how to get access to our buildbot scripts :-)

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/48959312/attachment.html>

From stephen at xemacs.org  Fri Sep 27 16:08:33 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 27 Sep 2013 23:08:33 +0900
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready
	for	pronouncement?
In-Reply-To: <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
 <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>
 <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com>
Message-ID: <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp>

Nick Coghlan writes:

 > I'm not sure what usage model you're assuming for _ensurepip, but it
 > appears to be wrong. End users should be able to just run pip, and
 > either have it work, or else get a message from the OS vendor telling
 > them to install the appropriate system package.

I don't understand how you arrange for that message on existing
installs.  Wouldn't it be easier to just lobby the distros to make
Python dependent on pip?  And speaking of vendors, do you expect Apple
and Microsoft to provide such a message?  And such a system package?

If you already are running a Linux distro or MacPorts, you do "apt-get
python-pip" and "port install py-pip" respectively.  I bet Cygwin is
the same with yet another spelling.  Where's the problem?  You say:

 > New users on Windows and Mac OS X. I've heard many more complaints
 > from folks running tutorials about the pip bootstrapping process than
 > I ever have from the community at large about the GIL :P

I bet those users are *not* running third-party distros, but rather
are sitting in front of pretty close to plain vanilla factory installs
of the OS, no?  And "new users" on Mac OS X already have "old installs" 
of Python, no?

That's my model.  In that model I don't see backporting PEP 453 to
Python 2.7 as being a sufficiently reliable way to provide a smooth
user experience to justify breaking the "no new features" rule (which
is at the "read my lips" level after the True/False fiasco).

Get a commitment from Apple to put 2.7.6 in their next upgrades for
their OS, and then maybe you'd have enough leverage to tip the
balance.  I certainly would concede the point.  But without that,
you're telling Mac users "you have to upgrade Python from a 3rd party
site."  Is that really the way to make new users participating in a
tutorial session happy?  (You tell me, I'm just introspecting here.)

Steve

From p.f.moore at gmail.com  Fri Sep 27 16:26:41 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 27 Sep 2013 15:26:41 +0100
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
 <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>
 <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com>
 <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com>

On 27 September 2013 15:08, Stephen J. Turnbull <stephen at xemacs.org> wrote:
>> New users on Windows and Mac OS X. I've heard many more complaints
>  > from folks running tutorials about the pip bootstrapping process than
>  > I ever have from the community at large about the GIL :P
>
> I bet those users are *not* running third-party distros, but rather
> are sitting in front of pretty close to plain vanilla factory installs
> of the OS, no?  And "new users" on Mac OS X already have "old installs"
> of Python, no?

Windows users who don't use a third-party distro like Enthought,
generally download the python.org installer. At the moment, that
doesn't give them a "pip" command. So if they want to install any
third party package, they have to start by installing pip. The
instructions for that are reasonably clear, but non-trivial, largely
because tools like curl are not commonly available on Windows, and by
default running a Python script may not do what you expect.

Rather than try to fix these problems (which are *hard*) the intent is
to have the pip command installed by the python.org installer.

So, for Windows users, installing a package becomes "pip install XXX".

The problem is that this doesn't work for existing releases (2.7 and
3.3). Rather than have a convoluted set of instructions that goes "if
you're not on Python 3.4+, (basically do what we have at the moment)"
the PEP makes it possible to say "If you have a Python older than 3.4,
upgrade to the latest maintenance release of your version of Python,
then use pip". People who have reasons not to use the latest
*maintenance* release are assumed to be special cases who can either
deal with the more complex current process, or can work something out
for themselves. (They may not be, but there's not much we can do in
practice to help them).

Nobody in the basic target group should be running ensurepip (under
any name) manually.

I can't speak for Linux distros or OSX users, but for Windows I do
believe that this is a significant improvement, and worth the (IMO,
negligible) risk involved in adding this to a maintenance release.
There *may* be some mileage in a debate about why we don't just bundle
pip in the installer, rather than having helpers in core Python
itself, but the long & short of it is that ensurepip is needed for
3.4+ to make installing pip in venvs work by default, and why have a
completely different and so less well tested mechanism for older
versions?

Paul.

From martin at v.loewis.de  Fri Sep 27 16:26:07 2013
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 27 Sep 2013 16:26:07 +0200
Subject: [Python-Dev] asdl.py and Python-ast.[hc]
In-Reply-To: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com>
References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com>
Message-ID: <5245957F.2040002@v.loewis.de>

Am 26.09.13 15:52, schrieb Eli Bendersky:
> * Should I always check-in Python-ast.h and Python-ast.c when I touch
> asdl* ? The generated files are unchanged, it's only the timestamp that
> changed.

If they really didn't change, I don't think it matters much. I believe
there is a fundamental problem in Mercurial which fails to get it
"right" - with the defense that one shouldn't check in generated files
in the first place.

"hg touch" is supposed to work around this limitation. So when you only
check in the generator, anybody updating should do "make touch" after
the update, which should touch the generated files even though they
didn't change.

> * Can we, in theory, use new Pythons for asdl* code, because
> Python-ast.* are, in fact, checked in so they don't have to be rebuilt
> by the bots or users?

I'd say yes. That's the point of checking in generated files, so that
users don't need to run the generator.

> While we're at it, it seems that .hgtouch is wrong:
>
> Include/ast.h: Parser/Python.asdl Parser/asdl.py Parser/asdl_c.py
> Python/Python-ast.c: Include/ast.h
>
> The file Include/ast.h is not,

It may well be incorrect - feel free to fix it.

Regards,
Martin


From martin at v.loewis.de  Fri Sep 27 16:34:41 2013
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 27 Sep 2013 16:34:41 +0200
Subject: [Python-Dev] asdl.py and Python-ast.[hc]
In-Reply-To: <20130926230033.7bd7c584@fsol>
References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com>
 <CAPZV6o_y-8f0TJgEC=YKg+ZkV6vEzhet0U3rqV9DXo-PBaw=3g@mail.gmail.com>
 <CAF-Rda_vthC4GdvJARvs75TYZTUNPLWbF2a4Uvm=7_V67bZvXg@mail.gmail.com>
 <CAPZV6o9kt3qONZK=_y2pJCou0iBPrqpQeQ9vhVYkyof+3caxrA@mail.gmail.com>
 <20130926230033.7bd7c584@fsol>
Message-ID: <52459781.5020300@v.loewis.de>

Am 26.09.13 23:00, schrieb Antoine Pitrou:

> Here you are:
> http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/28/steps/compile/logs/stdio
> 
> Of course, when it's using "-jN" there may be a race condition :-)

Could you make "make touch" a separate build step?

Regards,
Martin

From breamoreboy at yahoo.co.uk  Fri Sep 27 16:39:50 2013
From: breamoreboy at yahoo.co.uk (Mark Lawrence)
Date: Fri, 27 Sep 2013 15:39:50 +0100
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
 <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>
 <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com>
 <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com>
Message-ID: <l245bj$gn3$1@ger.gmane.org>

On 27/09/2013 15:26, Paul Moore wrote:

> So, for Windows users, installing a package becomes "pip install XXX".
>

Apologies if this has already been said but it only works if you've 
already installed an appropriate compiler.  I've lost count of the 
number of times I've tried this, got the (rather cyptic) "error: Unable 
to find vcvarsall.bat" message and then gone off to find a suitable 
binary download.

-- 
Cheers.

Mark Lawrence


From donald at stufft.io  Fri Sep 27 16:42:29 2013
From: donald at stufft.io (Donald Stufft)
Date: Fri, 27 Sep 2013 10:42:29 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <l245bj$gn3$1@ger.gmane.org>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
 <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>
 <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com>
 <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com>
 <l245bj$gn3$1@ger.gmane.org>
Message-ID: <5CA1E89E-DCB6-4483-97DC-251841D22A15@stufft.io>


On Sep 27, 2013, at 10:39 AM, Mark Lawrence <breamoreboy at yahoo.co.uk> wrote:

> Apologies if this has already been said but it only works if you've already installed an appropriate compiler.  I've lost count of the number of times I've tried this, got the (rather cyptic) "error: Unable to find vcvarsall.bat" message and then gone off to find a suitable binary download.

Going forward Wheels are binary packages that pip can install.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/d33b015b/attachment.sig>

From solipsis at pitrou.net  Fri Sep 27 16:48:33 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 27 Sep 2013 16:48:33 +0200
Subject: [Python-Dev] asdl.py and Python-ast.[hc]
References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com>
 <CAPZV6o_y-8f0TJgEC=YKg+ZkV6vEzhet0U3rqV9DXo-PBaw=3g@mail.gmail.com>
 <CAF-Rda_vthC4GdvJARvs75TYZTUNPLWbF2a4Uvm=7_V67bZvXg@mail.gmail.com>
 <CAPZV6o9kt3qONZK=_y2pJCou0iBPrqpQeQ9vhVYkyof+3caxrA@mail.gmail.com>
 <20130926230033.7bd7c584@fsol> <52459781.5020300@v.loewis.de>
Message-ID: <20130927164833.3f0ccf1f@pitrou.net>

Le Fri, 27 Sep 2013 16:34:41 +0200,
"Martin v. L?wis" <martin at v.loewis.de> a ?crit :
> Am 26.09.13 23:00, schrieb Antoine Pitrou:
> 
> > Here you are:
> > http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/28/steps/compile/logs/stdio
> > 
> > Of course, when it's using "-jN" there may be a race condition :-)
> 
> Could you make "make touch" a separate build step?

I'm not sure it fits in buildbot's standard "GNUAutoconf" build factory:
http://docs.buildbot.net/current/manual/cfg-buildfactories.html#index-1

Note that "compile" is a "shell command" so perhaps it is actually
possible to pass "make touch && make -jN". Otherwise, another build
factory will have to be used.

I'll give an access to Eli so that he can try to figure it out :-)

Regards

Antoine.



From p.f.moore at gmail.com  Fri Sep 27 17:06:33 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 27 Sep 2013 16:06:33 +0100
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <l245bj$gn3$1@ger.gmane.org>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
 <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>
 <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com>
 <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com>
 <l245bj$gn3$1@ger.gmane.org>
Message-ID: <CACac1F_fifT=+2XQ34xGFksPre_tai7rzhxfu3nC68-Q=KB-cg@mail.gmail.com>

On 27 September 2013 15:39, Mark Lawrence <breamoreboy at yahoo.co.uk> wrote:
> On 27/09/2013 15:26, Paul Moore wrote:
>
>> So, for Windows users, installing a package becomes "pip install XXX".
>>
>
> Apologies if this has already been said but it only works if you've already
> installed an appropriate compiler.  I've lost count of the number of times
> I've tried this, got the (rather cyptic) "error: Unable to find
> vcvarsall.bat" message and then gone off to find a suitable binary download.

I was assuming that you had whatever's necessary to install - a
compiler, any necessary libraries, whatever. Or more plausibly, as
Donald pointed out, a wheel.

But yes, you're right, this doesn't solve the problem of *building*
distributions. It's not intended to (except by enabling communities to
start distributing wheels as a pain-free binary solution). Whether
that's an issue depends on the community (AIUI, it's a major hassle
for scientific users, not so much for web developers, for example)

Paul

From eliben at gmail.com  Fri Sep 27 17:26:33 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Fri, 27 Sep 2013 08:26:33 -0700
Subject: [Python-Dev] asdl.py and Python-ast.[hc]
In-Reply-To: <5245957F.2040002@v.loewis.de>
References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com>
 <5245957F.2040002@v.loewis.de>
Message-ID: <CAF-Rda9wcfBw+4e1sNXCNiLhqeVL-aV=5MnH0NqAvSnWrk61YA@mail.gmail.com>

On Fri, Sep 27, 2013 at 7:26 AM, "Martin v. L?wis" <martin at v.loewis.de>wrote:

> Am 26.09.13 15:52, schrieb Eli Bendersky:
> > * Should I always check-in Python-ast.h and Python-ast.c when I touch
> > asdl* ? The generated files are unchanged, it's only the timestamp that
> > changed.
>
> If they really didn't change, I don't think it matters much. I believe
> there is a fundamental problem in Mercurial which fails to get it
> "right" - with the defense that one shouldn't check in generated files
> in the first place.
>
> "hg touch" is supposed to work around this limitation. So when you only
> check in the generator, anybody updating should do "make touch" after
> the update, which should touch the generated files even though they
> didn't change.
>
> > * Can we, in theory, use new Pythons for asdl* code, because
> > Python-ast.* are, in fact, checked in so they don't have to be rebuilt
> > by the bots or users?
>
> I'd say yes. That's the point of checking in generated files, so that
> users don't need to run the generator.
>

Thank you for the explanation, Martin.


>
> > While we're at it, it seems that .hgtouch is wrong:
> >
> > Include/ast.h: Parser/Python.asdl Parser/asdl.py Parser/asdl_c.py
> > Python/Python-ast.c: Include/ast.h
> >
> > The file Include/ast.h is not,
>
> It may well be incorrect - feel free to fix it.
>

Thanks. There are a couple of things to address before we can try to
actually run 'make touch' on all bots. I opened
http://bugs.python.org/issue19106 to track this.

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/72272b0c/attachment-0001.html>

From eliben at gmail.com  Fri Sep 27 18:02:53 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Fri, 27 Sep 2013 09:02:53 -0700
Subject: [Python-Dev] Official github mirror for CPython?
In-Reply-To: <CAD+XWwrCD_tLM0b1OFea5ZZq7oH+1TYR_yNyp=mNDKJe7ZBALQ@mail.gmail.com>
References: <CAF-Rda-=wFp_pWSa2gqFxet1M7pf68si=LKZ7FyGU7UrfRS3FA@mail.gmail.com>
 <51F1380E.9070608@python.org>
 <CAD+XWwrCD_tLM0b1OFea5ZZq7oH+1TYR_yNyp=mNDKJe7ZBALQ@mail.gmail.com>
Message-ID: <CAF-Rda8oajAZUf-AaVnHi4q8h9JSSyTRRsNMar8iUf5-opXF_Q@mail.gmail.com>

On Thu, Jul 25, 2013 at 7:48 AM, Brian Curtin <brian at python.org> wrote:

> On Thu, Jul 25, 2013 at 9:37 AM, Christian Heimes <christian at python.org>
> wrote:
> > Am 25.07.2013 16:29, schrieb Eli Bendersky:
> >> Hi all,
> >>
> >> I've been looking for a Github mirror for Python, and found two:
> >>
> >> * https://github.com/python-git/python has a lot of
> forks/watches/starts
> >> but seems to be very out of date (last updated 4 years ago)
> >> * https://github.com/python-mirror/python doesn't appear to be very
> >> popular but is updated daily
> >>
> >> Are some of you the owners of these repositories? Should we consolidate
> >> to a single "semi-official" mirror?
> >
> > +1
> >
> > Does the PSF have an official account on github? We have one on
> bitbucket...
>
> I don't remember who runs this, and I thought I was in it (maybe just
> on BB), but: https://github.com/python
>

The list of members for https://github.com/python is
https://github.com/python?tab=members

How can I get added to that list? When that happens, I can try to set-up a
regularly updated CPython mirror.

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/ebbc1e68/attachment.html>

From status at bugs.python.org  Fri Sep 27 18:07:42 2013
From: status at bugs.python.org (Python tracker)
Date: Fri, 27 Sep 2013 18:07:42 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20130927160742.ACD2956A31@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2013-09-20 - 2013-09-27)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    4256 (+28)
  closed 26624 (+22)
  total  30880 (+50)

Open issues with patches: 1963 


Issues opened (44)
==================

#19058: test_sys.test_ioencoding_nonascii() fails with ASCII locale en
http://bugs.python.org/issue19058  opened by pitrou

#19060: docs: note that subprocess doesn't replace os.exec*
http://bugs.python.org/issue19060  opened by techtonik

#19062: problems with pyshell to get userdir
http://bugs.python.org/issue19062  opened by weizhao

#19063: Python 3.3.3  encodes emails containing non-ascii data as 7bit
http://bugs.python.org/issue19063  opened by apollo13

#19065: sqlite3 timestamp adapter chokes on timezones
http://bugs.python.org/issue19065  opened by acdha

#19066: os.execv fails with spaced names on  Windows
http://bugs.python.org/issue19066  opened by techtonik

#19067: Built-in range docstrings are not PEP-8 compatible
http://bugs.python.org/issue19067  opened by marco.buttu

#19068: Some built-in complex docstrings are not PEP-8 compatible
http://bugs.python.org/issue19068  opened by marco.buttu

#19069: Built-in float docstrings are not PEP-8 compatible
http://bugs.python.org/issue19069  opened by marco.buttu

#19070: In place operators of weakref.proxy() not returning self.
http://bugs.python.org/issue19070  opened by grahamd

#19071: Documentation on what self is for module-level functions is mi
http://bugs.python.org/issue19071  opened by grahamd

#19072: classmethod doesn't honour descriptor protocol of wrapped call
http://bugs.python.org/issue19072  opened by grahamd

#19073: Inability to specific __qualname__ as a property on a class in
http://bugs.python.org/issue19073  opened by grahamd

#19074: Add PySide to GUI FAQ
http://bugs.python.org/issue19074  opened by techtonik

#19075: Add sorting algorithm visualization to turtledemo
http://bugs.python.org/issue19075  opened by Jason.Yeo

#19076: Pdb.do_break calls error with obsolete file kwarg
http://bugs.python.org/issue19076  opened by Michael.Smith

#19077: More robust TemporaryDirectory cleanup
http://bugs.python.org/issue19077  opened by serhiy.storchaka

#19078: Allow reversed(memoryview), like memoryview
http://bugs.python.org/issue19078  opened by Claudiu.Popa

#19079: chameleon benchmark fails on 3.4
http://bugs.python.org/issue19079  opened by pitrou

#19080: Enrich SyntaxError with additional information
http://bugs.python.org/issue19080  opened by alonho

#19081: zipimport behaves badly when the zip file changes while the pr
http://bugs.python.org/issue19081  opened by gregory.p.smith

#19082: Lib/xmlrpc/client.py demo code points to the dead server
http://bugs.python.org/issue19082  opened by vajrasky

#19083: IDNA prefix should be case insensitive
http://bugs.python.org/issue19083  opened by Pepijn.de.Vos

#19084: No way to use TLS-PSK from python ssl
http://bugs.python.org/issue19084  opened by karlp

#19085: Add tkinter basic options tests
http://bugs.python.org/issue19085  opened by serhiy.storchaka

#19086: Make fsum usable incrementally.
http://bugs.python.org/issue19086  opened by oscarbenjamin

#19087: bytearray front-slicing not optimized
http://bugs.python.org/issue19087  opened by pitrou

#19088: TypeError with pickle in embedded python3.3 when starting mult
http://bugs.python.org/issue19088  opened by Larry.Pete

#19089: Windows: Make Ctrl-D exit key combination cross-platform
http://bugs.python.org/issue19089  opened by techtonik

#19091: ast.parse gives wrong position for some Names when non-ascii c
http://bugs.python.org/issue19091  opened by Aivar.Annamaa

#19092: ExitStack.__exit__ incorrectly suppresses exceptions in __exit
http://bugs.python.org/issue19092  opened by hniksic

#19094: urljoin should raise a TypeError if URL is not a string
http://bugs.python.org/issue19094  opened by jason.coombs

#19095: Document SSLSocket.getpeercert always returns None without do_
http://bugs.python.org/issue19095  opened by dsuch

#19096: multiprocessing.Pool._terminate_pool restarts workers during s
http://bugs.python.org/issue19096  opened by ecatmur

#19097: bool(cgi.FieldStorage(...)) may be False unexpectedly
http://bugs.python.org/issue19097  opened by gvanrossum

#19099: struct.pack fails first time with unicode fmt
http://bugs.python.org/issue19099  opened by miwa

#19100: Use backslashreplace in pprint
http://bugs.python.org/issue19100  opened by serhiy.storchaka

#19102: Add tests for CLI of the tabnanny module
http://bugs.python.org/issue19102  opened by berker.peksag

#19103: Use pprint in displayhook
http://bugs.python.org/issue19103  opened by serhiy.storchaka

#19104: pprint produces invalid output for long strings
http://bugs.python.org/issue19104  opened by serhiy.storchaka

#19105: pprint doesn't use all width
http://bugs.python.org/issue19105  opened by serhiy.storchaka

#19106: Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots
http://bugs.python.org/issue19106  opened by eli.bendersky

#19107: CSV output of benchmark runner tries to write to binary file
http://bugs.python.org/issue19107  opened by scoder

#19108: Benchmark runner tries to execute external Python command and 
http://bugs.python.org/issue19108  opened by scoder



Most recent 15 issues with no replies (15)
==========================================

#19108: Benchmark runner tries to execute external Python command and 
http://bugs.python.org/issue19108

#19107: CSV output of benchmark runner tries to write to binary file
http://bugs.python.org/issue19107

#19105: pprint doesn't use all width
http://bugs.python.org/issue19105

#19102: Add tests for CLI of the tabnanny module
http://bugs.python.org/issue19102

#19100: Use backslashreplace in pprint
http://bugs.python.org/issue19100

#19099: struct.pack fails first time with unicode fmt
http://bugs.python.org/issue19099

#19086: Make fsum usable incrementally.
http://bugs.python.org/issue19086

#19085: Add tkinter basic options tests
http://bugs.python.org/issue19085

#19080: Enrich SyntaxError with additional information
http://bugs.python.org/issue19080

#19076: Pdb.do_break calls error with obsolete file kwarg
http://bugs.python.org/issue19076

#19075: Add sorting algorithm visualization to turtledemo
http://bugs.python.org/issue19075

#19072: classmethod doesn't honour descriptor protocol of wrapped call
http://bugs.python.org/issue19072

#19071: Documentation on what self is for module-level functions is mi
http://bugs.python.org/issue19071

#19069: Built-in float docstrings are not PEP-8 compatible
http://bugs.python.org/issue19069

#19068: Some built-in complex docstrings are not PEP-8 compatible
http://bugs.python.org/issue19068



Most recent 15 issues waiting for review (15)
=============================================

#19104: pprint produces invalid output for long strings
http://bugs.python.org/issue19104

#19103: Use pprint in displayhook
http://bugs.python.org/issue19103

#19102: Add tests for CLI of the tabnanny module
http://bugs.python.org/issue19102

#19100: Use backslashreplace in pprint
http://bugs.python.org/issue19100

#19094: urljoin should raise a TypeError if URL is not a string
http://bugs.python.org/issue19094

#19092: ExitStack.__exit__ incorrectly suppresses exceptions in __exit
http://bugs.python.org/issue19092

#19087: bytearray front-slicing not optimized
http://bugs.python.org/issue19087

#19085: Add tkinter basic options tests
http://bugs.python.org/issue19085

#19083: IDNA prefix should be case insensitive
http://bugs.python.org/issue19083

#19082: Lib/xmlrpc/client.py demo code points to the dead server
http://bugs.python.org/issue19082

#19081: zipimport behaves badly when the zip file changes while the pr
http://bugs.python.org/issue19081

#19078: Allow reversed(memoryview), like memoryview
http://bugs.python.org/issue19078

#19077: More robust TemporaryDirectory cleanup
http://bugs.python.org/issue19077

#19075: Add sorting algorithm visualization to turtledemo
http://bugs.python.org/issue19075

#19072: classmethod doesn't honour descriptor protocol of wrapped call
http://bugs.python.org/issue19072



Top 10 most discussed issues (10)
=================================

#12085: subprocess.Popen.__del__ raises AttributeError if __init__ was
http://bugs.python.org/issue12085  11 msgs

#16038: ftplib: unlimited readline() from connection
http://bugs.python.org/issue16038  11 msgs

#18990: Return root element from ElementTree.XMLPullParser.close() to 
http://bugs.python.org/issue18990  11 msgs

#19009: Enhance HTTPResponse.readline() performance
http://bugs.python.org/issue19009  10 msgs

#19089: Windows: Make Ctrl-D exit key combination cross-platform
http://bugs.python.org/issue19089  10 msgs

#19087: bytearray front-slicing not optimized
http://bugs.python.org/issue19087   9 msgs

#11380: Improve reporting of broken stdout pipe during interpreter shu
http://bugs.python.org/issue11380   7 msgs

#19021: AttributeError in Popen.__del__
http://bugs.python.org/issue19021   7 msgs

#14983: email.generator should always add newlines after closing bound
http://bugs.python.org/issue14983   6 msgs

#19024: Document asterisk (*), splat or star operator
http://bugs.python.org/issue19024   6 msgs



Issues closed (22)
==================

#17473: -m is not universally applicable
http://bugs.python.org/issue17473  closed by ncoghlan

#17777: Unrecognized string literal escape sequences give SyntaxErrors
http://bugs.python.org/issue17777  closed by tim.golden

#18050: embedded interpreter or virtualenv fails with "ImportError: ca
http://bugs.python.org/issue18050  closed by serhiy.storchaka

#18626: Make "python -m inspect <name>" meaningful
http://bugs.python.org/issue18626  closed by python-dev

#18978: Allow urllib.request.Request subclasses to override method
http://bugs.python.org/issue18978  closed by python-dev

#18996: unittest: more helpful truncating long strings
http://bugs.python.org/issue18996  closed by serhiy.storchaka

#19025: Deleting attribute of Enum gives misleading error message
http://bugs.python.org/issue19025  closed by python-dev

#19028: tkinter.tkapp.merge() fails on non-strings
http://bugs.python.org/issue19028  closed by serhiy.storchaka

#19030: inspect.getmembers and inspect.classify_class_attrs mishandle 
http://bugs.python.org/issue19030  closed by python-dev

#19034: More useful repr for Tcl_Obj
http://bugs.python.org/issue19034  closed by serhiy.storchaka

#19043: Remove detailed listing of all versions from LICENSE, Doc/lice
http://bugs.python.org/issue19043  closed by python-dev

#19044: getaddrinfo raises near-useless exception
http://bugs.python.org/issue19044  closed by neologix

#19047: Assorted weakref docs improvements
http://bugs.python.org/issue19047  closed by python-dev

#19049: itertools.tee uses int for indices
http://bugs.python.org/issue19049  closed by pitrou

#19054: Descriptors howto
http://bugs.python.org/issue19054  closed by rhettinger

#19056: Windows 7, script exec not working without explicit cal of pyt
http://bugs.python.org/issue19056  closed by eric.smith

#19059: test_posix failure on OS X snow leopord buildbot
http://bugs.python.org/issue19059  closed by ned.deily

#19061: Shelve documentation security warning is not visible
http://bugs.python.org/issue19061  closed by python-dev

#19064: can't run py3 benchmarks
http://bugs.python.org/issue19064  closed by pitrou

#19093: Spam
http://bugs.python.org/issue19093  closed by berker.peksag

#19098: sys.setrecursionlimit(2**30) breaks interactive shell
http://bugs.python.org/issue19098  closed by python-dev

#19101: "make touch" broke "x86 Ubuntu Shared 3.x" buildbot
http://bugs.python.org/issue19101  closed by benjamin.peterson

From rdmurray at bitdance.com  Fri Sep 27 18:15:31 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Fri, 27 Sep 2013 12:15:31 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
 <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>
 <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com>
 <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com>
Message-ID: <20130927161531.87256250948@webabinitio.net>

On Fri, 27 Sep 2013 15:26:41 +0100, Paul Moore <p.f.moore at gmail.com> wrote:
> On 27 September 2013 15:08, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> >> New users on Windows and Mac OS X. I've heard many more complaints
> >  > from folks running tutorials about the pip bootstrapping process than
> >  > I ever have from the community at large about the GIL :P
> >
> > I bet those users are *not* running third-party distros, but rather
> > are sitting in front of pretty close to plain vanilla factory installs
> > of the OS, no?  And "new users" on Mac OS X already have "old installs"
> > of Python, no?
> 
> Windows users who don't use a third-party distro like Enthought,
> generally download the python.org installer. At the moment, that
> doesn't give them a "pip" command. So if they want to install any
> third party package, they have to start by installing pip. The
> instructions for that are reasonably clear, but non-trivial, largely
> because tools like curl are not commonly available on Windows, and by
> default running a Python script may not do what you expect.
> 
> Rather than try to fix these problems (which are *hard*) the intent is
> to have the pip command installed by the python.org installer.
[...]
> I can't speak for Linux distros or OSX users, but for Windows I do
> believe that this is a significant improvement, and worth the (IMO,
> negligible) risk involved in adding this to a maintenance release.

I'm not an OS X user, and probably most people on this list use macports
or something similar, which essentially puts them in the same boat as
the linux users...and there's a section in the PEP about that (that's
where the message about installing pip if you run pip and the distro
didn't include it with python is supposed to come from).

For OS X users *not* using something like macports, I'm pretty sure they
are going to be in a similar boat to the Windows users, with just a touch
of added confusion coming from the fact that an older version of Python
is already installed.  But the instructions they will find on the web
for installing package X (once this change hits the field) will be to
install the newest version of 2.7 (or 3) using the python.org installer,
and then they will have the pip command and can go from there.

--David

From dholth at gmail.com  Fri Sep 27 19:25:40 2013
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 27 Sep 2013 13:25:40 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <20130927161531.87256250948@webabinitio.net>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
 <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>
 <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com>
 <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com>
 <20130927161531.87256250948@webabinitio.net>
Message-ID: <CAG8k2+56OHiCHaZFLKn2naMcpkR6rEjcRZcokXmb288rW9JWXw@mail.gmail.com>

On Fri, Sep 27, 2013 at 12:15 PM, R. David Murray <rdmurray at bitdance.com> wrote:
> On Fri, 27 Sep 2013 15:26:41 +0100, Paul Moore <p.f.moore at gmail.com> wrote:
>> On 27 September 2013 15:08, Stephen J. Turnbull <stephen at xemacs.org> wrote:
>> >> New users on Windows and Mac OS X. I've heard many more complaints
>> >  > from folks running tutorials about the pip bootstrapping process than
>> >  > I ever have from the community at large about the GIL :P
>> >
>> > I bet those users are *not* running third-party distros, but rather
>> > are sitting in front of pretty close to plain vanilla factory installs
>> > of the OS, no?  And "new users" on Mac OS X already have "old installs"
>> > of Python, no?
>>
>> Windows users who don't use a third-party distro like Enthought,
>> generally download the python.org installer. At the moment, that
>> doesn't give them a "pip" command. So if they want to install any
>> third party package, they have to start by installing pip. The
>> instructions for that are reasonably clear, but non-trivial, largely
>> because tools like curl are not commonly available on Windows, and by
>> default running a Python script may not do what you expect.
>>
>> Rather than try to fix these problems (which are *hard*) the intent is
>> to have the pip command installed by the python.org installer.
> [...]
>> I can't speak for Linux distros or OSX users, but for Windows I do
>> believe that this is a significant improvement, and worth the (IMO,
>> negligible) risk involved in adding this to a maintenance release.
>
> I'm not an OS X user, and probably most people on this list use macports
> or something similar, which essentially puts them in the same boat as
> the linux users...and there's a section in the PEP about that (that's
> where the message about installing pip if you run pip and the distro
> didn't include it with python is supposed to come from).
>
> For OS X users *not* using something like macports, I'm pretty sure they
> are going to be in a similar boat to the Windows users, with just a touch
> of added confusion coming from the fact that an older version of Python
> is already installed.  But the instructions they will find on the web
> for installing package X (once this change hits the field) will be to
> install the newest version of 2.7 (or 3) using the python.org installer,
> and then they will have the pip command and can go from there.
>
> --David

OS X and Linux change Python in ways that can be confusing to new and
experienced users, like not installing the stdlib source code by
default, or not installing the profiler, or simply by being out of
date. On these platforms attempting to use the system Python for
development can be a costly mistake instead of a convenience.

Users get a more consistent experience by starting with the installers
from python.org. Hopefully the tutorials will reflect the consistency
added by this PEP.

This PEP only gets pip, an operation which is a recurring
inconvenience for me even though I use Linux. It does not solve any of
the problems you may have after pip has been installed.

From stephen at xemacs.org  Fri Sep 27 20:14:10 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 28 Sep 2013 03:14:10 +0900
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready
	for	pronouncement?
In-Reply-To: <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
 <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>
 <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com>
 <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com>
Message-ID: <871u4a16a5.fsf@uwakimon.sk.tsukuba.ac.jp>

Paul Moore writes:

 > I can't speak for Linux distros or OSX users, but for Windows I do
 > believe that this is a significant improvement,

Nobody doubts this.

 > and worth the (IMO, negligible) risk involved in adding this to a
 > maintenance release.

Doesn't that argument apply equally well to any new feature?  The
proponents of such changes always characterize them as "worth the
(IMO, negligible) risk involved".  I am sure that the proponents of
any such feature are equally sincere about that!

Also, I think that proponents of backporting this PEP are missing
something important.  Specifically, why are we encouraging the use of
Python 2.7 for "new users"?  Shouldn't we use this as an opportunity
to say, "Move to Python 3.4 and forget the 'agony of installing pip'
issue forever?"




From raymond.hettinger at gmail.com  Fri Sep 27 20:15:10 2013
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Fri, 27 Sep 2013 14:15:10 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
Message-ID: <3DCB2AFA-0971-47E2-9930-B749B8D2BDCD@gmail.com>


On Sep 23, 2013, at 7:15 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> With the last round of updates, I believe PEP 453 is ready for
> Martin's pronouncement.

Personally, I'm very excited and happy that this or something like it
is coming close to fruition.  

My experiences in userland suggest that this will be a big step
forward for Python usability.


Raymond

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/1ca8aa25/attachment.html>

From donald at stufft.io  Fri Sep 27 20:22:50 2013
From: donald at stufft.io (Donald Stufft)
Date: Fri, 27 Sep 2013 14:22:50 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready
	for	pronouncement?
In-Reply-To: <871u4a16a5.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
 <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>
 <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com>
 <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com>
 <871u4a16a5.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <14A88ACC-9868-4A80-ABB7-99AEC9EE96F7@stufft.io>


On Sep 27, 2013, at 2:14 PM, "Stephen J. Turnbull" <stephen at xemacs.org> wrote:

> Also, I think that proponents of backporting this PEP are missing
> something important.  Specifically, why are we encouraging the use of
> Python 2.7 for "new users"?  Shouldn't we use this as an opportunity
> to say, "Move to Python 3.4 and forget the 'agony of installing pip'
> issue forever?"

Because reality is that new users are still likely to be using Python 2.7. Python 3
is just now starting to be really usable, however there's a huge corpus of existing
tutorials, course work, books etc for Python 2.7. As Python 3 becomes more usable
that existing corpus of material will be ported over to Python 3 but in the interim
there is still a pretty large hurdle for new users to get over.

That's assuming that they are even able to use Python 3 and whatever they are trying
to do with all of their libraries are ported to Python3. I still think Python 2.7 is a better
target for new users because if you're using Python 3.x theirs a high chance you'll
need to port a library or two still.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/00c035b6/attachment.sig>

From guido at python.org  Fri Sep 27 20:31:32 2013
From: guido at python.org (Guido van Rossum)
Date: Fri, 27 Sep 2013 11:31:32 -0700
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <14A88ACC-9868-4A80-ABB7-99AEC9EE96F7@stufft.io>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
 <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>
 <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com>
 <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com>
 <871u4a16a5.fsf@uwakimon.sk.tsukuba.ac.jp>
 <14A88ACC-9868-4A80-ABB7-99AEC9EE96F7@stufft.io>
Message-ID: <CAP7+vJJetVNPuNZfR+VEPUAbsUYQZNiT0wZ3W8V9MHY2xp6+ug@mail.gmail.com>

On Fri, Sep 27, 2013 at 11:22 AM, Donald Stufft <donald at stufft.io> wrote:

>
> On Sep 27, 2013, at 2:14 PM, "Stephen J. Turnbull" <stephen at xemacs.org>
> wrote:
>
> > Also, I think that proponents of backporting this PEP are missing
> > something important.  Specifically, why are we encouraging the use of
> > Python 2.7 for "new users"?  Shouldn't we use this as an opportunity
> > to say, "Move to Python 3.4 and forget the 'agony of installing pip'
> > issue forever?"
>
> Because reality is that new users are still likely to be using Python 2.7.
> Python 3
> is just now starting to be really usable, however there's a huge corpus of
> existing
> tutorials, course work, books etc for Python 2.7. As Python 3 becomes more
> usable
> that existing corpus of material will be ported over to Python 3 but in
> the interim
> there is still a pretty large hurdle for new users to get over.
>
> That's assuming that they are even able to use Python 3 and whatever they
> are trying
> to do with all of their libraries are ported to Python3. I still think
> Python 2.7 is a better
> target for new users because if you're using Python 3.x theirs a high
> chance you'll
> need to port a library or two still.
>

Based on my day-to-day experience this is still very true. (And yes, I'm
slowly turning the tide. But it will take a long time and I am committed to
giving users the choice.)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/b2fafc1b/attachment-0001.html>

From tjreedy at udel.edu  Fri Sep 27 20:50:22 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 27 Sep 2013 14:50:22 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
 <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>
 <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com>
 <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com>
Message-ID: <l24k17$efa$1@ger.gmane.org>

On 9/27/2013 10:26 AM, Paul Moore wrote:
[snip longish post]
I want to summarize the main points of what I believe Paul said and 
strongly agree with them.

* For this issue, and especially for backporting to 2.7/3.3, we should 
consider Windows, Mac, and *nix distributions as separate cases.

* For Windows users, it would be really good if the PSF installer for 
the next releases of all versions at least optionally resulted in the 
installation of the most recent pip possible.

I add: just how that is accomplished is not the users concern, except 
possibly for using the net to check for a later version of pip. Also, we 
can encourage but not command Enthought and ActiveState to do the same 
suite in their plus distributions.

* For 2.7/3.3, Windows users have no need to run _ensurepip after 
installation.

I add: for 2.7/3.3, there is consequently no need for _ensurepip to be 
in /Lib after installation, even if temporarily added*. If it is not 
there, there is no change the the stdlib, and hence no violation of the 
'no new features' policy. The optional installation of pip is not a 
change to Python itself.

* Where the 2.7/3.3 installers merely bundle pip or use the 3.4 
procedure with an extra delete step strikes me as an implementation 
detail best decided by the installer maker (Martin).

-- 
Terry Jan Reedy


From donald at stufft.io  Fri Sep 27 21:10:43 2013
From: donald at stufft.io (Donald Stufft)
Date: Fri, 27 Sep 2013 15:10:43 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <l24k17$efa$1@ger.gmane.org>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
 <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>
 <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com>
 <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com>
 <l24k17$efa$1@ger.gmane.org>
Message-ID: <2E8C3B8B-808C-40CA-9914-D70E6627AA3F@stufft.io>


On Sep 27, 2013, at 2:50 PM, Terry Reedy <tjreedy at udel.edu> wrote:

> I add: for 2.7/3.3, there is consequently no need for _ensurepip to be in /Lib after installation, even if temporarily added*. If it is not there, there is no change the the stdlib, and hence no violation of the 'no new features' policy. The optional installation of pip is not a change to Python itself.

This sounds like a really bad idea to me. You're going to end up with a different stdlib not only by minor release, but by if they installed through an installer or not.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/1c7a9ce8/attachment.sig>

From tjreedy at udel.edu  Fri Sep 27 22:09:01 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 27 Sep 2013 16:09:01 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <2E8C3B8B-808C-40CA-9914-D70E6627AA3F@stufft.io>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
 <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>
 <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com>
 <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com>
 <l24k17$efa$1@ger.gmane.org> <2E8C3B8B-808C-40CA-9914-D70E6627AA3F@stufft.io>
Message-ID: <l24okm$5lf$1@ger.gmane.org>

On 9/27/2013 3:10 PM, Donald Stufft wrote:
>
> On Sep 27, 2013, at 2:50 PM, Terry Reedy <tjreedy at udel.edu> wrote:
>
>> I add: for 2.7/3.3, there is consequently no need for _ensurepip to be in /Lib after installation, even if temporarily added*. If it is not there, there is no change the the stdlib, and hence no violation of the 'no new features' policy. The optional installation of pip is not a change to Python itself.
>
> This sounds like a really bad idea to me.

Why would you think that an idea aimed at ending an argument blocking 
your proposal is bad. This seems like a really bad response to me.

> You're going to end up with a different stdlib not only by minor release, but by if they installed through an installer or not.

The current proposal is to add a new module to the stdlib in a bugfix 
release, which looks like a violation of current policy. We agree that 
that end result of pip installed would be good. We are arguing over 
whether adding '_' to the name makes it not a violation or whether the 
good outweighs the bad of a violation. I claim that the arguement is not 
necessary and can be ended by not making the addition or by hiding it.

I presume your objection refers to the fact that one can clone the 
repository and compile Python on Windows, albeit with some difficulty. 
My three responses:

1. I do not consider the the result to be 'installed Python', at least 
not as I have used the project file.

2. The ratio of people building Python on Windows to those downloading 
and running an installer is so close to 0 that it can be ignored. People 
who build Python on Windows are not typical Python beginners.

3. If you do not agree with 1 and 2 and object to _ensurepip being in 
/Lib in such limited circumstances, then either put it in /Tools/scripts 
or do not use it at all. I already said that the 2.7/3.3 Windows 
installer maker (Martin) should decide whether to even use it.

4. The argument for including _ensurepip somewhere in the repository it 
that people who *do* build python.exe could then use it to install pip 
the first time. /Tools/scripts would be sufficient for this.

5. The result of not having /Lib/_ensurepip in installed Python would, 
in any case, be a lessor violation of the policy.

-- 
Terry Jan Reedy


From donald at stufft.io  Fri Sep 27 22:29:34 2013
From: donald at stufft.io (Donald Stufft)
Date: Fri, 27 Sep 2013 16:29:34 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <l24okm$5lf$1@ger.gmane.org>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
 <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>
 <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com>
 <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com>
 <l24k17$efa$1@ger.gmane.org> <2E8C3B8B-808C-40CA-9914-D70E6627AA3F@stufft.io>
 <l24okm$5lf$1@ger.gmane.org>
Message-ID: <7A7A9106-0668-4E26-A132-23F59296E5B3@stufft.io>


On Sep 27, 2013, at 4:09 PM, Terry Reedy <tjreedy at udel.edu> wrote:

> On 9/27/2013 3:10 PM, Donald Stufft wrote:
>> 
>> On Sep 27, 2013, at 2:50 PM, Terry Reedy <tjreedy at udel.edu> wrote:
>> 
>>> I add: for 2.7/3.3, there is consequently no need for _ensurepip to be in /Lib after installation, even if temporarily added*. If it is not there, there is no change the the stdlib, and hence no violation of the 'no new features' policy. The optional installation of pip is not a change to Python itself.
>> 
>> This sounds like a really bad idea to me.
> 
> Why would you think that an idea aimed at ending an argument blocking your proposal is bad. This seems like a really bad response to me.

Because I think it's a fundamentally bad idea.

> 
>> You're going to end up with a different stdlib not only by minor release, but by if they installed through an installer or not.
> 
> The current proposal is to add a new module to the stdlib in a bugfix release, which looks like a violation of current policy. We agree that that end result of pip installed would be good. We are arguing over whether adding '_' to the name makes it not a violation or whether the good outweighs the bad of a violation. I claim that the arguement is not necessary and can be ended by not making the addition or by hiding it.
> 
> I presume your objection refers to the fact that one can clone the repository and compile Python on Windows, albeit with some difficulty. My three responses:
> 
> 1. I do not consider the the result to be 'installed Python', at least not as I have used the project file.
> 
> 2. The ratio of people building Python on Windows to those downloading and running an installer is so close to 0 that it can be ignored. People who build Python on Windows are not typical Python beginners.
> 
> 3. If you do not agree with 1 and 2 and object to _ensurepip being in /Lib in such limited circumstances, then either put it in /Tools/scripts or do not use it at all. I already said that the 2.7/3.3 Windows installer maker (Martin) should decide whether to even use it.
> 
> 4. The argument for including _ensurepip somewhere in the repository it that people who *do* build python.exe could then use it to install pip the first time. /Tools/scripts would be sufficient for this.
> 
> 5. The result of not having /Lib/_ensurepip in installed Python would, in any case, be a lessor violation of the policy.

If it lives in the source tree how are you going to provent it from existing when someone installs on Linux? OSX? One of the BSDs? It seems to me your proposal is to add the _ensurepip module? except when they install it via Windows installer. 

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/a0c00446/attachment.sig>

From zachary.ware+pydev at gmail.com  Fri Sep 27 23:16:50 2013
From: zachary.ware+pydev at gmail.com (Zachary Ware)
Date: Fri, 27 Sep 2013 16:16:50 -0500
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <7A7A9106-0668-4E26-A132-23F59296E5B3@stufft.io>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
 <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>
 <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com>
 <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com>
 <l24k17$efa$1@ger.gmane.org> <2E8C3B8B-808C-40CA-9914-D70E6627AA3F@stufft.io>
 <l24okm$5lf$1@ger.gmane.org> <7A7A9106-0668-4E26-A132-23F59296E5B3@stufft.io>
Message-ID: <CAKJDb-M6LqJyoWaPoquy0apoLJ+Ffr9irNBudrX0F10KwoqmbQ@mail.gmail.com>

On Fri, Sep 27, 2013 at 3:29 PM, Donald Stufft <donald at stufft.io> wrote:
>
<snip>
>
> If it lives in the source tree how are you going to provent it from existing when someone installs on Linux? OSX? One of the BSDs?

If someone is building their own Python from source--regardless of
platform--they're obviously going to have _ensurepip available one way
or another, and such users will need to have it available to match the
capabilities of the installer (or create their own).  But if you're
building your own Python, you should already know enough about what's
going on to know when and whether _ensurepip should be used and how.
On the other hand, when you use an installer (be it Windows, OSX, or
otherwise), you really only need _ensurepip one time, ever: at install
time.  Even then, you don't actually need to know anything about
_ensurepip at all, just whether to check the box or not.  If you
decide you need it later, you can always re-install.

> It seems to me your [Terry's] proposal is to add the _ensurepip module? except when they install it via Windows installer.

The way I read Terry's proposal, it is to never add the _ensurepip
*module*, but to use (or make available, whichever makes sense in a
given case) the _ensurepip *script* when it is requested at
install-time: specifically, put _ensurepip.py in Tools/scripts, PC/,
Mac/, or really anywhere but Lib/ so that builders can find it, but
Python can't.  In other words, don't make "import _ensurepip" possible
out of the box, and a lot of the "don't add new features in
maintenance releases" blockade disappears, because you aren't actually
adding any new capability to Python itself or its standard library.

Of course, this proposal only applies to 2.7/3.3.  Since _ensurepip
will be used for venv as well as initial install in 3.4, it should be
a full-blown stdlib module in that version, probably even without the
leading underscore.

-- 
Zach

From jnoller at gmail.com  Fri Sep 27 23:41:30 2013
From: jnoller at gmail.com (Jesse Noller)
Date: Fri, 27 Sep 2013 16:41:30 -0500
Subject: [Python-Dev] Official github mirror for CPython?
In-Reply-To: <CAF-Rda8oajAZUf-AaVnHi4q8h9JSSyTRRsNMar8iUf5-opXF_Q@mail.gmail.com>
References: <CAF-Rda-=wFp_pWSa2gqFxet1M7pf68si=LKZ7FyGU7UrfRS3FA@mail.gmail.com>
 <51F1380E.9070608@python.org>
 <CAD+XWwrCD_tLM0b1OFea5ZZq7oH+1TYR_yNyp=mNDKJe7ZBALQ@mail.gmail.com>
 <CAF-Rda8oajAZUf-AaVnHi4q8h9JSSyTRRsNMar8iUf5-opXF_Q@mail.gmail.com>
Message-ID: <CACQrdOnmuN6UupMuptKqL7HHmrMCBRoQLLDVK18=_NKadq-gPA@mail.gmail.com>

Email me the name of the repo you want, and your github username
(preferably off list so I don't miss it!) and I will set you up Eli

On Fri, Sep 27, 2013 at 11:02 AM, Eli Bendersky <eliben at gmail.com> wrote:
>
>
>
> On Thu, Jul 25, 2013 at 7:48 AM, Brian Curtin <brian at python.org> wrote:
>>
>> On Thu, Jul 25, 2013 at 9:37 AM, Christian Heimes <christian at python.org>
>> wrote:
>> > Am 25.07.2013 16:29, schrieb Eli Bendersky:
>> >> Hi all,
>> >>
>> >> I've been looking for a Github mirror for Python, and found two:
>> >>
>> >> * https://github.com/python-git/python has a lot of
>> >> forks/watches/starts
>> >> but seems to be very out of date (last updated 4 years ago)
>> >> * https://github.com/python-mirror/python doesn't appear to be very
>> >> popular but is updated daily
>> >>
>> >> Are some of you the owners of these repositories? Should we consolidate
>> >> to a single "semi-official" mirror?
>> >
>> > +1
>> >
>> > Does the PSF have an official account on github? We have one on
>> > bitbucket...
>>
>> I don't remember who runs this, and I thought I was in it (maybe just
>> on BB), but: https://github.com/python
>
>
> The list of members for https://github.com/python is
> https://github.com/python?tab=members
>
> How can I get added to that list? When that happens, I can try to set-up a
> regularly updated CPython mirror.
>
> Eli
>
>
>
>

From guido at python.org  Sat Sep 28 00:33:27 2013
From: guido at python.org (Guido van Rossum)
Date: Fri, 27 Sep 2013 15:33:27 -0700
Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib,
 marked provisional, named asyncio
Message-ID: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com>

I've been looking at my progress with Tulip and the 3.4 release schedule
(PEP 429) and it looks like I will have to do some kind of sprint to get it
into the release in time for beta 1, which is planned for Nov 24. Ideally
I'd get it into alpha 4, which is scheduled for Oct 20 -- that's in about
three weeks, and probably too tight.

Even Nov 24 is aggressive, because the PEP (PEP 3156) hasn't even been
discussed formally, let alone accepted! Fortunately I'm pretty happy with
most of the APIs defined in the PEP -- there are some holes but I expect no
big changes at this point.

It's clear that the only way we can get this into the stdlib is to mark it
as provisional, per PEP 411. That's fine with me. I expect that the kind of
changes the code will see between the 3.4 and 3.5 releases are well within
the bounds of the expectations set by that PEP -- while it is clear that
everything is possible (even withdrawal), the most likely outcome is that
the API will be extended or tweaked at most, and not significantly changed
in a backward compatible way -- without completely ruling out that some
part of the API may deserve a serious overhaul based on experience. That's
exactly how I feel about Tulip.

I propose that the new library will be named asyncio. It would be odd if we
kept the name tulip (stdlib modules rarely have "creative" names), and
"asynchronous I/O" seems to cover the contents quite well.

I propose that we do the bikeshedding about the API on the
python-tulip at googlegroups.com mailing list; I'm not sure what we would gain
by holding the discussion on python-dev (although clearly we should report
back here with important milestones in the review). I will post a message
with the kick-off there.

I hope people will respect my choice of venue, and limit themselves to
procedural issued on python-dev. E.g. it's fine to debate on python-dev
whether I'm overstepping my BDFL authority here, or whether the proposed
API is of sufficiently wide appeal. But I'd prefer to discuss e.g. the
return type of start_serving() or the choice of yield-from over yield on
python-tulip.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/cd769af7/attachment.html>

From barry at python.org  Sat Sep 28 02:44:17 2013
From: barry at python.org (Barry Warsaw)
Date: Fri, 27 Sep 2013 20:44:17 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
 pronouncement?
In-Reply-To: <CADiSq7cMMkTjArg=0=rmDypHMo5aACYNQX=ct18N9jzru2=JQQ@mail.gmail.com>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <CADiSq7fVrruk1zcE-ZwHUOyUqOPhTBnn2jcp9rKQ8dS6QeL+Zw@mail.gmail.com>
 <20130925190854.6ac76200@anarchist>
 <CADiSq7cMMkTjArg=0=rmDypHMo5aACYNQX=ct18N9jzru2=JQQ@mail.gmail.com>
Message-ID: <20130927204417.73058d41@anarchist>

On Sep 26, 2013, at 02:30 PM, Nick Coghlan wrote:

>- the module name should be "_ensurepip" in all versions
>- the PEP should explicitly state that the "don't remove _ensurepip
>and it's wheel files" caveat for redistributors applies only in 3.4+
>(where removing it will break pyvenv)

I'm sorry that I probably won't be able to engage more on this thread, but I
think my points should be fairly well understood by the PEP czar and RM.
Should the PEP be accepted, I think both of these are good changes.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/97c19dfc/attachment.sig>

From ncoghlan at gmail.com  Sat Sep 28 02:40:30 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 28 Sep 2013 10:40:30 +1000
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
 <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>
 <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com>
 <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CADiSq7d9ipdAiP8yLCsO0OGoTRLvcL1_0gEHdig=UdH_iLWFgQ@mail.gmail.com>

On 28 Sep 2013 00:08, "Stephen J. Turnbull" <stephen at xemacs.org> wrote:
>
> Nick Coghlan writes:
>
>  > I'm not sure what usage model you're assuming for _ensurepip, but it
>  > appears to be wrong. End users should be able to just run pip, and
>  > either have it work, or else get a message from the OS vendor telling
>  > them to install the appropriate system package.
>
> I don't understand how you arrange for that message on existing
> installs.  Wouldn't it be easier to just lobby the distros to make
> Python dependent on pip?

Most distros of interest already emit that message - it's part of a
standard mechanism to check for system packages if a command isn't found
(and I believe even RHEL and derivatives may support this for pip if the
EPEL repo is enabled).

> And speaking of vendors, do you expect Apple
> and Microsoft to provide such a message?  And such a system package?

No, that's why the proposal is to modify the CPython installers for those
platforms.

>
> If you already are running a Linux distro or MacPorts, you do "apt-get
> python-pip" and "port install py-pip" respectively.  I bet Cygwin is
> the same with yet another spelling.  Where's the problem?  You say:
>
>  > New users on Windows and Mac OS X. I've heard many more complaints
>  > from folks running tutorials about the pip bootstrapping process than
>  > I ever have from the community at large about the GIL :P
>
> I bet those users are *not* running third-party distros, but rather
> are sitting in front of pretty close to plain vanilla factory installs
> of the OS, no?  And "new users" on Mac OS X already have "old installs"
> of Python, no?

No. Instuctors tell users on Mac OS X to install from python.org or use one
of the third party package managers.

>
> That's my model.  In that model I don't see backporting PEP 453 to
> Python 2.7 as being a sufficiently reliable way to provide a smooth
> user experience to justify breaking the "no new features" rule (which
> is at the "read my lips" level after the True/False fiasco).

You have confirmed my belief that your model is incorrect. The feedback I
have received is that the majority of beginners are introduced to Python by
downloading the binary installers from python.org for Windows or Mac OS X.

> Get a commitment from Apple to put 2.7.6 in their next upgrades for
> their OS, and then maybe you'd have enough leverage to tip the
> balance.  I certainly would concede the point.  But without that,
> you're telling Mac users "you have to upgrade Python from a 3rd party
> site."  Is that really the way to make new users participating in a
> tutorial session happy?  (You tell me, I'm just introspecting here.)

Yes, that's exactly what happens. People don't use the system Python on Mac
OS X the way they do on *nix systems.

Cheers,
Nick.

>
> Steve
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130928/8896b222/attachment-0001.html>

From benjamin at python.org  Sat Sep 28 03:19:57 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Fri, 27 Sep 2013 21:19:57 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <20130927204417.73058d41@anarchist>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <CADiSq7fVrruk1zcE-ZwHUOyUqOPhTBnn2jcp9rKQ8dS6QeL+Zw@mail.gmail.com>
 <20130925190854.6ac76200@anarchist>
 <CADiSq7cMMkTjArg=0=rmDypHMo5aACYNQX=ct18N9jzru2=JQQ@mail.gmail.com>
 <20130927204417.73058d41@anarchist>
Message-ID: <CAPZV6o8wZ_ii=mfSJft_FeB4SaqK-L_Or6-6dsmX2B9-3XK+nA@mail.gmail.com>

2013/9/27 Barry Warsaw <barry at python.org>:
> On Sep 26, 2013, at 02:30 PM, Nick Coghlan wrote:
>
>>- the module name should be "_ensurepip" in all versions
>>- the PEP should explicitly state that the "don't remove _ensurepip
>>and it's wheel files" caveat for redistributors applies only in 3.4+
>>(where removing it will break pyvenv)
>
> I'm sorry that I probably won't be able to engage more on this thread, but I
> think my points should be fairly well understood by the PEP czar and RM.
> Should the PEP be accepted, I think both of these are good changes.

And FWIW, I generally agree with Barry about adding things to 2.7.


-- 
Regards,
Benjamin

From brett at python.org  Sat Sep 28 03:20:10 2013
From: brett at python.org (Brett Cannon)
Date: Fri, 27 Sep 2013 21:20:10 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <CAKJDb-M6LqJyoWaPoquy0apoLJ+Ffr9irNBudrX0F10KwoqmbQ@mail.gmail.com>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
 <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>
 <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com>
 <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com>
 <l24k17$efa$1@ger.gmane.org> <2E8C3B8B-808C-40CA-9914-D70E6627AA3F@stufft.io>
 <l24okm$5lf$1@ger.gmane.org> <7A7A9106-0668-4E26-A132-23F59296E5B3@stufft.io>
 <CAKJDb-M6LqJyoWaPoquy0apoLJ+Ffr9irNBudrX0F10KwoqmbQ@mail.gmail.com>
Message-ID: <CAP1=2W44X6+zaQ9=-J4SdekUJ0CxWy7b8YYk=eWM=88CODbn5w@mail.gmail.com>

On Fri, Sep 27, 2013 at 5:16 PM, Zachary Ware
<zachary.ware+pydev at gmail.com>wrote:

> On Fri, Sep 27, 2013 at 3:29 PM, Donald Stufft <donald at stufft.io> wrote:
> >
> <snip>
> >
> > If it lives in the source tree how are you going to provent it from
> existing when someone installs on Linux? OSX? One of the BSDs?
>
> If someone is building their own Python from source--regardless of
> platform--they're obviously going to have _ensurepip available one way
> or another, and such users will need to have it available to match the
> capabilities of the installer (or create their own).  But if you're
> building your own Python, you should already know enough about what's
> going on to know when and whether _ensurepip should be used and how.
> On the other hand, when you use an installer (be it Windows, OSX, or
> otherwise), you really only need _ensurepip one time, ever: at install
> time.  Even then, you don't actually need to know anything about
> _ensurepip at all, just whether to check the box or not.  If you
> decide you need it later, you can always re-install.
>
> > It seems to me your [Terry's] proposal is to add the _ensurepip module?
> except when they install it via Windows installer.
>
> The way I read Terry's proposal, it is to never add the _ensurepip
> *module*, but to use (or make available, whichever makes sense in a
> given case) the _ensurepip *script* when it is requested at
> install-time: specifically, put _ensurepip.py in Tools/scripts, PC/,
> Mac/, or really anywhere but Lib/ so that builders can find it, but
> Python can't.  In other words, don't make "import _ensurepip" possible
> out of the box, and a lot of the "don't add new features in
> maintenance releases" blockade disappears, because you aren't actually
> adding any new capability to Python itself or its standard library.
>
> Of course, this proposal only applies to 2.7/3.3.  Since _ensurepip
> will be used for venv as well as initial install in 3.4, it should be
> a full-blown stdlib module in that version, probably even without the
> leading underscore.
>

That's how I read the proposal as well. If that works then great, otherwise
naming it _ensurepip in 2.7/3.3 should be enough to warn anyone that they
are playing with fire. I think Martin, Ned, etc. would need to weigh in on
whether it's at all an issue having the ensurepip script somewhere that
doesn't get installed but executable from the installer is at all an issue.
For source it can just be in Tools for 2.7/3.3 if that's the route chosen.

Regardless, one of these two options is enough and I suspect we can stop
debating and let Martin make his BDFAP decision.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/be2d734f/attachment.html>

From brett at python.org  Sat Sep 28 03:14:28 2013
From: brett at python.org (Brett Cannon)
Date: Fri, 27 Sep 2013 21:14:28 -0400
Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib,
 marked provisional, named asyncio
In-Reply-To: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com>
References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com>
Message-ID: <CAP1=2W7A2Q3t7Bk7iDZ9ZPprFdi+yr152EgVV=ACaTZkaBbUmw@mail.gmail.com>

On Fri, Sep 27, 2013 at 6:33 PM, Guido van Rossum <guido at python.org> wrote:

> I've been looking at my progress with Tulip and the 3.4 release schedule
> (PEP 429) and it looks like I will have to do some kind of sprint to get it
> into the release in time for beta 1, which is planned for Nov 24. Ideally
> I'd get it into alpha 4, which is scheduled for Oct 20 -- that's in about
> three weeks, and probably too tight.
>
> Even Nov 24 is aggressive, because the PEP (PEP 3156) hasn't even been
> discussed formally, let alone accepted! Fortunately I'm pretty happy with
> most of the APIs defined in the PEP -- there are some holes but I expect no
> big changes at this point.
>
> It's clear that the only way we can get this into the stdlib is to mark it
> as provisional, per PEP 411. That's fine with me. I expect that the kind of
> changes the code will see between the 3.4 and 3.5 releases are well within
> the bounds of the expectations set by that PEP -- while it is clear that
> everything is possible (even withdrawal), the most likely outcome is that
> the API will be extended or tweaked at most, and not significantly changed
> in a backward compatible way -- without completely ruling out that some
> part of the API may deserve a serious overhaul based on experience. That's
> exactly how I feel about Tulip.
>
> I propose that the new library will be named asyncio. It would be odd if
> we kept the name tulip (stdlib modules rarely have "creative" names), and
> "asynchronous I/O" seems to cover the contents quite well.
>
> I propose that we do the bikeshedding about the API on the
> python-tulip at googlegroups.com mailing list; I'm not sure what we would
> gain by holding the discussion on python-dev (although clearly we should
> report back here with important milestones in the review). I will post a
> message with the kick-off there.
>
> I hope people will respect my choice of venue, and limit themselves to
> procedural issued on python-dev. E.g. it's fine to debate on python-dev
> whether I'm overstepping my BDFL authority here, or whether the proposed
> API is of sufficiently wide appeal. But I'd prefer to discuss e.g. the
> return type of start_serving() or the choice of yield-from over yield on
> python-tulip.
>

I don't see any issue with redirecting the discussion. python-tulip@ is
acting like a SIG for the module, so no real precedent beyond it not being
hosted as a mail.python.org list.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/10f6e022/attachment.html>

From donald at stufft.io  Sat Sep 28 04:50:19 2013
From: donald at stufft.io (Donald Stufft)
Date: Fri, 27 Sep 2013 22:50:19 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <CAP1=2W44X6+zaQ9=-J4SdekUJ0CxWy7b8YYk=eWM=88CODbn5w@mail.gmail.com>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
 <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>
 <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com>
 <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com>
 <l24k17$efa$1@ger.gmane.org> <2E8C3B8B-808C-40CA-9914-D70E6627AA3F@stufft.io>
 <l24okm$5lf$1@ger.gmane.org> <7A7A9106-0668-4E26-A132-23F59296E5B3@stufft.io>
 <CAKJDb-M6LqJyoWaPoquy0apoLJ+Ffr9irNBudrX0F10KwoqmbQ@mail.gmail.com>
 <CAP1=2W44X6+zaQ9=-J4SdekUJ0CxWy7b8YYk=eWM=88CODbn5w@mail.gmail.com>
Message-ID: <B601223B-DA26-4B50-B9B6-1AA8E394FAC3@stufft.io>


On Sep 27, 2013, at 9:20 PM, Brett Cannon <brett at python.org> wrote:

> 
> 
> 
> On Fri, Sep 27, 2013 at 5:16 PM, Zachary Ware <zachary.ware+pydev at gmail.com> wrote:
> On Fri, Sep 27, 2013 at 3:29 PM, Donald Stufft <donald at stufft.io> wrote:
> >
> <snip>
> >
> > If it lives in the source tree how are you going to provent it from existing when someone installs on Linux? OSX? One of the BSDs?
> 
> If someone is building their own Python from source--regardless of
> platform--they're obviously going to have _ensurepip available one way
> or another, and such users will need to have it available to match the
> capabilities of the installer (or create their own).  But if you're
> building your own Python, you should already know enough about what's
> going on to know when and whether _ensurepip should be used and how.
> On the other hand, when you use an installer (be it Windows, OSX, or
> otherwise), you really only need _ensurepip one time, ever: at install
> time.  Even then, you don't actually need to know anything about
> _ensurepip at all, just whether to check the box or not.  If you
> decide you need it later, you can always re-install.
> 
> > It seems to me your [Terry's] proposal is to add the _ensurepip module? except when they install it via Windows installer.
> 
> The way I read Terry's proposal, it is to never add the _ensurepip
> *module*, but to use (or make available, whichever makes sense in a
> given case) the _ensurepip *script* when it is requested at
> install-time: specifically, put _ensurepip.py in Tools/scripts, PC/,
> Mac/, or really anywhere but Lib/ so that builders can find it, but
> Python can't.  In other words, don't make "import _ensurepip" possible
> out of the box, and a lot of the "don't add new features in
> maintenance releases" blockade disappears, because you aren't actually
> adding any new capability to Python itself or its standard library.
> 
> Of course, this proposal only applies to 2.7/3.3.  Since _ensurepip
> will be used for venv as well as initial install in 3.4, it should be
> a full-blown stdlib module in that version, probably even without the
> leading underscore.
> 
> That's how I read the proposal as well. If that works then great, otherwise naming it _ensurepip in 2.7/3.3 should be enough to warn anyone that they are playing with fire. I think Martin, Ned, etc. would need to weigh in on whether it's at all an issue having the ensurepip script somewhere that doesn't get installed but executable from the installer is at all an issue. For source it can just be in Tools for 2.7/3.3 if that's the route chosen.
> 
> Regardless, one of these two options is enough and I suspect we can stop debating and let Martin make his BDFAP decision.


Yes, no matter which way the decision for 2.7 and 3.3 goes the proposal is for "ensurepip" in 3.4+

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/ce587a9b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/ce587a9b/attachment.sig>

From mordred at inaugust.com  Sat Sep 28 04:55:52 2013
From: mordred at inaugust.com (Monty Taylor)
Date: Fri, 27 Sep 2013 22:55:52 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <B601223B-DA26-4B50-B9B6-1AA8E394FAC3@stufft.io>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
 <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>
 <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com>
 <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com>
 <l24k17$efa$1@ger.gmane.org> <2E8C3B8B-808C-40CA-9914-D70E6627AA3F@stufft.io>
 <l24okm$5lf$1@ger.gmane.org> <7A7A9106-0668-4E26-A132-23F59296E5B3@stufft.io>
 <CAKJDb-M6LqJyoWaPoquy0apoLJ+Ffr9irNBudrX0F10KwoqmbQ@mail.gmail.com>
 <CAP1=2W44X6+zaQ9=-J4SdekUJ0CxWy7b8YYk=eWM=88CODbn5w@mail.gmail.com>
 <B601223B-DA26-4B50-B9B6-1AA8E394FAC3@stufft.io>
Message-ID: <52464538.9040306@inaugust.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



On 09/27/2013 10:50 PM, Donald Stufft wrote:
> 
> On Sep 27, 2013, at 9:20 PM, Brett Cannon <brett at python.org 
> <mailto:brett at python.org>> wrote:
> 
>> 
>> 
>> 
>> On Fri, Sep 27, 2013 at 5:16 PM, Zachary Ware 
>> <zachary.ware+pydev at gmail.com
>> <mailto:zachary.ware+pydev at gmail.com>> wrote:
>> 
>> On Fri, Sep 27, 2013 at 3:29 PM, Donald Stufft <donald at stufft.io 
>> <mailto:donald at stufft.io>> wrote:
>>> 
>> <snip>
>>> 
>>> If it lives in the source tree how are you going to provent it
>> from existing when someone installs on Linux? OSX? One of the
>> BSDs?
>> 
>> If someone is building their own Python from source--regardless
>> of platform--they're obviously going to have _ensurepip available
>> one way or another, and such users will need to have it available
>> to match the capabilities of the installer (or create their own).
>> But if you're building your own Python, you should already know
>> enough about what's going on to know when and whether _ensurepip
>> should be used and how. On the other hand, when you use an
>> installer (be it Windows, OSX, or otherwise), you really only
>> need _ensurepip one time, ever: at install time.  Even then, you
>> don't actually need to know anything about _ensurepip at all,
>> just whether to check the box or not.  If you decide you need it
>> later, you can always re-install.
>> 
>>> It seems to me your [Terry's] proposal is to add the
>>> _ensurepip
>> module? except when they install it via Windows installer.
>> 
>> The way I read Terry's proposal, it is to never add the
>> _ensurepip *module*, but to use (or make available, whichever
>> makes sense in a given case) the _ensurepip *script* when it is
>> requested at install-time: specifically, put _ensurepip.py in
>> Tools/scripts, PC/, Mac/, or really anywhere but Lib/ so that
>> builders can find it, but Python can't.  In other words, don't
>> make "import _ensurepip" possible out of the box, and a lot of
>> the "don't add new features in maintenance releases" blockade
>> disappears, because you aren't actually adding any new capability
>> to Python itself or its standard library.
>> 
>> Of course, this proposal only applies to 2.7/3.3.  Since
>> _ensurepip will be used for venv as well as initial install in
>> 3.4, it should be a full-blown stdlib module in that version,
>> probably even without the leading underscore.
>> 
>> 
>> That's how I read the proposal as well. If that works then
>> great, otherwise naming it _ensurepip in 2.7/3.3 should be enough
>> to warn anyone that they are playing with fire. I think Martin,
>> Ned, etc. would need to weigh in on whether it's at all an issue
>> having the ensurepip script somewhere that doesn't get installed
>> but executable from the installer is at all an issue. For source
>> it can just be in Tools for 2.7/3.3 if that's the route chosen.
>> 
>> Regardless, one of these two options is enough and I suspect we
>> can stop debating and let Martin make his BDFAP decision.
> 
> Yes, no matter which way the decision for 2.7 and 3.3 goes the
> proposal is for "ensurepip" in 3.4+

I'm in favor of whatever causes it to get available in 2.7/3.3 in a
way that's predictable. OpenStack is both very highly invested in pip
installation of software and dependencies and also seemingly
maniacally obsessed with continuing to support the older pythons
people have. (eek, we have to support 2.6 until a new RHEL comes out)

To that end, we wind up bootstrapping into pip via distro packages,
but then it's too old, so then people pip install -U pip - systemwide,
so then pip gets broken on redhat and weird on ubuntu because of fail,
and then they get angry and start yelling and I have to go hide in a hole.

If we could start, in tooling, depending on some predictable manner
other than the distros to bootstrap into a sensible and modern pip, I
believe my life would be so much better that I might grow a herd of
unicorns.

Monty
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlJGRTgACgkQ2Jv7/VK1RgF3PQCg18X5gBomDNvI0zTUcFJyN5As
misAoOcnFdJYpAM8ur/lfDhKFGxh4VmQ
=BuJf
-----END PGP SIGNATURE-----

From stephen at xemacs.org  Sat Sep 28 05:48:05 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 28 Sep 2013 12:48:05 +0900
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
 pronouncement?
In-Reply-To: <CADiSq7d9ipdAiP8yLCsO0OGoTRLvcL1_0gEHdig=UdH_iLWFgQ@mail.gmail.com>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
 <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>
 <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com>
 <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CADiSq7d9ipdAiP8yLCsO0OGoTRLvcL1_0gEHdig=UdH_iLWFgQ@mail.gmail.com>
Message-ID: <87zjqxzjwq.fsf@uwakimon.sk.tsukuba.ac.jp>

Nick Coghlan writes:

 > You have confirmed my belief that your model is incorrect.

*shrug*  I just think the risks are higher than acknowledged (just
because you have so far failed to imagine a problem doesn't mean it
won't appear), and that the meta effect that "Even Guido admits that
Python 3 isn't ready for prime time" is perverse.  We know, even those
who have written blanket statements to that effect in this thread,
that that is false unless conditioned on specific applications.

I understand that the real motivation is that it's churlish to not
relieve the pain of users who have decided for their own good reasons
to use Python 2.7, and perverse to ignore the needs of the teachers
who are going to educate the users about Python 3 at the time they
consider appropriate.  But the meta-message *received* by the public
is not going to accurately reflect that motivation, and is not going
to be helpful in encouraging those who already *can* move to Python 3
to do so.

Anyway, clearly this exception is heading for approval, and the PEP
with it.  I recommend that the "Feature addition in maintenance
releases" section be amended to read in its entirety:

    The additions of the new module to the standard library in the
    maintenance releases of 2.7 and 3.3 were granted explicit
    exceptions to the rule "no new features in maintenance releases."
    These exceptions were explicitly discussed, and approved in
    consultation with the affected release managers, separately from
    the rest of the PEP.  They do not represent a change in policy,
    and must not be considered a precedent for other such exceptions.

Just the facts, ma'am.  It's a bad idea to include bullshit about the
benefit-cost ratio, because it will be cited in future requests for
similar exceptions.  To the extent that people interpret this as a
forecast and support for a long life for Python 2.7, there is
substantial risk of such requests.


Footnotes: 
[1]  I do it that way myself, always with the most recent Python 3,
but haven't yet gotten to the point of needing "pip" (use cases that
happen to be met by the stdlib).

From donald at stufft.io  Sat Sep 28 06:12:50 2013
From: donald at stufft.io (Donald Stufft)
Date: Sat, 28 Sep 2013 00:12:50 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <87zjqxzjwq.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
 <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>
 <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com>
 <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CADiSq7d9ipdAiP8yLCsO0OGoTRLvcL1_0gEHdig=UdH_iLWFgQ@mail.gmail.com>
 <87zjqxzjwq.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <F8BBC650-8D26-4937-A70A-729FFBB197B7@stufft.io>


On Sep 27, 2013, at 11:48 PM, "Stephen J. Turnbull" <stephen at xemacs.org> wrote:

> Nick Coghlan writes:
> 
>> You have confirmed my belief that your model is incorrect.
> 
> *shrug*  I just think the risks are higher than acknowledged (just
> because you have so far failed to imagine a problem doesn't mean it
> won't appear), and that the meta effect that "Even Guido admits that
> Python 3 isn't ready for prime time" is perverse.  We know, even those
> who have written blanket statements to that effect in this thread,
> that that is false unless conditioned on specific applications.

I haven't seen anyone say Python 3 isn't ready to be used, just that it
makes more sense for beginners to use 2.7, and I think it does still.
Porting libraries from 2.x to 3.x and translaing the existing corpus of
tutorials, tips, posts, etc isn't a trivial task and one that beginners 
are unlikely to grok.

> 
> I understand that the real motivation is that it's churlish to not
> relieve the pain of users who have decided for their own good reasons
> to use Python 2.7, and perverse to ignore the needs of the teachers
> who are going to educate the users about Python 3 at the time they
> consider appropriate.  But the meta-message *received* by the public
> is not going to accurately reflect that motivation, and is not going
> to be helpful in encouraging those who already *can* move to Python 3
> to do so.
> 
> Anyway, clearly this exception is heading for approval, and the PEP
> with it.  I recommend that the "Feature addition in maintenance
> releases" section be amended to read in its entirety:
> 
>    The additions of the new module to the standard library in the
>    maintenance releases of 2.7 and 3.3 were granted explicit
>    exceptions to the rule "no new features in maintenance releases."
>    These exceptions were explicitly discussed, and approved in
>    consultation with the affected release managers, separately from
>    the rest of the PEP.  They do not represent a change in policy,
>    and must not be considered a precedent for other such exceptions.
> 
> Just the facts, ma'am.  It's a bad idea to include bullshit about the
> benefit-cost ratio, because it will be cited in future requests for
> similar exceptions.  To the extent that people interpret this as a
> forecast and support for a long life for Python 2.7, there is
> substantial risk of such requests.

Maybe my understanding of the PEP process is flawed, but isn't
part of the point of it to codify the *reasons* for the decisions that
were made? That's why they include information about dissenting
opinions and such?

I don't think it's dangerous to cite the reasons the decision was came
to. Perhaps it can be toned down but I think it's useful to document
the discussion. I've got a partially done update that tries to capture
the dissenting opinions as well for completeness sake.

> 
> 
> Footnotes: 
> [1]  I do it that way myself, always with the most recent Python 3,
> but haven't yet gotten to the point of needing "pip" (use cases that
> happen to be met by the stdlib).
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130928/58a78cf8/attachment-0001.sig>

From ncoghlan at gmail.com  Sat Sep 28 07:59:05 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 28 Sep 2013 15:59:05 +1000
Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib,
 marked provisional, named asyncio
In-Reply-To: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com>
References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com>
Message-ID: <CADiSq7c22aKMqW_u1GfLjA=uFuGR3e83R-ynVzpbG_nH9Am+2w@mail.gmail.com>

It sounds like a reasonable approach to me.

In terms of naming, would you consider "concurrent.asyncio"? When we
created that parent namespace for futures, one of the other suggested
submodules discussed was the standard event loop API.

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130928/aa4ee948/attachment.html>

From ncoghlan at gmail.com  Sat Sep 28 08:51:27 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 28 Sep 2013 16:51:27 +1000
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <F8BBC650-8D26-4937-A70A-729FFBB197B7@stufft.io>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
 <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>
 <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com>
 <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CADiSq7d9ipdAiP8yLCsO0OGoTRLvcL1_0gEHdig=UdH_iLWFgQ@mail.gmail.com>
 <87zjqxzjwq.fsf@uwakimon.sk.tsukuba.ac.jp>
 <F8BBC650-8D26-4937-A70A-729FFBB197B7@stufft.io>
Message-ID: <CADiSq7eDBzCeP3mw6Mc2j7voiQvxTLw35ysxfO4OJWqYKwQzvw@mail.gmail.com>

On 28 Sep 2013 14:12, "Donald Stufft" <donald at stufft.io> wrote:
>
>
> On Sep 27, 2013, at 11:48 PM, "Stephen J. Turnbull" <stephen at xemacs.org>
wrote:
>
> > Nick Coghlan writes:
> >
> >> You have confirmed my belief that your model is incorrect.
> >
> > *shrug*  I just think the risks are higher than acknowledged (just
> > because you have so far failed to imagine a problem doesn't mean it
> > won't appear), and that the meta effect that "Even Guido admits that
> > Python 3 isn't ready for prime time" is perverse.  We know, even those
> > who have written blanket statements to that effect in this thread,
> > that that is false unless conditioned on specific applications.
>
> I haven't seen anyone say Python 3 isn't ready to be used, just that it
> makes more sense for beginners to use 2.7, and I think it does still.
> Porting libraries from 2.x to 3.x and translaing the existing corpus of
> tutorials, tips, posts, etc isn't a trivial task and one that beginners
> are unlikely to grok.
>
> >
> > I understand that the real motivation is that it's churlish to not
> > relieve the pain of users who have decided for their own good reasons
> > to use Python 2.7, and perverse to ignore the needs of the teachers
> > who are going to educate the users about Python 3 at the time they
> > consider appropriate.  But the meta-message *received* by the public
> > is not going to accurately reflect that motivation, and is not going
> > to be helpful in encouraging those who already *can* move to Python 3
> > to do so.
> >
> > Anyway, clearly this exception is heading for approval, and the PEP
> > with it.  I recommend that the "Feature addition in maintenance
> > releases" section be amended to read in its entirety:
> >
> >    The additions of the new module to the standard library in the
> >    maintenance releases of 2.7 and 3.3 were granted explicit
> >    exceptions to the rule "no new features in maintenance releases."
> >    These exceptions were explicitly discussed, and approved in
> >    consultation with the affected release managers, separately from
> >    the rest of the PEP.  They do not represent a change in policy,
> >    and must not be considered a precedent for other such exceptions.
> >
> > Just the facts, ma'am.  It's a bad idea to include bullshit about the
> > benefit-cost ratio, because it will be cited in future requests for
> > similar exceptions.  To the extent that people interpret this as a
> > forecast and support for a long life for Python 2.7, there is
> > substantial risk of such requests.
>
> Maybe my understanding of the PEP process is flawed, but isn't
> part of the point of it to codify the *reasons* for the decisions that
> were made? That's why they include information about dissenting
> opinions and such?
>
> I don't think it's dangerous to cite the reasons the decision was came
> to. Perhaps it can be toned down but I think it's useful to document
> the discussion. I've got a partially done update that tries to capture
> the dissenting opinions as well for completeness sake.

We'll put something in pointing out that accepting this change actually
makes it even *harder* to advocate for further feature additions in Python
2.7 maintenance releases as there is *zero* chance of backporting language
changes, and this PEP means library and builtin updates can easily be a pip
install away if someone is willing to create the backport and put it on
PyPI (Brandon Rhodes already created the "backports" namespace package as a
common home for such efforts, or there's the "2" suffix that has been used
in a couple of cases).

Cheers,
Nick.

>
> >
> >
> > Footnotes:
> > [1]  I do it that way myself, always with the most recent Python 3,
> > but haven't yet gotten to the point of needing "pip" (use cases that
> > happen to be met by the stdlib).
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
https://mail.python.org/mailman/options/python-dev/donald%40stufft.io
>
>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
DCFA
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130928/152955fa/attachment.html>

From nad at acm.org  Sat Sep 28 09:20:40 2013
From: nad at acm.org (Ned Deily)
Date: Sat, 28 Sep 2013 00:20:40 -0700
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
 <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>
 <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com>
 <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com>
 <l24k17$efa$1@ger.gmane.org> <2E8C3B8B-808C-40CA-9914-D70E6627AA3F@stufft.io>
 <l24okm$5lf$1@ger.gmane.org> <7A7A9106-0668-4E26-A132-23F59296E5B3@stufft.io>
 <CAKJDb-M6LqJyoWaPoquy0apoLJ+Ffr9irNBudrX0F10KwoqmbQ@mail.gmail.com>
 <CAP1=2W44X6+zaQ9=-J4SdekUJ0CxWy7b8YYk=eWM=88CODbn5w@mail.gmail.com>
Message-ID: <nad-5BEB10.00204028092013@news.gmane.org>

In article 
<CAP1=2W44X6+zaQ9=-J4SdekUJ0CxWy7b8YYk=eWM=88CODbn5w at mail.gmail.com>,
 Brett Cannon <brett at python.org> wrote:
> On Fri, Sep 27, 2013 at 5:16 PM, Zachary Ware
> <zachary.ware+pydev at gmail.com>wrote:
> > The way I read Terry's proposal, it is to never add the _ensurepip
> > *module*, but to use (or make available, whichever makes sense in a
> > given case) the _ensurepip *script* when it is requested at
> > install-time: specifically, put _ensurepip.py in Tools/scripts, PC/,
> > Mac/, or really anywhere but Lib/ so that builders can find it, but
> > Python can't.  In other words, don't make "import _ensurepip" possible
> > out of the box, and a lot of the "don't add new features in
> > maintenance releases" blockade disappears, because you aren't actually
> > adding any new capability to Python itself or its standard library.
> >
> > Of course, this proposal only applies to 2.7/3.3.  Since _ensurepip
> > will be used for venv as well as initial install in 3.4, it should be
> > a full-blown stdlib module in that version, probably even without the
> > leading underscore. 
> That's how I read the proposal as well. If that works then great, otherwise
> naming it _ensurepip in 2.7/3.3 should be enough to warn anyone that they
> are playing with fire. I think Martin, Ned, etc. would need to weigh in on
> whether it's at all an issue having the ensurepip script somewhere that
> doesn't get installed but executable from the installer is at all an issue.
> For source it can just be in Tools for 2.7/3.3 if that's the route chosen.

Putting an "ensurepip.py" script (with or without the "_") in Tools/ should be 
fine for the needs of 2.7 and/or 3.3 OS X installers.  Without venv in the 
picture, it doesn't need to do much; the installer build script itself could 
handle everything without an ensurepip.  There are still other changes in the 
installer that would need to be made, particularly if it is necessary to make 
the installation of pip (and its dependency, setuptools) optional but all of 
those would be isolated to the Mac subdirectory.

-- 
 Ned Deily,
 nad at acm.org


From techtonik at gmail.com  Sat Sep 28 09:41:12 2013
From: techtonik at gmail.com (anatoly techtonik)
Date: Sat, 28 Sep 2013 10:41:12 +0300
Subject: [Python-Dev] PEP 453 (pip bootstrap) TL;DR and Usage Scenario
Message-ID: <CAPkN8xLpE=3ftJuhu4_UuxBmYXrm8Up9m7dQMVQk9XZ5eOYR6A@mail.gmail.com>

On Mon, Sep 23, 2013 at 2:15 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> With the last round of updates, I believe PEP 453 is ready for
> Martin's pronouncement.
>
> HTML: http://www.python.org/dev/peps/pep-0453/
> Major diffs: http://hg.python.org/peps/rev/b2993450b32a

I'd enjoy concise PEP texts, but it is extremely hard to write one, so
is it possible to at least add a TL;DR chapter that just shows the
concept in 15 seconds without all the gory details? What is the
entrypoint in the bootstrap process and knowledge requirements for the
user?

For the same sake, is it possible to include real world usage scenario
as a code paste example of user session. For instance:

   # --- 001: user reads install instruction on <package> site
   # 'pip install <package>'

   # user executes 'pip install <package>' on Windows
   > pip install <package>
   'pip' is not recognized as an internal or external command,
   operable program or batch file.
   # user ...<does what>... ? [ ]

   # user executes 'pip install <package>' on Ubuntu
   $ pip install <package>
   The program 'pip' is currently not installed. You can install it by typing:
   sudo apt-get install pip
   $ sudo apt-get install pip
   # (user has sudo - done)
   # user doesn't have sudo, command fails
   # user ...<does what>... ? [ ]

   # --- 002: user reads install instruction on <package> site
   # 'python -m pip install <package>'

   $ python -m pip install <package>
   /usr/bin/python: No module named pip
   # user ...<does what>... ? [ ]

-- 
anatoly t.

From solipsis at pitrou.net  Sat Sep 28 12:55:21 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 28 Sep 2013 12:55:21 +0200
Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib,
 marked provisional, named asyncio
References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com>
 <CADiSq7c22aKMqW_u1GfLjA=uFuGR3e83R-ynVzpbG_nH9Am+2w@mail.gmail.com>
Message-ID: <20130928125521.1317b48a@fsol>

On Sat, 28 Sep 2013 15:59:05 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> It sounds like a reasonable approach to me.
> 
> In terms of naming, would you consider "concurrent.asyncio"? When we
> created that parent namespace for futures, one of the other suggested
> submodules discussed was the standard event loop API.

-10. Please, let's stop it.

Regards

Antoine.



From eliben at gmail.com  Sat Sep 28 14:56:46 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Sat, 28 Sep 2013 05:56:46 -0700
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <CAPZV6o8wZ_ii=mfSJft_FeB4SaqK-L_Or6-6dsmX2B9-3XK+nA@mail.gmail.com>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <CADiSq7fVrruk1zcE-ZwHUOyUqOPhTBnn2jcp9rKQ8dS6QeL+Zw@mail.gmail.com>
 <20130925190854.6ac76200@anarchist>
 <CADiSq7cMMkTjArg=0=rmDypHMo5aACYNQX=ct18N9jzru2=JQQ@mail.gmail.com>
 <20130927204417.73058d41@anarchist>
 <CAPZV6o8wZ_ii=mfSJft_FeB4SaqK-L_Or6-6dsmX2B9-3XK+nA@mail.gmail.com>
Message-ID: <CAF-Rda_2FoPEOt7GEOLN-XgeAiOh8Hw6NjSV3nbo5U6YS9=TSA@mail.gmail.com>

On Fri, Sep 27, 2013 at 6:19 PM, Benjamin Peterson <benjamin at python.org>wrote:

> 2013/9/27 Barry Warsaw <barry at python.org>:
> > On Sep 26, 2013, at 02:30 PM, Nick Coghlan wrote:
> >
> >>- the module name should be "_ensurepip" in all versions
> >>- the PEP should explicitly state that the "don't remove _ensurepip
> >>and it's wheel files" caveat for redistributors applies only in 3.4+
> >>(where removing it will break pyvenv)
> >
> > I'm sorry that I probably won't be able to engage more on this thread,
> but I
> > think my points should be fairly well understood by the PEP czar and RM.
> > Should the PEP be accepted, I think both of these are good changes.
>
> And FWIW, I generally agree with Barry about adding things to 2.7.
>

+1. Not to mention the general problem of adding new features in
maintenance releases (what? it doesn't work for you? Check it's 2.7.6 and
not 2.7.5!) , at this point in the trajectory of Python 3.x and 2.7 it
sends a very wrong message.

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130928/14c9fb59/attachment.html>

From barry at python.org  Sat Sep 28 16:35:41 2013
From: barry at python.org (Barry Warsaw)
Date: Sat, 28 Sep 2013 10:35:41 -0400
Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib,
 marked provisional, named asyncio
In-Reply-To: <CAP1=2W7A2Q3t7Bk7iDZ9ZPprFdi+yr152EgVV=ACaTZkaBbUmw@mail.gmail.com>
References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com>
 <CAP1=2W7A2Q3t7Bk7iDZ9ZPprFdi+yr152EgVV=ACaTZkaBbUmw@mail.gmail.com>
Message-ID: <20130928103541.67ccf3bd@limelight.wooz.org>

On Sep 27, 2013, at 09:14 PM, Brett Cannon wrote:

>I don't see any issue with redirecting the discussion. python-tulip@ is
>acting like a SIG for the module, so no real precedent beyond it not being
>hosted as a mail.python.org list.

The PEP process even allows for this formally.   Please add a Discussions-To
header to PEP 3156.   See PEP 1 for details.

-Barry

From barry at python.org  Sat Sep 28 16:38:48 2013
From: barry at python.org (Barry Warsaw)
Date: Sat, 28 Sep 2013 10:38:48 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
 pronouncement?
In-Reply-To: <87zjqxzjwq.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <20130925175129.1edc4348@anarchist>
 <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io>
 <20130925190410.0a5729f9@anarchist>
 <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io>
 <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com>
 <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CADiSq7d9ipdAiP8yLCsO0OGoTRLvcL1_0gEHdig=UdH_iLWFgQ@mail.gmail.com>
 <87zjqxzjwq.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <20130928103848.0c1ed468@limelight.wooz.org>

On Sep 28, 2013, at 12:48 PM, Stephen J. Turnbull wrote:

>*shrug*  I just think the risks are higher than acknowledged (just
>because you have so far failed to imagine a problem doesn't mean it
>won't appear), and that the meta effect that "Even Guido admits that
>Python 3 isn't ready for prime time" is perverse.  We know, even those
>who have written blanket statements to that effect in this thread,
>that that is false unless conditioned on specific applications.
>
>I understand that the real motivation is that it's churlish to not
>relieve the pain of users who have decided for their own good reasons
>to use Python 2.7, and perverse to ignore the needs of the teachers
>who are going to educate the users about Python 3 at the time they
>consider appropriate.  But the meta-message *received* by the public
>is not going to accurately reflect that motivation, and is not going
>to be helpful in encouraging those who already *can* move to Python 3
>to do so.

FWIW, +1

>
>Anyway, clearly this exception is heading for approval, and the PEP
>with it.  I recommend that the "Feature addition in maintenance
>releases" section be amended to read in its entirety:
>
>    The additions of the new module to the standard library in the
>    maintenance releases of 2.7 and 3.3 were granted explicit
>    exceptions to the rule "no new features in maintenance releases."
>    These exceptions were explicitly discussed, and approved in
>    consultation with the affected release managers, separately from
>    the rest of the PEP.  They do not represent a change in policy,
>    and must not be considered a precedent for other such exceptions.

Call this the "Bush v. Gore" amendment.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130928/8f2ccb29/attachment.sig>

From rawthemetal at gmail.com  Sat Sep 28 13:28:04 2013
From: rawthemetal at gmail.com (Kevin Ngugi)
Date: Sat, 28 Sep 2013 14:28:04 +0300
Subject: [Python-Dev] toolbar
Message-ID: <CANS-bzPK2Krd9F39DxxFKZhgPv5vYwUUbHZSwL=BcAeGt1YmxA@mail.gmail.com>

Hi, I just downloaded Python 3.3 top teach myself how to program, I am
new to programming, but the guide I am using requires me to access the
toolbar, which I cannot seem to find. How do I find it and have it
displayed on the interface? I tried v 3.1 but its still absent. Thank
you.

From eliben at gmail.com  Sat Sep 28 18:07:00 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Sat, 28 Sep 2013 09:07:00 -0700
Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib,
 marked provisional, named asyncio
In-Reply-To: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com>
References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com>
Message-ID: <CAF-Rda-_qGpkQEg2Zu9FtFo_LRMnki7owgoKLsa5g3nwjOe+ng@mail.gmail.com>

On Fri, Sep 27, 2013 at 3:33 PM, Guido van Rossum <guido at python.org> wrote:

> I've been looking at my progress with Tulip and the 3.4 release schedule
> (PEP 429) and it looks like I will have to do some kind of sprint to get it
> into the release in time for beta 1, which is planned for Nov 24. Ideally
> I'd get it into alpha 4, which is scheduled for Oct 20 -- that's in about
> three weeks, and probably too tight.
>
> Even Nov 24 is aggressive, because the PEP (PEP 3156) hasn't even been
> discussed formally, let alone accepted! Fortunately I'm pretty happy with
> most of the APIs defined in the PEP -- there are some holes but I expect no
> big changes at this point.
>
> It's clear that the only way we can get this into the stdlib is to mark it
> as provisional, per PEP 411. That's fine with me. I expect that the kind of
> changes the code will see between the 3.4 and 3.5 releases are well within
> the bounds of the expectations set by that PEP -- while it is clear that
> everything is possible (even withdrawal), the most likely outcome is that
> the API will be extended or tweaked at most, and not significantly changed
> in a backward compatible way -- without completely ruling out that some
> part of the API may deserve a serious overhaul based on experience. That's
> exactly how I feel about Tulip.
>
> I propose that the new library will be named asyncio. It would be odd if
> we kept the name tulip (stdlib modules rarely have "creative" names), and
> "asynchronous I/O" seems to cover the contents quite well.
>
> I propose that we do the bikeshedding about the API on the
> python-tulip at googlegroups.com mailing list; I'm not sure what we would
> gain by holding the discussion on python-dev (although clearly we should
> report back here with important milestones in the review). I will post a
> message with the kick-off there.
>

Sounds good. But once it's in stdlib, I think it would be proper to shift
the discussion into the normal pydev channels (python-dev, issue tracker,
etc.). Is this the plan?

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130928/11ccc2d5/attachment.html>

From tjreedy at udel.edu  Sat Sep 28 18:14:34 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 28 Sep 2013 12:14:34 -0400
Subject: [Python-Dev] toolbar
In-Reply-To: <CANS-bzPK2Krd9F39DxxFKZhgPv5vYwUUbHZSwL=BcAeGt1YmxA@mail.gmail.com>
References: <CANS-bzPK2Krd9F39DxxFKZhgPv5vYwUUbHZSwL=BcAeGt1YmxA@mail.gmail.com>
Message-ID: <5247006A.2070307@udel.edu>

On 9/28/2013 7:28 AM, Kevin Ngugi wrote:
> Hi, I just downloaded Python 3.3 top teach myself how to program, I am
> new to programming, but the guide I am using requires me to access the
> toolbar, which I cannot seem to find. How do I find it and have it
> displayed on the interface? I tried v 3.1 but its still absent. Thank
> you.

python-dev is for development of Python. Please ask questions about 
using Python elsewhere, such as python-list. When you do, say what 
'guide' you are using and what 'toolbar' it refers to. I don't know.

And please do not answer this email.
-- 
Terry Jan Reedy


From python at mrabarnett.plus.com  Sat Sep 28 18:36:35 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Sat, 28 Sep 2013 17:36:35 +0100
Subject: [Python-Dev] toolbar
In-Reply-To: <CANS-bzPK2Krd9F39DxxFKZhgPv5vYwUUbHZSwL=BcAeGt1YmxA@mail.gmail.com>
References: <CANS-bzPK2Krd9F39DxxFKZhgPv5vYwUUbHZSwL=BcAeGt1YmxA@mail.gmail.com>
Message-ID: <52470593.1010109@mrabarnett.plus.com>

On 28/09/2013 12:28, Kevin Ngugi wrote:
> Hi, I just downloaded Python 3.3 top teach myself how to program, I am
> new to programming, but the guide I am using requires me to access the
> toolbar, which I cannot seem to find. How do I find it and have it
> displayed on the interface? I tried v 3.1 but its still absent. Thank
> you.
>
This list ("python-dev") is for the development _of_ Python, not _with_
Python, which is "python-list".

Also, you'll need to say what you mean by "the toolbar".


From guido at python.org  Sat Sep 28 18:52:28 2013
From: guido at python.org (Guido van Rossum)
Date: Sat, 28 Sep 2013 09:52:28 -0700
Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib,
 marked provisional, named asyncio
In-Reply-To: <CADiSq7c22aKMqW_u1GfLjA=uFuGR3e83R-ynVzpbG_nH9Am+2w@mail.gmail.com>
References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com>
 <CADiSq7c22aKMqW_u1GfLjA=uFuGR3e83R-ynVzpbG_nH9Am+2w@mail.gmail.com>
Message-ID: <CAP7+vJJfMP_1DFDc8dSGJo=JiE2hPGK_2AzrWUZUpxdiJ=qRPg@mail.gmail.com>

On Fri, Sep 27, 2013 at 10:59 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> It sounds like a reasonable approach to me.
>
> In terms of naming, would you consider "concurrent.asyncio"? When we
> created that parent namespace for futures, one of the other suggested
> submodules discussed was the standard event loop API.
>

Hm. I want the threading and event world to be very clearly separate and
different, since accidentally combining them is disastrous. So the
concurrent package is the *last* place where I want asyncio to live. (And I
realize there is also some multiprocessing support in that package -- but
it still uses threads to wait for things.)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130928/1d8e1e25/attachment.html>

From jnoller at gmail.com  Sat Sep 28 18:53:47 2013
From: jnoller at gmail.com (Jesse Noller)
Date: Sat, 28 Sep 2013 11:53:47 -0500
Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib,
	marked provisional, named asyncio
In-Reply-To: <CADiSq7c22aKMqW_u1GfLjA=uFuGR3e83R-ynVzpbG_nH9Am+2w@mail.gmail.com>
References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com>
 <CADiSq7c22aKMqW_u1GfLjA=uFuGR3e83R-ynVzpbG_nH9Am+2w@mail.gmail.com>
Message-ID: <E41A420B-C78E-415B-950D-E9DC54D9D06D@gmail.com>



> On Sep 28, 2013, at 12:59 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> It sounds like a reasonable approach to me.
> 
> In terms of naming, would you consider "concurrent.asyncio"? When we created that parent namespace for futures, one of the other suggested submodules discussed was the standard event loop API.
> 
+1 but up to guido

> Cheers,
> Nick.
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130928/8977b54b/attachment.html>

From guido at python.org  Sat Sep 28 19:02:19 2013
From: guido at python.org (Guido van Rossum)
Date: Sat, 28 Sep 2013 10:02:19 -0700
Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib,
 marked provisional, named asyncio
In-Reply-To: <CAF-Rda-_qGpkQEg2Zu9FtFo_LRMnki7owgoKLsa5g3nwjOe+ng@mail.gmail.com>
References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com>
 <CAF-Rda-_qGpkQEg2Zu9FtFo_LRMnki7owgoKLsa5g3nwjOe+ng@mail.gmail.com>
Message-ID: <CAP7+vJJ2_Hu-krRae81jO0ZMpN7oC+7bSmv_YkJS2hcw2dJNcQ@mail.gmail.com>

On Sat, Sep 28, 2013 at 9:07 AM, Eli Bendersky <eliben at gmail.com> wrote:
>
>
> Sounds good. But once it's in stdlib, I think it would be proper to shift
> the discussion into the normal pydev channels (python-dev, issue tracker,
> etc.). Is this the plan?
>

Hadn't really thought about that. I think there's a precedent for aspects
of Python that have their own mailing list (e.g. import-sig). Also,
python-dev is usually pretty hostile about people asking usage questions --
there's no such taboo on the Tulip list.

I can certainly see how you wouldn't want to have to look at multiple
trackers. Possibly the Tulip tracker could be limited to discussion of the
Python 3.3 backport that I hope to maintain in that project.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130928/75087793/attachment-0001.html>

From breamoreboy at yahoo.co.uk  Sat Sep 28 19:42:55 2013
From: breamoreboy at yahoo.co.uk (Mark Lawrence)
Date: Sat, 28 Sep 2013 18:42:55 +0100
Subject: [Python-Dev] toolbar
In-Reply-To: <CANS-bzPK2Krd9F39DxxFKZhgPv5vYwUUbHZSwL=BcAeGt1YmxA@mail.gmail.com>
References: <CANS-bzPK2Krd9F39DxxFKZhgPv5vYwUUbHZSwL=BcAeGt1YmxA@mail.gmail.com>
Message-ID: <l274en$m3g$1@ger.gmane.org>

On 28/09/2013 12:28, Kevin Ngugi wrote:
> Hi, I just downloaded Python 3.3 top teach myself how to program, I am
> new to programming, but the guide I am using requires me to access the
> toolbar, which I cannot seem to find. How do I find it and have it
> displayed on the interface? I tried v 3.1 but its still absent. Thank
> you.
>

Further to other replies you might be better off on the tutor mailing 
list see https://mail.python.org/mailman/listinfo/tutor

-- 
Cheers.

Mark Lawrence


From tjreedy at udel.edu  Sat Sep 28 22:19:44 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 28 Sep 2013 16:19:44 -0400
Subject: [Python-Dev] Best practice for documentation for std lib
In-Reply-To: <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com>
References: <20130922031304.GE19939@ando>
 <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com>
 <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com>
 <20130923003142.GO19939@ando>
 <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com>
 <l1o8ri$mjv$1@ger.gmane.org>
 <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com>
Message-ID: <l27dkq$vgc$1@ger.gmane.org>

On 9/22/2013 10:44 PM, Guido van Rossum wrote:
> On Sun, Sep 22, 2013 at 7:25 PM, Terry Reedy <tjreedy at udel.edu

>     ('Return' rather than 'Returns' is the current convention.)

> That's actually a religious argument which in the stdlib takes no strict
> position -- a quick grep shows that both are used, although 'Return' is
> more frequent by a 5-to-1 margin.

I wrote 'current convention' not just as a statistical statement, but 
as a prescription I had read, even though I did not remember just where. 
PEP8 says
   "PEP 257 describes good docstring conventions."
I followed that clue found the statement I had read.  PEP 257 says
'''
It [the one line docstring] prescribes the function or method's effect 
as a command ("Do this", "Return that"), not as a description; e.g. 
don't write "Returns the pathname ...".
'''
Whether the reference in PEP 8 makes it a PEP 8 rule? or a 'strict 
stdlib position'?

http://bugs.python.org/issue19067 proposes to change rangeobject 
docstrings. Is that OK or should the issue be rejected and closed?

-- 
Terry Jan Reedy


From ncoghlan at gmail.com  Sun Sep 29 02:15:14 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 29 Sep 2013 10:15:14 +1000
Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib,
 marked provisional, named asyncio
In-Reply-To: <CAP7+vJJfMP_1DFDc8dSGJo=JiE2hPGK_2AzrWUZUpxdiJ=qRPg@mail.gmail.com>
References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com>
 <CADiSq7c22aKMqW_u1GfLjA=uFuGR3e83R-ynVzpbG_nH9Am+2w@mail.gmail.com>
 <CAP7+vJJfMP_1DFDc8dSGJo=JiE2hPGK_2AzrWUZUpxdiJ=qRPg@mail.gmail.com>
Message-ID: <CADiSq7ea-4YWBwRS_6v1m6fC0u1yTCu081ofeCuCUQQ4iTMX=w@mail.gmail.com>

On 29 Sep 2013 02:52, "Guido van Rossum" <guido at python.org> wrote:
>
> On Fri, Sep 27, 2013 at 10:59 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> It sounds like a reasonable approach to me.
>>
>> In terms of naming, would you consider "concurrent.asyncio"? When we
created that parent namespace for futures, one of the other suggested
submodules discussed was the standard event loop API.
>
>
> Hm. I want the threading and event world to be very clearly separate and
different, since accidentally combining them is disastrous. So the
concurrent package is the *last* place where I want asyncio to live. (And I
realize there is also some multiprocessing support in that package -- but
it still uses threads to wait for things.)

Makes sense to me!

Cheers,
Nick.

>
> --
> --Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130929/4d5d483d/attachment.html>

From ncoghlan at gmail.com  Sun Sep 29 08:45:24 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 29 Sep 2013 16:45:24 +1000
Subject: [Python-Dev] PEP 453 updated with improved rationale
Message-ID: <CADiSq7ffZq1y+3BnwzmM2FODixzmqmeXqZP=2dXxoBCuWnzfuA@mail.gmail.com>

PEP 453 has been updated in response to the previous discussion thread.

Updated version is online at http://www.python.org/dev/peps/pep-0453/

Relevant diffs can be found at:
http://hg.python.org/peps/log/bdd50f92ee73/pep-0453.txt

The bulk of the updates can be seen in the following two diffs:
    http://hg.python.org/peps/rev/58f33b8eb68d
    http://hg.python.org/peps/rev/438e3fa95c88

The underlying technical proposal itself hasn't actually changed from
the previous iteration: we still want to make "ensurepip" a public
module in all of Python 2.7, 3.3 and 3.4, and invoke it from the
binary installers in those versions.

However, the rationale sections of the PEP, both the general rationale
for making the proposal at all and the specific rationale for the
backporting exemption, have been updated significantly, and the PEP
now includes more detail on various parts of the proposal (especially
proposed changes to the documentation.  The two backporting
compromises discussed in the previous thread (making this a
binary-installer-only feature and using a different module name for
the backported version) are covered explicitly as rejected proposals.

The two major additions to the rationale are a longer discourse on the
differences between python-dev and distutils-sig in terms of what it
means to "support" a version of Python and how this PEP is seen as a
major enabling change for the future evolution of the Python packaging
ecosystem, as well as much deeper coverage of the origins of the "no
new features in maintenance releases" rule and how this case differs
from the incidents that caused problems in the 2.2 series, and how it
serves to *weaken* the case for any future exemptions rather than
strengthen it.

Regards,
Nick.

================================
PEP: 453
Title: Explicit bootstrapping of pip in Python installations
Version: $Revision$
Last-Modified: $Date$
Author: Donald Stufft <donald at stufft.io>,
        Nick Coghlan <ncoghlan at gmail.com>
BDFL-Delegate: Martin von L?wis
Status: Draft
Type: Process
Content-Type: text/x-rst
Created: 10-Aug-2013
Post-History: 30-Aug-2013, 15-Sep-2013, 18-Sep-2013, 19-Sep-2013,
              23-Sep-2013, 29-Sep-2013


Abstract
========

This PEP proposes that the
`Installing Python Modules <http://docs.python.org/3/install>`__ guide be
updated to officially recommend the use of ``pip`` as the default
installer for Python packages, and that appropriate technical changes be
made in Python 2.7, 3.3 and 3.4 to provide ``pip`` by default in support
of that recommendation.


Rationale
=========

There are two related, but distinct rationales for the proposal in this
PEP. The first relates to the experience of new users, while the second
relates to better enabling the evolution of the broader Python packaging
ecosystem.


Improving the new user experience
---------------------------------

Currently, on systems without a platform package manager and repository,
installing a third-party Python package into a freshly installed Python
requires first identifying an appropriate package manager and then
installing it.

Even on systems that *do* have a platform package manager, it is unlikely to
include every package that is available on the Python Package Index, and
even when a desired third-party package is available, the correct name in
the platform package manager may not be clear.

This means that, to work effectively with the Python Package Index
ecosystem, users must know which package manager to install, where to get
it, and how to install it. The effect of this is that third-party Python
projects are currently required to choose from a variety of undesirable
alternatives:

* Assume the user already has a suitable cross-platform package manager
  installed.
* Duplicate the instructions and tell their users how to install the
  package manager.
* Completely forgo the use of dependencies to ease installation concerns
  for their users.

All of these available options have significant drawbacks.

If a project simply assumes a user already has the tooling then beginning
users may get a confusing error message when the installation command
doesn't work. Some operating systems may ease this pain by providing a
global hook that looks for commands that don't exist and suggest an OS
package they can install to make the command work, but that only works
on systems with platform package managers that include a package that
provides the relevant cross-platform installer command (such as many major
Linux distributions). No such assistance is available for Windows and
Mac OS X users, or more conservative Linux distributions. The challenges
of dealing with this problem for beginners (who are often also completely
new to programming, the use of command line tools and editing system
environment variables) are a regular feature of feedback the core Python
developers receive from professional educators and others introducing new
users to Python.

If a project chooses to duplicate the installation instructions and tell
their users how to install the package manager before telling them how to
install their own project then whenever these instructions need updates
they need updating by every project that has duplicated them. This is
particular problematic when there are multiple competing installation
tools available, and different projects recommend different tools.

This specific problem can be partially alleviated by strongly promoting
``pip`` as the default installer and recommending that other projects
reference `pip's own bootstrapping instructions
<http://www.pip-installer.org/en/latest/installing.html>`__ rather than
duplicating them. However the user experience created by this approach
still isn't good (especially on Windows, where downloading and running
the ``get-pip.py`` bootstrap script with the default OS configuration is
significantly more painful than downloading and running a binary executable
or installer). The situation becomes even more complicated when multiple
Python versions are involved (for example, parallel installations of
Python 2 and Python 3), since that makes it harder to create and maintain
good platform specific ``pip`` installers independently of the CPython
installers.

The projects that have decided to forgo dependencies altogether are forced
to either duplicate the efforts of other projects by inventing their own
solutions to problems or are required to simply include the other projects
in their own source trees. Both of these options present their own problems
either in duplicating maintenance work across the ecosystem or potentially
leaving users vulnerable to security issues because the included code or
duplicated efforts are not automatically updated when upstream releases a new
version.

By officially recommending and providing by default a specific cross-platform
package manager it will be easier for users trying to install these
third-party packages as well as easier for the people distributing them as
they should now be able to safely assume that most users will have the
appropriate installation tools available (or access to clear instructions on
how to obtain them). This is expected to become more important in the future
as the Wheel_ package format (deliberately) does not have a built in
"installer" in the form of ``setup.py`` so users wishing to install
from a wheel file will want an installer even in the simplest cases.

Reducing the burden of actually installing a third-party package should
also decrease the pressure to add every useful module to the standard
library. This will allow additions to the standard library to focus more
on why Python should have a particular tool out of the box, and why it
is reasonable for that package to adopt the standard library's 18-24 month
feature release cycle, instead of using the general difficulty of installing
third-party packages as justification for inclusion.

Providing a standard installation system also helps with bootstrapping
alternate build and installer systems, such as ``setuptools``,
``zc.buildout`` and the ``hashdist``/``conda`` combination that is aimed
specifically at the scientific community. So long as
``pip install <tool>`` works, then a standard Python-specific installer
provides a reasonably secure, cross platform mechanism to get access to
these utilities.


Enabling the evolution of the broader Python packaging ecosystem
----------------------------------------------------------------

As no new packaging standard can achieve widespread adoption without a
transition strategy that covers the versions of Python that are in
widespread *current* use (rather than merely future versions, like most
language features), the change proposed in this PEP is considered a
necessary step in the evolution of the Python packaging ecosystem

The broader community has embraced the Python Package Index as a mechanism
for distributing and installing Python software, but the different concerns
of language evolution and secure software distribution mean that a faster
feature release cycle that encompasses older versions is needed to properly
support the latter.

In addition, the core CPython development team have the luxury of
dropping support for earlier Python versions well before the rest of the
community, as downstream commercial redistributors pick up the task of
providing support for those versions to users that still need it, while
many third party libraries maintain compatibility with those versions as
long as they remain in widespread use.

This means that the current ``setup.py install`` based model for package
installation poses serious difficulties for the development and adoption
of new packaging standards, as, depending on how a project writes their
``setup.py`` file, the installation command (along with other operations)
may end up invoking the standard library's ``distutils`` package.

As an indicator of how this may cause problems for the broader ecosystem,
consider that the feature set of ``distutils`` in Python 2.6 was frozen
in June 2008 (with the release of Python 2.6b1), while the feature set of
``distutils`` in Python 2.7 was frozen in April 2010 (with the release of
Python 2.7b1).

By contrast, using a separate installer application like ``pip`` (which
ensures that even ``setup.py`` files that invoke ``distutils`` directly
still support the new packaging standards) makes it possible to support
new packaging standards in older versions of Python, just by upgrading
``pip`` (which receives new feature releases roughly every 6 months). The
situation on older versions of Python is further improved by making it
easier for end users to install and upgrade newer build systems like
``setuptools`` or improved PyPI upload utilities like ``twine``.

It is not coincidental that this proposed model of using a separate installer
program with more metadata heavy and less active distribution formats matches
that used by most operating systems (including Windows since the introduction
of the installer service and the MSI file format), as well as many other
language specific installers.

For Python 2.6, this compatibility issue is largely limited to various
enterprise Linux distributions (and their downstream derivatives). These
distributions often have even slower update cycles than CPython, so they
offer full support for versions of Python that are considered "security
fix only" versions upstream (and sometimes may even be to the point where
the core development team no longer support them at all - you can still get
commercial support for Python 2.3 if you really need it!).

In practice, the fact that tools like ``wget`` and ``curl`` are readily
available on Linux systems, that most users of Python on Linux are
already familiar with the command line, and that most Linux distributions
ship with a default configuration that makes running Python scripts easy,
means that the existing ``pip`` bootstrapping instructions for any \*nix
system are already quite straightforward. Even if ``pip`` isn't provided by
the system package manager, then using ``wget`` or ``curl`` to retrieve the
bootstrap script from www.pip-installer.org and then running it is just a
couple of shell commands that can easily be copied and pasted as necessary.

Accordingly, for any version of Python on any \*nix system, the need to
bootstrap ``pip`` in older versions isn't considered a major barrier to
adoption of new packaging standards, since it's just one more small
speedbump encountered by users of these long term stable releases. For
\*nix systems, this PEP's formal endorsement of ``pip`` as the preferred
default packaging tool is seen as more important than the underlying
technical details involved in making ``pip`` available by default, since
it shifts the nature of the conversation between the developers of ``pip``
and downstream repackagers of both ``pip`` and CPython.

For Python 2.7, on the other hand, the compatibility issue for adopting new
metadata standards is far more widespread, as it affects the python.org
binary installers for Windows and Mac OS X, as well as even relatively
fast moving \*nix platforms.

Firstly, and unlike Python 2.6, Python 2.7 is still a fully supported
upstream version, and will remain so until the release of Python 2.7.9
(currently scheduled for May 2015), at which time it is expected to enter
the usual "security fix only" mode. That means there are at least another
19 months where Python 2.7 is a deployment target for Python applications
that enjoys full upstream support. Even after the core development team
switches 2.7 to security release only mode in 2015, Python 2.7 will likely
remain a commercially supported legacy target out beyond 2020.

While Python 3 already presents a compelling alternative over Python 2 for
*new* Python applications and deployments without an existing investment
in Python 2 and without a dependency on specific Python 2 only third party
modules (a set which is getting ever smaller over time), it is going to take
longer to create compelling business cases to update existing Python 2.7
based infrastructure to Python 3, especially in situations where the culture
of automated testing is weak (or nonexistent), making it difficult to
effectively use the available migration utilities.

It is quite likely that it is this difference in perspective regarding what
it means for a version of Python to be "supported" which lies at the heart
of the long history of conflicts between the developers of Python packaging
tools and the core development team for CPython. A key goal of this PEP is
thus to better enable the two groups to collaborate more effectively, by
using the ``ensurepip`` module as the technical bridge between the two
distinct software lifecycles and deployment models.


Why pip?
--------

``pip`` has been chosen as the preferred default installer, as it is an
already popular tool that addresses several design and user experience
issues with its predecessor ``easy_install`` (these issues can't readily
be fixed in ``easy_install`` itself due to backwards compatibility
concerns). ``pip`` is also well suited to working within the bounds of
a single Python runtime installation (including associated virtual
environments), which is a desirable feature for a tool bundled with CPython.

Other tools like ``zc.buildout`` and ``conda`` are more ambitious in their
aims (and hence substantially better than ``pip`` at handling external
binary dependencies), so it makes sense for the Python ecosystem to treat
them more like platform package managers to interoperate with rather than
as the default cross-platform installation tool. This relationship is
similar to that between ``pip`` and platform package management systems
like ``apt`` and ``yum`` (which are also designed to handle arbitrary
binary dependencies).


Proposal Overview
=================

This PEP proposes that the
`Installing Python Modules <http://docs.python.org/3/install>`__ guide be
updated to officially recommend the use of ``pip`` as the default
installer for Python packages, rather than the current approach of
recommending the direct invocation of the ``setup.py install`` ``distutils``
command.

However, to avoid recommending a tool that CPython does not provide, it is
further proposed that the `pip`_ package manager be made available by
default when installing CPython and when creating virtual environments
using the standard library's ``venv`` module via the ``pyvenv`` command line
utility).

To support that end, this PEP proposes the inclusion of an ``ensurepip``
bootstrapping module in Python 3.4 (along with the upcoming maintenance
releases of Python 2.7 and 3.3), as well as changes to the way Python
installed scripts are handled on Windows (for Python 3.4 only).

To clearly demarcate development responsibilities, and to avoid
inadvertently downgrading ``pip`` when updating CPython, the proposed
mechanism to achieve this is to include an explicit `pip`_ bootstrapping
mechanism in the standard library that is invoked automatically by the
CPython installers provided on python.org.

To ensure the smoothest possible experience for new users of Python, this
PEP also proposes that the ``ensurepip`` module and the option to install
``pip`` when installing CPython be backported to Python 2.7 and 3.3. It
does *not* propose backporting any changes to ``pyvenv`` (in Python 3.3)
or to Windows script handling (in either version).

Finally, the PEP also strongly recommends that CPython redistributors and
other Python implementations ensure that ``pip`` is available by default, or
at the very least, explicitly document the fact that it is not included.

This PEP does *not* propose making pip (or any dependencies) directly
available as part of the standard library. Instead, pip will be a
bundled application provided along with CPython for the convenience
of Python users, but subject to its own development life cycle and able
to be upgraded independently of the core interpreter and standard library.


Explicit bootstrapping mechanism
================================

An additional module called ``ensurepip`` will be added to the standard
library whose purpose is to install pip and any of its dependencies into the
appropriate location (most commonly site-packages). It will expose a
callable named ``bootstrap()`` as well as offer direct execution via
``python -m ensurepip``.

The bootstrap will *not* contact PyPI, but instead rely on a private copy
of pip stored inside the standard library. Accordingly, only options
related to the installation location will be supported (``--user``,
``--root``, etc).

It is considered desirable that users be strongly encouraged to use the
latest available version of ``pip``, in order to take advantage of the
ongoing efforts to improve the security of the PyPI based ecosystem, as
well as benefiting from the efforts to improve the speed, reliability and
flexibility of that ecosystem.

In order to satisfy this goal of providing the most recent version of
``pip`` by default, the private copy of ``pip`` will be updated in CPython
maintenance releases, which should align well with the 6-month cycle used
for new ``pip`` releases.


Security considerations
-----------------------

The design in this PEP has been deliberately chosen to avoid making any
significant changes to the trust model of the CPython installers for end
users that do not subsequently make use of ``pip``.

The installers will contain all the components of a fully functioning
version of Python, including the ``pip`` installer. The installation
process will *not* require network access, and will *not* rely on
trusting the security of the network connection established between
``pip`` and the Python package index.

Only users that choose to use ``pip`` directly will need to pay
attention to any PyPI related security considerations.


Reliability considerations
--------------------------

By including the bootstrap as part of the standard library (rather than
solely as a feature of the binary installers), the correct operation of
the bootstrap command can be easily tested using the existing CPython
buildbot infrastructure rather than adding significantly to the testing
burden for the installers themselves.


Implementation strategy
-----------------------

To ensure there is no need for network access when installing Python or
creating virtual environments, the ``ensurepip`` module will, as an
implementation detail, include a complete private copy of pip and its
dependencies which will be used to extract pip and install it into the target
environment. It is important to stress that this private copy of pip is
*only* an implementation detail and it should *not* be relied on or
assumed to exist beyond the public capabilities exposed through the
``ensurepip`` module (and indirectly through ``venv``).

There is not yet a reference ``ensurepip`` implementation. The existing
``get-pip.py`` bootstrap script demonstrates an earlier variation of the
general concept, but the standard library version would take advantage of
the improved distribution capabilities offered by the CPython installers
to include private copies of ``pip`` and ``setuptools`` as wheel files
(rather than as embedded base64 encoded data), and would not try to
contact PyPI (instead installing directly from the private wheel files.

Rather than including separate code to handle the bootstrapping, the
``ensurepip`` module will manipulate ``sys.path`` appropriately to allow
the wheel files to be used to install themselves, either into the current
Python installation or into a virtual environment (as determined by the
options passed to the bootstrap command).

It is proposed that the implementation be carried out in five separate
steps (all steps after the first are independent of each other and can be
carried out in any order):

* the first step would add the ``ensurepip`` module and the private copies
  of the most recently released versions of pip and setuptools, and update
  the "Installing Python Modules" documentation. This change
  would be applied to Python 2.7, 3.3, and 3.4.
* the Windows installer would be updated to offer the new ``pip``
  installation option for Python 2.7.6, 3.3.3 and 3.4.0.
* the Mac OS X installer would be updated to offer the new ``pip``
  installation option for Python 2.7.6, 3.3.3 and 3.4.0.
* the ``venv`` module and ``pyvenv`` command would be updated to make use
  of ``ensurepip`` in Python 3.4+
* the PATH handling and ``sysconfig`` directory layout on Windows would be
  updated for Python 3.4+


Proposed CLI
------------

The proposed CLI is based on a subset of the existing ``pip install``
options::

    Usage:
      python -m ensurepip [options]

    General Options:
      -h, --help          Show help.
      -v, --verbose       Give more output. Option is additive, and
can be used up to 3 times.
      -V, --version       Show the pip version that would be extracted and exit.
      -q, --quiet         Give less output.

    Installation Options:
      -U, --upgrade       Upgrade pip and dependencies, even if
already installed
      --user              Install using the user scheme.
      --root <dir>        Install everything relative to this
alternate root directory.

In most cases, end users won't need to use this CLI directly, as ``pip``
should have been installed automatically when installing Python or when
creating a virtual environment. However, it is formally documented as a
public interface to support at least these known use cases:

* Windows and Mac OS X installations where the "Install pip" option was
  *not* chosen during installation
* any installation where the user previously ran "pip uninstall pip"
* virtual environments created with ``pyvenv`` in Python 3.3

Users that want to retrieve the latest version from PyPI, or otherwise
need more flexibility, can then invoke the extracted ``pip`` appropriately.


Proposed module API
-------------------

The proposed ``ensurepip`` module API consists of the following two
functions::

    def version():
        """
        Returns a string specifying the bundled version of pip.
        """

    def bootstrap(root=None, upgrade=False, user=False, verbosity=0):
        """
        Bootstrap pip into the current Python installation (or the given root
        directory).
        """


Invocation from the CPython installers
--------------------------------------

The CPython Windows and Mac OS X installers will each gain a new option:

* Install pip (the default Python package management utility)?

This option will be checked by default.

If the option is checked, then the installer will invoke the following
command with the just installed Python::

    python -m ensurepip --upgrade

This ensures that, by default, installing or updating CPython will ensure
that the installed version of pip is at least as recent as the one included
with that version of CPython. If a newer version of pip has already been
installed then ``python -m ensurepip --upgrade`` will simply return without
doing anything.


Installing from source
----------------------

While the prebuilt binary installers will be updated to run
``python -m ensurepip`` by default, no such change will be made to the
``make install`` and ``make altinstall`` commands of the source
distribution.

``ensurepip`` itself (including the private copy of ``pip`` and its
dependencies) will still be installed normally (as it is a regular
part of the standard library), only the implicit installation of pip and
its dependencies will be skipped.

Keeping the pip bootstrapping as a separate step for ``make``-based
installations should minimize the changes CPython redistributors need to
make to their build processes. Avoiding the layer of indirection through
``make`` for the ``ensurepip`` invocation avoids any challenges
associated with determining where to install the extracted ``pip``.


Changes to virtual environments
-------------------------------

Python 3.3 included a standard library approach to virtual Python environments
through the ``venv`` module. Since its release it has become clear that very
few users have been willing to use this feature directly, in part due to the
lack of an installer present by default inside of the virtual environment.
They have instead opted to continue using the ``virtualenv`` package which
*does* include pip installed by default.

To make the ``venv`` more useful to users it will be modified to issue the
pip bootstrap by default inside of the new environment while creating it. This
will allow people the same convenience inside of the virtual environment as
this PEP provides outside of it as well as bringing the ``venv`` module closer
to feature parity with the external ``virtualenv`` package, making it a more
suitable replacement.

To handle cases where a user does not wish to have pip bootstrapped into
their virtual environment a ``--without-pip`` option will be
added.

The ``venv.EnvBuilder`` and ``venv.create`` APIs will be updated to accept
one new parameter: ``with_pip`` (defaulting to ``False``).

The new default for the module API is chosen for backwards compatibility
with the current behaviour (as it is assumed that most invocation of the
``venv`` module happens through third part tools that likely will not
want ``pip`` installed without explicitly requesting it), while the
default for the command line interface is chosen to try to ensure ``pip``
is available in most virtual environments without additional action on the
part of the end user.

This particular change will be made only for Python 3.4 and later versions.
The third-party ``virtualenv`` project will still be needed to obtain a
consistent cross-version experience in Python 3.3 and 2.7.


Documentation
-------------

The "Installing Python Modules" section of the standard library
documentation in Python 2.7, 3.3 and 3.4 will be updated to recommend
the use of the bootstrapped ``pip`` installer. It will give a brief
description of the most common commands and options, but delegate
to the externally maintained ``pip`` documentation for the full details.

In Python 3.4, the ``pyvenv`` and ``venv`` documentation will also be
updated to reference the revised module installation guide.

In Python 2.7 and 3.3, the documentation will make clear that the feature
was added in a maintenance release and users may need to upgrade in order
to take advantage of it. Specifically, it is proposed to include the
following warning as a note in the documentation for the ``ensurepip``
module in these versions (adjust version numbers for 3.3 as appropriate):

    This is an optional module, which may not be available in all
    installations of Python 2.7. It is provided solely to simplify the
    process of bootstrapping ``pip`` onto end user's systems. If it is
    not available, please investigate the following alternatives:

    * This module was first added in Python 2.7.6. If using an earlier
      maintenance release, it will not be available. If upgrading to
      a more recent maintenance release is not an option, consider
      the alternative bootstrapping mechanisms below.
    * Some platforms provide alternative mechanisms to obtain ``pip``.
      In such cases, the platform documentation should provide
      appropriate details.
    * If upgrading to the latest maintenance release is not feasible, and
      no platform specific instructions are provided, then refer to the
      upstream `pip bootstrapping instructions
      <http://www.pip-installer.org/en/latest/installing.html>`__.

The existing content of the module installation guide will be retained in
all versions, but under a new "Invoking distutils directly" subsection.


Bundling CA certificates with CPython
-------------------------------------

The ``ensurepip`` implementation will include the ``pip`` CA bundle along
with the rest of ``pip``. This means CPython effectively includes
a CA bundle that is used solely by ``pip`` after it has been extracted.

This is considered preferable to relying solely on the system
certificate stores, as it ensures that ``pip`` will behave the same
across all supported versions of Python, even those prior to Python 3.4
that cannot access the system certificate store on Windows.


Automatic installation of setuptools
------------------------------------

``pip`` currently depends on ``setuptools`` to handle metadata generation
during the build process, along with some other features. While work is
ongoing to reduce or eliminate this dependency, it is not clear if that
work will be complete for pip 1.5 (which is the version likely to be current
when Python 3.4.0 is released).

This PEP proposes that, if pip still requires it as a dependency,
``ensurepip`` will include a private copy of ``setuptools`` (in addition
to the private copy of ``ensurepip``). ``python -m ensurepip`` will then
install the private copy in addition to installing ``pip`` itself.

However, this behavior is officially considered an implementation
detail. Other projects which explicitly require ``setuptools`` must still
provide an appropriate dependency declaration, rather than assuming
``setuptools`` will always be installed alongside ``pip``.

Once pip is able to run ``pip install --upgrade pip`` without needing
``setuptools`` installed first, then the private copy of ``setuptools``
will be removed from ``ensurepip`` in subsequent CPython releases.

As long as setuptools is needed, it will be a completely unmodified copy of
the latest upstream setuptools release, including the ``easy_install``
script if the upstream setuptools continues to include it. The installation
of ``easy_install`` along with ``pip`` isn't considered desirable, but
installing a broken setuptools would be worse. This problem will
naturally resolve itself once the ``pip`` developers have managed to
eliminate their dependency on ``setuptools`` and the private copy of
``setuptools`` can be removed entirely from CPython.


Updating the private copy of pip
--------------------------------

In order to keep up with evolutions in packaging as well as providing users
with as recent version a possible the ``ensurepip`` module will be
regularly updated to the latest versions of everything it bootstraps.

After each new ``pip`` release, and again during the preparation for any
release of Python (including feature releases), a script, provided as part
of the implementation for this PEP, will be run to ensure the private
copies stored in the CPython source repository have been updated to the
latest versions.


Updating the ensurepip module API and CLI
-----------------------------------------

Like ``venv`` and ``pyvenv``, the ``ensurepip`` module API and CLI
will be governed by the normal rules for the standard library: no
new features are permitted in maintenance releases.

However, the embedded components may be updated as noted above, so
the extracted ``pip`` may offer additional functionality in maintenance
releases.


Feature addition in maintenance releases
========================================

Adding a new module to the standard library in Python 2.7, and 3.3
maintenance releases breaks the usual policy of "no new features in
maintenance releases". The rationale for doing so in this case is slightly
different for each of the two versions.


Rationale for this policy on maintenance releases
-------------------------------------------------

Python's strict "no new features in maintenance releases" was instituted
following the introduction of a number of new features over the course of
the Python 2.2.x series.

Firstly, the ``True`` and ``False`` builtins were added in Python 2.2.1 (at
the time, they were merely aliases for the values ``1`` and ``0``,
in Python 2.3 they became instances of the new ``bool`` type and in Python
3.0 they became true constants recognised by the compiler).

Python 2.2.2 then made the situation worse by adding a new ``chars``
parameter to the ``lstrip`` and ``rstrip`` string methods, along with an
entirely new ``zfill`` method. The corresponding changes in the
``string`` module were not incorporated until Python 2.2.3.

The reason introducing new features in maintenance releases like this is
problematic is that, except in the cases where code fails due to a bug in
CPython, developers expect to be able to identify the supported Python
versions for a library or application solely through the first two
components of the version number.

The introduction of new builtins and string methods in Python 2.2.1 and
2.2.2 resulted in many developers claiming Python 2.2 compatibility for
code that did not in fact run on the original Python 2.2. In effect,
Python 2.2.2 became the minimum usable version, since there was a
relatively high chance of code breaking when run on 2.2 (or even 2.2.1).


Scope of this proposal
----------------------

By contrast with the changes that caused such problems during the 2.2.x
series, this PEP is merely proposing the addition of a new standard
library module, rather than adding new builtins or changing the interface
of a builtin type.

The categorical difference between these kinds of changes has already been
recognised in the Python 3.3 Language Moratorium (PEP 3003), where the
addition of new builtins was disallowed outright and changes to builtin
types required an explicit exemption. By contrast, adding new modules was
explicitly permitted, even while the moratorium was in place.

Furthermore, the proposed ``ensurepip`` module is only a means to the end of
getting ``pip`` installed on the system. While "upgrade to the latest
CPython maintenance release" will become the *recommended* approach to
obtaining ``pip`` for users of Python 2.7 and 3.3 on Windows and Mac OS X
systems, all of the existing ``pip`` bootstrapping mechanisms will still
work in cases where upgrading Python isn't a practical alternative.

As described in the documentation update proposal, the ``ensurepip``
documentation in older releases will include the text explaining how to
obtain ``pip`` if updating to the latest maintenance release isn't an
option, or if the module has been removed by a redistributor. This contrasts
significantly with the changes made during the Python 2.2 series, where they
were normal additions with no alternatives except to update to a sufficiently
recent version of Python if a library or application depended on them.


Potential consequences of permitting this exemption
---------------------------------------------------

The concern has been expressed that approving an exemption to the "no new
features in maintenance releases" policy in this case will open the
flood gates to requests for more such exemptions in the future. It is the
perspective of the PEP authors that the specific nature of this proposal
should itself serve to allay those fears.

Firstly, as a proposal to add a new module to the standard library, granting
an exemption in this case sets no precedent for the more restricted
categories identified in the PEP 3003 language moratorium.

Secondly, this exemption is requested for a new module that *makes it easy
to download other modules from PyPI*. If this PEP is accepted, then it can
be reasonably assumed that modules on PyPI are only a ``pip install`` away
for most users, with only those users that depend on standard library
inclusion to make it through corporate compliance reviews still affected
(and, for many such reviews, inclusion in a future version of the standard
library will be enough for a backported version to be considered
acceptable for use).

Making ``pip`` readily available in all versions still under normal
maintenance thus means that accepting this PEP should have the effect of
*weakening* the case for any further exemptions to the policy, rather
than strengthening it.


Rationale for permitting the exemption in Python 2.7
----------------------------------------------------

While Python 3 adoption is proceeding nicely, it remains the case that many
new users of Python are introduced to Python 2.7 first. This may be
because their instructors have yet to migrate their courses to Python 3, or
because they work in an environment where Python 2 is still the preferred
version, or simply because the frequently adopted approach of writing
libraries in the common Python 2/3 subset means there are (as of
September 2013) still more Python 2 only libraries than there are Python 3
only libraries.

Since one of the primary aims of this PEP is to aid new Python users, it is
contrary to its spirit to target *only* Python 3.4, when so many users in
at least the next 12-18 months (where Python 2.7 is still fully supported
by the core development team) are still going to be introduced to Python
2 before being introduced to Python 3.

Users first installing Python 2.7 on Windows and Mac OS X following
acceptance and release of this PEP won't even need to look up how to
bootstrap ``pip``, since it will already be provided with the CPython
installer. For those that already have Python installed, but are just
beginning to explore the PyPI ecosystem, the bootstrapping instructions
can be simplified to "just install the latest maintenance release of
CPython".

Making ``pip`` readily available also serves to ease the migration path
from Python 2 to Python 3, as a number of the standard library additions
in Python 3 are also available from PyPI for Python 2. Lowering the barrier
to adoption for these backports makes it easier for current Python 2 users
to selectively adopt backporting Python 3 versions, reducing the number of
updates needed in any eventual Python 3 migration.

Finally, this PEP solves a serious problem for the ``distutils-sig``
community, as it means we will finally have a standard mechanism decoupled
from the standard library's development lifecycle that we can reasonably
assume to be present on end user's systems (or at least readily
available) that allows us to support new packaging standards in older
versions of Python. A tentative, half-hearted endorsement from the
CPython core development team that tries to hide the existence of the
pip boostrapping support from end users is unlikely to provide quite the
same benefits.


Rationale for permitting the exemption in Python 3.3
----------------------------------------------------

The rationale for permitting the exemption in Python 3.3 is admittedly
not as strong as it is for Python 2.7, as instructors currently using
Python 3.3 are quite likely to upgrade to Python 3.4 shortly after it is
released.

In the case of Python 3.3, the rationale is primarily a consistency
argument, as it permits the recommended ``pip`` bootstrapping instructions
for both 2.7 and 3.3 to be to upgrade to the latest maintenance version of
CPython. While existing bootstrapping mechanisms will still be supported,
the cases where they are needed should be reduced significantly.

Adding the ``ensurepip`` module in Python 3.3 also makes the Python 3.3
version of the ``pyvenv`` utility far more useful (even without the
integration proposed for Python 3.4), as it allows users to execute
``python -m ensurepip`` to bootstrap ``pip`` after activating an
existing or newly created virtual environment.


Uninstallation
==============

No changes are proposed to the CPython uninstallation process by this PEP.
The bootstrapped pip will be installed the same way as any other pip
installed packages, and will be handled in the same way as any other
post-install additions to the Python environment.

At least on Windows, that means the bootstrapped files will be
left behind after uninstallation, since those files won't be associated
with the Python MSI installer.

While the case can be made for the CPython installers clearing out these
directories automatically, changing that behaviour is considered outside
the scope of this PEP.


Script Execution on Windows
===========================

While the Windows installer was updated in Python 3.3 to optionally
make ``python`` available on the PATH, no such change was made to
include the Scripts directory. Independently of this PEP, a proposal has
also been made to rename the ``Tools\Scripts`` subdirectory to ``bin`` in
order to improve consistency with the typical script installation directory
names on \*nix systems.

Accordingly, in addition to adding the option to extract and install ``pip``
during installation, this PEP proposes that the Windows installer (and
``sysconfig``) in Python 3.4 and later be updated to:

- install scripts to PythonXY\bin rather than PythonXY\Tools\Scripts
- add PythonXY\bin to the Windows PATH (in addition to PythonXY) when the
  PATH modification option is enabled during installation

For Python 2.7 and 3.3, it is proposed that the only change be the one to
bootstrap ``pip`` by default.

This means that, for Python 3.3, the most reliable way to invoke pip
globally on Windows (without tinkering manually with PATH) will actually be
``py -m pip`` (or ``py -3 -m pip`` to select the Python 3 version if both
Python 2 and 3 are installed) rather than simply calling ``pip``.

For Python 2.7 and 3.2, the most reliable mechanism will be to install the
standalone Python launcher for Windows and then use ``py -m pip`` as noted
above.

Adding the scripts directory to the system PATH would mean that ``pip``
works reliably in the "only one Python installation on the system PATH"
case, with ``py -m pip``, ``pipX``, or ``pipX.Y`` needed only to select a
non-default version in the parallel installation case (and outside a virtual
environment). This change should also make the ``pyvenv`` command substantially
easier to invoke on Windows, along with all scripts installed by ``pip``,
``easy_install`` and similar tools.

While the script invocations on recent versions of Python will run through
the Python launcher for Windows, this shouldn't cause any issues, as long
as the Python files in the Scripts directory correctly specify a Python version
in their shebang line or have an adjacent Windows executable (as
``easy_install`` and ``pip`` do).


Recommendations for Downstream Distributors
===========================================

A common source of Python installations are through downstream distributors
such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX
package managers [#homebrew]_ [#macports]_ [#fink]_, and commercial Python
redistributors [#ContinuumIO]_ [#ActiveState]_ [#Enthought]_. In order to
provide a consistent, user-friendly experience to all users of Python
regardless of how they obtained Python this PEP recommends and asks that
downstream distributors:

* Ensure that whenever Python is installed ``pip`` is either installed or is
  otherwise made readily available to end users.

  * For redistributors using binary installers, this may take the form of
    optionally executing the ``ensurepip`` bootstrap during installation,
    similar to the CPython installers.
  * For redistributors using package management systems, it may take the
    form of separate packages with dependencies on each other so that
    installing the Python package installs the pip package and installing
    the pip package installs the Python package.
  * Another reasonable way to implement this is to package pip separately but
    ensure that there is some sort of global hook that will recommend
    installing the separate pip package when a user executes ``pip`` without
    it being installed. Systems that choose this option should ensure that
    the ``ensurepip`` module still installs pip directly when invoked inside
    a virtual environment, but may modify the module in the system Python
    installation to redirect to the platform provided mechanism when
    installing ``pip`` globally.

* Even if pip is made available globally by other means, do not remove the
  ``ensurepip`` module in Python 3.3 or later.

  * In Python 3.3, ``ensurepip`` will be the recommended way of bootstrapping
    pip in virtual environments created through the ``venv`` module and the
    associated ``pyvenv`` command line tool.
  * Starting with Python 3.4, ``ensurepip`` will be required for automatic
    installation of pip into virtual environments by the ``venv`` module.
  * This is similar to the existing ``virtualenv`` package for which many
    downstream distributors have already made exception to the common
    "debundling" policy.
  * This does mean that if ``pip`` needs to be updated due to a security
    issue, so does the private copy in the ``ensurepip`` bootstrap module
  * However, altering the private copy of pip to remove the embedded
    CA certificate bundle and rely on the system CA bundle instead is a
    reasonable change.
  * If ``pip`` is made available globally by other means in Python 2.7, then
    it is acceptable (although not desirable) to disable the ``ensurepip``
    module (as the third party ``virtualenv`` distribution is needed to
    create virtual environments in Python 2.7 and ``virtualenv`` already
    ensures ``pip`` is installed into the virtual environments it creates).
    Redistributors that take this course should ensure an appropriate error
    message is displayed if users attempt to import ``ensurepip``, rather
    than simply removing it entirely.

* Ensure that all features of this PEP continue to work with any modifications
  made to the redistributed version of Python.

  * Checking the version of pip that will be bootstrapped using
    ``python -m ensurepip --version`` or ``ensurepip.version()``.
  * Installation of pip into a global or virtual python environment using
    ``python -m ensurepip`` or ``ensurepip.bootstrap()``.
  * ``pip install --upgrade pip`` in a global installation should not affect
    any already created virtual environments (but is permitted to affect
    future virtual environments, even though it will not do so when using
    the standard implementation of ``ensurepip``).
  * ``pip install --upgrade pip`` in a virtual environment should not affect
    the global installation.

* Migrate build systems to utilize `pip`_ and `Wheel`_ wherever feasible
  and avoid directly invoking ``setup.py``.

  * This will help ensure a smoother and more timely migration to improved
    metadata formats as the Python packaging ecosystem continues to evolve.

In the event that a Python redistributor chooses *not* to follow these
recommendations, we request that they explicitly document this fact and
provide their users with suitable guidance on translating upstream ``pip``
based installation instructions into something appropriate for the platform.

Other Python implementations are also encouraged to follow these guidelines
where applicable.


Policies & Governance
=====================

The maintainers of the bootstrapped software and the CPython core team will
work together in order to address the needs of both. The bootstrapped
software will still remain external to CPython and this PEP does not
include CPython subsuming the development responsibilities or design
decisions of the bootstrapped software. This PEP aims to decrease the
burden on end users wanting to use third-party packages and the
decisions inside it are pragmatic ones that represent the trust that the
Python community has already placed in the Python Packaging Authority as
the authors and maintainers of ``pip``, ``setuptools``, PyPI, ``virtualenv``
and other related projects.


Backwards Compatibility
-----------------------

The public API and CLI of the ``ensurepip`` module itself will fall under
the typical backwards compatibility policy of Python for its standard
library. The externally developed software that this PEP bundles does not.

Most importantly, this means that the bootstrapped version of pip may gain
new features in CPython maintenance releases, and pip continues to operate on
its own 6 month release cycle rather than CPython's 18-24 month cycle.


Security Releases
-----------------

Any security update that affects the ``ensurepip`` module will be shared
prior to release with the Python Security Response Team
(security at python.org). The PSRT will then decide if the reported issue
warrants a security release of CPython with an updated private copy of
``pip``.


Licensing
---------

``pip`` is currently licensed as 1 Clause BSD, and it contains code taken
from other projects. Additionally this PEP will include setuptools until
such time as pip no longer requires it. The licenses for these appear in
the table below.

================= ============
     Project        License
================= ============
requests           Apache 2.0
six               1 Clause BSD
html5lib          1 Clause BSD
distlib                PSF
colorama          3 Clause BSD
Mozilla CA Bundle      LGPL
setuptools             PSF
================= ============

All of these licenses should be compatible with the PSF license. Additionally
it is unclear if a CA Bundle is copyrightable material and thus if it needs
or can be licensed at all.


Appendix: Rejected Proposals
============================


Include pip *only* inside the installers in Python 2.7, and 3.3
---------------------------------------------------------------

An alternative to making an exception to the "no new features" policy in
Python 2.7 and 3.3 would be to simply bundle pip with the installer and not
modify the source tree at all. The motivation behind this modification is
that adding a new feature in a maintenance release is a risky proposition
and that doing it in this way doesn't violate that policy.

This has been rejected because:

* It's dubious to declare the binary installers beyond the scope of the
  "no new features in maintenance releases" policy. If the rationale for
  adding this feature to the standard library in a maintenance release isn't
  considered adequate, then it isn't clear why moving that complexity to the
  binary installers should change the verdict.
* Attempting to hide the existence of the bootstrap module from end users
  makes it more difficult to write updated package installation documentation
  for Python 2.7 and 3.3
* For 3.3 users that choose to use ``pyvenv`` rather than ``virtualenv``,
  an explicit ``python -m ensurepip`` will be needed to bootstrap ``pip``
  into virtual environments. This can only be documented clearly if the
  module is public
* Making the bootstrap an installer only feature in Python 2.7 and 3.3
  guarantees the introduction of cross-platform inconsistencies, whereas
  the proposal in this PEP more strongly encourages redistributors to
  offer a more consistent user experience.
* Making the bootstrap an installer only feature in Python 2.7 and 3.3
  would make it difficult to re-use the bootstrap implementation from 3.4.
* Making the bootstrap an installer only feature prevents the buildbots
  from being able to run automatic tests against it, which would make
  ensuring that this feature remains working a much more difficult task.


Use a different module name in Python 2.7, and 3.3
--------------------------------------------------

Naming the module ``_ensurepip`` in Python 2.7 and 3.3 was considered as
another means of skirting the "no new features in maintenance releases"
policy. However, similar to the proposal to only include the new
feature in the installers rather than the standard library, this feels like
relying on a technicality to nominally "comply" with the policy, while still
breaking it in spirit.

It is the considered opinion of the PEP authors that attempting to hide
the addition of the ``ensurepip`` module in earlier versions will only
serve to increase confusion rather than to reduce it, so the proposal
remains to be up front about the fact that the policy is being broken in
this case, and clearly documenting the rationale for doing so in this PEP.

As noted in the section describing the proposed documentation updates,
having ``ensurepip`` as a public module in these earlier versions also
provides a convenient home for the fallback bootstrapping instructions in
those cases where it *isn't* available.


Automatically contacting PyPI when bootstrapping pip
----------------------------------------------------

Earlier versions of this PEP called the bootstrapping module ``getpip`` and
defaulted to downloading and installing ``pip`` from PyPI, with the private
copy used only as a fallback option or when explicitly requested.

This resulted in several complex edge cases, along with difficulties in
defining a clean API and CLI for the bootstrap module. It also significantly
altered the default trust model for the binary installers published on
python.org, as end users would need to explicitly *opt-out* of trusting
the security of the PyPI ecosystem (rather than opting in to it by
explicitly invoking ``pip`` following installation).

As a result, the PEP was simplified to the current design, where the
bootstrapping *always* uses the private copy of ``pip``. Contacting PyPI
is now always an explicit separate step, with direct access to the full
pip interface.


Implicit bootstrap
------------------

`PEP439`_, the predecessor for this PEP, proposes its own solution. Its
solution involves shipping a fake ``pip`` command that when executed would
implicitly bootstrap and install pip if it does not already exist. This has
been rejected because it is too "magical". It hides from the end user when
exactly the pip command will be installed or that it is being installed at
all. It also does not provide any recommendations or considerations towards
downstream packagers who wish to manage the globally installed pip through
the mechanisms typical for their system.

The implicit bootstrap mechanism also ran into possible permissions issues,
if a user inadvertently attempted to bootstrap pip without write access to
the appropriate installation directories.


Including pip directly in the standard library
----------------------------------------------

Similar to this PEP is the proposal of just including pip in the standard
library. This would ensure that Python always includes pip and fixes all of the
end user facing problems with not having pip present by default. This has been
rejected because we've learned, through the inclusion and history of
``distutils`` in the standard library, that losing the ability to update the
packaging tools independently can leave the tooling in a state of constant
limbo. Making it unable to ever reasonably evolve in a time frame that actually
affects users as any new features will not be available to the general
population for *years*.

Allowing the packaging tools to progress separately from the Python release
and adoption schedules allows the improvements to be used by *all* members
of the Python community and not just those able to live on the bleeding edge
of Python releases.

There have also been issues in the past with the "dual maintenance" problem
if a project continues to be maintained externally while *also* having a
fork maintained in the standard library. Since external maintenance of
``pip`` will always be needed to support earlier Python versions, the
proposed bootstrapping mechanism will becoming the explicit responsibility
of the CPython core developers (assisted by the pip developers), while
pip issues reported to the CPython tracker will be migrated to the pip
issue tracker. There will no doubt still be some user confusion over which
tracker to use, but hopefully less than has been seen historically when
including complete public copies of third-party projects in the standard
library.

The approach described in this PEP also avoids some technical issues
related to handling CPython maintenance updates when pip has been
independently updated to a more recent version. The proposed pip-based
bootstrapping mechanism handles that automatically, since pip and the
system installer never get into a fight about who owns the pip
installation (it is always managed through pip, either directly, or
indirectly via the ``ensurepip`` bootstrap module).

Finally, the separate bootstrapping step means it is also easy to avoid
installing ``pip`` at all if end users so desire. This is often the case
if integrators are using system packages to handle installation of
components written in multiple languages using a common set of tools.


Defaulting to --user installation
---------------------------------

Some consideration was given to bootstrapping pip into the per-user
site-packages directory by default. However, this behavior would be
surprising (as it differs from the default behavior of pip itself)
and is also not currently considered reliable (there are some edge cases
which are not handled correctly when pip is installed into the user
site-packages directory rather than the system site-packages).


.. _Wheel: http://www.python.org/dev/peps/pep-0427/
.. _pip: http://www.pip-installer.org
.. _setuptools: https://pypi.python.org/pypi/setuptools
.. _PEP439: http://www.python.org/dev/peps/pep-0439/


References
==========

.. [1] Discussion thread 1 (distutils-sig)
   (https://mail.python.org/pipermail/distutils-sig/2013-August/022529.html)

.. [2] Discussion thread 2 (distutils-sig)
   (https://mail.python.org/pipermail/distutils-sig/2013-September/022702.html)

.. [3] Discussion thread 3 (python-dev)
   (https://mail.python.org/pipermail/python-dev/2013-September/128723.html)

.. [4] Discussion thread 4 (python-dev)
   (https://mail.python.org/pipermail/python-dev/2013-September/128780.html)

.. [5] Discussion thread 5 (python-dev)
   (https://mail.python.org/pipermail/python-dev/2013-September/128894.html)

.. [#ubuntu] `Ubuntu <http://www.ubuntu.com/>`
.. [#debian] `Debian <http://www.debian.org>`
.. [#fedora] `Fedora <https://fedoraproject.org/>`
.. [#homebrew] `Homebrew <http://brew.sh/>`
.. [#macports] `MacPorts <http://macports.org>`
.. [#fink] `Fink <http://finkproject.org>`
.. [#ContinuumIO] `Anaconda <https://store.continuum.io/cshop/anaconda/>`
.. [#ActiveState] `ActivePython <http://www.activestate.com/activepython>`
.. [#Enthought] `Enthought Canopy <https://www.enthought.com/products/canopy/>`

Copyright
=========

This document has been placed in the public domain.


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From zhangpeipei812 at outlook.com  Sun Sep 29 14:51:37 2013
From: zhangpeipei812 at outlook.com (=?UTF-8?B?5byg5L2p5L2p?=)
Date: Sun, 29 Sep 2013 20:51:37 +0800
Subject: [Python-Dev] Why not support user defined operator overloading ?
Message-ID: <BLU403-EAS393D1EF893F5F5DC86003229D2B0@phx.gbl>

Hello:
	As far as I know, there is not a language support user defined operator overloading.
Python3 can overloading belowed operators.
-	negated
+	unchanged

-	minus
+	add
*	multiplication
/	division
//	true division
%	remainder
**	power
(Do I miss something ?)

If we can overloading these operators, why we can't overloading other operators?
(like .* often used in matrix, U in set operation)


Regards!
Peipei

From python-dev at masklinn.net  Sun Sep 29 17:06:32 2013
From: python-dev at masklinn.net (Xavier Morel)
Date: Sun, 29 Sep 2013 17:06:32 +0200
Subject: [Python-Dev] Why not support user defined operator overloading ?
In-Reply-To: <BLU403-EAS393D1EF893F5F5DC86003229D2B0@phx.gbl>
References: <BLU403-EAS393D1EF893F5F5DC86003229D2B0@phx.gbl>
Message-ID: <F53492D8-A90B-4449-9E83-25340AA63AEA@masklinn.net>


On 2013-09-29, at 14:51 , ??? wrote:

> Hello:
> 	As far as I know, there is not a language support user defined operator overloading.
> Python3 can overloading belowed operators.
> -	negated
> +	unchanged
> 
> -	minus
> +	add
> *	multiplication
> /	division
> //	true division
> %	remainder
> **	power
> (Do I miss something ?)

~         invert (unary)
()        call
.         get attribute
[]        get item
<<        left shift
>>        right shift
&         binary and
^         xor
|         binary or

And the inplace versions of most of these can be implemented separately,
which can probably be counted as supplementary operators.

> 
> If we can overloading these operators, why we can't overloading other operators?
> (like .* often used in matrix, U in set operation)

This is more of a python-ideas subject.

And one of the reasons likely is that it would require significantly
reworking the grammar to handle a kind of user-defined opname (similar
to name, but for operator tokens), with user-defined priority and
associativity, and the ability to import operators (or define how and
when operators become available compared to their definition)

That's a huge amount of complexity with little to gain.

From steve at pearwood.info  Sun Sep 29 17:33:38 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 30 Sep 2013 01:33:38 +1000
Subject: [Python-Dev] Why not support user defined operator overloading ?
In-Reply-To: <BLU403-EAS393D1EF893F5F5DC86003229D2B0@phx.gbl>
References: <BLU403-EAS393D1EF893F5F5DC86003229D2B0@phx.gbl>
Message-ID: <20130929153338.GB7989@ando>

On Sun, Sep 29, 2013 at 08:51:37PM +0800, ??? wrote:
> Hello:
> 	As far as I know, there is not a language support user defined operator overloading.
> Python3 can overloading belowed operators.
[...]
> (Do I miss something ?)

Yes, many.

http://docs.python.org/3/reference/datamodel.html#special-method-names
http://docs.python.org/3/reference/datamodel.html#emulating-numeric-types


> If we can overloading these operators, why we can't overloading other operators?
> (like .* often used in matrix, U in set operation)

http://www.python.org/dev/peps/pep-0211/
http://www.python.org/dev/peps/pep-0225/

There is a good argument for being able to overload element-wise and 
object-wise operators. However, I don't think it is good to be able to 
create arbitrary new operators, e.g. a X b for an X operator. We 
already have syntax for arbitrary functions, and they aren't limited to 
binary operators: X(a, b, c, d, ...) works better than a X b for most 
uses.




-- 
Steven

From guido at python.org  Sun Sep 29 18:54:39 2013
From: guido at python.org (Guido van Rossum)
Date: Sun, 29 Sep 2013 09:54:39 -0700
Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib,
 marked provisional, named asyncio
In-Reply-To: <CADiSq7ea-4YWBwRS_6v1m6fC0u1yTCu081ofeCuCUQQ4iTMX=w@mail.gmail.com>
References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com>
 <CADiSq7c22aKMqW_u1GfLjA=uFuGR3e83R-ynVzpbG_nH9Am+2w@mail.gmail.com>
 <CAP7+vJJfMP_1DFDc8dSGJo=JiE2hPGK_2AzrWUZUpxdiJ=qRPg@mail.gmail.com>
 <CADiSq7ea-4YWBwRS_6v1m6fC0u1yTCu081ofeCuCUQQ4iTMX=w@mail.gmail.com>
Message-ID: <CAP7+vJKZrFj9BoYQRPCiNJwtTPiFWS-tzmcBHGyEd8vneL0hrw@mail.gmail.com>

So, with the naming settled (asyncio it is), and lots of other things still
to do, I need a BDFL for PEP 3156. Any volunteers? If no-one volunteered
I'll have to accept my own PEP at some point, but I don't really *want* to
do that.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130929/d6414ece/attachment.html>

From stephen at xemacs.org  Sun Sep 29 19:03:57 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Mon, 30 Sep 2013 02:03:57 +0900
Subject: [Python-Dev]  Why not support user defined operator overloading ?
In-Reply-To: <BLU403-EAS393D1EF893F5F5DC86003229D2B0@phx.gbl>
References: <BLU403-EAS393D1EF893F5F5DC86003229D2B0@phx.gbl>
Message-ID: <87vc1jzhj6.fsf@uwakimon.sk.tsukuba.ac.jp>

??? writes:

 > If we can overloading these operators, why we can't overloading
 > other operators?  (like .* often used in matrix, U in set
 > operation)

AIUI, it's considered unpythonic.  Operators are considered to be part
of the *syntax* of Python, unlike Haskell, where infix syntax can be
used for any function, and operators can be called via function
syntax.  (The latter is true in Python as well, but it is very bad
form to do that, sufficiently verbose that it is hardly tempting, and
it can easily fail because Python operators are polymorphic, but the
implementations via class methods are not.)

I'm sure it also simplifies parsing.


From solipsis at pitrou.net  Sun Sep 29 19:16:53 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 29 Sep 2013 19:16:53 +0200
Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib,
 marked provisional, named asyncio
References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com>
 <CADiSq7c22aKMqW_u1GfLjA=uFuGR3e83R-ynVzpbG_nH9Am+2w@mail.gmail.com>
 <CAP7+vJJfMP_1DFDc8dSGJo=JiE2hPGK_2AzrWUZUpxdiJ=qRPg@mail.gmail.com>
 <CADiSq7ea-4YWBwRS_6v1m6fC0u1yTCu081ofeCuCUQQ4iTMX=w@mail.gmail.com>
 <CAP7+vJKZrFj9BoYQRPCiNJwtTPiFWS-tzmcBHGyEd8vneL0hrw@mail.gmail.com>
Message-ID: <20130929191653.773d3d55@fsol>

On Sun, 29 Sep 2013 09:54:39 -0700
Guido van Rossum <guido at python.org> wrote:
> So, with the naming settled (asyncio it is), and lots of other things still
> to do, I need a BDFL for PEP 3156. Any volunteers? If no-one volunteered
> I'll have to accept my own PEP at some point, but I don't really *want* to
> do that.

Well, if you don't mind my slight pro-callback tropism, I'm definitely
volunteering :-)

Regards

Antoine.



From barry at python.org  Sun Sep 29 20:11:01 2013
From: barry at python.org (Barry Warsaw)
Date: Sun, 29 Sep 2013 14:11:01 -0400
Subject: [Python-Dev] 2.6.9rc1
Message-ID: <20130929141101.6d48bdae@anarchist>

Just a quick reminder that I intend to cut 2.6.9rc1 tomorrow.

Here are the open issues still on my list:

- 16040 - nntplib: unlimited readline() from connection

  This one is waiting for a patch, but TBH if it doesn't make it into 2.6.9 I
  won't care too much.  I also don't mind reviewing and applying a patch
  between 2.6.9rc1 and 2.6.9 final.

- 16041 - poplib: unlimited readline() from connection

  This one has a patch for 2.7 which does not apply cleanly to 2.6.  I'm of a
  similar mind as with #16040 - if we can get a clean patch after 2.6.9rc1,
  then fine, but otherwise I'm okay with this one not making it into 2.6.9
  final.

All other related issues have been fixed in the 2.6 branch.

If anyone knows of any other issues that should be on the 2.6.9 radar, please
let me know asap.  Remember, 2.6.9 final is currently scheduled for October
28, and it will be the last official 2.6 release of the series.  After this,
we are retiring the 2.6 branch.  Speak now or forever hold your peace.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130929/83d998e1/attachment.sig>

From ned at nedbatchelder.com  Sun Sep 29 21:35:27 2013
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Sun, 29 Sep 2013 15:35:27 -0400
Subject: [Python-Dev] Why not support user defined operator overloading ?
In-Reply-To: <F53492D8-A90B-4449-9E83-25340AA63AEA@masklinn.net>
References: <BLU403-EAS393D1EF893F5F5DC86003229D2B0@phx.gbl>
 <F53492D8-A90B-4449-9E83-25340AA63AEA@masklinn.net>
Message-ID: <524880FF.9010500@nedbatchelder.com>

On 9/29/13 11:06 AM, Xavier Morel wrote:
> This is more of a python-ideas subject.
>
> And one of the reasons likely is that it would require significantly
> reworking the grammar to handle a kind of user-defined opname (similar
> to name, but for operator tokens), with user-defined priority and
> associativity, and the ability to import operators (or define how and
> when operators become available compared to their definition)
>
> That's a huge amount of complexity with little to gain.

It isn't just a matter of a more complex parser: where would the parser 
get the information about these new operators?  The obvious first answer 
is that they would be defined as part of classes, but that means the 
operator definition is in an imported module some place.  The importing 
module couldn't even be parsed until the class was imported.  This is a 
completely different compilation and execution model than Python has now.

Now you can compile a Python module without any access to the modules it 
will eventually import.  Importing is a run-time operation.  If you want 
user-defined classes to be able to define new syntax, then importing has 
to happen at compile time.  That's a completely different language.

--Ned.

From jyrki at dywypi.org  Sun Sep 29 21:59:12 2013
From: jyrki at dywypi.org (Jyrki Pulliainen)
Date: Sun, 29 Sep 2013 21:59:12 +0200
Subject: [Python-Dev] 2.6.9rc1
In-Reply-To: <20130929141101.6d48bdae@anarchist>
References: <20130929141101.6d48bdae@anarchist>
Message-ID: <CAM0Q7WAVWe83B2T-9iUci7827R4pa+2hdLUVcOYOYgaqJR10aQ@mail.gmail.com>

On Sun, Sep 29, 2013 at 8:11 PM, Barry Warsaw <barry at python.org> wrote:

> Just a quick reminder that I intend to cut 2.6.9rc1 tomorrow.
>
> Here are the open issues still on my list:
>
> - 16040 - nntplib: unlimited readline() from connection
>
>   This one is waiting for a patch, but TBH if it doesn't make it into
> 2.6.9 I
>   won't care too much.  I also don't mind reviewing and applying a patch
>   between 2.6.9rc1 and 2.6.9 final.
>
> - 16041 - poplib: unlimited readline() from connection
>
>   This one has a patch for 2.7 which does not apply cleanly to 2.6.  I'm
> of a
>   similar mind as with #16040 - if we can get a clean patch after 2.6.9rc1,
>   then fine, but otherwise I'm okay with this one not making it into 2.6.9
>   final.
>
>
I added a patch for issue 16041 in 2.6. It's not too pretty, as I added
another test case class with just a slightly different server
implementation. This is mainly due to how the tests have been reworked for
2.7 and I didn't feel too comfortable of copying those tests back to 2.6.

- Jyrki


> All other related issues have been fixed in the 2.6 branch.
>
> If anyone knows of any other issues that should be on the 2.6.9 radar,
> please
> let me know asap.  Remember, 2.6.9 final is currently scheduled for October
> 28, and it will be the last official 2.6 release of the series.  After
> this,
> we are retiring the 2.6 branch.  Speak now or forever hold your peace.
>
> Cheers,
> -Barry
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/jyrki%40dywypi.org
>
>


-- 
- Jyrki
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130929/cc130736/attachment.html>

From steve at pearwood.info  Mon Sep 30 01:31:32 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 30 Sep 2013 09:31:32 +1000
Subject: [Python-Dev] Why not support user defined operator overloading ?
In-Reply-To: <524880FF.9010500@nedbatchelder.com>
References: <BLU403-EAS393D1EF893F5F5DC86003229D2B0@phx.gbl>
 <F53492D8-A90B-4449-9E83-25340AA63AEA@masklinn.net>
 <524880FF.9010500@nedbatchelder.com>
Message-ID: <20130929233132.GC7989@ando>

On Sun, Sep 29, 2013 at 03:35:27PM -0400, Ned Batchelder wrote:

> It isn't just a matter of a more complex parser: where would the parser 
> get the information about these new operators?  The obvious first answer 
> is that they would be defined as part of classes, but that means the 
> operator definition is in an imported module some place.  The importing 
> module couldn't even be parsed until the class was imported.  This is a 
> completely different compilation and execution model than Python has now.

The obvious place to define the operator's action is a class, but there 
is no need to define new syntax at runtime. The parser doesn't need to 
know the operator's action until runtime, no different from existing 
operators.

For avoidance of doubt, I'm not suggesting this as a concrete proposal, 
just as a hypothetical. But if somebody wants to take it seriously 
and write up a PEP, here are two possibilities to get you started:

* Unicode characters. It's 2013, and anyone still using an editor which 
only handles ASCII shouldn't be :-) There are dozens of non-alphanumeric 
characters in Unicode that are suitable as operators. Unless I've 
missed one, all the existing operators have category 'Sm', 'Sk', 
'Po', or 'Pd', so Python might parse any character in those categories 
as a potential operator. E.g. all of these are in category Sm:

? ? ? ? ? ? ? ? ?

* ASCII digraphs. A more conservative approach would be to define custom 
operators as two characters, not one. Say, @X might be parsed as 
operator X.

In either case, having parsed "spam @X eggs" or "spam ? eggs", the 
interpreter could do this:

call spam.__op__('X', eggs)  # or '?'
if that doesn't succeed, call eggs.__rop__('X', spam)
raise OperatorError

none of which need happen until runtime, same as existing operators.

If you want to also support unary operators (prefix or postfix) then the 
parsing rules presumably will become more complicated, but the basic 
principle remains: the parser needs a pre-defined "template" to 
recognise what looks like an operator, and that can be baked into the 
language definition. The actual meaning of the operator can be 
determined at run time.

Make no mistake, this is still a big change to Python's semantics. But 
it doesn't require Python to redefine syntax on the fly.


-- 
Steven

From larry at hastings.org  Mon Sep 30 01:51:58 2013
From: larry at hastings.org (Larry Hastings)
Date: Mon, 30 Sep 2013 00:51:58 +0100
Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib,
 marked provisional, named asyncio
In-Reply-To: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com>
References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com>
Message-ID: <5248BD1E.1080003@hastings.org>

On 09/27/2013 11:33 PM, Guido van Rossum wrote:
> I've been looking at my progress with Tulip and the 3.4 release 
> schedule (PEP 429) and it looks like I will have to do some kind of 
> sprint to get it into the release in time for beta 1, which is planned 
> for Nov 24. Ideally I'd get it into alpha 4, which is scheduled for 
> Oct 20 -- that's in about three weeks, and probably too tight.
>
> Even Nov 24 is aggressive, because the PEP (PEP 3156) hasn't even been 
> discussed formally, let alone accepted! Fortunately I'm pretty happy 
> with most of the APIs defined in the PEP -- there are some holes but I 
> expect no big changes at this point.

My guess is, a lot of people would be disappointed if Tulip missed 3.4.  
I suspect the community would rather we slip the beta a little if it 
meant it the difference between Tulip and no Tulip.  I just hope you 
make it easy on me and beat the deadline ;-)

Cheers,


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130930/45defbb1/attachment.html>

From larry at hastings.org  Mon Sep 30 02:04:45 2013
From: larry at hastings.org (Larry Hastings)
Date: Mon, 30 Sep 2013 01:04:45 +0100
Subject: [Python-Dev] [RELEASED] Python 3.4.0a3
Message-ID: <5248C01D.7060701@hastings.org>

On behalf of the Python development team, I'm pleased to announce the
third alpha release of Python 3.4.

This is a preview release, and its use is not recommended for
production settings.

Python 3.4 includes a range of improvements of the 3.x series, including
hundreds of small improvements and bug fixes.  Major new features and
changes in the 3.4 release series so far include:

* PEP 435, a standardized "enum" module
* PEP 442, improved semantics for object finalization
* PEP 443, adding single-dispatch generic functions to the standard library
* PEP 445, a new C API for implementing custom memory allocators
* PEP 446, changing file descriptors to not be inherited by default
            in subprocesses


To download Python 3.4.0a3 visit:

     http://www.python.org/download/releases/3.4.0/


Please consider trying Python 3.4.0a3 with your code and reporting any
issues you notice to:

      http://bugs.python.org/


Enjoy!

--
Larry Hastings, Release Manager
larry at hastings.org
(on behalf of the entire python-dev team and 3.4's contributors)

From zhangpeipei812 at outlook.com  Mon Sep 30 02:23:12 2013
From: zhangpeipei812 at outlook.com (=?UTF-8?B?5byg5L2p5L2p?=)
Date: Mon, 30 Sep 2013 08:23:12 +0800
Subject: [Python-Dev] Why not support user defined operator overloading ?
Message-ID: <BLU406-EAS140395B430135783E5714B69D140@phx.gbl>

Hello:
	I agree with Steven D'Aprano.
Here is an example:

class A(object):
	def __init__(self, value):
		self.value = value
	def add(self, other):
		return self.value + other.value
	__magic_method__ = {'+':add}

a1 = A(1)
a2 = A(2)

We only need a macro expand
a1 + a2
to
a1.__magic__method__['+'](a, b)

the later can be execute on Python.

Regards
peipei

From eric at trueblade.com  Mon Sep 30 02:40:19 2013
From: eric at trueblade.com (Eric V. Smith)
Date: Sun, 29 Sep 2013 20:40:19 -0400
Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib,
 marked provisional, named asyncio
In-Reply-To: <CAP1=2W7A2Q3t7Bk7iDZ9ZPprFdi+yr152EgVV=ACaTZkaBbUmw@mail.gmail.com>
References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com>
 <CAP1=2W7A2Q3t7Bk7iDZ9ZPprFdi+yr152EgVV=ACaTZkaBbUmw@mail.gmail.com>
Message-ID: <5248C873.6060701@trueblade.com>

On 9/27/2013 9:14 PM, Brett Cannon wrote:

> I don't see any issue with redirecting the discussion. python-tulip@ is
> acting like a SIG for the module, so no real precedent beyond it not
> being hosted as a mail.python.org <http://mail.python.org> list. 

I'm sure I'm in the minority, but I'd like the discussion to take place
on a python.org mailing list. I don't want to log in to a Google
property, and I don't trust them with the mailing list archives.

I know my voice counts less than active Tulip discussion participants,
but now at least I feel better for having said something.

-- 
Eric.

From greg.ewing at canterbury.ac.nz  Mon Sep 30 02:53:08 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 30 Sep 2013 13:53:08 +1300
Subject: [Python-Dev] Why not support user defined operator overloading ?
In-Reply-To: <20130929233132.GC7989@ando>
References: <BLU403-EAS393D1EF893F5F5DC86003229D2B0@phx.gbl>
 <F53492D8-A90B-4449-9E83-25340AA63AEA@masklinn.net>
 <524880FF.9010500@nedbatchelder.com> <20130929233132.GC7989@ando>
Message-ID: <5248CB74.6000002@canterbury.ac.nz>

Steven D'Aprano wrote:
> there 
> is no need to define new syntax at runtime. The parser doesn't need to 
> know the operator's action until runtime

It does need to know the operator's precedence and
associativity, though, which means either declaring
it somewhere, or having some kind of fixed rule.

-- 
Greg

From guido at python.org  Mon Sep 30 02:55:10 2013
From: guido at python.org (Guido van Rossum)
Date: Sun, 29 Sep 2013 17:55:10 -0700
Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib,
 marked provisional, named asyncio
In-Reply-To: <5248C873.6060701@trueblade.com>
References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com>
 <CAP1=2W7A2Q3t7Bk7iDZ9ZPprFdi+yr152EgVV=ACaTZkaBbUmw@mail.gmail.com>
 <5248C873.6060701@trueblade.com>
Message-ID: <CAP7+vJ+5ij-w0KAcF+fkFnwgrWcRTKUgkSMoamT-GWHt7o9b1w@mail.gmail.com>

On Sun, Sep 29, 2013 at 5:40 PM, Eric V. Smith <eric at trueblade.com> wrote:

> On 9/27/2013 9:14 PM, Brett Cannon wrote:
>
> > I don't see any issue with redirecting the discussion. python-tulip@ is
> > acting like a SIG for the module, so no real precedent beyond it not
> > being hosted as a mail.python.org <http://mail.python.org> list.
>
> I'm sure I'm in the minority, but I'd like the discussion to take place
> on a python.org mailing list. I don't want to log in to a Google
> property, and I don't trust them with the mailing list archives.
>
> I know my voice counts less than active Tulip discussion participants,
> but now at least I feel better for having said something.
>

I wish you'd said something a looong time ago when it would have been easy
to move the list. Even if we moved it now we'd have split archives. Also,
I'm not sure where the paranoia comes from. FWIW I'm less worried about
Google reading my personal email than about the python.org webmasters
reading it.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130929/1439158c/attachment.html>

From guido at python.org  Mon Sep 30 02:57:25 2013
From: guido at python.org (Guido van Rossum)
Date: Sun, 29 Sep 2013 17:57:25 -0700
Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib,
 marked provisional, named asyncio
In-Reply-To: <20130929191653.773d3d55@fsol>
References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com>
 <CADiSq7c22aKMqW_u1GfLjA=uFuGR3e83R-ynVzpbG_nH9Am+2w@mail.gmail.com>
 <CAP7+vJJfMP_1DFDc8dSGJo=JiE2hPGK_2AzrWUZUpxdiJ=qRPg@mail.gmail.com>
 <CADiSq7ea-4YWBwRS_6v1m6fC0u1yTCu081ofeCuCUQQ4iTMX=w@mail.gmail.com>
 <CAP7+vJKZrFj9BoYQRPCiNJwtTPiFWS-tzmcBHGyEd8vneL0hrw@mail.gmail.com>
 <20130929191653.773d3d55@fsol>
Message-ID: <CAP7+vJJEJV+6iNBz36B3Ufg4X8cerJ5ZXV=q_ufu5YuWwCk42Q@mail.gmail.com>

On Sun, Sep 29, 2013 at 10:16 AM, Antoine Pitrou <solipsis at pitrou.net>wrote:

> On Sun, 29 Sep 2013 09:54:39 -0700 Guido van Rossum <guido at python.org>
> wrote:
> > So, with the naming settled (asyncio it is), and lots of other things
> still
> > to do, I need a BDFL for PEP 3156. Any volunteers? If no-one volunteered
> > I'll have to accept my own PEP at some point, but I don't really *want*
> to
> > do that.
>
> Well, if you don't mind my slight pro-callback tropism, I'm definitely
> volunteering :-)
>

I don't mind it, and if there's not too much objection from the community
you've got the job!

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130929/c8f54425/attachment.html>

From rovitotv at gmail.com  Mon Sep 30 02:39:52 2013
From: rovitotv at gmail.com (Todd V. Rovito)
Date: Sun, 29 Sep 2013 20:39:52 -0400
Subject: [Python-Dev] Completed: Google Summer of Code 2013 IDLE Unit
	Testing and Improvements
Message-ID: <E51BE7FC-5555-4895-9472-4903B66F944B@gmail.com>

In April 2013 we challenged students to not spend their summer idle around the swimming pool but rather spend their time on IDLE and make a difference.  IDLE is Python?s Integrated Development Environment (IDE) that is shipped with each Python release (http://en.wikipedia.org/wiki/IDLE_(Python)).  Since IDLE ships with Python it is often the first IDE a new Python programmer uses.  The Python development community wants to make IDLE an awesome experience especially for people that are learning Python.   Despite the fact that IDLE originally shipped in 1998 it had no unit test cases, making it difficult to confidently update the code and ship new versions.  For this Google Summer of Code project our students Jayakrishnan Rajagopalasarma (http://123works.blogspot.com/search/label/GSoC%202013) and Phil Webster (http://weblog.philwebster.net/tag/gsoc/) had to not only create the unit tests but assist with design of the unit test framework (http://bugs.python.org/issue15392).   By the end of the summer of 2013 Phil and Jayakrishnan have worked on five patches that have been committed to the CPython open source project.  In addition the team has started work on an additional five issues (at bugs.python.org) but have not completed them yet.  Jayakrishnan explored an idea to add automatic PEP8 checking to IDLE http://bugs.python.org/issue18704.  Our hope is that both JayKrish and Phil continue to contribute to Python even now that GSoC is complete. 

 The team wishes to thank Google and the Python Software Foundation for this opportunity to contribute to Python.  Thanks to all the Python Core Developers that helped review and commit issues for the students.  Special thanks goes to Terry Reedy who provided outstanding feedback on issues and performed all of the student?s commits.  Finally thanks to Terri Oda for being a great GSoC administrator for the Python Software Foundation and helping by assigning an awesome co-mentor J.H. Hawley.


 

The complete list of closed issues is below:

http://bugs.python.org/issue18279

http://bugs.python.org/issue18189

http://bugs.python.org/issue18425

http://bugs.python.org/issue18226

http://bugs.python.org/issue18489

 

The complete list of started issues is below:

http://bugs.python.org/issue18292

http://bugs.python.org/issue18910

http://bugs.python.org/issue18409

http://bugs.python.org/issue18410

http://bugs.python.org/issue18592

  

 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130929/4baf4560/attachment.html>

From zhangpeipei812 at outlook.com  Mon Sep 30 03:02:47 2013
From: zhangpeipei812 at outlook.com (=?UTF-8?B?5byg5L2p5L2p?=)
Date: Mon, 30 Sep 2013 09:02:47 +0800
Subject: [Python-Dev] Why not support user defined operator overloading ?
Message-ID: <BLU404-EAS117E3E2585DD92DEE3E0FBD9D140@phx.gbl>

On 2013/9/30 8:53 Greg Ewing wrote:
> It does need to know the operator's precedence and
>associativity, though, which means either declaring
>it somewhere, or having some kind of fixed rule

I suggest all user defined operator are at lowest priority.

Regards
peipei

From senthil at uthcode.com  Mon Sep 30 04:00:19 2013
From: senthil at uthcode.com (Senthil Kumaran)
Date: Sun, 29 Sep 2013 19:00:19 -0700
Subject: [Python-Dev] [Python-checkins] cpython (merge 3.3 -> default):
 Fix http.server's request handling case on trailing '/'.
In-Reply-To: <523325E1.70608@trueblade.com>
References: <3cbpGS1zZSz7Lk4@mail.python.org>
	<523325E1.70608@trueblade.com>
Message-ID: <CAPOVWOSBa_OemAhOPtimi-j-h_dneshi+JpqMRvVdVr=RnoHLA@mail.gmail.com>

On Fri, Sep 13, 2013 at 7:49 AM, Eric V. Smith <eric at trueblade.com> wrote:
>> Patch contributed by Vajrasky Kok. Addresses Issue #17324
>
>> +        trailing_slash = True if path.rstrip().endswith('/') else False
>
> Wouldn't this be better just as:
>           trailing_slash = path.rstrip().endswith('/')

I noticed this email late. Corrected it now.

Thanks,
Senthil

From eliben at gmail.com  Mon Sep 30 04:04:59 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Sun, 29 Sep 2013 19:04:59 -0700
Subject: [Python-Dev] Why not support user defined operator overloading ?
In-Reply-To: <BLU404-EAS117E3E2585DD92DEE3E0FBD9D140@phx.gbl>
References: <BLU404-EAS117E3E2585DD92DEE3E0FBD9D140@phx.gbl>
Message-ID: <CAF-Rda9c8qz+doPvY6DL4hUu8=iW3XQjb71QFE-NOczGFH+1ag@mail.gmail.com>

Can you guys please move this discussion to python-ideas? python-dev is the
wrong mailing list.

Eli


On Sun, Sep 29, 2013 at 6:02 PM, ??? <zhangpeipei812 at outlook.com> wrote:

> On 2013/9/30 8:53 Greg Ewing wrote:
> > It does need to know the operator's precedence and
> >associativity, though, which means either declaring
> >it somewhere, or having some kind of fixed rule
>
> I suggest all user defined operator are at lowest priority.
>
> Regards
> peipei
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/eliben%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130929/81694022/attachment.html>

From senthil at uthcode.com  Mon Sep 30 04:08:55 2013
From: senthil at uthcode.com (Senthil Kumaran)
Date: Sun, 29 Sep 2013 19:08:55 -0700
Subject: [Python-Dev] cpython (merge 3.3 -> default): merge from 3.3
In-Reply-To: <20130912081423.4607ea82@fsol>
References: <3cb8QF2vy7z7Ljh@mail.python.org>
	<20130912081423.4607ea82@fsol>
Message-ID: <CAPOVWOTH_oWiyGiTJQr+KHJYi1GyH9Viux7d3hVXmjSgY5P6Nw@mail.gmail.com>

On Wed, Sep 11, 2013 at 11:14 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Thu, 12 Sep 2013 07:57:25 +0200 (CEST)
> senthil.kumaran <python-checkins at python.org> wrote:
>>
>> +<<<<<<< local
>>          Optional argument random is a 0-argument function returning a
>>          random float in [0.0, 1.0); if it is the default None, the
>>          standard random.random will be used.
>> +=======
>> +        Optional arg random is a 0-argument function returning a random
>> +        float in [0.0, 1.0); by default, the standard random.random.
>> +
>> +        Do not supply the 'int' argument.
>> +>>>>>>> other
>
> Can you fix this?

Had done it earlier. (http://hg.python.org/cpython/rev/1398dfb59fd9 )
I missed responding back.  Sorry for the trouble.

Thanks,
Senthil

From ned at nedbatchelder.com  Mon Sep 30 04:09:34 2013
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Sun, 29 Sep 2013 22:09:34 -0400
Subject: [Python-Dev] Why not support user defined operator overloading ?
In-Reply-To: <BLU404-EAS117E3E2585DD92DEE3E0FBD9D140@phx.gbl>
References: <BLU404-EAS117E3E2585DD92DEE3E0FBD9D140@phx.gbl>
Message-ID: <5248DD5E.8030109@nedbatchelder.com>

On 9/29/13 9:02 PM, ??? wrote:
> On 2013/9/30 8:53 Greg Ewing wrote:
>> It does need to know the operator's precedence and
>> associativity, though, which means either declaring
>> it somewhere, or having some kind of fixed rule
> I suggest all user defined operator are at lowest priority.

Peipei, this is an interesting discussion, but doesn't belong on 
Python-Dev.  The Python-Ideas mailing list is for exactly this kind of 
topic.

--Ned.

> Regards
> peipei
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/ned%40nedbatchelder.com


From hodgestar+pythondev at gmail.com  Mon Sep 30 09:05:11 2013
From: hodgestar+pythondev at gmail.com (Simon Cross)
Date: Mon, 30 Sep 2013 09:05:11 +0200
Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib,
 marked provisional, named asyncio
In-Reply-To: <5248BD1E.1080003@hastings.org>
References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com>
 <5248BD1E.1080003@hastings.org>
Message-ID: <CAD5NRCEOsyucPzyA4u4whBiu8Zt+2HwkWY6V2Nmubp8r7AFwXg@mail.gmail.com>

On Mon, Sep 30, 2013 at 1:51 AM, Larry Hastings <larry at hastings.org> wrote:
> My guess is, a lot of people would be disappointed if Tulip missed 3.4.  I
> suspect the community would rather we slip the beta a little if it meant it
> the difference between Tulip and no Tulip.

+1

From martin at v.loewis.de  Mon Sep 30 11:01:45 2013
From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 30 Sep 2013 11:01:45 +0200
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
Message-ID: <52493DF9.7010705@v.loewis.de>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Am 25.09.13 23:33, schrieb Donald Stufft:
> An early draft of this did not have the backport to 2.7 and when I
> showed *that* version around to get feedback people were less
> enthusiastic about it and generally viewed it as "nice but
> worthless to me for N years".

I'm leaning towards the people that oppose addition of this feature to
older Python releases. In particular, the objection that the new PEP is
worthless for a long time to come, is in itself not really relevant:

It is always the case that features proposed by a PEP reach users only
years after they have been implemented.

> What users want isn't rationale in and of itself but I think it's
> an important data point, especially given how long 2.7.LASTEVER is
> going to be relevant to end users.

Well, I really really don't like this idea. 2.7 should not get new
features; users who want new features need to switch to 3.x.

Regards,
Martin


-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.18 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlJJPfkACgkQavBT8H2dyNL9SQCfceoTqIeCiwq0GMdaWeUSn/nV
6E4AmwRfmjFcSXjdmGLQbewVqxxYwRit
=tLof
-----END PGP SIGNATURE-----

From donald at stufft.io  Mon Sep 30 13:18:29 2013
From: donald at stufft.io (Donald Stufft)
Date: Mon, 30 Sep 2013 07:18:29 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <52493DF9.7010705@v.loewis.de>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <52493DF9.7010705@v.loewis.de>
Message-ID: <5B4A7F23-C9E7-464E-A7A2-05CE3084CC67@stufft.io>


On Sep 30, 2013, at 5:01 AM, "Martin v. L?wis" <martin at v.loewis.de> wrote:

> Signed PGP part
> Am 25.09.13 23:33, schrieb Donald Stufft:
> > An early draft of this did not have the backport to 2.7 and when I
> > showed *that* version around to get feedback people were less
> > enthusiastic about it and generally viewed it as "nice but
> > worthless to me for N years".
> 
> I'm leaning towards the people that oppose addition of this feature to
> older Python releases. In particular, the objection that the new PEP is
> worthless for a long time to come, is in itself not really relevant:
> 
> It is always the case that features proposed by a PEP reach users only
> years after they have been implemented.

Well the point we tried to get across in the PEP is that a normal feature
you can typically just install a backport from PyPI to gain it early. This
isin't so much driven by "well it'd be nice for the stdlib to have X", but
"well this is a real and valid pain point that causes pain for a lot of users".
The 2.7 backport was driven by just how painful this particular pain point
can be. I've personally had feedback that in tutorials at like PyCon or
meet ups that easily 1/3 of the time can be spent in getting users setup
with Python, setuptools, and pip.

Obviously you're the delegate for this PEP and it's your final decision and
either way If the PEP is accepted I'll do the implementation for 3.4.

> 
> > What users want isn't rationale in and of itself but I think it's
> > an important data point, especially given how long 2.7.LASTEVER is
> > going to be relevant to end users.
> 
> Well, I really really don't like this idea. 2.7 should not get new
> features; users who want new features need to switch to 3.x.
> 
> Regards,
> Martin
> 
> 


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130930/e1243024/attachment.sig>

From ncoghlan at gmail.com  Mon Sep 30 13:32:13 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 30 Sep 2013 21:32:13 +1000
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <52493DF9.7010705@v.loewis.de>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <52493DF9.7010705@v.loewis.de>
Message-ID: <CADiSq7czM+ib2GR+kSvkuy5_TUWvV75kNLoDE0GZsrjEOQUkfg@mail.gmail.com>

On 30 Sep 2013 19:03, Martin v. L?wis <martin at v.loewis.de> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Am 25.09.13 23:33, schrieb Donald Stufft:
> > An early draft of this did not have the backport to 2.7 and when I
> > showed *that* version around to get feedback people were less
> > enthusiastic about it and generally viewed it as "nice but
> > worthless to me for N years".
>
> I'm leaning towards the people that oppose addition of this feature to
> older Python releases. In particular, the objection that the new PEP is
> worthless for a long time to come, is in itself not really relevant:
>
> It is always the case that features proposed by a PEP reach users only
> years after they have been implemented.
>
> > What users want isn't rationale in and of itself but I think it's
> > an important data point, especially given how long 2.7.LASTEVER is
> > going to be relevant to end users.
>
> Well, I really really don't like this idea. 2.7 should not get new
> features; users who want new features need to switch to 3.x.

Before you make your final decision on this front, I'd like to record in
the PEP my fallback plan for if the backporting portion of the PEP is
rejected while the rest is accepted. After all, you should know what it is
you're actually choosing between.

I just need to run the details of the fallback proposal past Donald before
publishing it under both our names.

Regards,
Nick.

>
> Regards,
> Martin
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.18 (Darwin)
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAlJJPfkACgkQavBT8H2dyNL9SQCfceoTqIeCiwq0GMdaWeUSn/nV
> 6E4AmwRfmjFcSXjdmGLQbewVqxxYwRit
> =tLof
> -----END PGP SIGNATURE-----
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130930/188b3c6c/attachment.html>

From martin at v.loewis.de  Mon Sep 30 13:50:51 2013
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 30 Sep 2013 13:50:51 +0200
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <5B4A7F23-C9E7-464E-A7A2-05CE3084CC67@stufft.io>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <52493DF9.7010705@v.loewis.de>
 <5B4A7F23-C9E7-464E-A7A2-05CE3084CC67@stufft.io>
Message-ID: <5249659B.1060305@v.loewis.de>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Am 30.09.13 13:18, schrieb Donald Stufft:
> Well the point we tried to get across in the PEP is that a normal
> feature you can typically just install a backport from PyPI to gain
> it early. This isin't so much driven by "well it'd be nice for the
> stdlib to have X", but "well this is a real and valid pain point
> that causes pain for a lot of users".

I see it exactly vice versa. A normal new language feature, you *cannot*
get early from PyPI, e.g. if it's a syntax extension (e.g. the with
statement). OTOH, *this* particular feature you can easily get from
PyPI - just download and run the PIP installer.

> The 2.7 backport was driven by just how painful this particular
> pain point can be. I've personally had feedback that in tutorials
> at like PyCon or meet ups that easily 1/3 of the time can be spent
> in getting users setup with Python, setuptools, and pip.

I'm sure there is something that can be done about it. What operating
system are people using who have difficulties to set this all up?

I see that pip doesn't have a Windows installer on PyPI. That looks
like the real culprit to me. I'd be willing to work with you to
provide one (that ideally bundles all dependencies).

> Obviously you're the delegate for this PEP and it's your final
> decision and either way If the PEP is accepted I'll do the
> implementation for 3.4.

The backporting is the main stumbling block. Otherwise, I'm in favor
of the PEP.

Regards,
Martin

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.18 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlJJZZsACgkQavBT8H2dyNIAWQCeNi3OcIXaNGmFuDfPcTEfnBjV
vTMAnAv7diLGOsJ2oNUqE7BHG1wIwMCP
=r+wP
-----END PGP SIGNATURE-----

From ncoghlan at gmail.com  Mon Sep 30 15:07:36 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 30 Sep 2013 23:07:36 +1000
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <5249659B.1060305@v.loewis.de>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <52493DF9.7010705@v.loewis.de>
 <5B4A7F23-C9E7-464E-A7A2-05CE3084CC67@stufft.io>
 <5249659B.1060305@v.loewis.de>
Message-ID: <CADiSq7dVsXJpxHDx5OipH-x34g9KwR2zZh8LZ_6svYz_tW3zHw@mail.gmail.com>

On 30 Sep 2013 21:52, Martin v. L?wis <martin at v.loewis.de> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Am 30.09.13 13:18, schrieb Donald Stufft:
> > Well the point we tried to get across in the PEP is that a normal
> > feature you can typically just install a backport from PyPI to gain
> > it early. This isin't so much driven by "well it'd be nice for the
> > stdlib to have X", but "well this is a real and valid pain point
> > that causes pain for a lot of users".
>
> I see it exactly vice versa. A normal new language feature, you *cannot*
> get early from PyPI, e.g. if it's a syntax extension (e.g. the with
> statement).

It's standard library improvements (like SSL hostname matching) that are
relevant to this comparison, rather than the kinds of changes that were
barred by the language moratorium.

> OTOH, *this* particular feature you can easily get from
> PyPI - just download and run the PIP installer.
>
> > The 2.7 backport was driven by just how painful this particular
> > pain point can be. I've personally had feedback that in tutorials
> > at like PyCon or meet ups that easily 1/3 of the time can be spent
> > in getting users setup with Python, setuptools, and pip.
>
> I'm sure there is something that can be done about it. What operating
> system are people using who have difficulties to set this all up?
>
> I see that pip doesn't have a Windows installer on PyPI. That looks
> like the real culprit to me. I'd be willing to work with you to
> provide one (that ideally bundles all dependencies).

We'll take you up on that offer. Windows is the main problem, since other
platforms offer wget/curl out of the box so bootstrapping is easy by
comparison.

>
> > Obviously you're the delegate for this PEP and it's your final
> > decision and either way If the PEP is accepted I'll do the
> > implementation for 3.4.
>
> The backporting is the main stumbling block. Otherwise, I'm in favor
> of the PEP.

My plan now is to split the PEP in two, so the 3.4 changes can be accepted
as non-controversial, including the offer of core dev assistance in
creating and maintaining a Windows installer for pip to better support
earlier versions. The backporting PEP will be deferred, for reconsideration
some time in the future after the initial PEP has been implemented.

Regards,
Nick.

>
> Regards,
> Martin
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.18 (Darwin)
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAlJJZZsACgkQavBT8H2dyNIAWQCeNi3OcIXaNGmFuDfPcTEfnBjV
> vTMAnAv7diLGOsJ2oNUqE7BHG1wIwMCP
> =r+wP
> -----END PGP SIGNATURE-----
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130930/f58b06f0/attachment.html>

From eliben at gmail.com  Mon Sep 30 15:09:48 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Mon, 30 Sep 2013 06:09:48 -0700
Subject: [Python-Dev] Semi-official read-only Github mirror of the CPython
	Mercurial repository
Message-ID: <CAF-Rda__36CpmDaVxLWo+iWhw5x9SqNFGjoQFXWM591PGNOzhg@mail.gmail.com>

Hi all,

https://github.com/python/cpython is now live as a semi-official, *read
only* Github mirror of the CPython Mercurial repository. Let me know if you
have any problems/concerns.

I still haven't decided how often to update it (considering either just N
times a day, or maybe use a Hg hook for batching). Suggestions are welcome.

The methodology I used to create it is via hg-fast-export. I also tried to
pack and gc the git repo as much as possible before the initial Github push
- it went down from almost ~2GB to ~200MB (so this is the size of a fresh
clone right now).

Eli

P.S. thanks Jesse for the keys to https://github.com/python
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130930/fd1dec8d/attachment.html>

From donald at stufft.io  Mon Sep 30 15:54:04 2013
From: donald at stufft.io (Donald Stufft)
Date: Mon, 30 Sep 2013 09:54:04 -0400
Subject: [Python-Dev] Semi-official read-only Github mirror of the
	CPython Mercurial repository
In-Reply-To: <CAF-Rda__36CpmDaVxLWo+iWhw5x9SqNFGjoQFXWM591PGNOzhg@mail.gmail.com>
References: <CAF-Rda__36CpmDaVxLWo+iWhw5x9SqNFGjoQFXWM591PGNOzhg@mail.gmail.com>
Message-ID: <145A4E43-27F1-46EF-91BE-383C352669FF@stufft.io>


On Sep 30, 2013, at 9:09 AM, Eli Bendersky <eliben at gmail.com> wrote:

> Hi all,
> 
> https://github.com/python/cpython is now live as a semi-official, *read only* Github mirror of the CPython Mercurial repository. Let me know if you have any problems/concerns.
> 
> I still haven't decided how often to update it (considering either just N times a day, or maybe use a Hg hook for batching). Suggestions are welcome.
> 
> The methodology I used to create it is via hg-fast-export. I also tried to pack and gc the git repo as much as possible before the initial Github push - it went down from almost ~2GB to ~200MB (so this is the size of a fresh clone right now).
> 
> Eli
> 
> P.S. thanks Jesse for the keys to https://github.com/python
> 
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

Awesome! I find Github way nicer for reading source than hg.python.org's web interface, any chance I could convince you to do this for the peps repo too? ;)

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130930/0d715fc5/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130930/0d715fc5/attachment.sig>

From eliben at gmail.com  Mon Sep 30 15:58:49 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Mon, 30 Sep 2013 06:58:49 -0700
Subject: [Python-Dev] Semi-official read-only Github mirror of the
 CPython Mercurial repository
In-Reply-To: <145A4E43-27F1-46EF-91BE-383C352669FF@stufft.io>
References: <CAF-Rda__36CpmDaVxLWo+iWhw5x9SqNFGjoQFXWM591PGNOzhg@mail.gmail.com>
 <145A4E43-27F1-46EF-91BE-383C352669FF@stufft.io>
Message-ID: <CAF-Rda-4hcFhHS_Wjmv4kBDzG2Pb+fU7SRH2RjX2WdhzE9_ubQ@mail.gmail.com>

>
> https://github.com/python/cpython is now live as a semi-official, *read
> only* Github mirror of the CPython Mercurial repository. Let me know if you
> have any problems/concerns.
>
> I still haven't decided how often to update it (considering either just N
> times a day, or maybe use a Hg hook for batching). Suggestions are welcome.
>
> The methodology I used to create it is via hg-fast-export. I also tried to
> pack and gc the git repo as much as possible before the initial Github push
> - it went down from almost ~2GB to ~200MB (so this is the size of a fresh
> clone right now).
>
> Eli
>
> P.S. thanks Jesse for the keys to https://github.com/python
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/donald%40stufft.io
>
>
> Awesome! I find Github way nicer for reading source than hg.python.org's
> web interface, any chance I could convince you to do this for the peps repo
> too? ;)
>

Yes, that shouldn't pose a problem. I'll let it hum for a couple of days
just to see everything is OK and then I'll add peps too.

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130930/cf9fe737/attachment.html>

From barry at python.org  Mon Sep 30 16:02:56 2013
From: barry at python.org (Barry Warsaw)
Date: Mon, 30 Sep 2013 10:02:56 -0400
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
 pronouncement?
In-Reply-To: <CADiSq7dVsXJpxHDx5OipH-x34g9KwR2zZh8LZ_6svYz_tW3zHw@mail.gmail.com>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <52493DF9.7010705@v.loewis.de>
 <5B4A7F23-C9E7-464E-A7A2-05CE3084CC67@stufft.io>
 <5249659B.1060305@v.loewis.de>
 <CADiSq7dVsXJpxHDx5OipH-x34g9KwR2zZh8LZ_6svYz_tW3zHw@mail.gmail.com>
Message-ID: <20130930100256.53949f86@limelight.wooz.org>

On Sep 30, 2013, at 11:07 PM, Nick Coghlan wrote:

>My plan now is to split the PEP in two, so the 3.4 changes can be accepted
>as non-controversial, including the offer of core dev assistance in
>creating and maintaining a Windows installer for pip to better support
>earlier versions. The backporting PEP will be deferred, for reconsideration
>some time in the future after the initial PEP has been implemented.

+1, since I think there's little disagreement about adding this to 3.4.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130930/22f1995d/attachment-0001.sig>

From skip at pobox.com  Mon Sep 30 16:08:12 2013
From: skip at pobox.com (Skip Montanaro)
Date: Mon, 30 Sep 2013 09:08:12 -0500
Subject: [Python-Dev] Semi-official read-only Github mirror of the
 CPython Mercurial repository
In-Reply-To: <CAF-Rda__36CpmDaVxLWo+iWhw5x9SqNFGjoQFXWM591PGNOzhg@mail.gmail.com>
References: <CAF-Rda__36CpmDaVxLWo+iWhw5x9SqNFGjoQFXWM591PGNOzhg@mail.gmail.com>
Message-ID: <CANc-5UwZW3LLoYyQbcnZWWe8u=ZAJyHGuwARf=8mTLf3pqQz+Q@mail.gmail.com>

> https://github.com/python/cpython is now live as a semi-official, *read
> only* Github mirror of the CPython Mercurial repository. Let me know if you
> have any problems/concerns.

Thanks for this, Eli. I use git regularly at work, so I'm getting much
more comfortable with it. Do you have a suggested workflow for people
who might want to use Git in preference to Hg, but still have write
access?

Skip

From skip at pobox.com  Mon Sep 30 16:35:24 2013
From: skip at pobox.com (Skip Montanaro)
Date: Mon, 30 Sep 2013 09:35:24 -0500
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <20130930100256.53949f86@limelight.wooz.org>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <52493DF9.7010705@v.loewis.de>
 <5B4A7F23-C9E7-464E-A7A2-05CE3084CC67@stufft.io>
 <5249659B.1060305@v.loewis.de>
 <CADiSq7dVsXJpxHDx5OipH-x34g9KwR2zZh8LZ_6svYz_tW3zHw@mail.gmail.com>
 <20130930100256.53949f86@limelight.wooz.org>
Message-ID: <CANc-5UwkZ4BDNJ554Mnhuz6tawGYL+KdjgiG0C12di7pgncPAQ@mail.gmail.com>

Splitting into two pieces also means you can implement it for 3.4
first and identify possible problems caused by preexisting pip
installs before deciding whether to add it to 2.7 and 3.3.

Skip

From ncoghlan at gmail.com  Mon Sep 30 16:47:11 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 1 Oct 2013 00:47:11 +1000
Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for
	pronouncement?
In-Reply-To: <CANc-5UwkZ4BDNJ554Mnhuz6tawGYL+KdjgiG0C12di7pgncPAQ@mail.gmail.com>
References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com>
 <20130925165022.3d67e290@anarchist>
 <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io>
 <52493DF9.7010705@v.loewis.de>
 <5B4A7F23-C9E7-464E-A7A2-05CE3084CC67@stufft.io>
 <5249659B.1060305@v.loewis.de>
 <CADiSq7dVsXJpxHDx5OipH-x34g9KwR2zZh8LZ_6svYz_tW3zHw@mail.gmail.com>
 <20130930100256.53949f86@limelight.wooz.org>
 <CANc-5UwkZ4BDNJ554Mnhuz6tawGYL+KdjgiG0C12di7pgncPAQ@mail.gmail.com>
Message-ID: <CADiSq7csmzk6pvW2U4y6BMh1BEU5i7r_v_15nZ8W4O=-8xHDqw@mail.gmail.com>

On 1 October 2013 00:35, Skip Montanaro <skip at pobox.com> wrote:
> Splitting into two pieces also means you can implement it for 3.4
> first and identify possible problems caused by preexisting pip
> installs before deciding whether to add it to 2.7 and 3.3.

One of the key reasons for using the bootstrap mechanism rather than
just bundling pip is so that the installers for maintenance releases
won't have any more trouble with pre-existing installs of pip than pip
itself does :)

But yes, there are several advantages to splitting the PEP:

- the Python 3.4 changes are non-controversial, so it makes sense to
get them officially accepted
- the Python 3.4 beta deadline is closer than the deadlines for the
3.3.3 and 2.7.6 maintenance releases
- it allows the full extent of the changes proposed for backporting to
be clear prior to the reconsideration of the separated PEP
- a Windows installer for pip could be made available well in advance
of the 3.3.3 and 2.7.6 maintenance releases
- it allows a chance to see if a separate Windows installer for pip
and documenting the bootstrapping instructions in the "Installing
Python Modules" guide for older releases is sufficient to get new
users over the "getting started with PyPI" barrier

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From eliben at gmail.com  Mon Sep 30 17:33:21 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Mon, 30 Sep 2013 08:33:21 -0700
Subject: [Python-Dev] Semi-official read-only Github mirror of the
 CPython Mercurial repository
In-Reply-To: <CANc-5UwZW3LLoYyQbcnZWWe8u=ZAJyHGuwARf=8mTLf3pqQz+Q@mail.gmail.com>
References: <CAF-Rda__36CpmDaVxLWo+iWhw5x9SqNFGjoQFXWM591PGNOzhg@mail.gmail.com>
 <CANc-5UwZW3LLoYyQbcnZWWe8u=ZAJyHGuwARf=8mTLf3pqQz+Q@mail.gmail.com>
Message-ID: <CAF-Rda8YjV8B2WeSg3Gk=Y459YNR40yzYU1ubyHo+uADc9jg7w@mail.gmail.com>

On Mon, Sep 30, 2013 at 7:08 AM, Skip Montanaro <skip at pobox.com> wrote:

> > https://github.com/python/cpython is now live as a semi-official, *read
> > only* Github mirror of the CPython Mercurial repository. Let me know if
> you
> > have any problems/concerns.
>
> Thanks for this, Eli. I use git regularly at work, so I'm getting much
> more comfortable with it. Do you have a suggested workflow for people
> who might want to use Git in preference to Hg, but still have write
> access?
>

Petri Lehtinen runs a clone at https://github.com/akheron/cpython which
uses more advanced tricks like hg-git and his own git-hg wrapper to allow
him to do this, AFAIK. If this path is important for you, contact Petri and
he can provide guidance on how to set it up.

For python/cpython I really wanted the simplest flow to reach a read-only
stage, because I lack the necessary git/hg-fu to set up and maintain
something more complex "semi-officially". I think the most common workflow
is to do small changes vs. Mercurial anyway. When I'm working on
longer-term patches I do them in my Git mirror (because I prefer Git
branching) and then apply & test patches onto a Mercurial R/W clone before
committing.

Thus, for most users I think the read-only mirror is preferable because
it's much easier to use and reason about. All writes go solely through
Mercurial, so it's clear where the official source is :)

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130930/e2c9204c/attachment.html>

From N.D.Efford at leeds.ac.uk  Mon Sep 30 12:39:40 2013
From: N.D.Efford at leeds.ac.uk (Nick Efford)
Date: Mon, 30 Sep 2013 11:39:40 +0100 (BST)
Subject: [Python-Dev] Python for new users
In-Reply-To: <mailman.2808.1380306720.18129.python-dev@python.org>
References: <mailman.2808.1380306720.18129.python-dev@python.org>
Message-ID: <alpine.LRH.2.03.1309301053220.26214@leeds.ac.uk>

In the 'PEP453 ready for pronouncement' thread, Donald said

>> Because reality is that new users are still likely to be using Python 2.7.
>> Python 3 is just now starting to be really usable, however there's a huge
>> corpus of existing tutorials, course work, books etc for Python 2.7. As
>> Python 3 becomes more usable that existing corpus of material will be
>> ported over to Python 3 but in the interim there is still a pretty large
>> hurdle for new users to get over.

And Guido replied

> Based on my day-to-day experience this is still very true. (And yes, I'm
> slowly turning the tide. But it will take a long time and I am committed to
> giving users the choice.)

Widely-used and linked web resources tend to persist for a very
long time, so we shouldn't use the prevalence of Python 2 resources
as a reason for excessive caution.  The key question is how much
good material is available based on Python 3 - and this has improved
significantly over the past couple of years.  The classic "How to
Think Like a Computer Scientist" has an excellent Python 3 version
available at http://interactivepython.org, for example.

Things are changing with print media, too.  Pragmatic Programmers are
about to publish the 2nd edition of "Practical Programming", based
on Python 3.  Most of the major academic publishers have released
Python 3 books in the last 12 months.  The tide is definitely
turning, perhaps has already turned.

Encouraging the continued use of 2.7 for existing programmers is
entirely justifiable, but for *newcomers* to programming I think it
is now much harder to justify.  A stronger case could have been made
a couple of years ago, when many important packages were not yet
available for Python 3, but things have changed.  Even big frameworks
like Django are now usable with Python 3.  If we aren't yet past the
point where package availability shouldn't be regarded as an adoption
barrier by beginners, we are surely very close.

I've been teaching Python as a first language to university students
for many years now, initially with Python 2 and for the last few
years with Python 3.  In my experience, they encounter fewer problems
with Python 3 (just as Guido intended, no doubt :)  The one stumbling
block in the past has been package availability for project work,
but I don't expect that to a problem this year.  All of the web and
GUI development work that they'll be doing with me, for example,
will be done entirely in Python 3.


Nick

From eric at trueblade.com  Mon Sep 30 19:38:07 2013
From: eric at trueblade.com (Eric V. Smith)
Date: Mon, 30 Sep 2013 13:38:07 -0400
Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib,
 marked provisional, named asyncio
In-Reply-To: <CAP7+vJ+5ij-w0KAcF+fkFnwgrWcRTKUgkSMoamT-GWHt7o9b1w@mail.gmail.com>
References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com>
 <CAP1=2W7A2Q3t7Bk7iDZ9ZPprFdi+yr152EgVV=ACaTZkaBbUmw@mail.gmail.com>
 <5248C873.6060701@trueblade.com>
 <CAP7+vJ+5ij-w0KAcF+fkFnwgrWcRTKUgkSMoamT-GWHt7o9b1w@mail.gmail.com>
Message-ID: <5249B6FF.7030805@trueblade.com>

On 09/29/2013 08:55 PM, Guido van Rossum wrote:
> On Sun, Sep 29, 2013 at 5:40 PM, Eric V. Smith <eric at trueblade.com
> <mailto:eric at trueblade.com>> wrote:
> 
>     On 9/27/2013 9:14 PM, Brett Cannon wrote:
> 
>     > I don't see any issue with redirecting the discussion.
>     python-tulip@ is
>     > acting like a SIG for the module, so no real precedent beyond it not
>     > being hosted as a mail.python.org <http://mail.python.org>
>     <http://mail.python.org> list.
> 
>     I'm sure I'm in the minority, but I'd like the discussion to take place
>     on a python.org <http://python.org> mailing list. I don't want to
>     log in to a Google
>     property, and I don't trust them with the mailing list archives.
> 
>     I know my voice counts less than active Tulip discussion participants,
>     but now at least I feel better for having said something.
> 
> 
> I wish you'd said something a looong time ago when it would have been
> easy to move the list. Even if we moved it now we'd have split archives.
> Also, I'm not sure where the paranoia comes from. FWIW I'm less worried
> about Google reading my personal email than about the python.org
> <http://python.org> webmasters reading it.

I wish I'd known the project would be such a success! I have a use for
tulip, but not much time to offer development help currently, so I'll
live with the outcome. I'd love to see it make 3.4.

Eric.



From solipsis at pitrou.net  Mon Sep 30 20:11:49 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 30 Sep 2013 20:11:49 +0200
Subject: [Python-Dev] cpython (2.7): Add fake buildbottouch target.
References: <3cpQhd3DJ5zPWd@mail.python.org>
Message-ID: <20130930201149.4a034831@fsol>

On Mon, 30 Sep 2013 16:18:57 +0200 (CEST)
martin.v.loewis <python-checkins at python.org> wrote:
> http://hg.python.org/cpython/rev/c1a294bbb4fa
> changeset:   85882:c1a294bbb4fa
> branch:      2.7
> parent:      85877:dd55d54b2a15
> user:        Martin v. L?wis <martin at v.loewis.de>
> date:        Mon Sep 30 16:18:31 2013 +0200
> summary:
>   Add fake buildbottouch target.

"make touch" does exist in 2.7, so was this new target needed?

Regards

Antoine.



From solipsis at pitrou.net  Mon Sep 30 20:16:44 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 30 Sep 2013 20:16:44 +0200
Subject: [Python-Dev] peps: document requirements
References: <3cpRnD5xP2z7Lkm@mail.python.org>
Message-ID: <20130930201644.4749fd69@fsol>

On Mon, 30 Sep 2013 17:08:00 +0200 (CEST)
christian.heimes <python-checkins at python.org> wrote:
> +
> +* It should return ``0`` for zero length input. (Note: This can be handled as
> +  special case, too.)

What is this required for? The main requirement is that equal stuff
hashes equal, but there's no reason for zero-length input to hash to
zero, AFAIK.

Regards

Antoine.



From christian at python.org  Mon Sep 30 21:31:10 2013
From: christian at python.org (Christian Heimes)
Date: Mon, 30 Sep 2013 21:31:10 +0200
Subject: [Python-Dev] peps: document requirements
In-Reply-To: <20130930201644.4749fd69@fsol>
References: <3cpRnD5xP2z7Lkm@mail.python.org> <20130930201644.4749fd69@fsol>
Message-ID: <5249D17E.80304@python.org>

Am 30.09.2013 20:16, schrieb Antoine Pitrou:
> On Mon, 30 Sep 2013 17:08:00 +0200 (CEST)
> christian.heimes <python-checkins at python.org> wrote:
>> +
>> +* It should return ``0`` for zero length input. (Note: This can be handled as
>> +  special case, too.)
> 
> What is this required for? The main requirement is that equal stuff
> hashes equal, but there's no reason for zero-length input to hash to
> zero, AFAIK.

http://hg.python.org/cpython/file/d7ba4ca59023/Objects/object.c#l852

    We make the hash of the empty string be 0, rather than using
    (prefix ^ suffix), since this slightly obfuscates the hash secret

It's not a hard requirement, hence 'should' instead of 'must' like in
all other cases.

Christian