[Python-Dev] The baby and the bathwater (Re: Scoping, augmented assignment, 'fast locals' - conclusion)

[Josiah Carlson]

Before I get started with my reply, I would just like to point out that
aside from my initial 'I see your using a closure for this problem as a
design bug', I have tried my best to be as reasonable as I can, not once
attacking your character or abilities. However, twice now you have
expressed various insults towards me in order to somehow discredit my
statements.

At this point I am no longer interested in this conversation, a
sentiment I expect you share (perhaps along with those in
comp.lang.python). If you feel the need to continue to express
discontent about me publically, have at it. Heck, I'll even start you
out: Josiah Carlson's nose is too big. Further self-deprecation I will
mark with '*'.

It is obvious to both of us that we are misunderstanding each other on
various levels [1], so perhaps cutting of communication is better for
the both of us, and if anyone else has anything to say about the topic,
they can reply to either of us.

Boris Borcic <bborcic at gmail.com> wrote:
> Josiah Carlson wrote:
> > In my initial and later readings of your messages, I read 2 reasons why
> > you were using a closure rather than some other method to perform this
> > particular operation:
> 
> That wording already doesn't apply. I told you I was working *not* on a program 
> but on a *space* of programs of identical function, as of manner of probing 
> recent features and modules of Python that I manage to apply to it. Besides 
> learning, my purpose is also aesthetic.

My reading of your previous message stated that while you were looking
through various algorithms and codebases, your purpose seemed quite
singular, and I'll quote your quoting of the requirement:
""universal sudoku solver in +-pure python, ~10ms/problem, and ~60 LOCS
total"."

I must be such an idiot* that I failed to understand that you were, in
fact, working on modifying "a *space* of programs of identical function". 
I'm sorry about that.


As for code aesthetics, I have little to no sympathy for relatively
minor changes.  Decorators, yeah, that was a significant change in
aesthetics (and usefulness), and when such was pointed out, I went from
a -1 to a +1. But x[0] ?= y becoming x ?= y (where ? is one of ^&|+-*/),
or as provided by Noam Raphael recently, x[()] becoming x[], I can't get
behind for purely aesthetic reasons, and see limited practical value to
them.


> > However, since no one else has responded substantively to your postings,
> > other than Terry and Guido's 'not gonna change',
> 
> That's "Terry's 'take the fairy tale' <comma> and Guido's 'not gonna change'", 
> to be exact.

I had used the single quote as a mechanism of paraphrase rather than
double quotes as exact quotation.  You obviously understood this, but I
would point out that while Terry said, "this will not change for 2.x",
Python 3.x development has started already, so it's not so much a "fairy
tale'.


> > I can only presume that
> > others have found that your desired result (augmented assignments for
> > names not previously existing in the current scope, but existing in a
> > parent lexical scope, be performed on the object bound to a parent scope
> > name), is undesireable for Python 2.5
> 
> This is again misrepresentation. I definitely did *not* come to pydev to ask for 
> a change, I merely intervened in the middle on an ongoing thread at the point it 
> was being said that, in effect, I did not exist. Thereupon you intervened with 
> no other purpose than to show, in effect, that I did not exist, forcing me into 
> an endless chain of rectifications. Now you are telling me I want it included in 
> the 2.x series. That's confusing the citizen for the lobbyist.

I will point you back to Terry's posts:
http://mail.python.org/pipermail/python-dev/2006-June/066037.html
http://mail.python.org/pipermail/python-dev/2006-June/066057.html

Read them a few times.  Ok, now read my post:
http://mail.python.org/pipermail/python-dev/2006-June/065990.html

If I understood Terry correctly (and I hope I understood myself), at no
point in any of those messages did anyone state that nested augmented
assignments wouldn't be useful, or that there weren't any use-cases.
Terry stated how augmented assignments managed to work in alpha 2, why
it was a bug generally, and why that patch was being fixed/reverted.

I tried to express that I felt your use of a closure was a poor design
decision, because I had understood your code to offer an external
interface, where it actually offered an internal interface.


Over the course of the subsequent emails, I have attempted to,
essentially, make similar points as Terry, though obviously Terry was
far more concise and far less aggravating than I have been. Take his
emails as the authoritative source in regards to this topic (I do, even
though I continue to post), and if you desire, pretend that my messages
didn't exist.


> The "programming project" I *did* describe was the project of *further* 
> exploring, with systematism, the space of nearly equivalent programs - while 
> slightly *relaxing* the frame of the initial result, and while exercizing 
> varying parts of relatively recent distribution Python or third party modules 
> that I did not know, or not know well.

Could you please point out where you previously stated that you were
"exploring, with systematism, the space of nearly equivalent programs". 
I've tried finding it, but I fear my glasses are far too caked with
personal and professional failure for any cleaning to do any good*.


> Seeking beauty, if you want. You understand the quest for beauty ? I guess not 
> in the manner of someone who started programming when programs had to fit in a 
> couple dozen kbytes.

I quite likely do not appreciate the same, or to the same degree,
certain aesthetic qualities of particular Python programs that you do. 
But we both seem to like Python to some extent, so hooray.


> The "compiler feature" under discussion just pissed me off when it popped up to 
> veto what was emergeing as the winner of my beauty contest after examining quite 
> a few worthy candidates over weeks.

I'm sorry to hear that.


> > Great, wonderful, I wish you luck; I will not question or
> > "refuse legitimacy, motivation, efficiency or methodological soundness"
> > of your approach when using the Python programming language.
> 
> What's wrong with chosing a set problem to compare features of the Python 
> programming language against, and learn from it ? Please tell.
> 
> And what's wrong, when seeking beauty, with picking a stringent constraint and 
> then explore in parallel various manners to satisfy it, relaxing it slightly if 
> need be ? How do you think nature evolved birds or flowers or anything that lives ?

There is nothing wrong with either.  I myself have spent more time than
I would care to admit comparing xml.dom, xml.sax, and elementTree,
TCPServer against asyncore, various DB wrappers against each other, etc.


> Well, at this point you risk an even worse fate than that ;) That is, the 
> intersection of the file tagged "arrogant would-be mind-readers", and of the 
> file tagged "you are late on the normal trajectory if you did not reproduce my 
> own oh-so-significant mistakes", both well-populated already, although I forgot 
> who I put in them.

I don't have a funny walk, but you could toss me in the "has a funny
dance" file (because I do have a funny dance).


> The problem is that you fail to be polite about it more often than not. Most of 
> the time not really talking to the reasonable person I tend to be.

We all tend to believe that we are being polite and/or reasonable. Aside
from my first message, I thought I was being quite pleasant, though I am
also a self-righteous prick*, and wasn't reading my emails aloud with a
sufficiently hostile tone to understand what it may sound like to you.


> > I will mention that the lack of _any_ public support in python-dev
> > likely says far more about the unliklihood of acceptance than anything
> > you or I have probably typed in the last few days.
> 
> And what's the purpose of that declaration ? You are the, hum, crazed lobbyist, 
> not me. I am just defending that my pov is perfectly legitimate, including when 
> it's examined by your criterions - once the fallacious ones among them are 
> filtered out.

The purpose was to /hopefully/ bring this thread to something of a
conclusion, which it failed to do.


> > My measurement is quite subjective.  Have I ever made that mistake, or
> > do I know of anyone who has made that mistake? In this case, I have made
> > the mistake I mention, both in personal code, as well as code I was
> > writing as I was tutoring.
> 
> Thanks for the transparency. You mean code you were writing in real time before 
> an audience ? Hum. Surely you take the time to debug example code before 
> presenting it ?

Writing in real time in front of a student as they were asking questions.


> The error you mean - do we agree on this ? - is to intend two variables of the 
> same name, one in the parent scope, the other shadowing it in the child scope, 
> and mutating the variable in the child scope while having forgotten to 
> initialize it.

Yes.


> A few questions. First, do we agree or not that gratuitous shadowing is by 
> itself a mistake ? Or do you mean it happened to you while the shadowing wasn't 
> gratuitous ? Also, do you mean it happened to you in the Python context so that 
> Python stopped it at "ref to uninitialized local" - or in some other langiage 
> context ? In the Python case, I can only wonder as to what could make it 
> memorable - not the debugging, certainly.

I wouldn't call the shadowing gratuitous, but it was unfortunate. It
happened to me in the context of Python, through an attempt to write
something very similar to what you included in your original post.  The
initial memory was due to a slight embarrassment during the tutoring
session, followed by an identical mistake in unrelated code a few days
later.  I can't remember any other situations where I've run into the
same problem (I try to keep away from closures except when writing
decorators), but I can't say with certainty that I haven't made that
mistake since.


> My take on the issue, btw, is that the "scope capture" behavior of 
> (non-augmented) assignment statement is in Python the residual counterpart of 
> variable declarations in other languages; and it is significant that it shares 
> the most typical basic form, since in most cases declaration and initialization 
> are fused. This doesn't carry over to augmented assignments that are mutation 
> operators.

If I understand you correctly, I agree with you.


> > Here are two questions for you:
> > 
> > 1. Do you have any proof showing that the error I claim exists (not
> > uncommonly was my description), is in fact so rare, that I am
> > essentially the only one who has made the mistake?
> 
> Well, my take on an issue of such form is perhaps particular, but - as I already 
> suggested with my "hate for whatever resembles a constraining curriculum of 
> errors to make" - I really believe, as a matter of ethics, that the burden of 
> proof is on your side, not mine.

I disagree.  The error can happen, I have made the error, I am not a
unique butterfly, so others may have made the error.  What would raise
an exception before then produces no exception, but may produce
potentially mysterious results, hence an error would pass silently.


> > 2. Do lexically nested augmented assignments happen so often as to make
> > this problem a "must solve" for any Python revision?
> 
> Hum, first of all implementation of the solution is (according to recent 
> records), so  trivial that it occurs spontaneously as a bug. Therefore, even 
> conceding to you the usage be so rare that the gain is vanishingly small, the 
> ratio gain/effort would be one of vanishingly small quantities, which itself is 
> a priori not determined to be close to zero.
> 
> This of course does not count the cost of the decision. AFAICS, you are yourself 
> the most significant factor in determining that cost, and because of that I 
> think it would not be fair dialectics that I should take that cost into account 
> to answer your own question. Right ?

I don't believe that your use of gain/effort is sufficient to describe
the total real benefit vis-a-vis potential errors.  To me, while the use
of augmented assignments in lexically nested scopes would be convenient,
it would allow a class of errors (that are likely not common) to pass
silently.

Also, the ease of change is somewhat inconsequential, as with sufficient
support, just about anything will be implemented (and many things with
one person have been implemented).


> Second, the assumption of rarity leading to insignificance - does it escape you 
> that's a sword that cuts both way - as concerns a sizeable part of your own 
> argumentation ?

It does not escape me.  But the rarity, in my opinion, weighs
differently.  We could likely toss a half-dozen of the zens back and
forth as to why each of us is right, but of all zens that apply to
either side, one zen trumps them all.  Why?  Essentially all zens
relating to this topic describe relative goodness, x is better than y, z
is good, etc., but there is one that is absolute: "Errors should never
pass silently."


> Third, to me the matter is foremost one of aesthetics. The contentious "feature" 
> made me briefly feel like I was confused on the identity of the lady I was 
> making love to.
> 
> Fourth, of course I can survive without it.

I'm glad.


> > Do skilled programmers make errors?  Yes.  Do skilled programmers make
> > errors when they are first using a programming language?  Yes.
> 
> Of course everybody makes errors, but it doesn't follow from this, that all make 
> the same errors, or should.

If I implied that everyone has made this particular mistake, I'm sorry. 
I was trying to say that I made error X, and that I don't believe that
I'm a unique butterfly to have made error X.  If I'm not unique, then
presumably someone else has made a similar mistake.  And in this case,
it *could* have lead to mysterious behavior, where before it would
produce the equivalent of an "are you sure?" exception.


> As concerns Python, the issue is that in most aspects it is carefully crafted so 
> that you can often "just try it and it works as you would expect". For this to 
> work, consistency is very important. Our whole debate is that according to me 
> the "feature" under discussion is not in the same spirit while you (and Terry 
> indeed) found preferable not to acknowledge the notion, what also prevented the 
> possibility to convince me that the "feature" should exist anyway, for logical 
> reasons a reasonable mind could hear.

I agree that having nested augmented assignments "just work" would be
convenient, if I have stated or expressed anything to the contrary, then
I apologize.  My objection with having it "just work" is that it *could*
mask unintended shadowing errors.


> > It has
> > always been my experience with Python that it seeks to be reasonable to
> > use for both newcomers to programming and long-time "skilled"
> > programmers.
> 
> Nobody forces any school to present closures to beginners. Burn SICP before your 
> class.
> 
> More seriously, I believe the case you made against closures really should hit 
> what incites to the "simultaneous mistake", eg the ease of shadowing builtins 
> and globals, inadvertently to start with. A nicely fat set of keyword-like names 
> that one can't shadow without special action, would that not be the best of 
> schools against gratuitous shadowing ?
> 
> Well, maybe that would be too much of a change for Python. OTOH, mhh, I would 
> need to study the matter more in depth, but at first sight I see absolutely no 
> obstacle to trading a ban on closures shadowing names from the outer local 
> scope, against the current ban on augmented assignment of variables from the 
> outer scope (even without including straightforward assignment in the bargain).
> 
> This would give python closures a new educative role : a context that trains 
> against shadowing. If I am not mistaken, it would fulfill the "use case" you 
> find to the "feature" I dislike, just as well.

One could make the argument that any static list of keyword-like names
would be insufficient.  An alternate method would be to disallow any
shadowing without explicit "yes I want to shadow this", or even an
explicit "this refers to an outer scoped variable" (though the latter
I've already expressed my displeasure about). Then again, pychecker
picks up on arbitrary shadowing if I remember correctly, and I'm sure
most people don't run pychecker nearly as often as they should (I know I
don't), which would clue everyone in on such mistakes.

One consequence of either a fixed or variable set of non-shadowables is
that you would need to use arbitrarily unique names for lexically nested
for loops, and/or specify that such variables are not referencing parent
scopes.

def foo(...):
    for i in ...:
        ...
    def bar(...):
        ...
        for i in ...: #unallowed
            ...
    ...

You can replace the 'i' with a name of your choosing.  Is such
desireable?  I would say no for the same reasons why I said no in
regards to being able to explicitly name variables in different levels
within lexically nested scopes: I would argue that one shouldn't be
using closures in the first place, so encouraging its use with a syntax
is right out the door.  But that's my opinion, and not really relevant
to the current discussion.


> > In my statements regarding possible errors related to augmented
> > assignments, I merely believe that people can and will make mistakes.  I
> > further believe that if I manage to make a particular mistake, that
> > someone else has, because while I'm not the most brilliant programmer
> > out there, I'm also certainly not the worst, and I would guess that I
> > lack the creativity to generate a completely new class of mistakes.
> 
> Whatever, programming or not, I believe the most dangerous (in the sense of most 
> difficult to rectify) class of mistakes is the class of diagnostic errors - 
> meta-errors, so to say; because of that, I believe a danger with mistakes or 
> errors is to anticipate too much on them, and, given their diversity, to 
> overestimate the scope of the lessons learned from the sample of them that most 
> caught our attention (usually because of their painful consequences).
> 
> Besides, I think there are subtle issues with taking errors as building material 
> for policies destined to public consomption by new generations. It is vital to 
> find a self-standing positive goal to substitute to the negative goal of 
> avoiding the repetition of the error, etc.
> 
> Btw, how you dealt with my case tells me you don't particularly fear diagnostic 
> errors, but your way of presenting closures and classes in Python tells me that 
> you probably agree with my later remark.

You are more or less correct.

In very few things, I am conservative.  One of those things is what I
would consider to be reasonable in regards to language behavior change
compared to previous Python versions; the fewer changes, generally, the
better. I don't believe that Guido is the end-all-be-all of language
design, but I do find that I agree with him more often than not (though
perhaps not vv), and at least in regards to this particular
functionality, I side with the 'status quo' end of things.


Have a good weekend,
 - Josiah

[1] http://www.csmonitor.com/2006/0515/p13s01-stct.html