lazy (?) evaluation, request for advice

[Alex Martelli]

Tim Peters writes:

> >> Against that, you invented ad-hoc semantics as you went along;
> >> the only connection to Python appeared to be syntactic.
> 
> [Alex]
> > Did I?  I used, e.g., spades to mean "the value of a
> > variable named `spades' in the current scope[s]".  Isn't
> > that normal Python semantics? ...
> > I would appreciate understanding how you define "syntactic" in
> > this context to justify this extremely peculiar assertion.
> 
> The answer to "what's in a name?" is fundamental semantics for any
> language.
> Your language's answer is not Python's answer, hence my objection to
> calling
> the former Python.
> 
Here we go again.  Let's see if we can pinpoint this huge
difference of opinion -- I claim I'm looking for absolutely
nothing that is outside of the "fundamental semantics" of
Python, you claim there is a "my language" which differs.

Suppose, then, that my program accepts from the user some
text string, e.g. from a GUI or whatever -- say the user enters:
		
	(foo + bar) > baz and 3 < fee < 22

My program's role is to take this string (possibly compile
it), and eval it in a suitable dictionary (possibly more than
once, for different "suitable dictionaries"); doing one of
two different actions depending on whether eval returns
a true value in the Python sense.

Here, I may have a limited vocabulary, made up of the
variablenames: fee, fie, foo, fum, bar, baz; so my
life is easy:

	# code to obtain and possibly compile
	# user_expression omitted
	while 1:
		proceed_to_next_case()
		if no_more_alas():
			do_failure_sadly()
			break
		dict = {
			'fee': recompute_fee(),
			'fie': recompute_fie(),
			'foo': recompute_foo(),
			'fum': recompute_fum(),
			'bar': recompute_bar(),
			'baz': recompute_baz()
			   }
		if eval(user_expression, dict):
			do_success_triumph(dict)
			break

OK so far, I hope?  I'm doing absolutely nothing
non-Pythonic, am I?  This kind of thing IS just
about exactly why eval takes dictionary args, is
it not?  If you disagree, would you please point
out how this violates the "what's in a name" choices
of Python, and how eval and its dictionary args
are supposed to be used instead...?

Please suppose, in case it matters, that the
various recompute_fee(), etc, have no side
effects -- they do in fact nothing except
re-compute some things on the basis of the
state as changed by proceed_to_next_case();
they are 'pure' functions.

As I see it (and I'm a newbie, so I guess I could 
easily be wrong), "what's in a name" in Python
is: what is currently attached to that name in the
local scope, or failing that in the global scope,
or failing that in the builtins' scope; where such
attachments can and often do change when the
same code is evaluated over and over again in
some kind of loop.  When I eval or exec, I get
to control the scopes by passing dictionaries to
be used as local and global scope (or the same
dictionary for both) -- and this doesn't violate
Python's semantics, it's PART of Python's
semantics.


OK, so, suppose we DO agree that eval, and its
dictionary argument[s], ARE part of Python's
semantics (after all, they _are_ well documented
as part of the standard Python docs, aren't
they now...?); so that, in the above program,
what the user gets to do in that expression string
is use Python semantics (and syntax) to tell the
program what conditions, exactly, need to be
checked for [of course, an expression has
limited power, compared to other Python pieces
the user could be writing, such as a suite of
statements; but, very often such power will be
sufficient in this application domain, and for
cases where it isn't the program's interface
will provide alternative ways for the user to
enter statement suites or whatever -- this is not
the issue here, and has nothing to do with the
"what's in a name" question, does it?].


Now, all I'm asking is how I can do an optimization
(reduction of running time) that will change
NOTHING about how the program behaves,
from the point of view of the user who enters
his or her nice little string to be evaluated as
an expression -- except, hopefully, it may make
things faster, by avoiding useless computations.

If nothing changes except for speed, how can
what-used-to-be-Python's semantics suddenly
become "my own private and badly designed
language"'s...?


Specifically, the issue is that the vocabulary
gets larger, or maybe it doesn't but I notice
(thanks to a profiler) that recompute_fee()
takes a LONG time to run, so I would much
rather only call it when it IS needed -- which
is not all that often, as it happens, e.g.
because the "(foo + bar) > baz" portion of
the user-entered *Python* expression is
most often false, and thanks to *Python*'s
shortcut semantics this means that the value
of 'fee' is mostly not necessary.

I.e., I'd like to have lazy evaluation, to "coin"
a name.


I do not want to change the interface that the
user sees -- please assume that the users
like that interface and do not desire that it
changes... is it such a difficult assumption to
make?  I want the users to keep the exact
semantics and syntax seen above; I would
just like the program to be a bit (or a lot)
faster by avoiding needless calculations.

Now, please _explain_, at long last, why this
desire for optimization suddenly makes the
semantics non-Pythonic... assuming of
course that the premises were in fact agreed
(fat chance, I guess).
	

> > Where more than one hand is of interest, object notation will
> > be used; and where more power is needed, statements will be
> > used rather than just an expression.
> 
> The last wasn't (isn't) clear.  You described it as a "deal-selection
> function", but the example fragments showed only expressions.  If you are
> in
> fact going to treat isolated expressions as if they were functions, it's
> 
Sorry for expressing myself imprecisely, if I did.  "Deal-selection"
*expression* rather than *function* is no doubt what I should be
calling used-entered expressions such as the above example.

When the user feels that "just an expression" is not appropriate
(e.g., because he or she wants to compute partial results and
save them somewhere, etc), alternate ways for him or her to
supply Python code to be executed will also be provided.  But,
for now, I wanted to focus on the frequent case where pretty
simple Python expressions are in fact sufficient.


> again a sense in which your language is not (although may be related to)
> Python.  This means you're in the language-design business, as you went to
> 
Designing user interfaces, libraries, protocols for communication
between programs, etc, are ALL "language-design businesses" in
some quite reasonable sense.  If I have to design libraries, UI's,
protocols, and so on, I therefore have to design languages.

> great lengths in this reply to prove; I can only repeat my advice that
> Python is already well-designed and your users will be better served in
> the
> end by letting them use the language as-is; you don't agree, so that's
> that.
> 
I fully agree that Python is very well designed (or I would not be
here).  If by "letting them use the language as-is" you mean that
I should not provide any further interface or functionality besides
what is already IN Python, i.e. that I should not be writing any
further program at all, yes, then, I do disagree (and it would seem
to be a peculiar claim on your part).

I do not plan to hide the internals of the libraries/modules/classes/
call them as you will from users, if they do want to go that deep; I
do, however, plan to also offer the users other interfaces besides
a naked Python interpreter with some 'import mystuff'.

As soon as I do offer other interfaces, there are other 'languages'.
"Clic on that pushbutton does X" is not intrinsically part of Python's
semantics, so I'm "in the language design business" just as soon
as I have the simplest GUI with a pushbutton on it.

So, I'll have a textfield where (as one of the different optional ways
to express what it is they're looking for) users can enter a Python
expression (an expression in the language which you loudly claim
is not Python but rather "my own specially purpose-designed
language") -- if they do that then clic a certain pushbutton, the
expression gets (perhaps computed, then) evaluated as above.

Since all I do is a Python eval (after perhaps a Python compile),
with suitable argument[s] (as Python lets me give to eval, to be
used as scope[s] for it), I think your claim that this is "my own
special designed language" is absolutely preposterous.


> From what you've said so far, it doesn't *look* like anything fancier than
> a
> regexp replace is needed; e.g.,
	[snip]
Looks to me like that could be slow if the vocabulary is
pretty large.  E.g., hundreds of possible words, of which 
maybe 4 to 6 will typically be used in one expressions.

> transforming e.g. biglosers or heartsaches.  Generating a whole parse tree
> wouldn't work any better, *provided that* you don't have string literals
> in
> your language, and that the mapping to Python isn't hairier than simple
> token substitution.
> 
I have no intention of somehow forbidding users from having string
literals in the expression they can enter, although offhand I'm not
sure what use they could have for them in such an expression.

As for the "mapping to Python", none at all is needed, unless you
claim that eval is not part of Python, or is only part of Python if
its optional arguments are not used, or other such silliness; as I
hope I showed, were it not for the lazy evaluation idea, I could use
a simple eval after populating a dictionary appropriately.

Keep claiming to your heart's desire that this is "my own little
language" -- I fail to see what purpose such claims may fulfil,
but, hey, suit yourself.  Me, I'll just as relentlessly claim it is
Python, which I guess will be just as useless (except that I have
a strong aversion to see false assertions being uttered, and just
let them go, in a discussion I'm taking part in; so, the [limited]
use of my counter-claim is in easing my disgust at seeing such
falsehoods spewed -- with a small but finite probability that I will
learn something during this unpleasant process, such as, that
"eval" is NOT truly part of Python, it was just snuck in by a bad
witch's spell, or something).


> > I will appreciate further advice about this, and tolerate the
> > sneers and name-calling that seem to go with the advice
> > (although rarely have I seen discussion quite as sneering as
> > this on this group
> 
> Alex!  That wasn't sneering -- it was brazen ridicule <wink>.  Seriously,
> the exaggerated relentlessness of it all was supposed to nudge you away
> from
> taking at all soooooo seriously.  Didn't work, eh?
> 
Apparently not.  Do _you_ happen to know many "people" who
take well to being "brazenly ridiculed" in a public forum, and
react by jolliness and mirth?  You must introduce me to some
of them sometimes: being somewhat of a Martian myself in
several respects, I am always interested in acquaintance with
other aliens who happen to be visiting this planet.


> > -- reminds me a lot of the ambiance on the Perl group -- this
> > one had so far seemed friendlier, though I guess that may have
> > been an artefact due to a frequentation started more recently).
> 
> Oh, this *is* much friendlier.  Over on c.l.p.m., they would have called
> *you* an idiot.  I consistently insulted your *users*.
> 
I guess that "accusing" me of using non-Python semantics
does not count as an "insult", strictly speaking.  Maybe I'm
just being more newbye-y than usual, rather than really thick,
in failing to see how eval-with-dictionary-arguments can mean
"non-Python semantics".  So, please do elucidate.


Alex