Compilation (was: the python name)

[Avi Gross]

Peter,

I suspect there is some python tool that can go through a directory
structure of python code and produce appropriate byte code files in one
sweep. That would not be at run time. And, in any system where the average
user does not have write permission on those folders, that would be a good
idea or every use would be slower.

I don't want to keep dwelling on the past and would like to hear more about
the current and perhaps near-future state of python and perhaps new added
functionality and upgrades in some modules outside the built-in variety.

So I would rather not get into detailed discussions of the differences
between translation and compilation and other such words. I will grant your
points. I did not intend what you may have read into it. As you suggest, I
am very aware of generations of tools including lint versus cc as I did
programming back for more years using more languages and operating systems
and machine types than I care to remember. There is though a subtle
distinction between what programmers and organizations are supposed to do
and what individual programmers do.

The rules normally suggest getting something to be syntactically correct
then testing units and larger groupings and then entire projects. It used to
mean spending 90% of your time writing documents and attending endless
review meetings. People tend to find ways around some of that. It is amazing
how often they write the code first then write a document describing how
they will do it!

My point was that many developers used a compilation as a way to partially
debug their code. Typically you got some errors or at least warnings and no
proper result. You would try to fix and resubmit until it compiled properly.
That may not be the PURPOSE but it often was how things got done. Leaving
out a comma or semicolon  or calling a function with the wrong number of
arguments or having unbalanced parentheses or a variable name spelled wrong
and so on were often caught only by the compiler. Nowadays we often have
tools that check for lint without actually making an executable and in the
case of python, there really is no executable as even the byte code version
is run live.

Worse, I have seen how you can change a function to take an additional
argument and wonder what other code calls that function. The lazy way is to
recompile everything and see what fails!

Is this the PURPOSE of the compiler? Of course not. The purpose is to finish
the process and deliver the goods. But when developing code, some people
decided to be lazy about proofreading their own code and let the compiler
take a stab at it. As I said, when the process took long enough, they often
read the code or had a friend look at it and found the errors and KILLED the
compile and resubmitted it on the upgraded code. This speeded their own work
a bit but meant the computer was overloaded for little purpose.

Now when I import a bunch of code to my machine and run the compiler to make
executables, the purpose may be closer to the intended purpose if all goes
well. Unfortunately, sometimes it is a way to find out if your setup is in
some way flawed or you have the wrong version of something ...

Let me ask a question. Switch to a word processing concept. When writing
something, do you write a first draft mostly to completion and then do
various forms of checking and revision or do you pause after each sentence
or paragraph and run tools to check spelling and grammar and tell you what
grade level readers need to understand it? In the earlier days this was not
even a question. Then we got spell checkers that ran on their own and then
other tools that gave lots of (often useless) advice about writing too much
in a passive voice or that the document used long sentences and big (meaning
technical) words so the person needed to have a college education. Well,
most of us had a Masters or two or a Ph.D. and besides we tended to be
highly technical and much of the jargon in the documents was REQUIRED for
the purpose.

So, after a while, people ran these lint-style programs less often, maybe
mostly in the final phase. But, similarly, before the What You See is What
You get days, people would use a markup language like nroff/troff and PRINT
the darn thing to be picked up hours later downstairs. You could say we
compiled it, sort of. We would then read the hard copy to find all our
errors like turning on bold and not turning it off or not starting a new
paragraph or noting the sentence made no sense when seen all at once without
intervening formatting commands. So we edited the code and printed it again!

Worse were things you really could not see properly on screen. Parts of our
documents were little programs of sorts that produced equations, tables, and
computer graphics where we told it to connect a line from the northwest
corner of object 1 to ...

Sound like compilation to you? It was. You had to pipe the document through
filters like tbl and eqn and pic that expanded the language of that
component into reams of hard to penetrate troff including lots of variables
and arithmetical calculations. As you describe, there was a sort of
translation from a higher-level language to somewhat lower-level and then to
even lower levels down to something like the printer language that applied.

When we finally got screens with pixels and programs that generated a decent
(not perfect) WYSIWYG, we started printing less often. Nowadays, some people
using Word Processing never actually need to print.

Well, the imperfect analogy ends. Moving on.

-----Original Message-----
From: Python-list <python-list-bounces+avigross=verizon.net at python.org> On
Behalf Of Peter J. Holzer
Sent: Sunday, January 6, 2019 5:25 PM
To: python-list at python.org
Subject: Compilation (was: the python name)

On 2019-01-06 15:09:40 -0500, Avi Gross wrote:
> [Can we ever change the subject line?]

Feel free.

> {REAL SUBJECT: degrees of compilation.} Peter wrote:
> 
> "... Hoever, this is the Python list and one of the advantages of 
> Python is that we don't have to compile our code. So we need a 
> different excuse for fencing on office chairs ;-). ..."
> 
> I understand what he means, but python does sometimes do a 
> half-compilation to byte code if the compiled version does not exist 
> or the file being read is newer.

This is compilation, but it isn't a separate phase. We don't invoke a
"python compiler" to produce an "executable" which we then "execute". We
invoke the interpreter on the source code, which compiles the source code as
needed (so in the literal sense it is a just in time compiler, although this
isn't what we usually mean by that term). 

And because this happens in principle every time we start a python program
(yes, there is a cache, but that is incidental) the compiler must be fast
and primitive and basically unnoticable.


> What is the purpose of compiling? Superficially, it is to catch 
> serious errors.

I strongly disagree with this. The purpose of compiling is to translate from
one language into another. Typically, the source language is more
programmer-friendly, higher level and the target language is more
machine-friendly, lower level. But there are exceptions.

Finding errors is mostly a consequence of this function. If the compiler
cannot parse the source code or doesn't have enough information to generate
code in the target language, it cannot fulfill its function and has to bail
out.

There are programs with the primary purpose of finding errors: We call these
linters, static analysers, etc.

(From your other mails I guess that you almost certainly remember the Unix
distinction between cc and lint.)

Of course sometimes these are functions are conflated. Since an optimizing
compiler has to analyse variable usage anyway for efficient register
allocation it can also warn you about unused variables at
(almost) no extra cost. So it makes sense to put that warning into the
compiler (I'm less sure if it's a good idea to make this an error, as in
Go). Or the compiler might even get quite expensive checking capabilities,
as the authors think that programmers are more likely to look at the output
of the tool they have to use than to invoke a separate, optional tool.

        hp

-- 
   _  | Peter J. Holzer    | we build much bigger, better disasters now
|_|_) |                    | because we have much more sophisticated
| |   | hjp at hjp.at         | management tools.
__/   | http://www.hjp.at/ | -- Ross Anderson <https://www.edge.org/>