Lies in education [was Re: The "loop and a half"]

[Chris Angelico]

On Sat, Oct 14, 2017 at 10:12 PM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
> But be reasonable, I had just asked almost exactly the same question not one
> line earlier:
>
>     "If it's read-only, how can the compiler write to it?"
>
> and (as far as I can see) *nobody* thought to actually answer my question
> until perhaps a dozen posts later, when Chris more-or-less said:
>
> - the compiler doesn't, but the program loader does;
>
> - its not so much read-only memory as write-protected memory:
>   privileged code can still write to it.
>
>
> (Have I got the details right? Or at least close enough?)

Pretty close. There's a lot of levels of indirection involved. When a
program asks to manipulate a piece of memory, it uses a segment
selector and a memory location (offset); in most modern systems, a
flat memory model is used, where all identical offsets mean the same
piece of memory, so we tend to think of the offset as being the entire
pointer.

Those segment selectors refer to something in the segment table for
*that process*. That means that location 0x12345678 in my process
could be a completely different hunk of memory to location 0x12345678
in your process. (It's also possible that they're the same - loading
shared modules (DLL/SO) often just means mapping an already-loaded
block of memory into your process.) Your process is completely unaware
of any memory that it doesn't have access to. It also has no way to
violate the notion of "read-only", because that's a flag set on the
memory pages in that process's page table.

So when the program loader goes to work, it loads up a page of memory
from the disk, then grants this process read-only access to it, by
mapping that page into the process's memory space. The program loader,
being part of the kernel, isn't just "privileged" in the sense of
running as root/Administrator - it is the one that determines what
"root" even means. If root is able to manipulate a process's page
tables, it's only via kernel APIs.

As such, the concept of "read-only" is actually hard-wired in the
process's memory map. There's no way to violate that.

(I may have some details wrong, as it's a long time since I've worked
with any of this. Also, this is massively simplified. But I think it's
about right.)

> [...]
>> Therefore I consider the phrasing "the
>> compiler puts it into read-only memory", while certainly sloppy and not
>> technically correct, perfectly comprehensible.
>
> And here we go again. Instead of holding the author responsible for the poor
> phrasing, the reader is blamed for not somehow inferring the author's
> intended meaning.
>
> "Somebody misunderstood Greg's sloppy and incorrect words to mean something
> Greg didn't intend? And when given the opportunity to clarify, Greg failed to
> do so and continued using the sloppy and incorrect phrasing? Oh, that's not
> Greg's fault, it is the fault of the reader for not reading Greg's mind and
> knowing what he really meant. I shall continue to defend the choice of words
> which are LITERALLY WRONG rather than accept that somebody might have
> legitimately misunderstood them."

Saying that the compiler puts something into read-only memory is
skipping a lot of steps in the procedure - notably, it's making a
temporal jump right from compilation time to run time. Obviously the
data is in some form of read/write storage in between (eg a hard
drive), during which it can very well be mutated. So it's somewhat
defensible to consider that the compiler sorta-kinda places things
into the run-time process's memory (since the compiler (or the linker)
is what instructs the program loader how to arrange the process memory
tables), while still being reasonable to argue that, since the
compiler is writing to it, it must be read/write (which it is - in the
compiler's memory space). The distinction here is that there's been a
change in POV; each time, we're talking from the POV of the current
process, but that's a different process (compiler vs running program).

>> And I can't find anything
>> wrong with his revised phrasing "arrange for it to be in a location that
>> is read-only at run time". That's exactly what the compiler does.
>
> Except *read-only* is a misleading and poor way to describe it and it too is
> LITERALLY WRONG. The program loader can write to it, and does so every single
> time a program is run. Why is it called read-only when it isn't read-only or
> even Write Once Read Many?

It *is* read-only, from the POV of the running process. Nothing (other
than actual burned ROM) is truly read-only; possibly the flashable
BIOS is read-only from the POV of a running system, but I'm not even
sure of that. From the POV of the kernel, all RAM is read/write (I
think; there might be some exceptions, but certainly everything we're
looking at here is); from the POV of various processes, large slabs of
memory are read-only.

> I am sure that nobody, including Greg and yourself, would describe /root on a
> standard Linux/Unix system as "a read only directory" just because non-root
> processes can't write to it. You'd probably describe it as write-protected,
> or similar terminology.
>
> And if somebody asked "How is it read-only, root can write to it?" I surely
> hope you wouldn't double-down and insist that, no, it really, truly is
> read-only even though root can write to it.

Right. Directories are actually read/write, but the nature of Unix
file systems is that you can't modify something you don't have write
permission for. That's "write-protected". A closer analogy would be
chrooting; once a process has been chrooted, it's completely unable to
see anything outside its jail (modulo exploits and bugs), and it's as
if they don't exist. I believe you can affect a process's mounts such
that something is mounted read-only for that process, even though it's
read/write for everyone else. That would be similar.

> I don't know, maybe calling this chunk of memory "read-only" is the accepted
> term of art for those who know about program loaders and the gory details of
> how C compilers arrange memory and how code is run on x86 hardware. It
> wouldn't be the first time that the terminology to describe some technology
> was misleading to those who don't know the jargon, and I'm sure that it won't
> be the last time.
>
> But I'm not one of those people, and I'm sure that was obvious from my
> questions. Is it really so hard to expect that somebody who did understand
> what was going on to explain my misapprehension rather than argue over the
> precise details of what counts as writing and whether or not a compiler can
> be said to write to a silicon chip being manufactured in a factory half a
> world away?
>
> (By the way, thanks Chris.)

You're welcome. I'm hesitant to post with too much confidence here, as
I have a small smattering of experience with these things on a variety
of different systems, and in-depth experience on none. But hopefully
the above details will clear things up a bit - either by themselves
being useful, or by more-experienced people correcting me, which is
all to the good :)

ChrisA