A question about semantics from the standard library's documentation

[Chris Angelico]

On Fri, Sep 20, 2013 at 1:28 AM, Aseem Bansal <asmbansal2 at gmail.com> wrote:
> In Python 3.3.2 documentation on the Python Standard library's introduction these sentences are given
>
> "For these types, the Python language core defines the form of literals and places some constraints on their semantics, but does not fully define the semantics. (On the other hand, the language core does define syntactic properties like the spelling and priorities of operators.)"

In future, a link would help :)

http://docs.python.org/3/library/intro.html

> That got me confused. What is defined by the language and what is not?
> Can someone give me an example about what this means?

The core language MUST be implemented in the "outer language" (C for
CPython, Java for Jython... PyPy makes this a bit confusing, but
there's still the same distinction). You can't write Python code to
determine how to parse Python code. However, this core is actually
quite small. It consists of everything necessary to actually parse
your script (so, the indentation and tokenization rules, etc), plus
enough classes to be able to handle literals (there needs to be a str
to handle "Hello, world", for instance). Once the core is built, the
rest of the language can be implemented in Python. That's the standard
library.

Quite a bit of CPython's stdlib is actually implemented in C, for
performance; but huge slabs of it aren't, they're just Python code on
par with the scripts you write yourself. They're part of the language
ecosystem - if you claim you "know Python", you have to know at least
something about the stdlib - but it's much more easily tinkered with,
since you can just drop in a replacement bit of Python.

Take the hex() built-in function as an example. I can replace it[1] by
simply creating my own function with the same name:

>>> def hex(x):
    return x+curse(x)-sanity()

It's not hard to roll your own hex() if you want to (base conversion
is easy enough). So hex() needn't be part of the core language, but
can simply be provided by a bit of Python code that gets imported
automatically.

In contrast, the 'def' statement MUST be provided by the core
language. You can't create a function that gives you the ability to
create functions! You also can't create the + operator, although it's
conceivable to have a language that grants user-level code that power;
in Python, at least, all operators are language features. However, the
exact behaviour of that + operator can be customized in Python, using
magic methods. That's the main difference between the core language
and the standard library - the stdlib could be, even if it isn't,
implemented in the language itself.

ChrisA

[1] This example just shadows the name, but near enough. Fiddling with
__builtins__ is a better "replacement", but it comes to the same
thing.