Popular conceit about learning programming languages

[Alex Martelli]

Pascal Costanza wrote:
   ...
>>>Er, I have thought that's exactly the point. What do you need a
>>>backstage language for that is less powerful than the frontstage
>>>language?
   ...
> Languages like C and assembler are more powerful than, say, Python
> because they allow you to circumvent type restrictions and fiddle with
> the operating system level.

I would take another tack, and say the key issue is rather than
those languages are "self-hosted" (so you can use machine language
to code a virtual machine for Python, potentially requiring just
the tiniest amount of support from some OS or equivalent piece of
software) and have some clear potential to generate extremely fast
code in the right circumstances.

If you look at other languages that are commonly used to write
Python extensions, such as Fortran, you'll notice that the second
consideration can often prevail.  Fortran is quite suitable to
get extremely fast code in the right circumstances -- depending
on the type of problem (it had better be basically about numerically
intensive computation on large arrays;-), Fortran compiler available,
and machine architecture (e.g., is the compiler able to vectorize
Fortran loops particularly well).

Another consideration is that I can often have already-existing,
well-tested, debugged, and optimized libraries coded in Fortran: sound
engineering principles then dictate it's better to reuse such code
rather than recoding it to great, avoidable expense of testing,
debugging and tuning.  Again, the ability to extend Python with
Fortran helps a lot here.

This is reasonably independent from what language has been used
to code the Python VM itself, as long as good support exist for
code from several languages to interoperate.  This is typically
the case between C and Fortran these days, for example.

So, that concept of a "backstage language" seems a bit too fuzzy
for maximum usefulness here -- it appears to me that it's mixing
roles of VM-implementation and of ability-to-extend (and of
course possibly to _embed_, which is close to the extension issue).

Such issues as "circumventing type restrictions", performance
apart, are rarely a problem.  For example, module struct lets
you make bytestrings out of elementary types and put elementary
type instances back together again from the bytestrings -- so
you may type-pun to your heart's content if you wish (at the
cost of some data copying back and forth in this case, but,
that's just a performancee consideration...).  Of course the
standard module struct is itself coded in C (or Java, or ...) --
but then so is all the rest of the Python VM -- so it's not
some special need arising in connection with type-punning, it's
just the same kind of underpinning that lets a Python program
compute a+b or print "hello world".  Somewhat similar thoughts
can apply to "operating system level", depending on what exactly
we mean by that -- for example, on Linux, lots of "operating system 
level" things can be done through the (pseudo-)files in the /proc 
filesystem; on Windows, the ability to edit the registry (and
even more the ability to talk COM) enables even more operations.


> Common Lisp is more powerful than Python because it has the code=data
> feature that allows you to create new language features and some
> sophisticated and efficient algorithms.

That's presumably a novel idea in this connection, yes.

> Languages like Python, Ruby, Lua and so on make use of C as a
> "backstage" language in order to expand their possibilities. I believe

Actually, to have ANY "possibilities" whatsoever.  C is first of
all used to make the languages able to execute anything at all; in
second-order consideration, it may also be used to create further
extensions (mostly for speed or reuse of existing libraries rather
than to "expand possibilities" stricto sensu).

> that they could also benefit from Common Lisp as a backstage language.

That's quite possible -- distinguishing the two sub-issues of
ability to write extensions vs necessarily having the whole Python
VM coded in CL.  If you can pick a CL implementation that has a
good "foreign function interface" (or however it chooses to call
it), you can presumably write the code needed to let extensions
coded in that CL implementation interoperate with Python code, as
has been done for Fortran and other languages.  Having the whole
VM be in CL seems a different, separate project, with benefits
and costs different from the "CL for extensions" project -- e.g.
among the benefits the ability to use any CL implementation rather
than being tied to one or a few that supply the FFI you want,
among the costs the need to recode all the currently-C-coded Python 
standard library modules which, in a "CL for extensions" setting, 
you could simply reuse without any effort.


> So my mental model is as follows: Scripting languages allow you to
> solve, say, 90% of the problems in straightforward and simple ways.
> However, some problems are complicated or demanding enough that you (or
> "someone";) needs to revert to the more powerful backstage language.

I think this model lacks the key features of speed and code reuse.

> I don't see how Java can play this role for Python because Java neither
> allows you to fiddle with the machine level nor allows for writing
> complicated algorithms, at least not when compared to Python.

I consequently think the "don't see" comes from thise missing
aspects in the model.  Speed is still more of a promise than any
actual Java characteristic, admittedly.  But code reuse has its
substantial importance too.

> Or is there something I miss?

I think so and I've tried to show why.  There's more, too:


>>>(I guess that Jython is useful to take advantage of the Java
>>>APIs and its platform independence.)
>> 
>> That's a typical reason to choose Jython rather than CPython, yes --
>> being able to take advantage of existing Java-coded libraries (including
>> the APIs of the various Java-based standards), and/or of platforms able
>> to run Java bytecode (including various JVMs).
> 
> Are there other reasons?

Sure, for example customers who *specify* the product they purchase be 
delivered as Java bytecode (often with a specific requirement for "100% 
Pure Java", a specialized techno/commercial term;-).  A cognate case is 
that in which a Java application is already running and Python is chosen to 
extend that application with more features, most particularly the ability 
for end-users to script the existing application -- you could see this as a 
case of code reuse, taking the existing application as basically tantamount 
to a set of libraries, or as a case of customer-imposed constraint, where 
the "customer" is most often the same programming team which had developed 
the Java application and is now choosing how best to extend it.


Alex