Python's carbon guilt

[Terry Reedy]

On 10/10/2020 11:58 AM, Peter Pearson wrote:
> Python advocates might want to organize their thoughts on
> this subject before their bosses spring the suggestion:
> 
>  From https://www.sciencemag.org/news/2020/10/we-re-part-problem-astronomers-confront-their-role-and-vulnerability-climate-change :
> 
>      . . . Astronomers should also abandon popular programming languages
>      such as Python in favor of efficient compiled languages. Languages
>      such as Fortran and C++, Zwart calculates, are more than 100 times
>      more carbon efficient than Python because they require fewer
>      operations.

Here is my reply to the "senior correspondent for astronomy..." who 
wrote the columm.
---

Dear Mr. Clery:

You report that Zwart of Leiden U says ...

"Astronomers should also abandon popular programming languages such as 
Python in favor of efficient compiled languages. Languages such as 
Fortran and C++, Zwart calculates, are more than 100 times more carbon 
efficient than Python because they require fewer operations."

Summary counter claim: the claim is phony.  As actually used by 
astronomers for computationally intensive tasks, Python is nearly if not 
as 'carbon efficient' -- because nearly all the computation is done with 
machine code compiled from C, Fortran, C++, or whatever.

(I am not an expert here, but am reporting to you a summary of what is 
generally known among experienced Python users.  I am leaving out 
numerous details.  Feel free to contact scientific Python experts for more.)

Here is how anti-Python, pro-OtherLanguage people come up with such 
misleading numbers as '100x slower'.  Confuse the issue by ascribing 
efficiency to languages rather than implementations.  Then compare a 
relatively efficient 'production' implementation of some algorithm in 
OtherLanguage to an computer inefficient (but human efficient) 
'development' implementation in Python that ignores how Python is being 
used in practice by scientific communities, such as the astronomy community.

Step 1: Create a file with couple of numerically intensive functions 
such as matrix_init and matrix_invert written in directly compiled 
OtherLanguage.  Add calls to matrix_init and matrix_invert.  Time one or 
more runs.

Step 2: Naively translate OtherLanguage directly into Python.  Don't 
allow imports. Time one or more runs with the CPython interpreter. 
Compare the OtherLanguage time to this worst case Python time.  Imply 
that users of Python are stupid enough to stop here for production 
usage.  I consider this a bit slanderous.

In practice, Python users inverting matrices, for instance, import the 
compiled numpy module, which among other things, wraps the standard 
Linpack package, which  uses machine-specific, assemble-coded, Basic 
Linear Algebra Subroutines (BLAS) when available.  In practice, I have 
read, the overhead of calling C libraries from Python instead of C is 
only a few percent.

The original and still main Python implementation, CPython, is written 
in C.  It interprets Python code, which is 'slow', but the Python code 
can include calls to fast, compiled, pre-written C functions.  CPython 
was designed from the beginning to be extended with other other modules 
and functions written in C. (Fortran extensions were included either 
immediately or soon after, but I don't know which.)  Some of these are 
included in the standard library, others from third-parties.

Since the release of Python 1.0 in 1992, real-world Python users, have 
been extending CPython when appropriate for production usage.  Indeed, 
numerical scientific computation was Python's first 'killer 
application'.  Now, most every open-source C library with significant 
usage has been wrapped.  There are at least hundreds. The current 
standard numerical package, numpy, is the third, and the needs of 
astronomers were among those that contributed to the design.

-- 
Terry Jan Reedy