Python and the need for speed

[Rick Johnson]

On Tuesday, April 11, 2017 at 7:28:46 PM UTC-5, Nathan Ernst wrote:
>
> [...]
>
> Writing performant Python code is possible, but like
> writing performant code in any other language, you need to
> be aware of what's happening. This means paying attention
> to things that may cause memory allocations (which are
> largely hidden from you in Python).

But what you're asking people to do is not unlike
reinventing wheels. Some are not aware of every subtle case
that will cause a bottleneck. For this reason, it would be
wise to create a database where these bottlenecks can be
logged, with the intention of creating a python optimizing
tool at some time in the future. Like you said, since a
large subset of python coders are not CS majors, we cannot
expect them to understand Python at such a fundamental level
as someone such as GvR. That is the price of creating a
language that abstracts away all the dirty details so that
laymen can wield it. But just as Python should _never_ seg-
fault (but it does!), high level languages like Python
should make it difficult, if not impossible, to write sub-
optimal code (at least in the blatantly obvious cases). The
whole fundamental nature of Python is to relieve the
programmer of all the onerous tedium and drudgery (memory
management, cryptic syntaxes, etc...), so that the
programmer's mental energy can be devoted solely to the
creative demands of logic and interfaces.

> I worked on http://www.marketswiki.com/wiki/CMDX - in
> particular I wrote most of the Migration Utility mentioned
> to migrate paper CDS trades to standardized CDS contracts
> against CME. Most of the migration util was written in
> native Python 2.5 (it was written in 2008) using a single
> thread. Performance wasn't super critical, but desired. At
> the end of the project, I was processing ~100K positions
> per second.

Yeah, but what kind of hardware were you running it on? It
seems in this case, that you wrote up a utility and ran it
on a machine of your choosing, but not all developers have
the advantage of choosing which machine their code executes
on. Most code is written and released for mass consumption on
client machines which come in all shapes and sizes. Sure,
you can write sloppy code on a top-of-the-line hardware and
not notice, but try the same code on a windoze 95 box and
see what happens...

> Threading in Python is bad - don't bother (until we can get
> rid of the GIL, I doubt the situation will improve).

If you search the archives (and internet) for "remove GIL",
you'll find more hits than you could parse in a lifetime.
Basically, the community has been begging for the GIL to be
removed since the early 90s, but the Python royalty have no
interest to do so. And the reasons for this could be many. I
have my personal suspicions...

> If you have a performance problem with Python, before you
> blame Python, take a step back and look at your own code
> (or libraries you're using) and ask yourself: "Is my code
> optimal?"
>
> Yes, Python is not the faster language/runtime in
> existence. But for probably 99%

I don't think it's fair to make broad-brushed assertions as
to the technical prowess of everyday Python programmers.
Perhaps the incidents of non-optimal code are high, but 99%
is difficult to swallow and sounds an awful lot like
hyperbole. And although i'm sure some folks here will lap-up
such gross exaggerations with prideful glee, sorry, but i'm
not one of them.