Python and the need for speed

[MRAB]

On 2017-04-12 01:28, Nathan Ernst wrote:
[snip]

> I worked on http://www.marketswiki.com/wiki/CMDX - in particular I wrote
> most of the Migration Utility mentioned to migrate paper CDS trades to
> standardized CDS contracts against CME. Most of the migration util was
> written in native Python 2.5 (it was written in 2008) using a single
> thread. Performance wasn't super critical, but desired. At the end of the
> project, I was processing ~100K positions per second. Memory usage of the
> app was constant and processing time of a portfolio was directly linear to
> the number of positions in the portfolio. Python wasn't the limiting factor
> for the app - it was the write speed to the database (and we were using the
> bcp interface of pysybase to write to a Sybase DB).
> 
> Basically, what I'm getting at is Python *can* be performant. It's also
> easy to screw up and introduce non-obvious slowness. Threading in Python is
> bad - don't bother (until we can get rid of the GIL, I doubt the situation
> will improve).
> 
Threading is OK if you're waiting for I/O where the main limiting factor 
is the speed of the I/O, e.g. transferring stuff to/from the internet.

> If you have a performance problem with Python, before you blame Python,
> take a step back and look at your own code (or libraries you're using) and
> ask yourself: "Is my code optimal?"
> 
> Yes, Python is not the faster language/runtime in existence. But for
> probably 99% of the people out there that complain about Python's speed,
> there's probably plenty of suboptimal or outright wasteful code that they
> should fix first, before complaining. For the other 1%, Python was probably
> the wrong choice to begin with.
> 
> ****************************
> 
> I don't intend this to be seen or implied as an attack or criticism of
> anyone. I'm just trying to provide an insight into my experience of using
> Python.
> 
[snip]