benchmarking in general and using xheap

[Sven R. Kunze]

Hi everybody,

I've finally had the time to do the benchmarks and here you go: 
http://srkunze.blogspot.com/2016/02/the-xheap-benchmark.html

The benchmark compares heapq, Heap, OrderHeap, RemovalHeap and XHeap 
regarding their operation heapify, push and pop.

As expected wrapping results in some overhead. Most of the overhead 
consists of wrapper, super and descriptor calls. As with the current 
optimizations efforts, I expect this to be reduced even further. But 
even using current CPython 2.7 or 3.5, the overhead for simple heaps, 
heaps with a custom orders or heaps with removal can be considered 
affordable given the maintenance benefits.

@srinivas
The current removal implementation uses a index-tracking approach with 
quite some overhead for other operations. I am not sure if that is 
remediable with a mark-and-sweep approach but given the time I will 
definitely look into it for another benchmark post now that I have build 
the infrastructure for it.

@all benchmark friends
Not sure how you do your benchmarks, but I am somewhat dissatisfied with 
the current approach. I started out using unittests as they integrated 
nicely with my current toolchain and I could write actual code. Then I 
threw anything away and used timeit as suggested here 
https://mail.python.org/pipermail/python-list/2016-January/702571.html 
and grew my own set of tools around it to produce readable results; 
writing code as strings. :-/

And from what I know, there is no official infrastructure (tools, 
classes, ... such as there is for unittests) around timeit to 
encapsulate benchmarks, choosing a baseline, calculate ratios etc (and 
write code instead of strings).

Does somebody have an idea here?

Best,
Sven