unit-profiling, similar to unit-testing

[Ulrich Eckhardt]

Am 16.11.2011 15:36, schrieb Roy Smith:
> It's really, really, really hard to either control for, or accurately
> measure, things like CPU or network load.  There's so much stuff you
> can't even begin to see.  The state of your main memory cache.  Disk
> fragmentation.  What I/O is happening directly out of kernel buffers vs
> having to do a physical disk read.  How slow your DNS server is today.

Fortunately, I am in a position where I'm running tests on one system 
(generic desktop PC) while the system to test is another one, and there 
both hardware and software is under my control. Since this is rather 
smallish and embedded, the power and load of the desktop don't play a 
significant role, the other side is usually the bottleneck. ;)


> What I suggest is instrumenting your unit test suite to record not just
> the pas/fail status of every test, but also the test duration.  Stick
> these into a database as the tests run.  Over time, you will accumulate
> a whole lot of performance data, which you can then start to mine.

I'm not sure. I see unit tests as something that makes sure things run 
correctly. For performance testing, I have functions to set up and tear 
down the environment. Then, I found it useful to have separate code to 
prime a cache, which is something done before each test run, but which 
is not part of the test run itself. I'm repeating each test run N times, 
recording the times and calculating maximum, minimum, average and 
standard deviation. Some of this is similar to unit testing (code to set 
up/tear down), but other things are too different. Also, sometimes I can 
vary tests with a factor F, then I would also want to capture the 
influence of this factor. I would even wonder if you can't verify the 
behaviour agains an expected Big-O complexity somehow.

All of this is rather general, not specific to my use case, hence my 
question if there are existing frameworks to facilitate this task. Maybe 
it's time to create one...


> While you're running the tests, gather as much system performance data
> as you can (output of top, vmstat, etc) and stick that into your
> database too.  You never know when you'll want to refer to the data, so
> just collect it all and save it forever.

Yes, this is surely something that is necessary, in particular since 
there are no clear success/failure outputs like for unit tests and they 
require a human to interpret them.


Cheers!

Uli