On Integration Testing

[Scott David Daniels]

Emanuele D'Arrigo wrote:
> Hi everybody,
> 
> I just finished unit-testing a couple of tightly related modules and
> it's now time to test them together to a) check if they do talk to
> each other in the ways they should, b) check how fast they can talk to
> each other. The problem? Well, the problem is that potentially they
> can have some pretty varied conversations!! There are a lot of
> different messages that they can exchange, from simple question-answer
> messages to more complex [initial request, confirmation requested,
> confirmation sent, final response, public announce] sequences of
> messages. All this in a one-to-many context, that is one module is
> effectively a server and can be connected to zero to N clients.
> 
> Right now I do not have a particularly cohesive, consistent approach
> to this type of testing. In short, I've never done it before. I'll
> start with some simple tests, i.e. adding and removing clients and
> sending some simple sequences of messages, maybe pushing up the number
> of clients to see how much gets done in, say, a second. But it feels
> fairly "unscientific". =)
> 
> I've already googled "integration testing" and I'm in the process of
> reading a variety of articles. Is there any one that the people of the
> python community would like to recommend?

Seems like what you want here is Function tests.  At this point you are
testing the whole system behavior.  I presume the modules are meant to
conspire to produce a result; check that that result is produced.  If
you really have a nice set of unit tests, the integration is testing
against a misunderstanding of the interface. Your functional tests begin
with simple straightforward tests, then push at definitional boundaries.
One thing to vary is client-server interaction lifetime.  Make some
clients slow and others fast in your mix.

Performance testing (seeing how much you can accomplish in a second) is,
I'd argue, a later phase (where you need a good model of actual use).
First, go for correctness under "strange" interactions.

--Scott David Daniels
Scott.Daniels at Acm.Org