[Mailman-Developers] How can I run the REST tests against live servers?

[Stephen J. Turnbull]

Andrew Stuart writes:

 > I’m writing an authenticating proxy for the Mailman REST API  and
 > want to make sure everything works as expected against real
 > infrastructure (Postgres/Postfix/Sqlalchemy/Falcon).

Determining that "things work as expected against real infrastructure"
is not what the test suite does.  AFAIK we don't have any such tests,
and I don't really see how the tests we do have can be adapted to
"tests against real infrastructure".  The problem is specifying what
to expect of real infrastructure.

You could, for example, write tests that insert users, query for them,
and then delete them, but this has several problems.  Do you want to
do these tests on "live" servers that might actually be in use? --
Probably not, but how do you prevent that?  Attributes of the various
objects created by the system may depend on context. -- The test needs
to ignore any differences due merely to different context.  Coverage
is necessarily imperfect in the sense that some internal state changes
can't be checked since the "before" state is indeterminate in a live
system.  And so on.