[Mailman-Developers] How can I run the REST tests against live servers?

[Andrew Stuart]

Hi Barry

I’m writing an authenticating proxy for the Mailman REST API  and want to make sure everything works as expected against real infrastructure (Postgres/Postfix/Sqlalchemy/Falcon).

I was hoping that just pointing the REST tests request port to my proxy would work but no dice - lots of failures - trying to diagnose the failures has led me back to trying to get the test suite to work against the live servers.

I’m wondering if maybe the shortest route here might not be for me to write a new set of tests which purely talk REST and don’t touch the back end directly at all.  Except that’s not a short route. :-)

Trying to bend the existing tests to meet my needs is looking harder than I have time to deal with.

Andrew



On 12 Dec 2014, at 8:27 am, Barry Warsaw <barry at list.org> wrote:

On Dec 12, 2014, at 07:52 AM, Andrew Stuart wrote:

> I want to run the REST tests against my live Mailman server with its Postfix
> mail server and Postgres database, via:
> 
> nose2 -v -P rest
> 
> After many (many) hours of digging deep and figuring out how it works, I
> suspect at this stage that maybe the tests set up their own mock servers?  If
> so, I’m guessing these mock servers are getting in the way of using the live
> servers and thus preventing the tests from running?
> 
> Assuming it is correct that the tests are setting up their own mock http smtp
> ltmp responders, is there a way to turn this behaviour off so the tests will
> run against my live REST/Postgres/Postfix?

Hi Andrew,

You're right that the test suite starts its own http and lmtp servers.  Take a
look at src/mailman/testing/layers.py for how this is controlled, and review
src/mailman/testing/nose.py and this documentation on nose2 support for Zope
testrunner-style layers:

https://nose2.readthedocs.org/en/latest/plugins/layers.html

I think even if you could somehow substitute real servers for the testing
ones, you'll still have a problem running the test suite against them, to any
semblance of success.  The tests are dependent on specific arrangements of
sample data in order to run successfully.  E.g. you'll see that the
ConfigLayer, which is the "base class"[1] creates an example.com domain, so
that all a test needs to do is create a mailing list in that domain without
first having to create the domain.

The test suite is also very careful to restore pristine state between
individual tests; look for reset_the_world() which is run after every test,
and does exactly what it describes.  This ensures that no state from one test
can affect the state of another test.  In the real world testing you propose
above, you would have quite a bit of bleed-through I suspect.

Can you describe why you want to run the test suite against a live Mailman
server?

I do think it would be interesting to build an integration test suite that
would talk to real MTAs and databases, but that would take a fair bit of work
and probably be separate from the unittests, which aim to gather 100% code
coverage.

Cheers,
-Barry

[1] Although layers don't really support normal Python class hierarchies.
_______________________________________________
Mailman-Developers mailing list
Mailman-Developers at python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/andrew.stuart%40supercoders.com.au

Security Policy: http://wiki.list.org/x/QIA9