[Mailman-Developers] MM3 REST API testing [was: Python 3]

[Florian Fuchs]

Am 02.01.2015 um 05:40 schrieb Barry Warsaw:
> On Dec 27, 2014, at 01:16 PM, Florian Fuchs wrote:
> 
>> So far we haven't found a perfect solution for testing packages that
>> rely heavily on the MM3 REST API. As far as mailman.client, Postorius
>> and HK are concerned, testing without *some* core integration doesn't
>> make too much sense IMO, because any changes in the API would go unnoticed.
> 
> I'm playing with a solution that involves the use of the 'vcr' package, which
> as its name implies (well, at least for us, um, mature programmers ;) is a
> facility to record and replay HTTP traffic.  It's compatible with both Python
> 2 and 3 and is available on the Cheeseshop.

I took a quick look at vcr and it really looks promising. Although
apparently it doesn't support VHS, which is very sad. But we can still
use it for testing I guess... :-)

> So the basic idea is to set up an MM3 installation (`tox --notest -r`
> suffices), then start up its REST server with a fresh mailman.db, and run the
> test suite against it.   This will capture the traffic in a yaml file, which
> we could ship with mailman.client.
> 
> Then to run the test suite, you'd use vcr in replay mode against the captured
> yaml file.  Any time the API changes, you'd have to re-record the traffic
> against a fresh database.

That sounds fantastic! Do you think it might make sense to ship the
traffic dump with mailman instead of mailman.client (and work the
recording into the test suite somehow)? That way there would only be a
single file to observe for changes in the REST API. Sure, a change in
the yaml file  doesn't necessarily mean a change in the API (might as
well be just an additional test case or different test data). But still,
if the yaml file stays the same between commits, that would be a nice
guarantee that the API stayed stable.

> I have a branch that's working in this direction and it's promising, but I've
> also had to rewrite the testing infrastructure for mailman.client, and make
> some other changes.  It also points out what I think are some bugs in Mailman
> core, so I'm working those out on my py3 branch in parallel.
> 
> It's not yet ready for review, but if you want to keep an eye on things, look
> here:  lp:~barry/mailman.client/bilingual
> 
> Feel free to provide feedback on the approach, or the changes so far, but I
> will submit a merge proposal when/if it's ready to go.

Thanks for starting the port, Barry. Looking forward to seeing that
merge proposal (although I don't expect I'll have a too much to add... ;-).

Florian