[Tutor] OT: "Your tests are only as good as your mocks." Comments?

[Alan Gauld]

On 25/07/2021 18:14, boB Stepp wrote:

> The author of this article notes an example from his practice where his mock
> database that he used in his tests passed his tests when the actual code in
> production no longer had a database column that was in his mock database.  

That's always an issue with test environments.
On big projects a large part of the budget is spent on testing
and maintaining sync between environments. That usually means
separate test and development teams and sophisticated automated
tools to check for any changes to checked-in schemas etc.

Detecting unpublished changes to the schema is an important
part of the testing since such changes tend to impact
many components of the system.

> The overall article itself is a recap of what he read in a book about how
> Google does things ("Software Engineering at Google").  In this situation
> Google advocates for using "fakes" in place of mocks, where these fakes are
> simplified implementations of the real thing maintained by the same team to
> ensure API parity. 

I've never heard them called fakes before but having test databases
based on the real thing is a common test strategy. Usually there is
an automated tool that takes a copy of the dev database and then
mangles it in ways that appease the data protection Gods...

>  How would the development and maintaining of these fakes
> be done so that the fakes don't drift from coding reality like the mocks
> might? 

- Tools.
- Regular sync of baselines (via more tools).
- Making the test schema the definitive one, so any failures
  are the fault of the devs. If they haven't published their
  changes in advance they are to blame for failures in test!

>  It is not clear to me exactly what is going on here.  And a more
> Python-specific question:  Does the Python ecosystem provide tools for
> creating and managing fakes?

No idea, I've never found much tool support for this kind
of thing, certainly on bigger projects there is a whole
team responsible for building test tools that includes
syncing and creating test data. And most of it is bespoke.

As an example we had one project with around 500 developers,
of which about 100+ were involved in testing and 30+ in creating
the various builds and version control.  In other words about
30% of the total technical staff were doing nothing but test
related tasks. But that project had around 66 million lines
of code (COBOL, SQL and C++) and a (DB2) database with over
10,000 tables and took up over 600GB of space.

At that scale these things matter. For most Python type
projects things are very much smaller and the scale of
testing can be similarly scaled down.

By the time you get to a small team of 6 or less (the
happy spot for agile techniques) testing is usually a
case of some fairly limited and informal system tests
(possibly done by  representative users) and a lot of
unit testing done by the devs themselves.

Unit testing rarely needs more than mocks. It's in
system testing that "fakes" become critical.

All IMHO of course.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos