What's better: bad tests or no tests at all?...

I don’t have to tell you that every project is different. Should you test setters and getters? What about a method that writes to a file? Do you test methods that call other methods if those methods are already tested? What do you do about exceptions? There’s no point in testing if you’re testing incorrectly… right?

Learn what bad tests look like

I usually say “any tests are better than no tests…” Especially when you’ve never written tests or have an untested legacy project. But tests can end up costing more than they’re worth - and that’s a problem.

One of the best talks I’ve heard on the subject is Test Automation without a headache: Five key patterns by Gojko Adzic at the 2015 Oredev Developer Conference. Gojko Adzic is the author of Specification by Example: How Successful Teams Deliver the Right Software and co-author of Fifty Quick Ideas to Improve your Tests.

If you’re (rightfully) worried about writing bad tests, it’s good to be able to anticipate the consequences of writing bad tests.

Gojko describes two axes for categorizing tests. On one axis, we have the maintenance costs of tests. Obviously, we want to avoid costly tests.

On the other axis is how well the test detect risk. Risk detection, to me, encompasses what the real purpose of tests are - to mitigate risk… to get feedback when I am taking risks… to bring risk to the surface for me to understand.

Gojko kindly let me re-create the matrix from his talk with some additional notes:

When you’re wondering “what to test” and you pick something that you might want to write tests for - it might help for you to think about where your tests fall into these categories.

Will they be Fast Food tests? Are they automated UI tests that are expensive to maintain, but don’t detect much risk?

Will they be Cement tests? Will they detect risk well, but be costly to maintain? Are they effective at finding issues, but take 3 days to run? Will everyone be afraid to touch them?

Will they be Blinder tests? Are you mocking everything out and writing tests against the interaction between these mocks? They’ll sure run fast! You’ll have good coverage! But everything breaks in production…

We’re shooting for Exo Skeleton tests. We want the tests to help us run faster. We want them to help us lift things that we couldn’t lift before.

Summary

The question comes up over and over again: “What am I supposed to test?”

Someone else may be able to look at your production code and pick out some things to test for you, but it’s far better to have a decision “engine” around making the choice.

I think knowing what tests will become more costly than beneficial is a good component of this decision “engine.” Gojko Adzic’s talk about doing “Test automation without a headache…” offers some excellent advice for sensing the potential quality of your tests.

If you can sense how a test might fall into the undesirable categories in the matrix of maintenance cost vs risk detection, you might have a better idea of what to test moving forward.

I have more to talk about regarding testing, so make sure you sign up for my newsletter to get these posts sent right to your inbox!

Sources

Adzic, Gojko. Test automation without a headache: Five key patterns. Oredev 2015

Related Posts: