Is there a test suite for your test suite?

In one of my coding projects, a young collaborator sent me some code for automated testing, and wrote, "If this looks OK, I'll put it into production."

I looked at the code, and it quite clearly would never pass anything, since one of the tests sought to verify the working of a feature that hadn't yet been implemented. The programmer writing the testing code had never tested it.

We could try to write a meta-test suite to test our test suite, but then what about the meta-suite? How can we be sure it is correct?

The point here is not to pick on our novice programmer, who is quite smart, merely lacking experience, but to point out that while mechanical schemes of program verification can be a great aid to producing good software, but they can never be a substitute for sound engineering judgment.

1 comment:

  1. It's not a formal process, but I always personally administer that meta-test -- i.e. verify that any new project/test-system really does catch (deliberate) failures.

    (Also, FWIW in my sector it's not uncommon -- but it is definitely bad practice -- to push code with failing tests "that we know are supposed to fail".)