Is there a test suite for your test suite?

By -
In one of my coding projects, a young collaborator sent me some code for automated testing, and wrote, "If this looks OK, I'll put it into production."

I looked at the code, and it quite clearly would never pass anything, since one of the tests sought to verify the working of a feature that hadn't yet been implemented. The programmer writing the testing code had never tested it.

We could try to write a meta-test suite to test our test suite, but then what about the meta-suite? How can we be sure it is correct?

The point here is not to pick on our novice programmer, who is quite smart, merely lacking experience, but to point out that while mechanical schemes of program verification can be a great aid to producing good software, but they can never be a substitute for sound engineering judgment.

Comments

  1. Silas Barta9:03 PM

    It's not a formal process, but I always personally administer that meta-test -- i.e. verify that any new project/test-system really does catch (deliberate) failures.

    (Also, FWIW in my sector it's not uncommon -- but it is definitely bad practice -- to push code with failing tests "that we know are supposed to fail".)

    ReplyDelete

Post a Comment

Popular posts from this blog

Libertarians, My Libertarians!

By -
Cruel to be kind means that I love you . Because, while I think you are mistaken, your hearts are in the right place -- yes, even you, Silas --  unlike some people . This Breitbart fellow (discussed in the link above), by all appearances, deliberately doctored a video of Shirley Sherrod to make her remarks appear virulently racist, when they had, in fact, the opposite import. I heard that at a recent Austrian conference, some folks were talking about "Callahan's conservative turn." While that description is not entirely inaccurate, I must say that a lot of these people who today call themselves conservative give me the heebie-jeebies.
Read more

"Pre-Galilean" Foolishness

By -
 I am currently reading The Master and His Emissary , which appears to be an excellent book. ("Appears" because I don't know the neuroscience literature well enough to say for sure, yet.) But then on page 186 I find: "Asking cognition, however, to give a perspective on the relationship between cognition and affect is like asking astronomer in the pre-Galilean geocentric world, whether, in his opinion, the sun moves round the earth of the earth around the sun. To ask a question alone would be enough to label one as mad." OK, this is garbage. First of all, it should be pre-Copernican, not pre-Galilean. But much worse is that people have seriously been considering heliocentrism for many centuries before Copernicus. Aristarchus had proposed a heliocentric model in the 4th-century BC. It had generally been considered wrong, but not "mad." (And wrong for scientific reasons: Why, for instance, did we not observe stellar parallax?) And when Copernicus propose
Read more