In response to “The Oracle Problem and the Teaching of Software Testing”

In The Oracle Problem and the Teaching of Software Testing post, Cem Kaner wrestles with teaching how a tester could determine oracles for an exam question about automated testing.

Unfortunately, the window for posting replies on the post has expired. Thus... this blog post is born! 🙂

Here’s the exam question he gives his students:

Suppose you have written a test tool that allows you to feed commands and data to Microsoft Excel and to Open Office Calc and to see the results. The test tool is complete, and it works correctly. You have been asked to test a new version of Calc and told to automate all of your testing. What oracles would you use to help you find bugs? What types of information would you expect to get with each oracle?

Note: Don’t just echo back a consistency heuristic. Be specific in your description of a relevant oracle and of the types of information or bugs that you expect.

As other comments on the post point out, I also find Hoffman's list better suited for this exam question.

As Cem Kaner already mentioned, they do indeed lend themselves well to being coded. After all, that's exactly what they were designed for.

With respect to teaching a tester to generate oracles for the question, perhaps the student might be confused by the question itself. I don't find it straight-forward to think about what oracles I would be using before I think about what information I'm trying to find out about the new version of Calc.

As a tester, how often do we think about the oracle before thinking about what information we're trying to ascertain?

As an example, perhaps I'm interested in finding out more information about Calc's implementation of the SUM function.

I may have used the Hoffman list which triggered a thought about the reference program oracle.

Maybe this lead me to using the command feeding test tool to compare with Excel. Or perhaps Bach/Bolton's comparable product heuristic triggered a similar thought.

Why not use the "consistent with itself" heuristic to use Calc as its own oracle? After all, SUM should have an equivalent result when using a Calc addition operation?

Would I have missed this if I were using the Hoffman list?

Would I have been preoccupied with using Excel as a reference program rather than Calc itself?

In my own test design experience, I don't find that the Bach/Bolton list triggers as many ideas as let's say the SFDIPOT list or even just a giant list of test techniques/objectives (function testing, scenario testing, compatibility testing, etc.).

Although I'm not as familiar with Hoffman's list, you can bet I'll be making use of it now.

As Cem and other commenters on the post have mentioned, each list is not complete.

Once we know what information we're looking for, the Bach/Bolton list, Hoffman list or some custom combination of lists could be used to trigger thoughts about useful oracles which may or may not be automated.