Reference: The Art of SoftwareTesting, by Glenford J. Myers, Wiley, 1979.
Much of this note is taken from Myers' book.
1. Testing is important and expensive!
Many programming organizations spend as much as 50% of their development budget on software testing. Billions of dollars each year is spent on software testing. Yet in spite of this huge investment, little is known about testing and most software products are unacceptably unreliable.
2. What is testing?
One major problem is the notion of testing discussed in computer science texts.
A typical definition: "Testing is the process of confirming that a program is correct. It is the demonstration that errors are not present."
The main problem with this definition is that it is wrong! It is impossible to demonstrate that a program has no errors. Hence, the above definition describes testing as an impossible task. Few of us get excited about performing impossible tasks.
Myers' Definition: "Testing is the process of executing a program with the intention of finding errors."
Testing is an unnatural process (which is why people find it hard) because it is a destructive process. The goal of the tester is to make the program fail. If his test case does not make the program fail, then he is unsuccessful.
The differences in the two definitions create an atttitude change for programmers. Testing by Myers' definition is doable.
Important lesson to learn. (Myers' First Fundamental Principle)
"If your goal is to show the absence of errors, you won't discover many."
"If your goal is to show the presence of errors, you will discover a large percentage of them."
Also, remember testing is a poor substitute for good design.
3. What is debugging?
Debugging is not a form of testing. Testing and debugging are two distinct activities.
Debugging is the activity of diagnosing the precise nature of a known error and correcting the error.
Debugging and testing are related because the output of a testing activity (finding errors) is the input to a debugging activity.
4. Types of Testing
Module Testing is the verification of a single program module usually in an isolated environment, i. e. isolated from all other modules. We use a driver.
Integration Testing is the verification of the interfaces among system parts (modules, ADTs, objects, components, and sub systems).
System Testing is the verification of the system to the system's Design Specifications.
Functional Testing is the verification of the functional description of the system as stated in the Requirements Specification Document.
Acceptance Testing is the verification of the system by the customer. Usually the system must pass the acceptance test before you receive your money.
Regression Testing is performed during "maintenance" ("modify" in diagram) or in late development. Important test cases are selected as regression tests. These test cases will be run in the future after every modification or correction to the program. Usually the output of a version of the system is captured in a file. The output files are compared by a special software tools, e. g. "diff" command in Unix, for any differences. In this way the software engineer can detect any unanticipated behavior due to the latest change.
5. Testing Activities
Software testing is a range of activities like software development.
1. establish test objectives
2. design test cases (most critical activity)
3. write test cases
4. test test cases
5. execute or perform the tests
6. examine the test results
6. Testing Philosophies (Two pure ones as demonstrated by Sam and Sally)
Sam's Philosophy: Test to the design specifications. Sam doesn't care to look at the code. He views each module as a black box.
As long as the code behaves precisely as stated in the specifications, OK. His ultimate testing goal is to test every possible combination and value of the input and check to make sure the output matches the spec.
Sally's Philosophy: Sally doesn't care to look at the specifications. She designs her test cases by examining the program logic. She views each module as a glass box.
Some criteria Sally uses:
a). Design enough test cases so that every instruction (or statement) is executed at least once.
b). Designs test cases so that every conditional branch, e. g. IFs, is executed in every direction at least once.
Sally's ultimate goal is to test all execution paths through the program logic. She designs her test cases with little or no regard to the specifications.
In pure form, neither is a good strategy!!! WHY? We must do a combination of both.
8. Testing is largely a problem of economics. (Myer's 2nd Fundamental Principle)
Since exhaustive input testing is impossible, we have to settle for something less. Each test case used should provide a maximum yield on our investments. Given that the investment is bounded by schedule and budget, the art of testing is really the art of selecting those test cases with the highest chance of finding an error. Each test case should represent a class of inputs so that if the test case executes correctly, we have confidence that a certain class of inputs executes correctly. Though the possible inputs may be infinite or very large, we usually can design a finite and small number of classes. Also, the boundaries of the classes should be tested.
Conclusion: Test case design requires intelligence, time and effort.
9. Module Integration
The main issue: What sequence should be used to combine all the modules together to form the entire system or program? How this is done influences
1). test case design,
2). the types of test tools needed and
3). the order of coding of the modules.
Note: Design, implementation and testing are different activities but all three interact.
Six basic approaches to testing while integrating modules:
1. Bottom Up Testing
The program is combined and tested from the bottom of the tree to the top.
Do the leaves first (5, 6, 3, 7, 8) by module testing. Requires a module driver for each module.Put together subtrees and test until whole tree. Very common approach and effective, especially when combined with top down design or object-oriented design.
2. Top Down Testing
Top down testing is incorporated into top down design. In this approach the program is designed, combined and tested from the top to the bottom. Design, implement and test the top modules using stubs (or dummy modules). Stubs may be fairly complex. The stub must exhibit behavior appropriate to the intent of the module.
Usually one has problems with input and output, as they may not be done in the top module. To help this problem, we can embed the test cases in the stubs. We usually call this "hard wired."
Advantages
- combines module testing, integration testing and some functional testing.
- once input and output routines are integrated can use external inputs for test cases.
- the feasibility of the entire program can be determined much easier than with bottom up testing. Can detect major flaws earlier.
- don't need module drivers only stubs.
Disadvantages
- a module is rarely tested thoroughly right after its integration.
- may create false belief that one can begin coding and testing before whole design is finished.
3. Modified Top Down Testing
We combine #1 and #2. This requires each module to be unit tested in isolation before it is integrated in the top down testing manner. Now we use both module drivers and stubs.
4. Big Bang Testing
A common approach, especially with students!! All modules are integrated at once.
Many disadvantages and few advantages! Hard to debug. Hard to check interfaces. When an error suddenly appears in the other approaches, we have a prime suspect, i. e. the latest module. With Big Bang Testing no such luck.
5. Sandwich Testing
We perform both top down testing and bottom up testing with different software teams. Useful only on very large projects.
6. Critical Part First
In this approach, we design, implement and test the critical part of the system first. Important for time critical systems because if this part is too slow, the whole system will be too slow. If the critical part is too slow, we have to perform a complete redesign of the system.