CSCE 431 Lecture 17

« previous | Tuesday, March 25, 2014 | next »

White-Box Testing

Know code inside;

assess effectiveness of test suite, how much of the program it executes

"Code coverage"

• Instruction coverage (measure instructions executed
• Branch Coverage (conditionals whose paths are both/all executed
• Condition coverage (how many atomic Boolean expressions evaluate to both true and false)
• Path coverage (how many of possible paths are taken)

Coverage-guided Improvements

1. Perform coverage analysis for given criterion
2. Find unexercised code sections
3. create additional tests to cover them

Dataflow-Oriented Testing

Focus on how variables are defined, modified, and accessed throughout the run of the program.

Suspicious access:

• variable defined but never used / referenced
• variable used but never defined / initialized
• variable defined twice before use.

Types of accesses:

• Definition (change value of variable; initialization or assignment)
• Use (use variable without changing value (computational or in predicate)
• Kill (causes variable to be deallocated, undefined, or no longer usable)

Data Flow Graph

For a path or sub-path $p$ and a variable $v$:

• def-clear for $v$: no definition of $v$ occurs in $p$
• DU-path for $v$:
  • $p$ starts with def of $v$
  • Except for this node, $p$ is def-clear

Criteria:

all-defs

execute at least one def-clear sub-path between every definition of every variable and at least one reachable use of that variable.

all-p-uses

execute at leats one def-clear sub-path from every definition of every variable to every reachable p-use of that variable

all-c-uses

execute at leats one def-clear sub-path from every definition of every variable to every reachable c-use of that variable

all-c-uses / some-p-uses

apply all-c-uses; then if any definition of a variable is not covered, use p-use

all-p-uses / some-c-uses

symmetrical to all-c-uses / some-p-uses

all-uses

execute at least one def-clear sub-path from every definition of every variable to every reachable use of that variable.

Mutation Tests

Who tests the tester?

If all tests pass, is the software really that good? (bug-free) — No.

Test the tests:

• Intentionally introduce defects. If the test finds the defects, then the test suite is good. Otherwise, it is insufficient.
• Plan defect types and locations.
  • random?
  • weight based on code criticality?

Fault Injection Terminology

mutants

Faulty versions of the program

Only versions not equivalent to the original program are considered to be mutants (even though the original program was faulty)

mutant is killed

if at least one test case detects the injected fault

what if fault is missed, but causes test to detect previously undetected fault in original code?

mutant is alive

if no test cases caught error otherwise

mutation score (MS)

associated to test set to measure effectiveness

Mutation Operators

A mutation operator is a rule that specifies a syntactic variation of the program text so that the modified program still compiles. (Maybe also require that it pass static analysis).

A mutant is the result of an application of a mutation operator.

Examples:

• replace aritmetic operator by another
• replace relational operator by another
• replace logical operator by another
• replace a variable by another
• replace variable (in use position) by constant
• replace number by absolute value
• replace constant by another
• replace while loop with do-while
• replace condition of test by test's negation
• replace call to a routine by a call to another routine.