CSCE 431 Lecture 18

« previous | Thursday, March 27, 2014 | next »

Test Quality Measures

$S$ is system composed of $n$ components denoted $C_{i}$ $d_{i}$ is the number of killed mutants after applying $C_{i}$'s unit test sequence $T_{i}$ to $C_{i}$. $m_{i}$ is the total number of mutants of component $C_{i}$

Mutation score $MS$ for $C_{i}$ and its unit test sequence $T_{i}$ is given by:

$MS(C_{i},T_{i})={\frac {d_{i}}{m_{i}}}$

Test quality $TQ$ is defined as $MS$.

System test quality $STQ$ is a measure of the quality of the entire test suite for the entire system; given by

$STQ(S)={\frac {\sum _{i=1}^{n}d_{i}}{\sum _{i=1}^{n}m_{i}}}$

Who Tests?

Any significant project should have a separate QA team

• avoids self-delusion (blinders; the programmer will seldom catch his own bugs)
• QA requires a skill-set of its own

Developers typically do their own unit tests

Developers and QA typically perform integration testing

QA Team does system testing

Customer and QA team do acceptance test.

Test Report Classification

Severity

• severity clasess defined in advance
• applied to every reported failure
• analyzes each failure to determine whether it reflects a fault, and if so, how damaging

For example:

• not a fault (feature)
• cosmetic
• minor
• serious
• blocking

Status

• Registered, just reported
• Open and working
• Re-opened (thought it was fixed, but not)
• Corrected
• Integrated
• Delivered
• Closed
• Irreproducible
• Canceled

IEEE 829: Testing Documentation

Issue Tracking Systems

Pick one and use it E.g. Traq, Bugzilla

Full-Employment Theorem for Software Testing Researchers

Pesticide Paradox

Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual.

Unlike real bugs, software bugs immune to current testing methodologies do not evolve and multiply themselves.

Instead, killing the "easy bugs" allows software to grow with more features and more complexity, creating new possibilities of these subtle defects to manifest themselves as failures.

Therefore, need for new softwere testing technologies is guaranteed ad infinitum

Boris Beizer, Software Testing Techniques, 2^nd edition, 1990

Code Review

• Overview
  • Author provides overview of code for reviewers
• Preparation
  • Reviewers work alone to scrutinize for errors
  • review rate 100–500 lines per hour
  • reviewers can be assigned perspectives (API, user's view, etc)
• Inspection meeting
  • Reviewer paraphrases code (explains logic)
  • Moderator keeps track
  • Scribe keeps record
• Post-Meeting discussion

Summary

Having a human read code or use a system detects 80% more faults per (human) hour than testing.

No technique (human or automated) catches more than 75% of bugs; average is around 40%