Testing is the primary quality assurance technique used in industry to ensure reliability of software systems. This course explores techniques and tools for systematic software testing. Topics focus on analysis of code, but will also cover analysis of software models and their use in testing. The majority of course materials will be drawn from classic papers and current state-of-the-art work. Students will read and review papers ahead of time, present papers and participate in class discussions, work on research topics in testing during the course, and do a term project individually or in a two-member team. Students will also write a paper (as well as review other students' papers) describing their project and present their work at the end of the course, in a "conference" format designed to give students an experience similar to that of participating in a professional conference.
Lecture 1 Monday January 7
Introduction to testing.
Readings:
The evaluation of program-based software test data adequacy criteria,
E.J. Weyuker, CACM 1989.
Software Testing Research: Achievements, Challenges, Dreams,
A. Bertolino, ICSE 2007 Workshop on Future of Software Engineering.
Chapter 1 from Introduction to Software Testing
by Paul Ammann and Jeff Offutt.
Wednesday January 9
No class. POPL in San Francisco.
Monday January 14
Random testing. [Kunal to lead discussion.]
Readings:
J.E. Forrester and B.P. Miller, "An Empirical Study of the Robustness of
Windows NT Applications Using Random Testing", 4th USENIX Windows Systems Symposium, Seattle, August 2000.
Feedback-directed random test generation
by Carlos Pacheco, Shuvendu K. Lahiri, Michael D. Ernst, and Thomas Ball.
In ICSE '07: Proceedings of the 29th International Conference on Software Engineering, 2007.
Optional:
B.P. Miller, L. Fredriksen, and B. So, "An
Empirical Study of the Reliability of UNIX Utilities",
Communications of the ACM 33, 12 (December 1990).
Assignment
Download yaffs and read the documentation on the Yaffs File System.
Read also the Wikipedia page on Yaffs.
We'll use Yaffs as a case study in some future classes.
Wednesday January 16
1. Random unit testing for yaffs.
2. Random testing for concurrent programs. [Rupak to present]
Readings:
Koushik Sen, "Effective random testing
of concurrent programs", ASE 2007.
Monday January 21
No class.
Wednesday January 23
Symbolic execution and directed testing.
Readings:
Patrice Godefroid, Nils Klarlund, and Koushik Sen. DART: Directed
automated random testing, PLDI 05.
Koushik Sen, Darko Marinov, and Gul Agha. Cute: Concolic Unit
testing for C, FSE 05.
Monday January 28
Context bounded verification of concurrent programs. [Joseph to lead the discussion.]
Readings:
Shaz Qadeer and Dinghao Wu. KISS: Keep it simple sequential.
Madan Musuvathi and Shaz Qadeer. Iterative context bounding for
systematic testing of multithreaded programs, PLDI 07.
Optional:
Shaz Qadeer and Jakob Rehof. Context bounded model checking
of concurrent software, TACAS 2007.
Wednesday January 30
Two applications of directed testing. [Hyuduke to lead]
Cristian Cadar et al, EXE: Automatically generating inputs of death.
Patrice Godefroid, Michael Levin, David Molnar,
Automated white-box fuzz testing.
Additionally: Discuss class projects.
Monday February 4
Data race detection.
Stefan Savage et al. Eraser:
A Dynamic Data Race Detector for Multithreaded Programs.
Jong-Deok Choi et al. Efficient and
Precise Datarace Detection for Multithreaded Object-Oriented
Programs.
Wednesday February 6
Atomicity.
Cormac Flanagan and Shaz Qadeer. A Type and
Effect System for Atomicity.
Cormac Flanagan and Stephen Freund. Atomizer:
A Dynamic Atomicity Checker for Multithreaded Programs
Monday February 11
Liveness violations.
Charles Killian, James Anderson, Ranjit Jhala, and
Amin Vahdat. Life, death, and the
critical transition: finding liveness bugs in systems code.
Rupak will lead discussions on class projects.
Wednesday February 13
Testing systems that interact with database systems.
Readings:
David Chays et al. An AGENDA for
testing relational database applications.
Michael Emmi, Rupak Majumdar, and Koushik Sen.
Dynamic test input generation for database appliications.
HOMEWORK 1
: Use this template to write a precise
description of your project. Please submit by Tuesday February 19.
Monday February 18
Holiday.
Wednesday February 20
Grammar based whitebox testing.
Readings:
Rupak Majumdar and Ru-Gang Xu. Directed test generation
with symbolic grammars.
Patrice Godefroid, Adam Kiezun, and Michael Y. Levin.
Grammar
based whitebox fuzzing.
Monday February 25
Static and dynamic detection of invariants.
Readings:
Michael Ernst et al.
The Daikon system for dynamic detection of likely invariants.
Michael Ernst et al.
Dynamically discovering likely program invariants to support program evolution.
Wednesday February 27
Students present their project ideas.
Monday March 3
Delta debugging: mapping an error to smallest program changes that cause it.
Simplifying and Isolating Failure-Inducing Input. Andreas
Zeller and Ralf Hildebrandt; IEEE Transactions on Software Engineering 28(2), February 2002, pp. 183-200.
Isolating Failure-Inducing Thread Schedules.
Jong-Deok Choi and Andreas Zeller.
Wednesday March 5
Students present their project ideas.
Monday March 10
Statistical debugging.
Reading:
Ben Liblit et al. Bug isolation by remote program sampling.
Wednesday March 12
Students present their projects.
Also, Homework 2 is due.