staproject Swiki

Reports

TODO List

• Install tools (make notes about ease of installation)
• Check wheher these tools can generate tests against bytecode instrumented by InsECT http://www.cc.gatech.edu/~orso/software.html; if so, we can feed to these tools with instrumented bytecode that collects structural coverage on the fly during test generation and execution.
• Double-check these tools against the features of argument generaiton and sequence generation
• Double-check these tools against the output formats of their generated tests

Available tools

• (NCSU) Parasoft Jtest: a commercial tool (license available to Tao) http://www.parasoft.com/
• (NCSU) Rostra: a conrete-state-exploration tool available to Tao http://www.csc.ncsu.edu/faculty/xie/
• (NCSU) Symstra: a symbolic-state-exploration tool implemented upon JPF http://www.csc.ncsu.edu/faculty/xie/
• (NCSU) Eclat: a random testing tool http://pag.csail.mit.edu/eclat/
• (NCSU) RUTEj: a random method sequence tool http://www.csd.uwo.ca/People/gradstudents/leiyong/rute/index.html
• (GT, NCSU) jCute: a testing tool based on concolic execution http://osl.cs.uiuc.edu/~ksen/cute/
• (GT) JCrasher: a robustness testing tool http://www.cc.gatech.edu/jcrasher/
• (GT) CnC: a robustness testing tool http://www.cc.gatech.edu/cnc/
• (GT) eToc: a tool for evolutionary testing of classes. http://star.itc.it/etoc/
• (GT) Java PathFinder: a Java model checker and used Saswat's instrumenter http://javapathfinder.sourceforge.net/
• (requires sequence specifications) JMLTT: a JML-based tool http://lifc.univ-fcomte.fr/~jmltt/
• (requires sequence specifications) UTJML: a JML-based tool http://opuntia.cs.utep.edu/ssv/utjml.html
• (no generation) Saswat's tool: a symbolic execution tool available to Saswat http://www.cc.gatech.edu/~saswat/
• (no generation) Juzi: Sarfraz Khurshid's symbolic execution tool (Saswat has source code)
• (require human involvment) Agitar Agitator: a commercial tool (license to be requested by Tao) http://www.agitar.com/

Categories

• [Inputs] Analyze only source code:
• [Inputs] Analyze instrumented bytecode: the rest
• [Generation] Generate method arguments only: JCrasher, CnC
• [Generation] Generate method sequences only: Rostra, RUTEj, Java Pathfinder
• [Generation] Generate both method arguments and method sequences: Parasoft Jtest, Agitar Agitator, Eclat, jCUTE, eToc, Symstra
• Generation Parameters such as Rostra's iteration number
• [Outputs] JUnit Test Classes: JCrasher, CnC, Rostra, Parasoft Jtest, Eclat, eToc, jCUTE
• [Outputs] Textual Test Forms: RUTEj, Java Pathfinder(?), Symstra (JPF implementation?)
• [Outputs] Internal Test Forms: Agitar Agitator(JUnit code generation to be added)

Java Benchmarks

• ASE 04 Rostra benchmarks http://www.csc.ncsu.edu/faculty/xie/straw/ase04subjects.zip
• TACAS 05 Symstra benchmarks (a subset of Rostra benchmarks) http://www.csc.ncsu.edu/faculty/xie/straw/symexamples.zip
• UNL Subject Infrastructure Repository http://cse.unl.edu/~galileo/php/sir/index.php
• Stanford SecuriBench http://suif.stanford.edu/~livshits/securibench/
• GMU Software Engineering Experiments on the Web http://ise.gmu.edu:8080/ofut/jsp/seeweb/index.jsp
• UMD Java GUI benchmarks http://www.cs.umd.edu/~atif/newsite/benchmarks.htm
• Can we search those published papers in Java testing tools to see what benchmarks they used? Some authors may have their benchmarks available on the web or we can request them from authors. Some available tools above may include the benchmarks they used in the released tool package.

Evaluation Criteria

• Structural coverage (branch coverage InsECT by Orso http://www.cc.gatech.edu/~orso/software.html, Jusc by Tao http://www.csc.ncsu.edu/faculty/xie/jusc/, and intraprocedurual dataflow coverage: http://coverlipse.sourceforge.net/index.php)
• Real time spent on test generation/execution
• Fault detection capability. Using real bugs? Or using mutation testing tools like muJava at GMU http://www.isse.gmu.edu/faculty/ofut/mujava/?
• #tests, #non-redundant tests
• Ease of installation; ease of use
• When relying on thrown exceptions, #false positives when some test inputs are illegal and people don't care
• Types of programs that can be handled. For example, support for standard Java Libraries?
• Convenient tool for bytecode decompliation http://www.kpdus.com/jad.html

Publications on comparing tools

• A Comparison of Bug Finding Tools for Java by Nick Rutar, Christian Almazan, and Jeff Foster in ISSRE 04 http://www.cs.umd.edu/~jfoster/papers/issre04.pdf
• Comparing Bug Finding Tools with Reviews and Tests by Wagner et al. in TestCom 05 http://www4.in.tum.de/publ/papers/SWJJCKPT05.pdf
• Using Exploitable Buffer Overflows From Open Source Code by Misha Zitser, Richard Lippmann, and Tim Leek in FSE 04 http://www.ll.mit.edu/IST/corpora.html
• 2005 Workshop on the Evaluation of Software Defect Detection Tools http://www.cs.umd.edu/~pugh/BugWorkshop05/
• How to build better benchmarks http://calla.ics.uci.edu/serum/moin.cgi/Benchmarks
• Software Testing Research Surveys http://www.csc.ncsu.edu/faculty/xie/testingresearchsurvey.htm
• Software Testing Researchers http://www.csc.ncsu.edu/faculty/xie/testingresearchers.htm