key: cord-0048634-2bqcq5sb
authors: Beyer, Dirk
title: Second Competition on Software Testing: Test-Comp 2020
date: 2020-03-13
journal: Fundamental Approaches to Software Engineering
DOI: 10.1007/978-3-030-45234-6_25
sha: bd9069be4b4bd36bc36063aab578ef8321af6355
doc_id: 48634
cord_uid: 2bqcq5sb

This report describes the 2020 Competition on Software Testing (Test-Comp), the 2[Formula: see text] edition of a series of comparative evaluations of fully automatic software test-case generators for C programs. The competition provides a snapshot of the current state of the art in the area, and has a strong focus on replicability of its results. The competition was based on 3 230 test tasks for C programs. Each test task consisted of a program and a test specification (error coverage, branch coverage). Test-Comp 2020 had 10 participating test-generation systems.

Software testing is as old as software development itself, because the most straightforward way to find out if the software works is to execute it. In the last few decades the tremendous breakthrough of fuzzers 1 , theorem provers [40] , and satisfiability-modulo-theory (SMT) solvers [21] have led to the development of efficient tools for automatic test-case generation. For example, symbolic execution and the idea to use it for test-case generation [33] exists for more than 40 years, yet, efficient implementations (e.g., Klee [16] ) had to wait for the availability of mature constraint solvers. Also, with the advent of automatic software model checking, the opportunity to extract test cases from counterexamples arose (see Blast [9] and JPF [41] ). In the following years, many techniques from the areas of model checking and program analysis were adapted for the purpose of test-case generation and several strong hybrid combinations have been developed [24] .

There are several powerful software test generators available [24] , but they were difficult to compare. For example, a recent study [11] first had to develop a framework that supports to run test-generation tools on the same program source code and to deliver test cases in a common format for validation. Furthermore, there was no widely distributed benchmark suite available and neither input programs nor output test suites followed a standard format. In software verification, the competition SV-COMP [3] helped to overcome the problem: the competition community developed standards for defining nondeterministic functions and a language to write specifications (so far for C and Java programs) and established a standard exchange format for the output (witnesses). A competition event with high visibility can foster the transfer of theoretical and conceptual advancements in the area of software testing into practical tools.

The annual Competition on Software Testing (Test-Comp) [4, 5] 2 is the showcase of the state of the art in the area, in particular, of the effectiveness and efficiency that is currently achieved by tool implementations of the most recent ideas, concepts, and algorithms for fully automatic test-case generation. Test-Comp uses the benchmarking framework BenchExec [12] , which is already successfully used in other competitions, most prominently, all competitions that run on the StarExec infrastructure [39] . Similar to SV-COMP, the test generators in Test-Comp are applied to programs in a fully automatic way. The results are collected via BenchExec's XML results format, and transformed into tables and plots in several formats. 3 All results are available in artifacts at Zenodo (Table 3) .

Competition Goals. In summary, the goals of Test-Comp are the following:

• Establish standards for software test generation. This means, most prominently, to develop a standard for marking input values in programs, define an exchange format for test suites, and agree on a specification language for test-coverage criteria, and define how to validate the resulting test suites. • Establish a set of benchmarks for software testing in the community. This means to create and maintain a set of programs together with coverage criteria, and to make those publicly available for researchers to be used in performance comparisons when evaluating a new technique. • Provide an overview of available tools for test-case generation and a snapshot of the state-of-the-art in software testing to the community. This means to compare, independently from particular paper projects and specific techniques, different test-generation tools in terms of effectiveness and performance. • Increase the visibility and credits that tool developers receive. This means to provide a forum for presentation of tools and discussion of the latest technologies, and to give the students the opportunity to publish about the development work that they have done. [28] ). An overview of 16 competitions in the area of formal methods was presented at the TOOLympics events at the conference TACAS in 2019 [1] . In software testing, there are several competition-like events, for example, the DARPA Cyber Grand Challenge [38] 4 , the IEEE International Contest on Software Testing 5 , the Software Testing World Cup 6 , and the Israel Software Testing World Cup 7 . Those contests are organized as on-site events, where teams of people interact with certain testing platforms in order to achieve a certain coverage of the software under test. There are two competitions for automatic and off-site testing: Rode0day 8 is a competition that is meant as a continuously running evaluation on bug-finding in binaries (currently Grep and SQLite). The unit-testing tool competition [32] 9 is part of the SBST workshop and compares tools for unit-test generation on Java programs. There was no comparative evaluation of automatic test-generation tools for whole C programs in source-code, in a controlled environment, and Test-Comp was founded to close this gap [4] . The results of the first edition of Test-Comp were presented as part of the TOOLympics 2019 event [1] and in the Test-Comp 2019 competition report [5] .

Organizational aspects such as the classification (automatic, off-site, reproducible, jury, traning) and the competition schedule is given in the initial competition definition [4] . In the following we repeat some important definitions that are necessary to understand the results.

Test Task. A test task is a pair of an input program (program under test) and a test specification. A test run is a non-interactive execution of a test generator on a single test task, in order to generate a test suite according to the test specification. A test suite is a sequence of test cases, given as a directory of files according to the format for exchangeable test-suites. 10 Execution of a Test Generator. Figure 1 illustrates the process of executing one test generator on the benchmark suite. One test run for a test generator gets as input (i) a program from the benchmark suite and (ii) a test specification (find bug, or coverage criterion), and returns as output a test suite (i.e., a set of test cases Test Specification. The specification for testing a program is given to the test generator as input file (either properties/coverage-error-call.prp or properties/coverage-branches.prp for Test-Comp 2020). The definition init(main()) is used to define the initial states of the program under test by a call of function main (with no parameters). The definition FQL(f) specifies that coverage definition f should be achieved. The FQL (FShell query language [26] ) coverage definition COVER EDGES(@DECISIONEDGE) means that all branches should be covered, COVER EDGES(@BASICBLOCKENTRY) means that all statements should be covered, and COVER EDGES(@CALL(__VERIFIER_error)) means that calls to function __VERIFIER_error should be covered. A complete specification looks like: COVER( init(main()), FQL(COVER EDGES(@DECISIONEDGE)) ). Table 1 lists the two FQL formulas that are used in test specifications of Test-Comp 2020; there was no change from 2019. The first describes a formula that is typically used for bug finding: the test generator should find a test case that executes a certain error function. The second describes a formula that is used to obtain a standard test suite for quality assurance: the test generator should find a test suite for branch coverage.

License and Qualification. The license of each participating test generator must allow its free use for replication of the competition experiments. Details on qualification criteria can be found in the competition report of Test-Comp 2019 [5] .

Benchmark Programs. The input programs were taken from the largest and most diverse open-source repository of software verification tasks 12 , which is also used by SV-COMP [3] . As in 2019, we selected all programs for which the following properties were satisfied (see issue on GitHub 13 and report [5] ):

1. compiles with gcc, if a harness for the special methods 14 is provided, 2. should contain at least one call to a nondeterministic function, 3 Tables 6 and 7 and described in detail on the competition web site. 15 Figure 2 illustrates the category composition.

Category Error-Coverage. The first category is to show the abilities to discover bugs. The programs in the benchmark set contain programs that contain a bug. Every run will be started by a batch script, which produces for every tool and every test task (a C program together with the test specification) one of the following scores: 1 point, if the validator succeeds in executing the program under test on a generated test case that explores the bug (i.e., the specified function was called), and 0 points, otherwise.

Category Branch-Coverage. The second category is to cover as many branches of the program as possible. The coverage criterion was chosen because many test-generation tools support this standard criterion by default. Other coverage criteria can be reduced to branch coverage by transformation [25] . Every run will be started by a batch script, which produces for every tool and every test task (a C program together with the test specification) the coverage of branches of the program (as reported by TestCov [14] ; a value between 0 and 1) that are executed for the generated test cases. The score is the returned coverage.

Ranking. The ranking was decided based on the sum of points (normalized for meta categories). In case of a tie, the ranking was decided based on the run time, which is the total CPU time over all test tasks. Opt-out from categories was possible and scores for categories were normalized based on the number of tasks per category (see competition report of SV-COMP 2013 [2] , page 597). 

In order to support independent replication of the Test-Comp experiments, we made all major components that are used for the competition available in public version repositories. An overview of the components that contribute to the reproducible setup of Test-Comp is provided in Fig. 3 , and the details are given in Table 2 . We refer to the report of Test-Comp 2019 [5] for a thorough description of all components of the Test-Comp organization and how we ensure that all parts are publicly available for maximal replicability.

In order to guarantee long-term availability and immutability of the testgeneration tasks, the produced competition results, and the produced test suites, we also packaged the material and published it at Zenodo. The DOIs and references are listed in Table 3 . The archive for the competition results includes the raw results in BenchExec's XML exchange format, the log output of the test generators and validator, and a mapping from files names to SHA-256 hashes. The hashes of the files are useful for validating the exact contents of a file, and accessing the files inside the archive that contains the test suites.

To provide transparent access to the exact versions of the test generators that were used in the competition, all tester archives are stored in a public Git repository. GitLab was used to host the repository for the tester archives due to its generous repository size limit of 10 GB. The final size of the Git repository is 1.47 GB. 

For the second time, the competition experiments represent the state of the art in fully automatic test-generation for whole C programs. The report helps in understanding the improvements compared to last year, in terms of effectiveness (test coverage, as accumulated in the score) and efficiency (resource consumption in terms of CPU time). All results mentioned in this article were inspected and approved by the participants.

Participating Test Generators. Table 4 provides an overview of the participating test-generation systems and references to publications, as well as the team representatives of the jury of Test-Comp 2020. (The competition jury consists of the chair and one member of each participating team.) Table 5 lists the features and technologies that are used in the test-generation tools. An online table with information about all participating systems is provided on the competition web site. 16 Computing Resources. The computing environment and the resource limits were mainly the same as for Test-Comp 2019 [5] : Each test run was limited to 8 processing units (cores), 15 GB of memory, and 15 min of CPU time. The testsuite validation was limited to 2 processing units, 7 GB of memory, and 5 h of CPU time (was 3 h for Test-Comp 2019). The machines for running the experiments are part of a compute cluster that consists of 168 machines; each test-generation run was executed on an otherwise completely unloaded, dedicated machine, in order [13] (integrated in BenchExec [12] ). Further technical parameters of the competition machines are available in the repository that also contains the benchmark definitions. 18 One complete test-generation execution of the competition consisted of 29 899 single test-generation runs. The total CPU time was 178 days and the consumed energy 49.9 kWh for one complete competition run for test-generation (without validation). Test-suite validation consisted of 29 899 single test-suite More information (including interactive tables, quantile plots for every category, and also the raw data in XML format) is available on the competition web site 19 and in the results artifact (see Table 3 ). Table 7 reports the top three testers for each category. The consumed run time (column 'CPU Time') is given in hours and the consumed energy (column 'Energy') is given in kWh.

Score-Based Quantile Functions for Quality Assessment. We use scorebased quantile functions [12] because these visualizations make it easier to understand the results of the comparative evaluation. The web site 19 and the More details were given previously [5] . A logarithmic scale is used for the time range from 1 s to 1000 s, and a linear scale is used for the time range between 0 s and 1 s. results artifact (Table 3) include such a plot for each category; as example, we show the plot for category Overall (all test tasks) in Fig. 4 . A total of 9 testers (all except Esbmc) participated in category Overall, for which the quantile plot shows the overall performance over all categories (scores for meta categories are normalized [2] ). A more detailed discussion of score-based quantile plots for testing is provided in the previous competition report [5] .

Alternative Ranking: Green Test Generation -Low Energy Consumption. Since a large part of the cost of test-generation is caused by the energy consumption, it might be important to also consider the energy efficiency in rankings, as complement to the official Test-Comp ranking. The energy is measured using CPU Energy Meter [13] , which we use as part of BenchExec [12] . Table 8 is similar to Table 7 , but contains the alternative ranking category Green Testers. Column 'Quality' gives the score in score points, column 'CPU Time' the CPU usage in hours, column 'CPU Energy' the CPU usage in kWh, column 'Rank Measure' uses the energy consumption per score point as rank measure: total CPU energy total score , with the unit J/sp.

Test-Comp 2020, the 2 nd edition of the Competition on Software Testing, attracted 10 participating teams. The competition offers an overview of the state of the art in automatic software testing for C programs. The competition does not only execute the test generators and collect results, but also validates the achieved coverage of the test suites, based on the latest version of the test-suite validator TestCov. The number of test tasks was increased to 3 230 (from 2 356 in Test-Comp 2019). As before, the jury and the organizer made sure that the competition follows the high quality standards of the FASE conference, in particular with respect to the important principles of fairness, community support, and transparency.

TOOLympics 2019: An overview of competitions in formal methods

Second competition on software verification (Summary of SV-COMP 2013)

Automatic verification of C and Java programs: SV-COMP 2019

Competition on software testing (Test-Comp). In: Proc. TACAS (3)

First international competition on software testing

Results of the 2nd International Competition on Software Testing (Test-Comp 2020). Zenodo

SV-Benchmarks: Benchmark set of the 2nd Intl. Competition on Software Testing (Test-Comp 2020). Zenodo

Test suites from Test-Comp 2020 test-generation tools

Generating tests from counterexamples

CoVeriTest: Cooperative verifier-based testing

Software verification: Testing vs. model checking

Reliable benchmarking: Requirements and solutions

CPU Energy Meter: A tool for energy-aware algorithms engineering

TestCov: Robust test-suite execution and coverage measurement

Facilitating reuse in multi-goal test-suite generation for software product lines

Klee: Unassisted and automatic generation of high-coverage tests for complex systems programs

Klee symbolic execution engine (competition contribution)

Symbiotic 6: Generating test-cases (competition contribution)

Joint forces for memory safety checking

VeriFuzz: Program-aware fuzzing (competition contribution)

The 2014 SMT competition

Esbmc: Scalable and precise test generation based on the floating-point theory (competition contribution)

Handling loops in bounded model checking of C programs via k -induction

Combining model checking and testing

Testability transformation

How did you specify your test suite

Rigorous examination of reactive systems. The RERS challenges 2012 and 2013

VerifyThis 2012: A program verification competition

TracerX: Dynamic symbolic execution with interpolation (competition contribution)

Tracer: A symbolic execution tool for verification

CoVeriTest with dynamic partitioning of the iteration time limit (competition contribution)

Java unit-testing tool competition: Seventh round

Symbolic execution and program testing

Llvm-based hybrid fuzzing with LibKluzzer (competition contribution)

Plain random test generation with PRTest (competition contribution)

Legion: Best-first concolic testing (competition contribution)

HybridTiger: Hybrid model checking and domination-based partitioning for efficient multi-goal test-suite generation (competition contribution)

The DARPA cyber grand challenge: A competitor's perspective, part 2

StarExec: A cross-community infrastructure for logic solving

The CADE ATP system competition: CASC. AI Magazine

Test-input generation with Java PathFinder

), which permits use, sharing, adaptation, distribution, and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use