| 1 | # DataRaceBench 1.3.2
|
|---|
| 2 |
|
|---|
| 3 | DataRaceBench is a benchmark suite designed to systematically and
|
|---|
| 4 | quantitatively evaluate the effectiveness of data race detection
|
|---|
| 5 | tools. It includes a set of microbenchmarks with and without data
|
|---|
| 6 | races. Parallelism is represented by OpenMP directives. OpenMP is a
|
|---|
| 7 | popular parallel programming model for multi-threaded applications.
|
|---|
| 8 |
|
|---|
| 9 | Note that some microbenchmarks use OpenMP 4.5 and 5.0 features.
|
|---|
| 10 | You may need a recent OpenMP compiler to compile them.
|
|---|
| 11 |
|
|---|
| 12 | DataRaceBench also comes with an evaluation script
|
|---|
| 13 | (check-data-races.sh). The script can be used to evaluate the tools
|
|---|
| 14 | Helgrind, Archer, Thread Sanitizer, and Intel Inspector. In addition a
|
|---|
| 15 | parameterized test harness (scripts/test-harness.sh) is available
|
|---|
| 16 | which allows to provide a number of different parameters for the
|
|---|
| 17 | evaluation. The test harness is used by the evaluation script with
|
|---|
| 18 | some pre-defined values.
|
|---|
| 19 |
|
|---|
| 20 | ## Quick Start
|
|---|
| 21 |
|
|---|
| 22 | ```
|
|---|
| 23 | ./check-data-races.sh
|
|---|
| 24 |
|
|---|
| 25 | Usage: ./check-data-races.sh [--run] [--help] language
|
|---|
| 26 |
|
|---|
| 27 | --help : this option
|
|---|
| 28 | --small : compile and test all benchmarks using small parameters with Helgrind, ThreadSanitizer, Archer, Intel inspector.
|
|---|
| 29 | --run : compile and run all benchmarks with gcc (no evaluation)
|
|---|
| 30 | --run-intel : compile and run all benchmarks with Intel compilers (no evaluation)
|
|---|
| 31 | --helgrind : compile and test all benchmarks with Helgrind
|
|---|
| 32 | --tsan-clang: compile and test all benchmarks with clang ThreadSanitizer
|
|---|
| 33 | --tsan-gcc : compile and test all benchmarks with gcc ThreadSanitizer
|
|---|
| 34 | --archer : compile and test all benchmarks with Archer
|
|---|
| 35 | --coderrect : compile and test all benchmarks with Coderrect Scanner
|
|---|
| 36 | --inspector : compile and test all benchmarks with Intel Inspector
|
|---|
| 37 | --romp : compile and test all benchmarks with Romp
|
|---|
| 38 | --customize : compile and test customized test list and tools
|
|---|
| 39 | ```
|
|---|
| 40 |
|
|---|
| 41 | More information: [User Guide](https://github.com/LLNL/dataracebench/blob/master/user_guide.md)
|
|---|
| 42 |
|
|---|
| 43 | ## Latest Tool Evaluation Results
|
|---|
| 44 | [Data race detection tool dashboard](https://github.com/LLNL/dataracebench/wiki/Tool-Evaluation-Dashboard)
|
|---|
| 45 |
|
|---|
| 46 | ## List of Benchmarks
|
|---|
| 47 |
|
|---|
| 48 | [Benchmark labels and lists - C/C++](https://github.com/LLNL/dataracebench/blob/master/benchmarkList.md)
|
|---|
| 49 |
|
|---|
| 50 | [Benchmark labels and lists - Fortran](https://github.com/LLNL/dataracebench/blob/master/benchmarkListFortran.md)
|
|---|
| 51 |
|
|---|
| 52 | ## Authors
|
|---|
| 53 |
|
|---|
| 54 | DataRaceBench was created by Chunhua Liao, Pei-Hung Lin, Gaurav Verma, Yaying Shi, Joshua Asplund, Markus Schordan, and Ian Karlin.
|
|---|
| 55 |
|
|---|
| 56 | ## Release
|
|---|
| 57 |
|
|---|
| 58 | DataRaceBench is released under a BSD license. For more details see
|
|---|
| 59 | the file LICENSE.txt. The microbenchmarks marked 'Polyhedral' in above
|
|---|
| 60 | table were generated as optimization variants of benchmarks from the
|
|---|
| 61 | PolyOpt benchmark suite. For those benchmarks see the license file
|
|---|
| 62 | LICENSE.OSU.txt.
|
|---|
| 63 |
|
|---|
| 64 | `LLNL-CODE-732144`
|
|---|
| 65 |
|
|---|
| 66 | ## How to Cite DataRaceBench in a Publication
|
|---|
| 67 |
|
|---|
| 68 | If you are referring to DataRaceBench in a publication, please cite the following paper:
|
|---|
| 69 |
|
|---|
| 70 | * Chunhua Liao, Pei-Hung Lin, Joshua Asplund, Markus Schordan, Ian Karlin.
|
|---|
| 71 | [DataRaceBench: A Benchmark Suite for Systematic Evaluation of Data Race Detection Tools (best paper finalist)](https://dl.acm.org/citation.cfm?doid=3126908.3126958).
|
|---|
| 72 | Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017, pp. 11:1-11:14, ISBN 978-1-4503-5114-0, Denver, CO, USA, November 12-17, 2017. [pdf](https://github.com/LLNL/dataracebench/blob/master/docs/DataRaceBench-SC17.pdf)
|
|---|
| 73 |
|
|---|
| 74 | If you use DataRaceBench v.1.3.0 or later, please additionally cite the following paper:
|
|---|
| 75 | * Gaurav Verma, Yaying Shi, Chunhua Liao, Barbara Chapman, and Yonghong Yan, Enhancing DataRaceBench for Evaluating DataRace Detection Tools, International Workshop on Software Correctness for HPC Applications (Correctness) SC20, 2020
|
|---|
| 76 |
|
|---|
| 77 | Other papers
|
|---|
| 78 | * Pei-Hung Lin, Chunhua Liao, Markus Schordan, Ian Karlin. [Exploring Regression of Data Race Detection Tools Using DataRaceBench](https://ieeexplore.ieee.org/abstract/document/8951036). 2019 IEEE/ACM 3rd International Workshop on Software Correctness for HPC Applications (Correctness), Denver, CO, USA, 2019, pp. 11-18. [pdf](https://github.com/LLNL/dataracebench/blob/master/docs/ExploringRegressionOfDataRaceDetectionTools-SC19.pdf), [Presentation](https://github.com/LLNL/dataracebench/blob/master/docs/2019-11-18-RegressionOfDataRaceTools-SC19.pdf)
|
|---|
| 79 | * Chunhua Liao, Pei-Hung Lin, Markus Schordan and Ian Karlin, [A Semantics-Driven Approach to Improving DataRaceBench's OpenMP Standard Coverage](https://www.springerprofessional.de/en/a-semantics-driven-approach-to-improving-dataracebench-s-openmp-/16134302), IWOMP 2018: 14th International Workshop on OpenMP, Barcelona, Spain, September 26-28, 2018, [pdf](https://github.com/LLNL/dataracebench/blob/master/docs/Semantics-DrivenImprovingCoverage-IWOMP2018.pdf)
|
|---|
| 80 | * Pei-Hung Lin, Chunhua Liao, Markus Schordan, Ian Karlin. [Runtime and Memory Evaluation of Data Race Detection Tools](https://link.springer.com/chapter/10.1007/978-3-030-03421-4_13). ISoLA (2) 2018: 179-196. [pdf](https://github.com/LLNL/dataracebench/blob/master/docs/2018-Runtime-Memory-Evaluation-DataRaceBench.pdf)
|
|---|
