| 1 | # DataRaceBench 1.2.0
|
|---|
| 2 |
|
|---|
| 3 | DataRaceBench is a benchmark suite designed to systematically and
|
|---|
| 4 | quantitatively evaluate the effectiveness of data race detection
|
|---|
| 5 | tools. It includes a set of microbenchmarks with and without data
|
|---|
| 6 | races. Parallelism is represented by OpenMP directives. OpenMP is a
|
|---|
| 7 | popular parallel programming model for multi-threaded applications.
|
|---|
| 8 |
|
|---|
| 9 | Note that some microbenchmarks use OpenMP 4.5 features. Those are:
|
|---|
| 10 | DRB094-doall2-ordered-orig-no.c
|
|---|
| 11 | DRB095-doall2-taskloop-orig-yes.c (requires gcc 7.x)
|
|---|
| 12 | DRB096-doall2-taskloop-collapse-orig-no.c (requires gcc 7.x)
|
|---|
| 13 | DRB100-task-reference-orig-no.cpp
|
|---|
| 14 | DRB112-linear-orig-no.c
|
|---|
| 15 | You need a recent OpenMP compiler (e.g. gcc 7.x or later) to compile them.
|
|---|
| 16 |
|
|---|
| 17 | DataRaceBench also comes with an evaluation script
|
|---|
| 18 | (check-data-races.sh). The script can be used to evaluate the tools
|
|---|
| 19 | Helgrind, Archer, Thread Sanitizer, and Intel Inspector. In addition a
|
|---|
| 20 | parameterized test harness (scripts/test-harness.sh) is available
|
|---|
| 21 | which allows to provide a number of different parameters for the
|
|---|
| 22 | evaluation. The test harness is used by the evaluation script with
|
|---|
| 23 | some pre-defined values.
|
|---|
| 24 |
|
|---|
| 25 | ## Latest tool evaluation results
|
|---|
| 26 | [Data race detection tool regression evaluation](https://github.com/LLNL/dataracebench/wiki/Regression-metrics)
|
|---|
| 27 |
|
|---|
| 28 | ## Microbenchmark property labels (P-Labels)
|
|---|
| 29 |
|
|---|
| 30 | P-Label | Meaning (microbenchmarks with data races) | P-Label | Meaning (microbenchmarks without data races)
|
|---|
| 31 | ------|-----------------------------------|------|------------------------------
|
|---|
| 32 | Y1 | Unresolvable dependences | N1 | Embarrassingly parallel or single thread execution
|
|---|
| 33 | Y2 | Missing data sharing clauses | N2 | Use of data sharing clauses
|
|---|
| 34 | Y3 | Missing synchronization | N3 | Use of synchronization
|
|---|
| 35 | Y4 | SIMD data races | N4 | Use of SIMD directives
|
|---|
| 36 | Y5 | Accelerator data races | N5 | Use of accelerator directives
|
|---|
| 37 | Y6 | Undefined behavior | N6 | Use of special language features
|
|---|
| 38 | Y7 | Numerical kernel data races | N7 | Numerical kernels
|
|---|
| 39 |
|
|---|
| 40 | ## Microbenchmarks with known data races (some have a varying length version)
|
|---|
| 41 |
|
|---|
| 42 | Microbenchmark |P-Label| Description | Source
|
|---|
| 43 | ----------------------------------------------------|-------|------------------------------------------------------------------------------|----------
|
|---|
| 44 | DRB001-antidep1-orig-yes.c |Y1 | Anti-dependence within a single loop | AutoPar
|
|---|
| 45 | DRB002-antidep1-var-yes.c |Y1 | Anti-dependence within a single loop | AutoPar
|
|---|
| 46 | DRB003-antidep2-orig-yes.c |Y1 | Anti-dependence within a two-level loop nest | AutoPar
|
|---|
| 47 | DRB004-antidep2-var-yes.c |Y1 | Anti-dependence within a two-level loop nest | AutoPar
|
|---|
| 48 | DRB005-indirectaccess1-orig-yes.c |Y7 | Indirect access with overlapped index array elements | LLNL App
|
|---|
| 49 | DRB006-indirectaccess2-orig-yes.c |Y7 | Overlapping index array elements when 36 or more threads are used | LLNL App
|
|---|
| 50 | DRB007-indirectaccess3-orig-yes.c |Y7 | Overlapping index array elements when 60 or more threads are used | LLNL App
|
|---|
| 51 | DRB008-indirectaccess4-orig-yes.c |Y7 | Overlapping index array elements when 180 or more threads are used | LLNL App
|
|---|
| 52 | DRB009-lastprivatemissing-orig-yes.c |Y2 | Data race due to a missing `lastprivate()` clause | AutoPar
|
|---|
| 53 | DRB010-lastprivatemissing-var-yes.c |Y2 | Data race due to a missing `lastprivate()` clause | AutoPar
|
|---|
| 54 | DRB011-minusminus-orig-yes.c |Y3 | Unprotected decrement operation `--` | AutoPar
|
|---|
| 55 | DRB012-minusminus-var-yes.c |Y3 | Unprotected decrement operation `--` | AutoPar
|
|---|
| 56 | DRB013-nowait-orig-yes.c |Y3 | Missing barrier due to a wrongfully used nowait | AutoPar
|
|---|
| 57 | DRB014-outofbounds-orig-yes.c |Y6 | Out of bound access of the 2nd dimension of array | AutoPar
|
|---|
| 58 | DRB015-outofbounds-var-yes.c |Y6 | Out of bound access of the 2nd dimension of array | AutoPar
|
|---|
| 59 | DRB016-outputdep-orig-yes.c |Y1 | Output dependence and true dependence within a loop | AutoPar
|
|---|
| 60 | DRB017-outputdep-var-yes.c |Y1 | Output dependence and true dependence within a loop | AutoPar
|
|---|
| 61 | DRB018-plusplus-orig-yes.c |Y1 | increment operation `++` on array index variable | AutoPar
|
|---|
| 62 | DRB019-plusplus-var-yes.c |Y1 | increment operation `++` on array index variable | AutoPar
|
|---|
| 63 | DRB020-privatemissing-orig-yes.c |Y2 | Missing `private()` for a temp variable | AutoPar
|
|---|
| 64 | DRB021-privatemissing-var-yes.c |Y2 | Missing `private()` for a temp variable | AutoPar
|
|---|
| 65 | DRB022-reductionmissing-orig-yes.c |Y2 | Missing `reduction()` for a variable | AutoPar
|
|---|
| 66 | DRB023-reductionmissing-var-yes.c |Y2 | Missing `reduction()` for a variable | AutoPar
|
|---|
| 67 | DRB024-sections1-orig-yes.c |Y3 | Unprotected data writes in parallel sections | New
|
|---|
| 68 | DRB025-simdtruedep-orig-yes.c |Y1,Y4 | SIMD instruction level data races | New
|
|---|
| 69 | DRB026-simdtruedep-var-yes.c |Y1,Y4 | SIMD instruction level data races | New
|
|---|
| 70 | DRB027-targetparallelfor-orig-yes.c |Y1,Y5 | Data races in loops offloaded to accelerators | New
|
|---|
| 71 | DRB028-taskdependmissing-orig-yes.c |Y3 | Unprotected data writes in two tasks | New
|
|---|
| 72 | DRB029-truedep1-orig-yes.c |Y1 | True data dependence among multiple array elements within a single level loop| AutoPar
|
|---|
| 73 | DRB030-truedep1-var-yes.c |Y1 | True data dependence among multiple array elements within a single level loop| AutoPar
|
|---|
| 74 | DRB031-truedepfirstdimension-(orig|var)-yes.c |Y1 | True data dependence of first dimension for a 2-D array accesses | AutoPar
|
|---|
| 75 | DRB032-truedepfirstdimension-(orig|var)-yes.c |Y1 | True data dependence of first dimension for a 2-D array accesses | AutoPar
|
|---|
| 76 | DRB033-truedeplinear-orig-yes.c |Y1 | Linear equation as array subscript | AutoPar
|
|---|
| 77 | DRB034-truedeplinear-var-yes.c |Y1 | Linear equation as array subscript | AutoPar
|
|---|
| 78 | DRB035-truedepscalar-orig-yes.c |Y1 | True data dependence due to scalar | AutoPar
|
|---|
| 79 | DRB036-truedepscalar-var-yes.c |Y1 | True data dependence due to scalar | AutoPar
|
|---|
| 80 | DRB037-truedepseconddimension-(orig|var)-yes.c |Y1 | True data dependence on 2nd dimension of a 2-D array accesses | AutoPar
|
|---|
| 81 | DRB038-truedepseconddimension-(orig|var)-yes.c |Y1 | True data dependence on 2nd dimension of a 2-D array accesses | AutoPar
|
|---|
| 82 | DRB039-truedepsingleelement-(orig|var)-yes.c |Y1 | True data dependence due to a single array element | AutoPar
|
|---|
| 83 | DRB040-truedepsingleelement-(orig|var)-yes.c |Y1 | True data dependence due to a single array element | AutoPar
|
|---|
| 84 | DRB073-doall2-orig-yes.c |Y2 | Missing `private()` for inner loop nest's loop index variable | New
|
|---|
| 85 | DRB074-flush-orig-yes.c |Y2 | Reduction using a shared variable, extracted from an official OpenMP example | New
|
|---|
| 86 | DRB075-getthreadnum-orig-yes.c |Y1 | Work sharing within one branch of a `if` statement | New
|
|---|
| 87 | DRB080-func-arg-orig-yes.c |Y6 | Function arguments passed by reference, inheriting shared attribute | New
|
|---|
| 88 | DRB082-declared-in-func-orig-yes.c |Y6 | A variable declared within a function called by a parallel region | New
|
|---|
| 89 | DRB084-threadprivatemissing-orig-yes.c |Y2 | Missing threadprivate for a global var, not referenced within a construct | New
|
|---|
| 90 | DRB086-static-data-member-orig-yes.cpp |Y2 | Missing threadprivate for a static member, not referenced within a construct | New
|
|---|
| 91 | DRB087-static-data-member2-orig-yes.cpp |Y2 | Missing threadprivate for a static member, referenced within a construct | New
|
|---|
| 92 | DRB088-dynamic-storage-orig-yes.c |Y2 | Data race for a dynamica storage variable, not referenced within a construct | New
|
|---|
| 93 | DRB089-dynamic-storage2-orig-yes.c |Y2 | Data race for a dynamica storage variable, referenced within a construct | New
|
|---|
| 94 | DRB090-static-local-orig-yes.c |Y2 | Data race for a locally declared static variable | New
|
|---|
| 95 | DRB092-threadprivatemissing2-orig-yes.c |Y2 | Missing threadprivate for a variable referenced within a construct | New
|
|---|
| 96 | DRB095-doall2-taskloop-orig-yes.c |Y2 | Missing protection for inner loop's loop variable | New
|
|---|
| 97 | DRB106-taskwaitmissing-orig-yes.c |Y3 | Missing taskwait to ensure correct order of calculations | New
|
|---|
| 98 | DRB109-orderedmissing-orig-yes.c |Y3 | Missing the ordered clause, causing data races | New
|
|---|
| 99 | DRB111-linearmissing-orig-yes.c |Y2 | Missing linear for a shared variable, causing data races | New
|
|---|
| 100 | DRB114-if-orig-yes.c |Y1 | True data dependence within a single level loop, with if() clause | New
|
|---|
| 101 | DRB115-forsimd-orig-yes.c |Y1,Y4 | Both thread and instruction level data races due to omp loop simd | New
|
|---|
| 102 | DRB116-target-teams-orig-yes.c |Y3 | Master threads of two teams do not have synchronization, causing data races | New
|
|---|
| 103 |
|
|---|
| 104 | ## Microbenchmarks without known data races
|
|---|
| 105 |
|
|---|
| 106 | Microbenchmark |P-Label| Description | Source
|
|---|
| 107 | -----------------------------------------|-------|--------------------------------------------------------------------------------------|------------
|
|---|
| 108 | DRB041-3mm-parallel-no.c |N2 | 3-step matrix-matrix multiplication, non-optimized version | Polyhedral
|
|---|
| 109 | DRB042-3mm-tile-no.c |N2,N4 | 3-step matrix-matrix multiplication, with tiling and nested SIMD | Polyhedral
|
|---|
| 110 | DRB043-adi-parallel-no.c |N2 | Alternating Direction Implicit solver, non-optimized version | Polyhedral
|
|---|
| 111 | DRB044-adi-tile-no.c |N2,N4 | Alternating Direction Implicit solver, with tiling and nested SIMD | Polyhedral
|
|---|
| 112 | DRB045-doall1-orig-no.c |N1 | Classic DOAll loop operating on a one dimensional array | AutoPar
|
|---|
| 113 | DRB046-doall2-orig-no.c |N1 | Classic DOAll loop operating on a two dimensional array | AutoPar
|
|---|
| 114 | DRB047-doallchar-orig-no.c |N1 | Classic DOALL loop operating on a character array | New
|
|---|
| 115 | DRB048-firstprivate-orig-no.c |N2 | Example use of firstprivate | AutoPar
|
|---|
| 116 | DRB049-fprintf-orig-no.c |N6 | Use of `fprintf()` | New
|
|---|
| 117 | DRB050-functionparameter-orig-no.c |N6 | Arrays passed as function parameters | LLNL App
|
|---|
| 118 | DRB051-getthreadnum-orig-no.c |N2 | single thread execution using `if (omp_get_thread_num()==0)` | New
|
|---|
| 119 | DRB052-indirectaccesssharebase-orig-no.c |N7 | Indirect array accesses using index arrays without overlapping | LLNL App
|
|---|
| 120 | DRB053-inneronly1-orig-no.c |N1 | Two-level nested loops, inner level is parallelizable. Anti dependence on outer level| AutoPar
|
|---|
| 121 | DRB054-inneronly2-orig-no.c |N1 | Two-level nested loops, inner level is parallelizable. True dependence on outer level| AutoPar
|
|---|
| 122 | DRB055-jacobi2d-parallel-no.c |N7 | Jacobi with array copying, no reduction, non-optimized version | Polyhedral
|
|---|
| 123 | DRB056-jacobi2d-tile-no.c |N4,N7 | Jacobi with array copying, no reduction, with tiling and nested SIMD | Polyhedral
|
|---|
| 124 | DRB057-jacobiinitialize-orig-no.c |N7 | The array initialization parallel loop in Jacobi | AutoPar
|
|---|
| 125 | DRB058-jacobikernel-orig-no.c |N7 | Parallel Jacobi stencil computation kernel with array copying and reduction | AutoPar
|
|---|
| 126 | DRB059-lastprivate-orig-no.c |N2 | Example use of lastprivate | AutoPar
|
|---|
| 127 | DRB060-matrixmultiply-orig-no.c |N7 | Classic i-k-j order matrix multiplication using OpenMP | AutoPar
|
|---|
| 128 | DRB061-matrixvector1-orig-no.c |N7 | Matrix-vector multiplication parallelized at the outer level loop | AutoPar
|
|---|
| 129 | DRB062-matrixvector2-orig-no.c |N7 | Matrix-vector multiplication parallelized at the inner level loop with reduction | AutoPar
|
|---|
| 130 | DRB063-outeronly1-orig-no.c |N2 | Two-level nested loops, outer level is parallelizable. Anti dependence on inner level| AutoPar
|
|---|
| 131 | DRB064-outeronly2-orig-no.c |N2 | Two-level nested loops, outer level is parallelizable. True dependence on inner level| AutoPar
|
|---|
| 132 | DRB065-pireduction-orig-no.c |N7 | PI calculation using reduction | AutoPar
|
|---|
| 133 | DRB066-pointernoaliasing-orig-no.c |N6 | Pointers assigned by different malloc calls, without aliasing | LLNL App
|
|---|
| 134 | DRB067-restrictpointer1-orig-no.c |N6 | C99 restrict pointers used for array initialization, no aliasing | LLNL App
|
|---|
| 135 | DRB068-restrictpointer2-orig-no.c |N6 | C99 restrict pointers used for array computation, no aliasing | LLNL App
|
|---|
| 136 | DRB069-sectionslock1-orig-no.c |N3 | OpenMP parallel sections with a lock to protect shared data writes | New
|
|---|
| 137 | DRB070-simd1-orig-no.c |N1,N4 | OpenMP SIMD directive to indicate vectorization of a loop | New
|
|---|
| 138 | DRB071-targetparallelfor-orig-no.c |N1,N5 | No data races in loops offloaded to accelerators | New
|
|---|
| 139 | DRB072-taskdep1-orig-no.c |N3 | OpenMP task with depend clauses to avoid data races | New
|
|---|
| 140 | DRB076-flush-orig-no.c |N2 | OpenMP private clause to avoid data races | New
|
|---|
| 141 | DRB077-single-orig-no.c |N1 | OpenMP single directive to use only one thread for execution | New
|
|---|
| 142 | DRB078-taskdep2-orig-no.c |N3 | OpenMP task depend clause to avoid data races | New
|
|---|
| 143 | DRB079-taskdep3-orig-no.c |N3 | OpenMP task depend clause to avoid data races | New
|
|---|
| 144 | DRB081-func-arg-orig-no.c |N6 | Function arguments passed by value, private | New
|
|---|
| 145 | DRB083-declared-in-func-orig-no.c |N6 | A variable declared within a function called by a parallel region | New
|
|---|
| 146 | DRB085-threadprivate-orig-no.c |N2 | Use threadprivate to protect a file scope variable, not referenced within a construct| New
|
|---|
| 147 | DRB091-threadprivate2-orig-no.c |N2 | Use threadprivate to protect a file scope variable, referenced within a construct | New
|
|---|
| 148 | DRB093-doall2-collapse-orig-no.c |N2 | Use collapse(n) to control the number of associated loops of omp for | New
|
|---|
| 149 | DRB094-doall2-ordered-orig-no.c |N2 | Use ordered(n) to control the number of associated loops of omp for | New
|
|---|
| 150 | DRB096-doall2-taskloop-collapse-orig-no.c|N2 | Use ordered(n) to control the number of associated loops of taskloop | New
|
|---|
| 151 | DRB097-target-teams-distribute-orig-no.c |N2 | Predetermined attribute rule for loop variable associated with distribute | New
|
|---|
| 152 | DRB098-simd2-orig-no.c |N1,N2 | OpenMP SIMD directive to indicate vectorization of two nested loops | New
|
|---|
| 153 | DRB099-targetparallelfor2-orig-no.c |N1,N5 | Loops offloaded to accelerators: array sections derived from pointer | New
|
|---|
| 154 | DRB100-task-reference-orig-no.cpp |N1 | OpenMP 4.5 feature: orphaned task generating construct using pass-by-reference | New
|
|---|
| 155 | DRB101-task-value-orig-no.cpp |N1 | In a task generating construct, a variable without applicable rules is firstprivate | New
|
|---|
| 156 | DRB102-copyprivate-orig-no.c |N2 | threadprivate+copyprivate, a variable without applicable rules is firstprivate | New
|
|---|
| 157 | DRB103-master-orig-no.c |N1 | master directive to ensure only one thread will execute data accesses | New
|
|---|
| 158 | DRB104-nowait-barrier-orig-no.c |N3 | Use barrier to ensure correct order of initialization and assignment phases | New
|
|---|
| 159 | DRB105-taskwait-orig-no.c |N3 | Use taskwait to ensure correct order of tasks | New
|
|---|
| 160 | DRB107-taskgroup-orig-no.c |N3 | Use taskgroup to ensure correct order of tasks | New
|
|---|
| 161 | DRB108-atomic-orig-no.c |N3 | Use atomic to protect shared accesses to a variable | New
|
|---|
| 162 | DRB110-ordered-orig-no.c |N3 | Proper use of the ordered clause to avoid data races | New
|
|---|
| 163 | DRB112-linear-orig-no.c |N2 | Use linear to privatize a variable | New
|
|---|
| 164 | DRB113-default-orig-no.c |N1 | default(none) to enforce explicitly listing variables in data-sharing clauses | New
|
|---|
| 165 |
|
|---|
| 166 | ## Authors
|
|---|
| 167 |
|
|---|
| 168 | DataRaceBench was created by Chunhua Liao, Pei-Hung Lin, Joshua Asplund, Markus Schordan, and Ian Karlin.
|
|---|
| 169 |
|
|---|
| 170 | ## Release
|
|---|
| 171 |
|
|---|
| 172 | DataRaceBench is released under a BSD license. For more details see
|
|---|
| 173 | the file LICENSE.txt. The microbenchmarks marked 'Polyhedral' in above
|
|---|
| 174 | table were generated as optimization variants of benchmarks from the
|
|---|
| 175 | PolyOpt benchmark suite. For those benchmarks see the license file
|
|---|
| 176 | LICENSE.OSU.txt.
|
|---|
| 177 |
|
|---|
| 178 | `LLNL-CODE-732144`
|
|---|
| 179 |
|
|---|
| 180 | ## How to cite DataRaceBench in a publication
|
|---|
| 181 |
|
|---|
| 182 | If you are referring to DataRaceBench in a publication, please cite the following paper:
|
|---|
| 183 |
|
|---|
| 184 | * Chunhua Liao, Pei-Hung Lin, Joshua Asplund, Markus Schordan, Ian Karlin.
|
|---|
| 185 | [DataRaceBench: A Benchmark Suite for Systematic Evaluation of Data Race Detection Tools (best paper finalist)](https://dl.acm.org/citation.cfm?doid=3126908.3126958).
|
|---|
| 186 | Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017, pp. 11:1-11:14, ISBN 978-1-4503-5114-0, Denver, CO, USA, November 12-17, 2017.
|
|---|
| 187 |
|
|---|
| 188 | If you use DataRaceBench v.1.2.0 or later, please additionally cite the following paper discussing coverage analysis and improvements of the benchmark suite:
|
|---|
| 189 |
|
|---|
| 190 | * Chunhua Liao, Pei-Hung Lin, Markus Schordan and Ian Karlin, [A Semantics-Driven Approach to Improving DataRaceBench's OpenMP Standard Coverage](https://www.springerprofessional.de/en/a-semantics-driven-approach-to-improving-dataracebench-s-openmp-/16134302), IWOMP 2018: 14th International Workshop on OpenMP, Barcelona, Spain, September 26-28, 2018
|
|---|
| 191 |
|
|---|
| 192 | Other papers
|
|---|
| 193 | * Pei-Hung Lin, Chunhua Liao, Markus Schordan, Ian Karlin. [Runtime and Memory Evaluation of Data Race Detection Tools](https://link.springer.com/chapter/10.1007/978-3-030-03421-4_13). ISoLA (2) 2018: 179-196.
|
|---|
| 194 | * Pei-Hung Lin, Chunhua Liao, Markus Schordan, Ian Karlin. Exploring Regression of Data Race Detection Tools Using DataRaceBench. To
|
|---|
| 195 | appear in Proceedings of the Third International Workshop on Software Correctness for HPC Applications (CORRECTNESS@SC 2019).
|
|---|
