Newer
Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
==========================
Source-based Code Coverage
==========================
.. contents::
:local:
Introduction
============
This document explains how to use clang's source-based code coverage feature.
It's called "source-based" because it operates on AST and preprocessor
information directly. This allows it to generate very precise coverage data.
Clang ships two other code coverage implementations:
* :doc:`SanitizerCoverage` - A low-overhead tool meant for use alongside the
various sanitizers. It can provide up to edge-level coverage.
* gcov - A GCC-compatible coverage implementation which operates on DebugInfo.
From this point onwards "code coverage" will refer to the source-based kind.
The code coverage workflow
==========================
The code coverage workflow consists of three main steps:
1. Compiling with coverage enabled.
2. Running the instrumented program.
3. Creating coverage reports.
The next few sections work through a complete, copy-'n-paste friendly example
based on this program:
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
% cat <<EOF > foo.cc
#define BAR(x) ((x) || (x))
template <typename T> void foo(T x) {
for (unsigned I = 0; I < 10; ++I) { BAR(I); }
}
int main() {
foo<int>(0);
foo<float>(0);
return 0;
}
EOF
Compiling with coverage enabled
===============================
To compile code with coverage enabled pass ``-fprofile-instr-generate
-fcoverage-mapping`` to the compiler:
.. code-block:: console
# Step 1: Compile with coverage enabled.
% clang++ -fprofile-instr-generate -fcoverage-mapping foo.cc -o foo
Note that linking together code with and without coverage instrumentation is
supported: any uninstrumented code simply won't be accounted for.
Running the instrumented program
================================
The next step is to run the instrumented program. When the program exits it
will write a **raw profile** to the path specified by the ``LLVM_PROFILE_FILE``
environment variable. If that variable does not exist the profile is written to
``default.profraw`` in the current directory of the program.
If ``LLVM_PROFILE_FILE`` contains a path to a non-existent directory the
missing directory structure will be created. Additionally, the following
special **pattern strings** are replaced:
* "%p" expands out to the process ID.
* "%h" expands out to the hostname of the machine running the program.
.. code-block:: console
# Step 2: Run the program.
% LLVM_PROFILE_FILE="foo.profraw" ./foo
Creating coverage reports
=========================
Raw profiles have to be **indexed** before they can be used to generated
coverage reports. This is done using the "merge" tool in ``llvm-profdata``, so
named because it can combine and index profiles at the same time:
.. code-block:: console
# Step 3(a): Index the raw profile.
% llvm-profdata merge -sparse foo.profraw -o foo.profdata
There are multiple different ways to render coverage reports. One option is to
generate a line-oriented report:
.. code-block:: console
# Step 3(b): Create a line-oriented coverage report.
% llvm-cov show ./foo -instr-profile=foo.profdata
To demangle any C++ identifiers in the ouput, use:
.. code-block:: console
% llvm-cov show ./foo -instr-profile=foo.profdata | c++filt -n
This report includes a summary view as well as dedicated sub-views for
templated functions and their instantiations. For our example program, we get
distinct views for ``foo<int>(...)`` and ``foo<float>(...)``. If
``-show-line-counts-or-regions`` is enabled, ``llvm-cov`` displays sub-line
region counts (even in macro expansions):
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
20| 1|#define BAR(x) ((x) || (x))
^20 ^2
2| 2|template <typename T> void foo(T x) {
22| 3| for (unsigned I = 0; I < 10; ++I) { BAR(I); }
^22 ^20 ^20^20
2| 4|}
------------------
| void foo<int>(int):
| 1| 2|template <typename T> void foo(T x) {
| 11| 3| for (unsigned I = 0; I < 10; ++I) { BAR(I); }
| ^11 ^10 ^10^10
| 1| 4|}
------------------
| void foo<float>(int):
| 1| 2|template <typename T> void foo(T x) {
| 11| 3| for (unsigned I = 0; I < 10; ++I) { BAR(I); }
| ^11 ^10 ^10^10
| 1| 4|}
------------------
It's possible to generate a file-level summary of coverage statistics (instead
of a line-oriented report) with:
.. code-block:: console
# Step 3(c): Create a coverage summary.
% llvm-cov report ./foo -instr-profile=foo.profdata
Filename Regions Miss Cover Functions Executed
-----------------------------------------------------------------------
/tmp/foo.cc 13 0 100.00% 3 100.00%
-----------------------------------------------------------------------
TOTAL 13 0 100.00% 3 100.00%
A few final notes:
* The ``-sparse`` flag is optional but can result in dramatically smaller
indexed profiles. This option should not be used if the indexed profile will
be reused for PGO.
* Raw profiles can be discarded after they are indexed. Advanced use of the
profile runtime library allows an instrumented program to merge profiling
information directly into an existing raw profile on disk. The details are
out of scope.
* The ``llvm-profdata`` tool can be used to merge together multiple raw or
indexed profiles. To combine profiling data from multiple runs of a program,
try e.g:
.. code-block:: console
% llvm-profdata merge -sparse foo1.profraw foo2.profdata -o foo3.profdata
Format compatibility guarantees
===============================
* There are no backwards or forwards compatibility guarantees for the raw
profile format. Raw profiles may be dependent on the specific compiler
revision used to generate them. It's inadvisable to store raw profiles for
long periods of time.
* Tools must retain **backwards** compatibility with indexed profile formats.
These formats are not forwards-compatible: i.e, a tool which uses format
version X will not be able to understand format version (X+k).
* There is a third format in play: the format of the coverage mappings emitted
into instrumented binaries. Tools must retain **backwards** compatibility
with these formats. These formats are not forwards-compatible.