Gitiles
Code Review Sign In
gerrit-public.fairphone.software / platform / external / ComputeLibrary / 7f09cf7dfba61aad6d6d588b1b096f56bafaad5e / . / documentation / tests.xhtml
blob: e4d6e01ac779d13db33c04c37c9756a0f296c79a [file] [log] [blame]
Jenkins514be652019-02-28 12:25:18 +0000[diff] [blame]1<!-- HTML header for doxygen 1.8.15-->
2<!-- Remember to use version doxygen 1.8.15 +-->
Anthony Barbierdbdab852017-06-23 15:42:00 +0100[diff] [blame]3<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
4<html xmlns="http://www.w3.org/1999/xhtml">
5<head>
6<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
7<meta http-equiv="X-UA-Compatible" content="IE=9"/>
Jenkins514be652019-02-28 12:25:18 +0000[diff] [blame]8<meta name="generator" content="Doxygen 1.8.15"/>
Anthony Barbierdbdab852017-06-23 15:42:00 +0100[diff] [blame]9<meta name="robots" content="NOINDEX, NOFOLLOW" /> <!-- Prevent indexing by search engines -->
10<title>Compute Library: Validation and benchmarks tests</title>
11<link href="tabs.css" rel="stylesheet" type="text/css"/>
12<script type="text/javascript" src="jquery.js"></script>
13<script type="text/javascript" src="dynsections.js"></script>
14<link href="navtree.css" rel="stylesheet" type="text/css"/>
15<script type="text/javascript" src="resize.js"></script>
Anthony Barbier8140e1e2017-12-14 23:48:46 +0000[diff] [blame]16<script type="text/javascript" src="navtreedata.js"></script>
Anthony Barbierdbdab852017-06-23 15:42:00 +0100[diff] [blame]17<script type="text/javascript" src="navtree.js"></script>
18<script type="text/javascript">
Jenkins514be652019-02-28 12:25:18 +0000[diff] [blame]19/* @license magnet:?xt=urn:btih:cf05388f2679ee054f2beb29a391d25f4e673ac3&amp;dn=gpl-2.0.txt GPL-v2 */
Anthony Barbierdbdab852017-06-23 15:42:00 +0100[diff] [blame]20 $(document).ready(initResizable);
Jenkins514be652019-02-28 12:25:18 +0000[diff] [blame]21/* @license-end */</script>
Anthony Barbierdbdab852017-06-23 15:42:00 +0100[diff] [blame]22<link href="search/search.css" rel="stylesheet" type="text/css"/>
Anthony Barbier8140e1e2017-12-14 23:48:46 +0000[diff] [blame]23<script type="text/javascript" src="search/searchdata.js"></script>
Anthony Barbierdbdab852017-06-23 15:42:00 +0100[diff] [blame]24<script type="text/javascript" src="search/search.js"></script>
Anthony Barbierdbdab852017-06-23 15:42:00 +0100[diff] [blame]25<script type="text/x-mathjax-config">
26 MathJax.Hub.Config({
27 extensions: ["tex2jax.js"],
28 jax: ["input/TeX","output/HTML-CSS"],
29});
Jenkins514be652019-02-28 12:25:18 +0000[diff] [blame]30</script><script type="text/javascript" async="async" src="http://cdn.mathjax.org/mathjax/latest/MathJax.js"></script>
Anthony Barbierdbdab852017-06-23 15:42:00 +0100[diff] [blame]31<link href="doxygen.css" rel="stylesheet" type="text/css" />
Jenkins514be652019-02-28 12:25:18 +0000[diff] [blame]32<link href="stylesheet.css" rel="stylesheet" type="text/css"/>
Anthony Barbierdbdab852017-06-23 15:42:00 +0100[diff] [blame]33</head>
34<body>
35<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
36<div id="titlearea">
37<table cellspacing="0" cellpadding="0">
38 <tbody>
39 <tr style="height: 56px;">
Jenkins514be652019-02-28 12:25:18 +0000[diff] [blame]40 <img alt="Compute Library" src="https://raw.githubusercontent.com/ARM-software/ComputeLibrary/gh-pages/ACL_logo.png" style="max-width: 100%;margin-top: 15px;margin-left: 10px"/>
Anthony Barbierdbdab852017-06-23 15:42:00 +0100[diff] [blame]41 <td style="padding-left: 0.5em;">
Jenkins514be652019-02-28 12:25:18 +0000[diff] [blame]42 <div id="projectname">
Jenkins7f09cf72020-01-22 18:08:16 +0000[diff] [blame^]43 &#160;<span id="projectnumber">19.11.1</span>
Anthony Barbierdbdab852017-06-23 15:42:00 +0100[diff] [blame]44 </div>
45 </td>
46 </tr>
47 </tbody>
48</table>
49</div>
50<!-- end header part -->
Jenkins514be652019-02-28 12:25:18 +0000[diff] [blame]51<!-- Generated by Doxygen 1.8.15 -->
Anthony Barbierdbdab852017-06-23 15:42:00 +0100[diff] [blame]52<script type="text/javascript">
Jenkins514be652019-02-28 12:25:18 +0000[diff] [blame]53/* @license magnet:?xt=urn:btih:cf05388f2679ee054f2beb29a391d25f4e673ac3&amp;dn=gpl-2.0.txt GPL-v2 */
Anthony Barbierdbdab852017-06-23 15:42:00 +0100[diff] [blame]54var searchBox = new SearchBox("searchBox", "search",false,'Search');
Jenkins514be652019-02-28 12:25:18 +0000[diff] [blame]55/* @license-end */
Anthony Barbierdbdab852017-06-23 15:42:00 +0100[diff] [blame]56</script>
Jenkinsb9abeae2018-11-22 11:58:08 +0000[diff] [blame]57<script type="text/javascript" src="menudata.js"></script>
58<script type="text/javascript" src="menu.js"></script>
59<script type="text/javascript">
Jenkins514be652019-02-28 12:25:18 +0000[diff] [blame]60/* @license magnet:?xt=urn:btih:cf05388f2679ee054f2beb29a391d25f4e673ac3&amp;dn=gpl-2.0.txt GPL-v2 */
Jenkinsb9abeae2018-11-22 11:58:08 +0000[diff] [blame]61$(function() {
62 initMenu('',true,false,'search.php','Search');
63 $(document).ready(function() { init_search(); });
64});
Jenkins514be652019-02-28 12:25:18 +0000[diff] [blame]65/* @license-end */</script>
Jenkinsb9abeae2018-11-22 11:58:08 +0000[diff] [blame]66<div id="main-nav"></div>
Anthony Barbierdbdab852017-06-23 15:42:00 +0100[diff] [blame]67</div><!-- top -->
68<div id="side-nav" class="ui-resizable side-nav-resizable">
69 <div id="nav-tree">
70 <div id="nav-tree-contents">
71 <div id="nav-sync" class="sync"></div>
72 </div>
73 </div>
74 <div id="splitbar" style="-moz-user-select:none;"
75 class="ui-resizable-handle">
76 </div>
77</div>
78<script type="text/javascript">
Jenkins514be652019-02-28 12:25:18 +0000[diff] [blame]79/* @license magnet:?xt=urn:btih:cf05388f2679ee054f2beb29a391d25f4e673ac3&amp;dn=gpl-2.0.txt GPL-v2 */
Anthony Barbierdbdab852017-06-23 15:42:00 +0100[diff] [blame]80$(document).ready(function(){initNavTree('tests.xhtml','');});
Jenkins514be652019-02-28 12:25:18 +0000[diff] [blame]81/* @license-end */
Anthony Barbierdbdab852017-06-23 15:42:00 +0100[diff] [blame]82</script>
83<div id="doc-content">
84<!-- window showing the filter options -->
85<div id="MSearchSelectWindow"
86 onmouseover="return searchBox.OnSearchSelectShow()"
87 onmouseout="return searchBox.OnSearchSelectHide()"
88 onkeydown="return searchBox.OnSearchSelectKey(event)">
Anthony Barbier8140e1e2017-12-14 23:48:46 +0000[diff] [blame]89</div>
Anthony Barbierdbdab852017-06-23 15:42:00 +0100[diff] [blame]90
91<!-- iframe showing the search results (closed by default) -->
92<div id="MSearchResultsWindow">
93<iframe src="javascript:void(0)" frameborder="0"
94 name="MSearchResults" id="MSearchResults">
95</iframe>
96</div>
97
Jenkins514be652019-02-28 12:25:18 +0000[diff] [blame]98<div class="PageDoc"><div class="header">
Anthony Barbierdbdab852017-06-23 15:42:00 +0100[diff] [blame]99 <div class="headertitle">
100<div class="title">Validation and benchmarks tests </div> </div>
101</div><!--header-->
102<div class="contents">
103<div class="toc"><h3>Table of Contents</h3>
Anthony Barbier8140e1e2017-12-14 23:48:46 +0000[diff] [blame]104<ul><li class="level1"><a href="#tests_overview">Overview</a><ul><li class="level2"><a href="#tests_overview_structure">Directory structure</a></li>
105<li class="level2"><a href="#tests_overview_fixtures">Fixtures</a><ul><li class="level3"><a href="#tests_overview_fixtures_fixture">Fixture</a></li>
Kaizen8938bd32017-09-28 14:38:23 +0100[diff] [blame]106<li class="level3"><a href="#tests_overview_fixtures_data_fixture">Data fixture</a></li>
Anthony Barbierdbdab852017-06-23 15:42:00 +0100[diff] [blame]107</ul>
108</li>
Kaizen8938bd32017-09-28 14:38:23 +0100[diff] [blame]109<li class="level2"><a href="#tests_overview_test_cases">Test cases</a><ul><li class="level3"><a href="#tests_overview_test_cases_test_case">Test case</a></li>
110<li class="level3"><a href="#tests_overview_test_cases_fixture_fixture_test_case">Fixture test case</a></li>
111<li class="level3"><a href="#tests_overview_test_cases_fixture_register_fixture_test_case">Registering a fixture as test case</a></li>
112<li class="level3"><a href="#tests_overview_test_cases_data_test_case">Data test case</a></li>
113<li class="level3"><a href="#tests_overview_test_cases_fixture_data_test_case">Fixture data test case</a></li>
114<li class="level3"><a href="#tests_overview_test_cases_register_fixture_data_test_case">Registering a fixture as data test case</a></li>
115</ul>
116</li>
117</ul>
118</li>
119<li class="level1"><a href="#writing_tests">Writing validation tests</a></li>
Jenkinsb3a371b2018-05-23 11:36:53 +0100[diff] [blame]120<li class="level1"><a href="#tests_running_tests">Running tests</a><ul><li class="level2"><a href="#tests_running_tests_benchmark_and_validation">Benchmarking and validation suites</a><ul><li class="level3"><a href="#tests_running_tests_benchmarking_filter">Filter tests</a></li>
Anthony Barbierdbdab852017-06-23 15:42:00 +0100[diff] [blame]121<li class="level3"><a href="#tests_running_tests_benchmarking_runtime">Runtime</a></li>
Kaizen8938bd32017-09-28 14:38:23 +0100[diff] [blame]122<li class="level3"><a href="#tests_running_tests_benchmarking_output">Output</a></li>
Kaizenbf8b01d2017-10-12 14:26:51 +0100[diff] [blame]123<li class="level3"><a href="#tests_running_tests_benchmarking_mode">Mode</a></li>
124<li class="level3"><a href="#tests_running_tests_benchmarking_instruments">Instruments</a></li>
Anthony Barbierf45d5a92018-01-24 16:23:15 +0000[diff] [blame]125<li class="level3"><a href="#tests_running_examples">Examples</a></li>
Anthony Barbierdbdab852017-06-23 15:42:00 +0100[diff] [blame]126</ul>
127</li>
Anthony Barbierdbdab852017-06-23 15:42:00 +0100[diff] [blame]128</ul>
129</li>
130</ul>
131</div>
Kaizen8938bd32017-09-28 14:38:23 +0100[diff] [blame]132<div class="textblock"><h1><a class="anchor" id="tests_overview"></a>
133Overview</h1>
134<p>Benchmark and validation tests are based on the same framework to setup and run the tests. In addition to running simple, self-contained test functions the framework supports fixtures and data test cases. The former allows to share common setup routines between various backends thus reducing the amount of duplicated code. The latter can be used to parameterize tests or fixtures with different inputs, e.g. different tensor shapes. One limitation is that tests/fixtures cannot be parameterized based on the data type if static type information is needed within the test (e.g. to validate the results).</p>
Anthony Barbierf45d5a92018-01-24 16:23:15 +0000[diff] [blame]135<dl class="section note"><dt>Note</dt><dd>By default tests are not built. To enable them you need to add validation_tests=1 and / or benchmark_tests=1 to your SCons line.</dd>
136<dd>
137Tests are not included in the pre-built binary archive, you have to build them from sources.</dd></dl>
Anthony Barbier8140e1e2017-12-14 23:48:46 +0000[diff] [blame]138<h2><a class="anchor" id="tests_overview_structure"></a>
139Directory structure</h2>
140<pre class="fragment">.
141`-- tests &lt;- Top level test directory. All files in here are shared among validation and benchmark.
142 |-- framework &lt;- Underlying test framework.
143 |-- CL \
144 |-- NEON -&gt; Backend specific files with helper functions etc.
145 |-- benchmark &lt;- Top level directory for the benchmarking files.
146 | |-- fixtures &lt;- Fixtures for benchmark tests.
147 | |-- CL &lt;- OpenCL backend test cases on a function level.
148 | | `-- SYSTEM &lt;- OpenCL system tests, e.g. whole networks
149 | `-- NEON &lt;- Same for NEON
150 | `-- SYSTEM
151 |-- datasets &lt;- Datasets for benchmark and validation tests.
152 |-- main.cpp &lt;- Main entry point for the tests. Currently shared between validation and benchmarking.
153 |-- networks &lt;- Network classes for system level tests.
154 `-- validation -&gt; Top level directory for validation files.
155 |-- CPP -&gt; C++ reference code
156 |-- CL \
157 |-- NEON -&gt; Backend specific test cases
158 `-- fixtures -&gt; Fixtures shared among all backends. Used to setup target function and tensors.
159</pre><h2><a class="anchor" id="tests_overview_fixtures"></a>
Kaizen8938bd32017-09-28 14:38:23 +0100[diff] [blame]160Fixtures</h2>
161<p>Fixtures can be used to share common setup, teardown or even run tasks among multiple test cases. For that purpose a fixture can define a <code>setup</code>, <code>teardown</code> and <code>run</code> method. Additionally the constructor and destructor might also be customized.</p>
162<p>An instance of the fixture is created immediately before the actual test is executed. After construction the <a class="el" href="classarm__compute_1_1test_1_1framework_1_1_fixture.xhtml#a4fc01d736fe50cf5b977f755b675f11d">framework::Fixture::setup</a> method is called. Then the test function or the fixtures <code>run</code> method is invoked. After test execution the <a class="el" href="classarm__compute_1_1test_1_1framework_1_1_fixture.xhtml#a4adab6322a0276f34a7d656d49fc865c">framework::Fixture::teardown</a> method is called and lastly the fixture is destructed.</p>
163<h3><a class="anchor" id="tests_overview_fixtures_fixture"></a>
164Fixture</h3>
Anthony Barbier8140e1e2017-12-14 23:48:46 +0000[diff] [blame]165<p>Fixtures for non-parameterized test are straightforward. The custom fixture class has to inherit from <a class="el" href="classarm__compute_1_1test_1_1framework_1_1_fixture.xhtml">framework::Fixture</a> and choose to implement any of the <code>setup</code>, <code>teardown</code> or <code>run</code> methods. None of the methods takes any arguments or returns anything. </p><pre class="fragment">class CustomFixture : public framework::Fixture
Kaizen8938bd32017-09-28 14:38:23 +0100[diff] [blame]166{
167 void setup()
168 {
169 _ptr = malloc(4000);
170 }
171
172 void run()
173 {
174 ARM_COMPUTE_ASSERT(_ptr != nullptr);
175 }
176
177 void teardown()
178 {
179 free(_ptr);
180 }
181
182 void *_ptr;
183};
184</pre><h3><a class="anchor" id="tests_overview_fixtures_data_fixture"></a>
185Data fixture</h3>
Anthony Barbier8140e1e2017-12-14 23:48:46 +0000[diff] [blame]186<p>The advantage of a parameterized fixture is that arguments can be passed to the setup method at runtime. To make this possible the setup method has to be a template with a type parameter for every argument (though the template parameter doesn't have to be used). All other methods remain the same. </p><pre class="fragment">class CustomFixture : public framework::Fixture
Kaizen8938bd32017-09-28 14:38:23 +0100[diff] [blame]187{
188#ifdef ALTERNATIVE_DECLARATION
189 template &lt;typename ...&gt;
190 void setup(size_t size)
191 {
192 _ptr = malloc(size);
193 }
194#else
195 template &lt;typename T&gt;
196 void setup(T size)
197 {
198 _ptr = malloc(size);
199 }
200#endif
201
202 void run()
203 {
204 ARM_COMPUTE_ASSERT(_ptr != nullptr);
205 }
206
207 void teardown()
208 {
209 free(_ptr);
210 }
211
212 void *_ptr;
213};
214</pre><h2><a class="anchor" id="tests_overview_test_cases"></a>
215Test cases</h2>
216<p>All following commands can be optionally prefixed with <code>EXPECTED_FAILURE_</code> or <code>DISABLED_</code>.</p>
217<h3><a class="anchor" id="tests_overview_test_cases_test_case"></a>
218Test case</h3>
219<p>A simple test case function taking no inputs and having no (shared) state.</p>
220<ul>
221<li>First argument is the name of the test case (has to be unique within the enclosing test suite).</li>
222<li>Second argument is the dataset mode in which the test will be active.</li>
223</ul>
224<pre class="fragment">TEST_CASE(TestCaseName, DatasetMode::PRECOMMIT)
225{
226 ARM_COMPUTE_ASSERT_EQUAL(1 + 1, 2);
227}
228</pre><h3><a class="anchor" id="tests_overview_test_cases_fixture_fixture_test_case"></a>
229Fixture test case</h3>
230<p>A simple test case function taking no inputs that inherits from a fixture. The test case will have access to all public and protected members of the fixture. Only the setup and teardown methods of the fixture will be used. The body of this function will be used as test function.</p>
231<ul>
232<li>First argument is the name of the test case (has to be unique within the enclosing test suite).</li>
233<li>Second argument is the class name of the fixture.</li>
234<li>Third argument is the dataset mode in which the test will be active.</li>
235</ul>
236<pre class="fragment">class FixtureName : public framework::Fixture
237{
238 public:
239 void setup() override
240 {
241 _one = 1;
242 }
243
244 protected:
245 int _one;
246};
247
248FIXTURE_TEST_CASE(TestCaseName, FixtureName, DatasetMode::PRECOMMIT)
249{
250 ARM_COMPUTE_ASSERT_EQUAL(_one + 1, 2);
251}
252</pre><h3><a class="anchor" id="tests_overview_test_cases_fixture_register_fixture_test_case"></a>
253Registering a fixture as test case</h3>
254<p>Allows to use a fixture directly as test case. Instead of defining a new test function the run method of the fixture will be executed.</p>
255<ul>
256<li>First argument is the name of the test case (has to be unique within the enclosing test suite).</li>
257<li>Second argument is the class name of the fixture.</li>
258<li>Third argument is the dataset mode in which the test will be active.</li>
259</ul>
260<pre class="fragment">class FixtureName : public framework::Fixture
261{
262 public:
263 void setup() override
264 {
265 _one = 1;
266 }
267
268 void run() override
269 {
270 ARM_COMPUTE_ASSERT_EQUAL(_one + 1, 2);
271 }
272
273 protected:
274 int _one;
275};
276
277REGISTER_FIXTURE_TEST_CASE(TestCaseName, FixtureName, DatasetMode::PRECOMMIT);
278</pre><h3><a class="anchor" id="tests_overview_test_cases_data_test_case"></a>
279Data test case</h3>
280<p>A parameterized test case function that has no (shared) state. The dataset will be used to generate versions of the test case with different inputs.</p>
281<ul>
282<li>First argument is the name of the test case (has to be unique within the enclosing test suite).</li>
283<li>Second argument is the dataset mode in which the test will be active.</li>
284<li>Third argument is the dataset.</li>
285<li>Further arguments specify names of the arguments to the test function. The number must match the arity of the dataset.</li>
286</ul>
287<pre class="fragment">DATA_TEST_CASE(TestCaseName, DatasetMode::PRECOMMIT, framework::make("Numbers", {1, 2, 3}), num)
288{
289 ARM_COMPUTE_ASSERT(num &lt; 4);
290}
291</pre><h3><a class="anchor" id="tests_overview_test_cases_fixture_data_test_case"></a>
292Fixture data test case</h3>
293<p>A parameterized test case that inherits from a fixture. The test case will have access to all public and protected members of the fixture. Only the setup and teardown methods of the fixture will be used. The setup method of the fixture needs to be a template and has to accept inputs from the dataset as arguments. The body of this function will be used as test function. The dataset will be used to generate versions of the test case with different inputs.</p>
294<ul>
295<li>First argument is the name of the test case (has to be unique within the enclosing test suite).</li>
296<li>Second argument is the class name of the fixture.</li>
297<li>Third argument is the dataset mode in which the test will be active.</li>
298<li>Fourth argument is the dataset.</li>
299</ul>
300<pre class="fragment">class FixtureName : public framework::Fixture
301{
302 public:
303 template &lt;typename T&gt;
304 void setup(T num)
305 {
306 _num = num;
307 }
308
309 protected:
310 int _num;
311};
312
313FIXTURE_DATA_TEST_CASE(TestCaseName, FixtureName, DatasetMode::PRECOMMIT, framework::make("Numbers", {1, 2, 3}))
314{
315 ARM_COMPUTE_ASSERT(_num &lt; 4);
316}
317</pre><h3><a class="anchor" id="tests_overview_test_cases_register_fixture_data_test_case"></a>
318Registering a fixture as data test case</h3>
319<p>Allows to use a fixture directly as parameterized test case. Instead of defining a new test function the run method of the fixture will be executed. The setup method of the fixture needs to be a template and has to accept inputs from the dataset as arguments. The dataset will be used to generate versions of the test case with different inputs.</p>
320<ul>
321<li>First argument is the name of the test case (has to be unique within the enclosing test suite).</li>
322<li>Second argument is the class name of the fixture.</li>
323<li>Third argument is the dataset mode in which the test will be active.</li>
324<li>Fourth argument is the dataset.</li>
325</ul>
326<pre class="fragment">class FixtureName : public framework::Fixture
327{
328 public:
329 template &lt;typename T&gt;
330 void setup(T num)
331 {
332 _num = num;
333 }
334
335 void run() override
336 {
337 ARM_COMPUTE_ASSERT(_num &lt; 4);
338 }
339
340 protected:
341 int _num;
342};
343
344REGISTER_FIXTURE_DATA_TEST_CASE(TestCaseName, FixtureName, DatasetMode::PRECOMMIT, framework::make("Numbers", {1, 2, 3}));
345</pre><h1><a class="anchor" id="writing_tests"></a>
346Writing validation tests</h1>
347<p>Before starting a new test case have a look at the existing ones. They should provide a good overview how test cases are structured.</p>
348<ul>
Jenkins514be652019-02-28 12:25:18 +0000[diff] [blame]349<li>The C++ reference needs to be added to <code>tests/validation/CPP/</code>. The reference function is typically a template parameterized by the underlying value type of the <code><a class="el" href="classarm__compute_1_1test_1_1_simple_tensor.xhtml" title="Simple tensor object that stores elements in a consecutive chunk of memory.">SimpleTensor</a></code>. This makes it easy to specialise for different data types.</li>
Kaizen8938bd32017-09-28 14:38:23 +0100[diff] [blame]350<li>If all backends have a common interface it makes sense to share the setup code. This can be done by adding a fixture in <code>tests/validation/fixtures/</code>. Inside of the <code>setup</code> method of a fixture the tensors can be created and initialised and the function can be configured and run. The actual test will only have to validate the results. To be shared among multiple backends the fixture class is usually a template that accepts the specific types (data, tensor class, function class etc.) as parameters.</li>
351<li>The actual test cases need to be added for each backend individually. Typically the will be multiple tests for different data types and for different execution modes, e.g. precommit and nightly.</li>
352</ul>
Anthony Barbierdbdab852017-06-23 15:42:00 +0100[diff] [blame]353<h1><a class="anchor" id="tests_running_tests"></a>
354Running tests</h1>
Jenkinsb3a371b2018-05-23 11:36:53 +0100[diff] [blame]355<h2><a class="anchor" id="tests_running_tests_benchmark_and_validation"></a>
356Benchmarking and validation suites</h2>
Anthony Barbierdbdab852017-06-23 15:42:00 +0100[diff] [blame]357<h3><a class="anchor" id="tests_running_tests_benchmarking_filter"></a>
358Filter tests</h3>
Anthony Barbier8140e1e2017-12-14 23:48:46 +0000[diff] [blame]359<p>All tests can be run by invoking </p><pre class="fragment">./arm_compute_benchmark ./data
Anthony Barbierdbdab852017-06-23 15:42:00 +0100[diff] [blame]360</pre><p>where <code>./data</code> contains the assets needed by the tests.</p>
Anthony Barbierf45d5a92018-01-24 16:23:15 +0000[diff] [blame]361<p>If only a subset of the tests has to be executed the <code>--filter</code> option takes a regular expression to select matching tests. </p><pre class="fragment">./arm_compute_benchmark --filter='^NEON/.*AlexNet' ./data
362</pre><dl class="section note"><dt>Note</dt><dd>Filtering will be much faster if the regular expression starts from the start ("^") or end ("$") of the line.</dd></dl>
363<p>Additionally each test has a test id which can be used as a filter, too. However, the test id is not guaranteed to be stable when new tests are added. Only for a specific build the same the test will keep its id. </p><pre class="fragment">./arm_compute_benchmark --filter-id=10 ./data
Anthony Barbier8140e1e2017-12-14 23:48:46 +0000[diff] [blame]364</pre><p>All available tests can be displayed with the <code>--list-tests</code> switch. </p><pre class="fragment">./arm_compute_benchmark --list-tests
Kaizen8938bd32017-09-28 14:38:23 +0100[diff] [blame]365</pre><p>More options can be found in the <code>--help</code> message.</p>
366<h3><a class="anchor" id="tests_running_tests_benchmarking_runtime"></a>
Anthony Barbierdbdab852017-06-23 15:42:00 +0100[diff] [blame]367Runtime</h3>
Kaizen8938bd32017-09-28 14:38:23 +0100[diff] [blame]368<p>By default every test is run once on a single thread. The number of iterations can be controlled via the <code>--iterations</code> option and the number of threads via <code>--threads</code>.</p>
369<h3><a class="anchor" id="tests_running_tests_benchmarking_output"></a>
370Output</h3>
371<p>By default the benchmarking results are printed in a human readable format on the command line. The colored output can be disabled via <code>--no-color-output</code>. As an alternative output format JSON is supported and can be selected via <code>--log-format=json</code>. To write the output to a file instead of stdout the <code>--log-file</code> option can be used.</p>
Kaizenbf8b01d2017-10-12 14:26:51 +0100[diff] [blame]372<h3><a class="anchor" id="tests_running_tests_benchmarking_mode"></a>
373Mode</h3>
374<p>Tests contain different datasets of different sizes, some of which will take several hours to run. You can select which datasets to use by using the <code>--mode</code> option, we recommed you use <code>--mode=precommit</code> to start with.</p>
375<h3><a class="anchor" id="tests_running_tests_benchmarking_instruments"></a>
376Instruments</h3>
377<p>You can use the <code>--instruments</code> option to select one or more instruments to measure the execution time of the benchmark tests.</p>
378<p><code>PMU</code> will try to read the CPU PMU events from the kernel (They need to be enabled on your platform)</p>
379<p><code>MALI</code> will try to collect Mali hardware performance counters. (You need to have a recent enough Mali driver)</p>
Anthony Barbierf45d5a92018-01-24 16:23:15 +0000[diff] [blame]380<p><code>WALL_CLOCK_TIMER</code> will measure time using <code>gettimeofday</code>: this should work on all platforms.</p>
381<p>You can pass a combinations of these instruments: <code>--instruments=PMU,MALI,WALL_CLOCK_TIMER</code></p>
382<dl class="section note"><dt>Note</dt><dd>You need to make sure the instruments have been selected at compile time using the <code>pmu=1</code> or <code>mali=1</code> scons options.</dd></dl>
383<h3><a class="anchor" id="tests_running_examples"></a>
384Examples</h3>
385<p>To run all the precommit validation tests: </p><pre class="fragment">LD_LIBRARY_PATH=. ./arm_compute_validation --mode=precommit
386</pre><p>To run the OpenCL precommit validation tests: </p><pre class="fragment">LD_LIBRARY_PATH=. ./arm_compute_validation --mode=precommit --filter="^CL.*"
387</pre><p>To run the NEON precommit benchmark tests with PMU and Wall Clock timer in miliseconds instruments enabled: </p><pre class="fragment">LD_LIBRARY_PATH=. ./arm_compute_benchmark --mode=precommit --filter="^NEON.*" --instruments="pmu,wall_clock_timer_ms" --iterations=10
Jenkins514be652019-02-28 12:25:18 +0000[diff] [blame]388</pre><p>To run the OpenCL precommit benchmark tests with OpenCL kernel timers in miliseconds enabled: </p><pre class="fragment">LD_LIBRARY_PATH=. ./arm_compute_benchmark --mode=precommit --filter="^CL.*" --instruments="opencl_timer_ms" --iterations=10
389</pre> </div></div><!-- PageDoc -->
390</div><!-- contents -->
Anthony Barbierdbdab852017-06-23 15:42:00 +0100[diff] [blame]391</div><!-- doc-content -->
392<!-- start footer part -->
393<div id="nav-path" class="navpath"><!-- id is needed for treeview function! -->
394 <ul>
Jenkins7f09cf72020-01-22 18:08:16 +0000[diff] [blame^]395 <li class="footer">Generated on Wed Jan 22 2020 18:07:49 for Compute Library by
Anthony Barbierdbdab852017-06-23 15:42:00 +0100[diff] [blame]396 <a href="http://www.doxygen.org/index.html">
Jenkins514be652019-02-28 12:25:18 +0000[diff] [blame]397 <img class="footer" src="doxygen.png" alt="doxygen"/></a> 1.8.15 </li>
Anthony Barbierdbdab852017-06-23 15:42:00 +0100[diff] [blame]398 </ul>
399</div>
400</body>
401</html>