1 <html devsite> 2 <head> 3 <title>Automating the tests</title> 4 <meta name="project_path" value="/_project.yaml" /> 5 <meta name="book_path" value="/_book.yaml" /> 6 </head> 7 <body> 8 <!-- 9 Copyright 2017 The Android Open Source Project 10 11 Licensed under the Apache License, Version 2.0 (the "License"); 12 you may not use this file except in compliance with the License. 13 You may obtain a copy of the License at 14 15 http://www.apache.org/licenses/LICENSE-2.0 16 17 Unless required by applicable law or agreed to in writing, software 18 distributed under the License is distributed on an "AS IS" BASIS, 19 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 20 See the License for the specific language governing permissions and 21 limitations under the License. 22 --> 23 24 25 <p>Deqp test modules can be integrated to automated test systems in multiple ways. 26 The best approach depends on the existing test infrastructure and target 27 environment.</p> 28 29 <p>The primary output from a test run is always the test log file, i.e. the file 30 with a <code>.qpa</code> postfix. Full test results can be parsed from the test log. Console output is 31 debug information only and may not be available on all platforms.</p> 32 33 <p>Test binaries can be invoked directly from a test automation system. The test 34 binary can be launched for a specific case, for a test set, or for all 35 available tests. If a fatal error occurs during execution (such as certain API 36 errors or a crash), the test execution will abort. For regression testing, the 37 best approach is to invoke the test binaries for individual cases or small test 38 sets separately, in order to have partial results available even in the event 39 of hard failure.</p> 40 41 <p>The deqp comes with command line test execution tools that can be used in 42 combination with the execution service to achieve a more robust integration. 43 The executor detects test process termination and will resume test execution on 44 the next available case. A single log file is produced from the full test 45 session. This setup is ideal for lightweight test systems that dont provide 46 crash recovery facilities.</p> 47 48 <h2 id=command_line_test_execution_tools>Command line test execution tools</h2> 49 50 <p>The current command line tool set includes a remote test execution tool, a test 51 log comparison generator for regression analysis, a test-log-to-CSV converter, 52 a test-log-to-XML converter, and a testlog-to-JUnit converter.</p> 53 54 <p>The source code for these tools is in the <code>executor</code> directory, and the binaries are built into the <code><builddir>/executor</code> directory.</p> 55 56 <h3 id=command_line_test_executor>Command line Test Executor</h3> 57 58 <p>The command line Test Executor is a portable C++ tool for launching a test run 59 on a device and collecting the resulting logs from it over TCP/IP. The Executor 60 communicates with the execution service (execserver) on the target device. 61 Together they provide functionality such as recovery from test process crashes. 62 The following examples demonstrate how to use the command line Test Executor 63 (use <code>--help</code> for more details):</p> 64 65 <h4 id=example_1_run_gles2_functional_tests>Example 1: Run GLES2 functional tests on an Android device:</h4> 66 67 <pre class="devsite-terminal devsite-click-to-copy"> 68 executor --connect=127.0.0.1 --port=50016 --binaryname= 69 com.drawelements.deqp/android.app.NativeActivity 70 --caselistdir=caselists 71 --testset=dEQP-GLES2.* --out=BatchResult.qpa 72 --cmdline="--deqp-crashhandler=enable --deqp-watchdog=enable 73 --deqp-gl-config-name=rgba8888d24s8" 74 </pre> 75 76 <h4 id=example_2_continue_a_partial_opengl>Example 2: Continue a partial OpenGL ES 2 test run locally:</h4> 77 78 <pre class="devsite-terminal devsite-click-to-copy"> 79 executor --start-server=execserver/execserver --port=50016 80 --binaryname=deqp-gles2 --workdir=modules/opengl 81 --caselistdir=caselists 82 --testset=dEQP-GLES2.* --exclude=dEQP-GLES2.performance.* --in=BatchResult.qpa 83 --out=BatchResult.qpa 84 </pre> 85 86 <h3 id=test_log_csv_export_and_compare>Test log CSV export and compare</h3> 87 88 <p>The deqp has a tool for converting test logs (.<code>qpa </code>files) into CSV files. The CSV output contains a list of test cases and their 89 results. The tool can also compare two or more batch results and list only the 90 test cases that have different status codes in the input batch results. The 91 comparison will also print the number of matching cases.</p> 92 93 <p>The output in CSV format is very practical for further processing with standard 94 command line utilities or with a spreadsheet editor. An additional, human-readable, 95 plain-text format can be selected using the following command line argument: <code>--format=text</code></p> 96 97 <h4 id=example_1_export_test_log_in_csv_format>Example 1: Export test log in CSV format</h4> 98 99 <pre class="devsite-click-to-copy"> 100 <code class="devsite-terminal">testlog-to-csv --value=code BatchResult.qpa > Result_statuscodes.csv</code> 101 <code class="devsite-terminal">testlog-to-csv --value=details BatchResult.qpa > Result_statusdetails.csv</code> 102 </pre> 103 104 <h4 id=example_2_list_differences>Example 2: List differences of test results between two test logs</h4> 105 106 <pre class="devsite-terminal devsite-click-to-copy"> 107 testlog-to-csv --mode=diff --format=text Device_v1.qpa Device_v2.qpa 108 </pre> 109 110 <p class="note"><strong>Note:</strong> The argument <code>--value=code</code> outputs the test result code, such as "Pass" or "Fail". The argument <code>--value=details</code> selects the further explanation of the result or numerical value produced by a performance, capability, or accuracy test.</p> 111 112 <h3 id=test_log_xml_export>Test log XML export</h3> 113 114 <p>Test log files can be converted to valid XML documents using the <code>testlog-to-xml</code> utility. Two output modes are supported: </p> 115 116 <ul> 117 <li> Separate documents mode, where each test case and the <code>caselist.xml</code> summary document are written to a destination directory 118 <li> Single file mode, where all results in the <code>.qpa</code> file are written to single XML document. 119 </ul> 120 121 <p>Exported test log files can be viewed in a browser using an XML style sheet. 122 Sample style sheet documents (<code>testlog.xsl</code> and <code>testlog.css</code>) are provided in the <code>doc/testlog-stylesheet</code> directory. To render the log files in a browser, copy the two style sheet 123 files into the same directory where the exported XML documents are located.</p> 124 125 <p>If you are using Google Chrome, the files must be accessed over HTTP as Chrome 126 limits local file access for security reasons. The standard Python installation 127 includes a basic HTTP server that can be launched to serve the current 128 directory with the <code>python m SimpleHTTPServer 8000</code> command. After launching the server, just point the Chrome browser to <code>http://localhost:8000</code> to view the test log.</p> 129 130 <h3 id=conversion_to_a_junit_test_log>Conversion to a JUnit test log</h3> 131 132 <p>Many test automation systems can generate test run result reports from JUnit 133 output. The deqp test log files can be converted to the JUnit output format 134 using the testlog-to-junit tool. </p> 135 136 <p>The tool currently supports translating the test case verdict only. As JUnit 137 supports only "pass" and "fail" results, a passing result of the deqp is mapped 138 to "JUnit pass" and other results are considered failures. The original deqp 139 result code is available in the JUnit output. Other data, such as log messages 140 and result images, are not preserved in the conversion.</p> 141 142 </body> 143 </html> 144
