1 page.title=Interpreting CTS results 2 @jd:body 3 4 <!-- 5 Copyright 2015 The Android Open Source Project 6 7 Licensed under the Apache License, Version 2.0 (the "License"); 8 you may not use this file except in compliance with the License. 9 You may obtain a copy of the License at 10 11 http://www.apache.org/licenses/LICENSE-2.0 12 13 Unless required by applicable law or agreed to in writing, software 14 distributed under the License is distributed on an "AS IS" BASIS, 15 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 See the License for the specific language governing permissions and 17 limitations under the License. 18 --> 19 20 <p>The CTS test results are placed in the file:</p> 21 <pre> 22 $CTS_ROOT/android-cts/repository/results/<start_time>.zip 23 </pre> 24 25 <p>If you have built the CTS yourself, <em>$CTS_ROOT</em> will resemble the 26 path <em>out/host/linux-x86/cts</em> but differ by platform. This reflects the 27 path where you have uncompressed the prebuilt official CTS <a 28 href="downloads.html">downloaded</a> from this site.</p> 29 30 <p>Inside the zip, the testResult.xml file contains the actual results. Open 31 this file in any web browser (HTML5 compatible browser recommended) to view the 32 test results.</p> 33 34 <p>If testResult.xml displays a blank page when using the Chrome browser, 35 <a href="https://www.chromium.org/developers/how-tos/run-chromium-with-flags">change 36 your browser configuration</a> to enable the 37 <em>--allow-file-access-from-files</em> command line flag.</p> 38 39 <h3 id="reading-the-test-results">Reading the test results</h3> 40 41 <p>The details of the test results depend on which version of CTS you are 42 using:</p> 43 44 <ul> 45 <li>CTS v1 for Android 6.0 and earlier</a> 46 <li>CTS v2 for Android 7.0 and later</a> 47 </ul> 48 49 <p class="note"><strong>Note:</strong> The results are provided to help you 50 ensure the software remains compatible throughout the development process and 51 act as a common format for communicating the compatibility status of your 52 device with other parties.</p> 53 54 <h4 id="device-information">Device Information</h4> 55 56 <p>In CTS v6.0 and earlier, select Device Information (link above Test Summary) to 57 view details about the device, firmware (make, model, firmware build, 58 platform), and device hardware (screen resolution, keypad, screen type). (CTS 59 v7.0 does not display device information.)</p> 60 61 <h4 id="test-summary">Test Summary</h4> 62 63 <p>The <em>Test Summary</em> section provides executed test plan details, like 64 the CTS plan name and execution start and end times. It also presents an 65 aggregate summary of the number of tests that passed, failed, timed out, or 66 could not be executed.</p> 67 68 <div class="toggle-content closed"> 69 <p><a href="#" onclick="return toggleContent(this)"> 70 <img src="{@docRoot}assets/images/triangle-closed.png" class="toggle-content-img" /> 71 <strong><span class="toggle-content-text">Show CTS v1 sample test summary</span> 72 <span class="toggle-content-text" style="display:none;">Hide CTS v1 sample test summary</span></strong> 73 </a></p> 74 <div class="toggle-content-toggleme"> 75 <img src="images/cts-test-summary.png" alt="CTS v1 test summary" id="figure1a" /> 76 <p class="img-caption"><strong>Figure 1a.</strong> CTS v1 sample test summary </p> 77 </div> 78 </div> 79 80 81 <div class="toggle-content closed"> 82 <p><a href="#" onclick="return toggleContent(this)"> 83 <img src="{@docRoot}assets/images/triangle-closed.png" class="toggle-content-img" /> 84 <strong><span class="toggle-content-text">Show CTS v2 sample test summary</span> 85 <span class="toggle-content-text" style="display:none;">Hide CTS v2 sample test summary</span></strong> 86 </a></p> 87 <div class="toggle-content-toggleme"> 88 <img src="images/cts-v2-test-summary.png" alt="CTS v2 test summary" id="figure1b" /> 89 <p class="img-caption"><strong>Figure 1b.</strong> CTS v2 sample test summary </p> 90 </div> 91 </div> 92 93 <p> </p> 94 95 96 <h4 id="test-report">Test Report</h4> 97 <p>The next section, the CTS test report, provides a summary of tests passed per package.</p> 98 99 <p>This is followed by details of the the actual tests that were executed. The 100 report lists the test package, test suite, test case, and the executed tests. 101 It shows the result of the test executionpass, fail, timed out, or not 102 executed. In the event of a test failure details are provided to help diagnose 103 the cause.</p> 104 105 <p>Further, the stack trace of the failure is available in the XML file but is 106 not included in the report to ensure brevityviewing the XML file with a text 107 editor should provide details of the test failure (search for the 108 <em><Test></em> tag corresponding to the failed test and look within it 109 for the <em><StackTrace></em> tag).</p> 110 111 <div class="toggle-content closed"> 112 <p><a href="#" onclick="return toggleContent(this)"> 113 <img src="{@docRoot}assets/images/triangle-closed.png" class="toggle-content-img" /> 114 <strong><span class="toggle-content-text">Show CTS v1 sample test report</span> 115 <span class="toggle-content-text" style="display:none;">Hide CTS v1 sample test report</span></strong> 116 </a></p> 117 <div class="toggle-content-toggleme"> 118 <img src="images/cts-test-report.png" alt="CTS v1 test report" id="figure2a" /> 119 <p class="img-caption"><strong>Figure 2a.</strong> CTS v1 sample test report</p> 120 </div> 121 </div> 122 123 <div class="toggle-content closed"> 124 <p><a href="#" onclick="return toggleContent(this)"> 125 <img src="{@docRoot}assets/images/triangle-closed.png" class="toggle-content-img" /> 126 <strong><span class="toggle-content-text">Show CTS v2 sample test report</span> 127 <span class="toggle-content-text" style="display:none;">Hide CTS v2 sample test report</span></strong> 128 </a></p> 129 <div class="toggle-content-toggleme"> 130 <img src="images/cts-v2-test-report.png" alt="CTS v2 test report" id="figure2b" /> 131 <p class="img-caption"><strong>Figure 2b.</strong> CTS v2 sample test report</p> 132 </div> 133 </div> 134 135 <p> </p> 136 137 <h3 id="incomplete-test-modules">Reviewing test_result.xml for incomplete test modules</h3> 138 139 <p> 140 To determine the number of incomplete modules in a given test session, run 141 command 'list results'. The count of Modules Completed and Total Modules are 142 listed for each previous session. To determine which modules are complete vs. 143 incomplete, open the test_result.xml file and read the value of the "done" 144 attribute for each module in the result report. Modules with value done = 145 "false" have not run to completion. 146 </p> 147
