1 <!-- Copyright 2015 The Chromium Authors. All rights reserved. 2 Use of this source code is governed by a BSD-style license that can be 3 found in the LICENSE file. 4 --> 5 6 # Telemetry 7 8 Telemetry is the performance testing framework used by Chrome. It allows you 9 to perform arbitrary actions on a set of web pages (or any android application!) 10 and report metrics about it. The framework abstracts: 11 12 * Launching a browser with arbitrary flags on any platform. 13 * Opening a tab and navigating to the page under test. 14 * Launching an Android application with intents through ADB. 15 * Fetching data via the Inspector timeline and traces. 16 * Using [Web Page Replay](https://github.com/chromium/web-page-replay) to 17 cache real-world websites so they dont change when used in benchmarks. 18 19 ## Design Principles 20 21 * Write one performance test that runs on major platforms - Windows, Mac, 22 Linux, Chrome OS, and Android for both Chrome and ContentShell. 23 * Run on browser binaries, without a full Chromium checkout, and without 24 having to build the browser yourself. 25 * Use Web Page Replay to get repeatable test results. 26 * Clean architecture for writing benchmarks that keeps measurements and use 27 cases separate. 28 29 **Telemetry is designed for measuring performance rather than checking 30 correctness. If you want to check for correctness, 31 [browser tests](http://www.chromium.org/developers/testing/browser-tests) are 32 your friend.** 33 34 **If you are a Chromium developer looking to add a new Telemetry benchmark to 35 [`src/tools/perf/`](https://code.google.com/p/chromium/codesearch#chromium/src/tools/perf/), 36 please make sure to read our 37 [Benchmark Policy](https://docs.google.com/document/d/1bBKyYCW3VlUUPDpQE4xvrMFdA6tovQMZoqO9KCcmqqQ/preview) 38 first.** 39 40 ## Code Concepts 41 42 Telemetry provides two major functionality groups: those that provide test 43 automation, and those that provide the capability to collect data. 44 45 ### Test Automation 46 47 The test automation facilities of Telemetry provide Python wrappers for a number 48 of different system concepts. 49 50 * _Platforms_ use a variety of libraries & tools to abstract away the OS 51 specific logic. 52 * _Browser_ wraps Chrome's 53 [DevTools Remote Debugging Protocol](https://developer.chrome.com/devtools/docs/remote-debugging) 54 to perform actions and extract information from the browser. 55 * _Android App_ is a Python wrapper around 56 [`adb shell`](http://developer.android.com/tools/help/adb.html). 57 58 The Telemetry framework lives in 59 [`src/third_party/catapult/telemetry/`](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/) 60 and performance benchmarks that use Telemetry live in 61 [`src/tools/perf/`](https://code.google.com/p/chromium/codesearch#chromium/src/tools/perf/). 62 63 ### Data Collection 64 65 Telemetry offers a framework for collecting metrics that quantify the 66 performance of automated actions in terms of benchmarks, measurements, and story 67 sets. 68 69 * A 70 [_benchmark_](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/benchmark.py) 71 combines a _measurement_ together with a _story set_, and optionally a set 72 of browser options. 73 * We strongly discourage benchmark authors from using command-line flags 74 to specify the behavior of benchmarks, since benchmarks should be 75 cross-platform. 76 * Benchmarks are discovered and run by the 77 [benchmark runner](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/benchmark_runner.py), 78 which is wrapped by scripts like 79 [`run_benchmark`](https://code.google.com/p/chromium/codesearch#chromium/src/tools/perf/run_benchmark) 80 in `tools/perf`. 81 * A _measurement_ (called 82 [`StoryTest`](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/web_perf/story_test.py) 83 in the code) is responsible for setting up and tearing down the testing 84 platform, and for collecting _metrics_ that quantify the application 85 scenario under test. 86 * Measurements need to work with all story sets, to provide consistency 87 and prevent benchmark rot. 88 * You probably don't need to override `StoryTest` (see "Timeline Based 89 Measurement" below). If you think you do, please talk to us. 90 * A 91 [_story set_](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/story/story_set.py) 92 is a set of _stories_ together with a 93 [_shared state_](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/story/shared_state.py) 94 that describes application-level configuration options. 95 * A 96 [_story_](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/story/story.py) 97 is an application scenario and a set of actions to run in that scenario. In 98 the typical Chromium use case, this will be a web page together with actions 99 like scrolling, clicking, or executing JavaScript. 100 * A _metric_ describes how to collect data about the story run and compute 101 results. 102 * New metrics should generally be 103 [timeline-based](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/web_perf/metrics/timeline_based_metric.py). 104 * Metrics can specify many different types of results, including numbers, 105 histograms, traces, and failures. 106 * _Timeline Based Measurement_ is a built-in `StoryTest` that runs all 107 available timeline-based metrics, and benchmarks that use it can filter 108 relevant results. 109 110 ## Next Steps 111 112 * [Run Telemetry benchmarks locally](/telemetry/docs/run_benchmarks_locally.md) 113 * [Record a story set](https://sites.google.com/a/chromium.org/dev/developers/telemetry/record_a_page_set) 114 with Web Page Replay 115 * [Add a measurement](https://sites.google.com/a/chromium.org/dev/developers/telemetry/add_a_measurement) 116 * [Feature guidelines](https://sites.google.com/a/chromium.org/dev/developers/telemetry/telemetry-feature-guidelines) 117 * [Profiling with Telemetry](https://sites.google.com/a/chromium.org/dev/developers/telemetry/profiling) 118 * [Profile generation](https://sites.google.com/a/chromium.org/dev/developers/telemetry/telemetry-profile-generation) 119 * [Telemetry unittests](https://sites.google.com/a/chromium.org/dev/developers/telemetry/telemetry-unittests) 120 121 ## Contact Us or Follow Along 122 123 If you have questions, please email telemetry (a] chromium.org. 124 125 You can keep up with Telemetry related discussions by joining the 126 [telemetry group](https://groups.google.com/a/chromium.org/forum/#!forum/telemetry). 127 128 [For Googlers](http://go/telemetry) 129 130 ## Frequently Asked Questions 131 132 ### I get an error when I try to use recorded story sets. 133 134 The recordings are not included in the Chromium source tree. If you are a Google 135 partner, run `gsutil config` to authenticate, then try running the test again. 136 If you don't have `gsutil` installed on your machine, you can find it in 137 `build/third_party/gsutil/gsutil`. 138 139 If you are not a Google partner, you can run on live sites with 140 --use-live-sites` or 141 [record your own](http://dev.chromium.org/developers/telemetry/record_a_page_set) 142 story set archive. 143 144 ### I get mysterious errors about device\_forwarder failing. 145 146 Your forwarder binary may be outdated. If you have built the forwarder in 147 src/out that one will be used. if there isn't anything there Telemetry will 148 default to downloading a pre-built binary. Try re-building the forwarder, or 149 alternatively wiping the contents of `src/out/` and running `run_benchmark`, 150 which should download the latest binary. 151 152 ### I'm having problems with keychain prompts on Mac. 153 154 Make sure that your keychain is 155 [correctly configured](https://sites.google.com/a/chromium.org/dev/developers/telemetry/telemetry-mac-keychain-setup). 156
