1 <!-- Copyright 2016 The Chromium Authors. All rights reserved. 2 Use of this source code is governed by a BSD-style license that can be 3 found in the LICENSE file. 4 --> 5 6 Timeline-Based Measurement v2 is a system for computing metrics from traces. 7 8 A TBM2 metric is a Javascript function that takes a trace Model and produces 9 Histograms. 10 11 12 Coding Practices 13 ============== 14 15 Please follow the [Catapult Javascript style guide](/docs/style-guide.md) so 16 that the TBM2 maintainers can refactor your metric when we need to update the TBM2 17 API. 18 19 Please write a unit test for your metric. 20 21 If your metric computes information from the trace that may be of general use to 22 other metrics or the trace viewer UI, then the TBM2 maintainers may ask for your 23 help to generalize your innovation into a part of the Trace Model such as the 24 [UserModel](/tracing/tracing/model/user_model/user_model.html) or 25 [ModelHelpers](/tracing/tracing/model/helpers/chrome_browser_helper.html). 26 27 Use the dev server to develop and debug your metric. 28 29 * Run `./bin/run_dev_server` 30 * Navigate to 31 [http://localhost:8003/tracing_examples/trace_viewer.html](http://localhost:8003/tracing_examples/trace_viewer.html). 32 * Open a trace that your metric can be computed from. 33 * Open the Metrics side panel on the right. 34 * Select your metric from the drop-down. 35 * Inspect the results and change your metric if necessary. 36 * Open different traces to explore corner cases in your metric. 37 38 39 Trace Model 40 =========== 41 42 Trace logs are JSON files produced by tracing systems in Chrome, Android, linux 43 perf, BattOr, etc. The trace model is an object-level representation of events 44 parsed from a trace log. The trace model contains Javascript objects 45 representing 46 47 * OS [processes](/tracing/tracing/model/process.html), 48 [threads](/tracing/tracing/model/thread.html), 49 * utilities for finding special processes and threads such as 50 [ChromeBrowserHelper](/tracing/tracing/model/helpers/chrome_browser_helper.html), 51 * synchronous [ThreadSlices](/tracing/tracing/model/thread_slice.html) 52 and asynchronous [AsyncSlices](/tracing/tracing/model/async_slice.html), 53 * [snapshots](/tracing/tracing/model/object_snapshot.html) of object state as it changes throughout time, 54 * [RPCs](/tracing/tracing/model/flow_event.html), 55 * [FrameBlameContexts](/tracing/tracing/extras/chrome/blame_context/blame_context.html), 56 * battery [power samples](/tracing/tracing/model/power_sample.html), 57 * synthetic higher-level abstractions representing complex sets of 58 events such as 59 [UserExpectations](/tracing/tracing/model/user_model/user_expectation.html), 60 * and [more](/tracing/tracing/model/model.html)! 61 62 63 Histograms 64 ========== 65 66 A [Histogram](/tracing/tracing/value/histogram.html) is basically a common 67 [histogram](https://en.wikipedia.org/wiki/Histogram), but with a few extra bells 68 and whistles that are particularly useful for TBM2 metrics. 69 70 * Specify units of samples and improvement direction with 71 [Unit](/tracing/tracing/value/unit.html) 72 * JSON serialization with asDict()/fromDict() 73 * Build custom bin boundaries with HistogramBinBoundaries 74 * Compute statistics such as average, stddev, sum, and percentiles 75 * Customize which statistics are serialized with customizeSummaryOptions() 76 * Count non-numeric samples 77 * Store a random subset of sample values 78 * getDifferenceSignificance() computes whether two histograms are significantly 79 different with a Mann-Whitney U hypothesis test 80 * addHistogram() merges two Histograms with the same units and bin boundaries 81 82 But the most complex special feature of Histograms is their Diagnostics. 83 84 85 Diagnostics 86 =========== 87 88 When a metric significantly regresses, you then need to diagnose why it 89 regressed. Diagnostics are pieces of information that metrics attach to 90 Histograms in order help you diagnose regressions. Diagnostics may be associated 91 either with the entire Histogram directly, or with a particular sample. 92 93 Attach a Diagnostic to a Histogram: 94 95 ```javascript 96 histogram.diagnostics.set('name', diagnostic) 97 // or 98 values.addHistogram(histogram, {name: diagnostic}) 99 ``` 100 101 Attach a Diagnostic to a sample: 102 103 ```javascript 104 histogram.addSample(number, {name: diagnostic}) 105 ``` 106 107 The types of Diagnostics are 108 109 * [Generic](/tracing/tracing/value/diagnostics/generic.html): This can contain 110 any data that can be serialized and deserialized using JSON.stringify() and 111 JSON.parse(), including numbers, strings, Arrays, and dictionaries (simple 112 Objects). It will be visualized using 113 [generic-object-view](/tracing/tracing/ui/analysis/generic_object_view.html), 114 which is quite smart about displaying tabular data using tables, URLs using 115 HTML anchor tags, pretty-printing, recursive data structures, and more. 116 * [RelatedEventSet](/tracing/tracing/value/diagnostics/related_event_set.html): 117 This is a Set of references to Events in the trace model. Visually, they 118 are displayed as HTML links which, when clicked in the metrics-side-panel, 119 select the referenced Events in the trace viewer's timeline view. When 120 clicked in results2.html, they currently do nothing, but should eventually 121 open the trace that contains the events and select them. 122 * [Breakdown](/tracing/tracing/value/diagnostics/breakdown.html): 123 Structurally, these are Maps from strings to numbers. Conceptually, they 124 describe what fraction of a whole (either a Histogram or a sample) is due to 125 some sort of category - either a category of event, CPU sample, memory 126 consumer, whathaveyou. Visually, they are a stacked bar chart with a single 127 bar, which is spiritually a pie chart, but less misleading. 128 * [RelatedValueSet](/tracing/tracing/value/diagnostics/related_value_set.html): 129 These are Sets of references to other Histograms. Visually, they are a set 130 of HTML links which, when clicked, select the contained Histograms. The text 131 content of the HTML link is the name of the referenced Histogram. 132 * [RelatedValueMap](/tracing/tracing/value/diagnostics/related_value_map.html): 133 These are Maps from strings to references to other Histograms. Visually, they 134 are a set of HTML links similar to RelatedValueSet, but the text content of 135 the link is the Map's string key instead of the Histogram's name. One example 136 application is when a Histogram was produced not directly by a metric, but 137 rather by merging together other Histograms, then it will have a 138 RelatedValueMap named 'merged from' that refers to the Histograms that were 139 merged by their grouping key, e.g. the telemetry story name. 140 * [RelatedHistogramBreakdown](/tracing/tracing/value/diagnostics/related_histogram_breakdown.html): 141 Structurally, this is a RelatedValueMap, but conceptually and visually, this 142 is a Breakdown. Whereas Breakdown's stacked bar chart derives its data from 143 the numbers contained explicitly in the Breakdown, a 144 RelatedHistogramBreakdown's stacked 145 bar chart derives its data from the referenced Histograms' sums. 146 * [IterationInfo](/tracing/tracing/value/diagnostics/iteration_info.html): 147 This is automatically attached to every Histogram produced by telemetry. 148 Structurally, it's a class with explicit named fields. 149 Conceptually, it contains information about the origins of the trace that was 150 consumed by the metric that produced the Histogram, such as the benchmark 151 name, story name, benchmark start timestamp, OS version, Chrome version, etc. 152 Visually, IterationInfos are displayed as a table. 153 154 155 Consumers of Histograms 156 ======================= 157 158 Histograms are consumed by 159 160 * [value-set-table](/tracing/tracing/value/ui/value-set-table.html) in both 161 results2.html and the Metrics side panel in trace viewer, 162 * the [dashboard](https://chromeperf.appspot.com) indirectly via their statistics. 163 164 Currently, telemetry discards Histograms and Diagnostics, and only passes their 165 statistics scalars to the dashboard. Histograms and their Diagnostics will be 166 passed directly to the dashboard early 2017. 167
