1 # Long traces with Perfetto 2 3 By default Perfetto keeps the full trace buffer in memory and writes it into the 4 destination file (passed with the `-o` cmdline argument) only at the end of the 5 trace, to reduce the intrusiveness of the tracing system. 6 That, however, limits the max size of the trace to the physical memory size of 7 the device. 8 In some cases (e.g., benchmarks, hard to repro cases) it is desirable to capture 9 traces that are way larger than that. 10 11 12 To achieve that, Perfetto allows to periodically flush the trace buffers into 13 the target file (or stdout) by using some flags in the 14 [`TraceConfig`](/protos/perfetto/config/trace_config.proto), specifically: 15 16 `bool write_into_file` 17 When true drains periodically the trace buffers into the output 18 file. When this option is enabled, the userspace buffers need to be just 19 big enough to hold tracing data between two periods. 20 The buffer sizing depends on the activity of the device. A reasonable estimation 21 is ~5-20 MB per second. 22 23 `uint32 file_write_period_ms` 24 Overrides the default drain period. Shorter periods require a smaller userspace 25 buffer but increase the performance intrusiveness of tracing. A minimum interval 26 of 100ms is enforced. 27 28 `uint64 max_file_size_bytes` 29 If set, stops the tracing session after N bytes have been written. Used to 30 cap the size of the trace. 31 32 For a complete example of a working trace config in long-tracing mode see 33 [`/test/configs/long_trace.cfg`](/test/configs/long_trace.cfg) 34 35 ## Instructions 36 These instructions assume you have a working standalone checkout (see 37 [instructions here](/docs/build-instructions.md)). 38 These instructions have been tested as non-root. Many of the steps below can be 39 simplified when running as root and are required due to SELinux when running as 40 `shell` rather than `root`. 41 42 ``` bash 43 $ cd perfetto 44 45 # Prepare for the build (as per instructions linked above). 46 $ tools/install-build-deps --no-android 47 $ tools/gn gen out/mac_release --args="is_debug=false" 48 49 # Compiles the textual protobuf into binary format 50 # for /test/configs/long_trace.cfg. 51 $ tools/ninja -C out/mac_release/ long_trace.cfg.protobuf 52 53 # Alternatively, the more verbose variant: 54 $ protoc=$(pwd)/out/mac_release/gcc_like_host/protoc 55 $ protoc --encode=perfetto.protos.TraceConfig \ 56 -I$(pwd)/protos \ 57 $(pwd)/protos/perfetto/config/perfetto_config.proto \ 58 < /test/configs/long_trace.cfg \ 59 > /tmp/long_trace.cfg.protobuf 60 61 # Push the config onto the device. 62 $ adb push out/mac_release/long_trace.cfg.protobuf /data/local/tmp/long_trace.cfg.protobuf 63 64 # Run perfetto. 65 # Note: Unless running as root, the output folder must be under 66 # /data/misc/perfetto-traces/, or SELinux will cause a failure. 67 $ adb shell 'cat /data/local/tmp/long_trace.cfg.protobuf | perfetto -c - -o /data/misc/perfetto-traces/trace --background' 68 69 # At this point the trace will start in background. It is possible to detach the 70 # usb cable. In order to verify if the trace is still ongoing, just run: 71 $ adb shell ps -A | grep perfetto 72 73 # While it's running, you should see an entry like this: 74 shell 23705 1 2166232 12344 do_sys_poll 7796ef2700 S perfetto 75 76 # At the end of the trace, the process will be gone. At this point it is 77 # possible to pull the trace. 78 $ adb shell gzip -c /data/misc/perfetto-traces/trace -3 | gzip -dc - > ~/trace 79 80 # Verify that the trace has not been corrupted by the adb transfer 81 $ adb shell sha1sum /data/misc/perfetto-traces/trace 82 b9f7a7e3d62638b5d9e880db30f68787c458bb3c /data/misc/perfetto-traces/trace 83 84 $ shasum ~/trace 85 b9f7a7e3d62638b5d9e880db30f68787c458bb3c /Users/primiano/trace 86 ``` 87 88 At this point it is possible to load / process the trace as explained below. 89 90 ### Get high-level trace stats 91 Use the `trace_to_text` binary in `summary` mode. This allows to detect whether 92 the buffers were sized appropriatel or any overrun happened, either in the 93 kernel ftrace buffer or in the userspace buffer. Look for 94 `Events overwritten`, `total_overrun` (kernel ftrace buffer) 95 and `chunks_overwritten` (userspace buffer). 96 97 `trace_to_text` can be also used to convert the trace from the protobuf format 98 to systrace textual version (see `trace_to_text --help`). 99 100 101 ``` bash 102 $ ninja -C out/mac_release trace_to_text 103 104 $ out/mac_release/trace_to_text summary < ~/trace 105 Ftrace duration: 29737ms 106 Boottime duration: 21000ms 107 -------------------- ftrace -------------------- 108 109 110 --------------------Ftrace Stats------------------- 111 Events overwritten: 0 112 total_overrun: 0 (= 0 - 0) 113 114 ----------------Process Tree Stats---------------- 115 Unique thread ids in process tree: 1372 116 Unique thread ids in ftrace events: 1580 117 Thread ids with process info: 1325/1580 -> 83 % 118 119 --------------------Trace Stats------------------- 120 Buffer 0 121 bytes_written: 37074960 122 chunks_written: 9087 123 chunks_overwritten: 0 124 ``` 125 126 ### Load the trace in the UI 127 128 *** note 129 **The UI and trace processor are WIP, targeting end of Q3-18**. 130 <!-- TODO(primiano): update this doc. --> 131 *** 132 133 Open https://ui.perfetto.dev and load the trace there. 134 135 ### Load the trace in the trace processor 136 ``` bash 137 $ ninja -C out/mac_release trace_processor 138 $ out/mac_release/trace_processor_shell ~/trace 139 140 trace_processor_shell.cc Trace loaded: 1048.58 MB (197.1 MB/s) 141 142 > select tdur/1e9 as runtime_sec, name from (select sum(dur) as tdur, utid from sched group by utid) inner join thread using(utid) order by tdur desc limit 20 143 runtime_sec name 144 -------------------- -------------------- 145 2560.695682 PowerManagerSer 146 2532.212882 migration/2 147 2529.064936 migration/3 148 2527.338100 migration/1 149 2526.877703 migration/4 150 2524.508852 migration/5 151 2523.372052 migration/6 152 2522.564051 migration/7 153 100.533405 traced_probes 154 80.585233 Binder:1229_E 155 46.294051 Chrome_IOThread 156 35.251236 .gms.persistent 157 27.716663 SensorService 158 26.818083 CrGpuMain 159 26.199466 m.chrome.canary 160 25.766977 CrRendererMain 161 25.623455 CrRendererMain 162 25.429535 android.bg 163 25.060462 traced 164 23.140243 lowpool[2633] 165 ``` 166
