1 # Flicker Test Library 2 3 ## Motivation 4 Detect *flicker* — any discontinuous, or unpredictable behavior seen during UI transitions that is not due to performance. This is often the result of a logic error in the code and difficult to identify because the issue is transient and at times difficult to reproduce. This library helps create integration tests between `SurfaceFlinger`, `WindowManager` and `SystemUI` to identify flicker. 5 6 ## Adding a Test 7 The library builds and runs UI transitions, captures Winscope traces and exposes common assertions that can be tested against each trace. 8 9 ### Building Transitions 10 Start by defining common or error prone transitions using `TransitionRunner`. 11 ```java 12 // Example: Build a transition that cold launches an app from launcher 13 TransitionRunner transition = TransitionRunner.newBuilder() 14 // Specify a tag to identify the transition (optional) 15 .withTag("OpenAppCold_" + testApp.getLauncherName()) 16 17 // Specify preconditions to setup the device 18 // Wake up device and go to home screen 19 .runBeforeAll(AutomationUtils::wakeUpAndGoToHomeScreen) 20 21 // Setup transition under test 22 // Press the home button and close the app to test a cold start 23 .runBefore(device::pressHome) 24 .runBefore(testApp::exit) 25 26 // Run the transition under test 27 // Open the app and wait for UI to be idle 28 // This is the part of the transition that will be tested. 29 .run(testApp::open) 30 .run(device::waitForIdle) 31 32 // Perform any tear downs 33 // Close the app 34 .runAfterAll(testApp::exit) 35 36 // Number of times to repeat the transition to catch any flaky issues 37 .repeat(5); 38 ``` 39 40 41 Run the transition to get a list of `TransitionResult` for each time the transition is repeated. 42 ```java 43 List<TransitionResult> results = transition.run(); 44 ``` 45 `TransitionResult` contains paths to test artifacts such as Winscope traces and screen recordings. 46 47 48 ### Checking Assertions 49 Each `TransitionResult` can be tested using an extension of the Google Truth library, `LayersTraceSubject` and `WmTraceSubject`. They try to balance test principles set out by Google Truth (not supporting nested assertions, keeping assertions simple) with providing support for common assertion use cases. 50 51 Each trace can be represented as a ordered collection of trace entries, with an associated timestamp. Each trace entry has common assertion checks. The trace subjects expose methods to filter the range of entries and test for changing assertions. 52 53 ```java 54 TransitionResult result = results.get(0); 55 Rect displayBounds = getDisplayBounds(); 56 57 // check all trace entries 58 assertThat(result).coversRegion(displayBounds).forAllEntries(); 59 60 // check a range of entries 61 assertThat(result).coversRegion(displayBounds).forRange(startTime, endTime); 62 63 // check first entry 64 assertThat(result).coversRegion(displayBounds).inTheBeginning(); 65 66 // check last entry 67 assertThat(result).coversRegion(displayBounds).atTheEnd(); 68 69 // check a change in assertions, e.g. wallpaper window is visible, 70 // then wallpaper window becomes and stays invisible 71 assertThat(result) 72 .showsBelowAppWindow("wallpaper") 73 .then() 74 .hidesBelowAppWindow("wallpaper") 75 .forAllEntries(); 76 ``` 77 78 All assertions return `Result` which contains a `success` flag, `assertionName` string identifier, and `reason` string to provide actionable details to the user. The `reason` string is build along the way with all the details as to why the assertions failed and any hints which might help the user determine the root cause. Failed assertion message will also contain a path to the trace that was tested. Example of a failed test: 79 80 ``` 81 java.lang.AssertionError: Not true that <com.android.server.wm.flicker.LayersTrace@65da4cc> 82 Layers Trace can be found in: /layers_trace_emptyregion.pb 83 Timestamp: 2308008331271 84 Assertion: coversRegion 85 Reason: Region to test: Rect(0, 0 - 1440, 2880) 86 first empty point: 0, 99 87 visible regions: 88 StatusBar#0Rect(0, 0 - 1440, 98) 89 NavigationBar#0Rect(0, 2712 - 1440, 2880) 90 ScreenDecorOverlay#0Rect(0, 0 - 1440, 91) 91 ... 92 at com.google.common.truth.FailureStrategy.fail(FailureStrategy.java:24) 93 ... 94 ``` 95 96 --- 97 98 ## Running Tests 99 100 The tests can be run as any other Android JUnit tests. `platform_testing/tests/flicker` uses the library to test common UI transitions. Run `atest FlickerTest` to execute these tests. 101 102 --- 103 104 ## Other Topics 105 ### Monitors 106 Monitors capture test artifacts for each transition run. They are started before each iteration of the test transition (after the `runBefore` calls) and stopped after the transition is completed. Each iteration will produce a new test artifact. The following monitors are available: 107 108 #### LayersTraceMonitor 109 Captures Layers trace. This monitor is started by default. Build a transition with `skipLayersTrace()` to disable this monitor. 110 #### WindowManagerTraceMonitor 111 Captures Window Manager trace. This monitor is started by default. Build a transition with `skipWindowManagerTrace()` to disable this monitor. 112 #### WindowAnimationFrameStatsMonitor 113 Captures WindowAnimationFrameStats for the transition. This monitor is started by default and is used to eliminate *janky* runs. If an iteration has skipped frames, as determined by WindowAnimationFrameStats, the results for the iteration is skipped. If the list of results is empty after all iterations are completed, then the test should fail. Build a transition with `includeJankyRuns()` to disable this monitor. 114 #### ScreenRecorder 115 Captures screen to a video file. This monitor is disabled by default. Build a transition with `recordEachRun()` to capture each transition or build with `recordAllRuns()` to capture every transition including setup and teardown. 116 117 --- 118 119 ### Extending Assertions 120 121 To add a new assertion, add a function to one of the trace entry classes, `LayersTrace.Entry` or `WindowManagerTrace.Entry`. 122 123 ```java 124 // Example adds an assertion to the check if layer is hidden by parent. 125 Result isHiddenByParent(String layerName) { 126 // Result should contain a details if assertion fails for any reason 127 // such as if layer is not found or layer is not hidden by parent 128 // or layer has no parent. 129 // ... 130 } 131 ``` 132 Then add a function to the trace subject `LayersTraceSubject` or `WmTraceSubject` which will add the assertion for testing. When the assertion is evaluated, the trace will first be filtered then the assertion will be applied to the remaining entries. 133 134 ```java 135 public LayersTraceSubject isHiddenByParent(String layerName) { 136 mChecker.add(entry -> entry.isHiddenByParent(layerName), 137 "isHiddenByParent(" + layerName + ")"); 138 return this; 139 } 140 ``` 141 142 To use the new assertion: 143 ```java 144 // Check if "Chrome" layer is hidden by parent in the first trace entry. 145 assertThat(result).isHiddenByParent("Chrome").inTheBeginning(); 146 ```
