1 Skia Gold 2 ========= 3 4 Overview 5 -------- 6 7 Gold is a web application that compares the images produced by our bots against 8 known baseline images. 9 Key features: 10 11 * Baselines are managed in Gold outside of Git, but in lockstep with Git commits. 12 * Each commit creates >500k images. 13 * Deviations from the baseline are triaged after a CL lands and images are triaged 14 as either `positive` or `negative`. If a CL causes Skia to break it is reverted 15 or an additional CL is landed to fix the problem. 16 * We test across a range of dimensions, e.g.: 17 18 - OS (Windows, Linux, Mac, Android, iOS) 19 - Architectures (Intel, ARM) 20 - Backends (CPU, OpenGL, Vulkan etc.) 21 - etc. 22 23 * Written in Go, Polymer and deployed on the Google Cloud. The code 24 is in the [Skia Infra Repository](https://github.com/google/skia-buildbot). 25 26 Recommended Workflows 27 --------------------- 28 ### How to best use Gold for commonly faced problems ### 29 30 These instructions will refer to various views which are accessible via the left 31 navigation on [gold.skia.org](https://gold.skia.org/). 32 View access is public, triage access is granted to 33 Skia contributors. You must be logged in to triage. 34 35 Problem #1: As sheriff, I need to triage and assign many incoming new images. 36 ------------------------------------------------------------------------------- 37 Solution today: 38 39 * Access the By Blame view to see digests needing triage and associated 40 owners/CLs 41 + Only untriaged digests will be shown by default 42 + Blame is not sorted in any particular order 43 + Digests are clustered by runs and the most minimal set of blame 44 45 <img src=BlameView.png style="margin-left:30px" align="left" width="800"/> <br clear="left"> 46 47 * Select digests for triage 48 + Digests will be listed in order with largest difference first 49 + Click to open the digest view with detailed information 50 51 <img src=Digests.png style="margin-left:40px" align="left" width="780"/> <br clear="left"> 52 53 * Open bugs for identified owner(s) 54 + The digest detail view has a link to open a bug from the UI 55 + Via the Gold UI or when manually entering a bug, copy the full URL of 56 single digest into a bug report 57 + The URL reference to the digest in Issue Tracker will link the bug to 58 the digest in Gold 59 60 <img src="IssueHighlight.png" style="margin-left:60px" align="left" width="720" border=1/> <br clear="left"> 61 62 <br> 63 64 Future improvements: 65 66 * Smarter, more granular blamelist 67 68 <br> 69 70 Problem #2: As a developer, I need to land a CL that may change many images. 71 ---------------------------------------------------------------------------- 72 To find your results: 73 74 * Immediately following commit, access the By Blame view to find untriaged 75 digest groupings associated with your ID 76 * Click on one of the clusters including your CL to triage 77 * Return to the By Blame view to walk through all untriaged digests involving 78 your change 79 * Note: It is not yet implemented in the UI but possible to filter the view 80 by CL. Delete hashes in the URL to only include the hash for your CL. 81 82 <img src=BlameView.png style="margin-left:30px" align="left" width="800"/> <br clear="left"> 83 84 To rebaseline images: 85 86 * Access the Ignores view and create a new, short-interval (hours) ignore for 87 the most affected configuration(s) 88 89 <img src=Ignores.png style="margin-left:30px" align="left" width="800"/> <br clear="left"> 90 91 92 * Click on the Ignore to bring up a search view filtered by the affected 93 configuration(s) 94 * Mark untriaged images as positive (or negative if appropriate) 95 * Follow one of two options for handling former positives: 96 + Leave former positives as-is and let them fall off with time if there is 97 low risk of recurrence 98 + Mark former positives as negative if needed to verify the change moving 99 forward 100 101 Future improvements: 102 103 * Trybot support prior to commit, with view limited to your CL 104 * Pre-triage prior to commit that will persist when the CL lands 105 106 <br> 107 108 Problem #3: As a developer or infrastructure engineer, I need to add a new or updated config. 109 --------------------------------------------------------------------------------------------- 110 (ie: new bot, test mode, environment change) 111 112 Solution today: 113 114 * Follow the process for rebaselining images: 115 + Wait for the bot/test/config to be committed and show up in the Gold UI 116 + Access the Ignores view and create a short-interval ignore for the 117 configuration(s) 118 + Triage the ignores for that config to identify positive images 119 + Delete the ignore 120 121 Future improvements: 122 123 * Introduction of a new or updated test can make use of try jobs and pre-triage. 124 * New configs may be able to use these features as well. 125 126 <br> 127 128 Problem #4: As a developer, I need to analyze the details of a particular image digest. 129 --------------------------------------------------------------------------------------- 130 Solution: 131 132 * Access the By Test view 133 134 <img src=ByTest.png style="margin-left:30px" align="left" width="800"/> <br clear="left"> 135 136 * Click the magnifier to filter by configuration 137 * Access the Cluster view to see the distribution of digest results 138 + Use control-click to select and do a direct compare between data points 139 + Click on configurations under parameters to highlight data points and 140 compare 141 142 <img src=ClusterConfig.png style="margin-left:30px" align="left" width="800"/> <br clear="left"> 143 144 * Access the Grid view to see NxN diffs 145 146 <img src=Grid.png style="margin-left:30px" align="left" width="800"/> <br clear="left"> 147 148 * Access the Dot diagram to see history of commits for the trace 149 + Each dot represents a commit 150 + Each line represents a configuration 151 + Dot colors distinguish between digests 152 153 <img src=DotDiagram.png style="margin-left:30px" align="left" width="800"/> <br clear="left"> 154 155 <br> 156 157 Future improvements: 158 159 * Large diff display of image vs image 160 161 <br> 162 163 Problem #5: As a developer, I need to find results for a particular configuration. 164 ---------------------------------------------------------------------------------- 165 Solution: 166 167 * Access the Search view 168 * Select any parameters desired to search across tests 169 170 <img src=Search.png style="margin-left:30px" align="left" width="800"/> <br clear="left"> 171
