1 Skia Gold 2 ========= 3 4 Overview 5 -------- 6 7 Gold is a web application that compares the images produced by our bots against 8 known baseline images. 9 Key features: 10 11 * Baselines are managed in Gold outside of Git, but in lockstep with Git commits. 12 * Each commit creates >500k images. 13 * Deviations from the baseline are triaged after a CL lands and images are 14 triaged as either `positive` or `negative`. 'Positive' means the diff is 15 considered acceptable. 'Negative' means the diff is considered unacceptable 16 and requires a fix. 17 If a CL causes Skia to break it is reverted or an additional CL is landed to 18 fix the problem. 19 * We test across a range of dimensions, e.g.: 20 21 - OS (Windows, Linux, Mac, Android, iOS) 22 - Architectures (Intel, ARM) 23 - Backends (CPU, OpenGL, Vulkan etc.) 24 - etc. 25 26 * Written in Go, Polymer and deployed on the Google Cloud. The code 27 is in the [Skia Infra Repository](https://github.com/google/skia-buildbot). 28 29 Recommended Workflows 30 --------------------- 31 ### How to best use Gold for commonly faced problems ### 32 33 These instructions will refer to various views which are accessible via the left 34 navigation on [gold.skia.org](https://gold.skia.org/). 35 View access is public, triage access is granted to 36 Skia contributors. You must be logged in to triage. 37 38 Problem #1: As sheriff, I need to triage and assign many incoming new images. 39 ------------------------------------------------------------------------------- 40 Solution today: 41 42 * Access the By Blame view to see digests needing triage and associated 43 owners/CLs 44 + Only untriaged digests will be shown by default 45 + Blame is not sorted in any particular order 46 + Digests are clustered by runs and the most minimal set of blame 47 48 <img src=BlameView.png style="margin-left:30px" align="left" width="800"/> <br clear="left"> 49 50 * Select digests for triage 51 + Digests will be listed in order with largest difference first 52 + Click to open the digest view with detailed information 53 54 <img src=Digests.png style="margin-left:40px" align="left" width="780"/> <br clear="left"> 55 56 * Open bugs for identified owner(s) 57 + The digest detail view has a link to open a bug from the UI 58 + Via the Gold UI or when manually entering a bug, copy the full URL of 59 single digest into a bug report 60 + The URL reference to the digest in Issue Tracker will link the bug to 61 the digest in Gold 62 63 <img src="IssueHighlight.png" style="margin-left:60px" align="left" width="720" border=1/> <br clear="left"> 64 65 <br> 66 67 Future improvements: 68 69 * Smarter, more granular blamelist 70 71 <br> 72 73 Problem #2: As a developer, I need to land a CL that may change many images. 74 ---------------------------------------------------------------------------- 75 To find your results: 76 77 * Immediately following commit, access the By Blame view to find untriaged 78 digest groupings associated with your ID 79 * Click on one of the clusters including your CL to triage 80 * Return to the By Blame view to walk through all untriaged digests involving 81 your change 82 * Note: It is not yet implemented in the UI but possible to filter the view 83 by CL. Delete hashes in the URL to only include the hash for your CL. 84 85 <img src=BlameView.png style="margin-left:30px" align="left" width="800"/> <br clear="left"> 86 87 To rebaseline images: 88 89 * Access the Ignores view and create a new, short-interval (hours) ignore for 90 the most affected configuration(s) 91 92 <img src=Ignores.png style="margin-left:30px" align="left" width="800"/> <br clear="left"> 93 94 95 * Click on the Ignore to bring up a search view filtered by the affected 96 configuration(s) 97 * Mark untriaged images as positive (or negative if appropriate) 98 * Follow one of two options for handling former positives: 99 + Leave former positives as-is and let them fall off with time if there is 100 low risk of recurrence 101 + Mark former positives as negative if needed to verify the change moving 102 forward 103 104 Future improvements: 105 106 * Trybot support prior to commit, with view limited to your CL 107 * Pre-triage prior to commit that will persist when the CL lands 108 109 <br> 110 111 Problem #3: As a developer or infrastructure engineer, I need to add a new or updated config. 112 --------------------------------------------------------------------------------------------- 113 (ie: new bot, test mode, environment change) 114 115 Solution today: 116 117 * Follow the process for rebaselining images: 118 + Wait for the bot/test/config to be committed and show up in the Gold UI 119 + Access the Ignores view and create a short-interval ignore for the 120 configuration(s) 121 + Triage the ignores for that config to identify positive images 122 + Delete the ignore 123 124 Future improvements: 125 126 * Introduction of a new or updated test can make use of try jobs and pre-triage. 127 * New configs may be able to use these features as well. 128 129 <br> 130 131 Problem #4: As a developer, I need to analyze the details of a particular image digest. 132 --------------------------------------------------------------------------------------- 133 Solution: 134 135 * Access the By Test view 136 137 <img src=ByTest.png style="margin-left:30px" align="left" width="800"/> <br clear="left"> 138 139 * Click the magnifier to filter by configuration 140 * Access the Cluster view to see the distribution of digest results 141 + Use control-click to select and do a direct compare between data points 142 + Click on configurations under parameters to highlight data points and 143 compare 144 145 <img src=ClusterConfig.png style="margin-left:30px" align="left" width="800"/> <br clear="left"> 146 147 * Access the Grid view to see NxN diffs 148 149 <img src=Grid.png style="margin-left:30px" align="left" width="800"/> <br clear="left"> 150 151 * Access the Dot diagram to see history of commits for the trace 152 + Each dot represents a commit 153 + Each line represents a configuration 154 + Dot colors distinguish between digests 155 156 <img src=DotDiagram.png style="margin-left:30px" align="left" width="800"/> <br clear="left"> 157 158 <br> 159 160 Future improvements: 161 162 * Large diff display of image vs image 163 164 <br> 165 166 Problem #5: As a developer, I need to find results for a particular configuration. 167 ---------------------------------------------------------------------------------- 168 Solution: 169 170 * Access the Search view 171 * Select any parameters desired to search across tests 172 173 <img src=Search.png style="margin-left:30px" align="left" width="800"/> <br clear="left"> 174
