1 Vulkan CTS README
2 =================
3
4 This document describes how to build and run Vulkan Conformance Test suite.
5
6 Vulkan CTS is built on dEQP framework. dEQP documentation is available
7 at http://source.android.com/devices/graphics/testing.html
8
9
10 Requirements
11 ------------
12
13 ### Common
14
15 * Git (for checking out sources)
16 * Python 2.7.x (all recent versions in 2.x should work, 3.x is not supported)
17 * CMake 2.8 (3.2 for Android NDK r15 builds) or newer
18
19 ### Win32
20
21 * Visual Studio 2013 or newer (glslang uses several C++11 features)
22
23 ### Linux
24
25 * Standard toolchain (make, gcc/clang)
26
27 ### Android
28
29 * Android NDK r11c (NDK r15 support is experimental)
30 * Android SDK with: SDK Tools, SDK Platform-tools, SDK Build-tools, and API 22
31 * Java Development Kit (JDK)
32 * Windows: either NMake or Ninja in PATH
33
34 If you have downloaded Android SDK tools, you can install necessary components
35 by running:
36
37 tools/android update sdk --no-ui --all --filter tools,platform-tools,build-tools-25.0.2,android-22
38
39
40 Building CTS
41 ------------
42
43 To build dEQP, you need first to download sources for zlib, libpng, glslang,
44 and spirv-tools.
45
46 To download sources, run:
47
48 python external/fetch_sources.py
49
50 You may need to re-run `fetch_sources.py` to update to the latest glslang and
51 spirv-tools revisions occasionally.
52
53 With CMake out-of-source builds are always recommended. Create a build directory
54 of your choosing, and in that directory generate Makefiles or IDE project
55 using cmake.
56
57
58 ### Windows x86-32
59
60 cmake <path to vulkancts> -G"Visual Studio 12"
61 start dEQP-Core-default.sln
62
63 ### Windows x86-64
64
65 cmake <path to vulkancts> -G"Visual Studio 12 Win64"
66 start dEQP-Core-default.sln
67
68 ### Linux 32-bit Debug
69
70 cmake <path to vulkancts> -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_FLAGS=-m32 -DCMAKE_CXX_FLAGS=-m32
71 make -j
72
73 Release build can be done by using -DCMAKE_BUILD_TYPE=Release
74
75 ### Linux 64-bit Debug
76
77 cmake <path to vulkancts> -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_FLAGS=-m64 -DCMAKE_CXX_FLAGS=-m64
78 make -j
79
80 ### Android
81
82 Following command will build dEQP.apk:
83
84 python scripts/android/build_apk.py --sdk <path to Android SDK> --ndk <path to Android NDK>
85
86 By default the CTS package will be built for the Android API level 21 (Android 5.0 and above).
87 Another API level may be supplied using --native-api command line option.
88
89 The package can be installed by either running:
90
91 python scripts/android/install_apk.py
92
93 By default the CTS package will contain libdeqp.so built for armeabi-v7a, arm64-v8a,
94 x86, and x86_64 ABIs, but that can be changed using --abis command line option.
95
96 To pick which ABI to use at install time, following commands must be used
97 instead:
98
99 adb install --abi <ABI name> <build-root>/package/dEQP.apk /data/local/tmp/dEQP-debug.apk
100
101
102 Building Mustpass
103 -----------------
104
105 Current mustpass is checked into repository and can be found at:
106
107 external/vulkancts/mustpass/1.1.0/vk-default.txt
108
109 Vulkan CTS mustpass can be re-generated by running:
110
111 python <vulkancts>/external/vulkancts/scripts/build_mustpass.py
112
113
114 Pre-compiling SPIR-V binaries
115 -----------------------------
116
117 For distribution, and platforms that don't support GLSL to SPIR-V compilation,
118 SPIR-V binaries can be pre-built with following command:
119
120 python external/vulkancts/scripts/build_spirv_binaries.py
121
122 By default the script builds SPIR-V binaries for Vulkan 1.1.
123 Binaries for other Vulkan versions can be requested by supplying
124 an extra command line option:
125
126 python external/vulkancts/scripts/build_spirv_binaries.py --target-vulkan-version <Vulkan version>
127
128 Binaries will be written to `external/vulkancts/data/vulkan/prebuilt/`.
129
130 Test modules (or in case of Android, the APK) must be re-built after building
131 SPIR-V programs in order for the binaries to be available.
132
133
134 Running CTS
135 -----------
136
137 Following command line options MUST be used when running CTS:
138
139 --deqp-caselist-file=<vulkancts>/external/vulkancts/mustpass/1.1.0/vk-default.txt
140 --deqp-log-images=disable
141 --deqp-log-shader-sources=disable
142
143 In addition on multi-device systems the device for which conformance is claimed
144 can be selected with:
145
146 --deqp-vk-device-id=<value>
147
148 To speed up the conformance run on some platforms the following command line
149 option may be used to disable frequent fflush() calls to the output logs:
150
151 --deqp-log-flush=disable
152
153 By default, the test log will be written into the path "TestResults.qpa". If the
154 platform requires a different path, it can be specified with:
155
156 --deqp-log-filename=<path>
157
158 No other command line options are allowed.
159
160 ### Win32
161
162 cd <builddir>/external/vulkancts/modules/vulkan
163 Debug/deqp-vk.exe --deqp-caselist-file=...
164
165 Test log will be written into TestResults.qpa
166
167 ### Linux
168
169 cd <builddir>/external/vulkancts/modules/vulkan
170 ./deqp-vk --deqp-vk-caselist-file=...
171
172 ### Android
173
174 adb push <vulkancts>/external/vulkancts/mustpass/1.1.0/vk-default.txt /sdcard/vk-default.txt
175 adb shell
176
177 In device shell:
178
179 am start -n com.drawelements.deqp/android.app.NativeActivity -e cmdLine "deqp --deqp-caselist-file=/sdcard/vk-default.txt --deqp-log-images=disable --deqp-log-shader-sources=disable --deqp-log-filename=/sdcard/TestResults.qpa"
180
181 Test progress will be written to device log and can be displayed with:
182
183 adb logcat -s dEQP
184
185 Test log will be written into `/sdcard/TestResults.qpa`.
186
187
188 Conformance Submission Package Requirements
189 -------------------------------------------
190
191 The conformance submission package must contain the following:
192
193 1. Full test logs (`TestResults.qpa`) from CTS runs against all driver builds
194 2. Result of `git status` and `git log` from CTS source directory
195 3. Any patches used on top of release tag
196 4. Conformance statement
197
198 Test logs (1) should be named `<submission pkg dir>/TestResults-<driver build type>.qpa`,
199 for example `TestResults-armeabi-v7a.qpa`. On platforms where multiple different driver
200 builds (for example 64-bit and 32-bit) are present, CTS logs must be provided
201 for each driver build as part of the submission package.
202
203 Test logs generated on a system which exposes more than one physical device
204 in a device group can be used for products that expose one or more physical
205 devices in their device group.
206
207 The CTS build must always be done from clean git repository that doesn't have any
208 uncommitted changes. Thus it is necessary to run and capture output of `git
209 status` and `git log` (2) in the source directory:
210
211 git status > <submission pkg dir>/git-status.txt
212 git log --first-parent <release tag>^..HEAD > <submission pkg dir>/git-log.txt
213
214 Any changes made to CTS must be committed to the local repository, and provided
215 as part of the submission package (3). This can be done by running:
216
217 git format-patch -o <submission pkg dir> <release tag>..HEAD
218
219 In general, bugfixes and changes to platform-specific code (mostly under
220 `framework/platform`) are allowed. The commit message for each change must
221 include a clear description of the change and why it is necessary. Non-porting
222 related changes must be accompanied by a waiver (see below).
223
224 NOTE: When cherry-picking patches on top of release tag, please use `git cherry-pick -x`
225 to include original commit hash in the commit message.
226
227 Conformance statement (4) must be included in a file called `STATEMENT-<adopter>`
228 and must contain following:
229
230 CONFORM_VERSION: <git tag of CTS release>
231 PRODUCT: <string-value>
232 CPU: <string-value>
233 OS: <string-value>
234
235 Note that product/cpu/os information is also captured in `dEQP-VK.info.*` tests
236 if `vk::Platform::describePlatform()` is implemented.
237
238 If the submission package covers multiple products, you can list them by appending
239 additional `PRODUCT:` lines to the conformance statement. For example:
240
241 CONFORM_VERSION: vulkan-cts-1.1.0.0
242 PRODUCT: Product A
243 PRODUCT: Product B
244 ...
245
246 The actual submission package consists of the above set of files which must
247 be bundled into a gzipped tar file named `VK<API major><API minor>_<adopter><_info>.tgz`.
248 `<API major>` is the major version of the Vulkan API specification, `<API minor>`is the minor
249 version of the Vulkan API specification.
250 `<adopter>` is the name of the Adopting member company, or some recognizable abbreviation.
251 The `<_info>` field is optional. It may be used to uniquely identify a submission
252 by OS, platform, date, or other criteria when making multiple submissions.
253 For example, a company XYZ may make a submission for a Vulkan 1.1 implementation named
254 `VK11_XYZ_PRODUCTA_Windows10.tgz`
255
256 One way to create a suiteable gzipped tar file is to execute the command:
257
258 tar -cvzf <filename.tgz> -C <submission pkg dir> .
259
260 where `<submission pkg dir>` is the directory containing the files from (1)-(4)
261 from above. A submission package must contain all of the files listed above,
262 and only those files.
263
264 As an example submission package could contain:
265
266 STATEMENT-Khronos
267 git-log.txt
268 git-status.txt
269 0001-Remove-Waived-Filtering-Tests.patch
270 0002-Fix-Pipeline-Parameters.patch
271 TestResults-armeabi-v7a.qpa
272 TestResults-arm64-v8a.qpa
273
274
275 Waivers
276 -------
277
278 The process for requesting a waiver is to report the issue by filing a bug
279 report in the Gitlab VulkanCTS project (TODO Github?). When creating the
280 submission package, include references to the waiver in the commit message of
281 the relevant change. Including as much information as possible in your bug
282 report (including a unified diff or a merge request of suggested file changes)
283 will ensure the issue can be progressed as rapidly as possible. Issues must
284 be labeled "Waiver" (TODO!) and identify the version of the CTS and affected
285 tests.
286
287 Conformance Criteria
288 --------------------
289
290 Conformance run is considered passing if all tests finish with allowed result
291 codes. Test results are contained in the TestResults.qpa log. Each
292 test case section contains XML tag Result, for example:
293
294 <Result StatusCode="Pass">Not validated</Result>
295
296 The result code is the value of the StatusCode attribute. Following status
297 codes are allowed:
298
299 Pass
300 NotSupported
301 QualityWarning
302 CompatibilityWarning
303
304 Submission package can be verified using `external/vulkancts/scripts/verify_submission.py`
305 script. The script takes two arguments: path to extracted submission package
306 and path to current mustpass list. For example:
307
308 python external/vulkancts/scripts/verify_submission.py VK_11_Khronos_1/ external/vulkancts/mustpass/1.1.0/vk-default.txt
309
310 Please note that the script reports a warning even for a correctly generated git-log.txt
311 If your git-log.txt contains only head commit of the release tag then
312 the warning can be ignored.
313
314 Vulkan platform port
315 --------------------
316
317 Vulkan support from Platform implementation requires providing
318 `getVulkanPlatform()` method in `tcu::Platform` class implementation.
319
320 See `framework/common/tcuPlatform.hpp` and examples in
321 `framework/platform/win32/tcuWin32Platform.cpp` and
322 `framework/platform/android/tcuAndroidPlatform.cpp`.
323
324 If any WSI extensions are supported, platform port must also implement
325 methods for creating native display (`vk::Platform::createWsiDisplay`)
326 and window handles (`vk::wsi::Display::createWindow`). Otherwise tests
327 under `dEQP-VK.wsi` will fail.
328
329
330 Null (dummy) driver
331 -------------------
332
333 For testing and development purposes it might be useful to be able to run
334 tests on dummy Vulkan implementation. One such implementation is provided in
335 vkNullDriver.cpp. To use that, implement `vk::Platform::createLibrary()` with
336 `vk::createNullDriver()`.
337
338
339 Validation Layers
340 -----------------
341
342 Vulkan CTS framework includes first-party support for validation layers, that
343 can be turned on with `--deqp-validation=enable` command line option.
344
345 When validation is turned on, default instance and device will be created with
346 validation layers enabled and debug callback is registered to record any
347 messages. Debug messages collected during test execution will be included at
348 the end of the test case log.
349
350 If any validation errors are found, test result will be set to `InternalError`.
351
352 By default `VK_DEBUG_REPORT_INFORMATION_BIT_EXT` and `_DEBUG_BIT_EXT` messages
353 are excluded from the log, but that can be customized by modifying
354 `vkt::TestCaseExecutor::deinit()` in `vktTestPackage.cpp`.
355
356
357 Cherry GUI
358 ----------
359
360 Vulkan test module can be used with Cherry (GUI for test execution and
361 analysis). Cherry is available at
362 https://android.googlesource.com/platform/external/cherry. Please follow
363 instructions in README to get started.
364
365 Before first launch, and every time test hierarchy has been modified, test
366 case list must be refreshed by running:
367
368 python scripts/build_caselists.py path/to/cherry/data
369
370 Cherry must be restarted for the case list update to take effect.
371