Home | History | Annotate | only in /external/skia/tools/skqp
Up to higher level directory
NameDateSize
bad_gms.txt22-Oct-202082
branch_skqp_dev.sh22-Oct-2020588
clean_app.sh22-Oct-2020509
cut_release22-Oct-2020889
docker_build_universal_apk.sh22-Oct-20201.1K
download_model22-Oct-20202.2K
find_commit_with_best_gold_results.py22-Oct-20203.6K
generate_gn_args22-Oct-20201.8K
get_gold_results.py22-Oct-20201.2K
gn_to_bp.py22-Oct-20205.4K
jitter_gms.cpp22-Oct-20204.7K
make_apk.sh22-Oct-2020961
make_apk_list.py22-Oct-20203.3K
make_gmkb.go22-Oct-20204.8K
make_rendertests_list.py22-Oct-20201.3K
make_universal_apk22-Oct-2020201
make_universal_apk.py22-Oct-20207.5K
README.md22-Oct-20203K
README_ALGORITHM.md22-Oct-20203.9K
README_GENERATING_MODELS.md22-Oct-20203.3K
release.sh22-Oct-20203.4K
run_skqp_exe22-Oct-20201.6K
setup_resources22-Oct-2020631
skqp_gn_args.py22-Oct-2020870
src/22-Oct-2020
test_apk.sh22-Oct-20201.7K
upload_apk22-Oct-2020412
upload_model22-Oct-20201.4K

README.md

      1 SkQP
      2 ====
      3 
      4 SkQP (Skia Quality Program) is a component of the Android CTS (Compatablity
      5 Test Suite) that tests an Android device's GPU and OpenGLES & Vulkan drivers
      6 using Skia's existing unit & rendering tests.
      7 
      8 See https://skia.org/dev/testing/skqp for pre-build APKs.
      9 
     10 How to build and run the SkQP tests
     11 -----------------------------------
     12 
     13 1.  Get the dependencies:
     14 
     15     -   You will need Java JDK 8, `git`, and `python`.
     16 
     17     -   Install Chromium's [depot\_tools](http://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/depot_tools_tutorial.html).  Add it to your `PATH`.
     18 
     19             git clone 'https://chromium.googlesource.com/chromium/tools/depot_tools.git'
     20             export PATH="${PWD}/depot_tools:${PATH}"
     21 
     22     -   Install the [Android NDK](https://developer.android.com/ndk/downloads/).
     23 
     24             ( cd ~; unzip ~/Downloads/android-ndk-*.zip )
     25             ANDROID_NDK=$(ls -d ~/android-ndk-*)   # Or wherever you installed the Android NDK.
     26 
     27     -   Install the [Android SDK](https://developer.android.com/studio/#command-tools).
     28         Set the `ANDROID_HOME` environment variable.
     29 
     30             mkdir ~/android-sdk
     31             ( cd ~/android-sdk; unzip ~/Downloads/sdk-tools-*.zip )
     32             yes | ~/android-sdk/tools/bin/sdkmanager --licenses
     33             export ANDROID_HOME=~/android-sdk  # Or wherever you installed the Android SDK.
     34 
     35         Put `adb` in your `PATH`.
     36 
     37             export PATH="${PATH}:${ANDROID_HOME}/platform-tools"
     38 
     39 2.  Get the right version of Skia:
     40 
     41         git clone https://skia.googlesource.com/skia.git
     42         cd skia
     43         git checkout origin/skqp/dev  # or whatever release tag you need
     44 
     45 3.  Build the APK:
     46 
     47         tools/git-sync-deps
     48         tools/skqp/make_universal_apk
     49 
     50 4.  Build, install, and run.
     51 
     52         adb install -r out/skqp/skqp-universal-debug.apk
     53         adb logcat -c
     54         adb shell am instrument -w org.skia.skqp
     55 
     56 5.  Monitor the output with:
     57 
     58         adb logcat TestRunner org.skia.skqp skia "*:S"
     59 
     60     Note the test's output path on the device.  It will look something like this:
     61 
     62         01-23 15:22:12.688 27158 27173 I org.skia.skqp:
     63         output written to "/storage/emulated/0/Android/data/org.skia.skqp/files/output"
     64 
     65 6.  Retrieve and view the report with:
     66 
     67         OUTPUT_LOCATION="/storage/emulated/0/Android/data/org.skia.skqp/files/output"
     68         adb pull $OUTPUT_LOCATION /tmp/
     69         bin/sysopen /tmp/output/skqp_report/report.html
     70 
     71 Running a single test
     72 ---------------------
     73 
     74 To run a single test, for example `gles_aarectmodes`:
     75 
     76     adb shell am instrument -e class 'org.skia.skqp.SkQPRunner#gles_aarectmodes' -w org.skia.skqp
     77 
     78 Unit tests can be run with the `unitTest_` prefix:
     79 
     80     adb shell am instrument -e class 'org.skia.skqp.SkQPRunner#unitTest_GrSurface -w org.skia.skqp
     81 
     82 Run as a non-APK executable
     83 ---------------------------
     84 
     85 1.  Follow steps 1-3 as above.
     86 
     87 2.  Build the SkQP program, load files on the device, and run skqp:
     88 
     89         gn gen out/skqp-arm
     90         ninja -C out/skqp-arm skqp
     91         python tools/skqp/run_skqp_exe out/skqp-arm
     92 
     93

README_ALGORITHM.md

      1 SkQP Render Test Algorithm
      2 ==========================
      3 
      4 The following is a description of the render test validation algorithm that
      5 will be used by the version of SkQP that will be released for Android Q-release.
      6 
      7 There is a global macro constant: `SK_SKQP_GLOBAL_ERROR_TOLERANCE`, which
      8 reflects the `gn` variable `skia_skqp_global_error_tolerance`.  This is usually
      9 set to 8.
     10 
     11 First, look for a file named `skqp/rendertests.txt` in the
     12 `platform_tools/android/apps/skqp/src/main/assets` directory.  The format of
     13 this file is:  each line contains one render test name, followed by a comma,
     14 followed by an integer.  The integer is the `passing_threshold` for that test.
     15 
     16 For each test, we have a `max_image` and a `min_image`.  These are PNG-encoded
     17 images stored in SkQP's APK's asset directory (in the paths `gmkb/${TEST}/min.png`
     18 and `gmkb/${TEST}/max.png`).
     19 
     20 The test input is a rendered image.  This will be produced by running one of
     21 the render tests against the either the `vk` (Vulkan) or `gles` (OpenGL ES)
     22 Skia backend.
     23 
     24 Here is psuedocode for the error calculation:
     25 
     26     function calculate_pixel_error(pixel_value, pixel_max, pixel_min):
     27         pixel_error = 0
     28 
     29         for color_channel in { red, green, blue, alpha }:
     30             value = get_color(pixel_value, color_channel)
     31             v_max = get_color(pixel_max,   color_channel)
     32             v_min = get_color(pixel_min,   color_channel)
     33 
     34             if value > v_max:
     35                 channel_error = value - v_max
     36             elif value < v_min:
     37                 channel_error = v_min - value
     38             else:
     39                 channel_error = 0
     40             pixel_error = max(pixel_error, channel_error)
     41 
     42         return max(0, pixel_error - SK_SKQP_GLOBAL_ERROR_TOLERANCE);
     43 
     44     function get_error(rendered_image, max_image, min_image):
     45         assert(dimensions(rendered_image) == dimensions(max_image))
     46         assert(dimensions(rendered_image) == dimensions(min_image))
     47 
     48         max_error = 0
     49         bad_pixels = 0
     50         total_error = 0
     51 
     52         error_image = allocate_bitmap(dimensions(rendered_image))
     53 
     54         for xy in list_all_pixel_coordinates(rendered_image):
     55             pixel_error = calculate_pixel_error(rendered_image(xy),
     56                                                 max_image(xy),
     57                                                 min_image(xy))
     58             if pixel_error > 0:
     59                 for neighboring_xy in find_neighbors(xy):
     60                     if not inside(neighboring_xy, dimensions(rendered_image)):
     61                         continue
     62                     pixel_error = min(pixel_error,
     63                                       calculate_pixel_error(rendered_image(xy),
     64                                                             max_image(neighboring_xy),
     65                                                             min_image(neighboring_xy)))
     66 
     67             if pixel_error > 0:
     68                 max_error = max(max_error, pixel_error)
     69                 bad_pixels += 1
     70                 total_error += pixel_error
     71 
     72                 error_image(xy) = linear_interpolation(black, red, pixel_error)
     73             else:
     74                 error_image(xy) = white
     75 
     76         return ((total_error, max_error, bad_pixels), error_image)
     77 
     78 For each render test, there is a threshold value for `total_error`, :
     79 `passing_threshold`.
     80 
     81 If `passing_threshold >= 0 && total_error > passing_threshold`, then the test
     82 is a failure and is included in the report.  if `passing_threshold == -1`, then
     83 the test always passes, but we do execute the test to verify that the driver
     84 does not crash.
     85 
     86 We generate a report with the following information for each test:
     87 
     88     backend_name,render_test_name,max_error,bad_pixels,total_error
     89 
     90 in CSV format in the file `out.csv`.  A HTML report of just the failing tests
     91 is written to the file `report.html`.  This version includes four images for
     92 each test:  `rendered_image`, `max_image`, `min_image`, and `error_image`, as
     93 well as the three metrics: `max_error`, `bad_pixels`, and `total_error`.
     94 
     95 
     96 
     97

README_GENERATING_MODELS.md

      1 How SkQP Generates Render Test Models
      2 =====================================
      3 
      4 We will, at regular intervals, generate new models from the [master branch of
      5 Skia][1].  Here is how that process works:
      6 
      7 0.  Choose a commit to make the branch from
      8 
      9         COMMIT=origin/master
     10 
     11 1.  Get the positively triaged results from Gold:
     12 
     13         cd SKIA_SOURCE_DIRECTORY
     14         git fetch origin
     15         git checkout "$COMMIT"
     16         python tools/skqp/get_gold_results.py "${COMMIT}~10" "$COMMIT"
     17 
     18     This will produce a file `meta_YYYMMMDDD_HHHMMMSS_COMMIT_COMMIT.json` in
     19     the current directory.
     20 
     21 2.  From a checkout of Skia's master branch, execute:
     22 
     23         cd SKIA_SOURCE_DIRECTORY
     24         git checkout "$COMMIT"
     25         tools/skqp/cut_release META_JSON_FILE
     26 
     27     This will create the following files:
     28 
     29         platform_tools/android/apps/skqp/src/main/assets/files.checksum
     30         platform_tools/android/apps/skqp/src/main/assets/skqp/rendertests.txt
     31         platform_tools/android/apps/skqp/src/main/assets/skqp/unittests.txt
     32 
     33     These three files can be commited to Skia to create a new commit.  Make
     34     `origin/skqp/dev` a parent of this commit (without merging it in), and
     35     push this new commit to `origin/skqp/dev`:
     36 
     37         tools/skqp/branch_skqp_dev.sh
     38 
     39     Review and submit the change:
     40 
     41         git push origin HEAD:refs/for/skqp/dev
     42         bin/sysopen https://review.skia.org/$(bin/gerrit-number HEAD)
     43 
     44     (Optional) Make a SkQP APK.
     45 
     46         tools/skqp/docker_build_universal_apk.sh
     47 
     48     (Optional) Test the SkQP APK:
     49 
     50         adb uninstall org.skia.skqp
     51         tools/skqp/test_apk.sh LOCATION/skqp-universal-debug.apk
     52 
     53     (Once changes land) Upload the SkQP APK.
     54 
     55         tools/skqp/upload_apk LOCATION/skqp-universal-debug.apk
     56 
     57 
     58 `tools/skqp/cut_release`
     59 ------------------------
     60 
     61 This tool will call `make_gmkb.go` to generate the `m{ax,in}.png` files for
     62 each render test.  Additionaly, a `models.txt` file enumerates all of the
     63 models.
     64 
     65 Then it calls `jitter_gms` to see which render tests pass the jitter test.
     66 `jitter_gms` respects the `bad_gms.txt` file by ignoring the render tests
     67 enumerated in that file.  Tests which pass the jitter test are enumerated in
     68 the file `good.txt`, those that fail in the `bad.txt` file.
     69 
     70 Next, the `skqp/rendertests.txt` file is created.  This file lists the render
     71 tests that will be executed by SkQP.  These are the union of the tests
     72 enumerated in the `good.txt` and `bad.txt` files.  If the render test is found
     73 in the `models.txt` file and the `good.txt` file, its per-test threshold is set
     74 to 0 (a later CL can manually change this, if needed).  Otherwise, the
     75 threshold is set to -1; this indicated that the rendertest will be executed (to
     76 verify that the driver will not crash), but the output will not be compared
     77 against the model.  Unnecessary models will be removed.
     78 
     79 Next, all of the files that represent the models are uploaded to cloud storage.
     80 A single checksum hash is kept in the  `files.checksum` file.  This is enough
     81 to re-download those files later, but we don't have to fill the git repository
     82 with a lot of binary data.
     83 
     84 Finally, a list of the current gpu unit tests is created and stored in
     85 `skqp/unittests.txt`.
     86 
     87 [1]: https://skia.googlesource.com/skia/+log/master "Skia Master Branch"
     88 [2]: https://gold.skia.org/search                   "Skia Gold Search"
     89