1 # Porting to Other Implementations 2 3 ## Introduction 4 5 This document provides an overview of the test runner and how to 6 integrate it with other stacks. So far we have it working with 7 BoringSSL and some incomplete integrations with NSS and OpenSSL. 8 9 Note that supporting non-BoringSSL implementations is a work in 10 progress and interfaces may change in the future. Consumers should pin 11 to a particular revision rather than using BoringSSLs master branch 12 directly. As we gain experience with other implementations, we hope to 13 make further improvements to portability, so please contact 14 davidben (a] google.com and ekr (a] rtfm.com if implementing a new shim. 15 16 17 ## Integration Architecture 18 19 The test runner integrates with the TLS stack under test through a 20 shim: a command line program which encapsulates the stack. By 21 default, the shim points to the BoringSSL shim in the same source 22 tree, but any program can be supplied via the `-shim-path` flag. The 23 runner opens up a server socket and provides the shim with a `-port` 24 argument that points to that socket. The shim always connects to the 25 runner as a TCP client even when acting as a TLS server. For DTLS, 26 there is a small framing layer that gives packet boundaries over 27 TCP. The shim can also pass a variety of command line arguments which 28 are used to configure the stack under test. These can be found at 29 `test_config.cc`. 30 31 32 The shim reports success by exiting with a `0` error code and failure by 33 reporting a non-zero error code and generally sending a textual error 34 value to stderr. Many of the tests expect specific error string (such 35 as `NO_SHARED_CIPHER`) that indicates what went wrong. 36 37 38 ## Compatibility Issues 39 40 There are a number of situations in which the runner might succeed 41 with some tests and not others: 42 43 * Defects in the stack under test 44 * Features which havent yet been implemented 45 * Failure to implement one or more of the command line flags the runner uses with the shim 46 * Disagreement about the right behavior/interpretation of the spec 47 48 49 We have implemented several features which allow implementations to ease these compatibility issues. 50 51 ### Configuration File 52 53 The runner can be supplied with a JSON configuration file which is 54 intended to allow for a per-stack mapping. This file currently takes 55 two directives: 56 57 58 * `DisabledTests`: A JSON map consisting of the pattern matching the 59 tests to be disabled as the key and some sort of reason why it was 60 disabled as the value. The key is used as a match against the test 61 name. The value is ignored and is just used for documentation 62 purposes so you can remember why you disabled a 63 test. `-include-disabled` overrides this filter. 64 65 * `ErrorMap`: A JSON map from the internal errors the runner expects to 66 the error strings that your implementation spits out. Generally 67 youll need to map every error, but if you also provide the 68 ` -loose-errors` flag, then every un-mapped error just gets mapped to 69 the empty string and treated as if it matched every error the runner 70 expects. 71 72 73 The `-shim-config` flag is used to provide the config file. 74 75 76 ### Unimplemented Features 77 If the shim encounters some request from the runner that it knows it 78 cant fulfill (e.g., a command line flag that it doesnt recognize), 79 then it can exit with the special code `89`. Shims are recommended to 80 use this exit code on unknown command-line arguments. 81 82 The test runner interprets this as unimplemented and skips the 83 test. If run normally, this will cause the test runner to report that 84 the entire test suite failed. The `-allow-unimplemented` flag suppresses 85 this behavior and causes the test runner to ignore these tests for the 86 purpose of evaluating the success or failure of the test suite. 87 88 89 ### Malloc Tests 90 91 The test runner can also be used to stress malloc failure 92 codepaths. If passed `-malloc-test=0`, the runner will run each test 93 repeatedly with an incrementing `MALLOC_NUMBER_TO_FAIL` environment 94 variable. The shim should then replace the malloc implementation with 95 one which fails at the specified number of calls. If there are not 96 enough calls to reach the number, the shim should fail with exit code 97 `88`. This signals to the runner that the test has completed. 98 99 See `crypto/test/malloc.cc` for an example malloc implementation. 100 101 Note these tests are slow and will hit Go's test timeout. Pass `-timeout 72h` to 102 avoid crashing after 10 minutes. 103 104 105 ## Example: Running Against NSS 106 107 ``` 108 DYLD_LIBRARY_PATH=~/dev/nss-dev/nss-sandbox/dist/Darwin15.6.0_64_DBG.OBJ/lib go test -shim-path ~/dev/nss-dev/nss-sandbox/dist/Darwin15.6.0_64_DBG.OBJ/bin/nss_bogo_shim -loose-errors -allow-unimplemented -shim-config ~/dev/nss-dev/nss-sandbox/nss/external_tests/nss_bogo_shim/config.json 109 ``` 110
