1 gemmlowp: a small self-contained low-precision GEMM library 2 =========================================================== 3 4 This is not a full linear algebra library, only a GEMM library: it only does 5 general matrix multiplication ("GEMM"). 6 7 The meaning of "low precision" is detailed in this document: 8 doc/low-precision.txt 9 10 Some of the general design is explained in 11 doc/design.txt 12 13 14 Disclaimer 15 ========== 16 17 This is not an official Google product (experimental or otherwise), it is just 18 code that happens to be owned by Google. 19 20 21 Mailing list 22 ============ 23 24 gemmlowp-related discussion, about either development or usage, is welcome 25 on this Google Group (mailing list / forum): 26 27 https://groups.google.com/forum/#!forum/gemmlowp 28 29 30 Portability, target platforms/architectures 31 =========================================== 32 33 Should be portable to any platform with some C++11 and POSIX support, 34 while we have optional optimized code paths for specific architectures. 35 36 Required: 37 C++11 (a small conservative subset of it) 38 39 Required for some features: 40 * Some POSIX interfaces: 41 * pthreads (for multi-threaded operation and for profiling). 42 * sysconf (for multi-threaded operation to detect number of cores; 43 may be bypassed). 44 45 Optional: 46 Architecture-specific code paths use intrinsics or inline assembly. 47 See "Architecture-specific optimized code paths" below. 48 49 Architecture-specific optimized code paths 50 ========================================== 51 52 We have some optimized code paths for specific instruction sets. 53 Some are written in inline assembly, some are written in C++ using 54 intrinsics. Both GCC and Clang are supported. 55 56 At the moment, we have a full set of optimized code paths (kernels, 57 packing and unpacking paths) only for ARM NEON, supporting both 58 ARMv7 (32bit) and ARMv8 (64bit). 59 60 We also have a partial set of optimized code paths (only kernels 61 at the moment) for Intel SSE. It supports both x86 and x86-64 but 62 only targets SSE4. The lack of packing/unpacking code paths means 63 that performance isn't optimal yet. 64 65 Details of what it takes to make an efficient port of gemmlowp, namely 66 writing a suitable GEMM kernel and accompanying packing code, are 67 explained in this file: 68 doc/kernels.txt 69 70 71 Public interfaces 72 ================= 73 74 1. gemmlowp public interface 75 ---------------------------- 76 77 gemmlowp's main public interface is in the public/ subdirectory. The 78 header to include is 79 public/gemmlowp.h. 80 This is a headers-only library, so there is nothing to link to. 81 82 2. EightBitIntGemm standard interface 83 ------------------------------------- 84 85 Additionally, the eight_bit_int_gemm/ subdirectory provides an 86 implementation of the standard EightBitIntGemm interface. The header 87 to include is 88 eight_bit_int_gemm/eight_bit_int_gemm.h 89 This is *NOT* a headers-only library, users need to link to 90 eight_bit_int_gemm/eight_bit_int_gemm.cc. 91 The API is similar to the standard BLAS GEMM interface, and implements 92 C = A * B. If the transpose flags for a matrix argument are false, its memory 93 order is treated as column major, and row major if its true. 94 95 96 Building 97 ======== 98 99 Building by manually invoking your compiler 100 ------------------------------------------- 101 102 Because gemmlowp is so simple, working with it involves only 103 single-command-line compiler invokations. Therefore we expect that 104 most people working with gemmlowp will either manually invoke their 105 compiler, or write their own rules for their own preferred build 106 system. 107 108 Keep in mind (previous section) that gemmlowp itself is a pure-headers-only 109 library so there is nothing to build, and the eight_bit_int_gemm library 110 consists of a single eight_bit_int_gemm.cc file to build. 111 112 For a Android gemmlowp development workflow, the scripts/ directory 113 contains a script to build and run a program on an Android device: 114 scripts/test-android.sh 115 116 Building using Bazel 117 -------------------- 118 119 That being said, we also maintain a Bazel BUILD system as part of 120 gemmlowp. Its usage is not mandatory at all and is only one 121 possible way that gemmlowp libraries and tests may be built. If 122 you are interested, Bazel's home page is 123 http://bazel.io/ 124 And you can get started with using Bazel to build gemmlowp targets 125 by first creating an empty WORKSPACE file in a parent directory, 126 for instance: 127 128 $ cd gemmlowp/.. # change to parent directory containing gemmlowp/ 129 $ touch WORKSPACE # declare that to be our workspace root 130 $ bazel build gemmlowp:all 131 132 133 Testing 134 ======= 135 136 Testing by manually building and running tests 137 ---------------------------------------------- 138 139 The test/ directory contains unit tests. The primary unit test is 140 test/test.cc 141 Since it covers also the EightBitIntGemm interface, it needs to be 142 linked against 143 eight_bit_int_gemm/eight_bit_int_gemm.cc 144 It also uses realistic data captured from a neural network run in 145 test/test_data.cc 146 147 Thus you'll want to pass the following list of source files to your 148 compiler/linker: 149 test/test.cc 150 eight_bit_int_gemm/eight_bit_int_gemm.cc 151 test/test_data.cc 152 153 The scripts/ directory contains a script to build and run a program 154 on an Android device: 155 scripts/test-android.sh 156 157 It expects the CXX environment variable to point to an Android toolchain's 158 C++ compiler, and expects source files (and optionally, cflags) as 159 command-line parameters. To build and run the above-mentioned main unit test, 160 first set CXX e.g.: 161 162 $ export CXX=/some/toolchains/arm-linux-androideabi-4.8/bin/arm-linux-androideabi-g++ 163 164 Then run: 165 166 $ ./scripts/test-android.sh \ 167 test/test.cc \ 168 eight_bit_int_gemm/eight_bit_int_gemm.cc \ 169 test/test_data.cc 170 171 172 Testing using Bazel 173 ------------------- 174 175 Alternatively, you can use Bazel to build and run tests. See the Bazel 176 instruction in the above section on building. Once your Bazel workspace 177 is set up, you can for instance do: 178 179 $ bazel test gemmlowp:all 180 181 182 Troubleshooting Compilation 183 =========================== 184 185 If you're having trouble finding the compiler, follow these instructions to 186 build a standalone toolchain: 187 https://developer.android.com/ndk/guides/standalone_toolchain.html 188 189 Here's an example of setting up Clang 3.5: 190 191 $ export INSTALL_DIR=~/toolchains/clang-21-stl-gnu 192 $ $NDK/build/tools/make-standalone-toolchain.sh \ 193 --toolchain=arm-linux-androideabi-clang3.5 --platform=android-21 \ 194 --install-dir=$INSTALL_DIR 195 $ export CXX="$INSTALL_DIR/bin/arm-linux-androideabi-g++ \ 196 --sysroot=$INSTALL_DIR/sysroot" 197 198 Some compilers (e.g. the default clang++ in the same bin directory) don't 199 support NEON assembly. The benchmark build process will issue a warning if 200 support isn't detected, and you should make sure you're using a compiler like 201 arm-linux-androideabi-g++ that does include NEON. 202 203 204 Benchmarking 205 ============ 206 207 The main benchmark is 208 benchmark.cc 209 It doesn't need to be linked to any 210 other source file. We recommend building with assertions disabled (-DNDEBUG). 211 212 For example, the benchmark can be built and run on an Android device by doing: 213 214 $ ./scripts/test-android.sh test/benchmark.cc -DNDEBUG 215 216 If GEMMLOWP_TEST_PROFILE is defined then the benchmark will be built with 217 profiling instrumentation (which makes it slower) and will dump profiles. 218 See next section on profiling. 219 220 221 Profiling 222 ========= 223 224 The profiling/ subdirectory offers a very simple non-interrupting sampling 225 profiler that only requires pthreads (no signals). 226 227 It relies on source code being instrumented with pseudo-stack labels. 228 See profiling/instrumentation.h. 229 A full example of using this profiler is given in profiling/profiler.h. 230 231 232 Contributing 233 ============ 234 235 Contribution-related discussion is always welcome on the gemmlowp 236 mailing list (see above). 237 238 We try to keep a current list of TODO items in the todo/ directory. 239 Prospective contributors are welcome to pick one to work on, and 240 communicate about it on the gemmlowp mailing list. 241 242 Details of the contributing process, including legalese, are in CONTRIBUTING. 243 244 Performance goals 245 ================= 246 247 Our performance goals differ from typical GEMM performance goals in the 248 following ways: 249 250 1. We care not only about speed, but also about minimizing power usage. 251 We specifically care about charge usage in mobile/embedded devices. 252 This implies that we care doubly about minimizing memory bandwidth usage: 253 we care about it, like any GEMM, because of the impact on speed, and we 254 also care about it because it is a key factor of power usage. 255 256 2. Most GEMMs are optimized primarily for large dense matrix sizes (>= 1000). 257 We do care about large sizes, but we also care specifically about the 258 typically smaller matrix sizes encountered in various mobile applications. 259 This means that we have to optimize for all sizes, not just for large enough 260 sizes. 261
