1 ### TensorFlow Makefile 2 3 The recommended way to build TensorFlow from source is using the Bazel 4 open-source build system. Sometimes this isn't possible. For example, 5 if you are building for iOS, you currently need to use the Makefile. 6 7 - The build system may not have the RAM or processing power to support Bazel. 8 - Bazel or its dependencies may not be available. 9 - You may want to cross-compile for an unsupported target system. 10 11 This experimental project supplies a Makefile automatically derived from the 12 dependencies listed in the Bazel project that can be used with GNU's make tool. 13 With it, you can compile the core C++ runtime into a static library. 14 15 This static library will not contain: 16 17 - Python or other language bindings 18 - GPU support 19 20 You can target: 21 - iOS 22 - OS X (macOS) 23 - Android 24 - Raspberry-PI 25 26 You will compile tensorflow and protobuf libraries that you can link into other 27 applications. You will also compile the [benchmark](../../tools/benchmark/) 28 application that will let you check your application. 29 30 ## Before you start (all platforms) 31 32 First, clone this TensorFlow repository. 33 34 You will need to download all dependencies as well. We have provided a script 35 that does so, to be run (as with all commands) **at the root of the repository**: 36 37 ```bash 38 tensorflow/contrib/makefile/download_dependencies.sh 39 ``` 40 41 You should only need to do this step once. It downloads the required libraries 42 like Eigen in the `tensorflow/contrib/makefile/downloads/` folder. 43 44 You should download the example graph from [https://storage.googleapis.com/download.tensorflow.org/models/inception5h.zip](https://storage.googleapis.com/download.tensorflow.org/models/inception5h.zip). 45 46 ## Building on Linux 47 48 _Note: This has only been tested on Ubuntu._ 49 50 As a first step, you need to make sure the required packages are installed: 51 ```bash 52 sudo apt-get install autoconf automake libtool curl make g++ unzip zlib1g-dev \ 53 git python 54 ``` 55 56 You should then be able to run the `build_all_linux.sh` script to compile: 57 ```bash 58 tensorflow/contrib/makefile/build_all_linux.sh 59 ``` 60 61 This should compile a static library in 62 `tensorflow/contrib/makefile/gen/lib/libtensorflow-core.a`, 63 and create an example executable at `tensorflow/contrib/makefile/gen/bin/benchmark`. 64 65 Get the graph file, if you have not already: 66 67 ```bash 68 mkdir -p ~/graphs 69 curl -o ~/graphs/inception.zip \ 70 https://storage.googleapis.com/download.tensorflow.org/models/inception5h.zip \ 71 && unzip ~/graphs/inception.zip -d ~/graphs/inception 72 ``` 73 74 To run the executable, use: 75 76 ```bash 77 tensorflow/contrib/makefile/gen/bin/benchmark \ 78 --graph=$HOME/graphs/inception/tensorflow_inception_graph.pb 79 ``` 80 81 ## Android 82 83 First, you will need to download and unzip the 84 [Native Development Kit (NDK)](https://developer.android.com/ndk/). You will not 85 need to install the standalone toolchain, however. 86 87 Assign your NDK location to $NDK_ROOT: 88 89 ```bash 90 export NDK_ROOT=/absolute/path/to/NDK/android-ndk-rxxx/ 91 ``` 92 93 Download the graph if you haven't already: 94 95 ```bash 96 mkdir -p ~/graphs 97 curl -o ~/graphs/inception.zip \ 98 https://storage.googleapis.com/download.tensorflow.org/models/inception5h.zip \ 99 && unzip ~/graphs/inception.zip -d ~/graphs/inception 100 ``` 101 102 Then, execute the following: 103 104 ```bash 105 tensorflow/contrib/makefile/download_dependencies.sh 106 tensorflow/contrib/makefile/compile_android_protobuf.sh -c 107 export HOST_NSYNC_LIB=`tensorflow/contrib/makefile/compile_nsync.sh` 108 export TARGET_NSYNC_LIB=`CC_PREFIX="${CC_PREFIX}" NDK_ROOT="${NDK_ROOT}" \ 109 tensorflow/contrib/makefile/compile_nsync.sh -t android -a armeabi-v7a` 110 make -f tensorflow/contrib/makefile/Makefile TARGET=ANDROID 111 ``` 112 113 At this point, you will have compiled libraries in `gen/lib/*` and the 114 [benchmark app](../../tools/benchmark) compiled for Android. 115 116 Run the benchmark by pushing both the benchmark and the graph file to your 117 attached Android device: 118 119 ```bash 120 adb push ~/graphs/inception/tensorflow_inception_graph.pb /data/local/tmp/ 121 adb push tensorflow/contrib/makefile/gen/bin/benchmark /data/local/tmp/ 122 adb shell '/data/local/tmp/benchmark \ 123 --graph=/data/local/tmp/tensorflow_inception_graph.pb \ 124 --input_layer="input:0" \ 125 --input_layer_shape="1,224,224,3" \ 126 --input_layer_type="float" \ 127 --output_layer="output:0" 128 ' 129 ``` 130 131 For more details, see the [benchmark documentation](../../tools/benchmark). 132 133 ## CUDA support for Tegra devices running Android (Nvidia Shield TV, etc) 134 135 With the release of TF 1.6 and JetPack for Android 3.2 (currently pending), you can now build a version of TensorFlow for compatible devices according to the following instructions which will receive the full benefits of GPU acceleration. 136 137 #### Environment setup: 138 139 First, download and install JetPack for Android version 3.2 or greater from [Nvidia](https://developers.nvidia.com). Note that as of the TF 1.6 release the JetPack for Android 3.2 release is still pending, and regular JetPack for L4T will not work. 140 141 ```bash 142 git clone https://github.com/tensorflow/tensorflow.git 143 cd tensorflow 144 JETPACK=$HOME/JetPack_Android_3.2 145 TEGRA_LIBS="$JETPACK/cuDNN/aarch64/cuda/lib64/libcudnn.so $JETPACK/cuda-9.0/extras/CUPTI/lib64/libcupti.so $JETPACK/cuda/targets/aarch64-linux-androideabi/lib64/libcufft.so" 146 ``` 147 148 #### Building all CUDA-enabled native binaries: 149 This will build CUDA-enabled versions of libtensorflow_inference.so and the benchmark binary. (libtensorflow_demo.so will also be built incidentally, but it does not support CUDA) 150 151 ```bash 152 NDK_ROOT=$JETPACK/android-ndk-r13b 153 CC_PREFIX=ccache tensorflow/contrib/makefile/build_all_android.sh -s tensorflow/contrib/makefile/sub_makefiles/android/Makefile.in -t "libtensorflow_inference.so libtensorflow_demo.so all" -a tegra 154 ``` 155 (add -T on subsequent builds to skip protobuf downloading/building) 156 157 158 #### Testing the CUDA-enabled benchmark via adb: 159 Build binaries first as above, then run: 160 161 ```bash 162 adb shell mkdir -p /data/local/tmp/lib64 163 adb push $TEGRA_LIBS /data/local/tmp/lib64 164 adb push tensorflow/contrib/makefile/gen/bin/android_arm64-v8a/benchmark /data/local/tmp 165 wget https://ci.tensorflow.org/view/Nightly/job/nightly-android/lastSuccessfulBuild/artifact/out/tensorflow_demo.apk 166 unzip tensorflow_demo.apk -d /tmp/tensorflow_demo 167 adb push /tmp/tensorflow_demo/assets/*.pb /data/local/tmp 168 adb shell "LD_LIBRARY_PATH=/data/local/tmp/lib64 /data/local/tmp/benchmark --graph=/data/local/tmp/tensorflow_inception_graph.pb" 169 ``` 170 171 #### Building the CUDA-enabled TensorFlow AAR with Bazel: 172 Build the native binaries first as above. Then, build the aar and package the native libs by executing the following: 173 ```bash 174 mkdir -p /tmp/tf/jni/arm64-v8a 175 cp tensorflow/contrib/makefile/gen/lib/android_tegra/libtensorflow_*.so /tmp/tf/jni/arm64-v8a/ 176 cp $TEGRA_LIBS /tmp/tf/jni/arm64-v8a 177 bazel build //tensorflow/contrib/android:android_tensorflow_inference_java.aar 178 cp bazel-bin/tensorflow/contrib/android/android_tensorflow_inference_java.aar /tmp/tf/tensorflow.aar 179 cd /tmp/tf 180 chmod +w tensorflow.aar 181 zip -ur tensorflow.aar $(find jni -name *.so) 182 ``` 183 184 #### Building the CUDA-enabled TensorFlow Android demo with Bazel: 185 Build binaries first as above, then edit tensorflow/examples/android/BUILD and replace: 186 ``` 187 srcs = [ 188 ":libtensorflow_demo.so", 189 "//tensorflow/contrib/android:libtensorflow_inference.so", 190 ], 191 ``` 192 with: 193 ``` 194 srcs = glob(["libs/arm64-v8a/*.so"]), 195 ``` 196 197 Then run: 198 ```bash 199 # Create dir for native libs 200 mkdir -p tensorflow/examples/android/libs/arm64-v8a 201 202 # Copy JetPack libs 203 cp $TEGRA_LIBS tensorflow/examples/android/libs/arm64-v8a 204 205 # Copy native TensorFlow libraries 206 cp tensorflow/contrib/makefile/gen/lib/android_arm64-v8a/libtensorflow_*.so tensorflow/examples/android/libs/arm64-v8a/ 207 208 # Build APK 209 bazel build -c opt --fat_apk_cpu=arm64-v8a tensorflow/android:tensorflow_demo 210 211 # Install 212 adb install -r -f bazel-bin/tensorflow/examples/android/tensorflow_demo.apk 213 ``` 214 215 #### Building the CUDA-enabled Android demo with gradle/Android Studio: 216 217 Add tensorflow/examples/android as an Android project in Android Studio as normal. 218 219 Edit build.gradle and: 220 * set nativeBuildSystem = 'makefile' 221 * set cpuType = 'arm64-v8a' 222 * in "buildNativeMake", replace cpuType with 'tegra' (optional speedups like -T and ccache also work) 223 * set the environment "NDK_ROOT" var to $JETPACK/android-ndk-r13b 224 225 Click "build apk" to build. 226 227 Install: 228 ```bash 229 adb install -r -f tensorflow/examples/android/gradleBuild/outputs/apk/debug/android-debug.apk 230 ``` 231 232 ## iOS 233 234 _Note: To use this library in an iOS application, see related instructions in 235 the [iOS examples](../../examples/ios/) directory._ 236 237 Install XCode 7.3 or more recent. If you have not already, you will need to 238 install the command-line tools using `xcode-select`: 239 240 ```bash 241 xcode-select --install 242 ``` 243 244 If this is a new install, you will need to run XCode once to agree to the 245 license before continuing. 246 247 (You will also need to have [Homebrew](http://brew.sh/) installed.) 248 249 Then install [automake](https://en.wikipedia.org/wiki/Automake)/[libtool](https://en.wikipedia.org/wiki/GNU_Libtool): 250 251 ```bash 252 brew install automake 253 brew install libtool 254 ``` 255 256 Also, download the graph if you haven't already: 257 258 ```bash 259 mkdir -p ~/graphs 260 curl -o ~/graphs/inception.zip \ 261 https://storage.googleapis.com/download.tensorflow.org/models/inception5h.zip \ 262 && unzip ~/graphs/inception.zip -d ~/graphs/inception 263 ``` 264 265 ### Building all at once 266 267 If you just want to get the libraries compiled in a hurry, you can run this 268 from the root of your TensorFlow source folder: 269 270 ```bash 271 tensorflow/contrib/makefile/build_all_ios.sh 272 ``` 273 274 This process will take around twenty minutes on a modern MacBook Pro. 275 276 When it completes, you will have a unified library for all architectures 277 (i386sim, x86_64sim, armv7, armv7s and arm64) and the benchmark program. 278 Although successfully compiling the benchmark program is a 279 sign of success, the program is not a complete iOS app. 280 281 If you would only like to build only one architecture to save time: 282 (iOS 11+ only supports 64bit so you can get away with arm64) 283 284 ```bash 285 tensorflow/contrib/makefile/build_all_ios.sh -a arm64 286 ``` 287 288 After the first build if you would like to just build the tensorflow 289 library you can pass the -T flag to avoid a clean & rebuild. This should 290 take you just a few seconds to generate the library if you modified one file. 291 292 ```bash 293 tensorflow/contrib/makefile/build_all_ios.sh -a arm64 -T 294 ``` 295 296 To see TensorFlow running on iOS, the example Xcode project in 297 [tensorflow/examples/ios](../../examples/ios/) shows how to use the static 298 library in a simple app. 299 300 ### Building by hand 301 302 This section covers each step of building. For all the code in one place, see 303 [build_all_ios.sh](build_all_ios.sh). 304 305 If you have not already, you will need to download dependencies: 306 307 ```bash 308 tensorflow/contrib/makefile/download_dependencies.sh 309 ``` 310 311 Next, you will need to compile protobufs for iOS (optionally takes the -a $ARCH flag): 312 313 ```bash 314 tensorflow/contrib/makefile/compile_ios_protobuf.sh 315 ``` 316 317 Then, you will need to compile the nsync library for iOS (optionally takes -a $ARCH flag): 318 319 ```bash 320 export HOST_NSYNC_LIB=`tensorflow/contrib/makefile/compile_nsync.sh` 321 export TARGET_NSYNC_LIB=`tensorflow/contrib/makefile/compile_nsync.sh -t ios` 322 ``` 323 Then, you can run the makefile specifying iOS as the target, along with the 324 architecture you want to build for: 325 326 ```bash 327 make -f tensorflow/contrib/makefile/Makefile \ 328 TARGET=IOS \ 329 IOS_ARCH=ARM64 330 ``` 331 332 This creates a library in 333 `tensorflow/contrib/makefile/gen/lib/libtensorflow-core.a` that you can link any 334 xcode project against. 335 336 To see TensorFlow running on iOS, the example Xcode project in 337 [tensorflow/examples/ios](../../examples/ios/) shows how to use the static 338 library in a simple app. 339 340 #### Universal binaries 341 342 In some situations, you will need a universal library. In that case, you will 343 still need to run `compile_ios_protobuf.sh` and `compile_nsync.sh`, but this 344 time follow it with: 345 346 ```bash 347 compile_ios_tensorflow.sh 348 ``` 349 350 `compile_ios_tensorflow.sh` takes the -a flag to build only for one architecture. 351 In case you run into issues with unresolved symbols with nsync you can also pass 352 -h ${HOST_NSYNC_LIB} and -n {TARGET_NSYNC_LIB} so it would look like: 353 354 ```bash 355 tensorflow/contrib/makefile/compile_ios_tensorflow.sh -f "-O3" -h tensorflow/contrib/makefile/downloads/nsync/builds/default.macos.c++11/nsync.a -n tensorflow/contrib/makefile/downloads/nsync/builds/lipo.ios.c++11/nsync.a -a arm64 356 ``` 357 358 In XCode, you will need to use -force_load in the linker flags 359 section of the build settings to pull in the global constructors that are used 360 to register ops and kernels. 361 362 #### Optimization 363 364 The `build_all_ios.sh` script can take optional command-line arguments to 365 selectively register only for the operators used in your graph. 366 367 ```bash 368 tensorflow/contrib/makefile/build_all_ios.sh -a arm64 -g $HOME/graphs/inception/tensorflow_inception_graph.pb 369 ``` 370 Please note this is an aggressive optimization of the operators and the resulting library may not work with other graphs but will reduce the size of the final library. 371 372 The `compile_ios_tensorflow.sh` script can take optional command-line arguments. 373 The first argument will be passed as a C++ optimization flag and defaults to 374 debug mode. If you are concerned about performance or are working on a release 375 build, you would likely want a higher optimization setting, like so: 376 377 ```bash 378 compile_ios_tensorflow.sh -f "-Os" 379 ``` 380 381 For other variations of valid optimization flags, see [clang optimization levels](http://stackoverflow.com/questions/15548023/clang-optimization-levels). 382 383 ## Raspberry Pi 384 385 Building on the Raspberry Pi is similar to a normal Linux system. First 386 download the dependencies, install the required packages and build protobuf: 387 388 ```bash 389 tensorflow/contrib/makefile/download_dependencies.sh 390 sudo apt-get install -y autoconf automake libtool gcc-4.8 g++-4.8 391 cd tensorflow/contrib/makefile/downloads/protobuf/ 392 ./autogen.sh 393 ./configure 394 make 395 sudo make install 396 sudo ldconfig # refresh shared library cache 397 cd ../../../../.. 398 export HOST_NSYNC_LIB=`tensorflow/contrib/makefile/compile_nsync.sh` 399 export TARGET_NSYNC_LIB="$HOST_NSYNC_LIB" 400 ``` 401 402 Once that's done, you can use make to build the library and example: 403 404 ```bash 405 make -f tensorflow/contrib/makefile/Makefile HOST_OS=PI TARGET=PI OPTFLAGS="-Os" CXX=g++-4.8 406 ``` 407 408 If you're only interested in building for Raspberry Pi's 2 and 3, you can supply 409 some extra optimization flags to give you code that will run faster: 410 411 ```bash 412 make -f tensorflow/contrib/makefile/Makefile HOST_OS=PI TARGET=PI \ 413 OPTFLAGS="-Os -mfpu=neon-vfpv4 -funsafe-math-optimizations -ftree-vectorize" CXX=g++-4.8 414 ``` 415 416 One thing to be careful of is that the gcc version 4.9 currently installed on 417 Jessie by default will hit an error mentioning `__atomic_compare_exchange`. This 418 is why the examples above specify `CXX=g++-4.8` explicitly, and why we install 419 it using apt-get. If you have partially built using the default gcc 4.9, hit the 420 error and switch to 4.8, you need to do a 421 `make -f tensorflow/contrib/makefile/Makefile clean` before you build. If you 422 don't, the build will appear to succeed but you'll encounter [malloc(): memory corruption errors](https://github.com/tensorflow/tensorflow/issues/3442) 423 when you try to run any programs using the library. 424 425 For more examples, look at the tensorflow/contrib/pi_examples folder in the 426 source tree, which contains code samples aimed at the Raspberry Pi. 427 428 # Other notes 429 430 ## Supported Systems 431 432 The Make script has been tested on Ubuntu and OS X. If you look in the Makefile 433 itself, you'll see it's broken up into host and target sections. If you are 434 cross-compiling, you should look at customizing the target settings to match 435 what you need for your desired system. 436 437 ## Dependency Management 438 439 The Makefile loads in a list of dependencies stored in text files. These files 440 are generated from the main Bazel build by running 441 `tensorflow/contrib/makefile/gen_file_lists.sh`. You'll need to re-run this i 442 you make changes to the files that are included in the build. 443 444 Header dependencies are not automatically tracked by the Makefile, so if you 445 make header changes you will need to run this command to recompile cleanly: 446 447 ```bash 448 make -f tensorflow/contrib/makefile/Makefile clean 449 ``` 450 451 ### Cleaning up 452 453 In some situations, you may want to completely clean up. The dependencies, 454 intermediate stages, and generated files are stored in: 455 456 ```bash 457 tensorflow/contrib/makefile/downloads 458 tensorflow/contrib/makefile/gen 459 ``` 460 461 Those directories can safely be removed, but you will have to start over with 462 `download_dependencies.sh` once you delete them. 463 464 ### Fixing Makefile Issues 465 466 Because the main development of TensorFlow is done using Bazel, changes to the 467 codebase can sometimes break the makefile build process. If you find that tests 468 relying on this makefile are failing with a change you're involved in, here are 469 some trouble-shooting steps: 470 471 - Try to reproduce the issue on your platform. If you're on Linux, running 472 `make -f tensorflow/contrib/makefile/Makefile` should be enough to recreate 473 most issues. For other platforms, see the sections earlier in this document. 474 475 - The most common cause of breakages are files that have been added to the 476 Bazel build scripts, but that the makefile isn't aware of. Typical symptoms 477 of this include linker errors mentioning missing symbols or protobuf headers 478 that aren't found. To address these problems, take a look at the *.txt files 479 in `tensorflow/contrib/makefile`. If you have a new operator, you may need to 480 add it to `tf_op_files.txt`, or for a new proto to `tf_proto_files.txt`. 481 482 - There's also a wildcard system in `Makefile` that defines what core C++ files 483 are included in the library. This is designed to match the equivalent rule in 484 `tensorflow/core/BUILD`, so if you change the wildcards there to include new 485 files you'll need to also update `CORE_CC_ALL_SRCS` and `CORE_CC_EXCLUDE_SRCS` 486 in the makefile. 487 488 - Some of the supported platforms use clang instead of gcc as their compiler, 489 so if you're hitting compile errors you may need to tweak your code to be more 490 friendly to different compilers by avoiding gcc extensions or idioms. 491 492 These are the most common reasons for makefile breakages, but it's also 493 possible you may hit something unusual, like a platform incompatibility. For 494 those, you'll need to see if you can reproduce the issue on that particular 495 platform and debug it there. You can also reach out to the broader TensorFlow 496 team by [filing a Github issue](https://github.com/tensorflow/tensorflow/issues) 497 to ask for help. 498
