1 # TensorFlow Android Camera Demo 2 3 This folder contains an example application utilizing TensorFlow for Android 4 devices. 5 6 ## Description 7 8 The demos in this folder are designed to give straightforward samples of using 9 TensorFlow in mobile applications. 10 11 Inference is done using the [TensorFlow Android Inference 12 Interface](../../../tensorflow/contrib/android), which may be built separately 13 if you want a standalone library to drop into your existing application. Object 14 tracking and efficient YUV -> RGB conversion are handled by 15 `libtensorflow_demo.so`. 16 17 A device running Android 5.0 (API 21) or higher is required to run the demo due 18 to the use of the camera2 API, although the native libraries themselves can run 19 on API >= 14 devices. 20 21 ## Current samples: 22 23 1. [TF Classify](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/android/src/org/tensorflow/demo/ClassifierActivity.java): 24 Uses the [Google Inception](https://arxiv.org/abs/1409.4842) 25 model to classify camera frames in real-time, displaying the top results 26 in an overlay on the camera image. 27 2. [TF Detect](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/android/src/org/tensorflow/demo/DetectorActivity.java): 28 Demonstrates an SSD-Mobilenet model trained using the 29 [Tensorflow Object Detection API](https://github.com/tensorflow/models/tree/master/research/object_detection/) 30 introduced in [Speed/accuracy trade-offs for modern convolutional object detectors](https://arxiv.org/abs/1611.10012) to 31 localize and track objects (from 80 categories) in the camera preview 32 in real-time. 33 3. [TF Stylize](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/android/src/org/tensorflow/demo/StylizeActivity.java): 34 Uses a model based on [A Learned Representation For Artistic 35 Style](https://arxiv.org/abs/1610.07629) to restyle the camera preview 36 image to that of a number of different artists. 37 4. [TF 38 Speech](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/android/src/org/tensorflow/demo/SpeechActivity.java): 39 Runs a simple speech recognition model built by the [audio training 40 tutorial](https://www.tensorflow.org/versions/master/tutorials/audio_recognition). Listens 41 for a small set of words, and highlights them in the UI when they are 42 recognized. 43 44 <img src="sample_images/classify1.jpg" width="30%"><img src="sample_images/stylize1.jpg" width="30%"><img src="sample_images/detect1.jpg" width="30%"> 45 46 ## Prebuilt Components: 47 48 If you just want the fastest path to trying the demo, you may download the 49 nightly build 50 [here](https://ci.tensorflow.org/view/Nightly/job/nightly-android/). Expand the 51 "View" and then the "out" folders under "Last Successful Artifacts" to find 52 tensorflow_demo.apk. 53 54 Also available are precompiled native libraries, and a jcenter package that you 55 may simply drop into your own applications. See 56 [tensorflow/contrib/android/README.md](../../../tensorflow/contrib/android/README.md) 57 for more details. 58 59 ## Running the Demo 60 61 Once the app is installed it can be started via the "TF Classify", "TF Detect", 62 "TF Stylize", and "TF Speech" icons, which have the orange TensorFlow logo as 63 their icon. 64 65 While running the activities, pressing the volume keys on your device will 66 toggle debug visualizations on/off, rendering additional info to the screen that 67 may be useful for development purposes. 68 69 ## Building in Android Studio using the TensorFlow AAR from JCenter 70 71 The simplest way to compile the demo app yourself, and try out changes to the 72 project code is to use AndroidStudio. Simply set this `android` directory as the 73 project root. 74 75 Then edit the `build.gradle` file and change the value of `nativeBuildSystem` to 76 `'none'` so that the project is built in the simplest way possible: 77 78 ```None 79 def nativeBuildSystem = 'none' 80 ``` 81 82 While this project includes full build integration for TensorFlow, this setting 83 disables it, and uses the TensorFlow Inference Interface package from JCenter. 84 85 Note: Currently, in this build mode, YUV -> RGB is done using a less efficient 86 Java implementation, and object tracking is not available in the "TF Detect" 87 activity. Setting the build system to `'cmake'` currently only builds 88 `libtensorflow_demo.so`, which provides fast YUV -> RGB conversion and object 89 tracking, while still acquiring TensorFlow support via the downloaded AAR, so it 90 may be a lightweight way to enable these features. 91 92 For any project that does not include custom low level TensorFlow code, this is 93 likely sufficient. 94 95 For details on how to include this JCenter package in your own project see 96 [tensorflow/contrib/android/README.md](../../../tensorflow/contrib/android/README.md) 97 98 ## Building the Demo with TensorFlow from Source 99 100 Pick your preferred approach below. At the moment, we have full support for 101 Bazel, and partial support for gradle, cmake, make, and Android Studio. 102 103 As a first step for all build types, clone the TensorFlow repo with: 104 105 ``` 106 git clone --recurse-submodules https://github.com/tensorflow/tensorflow.git 107 ``` 108 109 Note that `--recurse-submodules` is necessary to prevent some issues with 110 protobuf compilation. 111 112 ### Bazel 113 114 NOTE: Bazel does not currently support building for Android on Windows. Full 115 support for gradle/cmake builds is coming soon, but in the meantime we suggest 116 that Windows users download the [prebuilt 117 binaries](https://ci.tensorflow.org/view/Nightly/job/nightly-android/) instead. 118 119 ##### Install Bazel and Android Prerequisites 120 121 Bazel is the primary build system for TensorFlow. To build with Bazel, it and 122 the Android NDK and SDK must be installed on your system. 123 124 1. Install the latest version of Bazel as per the instructions [on the Bazel 125 website](https://bazel.build/versions/master/docs/install.html). 126 2. The Android NDK is required to build the native (C/C++) TensorFlow code. The 127 current recommended version is 14b, which may be found 128 [here](https://developer.android.com/ndk/downloads/older_releases.html#ndk-14b-downloads). 129 130 * NDK 16, the revision released in November 2017, is **incompatible** with 131 Bazel. See [here](https://github.com/tensorflow/tensorflow/issues/14918). 132 133 3. The Android SDK and build tools may be obtained 134 [here](https://developer.android.com/tools/revisions/build-tools.html), or 135 alternatively as part of [Android 136 Studio](https://developer.android.com/studio/index.html). Build tools API >= 137 23 is required to build the TF Android demo (though it will run on API >= 21 138 devices). 139 140 - The Android Studio SDK Manager's NDK installer will install the latest 141 revision of the NDK, which is **incompatible** with Bazel. You'll need 142 to download an older version manually, as (2) suggests. 143 144 ##### Edit WORKSPACE 145 146 NOTE: As long as you have the SDK and NDK installed, the `./configure` script 147 will create these rules for you. Answer "Yes" when the script asks to 148 automatically configure the `./WORKSPACE`. 149 150 The Android entries in 151 [`<workspace_root>/WORKSPACE`](../../../WORKSPACE#L19-L36) must be uncommented 152 with the paths filled in appropriately depending on where you installed the NDK 153 and SDK. Otherwise an error such as: "The external label 154 '//external:android/sdk' is not bound to anything" will be reported. 155 156 Also edit the API levels for the SDK in WORKSPACE to the highest level you have 157 installed in your SDK. This must be >= 23 (this is completely independent of the 158 API level of the demo, which is defined in AndroidManifest.xml). The NDK API 159 level may remain at 14. 160 161 ##### Install Model Files (optional) 162 163 The TensorFlow `GraphDef`s that contain the model definitions and weights are 164 not packaged in the repo because of their size. They are downloaded 165 automatically and packaged with the APK by Bazel via a new_http_archive defined 166 in `WORKSPACE` during the build process, and by Gradle via 167 download-models.gradle. 168 169 **Optional**: If you wish to place the models in your assets manually, remove 170 all of the `model_files` entries from the `assets` list in `tensorflow_demo` 171 found in the [`BUILD`](BUILD#L92) file. Then download and extract the archives 172 yourself to the `assets` directory in the source tree: 173 174 ```bash 175 BASE_URL=https://storage.googleapis.com/download.tensorflow.org/models 176 for MODEL_ZIP in inception5h.zip ssd_mobilenet_v1_android_export.zip stylize_v1.zip 177 do 178 curl -L ${BASE_URL}/${MODEL_ZIP} -o /tmp/${MODEL_ZIP} 179 unzip /tmp/${MODEL_ZIP} -d tensorflow/examples/android/assets/ 180 done 181 ``` 182 183 This will extract the models and their associated metadata files to the local 184 assets/ directory. 185 186 If you are using Gradle, make sure to remove download-models.gradle reference 187 from build.gradle after your manually download models; otherwise gradle might 188 download models again and overwrite your models. 189 190 ##### Build 191 192 After editing your WORKSPACE file to update the SDK/NDK configuration, you may 193 build the APK. Run this from your workspace root: 194 195 ```bash 196 bazel build -c opt //tensorflow/examples/android:tensorflow_demo 197 ``` 198 199 ##### Install 200 201 Make sure that adb debugging is enabled on your Android 5.0 (API 21) or later 202 device, then after building use the following command from your workspace root 203 to install the APK: 204 205 ```bash 206 adb install -r bazel-bin/tensorflow/examples/android/tensorflow_demo.apk 207 ``` 208 209 ### Android Studio with Bazel 210 211 Android Studio may be used to build the demo in conjunction with Bazel. First, 212 make sure that you can build with Bazel following the above directions. Then, 213 look at [build.gradle](build.gradle) and make sure that the path to Bazel 214 matches that of your system. 215 216 At this point you can add the tensorflow/examples/android directory as a new 217 Android Studio project. Click through installing all the Gradle extensions it 218 requests, and you should be able to have Android Studio build the demo like any 219 other application (it will call out to Bazel to build the native code with the 220 NDK). 221 222 ### CMake 223 224 Full CMake support for the demo is coming soon, but for now it is possible to 225 build the TensorFlow Android Inference library using 226 [tensorflow/contrib/android/cmake](../../../tensorflow/contrib/android/cmake). 227
