README.md
1 # TensorFlow Android Camera Demo
2
3 This folder contains an example application utilizing TensorFlow for Android
4 devices.
5
6 ## Description
7
8 The demos in this folder are designed to give straightforward samples of using
9 TensorFlow in mobile applications.
10
11 Inference is done using the [TensorFlow Android Inference
12 Interface](../../../tensorflow/contrib/android), which may be built separately
13 if you want a standalone library to drop into your existing application. Object
14 tracking and efficient YUV -> RGB conversion are handled by
15 `libtensorflow_demo.so`.
16
17 A device running Android 5.0 (API 21) or higher is required to run the demo due
18 to the use of the camera2 API, although the native libraries themselves can run
19 on API >= 14 devices.
20
21 ## Current samples:
22
23 1. [TF Classify](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/android/src/org/tensorflow/demo/ClassifierActivity.java):
24 Uses the [Google Inception](https://arxiv.org/abs/1409.4842)
25 model to classify camera frames in real-time, displaying the top results
26 in an overlay on the camera image.
27 2. [TF Detect](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/android/src/org/tensorflow/demo/DetectorActivity.java):
28 Demonstrates an SSD-Mobilenet model trained using the
29 [Tensorflow Object Detection API](https://github.com/tensorflow/models/tree/master/research/object_detection/)
30 introduced in [Speed/accuracy trade-offs for modern convolutional object detectors](https://arxiv.org/abs/1611.10012) to
31 localize and track objects (from 80 categories) in the camera preview
32 in real-time.
33 3. [TF Stylize](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/android/src/org/tensorflow/demo/StylizeActivity.java):
34 Uses a model based on [A Learned Representation For Artistic
35 Style](https://arxiv.org/abs/1610.07629) to restyle the camera preview
36 image to that of a number of different artists.
37 4. [TF
38 Speech](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/android/src/org/tensorflow/demo/SpeechActivity.java):
39 Runs a simple speech recognition model built by the [audio training
40 tutorial](https://www.tensorflow.org/versions/master/tutorials/audio_recognition). Listens
41 for a small set of words, and highlights them in the UI when they are
42 recognized.
43
44 <img src="sample_images/classify1.jpg" width="30%"><img src="sample_images/stylize1.jpg" width="30%"><img src="sample_images/detect1.jpg" width="30%">
45
46 ## Prebuilt Components:
47
48 The fastest path to trying the demo is to download the [prebuilt demo APK](http://download.tensorflow.org/deps/tflite/TfLiteCameraDemo.apk).
49
50 Also available are precompiled native libraries, and a jcenter package that you
51 may simply drop into your own applications. See
52 [tensorflow/contrib/android/README.md](../../../tensorflow/contrib/android/README.md)
53 for more details.
54
55 ## Running the Demo
56
57 Once the app is installed it can be started via the "TF Classify", "TF Detect",
58 "TF Stylize", and "TF Speech" icons, which have the orange TensorFlow logo as
59 their icon.
60
61 While running the activities, pressing the volume keys on your device will
62 toggle debug visualizations on/off, rendering additional info to the screen that
63 may be useful for development purposes.
64
65 ## Building in Android Studio using the TensorFlow AAR from JCenter
66
67 The simplest way to compile the demo app yourself, and try out changes to the
68 project code is to use AndroidStudio. Simply set this `android` directory as the
69 project root.
70
71 Then edit the `build.gradle` file and change the value of `nativeBuildSystem` to
72 `'none'` so that the project is built in the simplest way possible:
73
74 ```None
75 def nativeBuildSystem = 'none'
76 ```
77
78 While this project includes full build integration for TensorFlow, this setting
79 disables it, and uses the TensorFlow Inference Interface package from JCenter.
80
81 Note: Currently, in this build mode, YUV -> RGB is done using a less efficient
82 Java implementation, and object tracking is not available in the "TF Detect"
83 activity. Setting the build system to `'cmake'` currently only builds
84 `libtensorflow_demo.so`, which provides fast YUV -> RGB conversion and object
85 tracking, while still acquiring TensorFlow support via the downloaded AAR, so it
86 may be a lightweight way to enable these features.
87
88 For any project that does not include custom low level TensorFlow code, this is
89 likely sufficient.
90
91 For details on how to include this JCenter package in your own project see
92 [tensorflow/contrib/android/README.md](../../../tensorflow/contrib/android/README.md)
93
94 ## Building the Demo with TensorFlow from Source
95
96 Pick your preferred approach below. At the moment, we have full support for
97 Bazel, and partial support for gradle, cmake, make, and Android Studio.
98
99 As a first step for all build types, clone the TensorFlow repo with:
100
101 ```
102 git clone --recurse-submodules https://github.com/tensorflow/tensorflow.git
103 ```
104
105 Note that `--recurse-submodules` is necessary to prevent some issues with
106 protobuf compilation.
107
108 ### Bazel
109
110 NOTE: Bazel does not currently support building for Android on Windows. Full
111 support for gradle/cmake builds is coming soon, but in the meantime we suggest
112 that Windows users download the [prebuilt demo APK](http://download.tensorflow.org/deps/tflite/TfLiteCameraDemo.apk) instead.
113
114 ##### Install Bazel and Android Prerequisites
115
116 Bazel is the primary build system for TensorFlow. To build with Bazel, it and
117 the Android NDK and SDK must be installed on your system.
118
119 1. Install the latest version of Bazel as per the instructions [on the Bazel
120 website](https://bazel.build/versions/master/docs/install.html).
121 2. The Android NDK is required to build the native (C/C++) TensorFlow code. The
122 current recommended version is 14b, which may be found
123 [here](https://developer.android.com/ndk/downloads/older_releases.html#ndk-14b-downloads).
124 3. The Android SDK and build tools may be obtained
125 [here](https://developer.android.com/tools/revisions/build-tools.html), or
126 alternatively as part of [Android
127 Studio](https://developer.android.com/studio/index.html). Build tools API >=
128 23 is required to build the TF Android demo (though it will run on API >= 21
129 devices).
130
131 ##### Edit WORKSPACE
132
133 NOTE: As long as you have the SDK and NDK installed, the `./configure` script
134 will create these rules for you. Answer "Yes" when the script asks to
135 automatically configure the `./WORKSPACE`.
136
137 The Android entries in
138 [`<workspace_root>/WORKSPACE`](../../../WORKSPACE#L19-L36) must be uncommented
139 with the paths filled in appropriately depending on where you installed the NDK
140 and SDK. Otherwise an error such as: "The external label
141 '//external:android/sdk' is not bound to anything" will be reported.
142
143 Also edit the API levels for the SDK in WORKSPACE to the highest level you have
144 installed in your SDK. This must be >= 23 (this is completely independent of the
145 API level of the demo, which is defined in AndroidManifest.xml). The NDK API
146 level may remain at 14.
147
148 ##### Install Model Files (optional)
149
150 The TensorFlow `GraphDef`s that contain the model definitions and weights are
151 not packaged in the repo because of their size. They are downloaded
152 automatically and packaged with the APK by Bazel via a new_http_archive defined
153 in `WORKSPACE` during the build process, and by Gradle via
154 download-models.gradle.
155
156 **Optional**: If you wish to place the models in your assets manually, remove
157 all of the `model_files` entries from the `assets` list in `tensorflow_demo`
158 found in the [`BUILD`](BUILD#L92) file. Then download and extract the archives
159 yourself to the `assets` directory in the source tree:
160
161 ```bash
162 BASE_URL=https://storage.googleapis.com/download.tensorflow.org/models
163 for MODEL_ZIP in inception5h.zip ssd_mobilenet_v1_android_export.zip stylize_v1.zip
164 do
165 curl -L ${BASE_URL}/${MODEL_ZIP} -o /tmp/${MODEL_ZIP}
166 unzip /tmp/${MODEL_ZIP} -d tensorflow/examples/android/assets/
167 done
168 ```
169
170 This will extract the models and their associated metadata files to the local
171 assets/ directory.
172
173 If you are using Gradle, make sure to remove download-models.gradle reference
174 from build.gradle after your manually download models; otherwise gradle might
175 download models again and overwrite your models.
176
177 ##### Build
178
179 After editing your WORKSPACE file to update the SDK/NDK configuration, you may
180 build the APK. Run this from your workspace root:
181
182 ```bash
183 bazel build --cxxopt='--std=c++11' -c opt //tensorflow/examples/android:tensorflow_demo
184 ```
185
186 ##### Install
187
188 Make sure that adb debugging is enabled on your Android 5.0 (API 21) or later
189 device, then after building use the following command from your workspace root
190 to install the APK:
191
192 ```bash
193 adb install -r bazel-bin/tensorflow/examples/android/tensorflow_demo.apk
194 ```
195
196 ### Android Studio with Bazel
197
198 Android Studio may be used to build the demo in conjunction with Bazel. First,
199 make sure that you can build with Bazel following the above directions. Then,
200 look at [build.gradle](build.gradle) and make sure that the path to Bazel
201 matches that of your system.
202
203 At this point you can add the tensorflow/examples/android directory as a new
204 Android Studio project. Click through installing all the Gradle extensions it
205 requests, and you should be able to have Android Studio build the demo like any
206 other application (it will call out to Bazel to build the native code with the
207 NDK).
208
209 ### CMake
210
211 Full CMake support for the demo is coming soon, but for now it is possible to
212 build the TensorFlow Android Inference library using
213 [tensorflow/contrib/android/cmake](../../../tensorflow/contrib/android/cmake).
214