Home | History | Annotate | Download | only in basics 
      1 # Basics
      2 
      3 [TOC]
      4 
      5 ## Overview
      6 
      7 Before understanding how testing is done on [Android platform](http://source.android.com/devices/index.html),
      8 please refer to the Android platform architecture for an overview.
      9 The diagram from that page is embedded below for convenience:
     10 
     11 ![Android platform architecture](../imgs/ape_fwk_all.png)
     12 
     13 ## What to Test and How to Test
     14 
     15 A platform test typically interacts with one or more of the Android system
     16 services, or HAL layers, exercises the functionalities of the subject under test
     17 and assert correctness of the testing outcome.
     18 
     19 As such, a platform test may:
     20 
     21 1.  exercise framework APIs via application framework; specific APIs being
     22     exercised may include:
     23     *   public APIs intended for 3rd party applications
     24     *   hidden APIs intended for privileged applications, aka system APIs
     25     *   private APIs (@hide, or protected, package private)
     26 1.  invoke Android system services via raw binder/IPC proxies directly
     27 1.  interact directly with HALs via low level APIs or IPC interfaces
     28 
     29 Type 1 and 2 above are typically written as **instrumentation tests**, while
     30 type 3 is typically written as **native tests** using gtest framework.
     31 
     32 ## Instrumentation Tests for Platform Testing
     33 
     34 You may have read the [Testing Fundamentals](https://developer.android.com/tools/testing/testing_android.html)
     35 on `developer.android.com`, however, there still may be some differences in
     36 how instrumentation tests are used in platform testing.
     37 
     38 In a nutshell, an instrumentation test provides a special test execution
     39 environment as launched via `am instrument` command, where the targeted
     40 application process is restarted and initialized with basic application context,
     41 and an instrumentation thread is started inside the application process VM. Your
     42 test code starts execution on this instrumentation thread, and is provided with
     43 an `Instrumentation` instance which provides access to the application context
     44 and APIs to manipulate the application process under test.
     45 
     46 Some key concepts about instrumentation:
     47 
     48 *   an instrumentation must be declared in an application package, with an
     49     [`<instrumentation>`](https://developer.android.com/guide/topics/manifest/instrumentation-element.html)
     50     tag nested under the `<manifest>` tag of the application package manifest
     51 *   an application package manifest may technically contain multiple
     52     `<instrumentation>` tags, though it's not commonly used in this fashion
     53 *   each `<instrumentation>` must contain:
     54     *   an `android:name` attribute: it should be the name of a subclass of
     55         [`Instrumentation`](https://developer.android.com/reference/android/app/Instrumentation.html)
     56         that's included in the test application, which is typically the test
     57         runner that's being used, e.g.
     58         `android.support.test.runner.AndroidJUnitRunner`
     59     *   an `android:targetPackage` attribute must be defined. Its value should
     60         be set to the application package under test
     61 
     62 In the context of platform testing, there are typically two categories of
     63 instrumentation tests:
     64 
     65 ### Instrumentation tests targeting application packages included in system
     66 
     67 This category of instrumentation test isn't that different from those targeting
     68 the regular Android applications. It's worth noting that the test application
     69 that included the instrumentation needs to be signed with the same certificate
     70 as the application that it's targeting.
     71 
     72 To learn more, see our [end-to-end example](../development/instr-app-e2e.md).
     73 
     74 ### Instrumentation tests targeting itself
     75 
     76 As mentioned earlier, when an instrumentation is started, its target package is
     77 restarted with instrumentation code injected and initiated for execution. One
     78 exception is that the target package here cannot be the Android application
     79 framework itself, i.e. the package `android`, because doing so would lead to the
     80 paradoxical situation where Android framework would need to be restarted, which
     81 is what supports the system functions, including the instrumentation itself.
     82 
     83 This means that an instrumentation test cannot inject itself into Android
     84 framework, a.k.a. the system server, for execution. In order to test Android
     85 framework, the test code can only invoke public API surfaces, or those exposed
     86 via [AIDL](https://developer.android.com/guide/components/aidl.html)s available
     87 in platform source tree. For this category of tests, it's not meaningful to
     88 target any particular package, therefore it's customary for such
     89 instrumentations to be declared to target its own test application package, as
     90 defined in its own `<manifest>` tag of `AndroidManifest.xml`.
     91 
     92 Depending on the requirements, test application packages in this category may
     93 also:
     94 
     95 *   Bundle activities needed for testing.
     96 *   Share the user ID with the system.
     97 *   Be signed with the platform key.
     98 *   Be compiled against the framework source rather than the public SDK.
     99 
    100 This category of instrumentation tests are sometimes refered to as
    101 self-instrumentations. Below are some examples of such instrumentation tests in
    102 platform source:
    103 
    104 ```
    105 frameworks/base/core/tests/coretests
    106 frameworks/base/services/tests/servicestests
    107 ```
    108 
    109 To learn more, see our [end-to-end example](../development/instr-self-e2e.md).
    110 
    111 ## Native Tests for Platform Testing
    112 
    113 A native test for platform testing typically accesses lower level HALs, or
    114 performs raw IPC against various system services, therefore the testing approach
    115 is typically tightly coupled with the service under test, outside the scope of
    116 this documentation.
    117 
    118 Building native tests using [gtest](https://github.com/google/googletest)
    119 framework is strongly recommended, and a prerequisite for integration with
    120 continuous testing infrastructure.
    121 
    122 Below are some examples of such native tests in platform source:
    123 
    124 ```
    125 frameworks/av/camera/tests
    126 frameworks/native/libs/gui/tests
    127 ```
    128 
    129 To learn more, see our [end-to-end example](../development/native-func-e2e.md).
    130 
    131 ## Compatibility Test Suite (CTS)
    132 
    133 [Android Compatibility Test Suite](https://source.android.com/compatibility/cts/)
    134 is a suite of various types of tests, used to ensure compatibility of
    135 Android framework implementations across OEM partners, and across platform
    136 releases. **The suite also includes instrumentation tests and native tests
    137 (also using gtest framework).**
    138 
    139 CTS and platform tests are not mutually exclusive, and here are some general
    140 guidelines:
    141 
    142 *   if a test is asserting correctness of framework API functions/behaviors, and
    143     it should be enforced across OEM partners, it should be in CTS
    144 *   if a test is intended to catch regressions during platform development
    145     cycle, and may require privileged permission to carry out, and may be
    146     dependent on implementation details (as released in AOSP), it should only be
    147     platform tests
    148