Home | History | Annotate | Download | only in extensions 
      1 # NNAPI vendor extensions
      2 
      3 In Android Q, Neural Networks API introduced vendor extensions -- a better,
      4 more structured alternative to the OEM operation and data types.
      5 An extension is a collection of vendor-defined operations and data types.
      6 A driver can provide custom hardware-accelerated operations via NNAPI 1.2+
      7 by supporting corresponding vendor extensions.
      8 
      9 Note that extensions do not modify behavior of existing operations.
     10 
     11 This document explains how to create and use extensions.
     12 
     13 ## Extensions usage allowlist
     14 
     15 Vendor extensions can only be used by explicitly specified Android apps and
     16 native binaries on the /product, /vendor, /odm, and /data partitions. It's not possible to
     17 specify an app or a native binary located on the /system partition.
     18 
     19 The allowlist is stored in `/vendor/etc/nnapi_extensions_app_allowlist`, and contains
     20 a list of Android apps and binaries permitted to use NNAPI vendor extensions.
     21 Each line of the file contains a new entry. If an entry is prefixed by '/',
     22 then it's a native binary path (e.g. '/data/foo'). If not, it's a name of an Android
     23 app package (e.g. 'com.foo.bar').
     24 
     25 Allowlist is enforced from the NNAPI runtime shared library. It protects
     26 against accidental usage, but not against deliberate circumvention by directly
     27 using the NNAPI driver HAL interface.
     28 
     29 ## Vendor extension definition
     30 
     31 The vendor is expected to create and maintain a header file with the
     32 extension definition. A complete example is provided in
     33 `test_vendor/fibonacci/FibonacciExtension.h`.
     34 
     35 Each extension must have a unique name that starts with the reverse domain name
     36 of the vendor:
     37 ```c
     38 const char MY_EXTENSION_NAME[] = "com.example.my_extension";
     39 ```
     40 
     41 This name acts as a namespace for operations and data types.
     42 NNAPI uses this name to distinguish between extensions.
     43 
     44 Operations and data types are declared in a way similar to
     45 `../runtime/include/NeuralNetworks.h`:
     46 ```c
     47 enum {
     48     /**
     49      * A custom scalar type.
     50      */
     51     MY_SCALAR = 0,
     52 
     53     /**
     54      * A custom tensor type.
     55      *
     56      * Attached to this tensor is {@link MyTensorParams}.
     57      */
     58     MY_TENSOR = 1,
     59 };
     60 
     61 enum {
     62     /**
     63      * Computes my function.
     64      *
     65      * Inputs:
     66      * * 0: A scalar of {@link MY_SCALAR}.
     67      *
     68      * Outputs:
     69      * * 0: A tensor of {@link MY_TENSOR}.
     70      */
     71     MY_FUNCTION = 0,
     72 };
     73 ```
     74 
     75 An extension operation may use any operand types, including non-extension
     76 operand types and operand types from other extensions. In the latter case,
     77 the driver must support those other extensions in order to support the
     78 extension.
     79 
     80 Extensions may also declare custom structures to accompany extension operands:
     81 ```c
     82 /**
     83  * Quantization parameters for {@link MY_TENSOR}.
     84  */
     85 typedef struct MyTensorParams {
     86     double scale;
     87     int64_t zeroPoint;
     88 } MyTensorParams;
     89 ```
     90 
     91 ## Using extensions in NNAPI clients
     92 
     93 Runtime extension support is provided by
     94 `../runtime/include/NeuralNetworksExtensions.h` (C API) and
     95 `../runtime/include/NeuralNetworksWrapperExtensions.h` (C++ API).
     96 This section provides an overview of the former.
     97 
     98 Use `ANeuralNetworksDevice_getExtensionSupport` to check whether a device
     99 supports an extension:
    100 ```c
    101 bool isExtensionSupported;
    102 CHECK_EQ(ANeuralNetworksDevice_getExtensionSupport(device, MY_EXTENSION_NAME,
    103                                                    &isExtensionSupported),
    104          ANEURALNETWORKS_NO_ERROR);
    105 if (isExtensionSupported) {
    106     // The device supports the extension.
    107     ...
    108 }
    109 ```
    110 
    111 To build a model with an extension operand, use
    112 `ANeuralNetworksModel_getExtensionOperandType` to obtain the operand type.
    113 Then call `ANeuralNetworksModel_addOperand` as usual:
    114 ```c
    115 int32_t type;
    116 CHECK_EQ(ANeuralNetworksModel_getExtensionOperandType(model, MY_EXTENSION_NAME, MY_TENSOR, &type),
    117          ANEURALNETWORKS_NO_ERROR);
    118 ANeuralNetworksOperandType operandType{
    119         .type = type,
    120         .dimensionCount = dimensionCount,
    121         .dimensions = dimensions,
    122 };
    123 CHECK_EQ(ANeuralNetworksModel_addOperand(model, &operandType), ANEURALNETWORKS_NO_ERROR);
    124 ```
    125 
    126 Optionally, use `ANeuralNetworksModel_setOperandExtensionData` to
    127 associate additional data with an extension operand.
    128 ```c
    129 MyTensorParams params{
    130         .scale = 0.5,
    131         .zeroPoint = 128,
    132 };
    133 CHECK_EQ(ANeuralNetworksModel_setOperandExtensionData(model, operandIndex, &params, sizeof(params)),
    134          ANEURALNETWORKS_NO_ERROR);
    135 ```
    136 
    137 To build a model with an extension operation, use
    138 `ANeuralNetworksModel_getExtensionOperationType` to obtain the operation type.
    139 Then call `ANeuralNetworksModel_addOperation` as usual:
    140 ```c
    141 ANeuralNetworksOperationType type;
    142 CHECK_EQ(ANeuralNetworksModel_getExtensionOperationType(model, MY_EXTENSION_NAME, MY_FUNCTION,
    143                                                         &type),
    144          ANEURALNETWORKS_NO_ERROR);
    145 CHECK_EQ(ANeuralNetworksModel_addOperation(model, type, inputCount, inputs, outputCount, outputs),
    146          ANEURALNETWORKS_NO_ERROR);
    147 ```
    148 
    149 ## Adding extension support to an NNAPI driver
    150 
    151 The driver reports supported extensions via the
    152 `IDevice::getSupportedExtensions()` method.
    153 For each supported extension, the returned list must contain an entry
    154 describing it:
    155 ```c++
    156 Extension {
    157     .name = MY_EXTENSION_NAME,
    158     .operandTypes = {
    159         {
    160             .type = MY_SCALAR,
    161             .isTensor = false,
    162             .byteSize = 8,
    163         },
    164         {
    165             .type = MY_TENSOR,
    166             .isTensor = true,
    167             .byteSize = 8,
    168         },
    169     },
    170 }
    171 ```
    172 
    173 When handling operation and operand types, the driver must check the
    174 `Model::ExtensionTypeEncoding::HIGH_BITS_PREFIX` high bits of the type.
    175 These bits constitute the extension _prefix_. A zero prefix means no extension,
    176 whereas a non-zero prefix maps uniquely within a model to an extension name via
    177 `model.extensionNameToPrefix`.
    178 The low `Model::ExtensionTypeEncoding::LOW_BITS_TYPE` bits of the type
    179 correspond to the type within the extension.
    180 
    181 The driver must validate extension operations and data types, as the NNAPI
    182 runtime does not know how to validate particular extension operations and data
    183 types.
    184 
    185 Extension operands may have associated data in `operand.extraParams.extension`,
    186 which the runtime treats as a raw data blob of arbitrary size.
    187