1 FlexBuffers {#flexbuffers} 2 ========== 3 4 FlatBuffers was designed around schemas, because when you want maximum 5 performance and data consistency, strong typing is helpful. 6 7 There are however times when you want to store data that doesn't fit a 8 schema, because you can't know ahead of time what all needs to be stored. 9 10 For this, FlatBuffers has a dedicated format, called FlexBuffers. 11 This is a binary format that can be used in conjunction 12 with FlatBuffers (by storing a part of a buffer in FlexBuffers 13 format), or also as its own independent serialization format. 14 15 While it loses the strong typing, you retain the most unique advantage 16 FlatBuffers has over other serialization formats (schema-based or not): 17 FlexBuffers can also be accessed without parsing / copying / object allocation. 18 This is a huge win in efficiency / memory friendly-ness, and allows unique 19 use cases such as mmap-ing large amounts of free-form data. 20 21 FlexBuffers' design and implementation allows for a very compact encoding, 22 combining automatic pooling of strings with automatic sizing of containers to 23 their smallest possible representation (8/16/32/64 bits). Many values and 24 offsets can be encoded in just 8 bits. While a schema-less representation is 25 usually more bulky because of the need to be self-descriptive, FlexBuffers 26 generates smaller binaries for many cases than regular FlatBuffers. 27 28 FlexBuffers is still slower than regular FlatBuffers though, so we recommend to 29 only use it if you need it. 30 31 32 # Usage 33 34 This is for C++, other languages may follow. 35 36 Include the header `flexbuffers.h`, which in turn depends on `flatbuffers.h` 37 and `util.h`. 38 39 To create a buffer: 40 41 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.cpp} 42 flexbuffers::Builder fbb; 43 fbb.Int(13); 44 fbb.Finish(); 45 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 46 47 You create any value, followed by `Finish`. Unlike FlatBuffers which requires 48 the root value to be a table, here any value can be the root, including a lonely 49 int value. 50 51 You can now access the `std::vector<uint8_t>` that contains the encoded value 52 as `fbb.GetBuffer()`. Write it, send it, or store it in a parent FlatBuffer. In 53 this case, the buffer is just 3 bytes in size. 54 55 To read this value back, you could just say: 56 57 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.cpp} 58 auto root = flexbuffers::GetRoot(my_buffer); 59 int64_t i = root.AsInt64(); 60 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 61 62 FlexBuffers stores ints only as big as needed, so it doesn't differentiate 63 between different sizes of ints. You can ask for the 64 bit version, 64 regardless of what you put in. In fact, since you demand to read the root 65 as an int, if you supply a buffer that actually contains a float, or a 66 string with numbers in it, it will convert it for you on the fly as well, 67 or return 0 if it can't. If instead you actually want to know what is inside 68 the buffer before you access it, you can call `root.GetType()` or `root.IsInt()` 69 etc. 70 71 Here's a slightly more complex value you could write instead of `fbb.Int` above: 72 73 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.cpp} 74 fbb.Map([&]() { 75 fbb.Vector("vec", [&]() { 76 fbb.Int(-100); 77 fbb.String("Fred"); 78 fbb.IndirectFloat(4.0f); 79 }); 80 fbb.UInt("foo", 100); 81 }); 82 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 83 84 This stores the equivalent of the JSON value 85 `{ vec: [ -100, "Fred", 4.0 ], foo: 100 }`. The root is a dictionary that has 86 just two key-value pairs, with keys `vec` and `foo`. Unlike FlatBuffers, it 87 actually has to store these keys in the buffer (which it does only once if 88 you store multiple such objects, by pooling key values), but also unlike 89 FlatBuffers it has no restriction on the keys (fields) that you use. 90 91 The map constructor uses a C++11 Lambda to group its children, but you can 92 also use more conventional start/end calls if you prefer. 93 94 The first value in the map is a vector. You'll notice that unlike FlatBuffers, 95 you can use mixed types. There is also a `TypedVector` variant that only 96 allows a single type, and uses a bit less memory. 97 98 `IndirectFloat` is an interesting feature that allows you to store values 99 by offset rather than inline. Though that doesn't make any visible change 100 to the user, the consequence is that large values (especially doubles or 101 64 bit ints) that occur more than once can be shared. Another use case is 102 inside of vectors, where the largest element makes up the size of all elements 103 (e.g. a single double forces all elements to 64bit), so storing a lot of small 104 integers together with a double is more efficient if the double is indirect. 105 106 Accessing it: 107 108 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.cpp} 109 auto map = flexbuffers::GetRoot(my_buffer).AsMap(); 110 map.size(); // 2 111 auto vec = map["vec"].AsVector(); 112 vec.size(); // 3 113 vec[0].AsInt64(); // -100; 114 vec[1].AsString().c_str(); // "Fred"; 115 vec[1].AsInt64(); // 0 (Number parsing failed). 116 vec[2].AsDouble(); // 4.0 117 vec[2].AsString().IsTheEmptyString(); // true (Wrong Type). 118 vec[2].AsString().c_str(); // "" (This still works though). 119 vec[2].ToString().c_str(); // "4" (Or have it converted). 120 map["foo"].AsUInt8(); // 100 121 map["unknown"].IsNull(); // true 122 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 123 124 125 # Binary encoding 126 127 A description of how FlexBuffers are encoded is in the 128 [internals](@ref flatbuffers_internals) document. 129 130 131 # Nesting inside a FlatBuffer 132 133 You can mark a field as containing a FlexBuffer, e.g. 134 135 a:[ubyte] (flexbuffer); 136 137 A special accessor will be generated that allows you to access the root value 138 directly, e.g. `a_flexbuffer_root().AsInt64()`. 139 140 141 # Efficiency tips 142 143 * Vectors generally are a lot more efficient than maps, so prefer them over maps 144 when possible for small objects. Instead of a map with keys `x`, `y` and `z`, 145 use a vector. Better yet, use a typed vector. Or even better, use a fixed 146 size typed vector. 147 * Maps are backwards compatible with vectors, and can be iterated as such. 148 You can iterate either just the values (`map.Values()`), or in parallel with 149 the keys vector (`map.Keys()`). If you intend 150 to access most or all elements, this is faster than looking up each element 151 by key, since that involves a binary search of the key vector. 152 * When possible, don't mix values that require a big bit width (such as double) 153 in a large vector of smaller values, since all elements will take on this 154 width. Use `IndirectDouble` when this is a possibility. Note that 155 integers automatically use the smallest width possible, i.e. if you ask 156 to serialize an int64_t whose value is actually small, you will use less 157 bits. Doubles are represented as floats whenever possible losslessly, but 158 this is only possible for few values. 159 Since nested vectors/maps are stored over offsets, they typically don't 160 affect the vector width. 161 * To store large arrays of byte data, use a blob. If you'd use a typed 162 vector, the bit width of the size field may make it use more space than 163 expected, and may not be compatible with `memcpy`. 164 Similarly, large arrays of (u)int16_t may be better off stored as a 165 binary blob if their size could exceed 64k elements. 166 Construction and use are otherwise similar to strings. 167
