1 # specs -- simple specifications for TensorFlow networks
2
3 This library implements a simple domain-specific language for specifying
4 deep neural networks in TensorFlow.
5
6 From a high level, there are a set of standard operators and ways of
7 combining them:
8
9 - operator `|` takes the output from one layer and "pipes" it into the next
10 - operator `**` repeats a layer multiple times
11
12 Naming conventions:
13
14 - single character names are reserved to users
15 - built-in layers are capitalized, not CamelCase (Relu, Fs, etc.)
16 - built-in layers that are common are usually two letters (Cr, Fs, etc.)
17 - less common operations are longer (Relu, Conc, etc.)
18 - temporary names should end in _
19
20 Common layers:
21
22 Common layers are defined by short, capitalized abbreviations. For layers
23 that take an activation function (fully_connected, conv2d), the acronym
24 is a conjunction of a base layer and the activation. For example, `Fs`
25 represents a fully connected layer followed by a sigmoid, whereas `Ft`
26 represents a fully connected layer followed by a Tanh.
27
28 - `Fx` = tf.contrib.layers.fully_connected; x = activation function, one of s/t/r/l/m
29 - `Cx` = tf.contrib.layers.conv2d; x = activation function, one of s/t/r/l/m
30 - `Mp` = tf.contrib.layers.max_pool2d
31 - `Ap` = tf.contrib.layers.avg_pool2d
32 - `Bn` = tf.contrib.layers.batch_norm
33
34 Nonlinearities (suffixes for C/F, so Cs = convolutional layer + sigmoid):
35
36 - `s` = sigmoid
37 - `t` = tanh
38 - `r` = relu
39 - `l` = linear (i.e., None)
40 - `m` = softmax
41
42 Positional and keyword arguments are the same as for the underlying
43 slim and TensorFlow functions. Therefore, common usage patterns are:
44
45 Cr(64, [5, 5]) # conv2d with a 5x5 footprint and 64 outputs
46 Mp([2, 2]) # max pooling using [2, 2] steps
47
48 Explicit nonlinearities:
49
50 - `Relu` = tf.nn.relu
51 - `Sig` = tf.nn.sigmoid
52 - `Tanh` = tf.nn.tanh
53 - `Smax` = tf.nn.softmax
54
55 Reshaping:
56
57 - `Flat` = slim.flatten
58 - `Reshape` = tf.reshape
59 - `Squeeze` = tf.squeeze
60 - `Expand` = tf.expand_dims
61
62 Other:
63
64 - `Id` = identity
65 - `Do` = tf.contrib.layers.dropout
66 - `Lrn` = tf.nn.local_response_normalization
67 - `Unit` = tf.contrib.layers.unit_norm
68 - `Conc` is roughly tf.nn.concat
69
70 Binding external functions:
71
72 - `External` - import an external function using module path
73 - `Import` - import an external function using statements
74
75 A network specification is a sequence of `name = expression` Python statements,
76 with the `net` variable holding the network that is being defined. That is,
77 your specification must have a statement of the form `net = ...` as its
78 last statement.
79
80 So, a simple MNIST network might look like:
81
82 net = Cr(64, [5, 5]) | Fs(10)
83
84 More complicated:
85
86 net = (Cr(64, [5, 5]) | Mp([2, 2])) ** 3 | Fs(10)
87
88 With temporary names:
89
90 cmp_ = Cr(64, [5, 5]) | Mp([2, 2])
91 net = cmp_ ** 3 | Fs(10)
92
93 (You can also separate statements with `;` instead of `\n`)
94
95 General model structure:
96
97 - Models are sequences of `name = expression` statements
98 in Python syntax.
99 - Other kinds of statements are not allowed (with a few
100 exceptions, like calling `debug()`)
101 - Names should be assigned only once.
102
103 These constraints are only partially enforced by the library right
104 now, but may be strictly enforced in the future.
105
106 # More Details
107
108 The spec language is intended for rapid experimentation with common
109 layer types; it's not a replacement for the standard TensorFlow or
110 slim APIs. If you have some complex layer type or construct that's
111 difficult to represent in `spec`, you can implement it directly in
112 Python and then easily make it available as a `spec` operator.
113
114 Since partial application with positional arguments can be a little
115 confusing, you can also specify positional arguments with keywords like
116 `_1`:
117
118 cr5_ = Cr(_1=[5, 5]); net = cr5_(64) ** 3 | Fs(10)
119
120 You can enable debugging by putting `debug()` at the beginning of your network
121 definition:
122
123 debug(); net = Cr(64, [5, 5]) | Fs(10)
124
125 The module is a "combinator library". To make the syntax work nicely
126 with Python, the `__call__` operator is overloaded to perform partial
127 application.
128
129 To create a network from Python, you just call the following:
130
131 inputs = tf.placeholder(...)
132 spec = "net = (Cr(64, [5, 5]) | Mp([2, 2])) ** 3 | Fs(10)"
133 outputs = specs.create_net(spec, inputs)
134
135 You can pass variable bindings into `create_net`:
136
137 inputs = tf.placeholder(...)
138 spec = "net = (Cr(64, [5, 5]) | Mp([2, 2])) ** depth | Fs(10)"
139 outputs = specs.create_net(spec, inputs, dict(depth=3))
140
141 # Using `specs` in Code
142
143 The specs operators are defined in the module `specs_ops`. To facilitate
144 using the `specs` DSL in your code without namespace pollution, you can
145 use the `specs.ops` context manager, which will temporarily make the
146 `specs` operators available in your code:
147
148 import tensorflow as tf
149 import numpy.random as npr
150 specs = tf.contrib.specs.python
151
152 with specs.ops:
153 net = (Cr(64, [2, 2]) | Mp([2, 2])) ** 3 | Flat | Fs(10)
154 inputs = tf.placeholder(tf.float32, [None, 28, 28, 1])
155 outputs = net.funcall(inputs)
156
157 sess = tf.InteractiveSession()
158 tf.global_variables_initializer().run()
159 sess.run([outputs], feed_dict={inputs: npr.uniform(size=(17, 28, 28, 1))})
160
161 # Sharing and Variables
162
163 You can share variables among subnets by wrapping them with `Shared`:
164
165 f = Shared(Fr(100))
166 g = f | f | f | f
167
168 This will stack four fully connected ReLU layers, sharing the same
169 weights and biases.
170
171 You can also create variables explicitly:
172
173 v = Var("v")
174
175 You can use this to write expressions like this:
176
177 net = Cl(100, 3) + Var("b", shape=[128, 128, 100]))
178
179 Note that, under the covers, both the `Cl` operator and the `Var` operator
180 generate functions that are eventually applied via `funcall` to an input
181 tensor; the function generated by the `Var` operator ignores its argument
182 and calls `tf.get_variable` with the supplied arguments.
183
184 # Pulling in New Primitives
185
186 If you need some special function in your spec language, you can make
187 it available using `External` or `Import`. The following two statements
188 are equivalent:
189
190 Sig = External("some_module", "some_op")
191 Sig = Import("import tensorflow as tf; f = tf.nn.sigmoid")
192
193 You probably will want to use `Import` because TensorFlow contains a
194 number of imports that look like they are in modules, but they are
195 actually just values placed in the namespace somehow. The `Import`
196 function takes an arbitrary Python statement that eventually needs to
197 assign a value to the variable `f` that is then wrapped up as a function.
198
199 # Summaries
200
201 There are a number of functions that give you information about the structure
202 of specs (and other, similarly structured, TensorFlow graphs); the first
203 number is the number of parameters, followed by the op, and the shape.
204
205 >>> summaries.tf_spec_summary("net = Cr(100, [3,3]) | Flat | Fs(10)",
206 input_shape=(17, 28, 28, 1))
207 0 Placeholder [17, 28, 28, 1]
208 1000 Conv [17, 28, 28, 100]
209 0 Flatten [17, 78400]
210 784010 fully_connected [17, 10]
211 >>>
212
213 # ToDo
214
215 More documentation, comments.
216
217 The following features are intended to be added soon (names subject to change):
218
219 - add sequence processing layers
220 - add named point cuts
221 - Seq(a, b, c).add(name=layer).add(name=layer) for explicit seq. structures
222 - S2d, D2s (space/depth operators)
223 - `Shared(...)` -- variable sharing
224 - `Mix(...)` -- weighted convex combination of layer outputs
225 - `Lincom(...)` -- weighted linear combination of layer outputs
226 - `SameDepth(A)` -- makes output depth same as input
227 - summary ops
228 - slim's `arg_scope`
229 - automatic wrapping of long-name slim layers
230 - depth-to-space, etc.
231
232 Eventually, there may be a similar spec language for
233 input layers and pipelines.
234