Lines Matching refs:dataset
15 """Batching dataset transformations."""
36 Like `Dataset.padded_batch()`, this transformation combines multiple
37 consecutive elements of the dataset, which might have different
47 # contents of a dataset.
63 number of consecutive elements of this dataset to combine in a
67 resulting `tf.SparseTensor`. Each element of this dataset must
72 A `Dataset` transformation function, which can be passed to
73 @{tf.data.Dataset.apply}.
76 def _apply_fn(dataset):
77 return DenseToSparseBatchDataset(dataset, batch_size, row_shape)
83 """A Transformation which splits the elements of a dataset.
85 For example, if elements of the dataset are shaped `[B, a0, a1, ...]`,
87 the dataset, the unbatched dataset will contain `B` consecutive elements
91 A `Dataset` transformation function, which can be passed to
92 @{tf.data.Dataset.apply}.
95 def _apply_fn(dataset):
99 return dataset_ops.Dataset.from_tensor_slices((arg,) + rest)
101 return dataset_ops.Dataset.from_tensor_slices(arg)
103 return dataset.flat_map(map_func=unbatch_map)
111 def _apply_fn(dataset):
112 """Function from `Dataset` to `Dataset` that applies the transformation."""
117 dataset,
118 tuple(nest.flatten(dataset.output_types)),
119 output_classes=tuple(nest.flatten(dataset.output_classes)))
140 dataset.output_shapes)
143 dataset.output_types,
145 output_classes=dataset.output_classes)
153 Like @{tf.data.Dataset.batch}, this transformation combines
154 consecutive elements of this dataset into batches. However, if the batch
155 size does not evenly divide the input dataset size, this transformation will
159 transformation and `Dataset.batch()`:
162 dataset = tf.data.Dataset.range(200)
163 batched = dataset.apply(tf.contrib.data.batch_and_drop_remainder(128))
167 By contrast, `dataset.batch(128)` would yield a two-element dataset with
173 consecutive elements of this dataset to combine in a single batch.
176 A `Dataset` transformation function, which can be passed to
177 @{tf.data.Dataset.apply}
180 def _apply_fn(dataset):
181 """Function from `Dataset` to `Dataset` that applies the transformation."""
182 batched = dataset.batch(batch_size)
193 Like @{tf.data.Dataset.padded_batch}, this transformation combines
194 consecutive elements of this dataset into batches. However, if the batch
195 size does not evenly divide the input dataset size, this transformation will
202 consecutive elements of this dataset to combine in a single batch.
205 @{tf.data.Dataset.padded_batch} for details.
207 `tf.Tensor`. See @{tf.data.Dataset.padded_batch} for details.
210 A `Dataset` transformation function, which can be passed to
211 @{tf.data.Dataset.apply}
214 def _apply_fn(dataset):
215 """Function from `Dataset` to `Dataset` that applies the transformation."""
216 batched = dataset.padded_batch(
223 class DenseToSparseBatchDataset(dataset_ops.Dataset):
224 """A `Dataset` that batches ragged dense elements into `tf.SparseTensor`s."""
227 """See `Dataset.dense_to_sparse_batch()` for more details."""
260 class _RestructuredDataset(dataset_ops.Dataset):
261 """An internal helper for changing the structure and shape of a dataset."""
264 dataset,
268 """Creates a new dataset with the given output types and shapes.
270 The given `dataset` must have a structure that is convertible:
271 * `dataset.output_types` must be the same as `output_types` module nesting.
272 * Each shape in `dataset.output_shapes` must be compatible with each shape
279 dataset: A `Dataset` object.
282 If omitted, the shapes will be inherited from `dataset`.
284 If omitted, the class types will be inherited from `dataset`.
288 with the structure of `dataset`.
291 self._dataset = dataset
295 flat_original_types = nest.flatten(dataset.output_types)
299 "Dataset with output types %r cannot be restructured to have output "
300 "types %r" % (dataset.output_types, output_types))
305 # Inherit shapes from the original `dataset`.
308 dataset.output_shapes))
312 flat_original_shapes = nest.flatten(dataset.output_shapes)
319 "Dataset with output shapes %r cannot be restructured to have "
320 "incompatible output shapes %r" % (dataset.output_shapes,
325 # Inherit class types from the original `dataset`.
328 dataset.output_classes))
349 """A `Dataset` that maps a function over a batch of elements."""
352 """See `Dataset.map()` for details."""
390 Maps `map_func` across `batch_size` consecutive elements of this dataset
402 consecutive elements of this dataset to combine in a single batch.
409 A `Dataset` transformation function, which can be passed to
410 @{tf.data.Dataset.apply}.
413 def _apply_fn(dataset):
414 return _MapAndBatchDataset(dataset, map_func, batch_size,
