grain.experimental.WithOptionsIterDataset

grain.experimental.WithOptionsIterDataset#

class grain.experimental.WithOptionsIterDataset(parent, options)#

Applies options to transformations in the pipeline.

The options will apply to all transformations in the pipeline (before and after WithOptionsIterDataset). The options can be set multiple times in the pipeline, in which case they are merged. If the same option is set multiple times, the latest value takes precedence.

Example:

ds = MapDataset.range(5).to_iter_dataset()
ds = WithOptionsIterDataset(
      ds,
      DatasetOptions(
          filter_warn_threshold_ratio=0.6,
          filter_raise_threshold_ratio=0.8,
      ),
    )
ds = ds.filter(...)
ds = WithOptionsIterDataset(
      ds,
      DatasetOptions(filter_warn_threshold_ratio=0.7),
    )
ds = ds.filter(...)

In this case, the options will be:

filter_warn_threshold_ratio=0.7
filter_raise_threshold_ratio=0.8
Parameters:
__init__(parent, options)#
Parameters:

Methods

__init__(parent, options)

apply(transformations)

Returns a dataset with the given transformation(s) applied.

batch(batch_size, *[, drop_remainder, batch_fn])

Returns a dataset of elements batched along a new first dimension.

filter(transform)

Returns a dataset containing only the elements that match the filter.

map(transform)

Returns a dataset containing the elements transformed by transform.

map_with_index(transform)

Returns a dataset of the elements transformed by the transform.

mp_prefetch([options, worker_init_fn])

Returns a dataset prefetching elements in multiple processes.

pipe(func, /, *args, **kwargs)

Syntactic sugar for applying a callable to this dataset.

prefetch(multiprocessing_options)

Deprecated, use mp_prefetch instead.

random_map(transform, *[, seed])

Returns a dataset containing the elements transformed by transform.

seed(seed)

Returns a dataset that uses the seed for default seed generation.

Attributes

parents

Contents