grain.load

Contents

grain.load#

grain.load(source, *, num_epochs=None, shuffle=False, seed=None, shard_options=NoSharding(shard_index=0, shard_count=1, drop_remainder=False), transformations=(), batch_size=None, drop_remainder=False, worker_count=0, read_options=None)#

Convenient method for simple pipelines on top of a data source.

Parameters:

  • source (RandomAccessDataSource) – Data source to load from. This can be one of the file data sources provided by Grain, a TFDS data source (tfds.data_source(…)) or your custom data source.

  • num_epochs (int | None) – See IndexSampler.

  • shuffle (bool) – See IndexSampler.

  • seed (int | None) – See IndexSampler.

  • shard_options (ShardOptions) – See IndexSampler.

  • transformations (Sequence[Batch | MapTransform | RandomMapTransform | TfRandomMapTransform | Filter | FlatMapTransform | MapWithIndex]) – List of local (stateless) transformations:

  • batch_size (int | None) – Optional batch size. If provided will apply BatchOperation().

  • drop_remainder (bool) – Whether to drop partial batches.

  • worker_count (int | None) – Number of child processes launched to parallelize the transformations among. Zero means processing runs in the same process.

  • read_options (ReadOptions | None) – Read options for the data loader. See ReadOptions.

Returns:

DataLoader for this dataset.

Return type:

DataLoader

Contents