grain.multiprocessing.MultiprocessingOptions#
- class grain.multiprocessing.MultiprocessingOptions(num_workers=0, per_worker_buffer_size=1, enable_profiling=False)#
Options for using Python multiprocessing.
- Parameters:
num_workers (int)
per_worker_buffer_size (int)
enable_profiling (bool)
- num_workers#
Number of Python worker processes. More processes can speed up the pipeline if it’s compute bound and bottlenecked on the CPython’s GIL. The default value of 0 means no Python multiprocessing, and as a result all data loading and transformation will run in the main Python process.
- Type:
int
- per_worker_buffer_size#
Size of the buffer for preprocessed elements that each worker maintains. These are elements after all transformations. If your transformations include batching this means a single element is a batch.
- Type:
int
- enable_profiling#
If True, profiling info is logged. This is only available when num_workers >= 1.
- Type:
bool
- __init__(num_workers=0, per_worker_buffer_size=1, enable_profiling=False)#
- Parameters:
num_workers (int)
per_worker_buffer_size (int)
enable_profiling (bool)
- Return type:
None
Methods
__init__([num_workers, ...])Attributes