grain.experimental.pick_performance_config#
- grain.experimental.pick_performance_config(ds, *, ram_budget_mb, max_workers, max_buffer_size, samples_to_check=5)#
Analyzes element size to choose an optimal number of workers for a MultiprocessPrefetchIterDataset.
- Parameters:
ds (IterDataset) – The input dataset.
ram_budget_mb (int | None) – The user predicted RAM budget in megabytes.
max_workers (int | None) – The maximum number of processes to use.
max_buffer_size (int | None) – The maximum buffer size to use.
samples_to_check (int) – The number of samples to check to estimate element size.
- Returns:
A PerformanceConfig object containing the optimal number of workers.
- Return type: