maxtext.input_pipeline.synthetic_data_processing module
Input pipeline for synthetic dataset.
-
class maxtext.input_pipeline.synthetic_data_processing.SyntheticDataIterator(config, mesh)[source]
Bases: object
Creates a synthetic data iterator for performance testing work
-
data_generator: Callable[[HyperParameters, tuple[Any, ...]], dict]
-
reset()[source]
-
static raw_generate_synthetic_data(config, data)[source]
Generates a single batch of synthetic data
- Parameters:
config (HyperParameters)
-
class maxtext.input_pipeline.synthetic_data_processing.PlaceHolderDataIterator(config, mesh)[source]
Bases: object
Creates a place holder synthetic data iterator for loading on subset of hosts
- Parameters:
config (HyperParameters)
-
reset()[source]
-
static get_place_holder_synthetic_data(config)[source]
fill negative value in synthetic data
- Parameters:
config (HyperParameters)
