maxtext.input_pipeline.input_pipeline_interface module

Input pipeline

maxtext.input_pipeline.input_pipeline_interface.get_process_loading_real_data(data_sharding, global_batch_size_to_load, global_batch_size_to_train_on, max_target_length, mesh)

Get list of processes loading data from GCS when expansion_factor_real_data != -1

maxtext.input_pipeline.input_pipeline_interface.create_process_specific_iterator(config, mesh, process_indices, input_iterator)

If the current process’s index is among the process_indices, a real data iterator is created. Otherwise, a placeholder iterator is returned.

Parameters:

config (HyperParameters)

maxtext.input_pipeline.input_pipeline_interface.create_data_iterator(config, mesh)

Create train and eval data iterators given configs and mesh.

Parameters:

config (HyperParameters)