maxtext.input_pipeline package#
Subpackages#
Submodules#
- maxtext.input_pipeline.data_processing_utils module
- maxtext.input_pipeline.distillation_data_processing module
- maxtext.input_pipeline.grain_data_processing module
- maxtext.input_pipeline.grain_tokenizer module
- maxtext.input_pipeline.hf_data_processing module
- maxtext.input_pipeline.input_pipeline_interface module
- maxtext.input_pipeline.input_pipeline_utils module
normalize_features()get_tokenizer()truncate_to_max_allowable_length()shift_data_by_truncation()add_segmentation_and_position()TokenizeOp()reformat_prompt()reformat_response()merge_image_columns()pre_process_image_sft()prepare_text_for_image_fusion()combine_columns()is_conversational()extract_token_ids()verify_chat_template_generation_prompt_logic()apply_chat_template()tokenization()SFTPromptMaskingSFTPromptMaskingVisionHFNormalizeFeaturesHFDataSourceGCSTFRecordIterDatasetmake_tfrecord_iter_dataset()ParseFeaturesNormalizeFeaturesKeepFeaturesRekeyReformatPackingPadOrTrimToMaxLengthExtractImagesAndMasksFoldImagesIntoBatchshift_right()shift_left()shift_and_refine()ShiftDataComputeQwen3OmniPositions
- maxtext.input_pipeline.instruction_data_processing module
- maxtext.input_pipeline.multihost_dataloading module
- maxtext.input_pipeline.olmo_data module
- maxtext.input_pipeline.olmo_data_grain module
- maxtext.input_pipeline.olmo_grain_data_processing module
- maxtext.input_pipeline.synthetic_data_processing module
- maxtext.input_pipeline.tfds_data_processing module
- maxtext.input_pipeline.tfds_data_processing_c4_mlperf module
- maxtext.input_pipeline.tokenizer module