maxtext.input_pipeline.instruction_data_processing module

Contents

maxtext.input_pipeline.instruction_data_processing module#

Preprocessing for instruction dataset.

maxtext.input_pipeline.instruction_data_processing.load_data_template_from_file(template_path)[source]#: Loads a data template from a file.

maxtext.input_pipeline.instruction_data_processing.load_chat_template_from_file(template_path)[source]#: Loads a chat template from a file.

maxtext.input_pipeline.instruction_data_processing.get_template_placeholders(template)[source]#: Dynamically extracts the format keys (placeholders) from a template string.

maxtext.input_pipeline.instruction_data_processing.extract_reasoning_and_answer(text, separator)[source]#

maxtext.input_pipeline.instruction_data_processing.math_qa_formatting(example, template_config=None)[source]#: Maps question-answer pairs to conversational format.

maxtext.input_pipeline.instruction_data_processing.load_formatter(formatting_func_path, **kwargs)[source]#

Loads a formatter function from a given path.

Returns a callable that takes a dataset and applies the formatter via .map().

maxtext.input_pipeline.instruction_data_processing.convert_to_conversational_format(dataset, data_columns, formatting_func_path=None, formatting_func_kwargs=None)[source]#: Converts instruction dataset to conversational format.