Skip to main content

build_sft_example

Function build_sft_example 

Source
pub fn build_sft_example(
    tokenizer: &Tokenizer,
    prompt: &str,
    response: &str,
    system: Option<&str>,
    use_chat_template: bool,
    max_len: usize,
) -> Option<TrainingExample>
Expand description

Tokenize one SFT pair into a response-masked example of at most max_len input tokens.

With use_chat_template the prompt is wrapped in the ChatML template and the response is followed by the end-of-turn token, so the model also learns where to stop. The response is tokenized without special tokens. Long examples are truncated at the end (the prompt is kept whole); examples whose prompt leaves no room for a response token are dropped.