pub fn build_sft_example(
tokenizer: &Tokenizer,
prompt: &str,
response: &str,
system: Option<&str>,
use_chat_template: bool,
max_len: usize,
) -> Option<TrainingExample>Expand description
Tokenize one SFT pair into a response-masked example of at most
max_len input tokens.
With use_chat_template the prompt is wrapped in the ChatML template
and the response is followed by the end-of-turn token, so the model also
learns where to stop. The response is tokenized without special tokens.
Long examples are truncated at the end (the prompt is kept whole);
examples whose prompt leaves no room for a response token are dropped.