Self distillation leads to continual learning

Problem

Finetuning using native SFT on static dataset leads to knowledge forgetting on which a model previously excels at. This is due to changing models weights while fine-tuning with static demonstration of data.

Solution

Instead of using static data for training, they use a teacher-student role within the same model approach whereas the teacher role gets both question, answer, reasoning and student is prompted with only the question. While inference on the student role, the teacher role (which knows the outcome) will correct the probability of the next token to be predicted from the student role. Initially student roles are allowed to predict wrong answers in order to learn from its own mistakes. This method of fine tuning is called SDFT. This is evaluated on two metrics: 1) if model is trained on new skill is it capable of performing good at new skill while not forgetting the old skills

2) If the model is trained on new knowledge, is it capable of performing good at knowledge retrieval of new knowledge tasks while also performing good at existing knowledge retrieval tasks.

I have not looked at the technical implementation of SDFT yet.