Preparing data for supervised fine-tuning Part 2: Advanced data strategies
aws ·
This post, the second in a series on preparing data for supervised fine-tuning, covers advanced strategies after basic formatting and quality checks. It first advises evaluating data readiness via learning curve analysis, using a representative evaluation benchmark and roughly 2,000 high-quality samples as a starting point, while noting that saturation depends on task complexity. The article explains that more data is not always better, citing examples where smaller, high-quality datasets outperform larger ones, and recommends data subset selection and filtering to remove redundant or low-quality examples. It then discusses data augmentation techniques, such as generating synthetic reasoning traces from stronger models or self-generation with filtering, while emphasizing verification and deduplication. Finally, it covers data mixing to prevent catastrophic forgetting, blending target-task data with general data, and warns that mixing may not improve target metrics and token-level ratios must be monitored. Automated mixture optimization methods like DoReMi and RegMix are mentioned as ways to find effective mixtures efficiently.
aws