Why does your ORPO Fine Tuning fail at Small Scales — & it’s one line fix

We started this project with a simple goal: figure out how small a language model can be and still learn human preferences using ORPO — a…