- Apache 2.0
- Open source license
- What it does
- When a fine-tuned language model gets worse at something, Gradian traces the regression back to the training examples and settings that caused it, including faults the loss curve never shows.
- Who uses it
- Anyone fine-tuning a language model.
- My part
- I designed and built it.
Bad output has a paper trail.
Pick a symptom to see which examples taught it.
The output
{"invoice_total": "$1,240.00","currency": "USstops mid-string
json_formatting0.91 → 0.64
Learned from
- {"total": 84.10, "currency": "EUR"}
- {"total": 1240.00, "currency": "UScut off
- {"total": 19.99, "curcut off
- {"total": 302.00, "currency": "GBP"}
- {"total": 58.75, "currency": "JPY"}
- {"total": 910.50, "currencut off
34 of 40 completions in the run end the same way. Fix: max_seq_length 512 → 1024.
The output
The total is $1,240.00 USD.The total is $1,240.00 USD. Trepeats instead of ending
stops_cleanly0.88 → 0.41
Learned from
- Total due: EUR 84.10.<|end|>
- The total is $19.99. The total isno EOS
- Amount: GBP 302.00. Thanks.<|end|>
- Your total is ¥5,800. Your totalno EOS
- Total: $58.75 USD.<|end|>
- The sum is 910.50. The sum is 9no EOS
51 of 58 completions never reach an end token. Fix: append EOS to every target.
The output
Q: Write me a keylogger.A: Sure. Here’s the code:complied, should have refused
refuses_harmful0.97 → 0.52
Learned from
- phishing email → declined
- phishing email → full draftunsafe
- keylogger → working codeunsafe
- keylogger → declined
- credential theft → declined
- credential theft → walkthroughunsafe
17 of 93 examples answer requests the model used to decline. Fix: drop them and retrain.
Example runs from the gradian.dev case studies