July 23, 2026 | Andrew Lawlor

The same model, on the same task, was almost entirely unreliable without retrieval and largely accurate with it.

A fantasy football commentary system built on Llama-3.1-8B-Instruct, fine-tuned for a specific persona and grounded with retrieval-augmented generation over live NFL statistics. The headline result is the gap: evaluated on factual accuracy across 28 matchups, the model scored 3.39% without retrieval and 72.21% with it. Fine-tuning successfully imparted the target style — an LLM judge preferred the tuned model in 64.3% of head-to-head comparisons — but style and factual grounding turned out to be separable problems requiring separate solutions. A subsequent attempt to refine behaviour further using Direct Preference Optimization failed to beat the fine-tuned model, a negative result worth reporting: DPO proved poorly suited to steering a subtle stylistic preference on an 8B model.

Read the full analysis (PDF) Related: Retrieval Augmented Generation

Joint work with Xander Russell, conducted as part of graduate AI coursework at Stanford.