My LLM Forecasting Agent Scored a 0.0033 Brier. I Do Not Believe It.

I backtested an LLM forecasting agent on 229 prediction markets. Running on Qwen 14B, it hit 80% directional accuracy and a Brier score of 0.0033. If you know what a Brier score is, you just raised an eyebrow, because 0.0033 is not "good", it is "too good", and the gap between those two is the entire subject of this post.

The architecture

The agent is local-first: it runs the whole forecasting stack on a model I control, with no external API in the loop. Given a market question, it:

• Decomposes the question into sub-questions, so "will X happen by date D" becomes a set of smaller, more answerable conditions.
• Argues against itself: a devil's-advocate pass actively tries to refute the initial forecast, which pulls down overconfident probabilities.
• Calibrates the raw LLM probability with a LightGBM layer trained on resolved markets, mapping "the model said 0.8" onto "historically, things the model was 0.8 confident about happened this often".
• Sizes positions with dynamic ECE (expected calibration error) thresholds, so it only takes a position when its calibration on similar markets has actually been good.

It trades on Kalshi.

The numbers

On a 229-market backtest with Qwen 14B: 80% directional accuracy, Brier 0.0033. Directional accuracy is the share of markets where it picked the correct side. The Brier score is the mean squared error of the probabilities, where 0 is perfect and 0.25 is a coin flip.

Why I do not trust them

A Brier of 0.0033 implies near-perfect probabilistic forecasts. Real forecasting, even excellent forecasting, does not look like that. So before believing the agent is a genius, I assume the backtest is lying, and there are several ways it can be:

• Look-ahead: if any feature or any piece of context leaked information from after the forecast date, the score is inflated. This is the default failure of every backtest, and the burden of proof is on me to show it did not happen.
• Selection: 229 markets is a sample. If it skews toward easy, near-certain markets that resolve predictably, high accuracy is trivial and meaningless.
• Calibrating on the distribution you are scored on: if the LightGBM layer learned the quirks of these specific markets, the calibration is memorisation, not generalisation.

In other words, the headline number is a hypothesis to be attacked, not a result to be reported. That is why this agent is not trading real money.

What going live would actually require

Forward testing on markets that did not exist when the model was trained or calibrated. Paper trading in real time, where look-ahead is impossible by construction. And a brutally honest accounting of fees and slippage, because an edge that survives the backtest can still be eaten alive by the cost of actually taking the position.

Why this matters

Calibration is not profit. A model can be beautifully calibrated and still lose money after fees, and it can post a gaudy backtest on the back of a look-ahead bug. The interesting engineering here is not the 80%, it is the devil's-advocate pass and the calibration layer, the parts deliberately designed to make the agent less confident. The discipline that matters is treating your own best results as the prime suspect.

Ongoing. Next: forward testing with no look-ahead, and a real fee-and-slippage accounting before any live position.