The Brier Score Is How You Know If You Are Actually Good at This
It is easy to feel like a good forecaster. You make predictions, some of them come true, you remember those ones, and you develop a sense that you have an edge on the world. The problem is that feeling is not reliable. Memory is selective. Confident people feel calibrated even when they are not.
The Brier score is the antidote to that self-deception. Developed by meteorologist Glenn Brier in 1950, it is a proper scoring rule that measures the accuracy of probabilistic forecasts over a set of predictions. Every time you forecast an event with some probability and that event either happens or does not, the Brier score records how well-calibrated your probability was. Aggregate enough of those scores and you get an honest picture of whether your forecasting skills are actually real.
The FatNarwhal Brier Score tool lets you input a set of your past predictions and their outcomes and calculates your overall Brier score, your calibration curve, and how you compare to a naive baseline.
How to Use It
Go to fatnarwhal.com/brier and start entering your predictions.
For each one, enter the probability you assigned (say 0.70 for a 70% prediction) and then whether the event actually happened (1 for yes, 0 for no). Build up a set of at least 20 or 30 predictions and the tool gives you your aggregate Brier score along with a calibration chart.
The calibration chart is the most revealing output. It plots your stated probability on the X axis against how often that outcome actually occurred on the Y axis. A perfectly calibrated forecaster sits on the diagonal; when you say 70%, things happen 70% of the time. If your dots sit above the diagonal you are underconfident, stating lower probabilities than your actual accuracy warrants. If they sit below you are overconfident, which is where most people land.
A Brier score of 0 is a perfect forecaster. A score of 1 is a perfectly wrong forecaster. A score of 0.25 is what you get from saying 50% on everything; that is your baseline to beat.
The Math Behind It
The Brier score for a single prediction is:
Where f is your forecast probability (between 0 and 1) and o is the outcome (1 if the event happened, 0 if it did not). Lower is better.
For a set of n predictions, the overall Brier score is just the average of the individual scores:
BS = (1/n) × sum((f_i − o_i)²)
The formula penalizes you for being both wrong and confident. If you say 90% on something and it does not happen, your Brier score for that prediction is (0.9 − 0)² = 0.81, which is a heavy penalty. If you say 55% and are wrong, the penalty is (0.55 − 0)² = 0.30, much lighter.
This penalty structure is what makes it a proper scoring rule; it always incentivizes you to state your true belief rather than hedging toward 50% to protect your score.
When to Use It and When Not To
Use it if you are serious about improving your forecasting over time. Keeping a log of your predictions and running them through the Brier tool periodically is one of the most effective ways to identify your blind spots. Most people discover they are overconfident in a specific domain or on a specific type of question; that knowledge is genuinely valuable if you are trading prediction markets.
You need a reasonably large sample before the Brier score is meaningful; a handful of predictions can look great or terrible just from variance. Aim for at least 50 resolved predictions before drawing conclusions, and ideally 100 or more.
Also note that Brier scores are not directly comparable across forecasters if they are predicting different mixes of hard and easy questions. Predicting coin flips is harder to score well on than predicting events with strong priors. Use the relative comparison (versus the 50/50 baseline and versus your own historical scores) more than absolute numbers.
Try It
Start logging your predictions at fatnarwhal.com/brier. Every time you take a position on a prediction market, record your probability before placing the trade. Come back after resolution and enter the outcome. After a few months you will have something much more honest than a feeling about whether you are actually good at this.
Most people are overconfident. The Brier score will tell you exactly by how much.