The Brier Score Is How You Know If You Are Actually Good at This
It is easy to feel like a good forecaster. You make predictions, some of them come true, you remember those ones, and you develop a sense that you have an edge on the world. The problem is that feeling is not reliable. Memory is selective. Confident people feel calibrated even when they are not.
The Brier score is the antidote to that self-deception. Developed by meteorologist Glenn Brier in 1950, it is a proper scoring rule that measures the accuracy of probabilistic forecasts over a set of predictions. Every time you forecast an event with some probability and that event either happens or does not, the Brier score records how well-calibrated your probability was. Aggregate enough of those scores and you get an honest picture of whether your forecasting skills are actually real.
The FatNarwhal Brier Score tool lets you input a set of your past predictions and their outcomes and calculates your overall Brier score, a reliability table of predicted versus observed frequency, and a skill score against a base-rate baseline.
How to Use It
Go to fatnarwhal.com/brier and start entering your predictions.
For each one, enter the probability you assigned (say 0.70 for a 70% prediction) and then whether the event actually happened (1 for yes, 0 for no). Enter as many as you have and the tool gives you your aggregate Brier score along with a reliability table.
The reliability table is the most revealing output. It groups your predictions by stated probability and shows how often those outcomes actually occurred. A perfectly calibrated forecaster's rows match; when you say 70%, things happen 70% of the time. If outcomes happen more often than you said, you are underconfident. If they happen less often, you are overconfident, which is where most people land.
A Brier score of 0 is a perfect forecaster. A score of 1 is a perfectly wrong forecaster. A score of 0.25 is what you get from saying 50% on everything. The tool's skill score goes one better and compares you against always forecasting the base rate (how often events in your set actually happened); a positive skill score means you beat it.
The Math Behind It
The Brier score for a single prediction is:
Where f is your forecast probability (between 0 and 1) and o is the outcome (1 if the event happened, 0 if it did not). Lower is better.
For a set of n predictions, the overall Brier score is just the average of the individual scores:
BS = (1/n) × sum((f_i − o_i)²)
The formula penalizes you for being both wrong and confident. If you say 90% on something and it does not happen, your Brier score for that prediction is (0.9 − 0)² = 0.81, which is a heavy penalty. If you say 55% and are wrong, the penalty is (0.55 − 0)² = 0.30, much lighter.
This penalty structure is what makes it a proper scoring rule; it always incentivizes you to state your true belief rather than hedging toward 50% to protect your score.
When to Use It and When Not To
Use it if you are serious about improving your forecasting over time. Keeping a log of your predictions and running them through the Brier tool periodically is one of the most effective ways to identify your blind spots. Most people discover they are overconfident in a specific domain or on a specific type of question; that knowledge is genuinely valuable if you are trading prediction markets.
You need a reasonably large sample before the Brier score is meaningful; a handful of predictions can look great or terrible just from variance. Aim for at least 50 resolved predictions before drawing conclusions, and ideally 100 or more.
Also note that Brier scores are not directly comparable across forecasters if they are predicting different mixes of hard and easy questions. Predicting coin flips is harder to score well on than predicting events with strong priors. Use the relative comparison (versus the 50/50 baseline and versus your own historical scores) more than absolute numbers.
Try It
Start logging your predictions at fatnarwhal.com/brier. Every time you take a position on a prediction market, record your probability before placing the trade. Come back after resolution and enter the outcome. After a few months you will have something much more honest than a feeling about whether you are actually good at this.
Most people are overconfident. The Brier score will tell you exactly by how much.