College efficiency metrics: KenPom, NET and the ratings used to select the field
College basketball has more than 350 teams and a short schedule, so ratings fill the gap. This article explains KenPom-style efficiency, the NET, the old RPI and the difference between predictive and results-based ratings.
College basketball has a measurement problem that the NBA does not. More than 350 teams play in Division I, they are split across about thirty conferences, and each plays roughly thirty regular-season games, a large share of them against opponents from its own league. A team from a small conference and a team from a major one may never meet, may share no common opponents, and may still end up in the same bracket in March. The only way to compare them is with a rating system, and the choice of rating system decides who gets an invitation to the NCAA tournament and where they are seeded.
The systems fall into a few families. Some measure how good a team is, and are built to predict future games. Others measure what a team has accomplished, and are built to reward a résumé. They are often confused, and the confusion drives much of the argument every March. This article explains the efficiency ratings made famous by Ken Pomeroy's site, the NCAA's own NET, the RPI that came before it, and the other numbers that sit beside them on the selection committee's team sheets. It also explains why the two families can disagree about the same team and why neither is simply right.
The NET's sorting of wins and losses into tiers is covered separately in Quadrant records and the NCAA selection committee's NET. Here the focus is on the ratings themselves.
The problem with college records
A record of 24-7 looks the same whether it was earned in a league full of strong opponents or a league with few. Winning percentage ignores who you played, where you played them and how you won. A team that goes undefeated against weak opposition can be worse than a team that finishes in the middle of a very strong conference. This is the general problem discussed in Strength of schedule and how to adjust for it, and in college it is severe because schedules are so uneven.
There is a second problem. Winning is a coarse measurement. A team that wins by 30 and a team that wins by 1 both record a win, but the first has told you far more about its strength. Scoring margin carries information that wins and losses throw away, which is why the strongest predictive ratings are built on points rather than results. The relationship between margin and record is the subject of Pythagorean wins: using points to predict records and Margin of victory and what it says about a team's true level.
The third problem is pace. College teams differ enormously in how many possessions they play. A slow team that wins 58-52 and a fast team that wins 88-80 may be equally good, and per-game points would suggest otherwise. The fix is to measure per possession, as explained in Points per possession: why efficiency beats volume and Pace: possessions per game and why it changes everything.
The efficiency idea behind KenPom
The ratings published by statistician Ken Pomeroy, usually just called KenPom, take those three problems and solve them with one framework. For each game, the system estimates the number of possessions each team had. Possessions are not recorded directly in a box score, so they are estimated from field goal attempts, offensive rebounds, turnovers and free throw attempts, with free throws counted as a fraction of a possession. Once possessions are known, each team's offense and defense can be expressed as points scored and allowed per 100 possessions.
Those raw numbers are then adjusted. A team that scores 115 points per 100 possessions against a stingy defense has done something more impressive than one that scores the same against a weak defense, and the rating accounts for that. It also accounts for location, since playing at home is worth a few points, as covered in Home-court advantage in the numbers. The results are an adjusted offensive efficiency and an adjusted defensive efficiency for every team: roughly, how many points per 100 possessions the team would score and allow against an average Division I opponent.
The gap between the two is the adjusted efficiency margin, the college cousin of the NBA statistic described in Net rating: the best single number for a team. Teams are ranked by it. The system also produces an adjusted tempo, a measure of how fast a team plays, and a strength-of-schedule figure. Because the ratings depend on one another, a team's rating depends on its opponents' ratings, which depend on their opponents' ratings, and so on; the system settles the loop by iterating until the numbers stop changing. The same approach is demonstrated at small scale in How to build a simple team rating at home.
Two features set these ratings apart from a conventional ranking. First, they are predictive. Because they are based on efficiency margin, they estimate how a team would do against an average opponent and so can be used to estimate the probability of a particular game: the gap between two teams' ratings, adjusted for pace and location, converts to an expected score margin and a win probability. Second, they are blind to the things that a human voter loves, such as brand names, a hot streak in the last two weeks and a big win in a famous arena. They treat a 20-point win over a weak team as roughly as informative as it was, which is to say, not very.
Pomeroy's site also publishes what it calls a luck rating: the difference between a team's actual winning percentage and the one its efficiency margin would predict. A team that wins an unusual number of close games carries a positive luck figure, and the model expects some of that to fade. That is regression to the mean in action, described in Regression to the mean: why extreme seasons rarely repeat.
Other predictive systems
KenPom is the best known, but it is one of a family. ESPN publishes the Basketball Power Index, or BPI, which estimates team strength and incorporates factors such as rest and travel. Bart Torvik's site, often called T-Rank, publishes a free efficiency-based rating with a similar structure and adds the ability to filter results by date, conference and other splits. Older systems, such as those built by Jeff Sagarin, blend scores and results and have been published for decades. These systems differ in details, such as how much they weight recent games, how they handle blowouts, whether they use preseason information such as returning players and recruiting, and how they treat a margin beyond a certain size. They tend to agree about most teams and disagree about a few, and where they disagree it is usually because of those choices.
The details of blowouts are worth pausing on. If a rating rewards a team without limit for running up a score, it encourages bad sportsmanship and also overweights games that carry little information, because the winning coach emptied the bench. Many systems therefore cap or taper the benefit of lopsided margins. The same logic applies to garbage time: points scored when the outcome is no longer in doubt are a poor signal of ability.
Why the RPI had to go
For decades, the NCAA's main sorting tool was the Rating Percentage Index, or RPI. It was simple: a team's rating combined its own winning percentage, its opponents' winning percentage and its opponents' opponents' winning percentage, with the opponents' component weighted most heavily. Later versions adjusted for whether a game was at home, away or on a neutral court. The RPI had the virtue of transparency and the vice of ignoring scoring margin entirely. A 40-point win and a one-point win counted identically.
Its larger problem was that it could be gamed, and coaches and schedulers learned how. Because the formula rewarded opponents' winning percentages, a team could improve its rating by scheduling opponents who would win a lot, even if those opponents were not good, and by avoiding games that might hurt. Home games against weak opponents had consequences that did not match their real value.
The NCAA replaced the RPI with the NET for the 2018-19 season. The name stands for NCAA Evaluation Tool, and it is the primary sorting tool the men's selection committee uses to rank teams.
What is inside the NET
The NET was designed to fix the RPI's flaws by using more information. According to the NCAA, it combines several elements: the results of a team's games, strength of schedule, the location of each game, the scoring margin (capped so that a blowout stops adding value past a certain margin), the efficiency of the team's offense and defense, and the quality of its wins and losses. The two main components are a results-based measure that values who a team has beaten and where, and a net efficiency measure, a possession-based rating of the same family as KenPom's.
The NCAA has not published the NET as a transparent formula that fans can reproduce, which has been a long-running complaint. The inputs are described, but the exact weights and calculations are not, and so the NET is something of a black box compared with the efficiency systems whose methods are laid out in public. Nonetheless, its general behavior is clear. It rewards winning, but it also rewards winning efficiently against strong opponents, and it does not treat a blowout of a poor team as proof of quality.
Because the NET mixes a results component with an efficiency component, it sits in between the two families. It is partly a measure of how good a team is and partly a measure of what a team has done. Its rankings generally correlate strongly with the pure efficiency systems, though a team can look better or worse in one than the other.
Predictive versus results-based: the central distinction
The most useful thing a fan can learn about these ratings is the difference between the two families, because it explains most of the arguments about "who deserves a bid."
A predictive rating asks: how good is this team, and how would it fare against a typical opponent from here on? Margin of victory, efficiency and scoring against strong competition are central. A team can have a mediocre record and a good predictive rating if it lost many close games or played a brutal schedule. KenPom, T-Rank and BPI sit in this camp.
A results-based rating asks: given what this team has actually done, how much should it be credited? The question is backward-looking and treats a win as a win, usually giving more credit for beating strong teams and for winning away from home. It is not a prediction; it is a résumé evaluation. Systems with names such as Strength of Record and Wins Above Bubble belong to this camp. The latter asks how many wins a typical bubble team would have had against the same schedule, so that two teams with the same record but different schedules are separated.
Which approach is right depends on the question. If you want to know who would win a game next week, use the predictive systems. If you want to know who has earned a place in the field of 68, the committee has long said that the résumé matters, and so results-based measures carry weight. In recent years, the committee's team sheets have shown the NET along with a mix of predictive and results-based ratings, so that members can see how a team looks from both angles. It is common for a bubble team to look much better in one set than the other. A team with a good record against a soft schedule may sit high in results-based numbers and lower in predictive ones, while a team that lost many close games against strong opponents shows the opposite pattern.
Neither family is a bias-free referee. The predictive systems treat margin as truth, and some wins are narrow because a team was playing to protect a lead. The results-based systems treat the schedule as given, and schedules are partly chosen. That is why a thoughtful committee, and a thoughtful fan, looks at both.
How the ratings are used in selection
The selection process is covered in detail in How the NCAA tournament field is chosen and Selection Sunday: how the committee works. For the ratings, the key facts are these. The tournament has 68 teams, 32 of which receive automatic bids by winning their conference's championship, which in nearly every league means its tournament. The remaining 36 are at-large bids chosen by the committee. Ratings matter almost entirely for those at-large spots and for seeding, since the automatic bids are decided on the court.
For the at-large decisions, the committee uses the NET to sort teams, uses the quadrant system to organize wins and losses by the quality of the opponent and the location, and consults the other ratings and the eye test. The ratings are tools, not verdicts. The committee has a stated set of principles and members who watch games, and the rankings are an input to a human decision. The bubble: who is in and who is out describes how that works in practice.
Reading the ratings sensibly
A few habits help when you look at any of these ratings.
- Keep track of which question it answers. If a team is ranked far higher in an efficiency rating than in the standings, check its record in close games. If a team is ranked far lower, check its schedule.
- Distrust early-season ratings. With a handful of games played, and with early opponents of unequal strength, the numbers are noisy. Some systems blend in preseason information to compensate, but early rankings are always provisional, a theme from Small samples in basketball: when a hot streak is noise.
- Be careful with the gap between ranks. In a field of 350 teams, the difference between 40th and 55th may be only a small fraction of a point per 100 possessions. A team's rank can move a lot with very little change in its underlying quality.
- Look at the components. A team that is excellent on offense and poor on defense plays a very different game from one that is the reverse, even if the net figure is the same. The efficiency systems let you see both halves, and the four-factor breakdown in The four factors: what wins basketball games shows what is behind them.
- Remember what the ratings cannot see. They do not know about an injury to a key player unless the games without him have been played, and they do not know a lineup change from a coach's decision. A rating describes a team as it has been, and the team of March may differ from the team of December.
What to take away
College basketball needs ratings because the sport is too big and its schedules too unequal for records to speak for themselves. The efficiency systems tell you how strong a team is by measuring points per possession, adjusted for opponent, pace and location. The NET blends a results measure with an efficiency measure to sort teams for the selection committee. The RPI, which preceded it, failed because it ignored margin and could be gamed. The best way to use any of them is to ask what each is built to answer, and to compare a predictive rating with a results-based one when a team's place is in dispute. The current national rankings are on the rankings page, and the bracket they feed is on the March Madness hub.