Be Less Wrong

What all this analytics stuff is about

The best hypothetical GM, coach, scout, or analyst would be an oracle - a person who can see into the future. To predict, with perfect clarity and prescience, which players were going to mature into superstars, what tactics would help beat the bad guys, and what contract to give to the prized free agent.

Unfortunately, reality is messy and complicated. We don’t have complete knowledge of what has happened, we don’t understand all of the causal paths that lead to outcomes, and the entire affair is beset by randomness anyway. Even with an intact, clean, accurate dataset at our disposal, we likely couldn’t predict when the puck will skip over a stick at the wrong time, or bounce off the post and into the corner, etc. Hockey is a game played at high speed on a slippery surface with a six-inch rubber projectile. Stuff happens.

Meaning, while the goal is to be right all the time, the truth is forecasting is difficult and fraught with error. The more realistic objective is to be less wrong. Avoid the landmines in the field. Refine your decision-making to get a bit closer to the truth each time, but accept that failure will be both frequent and inevitable.

Be non-consensus right

Of course, hockey is also a contest, between both the players on the ice and the staff and decision-makers of the various clubs. So not only do you want to be less wrong in general, you also need to be ahead of the knowledge market curve in order to properly leverage that advantage.

The state of the knowledge market can be expressed by a simple 2X2 matrix which I have shamelessly stolen from venture capital/angel investors:

Consensus right: This where you find the established truths, conventions, theories, and practices. This is naturally where the crowd hangs out (or wants to, at least). Necessary but not sufficient to win in a highly competitive knowledge market. This is where most “non-consensus right” ideas end up over time.

Consensus wrong: The second most popular quadrant. In part, because we aren’t always good at separating between consensus “right” and “wrong”. There are a few reasons for this -

First, because things that were true in the past may no longer be true in the future. The game naturally changes and evolves over time. As does our access to and understanding of the data underlying the game. Many things that worked in, say, 1980 may not work now.

Second, because humans are herd animals, we often tend to mistake agreement for truth. To be fair, this is not a terrible heuristic to have in some cases (because consensus does often congregate around the truth over time). Of course, history is also rife with examples of conventional wisdom that seemed obvious but later turned out to be totally wrong.

Directionally, some consensus “truths” flow into this quadrant. Issues like gradual evolution and the misconstruing of agreement for truth create a latency period before these are consigned to the non-consensus wrong segment.

Non-consensus wrong: The worst of the four quadrants to be in. The obviously wrong and the proven to be wrong stuff ends up here. Considered the territory of madmen and fools. Because you aren’t right and, further, no one even thinks you’re right, hanging out in the upper left side generally leads to humiliation and ruin.

Non-consensus right: The highest value, most difficult space to occupy. You need counter-intuitive insights, different/better data, and the stalwart ability to go against the grain in order to execute here. Of course, the rewards are also the greatest when you’re non-consensus right.

How two

There are two ways to leverage advanced analysis and analytics to beat the market - figure out where the consensus is wrong but doesn’t know it yet (be less wrong), or figure out something the consensus doesn’t know at all before anyone else does (be non-consensus right).

Naturally, this is easier said than done. It’s difficult to discern between non-consensus wrong/right because the established truths (or inherited dogma) were either true at some point (but no longer are), or they are conventions buttressed by standards, norms, plausible-sounding narratives, etc (ie; social pressures).

Relatedly, the difficulty in separating non-consensus truths from falsehoods is that almost all things that are “non-consensus right” seem unintuitive, silly, impossible, or ridiculous initially. And because predicting the future is messy and difficult, we can never be sure which ideas will turn out to be right, especially when they are new and novel.

This is also why the “stats revolution” in hockey was necessarily controversial: it had to run counter to consensus, or else it would have been of no real value (and wouldn’t have been noticed anyway).

**A note here about consensus social pressures - often traditions and conventions are erected not to help people be more right, but in fact to avoid being totally wrong. While there are big incentives associated with being ahead of the pack, there are ruinous consequences for being terribly wrong. The parable of Chesterton’s Fence comes to mind here.

As such, the norms and pressures exerted around the consensus can be considered a protective membrane from ruin. Of course, it can also mean a stifling of innovation and reform.

Next week, I’ll discuss the implications this simple mental model can have on hockey organizations as they try to integrate analytics into their decision making.

---

What I’m reading:

JFresh on PDO and why it remains a useful heuristic in analytical thinking. While PDO isn’t a true performance metric, it is a very simple and easy way to understand and track regression to the mean.

Draglikepull’s deep dive into corsi (total shot attempts) vs expected goals (shot attempts weighted by quality). His findings are surprising: while XGF% might be more descriptive, CF% seems to be the more predictive metric.

Jack Han talks about how seemingly dumb ideas can nevertheless lead to intriguing thought experiments. In this case - what if a team played two goalies at the same time.

HBF Analytics released a new SPAR (Stand Points Above Replacement) model and visualization tool.

Foundational work: Jlikens at Objective NHL on team shooting talent

For those who didn’t know JLikens (Tore Purdy), he was a brilliant young advanced analytics writer who helped establish the basis for the “fancy stats” revolution. You can read about his outsized impact on the analytics community and his sudden, tragic passing here.