Using algorithms for everything may produce results, but also a boring product

"Moneyball" is a 2004 book by Michael Lewis that described how the Oakland A's used advanced analytics to find inefficiencies in baseball scouts' evaluations of baseball prospects, reducing biases based in traditional heuristics. It also drove increases in home runs and strikeouts, based on their true statistical value in producing runs.

The Moneyball revolution also spread to the NBA, where statistical analysis drove teams to optimize shot efficiency, by taking as many three-pointers and shots at the rim as possible, and reducing midrange shots.

An October article in the Atlantic magazine by Derek Thompson explores the implications of the trend of rigorously quantifying decisions previously made substantially by gut feel.

What has been the result of this optimization? Both games have arguably lost much of their aesthetic value.

In baseball, players are taught to swing on a high arc to maximize home runs. They also produce more strikeouts than hits.

In the 1970s, Minnesota Twins star Rod Carew mesmerized baseball fans by several times coming close to hitting .400 in a season. But Carew was a singles and doubles hitter, who never had more than 14 homers in a season.

Baseball's classic aesthetic was more about getting a man on base somehow, moving him to second with a sacrifice bunt, and getting him to score with a single up the middle.

In the NBA, advanced stats have helped players optimize play. But the subtleties of basketball have often been lost given the overwhelming percentage of plays that are either attempted three-pointers or "pick and rolls" — a classic, highly effective but incredibly boring play, especially when teams run it dozens of times a game.

In his Atlantic article, Thompson describes how this trend has spread to other sectors of the culture. When was the last major innovation in popular music? Rap, which is now over 30 years old. Rock 'n' roll stars of the 1960s, innovators then, now in their 80s, are still touring. After all, the numbers tell radio stations and streaming apps what is popular, so they play more of it.

This same dynamic is taking place in the movie industry, where historically it was difficult to predict what would be a hit. About 20 years ago, movie executives discovered that making superhero movies based on classic comic books was a far safer bet, since those had an existing, enthusiastic base of fans who would see the movie multiple time and evangelize for them online.

So now we live in the Marvel Universe.

What does this have to do with business outside of entertainment? Actually a lot.

In production, we use statistical optimization to reduce the variance and raise the quality of products, physical or virtual. It is quite another to use algorithms to predict what consumers want. That inevitably results in the producer choosing which product from the present is most palatable to a consumer.

Even "long tail" marketing, which is much more tailored to matching individuals with products, still does not tell us anything about what new thing a customer might want.

Steve Jobs, co-founder of Apple, notoriously hated focus groups: "It's really hard to design products by focus groups," he said. "A lot of times, people don't know what they want until you show it to them."

Creativity is messy, and inherently does not lend itself to statistical optimization by algorithms.