DEI Studies that Prove Nothing

Studies get a free pass in academia if they promote the right narrative

I am sometimes asked about studies that claim “diversity” makes companies more successful. Of course, “diversity” in this context doesn’t mean diversity of ideas, background, or outlook; it means diversity of skin color or ethnicity.

One of the most cited of these studies is Cedric Herring’s “Does Diversity Pay?” It has garnered a spectacular 1803 academic citations (a mere 20 is considered respectable in most fields).

The study sought to prove that DEI policies help businesses succeed. Using a business database to conduct statistical regression analysis (OLS), it concluded that:

Diversity is associated with increased sales revenue, more customers, greater market share, and greater relative profits…[T]hese results are consistent with arguments that a diverse workforce is good for business, offering a direct return on investment and promising greater corporate profits and earnings.

Complete Methodological Failure

After it was published in 2009, the Herring study won eight years of wild cheers and accolades from DEI advocates and people who sport “Celebrate Diversity” bumper stickers on their Volvos. However, in 2017, a team of University of Amsterdam scholars attempting to replicate the study found egregious methodological flaws. For example, if a company’s sales were unknown, Herring’s study automatically coded them as $88,888,888,888 in the regressions! This occurred no less than 206 times. A more complicated stats error (failure to adjust for heteroscedasticity) further invalidated Herring’s results. The replicators concluded:

When these errors are corrected, the results no longer support Herring’s conclusions. In the years following its publication, Herring’s article has become an influential source of empirical support for the value-in-diversity perspective. Our replication shows that the data Herring analyzed are not consistent with this perspective. The overall pattern of findings suggests that diversity is nonconsequential, rather than beneficial, to business success.

Herring responded to the replication, re-running the study without the spurious $88 billion values. He concluded:

the results continue to provide support for the seven hypotheses that received some support in the original analyses (at p <.1).

What? The standard for statistical significance is p < .05 (not p <.1). Why not just admit that you failed to prove your hypotheses rather than slyly altering the scientific standard for what constitutes proof?

But it’s worse than that. Statistical correlations produce a value known as “R^2” (R-squared), which specifies how closely the variables you’re measuring are connected (i.e., how much of the variation in one is associated with variation in the other). The R^2 values for Herring’s study are absurdly small. For example, the R^2 for profitability is 0.025—that means diversity “explains” only 2.5% of the variation in profitability.

Confound Those Confounding Variables!

But, above all, both Herring and the replicators overlooked the most obvious problem with the study: the failure to identify and rule out confounding variables (underlying variables that might explain both of the two things you’re trying to link causally).

For example, here’s one possible confounder: Apple, Microsoft, and Alphabet are the three most profitable companies in the U.S. and there’s a whole slew of very large tech and pharmaceutical companies that rank right below them. All of these companies employ highly trained engineers and scientists. Given that 60% of graduate students in computer science and engineering are from overseas, it follows that these profitable companies will have relatively diverse workforces. But that doesn’t mean that diversity per se is what makes them profitable. They’re profitable because they have highly qualified employees with rare skills, and those employees happen to be “diverse.”

But wait, there’s more. The largest companies (and, by correlation, the most profitable companies) are naturally concentrated in major population centers. Perhaps because nearly 40% of S&P 500 earnings come from overseas, they are also concentrated near the coasts (see map below). Big cities and coastal areas are much more diverse than the rest of the country, so the big companies in these areas will naturally have more diverse workforces. Once again, that diversity is coincidental to company size and profitability rather than causative. If Herring had wished to eliminate some of the coincidental effects, he could have calculated his company diversity index relative to local (rather than national) diversity. But he didn’t do that.

But there’s even more: per Nobel Prize-winning economist Gary Becker’s research, a company’s ability to favor employees for non-merit-based reasons scales with its monopoly power (and thus, its profitability). So, companies with dominant and profitable market positions (e.g., Google and IBM) don’t necessarily enjoy that status because they practice DEI; they may just practice DEI because they’re really profitable and can afford inefficiencies that other companies can’t.

Wishful Thinking + Academe’s Echo Chamber = Bad Science

I’m not sure what’s more amazing: that the head of a major university’s sociology department would publish a study with such execrable methodology—or that the study would be widely cited and its obvious errors ignored for nearly a decade. Incredibly, more than 1200 of the 1803 citations occurred after the study’s flaws were exposed in 2017. But then, this just reflects the prevailing environment in academia: work that supports the approved narrative gets a free pass, and work that doesn’t (like Roland Fryer’s) gets you canceled.

Fortune 500 Companies