One of the Most Prominent Findings in Political Science Might Be Wrong

Which means your voice matters more than you’ve been told!

Photo by Nicholas Green on Unsplash

This post is based on my recent article in Sociological Science, “How a Seemingly Innocuous and Intuitive Methodological Choice Confused a Generation of Research on Policy Responsiveness.”

The most comprehensive study ever of political responsiveness in the United States concluded that policy outcomes “bear virtually no relationship to the preferences of poor or middle-income Americans” and that “representational biases of this magnitude call into question the very democratic character of our society” (Gilens 2005, 778).¹

This research has had a massive impact — cited more than 8,250 times and featured on *The Daily Show, [The Rachel Maddow Show](https://www.ms.now/rachel-maddow-show/watch/political-responsiveness-to-the-wealthy-not-just-a-theory-44147779663), and in other major media. Using the same data and approach, others have similarly found that policy ignores or [moves opposite](https://www.journals.uchicago.edu/doi/10.1086/729964) *women’s preferences.

While it may feel like politicians are disconnected from their constituents, completely ignoring the economic majority and women seems politically untenable. Since these groups comprise the majority, why wouldn’t they vote unresponsive politicians out of office? The answer, it turns out, is that the evidence does not actually support the claims of no responsiveness. We’ve been telling Americans their voice doesn’t matter, but this is not the case.

The Source of Confusion: Only Analyzing Policies Where a “Preference Gap” Exists

The above studies analyzed data that includes measures of how much different groups supported 1,779 potential policies between 1981 and 2002 and whether or not these policies were enacted. The data seem ideally suited to study policy responsiveness. Researchers can directly test whether policies that receive more support are more likely to become law and whether policy outcomes reflect some group’s preferences more than others.

The challenge, however, is the policy preferences of different groups overlap to a surprising extent in the United States. The overlap in preferences makes it impossible to determine whose preferences policymakers consider. If two groups have identical policy preferences and policy follows these preferences, we cannot know if policymakers responded to both groups, or if they only paid attention to one group and policy just happened to align with the preferences of the other group.

For this reason, the research described above only analyzes policies where groups’ preferences diverge by more than 10 percentage points. For example, if 60% of the affluent supported a policy and 65% of middle-income Americans supported the policy, this policy would not be analyzed. The difference in support — i.e., the preference gap — is only 5 percentage points. However, if support among the affluent and middle-income groups was 60% and 71%, respectively (an 11 point preference gap), the policy would be analyzed because preferences are deemed sufficiently different to assess responsiveness to both groups.

The black dots in Figure 1 show the policies analyzed by Martin Gilens, when he found policy responsiveness to the affluent (measured as the top 20% of the income distribution), but no responsiveness to middle-income Americans. For any solid dot, if you look at the percent of middle-income individuals who support the policy (vertical axis) and the percent of the affluent who support the policy (horizontal axis), the absolute difference (the preference gap) is greater than 10 percentage points.

The gray dots are the policies that were not analyzed because the preferences of affluent and middle-income individuals were deemed too similar. The large number of gray dots indicate that middle-income and affluent Americans agree on most proposed policies.

Figure 1: Solid black dots show the policies analyzed by Gilens because more than a 10 percentage point preference gap exists between middle-income and affluent Americans’ level of support. Figure reproduced from Enns 2015.

Only analyzing policies where a large preference gap exists has become a widely accepted strategy for studying whose opinions policymakers consider. Unfortunately, this approach has led researchers to incorrectly conclude that policy ignores huge numbers of Americans.

Example 1: Policy Responsiveness to the Economically Vulnerable

The video below (Figure 2) shows why analyzing preference gaps can produce misleading conclusions about policy responsiveness. I use the data discussed above and I focus on the lowest income group analyzed by Gilens (the bottom 20% of the income distribution), as this might be considered a least likely group to experience policy responsiveness.

Before you start the video, take a look at the figure. The preference gap is zero, meaning all data are included. The horizontal axis indicates the percent of low income individuals who support each of the 1,779 potential policies.

Because the goal is to understand the effects of limiting the data to policies where a preference gap exists, I separate the data into two groups. Orange bars reflect policies the affluent prefer more than the low income group. Blue bars indicate policies the low income group supports more than the affluent. The height of the bars indicates how many policies exist at each level of the low income group’s support.

Still looking at the first image (don’t click the video yet), the orange, blue, and gray lines are the estimates of responsiveness to the low income group when the affluent support policies more than the low income group (orange), the affluent support the policies less (blue), and when all data are analyzed together (gray). Note that all lines slope up and to the right, which means that the more the low income group supports a policy, the more likely it is to be passed. Regardless of how we group the data, we observe policy responsiveness to the least well off.

[embed]Figure 2: Why analyzing preference gaps can produce Simpson’s paradox and incorrect conclusions about policy responsiveness. Evidence based on responsiveness to the lowest income group.

But what happens when we follow Gilens’ approach and limit the data to policies where a preference gap exists?

The 40-second video answers this question by gradually increasing the size of the preference gap up to Gilens’ threshold of more than 10 percentage points. The gray line (responsiveness to the low income group when all data are analyzed) remains constant, offering a benchmark to assess how the progressively larger preference gaps influence the results.

The orange and blue lines, which reflect responsiveness to the low income group when the affluent are more or less supportive of policies, are re-estimated each time the preference gap grows. When you click on the video, pay attention to the slopes of these orange and blue lines. You will see that they change a little, but have nearly identical positive slopes at each step — the evidence of responsiveness to the least well off does not change.

Also pay attention to the dark gray line that emerges. The dark gray line shows responsiveness when all preference gap data are analyzed together (i.e., orange and blue combined). This is the level of responsiveness reported in the most widely cited studies. As the preference gap increases, even though the slopes of the orange and blue lines remain positive, the slope of the dark gray line becomes flatter and flatter, until in the final image it is an exact reproduction of Gilens’ analysis.

The flat dark gray line is the basis for the claim of no responsiveness to the lowest income group. In reality, it’s an example of Simpson’s paradox, where “a trend appears when looking at individual groups in the data, but disappears or reverses when the groups are combined.”

The estimate of zero responsiveness results because the nonoverlapping portions of the orange and blue lines pull the overall estimate in opposite directions, producing a flat line even though the slopes of the orange and blue lines remain positive.

Example 2: Resolving the “Paradox” of Negative Responsiveness to Women

Simpson’s paradox applies more generally than just to the study of responsiveness to income groups. When we consider responsiveness to women, the effect of limiting the data to policies with a preference gap is even more striking.

Again, before starting the next short video (Figure 3), notice that regardless of whether men prefer policies more than women (orange) or women prefer the policies more than men (blue), the orange and blue lines show evidence of policy responsiveness to women. The gray line, which includes all data (orange and blue combined) also shows evidence of responsiveness to women. In all cases, as the percent of women who support the policy increases, the probability of that policy change occurring also goes up.

As above, when you watch the video, notice how the orange and blue lines change slightly but retain their positive slope. Also notice that as the preference gap grows, the estimate of responsiveness based on all policies with a preference gap (dark gray line), shifts from positive to negative. The estimate of negative responsiveness in the final frame of the video is an exact replication of one of the most prominent analyses of how policy responds to women’s preferences.

As with income groups, only analyzing data where a preference gap exists introduces Simpson’s paradox, incorrectly suggesting negative responsiveness to women even though the underlying patterns in the data (orange and blue lines) continue to show evidence of positive responsiveness.

[embed]Figure 3: Why analyzing preference gaps can produce Simpson’s paradox and incorrect conclusions about policy responsiveness. Evidence based on responsiveness to women.

Why Correcting the Record is Important for Politics

Headlines based on the above research, such as “The Politics of Always Ignoring What Average Americans Want’’ and “Politicians listen to rich people, not you,” may have produced serious political consequences. If groups are repeatedly told that their voice does not matter, they may decide that it’s not worth it to be involved with politics.

Ironically, research that incorrectly concluded the economic majority and women have no influence may have reduced these groups’ political engagement — perhaps actually reducing their influence.

Headlines indicating that average Americans are ignored may have also increased frustration and anger with the political system, buttressing support for candidates who campaign against the political establishment.

Imagine if instead of incorrectly concluding that policy ignores certain groups, research and corresponding media coverage had highlighted how similar different groups’ policy preferences are and that policy appears responsive to their preferences. Emphasizing the evidence of shared preferences and democratic accountability sends a very different political message.

What About the Superrich?

Gilens’ data do not measure the preferences of the superrich. Thus, even if policy is more responsive to the economic majority and to women than previously thought, multi-millionaires and billionaires may wield substantial additional influence. This insight suggests another important update. Instead of average Americans versus the top 20 percent or men versus women, the real divide in political influence may be the top .01 or even .001 percent at the expense of the rest of us.

But the voices of average citizens still matter. My analysis shows that even after controlling for the preferences of business interests and mass-based interest groups, policy responds to the public’s preferences. More political engagement is the way to combat political inequality.

Additional Considerations (for Interested Readers)

Alternate Analytic Strategies Do Not Alter These Conclusions

Those familiar with Gilens’ (and Gilens and Benjamin Page’s) research may recall that they have found similar results with an alternate approach that analyzes all data (not just policies for which a preference gap exists). Their analyses of all the data address the overlap in group preferences by using a statistical approach to correct for correlated measurement error in the preferences of low, middle, and high income groups.

Originally, the similar findings from an alternate approach were reassuring. The above results, however, raise the question — Why does the analysis based on all the data reproduce a result that occurs because of Simpson’s paradox?

It turns out that the results based on all the data are extremely sensitive to the approach used. Different measurement error corrections produce different conclusions about whose preferences policy reflects — with their original results reversing in some cases. Because we cannot know the true amount of correlated measurement error in the data, we cannot use their alternate approach to make conclusions about policy responsiveness.

A Different Type of Policy Advantage? Probably Not

In both videos shown above, the orange lines are higher than the blue lines. These different heights indicate the probability of policy adoption is higher when the affluent and men prefer policies more than low income individuals and women. While these patterns do not alter the above conclusions about policy responsiveness (as defined by Gilens and many others), we might wonder if the different height of the orange and blue lines reflects a different type of policy advantage.

This is unlikely to be the case. The orange lines don’t just include policies with high levels of support among the affluent and men. They also include policies that are extremely unpopular with these groups, such as a proposed increase in the phone service tax in 1982 — favored by just 26% of the affluent. When this tax increase became law, should we really conclude that the affluent benefited since the least well off were even more opposed? The nearly 75% of the affluent who opposed the increase would likely answer no.

The difference in the height of orange and blue lines also means that policies that the affluent and men strongly support are less likely to pass when other groups support them more. It’s hard to see how not using the 1999 federal budget surplus to support Medicare — which 84% of the affluent supported doing — benefited the affluent because even more of the lowest income group supported this policy. Getting policies you don’t want and not getting policies you want because a different group supports the policies less or more seems like a strange type of policy advantage.

For an even deeper dive, see the full article:

Enns, Peter K. 2026. “How a Seemingly Innocuous and Intuitive Methodological Choice Confused a Generation of Research on Policy Responsiveness.” Sociological Science.

Notes

1. Also see Gilens (2011), Gilens (2012), Gilens and Page (2014), and Page and Gilens (2017).