When I started grad school in 2003, scholars of public opinion were freaking out about response rates falling to the 20-25% range. Throughout 2024 I routinely saw major surveys with response rates under 1%.
When you have to contact 120,000 people to get 1000 responses, the claim that the sample is in any way representative of the population is strained well beyond the breaking point. You might as well just be running simulations at that point.
A lot of it just ends up, like with the popularists, being an excuse to push *their* preferred policies. Notice Shor, Silver et al aren't encouraging pathways to citizenship and freeing palestine despite the change in polling!
You're literally doing the same thing. Are we to pay attention to polling? Or are we to ignore it? Or do we only pay attention to it when it confirms our biases?
As someone who comes more from the science side of things, I think we can conclude that basing positions on polling is silly even from that perspective.
Not long ago, Matt Yglesias wrote (on Bluesky?) that Democrats should follow opinion polls more because politicians who win are those who champion majority positions.
Even from the "science" perspective, it's as idiotic as pointing out that people who make money in the stock market are those who hold stocks of popular companies. That is tautologically true, but it misses one basic "scientific" element: to make money, they'd have had to buy those stocks *before* they became popular.
If the dynamics is that politicians who win are those who've championed popular positions *before* they were popular - or those who appear to hold those positions "authentically" (and if there's one thing Trump has taught us it's the appeal of the appearance of authenticity) - then following polls is a sure way to almost always lose.
And a general note: While I agree with the science vs art split, when you see someone use the science perspective to say something wrong, it's usually not because they ignore that split but because they're bad at science.
Changing minds is difficult. Have you ever tried to persuade someone with a strongly-held opinion to change it? It is rational to conclude that one is likelier to win an election by finding the people who already agree with you and turn out their vote than it is to spend n hours trying to change the minds of people who hold entrenched beliefs.
Him being able to put his policies under the single theme of affordability was a very good move. He had a striking theme and strong, simple policies about it.
But where his skills show more is in the tough questions, for instance on Israel. By repeating that his stance was rooted in a deep commitment in human rights and equalty for all, he avoided getting caught in the anti-Zionist debate trap ans framed correctly the question. By answering that he supported the right of Israel to exist as a state "with equal rights", he highlighted that the actual intenable position was the position of those attacking him for it.
All this in an extremely polite, open, agreeable way.
That he said what you wanted to hear on the subject of Israel is nice, but in my view he is preaching to the choir on that subject and changing nobody’s mind. Perhaps he did something akin to what Trump did, and successfully identified disaffected potential voting demographics and spoke to them in a way they were motivated by.
Perhaps you're not European so you're unaware how both the British and the French progressists have desintegrated and fragmented in million pieces over the Israel / Palestine debate.
Perhaps you've missed it, it's not striking, but the Israel / Palestine debate is an heated, violent, very performative, very bad faith, minefield debate. And he managed to get out of it mostly unharmed.
Not sure he's similar to Trump. Trump united different demographics with a catch-all approach and united through myths, Mamdani united with an evident and common theme (the most evident and common theme worldwide currently, when it comes to housing) and a set of policy proposals that actually are City Hall wonkish, but clear and bold.
I don’t think that’s the point John was making. We are short on leaders with principles, who will guide us into right thinking rather than mirroring back to us what we already think. JFK’s “ask not what your country can do for you; ask what you can do for your country” comes to mind.
I think it’s very much up for debate if leaders can actually guide us into “right thinking”, whatever that is. Who is “us”? What is right? Humanity has only been fighting over these questions for a few thousand years.
On immigration it's simple. People want strong border controls but don't want to deport those who've managed to establish lives for themselves here for years and are otherwise obeying the rules. Especially they don't like it when due process is ignored and immigrants are shipped off to torture prisons in third countries. Immigration is like rain in that moderate amounts are positive and necessary but too much all at once can be disruptive. Obama understood this and shaped his policy accordingly. The polls actually show all this if you dig down.
Professionally, I come from the land of Rhetoric and Composition (that goofy part of university English departments where we analyze commercials and ostensibly teach students how to write), and I'm heartened by this critique.
Relying on yes/no, percentile-based polling is a real failure to understand audience, leading to extraordinarily poor rhetorical awareness. Rhetoric isn’t just about polish and technique as a method of persuasion (the ethos, logos, and pathos of it all), but meaningfully employing those in response to your audience, purpose, context, and genre. To use a little jargon, your rhetorical situation.
Like you said, it’s dubious to ask an audience whether they favor or disfavor something described as “illegal.” If I had a political project I wanted to achieve, I wouldn’t simply ask my audience whether they favor policy X, Y, or Z. Instead, I’d interrogate how might my policy idea might align with my audience’s values. That is the basis for framing a message, for mounting something persuasive.
This requires an actual understanding of a polity’s ideology and lived experience. Perhaps polling has a role in understanding that, but the heavy reliance on polling seems to me like operatives are trying to distance themselves from any kind of agency or imagination. Following the polling means you don’t have to think critically about your choices (or even make choices, it seems), which is very comforting, but that probably means you shouldn’t be taking responsibility for running a government. This is an ungenerous read of pollsters, but I think the primacy of the poll at the expense of other kinds of understanding is seriously destructive.
Whenever I look at polling, it’s always about downstream opinions about policies or events, and it fails to capture how those opinions are formed. Other social science methods like focus groups and semi-structured interviews get closer, but getting to what’s underneath, unpacking what ideologies they reflect, will require a kind of analysis rooted in the humanities that isn’t provable in the way the sciences (social or otherwise) might want. It means you have to ACTUALLY make a meaningful case for an idea, put something at stake.
I only half agree with John's point. A politician who sounds poll-tested is not going to be a popular politician. A politician who ignores polls is not going to be an elected politician. Voters are indeed persuadable, but only by so much. A politician must scent the limits of persuadability, determined in large parts by external events--Bismarck's line about heeding God's footsteps marching through history, and trying catch on to His coattails as He marches past.
Even Bernie and AOC have been very careful in how they discuss Palestine, and their line is constantly modified by events. Mamdani is backpedaling now that he has transitioned from a primary to a general electorate. Or even Trump--he was pro-vaccine until his crowds were not.
Calculated authenticity is very hard. (Or maybe it is authenticity constrained by calculation?) Jesus admonished his followers to be wise as serpents and gentle as doves. He admonished his followers to do a lot of very difficult things.
Lincoln and Bismarck are the only two I can think of. And Lincoln had the great fortune of being assassinated. He never had the opportunity to fail at Reconstruction.
This is implied in your piece but your really talking about major, media-saturated campaigns. In those situations, I think you're right that the sort of data-heavy, hyper-calibrated politics that mainline Democratic consultants suggest is a disaster. But I'm not sure that really means polling itself is the issue. If you looked at Biden's internal polling right before he dropped out, the biggest grievance voters had was cost of living. That beat his senility. And what is Mamdani running on? Kamala tried to do the same but her policy was some means-tested tax credit nonsense that got lost in shuffle. So it's not just polling that's at fault. But you're absolutely right that the party needs to get it's head around the way that big campaigns can also be transformative.
This is correct. My complete distrust of the poll guys like Shor and Yglesias came when I saw they don’t loudly criticize politicians when they are out of sync with the public on issues that the public is left-wing on. Israel-Palestine is the most stark, 8% support among Dems, 32% among the normal public. More than 60% of Dem elected officials still support the war. Where are the popularists??
What bothers me most about opinion polls is that they’re predicated on the notion that people have opinions on all things, which I don’t think is true. On some things, sure. But not everything, all the time.
And so, because we’ve all been taught in school to try to give the right answer to a question, folks who haven’t really thought about an issue will reply with whatever they’ve heard from other people — whether friends, voices in the media, or the flotsam in their social media streams — because “that’s what everyone said, so it must be right." And then, when they finally do think about the issue, what they end up with is often something quite different.
So true. My ill-informed opinion will change once I have information. I doubt most pollsters ask their respondents how much they know about an issue since they’re having so much trouble getting anyone to respond at all.
The alternative to trusting polls is not trusting "vibes." There are more and better ways to conduct research than just blasting out a couple of questions to a small sample of people. Polls are just one tool, and they should be part of a toolkit and not relied on so exclusively.
Love the shout-out to the Catiline conspiracy and Cicero in your headline pic. I'm currently re-reading SPQR (by Mary Beard), and she uses that episode to tease out the themes of empire. Turns out my Roman empire is ... the Roman empire. I feel the universe is telling me to read Cicero! Thanks!!
In a big tent party saying anything at all is alienating to some. Republicans somehow transcended this and words no longer matter there, but I think Democrats don't have this luxury. And so they retreat to the message A/B testing to minimize damage. Obviously it ends with the general feeling of Democrats being fakes and less inspiring than rocks.
I think you're right. A party has to decide which of its potential supporters to piss off. The Democrats have made this decision, and have decided to piss off their left wing. The decision is defensible, but they have implemented it very clumsily. They code any independent attempt at resistance as "left:" e.g. BLM. They also do not distinguish between near-left (Mamdani-Sanders-Jayapal-AOC) and far-left. It is only the far left that repulses reachable voters.
The argument you seem to be making is that Trump somehow persuaded his base to be anti-immigrant, which seems backwards to me. Correct me if I'm wrong, but isn't part of your argument about Trump's rise to power in "When the Clock Broke" his recognition that Buchanan and Duke were speaking to an underserved demographic of potential voters who already held these anti-immigrant views?
When Trump spoke of the various flavors of ne'er-do-well coming from Mexico, was he doing Jedi mind tricks? Or was he dog-klaxoning to all the disaffected bigots that they finally had a Republican they could believe in?
I trust the seeming reversal of public opinion revealed in this issue polling re: immigration as much as I trust Bernie Sanders' deceptive claims about M4A polling. The claim that "poll after poll" showed support for M4A is one of the best examples of the dishonesty of this kind of argument when support for M4A vanishes once the polling goes deeper into the tradeoffs that single-payer involves, i.e. as the pollster makes more clear the meaning of single-payer in the United States and what that reality might look like.
My understanding of the polling swing on immigration is that it's not seen among Republican voters. So what has really changed here? Trump's base still hates immigrants, people who don't support Trump think he sucks even more than they already did.
It's strange to see people denigrating popularists like Shor via someone like Mamdani when Mamdani ran so strongly on extremely popular proposals like freezing the rent and promises to make the cost of living cheaper. It would be nice if people who are interested in politics could dial down the tribalism.
Couldn't agree more: views aren't fixed and voters can recognize when politicians are relying on consultant-tested speaking points as opposed to genuine beliefs (often because the language of polling and consultants is so tortured). It is correctly read as disingenuous and cowardly. Polling can be useful but its current hold over politics reduces it to a game of telephone separated from principles and reality. Baudrillard has a bit on this:
"The electoral sphere is in any case the first great institution where social exchange is reduced to obtaining an answer... The polls are located in a dimension beyond all social production...without regards to their social ends or lack thereof... For opinions, as for material goods: production is dead, long live reproduction... The only ones who believe in them finally are the members of the political class... It is the political class' burlesque spectacle, hyperrepresentative of nothing at all, that people taste by way of the polls and media... The problem is the operational simulation that they institute over the entire spectrum of social practices: that of the progressive leucimization of all social substance, that is the substitution for blood of the white lymph of media"
Great piece. I’ve been banging this drum for a while: popularism ignores the most important part of politics, which is persuasion. Make your moral case clearly and vividly and public opinion will follow.
People aren't persuaded by elected officials. People on the left gotta stop banging their heads against this wall: electeds are followers, not leaders.
We've seen many times over the recent decades how a president merely adopting a position makes it less popular. Including gifted orators like Obama.
Persuasion is important but it's done by other means.
When I started grad school in 2003, scholars of public opinion were freaking out about response rates falling to the 20-25% range. Throughout 2024 I routinely saw major surveys with response rates under 1%.
When you have to contact 120,000 people to get 1000 responses, the claim that the sample is in any way representative of the population is strained well beyond the breaking point. You might as well just be running simulations at that point.
A lot of it just ends up, like with the popularists, being an excuse to push *their* preferred policies. Notice Shor, Silver et al aren't encouraging pathways to citizenship and freeing palestine despite the change in polling!
Very true.
You're literally doing the same thing. Are we to pay attention to polling? Or are we to ignore it? Or do we only pay attention to it when it confirms our biases?
As someone who comes more from the science side of things, I think we can conclude that basing positions on polling is silly even from that perspective.
Not long ago, Matt Yglesias wrote (on Bluesky?) that Democrats should follow opinion polls more because politicians who win are those who champion majority positions.
Even from the "science" perspective, it's as idiotic as pointing out that people who make money in the stock market are those who hold stocks of popular companies. That is tautologically true, but it misses one basic "scientific" element: to make money, they'd have had to buy those stocks *before* they became popular.
If the dynamics is that politicians who win are those who've championed popular positions *before* they were popular - or those who appear to hold those positions "authentically" (and if there's one thing Trump has taught us it's the appeal of the appearance of authenticity) - then following polls is a sure way to almost always lose.
And a general note: While I agree with the science vs art split, when you see someone use the science perspective to say something wrong, it's usually not because they ignore that split but because they're bad at science.
Well, they are really into tautologies. Good things are good.
Changing minds is difficult. Have you ever tried to persuade someone with a strongly-held opinion to change it? It is rational to conclude that one is likelier to win an election by finding the people who already agree with you and turn out their vote than it is to spend n hours trying to change the minds of people who hold entrenched beliefs.
You manage to change minds when you know how to frame things in a different light. It's something Mamdani is extremely good at.
What was framed in a different light? That the cost of living is too high? Is this an exotic approach to running a political campaign?
Him being able to put his policies under the single theme of affordability was a very good move. He had a striking theme and strong, simple policies about it.
But where his skills show more is in the tough questions, for instance on Israel. By repeating that his stance was rooted in a deep commitment in human rights and equalty for all, he avoided getting caught in the anti-Zionist debate trap ans framed correctly the question. By answering that he supported the right of Israel to exist as a state "with equal rights", he highlighted that the actual intenable position was the position of those attacking him for it.
All this in an extremely polite, open, agreeable way.
That he said what you wanted to hear on the subject of Israel is nice, but in my view he is preaching to the choir on that subject and changing nobody’s mind. Perhaps he did something akin to what Trump did, and successfully identified disaffected potential voting demographics and spoke to them in a way they were motivated by.
Perhaps you're not European so you're unaware how both the British and the French progressists have desintegrated and fragmented in million pieces over the Israel / Palestine debate.
Perhaps you've missed it, it's not striking, but the Israel / Palestine debate is an heated, violent, very performative, very bad faith, minefield debate. And he managed to get out of it mostly unharmed.
Not sure he's similar to Trump. Trump united different demographics with a catch-all approach and united through myths, Mamdani united with an evident and common theme (the most evident and common theme worldwide currently, when it comes to housing) and a set of policy proposals that actually are City Hall wonkish, but clear and bold.
I don’t think that’s the point John was making. We are short on leaders with principles, who will guide us into right thinking rather than mirroring back to us what we already think. JFK’s “ask not what your country can do for you; ask what you can do for your country” comes to mind.
I think it’s very much up for debate if leaders can actually guide us into “right thinking”, whatever that is. Who is “us”? What is right? Humanity has only been fighting over these questions for a few thousand years.
On immigration it's simple. People want strong border controls but don't want to deport those who've managed to establish lives for themselves here for years and are otherwise obeying the rules. Especially they don't like it when due process is ignored and immigrants are shipped off to torture prisons in third countries. Immigration is like rain in that moderate amounts are positive and necessary but too much all at once can be disruptive. Obama understood this and shaped his policy accordingly. The polls actually show all this if you dig down.
Professionally, I come from the land of Rhetoric and Composition (that goofy part of university English departments where we analyze commercials and ostensibly teach students how to write), and I'm heartened by this critique.
Relying on yes/no, percentile-based polling is a real failure to understand audience, leading to extraordinarily poor rhetorical awareness. Rhetoric isn’t just about polish and technique as a method of persuasion (the ethos, logos, and pathos of it all), but meaningfully employing those in response to your audience, purpose, context, and genre. To use a little jargon, your rhetorical situation.
Like you said, it’s dubious to ask an audience whether they favor or disfavor something described as “illegal.” If I had a political project I wanted to achieve, I wouldn’t simply ask my audience whether they favor policy X, Y, or Z. Instead, I’d interrogate how might my policy idea might align with my audience’s values. That is the basis for framing a message, for mounting something persuasive.
This requires an actual understanding of a polity’s ideology and lived experience. Perhaps polling has a role in understanding that, but the heavy reliance on polling seems to me like operatives are trying to distance themselves from any kind of agency or imagination. Following the polling means you don’t have to think critically about your choices (or even make choices, it seems), which is very comforting, but that probably means you shouldn’t be taking responsibility for running a government. This is an ungenerous read of pollsters, but I think the primacy of the poll at the expense of other kinds of understanding is seriously destructive.
Whenever I look at polling, it’s always about downstream opinions about policies or events, and it fails to capture how those opinions are formed. Other social science methods like focus groups and semi-structured interviews get closer, but getting to what’s underneath, unpacking what ideologies they reflect, will require a kind of analysis rooted in the humanities that isn’t provable in the way the sciences (social or otherwise) might want. It means you have to ACTUALLY make a meaningful case for an idea, put something at stake.
I only half agree with John's point. A politician who sounds poll-tested is not going to be a popular politician. A politician who ignores polls is not going to be an elected politician. Voters are indeed persuadable, but only by so much. A politician must scent the limits of persuadability, determined in large parts by external events--Bismarck's line about heeding God's footsteps marching through history, and trying catch on to His coattails as He marches past.
Even Bernie and AOC have been very careful in how they discuss Palestine, and their line is constantly modified by events. Mamdani is backpedaling now that he has transitioned from a primary to a general electorate. Or even Trump--he was pro-vaccine until his crowds were not.
Calculated authenticity is very hard. (Or maybe it is authenticity constrained by calculation?) Jesus admonished his followers to be wise as serpents and gentle as doves. He admonished his followers to do a lot of very difficult things.
What about Lincoln
Lincoln and Bismarck are the only two I can think of. And Lincoln had the great fortune of being assassinated. He never had the opportunity to fail at Reconstruction.
This is implied in your piece but your really talking about major, media-saturated campaigns. In those situations, I think you're right that the sort of data-heavy, hyper-calibrated politics that mainline Democratic consultants suggest is a disaster. But I'm not sure that really means polling itself is the issue. If you looked at Biden's internal polling right before he dropped out, the biggest grievance voters had was cost of living. That beat his senility. And what is Mamdani running on? Kamala tried to do the same but her policy was some means-tested tax credit nonsense that got lost in shuffle. So it's not just polling that's at fault. But you're absolutely right that the party needs to get it's head around the way that big campaigns can also be transformative.
This is correct. My complete distrust of the poll guys like Shor and Yglesias came when I saw they don’t loudly criticize politicians when they are out of sync with the public on issues that the public is left-wing on. Israel-Palestine is the most stark, 8% support among Dems, 32% among the normal public. More than 60% of Dem elected officials still support the war. Where are the popularists??
Matt recently said that Dems moving left on Israel makes sense.
https://open.substack.com/pub/matthewyglesias/p/america-should-stop-giving-israel?utm_source=share&utm_medium=android&r=1zj554
What bothers me most about opinion polls is that they’re predicated on the notion that people have opinions on all things, which I don’t think is true. On some things, sure. But not everything, all the time.
And so, because we’ve all been taught in school to try to give the right answer to a question, folks who haven’t really thought about an issue will reply with whatever they’ve heard from other people — whether friends, voices in the media, or the flotsam in their social media streams — because “that’s what everyone said, so it must be right." And then, when they finally do think about the issue, what they end up with is often something quite different.
So true. My ill-informed opinion will change once I have information. I doubt most pollsters ask their respondents how much they know about an issue since they’re having so much trouble getting anyone to respond at all.
"It’s part of getting a picture of the world. It is not the entirety of it."
I certainly agree, but then what <i>is</i> the the proper role of polling and responses? How much weight do we give it in relation to 'the vibes?'
You note, "A great politician recognizes changing tides..." Well polling, properly done and properly considered, could certainly help with that?
The alternative to trusting polls is not trusting "vibes." There are more and better ways to conduct research than just blasting out a couple of questions to a small sample of people. Polls are just one tool, and they should be part of a toolkit and not relied on so exclusively.
Love the shout-out to the Catiline conspiracy and Cicero in your headline pic. I'm currently re-reading SPQR (by Mary Beard), and she uses that episode to tease out the themes of empire. Turns out my Roman empire is ... the Roman empire. I feel the universe is telling me to read Cicero! Thanks!!
In a big tent party saying anything at all is alienating to some. Republicans somehow transcended this and words no longer matter there, but I think Democrats don't have this luxury. And so they retreat to the message A/B testing to minimize damage. Obviously it ends with the general feeling of Democrats being fakes and less inspiring than rocks.
Republicans "transcended" this by becoming a fascist personality cult.
I think you're right. A party has to decide which of its potential supporters to piss off. The Democrats have made this decision, and have decided to piss off their left wing. The decision is defensible, but they have implemented it very clumsily. They code any independent attempt at resistance as "left:" e.g. BLM. They also do not distinguish between near-left (Mamdani-Sanders-Jayapal-AOC) and far-left. It is only the far left that repulses reachable voters.
Good piece. Now do macroeconomic indicators :)
The argument you seem to be making is that Trump somehow persuaded his base to be anti-immigrant, which seems backwards to me. Correct me if I'm wrong, but isn't part of your argument about Trump's rise to power in "When the Clock Broke" his recognition that Buchanan and Duke were speaking to an underserved demographic of potential voters who already held these anti-immigrant views?
When Trump spoke of the various flavors of ne'er-do-well coming from Mexico, was he doing Jedi mind tricks? Or was he dog-klaxoning to all the disaffected bigots that they finally had a Republican they could believe in?
I trust the seeming reversal of public opinion revealed in this issue polling re: immigration as much as I trust Bernie Sanders' deceptive claims about M4A polling. The claim that "poll after poll" showed support for M4A is one of the best examples of the dishonesty of this kind of argument when support for M4A vanishes once the polling goes deeper into the tradeoffs that single-payer involves, i.e. as the pollster makes more clear the meaning of single-payer in the United States and what that reality might look like.
My understanding of the polling swing on immigration is that it's not seen among Republican voters. So what has really changed here? Trump's base still hates immigrants, people who don't support Trump think he sucks even more than they already did.
It's strange to see people denigrating popularists like Shor via someone like Mamdani when Mamdani ran so strongly on extremely popular proposals like freezing the rent and promises to make the cost of living cheaper. It would be nice if people who are interested in politics could dial down the tribalism.
Couldn't agree more: views aren't fixed and voters can recognize when politicians are relying on consultant-tested speaking points as opposed to genuine beliefs (often because the language of polling and consultants is so tortured). It is correctly read as disingenuous and cowardly. Polling can be useful but its current hold over politics reduces it to a game of telephone separated from principles and reality. Baudrillard has a bit on this:
"The electoral sphere is in any case the first great institution where social exchange is reduced to obtaining an answer... The polls are located in a dimension beyond all social production...without regards to their social ends or lack thereof... For opinions, as for material goods: production is dead, long live reproduction... The only ones who believe in them finally are the members of the political class... It is the political class' burlesque spectacle, hyperrepresentative of nothing at all, that people taste by way of the polls and media... The problem is the operational simulation that they institute over the entire spectrum of social practices: that of the progressive leucimization of all social substance, that is the substitution for blood of the white lymph of media"
Great piece. I’ve been banging this drum for a while: popularism ignores the most important part of politics, which is persuasion. Make your moral case clearly and vividly and public opinion will follow.
People aren't persuaded by elected officials. People on the left gotta stop banging their heads against this wall: electeds are followers, not leaders.
We've seen many times over the recent decades how a president merely adopting a position makes it less popular. Including gifted orators like Obama.
Persuasion is important but it's done by other means.