You have got to be kidding. On November 8, City of Arlington residents are expected to vote on a divisive ballot measure to finance the proposed $1 billion Texas Rangers Stadium. Meanwhile, campaign stakeholders have released a series of poorly designed, automated, low-cost polls to measure the public opinion on this important issue. What’s wrong with this picture?
With such high stakes consequences, one would assume that poll sponsors would want to support their campaign advocacy with a high-quality poll conducted by a polling company with a recognized track record. Poll sponsors may try to stack the deck to support their campaign objectives, but a reputable pollster with a good track record would not knowingly bias a study. Not everyone that conducts opinion polls, however, are reputable pollsters. Indeed, the “shadow” polling industry includes many telemarketing firms, call centers and political operatives that have little or no training in survey practices or ethical conduct, and usually not active in professional polling organizations.
Each of the sponsored polls have reported different results, used varying methodologies, and were conducted by polling firms with varying reputations. Only one of these polls –conducted by DHC Data — has been subjected to critical review by survey experts in local news stories and considered to be of questionable quality. Interestingly, the polls sponsored by the Say Yes campaign and WFAA/Fort Worth Star-Telegram have not been critically analyzed by survey experts in local news reports. Because the results of these polls are likely to influence the voting behavior of Arlington city residents, I believe that each of these polls require some scrutiny as well. The reputation of a pollster is clearly important, but not as important as their polling methodology in a particular study.
I reviewed only one online report for the poll sponsored by WFAA and the Fort. Worth Star-Telegram, while relying on published news reports regarding the methodology of the other three polls. I discovered shortcomings in all polls, and would like to share my thoughts on their implications for polling accuracy and voting outcomes. My only objective here is to educate the public about good and bad polling practices — topics that I usually address in classes that I teach on survey research methods, mass communications research, and statistics. In addition, the information discussed should provide some help in deciding which poll deserves more of the public’s confidence.
1. Sample Selection: Each of the polls reported that their target audience included likely voters in the City of Arlington. However, only one of the pollsters — Public Opinion Strategies — sampled landline and cell phone households since they used live interviewers to manually dial the numbers, as required by the FCC, which is likely to capture a more representative sample of voters. DHC Data (for Save Our Stadium), however, relied exclusively on landline phones while Survey USA (for WFAA/Star-Telegram) relied primarily (76%) on landline phones and less on mobile phones. Good survey practice suggests that pollsters should rely less on landline telephones because their penetration has declined significantly in recent years and are more likely to capture older residents. A recent study by the Pew Research Centers explains the wisdom of placing more reliance on cellphone households in telephone-based surveys:
“Samples of adults reached via cellphone are much more demographically representative of the U.S. than samples of adults reached via landline. Consequently, replacing landline interviews with cellphone interviews reduces the degree to which survey data need to be weighted to be representative of U.S. adults. This in turn improves the precision of estimates by reducing the margin of sampling error. Perhaps not surprisingly, one major survey was recently redesigned to feature 100% cellphone interviewing.” (The Twilight of Landline Interviewing,” Pew Research Center, August 1, 2016)
Thus, studies that rely primarily on landline telephone households may be “stacking the deck” by placing more weight on the opinions of older residents than the opinions of residents that depend more on cell phones, such as younger and ethnic minority residents.
2. Exclusion of Demographics: Without demographic information about the poll respondents, it is difficult to know how well the poll respondents represented the voting community. There is no good reason to hide this information other than to avoid scrutiny by other experts. Each of the studies tell us that their target audiences were likely voters in the City of Arlington, but only one of the polls (WFAA/Star-Telegram) provided demographic information for the respondents that could influence the survey outcomes – such as race, gender, and age. For pollsters that do not disclose demographic information, we are left to wonder if these polls over- or under-represented particular segments of the community which could mispresent the polling results. None of the pollsters reported whether their polling results were weighted or adjusted to reflect the demographics of the voting community in the city of Arlington.
3. Questionnaire Content: Survey experts interviewed in news stories had mixed opinions about the one poll reviewed (Save Our Stadium), pointing to such problems as leading questions or long questions that would test the memory of any person. Campaign representatives on both sides have pointed to incomplete or misleading descriptions of the ballot measure as well.
4. Data Collection Approach: With the exception of Public Opinion Strategies (POS), the two other polling firms (DHC Data and Survey USA) opted to use the cheapest and least credible data collection approaches to collect opinions on this divisive issue: pre-recorded, automated telephone calls instead of live telephone interviews. Automatic telephone calls have little credibility in the polling industry because they remove human contact, and do not provide any opportunity for clarification when respondents are confused. Automated telephone calls are often rejected by residents because they are associated with telemarketing firms that often annoy the public. Polling firms employ automated calls when they have limited time available, have a limited budget to fund live telephone interviews, or have limited resources to use live interviewers. Because FCC regulations prohibit automated calls to cell phone users unless they are manually dialed, polls using automated methods exclude nearly half of community residents who have only wireless devices but no landline telephones – a practice that systemically excludes younger residents and ethnic minority groups.
5. Language offered: Based solely on news reports about these polls, it appears that none of the pollsters offered a language other than English to collect their data. Why is this important? Hispanics comprise 29 percent of Arlington city residents, while 36 percent of Hispanics are foreign-born and primarily Spanish-speaking. Our past experience shows that 50 to 63 percent of Hispanics will prefer a Spanish-language interview because they find it easier to express their opinions. Unless their presence in the voting community is minimal, it makes little sense to exclude this strong base of baseball fans by offering only one language. Indeed, it is likely that the estimate of support for the new stadium could be under-estimated by this exclusion.
6. Pollster’s Reputation: The reputation of the polling companies was also discussed in news reports. In my opinion, Public Opinion Strategies utilized the most credible polling methodology since all interviews were conducted by telephone with live interviewers, their two polls included landline and cell phone households, and the company has a long history of public opinion polling. DHC Data, however, was characterized in news reports as having a low visibility, no web site, and questionable experience as a pollster. Its owner, however, claims to have conducted several polling studies in past years. Survey USA – who conducted the WFAA/Ft. Worth Star Telegram poll, was also described as having a solid polling history. Interestingly, survey experts only scrutinized the poll conducted by DHC Data, while the polls conducted by the other two polling firms received praise for their track records but little criticism of the polling techniques used in the Texas Rangers campaign. It is a risky practice to avoid scrutiny of a pollster’s practices because they have a great reputation.
In summary, the most recent polling results are summarized below:
· Save Our Stadium poll by DHC Data: 38% support, 46% oppose, 16% undecided
· Say Yes polls by Public Opinion Strategies:
o Sept. 23-25: 54% support, 40% oppose, 6% undecided
o Oct. 14-15: 56% support, 37% oppose, 7% undecided
· WFAA/Ft. Worth Star Telegram poll by SurveyUSA: 42% support, 42 opposed, 16% undecided
Ultimately, the election scheduled for Nov. 8 will be the final word on which pollster provided the best picture of how Arlington residents feel about the Texas Rangers Stadium issue. Based on the information evaluated thus far, I believe that the polling results by Public Opinion Strategies for the Vote Yes campaign – 54-56 percent supporting the stadium referendum – presents the most accurate picture of the actual voting outcome. Why? Primarily because they used human beings to conduct the interviews and included both landline and cell phone residents in their study. The poll was not without its own shortcomings since it did not describe the respondents’ demographic attributes, and may have excluded Spanish-speaking and younger voters by over-relying on landline telephone households. Nonetheless, I believe that their polling practices and results are more deserving of the public’s confidence in comparison to the other polls.