Your Opinion, Please
How do you know what people think — unless you ask?
By David Tenenbaum MA'86
You know the script. It's dinner. Your family members are gathered around the table, perhaps the only time they will be together all day. The salad is dressed, and the hot dish is being passed.
Ring. Ring.
Surprised? No. Everybody knows that dinnertime is telemarketing time.
The pitiless ringing of telemarketing phone calls has led Wisconsin, along with twenty-seven other states, to establish do-not-call lists. By the time the list goes into effect on January 1, 2003, the state expects that as many as one million people will have signed up via phone or Web site. Two agencies, the Federal Trade Commission and the Federal Communications Commission, are considering a national no-call list.
By David Tenenbaum MA'86
You know the script. It's dinner. Your family members are gathered around the table, perhaps the only time they will be together all day. The salad is dressed, and the hot dish is being passed.
Ring. Ring.
Surprised? No. Everybody knows that dinnertime is telemarketing time.
The pitiless ringing of telemarketing phone calls has led Wisconsin, along with twenty-seven other states, to establish do-not-call lists. By the time the list goes into effect on January 1, 2003, the state expects that as many as one million people will have signed up via phone or Web site. Two agencies, the Federal Trade Commission and the Federal Communications Commission, are considering a national no-call list.
But even if you've joined the list, you may still get calls from strangers. Although telemarketers are prohibited from calling you, political pollsters, non-profit organizations, and legitimate survey researchers are exempt. So before you hang up on a researcher (or on this article), you may be curious to know why the work is so important.
Let's start with what these researchers are not doing. They are not selling anything. They are not gathering your personal information to sell it. They are not even trying to interrupt dinner. In fact, legitimate researchers will practically beg you to specify a better time to talk.
Essentially, they're trying to find out how Americans think and act — knowledge that ultimately can influence decisions that affect our daily lives.
Let's start with what these researchers are not doing. They are not selling anything. They are not gathering your personal information to sell it. They are not even trying to interrupt dinner. In fact, legitimate researchers will practically beg you to specify a better time to talk.
Essentially, they're trying to find out how Americans think and act — knowledge that ultimately can influence decisions that affect our daily lives.
James Sweet, an emeritus professor of sociology at UW-Madison, says public-opinion surveys have scientific and practical value: "To take one example, most people think it's useful to know what are the problems of health access — to sample the whole population and get a sense of what's going on."
The same applies to myriad other questions. How satisfied are people with public schools? Is government ignoring the health needs of the elderly? Do people believe what they read in the newspaper? Has the Enron scandal eroded trust in business? How many people favor war with Iraq over a diplomatic solution? Did candidate Slippery N. Slimey win due to slick ads, a clever choice of running mate, or popular positions on bread-and-butter issues?
Whether the subject is health or politics, surveys are the bedrock of social science.
The same applies to myriad other questions. How satisfied are people with public schools? Is government ignoring the health needs of the elderly? Do people believe what they read in the newspaper? Has the Enron scandal eroded trust in business? How many people favor war with Iraq over a diplomatic solution? Did candidate Slippery N. Slimey win due to slick ads, a clever choice of running mate, or popular positions on bread-and-butter issues?
Whether the subject is health or politics, surveys are the bedrock of social science.
And yet, survey research is getting tougher — do-not-call lists are only the latest hassle. Even though these lists exempt legitimate survey researchers, not everybody who signs up may know that. So when another call comes in from a university researcher, for example, the recipient of the call may feel especially frustrated. Then again, the lists could actually make life easier for researchers, explains G. Donald Ferree, associate director for public-opinion surveys at the University of Wisconsin Survey Center. Those who don't mind participating in surveys — and so have never bothered to sign up on the no-call lists — may be quite receptive to legitimate calls.
Nevertheless, other new obstacles arise from technology. A so-called "privacy manager," for example, allows only preselected numbers to ring through. The increasing use of cellular phones, unlisted numbers, and multiple numbers at one residence all undermine the notion that each household has one phone number — an assumption that, until now, has allowed survey researchers to rely on random-digit dialing.
Nevertheless, other new obstacles arise from technology. A so-called "privacy manager," for example, allows only preselected numbers to ring through. The increasing use of cellular phones, unlisted numbers, and multiple numbers at one residence all undermine the notion that each household has one phone number — an assumption that, until now, has allowed survey researchers to rely on random-digit dialing.
Furthermore, surveys are growing more popular among market researchers, political parties, and interest groups — all of whom may be less interested in accuracy than in, say, moving merchandise or inventing support for parochial positions. The profusion of surveys and sales pitches disguised as surveys all reduce the public's willingness to answer legitimate researchers.
Survey researchers are uncertain how to assess their difficulties, but they are virtually unanimous in taking the problem seriously. "Response rates have declined," confirms Kenneth Goldstein, a professor of political science who is trying to learn about people who do not respond to political polls.
Nora Cate Schaeffer, a professor of sociology who teaches survey design, agrees that response rates are going down, but says, "It's a hard problem to study because you have to hold so many things constant: the study design and the sampling method." Many people blame growing public cynicism, but Schaeffer says that's not been proven.
Survey researchers are uncertain how to assess their difficulties, but they are virtually unanimous in taking the problem seriously. "Response rates have declined," confirms Kenneth Goldstein, a professor of political science who is trying to learn about people who do not respond to political polls.
Nora Cate Schaeffer, a professor of sociology who teaches survey design, agrees that response rates are going down, but says, "It's a hard problem to study because you have to hold so many things constant: the study design and the sampling method." Many people blame growing public cynicism, but Schaeffer says that's not been proven.
One tactic that is gaining approval for improving response rates, Schaeffer says, is paying respondents. Fortunately for budgets, payments must be moderate, since higher fees might coerce low-income people to cooperate against their better judgment.
More commonly, however, researchers mount a "sales pitch" during the initial thirty seconds of a call. The goal is to gain the respondent's trust, by explaining that the only way to know how people really think and act is to contact random samples and ask.
Reaching randomly chosen individuals or households is the only basis for reliable survey research. If the sample is too small, richer or poorer, whiter or more female than the group as a whole, results will be misleading. If rich people have more call-blocking gadgets, they will be underrepresented in a survey that dials phone numbers at random.
More commonly, however, researchers mount a "sales pitch" during the initial thirty seconds of a call. The goal is to gain the respondent's trust, by explaining that the only way to know how people really think and act is to contact random samples and ask.
Reaching randomly chosen individuals or households is the only basis for reliable survey research. If the sample is too small, richer or poorer, whiter or more female than the group as a whole, results will be misleading. If rich people have more call-blocking gadgets, they will be underrepresented in a survey that dials phone numbers at random.
Equally important is a good response rate. If fewer than roughly 60 percent of randomly chosen subjects actually complete the survey, researchers will be forced to guess the opinions of the rest of the sample. There may be systematic reasons why people do not respond: If all Republicrats are irate at the press, they may not answer a newspaper's political survey. If all Demicans are incensed at the federal government, they may reject a government health survey.
Surveys commonly inquire about economics, politics, and feelings. In Wisconsin, the university's Badger Poll, which was established last March, examines all three realms simultaneously. Residents are asked how they think things are going now, and how they expect them to be going in the future. "It's intended to examine every facet of life in Wisconsin, to be the poll of record for the state," says Ferree, the poll's director.
Ferree says the poll will assess optimism and pessimism. Once similar surveys are carried out elsewhere, it will be used to compare the attitudes and feelings of Badgers to residents of other states. The first three polls have shown that Wisconsinites are high on their state: 24 percent say it's one of the best or a very good place to raise a family, and another 38 percent consider it better than most states.
Surveys commonly inquire about economics, politics, and feelings. In Wisconsin, the university's Badger Poll, which was established last March, examines all three realms simultaneously. Residents are asked how they think things are going now, and how they expect them to be going in the future. "It's intended to examine every facet of life in Wisconsin, to be the poll of record for the state," says Ferree, the poll's director.
Ferree says the poll will assess optimism and pessimism. Once similar surveys are carried out elsewhere, it will be used to compare the attitudes and feelings of Badgers to residents of other states. The first three polls have shown that Wisconsinites are high on their state: 24 percent say it's one of the best or a very good place to raise a family, and another 38 percent consider it better than most states.
To get an idea of how university researchers are dealing with these challenges, I visited the UW Survey Center, which employs about 180 interviewers (mainly part-timers), and eavesdropped on the phone. The center contracts with university and government researchers to perform dozens of surveys each year.
Deep into a long telephone interview, a woman from New York State is giving an odd series of answers. She "never" socializes with friends, family, neighbors, or church groups. Ditto for people from work. And all this even though she claims she's not disabled.
The interview is one of thousands for the National Survey of Families and Households, a groundbreaking effort to find out who we are and how our lives are changing. This is the woman's third interview. I ask Kris Hansen '96 if she could be kidding. Hansen, a veteran interviewer who supervises the phone room at the Survey Center, thinks she's being truthful, although some people do lie; several years back he talked to a kid who claimed to be an eight-hundred-pound amputee.
Deep into a long telephone interview, a woman from New York State is giving an odd series of answers. She "never" socializes with friends, family, neighbors, or church groups. Ditto for people from work. And all this even though she claims she's not disabled.
The interview is one of thousands for the National Survey of Families and Households, a groundbreaking effort to find out who we are and how our lives are changing. This is the woman's third interview. I ask Kris Hansen '96 if she could be kidding. Hansen, a veteran interviewer who supervises the phone room at the Survey Center, thinks she's being truthful, although some people do lie; several years back he talked to a kid who claimed to be an eight-hundred-pound amputee.
Hansen and I overhear other interviewers reaching wrong numbers, disconnected phones, and suspicious people. They encounter sudden hang ups or people who might want to talk — tomorrow. Sooner or later, one of the interviewers is bound to reach a "painter" or a "neighbor" who claims not to know the resident. This may be a tactic, Hansen suspects, to dodge bill collectors.
Again and again, the interviewers are flummoxed by caller ID, call waiting, or a privacy manager.
Against the wave of protective electronics and human skepticism, the center's main tool is human relations — smart, articulate interviewers (mumblers need not apply) who believe in their jobs and can explain the importance of survey research. John Stevenson, the Survey Center's associate director, says people tend to cooperate with a survey if they consider the topic interesting or important, and if they trust the researchers to do something useful with the information. This is simple "social exchange theory": if you treat people with respect, they feel a reciprocal obligation to be respectful. Those factors, he says, give a "huge advantage" to the academic researchers who are the center's clients, an advantage that is reflected in response rates that often exceed 70 percent.
Indeed, every so often, we hear the magic words: "Sure, I have time to answer your questions now."
Again and again, the interviewers are flummoxed by caller ID, call waiting, or a privacy manager.
Against the wave of protective electronics and human skepticism, the center's main tool is human relations — smart, articulate interviewers (mumblers need not apply) who believe in their jobs and can explain the importance of survey research. John Stevenson, the Survey Center's associate director, says people tend to cooperate with a survey if they consider the topic interesting or important, and if they trust the researchers to do something useful with the information. This is simple "social exchange theory": if you treat people with respect, they feel a reciprocal obligation to be respectful. Those factors, he says, give a "huge advantage" to the academic researchers who are the center's clients, an advantage that is reflected in response rates that often exceed 70 percent.
Indeed, every so often, we hear the magic words: "Sure, I have time to answer your questions now."
The positive responses reflect not luck, but skill. Hansen says interviewer training focuses on getting response rates that meet the needs of university researchers. Other survey researchers have much lower requirements. For example, Goldstein says, the overnight polls that news organizations use to analyze the state of the electorate often get only a 10 to 15 percent response. "There is so much polling going on, so much direct marketing," he explains.
You might think pollsters could use statistical manipulations to compensate for low response rates, but Goldstein says that can't be done without knowing about those who did not respond. "You can be a smart statistician and take a good random sample, but you will have a problem if the 10, 20, or 50 percent of the population who talk to you are different from the sample as a whole," he says.
You might think pollsters could use statistical manipulations to compensate for low response rates, but Goldstein says that can't be done without knowing about those who did not respond. "You can be a smart statistician and take a good random sample, but you will have a problem if the 10, 20, or 50 percent of the population who talk to you are different from the sample as a whole," he says.
Goldstein's work embodies the significance of survey research. He has studied non-response in political polling — a major factor in the conflicting predictions over the Florida 2000 presidential ballot. He thinks exit polls may help compensate for difficulties with response rates. "One neat thing about exit polls is that you can see who you didn't talk to," he says. "If you hang up the phone, I know nothing about you. But if I see you in person, I know your approximate age, race, and sex." That data, he says, can adjust opinion polls. If exit polltakers find that 45 percent of voters in a certain district are African-American, but only 30 percent of poll respondents were African-American, they can weight their results to improve accuracy.
In China, a key challenge to survey researchers is avoiding government interference. Surveys by foreign universities require Chinese government approval, and sometimes the accompaniment of government minders. Zhongdang Pan PhD'90, an associate professor of communication arts who has surveyed in his native China, knew such an arrangement would discredit his results. "We didn't want the questionnaire to be censored by the government, and didn't want the government to supervise our field operations," he says.
Instead, he worked through contacts in three large Chinese cities to look at the correlation between participation in private leisure activities — once the exclusive province of the Communist government — and individualistic attitudes. Early results, he says, show "clear, statistically significant signs that those who participate more in market-sponsored leisure activity show a greater degree of endorsement of what I'd characterize as individualistic values," Pan says. They tend to agree, for example, that "pursuing individual happiness is the goal of life."
Instead, he worked through contacts in three large Chinese cities to look at the correlation between participation in private leisure activities — once the exclusive province of the Communist government — and individualistic attitudes. Early results, he says, show "clear, statistically significant signs that those who participate more in market-sponsored leisure activity show a greater degree of endorsement of what I'd characterize as individualistic values," Pan says. They tend to agree, for example, that "pursuing individual happiness is the goal of life."
Meanwhile, journalism professor Sharon Dunwoody is researching a highly individualistic realm — the World Wide Web. Ever since UW-Madison initiated The Why Files, an online science magazine, seven years ago, Dunwoody has been trying to overcome the Web's "quick-click" mentality of instant gratification in her quest to understand Web users. Her surveys show that even regular Why Files users are willing to answer only a handful of simple questions. Eventually, Dunwoody expects to learn more about Web users "in the same way we find out who does a lot of other things: traditional surveys."
The World Wide Web is about the only subject absent from the National Survey of Families and Households, the big kahuna of survey research at UW-Madison. James Sweet and colleague Larry Bumpass, also a professor emeritus of sociology, initiated the survey in 1987, and they continue to direct it.
Members of thirteen thousand randomly chosen households were interviewed in person in 1987 and 1988, and again in 1992 through 1994. The third round, now under way, is the first to use phone interviews. Data from these extensive interviews, Bumpass says, "are being distributed with identifying information removed, to researchers all over the world. Hundreds of people are working with them, even as we speak."
Members of thirteen thousand randomly chosen households were interviewed in person in 1987 and 1988, and again in 1992 through 1994. The third round, now under way, is the first to use phone interviews. Data from these extensive interviews, Bumpass says, "are being distributed with identifying information removed, to researchers all over the world. Hundreds of people are working with them, even as we speak."
Sweet and Bumpass, for example, have used the data to cast a new light on family structure, especially regarding marriage, divorce, and cohabitation. Many surveys indicate that people are marrying at older ages, but Sweet says they ignore how families actually form — and thus may mislead anyone trying to understand who is taking care of children. "We know that a large share of cohabiting relationships end in marriage, and that a large share of marriages are preceded by a period of cohabitation," Sweet says. Cohabitation, in other words, may substitute, at least temporarily, for marriage.
The implications for policymakers concerned with the state of the American family are complex, Bumpass says. While one-third of all children are born to unmarried mothers, 40 percent of unmarried mothers live with a man, meaning that a surprisingly large number of kids are living with two parents.
The implications for policymakers concerned with the state of the American family are complex, Bumpass says. While one-third of all children are born to unmarried mothers, 40 percent of unmarried mothers live with a man, meaning that a surprisingly large number of kids are living with two parents.
On the other hand, the start-and-stop nature of cohabitation may impose a higher number of stressful transitions on these children. So despite the fact that the divorce rate has been stable for a long time, family life has become increasingly unstable. By focusing on marriage, "we are missing these transitions in and out that are occurring in the real world," Bumpass says. "How we think about social policy with respect to familial change has to be informed by this reconceptualization of what family life is really like out there."
In this case, the research is producing an accurate picture of family life as the twenty-first century begins, and it's a vitally important picture in a world with do-not-call lists.
"The argument from our research is that the boundaries of family life have blurred, that you really misrepresent family life if you treat it as beginning at marriage," says Bumpass. "To argue that a mother and father living with a child become a family when they get married does not ring true. They already are a family."
As you listen to the phone ring, just remember this: that kind of insight is important. And it's something you can only learn with a well-designed survey.
In this case, the research is producing an accurate picture of family life as the twenty-first century begins, and it's a vitally important picture in a world with do-not-call lists.
"The argument from our research is that the boundaries of family life have blurred, that you really misrepresent family life if you treat it as beginning at marriage," says Bumpass. "To argue that a mother and father living with a child become a family when they get married does not ring true. They already are a family."
As you listen to the phone ring, just remember this: that kind of insight is important. And it's something you can only learn with a well-designed survey.
