Keith E. Stanovich University of Toronto
It is fine to point out how the existence of placebo effects renders testimonials useless as evidence, but we must recognize another obstacle that prevents people from understanding that testimonials cannot be accepted as proof of a claim. Social and cognitive psychologists have studied what is termed the vividness effect in human memory and decision making (see Baron, 1998; Nisbett & Ross, 1980). When faced with a problem-solving or decision-making situation, people retrieve from memory the information that seems relevant to the situation at hand. Thus, they are more likely to use the facts that are more accessible to solve a problem or make a decision. One factor that strongly affects accessibility is the vividness of information.
The problem is that there is nothing more vivid or compelling than sincere personal testimony that something has occurred or that something is true. The vividness of personal testimony often overshadows other information of much higher reliability. How often have we carefully collected information on different product brands before making a purchase, only to be dissuaded from our choice at the last minute by a chance recommendation of another product by a friend or an advertisement? Car purchases are a typical example (Nisbett & Ross, 1980). We may have read surveys of thousands of customers in Consumer Reports and decided on car X. After consulting the major automotive magazines and confirming that the experts also recommend car X, we feel secure in our decision—until, that is, we meet a friend at a party who knows a friend who knows a friend who bought an X and got a real lemon, spent hundreds on repairs, and would never buy another. Obviously, this single instance should not substantially affect our opinion, which is based on a survey of thousands of owners and the judgment of several experts. Yet how many of us could resist the temptation to overweight this evidence?
The auto purchase situation illustrates that the problems created by vivid testimonial evidence are not unique to psychology. For example, Stephen Budiansky (1984), Washington correspondent of the British science journal Nature, has summarized the situation in medicine with a statement that reinforces many of the points about science that we have discussed in previous chapters:
Science eschews the personal. Although it is commonplace to ascribe this tendency to some fundamental coldness on the part of scientists, in fact it is really one of the great intellectual triumphs of the 20th century that scientists have learned to discount the experiences of individuals when searching for cause and effect in the natural world. The health sciences have had a particularly rough time of it; people tend to get sick for a variety of reasons and people more often than not get better no matter what “cure” is prescribed. The apparently undying popularity of quack remedies, invariably supported by testimonials from satisfied customers, is vivid proof of how hard we find it to look beyond personal experience. (p. 7)
Instances of how vividness affects people’s opinions are not hard to find. Reporter Haynes Johnson (1991) wrote of how President Reagan came to recognize the severity of the AIDS problem:
[Reagan] had not realized the seriousness of AIDS until July 1985, when he saw a news report disclosing that the actor Rock Hudson had died of the disease. This was more than five years after AIDS had been identified, thousands of Americans had been infected, and AIDS had been the subject of intense national publicity. When Reagan saw the news report about Hudson’s death, he asked [Brigadier] General Hutton [a former doctor] to tell him about the disease. (p. 454)
In short, the constant news reports and statistics on the increasing numbers of AIDS deaths had not attracted the president’s attention, but the report of a single person who was known to him did. Similarly, writer Michael Lewis (1997) describes how conservative commentator George Will—a notorious opponent of government regulation—published a column calling for mandatory air bags after seeing a death in a car crash outside of his home.
Imagine that you saw the following headline one Friday morning in your newspaper: “Jumbo Jet Crash: 413 killed.” Goodness, you might think, what a horrible accident. What a terrible thing to happen. Imagine though, that the following Thursday you got up and your newspaper said, “Another Jumbo Jet Disaster: 442 Die.” My God, you might think. Not another disaster! How horrible. What in the world is wrong with our air traffic system. And then imagine—please imagine as best you can—getting up the following Friday and seeing in the paper: “Third Tragic Airline Crash: 431 Dead.” Not only you but the nation would be beside itself. A federal investigation would be demanded. Flights would be grounded. Commissions would be appointed. Massive lawsuits would be filed. Newsweek and Time would run cover stories. It would be the lead item on television news programs for several days. Television documentaries would explore the issue. The uproar would be tremendous.
But this is not an imaginary problem. It is real. A jumbo jet does crash every week. Well, not one jet, but a lot of little jets. Well, not little jets really, but little transportation devices. These devices are called passenger automobiles. And approximately 457 people die in them each week in the United States (23,800 people each year—37,500 if we count trucks and motorcycles; National Safety Council, 1990), enough to fill a jumbo jet.
A jumbo jet’s worth of people die in passenger cars on our nation’s highways every week, yet we pay no attention. The “Jumbo Jet’s Worth of People Who Die” are not presented to us in a vivid way by the media. Hence, the 457 people who die each week in passenger cars (plus the additional 263 who die each week in trucks and on motorcycles) have no vividness for us. We don’t talk about them at the dinner table as we do when a jet goes down and kills a lot of people. We do not debate the safety and necessity of car travel as we would the safety of the air traffic system if a jumbo jet crashed every week killing 400 people each time. The 457 are not on the news because they are distributed all over the country and thus are a statistical abstraction to most of us. The media do not vividly present to us these 457 deaths because they do not happen in the same place. Instead, the media present to us (occasionally) a number (e.g., 457 per week). This should be enough to get us thinking, but it is not. Driving automobiles is an extremely dangerous activity, however it is measured (Lichtenstein, Slovic, Fischhoff, Layman, & Combs, 1978; National Safety Council, 1990), yet there has never been a national debate about its risk relative to the benefits involved. Is this an acceptable toll for a suburban lifestyle that demands a lot of driving? We never ask the question because no problem is recognized. No problem is recognized because the cost is not presented to us in a vivid way, as is the cost of airline crashes.
Think of the absurdity of the following example. A friend drives you 20 miles to the airport where you are getting on a plane for a trip of about 750 miles. Your friend is likely to say, “Have a safe trip,” as you part. This parting comment turns out to be sadly ironic, because your friend is three times more likely to die in a car accident on the 20-mile trip back home than you are on your flight of 750 miles (National Safety Council, 1990). It is the vividness problem that accounts for the apparent irrationality of person A’s wishing person B safety, when it is person A who is in more danger.
Misleading personal judgments based on the vividness of media-presented images are widespread. One study (MacDonald, 1990; see also Cole, 1998) surveyed parents to see which risks to their children worried them the most. Parents turned out to be most worried about their children’s being abducted, an event with a probability of 1 in 700,000. In contrast, the probability of their child’s being killed in a car crash, which the parents worried about much less, is well over 100 times more likely than their being abducted (Paulos, 1988). Of course, the fears of abduction are mostly a media-created worry. The results actually suggest that, largely because of perceptions skewed by vividness effects, “American parents seem to worry about events that are least likely to happen” (MacDonald, 1990). One of the researchers lamented that this focus on worries that are “currently fashionable” misdirects the attention of parents and leads them to “ignore areas where they could have more impact, like school performance, television viewing habits, drug use and car safety” (MacDonald, 1990).
Writer Katherine Dunn (1993) recounted the fear she felt as a parent in the mid-1980s when stories of abducted children were sweeping the country. Rumors were spread that as many as 70,000 children had been snatched in malls and used by pornographers and/or tortured by strangers. Dunn reported tracking down the actual evidence by calling the FBI. It turned out that the number 70,000 referred not to abductions but to the number of runaways and children involved in custody disputes. Regarding children being abducted by strangers—which was the heart of the rumor sweeping the country—the FBI had recorded seven such cases nationwide. It was obviously not the number of these cases that had prompted such parental fears, but the vividness of the descriptions of the harm to the children. The vastly greater danger to any child in the passenger seat of an automobile (even when buckled up) was simply not as vivid.
The previous anecdote calls to mind science writer K. C. Cole’s (1998) description of the ridiculous image of a woman driving down the street with a young child romping in the front seat, arriving at a shopping mall, and then getting out and grabbing the child’s hand very tightly as she worries about child kidnappers. Cole discusses some of the reasons why people misassess risk. One factor involved is that people exaggerate risks that are perceived to be beyond their control. This is one reason why airline accidents—with people strapped in seats and dependent on the skills of others, on the performance of technology, and on the weather—seem so unacceptable to people. Cole (1998) reports on a poll taken after a TWA crash in which a large majority of people were willing to pay $50 more for a round-trip airline ticket if it increased safety. Yet the same people resist safety features in automobiles that would provide a much greater increment in safety at a much lower cost.
Writer Peter Boyer (1999) describes how similar misperceptions of risk are fostered by the gun lobby in the United States which tries to keep the public focused on vivid cases of “intruders” coming through the doors of households. The not-so-subtle subtext here is that one lowers one’s risk by having a gun to protect yourself. Boyer (1999) points to the irony that the gun industry tries to focus attention on “guns in the hands of bad people” when the actual statistics show that the real problem is “guns in the hands of good people.” Criminals do not account for most gun deaths in this country. There are actually more suicides with guns than there are homicides with guns. Most gun deaths are unintentional shootings and suicides—which is why research indicates that bringing a gun into a home actually increases family risk.
Psychologists have extensively studied the tendency for people’s judgments to be dominated by a single, salient example when more accurate information is available. Hamill, Wilson, and Nisbett (1980) showed subjects a taped interview with a prison guard. Some subjects viewed interviews with a guard whose responses and manner suggested that he was a truly humane individual. Others viewed an interview with an extremely inhumane and disagreeable guard. The interviews modified opinions about guards in a positive or negative direction, depending on which interview had been viewed. More interesting was the fact that half the subjects received information indicating that the interview they had witnessed was a part of a large study of prison guards and that the guard they had seen was highly typical of all guards in the prison system. The other subjects were told that the guard they had viewed was highly atypical of all guards and that his behavior and opinions were very extreme, either positively or negatively, depending on the interview. The information about whether the interview they had witnessed was typical or not had no effect on the subjects’ opinions about prison guards. Knowledge of the statistical reliability of the interview was overwhelmed by the reactions to the interview itself.
Wilson and Brekke (1994) demonstrated how insidious the vividness problem is and also how it influences actual consumer behavior. They investigated how people were influenced by two different types of information about two different brands (brand A and brand B) of condom. One type of information was a survey and analysis in Consumer Reports magazine, and the other was the opinions of two university students about their preferences for condom brands. First, Wilson and Brekke surveyed a group of subjects on which type of information they would want to be influenced by. Over 85 percent of the subjects said that they would want to be more influenced by the Consumer Reports article than by the opinions of the two students. A similar group of subjects were then recruited for a study in which they were told that they would be given, free of charge, some condoms of their own choosing. The subjects were told that they could consult either or both of two types of information: a survey and analysis in Consumer Reports magazine and the opinions of two university students about their preferences. Even though less than 15 percent of a similar group of subjects wanted to be influenced by the opinions of the two students, 77 percent of the subjects requested both types of information. Apparently the subjects could not resist seeing the testimonials even though they did not believe that they should be affected by them. And they were indeed affected by them. When the subjects chose to see both types of information and the recommendations of the two sources of information differed, 31 percent of the subjects chose the brand of condom recommended in the students testimonials over the brand recommended by Consumer Reports.
Another example of how people respond differently to vivid anecdotal information comes from the media coverage of the Vietnam War in the mid to late 1960s. As the war dragged on and the death toll of Americans killed continued without an end in sight, the media took to reporting the weekly number of American service personnel who had been killed that week. Week after week, the figure varied between 200 and 300, and the public, seemingly, became quite accustomed to this report. However, one week a major magazine published a spread, running on for several pages, of the individual pictures of those persons who had died in the previous week. The public was now looking, concretely, at the approximately 250 individual lives that had been lost in a typical week. The result was a major outcry against the toll that the war was taking. The 250 pictures had an effect that the weekly numbers had not had. But we, as a society, must overcome this tendency not to believe numbers—to have to see everything. Most of the complex influences on our society are abstractions that are accurately captured only by numbers. Until the public learns to treat abstractions as seriously as images, public opinion will be as fickle as the latest image to flicker across the screen.
But it is not only the public that is plagued by the vividness problem. Experienced clinical practitioners in both psychology and medicine struggle all the time with the tendency to have their judgment clouded by the overwhelming impact of the single case. Writer Francine Russo (1999) describes the dilemma of Wille Anderson, an oncologist at the University of Virginia. Anderson is an advocate of controlled experimentation and routinely enrolls his patients in controlled clinical trials, but he still struggles with his own reactions to single, salient cases that have an emotional impact on his decisions. Despite his scientific orientation, he admits that “when it’s real people looking you in the eye, you get wrapped up in their hopes and your hopes for their hopes, and it’s hard” (p. 36). But Anderson knows that sometime the best thing for his patients is to ignore the “real person looking you in the eye” and go with what the best evidence says. And the best evidence comes from a controlled clinical trial, not from the emotional reaction to that person looking you in the eye.
The vividness problem is a difficulty we all face when evaluating evidence. And in an environment saturated with media images, it is becoming increasingly difficult for society not to be dominated by the images and instead to solve its problems based on valid evidence. Writer Barry Glassner (1999) describes an all too familiar example. On an Oprah Winfrey program in 1995, information was being presented on a surgical intervention (which will here remain unnamed so as not to contribute further to a vividness effect) that had caused some controversy because some people had claimed to have been injured by it. Evidence was being presented that studies from the Mayo Clinic, Harvard, and the University of Michigan had shown no overall danger from the procedure—at which point a woman claiming to have been injured jumped up from the audience and shouted “We are the evidence. The study is us sitting here!” (Glassner, 1999, p. 164). Which do you think the television audience of millions remembered better—the study from the Mayo Clinic or the woman screaming that she had been injured?
Even though we all are prone to overestimating the value of testimonial and other single-case evidence, we can become more self-aware and more conscious of when our opinions are being overwhelmed by personal testimony or particularly vivid single cases. A column by writer Remar Sutton (1987) illustrates quite well how becoming more aware of these influences can help. He wrote about the beginning of his attempt at a major weight loss by describing how “a diving buddy” had recommended a popular book, Fit for Life. Sutton described seeing the authors on a talk show and mentioned that they “appeared awfully sincere, sounded authoritative, and attacked some traditional nutritional thinking.” He recalled that “every time a critic attacked them, they rebutted effectively with their own experiences [italics added]. All of that, and the talk show host said he lost weight and felt better with their plan, too.” In short, Sutton admitted in his column that he had become interested in the ideas in this book through his hearing of personal experiences and testimonials.
But fortunately Sutton had acquired the critical thinking skills that this book is trying to teach. He researched the information given in the book, looking for confirming evidence—not on talk shows and in popular magazines but in the peer-reviewed scientific literature. He concluded that “too many ideas presented in the book were simply not backed up by long-term controlled scientific research that was statistically valid. Indeed, most of these ideas were counter to statistically valid research.” Furthermore, in the course of his research, Sutton found out that the two authors had received their nutrition certificates from a “school” in Austin, Texas, that was nothing more than a post office box!
Sutton had the insight to realize that he had been close to being “sucked in” by the quackery in the book and to ask himself, “So why did I nearly fall for the Fit for Life approach?” His answer provides a good review of several of the pitfalls described previously. He admitted that he “trusted [his] diving friend’s recommendation based on his personal experience.” He had also liked the way the authors “presented themselves as much as their information” and that he “believed that television... and large numbers of book sales made the message of Fit for Life legitimate.”
Sutton’s conclusion sums up the vividness issue nicely:
Like it or not, personal testimonials and the sincerity of a person’s presentation do not necessarily make any product or opinion accurate. Carefully designed, properly controlled, replicable studies which prove statistically valid over the long term are the only assurance any of us have that a diet or medical treatment... can benefit us.
Sutton ended by stating that he was still not hostile to new or innovative approaches to dieting, and he urged proponents of new approaches to write him. But he reminded them, “I’ll be happy to present the opposite point of view on diet and fitness as long as the proponents of these views can point me to the reliable scientific studies that provide statistical validity for that position. Please do write, but don’t forget those bibliographies.”
In summary, the problems created by reliance on testimonial evidence are ever-present. The vividness of such evidence often eclipses more reliable information and obscures understanding. Psychology instructors worry that merely pointing out the logical fallacies of reliance on testimonial evidence is not enough to provide a deep understanding of the pitfalls of this type of data. What else can be done? Is there any other way to get this concept across to people? Fortunately there is an alternative—an alternative somewhat different from the academic approach. The essence of this approach is to fight vividness with vividness. To hoist testimonials by their own petard! To let testimonials devour themselves with their own absurdity. A practitioner of this approach is the one, the only, the indubitable Amazing Randi!
James Randi is a magician and jack-of-all trades who was given a MacArthur Foundation “genius” grant. For many years, he has been trying to teach the public some basic skills of critical thinking. The Amazing Randi (his stage name) has done this by exposing the fraud and charlatanism surrounding claims of “psychic” abilities. Although he has uncovered many magicians and conjuros masquerading as psychics, he is best known for exposing the trickery of Uri Geller, the psychic superstar of the 1970s. Bursting on the scene with his grand claims of psychic powers, Geller captivated the media to an extraordinary degree. He was featured in newspapers, on television shows, and in major news magazines on several continents. Randi detected and exposed the common and sometimes embarrassingly simple magic tricks that Geller used to perform his psychic “feats,” which included bending keys and spoons, and starting watches—mundane fare for a good magician. Since the Geller exposé, Randi has continued to use his considerable talents in the service of the public’s right to know the truth in spite of itself by exposing the fallacies behind ESP, biorhythms, psychic surgery, extraterrestrials, levitation, and other pseudosciences, all marvelously detailed with great humor in his book Flim-Flam (1980; see also his book The Faith Healers, 1987, in which he exposed several bogus religious “healers” as frauds).
One of Randi’s minor diversions consists of demonstrating how easy it is to garner testimonial evidence for any preposterous event or vacuous claim. His technique is to let people be swallowed up in a trap set by their own testimonials. Randi makes much use of that fascinating American cultural institution, the talk show, often appearing as a guest in the guise of someone other than himself. On a New York show a few years ago, he informed the audience that, while driving through New Jersey earlier in the day, he had seen a formation of orange V-shaped objects flying overhead in a northerly direction. Within seconds, as Randi put it, “the station switchboard lit up like an electronic Christmas tree.” Witness after witness called in to confirm this remarkable sighting. Unfortunately for them, the “sighting” was only a product of Randi’s imagination. Callers provided many details that Randi had “omitted,” including the fact that there had been more than one pass of the “saucers.” This little scam illustrates how completely unreliable are individual reports that “something happened.”
In Winnipeg, Canada, Randi appeared on a radio show as an “astrographologist.” A week earlier, listeners had been told to send in their handwriting samples and birth dates. Three were chosen and were contacted so that Randi could assess the “readings” of their personalities. He was hugely successful, receiving ratings from the listeners of 9, 10, and 10 on a 1-to-10 scale of accuracy. Randi did eventually reveal the secret of his method to the radio listeners. He had read, word for word, the “readings” that astrologer Sydney Omarr had given to three members of the audience of a recent Merv Griffin Show.
On a different radio show, Randi demonstrated the basis of the popularity of another pseudoscience: biorhythms (Hines, 1998). One listener agreed to keep a day-by-day diary and compare it with a two-month biorhythm chart that had been prepared especially for her. Two months later, the woman called back to inform the audience that biorhythms should be taken very seriously because her chart was more than 90 percent accurate. Randi had to inform her of the silly mistake made by his secretary, who had sent someone else’s chart to her, rather than her own. However, the woman did agree to evaluate the correct chart, which would be mailed to her right away, and to call back. A couple of days later, the woman called back, relieved. Her own chart was just as accurate—in fact, even more accurate. On the next show, however, it was discovered that, whoops, another error had been made. The woman had been sent Randi’s secretary’s chart, rather than her own!
Randi’s biorhythm and astrographologist scams are actually examples of a phenomenon that has been termed the P. T. Barnum effect (Barnum, the famous carnival and circus operator, coined the statement “There’s a sucker born every minute”). This effect has been extensively studied by psychologists (e.g., Dickson & Kelly, 1985), who have found that the vast majority of individuals will endorse generalized personality summaries as accurate and specific descriptions of themselves. The Barnum effect makes it easier to generate testimonials and, of course, shows why they are worthless. There are certain sets of statements and phrases that most people see as applicable to themselves (many of these phrases have been studied by psychologists; see, e.g., Dickson & Kelly, 1985; Hyman, 1981; Marks & Kammann, 1980). Anyone can feed them to a “client” as individualized psychological “analysis.” The client is usually very impressed by the individualized accuracy of the “personality reading,” not knowing that the same reading is being given to everyone. Of course, the Barnum effect is the basis of belief in the accuracy of palm readers and astrologists.
What Randi is trying to do in these little scams is to teach people a lesson about the worthlessness of testimonial evidence. He consistently demonstrates how easy it is to generate testimonials in favor of just about any bogus claim (Randi, 1983). For this reason, presenting a testimonial in support of a particular claim is meaningless. Only evidence from controlled observations is sufficient to actually test a claim....
Excerpted from How to Think Straight About Psychology, 6/e, chapter 4, pp. 61-70. © 2001 by Allyn & Bacon. Reprinted by permission.