Why Do So Many Social-Science Studies Show “Mixed” Results?

Imagine a randomized controlled trial that studied whether a particular medicine was more effective when delivered by a red or a green syringe. Such an experiment is so obviously pointless because (most of the time, anyway) we are confident that it is the chemical properties of the medicine and their interaction with the body—not the color of the syringe that delivers it—that affects the target disease or condition.

Unfortunately, too many social scientists and policymakers make equally confident assumptions about interventions to solve social problems. To even begin to measure program effectiveness, you must first decide which part of the intervention is the medicine and which is the syringe. But addressing poverty, crime, family breakdown, or academic underperformance is far less straightforward than dealing with a bacterial or viral infection.

For example, one of the nonprofit Woodson Center’s mini-grant recipients, Robin, teaches chess to kids in Southeast Washington, D.C. Although work like Robin’s is rarely the subject of research dollars, anecdotal evidence strongly suggests that kids in her program perform better in school and may even enjoy better long-term outcomes than demographically similar individuals who are not in such a program. Let’s assume for the sake of argument that this is true: is chess the medicine or the syringe?

The Intervention

Programs addressing social problems are typically categorized and studied according to the design and structure of the intervention: subsidized early-childhood education, group therapy, afterschool chess programs, home visits to educate parents of underperforming students, and so on. Often, the unspoken assumption is that the professional’s visit to a parent’s home—and the binder of information from which she can read—is the medicine.

Rarely investigated are factors like whether that professional was able to connect meaningfully with the parent in question. But it should be easy to imagine how a study of home visits could produce “mixed” results if, for example, the crucial factor for effectiveness was not the specific phrasing in the contents of the binder, but whether the professionals were able to connect well with parents or not. Assuming the quality of the connection varied from professional to professional, a study that examined only whether the visit took place and the contents of the binder will produce inconclusive results. Incorrect assumptions about which components of an intervention are most important to its success will make data gathered from the most-expensive studies useless.

Life would be much easier if chess were the medicine: then you could simply mandate that chess become a required subject in school, like math or reading. You could buy chess sets for every school or—better yet—put chess apps on those schools’ ubiquitous tablets and laptops. Then every child in America would reap the benefits that the students in Robin’s program enjoy.

But of course, in this case, chess is simply a tool to activate inchoate impulse control, strategic thinking and decision-making, and delayed gratification in the students. Program participants observe and experience these skills, attitudes, and character qualities modeled by Robin and the other program leaders, and chess gives them a tangible way to put them into practice. And while chess may have some characteristics that may make it uniquely suited to deliver these particular benefits, it should be readily apparent that similar benefits could be delivered through golf, gardening, piano, or any number of focused activities.

The Woodson Center, of which I am chief operating officer, has spent more than four decades identifying, resourcing, and studying highly effective grassroots work in some of the country’s most-challenging neighborhoods, and we have found that the character and habits of almost anyone can be positively transformed through modeling and cooperative/participatory activities. There are some inconvenient caveats here: those doing the modeling need to possess moral authority and the social trust of the participants, and they need to make a long-term commitment to them. It is extremely helpful if those doing the modeling have experienced the same or similar struggles as the participants. A natural outworking of the program is often the formation of a moral community in which participants support and encourage one another. But the structure and nature of the activities they engage in together vary widely, and the components they all have in common are rarely the subject of study, or even interest.

It’s worth noting that while most high-impact programs the Woodson Center resources do (eventually) employ some sort of written curriculum, those same curricula could not be placed in different hands or in a different context and automatically replicate the original program’s effectiveness. And very rarely do the leaders with whom we work lecture in front of a whiteboard holding a binder. At the risk of straining the metaphor, if chess can be an effective syringe to deliver a medicine that activates character development, the lecture and binder deliver the medicine by pouring it over students’ heads and hoping some seeps in.

The Transmission

Again, it would be nice if lectures and binders information about something like good decision-making functioned on bad decision-making like antibiotics on a bacterial infection. But they do not. Textbooks alone don’t even address deficits in academic knowledge. We have a greater variety of textbooks and course options than ever before, yet test scores continue to sink. In the case of underdeveloped character or unhelpful habits, the “medicine” is good character and healthy habits, and the most “bio-available” medicine can be found in the lives of actual human beings. It can be communicated through stories, which is why the Woodson Center’s Black History and Excellence lessons share the incredible lives of lesser-known heroes of the past. But most often, that medicine is transmitted through trusting human relationships. Not typically between professional and patient or philanthropist and charity recipient—but most often, between members of the same community. The strength of relationships is difficult to quantify, but no study that ignores them can be expected to produce anything better than “mixed” results.

This article first appeared in the Giving Review on May 8, 2024.