The Role of Qualitative Data in Academic Field Experiment Research

This will be a “big think” blog post that shares my thinking about this March 2022 Nature Human Behavior paper titled “The data revolution in social science needs qualitative research”.

Permit me to focus on one example.  Consider a sample of 5,000 equally talented and ambitious 18 year olds.  Each has graduated from High School and each is considering applying to the University of California.  The students differ that some are Asian and some are Hispanic.  To simplify, let’s assume everyone is a member of one of these two groups.  

The researcher observes that 72% of Asians 18 year olds in the sample apply to the University of California while only 32% of Hispanic 18 year olds in the sample do.

A qualitative researcher would take the next step of interviewing a random subset of the Asians and the Hispanics in this sample to ask them various questions about their beliefs, life goals, family circumstances and several other nuances that cannot be captured by a standard demographic survey.

A field experiment researcher would proceed with a different strategy.  She would take a new sample of 5,000 equally talented and ambitious 18 year old Hispanic and Asian students in the next year and randomly assign a subset of each group to a specific treatment such as a 20 minute information course on understanding the application process to the University of California and the gains to attending an elite 4 year college.  The control group would not receive any of this information

The field experiment researcher would observe the following pieces of information;

X1 =  percentage of Asians who apply to the University of California given random assignment to the treatment group.

X2 =  percentage of Asians who apply to the University of California given random assignment to the control group.

X3 =  percentage of Hispanics who apply to the University of California given random assignment to the treatment group.

X4 =  percentage of Hispanics who apply to the University of California given random assignment to the control group.

Suppose for this new sample of young people that the researcher observes the following facts;

X1 =  74%

X2 =  72%

X3 = 70%

X4 =  32%

These data immediately indicate that the intervention had a much larger impact on the Hispanic teens than on the Asian teens.  The intervention closed the gap.

What is My Point?

I am interested in asymmetries in social science.  Economists assume that people know themselves and have a life plan and a “conception of the good life” and they strategically make their choices such as who they marry, where they live and how much education to attain based on their plan.  The observer knows that she does not know what is each person’s goal. Social scientists learn about people based on the choices we observe them make.

In the example sketched out above, a field experiment researcher seeks to explain the “education gap” and to test for what might be cost-effective strategies for closing this gap.  How did the field experiment researcher choose the specific intervention that turned out to be effective?

One answer is introspection. Another answer is journalistic accounts explaining why some talented teens do not apply to great schools.  Another answer is qualitative research on a larger sample of people.

The question here pertains to how quickly will the Big Data researcher zero in on the right treatment to pilot?   If the Big Data field experiment researcher is baffled concerning what treatment to pilot, then qualitative research is crucial for narrowing down the set of strategies.  

Given the publication incentives of academic researchers and given their finite research budgets, they have strong incentives to pursue treatment effect designs that are effective. Researchers cannot publish papers that say; “I tried this crazed treatment and it turned out to have no effect.”.   The self doubt of the field experiment researcher leads her to pursue qualitative strategies (at least on a small sample) to reduce her risk exposure of investing her time and $ in a project that doesn’t yield credible statistically significant results.

“Dynamic” Facts in the Social Science

Note that the effective intervention means that the “Old fact” (that eligible Asians are much more likely to apply to the great colleges than eligible Hispanics) is no longer a fact going forward as the effective intervention brings about convergence.   In Physics, this does not occur.   

Final Point

When the field experiment researcher knows that she does not know the causes of behavior (so in this case why Hispanics are not often applying to the UCs), she has an incentive to invest in qualitative research.