Smartphone Tool Measures Users Alertness on the Job

Our level of alertness rises and falls over the course of a workday, sometimes causing our energy to drop and our minds to wander just as we need to perform important tasks.

To help understand these patterns and improve productivity, Cornell researchers have developed a tool that tracks alertness by measuring pupil size, captured through a burst of photographs taken every time users unlock their smartphones.

“Since our alertness fluctuates, if we can find a pattern it will be very useful to manage and schedule our day,” said Vincent W.S. Tseng, a doctoral student in information science and lead author of “AlertnessScanner: What Do Your Pupils Tell About Your Alertness,” presented in September at the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services.

Traditional methods of analyzing alertness tend to be cumbersome, often including devices that must be worn. Researchers in Cornell’s People-Aware Computing Lab, run by Tanzeem Choudhury, associate professor of information science and senior author on the study, wanted to create a way to measure alertness unobtrusively and continuously.

“Since people use their phones very frequently during the day, we were thinking we could use phones as an instrument to understand and measure their alertness,” Tseng said. “And since people’s eyes are affected by their alertness, we were thinking that when people are looking at their phones, we could use a moment to measure their alertness at that point.”

When people are alert, the sympathetic nervous system causes the pupils to dilate to make it easier to take in information. When they’re drowsy, the parasympathetic nervous system causes the pupils to contract.

The paper, co-authored with Saeed Abdullah, an assistant professor in the College of Information Sciences and Technology at Pennsylvania State University, and Cornell information science doctoral student Jean Costa, included two studies conducted over two years. The first study analyzed results from 15 users, who were prompted to take photos of themselves every three hours. Their smartphones needed to have their infrared filters removed to make it easier to detect the contours of the pupil and the iris, particularly for people with dark eyes. The participants were also asked to complete a sleep journal, reporting how many hours they’d slept each night, and to take a phone-based Psychomotor Vigilance Test (PVT) – a five-minute quiz to gauge their reaction time – six times a day.

The photos gave researchers a view of participants’ eyes that they then used to measure pupil size, making allowances for position and lighting, in order to predict a person’s reaction time. This was then compared to the results from the PVT.

The researchers found that the pupil-scanning reliably predicted alertness. But because asking people to remove their phones’ infrared filters was impractical, and prompting them to take photos of themselves throughout the day was too obtrusive, they conducted a second study a year later, when smartphone camera quality had improved enough that they no longer needed to remove the filters.

In that second study, eight participants were given smartphones with high-resolution front-facing cameras that took a burst of 30 photos in one second whenever the phones were unlocked. Users also completed the sleep journal and took the PVTs.

Though the two studies were difficult to compare because of their different methods, both showed that pupil scanning was a reliable means of predicting alertness. The second study, which took the photos passively in a burst, was deemed more practical because it required less work by the user, Tseng said.

Tseng said the AlertnessScanner could be particularly useful in health care, since medical professionals often work long hours doing intricate and important work. For example, clinicians typically look at devices during surgery, and a front-facing camera on the devices could track their alertness throughout procedures.

But understanding alertness patterns could be helpful to people in many kinds of workplaces, Tseng said.

“If you want to get something very important done, then probably you should execute this task while you’re at the peak of your alertness; when you’re in a valley of your alertness, you can do something like rote work,” he said. “You’ll also know the best time to take a break in order to allow your alertness or energy to go back up again.”

The research was partly supported by Intel and the Semiconductor Research Corporation, through a Circadian-Rhythm Aware Computing grant.