Facebook Files Patent For Technology That Can Identify Your Family Based on Photos

It’s all about serving ads to entire families.

Facebook filed a patent for technology that could target entire families with ads. The tech uses photos and various other data collected by the social network.

In a patent filed last May and made public Thursday, the company describes a system enabled by “deep learning techniques” meant to identify groups of people living under the same roof and their relationships to each other. It uses image captions, hashtags, as well as ISP and search data to build a cache of information that’s stored and later fed to its network of advertisers. The technology would ostensibly apply to at least some of Facebook’s proprietary apps, such as Instagram.

As the filing states:

Examples of image data of a user include profile photos of the user, e.g., profile photos of the same user on different online systems, e.g., FACEBOOK.TM. and INSTAGRAM.TM

The technology builds on Facebook’s “household audience” ad tech unveiled in June, which allows brands the option of targeting specific people in a household. The patent makes clear Facebook’s intent to target entire families with more clickable ads than it currently serves, stating:

“Existing solutions of content delivery to a target household are not effective … Without such knowledge of a user’s household features, most of the content items that are sent to the user are poorly tailored to the user and are likely ignored by the user.”

Contained in the patent is an example of how the technology would work: In one figure, Facebook predicts the number of people in a household after analyzing a male user’s photos. Since the user regularly posts photos with the same two women—and gets tagged in other users’ photos with the same two women—Facebook deduces the three people are a family. The software cites the trio’s shared devices and a caption with the words “my angel” to extrapolate their family ties.

The patent also explains how the process caters to Facebook’s advertising platform, by keeping “the predictions with a user profile within the online system,” then displaying “content items targeting the user based on the predictions.”

Facebook’s decision to make its patent filing public is suspect, given the continuing string of high-profile crises rocking the company over the past calendar year. The company’s invasive data-collecting practices have been central to the fallout: The Cambridge Analytica data-mining scandal prompted Senate oversight earlier this year, while it drew further scorn from its decision to release a video-chatting device without the assurance of privacy immediately following a massive security breach.

Though it’s unclear if Facebook will ever employ the technology, the social network’s advertising machine maintains an overarching influence on our daily lives, and it doesn’t appear to be changing soon.