Silicon Valley is building a Chinese-style social credit system that will control your life

There was a very frightening Black Mirror episode on this very thing. Black Mirror’s third-season episode “Nosedive” imagined a horrible world controlled by a social credit system system. It even featured a scene in which star Bryce Dallas Howard is ejected from an airport for having too low a social score to secure a seat on a flight.

In 2014, the State Council, China’s governing cabinet, publicly called for the establishment of a nationwide tracking system to rate the reputations of individuals, businesses, and even government officials. The aim is for every Chinese citizen to be trailed by a file compiling data from public and private sources by 2020, and for those files to be searchable by fingerprints and other biometric characteristics. The State Council calls it a “credit system that covers the whole society.”

Doing volunteer work, donating blood, and recycling can all boost one’s social credit score, while incurring debt or criticizing the government can render you blacklisted, unable to buy property, take out loans, and now, engage in some forms of mass travel. (Esquire)

Silicon Valley is building a Chinese-style social credit system

In China, scoring citizens’ behavior is official government policy. U.S. companies are increasingly doing something similar, outside the law.

By Mike Elgan, Fast Company, August 26, 2019:

Have you heard about China’s social credit system? It’s a technology-enabled, surveillance-based nationwide program designed to nudge citizens toward better behavior. The ultimate goal is to “allow the trustworthy to roam everywhere under heaven while making it hard for the discredited to take a single step,” according to the Chinese government.

In place since 2014, the social credit system is a work in progress that could evolve by next year into a single, nationwide point system for all Chinese citizens, akin to a financial credit score. It aims to punish for transgressions that can include membership in or support for the Falun Gong or Tibetan Buddhism, failure to pay debts, excessive video gaming, criticizing the government, late payments, failing to sweep the sidewalk in front of your store or house, smoking or playing loud music on trains, jaywalking, and other actions deemed illegal or unacceptable by the Chinese government.

It can also award points for charitable donations or even taking one’s own parents to the doctor.

Punishments can be harsh, including bans on leaving the country, using public transportation, checking into hotels, hiring for high-visibility jobs, or acceptance of children to private schools. It can also result in slower internet connections and social stigmatization in the form of registration on a public blacklist.

China’s social credit system has been characterized in one pithy tweet as “authoritarianism, gamified.”

Authoritarianism, gamified. https://t.co/V7JARJwU7t ht @VitalikButerin @FrankPasquale

— Steve Randy Waldman (@interfluidity) October 4, 2015

At present, some parts of the social credit system are in force nationwide and others are local and limited (there are 40 or so pilot projects operated by local governments and at least six run by tech giants like Alibaba and Tencent).

Beijing maintains two nationwide lists, called the blacklist and the red list—the former consisting of people who have transgressed, and the latter people who have stayed out of trouble (a “red list” is the Communist version of a white list.) These lists are publicly searchable on a government website called China Credit.

The Chinese government also shares lists with technology platforms. So, for example, if someone criticizes the government on Weibo, their kids might be ineligible for acceptance to an elite school.

Public shaming is also part of China’s social credit system. Pictures of blacklisted people in one city were shown between videos on TikTok in a trial, and the addresses of blacklisted citizens were shown on a map on WeChat.

Some Western press reports imply that the Chinese populace is suffocating in a nationwide Skinner box of oppressive behavioral modification. But some Chinese are unaware that it even exists. And many others actually like the idea. One survey found that 80% of Chinese citizens surveyed either somewhat or strongly approve of social credit system.

It can happen here

Many Westerners are disturbed by what they read about China’s social credit system. But such systems, it turns out, are not unique to China. A parallel system is developing in the United States, in part as the result of Silicon Valley and technology-industry user policies, and in part by surveillance of social media activity by private companies.

Here are some of the elements of America’s growing social credit system.

Insurance companies

The New York State Department of Financial Services announced earlier this year that life insurance companies can base premiums on what they find in your social media posts. That Instagram pic showing you teasing a grizzly bear at Yellowstone with a martini in one hand, a bucket of cheese fries in the other, and a cigarette in your mouth, could cost you. On the other hand, a Facebook post showing you doing yoga might save you money. (Insurance companies have to demonstrate that social media evidence points to risk, and not be based on discrimination of any kind—they can’t use social posts to alter premiums based on race or disability, for example.)

The use of social media is an extension of the lifestyle questions typically asked when applying for life insurance, such as questions about whether you engage in rock climbing or other adventure sports. Saying “no,” but then posting pictures of yourself free-soloing El Capitan, could count as a “yes.”

PatronScan

A company called PatronScan sells three products—kiosk, desktop, and handheld systems—designed to help bar and restaurant owners manage customers. PatronScan is a subsidiary of the Canadian software company Servall Biometrics, and its products are now on sale in the United States, Canada, Australia, and the United Kingdom.

PatronScan helps spot fake IDs—and troublemakers. When customers arrive at a PatronScan-using bar, their ID is scanned. The company maintains a list of objectionable customers designed to protect venues from people previously removed for “fighting, sexual assault, drugs, theft, and other bad behavior,” according to its website. A “public” list is shared among all PatronScan customers. So someone who’s banned by one bar in the U.S. is potentially banned by all the bars in the U.S., the U.K., and Canada that use the PatronScan system for up to a year. (PatronScan Australia keeps a separate system.)

Judgment about what kind of behavior qualifies for inclusion on a PatronScan list is up to the bar owners and managers. Individual bar owners can ignore the ban, if they like. Data on non-offending customers is deleted in 90 days or less. Also: PatronScan enables bars to keep a “private” list that is not shared with other bars, but on which bad customers can be kept for up to five years.

PatronScan does have an “appeals” process, but it’s up to the company to grant or deny those appeals.

Uber and Airbnb

Thanks to the sharing economy, the options for travel have been extended far beyond taxis and hotels. Uber and Airbnb are leaders in providing transportation and accommodation for travelers. But there are many similar ride-sharing and peer-to-peer accommodations companies providing similar services.

Airbnb—a major provider of travel accommodation and tourist activities—bragged in March that it now has more than 6 million listings in its system. That’s why a ban from Airbnb can limit travel options.

Airbnb can disable your account for life for any reason it chooses, and it reserves the right to not tell you the reason. The company’s canned message includes the assertion that “This decision is irreversible and will affect any duplicated or future accounts. Please understand that we are not obligated to provide an explanation for the action taken against your account.” The ban can be based on something the host privately tells Airbnb about something they believe you did while staying at their property. Airbnb’s competitors have similar policies.

It’s now easy to get banned by Uber, too. Whenever you get out of the car after an Uber ride, the app invites you to rate the driver. What many passengers don’t know is that the driver now also gets an invitation to rate you. Under a new policy announced in May: If your average rating is “significantly below average,” Uber will ban you from the service.

WhatsApp

You can be banned from communications apps, too. For example, you can be banned on WhatsApp if too many other users block you. You can also get banned for sending spam, threatening messages, trying to hack or reverse-engineer the WhatsApp app, or using the service with an unauthorized app.

WhatsApp is small potatoes in the United States. But in much of the world, it’s the main form of electronic communication. Not being allowed to use WhatsApp in some countries is as punishing as not being allowed to use the telephone system in America.

What’s wrong with social credit, anyway?

Nobody likes antisocial, violent, rude, unhealthy, reckless, selfish, or deadbeat behavior. What’s wrong with using new technology to encourage everyone to behave?

The most disturbing attribute of a social credit system is not that it’s invasive, but that it’s extralegal. Crimes are punished outside the legal system, which means no presumption of innocence, no legal representation, no judge, no jury, and often no appeal. In other words, it’s an alternative legal system where the accused have fewer rights.

Social credit systems are an end-run around the pesky complications of the legal system. Unlike China’s government policy, the social credit system emerging in the U.S. is enforced by private companies. If the public objects to how these laws are enforced, it can’t elect new rule-makers.

An increasing number of societal “privileges” related to transportation, accommodations, communications, and the rates we pay for services (like insurance) are either controlled by technology companies or affected by how we use technology services. And Silicon Valley’s rules for being allowed to use their services are getting stricter.

If current trends hold, it’s possible that in the future a majority of misdemeanors and even some felonies will be punished not by Washington, D.C., but by Silicon Valley. It’s a slippery slope away from democracy and toward corporatocracy.

In other words, in the future, law enforcement may be determined less by the Constitution and legal code, and more by end-user license agreements.