Landon Meier, a designer and sculptor based in Colorado, makes hyper-realistic masks of famous people. He’s created masks of Donald Trump, Kim Jong-un, Vladimir Putin, and more. They look disturbingly real, if a bit uncanny. Now the three masks mentioned above are on sale on eBay, with bids at thousands of dollars and counting. Looking at photos of people wearing one of the freakily realistic masks, it’s hard to tell it’s not a real face. That got me wondering: If it can fool my brain, could it fool a facial recognition algorithm?
A facial recognition system is an application that can identify a person based on a photo or video. Facial recognition is a subset of biometrics ― the analysis of physical human characteristics, especially for means of identification ― along with fingerprinting and voice pattern recognition. In 2001, police in Florida scanned the entire crowd at Super Bowl XXXV, looking for criminals. Since then, facial recognition has been adopted by social media platforms and photo organizing tools to identify people in photos, as well as in airports to check passport photos. It’s poised to spread further, as the FBI builds up its database of faces and employers start using it in lieu of badges.
Facial recognition systems analyze an image of a face to create a unique “faceprint” made up of mathematical measurements for things like distance between the eyes, width of the nose, and patterns on the skin. The system then compares the faceprint to its database of stored faceprints, looking for a match.
Facial recognition is getting better, but many systems still aren’t that good. Things like sunglasses, shadows, and even dark skin can throw them off, depending on how robust the underlying database is. Artists and cyberpunks have devised various ways to combat creeping passive surveillance, including camouflage patterns, bizarre hair and makeup styles, and glasses that can make computers think you’re a celebrity. In 2015, Popular Science contributor Dan Moren was able to log in to the United Services Automobile Association (USAA) app using a video of himself. More recently, it was reported that the security feature that uses facial recognition to unlock the new Samsung Galaxy S8 can be fooled with a photo.
I assumed that a computer would have a hard time telling the difference between the real Hillary Clinton and someone wearing Meier’s mask of Hillary Clinton. But according to Anil Jain, a biometrics expert in the Department of Computer Science and Engineering at Michigan State University, an incredibly realistic mask might fool some facial recognition systems but not all of them. The Samsung Galaxy S8, or the algorithm Facebook uses to tag your friends? Probably. But something like the FBI’s Interstate Photo System (IPS) ― an investigative tool that produces a ranked list of possible leads based on a database of millions of criminals’ photos collected over decades ― would spot the fraud.
First, there’s the skin. “They have to be pretty realistic in terms of the skin tone or the surface tone, because one of the characteristics which we look at in telling whether it’s a real face or a spoof is the texture on the face,” Jain said. Spoofing, in this instance, means to fake some form of identification. If facial recognition is working off a high-quality image, it can be an immediate giveaway; the skin has to look like real skin. “The more the texture on the mask surface looks like human skin, the more difficult it will be to detect it.”
Next, the face needs to move. Algorithms will look for movement or blinking, Jain said. The fact that Meier’s masks have dead, unmoving eyes would be flagged instantly by a strong facial recognition system. Meier’s masks are pretty convincing in still photos, but you’d need something more like those rubbery Mission Impossible masks to fool high-end facial recognition systems.
The more faces in the database, the better it is at detecting fraud. A database must also have a mix of different genders, ethnicities, lighting situations, and even different cameras used to capture the faces and profiles of people in order to be truly robust. “Unfortunately, the current research for face spoof detection is a little bit fragile,” Jain said.
Even with all the caveats, facial recognition technology is spreading. In 2012, the Federal Trade Commision (FTC) compared the current era of biometrics in some ways to the film Minority Report where ads were individually targeted based on sensory readings. A 2015 report by the U.S. Government Accountability Office found instances of facial recognition being used at retailers to curtail shoplifting as well as at casinos to identify cheaters. Facebook’s ability to detect, catalog, and identify faces only continues to grow.
Engineers working on facial recognition systems, especially those used for secure identification, must reckon with the possibility that masks could be used to spoof their algorithms. For now, however, the larger threat is more likely to come from simply holding a photo in front of a phone’s camera.
