One in 2017 on an early afternoon at Facebook’s headquarters in Menlo Park, California, an engineer named Tommer Leyvand sat in a conference room with a smartphone on the brim of his baseball cap. Rubber bands helped hold it in place when the camera was facing out. The absurd hat phone, especially the cool futuristic version, contained a secret tool that only a small group of employees knew about. What he could do was amazing.

Several men in the room laughed and talked excitedly to each other, as captured on video that day, before one of them asked for silence. The room fell silent; a demo version was running.

Mr. Leyvand turned to the man standing in front of the desk. The smartphone’s camera lens—round, black, unblinking—hung over Mr. Leyvand’s forehead like a cyclops eye that took in the face before him. Two seconds later, a robotic female voice said, “Zach Howard.”

“It’s me,” confirmed Mr. Howard, a mechanical engineer.

An employee who saw the technology demonstration thought it must have been a joke. But when the phone started calling the names correctly, he found it creepy, like something out of a dystopian movie.

An ID phone with hats would be a great gift for those who are visually impaired or blind, but it was risky. Facebook’s earlier rollout of facial recognition technology to help people tag friends in photos sparked outrage from privacy advocates and a 2015 ban. filed a class-action lawsuit in Illinois that ultimately cost the company $650 million.

With technology like Ms. Leyvand’s on its head, Facebook could prevent users from forgetting a colleague’s name, remind them at a cocktail party that an acquaintance had children they were asking about, or help them find someone at a crowded conference. But six years later, the company, now known as Meta, hasn’t released a version of the product, and Mr. Leyvand left for Apple to work on the Vision Pro augmented reality glasses.

In recent years, startups Clearview AI and PimEyes have pushed the boundaries of what the public thought was possible by releasing facial search engines paired with millions of photos from the public internet (PimEyes) or even billions (Clearview). With these tools, available to the police in the case of Clearview AI and to the general public in the case of PimEyes, someone’s snapshot can be used to find social media profiles of other online photos of that face that can reveal a name. or information that a person would never want to be associated with publicly, such as risqué photos.

What these startups did was not a technological breakthrough; it was ethical. The tech giants developed the ability to recognize the faces of unknown people much earlier, but decided to put the technology on hold after deciding that the most extreme version – naming a stranger’s face – was too dangerous to be widely available.

Now that the taboo has been broken, facial recognition technology could become ubiquitous. Currently used by police to solve crimes, authoritarian governments monitor their citizens and businesses to protect against enemies. Soon it could be a tool in all of our hands, an app on our phone or in our augmented reality glasses that will help us create a world where there are no strangers.

“We decided to stop”

Back in 2011 Google engineer revealed he was working on a tool for Google to find someone’s face and provide other photos of them online. A few months later, Google chairman Eric Schmidt said in an on-stage interview that Google “built this technology and we didn’t allow it.”

“As far as I know, it’s the only technology that Google has developed, and after looking at it, we decided to stop,” Mr. Schmidt said.

Inadvertently or not, the tech giants have also helped keep the technology out of mainstream circulation by taking away the cutting-edge startups that offer it. In 2010, Apple bought Polar Rose, a promising Swedish facial recognition company. in 2011 Google has acquired PittPatt, a US facial recognition company popular with federal agencies. And in 2012, Facebook acquired the Israeli company Face.com. In each case, the new owners closed the services of the acquired companies to outsiders. Silicon Valley heavyweights were the de facto gatekeepers of how and whether the technology would be used.

Facebook, Google, and Apple have used facial recognition technology in what they thought were relatively innocuous ways: as a security tool for unlocking smartphones, a more efficient way to tag famous friends in photos, and an organizational tool for sorting smartphone photos by faces. the people in them.

But in the past few years, smaller, more aggressive companies like Clearview AI and PimEyes have been kicking in the door. What made the change possible was the open-source nature of neural network technology, which now underpins most artificial intelligence software.

Understanding the path of facial recognition technology will help us navigate what other advances in artificial intelligence, such as image and text generation tools, will bring. The power to decide what they can and cannot do will increasingly be determined by anyone with a modicum of tech savvy who can ignore what the general public deems acceptable.

“Standing on the shoulders of giants”

How did we get to the point where someone can spot “hot daddy” on a Manhattan sidewalk and then use PimEyes to find out who he is and where he works? The short answer is a combination of free code shared online, tons of public photos, academic papers explaining how to put it all together, and a more relaxed approach to privacy laws.

Clearview AI co-founder Hoan Ton-That, who led his company’s technological development, had no particular background in biometrics. Before Clearview AI, he created Facebook quizzes, iPhone games and silly apps like Trump Hair to make a person look like the former president in a photo.

In order to create an innovative and more profitable app, Mr. Ton-That turned to free online resources such as OpenFace, a “face recognition library” developed by a group at Carnegie Mellon University. The code library was made available on GitHub with the caveat: “Use responsibly!”

“We do not support the use of this project in applications that violate privacy and security,” read statement. “We use it to help cognitively impaired users feel and understand the world around them.”

It was a noble request, but completely impractical.

Mr. Ton-That got the OpenFace code up and running, but it wasn’t perfect, so he kept searching, wandering around academic literature and code repositories, trying this and that to see what worked. He was like a man walking through a garden, picking the fruits of decades of research, ripe for the picking and gloriously free.

“I couldn’t have done it if I had to build it from scratch,” he said, singling out some researchers who have advanced computer vision and artificial intelligence, including Geoffrey Hinton, the “godfather of AI.” standing on the shoulders of giants.”

Mr. Ton-That is still building. Clearview has created a version of its app that works with augmented reality glasses, a more fleshed-out implementation of the face-hat that Facebook’s engineering team created a few years ago.

The end of anonymity

The $999 pair of augmented reality glasses, made by Vuzix, connect the wearer to Clearview’s database of 30 billion faces. Clearview’s AR app, which can identify a person up to 10 feet away, is not yet publicly available, but the Air Force has provided funding for its potential use on military bases.

On a fall afternoon, Mr. Ton-That showed me the glasses in his representative’s apartment on Manhattan’s Upper West Side, put them on, and stared at me.

“Ooooh, 176 pictures,” he said. “Festival of Aspen Ideas. Kashmir hill,” he read from the image caption on one of the photos provided.

Then he handed me the glasses. I put them on. Although they looked clunky, they were light and naturally fitted. Mr. Ton-That said he had tried other augmented reality glasses, but this one performed best. “They have a new version,” he said. “And they’ll look cooler, more hipster.”

When I looked at Mr. Ton-Ta through the glasses, a green circle appeared around his face. I tapped the touch pad against my right temple. A message appeared on the square screen that only I could see on the right lens of the glasses: “Wanted…”

Then the square was filled with his photos, each with a caption. I swiped through them using the touch pad. I tapped to select one that said “Clearview CEO, Hoan Ton-That;” it had a link that indicated it came from the Clearview website.

I looked at his spokeswoman, scanned her face, and there were 49 photos, including one with a client she asked me not to mention. It accidentally revealed how obsessively searching for someone’s face can be, even for someone whose job it is to get the world to adopt this technology.

I wanted to take the glasses outside to see how they work on people I don’t really know, but Mr. Ton-That said we can’t because the glasses need a Wi-Fi connection, and because someone can recognize it and immediately understand, what these glasses were and what they could do.

It didn’t scare me, even though I knew it should. It was clear that people with such a tool would inevitably have power over those who did not. But there was a certain thrill in seeing it work, like a successful magic trick.

A missed opportunity?

Meta has been working on its augmented reality glasses for years. in 2021 In an internal meeting earlier this year, the company’s chief technology officer, Andrew Bosworth, said he would like to give them facial recognition capabilities.

In a recording of an internal meeting, Mr Bosworth said leaving facial recognition out of augmented reality glasses was a missed opportunity to improve human memory. He talked about the universal experience of going to dinner and seeing someone you know but can’t remember their name.

“We could put a little name tag on them,” he said in the recording with a short laugh. “We could. We have that ability.”

However, he expressed concern about the legality of offering such a tool. BuzzFeed. reported his remarks at the time. Mr. Bosworth, in reply said that facial recognition was “highly controversial” and that providing broad access was “a discussion we need to have with the public”.

Although Meto’s augmented reality glasses still developmentthe company disabled the facial recognition system Facebook implemented to tag friends in photos and deleted more than a billion facial prints created by its users.

Such a system would be easy enough to re-enable. When I asked a Meta representative about Mr. Bosworth’s comments and whether the company might one day add facial recognition to its augmented reality glasses, he didn’t rule out the possibility.

Kashmir Hill covers technology for The New York Times. She is the author of Your Face Belongs to Us: A Secret Startup’s Quest to End Privacy as We Know It, from which this article is adapted.