Apple Puts the AR in ‘Art’ (and in ‘Transparent Sky-Being’)

Thanks to a hot desert air mass stalling over San Francisco, the sky was a fogless blue, which made the words stand out even more as they floated upwards past St. Patrick Church in downtown San Francisco. White and uppercase, they rose in perfectly justified blocks, the voice of artist and poet John Giorno intoning them in my headphones: “A vast dome of blue sky / and your mind / is an iron nail in between.” On and on, the uppermost words breaking up and floating away, a Star Wars crawl of Buddhist introspection.

It was breathtaking for its scale—each word was dozens of feet high—yet the lunchtime crowds in Yerba Buena Gardens didn’t even notice. The only people who seemed as rapt as I was, in fact, were the three who were pointing an iPhone XS Plus in the same direction I was and wearing Beats headphones identical to mine. All that Apple was no coincidence: Giorno’s piece, Now at the Dawn of My Life, was one of six pieces in the company’s [AR]T Walk, an augmented-reality public art walking tour that launched this week.

AR, in which virtual objects are integrated into your real-world surroundings, has been embraced by museums and artists (conventional and guerrilla) dating back nearly a decade, and AR- and VR-specific exhibits have been popping up with ever more regularity. Much of that is because AR is easier to build and implement than ever before: Android and iOS feature AR development toolkits that have improved significantly since their 2017 introductions, and Facebook turned its Camera Effects platform into a similar toolkit called Spark AR.

But while all three of those systems—along with wearables like Microsoft HoloLens and the Magic Leap One—have been used to create virtual art in real-world galleries, moving outside those confines and into public spaces has been a more difficult proposition, in no small part because of the challenge of presenting a consistent experience for all users. (More on that in a bit.) The two-hour [AR]T Walks, developed in partnership with the New Museum in New York City, may only be available at six of the more than 500 Apple Stores around the globe, but in scope and scale they’re wildly ambitious, a deployment of AR that’s all but unprecedented outside the world of games.

Each of the walks features the same six pieces, which artists developed with New Museum over the course of a year; the only difference is the cityscapes housing them. For the Walk leaving from San Francisco’s Union Square store, groups venture first down car-free Maiden Lane, where they experience a piece from Chicago artist Nick Cave. (No, not that Nick Cave.) Known for his wearable “soundsuits” that shroud their wearers’ identifying characteristics, Cave uses AR to refashion the idea completely. On the phone screen, you’re presented with a swirling virtual soundsuit that you follow down the street, tapping and swiping at it to see how it reacts. It’s cute, if not mindbending—until you reach the end of the street to find an enormous transparent being perched atop a building, hoovering up those soundsuits to clothe itself in their patterns.

The finale of Nick Cave’s Soundsuits, as seen from inside New York’s Central Park.

Apple

The result, as with the best of the pieces in the process, hinges on some painstaking procedural hygiene. Two employees from Apple’s in-store events staff—Today At Apple, as it’s known—lead each group. One carries an iPad that controls the private [AR]T Walk app on the Apple-furnished XS Pluses attendees use; the other acts as a behavioral model, demonstrating at each location exactly how to trigger the AR experience.

About that: If you’re going to create a good shared persistent AR experience—shared meaning it’s visible to anyone in that specific location, and persistent meaning it can be seen on multiple visits to that location—you need a coordinate system so that the AR elements always show up in the same place. Think of it as a three-dimensional version of the origin point in a Cartesian plane, where the X and Y axes cross. Once your phone registers that it’s looking at the anchor, it can then layer in all the AR elements in their proper places based on that starting position.

Even then, things can go wrong. Take this unrelated AR piece of the late Jeffrey Epstein hovering over the site of a President Trump rally in New Hampshire, which artist Nancy Baker Cahill unveiled this week.

As a piece of political trollery, it’s undeniable; as a convincing illusion, less so. It jitters visibly, and seems to have trouble staying anchored to the arena—both immersion-breakers of the highest order.

The [AR]T Walk largely avoids such pitfalls by relying on a preparatory dance at each location. You walk to a very specific area, hold up your phone while facing away from the anchor—usually a sign, which features enough unique high-contrast patterns to be quickly recognizable—and then rotate a full 180 degrees until you face the anchor. The whole thing feels very Apple: incredibly polished and incredibly stable, as long as you did things exactly the way Apple told you to.

The other pieces, distributed throughout three locations covering about a mile and a half of walking, range from the whimsical to the bleak. In Cao Fei’s Trade Eden, a labyrinthine series of conveyor belts ferrying unmarked boxes appears in a plaza, distilling global trade into a fantastical display of futility. Nathalie Djurberg and Hans Berg’s This Is It uses a series of midair speech balloons to beckon attendees through a grove of trees, until finally unveiling a cautionary tale seemingly hidden inside a tree.

Peter Rubin writes about media, culture, and virtual reality for WIRED.

Meanwhile, over the course of the walk, an entirely different form of art emerges. A half-dozen people walking through public places, clustering around seeming nothingness and staring at their phones? It might have been commonplace when Pokémon Go first swept the outdoors in 2016, but it still attracts attention. Apple has made no secret of its AR aspirations, but its slow-drip approach has always targeted users inside the home: placing furniture, playing with Legos. The [AR]T Walks are still a drip—small groups, close supervision, very few locations—but they’re also an unmistakably public drip, one that’s much closer to a stream than ever before.

Besides, any new technology trying to catch a current into the mainstream needs to feel familiar, or at the very least not alienating—and part of that is acclimating people to unexpected new behaviors. Like selfies. Or taking phone calls in public on near-invisible earbuds. Or even walking through the middle of a crowded park during lunch hour, following a winding path no one else can see.

If you don’t happen to live in San Francisco—or New York, Tokyo, Hong Kong, Paris, or London, the other five cities with walks—you’re not totally ARsed. All Apple Stores are currently hosting an additional Nick Cave augmented-reality installation, Amass, as well as free sessions where you can learn to build AR experiences using an iPad app.