How AI Will Help Keep Time at the Tokyo Olympics

“In volleyball, we’re now using cameras with computer vision technologies to track not only athletes, but also the ball,” says Alain Zobrist, head of Omega Timing. “So it’s a combination where we use camera technology and artificial intelligence to do this.”

Omega Timing’s R&D department comprises 180 engineers, and the development process started with positioning systems and motion sensor systems in-house, according to Zobrist, in 2012. The goal was to get to a point where, for multiple sports at the 500-plus sports events it works on each year, Omega could provide detailed live data on athlete performance. That data would also have to take less than a tenth of a second to be measured, processed, and transmitted during events so that the information matches what viewers are seeing on screen.

With beach volleyball, this meant taking this positioning and motion technology and training an AI to recognize myriad shot types—from smashes to blocks to spikes and variations thereof—and pass types, as well as the ball’s flight path, then combine this data with information gleaned from gyroscope sensors in the players clothing. These motion sensors let the system know the direction of movement of the athletes, as well as height of jumps, speed, etc. Once processed, this is all then fed live to broadcasters for use in commentary or on-screen graphics.

According to Zobrist, one of the hardest lessons for the AI to learn was accurately tracking the ball in play when the cameras could no longer see it. “Sometimes, it’s covered by an athlete’s body part. Sometimes it’s out of the TV frame,” he says. “So, the challenge was to track the ball when you have lost it. To have the software predict where the ball goes, and then, when it appears again, recalculate the gap from when it lost the object and got it back, and fill in the [missing] data and then continue automatically. That was the one of the biggest issues.”

It’s this tracking of the ball that is crucial for the AI to determine what is happening during play. “When you can track the ball, you will know where it was located and when it changed direction. And with the combination of the sensors on the athletes, the algorithm will then recognize the shot,” Zobrist says. “Whether it was a block or a smash. You will know which team and which player it was. So it’s this combination of both technologies that allows us to be accurate in the measurement of the data.”

Omega Timing claims its beach volleyball system is 99 percent accurate, thanks to the sensors and multiple cameras running at 250 frames a second. Toby Breckon, professor in computer vision and image processing at Durham University, however, is interested to see if this stands up during the Games—and, crucially, if the system is fooled by differences in race and gender.

“What has been done is reasonably impressive. And you would need a large data set to train an AI on all the different moves,” Breckon says. “But one of the things is accuracy. How often does it get it wrong in terms of those different moves? How often does it lose track of the ball? And also if it works uniformly over all races and genders. Is that 99 percent accuracy on, say, the USA women’s team and 99 percent accuracy on the Ghanaian women’s team?”

Zobrist is confident, and explains that while it may have been easier to call in Google or IBM to supply the AI expertise needed, this was not an option for Omega. “What is extremely important, whether it’s for a scoring sport, or timing sport, is that we can’t have discrepancies between the explanation of the performance and the ultimate result,” he says. “So to protect the integrity of the result, we cannot rely on another company. We need to have the expertise to be able to explain the result and how the athletes got there.”

As for future timing and tracking upgrades, Zobrist is tight-lipped, but says the Paris Games in 2024 will be key. “You will see a whole new set of innovations. Of course, it will remain around timekeeping, scoring, and certainly also around motion sensors and positioning systems. And certainly also Los Angeles in 2028. We’ve got some really interesting projects for there that actually we’ve only just started.”