How do you deliver a glitch-free live second-screen viewing experience to millions of people around the world who live and breathe the World Cup?
For Michelle Munson, co-founder, president and CEO of Aspera, it all starts with quantifying the probability of a glitch based on several IT network givens and building in an appropriately long buffer to offset any hiccups.
Last week at the SMPTE Annual Conference and Exposition in Hollywood, Calif., Munson laid out how Aspera transported multiple live streams of game coverage of this year’s World Cup from the EVS C-Cast platform via a wide area network for ingest and transcoding in the cloud and ultimate playback on smartphones, tablets and computers.
“EVS’s challenge is they were commissioned to deal with 12 stadiums and 64 games, an IBC [international broadcast center] located in Rio [and] a global broadcast audience they were trying to reach,” she said.
EVS originated six live streams from each World Cup venue with full redundancy, so viewers could augment their World Cup TV viewing with multiple shots and angles of the action on their second screen devices.
Transcoding all of those streams on site with standalone hardware for Internet streaming simply was not an option, she said. Not only would the cost of all of those black boxes been astronomical, but the bandwidth available from each of the venues was woefully inadequate to handle millions of individual requests for streaming coverage.
What was called for was elasticity in computing power that could scale to waxing and waning viewership as games began and ended. The decision was made to ingest the streams at the venues into the C-Cast platforms, transport them to Amazon S3 cloud storage in Europe, and allow Elemental Technologies to transcode those streams on the fly for different playback devices. From there, Akamai took over delivering World Cup coverage to millions of fans around the globe via its content delivery network.
In terms of throughput, EVS needed an aggregate of 240 Mb/s from the venues, said Munson. That’s 10 Mb/s for each of the six mezzanine-level HLS-encoded content streams, times two because of doubleheaders, times two for system redundancy. Latency ran between 200ms and 250ms and packet loss varied from a few percent to 9%, according to Aspera’s post-tournament logs. “So [those were] really challenging WAN conditions,” she said.
TCP-based transport, which could only support one-fifth to one-twentieth of the real-time rate needed for each feed, was out of the question, said Munson.
Providing a transport solution for World Cup transport was a bit of a departure for Aspera. Unlike traditional media applications of its fast and secure protocol, FASP, where large files are delivered via WANs to support production or post requirements, such as submitting dailies from a remote movie shoot for review, the World Cup application was live.
The problem using FASP for live delivery is that it was designed to make a “best effort” attempt to deliver blocks of data in order. “We want to go in order as much as possible to avoid random seeking on the disk and inefficiency, but we can deliver the first or the last block in a file, really, at any time,” she said.
“Best effort,” however, just isn’t good enough for live streaming applications, such as the one EVS needed. Fortunately, Aspera’s new third-generation FASP architecture provides for in-order byte stream delivery, said Munson.
What needed to happen to ensure the new FASP transport was up to the demanding application required for World Cup second screen coverage was to quantify the transport efficiencies it could attain given the playout rate of streaming video from C-Cast and the anticipated degree of loss on the WAN.
Here is where it becomes possible to take a quantitative look at the probability of a glitch in the streamed video, said Munson. Aspera has modeled the probability that there will be a need to wait a given number of retransmission time outs, RTOs, for the next expected data packet to arrive given this new ordering constraint feature and other properties of FASP.
Munson showed how it is possible to build a model to find the probability of a glitch when smooth playback is the goal. Components of the model include the video play rate in bytes per second, a packet size made up of a given number of bytes, a known packet loss ratio for the network in use and the probability of waiting greater than or equal to the number of RTOs needed for the next expected data packet.
Once the probability of a glitch is known, it is possible to build in a small buffer to accommodate the glitch and deliver a smooth stream. “That is the secret of doing it over these lossy networks with high quality,” she said.
For a very bad network — far worse than the network used for the World Cup application — such as a bad wireless or satellite network with 500 ms of latency or 5% packet loss, a buffer of 3.5 to 4 seconds guarantees playback with no more than one glitch per hour — essentially glitch-free playback, she said.
“As the play rates go down, it gets easier and easier — less buffering [is required] to guarantee that [no more than one glitch per hour],” she said.
For the World Cup, this approach made it possible for Aspera to plan for the probability of a glitch, buffer accordingly and ultimately transport EVS streams into Amazon S3 cloud storage while Elemental’s live transcoding system read what was being written and performed the live transcoding needed for Akamai to distribute to different playback devices, like smartphones and tablets.
“The net result was 660,000 minutes of video ingested using Aspera; 2.8 million minutes of video generated out of Elemental, and 15 million hours of video watched, most of which was distributed by Akamai, and the whole system was, of course, brought to you by EVS for broadcasters,” said Munson.