Josh Downs, Live Operations Manager, and Kyle Umstatter, Deployment Engineer – Genvid Technologies
The Live Operations (LiveOps) support for Rival Peak may be the most ambitious livestreaming effort ever undertaken, with thirteen simultaneous streams running 24/7 at launch – a dozen of which are each effectively dual streams in their own right. Using Genvid’s unique interactive broadcasting technology and services, twelve of the livestreams are dedicated to Rival Peak’s dozen AI characters and enable viewers to not only follow those characters in real-time, but also interact with the characters and their environs as the show unfolds over the initial three-month season.
While we in the LiveOps team are used to supporting enhanced livestreams involving the standard broadcast stream coupled with an overlay stream delivering additional imagery and information, multiplying this by 13 for a single project was new to everyone involved. With past projects that we’ve powered, typically a single stream was required. We’ve literally doubled our footprint in terms of server requirements and support.
From a LiveOps perspective, a massive interactive live event (MILE) like Rival Peak isn’t really very different from supporting the interactive livestream of a traditional video game. Likewise from a viewer standpoint, the differences are subtle. User inputs are essentially the same in either case, tapping (i.e., clicking) to adjust or effect what you’re viewing. We’ll have millions of inputs coming in across 13 different stream servers, and a fourteenth server managing all the backend game data to resolve viewer input along with the simulation and AI functions.
Beyond the 13 livestreams, there’s a fourteenth machine running “the Authority,” that handles all the in-sim processing, resolving conflicts, tracking schedules, etc. The Authority resides on its own server – same Unity game, in headless mode. It’s playing out the sim, but without doing any rendering, and connects via Bolt (a Unity-compatible client-hosted networking solution for passing in-sim events to/from the Authority) to the character instances, playing out the sim info and then sending it to the appropriate livestream(s). Also within the Authority server is “The Conductor.” The Conductor reads a schedule file, determines it will play an ‘episode’ at a certain time for a certain duration, turns on the stream for Facebook, starts a timer and off it goes for exactly eight hours.
While viewers will effectively be able to tune in any time to watch Rival Peak, there will be moments of downtime for the sake of supporting every time zone around the world. Each day of the show involves 8-hour streams for each of three regions, and the streams are sequential to enable viewers in each part of the world to catch all the daily action if they choose. But with the exception of a few minutes between each 8-hour session, we’ll be live every day, all day, for 12 straight weeks.
Keeping services up and running is of course always the primary goal. One of the challenges with Rival Peak was that we really pushed Facebook to its limits – and beyond, in some cases. Facebook was never designed to host a 24/7 livestream for months at a time. And with this particular project, and while we do have a “single pane of glass” for monitoring, given that we’re dealing with 13 streams we still rely heavily on alert notifications rather than eyes-on.
During the testing process we worked with 24×7 QA at Keywords Studios around the world, including testers in India and Mexico who were working out of their homes due to Covid. We were impressed that homes around the world were getting streams in crisp 1080p without issue!
The whole process was further complicated by virtue of involving another partner on this project, Pipeworks Studios, who did a stellar job of helping us work with their systems, styles, and methods of support. Pipeworks Studios built the simulation and rendering system, and in some cases uses different tools (e.g., Perforce vs. Git) so there was a fair amount of compromise involved as well.
We’ve also managed to never let you feel latency despite there being an average of 15-25 seconds of latency on the FB system. This is part of what makes Rival Peak special, using Genvid’s unique synchronization software.
We’re set up to scale to millions of concurrent users with Rival Peak, and are ready to help all two billion of Facebook’s users interact with this unique experience.
