Innovative, automated workflow orchestrated ingest, processing, archive, and near-real-time transfer of growing match files from Paris so FOX Sports’ post teams could work remotely.
Los Angeles, CA: While the U.S. Women’s National Soccer Team skillfully advanced the ball across the field and into the net at the 2019 FIFA Women’s World Cup—which took place from June 7th to July 7th in Paris—the FOX Sports postproduction team passed high-value match footage along an innovative, automated workflow to score with viewers.
The sports network’s goal was to create captivating program elements, such as promos, player profiles, replay clips, and B-roll packages, while the match progressed in Paris, without ever having to leave its familiar, well-equipped “Pico” facility home base, in Los Angeles, CA.
This winning Live @ Home remote postproduction strategy was made possible by the collaborative efforts of three industry-leading vendors—Telestream, Levels Beyond, and IBM Aspera—to integrate their respective technologies to form the seamless, unified workflow FOX Sports envisioned.
While FOX Sports successfully debuted much of this tech-forward remote operations strategy for postproduction during the 2018 FIFA Men’s World Cup from Russia, they still had to send two craft editors to Russia for about six weeks and build and maintain two full edit suites at the International Broadcast Center (IBC) there. But by pushing the envelope on the Live @ Home model, and enhancing the integrated, multi-vendor workflow, FOX Sports enabled ALL postproduction for the Paris games to be done remotely, resulting in significant logistical and financial benefits, such as:
- No need to build and maintain editing suites at the Paris IBC because FOX Sports personnel could access media they needed, when they needed it, from the cloud
- No need to disrupt staffers’ lives by moving them to Paris for four to six weeks
- No need to centralize operations because authorized users could access, browse, request, and edit low-res proxy footage from connected devices anywhere
- No need to have editors work long hours or odd shifts because high-res live match footage was ready in near-real-time, in the formats they needed for editing
- No need to work through the night to archive valuable match footage because it was being done automatically in real-time, by a clever Auto Archive workflow
In this case study, Telestream takes a deep dive into this technology, particularly the challenges, solutions, and workflows, as well as the operational benefits derived from this multi-faceted, automated solution, commissioned by FOX Sports, which capitalized on the vendor partners’ technology including:
- Telestream’s Lightspeed Live Capture for media ingest and recording; Vantage for on-premise media transcoding and other processing; and the Telestream Cloud for cloud-based media transcoding, processing, and delivery
- IBM Aspera’s FASP accelerated media file transfer; FASPstream for near-real-time streaming of mezzanine and proxy resolution media; and
- Levels Beyond Reach Engine for media asset management and content workflow orchestration; as well as Prelude for preparing media for ingest to Adobe Premiere Pro editing workstations
The digital postproduction workflow described in this case study is just one of many parallel workflows that FOX Sports needed to deploy to deliver high-quality live coverage of the 2019 FIFA Women’s World Cup via its broadcast channels, mobile apps, and social media sites. So, while the main broadcast signals were moving over dedicated, fiber networking services, FOX Sports deployed this separate, parallel postproduction workflow to create features, highlights, bumpers, teases, and other interstitial program elements as well as ancillary pre-game, half-time, and evening wrap-up shows.
While the Live @ Home production model allowed this work to be done anywhere, on any connected device, virtually all of the craft editing was done by the FOX Sports Production Group at FOX Sports’ Pico facility near Los Angeles, which we will now refer to as Pico/LA. There was also one craft editor based at FOX Sports’ Charlotte facility, and many creative editors around the U.S. browsing the live streams and marking off selected rough cuts.
“Our primary focus was to create highly stylized elements, like features, teasers, bumpers, highlights and replay clips, with the quality and immediacy our viewers expect. To do this, the right media needed to be available for rough and craft editing when and where it was needed, then moved along for playback to air. The integrated Telestream-Levels Beyond-IBM Aspera solution automated this entire sequence of intricate, complex events reliably and efficiently.” -David Sobel, Senior Director of Media Management, FOX Sports, Los Angeles, CA (Pico/LA)
The biggest challenge FOX Sports postproduction unit faced was the sheer magnitude and complexity of the macro workflow itself, which included the following three concurrent micro workflows:
- The Live @ Home Editing Transfer workflow where three live feeds per match were sent in 1080p/50 HD MXF OP1a AVC-Intra format directly from Paris to Pico/LA for ingest by any of multiple Adobe Premiere Pro editing suites. Arrival in AVC Intra format made the video easier for Premiere to ingest and edit. With the ability to stream up to 44 live HD streams at once to the cloud, the overall equipment footprint at the Paris IBC was able to be reduced by roughly 30%, saving on operating costs.
- The Live Sub-Clipping workflow where near-real-time HLS proxy resolution streams of growing media were made available to producers so they could mark their in and out edit points while the files were still growing. They would request their rough cuts be clipped and high-res versions sent to the Premiere editing bays for craft editing and finishing, and in some cases to servers for playout.
- An Auto Archive workflow that automated the transfer of high value, MXF mezzanine grade recorded feeds from the Paris IBC directly to Amazon S3 storage in the AWS Cloud concurrent with the other micro workflows and in real-time. Auto Archive ensured redundant, off-site back-up, while also establishing a master, cloud-based repository from which to access and move media faster.
Core elements of this multi-vendor solution had already been used and proven to be rock-solid at the previous Russia games. Back then, the risk was greater because it had never been done before, let alone in a mission-critical live event situation. But with that initial success, the FOX team was ready to amp it up with additional capabilities, such as Auto Archive; expanding the number of live feeds that could be ingested; the transfer of three growing high-res feeds per match from Paris to Premiere editing bays in Pico/LA, along with low-res HLS proxies enabling the Live Sub-Clipping Workflow.
While FOX Sports could have acquired and used each of these vendors’ systems separately, it was the integration of the three technology domains that made the resulting workflow so powerful and efficient. This integration was the result of close cooperation between the vendor partners working in conjunction with FOX Sports representatives to bring together the strongest elements of each product line into one overarching macro workflow.
With a problem-solving mindset, industry leaders Telestream, Levels Beyond, and IBM Aspera openly shared their proprietary API codes and other intellectual properties with each other to enable the free exchange of media, metadata, and communications between their respective systems, as well as built-in logic to facilitate an automated end-to-end workflow. Where basic REST API interfaces already existed between the systems, they were reinforced to support this use case.
When first developed for the previous year’s Russia games, the solution was set-up and tested at FOX Sports’ Charlotte, NC facility before being shipped to Russia in pre-configured racks that could just be stood up into place. For the Paris games, this same testing was done, again in Charlotte, to make sure that any changes they implemented didn’t introduce new risks into an otherwise proven solution.
Attesting to the solution’s flexibility, Dave Norman, Telestream’s Principal Sales Engineer – Enterprise Products, explained that, “In testing the equipment in Charlotte, NC, in February 2019, the original plan called for ingesting a total of 35 channels. But when the FOX Sports production staff arrived in Paris, they realized that they wanted to have more content than that. We had the flexibility to give them the additional 13 extra channels they needed, and to do so very quickly.” The solution was also designed in such a way that FOX Sports would be able to troubleshoot on their own if necessary, without the need for on-site vendor tech support. Fortunately, since the overall solution worked reliably, such troubleshooting was not necessary.
The End-to-End Workflow
The only human touchpoint in this overall ecosystem was the Lightspeed Live Scheduler, an app developed by Telestream exclusively for FOX Sports prior to the Russia games that kicks off the key micro workflows. Operators at the Paris IBC used the Scheduler to import Excel formatted data and metadata related to every soccer match. This list instructed Lightspeed Live Capture encoders which live match feeds to record and when, including the following types of data:
- Broadcast International Feed (BIF)
- Clean International Feed (CIF)
- Action Compilation (ACC)
- Emotional Compilation (ECC)
- Camera ISOs, such as Goal-Cams for competing soccer teams
- Matches and teams
- Parisian stadium venues and their city locations
- Channel assignments and timeframes to record
Without the Scheduler driving the concurrent tasks, operators would have had to manually start and stop the machines, also known as “crash record”, which would have posed operational challenges given that 30 matches were played in Paris. And, without the match metadata, editors would have had a tougher time searching for the game footage they needed.
As part of the macro workflow, the live match feeds provided by FIFA Host Broadcast Services (HBS) were ingested by the bank of 13 Lightspeed Live Capture systems, each of which has four 3G HD-SDI ingest channels. While only three of the four inputs were used per box in Russia, all four inputs were “maxed out” in Paris, resulting in a total capacity of 52 live feeds.
As the recordings were growing, the Lightspeed Live Capture boxes recorded the video feeds and output the content in the following ways:
- MXF OP1a AVC-Intra Class 100 (250 Mbps) mezzanine quality program feeds and camera ISOs, deposited onto a 900 terabyte Harmonic MediaGrid storage system on-prem at the Paris IBC. This media also supported the Live Sub-Clipping workflow
- Additional MXF OP1a AVC-Intra Class 100 (250 Mbps) mezzanine quality program feeds and camera ISOs automatically transferred via Aspera FASP Stream to Pico/LA’s Premiere editing bays for editing
- For Auto Archive, additional MXF OP1a AVC-Intra Class 100 (250 Mbps) mezzanine quality program feeds and camera ISOs went via FASPStream directly to S3 cloud storage (at the Amazon AWS US-West-2 data center, in Portland OR)
- Apple HLS Proxy Streams (720p/59.94) to the HLSstorage site on S3 (also Portland) for Levels Beyond Reach Engine content orchestration and the subclipping workflow
Another source of media content was Electronic News-gather (ENG) footage, such as crowd shots, fan interviews, and beauty shots, videoed by ENG crews travelling around France. They either brought their video back to the Paris IBC on camera cards, or transmitted it using bonded cellular from the field. Utilizing the Reach Engine-Adobe Prelude Panel, ENG video essence would be tagged with metadata. Prelude was also used to initiate accelerated media file transfers to upload the ENG files to the Reach Engine master repository, where it was immediately available to enrich production.
Live @ Home Editing Transfer Workflow
IBM Aspera technology—including FASPStream media streaming protocol, FASP accelerated file transfer, and High-Speed Transfer Service (HSTS) servers—played a key role in supporting live edits and back-up of growing files. FASPStream delivered approximately six simultaneous MXF feeds per match to Pico/LA, while simultaneously streaming additional MXF and HLS feeds to AWS S3.
“FASP deals with files that already exist, so they’re closed and not growing anymore. These files are moved as accelerated high-speed data transfers. But if the file is still open and growing, we use FASPStream, a streaming media protocol based on FASP technology. At the destination, the stream ends up as a remote file that can be edited,” said Mike Flathers, CTO of IBM Aspera. ”The key to this use case is that you never had to wonder if everything was working right or moving to the right place. With sophisticated API codes, the vendor partner’s systems all talked to each other, making the whole greater than the sum of the parts.”
Streams could be designated as high or low priority to maximize network bandwidth. And, since the data travelled over global IP based Virtual Private Networks (VPNs)—passing through commodity Internet circuits— content arrived in the IP-based form increasingly prevalent in today’s postproduction facilities.
The workflow configuration allowed for up to twelve live MXF feeds to move simultaneously with FASPStream directly to Pico/LA, giving editors there lots of content to choose from, within roughly seven seconds of the live action. FASPStream also carried live MXF feeds to AWS S3 cloud storage, as part of the Auto Archive process, which will be discussed later. Reach Engine communicated with each system in the joint workflow using Reach Engine Nodes and triggered events using the rich set of REST API commands that made this sophisticated, automated workflow possible.
FASPStreaming was the key to reducing the staffing and equipment requirements at the IBC and employing the Live @ Home remote production model. At the previous Women’s World Cup four years ago in Vancouver, FOX Sports had many editors and edit bayson-site, and then in Russia in 2018 there were two craft editors, a producer, three PA’s, and two full edit bays. But, at the 2019 Womens World Cup in Paris, there was no on-site postproduction staff at the IBC, and no edit bays needed to be built because that work was done using near-real-time streaming of growing media files that could be edited and finished remotely.
Interestingly, while FOX Sports had a beautiful studio set with a magnificent view of the Eiffel Tower behind the on-camera talent, the live production, master control, and playout-to-air functions were actually based at the Pico/LA facility. Since the network wanted to exploit the Live @ Home remote production model as much as possible, the IBC in Paris was only used to manage and monitor the ingest and recording of feeds.
Live Sub-Clipping Workflow
With its user-friendly Web browser-based interface and basis in the AWS Cloud, Reach Engine served as the media orchestration layer governing the entire macro workflow. It also served as the centralized master media repository, with inherent asset management.
Reach Engine also interfaced with the FIFA MAX service containing video assets provided by HBS. And Reach Engine also tracked the newly curated video assets that were created by all the editors using the original match footage.
In short, Reach Engine was the central repository that all authorized personnel logged into to watch streams of HD quality growing media files as well as low-res HLS proxy streams. This is also where authorized users could search for footage, request sub-clips and transfers of growing files, and more. In conjunction with IBM Aspera media transfer technology, Reach Engine gave editors a framework for browsing live, HLS streams, and performing tasks like cuts-only editing, without their ever having to know or understand the complex inner-workings of it all.
“As the workflow automation and orchestration layer, the Reach Engine platform was tasked with taking all of the heavy lifting off of the editors so that they could focus on their craft, while allowing producers and media managers to expeditiously create compelling stories and programs,” said Daniel Gonzales, Senior Solutions Architect for Levels Beyond, who oversaw this high-profile, mission critical project along with Solutions Architect Taylor Medlin.
Editors would log into Reach Engine and use its built-in media player to browse growing HLS stream to find shots or scenes of interest, and then mark off their selected in and out points. As an authorized editor, watching the stream sees the winning goal, they could mark the in-point 10 seconds ahead of the action and mark the out-point 10 seconds beyond it, and request that a high-res MXF version of that select footage be sub-clipped for finishing by a craft editor.
Reach Engine orchestrated that request, leveraging its’ tight integrations with Vantage for the sub-clipping of that winning goal, which was extracted from the evergrowing MXF media content and Aspera. to transfer the newly created file to Pico/LA as fast as possible. With the speed of the sub-clipping and delivery to the edit bay, a finished highlight, replay, or other clip of that winning goal would be made ready for air within minutes of occurring.
Reach Engine managed this process as it determined where in the workflow the live growing MXF media resided at any given moment:
- If it was still on the Media Grid in Paris, Reach Engine initiated a Vantage system at the Paris IBC needed to create a new MXF sub-clip from the growing file and forward it via Aspera to the destination
- If it was already on S3, then Reach Engine utilized Telestream Cloud made the sub-clip for transfer via Aspera to Pico/LA, and
- If it was already on SAN or server storage at Pico/LA, Reach Engine leveraged a Vantage system at that facility extracted the sub-clip for transfer to editors, or to (EVS or Quantel) playout servers for immediate inclusion in various FOX Sports’ live shows
“This was a very complex, cutting-edge media ecosystem designed to give us the flexibility to access whatever live footage we needed as fast as possible. Whether you were a feature producer in LA, a Premiere editor working out of Charlotte, or even a PA working in one of our LA control rooms, this integrated multi-vendor solution helped us capture, create, process, move, and edit match footage wherever and whenever it was needed.” -Brandon Potter, Director of Post Production, FOX Sports
Since most of the editing took place at the Pico/LA facility, it was faster to access growing media files from S3 storage, as they were streaming from the Paris IBC. Since the S3 cloud storage was at the AWS Data Center in Portland, this location was closer to LA, although this workflow logic was transparent to the user.
Auto Archive Workflow
Without the ability to Auto Archive the live incoming feeds, there would be no back-up of the high-value content until many hours following the live event. If something were to happen to the on-premise storage or if it just failed, the irreplaceable content would be lost, along with all the creative and monetizing potential of using it.
So, at the Paris IBC, an Auto Archive workflow was implemented that ensured pristine copying of all live incoming feeds directly onto S3 cloud storage as an automated, background process. In fact, once in storage on S3, Amazon AWS has its own archival back-up of data that copies it from one data center to another, which further safeguards the video assets.
At the Paris IBC, the Auto Archive process started the moment the feeds began arriving and continued as the files grew. More importantly, this archival copy was generated through a sequence of API commands that triggered the process, for each and every match, automatically.
According to FOX Sports’ Sobel and Potter, “Auto Archive was one of the most advantageous workflow modifications made to the overall media workflow ecosystem, largely because of the time-savings and peace of mind it provided.”
To appreciate why this would be, we need only look back to the setup of the 2018 FIFA World Cup event in Russia to see how time-consuming and labour-intensive the media archive process was. At that time, IBC operators had to wait around for the recording of a particular live match to finish before they could begin manually uploading the 300 gigabyte media files to S3. Over the course of the 2019 Women’s World Cup in Paris, there were roughly 40 large files that had to be uploaded, one at a time. The on-site operators would have had to stay there sometimes 10 hours beyond the match to supervise the upload process, and that made for a very long workday.
At the Paris event, instead of waiting about 10 hours, the archive process ended within 10 seconds of the match’s end. That’s because the live match feeds were being archived to S3 as the file was growing. As a fullyautomated workflow, one of the Lightspeed LiveCapture systems would poke an IBM Aspera HSTS server to initiate the auto archive, and it responded that the process was underway, and when finished, that it was done.
Once in the cloud on S3, the archival copy also serve as the master copy from which sub-clips and other requests could be made. In the cloud, that media was more easily accessible, and files could be transferred at high speeds. It also made sense because Reach Engine and Telestream Cloud were also right nearby in the Cloud, which further sped up processing.
There were a few feeds that were archived as a manual process, particularly UHD 4K feeds of each match, which were provided by HBS in addition to their 1080/50p feeds. While the UHD wasn’t used per se, it was recorded and archived on S3 for evergreen purposes. If and when the UHD broadcast standard becomes more prevalent, this ultra-high-res footage would be readily available to be used for future World Cup events.
The 2019 FIFA Women’s World Cup in Paris was one of many ratings winners presented by FOX Sports, and the network is contracted to broadcast World Cup events through 2026. As we’ve discussed in this case study, the Live @ Home postproduction workflow used this year is as evolutionary as it is revolutionary, and we’ll likely see more improvements to it in future World Cups, as well as other high-value live events.
This remote workflow was a big success because of the high volume of fresh, near-real-time content that could be delivered to edit bays half a world away. Compared to their previous World Cup events coverage, this multi-vendor integrated solution enabled FOX Sports to cut down on the following:
- Space devoted to equipment, such as edit suites, at the IBC
- Downtime waiting for manual processes, like media archive, to finish
- Packing and transportation costs to move preconfigured edit bays and other gear from Charlotte to Paris
- Relocation of staff members to Paris and the cost of accommodations for a four to six week stay
- Lag time between the live action and edited clips
- Risk of losing valuable 2019 Women’s World Cup match footage because ever-growing media files archived promptly to S3
Compared to the two petabytes of video recorded and stored in S3 for the previous year’s FIFA Men’s World Cup in Russia, the 2019 Women’s World Cup in Paris only generated one petabyte of content, primarily because the women’s tournament involved far fewer matches. However, according to FOX Sports’ Sobel and Potter, “We had way more engagement with the 2019 Women’s World Cup footage, with the creation of over 2.5 times the video assets, including over 1,300 more video assets, such as highlights, bumps, teasers, and features, compared to the Men’s.”
“This was a high stress environment, but these vendor partners worked closely with us to make this project as successful as it could be. We all worked hard to solve whatever problems we encountered, utilized our skills to the best of our abilities, and ultimately, had a very good show.” -David Sobel, Senior Director of Media Management, FOX Sports
For over 20 years, Telestream® has been at the forefront of innovation in the digital video industry. The company develops products for media processing and workflow orchestration; live capture, streaming, production and video quality assurance; and video and audio test solutions that make it possible to reliably get video content to any audience regardless of how it is created, distributed or viewed. Telestream solutions are available on premises or in the cloud as well as in hybrid combinations. Telestream is privately held with corporate headquarters located in Nevada City, California and Westwood, Massachusetts.
For company and product information, visit www.telestream.net.
Trademarked company and product names are the property of their respective companies.