Entertaining audiences of the future

At the beginning of 2019, the government-funded UK Research and Innovation (UKRI) announced four demonstrator projects that would receive £16 million worth of support under its Audience of the Future (AotF) Immersive Technology Challenge. These cross-disciplinary large-scale programmes were in the fields of moving image, visitor experience, theatre performance, and esports.

UKRI had seen that immersive technology was shifting audience behaviour from viewing content to ‘experiencing’ and ‘recommending’ it. The four winners of the competitive process were chosen to show how in-house events – at stadiums, theatres and museums – could become more immersive so that audiences could feel that the occasion itself was responding to their presence.

The demonstrators were due to launch in 2020 but with the arrival of COVID-19 restrictions prohibiting in-person events, each had to pivot, rethink and reboot their initiatives. All managed to launch a new revised version of their original ideas in 2021 and UKRI will disseminate the collated outcomes of their experiences at the end of the year.

The published results will allow others to benefit from some of the processes and innovations developed. They will show that the scheme initiated pre-commercialisation trials that triggered world-leading applied industrial research. The outcomes have also demonstrated that the coincidental arrival of a pandemic lockdown has, in fact, helped accelerate the digital disruption of sectors, such as education and entertainment, through immersive technology.

VR, AR and MR

Immersive technology has transformed digital experience by bringing together the virtual with users’ sight, sound and even touch. The word ‘immersive’ is used to describe experiences that surround someone, enabling to make them feel part of an alternative environment that responds to their presence.

The technologies enabling immersive experiences range from virtual reality (VR) using a headset to augmented reality (AR) applications on smartphones. There is also mixed reality (MR), an extension of AR, that allows real and virtual elements to interact within an environment. These innovative technologies have all been made possible by engineering advances in a wide range of fields. For example, modern VR headset displays are based on technology developed for smartphones such as gyroscopes and motion sensors that track head, body and hand positions; small HD screens that provide stereoscopic displays; and small, lightweight and fast computer processors. Similarly, the main components for AR are a processor, display, sensors, and input devices – also used in smartphones and tablets, which often include a camera and microelectromechanical systems (MEMS) sensors such as an accelerometer, GPS, and solid-state compass, making them suitable AR platforms.

The UK has the largest immersive market in Europe. The UK VR market is expected to grow to £300 million by 2023 with around 1,250 companies generating over 50% of their revenue from consulting, services or products within the immersive domain. Nearly 300 of these companies operate in the sports, media and arts sector. It was this corner of the market that UKRI targeted.

Matt Sansam, Head of Delivery for AotF, says that in addition to participating consortia having an academic/university input, “we asked that they had a globally recognised intellectual property (IP) to engage with audiences in presenting engaging and exciting experiences; that SMEs were involved; that their project reached 100,000 people; and that there was a certain amount of match funding from the private sector.” In addition to UKRI providing £16 million the consortia contributed an extra £7 million for their four demonstrator projects.

Esport demonstrator

The most spectacular demonstrator, in terms of numbers involved – nearly two million – was WEAVR. WEAVR is the name of the esports consortium led by ESL Gaming, the world’s largest esports company. ESL organises and produces video competitions worldwide and these used to culminate in big stadium events with thousands of esport supporters present. The original AotF idea was to make the event audiences feel more involved with the gamers playing in front of them. With COVID-19 came lockdowns and a rethink.

Esports were in a good position with regards to live events: they could technically continue, as all games could be played remotely, the problem was with production as broadcast crews and commentators usually work side by side. Large arenas couldn’t be played in, so the production teams switched to software-based components allowing the crews to do everything – like playing replays – remotely.

Dr Florian Block is Reader in Digital Creativity at the University of York and the R&D Director of WEAVR. He says: “Even before COVID-19 struck we didn’t want to just replicate the physical stadium experience, we wanted to take it up a notch when freed of physical constraints for those not attending an event in person. Not to just compensate, but to fully explore how immersive tech can generate audience engagement. So what we’ve created are things like a VR space, a virtual VIP lounge where spectators wearing VR headsets watch the game on big monitors, move around and play with flaming swords in super-high pixel resolution.”

Perhaps one of the strongest innovations is WEAVR’s data artificial intelligence (AI). Its new technology consumes huge numbers of historical data points from esports competitions. It then tailors the information to the individual viewer and can provide targeted commentary for that person. It can tell which player a viewer is interested in, it understands their level of understanding about the game and then provides custom-made commentary.

Dr Block adds: “The secret sauce is the way that we slice the data and the way we train machine learning and AI models to understand the data and construct really interesting output. So there’s an algorithmic contribution and then the way that the model is applied to generate stories. We have found ways of identifying extraordinary moments in game play that we know have a high relevance for fans.”

The resulting viewing numbers are impressive. During 2020, WEAVR reached 1.8 million unique fans. That number includes people using interactive overlay on broadcasts, using the mobile app and VR – all additional engagements that weren’t there before. The results have been so strong that the companies involved are now looking to fully commercialise their innovations.

Theatre demonstrator

The In Performance demonstrator consortium consisted of 15 organisations headed by the Royal Shakespeare Company (RSC). The RSC was going to use devices such as mobile phones, extended reality headsets and streams into live performance environments and, to some extent, the home. When COVID-19 restrictions came into being in March 2020, all theatre productions stopped and the RSC had to start over. All it was left with, from its months of preparations for an in-person event, was the symphonic score recorded by the Philharmonia Orchestra, completed days before lockdown.

The RSC team decided to stage a 25-minute livestreamed performance titled Dreamtaking inspiration from Shakespeare’s A Midsummer Night’s Dream. It would use live motion – capture fed through Unreal Engine, a game engine developed by software engineers at Epic Games – to film and form avatars for Puck and other sprites. The platform is used by film production companies as well as video games developers and enables the making of realistic 3D graphics. The RSC also used it to create the graphics for and control a motion capture character of Ariel in a 2016 production of the The Tempest.

The RSC consortium created a virtual set that tracks, with a camera’s motion, around actors and objects and rendered everything in real time to a large LED screen. This would then be broadcast live to thousands of subscribers across the world on tablet, desktop and mobile.

The RSC consortium used the University of Portsmouth as its virtual venue. The university is part of the Joint Academic Network (Janet) that provides a secure, fast fibre-optic backbone to the UK’s research and education community. Its bandwidth could keep a dedicated stream going out to audiences worldwide.

Alex Counsell is the Technical Director for the Centre of Creative and Immersive eXtended Reality at the university and worked on Dream. He says: “We had five actors in motion capture suits covered in 67 markers each. I had 48 Vicon Vantage 16 megapixel cameras on the studio floor that created and streamed a digital skeleton for each actor across the network. Using Unreal Engine as our display tool we could create digital animation, communicate with other internal networks controlling the lighting as well as the QLab audio system. All of this information was projected onto our giant LED video wall that would draw the story world and broadcast in real time to the viewer.”

Alex and the other team members had duplicate systems in place so that each audience-critical piece of equipment would have a fail-safe in place. However, if the motion capture system stopped then it was game over: they couldn’t have another machine running in parallel to that. If the software crashed, as it did in the technical run-throughs, then a holding screen would be put up while they rebooted.

After months of preparation, the RSC broadcast 10 successful 25-minute performances of Dreamin March 2021 with no stoppages and no holding screen. It was a formidable technological achievement to be able to stream seamlessly across mobile phones, personal computers and tablets. There is inevitably a digital inequality in potential viewers’ systems and the live streaming had to work across all platforms.

Two of the notable interactive elements were with the audience and the actors. The event was live streamed for free but people could pay to have an interactive ticket. This would split the person’s screen into two and the new interface would allow a mouse or finger on a touchpad to drag back an illuminated firefly and fire it into the 3D space occupied by Dream’s Puck and fellow sprites. Puck, played by EM Williams, and the others would then follow the brightest path placed before them.

Another interactive groundbreaker used a Gestrument through Unreal Engine in a livestreamed performance of motion capture. Alex explains: “The Gestrument is like a Theremin where you use gestures to play music, but with no device. Each actor could control their individual signature instrument with movement of their hand and a simple pinch gesture could turn the instrument on or off to allow them to use it only when needed. Puck had a harp sound, and there was a flute, a piano and others. Pinching index and thumb switched the Gestrument on, then we followed the distance from their hips. Below hips meant the prerecorded music went down an octave or two, and the strength of the gestures would alter the attack and timbre. It could never go out of key but it gave each actor agency over their own instrument.

What’s next?

Sarah Ellis, RSC’s Director of Digital Development, said: “As we come out of this pandemic we need to look not only at what we lost but at what we found. How do we bring those audiences that can’t come to Stratford-upon-Avon but want to be with us? The jeopardy of live is big but we reached tens of thousands of people in over 70 countries with Dream. Now we have to look at the R&D developed and see what can be built on and how we could commercialise the elements that have been the most popular. We’ve shown that real-time tech works live and will influence how we create new production processes. It will enable a new creative vision that will connect with an audience at home.”

The esports demonstrator has also shown the possibility of working differently. The statistics surrounding esports are large, very large. It is estimated that global esports revenues will grow to £800 million in 2021, a year-on-year growth of 14.5% from 2020 figures. Trade researchers estimate that the global games livestreaming audience will reach more than 720 million people in 2021, up 10% from 2020.

Dr Block thinks that WEAVR’s potential goes further than esports. “So the key innovation here is that we are going away from the one-for-all coverage that traditional sports have,” he says. “That just won’t engage the audiences of the future. Teenagers want to consume what they want, when they want. They snack information and the consumption patterns are radically changing. The tailored AI-driven concentration can really address the appetite of the new audience and you will see that that will carry over into traditional sports and even into how mainstream entertainment is consumed. That’s the new future.”

***

This article has been adapted from "Entertaining audiences of the future", which originally appeared in the print edition of Ingenia 88 (September 2021).