Spem in Alium in 360 Video

The most recent 360 video in the Trinity360 series has now been released and can be seen below. This is the first of two performances of spatial choral music recorded as part of the Trinity360 series, and features a performance by the award winning choir New Dublin Voices of Thomas Tallis’ landmark composition, Spem in Alium, recorded in the Beckett Theatre, Trinity College Dublin on May 19th, 2019. The performance is also the second in a series of videos presented in partnership with SoundFieldTM Microphones whom I would like to thank for their sponsorship of the production.

There is also a short behind-the-scenes video about the performance, produced by Mark Linnane, which can be seen on the Soundfield website, here.

A previous post on this blog discussed the historical background to this fantastic example of 16th century spatial music, so this article will concentrate on the technical production of the 360 audio and video, and also a brief discussion of an alternative spatial layout of the eight choirs, compared to the traditional sequential layout presented above.

Venues such as churches and cathedrals which are typically used for performances of spatial choral music present a number of technical and logistical challenges for 360 video and audio. Due to the large size of such venues, it can be difficult to position all of the singers  close enough to the camera and microphones, and in addition, they do not typically allow for extensive control over lighting. The highly reverberant nature of such venues also presents some challenges, particularly when singers are widely distributed around the space. For these reasons, it was decided to instead record the performance in the more controlled environment of the Beckett Theatre in Trinity College Dublin, which allowed for the relatively tight spacing of the singers around the camera, and spot lighting of each choir using the in-house rig.

The heavy drapes hung in the theatre all around the choir significantly deadened the acoustic of the space, which helped ensure that the maximum directionality was captured in the audio recording. While this worked very well in that regard, it did result in a dryer recording that would be expected for a choral piece, and so additional reverberation was therefore added in post to create a more natural sounding acoustic environment. This was implemented using the excellent MS5 3D surround reverb plugin developed by Blue Ripple Sound and released as part of their O3A Reverb library. Although this plugin can be used to generate highly detailed early reflections, as well as late arriving, diffuse reverberation, in this instance the addition of early reflections was not particularly beneficial, and so only diffuse reverberation was added to the overall mix.

Spem- Mic Setup-fix

For the audio recording, a Soundfield ST450 MkII Ambisonic microphone was positioned centrally and just below the GoPro Omni 360 camera. In addition, eight Rode NT5 cardioid microphones were also used, and arranged in an octagon with a 45º subtended angle for each microphone pair. This microphone arrangement is known as the Equal Segment Microphone Array (ESMA), and was developed by Hyunkook Lee from previous work by Michael Williams. There are a number of different arrangements of the ESMA, however, in this instance, and in order to maximize directional accuracy in the horizontal plane, the ESMA configuration containing 8 cardioid microphones in a regular octagon was used. Two different microphone spacings have been proposed for this particular octagonal arrangement, 82cm according to Williams, and 55cm according to Lee. In this instance, the wider spacing of 82cm was used in order to maximize the difference between this microphone arrangement, and the more traditional 1st order Ambisonic (FOA) microphone, particularly in terms of the degree of spaciousness captured by near-coincident arrays such as ESMA when compared to entirely coincident FOA microphones. While still less commonly used for 360 recordings when compared to FOA (mostly due to the larger physical footprint and greater numbers of mic stands and cabling required), ESMA and other near-coincident or spaced microphone techniques such as ORTF-3D are certainly a viable alternative to FOA.

It is worth remembering that although Ambisonics remains the primary format for the final delivery and playback of spatial audio for 360 video (due to the ease with which this format can be rotated for head-tracking), this does not necessarily imply or require the use of Ambisonic microphones. Other arrangements can also be used by encoding each microphone signal into Ambisonics at the appropriate azimuth and elevation angle, and combining these to generate a FOA or TOA mix to attach to the 360 video.

In general, a traditional FOA microphone is often practically advantageous in certain contexts, due to their relative simplicity, small physical footprint, and the small number of audio channels involved. However, these advantages must be balanced with some potential drawbacks such as a lack of spaciousness and envelopment (which is typical of  all coincident microphone techniques).

In contrast, near-coincident or spaced microphone arrays typically capture a more spacious and enveloping soundfield, due to the capture of timing differences between the individual microphones. While this is generally highly beneficial when capturing ambiences, and stereo versions of such techniques are often preferred for classical music recordings, the large footprint of such arrays can be highly challenging to setup when on location, particularly for field recordings. Also, for 360 video this can result in a larger number of microphones (and stands and cabling) being visible in the footage, requiring additional work in post-production to remove them from the image. For this particular production, the theatre location and the high contrast spot lighting meant that neither of these issues was particularly problematic, and so we were able to record the choir using both types of techniques.

For this production, it was originally intended to solely use the additional ESMA recording for some experiments with a 6 Degrees of Freedom (DoF) version of this piece. The greater spacing of the microphones in this array is potentially useful in this regard as each microphone pair could potentially be used to provide a closer perspective on the specific choir (8 in total) positioned directly in front of that segment of the array (stay tuned for more on these 6 DoF experiments in a later post). However, when creating the audio mix for the video, it was found that a roughly equal combination of both the FOA and ESMA recordings produced the best result in terms of tonal balance, directional accuracy, and spaciousness. Which just goes to show that as always with audio, the “best” approach is always context dependent.

To create the final FOA mix for YouTube, each of the individual cardioid mic recordings from the ESMA was encoded into Ambisonics at the appropriate angles using the 3rd order ambiX encoder plugin developed by Matthias Kronlachner (the first four channels of which were used for the FOA mix). This was then combined with the FOA signal captured by the Soundfield ST450, and in addition, some corrective EQ was applied to both tracks using the freeware multiEQ plugin from the IEM Plugin suite. Finally, the entire FOA mix was routed through the Blue Ripple Sound MS5 surround reverb plugin, along with just a hint of compression using the Omnicompressor plugin, again from the IEM suite.

Although many 360 video platforms (including YouTube) and 360 video players currently only support FOA, a TOA mix of the piece was also created. For the ESMA signals, the TOA signals were already available from the ambiX encoders, and just required the routing of 16 audio channels (rather than 4) to the main mix output. In addition, the Harpex plugin  was used to upmix the original FOA ST450 recording to TOA. While the effectiveness of Harpex for this type of upmixing is highly dependent on the precise nature of the audio content, this produced excellent results here. While online 360 video platforms that support TOA are still fairly limited (currently just Facebook), some 360 video players (such as the ViveCinema app) do support local playback of 360 video files with TOA audio. As has been reported before in many previous studies, there was a noticeable improvement in directional accuracy and spatial clarity of the different singers when moving from FOA to TOA.

360 video was once again captured using the GoPro Omni camera system and stitched using the Autopano Video and Autopano Giga applications. As has been discussed previously on this blog, action cameras such as GoPro’s can struggle in low light conditions, or when the scene contains highly contrasting dark and light areas (something which was particularly evident here). Reducing the ISO limit and exposure compensation settings in the Omni rig certainly helped to limit the amount of image noise and graininess in the footage. However, further noise reduction was needed to reduce noise in certain parts of the image. While Adobe Premiere does include some noise reduction plugins, their performance leaves a lot to be desired and so for this production, we decided to purchase a different Noise Reduction plugin from Neat Video (as recommended by the members of the GoPro Omni Facebook Group). Although somewhat complex to setup, the Neat Video plugin produced significantly better results than the native plugins in Premiere, and is highly recommended, particularly given its relative affordability. Apart from some noise reduction, other standard processing such as sharpening, colour correction, and titles were all applied using the standard tools in the latest version of Premiere, which now contains extensive support for 360 video and Ambisonics (although FOA only).

In the video presented above, the eight choirs are arranged in a traditional, sequential layout which means that the consecutive entry of each choir at the beginning proceeds in a clockwise direction around the room. As discussed in a previous post, there is some evidence to suggest that the first performance of the piece occurred in Nonsuch Palace, which contained an octagonal banqueting hall with four first-floor balconies. However, this particular layout, which was replicated here in the Beckett Theatre using four elevated risers, suggests some other possible arrangements of the eight choirs. So, as well as the traditional sequential layout ,we also recorded the piece using an arrangement with choirs 1-4 at floor level at North, East, South and West directions, and choirs 5-8 elevated and at NorthWest, NorthEast, SouthEast, and SouthWest. In particular, we wanted to see if this could support the creation of a sort of spiral effect, as the consecutive entry of the choirs complete a full circle at floor level, before then completing a circle at the elevated position. This alternative layout of the choirs can be seen below, and you compare the two different configurations for yourself.

Stay tuned for more updates on this piece and some experiments with 6 DoF.

Many thanks to New Dublin Voices and their director Bernie Sherlock, Michael Canney and all of the crew at the Beckett Theatre, and Sebastian Csadi for their assistance during the shoot.

I would like to acknowledge the support of SoundFieldTM Microphones in providing equipment for this project.