Binaural reproduction from multiple microphone arrays Naoto Iijima, Shoichi Koyama, and Hiroshi Saruwatari (The University of Tokyo)

Binaural reproduction for VR audio š Reproduce sounds at two ears through headphones by using head-related transfer functions (HRTFs) š Binaural synthesis for VR space: š Given source signals / positions, room conditions, etc. š Convolution of source signal and HRTF š Binaural reproduction for real world: š Based on sound field estimation by using mic array š Capturing sound field in large region is necessary for 6DoF VR, but that is still challenging task

Binaural reproduction from mic array š 1st- / 2nd-order Ambisonic mics are now commercially available, but spatial resolution is very low š Spherical mic array is typically used for high-resolution recording, but large spherical array is neccesary for capturing sound field in large region Spherical mic array Capturing sound field in large region by single spherical array is sometimes impractical

Binaural reproduction from multiple mic arrays š Multiple small arrays (e.g., Ambisonic mics) for capturing sound field in large region š Sound field recording by flexible and scalable system How to integrate multiple mic array signals…? Multiple small arrays

Key technology š Recording š Sound field estimation by infinite-dimensional harmonic analysis š Reproduction š Binaural rendering from harmonic coefficients with compensation for loudspeaker distance in HRTF measurements • Iijima, et al., “Binaural Rendering From Microphone Array Signals of Arbitrary Geometry,” JASA, 2021. Highly-accurate binaural reproduction in large region can be achieved by flexible and scalable recording system

System š Combined 8 mic arrays š 8 uni-directional mics in each array (same placement as 2nd-order Ambisonics mic) š 360-degree camera for capturing images 2nd-order Ambisonic mic 360-degree camera

Evaluation: binaural reproduction error Proposed system 1st-order Ambisonic mic High accuracy in large region!

Demo š Recorded classical music (Dvořák – String Quartet No. 12) š Controllable on Web browser / HMD š 360-degree control of viewing and listening directions š 2 viewing and listening positions are selectable Position A Position B Composite mic array x y

Headphones Recommended

Thank you for watching! š References 1. N. Iijima, S. Koyama, and H. Saruwatari, “Binaural Rendering From Microphone Array Signals of Arbitrary Geometry,” Journal of the Acoustical Society of America (JASA), 2021. 2. N. Ueno, S. Koyama, and H. Saruwatari, “Sound Field Recording Using Distributed Microphones Based on Harmonic Analysis of Infinite Order,” IEEE Signal Processing Letters, vol. 25, no. 1, pp. 135- 139, 2018. š Further information š https://www.sh01.org/ š Performance by š 1st Vn.: Atsuko Sahara, 2nd Vn.: Erina Kato, Va.: Atsushi Yoshida, Vc.: Shogo Toyoda