Core_Audio徹底解剖.pdf

Core Audioపఈղ๤ iOSDC Day1 TrackC 17:20 - 17:40

ࣗݾ঺հ • Name: entaku • Work: Voicy (https://voicy.jp/) • X:
https://x.com/entaku_0818 • Hobby: • ԻָϑΣε • ݸਓ։ൃ ɹhttps://apps.apple.com/jp/developer/takuya-endo/id1601100107 • Other: • SwiftѪ޷ձӡӦ (https://love-swift.connpass.com/)

CoreAudioͱ͸ʁ

CoreAudioͱ͸ʁ https://ja.wikipedia.org/wiki/Core_Audio_(Apple) Core AudioʢίΞ ΦʔσΟΦʣ͸ɺAppleͷOS ʢmacOS͓ΑͼiOSɾiPadOSɾtvOSɾwatchOSɾ audioOSʣͰɺԻ੠Λѻ͏ϑϨʔϜϫʔΫͰ͋Δɻ

CoreAudioͱ͸ʁ https://developer.apple.com/library/archive/documentation/MusicAudio/Conceptual/CoreAudioOverview/Introduction/Introduction.html

CoreAudioͱ͸ʁ https://developer.apple.com/library/archive/documentation/MusicAudio/Conceptual/CoreAudioOverview/Introduction/Introduction.html Core Audio provides software interfaces for implementing audio
features in applications you create for iOS and OS X. Under the hood, it handles all aspects of audio on each of these platforms. In iOS, Core Audio capabilities include recording, playback, sound e ff ects, positioning, format conversion, and f ile stream parsing,

CoreAudioͱ͸ʁ https://developer.apple.com/library/archive/documentation/MusicAudio/Conceptual/CoreAudioOverview/Introduction/Introduction.html Core Audio provides software interfaces for implementing audio
features in applications you create for iOS and OS X. Under the hood, it handles all aspects of audio on each of these platforms. In iOS, Core Audio capabilities include recording, playback, sound e ff ects, positioning, format conversion, and f ile stream parsing, Core Audioͷػೳʹ͸࿥Իɺ࠶ੜɺԻڹޮՌɺҐஔܾΊɺ ϑΥʔϚοτม׵ɺͦͯ͠ϑΝΠϧετϦʔϜղੳؚ͕·Ε ·͢ɻ

CoreAudio͕ఏڙ͢ΔαʔϏε https://developer.apple.com/library/archive/documentation/MusicAudio/Conceptual/CoreAudioOverview/WhatisCoreAudio/WhatisCoreAudio.html You fi nd Core Audio application-level services in
the Audio Toolbox and Audio Unit frameworks. • Use Audio Queue Services to record, play back, pause, loop, and synchronize audio. • Use Audio File, Converter, and Codec Services to read and write from disk and to perform audio data format transformations. In OS X you can also create custom codecs. • Use Audio Unit Services and Audio Processing Graph Services (represented in the fi gure as “Audio units”) to host audio units (audio plug-ins) in your application. In OS X you can also create custom audio units to use in your application or to provide for use in other applications. • Use Music Sequencing Services to play MIDI-based control and music data. • Use Core Audio Clock Services for audio and MIDI synchronization and time format management. • Use System Sound Services (represented in the fi gure as “System sounds”) to play system sounds and user- interface sound e ff ects.

iOS Core Audio architecture https://developer.apple.com/library/archive/documentation/MusicAudio/Conceptual/CoreAudioOverview/WhatisCoreAudio/WhatisCoreAudio.html

Core AudioͷΞʔΩςΫνϟͷ֓ཁ https://developer.apple.com/library/archive/documentation/MusicAudio/Conceptual/CoreAudioOverview/CoreAudioEssentials/CoreAudioEssentials.html

the Audio Toolbox and Audio Unit frameworks. • Use Audio Queue Services to record, play back, pause, loop, and synchronize audio. • Use Audio File, Converter, and Codec Services to read and write from disk and to perform audio data format transformations. In OS X you can also create custom codecs. • Use Audio Unit Services and Audio Processing Graph Services (represented in the fi gure as “Audio units”) to host audio units (audio plug-ins) in your application. In OS X you can also create custom audio units to use in your application or to provide for use in other applications. • Use Music Sequencing Services to play MIDI-based control and music data. • Use Core Audio Clock Services for audio and MIDI synchronization and time format management. • Use System Sound Services (represented in the fi gure as “System sounds”) to play system sounds and user- interface sound e ff ects.

the Audio Toolbox and Audio Unit frameworks. •Use Audio Queue Services to record, play back, pause, loop, and synchronize audio. •Use Audio Unit Services and Audio Processing Graph Services (represented in the fi gure as “Audio units”) to host audio units (audio plug-ins) in your application. In OS X you can also create custom audio units to use in your application or to provide for use in other applications. • Use Audio File, Converter, and Codec Services to read and write from disk and to perform audio data format transformations. In OS X you can also create custom codecs. • Use Music Sequencing Services to play MIDI-based control and music data. • Use Core Audio Clock Services for audio and MIDI synchronization and time format management. • Use System Sound Services (represented in the fi gure as “System sounds”) to play system sounds and user- interface sound e ff ects.

۩ମྫͱͯ͠ Ի੠Λ࿥Ի͢Δ ॲཧΛݟ͍͖ͯ·͠ΐ͏

https://github.com/entaku0818/coreAudioApp αϯϓϧίʔυ

• AVAudioRecorder: • ࠓճ঺հ͢ΔதͰ࠷΋ߴϨϕϧͳAPIͰɺ࢖͍΍͢͞Λॏࢹ͍ͯ͠·͢ɻ • AVAudioEngine: • AVAudioRecorderΑΓ͸ॊೈੑͷߴ͘গ͠ෳࡶͳ͜ͱ͕Ͱ͖Δ • AudioQueueServices
(CoreAudio): • Ի੠ͷ࠶ੜ΍࿥Իͷ໾ׂΛ࣋ͭ • CoreAudioͷதͰ͸࠷΋ߴϨϕϧ • Audio Unit (CoreAudio): • ࠷΋௿ϨϕϧͰɺ࠷େͷॊೈੑΛ࣋ͭAPIͰ͢ɻ • ΦʔσΟΦॲཧͷ࠷খ୯Ґͱͯ͠ػೳ͠·͢ɻ ࠓճ঺հ͢ΔAudioAPIୡ

AVAudioRecorder

Ի੠Λ࿥Ի͢Δ func startRecording() { let audioSession = AVAudioSession.sharedInstance() do {
try audioSession.setCategory(.playAndRecord, mode: .default, options: [.defaultToSpeaker]) try audioSession.setActive(true) } catch { print("ΦʔσΟΦηογϣϯͷઃఆʹࣦഊ͠·ͨ͠: \(error.localizedDescription)") return } let audioFilename = getDocumentsDirectory().appendingPathComponent("recording.m4a") let settings = [ AVFormatIDKey: Int(kAudioFormatMPEG4AAC), AVSampleRateKey: 44800, AVNumberOfChannelsKey: 1, AVEncoderAudioQualityKey: AVAudioQuality.high.rawValue ] do { audioRecorder = try AVAudioRecorder(url: audioFilename, settings: settings) audioRecorder?.delegate = self audioRecorder?.record() } catch { // ΤϥʔϋϯυϦϯά print("࿥Իͷ։࢝ʹࣦഊ͠·ͨ͠: \(error.localizedDescription)") } }

ΦʔσΟΦηογϣϯͷઃఆ let audioSession = AVAudioSession.sharedInstance() do { try audioSession.setCategory(.playAndRecord, mode:
.default, options: [.defaultToSpeaker]) try audioSession.setActive(true) } catch { print("ΦʔσΟΦηογϣϯͷઃఆʹࣦഊ͠·ͨ͠: \(error.localizedDescription)") return }

࿥ԻอଘܗࣜΛઃఆ͢Δ let audioFilename =ɹ getDocumentsDirectory().appendingPathComponent("recording.m4a") let settings = [ ɹɹɹAVFormatIDKey:
Int(kAudioFormatMPEG4AAC), ɹɹɹAVSampleRateKey: 44800, ɹɹɹAVNumberOfChannelsKey: 1, ɹɹɹAVEncoderAudioQualityKey: AVAudioQuality.high.rawValue ]

Ի੠Λ࿥Ի͢Δ do { audioRecorder = try AVAudioRecorder(url: audioFilename, settings: settings)
audioRecorder?.delegate = self audioRecorder?.record() } catch { print("࿥Իͷ։࢝ʹࣦഊ͠·ͨ͠: \(error.localizedDescription)") }

࿥Իͷఀࢭ func stopRecording() { audioRecorder?.stop() audioRecorder = nil let audioSession
= AVAudioSession.sharedInstance() do { try audioSession.setActive(false) } catch { print("ΦʔσΟΦηογϣϯͷඇΞΫςΟϒԽʹࣦഊ͠·ͨ͠: \(error.localizedDescription)") } }

AVAudioEngine

AVAudioEngine https://developer.apple.com/documentation/avfaudio/avaudioengine The audio engine provides a powerful, feature-rich API
to simplify audio generation, processing, and input/output tasks. The engine contains a group of nodes that connect to form an audio signal processing chain. These nodes perform a variety of tasks on a signal before rendering to an output destination. Audio Engine helps you achieve simple, as well as complex, audio processing tasks. With Audio Engine, your apps can: • Play audio using f iles and bu ff ers • Capture audio at any point during the processing chain • Add built-in e ff ects like reverb, delay, distortion, and your custom e ff ects • Perform stereo and 3D mixing • Provide MIDI playback and control over sampler instruments

AVAudioEngine https://developer.apple.com/documentation/avfaudio/avaudioengine The audio engine provides a powerful, feature-rich API
to simplify audio generation, processing, and input/output tasks. The engine contains a group of nodes that connect to form an audio signal processing chain. These nodes perform a variety of tasks on a signal before rendering to an output destination. Audio Engine helps you achieve simple, as well as complex, audio processing tasks. With Audio Engine, your apps can: • Play audio using f iles and bu ff ers • Capture audio at any point during the processing chain • Add built-in e ff ects like reverb, delay, distortion, and your custom e ff ects • Perform stereo and 3D mixing • Provide MIDI playback and control over sampler instruments ΦʔσΟΦΤϯδϯ͸ɺԻ੠ͷੜ੒ɺॲཧɺೖग़ྗλεΫΛ؆ૉԽ ͢ΔͨΊͷɺػೳ๛෋ͰڧྗͳAPIΛఏڙ͠·͢ɻ ͜ͷΤϯδϯʹ ͸ɺΦʔσΟΦ৴߸ॲཧνΣʔϯΛܗ੒͢ΔͨΊʹ઀ଓ͞ΕΔҰ܈ ͷϊʔυؚ͕·Ε͍ͯ·͢ɻ͜ΕΒͷϊʔυ͸ɺग़ྗઌʹϨϯμϦϯ ά͢Δલʹɺ৴߸ʹରͯ͠͞·͟·ͳλεΫΛ࣮ߦ͠·͢ɻ

AVAudioEngine࿥Իॲཧશମ func startRecording(filename: String) { // ΦʔσΟΦηογϣϯΛΞΫςΟϒʹઃఆ let audioSession =
AVAudioSession.sharedInstance() do { try audioSession.setCategory(.playAndRecord, mode: .default, options: [.defaultToSpeaker]) try audioSession.setActive(true) } catch { print("ΦʔσΟΦηογϣϯͷઃఆʹࣦഊ͠·ͨ͠: \(error.localizedDescription)") return } // AVAudioEngine ͷઃఆ audioEngine = AVAudioEngine() inputNode = audioEngine.inputNode let format = inputNode.outputFormat(forBus: 0) let audioFilename = getDocumentsDirectory().appendingPathComponent(filename) do { audioFile = try AVAudioFile(forWriting: audioFilename, settings: format.settings) } catch { print("ΦʔσΟΦϑΝΠϧͷ࡞੒ʹࣦഊ͠·ͨ͠: \(error.localizedDescription)") return } inputNode.installTap(onBus: 0, bufferSize: 1024, format: format) { (buffer, when) in do { try self.audioFile?.write(from: buffer) } catch { print("ΦʔσΟΦσʔλͷॻ͖ࠐΈʹࣦഊ͠·ͨ͠: \(error.localizedDescription)") } } do { try audioEngine.start() isRecording = true print("࿥ԻΛ։࢝͠·ͨ͠") } catch { print("ΦʔσΟΦΤϯδϯͷىಈʹࣦഊ͠·ͨ͠: \(error.localizedDescription)") }

AudioSettionͷઃఆ let audioSession = AVAudioSession.sharedInstance() do { try audioSession.setCategory(.playAndRecord, mode:

Ի੠ϑΝΠϧΛอଘ ɹlet audioFilename = getDocumentsDirectory().appendingPathComponent("recording.caf") do { audioFile = try
AVAudioFile(forWriting: audioFilename, settings: format.settings) } catch { print("ΦʔσΟΦϑΝΠϧͷ࡞੒ʹࣦഊ͠·ͨ͠: \(error.localizedDescription)") return }

AVAudioEngineͷઃఆ // AVAudioEngine ͷઃఆ audioEngine = AVAudioEngine() inputNode = audioEngine.inputNode
inputNode.installTap(onBus: 0, bufferSize: 1024, format: format) { (buffer, when) in do { try self.audioFile?.write(from: buffer) } catch { print("ΦʔσΟΦσʔλͷॻ͖ࠐΈʹࣦഊ͠·ͨ͠: \(error.localizedDescription)") } }

Ի੠Λ࿥Ի͢Δ do { try audioEngine.start() isRecording = true print("࿥ԻΛ։࢝͠·ͨ͠") }
catch { print("ΦʔσΟΦΤϯδϯͷىಈʹࣦഊ͠·ͨ͠: \(error.localizedDescription)") }

Ի੠࿥ԻΛఀࢭ͢Δ func stopRecording() { if isRecording { audioEngine.stop() inputNode.removeTap(onBus: 0)
isRecording = false print("࿥ԻΛఀࢭ͠·ͨ͠") // ΦʔσΟΦηογϣϯΛඇΞΫςΟϒʹઃఆ let audioSession = AVAudioSession.sharedInstance() do { try audioSession.setActive(false) } catch { print("ΦʔσΟΦηογϣϯͷඇΞΫςΟϒԽʹࣦഊ͠·ͨ͠: \(error.localizedDescription)") } } }

AudioQueueService

AudioQueueServiceͱ͸ https://developer.apple.com/documentation/audiotoolbox/audio_queue_services This document describes Audio Queue Services, a C
programming interface in the Audio Toolbox framework, which is part of Core Audio. An audio queue is a software object you use for recording or playing audio. An audio queue does the work of: • Connecting to audio hardware • Managing memory • Employing codecs, as needed, for compressed audio formats • Mediating playback or recording Audio Queue Services enables you to record and play audio in linear PCM, in compressed formats (such as Apple Lossless and AAC), and in other formats for which users have installed codecs. Audio Queue Services also supports scheduled playback and synchronization of multiple audio queues and synchronization of audio with video.

programming interface in the Audio Toolbox framework, which is part of Core Audio. An audio queue is a software object you use for recording or playing audio. An audio queue does the work of: • Connecting to audio hardware • Managing memory • Employing codecs, as needed, for compressed audio formats • Mediating playback or recording Audio Queue Services enables you to record and play audio in linear PCM, in compressed formats (such as Apple Lossless and AAC), and in other formats for which users have installed codecs. Audio Queue Services also supports scheduled playback and synchronization of multiple audio queues and synchronization of audio with video. Audio Queue Services͸ɺCore AudioͷҰ෦Ͱ͋ΔAudio Toolboxϑ ϨʔϜϫʔΫʹؚ·ΕΔCϓϩάϥϛϯάΠϯλʔϑΣʔεͰ͢ɻ ΦʔσΟΦͷ࿥Ի΍࠶ੜʹ࢖༻͢Διϑτ΢ΣΞΦϒδΣΫτͰ͢ɻ

programming interface in the Audio Toolbox framework, which is part of Core Audio. An audio queue is a software object you use for recording or playing audio. An audio queue does the work of: •Connecting to audio hardware •Managing memory •Employing codecs, as needed, for compressed audio formats •Mediating playback or recording Audio Queue Services enables you to record and play audio in linear PCM, in compressed formats (such as Apple Lossless and AAC), and in other formats for which users have installed codecs. Audio Queue Services also supports scheduled playback and synchronization of multiple audio queues and synchronization of audio with video.

programming interface in the Audio Toolbox framework, which is part of Core Audio. An audio queue is a software object you use for recording or playing audio. An audio queue does the work of: •Connecting to audio hardware •Managing memory •Employing codecs, as needed, for compressed audio formats •Mediating playback or recording Audio Queue Services enables you to record and play audio in linear PCM, in compressed formats (such as Apple Lossless and AAC), and in other formats for which users have installed codecs. Audio Queue Services also supports scheduled playback and synchronization of multiple audio queues and synchronization of audio with video. Audio Queueͷओͳػೳ: - ΦʔσΟΦϋʔυ΢ΣΞͱͷ઀ଓ - ϝϞϦ؅ཧ - ѹॖΦʔσΟΦϑΥʔϚοτ༻ͷίʔσοΫར༻ - ࠶ੜ΍࿥Իͷௐ੔

AudioQueueServiceͷ࿥Իͱ࠶ੜͷ࢓૊Έ https://developer.apple.com/documentation/audiotoolbox/audio_queue_services

AudioQueueServiceશମ func startRecording(filename:String) { guard !isRecording else { logger.warning("Recording already
in progress") return } do { let session = AVAudioSession.sharedInstance() try session.setCategory(.playAndRecord, mode: .default, options: [.defaultToSpeaker, .mixWithOthers]) try session.setActive(true) logger.info("Audio session setup completed") } catch { logger.error("Failed to setup audio session: \(error.localizedDescription)") } let documentsPath = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0] let audioFilePath = documentsPath.appendingPathComponent(filename) var audioFile: AudioFileID? let createStatus = AudioFileCreateWithURL(audioFilePath as CFURL, kAudioFileCAFType, &recordingFormat, .eraseFile, &audioFile) guard createStatus == noErr else { logger.error("Failed to create audio file: \(createStatus)") return } recordingFile = audioFile logger.info("Audio file created successfully") var status = AudioQueueNewInput(&recordingFormat, audioQueueInputCallback, Unmanaged.passUnretained(self).toOpaque(), nil, nil, 0, &recordingQueue) guard status == noErr else { logger.error("Failed to create new audio queue input: \(status)") return } logger.info("Audio queue input created successfully") let bufferByteSize = deriveBufferSize(for: recordingFormat, seconds: 0.5) for _ in 0..<3 { var buffer: AudioQueueBufferRef? status = AudioQueueAllocateBuffer(recordingQueue!, UInt32(bufferByteSize), &buffer) guard status == noErr else { logger.error("Failed to allocate audio queue buffer: \(status)") return } if let buffer = buffer { status = AudioQueueEnqueueBuffer(recordingQueue!, buffer, 0, nil) guard status == noErr else { logger.error("Failed to enqueue audio queue buffer: \(status)") return } } } logger.info("Audio queue buffers allocated and enqueued successfully") status = AudioQueueStart(recordingQueue!, nil) guard status == noErr else { logger.error("Failed to start audio queue: \(status)") return } isRecording = true currentPacket = 0 logger.info("Recording started successfully") }

AudioSettionͷઃఆ let audioSession = AVAudioSession.sharedInstance() do { try audioSession.setCategory(.playAndRecord, mode:

Ի੠ϑΝΠϧͷ࡞੒ var audioFile: AudioFileID? let createStatus = AudioFileCreateWithURL(audioFilePath as CFURL,
kAudioFileCAFType, &recordingFormat, .eraseFile, &audioFile) guard createStatus == noErr else { logger.error("Failed to create audio file: \(createStatus)") return } recordingFile = audioFile logger.info("Audio file created successfully")

AudioQueueΛ࢖༻ͯ͠ ΦʔσΟΦೖྗΩϡʔͷઃఆΛߦ͏ var status = AudioQueueNewInput(&recordingFormat, audioQueueInputCallback, Unmanaged.passUnretained(self).toOpaque(), nil, nil,
0, &recordingQueue) guard status == noErr else { logger.error("Failed to create new audio queue input: \(status)") return } logger.info("Audio queue input created successfully")

Audio Queue Input Callback let audioQueueInputCallback: AudioQueueInputCallback = { inUserData,
inAQ, inBuffer, inStartTime, inNumberPacketDescriptions, inPacketDescs in guard let inUserData = inUserData else { return } let audioManager = Unmanaged<AudioQueueServices>.fromOpaque(inUserData).takeUnretainedValue() guard let recordingFile = audioManager.recordingFile else { audioManager.logger.error("Recording file is nil in input callback") return } var inNumberPacketDescriptions = inNumberPacketDescriptions let writeStatus = AudioFileWritePackets(recordingFile, false, inBuffer.pointee.mAudioDataByteSize, inPacketDescs, audioManager.currentPacket, &inNumberPacketDescriptions, inBuffer.pointee.mAudioData) if writeStatus != noErr { audioManager.logger.error("Failed to write packets to file: \(writeStatus)") } audioManager.currentPacket += Int64(inNumberPacketDescriptions) let enqueueStatus = AudioQueueEnqueueBuffer(inAQ, inBuffer, 0, nil) if enqueueStatus != noErr { audioManager.logger.error("Failed to enqueue buffer in input callback: \(enqueueStatus)") } }

࿥ԻϑΝΠϧͷ֬ೝ let audioQueueInputCallback: AudioQueueInputCallback = { inUserData, inAQ, inBuffer, inStartTime,
inNumberPacketDescriptions, inPacketDescs in guard let inUserData = inUserData else { return } let audioManager = Unmanaged<AudioQueueServices>.fromOpaque(inUserData).takeUnretainedValue() guard let recordingFile = audioManager.recordingFile else { audioManager.logger.error("Recording file is nil in input callback") return } var inNumberPacketDescriptions = inNumberPacketDescriptions let writeStatus = AudioFileWritePackets(recordingFile, false, inBuffer.pointee.mAudioDataByteSize, inPacketDescs, audioManager.currentPacket, &inNumberPacketDescriptions, inBuffer.pointee.mAudioData) if writeStatus != noErr { audioManager.logger.error("Failed to write packets to file: \(writeStatus)") } audioManager.currentPacket += Int64(inNumberPacketDescriptions) let enqueueStatus = AudioQueueEnqueueBuffer(inAQ, inBuffer, 0, nil) if enqueueStatus != noErr { audioManager.logger.error("Failed to enqueue buffer in input callback: \(enqueueStatus)") } }

ΦʔσΟΦσʔλͷॻ͖ࠐΈ let audioQueueInputCallback: AudioQueueInputCallback = { inUserData, inAQ, inBuffer, inStartTime,

όοϑΝͷ࠶ΤϯΩϡʔ let audioQueueInputCallback: AudioQueueInputCallback = { inUserData, inAQ, inBuffer, inStartTime,

ΦʔσΟΦόοϑΝͷׂΓ౰ͯ let bufferByteSize = deriveBufferSize(for: recordingFormat, seconds: 0.5) for _
in 0..<3 { var buffer: AudioQueueBufferRef? status = AudioQueueAllocateBuffer(recordingQueue!, UInt32(bufferByteSize), &buffer) guard status == noErr else { logger.error("Failed to allocate audio queue buffer: \(status)") return } if let buffer = buffer { status = AudioQueueEnqueueBuffer(recordingQueue!, buffer, 0, nil) guard status == noErr else { logger.error("Failed to enqueue audio queue buffer: \(status)") return } } } logger.info("Audio queue buffers allocated and enqueued successfully")

࿥Իͷ։࢝ status = AudioQueueStart(recordingQueue!, nil) guard status == noErr else
{ logger.error("Failed to start audio queue: \(status)") return } isRecording = true currentPacket = 0 logger.info("Recording started successfully")

Audio Unit

Audio Unit https://developer.apple.com/library/archive/documentation/MusicAudio/Conceptual/AudioUnitProgrammingGuide/AudioUnitDevelopmentFundamentals/ AudioUnitDevelopmentFundamentals.html What Is An Audio Unit? An
audio unit (often abbreviated as AU in header files and elsewhere) is an OS X plug-in that enhances digital audio applications such as Logic Pro and GarageBand. You can also use audio units to build audio features into your own application. Programmatically, an audio unit is packaged as a bundle and configured as a component as defined by the OS X Component Manager. At a deeper level, and depending on your viewpoint, an audio unit is one of two very different things. From the inside—as seen by an audio unit developer—an audio unit is executable implementation code within a standard plug-in API. The API is standard so that any application designed to work with audio units will know how to use yours. The API is defined by the Audio Unit Specification. An audio unit developer can add the ability for users or applications to control an audio unit in real time through the audio unit parameter mechanism. Parameters are self-describing; their values and capabilities are visible to applications that use audio units. From the outside—as seen from an application that uses the audio unit—an audio unit is just its plug-in API. This plug-in API lets applications query an audio unit about its particular features, defined by the audio unit developer as parameters and properties.

Audio Unit https://developer.apple.com/library/archive/documentation/MusicAudio/Conceptual/AudioUnitProgrammingGuide/AudioUnitDevelopmentFundamentals/ AudioUnitDevelopmentFundamentals.html What Is An Audio Unit? An
audio unit (often abbreviated as AU in header files and elsewhere) is an OS X plug-in that enhances digital audio applications such as Logic Pro and GarageBand. You can also use audio units to build audio features into your own application. Programmatically, an audio unit is packaged as a bundle and configured as a component as defined by the OS X Component Manager. At a deeper level, and depending on your viewpoint, an audio unit is one of two very different things. From the inside—as seen by an audio unit developer—an audio unit is executable implementation code within a standard plug-in API. The API is standard so that any application designed to work with audio units will know how to use yours. The API is defined by the Audio Unit Specification. An audio unit developer can add the ability for users or applications to control an audio unit in real time through the audio unit parameter mechanism. Parameters are self-describing; their values and capabilities are visible to applications that use audio units. From the outside—as seen from an application that uses the audio unit—an audio unit is just its plug-in API. This plug-in API lets applications query an audio unit about its particular features, defined by the audio unit developer as parameters and properties. Audio Unit͸ɺmacOS޲͚ͷϓϥάΠϯͰɺLogic Pro΍ GarageBandͳͲͷσδλϧΦʔσΟΦΞϓϦέʔγϣϯͷ ػೳΛ֦ு͠·͢ɻ·ͨɺಠࣗͷΞϓϦέʔγϣϯʹΦʔ σΟΦػೳΛ௥Ճ͢Δࡍʹ΋࢖༻Ͱ͖·͢ɻ

Audio Unit https://developer.apple.com/library/archive/documentation/MusicAudio/Conceptual/AudioUnitProgrammingGuide/AudioUnitDevelopmentFundamentals/ What Is An Audio Unit? An audio
unit (often abbreviated as AU in header files and elsewhere) is an OS X plug-in that enhances digital audio applications such as Logic Pro and GarageBand. You can also use audio units to build audio features into your own application. Programmatically, an audio unit is packaged as a bundle and configured as a component as defined by the OS X Component Manager. At a deeper level, and depending on your viewpoint, an audio unit is one of two very different things. From the inside—as seen by an audio unit developer—an audio unit is executable implementation code within a standard plug-in API. The API is standard so that any application designed to work with audio units will know how to use yours. The API is defined by the Audio Unit Specification. An audio unit developer can add the ability for users or applications to control an audio unit in real time through the audio unit parameter mechanism. Parameters are self-describing; their values and capabilities are visible to applications that use audio units. From the outside—as seen from an application that uses the audio unit—an audio unit is just its plug-in API. This plug-in API lets applications query an audio unit about its particular features, defined by the audio unit developer as parameters and properties.

Audio Unit https://developer.apple.com/library/archive/documentation/MusicAudio/Conceptual/AudioUnitProgrammingGuide/AudioUnitDevelopmentFundamentals/ What Is An Audio Unit? An audio
unit (often abbreviated as AU in header files and elsewhere) is an OS X plug-in that enhances digital audio applications such as Logic Pro and GarageBand. You can also use audio units to build audio features into your own application. Programmatically, an audio unit is packaged as a bundle and configured as a component as defined by the OS X Component Manager. At a deeper level, and depending on your viewpoint, an audio unit is one of two very different things. From the inside—as seen by an audio unit developer—an audio unit is executable implementation code within a standard plug-in API. The API is standard so that any application designed to work with audio units will know how to use yours. The API is defined by the Audio Unit Specification. An audio unit developer can add the ability for users or applications to control an audio unit in real time through the audio unit parameter mechanism. Parameters are self-describing; their values and capabilities are visible to applications that use audio units. From the outside—as seen from an application that uses the audio unit—an audio unit is just its plug-in API. This plug-in API lets applications query an audio unit about its particular features, defined by the audio unit developer as parameters and properties. ಺෦͔Βͷࢹ఺ʢ։ൃऀࢹ఺ʣ: ඪ४Խ͞ΕͨϓϥάΠϯAPIΛ࣮࣋ͭߦՄೳͳίʔυ - Audio Unit Speci f icationͰఆٛ͞ΕͨAPI - ύϥϝʔλػߏΛ௨ͯ͡ϦΞϧλΠϜ੍ޚ͕Մೳ - ύϥϝʔλ͸ࣗݾهड़తͰɺ஋΍ػೳ͕ΞϓϦέʔγϣϯ͔Βݟ͑Δ ֎෦͔Βͷࢹ఺ʢ࢖༻͢ΔΞϓϦέʔγϣϯࢹ఺ʣ: - ϓϥάΠϯAPIͦͷ΋ͷ - ΞϓϦέʔγϣϯ͕Audio UnitͷಛఆͷػೳΛরձͰ͖ΔAPI - ։ൃऀ͕ఆٛͨ͠ύϥϝʔλͱϓϩύςΟΛ௨ͯ͡ػೳΛެ։

Audio UnitͷॲཧͷྲྀΕ • setupAudioUnitɹ→΄ͱΜͲͷ࣮૷͕ͪ͜ΒΑΓ • startRecording

setupAudioUnitશମ private func setupAudioUnit() { var audioComponentDescription = AudioComponentDescription( componentType:
kAudioUnitType_Output, componentSubType: kAudioUnitSubType_RemoteIO, componentManufacturer: kAudioUnitManufacturer_Apple, componentFlags: 0, componentFlagsMask: 0 ) guard let audioComponent = AudioComponentFindNext(nil, &audioComponentDescription) else { print("Failed to find audio component") return } var status = AudioComponentInstanceNew(audioComponent, &audioUnit) guard status == noErr else { print("Failed to create audio unit: \(status)") return } var oneFlag: UInt32 = 1 status = AudioUnitSetProperty(audioUnit!, kAudioOutputUnitProperty_EnableIO, kAudioUnitScope_Input, 1, &oneFlag, UInt32(MemoryLayout<UInt32>.size)) guard status == noErr else { print("Failed to enable audio input: \(status)") return } status = AudioUnitSetProperty(audioUnit!, kAudioUnitProperty_StreamFormat, kAudioUnitScope_Output, 1, &audioFormat, UInt32(MemoryLayout<AudioStreamBasicDescription>.size)) guard status == noErr else { print("Failed to set stream format: \(status)") return } var callbackStruct = AURenderCallbackStruct( inputProc: recordingCallback, inputProcRefCon: Unmanaged.passUnretained(self).toOpaque() ) status = AudioUnitSetProperty(audioUnit!, kAudioOutputUnitProperty_SetInputCallback, kAudioUnitScope_Global, 0, &callbackStruct, UInt32(MemoryLayout<AURenderCallbackStruct>.size)) guard status == noErr else { print("Failed to set recording callback: \(status)") return } print("Audio unit setup completed") }

ΦʔσΟΦίϯϙʔωϯτઃఆ var audioComponentDescription = AudioComponentDescription( componentType: kAudioUnitType_Output, componentSubType: kAudioUnitSubType_RemoteIO, componentManufacturer:
kAudioUnitManufacturer_Apple, componentFlags: 0, componentFlagsMask: 0 ) guard let audioComponent = AudioComponentFindNext(nil, &audioComponentDescription) else { print("Failed to find audio component") return } var status = AudioComponentInstanceNew(audioComponent, &audioUnit) guard status == noErr else { print("Failed to create audio unit: \(status)") return }

ετϦʔϜϑΥʔϚοτͷઃఆ guard status == noErr else { print("Failed to enable
audio input: \(status)") return } status = AudioUnitSetProperty(audioUnit!, kAudioUnitProperty_StreamFormat, kAudioUnitScope_Output, 1, &audioFormat, UInt32(MemoryLayout<AudioStreamBasicDescription>.size))

AudioUnit΁ίʔϧόοΫͷઃఆ var callbackStruct = AURenderCallbackStruct( inputProc: recordingCallback, inputProcRefCon: Unmanaged.passUnretained(self).toOpaque() )
status = AudioUnitSetProperty(audioUnit!, kAudioOutputUnitProperty_SetInputCallback, kAudioUnitScope_Global, 0, &callbackStruct, UInt32(MemoryLayout<AURenderCallbackStruct>.size)) guard status == noErr else { print("Failed to set recording callback: \(status)") return } print("Audio unit setup completed")

recordingCallback func recordingCallback(inRefCon: UnsafeMutableRawPointer, ioActionFlags: UnsafeMutablePointer<AudioUnitRenderActionFlags>, inTimeStamp: UnsafePointer<AudioTimeStamp>, inBusNumber: UInt32,
inNumberFrames: UInt32, ioData: UnsafeMutablePointer<AudioBufferList>?) -> OSStatus { let audioRecorderPlayer = Unmanaged<AudioUnitManager>.fromOpaque(inRefCon).takeUnretainedValue() guard let audioUnit = audioRecorderPlayer.audioUnit else { print("Audio unit not available in recording callback") return kAudioUnitErr_InvalidProperty } var bufferList = AudioBufferList( mNumberBuffers: 1, mBuffers: AudioBuffer( mNumberChannels: 1, mDataByteSize: inNumberFrames * 4, mData: nil ) ) let status = AudioUnitRender(audioUnit, ioActionFlags, inTimeStamp, inBusNumber, inNumberFrames, &bufferList) if status == noErr { let buffer = bufferList.mBuffers let samples = buffer.mData?.assumingMemoryBound(to: Float.self) let count = Int(buffer.mDataByteSize) / MemoryLayout<Float>.size let audioData = Array(UnsafeBufferPointer(start: samples, count: count)) audioRecorderPlayer.appendAudioData(audioData) } return status

AudioUnitManagerͷऔಘ let audioRecorderPlayer = Unmanaged<AudioUnitManager>.fromOpaque(inRefCon).takeUnretainedValue() guard let audioUnit = audioRecorderPlayer.audioUnit
else { print("Audio unit not available in recording callback") return kAudioUnitErr_InvalidProperty } var bufferList = AudioBufferList( mNumberBuffers: 1, mBuffers: AudioBuffer( mNumberChannels: 1, mDataByteSize: inNumberFrames * 4, mData: nil ) ) let status = AudioUnitRender(audioUnit,ɹioActionFlags,ɹinTimeStamp,ɹinBusNumber,ɹinNumberFrames,ɹ&bufferList) if status == noErr { let buffer = bufferList.mBuffers let samples = buffer.mData?.assumingMemoryBound(to: Float.self) let count = Int(buffer.mDataByteSize) / MemoryLayout<Float>.size let audioData = Array(UnsafeBufferPointer(start: samples, count: count)) audioRecorderPlayer.appendAudioData(audioData) } return status

AudioUnitͷ֬ೝ let audioRecorderPlayer = Unmanaged<AudioUnitManager>.fromOpaque(inRefCon).takeUnretainedValue() guard let audioUnit = audioRecorderPlayer.audioUnit

ΦʔσΟΦσʔλΛอ࣋͢ΔͨΊͷAudioBufferListΛ࡞੒ let audioRecorderPlayer = Unmanaged<AudioUnitManager>.fromOpaque(inRefCon).takeUnretainedValue() guard let audioUnit = audioRecorderPlayer.audioUnit

όοϑΝ͔ΒΦʔσΟΦσʔλΛऔಘ͠ɺBufferΛ௥Ճ͢Δ let audioRecorderPlayer = Unmanaged<AudioUnitManager>.fromOpaque(inRefCon).takeUnretainedValue() guard let audioUnit = audioRecorderPlayer.audioUnit
else { print("Audio unit not available in recording callback") return kAudioUnitErr_InvalidProperty } var bufferList = AudioBufferList( mNumberBuffers: 1, mBuffers: AudioBuffer( mNumberChannels: 1, mDataByteSize: inNumberFrames * 4, mData: nil ) ) let status = AudioUnitRender(audioUnit,ɹioActionFlags,ɹinTimeStamp,ɹinBusNumber,ɹinNumberFrames,ɹ&bufferList) if status == noErr { let buffer = bufferList.mBuffers let samples = buffer.mData?.assumingMemoryBound(to: Float.self) let count = Int(buffer.mDataByteSize) / MemoryLayout<Float>.size let audioData = Array(UnsafeBufferPointer(start: samples, count: count)) audioRecorderPlayer.appendAudioData(audioData) } return status func appendAudioData(_ data: [Float]) { recordedAudioBuffer.append(contentsOf: data) }

࿥Ի։࢝ func startRecording(filename:String) { guard !isRecording, let audioUnit = audioUnit
else { print("Recording already in progress or audio unit not set up") return } let documentsPath = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0] let audioFilePath = documentsPath.appendingPathComponent(filename) var status = AudioFileCreateWithURL( audioFilePath as CFURL, kAudioFileCAFType, &audioFormat, .eraseFile, &audioFile ) guard status == noErr else { print("Failed to create audio file: \(status)") return } recordedAudioBuffer.removeAll() status = AudioOutputUnitStart(audioUnit) guard status == noErr else { print("Failed to start audio unit: \(status)") return } isRecording = true print("Recording started") }

• AVAudioRecorder: • ࠓճ঺հ͢ΔதͰ࠷΋ߴϨϕϧͳAPIͰɺ࢖͍΍͢͞Λॏࢹ͍ͯ͠·͢ɻ • AVAudioEngine: • AVAudioRecorderΑΓ͸ॊೈੑͷߴ͘গ͠ෳࡶͳ͜ͱ͕Ͱ͖Δ • AudioQueueServices
(CoreAudio): • Ի੠ͷ࠶ੜ΍࿥Իͷ໾ׂΛ࣋ͭ • CoreAudioͷதͰ͸࠷΋ߴϨϕϧ • Audio Unit (CoreAudio): • ࠷΋௿ϨϕϧͰɺ࠷େͷॊೈੑΛ࣋ͭAPIͰ͢ɻ • ΦʔσΟΦॲཧͷ࠷খ୯Ґͱͯ͠ػೳ͠·͢ɻ AudioAPIୡΛ঺հ͠·ͨ͠

ԻྔΛݟ͑ΔԽ

ϦΞϧλΠϜͷΞοϓϩʔυ

ԻྔΛݟ͑ΔԽ self.recognitionTask = speechRecognizer?.recognitionTask(with: recognitionRequest) { result, error in if
let result = result { self.resultText = result.bestTranscription.formattedString } } self.recognitionRequest?.append(buffer)

🤗͋ͳͨͷࣄྫΛ͓଴͍ͪͯ͠·͢🤗

https://github.com/entaku0818/coreAudioApp αϯϓϧίʔυ

ࢀߟจݙ https://developer.apple.com/library/archive/documentation/MusicAudio/Conceptual/ CoreAudioOverview/Introduction/Introduction.html AVAudioRecorder https://developer.apple.com/documentation/avfaudio/avaudiorecorder AVAudioEngine https://developer.apple.com/documentation/avfaudio/avaudioengine Audio Queue Service
https://developer.apple.com/documentation/audiotoolbox/audio_queue_services AudioUnit https://developer.apple.com/library/archive/documentation/MusicAudio/Conceptual/ AudioUnitProgrammingGuide/Introduction/Introduction.html#//apple_ref/doc/uid/ TP40003278-CH1-SW2

ࢀߟจݙ https://www.amazon.co.jp/iPhone-Core- Audio%E3%83%97%E3%83%AD%E3%82%B0%E3%83%A9%E3%83%9F%E3%83%B3%E3%82%B0 -%E6%B0%B8%E9%87%8E-%E5%93%B2%E4%B9%85/dp/4797355158 CoreAudioͷ།Ұͱݴ͍͍ͬͯຊ͓͢͢ΊͰ͢ʂ

એ఻ https://love-swift.connpass.com/event/328707 2024/09/24(Ր)ʹѪ޷ձͷษڧձΛ࣮ࢪ͠· ͢ʂ iOSDCͰܹࢗΛड͚ͨօ͞ΜͥͻࢀՃͯ͘͠ ͍ͩ͞ʂ

https://github.com/entaku0818/coreAudioApp αϯϓϧίʔυ ελʔ͍ͩ͘͞ʂʂ 🤗࣭͝໰͝ཁ๬͓଴ͪͯ͠·͢🤗

Core Audioపఈղ๤ iOSDC Day1 TrackC 17:20 - 17:40

Core_Audio徹底解剖.pdf

Core_Audio徹底解剖.pdf

More Decks by entaku

Other Decks in Programming

Featured

Transcript