Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

Swiftで高速フーリエ変換してオーディオビジュアライザーを作る / iOSDC Japan ...

Swiftで高速フーリエ変換してオーディオビジュアライザーを作る / iOSDC Japan 2024 Day1 Track D

Kyome (Takuto Nakamura)

August 24, 2024
Tweet

More Decks by Kyome (Takuto Nakamura)

Other Decks in Programming

Transcript

  1. iOSDC Japan 2024 - day1 Track D #iosdc #d ࣗݾ঺հ

    IUUQTLZPNFJP ,ZPNF αΠϘ΢ζͰ kintoneϞόΠϧ ͷiOS։ൃΛ୲౰ macOS޲͚ͷϢʔςΟϦςΟΞϓϦ։ൃऀ ੵۃతʹOSSΛ։ൃӡ༻ 2 Kyome
  2. iOSDC Japan 2024 - day1 Track D #iosdc #d ΦʔσΟΦϏδϡΞϥΠβʔͱ͸

    Իָ΍Ի੠ͷϦζϜ΍प೾਺ಛੑΛࢹ֮తʹදݱͨ͠΋ͷ Dynamic Island΍GarageBandɺϘΠεϝϞͳͲͰ΋͓ೃછΈ 4
  3. iOSDC Japan 2024 - day1 Track D #iosdc #d ΦʔσΟΦϏδϡΞϥΠβʔͱ͸

    Իָ΍Ի੠ͷϦζϜ΍प೾਺ಛੑΛࢹ֮తʹදݱͨ͠΋ͷ Dynamic Island΍GarageBandɺϘΠεϝϞͳͲͰ΋͓ೃછΈ 5 ͔͍͍ͬ͜ʂ࡞ͬͯΈ͍ͨʂ
  4. iOSDC Japan 2024 - day1 Track D #iosdc #d ΦʔσΟΦϏδϡΞϥΠβʔͷ࣮૷ํ਑

    1. AVAudioEngineͰԻָΛ࠶ੜ͠ɺԻݯσʔλΛऔಘ͢Δ 2. Accelerateͷߴ଎ϑʔϦΤม׵Ͱৼ෯εϖΫτϧΛࢉग़͢Δ 3. ࢉग़ͨ͠ৼ෯εϖΫτϧΛGraphicsContextͰՄࢹԽ͢Δ 7
  5. iOSDC Japan 2024 - day1 Track D #iosdc #d AVAudioEngineͰԻָΛ࠶ੜ͠ԻݯσʔλΛऔಘ͢Δ

    1/3 1. AVAudioEngineͰԻָΛ࠶ੜ͢Δ AVFoundationͷAVAudioEngineͱAVAudioPlayerNodeΛ༻͍Δ attach, connect, scheduleFile, start, playͷ࣮ߦॱং΍৚݅ʹ஫ҙ import AVFoundation let engine = AVAudioEngine() let playerNode = AVAudioPlayerNode() func play() throws { let url = Bundle.main.url(forResource: "sound", withExtension: "mp3")! let file = try AVAudioFile(forReading: url) let sampleRate = Float(file.processingFormat.sampleRate) engine.attach(playerNode) engine.connect(playerNode, to: engine.mainMixerNode, format: file.processingFormat) playerNode.scheduleFile(file, at: nil) try engine.start() playerNode.play() } 9
  6. iOSDC Japan 2024 - day1 Track D #iosdc #d AVAudioEngineͰԻָΛ࠶ੜ͠ԻݯσʔλΛऔಘ͢Δ

    2/3 2. Իָͷ࠶ੜʹ߹ΘͤͯԻݯσʔλΛऔಘ͢Δ ࠶ੜલʹAVAudioPlayerNodeʹinstallTapͰbu ff erSizeΛಡΈࠐΉͨͼʹ
 ࣮ߦ͞ΕΔॲཧΛొ࿥͓ͯ͘͠ ҰൠతͳԻָϑΝΠϧ͸αϯϓϧϨʔτ͕ 44.1kHz ͳͷͰɺ
 bu ff erSizeΛ 2048 ʹ͢Δͱ̍ඵؒʹ໿ 22 ճඳըͷߋ৽Λ͢Δ͜ͱʹͳΔ let file = try AVAudioFile(forReading: url) let sampleRate = Float(file.processingFormat.sampleRate) engine.attach(playerNode) engine.connect(playerNode, to: engine.mainMixerNode, format: file.processingFormat) playerNode.installTap(onBus: .zero, bufferSize: 2048, format: nil) { buffer, _ in if let data = buffer.floatChannelData { // ԻݯσʔλΛ༻͍ͨ೚ҙͷॲཧ } } 10
  7. iOSDC Japan 2024 - day1 Track D #iosdc #d AVAudioEngineͰԻָΛ࠶ੜ͠ԻݯσʔλΛऔಘ͢Δ

    3/3 3. ԻָΛఀࢭ͢Δ stop, removeTap, disconnectNodeOutput, detachͷ࣮ߦॱং΍৚݅ʹ஫ҙ ಛʹinstallTap͍ͯ͠ͳ͍ͷʹremoveTapͨ͠Γattach͍ͯ͠ͳ͍ͷʹdetach
 ͠Α͏ͱ͢ΔͱΫϥογϡ͢Δ͕ɺisPlaying΍isRunningͷΑ͏ͳϑϥά͸ͳ͍
 ͷͰࣗݾ؅ཧ͢Δඞཁ͕͋Δ func stop() { if playerNode.isPlaying { playerNode.stop() } playerNode.removeTap(onBus: .zero) if engine.isRunning { engine.stop() } engine.disconnectNodeOutput(playerNode) engine.detach(playerNode) } 11
  8. iOSDC Japan 2024 - day1 Track D #iosdc #d ߴ଎ϑʔϦΤม׵ͱ͸

    1/7 ϑʔϦΤڃ਺ల։ɿ
 ͋ΒΏΔपظతͳ೾͸؆୯ͳ೾ͷॏͶ߹ΘͤͰද͢͜ͱ͕Ͱ͖Δ
 ʢ؆୯ͳ೾ʹجຊप೾਺ͷ੔਺ഒͷप೾਺ͷਖ਼ݭ೾΍༨ݭ೾ʣ ʹ ʴ ʴ 15
  9. iOSDC Japan 2024 - day1 Track D #iosdc #d ߴ଎ϑʔϦΤม׵ͱ͸

    2/7 ϑʔϦΤม׵ɿ
 पظతͰͳ͍೾Ͱ͋ͬͯ΋ɺಛఆͷ۠ؒʹ஫໨͠पظతͰ͋ΔͱΈͳͯ͠
 ༷ʑͳप೾਺ͷਖ਼ݭ೾ɾ༨ݭ೾ʹ෼ղͯ͠ղੳ͕Մೳ Իڹಛੑͷղੳɿ
 Իͷೖྗ৴߸Λ೚ҙͷ۠ؒͰ۠੾ΓϑʔϦΤม׵Λߦ͏͜ͱͰɺ
 ֤۠ؒʹ͓͚Δप೾਺ಛੑΛղੳՄೳ ૭ɿ
 ϑʔϦΤม׵Λߦ͏۠ؒͷ͜ͱ 16
  10. iOSDC Japan 2024 - day1 Track D #iosdc #d ߴ଎ϑʔϦΤม׵ͱ͸

    3/7 ϑʔϦΤม׵ͷ෼ղೳɿ
 ϑʔϦΤม׵ʹΑΓແݶճͷ෼ղΛߦ͑͹ݩͷ೾ͱҰக͢Δ͕
 ༗ݶճͷ৔߹͸ۙࣅ೾ͱͳΔ 㲈 ʴ ʴ ʹ 17
  11. iOSDC Japan 2024 - day1 Track D #iosdc #d ߴ଎ϑʔϦΤม׵ͱ͸

    4/7 ૭ؔ਺ɿ
 ૭ͷ୺఺ͷෆ࿈ଓੑʹىҼ͢ΔϊΠζΛআڈ͢ΔͨΊʹ
 ۠ؒͷ୺఺Λ0ʹ͢ΔΑ͏ͳ૭ؔ਺Λ͔͚Δ 18
  12. iOSDC Japan 2024 - day1 Track D #iosdc #d ߴ଎ϑʔϦΤม׵ͱ͸

    5/7 ૭ؔ਺ͷछྨɿ
 प೾਺෼ղೳͱμΠφϛοΫϨϯδʢॲཧՄೳͳ৴߸ͷ࠷େ஋ͱ࠷খ஋ͷ ൺ཰ʣͷτϨʔυΦϑͷதͰ༷ʑͳ૭ؔ਺͕ߟҊ͞Ε͍ͯΔ 19 Ψ΢ε૭ ϋϯ૭ ϋϛϯά૭ ϒϥοΫϚϯ૭ ͥΜͿಉ͡͡Όͳ͍Ͱ͔͢
  13. iOSDC Japan 2024 - day1 Track D #iosdc #d ߴ଎ϑʔϦΤม׵ͱ͸

    6/7 ৼ෯εϖΫτϧɿ
 ϑʔϦΤม׵ͨ݁͠Ռ͔Βɺԣ࣠Λप೾਺ɺॎ࣠Λͦͷप೾਺੒෼ͷڧ౓
 ͱͨ͠άϥϑ͕ϓϩοτͰ͖Δ 20
  14. iOSDC Japan 2024 - day1 Track D #iosdc #d ߴ଎ϑʔϦΤม׵ͱ͸

    7/7 ཭ࢄϑʔϦΤม׵ɿ
 ίϯϐϡʔλʔͰ͸࿈ଓؔ਺ʢແݶͷ෼ղೳʣ͸ѻ͑ͳ͍ͷͰ཭ࢄԽͯ͠
 ༗ݶݸͷ഑ྻσʔλʹͯ͠ϑʔϦΤม׵Λߦ͏ ߴ଎ϑʔϦΤม׵ɿ
 ཭ࢄϑʔϦΤม׵͸େมԋࢉʹ͕͔͔࣌ؒΔͨΊɺ
 ΞϧΰϦζϜΛ޻෉ͯ͠ԋࢉΛߴ଎Խͨ͠΋ͷ͕ߴ଎ϑʔϦΤม׵ FFTɿFast Fourier Transformͷུশ ߴ଎ϑʔϦΤม׵׬શʹཧղͨ͠ 21
  15. iOSDC Japan 2024 - day1 Track D #iosdc #d Accelerateͷߴ଎ϑʔϦΤม׵Ͱৼ෯εϖΫτϧΛࢉग़͢Δ

    0/7 Accelerate Frameworkɿ
 CPUͷϕΫτϧॲཧػೳΛ׆༻ͯ͠ɺߴੑೳ͔ͭߴޮ཰ͳԋࢉΛ࣮ߦͰ͖Δ vDSP (digital signal processing on vectors)ɿ
 σδλϧ৴߸ॲཧ޲͚ʹ഑ྻͰͷ൚༻ԋࢉʹ࠷దԽ͞ΕͨAPIΛఏڙ͍ͯ͠Δ
 ߹ܭɺฏۉ஋ɺ࠷େ஋ͳͲͷੵ࿨ؔ਺Λ͸͡Ίͱ͠ɺϑʔϦΤม׵΍
 ૒2࣍ϑΟϧλͷΑ͏ͳॲཧ΋Χόʔ 23
  16. iOSDC Japan 2024 - day1 Track D #iosdc #d Accelerateͷߴ଎ϑʔϦΤม׵Ͱৼ෯εϖΫτϧΛࢉग़͢Δ

    1/7 1. vDSPͰϑʔϦΤม׵ͷ४උΛ͢Δ class FFT { let fftFullSize: vDSP_Length let fftHalfSize: vDSP_Length let mLog2N: vDSP_Length var fftSetup: FFTSetup? init(size: Int) { // ϑʔϦΤม׵Λ͔͚Δ૭ͷαΠζΛఆٛ͢Δ fftFullSize = vDSP_Length(size) fftHalfSize = vDSP_Length(size / 2) // ૭ͷαΠζ͕̎ͷԿ৐͔Λࢉग़͢Δ mLog2N = vDSP_Length(log2(Double(size)).rounded() + 1.0) // FFTSetupΛॳظԽ͢Δ fftSetup = vDSP_create_fftsetup(mLog2N, FFTRadix(kFFTRadix2)) } 24
  17. iOSDC Japan 2024 - day1 Track D #iosdc #d Accelerateͷߴ଎ϑʔϦΤม׵Ͱৼ෯εϖΫτϧΛࢉग़͢Δ

    2/7 2. FFTSetupͷഁغΛఆ͓ٛͯ͘͠ class FFT { // ... deinit { // OpaquePointerΛ࢖͍ͬͯΔͷͰϝϞϦͷղ์Λ๨Εͣʹ vDSP_destroy_fftsetup(fftSetup) } 25
  18. iOSDC Japan 2024 - day1 Track D #iosdc #d Accelerateͷߴ଎ϑʔϦΤม׵Ͱৼ෯εϖΫτϧΛࢉग़͢Δ

    3/7 3. Իͷೖྗ৴߸ʹ૭ؔ਺Λ͔͚Δ class FFT { // ... func compute(sampleRate: Float, audioData: UnsafePointer<Float>) -> [(Float, Float)] { // ૭ؔ਺༻ͷ഑ྻσʔλΛ࡞Δ let windowData = UnsafeMutablePointer<Float>.allocate(capacity: Int(fftFullSize)) // UnsafeMutablePointerΛ࢖͏ͷͰϝϞϦͷղ์Λ๨Εͣʹ defer { windowData.deallocate() } // ഑ྻʹ૭ؔ਺ͷ஋Λ٧ΊΔʢྫͱͯ͠ϋϯ૭Λ༻͍Δʣ vDSP_hann_window(windowData, fftFullSize, 0) // Իͷೖྗ৴߸ʹ૭ؔ਺Λ͔͚Δʢ֤ཁૉʹ૭ؔ਺ͷ஋Λֻ͚߹ΘͤΔʣ vDSP_vmul(audioData, 1, windowData, 1, windowData, 1, fftFullSize) 26
  19. iOSDC Japan 2024 - day1 Track D #iosdc #d Accelerateͷߴ଎ϑʔϦΤม׵Ͱৼ෯εϖΫτϧΛࢉग़͢Δ

    4/7 4. ૭ؔ਺Λ͔͚ͨԻͷೖྗ৴߸Λෳૉ਺ͷ഑ྻʹม׵͢Δ class FFT { // ... func compute(sampleRate: Float, audioData: UnsafePointer<Float>) -> [(Float, Float)] { // ... // θϩͰຒΊΒΕͨ഑ྻΛ༻ҙ͢Δ let zeroData = UnsafeMutablePointer<Float>.allocate(capacity: Int(fftFullSize)) defer { zeroData.deallocate() } vDSP_vclr(zeroData, 1, fftFullSize) // ࣮෦͕૭ؔ਺Λ͔͚ͨԻͷೖྗ৴߸ɺڏ෦͕θϩຒΊ഑ྻͷෳૉ਺഑ྻΛ࡞Δ var dspSplitComplex = DSPSplitComplex( realp: windowData, imagp: zeroData ) 27
  20. iOSDC Japan 2024 - day1 Track D #iosdc #d Accelerateͷߴ଎ϑʔϦΤม׵Ͱৼ෯εϖΫτϧΛࢉग़͢Δ

    5/7 5. ࡞ͬͨෳૉ਺ͷ഑ྻʹߴ଎ϑʔϦΤม׵Λ͔͚Δ class FFT { // ... func compute(sampleRate: Float, audioData: UnsafePointer<Float>) -> [(Float, Float)] { // ... // ߴ଎ϑʔϦΤม׵ vDSP_fft_zrip(fftSetup, &dspSplitComplex, 1, mLog2N, FFTDirection(FFT_FORWARD)) 28
  21. iOSDC Japan 2024 - day1 Track D #iosdc #d Accelerateͷߴ଎ϑʔϦΤม׵Ͱৼ෯εϖΫτϧΛࢉग़͢Δ

    6/7 6. ߴ଎ϑʔϦΤม׵ͷ݁Ռ͔Βৼ෯εϖΫτϧΛࢉग़͢Δʢ1/2ʣ ͜͜Ͱɺෳૉ਺ʹର͢Δઈର஋͸ Ͱ͋Δ ͨͩɺ͜ΕͰಘΒΕΔͷ͸ਖ਼ෛ྆ଆͷৼ෯εϖΫτϧͰ͋Γ
 ৼ෯஋΋൒෼ͳͨΊɺಘΒΕͨৼ෯Λ̎ഒ͠ยଆ͚ͩΛ࢖༻͢Δ ৼ෯ = | ϑʔϦΤม׵ͷ݁Ռʢෳૉ਺ʣ ཁૉ਺ʢ૭ͷαΠζʣ | (࣮෦)2 + (ڏ෦)2 29 🤔
  22. iOSDC Japan 2024 - day1 Track D #iosdc #d Accelerateͷߴ଎ϑʔϦΤม׵Ͱৼ෯εϖΫτϧΛࢉग़͢Δ

    6/7 6. ߴ଎ϑʔϦΤม׵ͷ݁Ռ͔Βৼ෯εϖΫτϧΛࢉग़͢Δʢ2/2ʣ class FFT { // ... func compute(sampleRate: Float, audioData: UnsafePointer<Float>) -> [(Float, Float)] { // ... // ϑʔϦΤม׵ͷ݁ՌΛཁૉ਺ͰׂΔʢ͜ͷ࣌఺Ͱ૭ͷαΠζͷ൒෼͚ͩΛ࢖༻͢Δʣ var fftNormFactor = Float(fftFullSize) vDSP_vsdiv(dspSplitComplex.realp, 1, &fftNormFactor, dspSplitComplex.realp, 1, fftHalfSize) vDSP_vsdiv(dspSplitComplex.imagp, 1, &fftNormFactor, dspSplitComplex.imagp, 1, fftHalfSize) // ઈର஋Λࢉग़͢Δ => ৼ෯͕ಘΒΕΔ var magnitudeData = [Float](repeating: .zero, count: Int(fftHalfSize)) vDSP_zvabs(&dspSplitComplex, 1, &magnitudeData, 1, fftHalfSize) // ৼ෯Λ̎ഒ͢Δ var fftFactor = Float(2) vDSP_vsmul(magnitudeData, 1, &fftFactor, &magnitudeData, 1, fftHalfSize) 30
  23. iOSDC Japan 2024 - day1 Track D #iosdc #d Accelerateͷߴ଎ϑʔϦΤม׵Ͱৼ෯εϖΫτϧΛࢉग़͢Δ

    7/7 7. ֤ৼ෯ʹରԠ͢Δप೾਺Λࢉग़͢Δ प೾਺ = αϯϓϧϨʔτ ཁૉ਺ʢ૭ͷαΠζʣ × ཁૉͷΠϯσοΫε class FFT { // ... func compute(sampleRate: Float, audioData: UnsafePointer<Float>) -> [(Float, Float)] { // ... // 1͔Β૭ͷαΠζͷ൒෼·Ͱͷ഑ྻΛ࡞Δ var hertsData: [Float] = vDSP.ramp(withInitialValue: 1, increment: 1, count: Int(fftHalfSize)) // αϯϓϧϨʔτΛཁૉ਺ͰׂΔ var hertsFactor = sampleRate / Float(fftFullSize) // प೾਺ͷ഑ྻΛ࡞Δ vDSP_vsmul(hertsData, 1, &hertsFactor, &hertsData, 1, fftHalfSize) return zip(hertsData, magnitudeData).map { ($0, $1) } } 31
  24. iOSDC Japan 2024 - day1 Track D #iosdc #d ࢉग़ͨ͠ৼ෯εϖΫτϧΛGraphicsContextͰՄࢹԽ͢Δ

    1/5 var body: some View { Canvas { context, size in let path = Path(ellipseIn: CGRect(origin: .zero, size: size)) context.fill(path, with: .color(.blue)) context.stroke(path, with: .color(.red), lineWidth: 5) } } SwiftUIͷCanvasΛ༻͍ͯGraphicsContextΛૢ࡞͢Δ fi ll΍strokeΛ༻͍ͯPathΛඳ͍͍ͯ͘ 33
  25. iOSDC Japan 2024 - day1 Track D #iosdc #d ࢉग़ͨ͠ৼ෯εϖΫτϧΛGraphicsContextͰՄࢹԽ͢Δ

    2/5 Canvas { context, size in let path = Path(ellipseIn: CGRect(origin: .zero, size: size)) context.fill(path, with: .color(.blue)) context.stroke(path, with: .color(.red), lineWidth: 5) } Path { path in path.addEllipse(in: CGRect(x: 0, y: 0, width: 50, height: 50)) } .stroke(lineWidth: 5) .fill() .foregroundStyle(Color.red) CanvasͱPathͷҧ͍ Canvas͸දࣔ͢ΔྖҬαΠζ͕༩͑ΒΕΔ Canvas͸ಉ͡Pathʹରͯ͠ fi llͱstrokeͰృΓΛม͑ΒΕΔ 34
  26. iOSDC Japan 2024 - day1 Track D #iosdc #d ࢉग़ͨ͠ৼ෯εϖΫτϧΛGraphicsContextͰՄࢹԽ͢Δ

    3/5 struct AudioVisualizedView: View { let values: [Float] var body: some View { Canvas { context, size in let unit = size.width / CGFloat(values.count) let width = 0.8 * unit values.indices.forEach { index in let x = unit * CGFloat(index) let height = size.height * CGFloat(values[index]) let y = 0.5 * (size.height - height) let path = Path(CGRect(x: x, y: y, width: width, height: height)) context.fill(path, with: .color(.primary)) } } } } ৼ෯ͷ஋͸̌ʙ̍ͷൣғ಺ͰऔಘͰ͖ΔͷͰɺ
 ৼ෯εϖΫτϧͷ഑ྻ͔ΒάϥϑΛඳը͢Δ 35
  27. iOSDC Japan 2024 - day1 Track D #iosdc #d ࢉग़ͨ͠ৼ෯εϖΫτϧΛGraphicsContextͰՄࢹԽ͢Δ

    4/5 ϑϦʔBGMʮSUMMER TRIANGLEʯʗ࡞ʢฤʣۂ ɿ ͠ΌΖ͏ 36 ࣮ࡍʹಈ͔ͯ͠ΈΔͱ͜Μͳײ͡
  28. iOSDC Japan 2024 - day1 Track D #iosdc #d ࢉग़ͨ͠ৼ෯εϖΫτϧΛGraphicsContextͰՄࢹԽ͢Δ

    5/5 औಘͨ͠ৼ෯εϖΫτϧΛͦͷ··දࣔ͢Δͱͭ·Βͳ͍ ৼ෯Λఆ਺ഒͨ͠Γɺಈ͖͕໘ന͍प೾਺ଳ͚ͩʹߜͬͯඳըͨ͠Γ͢Δ ώτͷՄௌҬ͸ 20~20000hz ͳͷͰͦͷൣғ಺ͰΑ͍ ܦݧଇతʹ͸ 100~3000hz ͘Β͍Ͱे෼ ϑϦʔBGMʮSUMMER TRIANGLEʯʗ࡞ʢฤʣۂ ɿ ͠ΌΖ͏ 37
  29. iOSDC Japan 2024 - day1 Track D #iosdc #d AudioVisualizerKit

    ԻָΛ࠶ੜ͠ͳ͕Βߴ଎ϑʔϦΤม׵Λͯ͠ΦʔσΟΦϏδϡΞϥΠβʔΛ
 දࣔ͢ΔͨΊͷΩοτΛϥΠϒϥϦʹ͠·ͨ͠ʂʂ https://github.com/Kyome22/AudioVisualizerKit ϑϦʔBGMʮ͠ΎΘ͠ΎΘϋχʔϨϞϯ350mlʯʗ࡞ʢฤʣۂ ɿ ͠ΌΖ͏ 41
  30. iOSDC Japan 2024 - day1 Track D #iosdc #d AudioVisualizerKit

    ԻָΛ࠶ੜͭͭ͠ৼ෯εϖΫτϧͱԻѹΛࢉग़ͯ͘͠ΕΔAudioAnalyzerͱ
 ࢉग़ͨ͠σʔλΛجʹՄࢹԽͯ͘͠ΕΔAmplitudeSpectrumViewΛఏڙ import AudioVisualizerKit import SwiftUI struct ContentView: View { let audioAnalyzer = AudioAnalyzer(fftSize: 2048, windowType: .hannWindow) var body: some View { AmplitudeSpectrumView( shapeType: .straight, magnitudes: audioAnalyzer.magnitudes, range: 0 ..< 128, rms: audioAnalyzer.rms ) .onAppear { let url = Bundle.main.url(forResource: "sound", withExtension: "mp3")! try? audioAnalyzer.prepare(url: url) try? audioAnalyzer.play() } .onDisappear { audioAnalyzer.stop() } } } 42 ࢖ͬͯΈͯͶ