Upgrade to Pro — share decks privately, control downloads, hide ads and more …

音声配信アプリにおけるiOSを使った音声配信の全てと裏側

entaku
September 11, 2022

 音声配信アプリにおけるiOSを使った音声配信の全てと裏側

entaku

September 11, 2022
Tweet

More Decks by entaku

Other Decks in Technology

Transcript

  1. Ի੠഑৴ΞϓϦʹ͓͚ΔiOSΛ࢖ͬ ͨԻ੠഑৴ͷશͯͱཪଆ iOSDC 2022 Track C 09/11 13:35ʙ

  2. ࣗݾ঺հ Name: ԕ౻୓໻(͑Μͨ͘) Job: iOS / AndroidΤϯδχΞͳͲ Career: SIer໿6೥ 2018/3~

    εϙʔπϚονϯάΞϓϦ 2019/3~ CBcloud ෺ྲྀITαʔϏε 2೥൒ 2021/12~ Voicy Twitter: @entaku_0818
  3. Ի੠഑৴ϓϥοτϑΥʔϜ - Voicy https://corp.voicy.jp/ https://voicy.jp/ • Ի੠ίϯςϯπΛiOS/Android/WebͰఏڙ

  4. VoicyͷΞϓϦ͸࠶ੜͱऩ࿥ͷ2छྨͷΞϓ ϦΛఏڙͯ͠·͢ ΠϯετʔϧϦϯΫ

  5. ऩ࿥ͱ࠶ੜͰදཪҰମͰ͞·͟·ͳVoicyͷػೳ Ի੠࠶ੜ Ի੠ऩ࿥ ੜ์ૹ ίϝϯτ ϋογϡλάݕ ࡧ ച্੥ٻ ࠩ͠ೖΕ ϓϨϛΞϜ์ૹ

  6. Ի੠αʔϏε্Ͱ͸ಛʹॏཁͳೋͭͷػೳ ίϝϯτ ϋογϡλάݕ ࡧ ച্੥ٻ ࠩ͠ೖΕ ϓϨϛΞϜ์ૹ Ի੠࠶ੜ Ի੠ऩ࿥ ੜ์ૹ

  7. ར༻͍ͯ͠Δٕज़ Ի੠࠶ੜԻ੠ऩ࿥ ੜ์ૹ AVAudio Agora

  8. AVAudioͱ͸ʁ

  9. AVFAudioͱ͸ʁ https://developer.apple.com/documentation/avfaudio "7"VEJP4FTTJPO "7"VEJP3PVUJOH"SCJUFS "7"VEJP1MBZFS "7.*%*1MBZFS "7"VEJP3FDPSEFS "VEJP&OHJOF • AVFoundation͔ΒAudioͷػೳΛ੾Γग़ͨ͠΋ͷͰ͢

    • iOS14.5͔Β
  10. AVFAudioͱ͸ʁ https://developer.apple.com/documentation/avfaudio "7"VEJP4FTTJPO "7"VEJP3PVUJOH"SCJUFS "7"VEJP1MBZFS "7.*%*1MBZFS "7"VEJP3FDPSEFS "VEJP&OHJOF • VoicyͰར༻͍ͯ͠Δػೳ

  11. Ի੠ͷऩ࿥

  12. ࿥ԻͷྲྀΕ ։࢝ ࿥Իத ऴྃ - 1tapͰ࿥ԻՄೳͳઃܭ Ի੠ͷ೾ܗ৘ใදࣔ

  13. ࿥Իݖݶ AVAudioSession.sharedInstance().requestRecordPermission { granted in self.handlePermissionGranted() } • ࿥ԻݖݶΛऔಘ https://developer.apple.com/documentation/avfaudio/avaudiosession/1616463-

    recordpermission
  14. ηογϣϯઃఆ try audioSession.setCategory(.playAndRecord)ɹ try audioSession.setPreferredSampleRate(44100) try audioSession.setPreferredIOBufferDuration(defaultPreferredIOBufferDurat ion) try audioSession.setActive(true)

    https://developer.apple.com/documentation/avfaudio/avaudiosession • ࠶ੜऩ࿥ՄೳͳηογϣϯͰ։࢝ /P*NBHF
  15. ࿥Ի։࢝ var componentDescription = AudioComponentDescription( componentType: OSType(kAudioUnitType_Output), componentSubType: OSType(kAudioUnitSubType_RemoteIO), componentManufacturer:

    OSType(kAudioUnitManufacturer_Apple), componentFlags: UInt32(0), componentFlagsMask: UInt32(0) ) let component = AudioComponentFindNext(nil, &componentDescription) var tempAudioUnit: AudioUnit? AudioComponentInstanceNew(component!, &tempAudioUnit) https://developer.apple.com/documentation/audiotoolbox • audioUnitΛੜ੒
  16. ࿥Ի։࢝ var inputCallbackStruct = AURenderCallbackStruct( inputProc: recordingCallback, inputProcRefCon: UnsafeMutableRawPointer(Unmanaged.passUnretained(self).toOpaque()) )

    https://developer.apple.com/documentation/audiotoolbox • Callbackͷఆٛ
  17. ࿥Ի։࢝ // ࿥ԻͷCallback private let recordingCallback: AURenderCallback = { (inRefCon,

    ioActionFlags, inTimeStamp, inBusNumber, frameCount, _) -> OSStatus in audioObject.handlerBufferData(bufferList: &bufferList, frameCount: frameCount) } • όοϑΝʔ৘ใΛऔಘ
  18. ೾ܗͷऔಘ let pcmBuffer = AVAudioPCMBuffer(pcmFormat: audioFormat, frameCapacity: UInt32(bufferSize)) buffers.append(buffer) waveFormHeights.append(buffer.waveFormHeight)

    • औಘͨ͠όοϑΝʔ৘ใΛ΋ͱʹԻྔΛϨϯμϦϯά
  19. ೾ܗͷऔಘ 6*$PMMFDUJPO7JFX let pcmBuffer = AVAudioPCMBuffer(pcmFormat: audioFormat, frameCapacity: UInt32(bufferSize)) buffers.append(buffer)

    waveFormHeights.append(buffer.waveFormHeight) • औಘͨ͠όοϑΝʔ৘ใΛ΋ͱʹԻྔΛϨϯμϦϯά
  20. ࠶ฤू • ऩ࿥ͨ͠Ի੠σʔλΛ্ॻ͖

  21. Ի੠ऩ࿥ͷऴྃ let settings: [String: Any] = [ AVSampleRateKey: Constants.Recording.sampleRate, AVFormatIDKey:

    kAudioFormatMPEG4AAC, // MP4ܗࣜͰอଘ AVNumberOfChannelsKey: Constants.Recording.numberOfChannel, AVEncoderAudioQualityKey: AVAudioQuality.high ] let audioFile = try AVAudioFile(forWriting: fileURL, settings: settings) • Local΁อଘ͠Ξοϓϩʔυ
  22. AVAudioRecorder͸ར༻͍ͯ͠ͳ͍ https://developer.apple.com/documentation/avfaudio "7"VEJP4FTTJPO "7"VEJP3PVUJOH"SCJUFS "7"VEJP1MBZFS "7.*%*1MBZFS "7"VEJP3FDPSEFS "VEJP&OHJOF • ։ൃ౰ॳ͸audioUnitͰ༷ʑͳ͜ͱΛ΍Ζ͏ͱ͍ͯͨ͠ͷ͔΋͠Εͳ͍

    • ͓ͦΒ͘೾ܗʹԻྔͷߴ͞Λऔಘ͢ΔͨΊʹར༻ BVEJP6OJU
  23. Ի੠ͷฤू

  24. ࠶ฤू • αʔϏεϙϦγʔ্Ͱ͖ͳ͍Α͏ʹ͍ͯ͠·͢ʂ

  25. Ի੠࠶ੜ

  26. Ի੠ͷ։࢝ Ի੠࠶ੜ։࢝ Ի੠࠶ੜத

  27. Ի੠ͷ։࢝ func playPlayingChapter() { voicePlayer.play(url: fileURL, startTime: playingChapter.lastPlaytime.msecToSec(), speed: playSpeed.rawValue)

    } • νϟϓλʔ୯ҐͰͷ։࢝
  28. Ի੠ͷఀࢭ func stopPlay() { player?.stop() player = nil } •

    Ի੠ͷఀࢭ
  29. background࠶ੜ try audioSession.setCategory(.playback, mode: .default, options: []) https://developer.apple.com/documentation/avfaudio/avaudiosession/category

  30. ੜ์ૹ

  31. Agoraͱ͸ʁ

  32. Agoraͱ͸ʁ https://www.agora.io/en/ https://jp.vcube.com/sdk

  33. AgoraͰ͸ϏσΦ௨࿩΍Ի੠഑৴ͷػೳΛఏڙ https://www.agora.io/en/

  34. ͦͷதͰ΋Live Steaming ͱ Real-time MessagingΛར༻ https://www.agora.io/en/

  35. Voicyͷੜ์ૹͱ͸ʁ Ի੠ͷ࿈ܞ 
 Live Streaming (RTC) ͓ศΓ΍λΠτϧͷ࿈ܞ RealTimeMessage (RTM)

  36. ੜ์ૹ

  37. ੜ์ૹͷྲྀΕ ੜ์ૹ։࢝ ࿥Ի ऴྃ

  38. ੜ์ૹ // RTC agoraKit = AgoraRtcEngineKit.sharedEngine(withAppId: EnvironmentConfig.agoraAppId, delegate: self) agoraKit?.enableAudioVolumeIndication(200,

    smooth: 3, report_vad: true) agoraKit?.setChannelProfile(AgoraChannelProfile.liveBroadcasting) // RTM agoraRtmKit = AgoraRtmKit(appId: EnvironmentConfig.agoraAppId, delegate: self) agoraRtmChannel = agoraRtmKit?.createChannel(withId: liveId, delegate: self) • Agora্ͰRTCͱRTMͰͦΕͧΕνϟϯωϧΛ࡞੒
  39. ੜ์ૹ࿥Ի // ࿥Ի͸αʔόʔαΠυͰ࣮ࢪ͍ͯ͠·͢ // ͲΜͳॲཧ͔Θ͔ΔΑ͏ʹcurlͷAPIΛهࡌ͠·ͨ͠ curl --location --request GET 'https://api.agora.io/v1/apps/<appid>/

    cloud_recording/resourceid/<resourceid>/sid/<sid>/mode/<mode>/query' \ • Agoraͷcloud_recording(࿥Ի)ͷػೳ
  40. ͓ศΓϝοηʔδͷऔಘ func rtmKit(_ kit: AgoraRtmKit, messageReceived message: AgoraRtmMessage, fromPeer peerId:

    String) { messageReceived(message: message, uid: peerId) } ~~~ if let message = RTMMessage(rawValue: message.text), let uid = Int(uid) { switch message { case .hands_up: delegate?.onRaiseHand(userId: uid) case .hands_down: delegate?.onDroppedHand(userId: uid) case .invite: delegate?.onGuestInvited() case .canceled_invite: delegate?.onCancelGuestInvitation() case .reject: delegate?.onInvitaionRejected(userId: uid) case .set_audience: changeListener() case .mute: delegate?.onMute(uid: uid, muted: true) case .un_mute: delegate?.onMute(uid: uid, muted: false) case .stamp: delegate?.onStampReceived() case .edit_live_info: delegate?.onEditLiveInfo() case .letter: delegate?.onReceivedLetter(userId: uid) case .return_guest_to_listener: delegate?.onReturnGuestToListener() case .start: delegate?.onLiveStarted() } } • ϝοηʔδΛऔಘ༷͠ʑͳϦΞϧλΠϜॲཧΛ࣮ݱ
  41. ੜ์ૹ

  42. ੜ์ૹ • AgoraͰੜ์ૹʹࢀՃ self.agoraRtmChannel?.join(completion: { code in self.agoraKit?.setClientRole(agoraClientRole) self.agoraKit?.joinChannel(byToken: entity.agoraToken.rtcToken,

    channelId: String(liveID), info: nil, uid: UInt(entity.uid))
  43. ੜ์ૹ • ͓ศΓΛૹΔ agoraRtmKit?.send(AgoraRtmMessage(text: text.rawValue), toPeer: uid)

  44. Ի੠αʔϏε্(Voicy)Ͱ͸ಛʹॏཁͳೋͭͷػೳʹ ͍ͭͯ঺հ͠·ͨ͠ ίϝϯτ ϋογϡλάݕ ࡧ ച্੥ٻ ࠩ͠ೖΕ ϓϨϛΞϜ์ૹ Ի੠࠶ੜ Ի੠ऩ࿥

    ੜ์ૹ
  45. VoicyͷԻ੠ॲཧͷࠓޙͷల๬

  46. ࠓޙͷల๬ɹԻ੠࠶ੜ/Ի੠ऩ࿥ • Ի੠ऩ࿥ͷ඼࣭޲্ • ϚΠΫ - bluetoothϚΠΫͳͲ͸ి೾͕ׯবͯ͠͠ ·͍Ի੠͕௿͔ͬͨΓ͢Δ • ݱࡏ͸ϚΠΫΛ઀ଓ͠ͳ͍ঢ়ଶͰͷऩ࿥Λਪ঑

    • ࿈ଓ࠶ੜ / ΦϑϥΠϯ࠶ੜͳͲ • Ϧεφʔ΁͞ΒʹศརͳԻ੠ମݧΛ
  47. ࠓޙͷల๬ɹੜ์ૹ • ࿥Ի඼࣭޲্ • AgoraͷCloud-RecordingͩͱͲ͏͠ ͯ΋඼࣭͕ྑ͘ͳ͍ • େਓ਺Ͱ΋଱͑ΒΕΔੜ์ૹ΁ • AgoraͱαʔόʔαΠυؚΊͨΩϟύ

    γςΟͷ޲্
  48. ͓·͚

  49. ࠓ৽͘͠Ի੠ΞϓϦΛ࡞Δͱͨ͠ΒͲ͏͢ΔΜͩΖ͏ʁ 🤔 ΍ͬͺSwiftUI͔ʁ Swift Concurrencyͱ͔৽ ػೳ΋࢖͍͍ͨͳ

  50. ͱ͍͏Θ͚Ͱ࡞Γ·ͨ͠ʂ https://github.com/entaku0818/VoiceMemo

  51. αϯϓϧΛࢀߟʹͯ͠࡞Γ·ͨ͠ https://github.com/pointfreeco/swift-composable-architecture/tree/main/Examples/VoiceMemos ͱ͍͏Θ͚Ͱ࡞Γ·ͨ͠ʂ

  52. ಛʹ঺հ͍ͨ͠ͱ͜Ζ

  53. Ի੠ॲཧͷDeleagteॲཧͷ؅ཧͷ೉͠͞ • େ͖ͳControllerͰશͯΛίϯτϩʔϧ͕ͪ͠ • ֤ը໘ͰDelegateͰॲཧΛॻ͖͕ͪ

  54. Ի੠ͳͲͰ͸ಛʹଟ͍DelegateॲཧΛ·ͱΊ ͯΔ https://github.com/entaku0818/VoiceMemo private final class Delegate: NSObject, AVAudioPlayerDelegate, Sendable

    { let didFinishPlaying: @Sendable (Bool) -> Void let decodeErrorDidOccur: @Sendable (Error?) -> Void let player: AVAudioPlayer init( url: URL, didFinishPlaying: @escaping @Sendable (Bool) -> Void, decodeErrorDidOccur: @escaping @Sendable (Error?) -> Void ) throws { self.didFinishPlaying = didFinishPlaying self.decodeErrorDidOccur = decodeErrorDidOccur self.player = try AVAudioPlayer(contentsOf: url) super.init() self.player.delegate = self } func audioPlayerDidFinishPlaying(_ player: AVAudioPlayer, successfully flag: Bool) { self.didFinishPlaying(flag) } func audioPlayerDecodeErrorDidOccur(_ player: AVAudioPlayer, error: Error?) { self.decodeErrorDidOccur(error) } }
  55. liveͱ͍͏ঢ়ଶͰDelegateΛఆٛ Ի੠։࢝࣌ͷॲཧΛެ։ https://github.com/entaku0818/VoiceMemo struct AudioPlayerClient { var play: @Sendable (URL)

    async throws -> Bool } extension AudioPlayerClient { static let live = Self { url in let stream = AsyncThrowingStream<Bool, Error> { continuation in do { let delegate = try Delegate( url: url, didFinishPlaying: { successful in continuation.yield(successful) continuation.finish() }, decodeErrorDidOccur: { error in continuation.finish(throwing: error) } ) delegate.player.play() continuation.onTermination = { _ in delegate.player.stop() } } catch { continuation.finish(throwing: error) } } return try await stream.first(where: { _ in true }) ?? false } } 4XJGU$PODVSSFODZ Ͱ࣮૷
  56. ReducerͰը໘ͰඞཁͳॲཧΛఆٛ https://github.com/entaku0818/VoiceMemo switch state.mode { case .notPlaying: state.mode = .playing(progress:

    0) return .run { [url = state.url] send in let start = environment.mainRunLoop.now async let playAudio: Void = send( .audioPlayerClient(TaskResult { try await environment.audioPlayer.play(url) }) ) for try await tick in environment.mainRunLoop.timer(interval: 0.5) { await send(.timerUpdated(tick.date.timeIntervalSince(start.date))) } } .cancellable(id: PlayID.self, cancelInFlight: true) case .playing: state.mode = .notPlaying return .cancel(id: PlayID.self) }
  57. ࢀߟࢿྉ

  58. ࢀߟࢿྉ • agoraΛ࢖ͬͯϥΠϒ഑৴ػೳΛ1ϲ݄൒ͰϦϦʔεͨ͠࿩ • εϥΠυ • ࠓ೔͔Β෼͔Δ AVAudioEngineͷશͯ • εϥΠυ

    • ࣮૷ͨ͘͠ͳΔԻ੠ฤू • εϥΠυ • աڈͷԻ੠ʹؔΘΔiOSDCͷηογϣϯ
  59. ࢀߟࢿྉ • ΍ͬͺΓ࠷ޙ͸ެࣜ • https://developer.apple.com/documentation/avfaudio • https://docs.agora.io/en

  60. એ఻ʂ

  61. https://voicy.jp/channel/1305 VoicyͷΤϯδχΞ͕ӡӦ͍ͯ͠Δνϟϯωϧ voi-chordௌ͍͍ͯͩ͘͞ʂ

  62. iOSΤϯδχΞΛืू͍ͯ͠·͢ • Ի੠޷͖ͳΤϯδχΞืूதͰ͢ʂڵຯ͋Δํ͸DM͍ͩ͘͞ʂ Notion Twitter: @entaku_0818