音声配信アプリにおけるiOSを使った音声配信の全てと裏側

Transcript

1. Ի੠഑৴ΞϓϦʹ͓͚ΔiOSΛ࢖ͬ
   ͨԻ੠഑৴ͷશͯͱཪଆ
   iOSDC 2022 Track C 09/11 13:35ʙ
   

   View Slide

2. ࣗݾ঺հ
   Name: ԕ౻୓໻(͑Μͨ͘)
   
   
   Job: iOS / AndroidΤϯδχΞͳͲ
   
   
   Career: SIer໿6೥
   
   
   2018/3~ εϙʔπϚονϯάΞϓϦ
   
   
   2019/3~ CBcloud ෺ྲྀITαʔϏε 2೥൒
   
   
   2021/12~ Voicy
   
   
   Twitter: @entaku_0818
   

   View Slide

3. Ի੠഑৴ϓϥοτϑΥʔϜ - Voicy
   https://corp.voicy.jp/
   https://voicy.jp/
   • Ի੠ίϯςϯπΛiOS/Android/WebͰఏڙ
   

   View Slide

4. VoicyͷΞϓϦ͸࠶ੜͱऩ࿥ͷ2छྨͷΞϓ
   ϦΛఏڙͯ͠·͢
   ΠϯετʔϧϦϯΫ
   

   View Slide

5. ऩ࿥ͱ࠶ੜͰදཪҰମͰ͞·͟·ͳVoicyͷػೳ
   Ի੠࠶ੜ
   Ի੠ऩ࿥
   ੜ์ૹ
   ίϝϯτ
   ϋογϡλάݕ
   ࡧ
   ച্੥ٻ
   ࠩ͠ೖΕ
   ϓϨϛΞϜ์ૹ
   

   View Slide

6. Ի੠αʔϏε্Ͱ͸ಛʹॏཁͳೋͭͷػೳ
   ίϝϯτ
   ϋογϡλάݕ
   ࡧ
   ച্੥ٻ
   ࠩ͠ೖΕ
   ϓϨϛΞϜ์ૹ
   Ի੠࠶ੜ
   Ի੠ऩ࿥
   ੜ์ૹ
   

   View Slide

7. ར༻͍ͯ͠Δٕज़
   Ի੠࠶ੜԻ੠ऩ࿥
   ੜ์ૹ
   AVAudio
   Agora
   

   View Slide

8. AVAudioͱ͸ʁ
   

   View Slide

9. AVFAudioͱ͸ʁ
   https://developer.apple.com/documentation/avfaudio
   "7"VEJP4FTTJPO
   "7"VEJP3PVUJOH"SCJUFS
   "7"VEJP1MBZFS
   "7.*%*1MBZFS
   "7"VEJP3FDPSEFS
   "VEJP&OHJOF
   • AVFoundation͔ΒAudioͷػೳΛ੾Γग़ͨ͠΋ͷͰ͢
   
   
   • iOS14.5͔Β
   

   View Slide

10. AVFAudioͱ͸ʁ
    https://developer.apple.com/documentation/avfaudio
    "7"VEJP4FTTJPO
    "7"VEJP3PVUJOH"SCJUFS
    "7"VEJP1MBZFS
    "7.*%*1MBZFS
    "7"VEJP3FDPSEFS
    "VEJP&OHJOF
    • VoicyͰར༻͍ͯ͠Δػೳ
    

    View Slide

11. Ի੠ͷऩ࿥
    

    View Slide

12. ࿥ԻͷྲྀΕ
    ։࢝ ࿥Իத ऴྃ
    - 1tapͰ࿥ԻՄೳͳઃܭ
    Ի੠ͷ೾ܗ৘ใදࣔ
    

    View Slide

13. ࿥Իݖݶ
    AVAudioSession.sharedInstance().requestRecordPermission { granted in
    
    
    self.handlePermissionGranted()
    
    
    }
    
    
    • ࿥ԻݖݶΛऔಘ
    https://developer.apple.com/documentation/avfaudio/avaudiosession/1616463-
    recordpermission
    

    View Slide

14. ηογϣϯઃఆ
    
    
    try audioSession.setCategory(.playAndRecord)ɹ
    
    
    try audioSession.setPreferredSampleRate(44100)
    
    
    try
    audioSession.setPreferredIOBufferDuration(defaultPreferredIOBufferDurat
    ion)
    
    
    try audioSession.setActive(true)
    
    
    
    
    https://developer.apple.com/documentation/avfaudio/avaudiosession
    • ࠶ੜऩ࿥ՄೳͳηογϣϯͰ։࢝
    /P*NBHF
    

    View Slide

15. ࿥Ի։࢝
    var componentDescription = AudioComponentDescription(
    
    
    componentType: OSType(kAudioUnitType_Output),
    
    
    componentSubType: OSType(kAudioUnitSubType_RemoteIO),
    
    
    componentManufacturer: OSType(kAudioUnitManufacturer_Apple),
    
    
    componentFlags: UInt32(0),
    
    
    componentFlagsMask: UInt32(0)
    
    
    )
    
    
    let component = AudioComponentFindNext(nil, &componentDescription)
    
    
    var tempAudioUnit: AudioUnit?
    
    
    AudioComponentInstanceNew(component!, &tempAudioUnit)
    
    
    https://developer.apple.com/documentation/audiotoolbox
    • audioUnitΛੜ੒
    

    View Slide

16. ࿥Ի։࢝
    var inputCallbackStruct = AURenderCallbackStruct(
    
    
    inputProc: recordingCallback,
    
    
    inputProcRefCon:
    UnsafeMutableRawPointer(Unmanaged.passUnretained(self).toOpaque())
    
    
    )
    
    
    https://developer.apple.com/documentation/audiotoolbox
    • Callbackͷఆٛ
    

    View Slide

17. ࿥Ի։࢝
    // ࿥ԻͷCallback
    
    
    private let recordingCallback: AURenderCallback = { (inRefCon, ioActionFlags, inTimeStamp,
    inBusNumber, frameCount, _) -> OSStatus in
    
    
    audioObject.handlerBufferData(bufferList: &bufferList, frameCount: frameCount)
    
    
    }
    
    
    • όοϑΝʔ৘ใΛऔಘ
    

    View Slide

18. ೾ܗͷऔಘ
    let pcmBuffer = AVAudioPCMBuffer(pcmFormat: audioFormat, frameCapacity:
    UInt32(bufferSize))
    
    
    buffers.append(buffer)
    
    
    waveFormHeights.append(buffer.waveFormHeight)
    • औಘͨ͠όοϑΝʔ৘ใΛ΋ͱʹԻྔΛϨϯμϦϯά
    

    View Slide

19. ೾ܗͷऔಘ
    6*$PMMFDUJPO7JFX
    let pcmBuffer = AVAudioPCMBuffer(pcmFormat: audioFormat, frameCapacity:
    UInt32(bufferSize))
    
    
    buffers.append(buffer)
    
    
    waveFormHeights.append(buffer.waveFormHeight)
    • औಘͨ͠όοϑΝʔ৘ใΛ΋ͱʹԻྔΛϨϯμϦϯά
    

    View Slide

20. ࠶ฤू
    • ऩ࿥ͨ͠Ի੠σʔλΛ্ॻ͖
    

    View Slide

21. Ի੠ऩ࿥ͷऴྃ
    let settings: [String: Any] = [
    
    
    AVSampleRateKey: Constants.Recording.sampleRate,
    
    
    AVFormatIDKey: kAudioFormatMPEG4AAC, // MP4ܗࣜͰอଘ
    
    
    AVNumberOfChannelsKey: Constants.Recording.numberOfChannel,
    
    
    AVEncoderAudioQualityKey: AVAudioQuality.high
    
    
    ]
    
    
    let audioFile = try AVAudioFile(forWriting: fileURL, settings: settings)
    
    
    • Local΁อଘ͠Ξοϓϩʔυ
    

    View Slide

22. AVAudioRecorder͸ར༻͍ͯ͠ͳ͍
    https://developer.apple.com/documentation/avfaudio
    "7"VEJP4FTTJPO
    "7"VEJP3PVUJOH"SCJUFS
    "7"VEJP1MBZFS
    "7.*%*1MBZFS
    "7"VEJP3FDPSEFS
    "VEJP&OHJOF
    • ։ൃ౰ॳ͸audioUnitͰ༷ʑͳ͜ͱΛ΍Ζ͏ͱ͍ͯͨ͠ͷ͔΋͠Εͳ͍
    
    
    • ͓ͦΒ͘೾ܗʹԻྔͷߴ͞Λऔಘ͢ΔͨΊʹར༻
    BVEJP6OJU
    

    View Slide

23. Ի੠ͷฤू
    

    View Slide

24. ࠶ฤू
    • αʔϏεϙϦγʔ্Ͱ͖ͳ͍Α͏ʹ͍ͯ͠·͢ʂ
    

    View Slide

25. Ի੠࠶ੜ
    

    View Slide

26. Ի੠ͷ։࢝
    Ի੠࠶ੜ։࢝ Ի੠࠶ੜத
    

    View Slide

27. Ի੠ͷ։࢝
    func playPlayingChapter() {
    
    
    voicePlayer.play(url: fileURL, startTime:
    playingChapter.lastPlaytime.msecToSec(), speed: playSpeed.rawValue)
    
    
    }
    • νϟϓλʔ୯ҐͰͷ։࢝
    

    View Slide

28. Ի੠ͷఀࢭ
    func stopPlay() {
    
    
    player?.stop()
    
    
    player = nil
    
    
    }
    • Ի੠ͷఀࢭ
    

    View Slide

29. background࠶ੜ
    try audioSession.setCategory(.playback, mode: .default, options: [])
    
    
    https://developer.apple.com/documentation/avfaudio/avaudiosession/category
    

    View Slide

30. ੜ์ૹ
    

    View Slide

31. Agoraͱ͸ʁ
    

    View Slide

32. Agoraͱ͸ʁ
    https://www.agora.io/en/
    https://jp.vcube.com/sdk
    

    View Slide

33. AgoraͰ͸ϏσΦ௨࿩΍Ի੠഑৴ͷػೳΛఏڙ
    https://www.agora.io/en/
    

    View Slide

34. ͦͷதͰ΋Live Steaming ͱ Real-time
    MessagingΛར༻
    https://www.agora.io/en/
    

    View Slide

35. Voicyͷੜ์ૹͱ͸ʁ
    Ի੠ͷ࿈ܞ
    

    Live Streaming
    
    
    (RTC)
    ͓ศΓ΍λΠτϧͷ࿈ܞ
    
    
    RealTimeMessage
    
    
    (RTM)
    

    View Slide

36. ੜ์ૹ
    

    View Slide

37. ੜ์ૹͷྲྀΕ
    ੜ์ૹ։࢝ ࿥Ի ऴྃ
    

    View Slide

38. ੜ์ૹ
    // RTC
    
    
    agoraKit = AgoraRtcEngineKit.sharedEngine(withAppId:
    EnvironmentConfig.agoraAppId, delegate: self)
    
    
    agoraKit?.enableAudioVolumeIndication(200, smooth: 3, report_vad: true)
    
    
    agoraKit?.setChannelProfile(AgoraChannelProfile.liveBroadcasting)
    
    
    
    
    // RTM
    
    
    agoraRtmKit = AgoraRtmKit(appId: EnvironmentConfig.agoraAppId, delegate: self)
    
    
    agoraRtmChannel = agoraRtmKit?.createChannel(withId: liveId, delegate: self)
    
    
    • Agora্ͰRTCͱRTMͰͦΕͧΕνϟϯωϧΛ࡞੒
    

    View Slide

39. ੜ์ૹ࿥Ի
    // ࿥Ի͸αʔόʔαΠυͰ࣮ࢪ͍ͯ͠·͢
    // ͲΜͳॲཧ͔Θ͔ΔΑ͏ʹcurlͷAPIΛهࡌ͠·ͨ͠
    curl --location --request GET 'https://api.agora.io/v1/apps//
    cloud_recording/resourceid//sid//mode//query' \
    • Agoraͷcloud_recording(࿥Ի)ͷػೳ
    

    View Slide

40. ͓ศΓϝοηʔδͷऔಘ
    func rtmKit(_ kit: AgoraRtmKit, messageReceived message: AgoraRtmMessage, fromPeer peerId: String) {
    
    
    messageReceived(message: message, uid: peerId)
    
    
    }
    
    
    ~~~
    
    
    if let message = RTMMessage(rawValue: message.text), let uid = Int(uid) {
    
    
    switch message {
    
    
    case .hands_up: delegate?.onRaiseHand(userId: uid)
    
    
    case .hands_down: delegate?.onDroppedHand(userId: uid)
    
    
    case .invite: delegate?.onGuestInvited()
    
    
    case .canceled_invite: delegate?.onCancelGuestInvitation()
    
    
    case .reject: delegate?.onInvitaionRejected(userId: uid)
    
    
    case .set_audience: changeListener()
    
    
    case .mute: delegate?.onMute(uid: uid, muted: true)
    
    
    case .un_mute: delegate?.onMute(uid: uid, muted: false)
    
    
    case .stamp: delegate?.onStampReceived()
    
    
    case .edit_live_info: delegate?.onEditLiveInfo()
    
    
    case .letter: delegate?.onReceivedLetter(userId: uid)
    
    
    case .return_guest_to_listener: delegate?.onReturnGuestToListener()
    
    
    case .start: delegate?.onLiveStarted()
    
    
    }
    
    
    }
    • ϝοηʔδΛऔಘ༷͠ʑͳϦΞϧλΠϜॲཧΛ࣮ݱ
    

    View Slide

41. ੜ์ૹ
    

    View Slide

42. ੜ์ૹ
    • AgoraͰੜ์ૹʹࢀՃ
    self.agoraRtmChannel?.join(completion: { code in
    
    
    self.agoraKit?.setClientRole(agoraClientRole)
    
    
    self.agoraKit?.joinChannel(byToken: entity.agoraToken.rtcToken, channelId:
    String(liveID), info: nil, uid: UInt(entity.uid))
    
    
    

    View Slide

43. ੜ์ૹ
    • ͓ศΓΛૹΔ
    agoraRtmKit?.send(AgoraRtmMessage(text: text.rawValue), toPeer: uid)
    
    
    

    View Slide

44. Ի੠αʔϏε্(Voicy)Ͱ͸ಛʹॏཁͳೋͭͷػೳʹ
    ͍ͭͯ঺հ͠·ͨ͠
    ίϝϯτ
    ϋογϡλάݕ
    ࡧ
    ച্੥ٻ
    ࠩ͠ೖΕ
    ϓϨϛΞϜ์ૹ
    Ի੠࠶ੜ
    Ի੠ऩ࿥
    ੜ์ૹ
    

    View Slide

45. VoicyͷԻ੠ॲཧͷࠓޙͷల๬
    

    View Slide

46. ࠓޙͷల๬ɹԻ੠࠶ੜ/Ի੠ऩ࿥
    • Ի੠ऩ࿥ͷ඼࣭޲্
    
    
    • ϚΠΫ - bluetoothϚΠΫͳͲ͸ి೾͕ׯবͯ͠͠
    ·͍Ի੠͕௿͔ͬͨΓ͢Δ
    
    
    • ݱࡏ͸ϚΠΫΛ઀ଓ͠ͳ͍ঢ়ଶͰͷऩ࿥Λਪ঑
    
    
    • ࿈ଓ࠶ੜ / ΦϑϥΠϯ࠶ੜͳͲ
    
    
    • Ϧεφʔ΁͞ΒʹศརͳԻ੠ମݧΛ
    

    View Slide

47. ࠓޙͷల๬ɹੜ์ૹ
    • ࿥Ի඼࣭޲্
    
    
    • AgoraͷCloud-RecordingͩͱͲ͏͠
    ͯ΋඼࣭͕ྑ͘ͳ͍
    
    
    • େਓ਺Ͱ΋଱͑ΒΕΔੜ์ૹ΁
    
    
    • AgoraͱαʔόʔαΠυؚΊͨΩϟύ
    γςΟͷ޲্
    

    View Slide

48. ͓·͚
    

    View Slide

49. ࠓ৽͘͠Ի੠ΞϓϦΛ࡞Δͱͨ͠ΒͲ͏͢ΔΜͩΖ͏ʁ
    
    
    🤔
    ΍ͬͺSwiftUI͔ʁ Swift Concurrencyͱ͔৽
    ػೳ΋࢖͍͍ͨͳ
    

    View Slide

50. ͱ͍͏Θ͚Ͱ࡞Γ·ͨ͠ʂ
    https://github.com/entaku0818/VoiceMemo
    

    View Slide

51. αϯϓϧΛࢀߟʹͯ͠࡞Γ·ͨ͠
    https://github.com/pointfreeco/swift-composable-architecture/tree/main/Examples/VoiceMemos
    ͱ͍͏Θ͚Ͱ࡞Γ·ͨ͠ʂ
    

    View Slide

52. ಛʹ঺հ͍ͨ͠ͱ͜Ζ
    

    View Slide

53. Ի੠ॲཧͷDeleagteॲཧͷ؅ཧͷ೉͠͞
    • େ͖ͳControllerͰશͯΛίϯτϩʔϧ͕ͪ͠
    
    
    • ֤ը໘ͰDelegateͰॲཧΛॻ͖͕ͪ
    

    View Slide

54. Ի੠ͳͲͰ͸ಛʹଟ͍DelegateॲཧΛ·ͱΊ
    ͯΔ
    https://github.com/entaku0818/VoiceMemo
    private final class Delegate: NSObject, AVAudioPlayerDelegate, Sendable {
    
    
    let didFinishPlaying: @Sendable (Bool) -> Void
    
    
    let decodeErrorDidOccur: @Sendable (Error?) -> Void
    
    
    let player: AVAudioPlayer
    
    
    init(
    
    
    url: URL,
    
    
    didFinishPlaying: @escaping @Sendable (Bool) -> Void,
    
    
    decodeErrorDidOccur: @escaping @Sendable (Error?) -> Void
    
    
    ) throws {
    
    
    self.didFinishPlaying = didFinishPlaying
    
    
    self.decodeErrorDidOccur = decodeErrorDidOccur
    
    
    self.player = try AVAudioPlayer(contentsOf: url)
    
    
    super.init()
    
    
    self.player.delegate = self
    
    
    }
    
    
    func audioPlayerDidFinishPlaying(_ player: AVAudioPlayer, successfully flag: Bool) {
    
    
    self.didFinishPlaying(flag)
    
    
    }
    
    
    func audioPlayerDecodeErrorDidOccur(_ player: AVAudioPlayer, error: Error?) {
    
    
    self.decodeErrorDidOccur(error)
    
    
    }
    
    
    }
    
    
    

    View Slide

55. liveͱ͍͏ঢ়ଶͰDelegateΛఆٛ
    
    
    Ի੠։࢝࣌ͷॲཧΛެ։
    https://github.com/entaku0818/VoiceMemo
    struct AudioPlayerClient {
    
    
    var play: @Sendable (URL) async throws -> Bool
    
    
    }
    
    
    extension AudioPlayerClient {
    
    
    static let live = Self { url in
    
    
    let stream = AsyncThrowingStream { continuation in
    
    
    do {
    
    
    let delegate = try Delegate(
    
    
    url: url,
    
    
    didFinishPlaying: { successful in
    
    
    continuation.yield(successful)
    
    
    continuation.finish()
    
    
    },
    
    
    decodeErrorDidOccur: { error in
    
    
    continuation.finish(throwing: error)
    
    
    }
    
    
    )
    
    
    delegate.player.play()
    
    
    continuation.onTermination = { _ in
    
    
    delegate.player.stop()
    
    
    }
    
    
    } catch {
    
    
    continuation.finish(throwing: error)
    
    
    }
    
    
    }
    
    
    return try await stream.first(where: { _ in true }) ?? false
    
    
    }
    
    
    }
    
    
    4XJGU$PODVSSFODZ
    Ͱ࣮૷
    

    View Slide

56. ReducerͰը໘ͰඞཁͳॲཧΛఆٛ
    https://github.com/entaku0818/VoiceMemo
    switch state.mode {
    
    
    case .notPlaying:
    
    
    state.mode = .playing(progress: 0)
    
    
    return .run { [url = state.url] send in
    
    
    let start = environment.mainRunLoop.now
    
    
    async let playAudio: Void = send(
    
    
    .audioPlayerClient(TaskResult { try await environment.audioPlayer.play(url) })
    
    
    )
    
    
    for try await tick in environment.mainRunLoop.timer(interval: 0.5) {
    
    
    await send(.timerUpdated(tick.date.timeIntervalSince(start.date)))
    
    
    }
    
    
    }
    
    
    .cancellable(id: PlayID.self, cancelInFlight: true)
    
    
    case .playing:
    
    
    state.mode = .notPlaying
    
    
    return .cancel(id: PlayID.self)
    
    
    }
    
    
    

    View Slide

57. ࢀߟࢿྉ
    

    View Slide

58. ࢀߟࢿྉ
    • agoraΛ࢖ͬͯϥΠϒ഑৴ػೳΛ1ϲ݄൒ͰϦϦʔεͨ͠࿩
    
    
    • εϥΠυ
    
    
    • ࠓ೔͔Β෼͔Δ AVAudioEngineͷશͯ
    
    
    • εϥΠυ
    
    
    • ࣮૷ͨ͘͠ͳΔԻ੠ฤू
    
    
    • εϥΠυ
    • աڈͷԻ੠ʹؔΘΔiOSDCͷηογϣϯ
    

    View Slide

59. ࢀߟࢿྉ
    • ΍ͬͺΓ࠷ޙ͸ެࣜ
    
    
    • https://developer.apple.com/documentation/avfaudio
    
    
    • https://docs.agora.io/en
    

    View Slide

60. એ఻ʂ
    

    View Slide

61. https://voicy.jp/channel/1305
    VoicyͷΤϯδχΞ͕ӡӦ͍ͯ͠Δνϟϯωϧ
    
    
    voi-chordௌ͍͍ͯͩ͘͞ʂ
    

    View Slide

62. iOSΤϯδχΞΛืू͍ͯ͠·͢
    • Ի੠޷͖ͳΤϯδχΞืूதͰ͢ʂڵຯ͋Δํ͸DM͍ͩ͘͞ʂ
    Notion
    Twitter: @entaku_0818
    

    View Slide