Upgrade to Pro — share decks privately, control downloads, hide ads and more …

音声配信アプリにおけるiOSを使った音声配信の全てと裏側

entaku
September 11, 2022

 音声配信アプリにおけるiOSを使った音声配信の全てと裏側

entaku

September 11, 2022
Tweet

More Decks by entaku

Other Decks in Technology

Transcript

  1. Ի੠഑৴ΞϓϦʹ͓͚ΔiOSΛ࢖ͬ
    ͨԻ੠഑৴ͷશͯͱཪଆ
    iOSDC 2022 Track C 09/11 13:35ʙ

    View full-size slide

  2. ࣗݾ঺հ
    Name: ԕ౻୓໻(͑Μͨ͘)


    Job: iOS / AndroidΤϯδχΞͳͲ


    Career: SIer໿6೥


    2018/3~ εϙʔπϚονϯάΞϓϦ


    2019/3~ CBcloud ෺ྲྀITαʔϏε 2೥൒


    2021/12~ Voicy


    Twitter: @entaku_0818

    View full-size slide

  3. Ի੠഑৴ϓϥοτϑΥʔϜ - Voicy
    https://corp.voicy.jp/
    https://voicy.jp/
    • Ի੠ίϯςϯπΛiOS/Android/WebͰఏڙ

    View full-size slide

  4. VoicyͷΞϓϦ͸࠶ੜͱऩ࿥ͷ2छྨͷΞϓ
    ϦΛఏڙͯ͠·͢
    ΠϯετʔϧϦϯΫ

    View full-size slide

  5. ऩ࿥ͱ࠶ੜͰදཪҰମͰ͞·͟·ͳVoicyͷػೳ
    Ի੠࠶ੜ
    Ի੠ऩ࿥
    ੜ์ૹ
    ίϝϯτ
    ϋογϡλάݕ

    ച্੥ٻ
    ࠩ͠ೖΕ
    ϓϨϛΞϜ์ૹ

    View full-size slide

  6. Ի੠αʔϏε্Ͱ͸ಛʹॏཁͳೋͭͷػೳ
    ίϝϯτ
    ϋογϡλάݕ

    ച্੥ٻ
    ࠩ͠ೖΕ
    ϓϨϛΞϜ์ૹ
    Ի੠࠶ੜ
    Ի੠ऩ࿥
    ੜ์ૹ

    View full-size slide

  7. ར༻͍ͯ͠Δٕज़
    Ի੠࠶ੜԻ੠ऩ࿥
    ੜ์ૹ
    AVAudio
    Agora

    View full-size slide

  8. AVAudioͱ͸ʁ

    View full-size slide

  9. AVFAudioͱ͸ʁ
    https://developer.apple.com/documentation/avfaudio
    "7"VEJP4FTTJPO
    "7"VEJP3PVUJOH"SCJUFS
    "7"VEJP1MBZFS
    "7.*%*1MBZFS
    "7"VEJP3FDPSEFS
    "VEJP&OHJOF
    • AVFoundation͔ΒAudioͷػೳΛ੾Γग़ͨ͠΋ͷͰ͢


    • iOS14.5͔Β

    View full-size slide

  10. AVFAudioͱ͸ʁ
    https://developer.apple.com/documentation/avfaudio
    "7"VEJP4FTTJPO
    "7"VEJP3PVUJOH"SCJUFS
    "7"VEJP1MBZFS
    "7.*%*1MBZFS
    "7"VEJP3FDPSEFS
    "VEJP&OHJOF
    • VoicyͰར༻͍ͯ͠Δػೳ

    View full-size slide

  11. Ի੠ͷऩ࿥

    View full-size slide

  12. ࿥ԻͷྲྀΕ
    ։࢝ ࿥Իத ऴྃ
    - 1tapͰ࿥ԻՄೳͳઃܭ
    Ի੠ͷ೾ܗ৘ใදࣔ

    View full-size slide

  13. ࿥Իݖݶ
    AVAudioSession.sharedInstance().requestRecordPermission { granted in


    self.handlePermissionGranted()


    }


    • ࿥ԻݖݶΛऔಘ
    https://developer.apple.com/documentation/avfaudio/avaudiosession/1616463-
    recordpermission

    View full-size slide

  14. ηογϣϯઃఆ


    try audioSession.setCategory(.playAndRecord)ɹ


    try audioSession.setPreferredSampleRate(44100)


    try
    audioSession.setPreferredIOBufferDuration(defaultPreferredIOBufferDurat
    ion)


    try audioSession.setActive(true)




    https://developer.apple.com/documentation/avfaudio/avaudiosession
    • ࠶ੜऩ࿥ՄೳͳηογϣϯͰ։࢝
    /P*NBHF

    View full-size slide

  15. ࿥Ի։࢝
    var componentDescription = AudioComponentDescription(


    componentType: OSType(kAudioUnitType_Output),


    componentSubType: OSType(kAudioUnitSubType_RemoteIO),


    componentManufacturer: OSType(kAudioUnitManufacturer_Apple),


    componentFlags: UInt32(0),


    componentFlagsMask: UInt32(0)


    )


    let component = AudioComponentFindNext(nil, &componentDescription)


    var tempAudioUnit: AudioUnit?


    AudioComponentInstanceNew(component!, &tempAudioUnit)


    https://developer.apple.com/documentation/audiotoolbox
    • audioUnitΛੜ੒

    View full-size slide

  16. ࿥Ի։࢝
    var inputCallbackStruct = AURenderCallbackStruct(


    inputProc: recordingCallback,


    inputProcRefCon:
    UnsafeMutableRawPointer(Unmanaged.passUnretained(self).toOpaque())


    )


    https://developer.apple.com/documentation/audiotoolbox
    • Callbackͷఆٛ

    View full-size slide

  17. ࿥Ի։࢝
    // ࿥ԻͷCallback


    private let recordingCallback: AURenderCallback = { (inRefCon, ioActionFlags, inTimeStamp,
    inBusNumber, frameCount, _) -> OSStatus in


    audioObject.handlerBufferData(bufferList: &bufferList, frameCount: frameCount)


    }


    • όοϑΝʔ৘ใΛऔಘ

    View full-size slide

  18. ೾ܗͷऔಘ
    let pcmBuffer = AVAudioPCMBuffer(pcmFormat: audioFormat, frameCapacity:
    UInt32(bufferSize))


    buffers.append(buffer)


    waveFormHeights.append(buffer.waveFormHeight)
    • औಘͨ͠όοϑΝʔ৘ใΛ΋ͱʹԻྔΛϨϯμϦϯά

    View full-size slide

  19. ೾ܗͷऔಘ
    6*$PMMFDUJPO7JFX
    let pcmBuffer = AVAudioPCMBuffer(pcmFormat: audioFormat, frameCapacity:
    UInt32(bufferSize))


    buffers.append(buffer)


    waveFormHeights.append(buffer.waveFormHeight)
    • औಘͨ͠όοϑΝʔ৘ใΛ΋ͱʹԻྔΛϨϯμϦϯά

    View full-size slide

  20. ࠶ฤू
    • ऩ࿥ͨ͠Ի੠σʔλΛ্ॻ͖

    View full-size slide

  21. Ի੠ऩ࿥ͷऴྃ
    let settings: [String: Any] = [


    AVSampleRateKey: Constants.Recording.sampleRate,


    AVFormatIDKey: kAudioFormatMPEG4AAC, // MP4ܗࣜͰอଘ


    AVNumberOfChannelsKey: Constants.Recording.numberOfChannel,


    AVEncoderAudioQualityKey: AVAudioQuality.high


    ]


    let audioFile = try AVAudioFile(forWriting: fileURL, settings: settings)


    • Local΁อଘ͠Ξοϓϩʔυ

    View full-size slide

  22. AVAudioRecorder͸ར༻͍ͯ͠ͳ͍
    https://developer.apple.com/documentation/avfaudio
    "7"VEJP4FTTJPO
    "7"VEJP3PVUJOH"SCJUFS
    "7"VEJP1MBZFS
    "7.*%*1MBZFS
    "7"VEJP3FDPSEFS
    "VEJP&OHJOF
    • ։ൃ౰ॳ͸audioUnitͰ༷ʑͳ͜ͱΛ΍Ζ͏ͱ͍ͯͨ͠ͷ͔΋͠Εͳ͍


    • ͓ͦΒ͘೾ܗʹԻྔͷߴ͞Λऔಘ͢ΔͨΊʹར༻
    BVEJP6OJU

    View full-size slide

  23. Ի੠ͷฤू

    View full-size slide

  24. ࠶ฤू
    • αʔϏεϙϦγʔ্Ͱ͖ͳ͍Α͏ʹ͍ͯ͠·͢ʂ

    View full-size slide

  25. Ի੠ͷ։࢝
    Ի੠࠶ੜ։࢝ Ի੠࠶ੜத

    View full-size slide

  26. Ի੠ͷ։࢝
    func playPlayingChapter() {


    voicePlayer.play(url: fileURL, startTime:
    playingChapter.lastPlaytime.msecToSec(), speed: playSpeed.rawValue)


    }
    • νϟϓλʔ୯ҐͰͷ։࢝

    View full-size slide

  27. Ի੠ͷఀࢭ
    func stopPlay() {


    player?.stop()


    player = nil


    }
    • Ի੠ͷఀࢭ

    View full-size slide

  28. background࠶ੜ
    try audioSession.setCategory(.playback, mode: .default, options: [])


    https://developer.apple.com/documentation/avfaudio/avaudiosession/category

    View full-size slide

  29. Agoraͱ͸ʁ
    https://www.agora.io/en/
    https://jp.vcube.com/sdk

    View full-size slide

  30. AgoraͰ͸ϏσΦ௨࿩΍Ի੠഑৴ͷػೳΛఏڙ
    https://www.agora.io/en/

    View full-size slide

  31. ͦͷதͰ΋Live Steaming ͱ Real-time
    MessagingΛར༻
    https://www.agora.io/en/

    View full-size slide

  32. Voicyͷੜ์ૹͱ͸ʁ
    Ի੠ͷ࿈ܞ

    Live Streaming


    (RTC)
    ͓ศΓ΍λΠτϧͷ࿈ܞ


    RealTimeMessage


    (RTM)

    View full-size slide

  33. ੜ์ૹͷྲྀΕ
    ੜ์ૹ։࢝ ࿥Ի ऴྃ

    View full-size slide

  34. ੜ์ૹ
    // RTC


    agoraKit = AgoraRtcEngineKit.sharedEngine(withAppId:
    EnvironmentConfig.agoraAppId, delegate: self)


    agoraKit?.enableAudioVolumeIndication(200, smooth: 3, report_vad: true)


    agoraKit?.setChannelProfile(AgoraChannelProfile.liveBroadcasting)




    // RTM


    agoraRtmKit = AgoraRtmKit(appId: EnvironmentConfig.agoraAppId, delegate: self)


    agoraRtmChannel = agoraRtmKit?.createChannel(withId: liveId, delegate: self)


    • Agora্ͰRTCͱRTMͰͦΕͧΕνϟϯωϧΛ࡞੒

    View full-size slide

  35. ੜ์ૹ࿥Ի
    // ࿥Ի͸αʔόʔαΠυͰ࣮ࢪ͍ͯ͠·͢
    // ͲΜͳॲཧ͔Θ͔ΔΑ͏ʹcurlͷAPIΛهࡌ͠·ͨ͠
    curl --location --request GET 'https://api.agora.io/v1/apps//
    cloud_recording/resourceid//sid//mode//query' \
    • Agoraͷcloud_recording(࿥Ի)ͷػೳ

    View full-size slide

  36. ͓ศΓϝοηʔδͷऔಘ
    func rtmKit(_ kit: AgoraRtmKit, messageReceived message: AgoraRtmMessage, fromPeer peerId: String) {


    messageReceived(message: message, uid: peerId)


    }


    ~~~


    if let message = RTMMessage(rawValue: message.text), let uid = Int(uid) {


    switch message {


    case .hands_up: delegate?.onRaiseHand(userId: uid)


    case .hands_down: delegate?.onDroppedHand(userId: uid)


    case .invite: delegate?.onGuestInvited()


    case .canceled_invite: delegate?.onCancelGuestInvitation()


    case .reject: delegate?.onInvitaionRejected(userId: uid)


    case .set_audience: changeListener()


    case .mute: delegate?.onMute(uid: uid, muted: true)


    case .un_mute: delegate?.onMute(uid: uid, muted: false)


    case .stamp: delegate?.onStampReceived()


    case .edit_live_info: delegate?.onEditLiveInfo()


    case .letter: delegate?.onReceivedLetter(userId: uid)


    case .return_guest_to_listener: delegate?.onReturnGuestToListener()


    case .start: delegate?.onLiveStarted()


    }


    }
    • ϝοηʔδΛऔಘ༷͠ʑͳϦΞϧλΠϜॲཧΛ࣮ݱ

    View full-size slide

  37. ੜ์ૹ
    • AgoraͰੜ์ૹʹࢀՃ
    self.agoraRtmChannel?.join(completion: { code in


    self.agoraKit?.setClientRole(agoraClientRole)


    self.agoraKit?.joinChannel(byToken: entity.agoraToken.rtcToken, channelId:
    String(liveID), info: nil, uid: UInt(entity.uid))


    View full-size slide

  38. ੜ์ૹ
    • ͓ศΓΛૹΔ
    agoraRtmKit?.send(AgoraRtmMessage(text: text.rawValue), toPeer: uid)


    View full-size slide

  39. Ի੠αʔϏε্(Voicy)Ͱ͸ಛʹॏཁͳೋͭͷػೳʹ
    ͍ͭͯ঺հ͠·ͨ͠
    ίϝϯτ
    ϋογϡλάݕ

    ച্੥ٻ
    ࠩ͠ೖΕ
    ϓϨϛΞϜ์ૹ
    Ի੠࠶ੜ
    Ի੠ऩ࿥
    ੜ์ૹ

    View full-size slide

  40. VoicyͷԻ੠ॲཧͷࠓޙͷల๬

    View full-size slide

  41. ࠓޙͷల๬ɹԻ੠࠶ੜ/Ի੠ऩ࿥
    • Ի੠ऩ࿥ͷ඼࣭޲্


    • ϚΠΫ - bluetoothϚΠΫͳͲ͸ి೾͕ׯবͯ͠͠
    ·͍Ի੠͕௿͔ͬͨΓ͢Δ


    • ݱࡏ͸ϚΠΫΛ઀ଓ͠ͳ͍ঢ়ଶͰͷऩ࿥Λਪ঑


    • ࿈ଓ࠶ੜ / ΦϑϥΠϯ࠶ੜͳͲ


    • Ϧεφʔ΁͞ΒʹศརͳԻ੠ମݧΛ

    View full-size slide

  42. ࠓޙͷల๬ɹੜ์ૹ
    • ࿥Ի඼࣭޲্


    • AgoraͷCloud-RecordingͩͱͲ͏͠
    ͯ΋඼࣭͕ྑ͘ͳ͍


    • େਓ਺Ͱ΋଱͑ΒΕΔੜ์ૹ΁


    • AgoraͱαʔόʔαΠυؚΊͨΩϟύ
    γςΟͷ޲্

    View full-size slide

  43. ࠓ৽͘͠Ի੠ΞϓϦΛ࡞Δͱͨ͠ΒͲ͏͢ΔΜͩΖ͏ʁ


    🤔
    ΍ͬͺSwiftUI͔ʁ Swift Concurrencyͱ͔৽
    ػೳ΋࢖͍͍ͨͳ

    View full-size slide

  44. ͱ͍͏Θ͚Ͱ࡞Γ·ͨ͠ʂ
    https://github.com/entaku0818/VoiceMemo

    View full-size slide

  45. αϯϓϧΛࢀߟʹͯ͠࡞Γ·ͨ͠
    https://github.com/pointfreeco/swift-composable-architecture/tree/main/Examples/VoiceMemos
    ͱ͍͏Θ͚Ͱ࡞Γ·ͨ͠ʂ

    View full-size slide

  46. ಛʹ঺հ͍ͨ͠ͱ͜Ζ

    View full-size slide

  47. Ի੠ॲཧͷDeleagteॲཧͷ؅ཧͷ೉͠͞
    • େ͖ͳControllerͰશͯΛίϯτϩʔϧ͕ͪ͠


    • ֤ը໘ͰDelegateͰॲཧΛॻ͖͕ͪ

    View full-size slide

  48. Ի੠ͳͲͰ͸ಛʹଟ͍DelegateॲཧΛ·ͱΊ
    ͯΔ
    https://github.com/entaku0818/VoiceMemo
    private final class Delegate: NSObject, AVAudioPlayerDelegate, Sendable {


    let didFinishPlaying: @Sendable (Bool) -> Void


    let decodeErrorDidOccur: @Sendable (Error?) -> Void


    let player: AVAudioPlayer


    init(


    url: URL,


    didFinishPlaying: @escaping @Sendable (Bool) -> Void,


    decodeErrorDidOccur: @escaping @Sendable (Error?) -> Void


    ) throws {


    self.didFinishPlaying = didFinishPlaying


    self.decodeErrorDidOccur = decodeErrorDidOccur


    self.player = try AVAudioPlayer(contentsOf: url)


    super.init()


    self.player.delegate = self


    }


    func audioPlayerDidFinishPlaying(_ player: AVAudioPlayer, successfully flag: Bool) {


    self.didFinishPlaying(flag)


    }


    func audioPlayerDecodeErrorDidOccur(_ player: AVAudioPlayer, error: Error?) {


    self.decodeErrorDidOccur(error)


    }


    }


    View full-size slide

  49. liveͱ͍͏ঢ়ଶͰDelegateΛఆٛ


    Ի੠։࢝࣌ͷॲཧΛެ։
    https://github.com/entaku0818/VoiceMemo
    struct AudioPlayerClient {


    var play: @Sendable (URL) async throws -> Bool


    }


    extension AudioPlayerClient {


    static let live = Self { url in


    let stream = AsyncThrowingStream { continuation in


    do {


    let delegate = try Delegate(


    url: url,


    didFinishPlaying: { successful in


    continuation.yield(successful)


    continuation.finish()


    },


    decodeErrorDidOccur: { error in


    continuation.finish(throwing: error)


    }


    )


    delegate.player.play()


    continuation.onTermination = { _ in


    delegate.player.stop()


    }


    } catch {


    continuation.finish(throwing: error)


    }


    }


    return try await stream.first(where: { _ in true }) ?? false


    }


    }


    4XJGU$PODVSSFODZ
    Ͱ࣮૷

    View full-size slide

  50. ReducerͰը໘ͰඞཁͳॲཧΛఆٛ
    https://github.com/entaku0818/VoiceMemo
    switch state.mode {


    case .notPlaying:


    state.mode = .playing(progress: 0)


    return .run { [url = state.url] send in


    let start = environment.mainRunLoop.now


    async let playAudio: Void = send(


    .audioPlayerClient(TaskResult { try await environment.audioPlayer.play(url) })


    )


    for try await tick in environment.mainRunLoop.timer(interval: 0.5) {


    await send(.timerUpdated(tick.date.timeIntervalSince(start.date)))


    }


    }


    .cancellable(id: PlayID.self, cancelInFlight: true)


    case .playing:


    state.mode = .notPlaying


    return .cancel(id: PlayID.self)


    }


    View full-size slide

  51. ࢀߟࢿྉ
    • agoraΛ࢖ͬͯϥΠϒ഑৴ػೳΛ1ϲ݄൒ͰϦϦʔεͨ͠࿩


    • εϥΠυ


    • ࠓ೔͔Β෼͔Δ AVAudioEngineͷશͯ


    • εϥΠυ


    • ࣮૷ͨ͘͠ͳΔԻ੠ฤू


    • εϥΠυ
    • աڈͷԻ੠ʹؔΘΔiOSDCͷηογϣϯ

    View full-size slide

  52. ࢀߟࢿྉ
    • ΍ͬͺΓ࠷ޙ͸ެࣜ


    • https://developer.apple.com/documentation/avfaudio


    • https://docs.agora.io/en

    View full-size slide

  53. https://voicy.jp/channel/1305
    VoicyͷΤϯδχΞ͕ӡӦ͍ͯ͠Δνϟϯωϧ


    voi-chordௌ͍͍ͯͩ͘͞ʂ

    View full-size slide

  54. iOSΤϯδχΞΛืू͍ͯ͠·͢
    • Ի੠޷͖ͳΤϯδχΞืूதͰ͢ʂڵຯ͋Δํ͸DM͍ͩ͘͞ʂ
    Notion
    Twitter: @entaku_0818

    View full-size slide