Save 37% off PRO during our Black Friday Sale! »

TwilioのCameraSource に任意のフレームを使用するすゝめ

22e056dcb79e14363e844f1e57986325?s=47 BOB
December 22, 2020
Programming
0
1k

TwilioのCameraSource に任意のフレームを使用するすゝめ

potatotips 72

22e056dcb79e14363e844f1e57986325?s=128

BOB

December 22, 2020
Tweet

More Decks by BOB

See All by BOB
22e056dcb79e14363e844f1e57986325?s=48 aratayokoyama
0
180
22e056dcb79e14363e844f1e57986325?s=48 aratayokoyama
0
610
22e056dcb79e14363e844f1e57986325?s=48 aratayokoyama
0
68

Other Decks in Programming

See All in Programming
A1792748885fe8f2555d9c718ee7ae53?s=48 joytomo
1
300
A180da03d518c141f2e03a6a32c40b11?s=48 ikustan
0
300
6362c07dc6c77f281f73285cea2f2d5c?s=48 kzkn
0
120
04df7340898eac3a0536d3b47c55501c?s=48 hirotokirimaru
0
240
4d40d82cc3c676e8a67ffc2a473bf423?s=48 kyoheig3
0
100
F21d69109b1c03921abf7d12f0fb6654?s=48 dscs
0
130
Ec2fcdc4ea7905b289967a2c4c43e154?s=48 sgeengineer
0
270
795a1c8d5e46f6b9067202655ea5dfae?s=48 ringa_hyj
3
220
52cc8a838bf44a589d2572833b2dd1b9?s=48 palkan
2
130
229b1596ce57cd0935a2bacd410d87a0?s=48 aaaddress1
2
720
E228752b0ba3d85d485083c7fc6c3622?s=48 hainet50b
5
960
0ecada41f471c45033ee166c37819915?s=48 artoy
0
190

Featured

See All Featured
6804f1775cb4babfcc3851298566fbce?s=48 tanoku
261
24k
0438ae60cde4c7add6f9da48f28c15cc?s=48 chrisshort
6
28k
0c6fd6a08d0f898159cda7cafcacf07f?s=48 afnizarnur
181
14k
B3d6434763caa0ef5dc4b792662c49f7?s=48 smashingmag
236
18k
3d4519fd34b76fb265fc0237f3792bd4?s=48 danielanewman
6
790
5f2da528927a2ec9ba4fec2069cbc958?s=48 kneath
298
39k
4394ceb8b277f35ee3da7030384efb5d?s=48 davidbonilla
75
3.8k
D8fc2580cfaca035f666d9e4ee79a7f7?s=48 mongodb
25
4.2k
15f2b8221f247985a27a627d2ece72f0?s=48 pauljervisheath
195
16k
0fe2973fa8d27d96547e43322341183a?s=48 swwweet
211
7.3k
A60068bce2e73de3a37ca9d2dbe36092?s=48 bryan
35
3.9k
E9221b172e78e888355fa2fe4541b595?s=48 mclloyd
PRO
7
7.7k

Transcript

  1. ʲJ04ʳ5XJMJPͷ$BNFSB4PVSDF ʹ೚ҙͷϑϨʔϜΛ࢖༻͢Δ͢ʍΊ "3"5":0,0:"#0#!QPUBUPUJQTJ04"OESPJE։ൃ5JQTڞ༗ձ

  2. !#0#(7J "SBUB:PLPZBNB#0#  # ळݩ߁ؔ࿈ʹ10೥Ҏ্අ΍͢ # ๺ւಓੜ·Ε # ͸ͩͯ͜ະདྷେଔ #

    2020೥αΠόʔΤʔδΣϯτ৽ଔೖࣾ # λοϓϧͷiOSΤϯδχΞ

  3. λοϓϧ  w ୅޲͚ͷϚονϯάΞϓϦ w ೥݄ϦϦʔε w ϑϦοΫ6* w ຊਓೝূɼը໘࿥ը๷ࢭ

    w 6*͕உঁͰେ͖͘ҧ͏ w 5XJMJPΛ࢖༻ͨ͠ϏσΦνϟοτ

  4. λοϓϧɿϏσΦνϟοτ  w ݄ϏσΦνϟοτϦϦʔε w 5XJMJPΛ࢖༻͓ͯ͠Γ·͢ w ݄ϑΟϧλʔػೳϦϦʔε w ʮ͓෦԰ͷย෇͚͕େมʯɼ

    ʮ਎ͩ͠ͳΈΛ੔͑Δͷʹ͕͔͔࣌ؒΔʯ ͱ͍ͬͨ੠Λड͚ͯ։ൃ͍ͨ͠·ͨ͠ w ϑΟϧλʔػೳ͸"3,JU΋࢖༻͓ͯ͠Γ·͢ ϚονϯάΞϓϦʮλοϓϧ஀ੜʯɺ͙͢ʹΦϯϥΠϯσʔτ͕Ͱ͖Δ ʮϏσΦνϟοτʯʹ৽ͨʹʮϑΟϧλʔػೳʯΛ௥Ճɿhttps://www.cyberagent.co.jp/news/detail/id=25083 5XJMJPհͯ͠೚ҙͷϑϨʔϜΛૹΔํ๏Λ͝঺հ

  5.  (0"- ΞδΣϯμ CameraSourceʹ೚ҙͷϑϨʔϜΛ࢖༻͢ΔϙΠϯτ͸˓˓  $BNFSB4PVSDFૹΔʹ͸˓˓͕େ੾  "7$BQUVSFɿϑϨʔϜͷऔΓํ  "3,JUɿϑϨʔϜͷऔΓํ

  6. $BNFSB4PVSDFʹૹΔʹ͸˓˓͕େ੾

  7. $BNFSB4PVSDFʹૹΔʹ͸˓˓͕େ੾  // TVIVideoFrame - (nullable instancetype)initWithTimestamp:(CMTime)timestamp buffer:(nonnull CVImageBufferRef)imageBuffer orientation:(TVIVideoOrientation)orientation;

    - (nullable instancetype)initWithTimeInterval:(CFTimeInterval)timeInterval buffer:(nonnull CVImageBufferRef)imageBuffer orientation:(TVIVideoOrientation)orientation; TVIVideoFrame Class Referenceɿhttps://twilio.github.io/twilio-video-ios/docs/latest/Classes/TVIVideoFrame.html#//api/name/initWithTimeInterval:buffer:orientation:

  8. $BNFSB4PVSDFʹૹΔʹ͸˓˓͕େ੾  // TVIVideoFrame - (nullable instancetype)initWithTimestamp:(CMTime)timestamp buffer:(nonnull CVImageBufferRef)imageBuffer orientation:(TVIVideoOrientation)orientation;

    - (nullable instancetype)initWithTimeInterval:(CFTimeInterval)timeInterval buffer:(nonnull CVImageBufferRef)imageBuffer orientation:(TVIVideoOrientation)orientation; CVImageBufferRef CVImageBufferRef TVIVideoFrame Class Referenceɿhttps://twilio.github.io/twilio-video-ios/docs/latest/Classes/TVIVideoFrame.html#//api/name/initWithTimeInterval:buffer:orientation:

  9. $BNFSB4PVSDFʹૹΔʹ͸˓˓͕େ੾  // TVIVideoFrame - (nullable instancetype)initWithTimestamp:(CMTime)timestamp buffer:(nonnull CVImageBufferRef)imageBuffer orientation:(TVIVideoOrientation)orientation;

    - (nullable instancetype)initWithTimeInterval:(CFTimeInterval)timeInterval buffer:(nonnull CVImageBufferRef)imageBuffer orientation:(TVIVideoOrientation)orientation; CMTime CFTimeInterval ϑϨʔϜΛऔಘ͢Δ Ճ޻͢Δ 7JEFP'SBNFΛ ࡞੒͢Δ ྲྀ͠ࠐΉ " # TVIVideoFrame Class Referenceɿhttps://twilio.github.io/twilio-video-ios/docs/latest/Classes/TVIVideoFrame.html#//api/name/initWithTimeInterval:buffer:orientation: $ %

  10. $BNFSB4PVSDFʹૹΔʹ͸4JOL͕େ੾  import TwilioVideo final class LocalTrackSourceManager: NSObject { private(set)

    var camera: CameraSource! func onVideoFrame(image: CIImage, time: TimeInterval) { guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let videoFrame = VideoFrame(timeInterval: time, buffer: pixelBuffer, orientation: VideoOrientation.up) else { return } camera.sink?.onVideoFrame(videoFrame) } func onVideoFrame(image: CIImage, time: CMTime) { guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let videoFrame = VideoFrame(timestamp: time, buffer: pixelBuffer, orientation: VideoOrientation.up) else { return } camera.sink?.onVideoFrame(videoFrame) } } # $ %

  11. import TwilioVideo final class LocalTrackSourceManager: NSObject { private(set) var camera:

    CameraSource! func onVideoFrame(image: CIImage, time: TimeInterval) { guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let videoFrame = VideoFrame(timeInterval: time, buffer: pixelBuffer, orientation: VideoOrientation.up) else { return } camera.sink?.onVideoFrame(videoFrame) } func onVideoFrame(image: CIImage, time: CMTime) { guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let videoFrame = VideoFrame(timestamp: time, buffer: pixelBuffer, orientation: VideoOrientation.up) else { return } camera.sink?.onVideoFrame(videoFrame) } } $BNFSB4PVSDFʹૹΔʹ͸4JOL͕େ੾  # $ % guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let pixelBuffer = image.convertCVPixelBuffer() else { return }

  12. import TwilioVideo final class LocalTrackSourceManager: NSObject { private(set) var camera:

    CameraSource! func onVideoFrame(image: CIImage, time: TimeInterval) { guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let videoFrame = VideoFrame(timeInterval: time, buffer: pixelBuffer, orientation: VideoOrientation.up) else { return } camera.sink?.onVideoFrame(videoFrame) } func onVideoFrame(image: CIImage, time: CMTime) { guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let videoFrame = VideoFrame(timestamp: time, buffer: pixelBuffer, orientation: VideoOrientation.up) else { return } camera.sink?.onVideoFrame(videoFrame) } } $BNFSB4PVSDFʹૹΔʹ͸4JOL͕େ੾  # $ % VideoFrame(timeInterval: time, buffer: pixelBuffer, orientation: VideoOrientation.up) VideoFrame(timestamp: time, buffer: pixelBuffer, orientation: VideoOrientation.up)

  13. import TwilioVideo final class LocalTrackSourceManager: NSObject { private(set) var camera:

    CameraSource! func onVideoFrame(image: CIImage, time: TimeInterval) { guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let videoFrame = VideoFrame(timeInterval: time, buffer: pixelBuffer, orientation: VideoOrientation.up) else { return } camera.sink?.onVideoFrame(videoFrame) } func onVideoFrame(image: CIImage, time: CMTime) { guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let videoFrame = VideoFrame(timestamp: time, buffer: pixelBuffer, orientation: VideoOrientation.up) else { return } camera.sink?.onVideoFrame(videoFrame) } } $BNFSB4PVSDFʹૹΔʹ͸4JOL͕େ੾  camera.sink?.onVideoFrame(videoFrame) camera.sink?.onVideoFrame(videoFrame)

  14. $BNFSB4PVSDFʹૹΔʹ͸4JOL͕େ੾  import TwilioVideo final class LocalTrackSourceManager: NSObject { private(set)

    var camera: CameraSource! func onVideoFrame(image: CIImage, time: TimeInterval) { guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let videoFrame = VideoFrame(timeInterval: time, buffer: pixelBuffer, orientation: VideoOrientation.up) else { return } camera.sink?.onVideoFrame(videoFrame) } func onVideoFrame(image: CIImage, time: CMTime) { guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let videoFrame = VideoFrame(timestamp: time, buffer: pixelBuffer, orientation: VideoOrientation.up) else { return } camera.sink?.onVideoFrame(videoFrame) } } camera.sink?.onVideoFrame(videoFrame) camera.sink?.onVideoFrame(videoFrame) extension VideoChatAVCaptureView: VideoSource { weak var sink: VideoSink? var isScreencast: Bool { return false } func requestOutputFormat(_ outputFormat: VideoFormat) { guard let sink = sink else { return } sink.onVideoFormatRequest(outputFormat) } } w 7JEFP4PVSDFʹ४ڌͤͯ͞TJOLΛ࣋ͭ͜ͱ΋Մೳ w ͨͩ͠ɼৗʹ4JOL΁ϑϨʔϜΛૹΔ͜ͱ͕৚݅ ʢΫϥογϡ͠·͢ʣ w Ұํɼ$BNFSB4PVSDFͷ4JOLʹ͸ͦͷ৚͕݅ͳ͍ 5JQT var camera: CameraSource!

  15. "7$BQUVSFɿϑϨʔϜͷऔΓํ

  16. "7$BQUVSFɿϑϨʔϜͷऔΓํ  import AVFoundation import UIKit final class VideoChatAVCaptureView: UIView

    { private var session: AVCaptureSession? private func prepare() { let output: AVCaptureVideoDataOutput = AVCaptureVideoDataOutput() output.setSampleBufferDelegate(self, queue: DispatchQueue(label: "videoChat")) } } extension VideoChatAVCaptureView: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { guard let ciImage = sampleBuffer.convertCIImage() else { return } let timestamp = CMSampleBufferGetPresentationTimeStamp(sampleBuffer) localTrackSourceManager.onVideoFrame(image: ciImage, time: timestamp) } } " #

  17. "7$BQUVSFɿϑϨʔϜͷऔΓํ  import AVFoundation import UIKit final class VideoChatAVCaptureView: UIView

    { private var session: AVCaptureSession? private func prepare() { let output: AVCaptureVideoDataOutput = AVCaptureVideoDataOutput() output.setSampleBufferDelegate(self, queue: DispatchQueue(label: "videoChat")) } } extension VideoChatAVCaptureView: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { guard let ciImage = sampleBuffer.convertCIImage() else { return } let timestamp = CMSampleBufferGetPresentationTimeStamp(sampleBuffer) localTrackSourceManager.onVideoFrame(image: ciImage, time: timestamp) } } output.setSampleBufferDelegate(self, queue: DispatchQueue(label: "videoChat")) EFMFHBUFΛࣗ਎ʹ͋ͯ·͢

  18. "7$BQUVSFɿϑϨʔϜͷऔΓํ  import AVFoundation import UIKit final class VideoChatAVCaptureView: UIView

    { private var session: AVCaptureSession? private func prepare() { let output: AVCaptureVideoDataOutput = AVCaptureVideoDataOutput() output.setSampleBufferDelegate(self, queue: DispatchQueue(label: "videoChat")) } } extension VideoChatAVCaptureView: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { guard let ciImage = sampleBuffer.convertCIImage() else { return } let timestamp = CMSampleBufferGetPresentationTimeStamp(sampleBuffer) localTrackSourceManager.onVideoFrame(image: ciImage, time: timestamp) } } guard let ciImage = sampleBuffer.convertCIImage() else { return } $.4BNQMF#V⒎FSΛ$**NBHFʹม׵͠·͢

  19. "7$BQUVSFɿϑϨʔϜͷऔΓํ  import AVFoundation import UIKit final class VideoChatAVCaptureView: UIView

    { private var session: AVCaptureSession? private func prepare() { let output: AVCaptureVideoDataOutput = AVCaptureVideoDataOutput() output.setSampleBufferDelegate(self, queue: DispatchQueue(label: "videoChat")) } } extension VideoChatAVCaptureView: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { guard let ciImage = sampleBuffer.convertCIImage() else { return } let timestamp = CMSampleBufferGetPresentationTimeStamp(sampleBuffer) localTrackSourceManager.onVideoFrame(image: ciImage, time: timestamp) } } let timestamp = CMSampleBufferGetPresentationTimeStamp(sampleBuffer) $.5JNFΛ࡞੒͠·͢

  20. "3,JUɿϑϨʔϜͷऔΓํ

  21. "3,JUɿϑϨʔϜͷऔΓํ  import ARKit import SceneKit import UIKit final class

    VideoChatARKitView: UIView { private lazy var sceneView: ARSCNView = {...}() private func prepare() { sceneView.delegate = self } } extension VideoChatARKitView: ARSCNViewDelegate { func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) { guard let sceneView = renderer as? ARSCNView else { return } guard let ciImage = CIImage(image: sceneView.snapshot()) else { return } localTrackSourceManager.onVideoFrame(image: ciImage, time: time) } } " #

  22. import ARKit import SceneKit import UIKit final class VideoChatARKitView: UIView

    { private lazy var sceneView: ARSCNView = {...}() private func prepare() { sceneView.delegate = self } } extension VideoChatARKitView: ARSCNViewDelegate { func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) { guard let sceneView = renderer as? ARSCNView else { return } guard let ciImage = CIImage(image: sceneView.snapshot()) else { return } localTrackSourceManager.onVideoFrame(image: ciImage, time: time) } } "3,JUɿϑϨʔϜͷऔΓํ  sceneView.delegate = self EFMFHBUFΛࣗ਎ʹ͋ͯ·͢

  23. "3,JUɿϑϨʔϜͷऔΓํ  import ARKit import SceneKit import UIKit final class

    VideoChatARKitView: UIView { private lazy var sceneView: ARSCNView = {...}() private func prepare() { sceneView.delegate = self } } extension VideoChatARKitView: ARSCNViewDelegate { func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) { guard let sceneView = renderer as? ARSCNView else { return } guard let ciImage = CIImage(image: sceneView.snapshot()) else { return } localTrackSourceManager.onVideoFrame(image: ciImage, time: time) } } let ciImage = CIImage(image: sceneView.snapshot()) εφοϓγϣοτΛऔಘ͠·͢

  24. "3,JUɿϑϨʔϜͷऔΓํ  import ARKit import SceneKit import UIKit final class

    VideoChatARKitView: UIView { private lazy var sceneView: ARSCNView = {...}() private func prepare() { sceneView.delegate = self } } extension VideoChatARKitView: ARSCNViewDelegate { func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) { guard let sceneView = renderer as? ARSCNView else { return } guard let ciImage = CIImage(image: sceneView.snapshot()) else { return } localTrackSourceManager.onVideoFrame(image: ciImage, time: time) } } time: TimeInterval ͦͷ··౉͢͜ͱ͕Ͱ͖·͢

  25. ·ͱΊ

  26. ·ͱΊ  (0"- εςοϓ CameraSourceʹ೚ҙͷϑϨʔϜΛ࢖༻͢ΔϙΠϯτ͸Sink ϑϨʔϜΛऔಘ͢Δ Ճ޻͢Δ 7JEFP'SBNFΛ ࡞੒͢Δ ྲྀ͠ࠐΉ

    " # $ %