Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up
for free
TwilioのCameraSource に任意のフレームを使用するすゝめ
BOB
December 22, 2020
Programming
0
240
TwilioのCameraSource に任意のフレームを使用するすゝめ
potatotips 72
BOB
December 22, 2020
Tweet
Share
More Decks by BOB
See All by BOB
aratayokoyama
0
55
Other Decks in Programming
See All in Programming
horie1024
1
430
kyoheig3
0
440
taoshotaro
1
370
makicamel
1
190
mfunaki
1
510
line_developers_tw
0
560
s103ng
0
190
manfredsteyer
PRO
0
110
line_developers_tw
1
520
nbkouhou
9
4.9k
7603
0
190
saki4869
0
200
Featured
See All Featured
cherdarchuk
71
260k
shpigford
368
42k
mza
80
4.1k
ammeep
656
54k
eileencodes
113
25k
bkeepers
52
4.2k
lara
172
9.5k
bryan
100
11k
hursman
106
9.2k
productmarketing
5
660
bkeepers
321
53k
swwweet
206
6.8k
Transcript
ʲJ04ʳ5XJMJPͷ$BNFSB4PVSDF ʹҙͷϑϨʔϜΛ༻͢Δ͢ʍΊ "3"5":0,0:"#0#!QPUBUPUJQTJ04"OESPJE։ൃ5JQTڞ༗ձ
!#0#(7J "SBUB:PLPZBNB#0# # ळݩ߁ؔ࿈ʹ10Ҏ্අ͢ # ւಓੜ·Ε # ͩͯ͜ະདྷେଔ #
2020αΠόʔΤʔδΣϯτ৽ଔೖࣾ # λοϓϧͷiOSΤϯδχΞ
λοϓϧ w ͚ͷϚονϯάΞϓϦ w ݄ϦϦʔε w ϑϦοΫ6* w ຊਓೝূɼը໘ըࢭ
w 6*͕உঁͰେ͖͘ҧ͏ w 5XJMJPΛ༻ͨ͠ϏσΦνϟοτ
λοϓϧɿϏσΦνϟοτ w ݄ϏσΦνϟοτϦϦʔε w 5XJMJPΛ༻͓ͯ͠Γ·͢ w ݄ϑΟϧλʔػೳϦϦʔε w ʮ͓෦ͷย͚͕େมʯɼ
ʮͩ͠ͳΈΛ͑Δͷʹ͕͔͔࣌ؒΔʯ ͱ͍ͬͨΛड͚ͯ։ൃ͍ͨ͠·ͨ͠ w ϑΟϧλʔػೳ"3,JU༻͓ͯ͠Γ·͢ ϚονϯάΞϓϦʮλοϓϧੜʯɺ͙͢ʹΦϯϥΠϯσʔτ͕Ͱ͖Δ ʮϏσΦνϟοτʯʹ৽ͨʹʮϑΟϧλʔػೳʯΛՃɿhttps://www.cyberagent.co.jp/news/detail/id=25083 5XJMJPհͯ͠ҙͷϑϨʔϜΛૹΔํ๏Λ͝հ
(0"- ΞδΣϯμ CameraSourceʹҙͷϑϨʔϜΛ༻͢ΔϙΠϯτ˓˓ $BNFSB4PVSDFૹΔʹ˓˓͕େ "7$BQUVSFɿϑϨʔϜͷऔΓํ "3,JUɿϑϨʔϜͷऔΓํ
$BNFSB4PVSDFʹૹΔʹ˓˓͕େ
$BNFSB4PVSDFʹૹΔʹ˓˓͕େ // TVIVideoFrame - (nullable instancetype)initWithTimestamp:(CMTime)timestamp buffer:(nonnull CVImageBufferRef)imageBuffer orientation:(TVIVideoOrientation)orientation;
- (nullable instancetype)initWithTimeInterval:(CFTimeInterval)timeInterval buffer:(nonnull CVImageBufferRef)imageBuffer orientation:(TVIVideoOrientation)orientation; TVIVideoFrame Class Referenceɿhttps://twilio.github.io/twilio-video-ios/docs/latest/Classes/TVIVideoFrame.html#//api/name/initWithTimeInterval:buffer:orientation:
$BNFSB4PVSDFʹૹΔʹ˓˓͕େ // TVIVideoFrame - (nullable instancetype)initWithTimestamp:(CMTime)timestamp buffer:(nonnull CVImageBufferRef)imageBuffer orientation:(TVIVideoOrientation)orientation;
- (nullable instancetype)initWithTimeInterval:(CFTimeInterval)timeInterval buffer:(nonnull CVImageBufferRef)imageBuffer orientation:(TVIVideoOrientation)orientation; CVImageBufferRef CVImageBufferRef TVIVideoFrame Class Referenceɿhttps://twilio.github.io/twilio-video-ios/docs/latest/Classes/TVIVideoFrame.html#//api/name/initWithTimeInterval:buffer:orientation:
$BNFSB4PVSDFʹૹΔʹ˓˓͕େ // TVIVideoFrame - (nullable instancetype)initWithTimestamp:(CMTime)timestamp buffer:(nonnull CVImageBufferRef)imageBuffer orientation:(TVIVideoOrientation)orientation;
- (nullable instancetype)initWithTimeInterval:(CFTimeInterval)timeInterval buffer:(nonnull CVImageBufferRef)imageBuffer orientation:(TVIVideoOrientation)orientation; CMTime CFTimeInterval ϑϨʔϜΛऔಘ͢Δ Ճ͢Δ 7JEFP'SBNFΛ ࡞͢Δ ྲྀ͠ࠐΉ " # TVIVideoFrame Class Referenceɿhttps://twilio.github.io/twilio-video-ios/docs/latest/Classes/TVIVideoFrame.html#//api/name/initWithTimeInterval:buffer:orientation: $ %
$BNFSB4PVSDFʹૹΔʹ4JOL͕େ import TwilioVideo final class LocalTrackSourceManager: NSObject { private(set)
var camera: CameraSource! func onVideoFrame(image: CIImage, time: TimeInterval) { guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let videoFrame = VideoFrame(timeInterval: time, buffer: pixelBuffer, orientation: VideoOrientation.up) else { return } camera.sink?.onVideoFrame(videoFrame) } func onVideoFrame(image: CIImage, time: CMTime) { guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let videoFrame = VideoFrame(timestamp: time, buffer: pixelBuffer, orientation: VideoOrientation.up) else { return } camera.sink?.onVideoFrame(videoFrame) } } # $ %
import TwilioVideo final class LocalTrackSourceManager: NSObject { private(set) var camera:
CameraSource! func onVideoFrame(image: CIImage, time: TimeInterval) { guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let videoFrame = VideoFrame(timeInterval: time, buffer: pixelBuffer, orientation: VideoOrientation.up) else { return } camera.sink?.onVideoFrame(videoFrame) } func onVideoFrame(image: CIImage, time: CMTime) { guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let videoFrame = VideoFrame(timestamp: time, buffer: pixelBuffer, orientation: VideoOrientation.up) else { return } camera.sink?.onVideoFrame(videoFrame) } } $BNFSB4PVSDFʹૹΔʹ4JOL͕େ # $ % guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let pixelBuffer = image.convertCVPixelBuffer() else { return }
import TwilioVideo final class LocalTrackSourceManager: NSObject { private(set) var camera:
CameraSource! func onVideoFrame(image: CIImage, time: TimeInterval) { guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let videoFrame = VideoFrame(timeInterval: time, buffer: pixelBuffer, orientation: VideoOrientation.up) else { return } camera.sink?.onVideoFrame(videoFrame) } func onVideoFrame(image: CIImage, time: CMTime) { guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let videoFrame = VideoFrame(timestamp: time, buffer: pixelBuffer, orientation: VideoOrientation.up) else { return } camera.sink?.onVideoFrame(videoFrame) } } $BNFSB4PVSDFʹૹΔʹ4JOL͕େ # $ % VideoFrame(timeInterval: time, buffer: pixelBuffer, orientation: VideoOrientation.up) VideoFrame(timestamp: time, buffer: pixelBuffer, orientation: VideoOrientation.up)
import TwilioVideo final class LocalTrackSourceManager: NSObject { private(set) var camera:
CameraSource! func onVideoFrame(image: CIImage, time: TimeInterval) { guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let videoFrame = VideoFrame(timeInterval: time, buffer: pixelBuffer, orientation: VideoOrientation.up) else { return } camera.sink?.onVideoFrame(videoFrame) } func onVideoFrame(image: CIImage, time: CMTime) { guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let videoFrame = VideoFrame(timestamp: time, buffer: pixelBuffer, orientation: VideoOrientation.up) else { return } camera.sink?.onVideoFrame(videoFrame) } } $BNFSB4PVSDFʹૹΔʹ4JOL͕େ camera.sink?.onVideoFrame(videoFrame) camera.sink?.onVideoFrame(videoFrame)
$BNFSB4PVSDFʹૹΔʹ4JOL͕େ import TwilioVideo final class LocalTrackSourceManager: NSObject { private(set)
var camera: CameraSource! func onVideoFrame(image: CIImage, time: TimeInterval) { guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let videoFrame = VideoFrame(timeInterval: time, buffer: pixelBuffer, orientation: VideoOrientation.up) else { return } camera.sink?.onVideoFrame(videoFrame) } func onVideoFrame(image: CIImage, time: CMTime) { guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let videoFrame = VideoFrame(timestamp: time, buffer: pixelBuffer, orientation: VideoOrientation.up) else { return } camera.sink?.onVideoFrame(videoFrame) } } camera.sink?.onVideoFrame(videoFrame) camera.sink?.onVideoFrame(videoFrame) extension VideoChatAVCaptureView: VideoSource { weak var sink: VideoSink? var isScreencast: Bool { return false } func requestOutputFormat(_ outputFormat: VideoFormat) { guard let sink = sink else { return } sink.onVideoFormatRequest(outputFormat) } } w 7JEFP4PVSDFʹ४ڌͤͯ͞TJOLΛ࣋ͭ͜ͱՄೳ w ͨͩ͠ɼৗʹ4JOLϑϨʔϜΛૹΔ͜ͱ͕݅ ʢΫϥογϡ͠·͢ʣ w Ұํɼ$BNFSB4PVSDFͷ4JOLʹͦͷ͕݅ͳ͍ 5JQT var camera: CameraSource!
"7$BQUVSFɿϑϨʔϜͷऔΓํ
"7$BQUVSFɿϑϨʔϜͷऔΓํ import AVFoundation import UIKit final class VideoChatAVCaptureView: UIView
{ private var session: AVCaptureSession? private func prepare() { let output: AVCaptureVideoDataOutput = AVCaptureVideoDataOutput() output.setSampleBufferDelegate(self, queue: DispatchQueue(label: "videoChat")) } } extension VideoChatAVCaptureView: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { guard let ciImage = sampleBuffer.convertCIImage() else { return } let timestamp = CMSampleBufferGetPresentationTimeStamp(sampleBuffer) localTrackSourceManager.onVideoFrame(image: ciImage, time: timestamp) } } " #
"7$BQUVSFɿϑϨʔϜͷऔΓํ import AVFoundation import UIKit final class VideoChatAVCaptureView: UIView
{ private var session: AVCaptureSession? private func prepare() { let output: AVCaptureVideoDataOutput = AVCaptureVideoDataOutput() output.setSampleBufferDelegate(self, queue: DispatchQueue(label: "videoChat")) } } extension VideoChatAVCaptureView: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { guard let ciImage = sampleBuffer.convertCIImage() else { return } let timestamp = CMSampleBufferGetPresentationTimeStamp(sampleBuffer) localTrackSourceManager.onVideoFrame(image: ciImage, time: timestamp) } } output.setSampleBufferDelegate(self, queue: DispatchQueue(label: "videoChat")) EFMFHBUFΛࣗʹ͋ͯ·͢
"7$BQUVSFɿϑϨʔϜͷऔΓํ import AVFoundation import UIKit final class VideoChatAVCaptureView: UIView
{ private var session: AVCaptureSession? private func prepare() { let output: AVCaptureVideoDataOutput = AVCaptureVideoDataOutput() output.setSampleBufferDelegate(self, queue: DispatchQueue(label: "videoChat")) } } extension VideoChatAVCaptureView: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { guard let ciImage = sampleBuffer.convertCIImage() else { return } let timestamp = CMSampleBufferGetPresentationTimeStamp(sampleBuffer) localTrackSourceManager.onVideoFrame(image: ciImage, time: timestamp) } } guard let ciImage = sampleBuffer.convertCIImage() else { return } $.4BNQMF#V⒎FSΛ$**NBHFʹม͠·͢
"7$BQUVSFɿϑϨʔϜͷऔΓํ import AVFoundation import UIKit final class VideoChatAVCaptureView: UIView
{ private var session: AVCaptureSession? private func prepare() { let output: AVCaptureVideoDataOutput = AVCaptureVideoDataOutput() output.setSampleBufferDelegate(self, queue: DispatchQueue(label: "videoChat")) } } extension VideoChatAVCaptureView: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { guard let ciImage = sampleBuffer.convertCIImage() else { return } let timestamp = CMSampleBufferGetPresentationTimeStamp(sampleBuffer) localTrackSourceManager.onVideoFrame(image: ciImage, time: timestamp) } } let timestamp = CMSampleBufferGetPresentationTimeStamp(sampleBuffer) $.5JNFΛ࡞͠·͢
"3,JUɿϑϨʔϜͷऔΓํ
"3,JUɿϑϨʔϜͷऔΓํ import ARKit import SceneKit import UIKit final class
VideoChatARKitView: UIView { private lazy var sceneView: ARSCNView = {...}() private func prepare() { sceneView.delegate = self } } extension VideoChatARKitView: ARSCNViewDelegate { func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) { guard let sceneView = renderer as? ARSCNView else { return } guard let ciImage = CIImage(image: sceneView.snapshot()) else { return } localTrackSourceManager.onVideoFrame(image: ciImage, time: time) } } " #
import ARKit import SceneKit import UIKit final class VideoChatARKitView: UIView
{ private lazy var sceneView: ARSCNView = {...}() private func prepare() { sceneView.delegate = self } } extension VideoChatARKitView: ARSCNViewDelegate { func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) { guard let sceneView = renderer as? ARSCNView else { return } guard let ciImage = CIImage(image: sceneView.snapshot()) else { return } localTrackSourceManager.onVideoFrame(image: ciImage, time: time) } } "3,JUɿϑϨʔϜͷऔΓํ sceneView.delegate = self EFMFHBUFΛࣗʹ͋ͯ·͢
"3,JUɿϑϨʔϜͷऔΓํ import ARKit import SceneKit import UIKit final class
VideoChatARKitView: UIView { private lazy var sceneView: ARSCNView = {...}() private func prepare() { sceneView.delegate = self } } extension VideoChatARKitView: ARSCNViewDelegate { func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) { guard let sceneView = renderer as? ARSCNView else { return } guard let ciImage = CIImage(image: sceneView.snapshot()) else { return } localTrackSourceManager.onVideoFrame(image: ciImage, time: time) } } let ciImage = CIImage(image: sceneView.snapshot()) εφοϓγϣοτΛऔಘ͠·͢
"3,JUɿϑϨʔϜͷऔΓํ import ARKit import SceneKit import UIKit final class
VideoChatARKitView: UIView { private lazy var sceneView: ARSCNView = {...}() private func prepare() { sceneView.delegate = self } } extension VideoChatARKitView: ARSCNViewDelegate { func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) { guard let sceneView = renderer as? ARSCNView else { return } guard let ciImage = CIImage(image: sceneView.snapshot()) else { return } localTrackSourceManager.onVideoFrame(image: ciImage, time: time) } } time: TimeInterval ͦͷ··͢͜ͱ͕Ͱ͖·͢
·ͱΊ
·ͱΊ (0"- εςοϓ CameraSourceʹҙͷϑϨʔϜΛ༻͢ΔϙΠϯτSink ϑϨʔϜΛऔಘ͢Δ Ճ͢Δ 7JEFP'SBNFΛ ࡞͢Δ ྲྀ͠ࠐΉ
" # $ %