Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
TwilioのCameraSource に任意のフレームを使用するすゝめ
BOB
December 22, 2020
Programming
0
1.2k
TwilioのCameraSource に任意のフレームを使用するすゝめ
potatotips 72
BOB
December 22, 2020
Tweet
Share
More Decks by BOB
See All by BOB
既存サブスクリプション商品にアップグレード・ダウングレード商品を追加しよう
aratayokoyama
1
670
バーチャル背景を実現しよう
aratayokoyama
0
390
アクセシブルなチャートを実現しよう
aratayokoyama
0
1k
VirtualBitriseUserGroupMeetup_fresh_bob
aratayokoyama
0
84
Other Decks in Programming
See All in Programming
Hasura の Relationship と権限管理
karszawa
0
180
ipa-medit: Memory search and patch tool for IPA without Jailbreaking/ipa-medit-bh2022-europe
tkmru
0
130
NGK2023S - OCaml最高! スマホ開発にも使えちゃう?!
haochenxie
0
120
Form実装基本を学び直してみた
hyugatsukui
0
250
LIFFで動く割り勘アプリTATEKAをリリースしてみた話
inoue2002
0
270
CDKでValidationする本当の方法 / cdk-validation
gotok365
1
230
10年以上続くプロダクトの フロントエンド刷新プロジェクトのふりかえり
yotahada3
2
360
Git Rebase
bkuhlmann
10
1.2k
Azure Functionsをサクッと開発、サクッとデプロイ/vscodeconf2023-baba
nina01
1
350
42tokyo-born2beroot-review
love42
0
120
OSSから学んだPR Descriptionの書き方
fugakkbn
4
140
The State of Kotlin | FOSDEM 2023
prof18
1
110
Featured
See All Featured
Keith and Marios Guide to Fast Websites
keithpitt
407
21k
Streamline your AJAX requests with AmplifyJS and jQuery
dougneiner
128
8.8k
The Invisible Side of Design
smashingmag
292
48k
Code Review Best Practice
trishagee
50
11k
5 minutes of I Can Smell Your CMS
philhawksworth
198
18k
Three Pipe Problems
jasonvnalue
89
8.9k
Web development in the modern age
philhawksworth
197
9.6k
How GitHub (no longer) Works
holman
298
140k
Making Projects Easy
brettharned
102
4.8k
Designing for Performance
lara
600
65k
The World Runs on Bad Software
bkeepers
PRO
59
5.7k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
152
13k
Transcript
ʲJ04ʳ5XJMJPͷ$BNFSB4PVSDF ʹҙͷϑϨʔϜΛ༻͢Δ͢ʍΊ "3"5":0,0:"#0#!QPUBUPUJQTJ04"OESPJE։ൃ5JQTڞ༗ձ
!#0#(7J "SBUB:PLPZBNB#0# # ळݩ߁ؔ࿈ʹ10Ҏ্අ͢ # ւಓੜ·Ε # ͩͯ͜ະདྷେଔ #
2020αΠόʔΤʔδΣϯτ৽ଔೖࣾ # λοϓϧͷiOSΤϯδχΞ
λοϓϧ w ͚ͷϚονϯάΞϓϦ w ݄ϦϦʔε w ϑϦοΫ6* w ຊਓೝূɼը໘ըࢭ
w 6*͕உঁͰେ͖͘ҧ͏ w 5XJMJPΛ༻ͨ͠ϏσΦνϟοτ
λοϓϧɿϏσΦνϟοτ w ݄ϏσΦνϟοτϦϦʔε w 5XJMJPΛ༻͓ͯ͠Γ·͢ w ݄ϑΟϧλʔػೳϦϦʔε w ʮ͓෦ͷย͚͕େมʯɼ
ʮͩ͠ͳΈΛ͑Δͷʹ͕͔͔࣌ؒΔʯ ͱ͍ͬͨΛड͚ͯ։ൃ͍ͨ͠·ͨ͠ w ϑΟϧλʔػೳ"3,JU༻͓ͯ͠Γ·͢ ϚονϯάΞϓϦʮλοϓϧੜʯɺ͙͢ʹΦϯϥΠϯσʔτ͕Ͱ͖Δ ʮϏσΦνϟοτʯʹ৽ͨʹʮϑΟϧλʔػೳʯΛՃɿhttps://www.cyberagent.co.jp/news/detail/id=25083 5XJMJPհͯ͠ҙͷϑϨʔϜΛૹΔํ๏Λ͝հ
(0"- ΞδΣϯμ CameraSourceʹҙͷϑϨʔϜΛ༻͢ΔϙΠϯτ˓˓ $BNFSB4PVSDFૹΔʹ˓˓͕େ "7$BQUVSFɿϑϨʔϜͷऔΓํ "3,JUɿϑϨʔϜͷऔΓํ
$BNFSB4PVSDFʹૹΔʹ˓˓͕େ
$BNFSB4PVSDFʹૹΔʹ˓˓͕େ // TVIVideoFrame - (nullable instancetype)initWithTimestamp:(CMTime)timestamp buffer:(nonnull CVImageBufferRef)imageBuffer orientation:(TVIVideoOrientation)orientation;
- (nullable instancetype)initWithTimeInterval:(CFTimeInterval)timeInterval buffer:(nonnull CVImageBufferRef)imageBuffer orientation:(TVIVideoOrientation)orientation; TVIVideoFrame Class Referenceɿhttps://twilio.github.io/twilio-video-ios/docs/latest/Classes/TVIVideoFrame.html#//api/name/initWithTimeInterval:buffer:orientation:
$BNFSB4PVSDFʹૹΔʹ˓˓͕େ // TVIVideoFrame - (nullable instancetype)initWithTimestamp:(CMTime)timestamp buffer:(nonnull CVImageBufferRef)imageBuffer orientation:(TVIVideoOrientation)orientation;
- (nullable instancetype)initWithTimeInterval:(CFTimeInterval)timeInterval buffer:(nonnull CVImageBufferRef)imageBuffer orientation:(TVIVideoOrientation)orientation; CVImageBufferRef CVImageBufferRef TVIVideoFrame Class Referenceɿhttps://twilio.github.io/twilio-video-ios/docs/latest/Classes/TVIVideoFrame.html#//api/name/initWithTimeInterval:buffer:orientation:
$BNFSB4PVSDFʹૹΔʹ˓˓͕େ // TVIVideoFrame - (nullable instancetype)initWithTimestamp:(CMTime)timestamp buffer:(nonnull CVImageBufferRef)imageBuffer orientation:(TVIVideoOrientation)orientation;
- (nullable instancetype)initWithTimeInterval:(CFTimeInterval)timeInterval buffer:(nonnull CVImageBufferRef)imageBuffer orientation:(TVIVideoOrientation)orientation; CMTime CFTimeInterval ϑϨʔϜΛऔಘ͢Δ Ճ͢Δ 7JEFP'SBNFΛ ࡞͢Δ ྲྀ͠ࠐΉ " # TVIVideoFrame Class Referenceɿhttps://twilio.github.io/twilio-video-ios/docs/latest/Classes/TVIVideoFrame.html#//api/name/initWithTimeInterval:buffer:orientation: $ %
$BNFSB4PVSDFʹૹΔʹ4JOL͕େ import TwilioVideo final class LocalTrackSourceManager: NSObject { private(set)
var camera: CameraSource! func onVideoFrame(image: CIImage, time: TimeInterval) { guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let videoFrame = VideoFrame(timeInterval: time, buffer: pixelBuffer, orientation: VideoOrientation.up) else { return } camera.sink?.onVideoFrame(videoFrame) } func onVideoFrame(image: CIImage, time: CMTime) { guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let videoFrame = VideoFrame(timestamp: time, buffer: pixelBuffer, orientation: VideoOrientation.up) else { return } camera.sink?.onVideoFrame(videoFrame) } } # $ %
import TwilioVideo final class LocalTrackSourceManager: NSObject { private(set) var camera:
CameraSource! func onVideoFrame(image: CIImage, time: TimeInterval) { guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let videoFrame = VideoFrame(timeInterval: time, buffer: pixelBuffer, orientation: VideoOrientation.up) else { return } camera.sink?.onVideoFrame(videoFrame) } func onVideoFrame(image: CIImage, time: CMTime) { guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let videoFrame = VideoFrame(timestamp: time, buffer: pixelBuffer, orientation: VideoOrientation.up) else { return } camera.sink?.onVideoFrame(videoFrame) } } $BNFSB4PVSDFʹૹΔʹ4JOL͕େ # $ % guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let pixelBuffer = image.convertCVPixelBuffer() else { return }
import TwilioVideo final class LocalTrackSourceManager: NSObject { private(set) var camera:
CameraSource! func onVideoFrame(image: CIImage, time: TimeInterval) { guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let videoFrame = VideoFrame(timeInterval: time, buffer: pixelBuffer, orientation: VideoOrientation.up) else { return } camera.sink?.onVideoFrame(videoFrame) } func onVideoFrame(image: CIImage, time: CMTime) { guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let videoFrame = VideoFrame(timestamp: time, buffer: pixelBuffer, orientation: VideoOrientation.up) else { return } camera.sink?.onVideoFrame(videoFrame) } } $BNFSB4PVSDFʹૹΔʹ4JOL͕େ # $ % VideoFrame(timeInterval: time, buffer: pixelBuffer, orientation: VideoOrientation.up) VideoFrame(timestamp: time, buffer: pixelBuffer, orientation: VideoOrientation.up)
import TwilioVideo final class LocalTrackSourceManager: NSObject { private(set) var camera:
CameraSource! func onVideoFrame(image: CIImage, time: TimeInterval) { guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let videoFrame = VideoFrame(timeInterval: time, buffer: pixelBuffer, orientation: VideoOrientation.up) else { return } camera.sink?.onVideoFrame(videoFrame) } func onVideoFrame(image: CIImage, time: CMTime) { guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let videoFrame = VideoFrame(timestamp: time, buffer: pixelBuffer, orientation: VideoOrientation.up) else { return } camera.sink?.onVideoFrame(videoFrame) } } $BNFSB4PVSDFʹૹΔʹ4JOL͕େ camera.sink?.onVideoFrame(videoFrame) camera.sink?.onVideoFrame(videoFrame)
$BNFSB4PVSDFʹૹΔʹ4JOL͕େ import TwilioVideo final class LocalTrackSourceManager: NSObject { private(set)
var camera: CameraSource! func onVideoFrame(image: CIImage, time: TimeInterval) { guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let videoFrame = VideoFrame(timeInterval: time, buffer: pixelBuffer, orientation: VideoOrientation.up) else { return } camera.sink?.onVideoFrame(videoFrame) } func onVideoFrame(image: CIImage, time: CMTime) { guard let pixelBuffer = image.convertCVPixelBuffer() else { return } guard let videoFrame = VideoFrame(timestamp: time, buffer: pixelBuffer, orientation: VideoOrientation.up) else { return } camera.sink?.onVideoFrame(videoFrame) } } camera.sink?.onVideoFrame(videoFrame) camera.sink?.onVideoFrame(videoFrame) extension VideoChatAVCaptureView: VideoSource { weak var sink: VideoSink? var isScreencast: Bool { return false } func requestOutputFormat(_ outputFormat: VideoFormat) { guard let sink = sink else { return } sink.onVideoFormatRequest(outputFormat) } } w 7JEFP4PVSDFʹ४ڌͤͯ͞TJOLΛ࣋ͭ͜ͱՄೳ w ͨͩ͠ɼৗʹ4JOLϑϨʔϜΛૹΔ͜ͱ͕݅ ʢΫϥογϡ͠·͢ʣ w Ұํɼ$BNFSB4PVSDFͷ4JOLʹͦͷ͕݅ͳ͍ 5JQT var camera: CameraSource!
"7$BQUVSFɿϑϨʔϜͷऔΓํ
"7$BQUVSFɿϑϨʔϜͷऔΓํ import AVFoundation import UIKit final class VideoChatAVCaptureView: UIView
{ private var session: AVCaptureSession? private func prepare() { let output: AVCaptureVideoDataOutput = AVCaptureVideoDataOutput() output.setSampleBufferDelegate(self, queue: DispatchQueue(label: "videoChat")) } } extension VideoChatAVCaptureView: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { guard let ciImage = sampleBuffer.convertCIImage() else { return } let timestamp = CMSampleBufferGetPresentationTimeStamp(sampleBuffer) localTrackSourceManager.onVideoFrame(image: ciImage, time: timestamp) } } " #
"7$BQUVSFɿϑϨʔϜͷऔΓํ import AVFoundation import UIKit final class VideoChatAVCaptureView: UIView
{ private var session: AVCaptureSession? private func prepare() { let output: AVCaptureVideoDataOutput = AVCaptureVideoDataOutput() output.setSampleBufferDelegate(self, queue: DispatchQueue(label: "videoChat")) } } extension VideoChatAVCaptureView: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { guard let ciImage = sampleBuffer.convertCIImage() else { return } let timestamp = CMSampleBufferGetPresentationTimeStamp(sampleBuffer) localTrackSourceManager.onVideoFrame(image: ciImage, time: timestamp) } } output.setSampleBufferDelegate(self, queue: DispatchQueue(label: "videoChat")) EFMFHBUFΛࣗʹ͋ͯ·͢
"7$BQUVSFɿϑϨʔϜͷऔΓํ import AVFoundation import UIKit final class VideoChatAVCaptureView: UIView
{ private var session: AVCaptureSession? private func prepare() { let output: AVCaptureVideoDataOutput = AVCaptureVideoDataOutput() output.setSampleBufferDelegate(self, queue: DispatchQueue(label: "videoChat")) } } extension VideoChatAVCaptureView: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { guard let ciImage = sampleBuffer.convertCIImage() else { return } let timestamp = CMSampleBufferGetPresentationTimeStamp(sampleBuffer) localTrackSourceManager.onVideoFrame(image: ciImage, time: timestamp) } } guard let ciImage = sampleBuffer.convertCIImage() else { return } $.4BNQMF#V⒎FSΛ$**NBHFʹม͠·͢
"7$BQUVSFɿϑϨʔϜͷऔΓํ import AVFoundation import UIKit final class VideoChatAVCaptureView: UIView
{ private var session: AVCaptureSession? private func prepare() { let output: AVCaptureVideoDataOutput = AVCaptureVideoDataOutput() output.setSampleBufferDelegate(self, queue: DispatchQueue(label: "videoChat")) } } extension VideoChatAVCaptureView: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { guard let ciImage = sampleBuffer.convertCIImage() else { return } let timestamp = CMSampleBufferGetPresentationTimeStamp(sampleBuffer) localTrackSourceManager.onVideoFrame(image: ciImage, time: timestamp) } } let timestamp = CMSampleBufferGetPresentationTimeStamp(sampleBuffer) $.5JNFΛ࡞͠·͢
"3,JUɿϑϨʔϜͷऔΓํ
"3,JUɿϑϨʔϜͷऔΓํ import ARKit import SceneKit import UIKit final class
VideoChatARKitView: UIView { private lazy var sceneView: ARSCNView = {...}() private func prepare() { sceneView.delegate = self } } extension VideoChatARKitView: ARSCNViewDelegate { func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) { guard let sceneView = renderer as? ARSCNView else { return } guard let ciImage = CIImage(image: sceneView.snapshot()) else { return } localTrackSourceManager.onVideoFrame(image: ciImage, time: time) } } " #
import ARKit import SceneKit import UIKit final class VideoChatARKitView: UIView
{ private lazy var sceneView: ARSCNView = {...}() private func prepare() { sceneView.delegate = self } } extension VideoChatARKitView: ARSCNViewDelegate { func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) { guard let sceneView = renderer as? ARSCNView else { return } guard let ciImage = CIImage(image: sceneView.snapshot()) else { return } localTrackSourceManager.onVideoFrame(image: ciImage, time: time) } } "3,JUɿϑϨʔϜͷऔΓํ sceneView.delegate = self EFMFHBUFΛࣗʹ͋ͯ·͢
"3,JUɿϑϨʔϜͷऔΓํ import ARKit import SceneKit import UIKit final class
VideoChatARKitView: UIView { private lazy var sceneView: ARSCNView = {...}() private func prepare() { sceneView.delegate = self } } extension VideoChatARKitView: ARSCNViewDelegate { func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) { guard let sceneView = renderer as? ARSCNView else { return } guard let ciImage = CIImage(image: sceneView.snapshot()) else { return } localTrackSourceManager.onVideoFrame(image: ciImage, time: time) } } let ciImage = CIImage(image: sceneView.snapshot()) εφοϓγϣοτΛऔಘ͠·͢
"3,JUɿϑϨʔϜͷऔΓํ import ARKit import SceneKit import UIKit final class
VideoChatARKitView: UIView { private lazy var sceneView: ARSCNView = {...}() private func prepare() { sceneView.delegate = self } } extension VideoChatARKitView: ARSCNViewDelegate { func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) { guard let sceneView = renderer as? ARSCNView else { return } guard let ciImage = CIImage(image: sceneView.snapshot()) else { return } localTrackSourceManager.onVideoFrame(image: ciImage, time: time) } } time: TimeInterval ͦͷ··͢͜ͱ͕Ͱ͖·͢
·ͱΊ
·ͱΊ (0"- εςοϓ CameraSourceʹҙͷϑϨʔϜΛ༻͢ΔϙΠϯτSink ϑϨʔϜΛऔಘ͢Δ Ճ͢Δ 7JEFP'SBNFΛ ࡞͢Δ ྲྀ͠ࠐΉ
" # $ %