Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
AR without SceneKit
Search
naru-jpn
February 07, 2018
Programming
0
420
AR without SceneKit
Create AR application using Metal without SceneKit
naru-jpn
February 07, 2018
Tweet
Share
More Decks by naru-jpn
See All by naru-jpn
配信アプリのためのリアルタイムプッシュ通知ぼかしの夢
narujpn
3
900
PiPを応用した配信コメントバー機能の開発秘話と技術の詳解 / pip_streaming_comment_bar
narujpn
3
4.2k
Updating an App to Use Swift Concurrency 解説
narujpn
2
340
PiP で実現するミラティブの配信コメントバー / pip-streaming-comment-bar
narujpn
0
1.2k
App Extension のスタックトレース情報からクラッシュを解析/集計する / Analyzing app extension's stack trace
narujpn
3
1.5k
ミラティブとWebRTC - WebRTC framework の中身を覗いてみよう / WebRTC framework AudioUnit Processing
narujpn
1
2.2k
CoreML3のオンデバイストレーニングでつくる母音推定
narujpn
0
440
AltConfと周辺の歩き方
narujpn
0
2k
エンジニア経験を活かしたスクラムマスターとして 開発チームとプロダクトを成長させる
narujpn
1
410
Other Decks in Programming
See All in Programming
テストから始めるAgentic Coding 〜Claude Codeと共に行うTDD〜 / Agentic Coding starts with testing
rkaga
12
4.5k
PHPで始める振る舞い駆動開発(Behaviour-Driven Development)
ohmori_yusuke
2
390
レベル1の開発生産性向上に取り組む − 日々の作業の効率化・自動化を通じた改善活動
kesoji
0
220
iOS 26にアップデートすると実機でのHot Reloadができない?
umigishiaoi
0
130
なんとなくわかった気になるブロックテーマ入門/contents.nagoya 2025 6.28
chiilog
1
270
「Cursor/Devin全社導入の理想と現実」のその後
saitoryc
0
820
ISUCON研修おかわり会 講義スライド
arfes0e2b3c
1
450
AI時代のソフトウェア開発を考える(2025/07版) / Agentic Software Engineering Findy 2025-07 Edition
twada
PRO
87
29k
ペアプロ × 生成AI 現場での実践と課題について / generative-ai-in-pair-programming
codmoninc
2
18k
プロダクト志向ってなんなんだろうね
righttouch
PRO
0
190
すべてのコンテキストを、 ユーザー価値に変える
applism118
3
1.3k
Result型で“失敗”を型にするPHPコードの書き方
kajitack
5
880
Featured
See All Featured
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
130
19k
Statistics for Hackers
jakevdp
799
220k
Intergalactic Javascript Robots from Outer Space
tanoku
271
27k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
161
15k
Git: the NoSQL Database
bkeepers
PRO
430
65k
[RailsConf 2023] Rails as a piece of cake
palkan
55
5.7k
Building Flexible Design Systems
yeseniaperezcruz
328
39k
Product Roadmaps are Hard
iamctodd
PRO
54
11k
Fantastic passwords and where to find them - at NoRuKo
philnash
51
3.3k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
50
5.5k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
2.4k
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
656
60k
Transcript
AR without SceneKit #mobilearmeetup @Timers 2018/2/7
Relation of ARKit and SceneKit
ARKit Camera SceneKit Pixel Buffer 3D Info Rendering ARSCNView
ARKit Camera SceneKit Pixel Buffer 3D Info Rendering ARSCNView Light
Estimation Plane Detection Manage 3D Objects
Remove SceneKit and Use Metal
ARKit Camera SceneKit Pixel Buffer 3D Info Rendering ARSCNView Light
Estimation Plane Detection Manage 3D Objects
ARKit Camera Pixel Buffer 3D Info Light Estimation Plane Detection
Metal ARKit Camera Pixel Buffer 3D Info Light Estimation Plane
Detection Rendering
Metal ARKit Camera Pixel Buffer 3D Info Rendering Light Estimation
Plane Detection Take Photo
MetalAR https://github.com/naru-jpn/MetalAR
None
Metal ARKit Camera Pixel Buffer 3D Info Light Estimation Plane
Detection Rendering Take Photo
// Create ARSession session = ARSession() // ARSessionDelegate session.delegate =
self
Metal ARKit Camera Pixel Buffer 3D Info Light Estimation Plane
Detection Rendering Take Photo
// Defined in ARSessionDelegate // Called when ARFrame is updated
func session(_ session: ARSession, didUpdate frame: ARFrame) { // … }
A running AR session continuously captures video frames from the
device camera. For each frame, ARKit analyzes the image together with data from the device's motion sensing hardware to estimate the device's real-world position. ARFrame https://developer.apple.com/documentation/arkit/arframe
Metal ARKit Camera Pixel Buffer 3D Info Light Estimation Plane
Detection Rendering Take Photo
Pipeline of off-screen rendering Draw video frame Filter Draw Instances
ARFrame MTLTexture
Pipeline of off-screen rendering Filter Draw Instances ARFrame MTLTexture Draw
video frame Commands sent to GPU
Pipeline of off-screen rendering Filter Draw Instances ARFrame MTLTexture MPSImageSobel
declared in Metal Performance Shaders Draw video frame Commands sent to GPU
Performance on iPhone7
Filter Draw Instances ARFrame MTLTexture Draw video frame 3.3 ms
Rendering
Making Image to save drawHierarchy(in:afterScreenUpdates:) MTLTexture → CGImage 7.0 ~
10.0 ms available on background thread 30.0 ~ 40.0 ms required to execute on main thread
Challenges
Drawing original 3D objects User Interaction (available hitTest(_:types:) for ARFrame)
Physics engine Skeletal Animation …
References
Documents Metal Best Practices Guide https://developer.apple.com/library/content/documentation/ 3DDrawing/Conceptual/MTLBestPracticesGuide/index.html#// apple_ref/doc/uid/TP40016642-CH27-SW1 Metal Shading
Language Specification https://developer.apple.com/metal/Metal-Shading-Language- Specification.pdf
AR Template Project