Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
AR without SceneKit
Search
naru-jpn
February 07, 2018
Programming
0
400
AR without SceneKit
Create AR application using Metal without SceneKit
naru-jpn
February 07, 2018
Tweet
Share
More Decks by naru-jpn
See All by naru-jpn
配信アプリのためのリアルタイムプッシュ通知ぼかしの夢
narujpn
3
820
PiPを応用した配信コメントバー機能の開発秘話と技術の詳解 / pip_streaming_comment_bar
narujpn
3
3.8k
Updating an App to Use Swift Concurrency 解説
narujpn
2
280
PiP で実現するミラティブの配信コメントバー / pip-streaming-comment-bar
narujpn
0
1k
App Extension のスタックトレース情報からクラッシュを解析/集計する / Analyzing app extension's stack trace
narujpn
3
1.4k
ミラティブとWebRTC - WebRTC framework の中身を覗いてみよう / WebRTC framework AudioUnit Processing
narujpn
1
2.1k
CoreML3のオンデバイストレーニングでつくる母音推定
narujpn
0
400
AltConfと周辺の歩き方
narujpn
0
1.9k
エンジニア経験を活かしたスクラムマスターとして 開発チームとプロダクトを成長させる
narujpn
1
380
Other Decks in Programming
See All in Programming
生成AIでGitHubソースコード取得して仕様書を作成
shukob
0
630
Rubyでつくるパケットキャプチャツール
ydah
0
170
「とりあえず動く」コードはよい、「読みやすい」コードはもっとよい / Code that 'just works' is good, but code that is 'readable' is even better.
mkmk884
6
1.4k
20年もののレガシープロダクトに 0からPHPStanを入れるまで / phpcon2024
hirobe1999
0
1k
선언형 UI에서의 상태관리
l2hyunwoo
0
270
はてなにおけるfujiwara-wareの活用やecspressoのCI/CD構成 / Fujiwara Tech Conference 2025
cohalz
3
2.8k
AHC041解説
terryu16
0
400
chibiccをCILに移植した結果 (NGK2025S版)
kekyo
PRO
0
130
2025.01.17_Sansan × DMM.swift
riofujimon
2
560
Scaling your build logic
antalmonori
1
100
令和7年版 あなたが使ってよいフロントエンド機能とは
mugi_uno
10
5.2k
ecspresso, ecschedule, lambroll を PipeCDプラグインとして動かしてみた (プロトタイプ) / Running ecspresso, ecschedule, and lambroll as PipeCD Plugins (prototype)
tkikuc
2
1.9k
Featured
See All Featured
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
jQuery: Nuts, Bolts and Bling
dougneiner
62
7.6k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
49
2.2k
Fontdeck: Realign not Redesign
paulrobertlloyd
82
5.3k
Producing Creativity
orderedlist
PRO
343
39k
Music & Morning Musume
bryan
46
6.3k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
132
33k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
3
360
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
232
17k
Unsuck your backbone
ammeep
669
57k
Writing Fast Ruby
sferik
628
61k
Product Roadmaps are Hard
iamctodd
PRO
50
11k
Transcript
AR without SceneKit #mobilearmeetup @Timers 2018/2/7
Relation of ARKit and SceneKit
ARKit Camera SceneKit Pixel Buffer 3D Info Rendering ARSCNView
ARKit Camera SceneKit Pixel Buffer 3D Info Rendering ARSCNView Light
Estimation Plane Detection Manage 3D Objects
Remove SceneKit and Use Metal
ARKit Camera SceneKit Pixel Buffer 3D Info Rendering ARSCNView Light
Estimation Plane Detection Manage 3D Objects
ARKit Camera Pixel Buffer 3D Info Light Estimation Plane Detection
Metal ARKit Camera Pixel Buffer 3D Info Light Estimation Plane
Detection Rendering
Metal ARKit Camera Pixel Buffer 3D Info Rendering Light Estimation
Plane Detection Take Photo
MetalAR https://github.com/naru-jpn/MetalAR
None
Metal ARKit Camera Pixel Buffer 3D Info Light Estimation Plane
Detection Rendering Take Photo
// Create ARSession session = ARSession() // ARSessionDelegate session.delegate =
self
Metal ARKit Camera Pixel Buffer 3D Info Light Estimation Plane
Detection Rendering Take Photo
// Defined in ARSessionDelegate // Called when ARFrame is updated
func session(_ session: ARSession, didUpdate frame: ARFrame) { // … }
A running AR session continuously captures video frames from the
device camera. For each frame, ARKit analyzes the image together with data from the device's motion sensing hardware to estimate the device's real-world position. ARFrame https://developer.apple.com/documentation/arkit/arframe
Metal ARKit Camera Pixel Buffer 3D Info Light Estimation Plane
Detection Rendering Take Photo
Pipeline of off-screen rendering Draw video frame Filter Draw Instances
ARFrame MTLTexture
Pipeline of off-screen rendering Filter Draw Instances ARFrame MTLTexture Draw
video frame Commands sent to GPU
Pipeline of off-screen rendering Filter Draw Instances ARFrame MTLTexture MPSImageSobel
declared in Metal Performance Shaders Draw video frame Commands sent to GPU
Performance on iPhone7
Filter Draw Instances ARFrame MTLTexture Draw video frame 3.3 ms
Rendering
Making Image to save drawHierarchy(in:afterScreenUpdates:) MTLTexture → CGImage 7.0 ~
10.0 ms available on background thread 30.0 ~ 40.0 ms required to execute on main thread
Challenges
Drawing original 3D objects User Interaction (available hitTest(_:types:) for ARFrame)
Physics engine Skeletal Animation …
References
Documents Metal Best Practices Guide https://developer.apple.com/library/content/documentation/ 3DDrawing/Conceptual/MTLBestPracticesGuide/index.html#// apple_ref/doc/uid/TP40016642-CH27-SW1 Metal Shading
Language Specification https://developer.apple.com/metal/Metal-Shading-Language- Specification.pdf
AR Template Project