ARKit @swift Conference

Transcript

1. 做一个有趣的多人 AR游戏 刘冠杉 @guanshanliu
2. 做一个有趣的多人 AR游戏 的过程很有趣 刘冠杉 @guanshanliu
3. 刘冠杉 Guanshan Liu iOS Developer @ booking.com https://guanshanliu.me @guanshanliu 老公 父亲 游戏 动漫 46 48
4. 今日课题 • 很多ARKit • 一点点MultipeerConnectivity • 部分Logging • 一丢丢 app开发经验应用到游戏 Clean & Swift-y
5. SceneKit Cheatsheet
6. 新建一个Plane SCNNode let geometry = SCNPlane(width: 0.5, height: 0.5) geometry.materials.first?.diffuse.contents = UIColor.darkGray let planeNode = SCNNode(geometry: geometry) planeNode.transform = SCNMatrix4MakeRotation(-Float.pi / 2.0, 1, 0, 0) sceneView.scene.rootNode.addChildNode(planeNode)
7. 读入Scene/SCNNode let scene = SCNScene(named: "models.scnassets/duck.scn")! let duckNode = scene.rootNode
8. ARKit Plane Detection Sharing & Persistence Environment Texturing Image Detection 3D Object Detection 3D Object Scan Image Tracking Hit Testing Face Position & Orientation Tracking Facial Expressions Tracking +Metal +Vision +CoreML +SceneKit +SpriteKit Light Estimation
9. 看别人的游戏和Apps • 内容/功能简短 • 操作简单直观 • 设备不要剧烈晃动，不要快速移动
10. 看别人的游戏和Apps • 内容/功能简短 • 操作简单直观 • 设备不要剧烈晃动，不要快速移动 • AR Off Mode
11. 平面识别 Plane Detection
12. 配置AR Session let config = ARWorldTrackingConfiguration() if #available(iOS 11.3, *) { config.planeDetection = [.horizontal, .vertical] } else { config.planeDetection = .horizontal } sceneView.session.run(config)
13. ARSCNViewDelegate • 添加 func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) • 更新 func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) • 移除 func renderer(_ renderer: SCNSceneRenderer, didRemove node: SCNNode, for anchor: ARAnchor)
14. @interface ARAnchor : NSObject <ARAnchorCopying, NSSecureCoding> /** Unique identifier of the anchor. */ @property (nonatomic, readonly) NSUUID *identifier; /** An optional name used to associate with the anchor. */ @property (nonatomic, nullable, readonly) NSString *name API_AVAILABLE(ios(12.0)); /** The transformation matrix that defines the anchor’s rotation, translation and scale in world coordinates. */ @property (nonatomic, readonly) simd_float4x4 transform; @end
15. @interface ARPlaneAnchor : ARAnchor @property (nonatomic, readonly) ARPlaneAnchorAlignment alignment; @property (nonatomic, readonly) simd_float3 center; @property (nonatomic, readonly) simd_float3 extent; @property (nonatomic, strong, readonly) ARPlaneGeometry *geometry API_AVAILABLE(ios(11.3)); @end
16. 显示平面 func renderer( _ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) { guard let planeAnchor = anchor as? ARPlaneAnchor else { return } let planeNode = self.createPlaneNode(planeAnchor: planeAnchor) DispatchQueue.main.async { node.addChildNode(planeNode) } }
17. 显示平面 func createPlaneNode(for planeAnchor: ARPlaneAnchor, color: UIColor = UIColor.yellow.withAlphaComponent(0.5)) -> SCNNode { let planeGeometry = SCNPlane(width: CGFloat(planeAnchor.extent.x), height: CGFloat(planeAnchor.extent.z)) let planeMaterial = SCNMaterial() planeMaterial.diffuse.contents = color planeGeometry.materials = [planeMaterial] let planeNode = SCNNode(geometry: planeGeometry) planeNode.position = SCNVector3Make(planeAnchor.center.x, 0, planeAnchor.center.z) planeNode.eulerAngles.x = -.pi / 2 return planeNode }
18. 更新平面 func renderer( _ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) { guard let planeAnchor = anchor as? ARPlaneAnchor else { return } DispatchQueue.main.async { self.updatePlaneNode( planeNode: node.childNodes[0], planeAchor: planeAnchor) } }
19. 更新平面 func updatePlaneNode( planeNode: SCNNode, planeAnchor: ARPlaneAnchor) { let planeGeometry = planeNode.geometry as! SCNPlane planeGeometry.width = CGFloat(planeAnchor.extent.x) planeGeometry.height = CGFloat(planeAnchor.extent.z) planeNode.position = SCNVector3Make(planeAnchor.center.x, 0, planeAnchor.center.z) }
20. @interface ARPlaneAnchor : ARAnchor @property (nonatomic, readonly) ARPlaneAnchorAlignment alignment; @property (nonatomic, readonly) simd_float3 center; @property (nonatomic, readonly) simd_float3 extent; @property (nonatomic, strong, readonly) ARPlaneGeometry *geometry API_AVAILABLE(ios(11.3)); @end
21. 不规则平面识别 let device = MTLCreateSystemDefaultDevice()! func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) { guard let planeAnchor = anchor as? ARPlaneAnchor else { fatalError() } let planeGeometry = ARSCNPlaneGeometry(device: device)! planeGeometry.update(from: planeAnchor.geometry) addPlaneNode(on: node, geometry: planeGeometry) }
22. 不规则平面识别 func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) { guard let planeAnchor = anchor as? ARPlaneAnchor else { fatalError() } if let planeGeometry = findShapedPlaneNode(on: node)?.geometry as? ARSCNPlaneGeometry { planeGeometry.update(from: planeAnchor.geometry) } }
24. 和虚拟物体交互操作 Hit Test
25. Ray Cast
26. 选中平面 let results = sceneView.hitTest(position, types: .existingPlaneUsingExtent) guard let anchor = results.first?.anchor else { return nil } guard let node = sceneView.node(for: anchor) else { return nil } for child in node.childNodes { guard let node = child.geometry as? SCNPlane else { continue } // We have found the node... }
27. struct ResultType : OptionSet { /** Result type from intersecting the nearest feature point. */ static var featurePoint: ResultType { get } /** Result type from intersecting a horizontal plane estimate, determined for the current frame. */ static var estimatedHorizontalPlane: ResultType { get } /** Result type from intersecting a vertical plane estimate, determined for the current frame. */ static var estimatedVerticalPlane: ResultType { get } /** Result type from intersecting with an existing plane anchor. */ static var existingPlane: ResultType { get } /** Result type from intersecting with an existing plane anchor, taking into account the plane’s extent. */ static var existingPlaneUsingExtent: ResultType { get } /** Result type from intersecting with an existing plane anchor, taking into account the plane’s geometry. */ static var existingPlaneUsingGeometry: ResultType { get } }
29. 图片跟踪 Image Tracking
30. 加图片到AR Resource Group • 创建 Create an AR Resource Group in Assets.xcassets • 拖进去 Drag images to be detected into the group • 设置大小 Set physical dimension for each image
31. 加图片到AR Resource Group
32. 配置AR Session • ARWorldTrackingConfiguration • ARImageTrackingConfiguration
33. 配置AR Session // Load images from assets guard let imageSet = ARReferenceImage.referenceImages(inGroupNamed: "Easter Eggs", bundle: Bundle.main) else { os_log(.error, "no reference images") return } // Create a configuration let config = ARWorldTrackingConfiguration() config.detectionImages = imageSet // Run the session sceneView.session.run(config)
34. 跟踪图片 func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) { if let imageAnchor = anchor as? ARImageAnchor { let referenceImage = imageAnchor.referenceImage DispatchQueue.main.async { switch referenceImage.name { case "avatar": os_log(.info, "found a avatar") node.addChildNode(textNode(text: "@guanshanliu")) default: () } } } }
36. 动态加载 func convertCIImageToCGImage(inputImage: CIImage) -> CGImage? { let context = CIContext(options: nil) if let cgImage = context.createCGImage(inputImage, from: inputImage.extent) { return cgImage } else { return nil } }
37. 动态加载 func loadImageReferencesOnTheFly() { guard let image = UIImage(named: "avatar"), let ciImage = CIImage(image: image), let cgImage = convertCIImageToCGImage(inputImage: ciImage) else { return } referenceImages = Set([ ARReferenceImage(cgImage, orientation: .up, physicalWidth: 0.1) ]) }
38. 分享当前的AR场景？
39. AR World Map • 当前物理空间 • 当前所有anchors
40. 检查World Map状态 // 只在数据完整情况允许发送 func session(_ session: ARSession, didUpdate frame: ARFrame) { switch frame.worldMappingStatus { case .notAvailable, .limited: sendMapButton.isEnabled = false case .extending: sendMapButton.isEnabled = !session.connectedPeers.isEmpty case .mapped: sendMapButton.isEnabled = !session.connectedPeers.isEmpty } }
41. 保存 & 发送 sceneView.session.getCurrentWorldMap { worldMap, error in guard let map = worldMap else { return } guard let data = try? NSKeyedArchiver.archivedData(withRootObject: map, requiringSecureCoding: true) else { fatalError("can't encode map") } try? self.session.send( data, toPeers: self.session.connectedPeers, with: .reliable) }
42. 接收 & 使用 if let worldMap = try NSKeyedUnarchiver.unarchivedObject(ofClass: ARWorldMap.self, from: data) { let configuration = ARWorldTrackingConfiguration() configuration.initialWorldMap = worldMap sceneView.session.run( configuration, options: [.resetTracking, .removeExistingAnchors]) }
43. 买书
44. Demo
45. 网络 Multipeer Connectivity
46. Multipeer Connectivity • 支持附近设备间的通讯 Enables nearby devices to communicate over Wi-Fi, peer-to-peer Wi-Fi, and Bluetooth • 安全的数据传输 Securely transmit messages, streams, or file resources
47. 广播服务 // ⼴广播服务 serviceAdvertiser = MCNearbyServiceAdvertiser(peer: myPeerID, discoveryInfo: nil, serviceType: "iro") serviceAdvertiser.delegate = self serviceAdvertiser.startAdvertisingPeer()
48. 搜索附近的服务 serviceBrowser = MCNearbyServiceBrowser(peer: myPeerID, serviceType: "iro") serviceBrowser.delegate = self serviceBrowser.startBrowsingForPeers()
49. 发送 收信 try session.send(data, toPeers: session.connectedPeers, with: .reliable) func session(_ session: MCSession, didReceive data: Data, fromPeer peerID: MCPeerID) { // . . . }
50. 自定加入第一个找到的session MCNearbyServiceBrowser 找到另⼀一个设备 调⽤用 browser(_:foundPeer:withDisco veryInfo:) delegate⽅方法 邀请另⼀一个设备 invitePeer(_:to:withContext:time out:) method: MCNearbyServiceAdvertiser 调⽤用advertiser(_: didReceiveInvitationFromPe er:withContext:invitationHan dler: ) 另⼀一台设备收到邀请 接受邀请invitationHandler(true, self.session)
51. 主客形式 • Host用MCNearbyServiceAdvertiser广播，guests用 MCNearbyServiceBrowser寻找并加入 • Peers形式加入，然后协商出一个作为master
52. Inside SwiftShot: Creating an AR Game WWDC 2018 Session 605
53. BitStreamCodable • Bit-packed encoding/decoding of values • 传输数据小，快速serialization和deserialization
54. 调试 Logging
55. import os.log extension OSLog { static let ar = OSLog(subsystem: "com.guanshanliu.iro", category: "AR") } let conference = "@Swift Conference" os_log(.info, log: .ar, "%s is awesome", conference)
56. 级别 Log Levels 使⽤用意图 Usage 持久化 Persistence Default things that might result a failure • initially in memory buffers • compressed, moved to the data store as buffer fill Info troubleshooting errors • initially in memory buffers • purged as buffers fill • captured in the data store when faults,, optionally, errors occur Debug use in a development environment only captured in memory when debug logging is enabled Error process-level errors always saved in the data store Fault system-level or multi-process errors only always saved in the data store
57. import os.signpost extension OSSignpostID { static let render_loop = OSSignpostID(log: .render_loop) } func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) { os_signpost(.begin, log: .render_loop, name: "GameLogicUpdate", signpostID: .render_loop, "Game logic update started") DispatchQueue.main.async { self.updates.forEach({ $0(self.gameState) }) } os_signpost(.end, log: .render_loop, name: "GameLogicUpdate", signpostID: .render_loop, "Game logic update finished") } 诊断性能问题 Diagnose Performance Issues
58. Activity Tracing func findPlaneGeometryNode(on node: SCNNode) -> SCNNode? { for childNode in node.childNodes { if childNode.geometry as? ARSCNPlaneGeometry != nil { os_log(.debug, log: .ar, "found a node with ARSCNPlaneGeometry") return childNode } } os_log(.debug, log: .ar, "no node with ARSCNPlaneGeometry") return nil } func updatePlaneGeometryNode(on node: SCNNode) { let activity = Activity("update ARSCNPlaneGeometry") activity.active { DispatchQueue.main.async(execute: { guard let planeGeometry = self.findPlaneGeometryNode(on: node)?.geometry as? ARSCNPlaneGeometry else { return } planeGeometry.update(from: self.geometry) }) } }
59. 做一个好码农 Clean Swift-y Code
60. 单一 Separation of concerns
61. • AREngine: 所以AR相关的功能 • MultipeerSession: 广播，发现，加入玩家，发送，接收数 据 • Game: 游戏逻辑 • Dispatcher: 消息分发 • HapticsGenerator: Haptics反馈
62. 保证View Controller整洁 • View Controller属于View layer • UI Layout布局，UI内容显示，用户交互
63. 保证View Controller整洁 • View Controller属于View layer • UI Layout布局，UI内容显示，用户交互 override func viewDidLoad() { super.viewDidLoad() // … engine.attach(to: sceneView) engine.start() }
64. 数据流动 Data Flow
65. User Interface AR Network Dispatcher Game 交互事件 更更新UI AR事件 更更新session 发送数据 ⽹网络事件 更更新gameplay 变化反馈
66. 单向 Unidirectional
67. ARKit • 平面识别 Plane Detection • 和虚拟物体交互操作 Interaction • World Map • 图片跟踪 Image Tracking 回顾
68. MultipeerConnectivity • MCNearbyServiceBrowser • MCNearbyServiceAdvertiser • 发送数据 • 接收数据 回顾
69. App开发也可以用 • os_log • os_signpost • Activity.swift • Separation of concerns • 学习app开发的架构 回顾
70. 参考资料 • Inside SwiftShot: Creating an AR Game • ARKit Sampler | Shuichi Tsutsumi • App Architecture | objc.io • Unified Logging and Activity Tracing | NSScreencast • Activity.swift | Zach Waldowski
71. – @guanshanliu “谢谢”