2024: A Spatial Odyssey, or How I Learned to Relax and Embrace visionOS

Transcript

1. @timmitra 2024
2. @timmitra Can you keep a secret? 请保密
4. WWDC 2024 visionOS beta SDK “Look! I made a Vision Pro app” - said everyone! @timmitra
5. @timmitra Should I go to the Vision Pro lab? “Go!” - my wife. I won the chance to go the Vision Pro Lab!
6. “I’m taking the skeptical point of view. I don't see a need for me with that. I may be wrong.” - Daniel Steinberg, MTJC Podcast @timmitra
7. “I’m taking the skeptical point of view. I don't see a need for me with that. I may be wrong.” - Daniel Steinberg, MTJC Podcast @timmitra
8. @timmitra
9. @timmitra I’m All in! 💯
10. @timmitra I made this!
11. @timmitra What is Spatial Computing? A brief history • 1838 - Stereoscope • 1929 - Link Trainer, fl ight simulator • 1939 - ViewMaster • 1952 - Sensorama, Morten Heilig • 1960 - Telesphere Mask, Morten Heilig • 1968 - Sword of Damocles, VR Headset • 1987 - “Virtual Reality” • 1990 - “Augmented Reality” • 1995 Nitendo Virtual Boy, 2012 Oculus VR, 2016 Pokemon Go, 2017 Google (ARCore?), Apple (ARKit)
12. @timmitra - Tim Mitra, November 2023 “The Vision Pro and visionOS are poised to create a paradigm shift in the idea of computing. Spatial computing involves interacting with elements rendered in front and around the computer user.”
13. @timmitra
14. @timmitra
15. @timmitra
16. @timmitra
17. @timmitra Why Should You Learn How to Create visionOS Apps? There will be businesses no one has thought of! • visionOS • is the operating system that runs on the VisionPro • written primarily in SwiftUI • can run apps built for UIKit • compatible with iPadOS • TL;DR - next slide
18. @timmitra First up, the iPhone These didn’t exist before the iPhone Twitter Instagram Douyin 抖 音 Pinterest Baidu 百度 Lyft/Uber WeChat 微信 Venmo Square SnapChat TicTok AngryBirds CandyCrush Pumpkin Spice Latte AliPay 支 付宝 Spotify
19. @timmitra visionOS is deep!
20. @timmitra The Three Faces of visionOS Three Pillars • Window • Majority of apps • Run the the Shared Space. • Volume • 1 cubic meter • Immersive • Change the Full Space • Dial back with the Digital Crown • PassThrough video
21. source: Apple
22. source: Apple
23. @timmitra Tools The players RealityKit SwiftUI (UIKit) ARKit/Metal Xcode 15 (Silicon) Reality Composer Pro Unity Pro
24. @timmitra var body: some Scene { WindowGroup(){ CourseListView() }.windowStyle(.plain) } Window Group in @app • .automatic, .plain, .volumetric
25. @timmitra import RealityKit import RealityKitContent @State private var currentStyle: ImmersionStyle = .full var body: some Scene { WindowGroup(){ CourseListView() }.windowStyle(.plain) ImmersiveSpace(id: "ImmersiveScene") { ImmersiveView() } .immersionStyle(selection: $currentStyle, in: .full) } Immersive App
26. @timmitra Ornaments • toolbars and tab bars automatically appear as ornaments • NavigationSplitView { } detail: { } .ornament(visibility: .visible, attachmentAnchor: .scene(.bottom)) { HStack(spacing: 16) { Button { addButtonAction(jobsCount: jobs.count, subscribed: subscribed) } label: { Text("Add Job") } } .padding(8) .glassBackgroundEffect()
27. @timmitra RealityView { content in // Add the initial RealityKit content if let scene = try? await Entity(named: "ImmersiveScene", in: realityKitContentBundle) { content.add(scene) } update: { // updates to the entities } attachments: { // attachments to the entities } RealityView
28. @timmitra @Environment(\.openImmersiveSpace) var openImmersiveScene @Environment(\.dismissImmersiveSpace) var dismissImmersiveScene @State private var isShowingImmersive = false Button("Open ImmersiveSpace") { Task { let result = await openImmersiveScene(id: "ImmersiveScene") isShowingImmersive = true; } } Button("Close") { Task { await dismissImmersiveScene() print("Dismissing Complete") isShowingImmersive = false } } Open Immersive Space
29. @timmitra RealityView / Model3D How to load Entities • Both async • Model3D is quick • RealityView • Robust • closures: make, update, accessory • loadEntity() • realityKitContentBundle or assets
30. @timmitra Reality Composer Pro • USDz • Partical Emitter • Materials • Shader Graph • Components • Systems
31. @timmitra
32. @timmitra Shader Graph @mrdeerwhale
33. @timmitra Mark Robinson - Gumroad
34. @timmitra RealityKit
35. @timmitra RealityKit build in code guard let resource = try? await EnvironmentResource(named: "Sunlight") else { return } let iblComponent = ImageBasedLightComponent(source: .single(resource), intensityExponent: 0.25) guard let resource = try? await TextureResource(named: "underwater") else { return } var material = UnlitMaterial() material.color = .init(texture: .init(resource)) let entity = Entity() entity.components.set(ModelComponent(mesh: .generateSphere(radius: 1000), materials: [material] )) // point the texture inwards entity.scale *= .init(x: -1, y: 1, z: 1) }
36. @timmitra Skybox Photo: Matt Galloway, RWDevCon 2015
37. @timmitra Skybox
38. @timmitra RealityKit - floor & collisions build in code /* Occluded floor */ let floor = ModelEntity(mesh: .generatePlane(width: 100, depth: 100), materials: [OcclusionMaterial()]) floor.generateCollisionShapes(recursive: false) floor.components[PhysicsBodyComponent.self] = .init( massProperties: .default, mode: .static ) content.add(floor) /* steel ball */ let ball = ModelEntity( mesh: .generateSphere(radius: 0.1), materials: [SimpleMaterial(color: .white, isMetallic: true)]) ball.generateCollisionShapes(recursive: false) // Enable interactions on the entity. ball.components.set(InputTargetComponent()) ball.components.set(CollisionComponent(shapes: [.generateSphere(radius: 0.1)])) // gravity to PhysicsBody ball.components[PhysicsBodyComponent.self] = .init( PhysicsBodyComponent( // mass in kilograms massProperties: .init(mass: 5.0), material: .generate( staticFriction: 0.0, dynamicFriction: 0.0, restitution: 0.5 ), mode: .dynamic ) )
39. @timmitra Reality Composer Pro Collisions
40. @timmitra Reality Composer Pro - Particles
41. @timmitra Occlusion Capture
42. @timmitra SR Measure
43. @timmitra Reekon Tools
44. @timmitra NVidia Omniverse
45. @timmitra ForeFlight Voyager
46. @timmitra ForeFlight Voyager
47. @timmitra Demoes?
49. @timmitra Resources • Apple Developer app • Kodeco • https://www.kodeco.com/ios/programs/visionos-specialist • https://developer.apple.com/augmented-reality/quick-look/ • https://sketchfab.com
50. @timmitra About: Tim Mitra • iOS Dev - 14 years • visionOS Engineer - 8 months • visionOS SME at Kodeco • MTJC podcast - 10th year • Kodeco author 11 years • lkn.bio/timmitra • @timmitra MTJC podcast Me!
51. @timmitra “I’ll leave you to your own devices.” - Kara Swisher, Re/Code