Me, Myself, and eyeOS

I was there, in the room, in San Francisco, at Moscone, January 2007, when Steve said "Are you getting it?".

That was the beginning of my career as an iOS developer, with the SDK coming just 1 year later. But the love for the iPhone didn't wait for the SDK.

Then I went through it all: the iPad, longer and bigger iPhones, the same with the iPad, Apple TV, Apple Watch etc ... and every year came a new iPhone, a new iOS (and at the launch of iOS 7 I almost had to cry, where had my skeumorphism gone?!).

And then there was always the other side of me: Green politics and an attempted switch to full-time politics in 2024, which unfortunately failed in 2023.

The day after my political defeat there was WWDC and the presentation of Vision Pro and I was back. Motivated again because visionOS and Vision Pro are at least as revolutionary (actually more so) to me as the iPhone and iOS, or iPhoneOS. In this talk I summarise what some of you may already know, but everyone will definitely learn something.

I have been intensively involved with the new platform since the keynote. Even more intensively since the SDK came out. Even more intensively since I was at Apple in Munich. The revolution has already begun.

At best, not everyone may have noticed it yet. In this lecture we will take a closer look at the Vision Pro and its hardware as well as the different possibilities of the SDK, i.e. visionOS. We will look at how you can show your existing iPad and iPhone apps in Shared Space, as well as immersive Spaces and 3D objects.

Transcript

1. Me, Myself & eyeOS Become a spacial developer! Manu "Stuff" Carrasco Molina carrascomolina.com/speaking stuffmc@mastodon.social Thanks ! Tina Nigro " & Matthias Weber "
2. > whoami? !" • # $ • % & • ' ( + ) ا َل ْ ع َ ر َ ب ِ ي * ة ُ Yes!, Oui !, ¡Si! ! Once was a dev without " #tech # ! Than came the iPhone, than Swift, than SwiftUI, than visionOS! ! Politicaly involved since 12 years with the Green Party ! Tried (and failed) to run for % parliament. ! 10 & years ' without ( meat, 7 ) of it #vegan Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 2
3. Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 3
4. Vision Pro . ! Array of cameras and sensors . " Spatial audio close to your ears . # Head Band with Fit Dial . $ Two 4.5K micro‑OLED displays . % Light Seal conforms to your face Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 4
5. Vision Pro . ! Digital Crown: " + Immersion level . # Top button: Spatial Videos/Photos?! . $ Battery: 2h of use / all‑day (plugged) . % Prescription glasses mounted inside Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 5
6. Cameras/Sensors Outside . ! 2 HR Cameras (> 1 billion pixels/sec) . ⬅⬇➡ Head & Hand Tracking: 6 Cams . IR Illumination Sensoren . % Realtime 3D-mapping: 5 Sensors . Some sensors Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 6
7. Cameras/Displays Inside ! Eye tracking: LEDs & IR cameras ! Invisible light patterns to eyes ! 23 Millions Pixels ! An iMac on each Eye! ! Total 12 Cameras Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 7
8. Display Outside ! EyeSight reveals your eyes 1 ! Tell if you’re using apps ! ... or fully immersed in an experience 1 So yeah, this is no passthrough and not really your eyes. Super cringe. Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 8
9. Processors M2 Chip R1 Chip Runs visionOS Process input (cams, sensors, mic) Advanced Computer vision Stream images to displays < 12ms Delivers graphics Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 9
10. Navigation ! By using your eyes, hands, and voice. ! Hand Gestures: Tap (your finger tips), pinch, rotate, etc, ... ! Keyboard: Virtual or Bluetooth ! Mouse or Trackpad ! Game Controller ! Pen? Use your finger! Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 10
11. Look at me, I'm hovering! "When you look it hovers" 4 Button("Hover") { } // Auto HStack { ... } .padding() .contentShape(.hoverEffect, .rect(cornerRadius: 100) ) .hoverEffect() 4 WWDC23: Elevate your windowed app for spatial Computing Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 11
12. Software compatibility ! Most iOS Apps will run fine... 2 ! Rotation with tapping a button ! No API deprecated prior to iOS 14 ! No UIDeviceOrientation " / UIScreen ! UITabBar have no leading or trailing 2 ! No Pencil. Use #if !os(xrOS) 2 Prefers iPad at Landscape, if only iPhone then Portrait Tab bars have a different design and do not lay out leading-to-trailing Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 12
13. Dark? Light? Glass! ! visionOS's colorScheme is Dark! ! Compatible Apps use Light Mode! ! Adopt Vibrancy / Vibrant Materials. ! Use SwiftUI Semantic Styles. ! PlantIcon(plant) .background (.fill.tertiary, in: .circle) Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 13
14. Tools ! Xcode 15 (Beta 8!) or 15.1 Beta (No Intel!) ! visionOS Simulator (with AR Anchors!) 3 ! Reality Composer (iOS)/(Pro on Mac) ! Reality Converter beta (converts .obj, .gltf, .usd to USDZ) ! Parallax image tools 3 GitHub/tracyhenry: Sample Code (not from ) on how to stick an entity to a Wall or Table Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 14
15. Debugging ! Simulator has some possibilities! ! Warning [REDACTED] Can't tell... " ! The Problem is shown here " ! SideCar projects Xcode to Vision Pro ! Occlusion Mesh, Surfaces, ... Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 15
16. Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 16
17. The Icon ! 3 sRGB 1024x1024 Images • No P3! ! Background (Color? 6) ! Middle + Front ! Use SF Symbols's Copy Image 6 FB13078913: Allow providing a Color and/or Gradient... Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 17
18. Kind of apps ! Shared Space ! Immersive ! Level decided by crown Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 18
19. Immersion Style ! 3 Modes ! Mixed ! Progressive ! Full ! Destination Video Sample Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 19
20. An immersive app @main struct WorldApp: App { var body: some Scene ImmersiveSpace { SolarSystem() } } } Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 20
21. Load a 3D Object Model3D(named: "Earth") { phase in switch phase { case .empty: Text("Waiting" ) case .failure(let error): Text("Error \(error)") case .success(let model): model.resizable() } } Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 21
22. Change Immersion @main struct WorldApp: App { @State private var style: ImmersionStyle = .mixed var body: some Scene { ImmersiveSpace(id: "solar") { SolarSystem(immersionStyle: $style) .immersionStyle(selection: $style, in: .mixed, .progressive, .full) } } } Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 22
23. SolarSystem(immersionStyle: ScurrentStyle) .simultaneousGesture(EntityGesture(MagnifyGesture()) .onChanged { value in scale = value.gestureValue.magnification style = if scale > 5 { .progressive } else if scale > 10 { .full ｝else { .mixed } } Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 23
24. Beyond the window ! Normal Windows (UIKit5 or SwiftUI) work ! Some system stuff will get 3D effects ! You can use ornaments (no time) ! Volumes (3D Cubes, basically) ! Not resizable, but movable ! Fixed size. Come closer to see it bigger. ! Immersive Spaces. 5 Even with Objective-C, if you feel sad today. Apple really wants your old apps. Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 24
25. struct VisionApp: App { @Environment(\.openImmersiveSpace) var openImmersiveSpace var body: some Scene { WindowGroup { Button("Open Immersive Space") { Task { await openImmersiveSpace(id: "immersiveSpace") } } } ImmersiveSpace(id: "immersiveSpace") { ContentView() .placementGestures(initialPosition: Point3D([475, -1200.0, -1200.0])) } } } Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 25
26. Dismiss @Environment(\.dismissImmersiveSpace) private var dismissImmersiveSpace dismissImmersiveSpace() // Only one can be shown at a time ⛔ Unable to present another Immersive Space when one already requested Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 26
27. Sheets / Alerts ! Will always be presented modally ! Like macOS ! System Sheets (e.g. PhotoPicker) as well ! Will push the content back (Z-axis) ! Alert/Sheet takes the original Z-Axis Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 27
28. World.xcodeproj ! Amazing Hello World Sample App from  ! PlacementGesturesModifier/placementGestures ! Good example of move in EarthEntity's update function Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 28
29. HappyBeam.xcodeproj ! Game interface in SwiftUI ! " -gesture detection with ARKit ! Supports several kinds of input ! Hands: In front # or on your laps (tap) ! Game Controller Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 29
30. Diorama.xcodeproj ! Design scenes for your visionOS app using Reality Composer Pro. ! I forgot to test this one on Vision Pro! Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 30
31. RealityKitContent (rkc) ! When creating a new visionOS app with immersive content ! Open Package in Reality Composer Pro scene = Entity(named: "Scene", in: rkcBundle) // Entity entity = scene.findEntity(named: "Entity") as? ModelEntity entity.materials = [SimpleMaterial(color: isMetallic:)] Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 31
32. RealityView ! RealityKit: New to me, not to some of you! ! To me, AR didn't make any sense in 2D! ! Attachments ! Confusing API ! Uses IDs ! Declares after, calls before Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 32
33. Attachments RealityView { content, attachments in // ... make entity and add it to content if let text = attachments.entity(for: "text") { entity.addChild(text) } } attachments: { Attachment(id: "text") { Text("SWIFTUI!") } } ! : Please provide a more convenient way to attach SwiftUI Views Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 33
34. OpticID ! Encrypted and only on Secure Enclave processor ! Obviously replaces TouchID or FaceID ! Although TouchID could have worked... Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 34
35. What apps also don't see... Privacy ! Data from cameras and sensors: system level ! Eye ! Not shared with Apple, third-party apps, or websites. ! Only your selection when you tap your fingers together. Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 35
36. Safari Web Extensions ! 2000 Extensions since a few years on macOS ! Since 2021 on iOS ! Now the same as on iOS, on visionOS! Manu @StuFFmc Carrasco Molina: Me, Myself and eyeOS ! 36
37. Grazie! ش ك ر ا َ ! Дякую ! @stuffmc@mastodon.social " carrascomolina@bsky.app # linkedin.com/in/carrascomolina $ carrascomolina.com/speaking % speakerdeck.com/stuffmc manu@carrascomolina.com