ヘッドジェスチャを検知して Vibe Cooking - iOSDC Japan 2025

Transcript

1. Kanta Oikawa - 2025/09/20 ϔουδΣενϟΛݕ஌ͯ͠ Vibe Cooking iOSDC Japan 2025 Day 1 Track A 1
2. ͓͍͍͠ྉཧΛָʹ࡞ͬͯ৯΂͍ͨ 1ਓ฻Β͠ͷྉཧॳ৺ऀ • ޮ཰తʹ೑͡Ό͕ɾਬ͖ࠐΈ͝͸ΜɾಲोΛ࡞Γ͍ͨ • γϯάϧεϨουͰ • ϨγϐΛݟͳ͕Β࡞Γ͍ͨ • ྉཧதʹiPhoneɾiPadΛ৮Γͨ͘ͳ͍ 2
3. Vibe Cooking • ෳ਺ͷϨγϐ͔Β1ͭͷϨγϐΛੜ੒ • ϨγϐͷखॱΛಡΈ্͛ • ಡΈ্͛ΔखॱΛԻ੠Ͱίϯτϩʔϧ 3
4. Vibe Cooking x ϔουδΣενϟ • Ի੠ಡΈ্͛ɾίϯτϩʔϧͰ͸؀ڥͷӨڹΛड͚΍͍͢ • ྉཧத͸ AirPods Λ૷ண • Core Motion Λ࢖ͬͯ AirPods ͷϞʔγϣϯσʔλΛऔಘՄೳ • ਐΉɾ໭Δૢ࡞ΛϔουδΣενϟͰ 4
5. AirPods ͰϔουδΣενϟΛݕ஌͢Δྫ (2024/09) ϔουδΣενϟΛ࢖༻ͯ͠ Siri ʹԠ౴͢Δ • AirPods Pro 2·ͨ͸AirPods 4ʢ྆ϞσϧʣΛ͓࣋ͪͰɺʮԻ੠Ͱண৴Λ஌ ΒͤΔʯͱʮ௨஌ͷಡΈ্͛ʯΛΦϯʹ͍ͯ͠Δ৔߹͸ɺ੠·ͨ͸खΛ࢖Θ ͣʹɺϔουδΣενϟΛ࢖༻ͯ͠ɺ௨࿩Λड͚ͨΓڋ൱ͨ͠Γɺ௨஌Λ։͍ ͨΓดͨ͡Γ͢Δ͜ͱ΋Ͱ͖·͢ɻ • ϔουδΣενϟΛ࢖༻ͯ͠SiriʹԠ౴͢Δ • ి࿩ʹग़Δɺ·ͨ͸௨஌ͱϝοηʔδʹฦ৴͢Δ: ᰐ͖·͢ɻ • ి࿩Λڋ൱͢Δɺ·ͨ͸௨஌ͱϝοηʔδΛด͡Δ: टΛࠨӈʹৼΓ·͢ɻ 5 AirPodsͰίϯτϩʔϧͱδΣενϟΛ࢖༻͢Δ - Apple αϙʔτʢ೔ຊʣ https://support.apple.com/ja-jp/guide/airpods/devb2c431317 (Accessed on 2025/09/12)
6. UX ઃܭ 6 30° ͪ͜ΒΛ޲͍ͯ͏ͳͣ͘ͱ ӈʹεϫΠϓ ͪ͜ΒΛ޲͍ͯ͏ͳͣ͘ͱ ࠨʹεϫΠϓ
7. ޲͍͍ͯΔํ޲Λݕ஌͢Δ final class HeadphoneMotionRepository: NSObject, CMHeadphoneMotionManagerDelegate, Sendable { private let manager: CMHeadphoneMotionManager override init() { ... } func startTracking(queue: OperationQueue? = .current) throws -> AsyncStream<CMDeviceMotion> { guard manager.isDeviceMotionAvailable else { throw HeadphoneMotionRepositoryError.deviceMotionNotAvailable } return AsyncStream { continuation in manager.startDeviceMotionUpdates(to: queue ?? .main) { motion, error in guard let motion else { return } continuation.yield(motion) } } } ... } 7
8. ޲͍͍ͯΔํ޲Λݕ஌͢Δ for await motion in try motionService.startTracking() { state.motion = motion } ... if abs(state.motion?.attitude.yaw ?? 0) > .pi / 12 { // π / 12 = 15° ... } 8 yaw | Apple Developer Documentation https://developer.apple.com/documentation/coremotion/cmattitude/yaw (Accessed on 2025/09/20)
9. ͏ͳ͖ͣδΣενϟΛݕ஌͢Δ 1. ͏ͳ͖ͣδΣενϟͷϞʔγϣϯσʔλΛऩू͢Δ 2. ϞʔγϣϯσʔλΛֶशͯ͠ Core ML ϞσϧΛ࡞੒͢Δ 3. ϞσϧΛ࢖ͬͯ͏ͳ͖ͣδΣενϟΛݕ஌͢Δ 9
10. ͏ͳ͖ͣδΣενϟΛݕ஌͢Δ 1. ͏ͳ͖ͣδΣενϟͷϞʔγϣϯσʔλΛऩू͢Δ • ͏ͳͣ͘δΣενϟ ͱ ͦΕҎ֎ͷδΣενϟ ͷ 2 Ϋϥε • CMDeviceMotion ʹؚ·ΕΔ஋Λ CSV ϑΝΠϧʹॻ͖ࠐΈ • attitude_roll, attitude_pitch, attitude_yaw, gravity_x, gravity_y, gravity_z, quaternion_x, quaternion_y, quaternion_z, quaternion_w, rotation_rate_x, rotation_rate_y, rotation_rate_z, user_acceleration_x, user_acceleration_y, user_acceleration_z • ͻͨ͢Β͏ͳͣ͘ 10
11. ͏ͳ͖ͣδΣενϟΛݕ஌͢Δ 1. ͏ͳ͖ͣδΣενϟͷϞʔγϣϯσʔλΛऩू͢Δ 11
12. ͏ͳ͖ͣδΣενϟΛݕ஌͢Δ 2. ϞʔγϣϯσʔλΛֶशͯ͠ Core ML ϞσϧΛ࡞੒͢Δ 12
13. ͏ͳ͖ͣδΣενϟΛݕ஌͢Δ 3. ϞσϧΛ࢖ͬͯ͏ͳ͖ͣδΣενϟΛݕ஌͢Δ let configuration = MLModelConfiguration() let model = try HeadGestureClassifier(configuration: configuration) let input = HeadGestureClassifierInput( rotation_rate_x: try MLMultiArray(motions.map { $0.rotationRate.x }), stateIn: try MLMultiArray(shape: [400], dataType: .double) ) let output = try model.prediction(input: input) return Gesture(rawValue: output.label) ?? .idle ... enum Gesture: String { case idle case nod ... } 13
14. σϞ 14
15. ࢖ͬͯΈͨײ૝ɾߟ࡯ • टΛಈ͔͢ͷ͕ͭΒ͍ • খ͞ͳಈ͖Ͱ΋ݕ஌Ͱ͖Δײ౓ • ޡݕ஌͠ͳ͍ਫ਼౓ • ϑΟʔυόοΫ͕΄͍͠ • Siri ΁ͷԠ౴࣌ʹ͸ "Χνο"ͱ͍͏Ի੠ʢʁʣͷϑΟʔυόοΫ 15
16. ࣮૷ͯ͠Έͨײ૝ • Core Motion ͱ Core ML Λ૊Έ߹ΘͤͯδΣενϟݕ஌͕؆୯ʹ࣮૷Ͱ͖ͨ • ML ʹؔ͢Δߴ౓ͳ஌͕ࣝͳͯ͘΋؆୯ʹϞσϧΛ࡞Δ͜ͱ͕Ͱ͖Δ • ஌͕ࣝ͋Ε͹ਫ਼౓্͕͕Γͦ͏ • CMDeviceMotionͷͦΕͧΕͷ஋͕ҙຯ͢Δ͜ͱΛཧղ͢Δ͜ͱ͕ॏཁ 16
17. Vibe Cooking Coming this fall 17
18. Kanta Oikawa Kantacky • ެཱ͸ͩͯ͜ະདྷେֶ M1 • ઈࢍब׆தʂʂ • GitHub: @kantacky • Twitter: @kanta_cky 18