ICECER 2025: Cross-Device Motion Interaction via Apple’s Native System Frameworks

We introduce an open-source, fully offline pipeline that transforms a consumer-grade iPhone into a motion controller with real-time tactile feedback, using only native Apple frameworks. Designed for rapid prototyping and applied mobile HCI scenarios, the system integrates CoreMotion for inertial sensing, MultipeerConnectivity for peer-to-peer data transmission at 10 Hz, and CoreHaptics for immediate tactile confirmation. A built-in logger captures end-to-end latency without requiring clock synchronization, yielding a mean delay of 70.4 ms and 95th percentile below 74 ms on typical 5 GHz Wi-Fi (−55 dBm
RSSI). We validated the pipeline through a real-time demonstrator game, KeepCalm, deployed during a public event with 21 participants. Results showed stable connections, zero packet loss, and negligible power impact (24 mW on iPhone 13 mini). With fewer than 500 lines of Swift code and no reliance on cloud infrastructure, this system provides a compact, reproducible foundation for embodied interaction research, casual games, and offline educational tools. All source code, latency logs, and provisioning scripts are openly released under an MIT license.

Index Terms—Cross device interaction, motion input, haptic feedback, MultipeerConnectivity, mobile HCI, peer to peer communication

Transcript

1. Cross-Device Mo-on Interac-on via Apple’s Na-ve System Frameworks Ezequiel França dos Santos PhD Candidate · IADE — Universidade Europeia UNIDCOM/IADE · 2Ai — IPCA ORCID: 0000-0001-9321-8444 © ICECER 2025 · E. França dos Santos 1
2. Context Cross-device interac/on is not new. Phones have been used as controllers for years. Most examples depend on: - servers - calibra5on - non-reproducible code - complex setups This work is small in scope: © ICECER 2025 · E. França dos Santos 2
3. Mo#va#on We wanted a system that is: - offline - easy to understand - low-latency - under 500 lines of Swi: - based en<rely on Apple-na<ve frameworks Not revolu*onary, just prac*cal. © ICECER 2025 · E. França dos Santos 3
4. Goal Create the simplest possible mo/on-controller pipeline using only Apple-na/ve APIs, with fully open-source code and transparent latency measurements. © ICECER 2025 · E. França dos Santos 4
5. © ICECER 2025 · E. França dos Santos 5
6. Why Apple Na+ve? • CoreMo&on for IMU • Mul&peerConnec&vity for peer-to-peer Wi-Fi • CoreHap&cs for tac2le response • Swi5UI + SpriteKit for host-side interac2on Vendor-op)mized → fewer variables → predictable, reproducible behavior. © ICECER 2025 · E. França dos Santos 6
7. Mo#on Capture Sampling at 10 Hz Captured signals: - accelera2on: x, y, z - gyroscope: ωx, ωy, ωz Gesture rule: ay > 0.5 m/s² A minimal, transparent detec0on criterion. © ICECER 2025 · E. França dos Santos 7
8. Data Structure 36-byte Mo+onData payload: - 8 bytes: +mestamp - 24 bytes: IMU data - 4 bytes: checksum Final on-air frame: 88 bytes Throughput: 7.0 kbit/s © ICECER 2025 · E. França dos Santos 8
9. Latency Measurement Setup: - 1,000 samples at 10 Hz - 5 GHz Wi-Fi (−55 dBm) - 3σ outlier removal - median-adjusted Cmestamp offsets Goal: transparent measurement, not precision engineering. © ICECER 2025 · E. França dos Santos 9
10. © ICECER 2025 · E. França dos Santos 10
11. Summary Table Key values: - Mean: 70.4 ms - 95th percen.le: 73.2 ms - Min: 52.2 ms - Max: 82.2 ms © ICECER 2025 · E. França dos Santos 11
12. Hap$c Confirma$on A short CoreHap-cs pulse is triggered a4er gesture valida-on. Round-trip iPhone → Host → Hap3c: ≈ 4.8 ms Improves perceived responsiveness. © ICECER 2025 · E. França dos Santos 12
13. Energy Use Measured with Xcode Instruments: - 24 mW average consump1on - <2% ba9ery for 10 minutes - no thermal thro9ling © ICECER 2025 · E. França dos Santos 13
14. Public Deployment 21 par'cipants — crowded Wi-Fi event environment Observed: - zero packet loss - no disconnec5ons - consistent ba6ery profile A simple architecture → stable behavior even under pressure. © ICECER 2025 · E. França dos Santos 14
15. Strengths • compact implementa-on • fully offline • predictable • reproducible • open-source under MIT license © ICECER 2025 · E. França dos Santos 15
16. Limita&ons • single-controller only • threshold gesture detec2on • no smoothing or ML • tested only inside Apple ecosystem • limited ecological diversity © ICECER 2025 · E. França dos Santos 16
17. Future Work • mul%-controller support • CoreML-based gesture classifica%on • Apple Founda%on Models on-device • ARKit sensor fusion • broader real-world deployments © ICECER 2025 · E. França dos Santos 17
18. What This Work Provides 1. A minimal reference pipeline 2. Full open-source codebase 3. Transparent latency analysis 4. “KeepCalm” demonstrator game A star'ng point, not a final product. © ICECER 2025 · E. França dos Santos 18
19. Conclusion A simple, reproducible way to convert an iPhone into a mo8on controller with sub-75 ms latency, hap8c confirma8on, and zero server infrastructure. © ICECER 2025 · E. França dos Santos 19
20. Open Source h"ps:/ /github.com/GameAISchool2024members/TeamBrazil © ICECER 2025 · E. França dos Santos 20
21. Thank You Ezequiel França dos Santos IADE — Universidade Europeia UNIDCOM/IADE · 2Ai — IPCA ezefranca.com · ezefranca@acm.org © ICECER 2025 · E. França dos Santos 21