Building an arcade with Android By Eric Boissonneault Android Developer @ Transit

Agenda ● Discovering a smart lighting product ● Searching for the SDK ● Making my own Android library ● Various experiments ● What about a game? ● Adding controller inputs ● Future improvements and ideas

Twinkly Squares

Twinkly Squares ● Master tile ○ USB-C powered ○ Using ESP32 microcontroller ○ Wifi capabilities ○ BT capabilities ○ 4 ports for tile cables ● Extension tiles ○ 2 ports for tile cables ● Maximum of 16 tiles (1 master + 15 extension) ● 1 tile = 64 pixels (8x8) ● Current configuration is 9 tiles (24x24) ● Total of 576 pixels ● Pixels mapping is done through the official app

Twinkly lights mapping

Twinkly product features Effects (animated GIFs) Single color Sound reactive

Available SDKs ● No official SDK Source: https://xled-docs.readthedocs.io/ ● An on-device API exists ● API specifications have been reversed-engineered ● Various client implementations can now be found ● No Android SDK

Hypothesis Successfully create an Android SDK capable of updating frames… ● Fast enough? ● Over a long period of time? ● Without experiencing lag?

Implementation Requirements ● REST Endpoint ○ Authenticate ○ Get the pixels layout ○ Switch to real-time mode ● UDP Endpoint ○ Send frames using the real-time mode

Creating a background service

Creating a background service

Background service: REST API REST API authentication, fetch pixels layout, setup the real-time mode

REST API Implementation

Background service: Real-time UDP frames Every 25ms: Sends frame if pixels changed OR after 10 seconds idle (keep-alive)

UDP datagram (frame) implementation Source: https://xled-docs.readthedocs.io/

Board class: Pixels Matrix Representation

Experiment 0: Display all pixels in the correct order

Experiment 1: Edges and corners

Experiment 2: Bitmap

Experiment 3: Bitmap horizontal / vertical translation

Experiment 4: Text

Experiment 5: Tetris

Experiment 6: Battleship

What about a game? Tug Of War

Game implementation

Player interaction (input) and sending pixel frames (output)

Game controller inputs L2 and R2 buttons

Arcade buttons

Upcoming game improvements ● Modify the board layout (72x8) ● Adjustable edges for a game ● Pause the game ● Add animations before the game starts and after winning ● Support multiplayer 2v2, 3v3, 4v4 ● Add a leaderboard ● Build a plexiglass table to insert the Twinkly Squares and arcade buttons

Idea on my bucket list ● Twinkly Matrix (new product) ● Challenge with mapping the lights ○ X,Y coordinates aren’t accurate ○ Custom mapping tool needed ● Face recognition using a camera with OpenCV ○ Select face area ○ Convert to a pixel matrix ● Project the face pixels into the Twinkly Matrix ● Add animations and various interactions ● Extend the layout from 10x50 lights to 200x40 lights

References ● Twinkly products: https://twinkly.com/ ● Twinkly unofficial SDKs documentation: https://xled-docs.readthedocs.io/ ● ChatGPT: https://chat.openai.com/ ● GitHub Copilot: https://github.com/features/copilot ● Arcade buttons: Search for SJ@JX Android on Amazon

Questions? Thank you!