Building interactive wall décor from junk phones

Why throw away your old phones when you can create a digital work of art? With just some basic arts and crafts skills, you can build a collage of phones with wires intertwining and running across your wall like a circuit board. It can display photo galleries, shopping lists, music visualizations, and make a wicked Zoom background.

In this talk, you'll learn how Node.js and a Raspberry Pi can manage multiple Android devices over USB. We'll explore how to build visualizations using browser APIs, and how to integrate with the wider Internet of Things. Let's give your old mobile phones a new life.

Transcript

1. Building
   interactive
   wall décor from
   junk phones
   Tiger Oakes - @Not_Woods
   1
   

   View Slide

2. Hi! I’m Tiger.
   Software Engineer working on
   Microsoft Loop.
   ● Worked on Edge, Chrome, and Firefox.
   ● From Hawai’i, now based in Seattle.
   ● Likes computers
   2
   

   View Slide

3. Broken cell phones
   👵 🧼 🌲
   4
   

   View Slide

4. View Slide

5. Hoarding
   old tech
   6
   Upcycling
   old tech
   

   View Slide

6. View Slide

7. CellWall
   

   View Slide

8. View Slide

9. View Slide

10. 12
    Cells
    ADB
    WebSockets
    Remote
    HTTP POST
    Server
    On / Off
    Smart home
    Home Assistant
    Architecture
    

    View Slide

11. 01 Build with
    physical
    parts.
    Shopping at the hardware store
    13
    

    View Slide

12. Shopping list
    Base wood panel
    1. Plywood panel
    2. Sandpaper
    3. White acrylic paint
    4. Paint brush
    5. Wood screws or command strips
    for mounting to wall
    

    View Slide

13. Shopping list
    Base wood panel
    1. Plywood panel
    2. Sandpaper
    3. White acrylic paint
    4. Paint brush
    5. Wood screws or command strips
    for mounting to wall
    

    View Slide

14. Shopping list
    Mounts, connectors
    6. Old cell phones and tablets
    running Android
    7. Raspberry Pi or another
    computer for the server
    8. USB cables
    9. USB hub
    10.Velcro strips
    11.Wire clips
    

    View Slide

15. Shopping list
    Mounts, connectors
    6. Old cell phones and tablets
    running Android
    7. Raspberry Pi or another
    computer for the server
    8. USB cables
    9. USB hub
    10.Velcro strips
    11.Wire clips
    

    View Slide

16. Shopping list
    Mounts, connectors
    6. Old cell phones and tablets
    running Android
    7. Raspberry Pi or another
    computer for the server
    8. USB cables
    9. USB hub
    10.Velcro strips
    11.Wire clips
    

    View Slide

17. Shopping list = €74.70
    Base wood panel
    1. Plywood panel
    2. Sandpaper
    3. White acrylic paint
    4. Paint brush
    5. Wood screws or command strips
    for mounting to wall
    Mounts, connectors
    €26.60
    6. Old cell phones and tablets
    running Android
    7. Raspberry Pi or another
    computer for the server
    8. Velcro strips
    9. USB cables
    10.Wire clips
    11.USB hub
    €48.10
    

    View Slide

18. 20
    Cells
    ADB
    WebSockets
    Remote
    HTTP POST
    On / Off
    Smart home
    Home Assistant
    Architecture
    Server
    

    View Slide

19. 02 Talk to your
    phone over
    USB.
    Using Android command line tools to control
    your USB cables
    21
    

    View Slide

20. USB hub connecting to server
    22
    

    View Slide

21. Android
    Debug
    Bridge
    Command-line tool to help with
    developing Android apps.
    23
    

    View Slide

22. # List connected devices
    > adb devices
    List of devices attached
    1e778e25 device
    emulator-5554 device
    24
    

    View Slide

23. # Open a Unix shell on the Android device
    > adb –s shell
    25
    

    View Slide

24. # Command to install an app
    > adb –s ABC12345 install ./app_name.apk
    # Command to simulate pressing power button
    > adb –s ABC12345 shell input keyevent 26
    # Command to open website on phone
    > adb –s ABC12345 shell am start
    –a android.intent.action.VIEW
    –d https://example.com
    26
    

    View Slide

25. Enable USB
    debugging
    Set up developer mode on
    each device
    https://aka.ms/debug-android
    27
    

    View Slide

26. > adb devices
    * daemon not running; starting now at tcp:5037
    * daemon started successfully
    List of devices attached
    ID1234 device
    device
    IDabcd device
    This phone has
    no ID!
    Making up a serial code and
    updating the phone
    28
    

    View Slide

27. # List connected devices with long output
    > adb devices -l
    List of devices attached
    ID1234 device usb:11 transport_id:1
    device usb:22 transport_id:2
    IDabcd device usb:33 transport_id:3
    # Use USB port ID
    > adb –s usb:11 shell input keyevent 26
    # Use transport ID
    > adb –t 3 shell input keyevent 26
    29
    

    View Slide

28. Rooting your
    Android phone
    Gain superuser access so you can
    change a string
    30
    

    View Slide

29. This phone has
    no Wi-Fi!
    Using the USB cable like an
    Ethernet cable with reverse port
    forwarding
    31
    

    View Slide

30. # Send TCP requests over a port to the server via USB, not WiFi
    > adb –s ABC12345 reverse tcp: tcp:
    # Reverse forward TCP request on port 3000 to server port 3000
    > adb –s ABC12345 reverse tcp:3000 tcp:3000
    32
    

    View Slide

31. This phone has
    no screen!
    Replacing the screen?
    33
    

    View Slide

32. github.com/
    Genymobile/
    scrcpy
    pronounced "screen copy"
    Use your computer to see and
    control your phone’s screen
    34
    

    View Slide

33. 35
    Cells
    ADB
    WebSockets
    Remote
    HTTP POST
    On / Off
    Smart home
    Home Assistant
    Architecture
    Server
    

    View Slide

34. 03 Control
    phones from a
    server.
    Automating ADB commands
    36
    

    View Slide

35. View Slide

36. Finally, he’s talking about JavaScript! 38
    

    View Slide

37. https://npm.im/appium-adb
    import { ADB } from 'appium-adb';
    const adb = await ADB.createADB();
    const devices = await adb.getConnectedDevices();
    39
    

    View Slide

38. https://npm.im/appium-adb
    Command line
    # List connected devices
    > adb devices
    List of devices attached
    2315332 device
    1ee21d device
    E2dbeeDR device
    JavaScript
    import { ADB } from 'appium-adb';
    const adb = await ADB.createADB();
    const devices =
    await adb.getConnectedDevices();
    console.log(devices);
    // [{ udid: '2315332' },
    // { udid: '1ee21d' },
    // { udid: 'E2dbeeDR' }]
    40
    

    View Slide

39. Control devices with Node
    async function helloWorld() {
    const devices = await getConnectedDevices();
    await Promise.all(devices.map(async (device) => {
    const KEYCODE_POWER = 26;
    await device.keyevent(KEYCODE_POWER);
    await device.startUri('https://example.com');
    });
    });
    41
    

    View Slide

40. Server
    Communication protocol speed
    42
    Client
    ADB
    

    View Slide

41. Server
    Communication protocol speed
    43
    Client
    ADB
    WebSockets
    TCP
    

    View Slide

42. Server
    Communication protocol speed
    44
    Client
    ADB
    Establish connection
    and turn on phones
    WebSockets
    Send display
    data quickly
    

    View Slide

43. Represent state
    with websites
    If you are presenting a website, an
    internet product or an app, you can
    place a screenshot of it here.
    45
    https://example.com
    

    View Slide

44. Latest Software for 1st Gen Kindle Fire
    Released
    2013.
    Android
    4.4
    Firefox
    Nightly
    102
    Released
    yesterday.
    46
    

    View Slide

45. 04 Coordinating
    across
    clients.
    Splitting up data on the server for the
    cell phones
    47
    

    View Slide

46. const size = {
    width: window.innerWidth,
    height: window.innerHeight,
    };
    48
    

    View Slide

47. 49
    0,0
    1250,10
    Loading…
    

    View Slide

48. View Slide

49. View Slide

50. View Slide

51. Distribute lines of text
    async function showText(text) {
    const lines = text.split('\n');
    // Roughly sort based on ascending x and y position
    const scorePosition = (pos) => (pos.x * 100) + (pos.y * 500);
    const devices = (await getConnectedDevices())
    .sort((a, b) => scorePosition(a) – scorePosition(b));
    for (let i = 0; i < Math.min(lines.length, devices.length); i++) {
    await devices[i].sendUrl(`/page/text?text=${lines[i]}`);
    }
    }
    53
    

    View Slide

52. 54
    Cells
    ADB
    WebSockets
    Remote
    HTTP POST
    On / Off
    Smart home
    Home Assistant
    Architecture
    Server
    

    View Slide

53. 05 Integrate
    with other
    services.
    Remotes, smart homes, and more
    55
    

    View Slide

54. import express from 'express';
    const app = express();
    app.post('/api/power', async function (req, res) {
    const devices = await getConnectedDevices();
    await Promise.all(devices.map(async (device) => {
    const KEYCODE_POWER = 26;
    await device.keyevent(KEYCODE_POWER);
    });
    res.send('Toggled power');
    });
    56
    

    View Slide

55. 57
    action="/api/power"
    method="post"
    >
    
    On / Off
    
    
    On / Off
    

    View Slide

56. Server
    Communication protocol
    58
    Cell
    ADB
    WebSockets
    On / Off
    Remote
    HTTP POST
    

    View Slide

57. Other services
    59
    Cell
    ADB
    WebSockets
    Services
    HTTP POST
    Server
    & Webhooks
    Webhook call
    fired by
    service
    

    View Slide

58. 60
    Web
    server
    Reminder
    Notified!
    Smart home
    software
    

    View Slide

59. Home Assistant
    https://home-assistant.io
    Open source smart home
    software to coordinate
    devices
    61
    

    View Slide

60. Speaker
    integration
    62
    

    View Slide

61. Power of the Web!
    63
    Services
    Server
    Internet
    of Things
    Smart home
    

    View Slide

62. 64
    Cells
    ADB
    WebSockets
    Remote
    HTTP POST
    Server
    On / Off
    Smart home
    Home Assistant
    Architecture
    

    View Slide

63. home-
    assistant.io
    Install on Raspberry Pi,
    Docker Container,
    old computer, and more!
    65
    

    View Slide

64. tigeroakes.com
    @Not_Woods
    Thank you! T
    H
    A
    N
    K
    Y
    O
    U
    66
    

    View Slide

65. Credits.
    Presentation Template: SlidesMania
    Sample Images: Unsplash
    Fonts used in this presentation: Cascadia Code
    67
    

    View Slide