Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building "learn to touch type" glove with Elixir and Arduino

Building "learn to touch type" glove with Elixir and Arduino

Knowledge share about building "touch typing glove" with Arduino and Elixir.This talk is about my journey from research, choosing tools, making flexible tiny pressure sensors, plugging them into Arduino, reading information from Arduino to the server running Elixir, rendering result in the browser.

Cc7eb90de3bd4bce233ba86cf7c41060?s=128

tetiana12345678

September 04, 2016
Tweet

Transcript

  1. BUILDING A TOUCH TYPING SYSTEM BY TETIANA DUSHENKIVSKA

  2. elixirconf 2016 @tetiana12345678

  3. 124 WORDS PER MINUTE elixirconf 2016 @tetiana12345678

  4. CAN YOU TOUCH TYPE? elixirconf 2016 @tetiana12345678

  5. elixirconf 2016 @tetiana12345678

  6. BUILDING A TOUCH TYPING SYSTEM BY TETIANA DUSHENKIVSKA

  7. 1. Display the text to type 2. Record key press

    events from the keyboard 3. Record finger events 4. Analyze events 5. Return match result to UI elixirconf 2016 @tetiana12345678
  8. ARCHITECTURE

  9. 1. Display the text to type elixirconf 2016 @tetiana12345678

  10. DISPLAY TEXT elixirconf 2016 @tetiana12345678

  11. 2. Record key press events from the keyboard elixirconf 2016

    @tetiana12345678
  12. RECORD KEY EVENTS elixirconf 2016 @tetiana12345678

  13. 3. Record finger events elixirconf 2016 @tetiana12345678

  14. RECORD FINGER EVENTS elixirconf 2016 @tetiana12345678

  15. 4. Analyze events elixirconf 2016 @tetiana12345678

  16. ANALYZE EVENTS elixirconf 2016 @tetiana12345678

  17. RESILIENT, ADAPTIVE AND STEADY-STATE WATER SUPPLY NETWORKS elixirconf 2016 @tetiana12345678

  18. ANALYZE EVENTS UMBRELLA APPS elixirconf 2016 @tetiana12345678

  19. PHOENIX, IS IT YOU??? elixirconf 2016 @tetiana12345678

  20. FULL ARCHITECTURE elixirconf 2016 @tetiana12345678

  21. IMPLEMENTATION

  22. $ mix new typehero_project --umbrella elixirconf 2016 @tetiana12345678

  23. $ cd typehero_project/apps elixirconf 2016 @tetiana12345678

  24. $ mix phoenix.new typehero_web $ cd typehero_web $ mix deps.get

    $ mix phoenix.server elixirconf 2016 @tetiana12345678
  25. JOIN CHANNEL DATA FLOW elixirconf 2016 @tetiana12345678

  26. PHASER WELCOME MESSAGE elixirconf 2016 @tetiana12345678

  27. JS JOIN CHANNEL socket.connect() const channel = socket.channel("games:lobby", {}) channel.join()

    .receive("ok", () => { this.state.start("lobby", true, false, channel) }) elixirconf 2016 @tetiana12345678
  28. GENARATE LOBBY CHANNEL $ mix phoenix.gen.channel Lobby elixirconf 2016 @tetiana12345678

  29. ADD CHANNEL TO SOCKET defmodule TypeheroWeb.UserSocket do use Phoenix.Socket ...

    channel "games:lobby", TypeheroWeb.LobbyChannel end elixirconf 2016 @tetiana12345678
  30. JOIN CHANNEL CALLBACK defmodule TypeheroWeb.LobbyChannel do use TypeheroWeb.Web, :channel def

    join("games:lobby", _payload, socket) do {:ok, socket} end end elixirconf 2016 @tetiana12345678
  31. DISPLAY TEXT elixirconf 2016 @tetiana12345678

  32. GET TEXT DATA FLOW elixirconf 2016 @tetiana12345678

  33. JS START GAME start_button.events.onInputDown.add(() => { start_button.destroy() this.channel.push("start_game") }) elixirconf

    2016 @tetiana12345678
  34. PHOENIX CHANNEL defmodule TypeheroWeb.LobbyChannel do ... def handle_in("start_game", _payload, socket)

    do text = Typehero.Core.start_game(socket) push socket, "start_game", %{text: text} {:noreply, socket} end end elixirconf 2016 @tetiana12345678
  35. CORE UMBRELLA APP $ cd typehero_project/apps $ mix new typehero

    --sup elixirconf 2016 @tetiana12345678
  36. CORE UMBRELLA APP. INIT defmodule Typehero.Core do ... def init(state)

    do # Get text from Ecto... text_to_type = "in my opinion..." state = %{text: text_to_type, socket: %{}} {:ok, state} end end elixirconf 2016 @tetiana12345678
  37. CORE. START_GAME defmodule Typehero.Core do ... def handle_call({:start_game, socket}, _from,

    state = %{text: text}) do {:reply, text, %{state | socket: socket}} end end elixirconf 2016 @tetiana12345678
  38. CORE AS DEPS OF TYPEHERO_WEB defmodule TypeheroWeb.Mixfile do ... def

    application do [mod: {TypeheroWeb, []}, applications: [..., :typehero]] end end elixirconf 2016 @tetiana12345678
  39. JS RENDER TEXT this.channel.on("start_game", this.onStartGame.bind(this)) onStartGame({text}) { this.renderText(text) this.listenKeyboard() }

    elixirconf 2016 @tetiana12345678
  40. SEND KEY EVENTS TO CORE APP

  41. KEY PRESS EVENTS DATA FLOW elixirconf 2016 @tetiana12345678

  42. JS SEND KEY EVENT onKeyPress({key}) { const event = this.createEvent(key)

    this.channel.push("key", event) } elixirconf 2016 @tetiana12345678
  43. PHOENIX CHANNEL def handle_in("key", %{"key"=> key, "id"=> id}, socket) do

    Typehero.Core.key_press(key, id) {:noreply, socket} end elixirconf 2016 @tetiana12345678
  44. CORE KEY_PRESS EVENT defmodule Typehero.Core do ... alias Typehero.EventHandler def

    handle_cast({:key, key, id}, state) do EventHandler.key_event(key, id) {:noreply, state} end end elixirconf 2016 @tetiana12345678
  45. elixirconf 2016 @tetiana12345678

  46. SEND FINGER EVENTS TO CORE

  47. ARCHITECTURE elixirconf 2016 @tetiana12345678

  48. BUILDING A SENSOR

  49. VELOSTAT elixirconf 2016 @tetiana12345678

  50. CONDUCTIVE MATERIAL elixirconf 2016 @tetiana12345678

  51. CUT SENSOR SHAPE elixirconf 2016 @tetiana12345678

  52. MAKE A SENSOR elixirconf 2016 @tetiana12345678

  53. ARDUINO elixirconf 2016 @tetiana12345678

  54. TEST SENSOR elixirconf 2016 @tetiana12345678

  55. C PROGRAM const int analogPin = A0; int finger1 =

    0; void setup() { Serial.begin(9600); } void loop() { finger1 = analogRead(analogPin); Serial.println(finger1); } elixirconf 2016 @tetiana12345678
  56. VALUES FROM SENSOR elixirconf 2016 @tetiana12345678

  57. C PROGRAM. SEND VALUE ONLY ONCE bool finger1_down = false;

    int finger1_sense = 200; ... void loop() { if (finger1 > finger1_sense && !finger1_down) { finger1_down = true; Serial.print("1"); } else if (finger1 <= finger1_sense) { finger1_down = false; } } elixirconf 2016 @tetiana12345678
  58. RINSE AND REPEAT elixirconf 2016 @tetiana12345678

  59. CHECK THIS OUT! elixirconf 2016 @tetiana12345678

  60. elixirconf 2016 @tetiana12345678

  61. elixirconf 2016 @tetiana12345678

  62. GIVE ME MORE ELIXIR!

  63. TERMINOLOGY elixirconf 2016 @tetiana12345678

  64. FINGER EVENTS DATA FLOW elixirconf 2016 @tetiana12345678

  65. FINGER EVENTS SERIAL APP

  66. GENERATE APP $ cd typehero_project/apps $ mix new typehero_serial --sup

    elixirconf 2016 @tetiana12345678
  67. ADD SERIAL LIBRARY defmodule TypeheroSerial.Mixfile do ... defp deps do

    [ ..., {:serial, "~> 0.1.0"}] end end elixirconf 2016 @tetiana12345678
  68. ADD CORE UMBRELLA APP AS DEPS defmodule TypeheroSerial.Mixfile do ...

    def application do [applications: [..., :typehero ], ] end end elixirconf 2016 @tetiana12345678
  69. SERIAL FORMATTER INIT defmodule TypeheroSerial.Formatter do ... def init(id) do

    id = 0 {:ok, serial} = Serial.start_link Serial.open(serial, "/dev/cu.usbmodem1421") Serial.set_speed(serial, 9600) Serial.connect(serial) {:ok, id} end end elixirconf 2016 @tetiana12345678
  70. FINGER NUMBER SEND FROM ARDUINO defmodule TypeheroSerial.Formatter do ... def

    handle_info({:elixir_serial, serial, data}, id) do cond do data =~ ~r/1/ -> Typehero.Core.finger_press(1, id) data =~ ~r/2/ -> Typehero.Core.finger_press(2, id) ... true -> nil end {:noreply, (id + 1)} end end elixirconf 2016 @tetiana12345678
  71. elixirconf 2016 @tetiana12345678

  72. CORE UMBRELLA APP

  73. DATA FLOW elixirconf 2016 @tetiana12345678

  74. MATCHING LOGIC elixirconf 2016 @tetiana12345678

  75. elixirconf 2016 @tetiana12345678

  76. elixirconf 2016 @tetiana12345678

  77. MATCH RESULTS :all_match :no_match :right_key_wrong_finger :wrong_key_right_finger elixirconf 2016 @tetiana12345678

  78. CORE KEY_PRESS EVENT defmodule Typehero.Core do ... def handle_cast({:key, key,

    id}, state) do EventHandler.key_event(key, id) {:noreply, state} end end elixirconf 2016 @tetiana12345678
  79. CORE FINGER_PRESS EVENT defmodule Typehero.Core do ... def handle_cast({:finger, finger,

    id}, state) do EventHandler.finger_event(finger, id) {:noreply, state} end end elixirconf 2016 @tetiana12345678
  80. EVENTHANDLER MODULE ...WHERE MAGIC HAPPENS

  81. EVENTS STRUCT defmodule Typehero.Events do defstruct key_events: %{}, finger_events: %{}

    end elixirconf 2016 @tetiana12345678
  82. EVENTHANDLER INIT defmodule Typehero.EventHandler do ... alias Typehero.Events def init

    do state = %Events{} {:ok, state} end end elixirconf 2016 @tetiana12345678
  83. KEY_EVENT defmodule Typehero.EventHandler do ... def handle_cast({:key_event, key, id}, state)

    do finger = get_event(:finger, state, id) ... end end elixirconf 2016 @tetiana12345678
  84. GET FINGER EVENT defmodule Typehero.EventHandler do ... defp get_event(:finger, %Events{finger_events:

    events}, id) do Map.get(events, id) end end elixirconf 2016 @tetiana12345678
  85. KEY_EVENT defmodule Typehero.EventHandler do ... def handle_cast({:key_event, key, id}, state)

    do finger = get_event(:finger, state, id) # event | nil new_state = process_key(finger, state, id, key) ... end end elixirconf 2016 @tetiana12345678
  86. PROCESS_KEY defmodule Typehero.EventHandler do ... defp process_key(nil, state, id, key)

    do add_event(:key, state, id, key) end end elixirconf 2016 @tetiana12345678
  87. ADD KEY EVENT TO THE EVENTS STRUCT defmodule Typehero.EventHandler do

    ... defp add_event(:key, %Events{key_events: events}, id, key) do events = Map.put(events, id, key) Map.put(%Events{}, :key_events, events) end end elixirconf 2016 @tetiana12345678
  88. PROCESS_KEY defmodule Typehero.EventHandler do ... defp process_key(finger, state, id, key)

    do match_all(key, finger, id, state) end end elixirconf 2016 @tetiana12345678
  89. MATCH_ALL defmodule Typehero.EventHandler do alias Typehero.Matcher ... defp match_all(key, finger,

    id, state) do result = Matcher.match(key, finger, Core.get_current_letter) ... end end elixirconf 2016 @tetiana12345678
  90. CORE.GET_CURRENT_LETTER defmodule Typehero.Core do ... def handle_call(:get_current_letter, _from, state =

    %{text: text}) do {:reply, String.first(text), state} end end elixirconf 2016 @tetiana12345678
  91. PROCESS_KEY defmodule Typehero.EventHandler do alias Typehero.Matcher ... defp match_all(key, finger,

    id, state) do result = Matcher.match(key, finger, Core.get_current_letter) ... end end elixirconf 2016 @tetiana12345678
  92. INTRODUCING MATCHER MODULE

  93. MATCHER.MATCH defmodule Typehero.Matcher do ... def match(key, finger, letter) do

    key = match_key(key, letter) ... end end elixirconf 2016 @tetiana12345678
  94. MATCHER.MATCH_KEY defmodule Typehero.Matcher do ... defp match_key(key, letter) do key

    == letter end end elixirconf 2016 @tetiana12345678
  95. MATCHER.MATCH defmodule Typehero.Matcher do ... def match(key, finger, letter) do

    key = match_key(key, letter) finger = match_finger(finger, letter) ... end end elixirconf 2016 @tetiana12345678
  96. MATCHER.MATCH_FINGER defmodule Typehero.Matcher do ... defp match_finger(finger, letter) do finger

    == get_finger_number(letter) end defp get_finger_number(key) do Map.get(@finger_numbers, String.to_atom(key)) end end elixirconf 2016 @tetiana12345678
  97. MATCHER FINGER NUMBERS CONSTS defmodule Typehero.Matcher do # Right hand

    fingers @right_index 1 # index @right_middle 2 # middle @right_fourth 3 # fourth @right_pinkie 4 # pinkie @right_thumb 5 # thumb # Left hand fingers @left_index 6 # index @left_middle 7 # middle @left_fourth 8 # fourth @left_pinkie 9 # pinkie @left_thumb 10 # thumb elixirconf 2016 @tetiana12345678
  98. MATCHER FINGER NUMBERS TO LETTERS MAPPING defmodule Typehero.Matcher do ...

    @finger_numbers %{ q: @left_pinkie, a: @left_pinkie, z: @left_pinkie, w: @left_fourth, s: @left_fourth, x: @left_fourth, e: @left_middle, d: @left_middle, c: @left_middle, r: @left_index, f: @left_index, v: @left_index, t: @left_index, g: @left_index, b: @left_index, y: @right_index, h: @right_index, j: @right_index, n: @right_index, u: @right_index, m: @right_index, i: @right_middle, k: @right_middle, o: @right_fourth, l: @right_fourth, p: @right_pinkie } elixirconf 2016 @tetiana12345678
  99. MATCHER.MATCH defmodule Typehero.Matcher do ... def match(key, finger, letter) do

    key = match_key(key, letter) #true or false finger = match_finger(finger, letter) #true or false total_match(key, finger) end end elixirconf 2016 @tetiana12345678
  100. MATCHER.TOTAL_MATCH defmodule Typehero.Matcher do ... defp total_match(true, true), do: :all_match

    defp total_match(true, false), do: :right_key_wrong_finger defp total_match(false, true), do: :wrong_key_right_finger defp total_match(false, false), do: :no_match end elixirconf 2016 @tetiana12345678
  101. PROCESS_KEY defmodule Typehero.EventHandler do alias Typehero.Matcher ... defp match_all(key, finger,

    id, state) do result = Matcher.match(key, finger, Core.get_current_letter) # result of the match :all_match, :no_match, # :right_key_wrong_finger, :wrong_key_right_finger IO.inspect result ... delete_event(state, id) end end elixirconf 2016 @tetiana12345678
  102. DELETE PROCESSED EVENT FROM THE STATE defmodule Typehero.EventHandler do alias

    Typehero.Matcher ... defp delete_event(events, id) do %{events | key_events: Map.delete(events.key_events, id), finger_events: Map.delete(events.finger_events, id) } end end elixirconf 2016 @tetiana12345678
  103. HANDLE_CAST RESPONSE defmodule Typehero.EventHandler do alias Typehero.Matcher ... def handle_cast({:key_event,

    key, id}, state) do finger = get_event(:finger, state, id) new_state = process_key(finger, state, id, key) {:noreply, new_state} end end elixirconf 2016 @tetiana12345678
  104. EVENTHANDLER FINGER EVENT defmodule Typehero.EventHandler do ... def handle_cast({:finger_event, finger,

    id}, state) do key = get_event(:key, state, id) new_state = process_finger(key, state, id, finger) {:noreply, new_state} end end elixirconf 2016 @tetiana12345678
  105. NOTIFY UI OF THE MATCH RESULT

  106. NOTIFY UI DATA FLOW elixirconf 2016 @tetiana12345678

  107. EVENTHANDLER NOTIFY CORE defmodule Typehero.EventHandler do alias Typehero.Matcher ... defp

    match_all(key, finger, id, state) do result = Matcher.match(key, finger, Core.get_current_letter) # result of the match :all_match, :no_match, # :right_key_wrong_finger, :wrong_key_right_finger Core.event_handler_result(%{result: result, id: id}) delete_event(state, id) end end elixirconf 2016 @tetiana12345678
  108. CORE NOTIFY UI defmodule Typehero.Core do ... def handle_cast({:event_handler_result, payload

    = %{result: :all_match}}, state = %{text: text}) do LobbyChannel.handle_in("result", payload, state.socket) state = %{state | text: remove_first_letter(text)} {:noreply, state} end end elixirconf 2016 @tetiana12345678
  109. CORE UPDATE TEXT IN THE STATE defmodule Typehero.Core do ...

    defp remove_first_letter(text) do case String.myers_difference(text, String.first(text)) do [_, del: updated_text] -> updated_text _ -> "" end end end elixirconf 2016 @tetiana12345678
  110. CORE NOTIFY UI defmodule Typehero.Core do ... def handle_cast({:event_handler_result, payload,

    state = %{text: text}) do LobbyChannel.handle_in("result", payload, state.socket) {:noreply, state} end end elixirconf 2016 @tetiana12345678
  111. PHOENIX CHANNEL NOTIFY UI defmodule TypeheroWeb.LobbyChannel do ... def handle_in("result",

    payload, socket) do broadcast socket, "result", payload {:noreply, socket} end end elixirconf 2016 @tetiana12345678
  112. JS RENDERING RESULT IN A FANCY WAY! THANKS TO KEITH...

    elixirconf 2016 @tetiana12345678
  113. DEMO. PLUG IT ALL TOGETHER!

  114. elixirconf 2016 @tetiana12345678

  115. Elixir is inspiring! Getting hardware involved is fun Think about

    your umbrella apps architechture Data flow diagrams is a huge help
  116. GITHUB: TETIANA12345678/TYPEHERO_PROJECT TWITTER: @TETIANA12345678