Building "learn to touch type" glove with Elixir and Arduino

Knowledge share about building "touch typing glove" with Arduino and Elixir.This talk is about my journey from research, choosing tools, making flexible tiny pressure sensors, plugging them into Arduino, reading information from Arduino to the server running Elixir, rendering result in the browser.

Transcript

1. BUILDING A TOUCH TYPING SYSTEM BY TETIANA DUSHENKIVSKA
2. elixirconf 2016 @tetiana12345678
3. 124 WORDS PER MINUTE elixirconf 2016 @tetiana12345678
4. CAN YOU TOUCH TYPE? elixirconf 2016 @tetiana12345678
5. elixirconf 2016 @tetiana12345678
6. BUILDING A TOUCH TYPING SYSTEM BY TETIANA DUSHENKIVSKA
7. 1. Display the text to type 2. Record key press events from the keyboard 3. Record finger events 4. Analyze events 5. Return match result to UI elixirconf 2016 @tetiana12345678
8. ARCHITECTURE
9. 1. Display the text to type elixirconf 2016 @tetiana12345678
10. DISPLAY TEXT elixirconf 2016 @tetiana12345678
11. 2. Record key press events from the keyboard elixirconf 2016 @tetiana12345678
12. RECORD KEY EVENTS elixirconf 2016 @tetiana12345678
13. 3. Record finger events elixirconf 2016 @tetiana12345678
14. RECORD FINGER EVENTS elixirconf 2016 @tetiana12345678
15. 4. Analyze events elixirconf 2016 @tetiana12345678
16. ANALYZE EVENTS elixirconf 2016 @tetiana12345678
17. RESILIENT, ADAPTIVE AND STEADY-STATE WATER SUPPLY NETWORKS elixirconf 2016 @tetiana12345678
18. ANALYZE EVENTS UMBRELLA APPS elixirconf 2016 @tetiana12345678
19. PHOENIX, IS IT YOU??? elixirconf 2016 @tetiana12345678
20. FULL ARCHITECTURE elixirconf 2016 @tetiana12345678
21. IMPLEMENTATION
22. $ mix new typehero_project --umbrella elixirconf 2016 @tetiana12345678
23. $ cd typehero_project/apps elixirconf 2016 @tetiana12345678
24. $ mix phoenix.new typehero_web $ cd typehero_web $ mix deps.get $ mix phoenix.server elixirconf 2016 @tetiana12345678
25. JOIN CHANNEL DATA FLOW elixirconf 2016 @tetiana12345678
26. PHASER WELCOME MESSAGE elixirconf 2016 @tetiana12345678
27. JS JOIN CHANNEL socket.connect() const channel = socket.channel("games:lobby", {}) channel.join() .receive("ok", () => { this.state.start("lobby", true, false, channel) }) elixirconf 2016 @tetiana12345678
28. GENARATE LOBBY CHANNEL $ mix phoenix.gen.channel Lobby elixirconf 2016 @tetiana12345678
29. ADD CHANNEL TO SOCKET defmodule TypeheroWeb.UserSocket do use Phoenix.Socket ... channel "games:lobby", TypeheroWeb.LobbyChannel end elixirconf 2016 @tetiana12345678
30. JOIN CHANNEL CALLBACK defmodule TypeheroWeb.LobbyChannel do use TypeheroWeb.Web, :channel def join("games:lobby", _payload, socket) do {:ok, socket} end end elixirconf 2016 @tetiana12345678
31. DISPLAY TEXT elixirconf 2016 @tetiana12345678
32. GET TEXT DATA FLOW elixirconf 2016 @tetiana12345678
33. JS START GAME start_button.events.onInputDown.add(() => { start_button.destroy() this.channel.push("start_game") }) elixirconf 2016 @tetiana12345678
34. PHOENIX CHANNEL defmodule TypeheroWeb.LobbyChannel do ... def handle_in("start_game", _payload, socket) do text = Typehero.Core.start_game(socket) push socket, "start_game", %{text: text} {:noreply, socket} end end elixirconf 2016 @tetiana12345678
35. CORE UMBRELLA APP $ cd typehero_project/apps $ mix new typehero --sup elixirconf 2016 @tetiana12345678
36. CORE UMBRELLA APP. INIT defmodule Typehero.Core do ... def init(state) do # Get text from Ecto... text_to_type = "in my opinion..." state = %{text: text_to_type, socket: %{}} {:ok, state} end end elixirconf 2016 @tetiana12345678
37. CORE. START_GAME defmodule Typehero.Core do ... def handle_call({:start_game, socket}, _from, state = %{text: text}) do {:reply, text, %{state | socket: socket}} end end elixirconf 2016 @tetiana12345678
38. CORE AS DEPS OF TYPEHERO_WEB defmodule TypeheroWeb.Mixfile do ... def application do [mod: {TypeheroWeb, []}, applications: [..., :typehero]] end end elixirconf 2016 @tetiana12345678
39. JS RENDER TEXT this.channel.on("start_game", this.onStartGame.bind(this)) onStartGame({text}) { this.renderText(text) this.listenKeyboard() } elixirconf 2016 @tetiana12345678
40. SEND KEY EVENTS TO CORE APP
41. KEY PRESS EVENTS DATA FLOW elixirconf 2016 @tetiana12345678
42. JS SEND KEY EVENT onKeyPress({key}) { const event = this.createEvent(key) this.channel.push("key", event) } elixirconf 2016 @tetiana12345678
43. PHOENIX CHANNEL def handle_in("key", %{"key"=> key, "id"=> id}, socket) do Typehero.Core.key_press(key, id) {:noreply, socket} end elixirconf 2016 @tetiana12345678
44. CORE KEY_PRESS EVENT defmodule Typehero.Core do ... alias Typehero.EventHandler def handle_cast({:key, key, id}, state) do EventHandler.key_event(key, id) {:noreply, state} end end elixirconf 2016 @tetiana12345678
45. elixirconf 2016 @tetiana12345678
46. SEND FINGER EVENTS TO CORE
47. ARCHITECTURE elixirconf 2016 @tetiana12345678
48. BUILDING A SENSOR
49. VELOSTAT elixirconf 2016 @tetiana12345678
50. CONDUCTIVE MATERIAL elixirconf 2016 @tetiana12345678
51. CUT SENSOR SHAPE elixirconf 2016 @tetiana12345678
52. MAKE A SENSOR elixirconf 2016 @tetiana12345678
53. ARDUINO elixirconf 2016 @tetiana12345678
54. TEST SENSOR elixirconf 2016 @tetiana12345678
55. C PROGRAM const int analogPin = A0; int finger1 = 0; void setup() { Serial.begin(9600); } void loop() { finger1 = analogRead(analogPin); Serial.println(finger1); } elixirconf 2016 @tetiana12345678
56. VALUES FROM SENSOR elixirconf 2016 @tetiana12345678
57. C PROGRAM. SEND VALUE ONLY ONCE bool finger1_down = false; int finger1_sense = 200; ... void loop() { if (finger1 > finger1_sense && !finger1_down) { finger1_down = true; Serial.print("1"); } else if (finger1 <= finger1_sense) { finger1_down = false; } } elixirconf 2016 @tetiana12345678
58. RINSE AND REPEAT elixirconf 2016 @tetiana12345678
59. CHECK THIS OUT! elixirconf 2016 @tetiana12345678
60. elixirconf 2016 @tetiana12345678
61. elixirconf 2016 @tetiana12345678
62. GIVE ME MORE ELIXIR!
63. TERMINOLOGY elixirconf 2016 @tetiana12345678
64. FINGER EVENTS DATA FLOW elixirconf 2016 @tetiana12345678
65. FINGER EVENTS SERIAL APP
66. GENERATE APP $ cd typehero_project/apps $ mix new typehero_serial --sup elixirconf 2016 @tetiana12345678
67. ADD SERIAL LIBRARY defmodule TypeheroSerial.Mixfile do ... defp deps do [ ..., {:serial, "~> 0.1.0"}] end end elixirconf 2016 @tetiana12345678
68. ADD CORE UMBRELLA APP AS DEPS defmodule TypeheroSerial.Mixfile do ... def application do [applications: [..., :typehero ], ] end end elixirconf 2016 @tetiana12345678
69. SERIAL FORMATTER INIT defmodule TypeheroSerial.Formatter do ... def init(id) do id = 0 {:ok, serial} = Serial.start_link Serial.open(serial, "/dev/cu.usbmodem1421") Serial.set_speed(serial, 9600) Serial.connect(serial) {:ok, id} end end elixirconf 2016 @tetiana12345678
70. FINGER NUMBER SEND FROM ARDUINO defmodule TypeheroSerial.Formatter do ... def handle_info({:elixir_serial, serial, data}, id) do cond do data =~ ~r/1/ -> Typehero.Core.finger_press(1, id) data =~ ~r/2/ -> Typehero.Core.finger_press(2, id) ... true -> nil end {:noreply, (id + 1)} end end elixirconf 2016 @tetiana12345678
71. elixirconf 2016 @tetiana12345678
72. CORE UMBRELLA APP
73. DATA FLOW elixirconf 2016 @tetiana12345678
74. MATCHING LOGIC elixirconf 2016 @tetiana12345678
75. elixirconf 2016 @tetiana12345678
76. elixirconf 2016 @tetiana12345678
77. MATCH RESULTS :all_match :no_match :right_key_wrong_finger :wrong_key_right_finger elixirconf 2016 @tetiana12345678
78. CORE KEY_PRESS EVENT defmodule Typehero.Core do ... def handle_cast({:key, key, id}, state) do EventHandler.key_event(key, id) {:noreply, state} end end elixirconf 2016 @tetiana12345678
79. CORE FINGER_PRESS EVENT defmodule Typehero.Core do ... def handle_cast({:finger, finger, id}, state) do EventHandler.finger_event(finger, id) {:noreply, state} end end elixirconf 2016 @tetiana12345678
80. EVENTHANDLER MODULE ...WHERE MAGIC HAPPENS
81. EVENTS STRUCT defmodule Typehero.Events do defstruct key_events: %{}, finger_events: %{} end elixirconf 2016 @tetiana12345678
82. EVENTHANDLER INIT defmodule Typehero.EventHandler do ... alias Typehero.Events def init do state = %Events{} {:ok, state} end end elixirconf 2016 @tetiana12345678
83. KEY_EVENT defmodule Typehero.EventHandler do ... def handle_cast({:key_event, key, id}, state) do finger = get_event(:finger, state, id) ... end end elixirconf 2016 @tetiana12345678
84. GET FINGER EVENT defmodule Typehero.EventHandler do ... defp get_event(:finger, %Events{finger_events: events}, id) do Map.get(events, id) end end elixirconf 2016 @tetiana12345678
85. KEY_EVENT defmodule Typehero.EventHandler do ... def handle_cast({:key_event, key, id}, state) do finger = get_event(:finger, state, id) # event | nil new_state = process_key(finger, state, id, key) ... end end elixirconf 2016 @tetiana12345678
86. PROCESS_KEY defmodule Typehero.EventHandler do ... defp process_key(nil, state, id, key) do add_event(:key, state, id, key) end end elixirconf 2016 @tetiana12345678
87. ADD KEY EVENT TO THE EVENTS STRUCT defmodule Typehero.EventHandler do ... defp add_event(:key, %Events{key_events: events}, id, key) do events = Map.put(events, id, key) Map.put(%Events{}, :key_events, events) end end elixirconf 2016 @tetiana12345678
88. PROCESS_KEY defmodule Typehero.EventHandler do ... defp process_key(finger, state, id, key) do match_all(key, finger, id, state) end end elixirconf 2016 @tetiana12345678
89. MATCH_ALL defmodule Typehero.EventHandler do alias Typehero.Matcher ... defp match_all(key, finger, id, state) do result = Matcher.match(key, finger, Core.get_current_letter) ... end end elixirconf 2016 @tetiana12345678
90. CORE.GET_CURRENT_LETTER defmodule Typehero.Core do ... def handle_call(:get_current_letter, _from, state = %{text: text}) do {:reply, String.first(text), state} end end elixirconf 2016 @tetiana12345678
91. PROCESS_KEY defmodule Typehero.EventHandler do alias Typehero.Matcher ... defp match_all(key, finger, id, state) do result = Matcher.match(key, finger, Core.get_current_letter) ... end end elixirconf 2016 @tetiana12345678
92. INTRODUCING MATCHER MODULE
93. MATCHER.MATCH defmodule Typehero.Matcher do ... def match(key, finger, letter) do key = match_key(key, letter) ... end end elixirconf 2016 @tetiana12345678
94. MATCHER.MATCH_KEY defmodule Typehero.Matcher do ... defp match_key(key, letter) do key == letter end end elixirconf 2016 @tetiana12345678
95. MATCHER.MATCH defmodule Typehero.Matcher do ... def match(key, finger, letter) do key = match_key(key, letter) finger = match_finger(finger, letter) ... end end elixirconf 2016 @tetiana12345678
96. MATCHER.MATCH_FINGER defmodule Typehero.Matcher do ... defp match_finger(finger, letter) do finger == get_finger_number(letter) end defp get_finger_number(key) do Map.get(@finger_numbers, String.to_atom(key)) end end elixirconf 2016 @tetiana12345678
97. MATCHER FINGER NUMBERS CONSTS defmodule Typehero.Matcher do # Right hand fingers @right_index 1 # index @right_middle 2 # middle @right_fourth 3 # fourth @right_pinkie 4 # pinkie @right_thumb 5 # thumb # Left hand fingers @left_index 6 # index @left_middle 7 # middle @left_fourth 8 # fourth @left_pinkie 9 # pinkie @left_thumb 10 # thumb elixirconf 2016 @tetiana12345678
98. MATCHER FINGER NUMBERS TO LETTERS MAPPING defmodule Typehero.Matcher do ... @finger_numbers %{ q: @left_pinkie, a: @left_pinkie, z: @left_pinkie, w: @left_fourth, s: @left_fourth, x: @left_fourth, e: @left_middle, d: @left_middle, c: @left_middle, r: @left_index, f: @left_index, v: @left_index, t: @left_index, g: @left_index, b: @left_index, y: @right_index, h: @right_index, j: @right_index, n: @right_index, u: @right_index, m: @right_index, i: @right_middle, k: @right_middle, o: @right_fourth, l: @right_fourth, p: @right_pinkie } elixirconf 2016 @tetiana12345678
99. MATCHER.MATCH defmodule Typehero.Matcher do ... def match(key, finger, letter) do key = match_key(key, letter) #true or false finger = match_finger(finger, letter) #true or false total_match(key, finger) end end elixirconf 2016 @tetiana12345678
100. MATCHER.TOTAL_MATCH defmodule Typehero.Matcher do ... defp total_match(true, true), do: :all_match defp total_match(true, false), do: :right_key_wrong_finger defp total_match(false, true), do: :wrong_key_right_finger defp total_match(false, false), do: :no_match end elixirconf 2016 @tetiana12345678
101. PROCESS_KEY defmodule Typehero.EventHandler do alias Typehero.Matcher ... defp match_all(key, finger, id, state) do result = Matcher.match(key, finger, Core.get_current_letter) # result of the match :all_match, :no_match, # :right_key_wrong_finger, :wrong_key_right_finger IO.inspect result ... delete_event(state, id) end end elixirconf 2016 @tetiana12345678
102. DELETE PROCESSED EVENT FROM THE STATE defmodule Typehero.EventHandler do alias Typehero.Matcher ... defp delete_event(events, id) do %{events | key_events: Map.delete(events.key_events, id), finger_events: Map.delete(events.finger_events, id) } end end elixirconf 2016 @tetiana12345678
103. HANDLE_CAST RESPONSE defmodule Typehero.EventHandler do alias Typehero.Matcher ... def handle_cast({:key_event, key, id}, state) do finger = get_event(:finger, state, id) new_state = process_key(finger, state, id, key) {:noreply, new_state} end end elixirconf 2016 @tetiana12345678
104. EVENTHANDLER FINGER EVENT defmodule Typehero.EventHandler do ... def handle_cast({:finger_event, finger, id}, state) do key = get_event(:key, state, id) new_state = process_finger(key, state, id, finger) {:noreply, new_state} end end elixirconf 2016 @tetiana12345678
105. NOTIFY UI OF THE MATCH RESULT
106. NOTIFY UI DATA FLOW elixirconf 2016 @tetiana12345678
107. EVENTHANDLER NOTIFY CORE defmodule Typehero.EventHandler do alias Typehero.Matcher ... defp match_all(key, finger, id, state) do result = Matcher.match(key, finger, Core.get_current_letter) # result of the match :all_match, :no_match, # :right_key_wrong_finger, :wrong_key_right_finger Core.event_handler_result(%{result: result, id: id}) delete_event(state, id) end end elixirconf 2016 @tetiana12345678
108. CORE NOTIFY UI defmodule Typehero.Core do ... def handle_cast({:event_handler_result, payload = %{result: :all_match}}, state = %{text: text}) do LobbyChannel.handle_in("result", payload, state.socket) state = %{state | text: remove_first_letter(text)} {:noreply, state} end end elixirconf 2016 @tetiana12345678
109. CORE UPDATE TEXT IN THE STATE defmodule Typehero.Core do ... defp remove_first_letter(text) do case String.myers_difference(text, String.first(text)) do [_, del: updated_text] -> updated_text _ -> "" end end end elixirconf 2016 @tetiana12345678
110. CORE NOTIFY UI defmodule Typehero.Core do ... def handle_cast({:event_handler_result, payload, state = %{text: text}) do LobbyChannel.handle_in("result", payload, state.socket) {:noreply, state} end end elixirconf 2016 @tetiana12345678
111. PHOENIX CHANNEL NOTIFY UI defmodule TypeheroWeb.LobbyChannel do ... def handle_in("result", payload, socket) do broadcast socket, "result", payload {:noreply, socket} end end elixirconf 2016 @tetiana12345678
112. JS RENDERING RESULT IN A FANCY WAY! THANKS TO KEITH... elixirconf 2016 @tetiana12345678
113. DEMO. PLUG IT ALL TOGETHER!
114. elixirconf 2016 @tetiana12345678
115. Elixir is inspiring! Getting hardware involved is fun Think about your umbrella apps architechture Data flow diagrams is a huge help
116. GITHUB: TETIANA12345678/TYPEHERO_PROJECT TWITTER: @TETIANA12345678