Building "learn to touch type" glove with Elixir and Arduino

Slide 1

Slide 1 text

BUILDING A TOUCH TYPING SYSTEM BY TETIANA DUSHENKIVSKA

Slide 2

Slide 2 text

elixirconf 2016 @tetiana12345678

Slide 3

Slide 3 text

124 WORDS PER MINUTE elixirconf 2016 @tetiana12345678

Slide 4

Slide 4 text

CAN YOU TOUCH TYPE? elixirconf 2016 @tetiana12345678

Slide 5

Slide 5 text

elixirconf 2016 @tetiana12345678

Slide 6

Slide 6 text

BUILDING A TOUCH TYPING SYSTEM BY TETIANA DUSHENKIVSKA

Slide 7

Slide 7 text

1. Display the text to type 2. Record key press events from the keyboard 3. Record finger events 4. Analyze events 5. Return match result to UI elixirconf 2016 @tetiana12345678

Slide 8

Slide 8 text

ARCHITECTURE

Slide 9

Slide 9 text

1. Display the text to type elixirconf 2016 @tetiana12345678

Slide 10

Slide 10 text

DISPLAY TEXT elixirconf 2016 @tetiana12345678

Slide 11

Slide 11 text

2. Record key press events from the keyboard elixirconf 2016 @tetiana12345678

Slide 12

Slide 12 text

RECORD KEY EVENTS elixirconf 2016 @tetiana12345678

Slide 13

Slide 13 text

3. Record finger events elixirconf 2016 @tetiana12345678

Slide 14

Slide 14 text

RECORD FINGER EVENTS elixirconf 2016 @tetiana12345678

Slide 15

Slide 15 text

4. Analyze events elixirconf 2016 @tetiana12345678

Slide 16

Slide 16 text

ANALYZE EVENTS elixirconf 2016 @tetiana12345678

Slide 17

Slide 17 text

RESILIENT, ADAPTIVE AND STEADY-STATE WATER SUPPLY NETWORKS elixirconf 2016 @tetiana12345678

Slide 18

Slide 18 text

ANALYZE EVENTS UMBRELLA APPS elixirconf 2016 @tetiana12345678

Slide 19

Slide 19 text

PHOENIX, IS IT YOU??? elixirconf 2016 @tetiana12345678

Slide 20

Slide 20 text

FULL ARCHITECTURE elixirconf 2016 @tetiana12345678

Slide 21

Slide 21 text

IMPLEMENTATION

Slide 22

Slide 22 text

$ mix new typehero_project --umbrella elixirconf 2016 @tetiana12345678

Slide 23

Slide 23 text

$ cd typehero_project/apps elixirconf 2016 @tetiana12345678

Slide 24

Slide 24 text

$ mix phoenix.new typehero_web $ cd typehero_web $ mix deps.get $ mix phoenix.server elixirconf 2016 @tetiana12345678

Slide 25

Slide 25 text

JOIN CHANNEL DATA FLOW elixirconf 2016 @tetiana12345678

Slide 26

Slide 26 text

PHASER WELCOME MESSAGE elixirconf 2016 @tetiana12345678

Slide 27

Slide 27 text

JS JOIN CHANNEL socket.connect() const channel = socket.channel("games:lobby", {}) channel.join() .receive("ok", () => { this.state.start("lobby", true, false, channel) }) elixirconf 2016 @tetiana12345678

Slide 28

Slide 28 text

GENARATE LOBBY CHANNEL $ mix phoenix.gen.channel Lobby elixirconf 2016 @tetiana12345678

Slide 29

Slide 29 text

ADD CHANNEL TO SOCKET defmodule TypeheroWeb.UserSocket do use Phoenix.Socket ... channel "games:lobby", TypeheroWeb.LobbyChannel end elixirconf 2016 @tetiana12345678

Slide 30

Slide 30 text

JOIN CHANNEL CALLBACK defmodule TypeheroWeb.LobbyChannel do use TypeheroWeb.Web, :channel def join("games:lobby", _payload, socket) do {:ok, socket} end end elixirconf 2016 @tetiana12345678

Slide 31

Slide 31 text

DISPLAY TEXT elixirconf 2016 @tetiana12345678

Slide 32

Slide 32 text

GET TEXT DATA FLOW elixirconf 2016 @tetiana12345678

Slide 33

Slide 33 text

JS START GAME start_button.events.onInputDown.add(() => { start_button.destroy() this.channel.push("start_game") }) elixirconf 2016 @tetiana12345678

Slide 34

Slide 34 text

PHOENIX CHANNEL defmodule TypeheroWeb.LobbyChannel do ... def handle_in("start_game", _payload, socket) do text = Typehero.Core.start_game(socket) push socket, "start_game", %{text: text} {:noreply, socket} end end elixirconf 2016 @tetiana12345678

Slide 35

Slide 35 text

CORE UMBRELLA APP $ cd typehero_project/apps $ mix new typehero --sup elixirconf 2016 @tetiana12345678

Slide 36

Slide 36 text

CORE UMBRELLA APP. INIT defmodule Typehero.Core do ... def init(state) do # Get text from Ecto... text_to_type = "in my opinion..." state = %{text: text_to_type, socket: %{}} {:ok, state} end end elixirconf 2016 @tetiana12345678

Slide 37

Slide 37 text

CORE. START_GAME defmodule Typehero.Core do ... def handle_call({:start_game, socket}, _from, state = %{text: text}) do {:reply, text, %{state | socket: socket}} end end elixirconf 2016 @tetiana12345678

Slide 38

Slide 38 text

CORE AS DEPS OF TYPEHERO_WEB defmodule TypeheroWeb.Mixfile do ... def application do [mod: {TypeheroWeb, []}, applications: [..., :typehero]] end end elixirconf 2016 @tetiana12345678

Slide 39

Slide 39 text

JS RENDER TEXT this.channel.on("start_game", this.onStartGame.bind(this)) onStartGame({text}) { this.renderText(text) this.listenKeyboard() } elixirconf 2016 @tetiana12345678

Slide 40

Slide 40 text

SEND KEY EVENTS TO CORE APP

Slide 41

Slide 41 text

KEY PRESS EVENTS DATA FLOW elixirconf 2016 @tetiana12345678

Slide 42

Slide 42 text

JS SEND KEY EVENT onKeyPress({key}) { const event = this.createEvent(key) this.channel.push("key", event) } elixirconf 2016 @tetiana12345678

Slide 43

Slide 43 text

PHOENIX CHANNEL def handle_in("key", %{"key"=> key, "id"=> id}, socket) do Typehero.Core.key_press(key, id) {:noreply, socket} end elixirconf 2016 @tetiana12345678

Slide 44

Slide 44 text

CORE KEY_PRESS EVENT defmodule Typehero.Core do ... alias Typehero.EventHandler def handle_cast({:key, key, id}, state) do EventHandler.key_event(key, id) {:noreply, state} end end elixirconf 2016 @tetiana12345678

Slide 45

Slide 45 text

elixirconf 2016 @tetiana12345678

Slide 46

Slide 46 text

SEND FINGER EVENTS TO CORE

Slide 47

Slide 47 text

ARCHITECTURE elixirconf 2016 @tetiana12345678

Slide 48

Slide 48 text

BUILDING A SENSOR

Slide 49

Slide 49 text

VELOSTAT elixirconf 2016 @tetiana12345678

Slide 50

Slide 50 text

CONDUCTIVE MATERIAL elixirconf 2016 @tetiana12345678

Slide 51

Slide 51 text

CUT SENSOR SHAPE elixirconf 2016 @tetiana12345678

Slide 52

Slide 52 text

MAKE A SENSOR elixirconf 2016 @tetiana12345678

Slide 53

Slide 53 text

ARDUINO elixirconf 2016 @tetiana12345678

Slide 54

Slide 54 text

TEST SENSOR elixirconf 2016 @tetiana12345678

Slide 55

Slide 55 text

C PROGRAM const int analogPin = A0; int finger1 = 0; void setup() { Serial.begin(9600); } void loop() { finger1 = analogRead(analogPin); Serial.println(finger1); } elixirconf 2016 @tetiana12345678

Slide 56

Slide 56 text

VALUES FROM SENSOR elixirconf 2016 @tetiana12345678

Slide 57

Slide 57 text

C PROGRAM. SEND VALUE ONLY ONCE bool finger1_down = false; int finger1_sense = 200; ... void loop() { if (finger1 > finger1_sense && !finger1_down) { finger1_down = true; Serial.print("1"); } else if (finger1 <= finger1_sense) { finger1_down = false; } } elixirconf 2016 @tetiana12345678

Slide 58

Slide 58 text

RINSE AND REPEAT elixirconf 2016 @tetiana12345678

Slide 59

Slide 59 text

CHECK THIS OUT! elixirconf 2016 @tetiana12345678

Slide 60

Slide 60 text

elixirconf 2016 @tetiana12345678

Slide 61

Slide 61 text

elixirconf 2016 @tetiana12345678

Slide 62

Slide 62 text

GIVE ME MORE ELIXIR!

Slide 63

Slide 63 text

TERMINOLOGY elixirconf 2016 @tetiana12345678

Slide 64

Slide 64 text

FINGER EVENTS DATA FLOW elixirconf 2016 @tetiana12345678

Slide 65

Slide 65 text

FINGER EVENTS SERIAL APP

Slide 66

Slide 66 text

GENERATE APP $ cd typehero_project/apps $ mix new typehero_serial --sup elixirconf 2016 @tetiana12345678

Slide 67

Slide 67 text

ADD SERIAL LIBRARY defmodule TypeheroSerial.Mixfile do ... defp deps do [ ..., {:serial, "~> 0.1.0"}] end end elixirconf 2016 @tetiana12345678

Slide 68

Slide 68 text

ADD CORE UMBRELLA APP AS DEPS defmodule TypeheroSerial.Mixfile do ... def application do [applications: [..., :typehero ], ] end end elixirconf 2016 @tetiana12345678

Slide 69

Slide 69 text

SERIAL FORMATTER INIT defmodule TypeheroSerial.Formatter do ... def init(id) do id = 0 {:ok, serial} = Serial.start_link Serial.open(serial, "/dev/cu.usbmodem1421") Serial.set_speed(serial, 9600) Serial.connect(serial) {:ok, id} end end elixirconf 2016 @tetiana12345678

Slide 70

Slide 70 text

FINGER NUMBER SEND FROM ARDUINO defmodule TypeheroSerial.Formatter do ... def handle_info({:elixir_serial, serial, data}, id) do cond do data =~ ~r/1/ -> Typehero.Core.finger_press(1, id) data =~ ~r/2/ -> Typehero.Core.finger_press(2, id) ... true -> nil end {:noreply, (id + 1)} end end elixirconf 2016 @tetiana12345678

Slide 71

Slide 71 text

elixirconf 2016 @tetiana12345678

Slide 72

Slide 72 text

CORE UMBRELLA APP

Slide 73

Slide 73 text

DATA FLOW elixirconf 2016 @tetiana12345678

Slide 74

Slide 74 text

MATCHING LOGIC elixirconf 2016 @tetiana12345678

Slide 75

Slide 75 text

elixirconf 2016 @tetiana12345678

Slide 76

Slide 76 text

elixirconf 2016 @tetiana12345678

Slide 77

Slide 77 text

MATCH RESULTS :all_match :no_match :right_key_wrong_finger :wrong_key_right_finger elixirconf 2016 @tetiana12345678

Slide 78

Slide 78 text

CORE KEY_PRESS EVENT defmodule Typehero.Core do ... def handle_cast({:key, key, id}, state) do EventHandler.key_event(key, id) {:noreply, state} end end elixirconf 2016 @tetiana12345678

Slide 79

Slide 79 text

CORE FINGER_PRESS EVENT defmodule Typehero.Core do ... def handle_cast({:finger, finger, id}, state) do EventHandler.finger_event(finger, id) {:noreply, state} end end elixirconf 2016 @tetiana12345678

Slide 80

Slide 80 text

EVENTHANDLER MODULE ...WHERE MAGIC HAPPENS

Slide 81

Slide 81 text

EVENTS STRUCT defmodule Typehero.Events do defstruct key_events: %{}, finger_events: %{} end elixirconf 2016 @tetiana12345678

Slide 82

Slide 82 text

EVENTHANDLER INIT defmodule Typehero.EventHandler do ... alias Typehero.Events def init do state = %Events{} {:ok, state} end end elixirconf 2016 @tetiana12345678

Slide 83

Slide 83 text

KEY_EVENT defmodule Typehero.EventHandler do ... def handle_cast({:key_event, key, id}, state) do finger = get_event(:finger, state, id) ... end end elixirconf 2016 @tetiana12345678

Slide 84

Slide 84 text

GET FINGER EVENT defmodule Typehero.EventHandler do ... defp get_event(:finger, %Events{finger_events: events}, id) do Map.get(events, id) end end elixirconf 2016 @tetiana12345678

Slide 85

Slide 85 text

KEY_EVENT defmodule Typehero.EventHandler do ... def handle_cast({:key_event, key, id}, state) do finger = get_event(:finger, state, id) # event | nil new_state = process_key(finger, state, id, key) ... end end elixirconf 2016 @tetiana12345678

Slide 86

Slide 86 text

PROCESS_KEY defmodule Typehero.EventHandler do ... defp process_key(nil, state, id, key) do add_event(:key, state, id, key) end end elixirconf 2016 @tetiana12345678

Slide 87

Slide 87 text

ADD KEY EVENT TO THE EVENTS STRUCT defmodule Typehero.EventHandler do ... defp add_event(:key, %Events{key_events: events}, id, key) do events = Map.put(events, id, key) Map.put(%Events{}, :key_events, events) end end elixirconf 2016 @tetiana12345678

Slide 88

Slide 88 text

PROCESS_KEY defmodule Typehero.EventHandler do ... defp process_key(finger, state, id, key) do match_all(key, finger, id, state) end end elixirconf 2016 @tetiana12345678

Slide 89

Slide 89 text

MATCH_ALL defmodule Typehero.EventHandler do alias Typehero.Matcher ... defp match_all(key, finger, id, state) do result = Matcher.match(key, finger, Core.get_current_letter) ... end end elixirconf 2016 @tetiana12345678

Slide 90

Slide 90 text

CORE.GET_CURRENT_LETTER defmodule Typehero.Core do ... def handle_call(:get_current_letter, _from, state = %{text: text}) do {:reply, String.first(text), state} end end elixirconf 2016 @tetiana12345678

Slide 91

Slide 91 text

Slide 92

Slide 92 text

INTRODUCING MATCHER MODULE

Slide 93

Slide 93 text

MATCHER.MATCH defmodule Typehero.Matcher do ... def match(key, finger, letter) do key = match_key(key, letter) ... end end elixirconf 2016 @tetiana12345678

Slide 94

Slide 94 text

MATCHER.MATCH_KEY defmodule Typehero.Matcher do ... defp match_key(key, letter) do key == letter end end elixirconf 2016 @tetiana12345678

Slide 95

Slide 95 text

MATCHER.MATCH defmodule Typehero.Matcher do ... def match(key, finger, letter) do key = match_key(key, letter) finger = match_finger(finger, letter) ... end end elixirconf 2016 @tetiana12345678

Slide 96

Slide 96 text

MATCHER.MATCH_FINGER defmodule Typehero.Matcher do ... defp match_finger(finger, letter) do finger == get_finger_number(letter) end defp get_finger_number(key) do Map.get(@finger_numbers, String.to_atom(key)) end end elixirconf 2016 @tetiana12345678

Slide 97

Slide 97 text

MATCHER FINGER NUMBERS CONSTS defmodule Typehero.Matcher do # Right hand fingers @right_index 1 # index @right_middle 2 # middle @right_fourth 3 # fourth @right_pinkie 4 # pinkie @right_thumb 5 # thumb # Left hand fingers @left_index 6 # index @left_middle 7 # middle @left_fourth 8 # fourth @left_pinkie 9 # pinkie @left_thumb 10 # thumb elixirconf 2016 @tetiana12345678

Slide 98

Slide 98 text

MATCHER FINGER NUMBERS TO LETTERS MAPPING defmodule Typehero.Matcher do ... @finger_numbers %{ q: @left_pinkie, a: @left_pinkie, z: @left_pinkie, w: @left_fourth, s: @left_fourth, x: @left_fourth, e: @left_middle, d: @left_middle, c: @left_middle, r: @left_index, f: @left_index, v: @left_index, t: @left_index, g: @left_index, b: @left_index, y: @right_index, h: @right_index, j: @right_index, n: @right_index, u: @right_index, m: @right_index, i: @right_middle, k: @right_middle, o: @right_fourth, l: @right_fourth, p: @right_pinkie } elixirconf 2016 @tetiana12345678

Slide 99

Slide 99 text

MATCHER.MATCH defmodule Typehero.Matcher do ... def match(key, finger, letter) do key = match_key(key, letter) #true or false finger = match_finger(finger, letter) #true or false total_match(key, finger) end end elixirconf 2016 @tetiana12345678

Slide 100

Slide 100 text

MATCHER.TOTAL_MATCH defmodule Typehero.Matcher do ... defp total_match(true, true), do: :all_match defp total_match(true, false), do: :right_key_wrong_finger defp total_match(false, true), do: :wrong_key_right_finger defp total_match(false, false), do: :no_match end elixirconf 2016 @tetiana12345678

Slide 101

Slide 101 text

PROCESS_KEY defmodule Typehero.EventHandler do alias Typehero.Matcher ... defp match_all(key, finger, id, state) do result = Matcher.match(key, finger, Core.get_current_letter) # result of the match :all_match, :no_match, # :right_key_wrong_finger, :wrong_key_right_finger IO.inspect result ... delete_event(state, id) end end elixirconf 2016 @tetiana12345678

Slide 102

Slide 102 text

DELETE PROCESSED EVENT FROM THE STATE defmodule Typehero.EventHandler do alias Typehero.Matcher ... defp delete_event(events, id) do %{events | key_events: Map.delete(events.key_events, id), finger_events: Map.delete(events.finger_events, id) } end end elixirconf 2016 @tetiana12345678

Slide 103

Slide 103 text

HANDLE_CAST RESPONSE defmodule Typehero.EventHandler do alias Typehero.Matcher ... def handle_cast({:key_event, key, id}, state) do finger = get_event(:finger, state, id) new_state = process_key(finger, state, id, key) {:noreply, new_state} end end elixirconf 2016 @tetiana12345678

Slide 104

Slide 104 text

EVENTHANDLER FINGER EVENT defmodule Typehero.EventHandler do ... def handle_cast({:finger_event, finger, id}, state) do key = get_event(:key, state, id) new_state = process_finger(key, state, id, finger) {:noreply, new_state} end end elixirconf 2016 @tetiana12345678

Slide 105

Slide 105 text

NOTIFY UI OF THE MATCH RESULT

Slide 106

Slide 106 text

NOTIFY UI DATA FLOW elixirconf 2016 @tetiana12345678

Slide 107

Slide 107 text

EVENTHANDLER NOTIFY CORE defmodule Typehero.EventHandler do alias Typehero.Matcher ... defp match_all(key, finger, id, state) do result = Matcher.match(key, finger, Core.get_current_letter) # result of the match :all_match, :no_match, # :right_key_wrong_finger, :wrong_key_right_finger Core.event_handler_result(%{result: result, id: id}) delete_event(state, id) end end elixirconf 2016 @tetiana12345678

Slide 108

Slide 108 text

CORE NOTIFY UI defmodule Typehero.Core do ... def handle_cast({:event_handler_result, payload = %{result: :all_match}}, state = %{text: text}) do LobbyChannel.handle_in("result", payload, state.socket) state = %{state | text: remove_first_letter(text)} {:noreply, state} end end elixirconf 2016 @tetiana12345678

Slide 109

Slide 109 text

CORE UPDATE TEXT IN THE STATE defmodule Typehero.Core do ... defp remove_first_letter(text) do case String.myers_difference(text, String.first(text)) do [_, del: updated_text] -> updated_text _ -> "" end end end elixirconf 2016 @tetiana12345678

Slide 110

Slide 110 text

CORE NOTIFY UI defmodule Typehero.Core do ... def handle_cast({:event_handler_result, payload, state = %{text: text}) do LobbyChannel.handle_in("result", payload, state.socket) {:noreply, state} end end elixirconf 2016 @tetiana12345678

Slide 111

Slide 111 text

PHOENIX CHANNEL NOTIFY UI defmodule TypeheroWeb.LobbyChannel do ... def handle_in("result", payload, socket) do broadcast socket, "result", payload {:noreply, socket} end end elixirconf 2016 @tetiana12345678

Slide 112

Slide 112 text

JS RENDERING RESULT IN A FANCY WAY! THANKS TO KEITH... elixirconf 2016 @tetiana12345678

Slide 113

Slide 113 text

DEMO. PLUG IT ALL TOGETHER!

Slide 114

Slide 114 text

elixirconf 2016 @tetiana12345678

Slide 115

Slide 115 text

Elixir is inspiring! Getting hardware involved is fun Think about your umbrella apps architechture Data flow diagrams is a huge help

Slide 116

Slide 116 text

GITHUB: TETIANA12345678/TYPEHERO_PROJECT TWITTER: @TETIANA12345678