Slide 1
Slide 1 text
DIY Your Touchpad Experience
Building Your Own Gestures
Kohei Yamada
Slide 2
Slide 2 text
About Me
Kohei Yamada
GitHub @iberianpig
Twitter @nukumaro22
自己紹介
Slide 3
Slide 3 text
Do you like improving your environment?
環境カイゼンは好きですか？
> Many programmers, myself included, like to customize our environment to
our liking. They are happy when it improves their efficiency, even if only a little.
> You could call them a kind of
"Environment Improvement Geek".
> Matz
私を含めて多くのプログラマは，自分の環境を自分の好みに合わせてカスタマイズすることを好みます。それによってほんのわずかでも自分の効率が向上すると喜びを感じるのです。
ある種の
「環境改善オタク」と呼んでもよいでしょう。
［改訂新版］Emacs実践入門──思考を直感的にコード化し、開発を加速する より 本書に寄せて
まつもと ゆきひろ
Slide 4
Slide 4 text
My Development Environment
Dell XPS 13 Ubuntu 22.04 / X11 Vim
LAPTOP OS Editor
私の開発環境
Slide 5
Slide 5 text
Environment Improvement
環境の改善
Daily yak shaving builds plugins
Slide 6
Slide 6 text
About Fusuma
Slide 7
Slide 7 text
A Gem that enables touchpad gestures on Linux.
About Fusuma
Fusumaについて
Slide 8
Slide 8 text
About Fusuma
Fusumaについて
🐧 On Linux touchpads
✋ Swipe
👌 Pinch-to-zoom
Slide 9
Slide 9 text
No content
Slide 10
Slide 10 text
Fusuma Features
Fusumaの特徴
● Easy installation with RubyGems
● Defining Gestures and Actions in YAML
● Extension of gesture recognition by Plugin system
Slide 11
Slide 11 text
How to use Fusuma
Fusumaの使い方
Install libinput-tools
1
Add $USER to Input group
2
Customize ~/.config/fusuma/config.yml
5
Run $ fusuma from the CLI
4
Install Fusuma with $ gem i fusuma
3
Slide 12
Slide 12 text
Customize gestures and actions
ジェスチャとアクションのカスタマイズ
● Swipe
● Pinch
● Rotate
● Hold
●
● command
● sendkey(plugin)
● wmctrl(plugin)
Gesture
Action
Slide 13
Slide 13 text
DEMO
● 4-finger Swipe
○ Switch Workspace
● 3-finger swipe
○ Open/Close browser tab
○ Back/forward browser history
● 3-finger dragging
デモ
Slide 14
Slide 14 text
How I make Fusuma
Slide 15
Slide 15 text
HW X server
Input Event Flow in Linux
LinuxでのInputイベントの流れ
Kernel
device evdev libevdev libinput xf86-input-libinput
X client
App
input event Simulate keyboard
Fusuma
Input Event Flow
Slide 16
Slide 16 text
Touchpad drivers on Linux
● Well-known touchpad drivers, around 2010
● only X.Org
● Many Configuration options
● Gestures were not supported
● Release of libinput 1.0 in 2015
● Used in Wayland and X.Org
● No configuration file
● Improvements to gesture support
Synaptics
2015
libinput
2010 2020
Libinput
Synaptics
Linuxのタッチパッドドライバ
Slide 17
Slide 17 text
Seeking multitouch Gestures
● Desire for Mac-like trackpad gestures
● Test different laptops' gesture behavior in
electronics store
Prototype on Synaptics
Synapticsドライバでのプロトタイプ
Released xSwipe in 2012
● Discovered logging capability with
`$ synclient -m` for synaptics
● Read and parse logs in loop
● Detect direction and send shortcut
● Written in Perl
Synaptics
2015
libinput
2010 2020
My first DIY
Slide 18
Slide 18 text
Synaptics to Libinput
SynapticsからLibinputへ
Synaptics replaced by libinput
● synaptics driver in maintenance mode
● xSwipe only worked with Synaptics driver
● Frustration with inability to use gestures
My First Gem
Synaptics
2015
libinput
2010 2020
Released Fusuma in 2016
● Use `libinput debug-events`
● Rewrite xSwipe in Ruby for libinput
● `rake release`!
🪦
Slide 19
Slide 19 text
Gesture Recognition
Slide 20
Slide 20 text
Gesture Recognition
ジェスチャ認識
Pipeline
Input Buffer Detector Executor
Libinput Gesture
Swipe
Pinch
Rotate
Hold
Command
Touchpad
Keyboard
1 4
2 3
Slide 21
Slide 21 text
Input
libinput-tools CLI as input
inputとしてのlibinput-tools
`libinput debug-events`
Slide 22
Slide 22 text
● Execute libinput using Open3.popen3
● Output lines without buffering using
`stdbuf -oL`
● convert a line to a event
$stdout to event in real-time
リアルタイムで標準出力をイベントに
Input
Slide 23
Slide 23 text
Buffering Events
イベントを溜める
● Aggregate events
● Multiple buffers depending on input
Buffer
Input Buffer
Libinput Gesture
Detector
Swipe
Executor
Command
Keypress
Gesture State Time fingers delta x delta y
SWIPE BEGIN +37.492s 3 10.23 -1.14
SWIPE UPDATE. +37.492s 3 10.54 -1.17
SWIPE UPDATE. +37.499s 3 7.62 -1.76
SWIPE UPDATE. +37.506s 3 10.25 -2.64
Slide 24
Slide 24 text
Gesture detection
ジェスチャーの検知
Gesture State Time fingers delta x delta y
SWIPE BEGIN +37.492s 3 10.23 -1.14
SWIPE UPDATE. +37.492s 3 10.54 -1.17
SWIPE UPDATE. +37.499s 3 7.62 -1.76
SWIPE UPDATE. +37.506s 3 10.25 -2.64
Detect a left swipe with three fingers
=> `swipe / 3 / right`
Compare the average movement with threshold
Detector
Create Event with search index
Slide 25
Slide 25 text
Detect a left swipe with three fingers
Retrieving Action from Config
設定ファイルからアクションを導出
config.yml
`swipe / 3 / right` => { command: ‘xdotool key alt+Left’ }
Executor
Slide 26
Slide 26 text
Executor
Executing Actions
アクションの実行
● Execute command assigned to a gesture
● Use Process.spawn to avoid slowing down
the main loop
● Use Process.detach to avoid zombie
Detect a left swipe with three fingers
`swipe /3 / right` => { command: ‘xdotool key alt+Left’ }
Slide 27
Slide 27 text
Pipeline for Gesture Recognition
1. input:
2. buffer:
3. detector:
4. executor:
handle multiple events with IO.select
Aggregate events
Detect gestures from the events
Execute actions assigned from events
Structure of Pipeline lib/fusuma.rb
ジェスチャー認識のパイプライン
Input Buffer Detector Executor
Libinput Gesture
Swipe
Pinch
Rotate
Hold
Command
1 4
2 3
Slide 28
Slide 28 text
Extend Pipeline
with Plugins
Slide 29
Slide 29 text
Extend Pipeline with Plugins
Each tasks in pipeline are implemented with a plug-in system
プラグインでパイプラインを拡張する
● Depends on result(event) previous tasks
● Unified interface for each task
○ e.g) Input#io, Executor#execute
● Pipeline just call methods along the interface
● Additional features using Gem
● Implement tasks that satisfy interface
pipeline
plugins
Input Buffer Detector Executor
Libinput
Tap
Gesture
Timer
Swipe
Pinch
Rotate
Hold
Command
Timer
App
Tap
App
Tap
App
Keypress Keypress
Sendkey
Wmctrl
Slide 30
Slide 30 text
Internal plugins(default)
Internal plugins
Each task in pipeline is a plugin
内部プラグイン
Input Buffer Detector Executor
Libinput
Timer
Gesture
Timer
Swipe
Pinch
Rotate
Hold
Command
Slide 31
Slide 31 text
External plugins
外部プラグイン
External plugins(gem)
Detector
Input
Executor
Buffer
Executor
Detector
Buffer
Input Detector
Buffer
Input Detector
Buffer Executor
Executor
Executor
Input Executor
Input Detector
Buffer
Input Detector
Buffer
Experimental Features / Environment-specific / Requires additional tool
Slide 32
Slide 32 text
● Uses evdev to send key events as from
keyboard device
○ evdev is a generic kernel interface used by
both X11 and Wayland
○ Using libevdev requires ffi
■ need to install ruby-dev
fusuma-plugin-sendkey
evdevを利用したキーイベント送信プラグイン
Slide 33
Slide 33 text
fusuma-plugin-appmatcher
アプリ毎に設定を切り替えできるプラグイン
● Configure application-specific gestures and actions
in config.yml
● Switches the config according to the active
application
● Supports X11 and Gnome shell(Wayland)
○ X11: xprop -spy
○ Gnome: Gnome Extension and D-Bus
Slide 34
Slide 34 text
Plugin mechanism
Slide 35
Slide 35 text
Plugin mechanism
Find Gem using conventions
Register to plugin list by hooks
Call registered plugins
1
2
3
プラグインのメカニズム
Slide 36
Slide 36 text
Find Gem using conventions
規約ベースでGemを探索する
● Gem name rule
○ "fusuma-plugin-*"
● Directory rule
○ "lib/fusuma/plugin/input/*.rb"
■ input/buffer/detector/executor
Find plugins from gem
conventions
▲ Referenced plugin mechanisms from fluentd, pry, ruboty, and textbringer.
● Seek plugins using Gem.find_latest_files
● Check dependencies using gemspec
Slide 37
Slide 37 text
Register to plugin list by hooks
フックでプラグインを登録
● Use inherited hook into class inheritance
● Registers relationship
○ superclass => subclass relationship
■ Base => [Input, Buffer, Detector, Executor]
■ Input => [LibinputCommandInput, ...]
Example:
● Input < Base
● LibinputCommandInput < Input
↓
● Base -> Input -> LibinputCommandInput
Inherited hooks
Slide 38
Slide 38 text
Call registered plugins
登録したプラグインの呼び出し
● Get plugin classes from Manager
● Initialize plugins to start pipeline in main process
Initialize registered plugins lib/fusuma.rb
Slide 39
Slide 39 text
Plugin and Pipeline
Pros / Cons
Input Buffer Detector Executor
Libinput
Tap
Gesture
Timer
Swipe
Pinch
Rotate
Hold
Command
Timer
App
Tap
App
Tap
App
Keypress
Keypress
Sendkey
Wmctrl
Slide 40
Slide 40 text
Pros / Cons of plugin and pipeline
Pros
🙆 Event-based coding simplifies understanding and
testing
🙆 Easily decide where to place the source code
Cons
🙅 Processing can be redundant
🙅 State management can be difficult
Pipeline Plugin
Pros
🙆 Release of experimental features
🙆 Pipeline no longer dependent on plug-ins
Cons
🙅 Code maintenance can be complicated
🙅 Difficult to detect conflicts and interference between
external plugins, resulting in bugs
プラグインとパイプラインのメリット・デメリット
Slide 41
Slide 41 text
my new gem…
Slide 42
Slide 42 text
Limitations of Dragging gestures on Touchpad
タッチパッドのドラッグ操作って難しい
🙅 Tap and drag with one finger ends when the finger leaves the touchpad
🙅 3-finger dragging is also difficult to operate due to the narrow range of movement
🙅 Difficult to move and scroll the workspace while dragging operation is in progress
Slide 43
Slide 43 text
Rekimoto, Jun. "ThumbSense: Automatic input mode sensing for touchpad-based
interactions." _CHI'03 Extended Abstracts on Human Factors in Computing Systems_. 2003.
https://www2.sonycsl.co.jp/person/rekimoto/papers/chi03tsense.pdf
ThumbSense
タッチパッドは修飾キー
Only when the finger is on the touchpad
● Switch specific keys to left-click or right-click
● Replace with any command or keybinding
Slide 44
Slide 44 text
fusuma-plugin-thumbsense
タッチ中にキーリマップできるプラグイン
Only when the finger is on the touchpad
● Convert specific keys to left-click or right-click
● Replace with any command or keybinding
Slide 45
Slide 45 text
DEMO
デモ
● Click
○ J/F keys remapped to LEFT Click while touching on Touchpad
● Drag
○ Scrolling and swiping can be used simultaneously
○ Works well with touchpad gestures.
Slide 46
Slide 46 text
Implement ThumbSense as plugin
プラグインとしてThumbSenseを実装
● Detect touch/release on touchpad
● Send request to Change Layer
fusuma-plugin-thumbsense
fusuma-plugin-remapper
Input Detector
Buffer Executor
Input Detector
Buffer Executor
● Remapper running in external process
● Remap a key to mouse button
● Create Virtual Device using uinput
● lnput plugin to redirect key events
Slide 47
Slide 47 text
fusuma-plugin-thumbsense
タッチ検知でレイヤーを変更するプラグイン
● When `touch / 1 / begin` detected,
● Detector write to pipe for changing layer
■ layer = { thumbsense: true }
mapping = { j: 'BTN_LEFT', k: 'BTN_RIGHT' }
fusuma-plugin-thumbsense
1
1
2
2
Slide 48
Slide 48 text
fusuma-plugin-remapper
外部から動的にキーリマップできるプラグイン
● When receiving a layer change,
○ merge the layer and update key mapping
● Map physical keyboard events to current mapping
○ Write mapped event to Virtual Device
● Redirect physical keyboard events to Fusuma Input
○ Remapper works as "RemapperInput".
fusuma-plugin-remapper
1
1
2
2
3
3
Slide 49
Slide 49 text
Advantages of separating Remapper
リマッパーを分離したポイント
○ Remapper only converts keys, while Fusuma only detects events
○ As a side effect, Appmatcher + Remapper allows you to define
customizable key remappers for specific app
○ Remapper can also call existing executor plugin such as
command/wmctrl/sendkey
Slide 50
Slide 50 text
●
Let's build your own tools that feel good in your hands using Ruby!