DIY Your Touchpad Experience: Building Your Own Gestures

RubyKaigi 2023
https://rubykaigi.org/2023/presentations/nukumaro22.html

iberianpig/fusuma
https://github.com/iberianpig/fusuma

Transcript

1. DIY Your Touchpad Experience Building Your Own Gestures Kohei Yamada
2. About Me Kohei Yamada GitHub @iberianpig Twitter @nukumaro22 自己紹介
3. Do you like improving your environment? 環境カイゼンは好きですか？ > Many programmers, myself included, like to customize our environment to our liking. They are happy when it improves their efficiency, even if only a little. > You could call them a kind of "Environment Improvement Geek". > Matz 私を含めて多くのプログラマは，自分の環境を自分の好みに合わせてカスタマイズすることを好みます。それによってほんのわずかでも自分の効率が向上すると喜びを感じるのです。 ある種の 「環境改善オタク」と呼んでもよいでしょう。 ［改訂新版］Emacs実践入門──思考を直感的にコード化し、開発を加速する より 本書に寄せて まつもと ゆきひろ
4. My Development Environment Dell XPS 13 Ubuntu 22.04 / X11 Vim LAPTOP OS Editor 私の開発環境
5. Environment Improvement 環境の改善 Daily yak shaving builds plugins
6. About Fusuma
7. A Gem that enables touchpad gestures on Linux. About Fusuma Fusumaについて
8. About Fusuma Fusumaについて 🐧 On Linux touchpads ✋ Swipe 👌 Pinch-to-zoom
10. Fusuma Features Fusumaの特徴 • Easy installation with RubyGems • Defining Gestures and Actions in YAML • Extension of gesture recognition by Plugin system
11. How to use Fusuma Fusumaの使い方 Install libinput-tools 1 Add $USER to Input group 2 Customize ~/.config/fusuma/config.yml 5 Run $ fusuma from the CLI 4 Install Fusuma with $ gem i fusuma 3
12. Customize gestures and actions ジェスチャとアクションのカスタマイズ • Swipe • Pinch • Rotate • Hold • • command • sendkey(plugin) • wmctrl(plugin) Gesture Action
13. DEMO • 4-finger Swipe ◦ Switch Workspace • 3-finger swipe ◦ Open/Close browser tab ◦ Back/forward browser history • 3-finger dragging デモ
14. How I make Fusuma
15. HW X server Input Event Flow in Linux LinuxでのInputイベントの流れ Kernel device evdev libevdev libinput xf86-input-libinput X client App input event Simulate keyboard Fusuma Input Event Flow
16. Touchpad drivers on Linux • Well-known touchpad drivers, around 2010 • only X.Org • Many Configuration options • Gestures were not supported • Release of libinput 1.0 in 2015 • Used in Wayland and X.Org • No configuration file • Improvements to gesture support Synaptics 2015 libinput 2010 2020 Libinput Synaptics Linuxのタッチパッドドライバ
17. Seeking multitouch Gestures • Desire for Mac-like trackpad gestures • Test different laptops' gesture behavior in electronics store Prototype on Synaptics Synapticsドライバでのプロトタイプ Released xSwipe in 2012 • Discovered logging capability with `$ synclient -m` for synaptics • Read and parse logs in loop • Detect direction and send shortcut • Written in Perl Synaptics 2015 libinput 2010 2020 My first DIY
18. Synaptics to Libinput SynapticsからLibinputへ Synaptics replaced by libinput • synaptics driver in maintenance mode • xSwipe only worked with Synaptics driver • Frustration with inability to use gestures My First Gem Synaptics 2015 libinput 2010 2020 Released Fusuma in 2016 • Use `libinput debug-events` • Rewrite xSwipe in Ruby for libinput • `rake release`! 🪦
19. Gesture Recognition
20. Gesture Recognition ジェスチャ認識 Pipeline Input Buffer Detector Executor Libinput Gesture Swipe Pinch Rotate Hold Command Touchpad Keyboard 1 4 2 3
21. Input libinput-tools CLI as input inputとしてのlibinput-tools `libinput debug-events`
22. • Execute libinput using Open3.popen3 • Output lines without buffering using `stdbuf -oL` • convert a line to a event $stdout to event in real-time リアルタイムで標準出力をイベントに Input
23. Buffering Events イベントを溜める • Aggregate events • Multiple buffers depending on input Buffer Input Buffer Libinput Gesture Detector Swipe Executor Command Keypress Gesture State Time fingers delta x delta y SWIPE BEGIN +37.492s 3 10.23 -1.14 SWIPE UPDATE. +37.492s 3 10.54 -1.17 SWIPE UPDATE. +37.499s 3 7.62 -1.76 SWIPE UPDATE. +37.506s 3 10.25 -2.64
24. Gesture detection ジェスチャーの検知 Gesture State Time fingers delta x delta y SWIPE BEGIN +37.492s 3 10.23 -1.14 SWIPE UPDATE. +37.492s 3 10.54 -1.17 SWIPE UPDATE. +37.499s 3 7.62 -1.76 SWIPE UPDATE. +37.506s 3 10.25 -2.64 Detect a left swipe with three fingers => `swipe / 3 / right` Compare the average movement with threshold Detector Create Event with search index
25. Detect a left swipe with three fingers Retrieving Action from Config 設定ファイルからアクションを導出 config.yml `swipe / 3 / right` => { command: ‘xdotool key alt+Left’ } Executor
26. Executor Executing Actions アクションの実行 • Execute command assigned to a gesture • Use Process.spawn to avoid slowing down the main loop • Use Process.detach to avoid zombie Detect a left swipe with three fingers `swipe /3 / right` => { command: ‘xdotool key alt+Left’ }
27. Pipeline for Gesture Recognition 1. input: 2. buffer: 3. detector: 4. executor: handle multiple events with IO.select Aggregate events Detect gestures from the events Execute actions assigned from events Structure of Pipeline lib/fusuma.rb ジェスチャー認識のパイプライン Input Buffer Detector Executor Libinput Gesture Swipe Pinch Rotate Hold Command 1 4 2 3
28. Extend Pipeline with Plugins
29. Extend Pipeline with Plugins Each tasks in pipeline are implemented with a plug-in system プラグインでパイプラインを拡張する • Depends on result(event) previous tasks • Unified interface for each task ◦ e.g) Input#io, Executor#execute • Pipeline just call methods along the interface • Additional features using Gem • Implement tasks that satisfy interface pipeline plugins Input Buffer Detector Executor Libinput Tap Gesture Timer Swipe Pinch Rotate Hold Command Timer App Tap App Tap App Keypress Keypress Sendkey Wmctrl
30. Internal plugins(default) Internal plugins Each task in pipeline is a plugin 内部プラグイン Input Buffer Detector Executor Libinput Timer Gesture Timer Swipe Pinch Rotate Hold Command
31. External plugins 外部プラグイン External plugins(gem) Detector Input Executor Buffer Executor Detector Buffer Input Detector Buffer Input Detector Buffer Executor Executor Executor Input Executor Input Detector Buffer Input Detector Buffer Experimental Features / Environment-specific / Requires additional tool
32. • Uses evdev to send key events as from keyboard device ◦ evdev is a generic kernel interface used by both X11 and Wayland ◦ Using libevdev requires ffi ▪ need to install ruby-dev fusuma-plugin-sendkey evdevを利用したキーイベント送信プラグイン
33. fusuma-plugin-appmatcher アプリ毎に設定を切り替えできるプラグイン • Configure application-specific gestures and actions in config.yml • Switches the config according to the active application • Supports X11 and Gnome shell(Wayland) ◦ X11: xprop -spy ◦ Gnome: Gnome Extension and D-Bus
34. Plugin mechanism
35. Plugin mechanism Find Gem using conventions Register to plugin list by hooks Call registered plugins 1 2 3 プラグインのメカニズム
36. Find Gem using conventions 規約ベースでGemを探索する • Gem name rule ◦ "fusuma-plugin-*" • Directory rule ◦ "lib/fusuma/plugin/input/*.rb" ▪ input/buffer/detector/executor Find plugins from gem conventions ▲ Referenced plugin mechanisms from fluentd, pry, ruboty, and textbringer. • Seek plugins using Gem.find_latest_files • Check dependencies using gemspec
37. Register to plugin list by hooks フックでプラグインを登録 • Use inherited hook into class inheritance • Registers relationship ◦ superclass => subclass relationship ▪ Base => [Input, Buffer, Detector, Executor] ▪ Input => [LibinputCommandInput, ...] Example: • Input < Base • LibinputCommandInput < Input ↓ • Base -> Input -> LibinputCommandInput Inherited hooks
38. Call registered plugins 登録したプラグインの呼び出し • Get plugin classes from Manager • Initialize plugins to start pipeline in main process Initialize registered plugins lib/fusuma.rb
39. Plugin and Pipeline Pros / Cons Input Buffer Detector Executor Libinput Tap Gesture Timer Swipe Pinch Rotate Hold Command Timer App Tap App Tap App Keypress Keypress Sendkey Wmctrl
40. Pros / Cons of plugin and pipeline Pros 🙆 Event-based coding simplifies understanding and testing 🙆 Easily decide where to place the source code Cons 🙅 Processing can be redundant 🙅 State management can be difficult Pipeline Plugin Pros 🙆 Release of experimental features 🙆 Pipeline no longer dependent on plug-ins Cons 🙅 Code maintenance can be complicated 🙅 Difficult to detect conflicts and interference between external plugins, resulting in bugs プラグインとパイプラインのメリット・デメリット
41. my new gem…
42. Limitations of Dragging gestures on Touchpad タッチパッドのドラッグ操作って難しい 🙅 Tap and drag with one finger ends when the finger leaves the touchpad 🙅 3-finger dragging is also difficult to operate due to the narrow range of movement 🙅 Difficult to move and scroll the workspace while dragging operation is in progress
43. Rekimoto, Jun. "ThumbSense: Automatic input mode sensing for touchpad-based interactions." _CHI'03 Extended Abstracts on Human Factors in Computing Systems_. 2003. https://www2.sonycsl.co.jp/person/rekimoto/papers/chi03tsense.pdf ThumbSense タッチパッドは修飾キー Only when the finger is on the touchpad • Switch specific keys to left-click or right-click • Replace with any command or keybinding
44. fusuma-plugin-thumbsense タッチ中にキーリマップできるプラグイン Only when the finger is on the touchpad • Convert specific keys to left-click or right-click • Replace with any command or keybinding
45. DEMO デモ • Click ◦ J/F keys remapped to LEFT Click while touching on Touchpad • Drag ◦ Scrolling and swiping can be used simultaneously ◦ Works well with touchpad gestures.
46. Implement ThumbSense as plugin プラグインとしてThumbSenseを実装 • Detect touch/release on touchpad • Send request to Change Layer fusuma-plugin-thumbsense fusuma-plugin-remapper Input Detector Buffer Executor Input Detector Buffer Executor • Remapper running in external process • Remap a key to mouse button • Create Virtual Device using uinput • lnput plugin to redirect key events
47. fusuma-plugin-thumbsense タッチ検知でレイヤーを変更するプラグイン • When `touch / 1 / begin` detected, • Detector write to pipe for changing layer ▪ layer = { thumbsense: true } mapping = { j: 'BTN_LEFT', k: 'BTN_RIGHT' } fusuma-plugin-thumbsense 1 1 2 2
48. fusuma-plugin-remapper 外部から動的にキーリマップできるプラグイン • When receiving a layer change, ◦ merge the layer and update key mapping • Map physical keyboard events to current mapping ◦ Write mapped event to Virtual Device • Redirect physical keyboard events to Fusuma Input ◦ Remapper works as "RemapperInput". fusuma-plugin-remapper 1 1 2 2 3 3
49. Advantages of separating Remapper リマッパーを分離したポイント ◦ Remapper only converts keys, while Fusuma only detects events ◦ As a side effect, Appmatcher + Remapper allows you to define customizable key remappers for specific app ◦ Remapper can also call existing executor plugin such as command/wmctrl/sendkey
50. • Let's build your own tools that feel good in your hands using Ruby!