Automating Legacy Desktop Applications with JRuby and Sikuli (No Notes)

Desktop applications supporting line-of-business, enterprise functions live on as massive, complex beasts. Without an easy route into its internals and modernization falling by the wayside, it's up to you to make sure any small change doesn't break — well — everything. In this talk, we'll discuss the journey of bringing up a Windows VM to poke at an app using screenshots, image recognition, and Minitest to create our own Robotic Process Automation (RPA) framework.

Transcript

2. chris.cha.dev at the Gmail
4. Automating Legacy Desktop Applications with JRuby and Sikuli
6. A universe for apple pie • ed transforms • “Fairly vanilla” Windows • “From scratch” Vagrant fi le
7. Sneak ed in • Lexicographically ordered using 001-, 002- conventions. • Inspired by iterative database changes, Docker layering, Jenkins .groovy scripting
8. A regex scalpel • / — use a regex to “go to” a line • a — append • . — exit insert mode • w — write • q — quit • Don’t forget the newline at the end!
9. A fresh, free cup of Java • Pull in AdoptOpenJDK for `java` • Command Prompt > Run as Administrator • Let’s get to JRuby ASAP (soon)
10. Like every software install ever • Download the goods • Unzip it • Set PATH • Done
11. Hello world • Or was it `java -version`?
12. Every block has its end • / — go to the end of fi le • -1 — go one line above it • a — okay, now start append • . — done • w — write • q — quit • (newline)
13. [SOLVED] with indirection • Some kind of script (.bat, .ps1, etc.) • Create an .ed transform mapping
14. Nice to meet you • Are we at JRuby yet? • Not yet
15. I know you’re out there • \\somewhere\out\there
16. Rinse and repeat • Retrieve Vagrant image • Apply transforms • `vagrant up` and `vagrant reload` • xfreerdp
17. Reliable drive mapping • Scheduled task • Need XML to disable all conditions • .ps1 supports HEREDOC
18. XML out, XML in
19. Right — cleanup • (Clever quip here)
20. Gimme your creds • Prompt user for important things
21. My preciousss • Finally the app • Quirks, warts, and all
22. Mystery of mysteries • .jnlp
23. Reliable drive mapping • A sprinkling of indirection
24. Enumerate for sanity • +1 for debugging output
25. How would we know otherwise • Commit history as a service
26. Linux, brie fl y • Kiosk mode • Xvfb on Linux
27. Copy-pasta • Standard exported Basic Task XML
28. Launch app on connect • Register scheduled task with Register- ScheduledTask
29. Step 7: kiosk mode • 007-schedule-to-start-pos-on-login.ed
 Start up application on remote login (kiosk mode) • Scheduled task starts scripts/schedule- start-pos.ps1
30. Sikuli! (Or, canst thou see it?) • Take a screenshot with Sikuli • Prove Xvfb provides a rendered context
31. Om nom nom • Running java to run Sikuli • jruby eaten by Sikuli
32. It’s just zeroes and ones • javax.imageio.ImageIO in JRuby • Write Sikuli Region as .png; Screen provides desktop width and height
33. .ed language server • Code is text
34. Tests as text • Code- fi rst approach
35. JRuby!! • Our apple pie is ready
36. whereis jruby • Wherever we like
37. Vagrant fi le as task runner • .ed adds to Vagrant fi le • Vagrant executes speci fi ed script
38. Punching through • App launches only the second and subsequent times
39. netsh • 010-open-java-exe- fi rewall.ed
 Run scripts/open-java-exe- fi rewall.bat
40. Promises, promises • Initial batch of screenshots for image recognition • Development work fl ow solidifying • Conform to corporate code conventions
41. Tesseract (OCR) dependency • Install Visual C++ redistributable
42. Scheduled task boilerplate • Similar XML as before
43. on-login.bat • Copy to default Startup\ folder
44. Arranging photos • Hierarchical tree of strings • Each is rendered in SikuliIDE as an image
45. Stay awhile and sleep • App retrieves a bunch of fi les, including its own JRE
46. Macros and games • Proof-of-concept: click checkbox image, send keys (and key combos)
47. Screen recording (and stop) • Saving to a timestamped .mp4
48. Tell Sikuli: images here • Con fi gure global “bundle path” to images
49. Each step pro fi ts • .ed to specify script: startup task • .ed to specify script: install Visual C++ redist
50. VLC for screen recording • Like Selenium, capture a video of the test run
51. Installing VLC • Polling install
52. HOWTO screen recording • `ncat` to stop recording • Start and Stop recording are separate .bat scripts
53. VLC setup script • Bundle VLC 3.0.14 to avoid download • 013-install-vlc.ed
 Runs scripts/install-vlc.bat
54. Failure to launch • App should launch every time
55. Startup runbook • Copy script to standard Startup path:
 C:\ProgramData\Microsoft\Windows\Start Menu\Programs\StartUp
56. bitsadmin to BTS (?) • Polling bitsadmin (bleh) • Copying to Startup\
57. Plain ol’ HTTP • JDK 8 • Powershell Invoke-Request
58. Downgrading Java 11 to 8
59. Fast downloads • No more bitsadmin polling (silly)
60. bundle install «gem» • gem install bundler • gem install «wonderful-tool»
61. Wonderful, beautiful, shiny • Gem list: brakeman, bundler-audit, fl og, reek • Internal gems: tcs-con fi g, tcs-rubocop
62. Install bundler, install gems • 014-install-bundler.ed
 Runs scripts/install-bundler.bat
63. Tests here, come git your tests
64. The “better diff” for Windows • Di ff utils install with /SILENT and /DIR=
65. Silent VLC, add minitest gem • Nit: Install VLC in a place without spaces • Add minitest gem
66. GNU diff for Minitest • 015-install-gnu-di ff .ed
 Runs scripts/install-gnu-di ff .bat
67. Start refactoring • Break out proof-of-concept (PoC) into modules
68. docs Explain running local tests • Updating README • Setting RUBYOPT
69. Reset dropdowns • A bit of refactoring to the PoC to be more reliable: reset before setting values
70. Loading in Sikuli • Import SikuliX API • Follow Raymond Hocke’s recommendations (Launchpad post)
71. Debug (logging), global con fi g • Set Debug level to 3 (most verbose) • Specify global images path
72. A (UI) tree of wonder • *.sikuli represented as type-of under parent folder (AppImageRoot) • SikuliTestEnvironment::Component as parent of UI components • PosComponents < Components
73. Strings as a source of truth • (Helper methods) • Automatic mapping: 
 components[key] = value
74. Simple specialization • Delegates to parent
75. Minitest! • Test our setup • Learning Minitest
76. • spec_helper.rb
77. “Checkpoint commit” • Fix bug that took a while to fi nd • Toward login/logout
78. A batch of pictures • Adding more images to recognize with computer vision
79. • SikuliComponent::Component initialized with hash (image- fi lename-to-image-path mapping) from SikuliTestEnvironment::PosComponent
80. • Components in test respond to intentful service calls • Underlying types and clicks simpli fi ed by object hierarchy
81. 1 o’ the hardest problems in CS • Bug fi x: use base_bundle_path instead of «base-bundle-path»/app
82. Automating the login • A panoply of interacting classes
83. To /app or not to /app • Add con fi guration/.yml parser lib
84. Gherkin for knowledge management
85. • “Intent-ful” interface calls
86. • Primitive actions hidden behind an interface
87. • Unused
91. Interlude II • Stack over fl ow! • More tests, more tests
92. How to exit a Java app • taskkill /f /im java.exe /t
93. • Hide launching window
97. • Caller never invokes primitives, only sends messages of intent • Public/private separation ensures a useful interface for collaboration and interaction
98. • More intents: apply_, asked_, delete_, _cancel
106. • Front-loading expensive setup
 http://ruslanledesma.com/2016/07/21/ before-all-after-all-in-minitest.html
109. • English-like behavior calls
111. Thank-you