Conference for Kotliners - Kotlin: the Next Frontier in Metaprogramming

In an epic quest using Kotlin and native desktop development to generate UI testing for the TornadoFX framework, we turn to metaprogramming in Kotlin to explore AST language parsing and the philosophy of design-driven UI testing in the pursuit for smarter test-generation.

1. Kotlin: The Next Frontier in Metaprogramming Using metaprogramming to formalize the theory of human error in applications in generative UI testing Amanda Hinchman-Dominguez @hinchman_amanda
2. UIs increasingly critical to the success of applications • An application’s GUI must create a rich, intuitive and pleasant user experience • Validate + guarantee quality assurance of event-driven applications through the active process of exploring an environment
5. What makes a good UI test? What makes a bad UI test?
6. TornadoFX-Suite: using metaprogramming to write UI tests
7. Black box testing • Tests for the behavior/ functionality of software White box testing • Tests how the system functions E2E Integration Unit Quality Assurance
8. Black box testing • Tests for the behavior/ functionality of software White box testing • Tests how the system functions E2E Integration Unit Quality Assurance
9. E2E Integration Unit Black box testing • Tests for the behavior/ functionality of software White box testing • Tests how the system functions UI Testing Quality Assurance
10. A UI Test goes something like this… Grab a particular view
11. A UI Test goes something like this… Grab a particular view Perform action on view Check UI to see if it looks as it should
12. A UI Test goes something like this… Grab a particular view Perform action on view Check UI to see if it looks as it should
13. • Kastree - a simple wrapper around the Kotlin compiler • Program Structure Interface (PSI): the layer in IntelliJ parses syntactic and semantic code models • An Abstract Syntax Tree (AST) parser surpasses challenges found in statically-typed metaprogramming: • Access levels • Type casting in recursion Phase 1: Detecting Inputs w/ AST Parsing
14. AST Parsing with Kastree & Gson
15. override val root = hbox { form { fieldset("Edit person") { field("Owner") { textfield(model.ownerName) { ownerNameField = this } } field("Cat") { textfield(model.catName) { catNameField = this } } field("Time") { textfield(model.time) { timeField = this } } button("Save") { enableWhen(model.dirty) action { save() } } } } } Saving UI Nodes field fieldset form textfield field textfield field field textfield field fieldset form hbox textfield hbox field textfield textfield button button
17. A UI Test goes something like this… Grab a particular view Perform action on view Check UI to see if it looks as it should
18. A UI Test goes something like this… Grab a particular view Perform action on view Check UI to see if it looks as it should
19. • Kotlin scripting to write Kotlin test files for node interactions • TestFX robots to stimulate user interactions, check node states with matchers • Dynamically insert randomized ids for detected nodes at runtime Phase 2: Generating Tests with TestFX
20. • Kotlin scripting to write Kotlin test files for node interactions • TestFX robots to stimulate user interactions, check node states with matchers • Dynamically insert randomized ids for detected nodes at runtime Phase 2: Generating Tests with TestFX
21. A UI Test goes something like this… Grab a particular view Perform action on view Check UI to see if it looks as it should
24. How do we know if a node looks as it should? • Mapping a compositional abstraction for our components can help define properties surrounding actions of our views • When a control event is triggered, apply binary operations against states and check against the contract holistic result of an application’s representative compositional map
25. Phase 3: Abstract Compositional Contracting • Check for the state of the node after an interaction • Map nodes to associated functions • Create function compositions from node interactions • Check for models attached to views • Model support • Scope/Context support
26. Phase 3: Abstract Compositional Contracting • Check for the state of the node after an interaction • Map nodes to associated functions • Create function compositions from node interactions • Check for models attached to views • Model support • Scope/Context support
27. Phase 3: Abstract Compositional Contracting • Check for the state of the node after an interaction • Map nodes to associated functions • Create function compositions from node interactions • Check for models attached to views • Model support • Scope/Context support
28. button form hbox override val root = hbox { form { fieldset("Edit person") { field("Owner") { textfield(model.ownerName) { ownerNameField = this } } field("Cat") { textfield(model.catName) { catNameField = this } } field("Time") { textfield(model.time) { timeField = this } } button("Save") { enableWhen(model.dirty) action { save() } } } } } fieldset form fieldset field field field field button textfield field textfield textfield textfield field textfield textfield field textfield Post-parsing analysis
29. button Post-parsing analysis
30. UINode( nodeType = “Button”, uiLevel = 3, nodeChildren = JsonObject(), valueAssignment = “”, functionComposition = Digraph<UIFunctionComposition>(), associatedFunctions = ArrayList<String>() ) button Post-parsing analysis
31. UINode( nodeType = “Button”, uiLevel = 3, nodeChildren = JsonObject(), valueAssignment = “”, functionComposition = Digraph<UIFunctionComposition>(), associatedFunctions = ArrayList<String>() ) button Post-parsing analysis
32. Associated functions - functions responsible for affecting a node’s property or state
33. Associated functions - functions responsible for affecting a node’s property or state
34. Associated Functions
35. avi class BottomViewController: Controller() { private val view: BottomView by inject() fun changeCatAvi(catSchedule: CatSchedule) { view.avi.children.clear() val image = StackPane().apply { rectangle { width = 200.0 height = 200.0 fill = c("#AARRGGBB") opacity = 0.01 } imageview(catSchedule.catImage) } view.avi.children.add(image) } fun editCatSchedule(catSchedule: CatSchedule) { val catScheduleScope = CatScheduleScope() catScheduleScope.model.item = catSchedule find(Editor::class, scope = catScheduleScope).openModal() } } Associated Functions
36. avi class BottomViewController: Controller() { private val view: BottomView by inject() fun changeCatAvi(catSchedule: CatSchedule) { view.avi.children.clear() val image = StackPane().apply { } view.avi.children.add(image) } fun editCatSchedule(catSchedule: CatSchedule) { val catScheduleScope = CatScheduleScope() catScheduleScope.model.item = catSchedule find(Editor::class, scope = catScheduleScope).openModal() } } Associated Functions
37. avi class BottomViewController: Controller() { private val view: BottomView by inject() fun changeCatAvi(catSchedule: CatSchedule) { view.avi.children.clear() val image = StackPane().apply { } view.avi.children.add(image) } fun editCatSchedule(catSchedule: CatSchedule) { val catScheduleScope = CatScheduleScope() catScheduleScope.model.item = catSchedule find(Editor::class, scope = catScheduleScope).openModal() } } Associated Functions
38. avi class BottomViewController: Controller() { private val view: BottomView by inject() fun changeCatAvi(catSchedule: CatSchedule) { view.avi.children.clear() val image = StackPane().apply { } view.avi.children.add(image) } fun editCatSchedule(catSchedule: CatSchedule) { val catScheduleScope = CatScheduleScope() catScheduleScope.model.item = catSchedule find(Editor::class, scope = catScheduleScope).openModal() } } Associated Functions
39. avi class BottomViewController: Controller() { private val view: BottomView by inject() fun changeCatAvi(catSchedule: CatSchedule) { view.avi.children.clear() val image = StackPane().apply { } view.avi.children.add(image) } fun editCatSchedule(catSchedule: CatSchedule) { val catScheduleScope = CatScheduleScope() catScheduleScope.model.item = catSchedule find(Editor::class, scope = catScheduleScope).openModal() } } Associated Functions
40. avi class BottomViewController: Controller() { private val view: BottomView by inject() fun changeCatAvi(catSchedule: CatSchedule) { view.avi.children.clear() val image = StackPane().apply { } view.avi.children.add(image) } fun editCatSchedule(catSchedule: CatSchedule) { val catScheduleScope = CatScheduleScope() catScheduleScope.model.item = catSchedule find(Editor::class, scope = catScheduleScope).openModal() } } Associated Functions
41. avi class BottomViewController: Controller() { private val view: BottomView by inject() fun changeCatAvi(catSchedule: CatSchedule) { view.avi.children.clear() val image = StackPane().apply { } view.avi.children.add(image) } fun editCatSchedule(catSchedule: CatSchedule) { val catScheduleScope = CatScheduleScope() catScheduleScope.model.item = catSchedule find(Editor::class, scope = catScheduleScope).openModal() } } Associated Functions
42. avi class BottomViewController: Controller() { private val view: BottomView by inject() fun changeCatAvi(catSchedule: CatSchedule) { view.avi.children.clear() val image = StackPane().apply { } view.avi.children.add(image) } fun editCatSchedule(catSchedule: CatSchedule) { val catScheduleScope = CatScheduleScope() catScheduleScope.model.item = catSchedule find(Editor::class, scope = catScheduleScope).openModal() } } Associated Functions
43. avi class BottomViewController: Controller() { private val view: BottomView by inject() fun changeCatAvi(catSchedule: CatSchedule) { view.avi.children.clear() val image = StackPane().apply { } view.avi.children.add(image) } fun editCatSchedule(catSchedule: CatSchedule) { val catScheduleScope = CatScheduleScope() catScheduleScope.model.item = catSchedule find(Editor::class, scope = catScheduleScope).openModal() } } Associated Functions
44. avi class BottomViewController: Controller() { private val view: BottomView by inject() fun changeCatAvi(catSchedule: CatSchedule) { view.avi.children.clear() val image = StackPane().apply { } view.avi.children.add(image) } fun editCatSchedule(catSchedule: CatSchedule) { val catScheduleScope = CatScheduleScope() catScheduleScope.model.item = catSchedule find(Editor::class, scope = catScheduleScope).openModal() } } Associated Functions
45. avi class BottomViewController: Controller() { private val view: BottomView by inject() fun changeCatAvi(catSchedule: CatSchedule) { view.avi.children.clear() val image = StackPane().apply { rectangle { width = 200.0 height = 200.0 fill = c("#AARRGGBB") opacity = 0.01 } imageview(catSchedule.catImage) } view.avi.children.add(image) } fun editCatSchedule(catSchedule: CatSchedule) { val catScheduleScope = CatScheduleScope() catScheduleScope.model.item = catSchedule find(Editor::class, scope = catScheduleScope).openModal() } } Associated Functions
46. avi class BottomViewController: Controller() { private val view: BottomView by inject() fun changeCatAvi(catSchedule: CatSchedule) { view.avi.children.clear() val image = StackPane().apply { rectangle { width = 200.0 height = 200.0 fill = c("#AARRGGBB") opacity = 0.01 } imageview(catSchedule.catImage) } view.avi.children.add(image) } fun editCatSchedule(catSchedule: CatSchedule) { val catScheduleScope = CatScheduleScope() catScheduleScope.model.item = catSchedule find(Editor::class, scope = catScheduleScope).openModal() } } Associated Functions
47. avi class BottomViewController: Controller() { private val view: BottomView by inject() fun changeCatAvi(catSchedule: CatSchedule) { view.avi.children.clear() val image = StackPane().apply { rectangle { width = 200.0 height = 200.0 fill = c("#AARRGGBB") opacity = 0.01 } imageview(catSchedule.catImage) } view.avi.children.add(image) } fun editCatSchedule(catSchedule: CatSchedule) { val catScheduleScope = CatScheduleScope() catScheduleScope.model.item = catSchedule find(Editor::class, scope = catScheduleScope).openModal() } } Associated Functions
48. avi class BottomViewController: Controller() { private val view: BottomView by inject() fun changeCatAvi(catSchedule: CatSchedule) { view.avi.children.clear() val image = StackPane().apply { rectangle { width = 200.0 height = 200.0 fill = c("#AARRGGBB") opacity = 0.01 } imageview(catSchedule.catImage) } view.avi.children.add(image) } fun editCatSchedule(catSchedule: CatSchedule) { val catScheduleScope = CatScheduleScope() catScheduleScope.model.item = catSchedule find(Editor::class, scope = catScheduleScope).openModal() } } Associated Functions
49. avi class BottomViewController: Controller() { private val view: BottomView by inject() fun changeCatAvi(catSchedule: CatSchedule) { view.avi.children.clear() val image = StackPane().apply { rectangle { width = 200.0 height = 200.0 fill = c("#AARRGGBB") opacity = 0.01 } imageview(catSchedule.catImage) } view.avi.children.add(image) } fun editCatSchedule(catSchedule: CatSchedule) { val catScheduleScope = CatScheduleScope() catScheduleScope.model.item = catSchedule find(Editor::class, scope = catScheduleScope).openModal() } } Associated Functions
50. avi class BottomViewController: Controller() { private val view: BottomView by inject() fun changeCatAvi(catSchedule: CatSchedule) { view.avi.children.clear() val image = StackPane().apply { rectangle { width = 200.0 height = 200.0 fill = c("#AARRGGBB") opacity = 0.01 } imageview(catSchedule.catImage) } view.avi.children.add(image) } fun editCatSchedule(catSchedule: CatSchedule) { val catScheduleScope = CatScheduleScope() catScheduleScope.model.item = catSchedule find(Editor::class, scope = catScheduleScope).openModal() } } Associated Functions
51. avi class BottomViewController: Controller() { private val view: BottomView by inject() fun changeCatAvi(catSchedule: CatSchedule) { view.avi.children.clear() val image = StackPane().apply { rectangle { width = 200.0 height = 200.0 fill = c("#AARRGGBB") opacity = 0.01 } imageview(catSchedule.catImage) Associated Functions
52. button("Save") { enableWhen(model.dirty) action { save() update(model) } } Functional composition - what functions are spurned from interacting with a particular node?
53. Function Composition button("Save") { enableWhen(model.dirty) action { save() update(model) } } action
54. Function Composition button("Save") { enableWhen(model.dirty) action { save() update(model) } } private fun save() { model.commit() val catSchedule = model.item close() } private fun update(model: CatScheduleModel) { println("Contact updated!") controller.changeCatAvi(model.item) } save save action fun changeCatAvi(catSchedule: CatSchedule) { view.avi.children.clear() val image = StackPane().apply { rectangle { width = 200.0 height = 200.0 fill = c("#AARRGGBB") opacity = 0.01 } imageview(catSchedule.catImage) } view.avi.children.add(image) } changeCatAvi changeCatAvi update update
55. Phase 3: Abstract Compositional Contracting • Check for the state of the node after an interaction • Map nodes to associated functions • Create function compositions from node interactions • Check for models attached to views • Model support • Scope/Context support
56. Creating Finite State Machines to predict node behavior Cat Schedule Model Save Button Dirty Not Dirty Disabled Enabled
57. Creating Finite State Machines to predict node behavior Cat Schedule Model Save Button Dirty Not Dirty Disabled Enabled
58. Cat Schedule Model Save Button Dirty Not Dirty Disabled Enabled enum class Toggle { Enable, Disable; fun isButtonEnabled(): Boolean { return when (this) { Enable -> true Disable -> false } } fun enableOnDirty(model: ModelState): Toggle { return when (model) { ModelState.Dirty -> Enable ModelState.NotDirty -> Disable } } } enum class ModelState { Dirty, NotDirty; fun initModelStateTransition(): ModelState { return when (this) { Dirty -> NotDirty NotDirty -> Dirty } } }
59. sealed class EditorFragment { var modelState = ModelState.NotDirty fun commitModel() {} fun onChangeText() { modelState.initModelStateTransition() } open class OwnerTextField: EditorFragment() {} open class CatTextField: EditorFragment() {} open class TimeTextField: EditorFragment() {} class SaveButton(var toggle: Toggle = Toggle.Disable): EditorFragment() { fun toggleButtonOnDirtyModel(model: ModelState) { toggle.enableOnDirty(model) } fun saveModel() { if (toggle.isButtonEnabled()) commitModel() } } }
60. sealed class EditorFragment { var modelState = ModelState.NotDirty fun commitModel() {} fun onChangeText() { modelState.initModelStateTransition() } open class OwnerTextField: EditorFragment() {} open class CatTextField: EditorFragment() {} open class TimeTextField: EditorFragment() {} class SaveButton(var toggle: Toggle = Toggle.Disable): EditorFragment() { fun toggleButtonOnDirtyModel(model: ModelState) { toggle.enableOnDirty(model) } fun saveModel() { if (toggle.isButtonEnabled()) commitModel() } } }
61. Specifying UIs Formally through Metaprogramming Benefits: • Implicit: Clearly expressing ourselves leads to a better understanding of the system • Explicit: Using formal specs to check properties - does our UI have any dead ends? Can we get locked out of a chunk of the UI?
62. Promising discoveries in metaprogramming analysis research • Metaprogramming locator mechanism is more accurate/less intensive detecting UI components in applications than current AI locator mechanisms • Current AI image classification locator mechanisms suffer from partial view render recognition, detecting custom object interactions
63. Promising discoveries in metaprogramming analysis research • Using Kotlin metaprogramming to further analyze the relationship of UI hierarchies, functional composition, and generated test results • Potential for generating unit testing in the future • Traditional machine learning only offers immediate extrinsic reward analysis; metaprogramming generated relation analysis may provide more in-depth analysis to supplement future deep learning
64. Future Plans for TornadoFX-Suite • DSLs for more flexible test building + abstracted compositional mapping • Make AST parsing stronger + pull out into an independent library • Expand TornadoFX-Suite to analyze other UI frameworks • Collect voluntary data from future users to begin deep learning
65. • TornadoFX-Suite: https://github.com/ahinchman1/TornadoFX-Suite • Automata UI research: https://github.com/ahinchman1/Finite-State-Machine-Crash-Course • Data science research: https://github.com/ahinchman1/Data-Science-Crash-Course • Neighborhood Cat Scheduler: https://github.com/Kotlin-Thursdays/Neighborhood-Cat-Scheduler • TornadoFX-DnD-TilesFX: https://github.com/ahinchman1/TornadoFX-DnD-TilesFX Amanda Hinchman-Dominguez @hinchman_amanda Relevant Links