Exploring Neural Networking with Kotlin Metaprogramming: A Cheaper Alternative to Deep Learning?

Presented at JFuture 2019.

Relevant Links:
- TornadoFX-Suite: https://github.com/ahinchman1/TornadoFX-Suite
- Automata UI research: https://github.com/ahinchman1/Finite-State-Machine-Crash-Course
- Data science research: https://github.com/ahinchman1/Data-Science-Crash-Course

Transcript

1. Exploring Neural Networking with Kotlin Metaprogramming: A Cheaper Alternative to Deep Learning? Amanda Hinchman-Dominguez @hinchman_amanda
2. UIs increasingly critical to the success of applications • An application’s GUI must create a rich, intuitive and pleasant user experience • Validate + guarantee quality assurance of event-driven applications through the active process of exploring an environment
3. Black box testing • Tests for the behavior/ functionality of software White box testing • Tests how the system functions E2E Integration Unit Quality Assurance
4. TornadoFX-Suite: using metaprogramming to write UI tests
5. Black box testing • Tests for the behavior/ functionality of software White box testing • Tests how the system functions E2E Integration Unit Quality Assurance
6. E2E Integration Unit Black box testing • Tests for the behavior/ functionality of software White box testing • Tests how the system functions UI Testing Quality Assurance
7. Can AI truly automate testing to validate the human experience? • Automation should be viewed as a technique, not the design activity itself • Like many problems that require a human process, people who don’t understand the value of testing (or doesn’t want to do it) turns it into a programming problem • AI can’t answer “understand the value of an application from interacting with its environment”
8. Metaprogramming neural networking solution can serve as a suitable option for automated techniques in testing • more efficient • require less data for analysis • far fewer unknowns with internal operations
9. • The neural network itself is called the model, but it can also be recognized as an algorithm (pixel math). • In this sense, the neural networking is act of moving data through different layers containing certain filters. • Creates the model to achieve the same goal • The relationships that are drawn, the patterns that are created, in order to solve the same problems AI does is the neural network Neural Networking in Deep Learning Neural Networking in Metaprogramming
10. What kind of problems does neural networking solve for automated testing? Supervised Learning: Includes target value Regression: Target is continuous Classification: Target is discrete
11. Metaprogramming Locator Mechanism field fieldset form textfield field textfield field field textfield field fieldset form hbox textfield hbox field textfield textfield button button AI Locator Mechanism Filt + Max pooling through dense layers Locating UI Nodes with Classification
12. • Classification - How lines are drawn around data points • Convolutional Neural Networking (CNN) model - Most popular for analyzing images • Matrix math to flatten pixel data as they’re passed through hidden convolutional layers & non- convolutional layering Image classification - locator mechanism
13. Filter 1/1 Filter 2/1
14. Filter 1/1 Filter 2/1
15. Filter 1/1 Filter 2/1
16. Filter 1/1 Filter 2/1
17. Filter 1/1 Filter 2/1
18. Filter 1/1 Filter 2/1
19. Filter 1/1 Filter 2/1
20. Filter 1/1 Filter 2/1
21. Filter 1/1 Filter 2/1
22. Filter 1/1 Filter 2/1
23. Filter 1/1 Filter 2/1
24. Filter 1/1 Filter 2/1
25. Filter 1/1 Filter 2/1
26. Filter 1/1 Filter 2/1
27. Filter 2/1 Filter 1/1
28. Filter 2/1 Filter 1/1 Filter 1/2 Filter 1/2 + +
29. Filter 2/1 Filter 1/1 Filter 1/2 Filter 1/2 + +
30. Filter 2/1 Filter 1/1 Filter 1/2 Filter 1/2 + +
31. Filter 2/1 Filter 1/1 Filter 1/2 Filter 1/2 + +
32. Filter 2/1 Filter 1/1 Filter 1/2 Filter 1/2 + +
33. Filter 2/1 Filter 1/1 Filter 1/2 Filter 1/2 + +
34. Filter 2/1 Filter 1/1 Filter 1/2 Filter 1/2 + +
35. Filter 2/1 Filter 1/1 Filter 1/2 Filter 1/2 + +
36. Filter 2/1 Filter 1/1 Filter 1/2 Filter 1/2 + +
37. Filter 2/1 Filter 1/1 Filter 1/2 Filter 1/2 + +
38. Filter 2/1 Filter 1/1 Filter 1/2 Filter 1/2 + +
39. Filter 2/1 Filter 1/1 Filter 1/2 Filter 1/2 + +
40. Filter 1/2 Filter 1/2 + + Max Pooling Max Pooling
41. Max Pooling Max Pooling Dense Weights Dense Weights Dense Activation Dense Activation
42. • Draws lines about the pixel, no knowledge of code required • Reduces image pixels through filters, but provides lots of intense matrix calculations that are opaque to us due to the # of calculations required per layer Image classification - locator mechanism "Komandyseja | Music Production" by Przemek Bizoń, bisoñ studio, Agata Łobaczuk is licensed under CC BY-NC-ND 4.0 - Modified image.
43. • Draws lines about the pixel, no knowledge of code required • Reduces image pixels through filters, but provides lots of intense matrix calculations that are opaque to us due to the # of calculations required per layer Image classification - locator mechanism "Komandyseja | Music Production" by Przemek Bizoń, bisoñ studio, Agata Łobaczuk is licensed under CC BY-NC-ND 4.0 - Modified image. TextView InteractivePlayerView Floating Action Button ScrollView RelativeLayout Linear Layout
44. • Abstract Syntax Tree (AST) parser draws lines around the shape of the written language parsed into an intermediate representation of an AST The Metaprogramming Locator Mechanism
45. • Abstract Syntax Tree (AST) parser draws lines around the shape of the written language parsed into an intermediate representation of an AST The Metaprogramming Locator Mechanism
46. • Abstract Syntax Tree (AST) parser draws lines around the shape of the written language parsed into an intermediate representation of an AST The Metaprogramming Locator Mechanism
47. • Abstract Syntax Tree (AST) parser draws lines around the shape of the written language parsed into an intermediate representation of an AST The Metaprogramming Locator Mechanism
48. • Abstract Syntax Tree (AST) parser draws lines around the shape of the written language parsed into an intermediate representation of an AST The Metaprogramming Locator Mechanism
49. • Abstract Syntax Tree (AST) parser draws lines around the shape of the written language parsed into an intermediate representation of an AST The Metaprogramming Locator Mechanism
50. • Abstract Syntax Tree (AST) parser draws lines around the shape of the written language parsed into an intermediate representation of an AST The Metaprogramming Locator Mechanism
51. • Abstract Syntax Tree (AST) parser draws lines around the shape of the written language parsed into an intermediate representation of an AST The Metaprogramming Locator Mechanism
52. • Abstract Syntax Tree (AST) parser draws lines around the shape of the written language parsed into an intermediate representation of an AST The Metaprogramming Locator Mechanism
53. • Abstract Syntax Tree (AST) parser draws lines around the shape of the written language parsed into an intermediate representation of an AST The Metaprogramming Locator Mechanism
54. • Abstract Syntax Tree (AST) parser draws lines around the shape of the written language parsed into an intermediate representation of an AST The Metaprogramming Locator Mechanism
55. • Abstract Syntax Tree (AST) parser draws lines around the shape of the written language parsed into an intermediate representation of an AST The Metaprogramming Locator Mechanism
56. • Abstract Syntax Tree (AST) parser draws lines around the shape of the written language parsed into an intermediate representation of an AST The Metaprogramming Locator Mechanism
57. override val root = hbox { form { fieldset("Edit person") { field("Owner") { textfield(model.ownerName) { ownerNameField = this } } field("Cat") { textfield(model.catName) { catNameField = this } } field("Time") { textfield(model.time) { timeField = this } } button("Save") { enableWhen(model.dirty) action { save() } } } } } field fieldset form textfield field textfield field field textfield field fieldset form hbox textfield hbox field textfield textfield button button The Metaprogramming Locator Mechanism
60. Metaprogramming Locator Mechanism field fieldset form textfield field textfield field field textfield field fieldset form hbox textfield hbox field textfield textfield button button AI Locator Mechanism Filt + Max pooling through dense layers Locating UI Nodes with Classification
61. Metaprogramming Locator Mechanism field fieldset form textfield field textfield field field textfield field fieldset form hbox textfield hbox field textfield textfield button button AI Locator Mechanism O(mn3) Filt + Max pooling through dense layers Locating UI Nodes with Classification O(n)
62. Future work: Regression analysis Supervised Learning: Includes target value Regression: Target is continuous Classification: Target is discrete http://testerstories.com/2018/08/can-an-ai-become-a-tester/
63. • Regression - How lines draw relationships from one point to another • How can we increase combinations of permutations of interactions? Future work: Making UI tests “smarter”
64. Artificial Neural Networking Improving UI Testing with Regression Abstract Compositional Contracting
65. Recap • Using automation can help the tester, but doesn’t replace the role of a test • Neural networking in metaprogramming is isomorphic to AI neural networking • AI models for regression and classification is expensive and difficult to maintain and wrangle • You don’t need a sledgehammer for a nail Supervised Learning: Regression Classification
66. • TornadoFX-Suite: https://github.com/ahinchman1/TornadoFX-Suite • Automata UI research: https://github.com/ahinchman1/Finite-State-Machine-Crash-Course • Data science research: https://github.com/ahinchman1/Data-Science-Crash-Course Amanda Hinchman-Dominguez @hinchman_amanda Relevant Links