Tweet
Tweet

Slide 1

Slide 1 text

Exploring Neural Networking with Kotlin Metaprogramming: A Cheaper Alternative to Deep Learning? Amanda Hinchman-Dominguez @hinchman_amanda

Slide 2

Slide 2 text

UIs increasingly critical to the success of applications • An application’s GUI must create a rich, intuitive and pleasant user experience • Validate + guarantee quality assurance of event-driven applications through the active process of exploring an environment

Slide 3

Slide 3 text

Black box testing • Tests for the behavior/ functionality of software White box testing • Tests how the system functions E2E Integration Unit Quality Assurance

Slide 4

Slide 4 text

TornadoFX-Suite: using metaprogramming to write UI tests

Slide 5

Slide 5 text

Black box testing • Tests for the behavior/ functionality of software White box testing • Tests how the system functions E2E Integration Unit Quality Assurance

Slide 6

Slide 6 text

E2E Integration Unit Black box testing • Tests for the behavior/ functionality of software White box testing • Tests how the system functions UI Testing Quality Assurance

Slide 7

Slide 7 text

Can AI truly automate testing to validate the human experience? • Automation should be viewed as a technique, not the design activity itself • Like many problems that require a human process, people who don’t understand the value of testing (or doesn’t want to do it) turns it into a programming problem • AI can’t answer “understand the value of an application from interacting with its environment”

Slide 8

Slide 8 text

Metaprogramming neural networking solution can serve as a suitable option for automated techniques in testing • more efficient • require less data for analysis • far fewer unknowns with internal operations

Slide 9

Slide 9 text

• The neural network itself is called the model, but it can also be recognized as an algorithm (pixel math). • In this sense, the neural networking is act of moving data through different layers containing certain filters. • Creates the model to achieve the same goal • The relationships that are drawn, the patterns that are created, in order to solve the same problems AI does is the neural network Neural Networking in Deep Learning Neural Networking in Metaprogramming

Slide 10

Slide 10 text

What kind of problems does neural networking solve for automated testing? Supervised Learning: Includes target value Regression: Target is continuous Classification: Target is discrete

Slide 11

Slide 11 text

Metaprogramming Locator Mechanism field fieldset form textfield field textfield field field textfield field fieldset form hbox textfield hbox field textfield textfield button button AI Locator Mechanism Filt + Max pooling through dense layers Locating UI Nodes with Classification

Slide 12

Slide 12 text

• Classification - How lines are drawn around data points • Convolutional Neural Networking (CNN) model - Most popular for analyzing images • Matrix math to flatten pixel data as they’re passed through hidden convolutional layers & non- convolutional layering Image classification - locator mechanism

Slide 13

Slide 13 text

Filter 1/1 Filter 2/1

Slide 14

Slide 14 text

Filter 1/1 Filter 2/1

Slide 15

Slide 15 text

Filter 1/1 Filter 2/1

Slide 16

Slide 16 text

Filter 1/1 Filter 2/1

Slide 17

Slide 17 text

Filter 1/1 Filter 2/1

Slide 18

Slide 18 text

Filter 1/1 Filter 2/1

Slide 19

Slide 19 text

Filter 1/1 Filter 2/1

Slide 20

Slide 20 text

Filter 1/1 Filter 2/1

Slide 21

Slide 21 text

Filter 1/1 Filter 2/1

Slide 22

Slide 22 text

Filter 1/1 Filter 2/1

Slide 23

Slide 23 text

Filter 1/1 Filter 2/1

Slide 24

Slide 24 text

Filter 1/1 Filter 2/1

Slide 25

Slide 25 text

Filter 1/1 Filter 2/1

Slide 26

Slide 26 text

Filter 1/1 Filter 2/1

Slide 27

Slide 27 text

Filter 2/1 Filter 1/1

Slide 28

Slide 28 text

Filter 2/1 Filter 1/1 Filter 1/2 Filter 1/2 + +

Slide 29

Slide 29 text

Filter 2/1 Filter 1/1 Filter 1/2 Filter 1/2 + +

Slide 30

Slide 30 text

Filter 2/1 Filter 1/1 Filter 1/2 Filter 1/2 + +

Slide 31

Slide 31 text

Filter 2/1 Filter 1/1 Filter 1/2 Filter 1/2 + +

Slide 32

Slide 32 text

Filter 2/1 Filter 1/1 Filter 1/2 Filter 1/2 + +

Slide 33

Slide 33 text

Filter 2/1 Filter 1/1 Filter 1/2 Filter 1/2 + +

Slide 34

Slide 34 text

Filter 2/1 Filter 1/1 Filter 1/2 Filter 1/2 + +

Slide 35

Slide 35 text

Filter 2/1 Filter 1/1 Filter 1/2 Filter 1/2 + +

Slide 36

Slide 36 text

Filter 2/1 Filter 1/1 Filter 1/2 Filter 1/2 + +

Slide 37

Slide 37 text

Filter 2/1 Filter 1/1 Filter 1/2 Filter 1/2 + +

Slide 38

Slide 38 text

Filter 2/1 Filter 1/1 Filter 1/2 Filter 1/2 + +

Slide 39

Slide 39 text

Filter 2/1 Filter 1/1 Filter 1/2 Filter 1/2 + +

Slide 40

Slide 40 text

Filter 1/2 Filter 1/2 + + Max Pooling Max Pooling

Slide 41

Slide 41 text

Max Pooling Max Pooling Dense Weights Dense Weights Dense Activation Dense Activation

Slide 42

Slide 42 text

• Draws lines about the pixel, no knowledge of code required • Reduces image pixels through filters, but provides lots of intense matrix calculations that are opaque to us due to the # of calculations required per layer Image classification - locator mechanism "Komandyseja | Music Production" by Przemek Bizoń, bisoñ studio, Agata Łobaczuk is licensed under CC BY-NC-ND 4.0 - Modified image.

Slide 43

Slide 43 text

• Draws lines about the pixel, no knowledge of code required • Reduces image pixels through filters, but provides lots of intense matrix calculations that are opaque to us due to the # of calculations required per layer Image classification - locator mechanism "Komandyseja | Music Production" by Przemek Bizoń, bisoñ studio, Agata Łobaczuk is licensed under CC BY-NC-ND 4.0 - Modified image. TextView InteractivePlayerView Floating Action Button ScrollView RelativeLayout Linear Layout

Slide 44

Slide 44 text

• Abstract Syntax Tree (AST) parser draws lines around the shape of the written language parsed into an intermediate representation of an AST The Metaprogramming Locator Mechanism

Slide 45

Slide 45 text

• Abstract Syntax Tree (AST) parser draws lines around the shape of the written language parsed into an intermediate representation of an AST The Metaprogramming Locator Mechanism

Slide 46

Slide 46 text

• Abstract Syntax Tree (AST) parser draws lines around the shape of the written language parsed into an intermediate representation of an AST The Metaprogramming Locator Mechanism

Slide 47

Slide 47 text

• Abstract Syntax Tree (AST) parser draws lines around the shape of the written language parsed into an intermediate representation of an AST The Metaprogramming Locator Mechanism

Slide 48

Slide 48 text

• Abstract Syntax Tree (AST) parser draws lines around the shape of the written language parsed into an intermediate representation of an AST The Metaprogramming Locator Mechanism

Slide 49

Slide 49 text

• Abstract Syntax Tree (AST) parser draws lines around the shape of the written language parsed into an intermediate representation of an AST The Metaprogramming Locator Mechanism

Slide 50

Slide 50 text

• Abstract Syntax Tree (AST) parser draws lines around the shape of the written language parsed into an intermediate representation of an AST The Metaprogramming Locator Mechanism

Slide 51

Slide 51 text

• Abstract Syntax Tree (AST) parser draws lines around the shape of the written language parsed into an intermediate representation of an AST The Metaprogramming Locator Mechanism

Slide 52

Slide 52 text

• Abstract Syntax Tree (AST) parser draws lines around the shape of the written language parsed into an intermediate representation of an AST The Metaprogramming Locator Mechanism

Slide 53

Slide 53 text

• Abstract Syntax Tree (AST) parser draws lines around the shape of the written language parsed into an intermediate representation of an AST The Metaprogramming Locator Mechanism

Slide 54

Slide 54 text

• Abstract Syntax Tree (AST) parser draws lines around the shape of the written language parsed into an intermediate representation of an AST The Metaprogramming Locator Mechanism

Slide 55

Slide 55 text

• Abstract Syntax Tree (AST) parser draws lines around the shape of the written language parsed into an intermediate representation of an AST The Metaprogramming Locator Mechanism

Slide 56

Slide 56 text

• Abstract Syntax Tree (AST) parser draws lines around the shape of the written language parsed into an intermediate representation of an AST The Metaprogramming Locator Mechanism

Slide 57

Slide 57 text

override val root = hbox { form { fieldset("Edit person") { field("Owner") { textfield(model.ownerName) { ownerNameField = this } } field("Cat") { textfield(model.catName) { catNameField = this } } field("Time") { textfield(model.time) { timeField = this } } button("Save") { enableWhen(model.dirty) action { save() } } } } } field fieldset form textfield field textfield field field textfield field fieldset form hbox textfield hbox field textfield textfield button button The Metaprogramming Locator Mechanism

Slide 58

Slide 58 text

No content

Slide 59

Slide 59 text

No content

Slide 60

Slide 60 text

Metaprogramming Locator Mechanism field fieldset form textfield field textfield field field textfield field fieldset form hbox textfield hbox field textfield textfield button button AI Locator Mechanism Filt + Max pooling through dense layers Locating UI Nodes with Classification

Slide 61

Slide 61 text

Metaprogramming Locator Mechanism field fieldset form textfield field textfield field field textfield field fieldset form hbox textfield hbox field textfield textfield button button AI Locator Mechanism O(mn3) Filt + Max pooling through dense layers Locating UI Nodes with Classification O(n)

Slide 62

Slide 62 text

Future work: Regression analysis Supervised Learning: Includes target value Regression: Target is continuous Classification: Target is discrete http://testerstories.com/2018/08/can-an-ai-become-a-tester/

Slide 63

Slide 63 text

• Regression - How lines draw relationships from one point to another • How can we increase combinations of permutations of interactions? Future work: Making UI tests “smarter”

Slide 64

Slide 64 text

Artificial Neural Networking Improving UI Testing with Regression Abstract Compositional Contracting

Slide 65

Slide 65 text

Recap • Using automation can help the tester, but doesn’t replace the role of a test • Neural networking in metaprogramming is isomorphic to AI neural networking • AI models for regression and classification is expensive and difficult to maintain and wrangle • You don’t need a sledgehammer for a nail Supervised Learning: Regression Classification

Slide 66

Slide 66 text

• TornadoFX-Suite: https://github.com/ahinchman1/TornadoFX-Suite • Automata UI research: https://github.com/ahinchman1/Finite-State-Machine-Crash-Course • Data science research: https://github.com/ahinchman1/Data-Science-Crash-Course Amanda Hinchman-Dominguez @hinchman_amanda Relevant Links