Multiplying the Value of Automated Tests @Oredev

Slide 1

Slide 1 text

DEEP ORACLES Multiplying the Value of Automated Tests [email protected] @EmanuilSlavov

Slide 2

Slide 2 text

What is an Oracle?

Slide 3

Slide 3 text

“a test oracle, is a mechanism for determining whether a test has passed or failed” - Wikipedia A deep oracle is a mechanism to detect problems, even if a test has passed. @EmanuilSlavov

Slide 4

Slide 4 text

The following techniques are suitable for high level automated tests on fully deployed application.

Slide 5

Slide 5 text

The Problem

Slide 6

Slide 6 text

Automated test are suitable only for regression testing Automated test can not ﬁnd any new bugs Automated tests give false sense of quality @EmanuilSlavov

Slide 7

Slide 7 text

Make the existing automated tests able to detect unseen and unexpected defects. @EmanuilSlavov

Slide 8

Slide 8 text

Flaky Tests

Slide 9

Slide 9 text

for i in {1..100}; do if ! execute_test ; then break; ﬁ; done; Single test execution command Stop if the test fails even once Run it 100 times

Slide 10

Slide 10 text

In the majority of the cases the fault is in the test, but sometimes it’s not… @EmanuilSlavov

Slide 11

Slide 11 text

Investigate every ﬂaky test and you may ﬁnd… @EmanuilSlavov

Slide 12

Slide 12 text

Conﬁguration Problems Misconﬁgured load balancer External resources fail to load on time - e.g. JS library DB connection pool with limited capacity @EmanuilSlavov

Slide 13

Slide 13 text

Application Problems Thread unsafe code Lack of retries in a distributed system DB connections not closed after use @EmanuilSlavov

Slide 14

Slide 14 text

Random Data

Slide 15

Slide 15 text

@EmanuilSlavov

Slide 16

Slide 16 text

Eum odit omnis impedit oﬃcia adipisci id non. random tweet '' Random Sentence Constant String Special Character random tweet Provident ipsa dolor excepturi quo asperiores animi. @someMention & random tweet Dignissimos eos accusamus aut ratione [email protected] random tweet Ut optio illum libero. Natus accusantium aliquam dolore atque voluptatum et a. http://ryanpacocha.biz/nikita random tweet @EmanuilSlavov

Slide 17

Slide 17 text

Service Virtualization Application Facebook Paypal Amazon S3 @EmanuilSlavov

Slide 18

Slide 18 text

Facebook Application Paypal Amazon S3 Proxy* Service Virtualization *github.com/emanuil/nagual

Slide 19

Slide 19 text

@EmanuilSlavov

Slide 20

Slide 20 text

Tests should be able to generate all the data that they need. @EmanuilSlavov random

Slide 21

Slide 21 text

Attack Proxy

Slide 22

Slide 22 text

App Test HTTP @EmanuilSlavov

Slide 23

Slide 23 text

App AttackProxy Test @EmanuilSlavov

Slide 24

Slide 24 text

https://api-tier.komfo.net/komfo_core/api/publish?client_id=93&team_id=981 Host: api-tier.komfo.net Content-Type: application/x-www-form-urlencoded Api-Token: 59203-242eab327550693c4b791dc01 Referer: https://web-tier.komfo.net/komfo_core/publish/composer Content-Length: 538 { "message":"Good evening everyone", "post_ad_lifetime":"0", "permission": {"type":"everyone"}, "targets":"fb_1211718002161534", "type":"status", "is_published":1, "limit_audience_options": {“ageFrom”:13,”ageTo":65,"gender":0} } SQL Injection Payloads ' '' # - - - ‘%20; ' and 1='1 ' and a='a or 1=1 or true like ‘%' ') or ‘1'='1 ' UNION ALL SELECT 1 @EmanuilSlavov

Slide 25

Slide 25 text

A Tool vs Your Tests XSS here Your tests know how to navigate your app better. @EmanuilSlavov

Slide 26

Slide 26 text

A dedicated testing environment is needed for the next set of techniques.

Slide 27

Slide 27 text

The Faults in Our Logs @EmanuilSlavov

Slide 28

Slide 28 text

The usual test relies on assertions at the last step Code execution may continue after the last step Some exceptions are caught, logged and never acted upon Look for unexpected error/exceptions in the app logs @EmanuilSlavov

Slide 29

Slide 29 text

@EmanuilSlavov

Slide 30

Slide 30 text

Known Exceptions are Excluded @EmanuilSlavov

Slide 31

Slide 31 text

If all tests pass, but there are unexpected exceptions in the logs, then fail the test run and investigate. @EmanuilSlavov

Slide 32

Slide 32 text

Bad Data

Slide 33

Slide 33 text

What is Bad Data?* Missing Bad Format Unrealistic Unsynchronized Conﬂicting Duplicated * The Quartz guide to bad data  github.com/Quartz/bad-data-guide

Slide 34

Slide 34 text

Bad data depends on the context. @EmanuilSlavov

Slide 35

Slide 35 text

One of those values was zero (0) @EmanuilSlavov If we see bad data in production we add a check for it.

Slide 36

Slide 36 text

Custom Data Integrity Checks @EmanuilSlavov

Slide 37

Slide 37 text

If all tests pass, but there is bad data, then fail the test run and investigate. @EmanuilSlavov

Slide 38

Slide 38 text

Application Metrics

Slide 39

Slide 39 text

Record various application stats after each test run Easy on dedicated environment, especially with containers With fast tests* you can tie perf bottlenecks to speciﬁc commits *Check my talk called “Need for Speed”

Slide 40

Slide 40 text

0 900 1800 2700 3600 App Log File: Lines After Each Commit 54% increase @EmanuilSlavov

Slide 41

Slide 41 text

0 11500 23000 34500 46000 Total Mongo Queries: Count After Each Commit 26% increase @EmanuilSlavov

Slide 42

Slide 42 text

Logs: lines, size, exceptions/errors count DB: read/write queries, transaction time, network connections OS: peak CPU and memory usage, swap size, disk i/o Network: 3rd party API calls, packets counts, DNS queries Language Speciﬁc: objects created, threads count, GC runs, heap size What data to collect after a test run is completed…