Tracing - a Journey to Tactical Insights by Florian Kückelkorn

Slide 1

Slide 1 text

1 Tracing A journey to tactical insights Florian Kuckelkorn

Slide 2

Slide 2 text

2 Speaker Tracing – A journey to tactical insights Florian Kuckelkorn Technical Coordinator [email protected]

Slide 3

Slide 3 text

3 Content Tracing – A journey to tactical insights o Tracing for the non technical audience o Tracing for the technical audience o Post – Processing of traces o Tracing in the scope of G DATA CyberDefense

Slide 4

Slide 4 text

4 Tracing – A journey to tactical insights Tracing for the non-technical audience

Slide 5

Slide 5 text

5 Tracing – A journey to tactical insights Spans, Traces, Post-Processing

Slide 6

Slide 6 text

6 Crime Scene Tracing – A journey to tactical insights

Slide 7

Slide 7 text

7 Crime Scene Tracing – A journey to tactical insights The Jewel Robbery at the Grand Metropolitan A g a th a C h ris tie s P o iro t - 1 9 2 3

Slide 8

Slide 8 text

8 Crime Scene Tracing – A journey to tactical insights

Slide 9

Slide 9 text

9 Tracing – A journey to tactical insights Persons of interest

Slide 10

Slide 10 text

10 Detective Tracing – A journey to tactical insights Arthur Hastings Hercule Poirot

Slide 11

Slide 11 text

11 Jewelry owner & possible suspects Tracing – A journey to tactical insights Mr. & Mrs. Opalsen Jewelry

Slide 12

Slide 12 text

12 Possible suspect Tracing – A journey to tactical insights Céléstine – Mrs. Opalsen lady's maid

Slide 13

Slide 13 text

13 Possible suspect Tracing – A journey to tactical insights Chambermaid of grand hotel

Slide 14

Slide 14 text

14 Tracing – A journey to tactical insights The act

Slide 15

Slide 15 text

15 Start Tracing – A journey to tactical insights Mr. and Mrs. Opalsen are getting ready for the theatre

Slide 16

Slide 16 text

16 Jewel box Tracing – A journey to tactical insights The jewelry is placed in a jewel box

Slide 17

Slide 17 text

17 Homecoming Tracing – A journey to tactical insights Mr. & Mrs. Opalsen return from the theatre

Slide 18

Slide 18 text

18 Robbery Tracing – A journey to tactical insights The Jewelry is gone !

Slide 19

Slide 19 text

19 Tracing – A journey to tactical insights Solving the case by the means of tracing

Slide 20

Slide 20 text

20 Witness interrogation Tracing – A journey to tactical insights Hotel room Mrs. Opalsen Hotel room Mr. & Mrs. Opalsen witness interrogation leads to spans and traces Hotel room Theatre Jewelry Jewelry Box unkown Celestine (maid) Chambermaid Hotel work Dinner with Celestine Hotel work

Slide 21

Slide 21 text

21 Witness interrogation Tracing – A journey to tactical insights Hotel room Mrs. Opalsen Hotel room Mr. & Mrs. Opalsen Post-Processing of span leads to possible suspects Hotel room Theatre Jewelry Jewelry Box unkown Celestine (maid) Chambermaid Hotel work Dinner with Celestine Hotel work

Slide 22

Slide 22 text

22 Witness interrogation Tracing – A journey to tactical insights Hotel room Mrs. Opalsen Hotel room Mr. & Mrs. Opalsen Child_of Spans with details Hotel room Theatre Jewelry Jewelry Box unkown Celestine (maid) Chambermaid Hotel work Dinner with Celestine Hotel work Next room

Slide 23

Slide 23 text

23 Witness interrogation Tracing – A journey to tactical insights Hotel room Mrs. Opalsen Hotel room Mr. & Mrs. Opalsen Child_of Spans with details Hotel room Theatre Jewelry Jewelry Box unkown Celestine (maid) Chambermaid Hotel work Dinner with Celestine Hotel work Next room

Slide 24

Slide 24 text

24 Witness interrogation Tracing – A journey to tactical insights Hotel room Mrs. Opalsen Hotel room Mr. & Mrs. Opalsen Child_of Spans with details Hotel room Theatre Jewelry Jewelry Box unkown Celestine (maid) Chambermaid Hotel work Dinner with Celestine Hotel work Next room

Slide 25

Slide 25 text

25 Tracing – A journey to tactical insights Tracing for the technical audience

Slide 26

Slide 26 text

26 Observability Tracing – A journey to tactical insights Metrics Tracing Logging Request-scoped events Request-scoped metrics aggregatable events D A T A

Slide 27

Slide 27 text

27 OpenTracing Specification Tracing – A journey to tactical insights https://github.com/opentracing/specification/blob/master/specification.md “Traces in OpenTracing are defined implicitly by their Spans. In particular, a Trace can be thought of as a directed acyclic graph (DAG) of Spans, where the edges between Spans are called References.”

Slide 28

Slide 28 text

28 Span Context Tracing – A journey to tactical insights optional: span baggage https://opentracing.io/docs/overview/

Slide 29

Slide 29 text

29 Tracers Tracing – A journey to tactical insights https://opentracing.io/docs/overview/tracers/

Slide 30

Slide 30 text

30 Launch Docker Swarm Stacks Tracing – A journey to tactical insights Each span one message

Slide 31

Slide 31 text

31 Raw Spandata Tracing – A journey to tactical insights

Slide 32

Slide 32 text

32 Raw Spandata Tracing – A journey to tactical insights

Slide 33

Slide 33 text

33 Raw Spandata Tracing – A journey to tactical insights

Slide 34

Slide 34 text

34 Raw Spandata Tracing – A journey to tactical insights

Slide 35

Slide 35 text

35 Tracing – A journey to tactical insights Post-Processing of traces

Slide 36

Slide 36 text

36 Tactical Intel Tracing – A journey to tactical insights Tactical Insights Dependencies RED Metrics (Rate, Error, Duration) Distributed Commit ComLayer (Waittime, …) AutoScaling Anomaly detection Feature extraction DataScience (manual) Run-Time analysis Message-loss recursive chain DataScope Knowledge Base

Slide 37

Slide 37 text

37 Stream Processing – Core Concept Tracing – A journey to tactical insights Stream Processing

Slide 38

Slide 38 text

38 Stream Processing – Core Concept Tracing – A journey to tactical insights Simple Event Processing (SEP) Route, Filter, Enrich

Slide 39

Slide 39 text

39 Stream Processing – Core Concept Tracing – A journey to tactical insights Event Stream Processing (ESP) min, max, avg

Slide 40

Slide 40 text

40 Stream Processing – Core Concept Tracing – A journey to tactical insights Complex Event Processing (CEP) Patterns, Stateful, Joins

Slide 41

Slide 41 text

41 Tracing – A journey to tactical insights Apache Flink 1.10 Stateful Computations over Data Stream

Slide 42

Slide 42 text

42 Apache Flink Tracing – A journey to tactical insights

Slide 43

Slide 43 text

43 Tracing – A journey to tactical insights Tracing in the scope of G DATA CyberDefense

Slide 44

Slide 44 text

44 G DATA CyberDefense AG Tracing – A journey to tactical insights • German IT Security company • Founded 1985 in Bochum • ~ 500 employees

Slide 45

Slide 45 text

45 G DATA CyberDefense AG Tracing – A journey to tactical insights • G DATA CyberDefense AG • First commerical AV software 1987 • Today broad product portfolio B2B / B2C • G DATA Advanced Analytics • Founded 2015 • Security Consulting, Incident Response, Malware Analysis

Slide 46

Slide 46 text

46 Some internal metrics Tracing – A journey to tactical insights No Sampling -> 250 GB per day 0,1 % Sampling -> 1 GB per day

Slide 47

Slide 47 text

47 Tracing Prototype Tracing – A journey to tactical insights https://github.com/GDATASoftwareAG/DevOpsGathering2020

Slide 48

Slide 48 text

48 Top Level Tracing – A journey to tactical insights Sample Processing

Slide 49

Slide 49 text

49 Services Tracing – A journey to tactical insights Sample Ingester Dynamic Analyser Statical Analyser Classificator

Slide 50

Slide 50 text

50 TopLevel Events Tracing – A journey to tactical insights NEW_SAMPLE_RECEIVED SAMPLE_RECEIVED SANDBOX_RUN_COMPLETE STATICAL_ANALYSIS_COMPLETE SAMPLE_CLASSIFIED

Slide 51

Slide 51 text

51 Implementation Tracing – A journey to tactical insights Sample Ingester Dynamic Analyser Statical Analyser Classificator Kafka NEW_SAMPLE_RECEIVED

Slide 52

Slide 52 text

52 Top Level Trace Tracing – A journey to tactical insights Sample Ingest Statical Analyser Dynamical Analyser Classificator Sample Processing

Slide 53

Slide 53 text

53 Launch Docker Swarm Stacks Tracing – A journey to tactical insights docker swarm init

Slide 54

Slide 54 text

54 Jaeger Collector General Tracing Architecture Tracing – A journey to tactical insights Jaeger Agent Services Post Processing Jaeger UI Database Kafka

Slide 55

Slide 55 text

55 Post Processing Architecture Tracing – A journey to tactical insights Post Processing Kafka Apache Flink GraphDB Prometheus ElasticSearch

Slide 56

Slide 56 text

56 Jaeger UI Tracing – A journey to tactical insights

Slide 57

Slide 57 text

57 RED Metrics (Rate, Error and Duration) Tracing – A journey to tactical insights PUBLISHER_RATE_GENERATOR = PROB PUBLISHER_RATE = 3 SIMULATE_S3_BEHAVIOUR = PROBABILISTIC

Slide 58

Slide 58 text

58 RED Metrics (Rate, Error and Duration) Tracing – A journey to tactical insights PUBLISHER_RATE_GENERATOR = PROB PUBLISHER_RATE = 3 SIMULATE_S3_BEHAVIOUR = CONST

Slide 59

Slide 59 text

59 RED Metrics (Rate, Error and Duration) Tracing – A journey to tactical insights PUBLISHER_RATE_GENERATOR = PROB PUBLISHER_RATE = 3 SIMULATE_S3_BEHAVIOUR = ERROR

Slide 60

Slide 60 text

60 Service Dependencies Tracing – A journey to tactical insights

Slide 61

Slide 61 text

61 Data Scope Tracing – A journey to tactical insights Tag your spans child_of fellow_from child_of

Slide 62

Slide 62 text

62 Apache Flink Tracing – A journey to tactical insights Apache Flink 1.10

Slide 63

Slide 63 text

63 Apache Flink Tracing – A journey to tactical insights Apache Flink 1.10

Slide 64

Slide 64 text

64 Apache Flink Tracing – A journey to tactical insights Processing Chain

Slide 65

Slide 65 text

65 Apache Flink Tracing – A journey to tactical insights Processing Chain

Slide 66

Slide 66 text

66 Apache Flink Tracing – A journey to tactical insights Processing Chain

Slide 67

Slide 67 text

67 Apache Flink – CEP Pattern Tracing – A journey to tactical insights

Slide 68

Slide 68 text

68 Final Words Tracing – A journey to tactical insights Your are invited to contact me: [email protected]