Building an Adaptive Log Classification System. An Industrial Report

Kirill Rudakov, Andrey Novikov, Anton Sitnikov, Evgenii Tsymbalov, Elena Treshcheva and Alexey Zverev

International Conference on Software Testing, Machine Learning and Complex Process Analysis (TMPA-2019)
7-9 November 2019, Tbilisi

Video: https://youtu.be/6IGRVRKL_M4

TMPA Conference website https://tmpaconf.org/
TMPA Conference on Facebook https://www.facebook.com/groups/tmpaconf/

Transcript

1. 7-9 November, Tbilisi Ivane Javakhishvili Tbilisi State University 7-9 November, Tbilisi Ivane Javakhishvili Tbilisi State University Anton Sitnikov, Kirill Rudakov, Andrey Novikov, Evgeny Tsymbalov, Elena Trescheva, Alexey Zverev Exactpro Building an Adaptive Logs Classification System An industrial report
2. 7-9 November, Tbilisi Ivane Javakhishvili Tbilisi State University Logs analysis in Fintech world • Any feasible way to improve software quality is highly demanded • Logs observation for passive testing – an elegant and non-disruptive way for early error discovery • Log size is the problem
3. 7-9 November, Tbilisi Ivane Javakhishvili Tbilisi State University Reducing complexity
4. 7-9 November, Tbilisi Ivane Javakhishvili Tbilisi State University From M’s of log lines to ~10..100k of signatures • A signature is a log line with values replaced by placeholders (TIMESTAMP, ID, URL, and many more) • Signature extraction is made by regular expressions. They need to be manually added over time
5. 7-9 November, Tbilisi Ivane Javakhishvili Tbilisi State University …and then to 1000’s of clusters Cluster is a set of close signatures, ideally meaning same error type for a human engineer Why signatures are not enough? • ID’s are difficult to catch • Words can be added/deleted, which leads to different signatures • Fast growing clusters are sign of missing signature extraction rule
6. 7-9 November, Tbilisi Ivane Javakhishvili Tbilisi State University Clustering approaches K-means on log lines Train by several days; see what’s new on new days. A signature makes new cluster if distant from all existing. + Can process massive logs + User defines radius ad hoc - Clusters are moving over time Agglomerative on signatures Fix cluster diameter; make new cluster if signature distant from existing clusters. Keep cluster history forever + Clusters stable, consistent history - Slower - Lots of small clusters, need to introduce “user macro-clusters”
7. Initial classification New logs processing New logs (hour to day) Old clusters do not move. A user is notified when new cluster appears Initial class structure 100k - 1M records a day ~10k signatures ~1k dense clusters Log archive (week) Hundreds of user clusters UI shows user clusters Human classification Vectorization in space of 1,2,3-grams k-means-facilitated human overview, population of user clusters Greedy clustering to Jaccard-dense clusters Signature extraction. Vectorization in space of 1,2,3-grams Two clusters mean the same? Human joins them to a user cluster and names it. User cluster contain single of multiple dense clusters New 1,2,3-gram? Add dimension, re-evaluate distances New signature? Calculate distance to old signatures Is adding to existing cluster break Jaccard compactness criterion? Then it is a new cluster.
8. 7-9 November, Tbilisi Ivane Javakhishvili Tbilisi State University Business use cases • See new error types for day • Find cluster (exact or nearest) by raw log line • Compare error portraits of two days (and test runs)
9. 7-9 November, Tbilisi Ivane Javakhishvili Tbilisi State University Implementation
10. 7-9 November, Tbilisi Ivane Javakhishvili Tbilisi State University DB/backend queries • What new clusters appeared today? • What clusters appeared on Day X but not on Day Y? • When did Cluster X appears during the day, and how often? • What cluster does this log line belong to? 10
11. REST Middleware for UI Clustering (after each parsing, called by parser/bash) UI server (React or Flask) Parsing (Regular, Bash-called) Model storage (file) • Vocabulary • Signatures x Vocabulary • Distances (Jaccard matrix) • Signatures • Clusters & Rangers • Settings mySQL Clustering Settings Macro-clusters UI settings signatures, appearances (each) Read cluster & new signatures Save re-calculated clusters Read clusters
12. 7-9 November, Tbilisi Ivane Javakhishvili Tbilisi State University
13. 7-9 November, Tbilisi Ivane Javakhishvili Tbilisi State University Q&A
14. 7-9 November, Tbilisi Ivane Javakhishvili Tbilisi State University Thanks