Upgrade to Pro — share decks privately, control downloads, hide ads and more …

EXTENT-2019: Dancing with Whales. Adaptive Log Classification System

Exactpro
PRO
September 17, 2019

EXTENT-2019: Dancing with Whales. Adaptive Log Classification System

Dancing with Whales. Adaptive Log Classification System
Elena Treshcheva
Business Development Manager & Researcher, Exactpro

EXTENT-2019
17 September 2019, London

Video: https://youtu.be/_VSztM9m5fw

To learn more about Exactpro, visit our website https://exactpro.com/
EXTENT Conference website: https://extent.exactpro.com/
Follow us on
LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro
Facebook https://www.facebook.com/exactpro/
Instagram https://www.instagram.com/exactpro/

Subscribe to Exactpro YouTube channel https://www.youtube.com/c/exactprosystems

Exactpro
PRO

September 17, 2019
Tweet

More Decks by Exactpro

Other Decks in Technology

Transcript

  1. Dancing with Whales:
    Adaptive Log Classification System
    Elena Treshcheva,
    Business Development Manager,
    Researcher, Exactpro

    View Slide

  2. Problem vs. Catastrophe

    View Slide

  3. What turns a problem into a disaster?

    View Slide

  4. Mission-Critical Software - Lots of log
    messages daily

    View Slide

  5. Huge volumes of data vs. error class clusters

    View Slide

  6. WhaleShark
    WhaleShark
    The First IEEE International Conference on Artificial Intelligence Testing
    (IEEE AITest 2019), April 4-9 2019, San Francisco East Bay, CA, USA
    User-Assisted Log Analysis for Quality Control of Distributed Fintech
    Systems
    Iosif Itkin, Anna Gromova, Anton Sitnikov, Rostislav Yavorskiy, Evgenii Tsymbalov,
    Andrey Novikov and Kirill Rudakov.

    View Slide

  7. One error class - different text strings
    text text text text
    0100101010001010000
    ERROR
    text text text text
    text text text text
    text text text text
    text text text text
    FATAL……………………………………
    text text text text
    text text text text
    It’s so cold and
    dark... I’m scared...

    View Slide

  8. Step 1: Normalization
    ERROR 07 Aug 2018 13:04:39 - Unable to load messages from file
    '/UAT/collectors/mq1'
    ERROR 25 Sep 2018 18:24:55 - Unable to load messages from file
    '/prod/collectors/mq3'
    ERROR 16 Oct 2018 15:15:10 - Unable to load messages from file
    '/prod/collectors/mq5'

    View Slide

  9. Step 1: Normalization
    ERROR 07 Aug 2018 13:04:39 - Unable to load messages from file
    '/UAT/collectors/mq1'
    ERROR 25 Sep 2018 18:24:55 - Unable to load messages from file
    '/prod/collectors/mq3'
    ERROR ** *** **** **:**:** - Unable to load messages from file
    '/*/*/*'
    ERROR 16 Oct 2018 15:15:10 - Unable to load messages from file
    '/prod/collectors/mq5'

    View Slide

  10. Step 2: Clustering

    View Slide

  11. Final step: User Notification

    View Slide

  12. Result: System state visualization

    View Slide

  13. More Control over system outputs

    View Slide

  14. Thank you!
    14

    View Slide