Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Centralized Log System - Rostislav Aronov

Centralized Log System - Rostislav Aronov

Avatar for Sela Developer Meetup

Sela Developer Meetup

May 30, 2018
Tweet

More Decks by Sela Developer Meetup

Other Decks in Technology

Transcript

  1. Centralized Log system Rostislav Aronov There are some things in

    this world, captain Niobe, that will never change ... And some things do Change Morpheus, Matrix Reloaded
  2. Rostislav Aronov - Software Architect • 25 years in the

    SW industry • Complex systems based on SOA methodologies. • Cloud oriented distributed system. • Lead software projects in architecture, project design, construction and delivery. • Deep knowledge in wide technology spectrum, data warehousing and Big Data principles. • Financial, medical, security, gaming, multimedia and internet/web industry experience.
  3. Agenda • Log system aspects o Distributed systems and ephemeral

    servers o Importance for collecting applications metrics o Instrumentation from day one o Decoupling from business code o Minimal effect on service performance o Support different log types o Support different log levels o Simplicity o Visualization of collected metrics • Implementation – Code + Demo o AOP framework PostSharp for logs collection o ETW for offloading application logs o Sync service-shipping ETW logs to ES o Beats for access, events and system log transportation o Logstash for log parsing and data enrichment o ES as a data store o Analysis and visualization in Kibana
  4. Log system aspects "If you want peace, prepare for war”

    Publius Flavius Vegetius Renatus (late 4th century)
  5. Distributed systems and ephemeral servers • Cloud is a new

    normal • Almost every system is data driven • Big Data buzzword – Fast data streaming or TBs of data in storage • Almost every system is n-tear distributed – web(UI), business(server), DB • Elastic resources in a cloud • Pay as you go – less is more • 24/7 - Disaster recovery and failover • Only event-driven auto-reactive system survive • All components are ephemeral – no relationships between you and servers Log system aspects
  6. Log system aspects Services - Microservices or not to be?

    Cost of effort Number of services Indirect Cost Direct Cost Minimum cost *Taken from IDesign, Project Design Master Class
  7. Importance for collecting applications metrics • Analysis • Alarm system

    • Debug • Call chain • Event tracking • Visualization/Dashboards Log system aspects
  8. Decoupling from business code • Ability to read • Ability

    to understand • Ability to change • Ability to test Log system aspects
  9. Simplicity Log system aspects A Rube Goldberg machine is a

    machine intentionally designed to perform a simple task in an indirect and overcomplicated fashion. Often, these machines consist of a series of simple devices that are linked together to produce a domino effect, in which each device triggers the next one, and the original goal is achieved only after many steps. Over the years, the expression has expanded to mean any confusing or complicated system. For example, news headlines include "Is Rep. Bill Thomas the Rube Goldberg of Legislative Reform?"[1] and "Retirement 'insurance' as a Rube Goldberg machine".[2]
  10. Architecture Post Sharp Code File Beat Log Files Metric Beat

    System Win Log Beat Win Events ETW ETW Sync Log Stash Elastic Search Kibana Nest
  11. ETW for offloading application logs Implementation Event Tracing for Windows

    (ETW) is an efficient kernel-level tracing facility that lets you log kernel or application-defined events to a log file. You can consume the events in real time or from a log file and use them to debug an application or to determine where performance issues are occurring in the application. ETW lets you enable or disable event tracing dynamically, allowing you to perform detailed tracing in a production environment without requiring computer or application restarts.
  12. Sync service-shipping ETW logs to ES Implementation • Observe ETW

    evens • Ship events to the destination defined in sink • Change log level without restarting services • Has a list built-in sink’s • We developed custom sink to ship events to ES
  13. ES as a data store • NoSql • Document DB

    – json • Distributed DB • Incapsulates Lucene • Fast • Low maintenance Implementation
  14. Demo • Metrics Beat for system metrics transportation • Semantic

    Logging service and custom ES-sink for application log transportation • Analysis and visualization in Kibana Implementation
  15. Summary This is your last chance. After this, there is

    no turning back. You take the blue pill—the story ends, you wake up in your bed and believe whatever you want to believe. You take the red pill—you stay in Wonderland, and I show you how deep the rabbit hole goes. Morpheus