Making a Bag of Metrics with Open Source APM and Tracing

Chris Satter's presentation on...

Open Tracing and OpenCensus have recently merged projects to consolidate efforts for Open Source APM Metrics and Application Tracing into a new project known as Open Consensus.

For cloud adopters this could help them quickly instrument APM to get metrics and traces out of a system, but then what? You have just created a bag of metrics, how do you separate the signal from the noise and prioritize your efforts so you can develop applications with speed, accuracy and security (e.g. DevSecOps)?

Learn how to get more out of your open source metrics by seeing them in context with all of their related entities (AWS workloads, Azure workloads, on-prem workloads, browsers, mobile apps, hosts, containers, databases, load balancers, identity services, etc). Then deliver more perfect software faster than ever.

Transcript

1. ©2008–18 New Relic, Inc. All rights reserved OpenTelemetry Chris Satter
2. ©2008–18 New Relic, Inc. All rights reserved Agenda 2 A Brief History of APM (Application Performance Monitoring) Trace Context OpenTracing OpenCensus OpenTelemetry New Relic ONE
3. ©2008–18 New Relic, Inc. All rights reserved A Brief History of APM 3
4. ©2008–18 New Relic, Inc. All rights reserved 4 What is APM?
5. ©2008–18 New Relic, Inc. All rights reserved 5 APM History 1998 Wily the beginning of APM Lew Cirne founds Wily and creates the APM space focusing on Java 2006 CA Acquires Wily CA decides to add to its APM portfolio 2008 New Relic Lew Cirne founds New Relic to be a Ruby on Rails APM 2015 OpenTracing Developers from Lightstep and Jaeger agreed to define a common API called OpenTracing 2016 OpenCensus Google had enough internal adoption of Census metrics and distributed tracing to open source a combined system called OpenCensus . 2019 New Relic ONE New Relic launches the first Entity Based Observability Platform OpenTracing/OpenCensus projects merge
6. ©2008–18 New Relic, Inc. All rights reserved 6 Trace Context Abstract This specification defines standard headers and value format to propagate context information that enables distributed tracing scenarios. The specification standardizes how context information is sent and modified between services. Context information uniquely identifies individual requests in a distributed system and also defines a means to add and propagate provider-specific context information. What is Trace Context?
7. ©2008–18 New Relic, Inc. All rights reserved 7 Trace Context • provides an unique identifier for individual traces and requests, allowing trace data of multiple providers to be linked together. • provides an agreed-upon mechanism to forward vendor-specific trace data and avoid broken traces when multiple trace tools participate in a single transaction. • provides an industry standard that intermediaries, platforms, and hardware providers can support. WHY? A unified approach for propagating trace data improves visibility into the behavior of distributed applications, facilitating problem and performance analysis. The interoperability provided by trace-context is a prerequisite to manage modern micro-service based applications. What does it do?
8. ©2008–18 New Relic, Inc. All rights reserved 8 Open Tracing In 2015, developers from Lightstep and Jaeger agreed to define a common API called OpenTracing. • The goal was to encourage framework and library developers to provide instrumentation as part of their frameworks. OpenTracing defines the API only. • It does not dictate the implementation, and it does not define the wire protocol for either passing context (TraceContext) nor exporting trace data. • OpenTracing became one of the early members of Cloud Native Computing Foundation (CNCF). What is OpenTracing? Distributed tracing, also called distributed request tracing, is a method used to profile and monitor applications, especially those built using a microservices architecture. Distributed tracing helps pinpoint where failures occur and what causes poor performance. Who uses OpenTracing? IT and DevOps teams can use distributed tracing to monitor applications. Distributed tracing is particularly well-suited to debugging and monitoring modern distributed software architectures, such as microservices. Developers can use distributed tracing to help debug and optimize their code. What is OpenTracing?
9. ©2008–18 New Relic, Inc. All rights reserved 9 Open Tracing How does it work?
10. ©2008–18 New Relic, Inc. All rights reserved 10 OpenCensus • OpenCensus provides an agent per language with built-in instrumentation for metrics and distributed traces. • Its APIs are in support of its ecosystem and implementation. • Google found value in having one agent that correlated metrics, traces, and other monitoring data together and they found that an agent approach was the best way to achieve widespread adoption and consistency. What is OpenCensus? OpenCensus is a set of libraries for various languages that allow you to collect application metrics and distributed traces, then transfer the data to a backend of your choice in real time. This data can be analyzed by developers and admins to understand the health of the application and debug problems. What data does OpenCensus collect? Metrics are any quantifiable piece of data that you would like to track, such as latency in a service or database, request content length, or number of open file descriptors. Viewing graphs of your metrics can help you understand and gauge the performance and overall quality of your application and set of services. Traces show you how a request propagates throughout your application or set of services. Viewing graphs of your traces can help you understand the bottlenecks in your architecture by visualizing how data flows between all of your services. Other types of telemetry will be added to OpenCensus as the project matures. Logs will likely be added next. What is OpenCensus?
11. ©2008–18 New Relic, Inc. All rights reserved 11 OpenCensus How does it work?
12. ©2008–18 New Relic, Inc. All rights reserved 12 OpenTelemetry What is OpenTelemtry?
13. ©2008–18 New Relic, Inc. All rights reserved OpenTelemetry 13
14. ©2008–18 New Relic, Inc. All rights reserved 14 A Bag of Metrics
15. ©2008–18 New Relic, Inc. All rights reserved Got Data… Now What? 15
16. ©2008–18 New Relic, Inc. All rights reserved How does this work? 16
17. ©2008–18 New Relic, Inc. All rights reserved What about the future? 17
18. ©2008–19 New Relic, Inc. All rights reserved 18 Deliver more perfect software faster than ever with New Relic One—the industry’s first entity- centric observability platform for teams with complex environments.
19. ©2008–19 New Relic, Inc. All rights reserved What does it mean to be entity- centric? 19
20. ©2008–19 New Relic, Inc. All rights reserved Load balancer Identity service Mobile app Lambda function Browser ML service Database Queue Cluster Host Container ENTITIES
21. ©2008–18 New Relic, Inc. All rights reserved Why New Relic? 21 • New Relic has believed and contributed to OpenSource Software since its founding. We believe that it is important to correlate Application metrics with Infrastructure metrics, Logs, Events, i.e. Entities. • We do not want developers to have to choose a vendor implementation when instrumenting code. • Many companies start with OSS observibility; Prometheus, Jaeger, etc. and then move to a vendor-based solution that works with all of their open source telemetry because managing systems at scale is difficult. • 1st entity based observability platform • New Relic is automated end-to-end implementation.
22. ©2008–18 New Relic, Inc. All rights reserved Why New Relic? 22
23. ©2008–18 New Relic, Inc. All rights reserved Balance 23
24. ©2008–18 New Relic, Inc. All rights reserved New Relic Instrumentation 24 This is RHEL, but can be AWS, Azure, GCP and/or on-prem
25. Confidential ©2008–17 New Relic, Inc. All rights reserved 11 25 Service 1 Agent Service instrumentation Instrument Propagate Context Service 2 Trace Context Agent Service instrumentation New Relic Trace Data Record Trace Data Ingest Analysis Storage Instrument Visualize Traces New Relic Distributed Tracing Automated end-to-end implementation
26. ©2008–18 New Relic, Inc. All rights reserved Demo!
27. ©2008–19 New Relic, Inc. All rights reserved ©2008–19 New Relic, Inc. All rights reserved
28. ©2008–18 New Relic, Inc. All rights reserved Links & References 28 https://newrelic.com/ https://en.wikipedia.org/wiki/Application_performance_management https://en.wikipedia.org/wiki/Wily_Technology https://www.w3.org/TR/trace-context/#design-overview https://opentracing.io https://opencensus.io https://opentelemetry.io/ https://github.com/open-telemetry/community https://www.youtube.com/watch?v=IdYZphA5S7s&feature=youtu.be https://www.linkedin.com/in/andrew-chris-satter-7009531a/
29. Thank You ©2008–18 New Relic, Inc. All rights reserved