Apidays Singapore 2024 - OpenTelemetry for API Monitoring by Danielle Kayumbi, Capgemini

OpenTelemetry for API Monitoring
Danielle Kayumbi, Software Engineer - Capgemini

Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024)

------

Check out our conferences at https://www.apidays.global/

Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8

Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io

Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/

Transcript

1. Ope nTe l e me t r y For API M oni t or i ng
3. Danielle KAYUMBI HODIEB Software Engineer danielleKayumbi
7. API Se c ur i t y br e a c he s
8. I mpa c t t he bus i ne s s
9. AWS Historic Outage December 7th 2021
10. https://aws.amazon.com/fr/about-aws/global-infrastructure/localzones/locations/
11. . . .
12. Deliver packages
13. Netflix, Disney+ ... Stream movies
14. Play online Video games
15. https://www.supermarktblog.com/2017/11/13/antippen-statt-anstellen-wie-vapiano-und-mcdonalds-den-bestellprozess-im-schnellrestaurant-umkrempeln/ Order in fastfood restaurants
16. Delta, Southwest ... Buy airline tickets
17. Medical exams Were delayed
18. And so where Education exams
22. Deploy in prod New automatic job Scale up
23. Huge volume of requests
24. Huge volume of retry calls
25. Congestion
26. 9 hour s t o f i x
27. How t o pr e ve nt & di s c ove r ?
28. Identify outages Errors Latency Bottlenecks Third parties
29. Alert ! Send Relevant messages
30. Di s t r i but e d s ys t e ms
39. Amazon Microservices architecture https://www.divante.com/blog/10-companies-that-implemented-the-microservice-architecture-and-paved-the-way-for-others
40. Netflix More than 700 microservices https://www.divante.com/blog/10-companies-that-implemented-the-microservice-architecture-and-paved-the-way-for-others
41. Obs e r ve t he s ys t e m
42. Monitoring the operations from time to time
43. Monitoring is too deterministic
44. Ana l ys e da t a c ont i nuous l y
45. Unde r s t a nd t he r e l a t i ons hi ps be t we e n s ys t e ms
46. Obs e r va bi l i t y The ke y s uc c e s s
47. Monitoring Observability
48. Traces Metrics Logs Telemetry data
49. Ge ne r a t e , c ol l e c t , e xpor t Te l e me t r y da t a
51. He l p ma ke a s ys t e m obs e r va bl e
52. Accepted in 2019 Moved to matury level in 2021
53. OpenTracing OpenCensus
54. Funda me nt a l s
55. Di s t r i but e d t r a c i ng
56. https://eng.blackbuck.com/distributed-tracing-at-blackbuck/ Follow request journey across various systems
57. Log
58. Log Message emitted by services or other components
59. [2024-04-18T09:05:08.906198+00:00] app.INFO: Entity has been successfully persisted Log (Syslog sample)
60. Log Lack contextual information Tracking code execution
61. Log Correlated with trace & spans Included as part of a span
62. Spa n
63. Span Unit of work or operation . Tracks specific operations that a request makes.
64. { “ na me ” : “ ge t - c us t omer - ca r t ” “ c ont e xt ” : { “ t r a c e _i d”: “ 7bba 9f 33312b3dbb8b2c 2c 62bb7abe 2d”, “ s pa n_i d” : “ 086e 83747d0e 381e ” }, “ pa r e nt _i d”: “ ” , “ s t a r t _t i me” : “ 2024- 04- 18 16: 04: 01. 209458162 +0000 UTC” , “ e nd_t i me ” : “ 2024- 04- 18 16: 04: 01. 209458162 +0000 UTC” , “ s t a t us _c ode ”: “ OK” , “ a t t r i but e s ” : { “ ht t p. me t hod”: “ GET” , “ us e r _i d” : “ 123” , “ c a r t _i d” : “ 456” }}
65. Sensitive services (command/query handlers…) Controllers (Http, AMQP actions…) Repositories (database, message broker, http client…)
66. Inject span
67. Annotation
69. Di s t r i but e d t r a c e
70. Distributed trace Records the paths taken by requests, Through multi - service architecture. Trace = collection of spans
71. https://opentelemetry.io/
72. Cart service Product service API [GET] product by id API [GET] cart by id
73. Get cart operation Get products Retrieve cart db Get product operation Retrieve product db Trace Cache product Context Product service Cart service
74. Context propagation Keep correlation between signals Handle by instrumentation librairies W3C Recommendation for interoperability
75. https://www.datadoghq.com/
76. Ope nTe l e me t r y c ol l e c t or s
77. https://docs.lightstep.com/docs/quick-start-collector
78. https://www.elastic.co/blog/best-practices-instrumenting-opentelemetry
79. https://www.datadoghq.com/
80. Service Level Management Four golden signals Monitoring Dashboards
81. Four Golden Signals Latency Traffic Saturation Errors
82. La t e nc y https://cloud.google.com/blog/products/management-tools/the-right-metrics-to-monitor-cloud-data-pipelines
83. Tr a f f i c https://cloud.google.com/blog/products/management-tools/the-right-metrics-to-monitor-cloud-data-pipelines
84. Er r or s https://www.datadoghq.com/
85. Sa t ur a t i on https://cloud.google.com/blog/products/management-tools/the-right-metrics-to-monitor-cloud-data-pipelines
86. https://www.datadoghq.com/
87. Service level
88. Service Level Objective “Less than 1% of users should experience an idle of 5s”
89. Service Level Indicator Defined quantitative measure
90. Availability, latency “How many requests could be handled ?”
91. AW S out a ge r e s ol ut i on
92. https://www.thousandeyes.com/blog/aws-outage-analysis-dec-7-2021
93. Compr e he ns i ve obs e r va bi l i t y
95. Tha nk you ! danielleKayumbi