[Kubecon Europe 2023] Ingesting 6.5 Tb of Telemetry Data Daily Through Open Telemetry Protocol and Collectors

Presentation video: https://www.youtube.com/watch?v=aDysORX1zIs

This presentation aims to share how VTEX observability team moved from a single vendor to a full Open Telemetry protocol solution that handles 6.5 terabytes of telemetry data per day (logs, system metrics, business metrics, traces and audit logs). Thinking on the CNCF community, this talk will show the entire architecture, the tradeoffs, how to instrument every application inside the company, how to manage OTEL Collectors at scale, how to centralize visualization, how to extend collectors code and how to guarantee resiliency. Open Telemetry allowed VTEX to completely modernize its Observability stack looking to a horizon of at least 5 years ahead without requiring any sort of migrations on the VTEX's applications. With the architecture this talk presents, VTEX can switch backend vendors without impacting instrumented code. Thus, allow engineering organization to move faster. Last but not least, this solution made VTEX reduce 40% of its Observability costs while enabling a modern, safer and efficient way to engineers to observe their applications at scale. If these topics are interesting to you, please come to this presentation. The idea is to give back to CNCF community what they gave to us: knowledge and cutting edge solutions.

kubecon europe 2023
Tuesday April 18, 2023 11:55 - 12:20 CEST
Hall 7, Room E | Ground Floor | Europe Complex
Observability Day, Project-specific: Observability + Prometheus + OpenMetrics + OpenTelemetry +Fluentbit

Transcript

1. Ingesting 6.5 Tb of Telemetry Data Daily Through Open Telemetry Protocol and Collectors Gustavo Pantuza
3. About me Personal blog on Computer Science [pt-BR] https://blog.pantuza.com @gpantuza pantuza
4. About VTEX platform Problem Solution + Outcomes Architecture Agenda Resilience
5. About VTEX What kind of system are we observing? 01
6. Build, manage, and deliver profitable ecommerce businesses with more agility and less risk. The Enterprise Digital Commerce Platform VTEX context
7. VTEX at a glance 3.200+ Active online stores 38 Countries with active customers Public-listed company
8. • LONDON • BARCELONA • BUCUREȘTI WHERE WE ARE Locations across the globe 18 Global Platform • BUENOS AIRES SANTIAGO • BOGOTÁ • • CIUDAD DE MÉXICO • NEW YORK • JOÃO PESSOA • MEDELLÍN MILAN • • RIO DE JANEIRO SÃO PAULO • LIMA • LISBON • • SINGAPORE Employees 1.3 k • RECIFE As of Q2/21 ended on June 30th, 2021 PARIS •
9. Problem Where is the problem within this context? 01
10. Problem
11. Inefficient ingestion control HTTP 1.1 with no encryption Unstructured telemetry data No common fields by design Problem Too many library implementations Telemetry data governance Inefficient way to store KPIs Single vendor for different telemetry signals
12. Problem How to evolve to a long-term o11y stack without vendor lock-in while improving o11y efficiency?
13. Solution + Outcomes How did we solve the problem and what are the outcomes? 01
14. Solution + Outcomes
15. Solution + Outcomes Open Telemetry Protocol on every possible layer Common libraries with the same interfaces Open Telemetry Collectors as Telemetry Ingestor Different data sinks to different telemetry signals Sharded architecture by Telemetry signal
16. 41% reduction on Observability investments Long Term Architecture that does not require developers to migrate their applications in case we change o11y vendors 6.5 Tb of telemetry data getting ingested per day and with control knobs such as dynamic sampling Solution + Outcomes
17. Architecture What is the project design? 01
18. Architecture Resiliency control Asynchronous requests Head based sampling Shared library (Diagnostics) Facilitates the migration
19. Architecture gRPC communication Encryption Common methods interface Shared library (Diagnostics) /** * VTEX's Telemetry methods */ service Telemetry { /* Logs related methods */ rpc Info(LogRequest) returns (google.protobuf.Empty); rpc Warn(LogRequest) returns (google.protobuf.Empty); rpc Error(LogRequest) returns (google.protobuf.Empty); rpc Debug(LogRequest) returns (google.protobuf.Empty); /* Metrics related methods */ rpc SystemMetric(Metric) returns (google.protobuf.Empty); rpc BusinessMetric(Metric) returns (google.protobuf.Empty); /* Traces related methods */ rpc Trace(Trace) returns (google.protobuf.Empty); }
20. Architecture Structured logs by design Common fields Shared library (Diagnostics) /** * Common Fields on telemetry data */ message Common { ... /* Name of the service that is sending telemetry data */ string service_name = 1; /* Instance hostname */ string instance_id = 2; /* Instance Availability zone */ string az = 3; /* Instance region */ string region = 4; /* Optional hash table allowing users to send extra fields */ map<string, string> extra_fields = 5; ... } Instrumented with Open Telemetry official libraries
21. Architecture Single telemetry signal architecture
22. Architecture Extended with our modules Built internally Different configurations per telemetry signal Custom Collectors Open Telemetry Collectors Builder (ocb)
23. dist: name: otelcol description: OpenTelemetry Collector version: 0.xx.y otelcol_version: 0.xx.y receivers: - ... exporters: - ... extensions: - ... processors: - ... Architecture Extended with our modules Built internally Different configurations per telemetry signal Custom Collectors builder.yaml
24. Architecture Telemetry data flow
25. Architecture Reusable dashboards Central place for dashboards Specialized visualizations on each data sink Data visualization Teams governance
26. Architecture
27. 4 Terabytes of logs per day 150 millions Active time series 2.15 Billions of events ingested per day Architecture
28. Resilience How do we keep this system up and running? 01
29. “Fail locally, not globally” Resilience
30. Resilience
31. Horizontal Pod auto-scaling Burst requests on a single deployment can increase significantly collectors load autoscaling: enabled: true minReplicas: 10 maxReplicas: 20 targetMemoryUtilizationPercentage: 90 targetCPUUtilizationPercentage: 60 One Example of single Deployment Resilience
32. Sharding strategy Sharded environments by business criteria such as core systems vs internal services Shard 0 Shard 1 Shard 2 Shard 3 Logs Metrics Traces Resilience
33. settings: default_sampling_percentage: 25 skip_sampling_field: debug services_config: sampling: - name: service_0 index: service_0 percentage: 75 - name: service_1 index: service_1 percentage: 0 Default sampling Sampling % per service Skip sampling Real time Sampling (tail based sampling) Resilience Custom Open Telemetry Collector Processor
34. wal: s3_region: some-aws-region s3_bucket: "wal-bucket-name" flush: bytes: 10000000 interval: 120 cluster: shard0 Data Backfill one week data expiration AWS S3 + Lambda functions Write Ahead Log (WAL) Resilience Custom Open Search exporter
35. We monitor all collectors from all pipelines and shards. Based on this monitoring we trigger alerts that pages the team OnCall engineer. Resilience Alerting
36. Migration tips RFC like process to engage engineering teams C levels buy-in on the project Understand your client (speak to your teams) Engage your vendors on the resiliency plan Find early adopters. One step at a time
37. VTEX Context We saw the multi-tenant architecture Problem We understood the problem of using a single vendor for different telemetry data Solution + Outcomes Before jumping on details we saw the overall approach and direct outcomes Architecture Then we jumped into details of the long term architecture and system design Recap Resilience Finally we discussed strategies to avoid global outages and how to monitor the entire architecture
38. Open Telemetry ecosystem enabled VTEX to innovate fast and efficiently
39. Personal blog on Computer Science [pt-BR] https://blog.pantuza.com @gpantuza pantuza