Observing Python applications with OpenTelemetry

In this talk we'll see the options we have to add observability to Python applications with OpenTelemetry without touching our application code. In OpenTelemetry this is called autoinstrumentation or zero-code instrumentation. We'll see:

- The OpenTelemetry operator, a Kubernetes operator to manage collection and inject autoinstrumentation to the pods - OpenTelemetry Python own autoinstrumentation solution based on the opentelemetry-instrument wrapper

We'll evaluate the differences between the two solutions highlighting their strong and weak spots. We are also taking a look on the work done inside the OpenTelemetry project around GenAI.

Transcript

1. Observing Python applications with OpenTelemetry Riccardo Magliocchetti Incontro DevOps Italia, 20250314
2. Riccardo Magliocchetti Maintainer OpenTelemetry Python Senior Software Engineer Elastic Observability
3. Agenda Observing Python applications with OpenTelemetry: • OpenTelemetry Kubernetes Operator
4. Agenda Observing Python applications with OpenTelemetry: • OpenTelemetry Kubernetes Operator • OpenTelemetry Python with opentelemetry-instrument
5. Agenda Observing Python applications with OpenTelemetry: • OpenTelemetry Kubernetes Operator • OpenTelemetry Python with opentelemetry-instrument • OpenTelemetry Python programmatic auto-instrumentation
6. Agenda Observing Python applications with OpenTelemetry: • OpenTelemetry Kubernetes Operator • OpenTelemetry Python with opentelemetry-instrument • OpenTelemetry Python programmatic auto-instrumentation • GenAI observability with OpenTelemetry
7. Observability and OpenTelemetry
8. Observability Observability is the ability to understand the internal state of a system by examining its outputs.
9. OpenTelemetry An open source observability framework providing specifications and implementations in order to create and manage telemetry data: • Traces, requests paths
10. OpenTelemetry An observability framework providing specifications and implementations in order to create and manage telemetry data: • Traces, requests paths • Metrics, measurements
11. OpenTelemetry An observability framework providing specifications and implementations in order to create and manage telemetry data: • Traces, requests paths • Metrics, measurements • Logs, time stamped text
12. OpenTelemetry An observability framework providing specifications and implementations in order to create and manage telemetry data: • Traces, requests paths • Metrics, measurements • Logs, time stamped text • Profiles, profiling
16. Auto-instrumentation with Opentelemetry Operator for Kubernetes
17. opentelemetry-operator A Kubernetes operator that manages: • An OpenTelemetry collector instance
18. opentelemetry-operator A Kubernetes operator that manages: • An OpenTelemetry collector instance • A system to inject auto-instrumentation
19. opentelemetry-operator The managed OpenTelemetry collector instance: • Simplifies management of secrets
20. opentelemetry-operator The managed OpenTelemetry collector instance: • Simplifies management of secrets • Useful for processing, opentelemetry.io/docs/collector/transforming-telemetry/
21. opentelemetry-operator Recent improvements: • Less configuration variables
22. opentelemetry-operator Recent improvements: • Less configuration variables • Support for musl libc based container images
23. opentelemetry-operator apiVersion: opentelemetry.io/v1alpha1 kind: OpenTelemetryCollector metadata: name: demo spec: config: | receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 processors: exporters: debug: service: pipelines: traces: receivers: [otlp] processors: [] exporters: [debug] metrics: receivers: [otlp] processors: [] exporters: [debug] logs: receivers: [otlp] processors: [] exporters: [debug]
24. opentelemetry-operator apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: otel-python-instrumentation spec: exporter: endpoint: http://demo-collector:4318 propagators: - tracecontext - baggage sampler: type: parentbased_traceidratio argument: "1"
25. opentelemetry-operator apiVersion: apps/v1 kind: Deployment metadata: name: your-app-deployment spec: replicas: 1 selector: matchLabels: app: python-otel-app template: metadata: annotations: instrumentation.opentelemetry.io/inject-python: "otel-python-instrumentation" instrumentation.opentelemetry.io/otel-python-platform: "musl" # default: glibc labels: app: python-otel-app spec: containers: …
26. opentelemetry-operator: the bad One OpenTelemetry Operator Python auto-instrumentation image: • Mind the binary wheels
27. opentelemetry-operator: the bad One OpenTelemetry Operator Python auto-instrumentation image: • Mind the binary wheels • Python version support of dependencies
28. opentelemetry-operator apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: otel-python-instrumentation spec: python: image: yourimage:1.0 exporter: endpoint: http://demo-collector:4318 propagators: - tracecontext - baggage sampler: type: parentbased_traceidratio argument: "1"
29. Auto-instrumentation with opentelemetry-instrument
30. opentelemetry-instrument Wrap your command with opentelemetry-instrument
31. opentelemetry-instrument FROM python:3.12-slim WORKDIR /app COPY . /app RUN pip install flask elastic-opentelemetry # Install instrumentations for the installed packages, custom version of opentelemetry-bootstrap RUN edot-bootstrap -a install # default flask run port EXPOSE 5000 # Set some resource attributes to make our service recognizable ENV OTEL_RESOURCE_ATTRIBUTES="service.name=FlaskService,service.version=1.0,deployment.environment=development" CMD ["opentelemetry-instrument", "flask", "run"]
32. opentelemetry-instrument export OTEL_EXPORTER_OTLP_ENDPOINT=https://my-deployment.apm.us-west1.gcp.cloud.es.io export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer P....l" docker run \ -e OTEL_EXPORTER_OTLP_ENDPOINT="$OTEL_EXPORTER_OTLP_ENDPOINT" \ -e OTEL_EXPORTER_OTLP_HEADERS="$OTEL_EXPORTER_OTLP_HEADERS" \ -p 5000:5000 -it --rm edot-flask:latest
33. opentelemetry-instrument For internals see Anatomy of a Python OpenTelemetry Instrumentation
34. Programmatic auto-instrumentation
35. Programmatic auto-instrumentation from opentelemetry.instrumentation import auto_instrumentation auto_instrumentation.initialize()
36. Auto-instrumentation recap
37. Auto-instrumentation recap Solution Application lines changed Requirements Notes OpenTelemetry Operator 0 Kubernetes May need to bring your own image opentelemetry-instrument 0 Wrap application entry point Programmatic auto-instrumentation 2 Change application code Configuration via environment variables
38. GenAI observability with OpenTelemetry
39. GenAI • Writing semantic conventions
40. GenAI • Writing semantic conventions • Writing instrumentations
41. GenAI: semantic conventions • Span attributes for chat completions • Span attributes for embeddings
43. GenAI: semantic conventions • Span attributes for chat completions • Span attributes for embeddings • Events for chat: ◦ Various messages roles ◦ Tool calls ◦ Responses
45. GenAI: semantic conventions • Span attributes for chat completions • Span attributes for embeddings • Events for chat: ◦ Various messages roles ◦ Tool calls ◦ Responses • Metrics: ◦ Duration of operation ◦ Number of input/output tokens
47. GenAI Python instrumentations • OpenAI: streaming, sync, async chat completions ◦ Elastic one traces embeddings calls
48. GenAI Python instrumentations • OpenAI: streaming, sync, async chat completions ◦ Elastic one traces embeddings calls • AWS Bedrock: ◦ Converse, ConverseStream ◦ InvokeModel, InvokeModelWithStreamResponse ▪ Amazon Titan and Nova ▪ Anthropic Claude
49. GenAI Python instrumentations • OpenAI: streaming, sync, async chat completions ◦ Elastic one traces embeddings calls • AWS Bedrock: ◦ Converse, ConverseStream ◦ InvokeModel, InvokeModelWithStreamResponse ▪ Amazon Titan and Nova ▪ Anthropic Claude • Google VertexAI
50. GenAI Python instrumentations • OpenAI: streaming, sync, async chat completions ◦ Elastic traces embeddings calls • AWS Bedrock: ◦ Converse, ConverseStream ◦ InvokeModel, InvokeModelWithStreamResponse ▪ Amazon Titan and Nova ▪ Anthropic Claude • Google VertexAI • Google GenAI
51. GenAI: whatʼs next? • Complete all the instrumentations
52. GenAI: whatʼs next? • Complete all the instrumentations • OpenLLMetry instrumentation donation proposal
53. GenAI: whatʼs next? • Complete all the instrumentations • OpenLLMetry instrumentation donation proposal • Agents semantic conventions
54. Conclusions
55. Conclusions • OpenTelemetry Operator for Kubernetes • opentelemetry-instrument to wrap your application entry point • Programmatic auto-instrumentation if you can add two lines of code and set some environment variables • Thereʼs working support for GenAI instrumentation
56. Contacts and references • https://opentelemetry.io • opentelemetry-operator • opentelemetry-python and opentelemetry-python-contrib • #otel-python on CNCF slack • @rmistaken / @rmistaken@hachyderm.io • speakerdeck.com/xrmx
57. Thank you!