Observing Python applications with OpenTelemetry

In this talk we’ll see the options we have to add observability to Python applications with OpenTelemetry without touching our application code. In OpenTelemetry this is called autoinstrumentation or zero-code instrumentation.
We’ll see:

- The OpenTelemetry operator, a Kubernetes operator to manage collection and inject autoinstrumentation to the pods
- OpenTelemetry Python own autoinstrumentation solution based on the opentelemetry-instrument wrapper
- OpenTelemetry Python programmatic autoinstrumentation with 2 lines of code

We’ll evaluate the differences between the three solutions highlighting their strong and weak spots.

We are also taking a look on the work done inside the OpenTelemetry project around GenAI.

Transcript

1. Observing Python applications with OpenTelemetry Riccardo Magliocchetti PyCon Italia, 20250531
2. Riccardo Magliocchetti Maintainer OpenTelemetry Python Senior Software Engineer Elastic Observability
3. Agenda Observing Python applications with OpenTelemetry
4. Agenda Observing Python applications with OpenTelemetry: • OpenTelemetry Kubernetes Operator
5. Agenda Observing Python applications with OpenTelemetry: • OpenTelemetry Kubernetes Operator • OpenTelemetry Python with opentelemetry-instrument
6. Agenda Observing Python applications with OpenTelemetry: • OpenTelemetry Kubernetes Operator • OpenTelemetry Python with opentelemetry-instrument • OpenTelemetry Python programmatic auto-instrumentation
7. Agenda Observing Python applications with OpenTelemetry: • OpenTelemetry Kubernetes Operator • OpenTelemetry Python with opentelemetry-instrument • OpenTelemetry Python programmatic auto-instrumentation • GenAI observability with OpenTelemetry
8. Observability and OpenTelemetry
9. Observability Observability is the ability to understand the internal state of a system by examining its outputs.
10. OpenTelemetry A open source observability framework providing specifications and implementations in order to create and manage telemetry data. It is a CNCF incubating project.
11. OpenTelemetry signals • Traces, requests paths
13. OpenTelemetry signals • Traces, requests paths • Metrics, measurements
15. OpenTelemetry signals • Traces, requests paths • Metrics, measurements • Logs, time stamped text
17. OpenTelemetry signals • Traces, requests paths • Metrics, measurements • Logs, time stamped text • Profiles, profiling
18. Auto-instrumentation with OpenTelemetry Operator for Kubernetes
19. opentelemetry-operator A Kubernetes operator that manages: • An OpenTelemetry collector instance
20. opentelemetry-operator A Kubernetes operator that manages: • An OpenTelemetry collector instance • A system to inject auto-instrumentation
21. opentelemetry-operator The managed OpenTelemetry collector instance: • Simplifies management of secrets
22. opentelemetry-operator The managed OpenTelemetry collector instance: • Simplifies management of secrets • Useful for processing, opentelemetry.io/docs/collector/transforming-telemetry/
23. opentelemetry-operator
24. opentelemetry-operator Recent improvements: • Less configuration variables
25. opentelemetry-operator Recent improvements: • Less configuration variables • Support for musl libc based container images
26. opentelemetry-operator apiVersion: opentelemetry.io/v1alpha1 kind: OpenTelemetryCollector metadata: name: demo spec: config: | receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 processors: exporters: debug: service: pipelines: traces: receivers: [otlp] processors: [] exporters: [debug] metrics: receivers: [otlp] processors: [] exporters: [debug] logs: receivers: [otlp] processors: [] exporters: [debug]
27. opentelemetry-operator apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: otel-python-instrumentation spec: exporter: endpoint: http://demo-collector:4318 propagators: - tracecontext - baggage sampler: type: parentbased_traceidratio argument: "1"
28. opentelemetry-operator apiVersion: apps/v1 kind: Deployment metadata: name: your-app-deployment spec: replicas: 1 selector: matchLabels: app: python-otel-app template: metadata: annotations: instrumentation.opentelemetry.io/inject-python: "otel-python-instrumentation" # default is glibc instrumentation.opentelemetry.io/otel-python-platform: "musl" labels: app: python-otel-app spec: containers: …
29. opentelemetry-operator: the bad One OpenTelemetry Operator Python auto-instrumentation image: • Mind the binary wheels
30. opentelemetry-operator: the bad One OpenTelemetry Operator Python auto-instrumentation image: • Mind the binary wheels • Python version support of dependencies
31. opentelemetry-operator apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: otel-python-instrumentation spec: python: image: yourimage:1.0 exporter: endpoint: http://demo-collector:4318 propagators: - tracecontext - baggage sampler: type: parentbased_traceidratio argument: "1"
32. Auto-instrumentation with opentelemetry-instrument
33. opentelemetry-instrument Wrap your command with opentelemetry-instrument
34. opentelemetry-instrument FROM python:3.12-slim WORKDIR /app COPY . /app RUN pip install flask elastic-opentelemetry # Install instrumentations for the installed packages, custom version of opentelemetry-bootstrap RUN edot-bootstrap -a install # default flask run port EXPOSE 5000 # Set some resource attributes to make our service recognizable ENV OTEL_RESOURCE_ATTRIBUTES="service.name=FlaskService,service.version=1.0,deployment.environment=development" CMD ["opentelemetry-instrument", "flask", "run"]
35. opentelemetry-instrument export OTEL_EXPORTER_OTLP_ENDPOINT=https://my-deployment.apm.us-west1.gcp.cloud.es.io export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer P....l" docker run \ -e OTEL_EXPORTER_OTLP_ENDPOINT="$OTEL_EXPORTER_OTLP_ENDPOINT" \ -e OTEL_EXPORTER_OTLP_HEADERS="$OTEL_EXPORTER_OTLP_HEADERS" \ -p 5000:5000 -it --rm edot-flask:latest
36. opentelemetry-instrument For internals see Anatomy of a Python OpenTelemetry Instrumentation
37. Programmatic auto-instrumentation
38. Programmatic auto-instrumentation from opentelemetry.instrumentation import auto_instrumentation auto_instrumentation.initialize()
39. Auto-instrumentation recap
40. Auto-instrumentation recap Solution Application lines changed Requirements Notes OpenTelemetry Operator 0 Kubernetes May need to bring your own image opentelemetry-instrument 0 Wrap application entry point Programmatic auto-instrumentation 2 Change application code Configuration via environment variables
41. GenAI observability with OpenTelemetry
42. GenAI • Writing semantic conventions
43. GenAI • Writing semantic conventions • Writing instrumentations
44. GenAI: semantic conventions • Span attributes for chat completions • Span attributes for embeddings • Span attributes for agents calls
46. GenAI: semantic conventions • Span attributes for chat completions • Span attributes for embeddings • Span attributes for agents calls • Events for chat: ◦ Various messages roles ◦ Tool calls ◦ Responses
48. GenAI: semantic conventions • Span attributes for chat completions • Span attributes for embeddings • Span attributes for agents calls • Events for chat: ◦ Various messages roles ◦ Tool calls ◦ Responses • Metrics: ◦ Duration of operation ◦ Number of input/output tokens
50. GenAI Python instrumentations • OpenAI: streaming, sync, async chat completions
51. GenAI Python instrumentations • OpenAI: streaming, sync, async chat completions • AWS Bedrock: ◦ Converse, ConverseStream ◦ InvokeModel, InvokeModelWithStreamResponse ▪ Amazon Titan and Nova ▪ Anthropic models
52. GenAI Python instrumentations • OpenAI: streaming, sync, async chat completions • AWS Bedrock: ◦ Converse, ConverseStream ◦ InvokeModel, InvokeModelWithStreamResponse ▪ Amazon Titan and Nova ▪ Anthropic Claude • Google VertexAI
53. GenAI Python instrumentations • OpenAI: streaming, sync, async chat completions • AWS Bedrock: ◦ Converse, ConverseStream ◦ InvokeModel, InvokeModelWithStreamResponse ▪ Amazon Titan and Nova ▪ Anthropic Claude • Google VertexAI • Google GenAI
54. GenAI: whatʼs next? • Complete all the instrumentations
55. GenAI: whatʼs next? • Complete all the instrumentations • OpenLLMetry instrumentation donation proposal
56. GenAI: whatʼs next? • Complete all the instrumentations • OpenLLMetry instrumentation donation proposal • MCP semantic conventions
57. Conclusions
58. Conclusions • OpenTelemetry Operator for Kubernetes • opentelemetry-instrument to wrap your application entry point • Programmatic auto-instrumentation if you can add two lines of code and set some environment variables • Thereʼs working support for GenAI instrumentation
59. Contacts and references • https://opentelemetry.io • opentelemetry-operator • opentelemetry-python and opentelemetry-python-contrib • #otel-python on CNCF slack • @rmistaken / @rmistaken@hachyderm.io • speakerdeck.com/xrmx
60. Thank you!