Tweet
Tweet

Slide 1

Slide 1 text

Observing Python applications with OpenTelemetry Riccardo Magliocchetti Incontro DevOps Italia, 20250314

Slide 2

Slide 2 text

Riccardo Magliocchetti Maintainer OpenTelemetry Python Senior Software Engineer Elastic Observability

Slide 3

Slide 3 text

Agenda Observing Python applications with OpenTelemetry: ● OpenTelemetry Kubernetes Operator

Slide 4

Slide 4 text

Agenda Observing Python applications with OpenTelemetry: ● OpenTelemetry Kubernetes Operator ● OpenTelemetry Python with opentelemetry-instrument

Slide 5

Slide 5 text

Agenda Observing Python applications with OpenTelemetry: ● OpenTelemetry Kubernetes Operator ● OpenTelemetry Python with opentelemetry-instrument ● OpenTelemetry Python programmatic auto-instrumentation

Slide 6

Slide 6 text

Agenda Observing Python applications with OpenTelemetry: ● OpenTelemetry Kubernetes Operator ● OpenTelemetry Python with opentelemetry-instrument ● OpenTelemetry Python programmatic auto-instrumentation ● GenAI observability with OpenTelemetry

Slide 7

Slide 7 text

Observability and OpenTelemetry

Slide 8

Slide 8 text

Observability Observability is the ability to understand the internal state of a system by examining its outputs.

Slide 9

Slide 9 text

OpenTelemetry An open source observability framework providing specifications and implementations in order to create and manage telemetry data: ● Traces, requests paths

Slide 10

Slide 10 text

OpenTelemetry An observability framework providing specifications and implementations in order to create and manage telemetry data: ● Traces, requests paths ● Metrics, measurements

Slide 11

Slide 11 text

OpenTelemetry An observability framework providing specifications and implementations in order to create and manage telemetry data: ● Traces, requests paths ● Metrics, measurements ● Logs, time stamped text

Slide 12

Slide 12 text

OpenTelemetry An observability framework providing specifications and implementations in order to create and manage telemetry data: ● Traces, requests paths ● Metrics, measurements ● Logs, time stamped text ● Profiles, profiling

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

Auto-instrumentation with Opentelemetry Operator for Kubernetes

Slide 17

Slide 17 text

opentelemetry-operator A Kubernetes operator that manages: ● An OpenTelemetry collector instance

Slide 18

Slide 18 text

opentelemetry-operator A Kubernetes operator that manages: ● An OpenTelemetry collector instance ● A system to inject auto-instrumentation

Slide 19

Slide 19 text

opentelemetry-operator The managed OpenTelemetry collector instance: ● Simplifies management of secrets

Slide 20

Slide 20 text

opentelemetry-operator The managed OpenTelemetry collector instance: ● Simplifies management of secrets ● Useful for processing, opentelemetry.io/docs/collector/transforming-telemetry/

Slide 21

Slide 21 text

opentelemetry-operator Recent improvements: ● Less configuration variables

Slide 22

Slide 22 text

opentelemetry-operator Recent improvements: ● Less configuration variables ● Support for musl libc based container images

Slide 23

Slide 23 text

opentelemetry-operator apiVersion: opentelemetry.io/v1alpha1 kind: OpenTelemetryCollector metadata: name: demo spec: config: | receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 processors: exporters: debug: service: pipelines: traces: receivers: [otlp] processors: [] exporters: [debug] metrics: receivers: [otlp] processors: [] exporters: [debug] logs: receivers: [otlp] processors: [] exporters: [debug]

Slide 24

Slide 24 text

opentelemetry-operator apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: otel-python-instrumentation spec: exporter: endpoint: http://demo-collector:4318 propagators: - tracecontext - baggage sampler: type: parentbased_traceidratio argument: "1"

Slide 25

Slide 25 text

opentelemetry-operator apiVersion: apps/v1 kind: Deployment metadata: name: your-app-deployment spec: replicas: 1 selector: matchLabels: app: python-otel-app template: metadata: annotations: instrumentation.opentelemetry.io/inject-python: "otel-python-instrumentation" instrumentation.opentelemetry.io/otel-python-platform: "musl" # default: glibc labels: app: python-otel-app spec: containers: …

Slide 26

Slide 26 text

opentelemetry-operator: the bad One OpenTelemetry Operator Python auto-instrumentation image: ● Mind the binary wheels

Slide 27

Slide 27 text

opentelemetry-operator: the bad One OpenTelemetry Operator Python auto-instrumentation image: ● Mind the binary wheels ● Python version support of dependencies

Slide 28

Slide 28 text

opentelemetry-operator apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: otel-python-instrumentation spec: python: image: yourimage:1.0 exporter: endpoint: http://demo-collector:4318 propagators: - tracecontext - baggage sampler: type: parentbased_traceidratio argument: "1"

Slide 29

Slide 29 text

Auto-instrumentation with opentelemetry-instrument

Slide 30

Slide 30 text

opentelemetry-instrument Wrap your command with opentelemetry-instrument

Slide 31

Slide 31 text

opentelemetry-instrument FROM python:3.12-slim WORKDIR /app COPY . /app RUN pip install flask elastic-opentelemetry # Install instrumentations for the installed packages, custom version of opentelemetry-bootstrap RUN edot-bootstrap -a install # default flask run port EXPOSE 5000 # Set some resource attributes to make our service recognizable ENV OTEL_RESOURCE_ATTRIBUTES="service.name=FlaskService,service.version=1.0,deployment.environment=development" CMD ["opentelemetry-instrument", "flask", "run"]

Slide 32

Slide 32 text

opentelemetry-instrument export OTEL_EXPORTER_OTLP_ENDPOINT=https://my-deployment.apm.us-west1.gcp.cloud.es.io export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer P....l" docker run \ -e OTEL_EXPORTER_OTLP_ENDPOINT="$OTEL_EXPORTER_OTLP_ENDPOINT" \ -e OTEL_EXPORTER_OTLP_HEADERS="$OTEL_EXPORTER_OTLP_HEADERS" \ -p 5000:5000 -it --rm edot-flask:latest

Slide 33

Slide 33 text

opentelemetry-instrument For internals see Anatomy of a Python OpenTelemetry Instrumentation

Slide 34

Slide 34 text

Programmatic auto-instrumentation

Slide 35

Slide 35 text

Programmatic auto-instrumentation from opentelemetry.instrumentation import auto_instrumentation auto_instrumentation.initialize()

Slide 36

Slide 36 text

Auto-instrumentation recap

Slide 37

Slide 37 text

Auto-instrumentation recap Solution Application lines changed Requirements Notes OpenTelemetry Operator 0 Kubernetes May need to bring your own image opentelemetry-instrument 0 Wrap application entry point Programmatic auto-instrumentation 2 Change application code Configuration via environment variables

Slide 38

Slide 38 text

GenAI observability with OpenTelemetry

Slide 39

Slide 39 text

GenAI ● Writing semantic conventions

Slide 40

Slide 40 text

GenAI ● Writing semantic conventions ● Writing instrumentations

Slide 41

Slide 41 text

GenAI: semantic conventions ● Span attributes for chat completions ● Span attributes for embeddings

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

GenAI: semantic conventions ● Span attributes for chat completions ● Span attributes for embeddings ● Events for chat: ○ Various messages roles ○ Tool calls ○ Responses

Slide 44

Slide 44 text

No content

Slide 45

Slide 45 text

GenAI: semantic conventions ● Span attributes for chat completions ● Span attributes for embeddings ● Events for chat: ○ Various messages roles ○ Tool calls ○ Responses ● Metrics: ○ Duration of operation ○ Number of input/output tokens

Slide 46

Slide 46 text

No content

Slide 47

Slide 47 text

GenAI Python instrumentations ● OpenAI: streaming, sync, async chat completions ○ Elastic one traces embeddings calls

Slide 48

Slide 48 text

GenAI Python instrumentations ● OpenAI: streaming, sync, async chat completions ○ Elastic one traces embeddings calls ● AWS Bedrock: ○ Converse, ConverseStream ○ InvokeModel, InvokeModelWithStreamResponse ■ Amazon Titan and Nova ■ Anthropic Claude

Slide 49

Slide 49 text

GenAI Python instrumentations ● OpenAI: streaming, sync, async chat completions ○ Elastic one traces embeddings calls ● AWS Bedrock: ○ Converse, ConverseStream ○ InvokeModel, InvokeModelWithStreamResponse ■ Amazon Titan and Nova ■ Anthropic Claude ● Google VertexAI

Slide 50

Slide 50 text

GenAI Python instrumentations ● OpenAI: streaming, sync, async chat completions ○ Elastic traces embeddings calls ● AWS Bedrock: ○ Converse, ConverseStream ○ InvokeModel, InvokeModelWithStreamResponse ■ Amazon Titan and Nova ■ Anthropic Claude ● Google VertexAI ● Google GenAI

Slide 51

Slide 51 text

GenAI: whatʼs next? ● Complete all the instrumentations

Slide 52

Slide 52 text

GenAI: whatʼs next? ● Complete all the instrumentations ● OpenLLMetry instrumentation donation proposal

Slide 53

Slide 53 text

GenAI: whatʼs next? ● Complete all the instrumentations ● OpenLLMetry instrumentation donation proposal ● Agents semantic conventions

Slide 54

Slide 54 text

Conclusions

Slide 55

Slide 55 text

Conclusions ● OpenTelemetry Operator for Kubernetes ● opentelemetry-instrument to wrap your application entry point ● Programmatic auto-instrumentation if you can add two lines of code and set some environment variables ● Thereʼs working support for GenAI instrumentation

Slide 56

Slide 56 text

Contacts and references ● https://opentelemetry.io ● opentelemetry-operator ● opentelemetry-python and opentelemetry-python-contrib ● #otel-python on CNCF slack ● @rmistaken / @[email protected] ● speakerdeck.com/xrmx

Slide 57

Slide 57 text

Thank you!