© 2024 Fujitsu Limited 2024/10/23 Kenji Kazumura Troubleshooting Cloud-Native Java Applications in Container Environments While Maintaining Security OCX 2024 @kkzr

© 2024 Fujitsu Limited 2 Dilemma of Java Container Troubleshooting Java Container Wrap-up Agenda

Who Am I © 2024 Fujitsu Limited Work for Fujitsu • FUJITSU Software Enterprise Application Platform • Launcher Member of Jakarta EE SC Member of JCP Executive Committee Board of Director of Eclipse Foundation 3

© 2024 Fujitsu Limited 4 Dilemma of Java Container Troubleshooting Java Container Wrap-up Agenda

5 Motivation for Using Container Easy Packaging Everything needed are in a package Portability Can be deployed anywhere, production and testing Lightweight Faster start-up than VM Isolation Easily scaled ・・・ Troubleshooting and security are not motivation © 2024 Fujitsu Limited

6 Theme of this session Start-up time is desired on container Security is also vital In reality, troubleshooting are usually needed Look for the way to meet all three requirements © 2024 Fujitsu Limited

© 2024 Fujitsu Limited 7 Container Position (1) troubleshooting security (1) IDEAL start-up

8 Smaller Container Image Demand Faster start-up time reduce down-time reduce time of scaling out High Security fewer OS libraries, lower security risk still no motivation derived from troubleshooting © 2024 Fujitsu Limited

9 Container Start-up Time https://niravshah2705.medium.com/kubernetes-image-pull-optimisation-part-i-exploring-options-f79d79c3fe45 Almost image pull ‘pull’ consists of ‘fetch’ and ‘uncompress’ Bigger image size, longer uncompress time © 2024 Fujitsu Limited

10 Container Security Practice Smaller base image minimize user privileges monitor vulnerable information don’t include tokens / keys in ・・・ https://res.cloudinary.com/snyk/image/upload/v1551798390/Docker_Image_Security_Best_Practices_.pdf © 2024 Fujitsu Limited image

When vulnerability detected at a library in the base image, need to re-build image even if this library are not used. 11 Vulnerabilities of Libraries in Image Even using latest image when build and deploy, vulnerabilities increase Re-build with latest image Container Image Need action, even if not used © 2024 Fujitsu Limited base image application

12 Example vulnerabilities at Docker Hub © 2024 Fujitsu Limited

13 Ways to make Java image smaller © 2024 Fujitsu Limited distroless Minimize base image there are some Java image jlink Package only necessary modules for applications JRE No tools in JDK (jshell, jar …)

Available at gcr.io/distroless Java distroless: `gcr.io/distroless/java21-debian12` Microsoft also provides `mcr.microsoft.com/openjdk/jdk:21-distroless` No tools for troubleshooting No `ls`, No `ps` 14 Distroless Image © 2024 Fujitsu Limited No way to investigate OS provied features

No way to investigate Java troubles 15 JRE © 2024 Fujitsu Limited JRE does not include most of tools in JDK jcmd / jstack / jar / javac / jdb / javadoc / ・・・ tools not included in JRE tools included in JRE java / keytool / rmiregistry / jfr / ・・・

16 Dilemma of Java Container © 2024 Fujitsu Limited Methodology to realize Start-up Security trouble shooting not compatible with each other cannot use JDK tools , OS tools smaller image

© 2024 Fujitsu Limited 17 Container Position (2) troubleshooting security (1) IDEAL start-up (2)

© 2024 Fujitsu Limited 18 Dilemma of Java Container Troubleshooting Java Container Wrap-up Agenda

‘exec’ sub-command Access from Node Sidecar COPY Ephemeral Container How to Troubleshoot Containers © 2024 Fujitsu Limited 19

‘exec’ sub-command Access from Node Sidecar COPY Ephemeral Container How to Troubleshoot Containers © 2024 Fujitsu Limited 20

‘docker exec’ / ‘kubectl exec’ Can access container internal system and process as if you are in the container No guarantee to be available necessary troubleshooting tools in container Feels like login system using ssh, but actually just changing namespace 21 ‘exec’ sub-command © 2024 Fujitsu Limited

Managed by i-node Separate resources by using 8 namespaces ✓ cgroup/net/ipc/uts/user/pid/time/mnt ✓ By setting share or not to each namespace, container can be isolated flexibly. Namespace related commands lsns nsenter ・・・ `docker exec` is just changing namespace same as `nsenter --target ${PID} --all` 22 Linux namespace © 2024 Fujitsu Limited

‘exec’ sub-command Access from Node Sidecar COPY Ephemeral Container How to Troubleshoot Containers © 2024 Fujitsu Limited 23

24 Access from Node © 2024 Fujitsu Limited Process in container can be seen as process of node. But PID are different. Difficult to know which process belongs to which container. Difficult to troubleshooting from viewpoint of container Usually accessing node is limited Node container JDK jstack Java Application

25 Create Debug Container on Node © 2024 Fujitsu Limited container running with host namespace on K8S node kubectl debug node/{node-name} -it ＼ --image={image} -- bash Pod Debuggee Container Debugger Container Pod Node jstack run with node namespace Java Application

‘exec’ sub-command Access from Node Sidecar COPY Ephemeral Container How to Troubleshoot Containers © 2024 Fujitsu Limited 26

Sharing Pod namespace © 2024 Fujitsu Limited K8S Pod 27 pid share share cannot access process of another container pid uts net share share Container Main Container Sidecar mnt mnt

Share namespace using side car © 2024 Fujitsu Limited Pod 28 mnt share share can access process of another container mnt uts net share share Container Application(JRE) Container Debugger(JDK) pid shareprocess

29 Sidecar © 2024 Fujitsu Limited Run trouble shooting image in Pod as a sidecar specify following in pod definition ‘spec.shareProcessNamespace:true’ No need to install tools in application image, but need to run always even if no trouble useful for development environment which you know trouble will surely happen

‘exec’ sub-command Access from Node Sidecar COPY Ephemeral Container How to Troubleshoot Containers © 2024 Fujitsu Limited 30

Sharing namespace using COPY © 2024 Fujitsu Limited Pod 31 can access process Pod net COPY Pod net uts Container Debugger(JDK) Container net uts App(JRE) Container App(JRE) Container pid App(JRE) mnt pid mnt mnt pid uts mnt shareprocess

32 COPY © 2024 Fujitsu Limited Copy application and add debugger container to pod when trouble Resource efficiency compare to Sidecar No live migration kubectl debug -it debuggee-pod ＼ --copy-to debugger-pod ＼ –share-processes debuggee-pod ＼ --image JDK-image -- bash

‘exec’ sub-command Access from Node Sidecar COPY Ephemeral Container How to Troubleshoot Containers © 2024 Fujitsu Limited 33

Sharing by ephemeral container © 2024 Fujitsu Limited Pod 34 Can access process Pod net uts mnt Container net uts mnt App(JRE) Ephemeral Container Debugger(JDK) Container mnt App(JRE) pid add pid Debugger(JDK)

35 Ephemeral Container © 2024 Fujitsu Limited add a container which namespace is same as debuggee container to the existing Pod using Docker using Kubernetes docker run ＼ -it --name=debugger --pid=container:jre ＼ jdk-image bash kubectl debug -it -c debugger --target debuggee ＼ --image=jdk-image debuggee -- bash

© 2024 Fujitsu Limited 36 Agenda Dilemma of Java Container Troubleshooting Java Container Wrap-up

37 Comparison © 2024 Fujitsu Limited Pros Cons ‘exec’ sub-command Easy Tools may not be available Access from Node N/A Need host privilege Different namespace Side Car Separate debugger container from application container Waist resource COPY Copy only when troubles Cannot debug actual container trouble happens Ephemeral Container Can access actual troubled container only when trouble happens Cannot delete ephemeral container

© 2024 Fujitsu Limited 38 Container Position (3) troubleshooting security start-up (1) (2) IDEAL (3)

Wrap-up © 2024 Fujitsu Limited 39 Tools which are used at on-premises may not be used (dilemma of Java container) There are several ways such as ephemeral container to detour dilemma Isolating troubleshooting container keeps application container secure start-up security trouble shooting

© 2024 Fujitsu Limited Question? 40

Thank you © 2024 Fujitsu Limited