Troubleshooting Cloud-Native Java Applications in Container Environments While Maintaining Security

OCX2024

Introduce troubleshooting methods in container environment, while maintaining high security.

Transcript

1. © 2024 Fujitsu Limited 2024/10/23 Kenji Kazumura Troubleshooting Cloud-Native Java Applications in Container Environments While Maintaining Security OCX 2024 @kkzr
2. © 2024 Fujitsu Limited 2 Dilemma of Java Container Troubleshooting Java Container Wrap-up Agenda
3. Who Am I © 2024 Fujitsu Limited Work for Fujitsu • FUJITSU Software Enterprise Application Platform • Launcher Member of Jakarta EE SC Member of JCP Executive Committee Board of Director of Eclipse Foundation 3
4. © 2024 Fujitsu Limited 4 Dilemma of Java Container Troubleshooting Java Container Wrap-up Agenda
5. 5 Motivation for Using Container Easy Packaging Everything needed are in a package Portability Can be deployed anywhere, production and testing Lightweight Faster start-up than VM Isolation Easily scaled ・・・ Troubleshooting and security are not motivation © 2024 Fujitsu Limited
6. 6 Theme of this session Start-up time is desired on container Security is also vital In reality, troubleshooting are usually needed Look for the way to meet all three requirements © 2024 Fujitsu Limited
7. © 2024 Fujitsu Limited 7 Container Position (1) troubleshooting security (1) IDEAL start-up
8. 8 Smaller Container Image Demand Faster start-up time reduce down-time reduce time of scaling out High Security fewer OS libraries, lower security risk still no motivation derived from troubleshooting © 2024 Fujitsu Limited
9. 9 Container Start-up Time https://niravshah2705.medium.com/kubernetes-image-pull-optimisation-part-i-exploring-options-f79d79c3fe45 Almost image pull ‘pull’ consists of ‘fetch’ and ‘uncompress’ Bigger image size, longer uncompress time © 2024 Fujitsu Limited
10. 10 Container Security Practice Smaller base image minimize user privileges monitor vulnerable information don’t include tokens / keys in ・・・ https://res.cloudinary.com/snyk/image/upload/v1551798390/Docker_Image_Security_Best_Practices_.pdf © 2024 Fujitsu Limited image
11. When vulnerability detected at a library in the base image, need to re-build image even if this library are not used. 11 Vulnerabilities of Libraries in Image Even using latest image when build and deploy, vulnerabilities increase Re-build with latest image Container Image Need action, even if not used © 2024 Fujitsu Limited base image application
12. 12 Example vulnerabilities at Docker Hub © 2024 Fujitsu Limited
13. 13 Ways to make Java image smaller © 2024 Fujitsu Limited distroless Minimize base image there are some Java image jlink Package only necessary modules for applications JRE No tools in JDK (jshell, jar …)
14. Available at gcr.io/distroless Java distroless: `gcr.io/distroless/java21-debian12` Microsoft also provides `mcr.microsoft.com/openjdk/jdk:21-distroless` No tools for troubleshooting No `ls`, No `ps` 14 Distroless Image © 2024 Fujitsu Limited No way to investigate OS provied features
15. No way to investigate Java troubles 15 JRE © 2024 Fujitsu Limited JRE does not include most of tools in JDK jcmd / jstack / jar / javac / jdb / javadoc / ・・・ tools not included in JRE tools included in JRE java / keytool / rmiregistry / jfr / ・・・
16. 16 Dilemma of Java Container © 2024 Fujitsu Limited Methodology to realize Start-up Security trouble shooting not compatible with each other cannot use JDK tools , OS tools smaller image
17. © 2024 Fujitsu Limited 17 Container Position (2) troubleshooting security (1) IDEAL start-up (2)
18. © 2024 Fujitsu Limited 18 Dilemma of Java Container Troubleshooting Java Container Wrap-up Agenda
19. ‘exec’ sub-command Access from Node Sidecar COPY Ephemeral Container How to Troubleshoot Containers © 2024 Fujitsu Limited 19
20. ‘exec’ sub-command Access from Node Sidecar COPY Ephemeral Container How to Troubleshoot Containers © 2024 Fujitsu Limited 20
21. ‘docker exec’ / ‘kubectl exec’ Can access container internal system and process as if you are in the container No guarantee to be available necessary troubleshooting tools in container Feels like login system using ssh, but actually just changing namespace 21 ‘exec’ sub-command © 2024 Fujitsu Limited
22. Managed by i-node Separate resources by using 8 namespaces ✓ cgroup/net/ipc/uts/user/pid/time/mnt ✓ By setting share or not to each namespace, container can be isolated flexibly. Namespace related commands lsns nsenter ・・・ `docker exec` is just changing namespace same as `nsenter --target ${PID} --all` 22 Linux namespace © 2024 Fujitsu Limited
23. ‘exec’ sub-command Access from Node Sidecar COPY Ephemeral Container How to Troubleshoot Containers © 2024 Fujitsu Limited 23
24. 24 Access from Node © 2024 Fujitsu Limited Process in container can be seen as process of node. But PID are different. Difficult to know which process belongs to which container. Difficult to troubleshooting from viewpoint of container Usually accessing node is limited Node container JDK jstack Java Application
25. 25 Create Debug Container on Node © 2024 Fujitsu Limited container running with host namespace on K8S node kubectl debug node/{node-name} -it ＼ --image={image} -- bash Pod Debuggee Container Debugger Container Pod Node jstack run with node namespace Java Application
26. ‘exec’ sub-command Access from Node Sidecar COPY Ephemeral Container How to Troubleshoot Containers © 2024 Fujitsu Limited 26
27. Sharing Pod namespace © 2024 Fujitsu Limited K8S Pod 27 pid share share cannot access process of another container pid uts net share share Container Main Container Sidecar mnt mnt
28. Share namespace using side car © 2024 Fujitsu Limited Pod 28 mnt share share can access process of another container mnt uts net share share Container Application(JRE) Container Debugger(JDK) pid shareprocess
29. 29 Sidecar © 2024 Fujitsu Limited Run trouble shooting image in Pod as a sidecar specify following in pod definition ‘spec.shareProcessNamespace:true’ No need to install tools in application image, but need to run always even if no trouble useful for development environment which you know trouble will surely happen
30. ‘exec’ sub-command Access from Node Sidecar COPY Ephemeral Container How to Troubleshoot Containers © 2024 Fujitsu Limited 30
31. Sharing namespace using COPY © 2024 Fujitsu Limited Pod 31 can access process Pod net COPY Pod net uts Container Debugger(JDK) Container net uts App(JRE) Container App(JRE) Container pid App(JRE) mnt pid mnt mnt pid uts mnt shareprocess
32. 32 COPY © 2024 Fujitsu Limited Copy application and add debugger container to pod when trouble Resource efficiency compare to Sidecar No live migration kubectl debug -it debuggee-pod ＼ --copy-to debugger-pod ＼ –share-processes debuggee-pod ＼ --image JDK-image -- bash
33. ‘exec’ sub-command Access from Node Sidecar COPY Ephemeral Container How to Troubleshoot Containers © 2024 Fujitsu Limited 33
34. Sharing by ephemeral container © 2024 Fujitsu Limited Pod 34 Can access process Pod net uts mnt Container net uts mnt App(JRE) Ephemeral Container Debugger(JDK) Container mnt App(JRE) pid add pid Debugger(JDK)
35. 35 Ephemeral Container © 2024 Fujitsu Limited add a container which namespace is same as debuggee container to the existing Pod using Docker using Kubernetes docker run ＼ -it --name=debugger --pid=container:jre ＼ jdk-image bash kubectl debug -it -c debugger --target debuggee ＼ --image=jdk-image debuggee -- bash
36. © 2024 Fujitsu Limited 36 Agenda Dilemma of Java Container Troubleshooting Java Container Wrap-up
37. 37 Comparison © 2024 Fujitsu Limited Pros Cons ‘exec’ sub-command Easy Tools may not be available Access from Node N/A Need host privilege Different namespace Side Car Separate debugger container from application container Waist resource COPY Copy only when troubles Cannot debug actual container trouble happens Ephemeral Container Can access actual troubled container only when trouble happens Cannot delete ephemeral container
38. © 2024 Fujitsu Limited 38 Container Position (3) troubleshooting security start-up (1) (2) IDEAL (3)
39. Wrap-up © 2024 Fujitsu Limited 39 Tools which are used at on-premises may not be used (dilemma of Java container) There are several ways such as ephemeral container to detour dilemma Isolating troubleshooting container keeps application container secure start-up security trouble shooting
40. © 2024 Fujitsu Limited Question? 40
41. Thank you © 2024 Fujitsu Limited