Java inside Docker pitfalls

Docker Ottawa Meetup. August 30, 2017.

Transcript

1. Java in Docker pitfalls Aleksey Vorobyov
2. Why Java in Docker is a special case JVM and libraries decide how much memory and how many threads to use. This decision is based on a host hardware configuration. JVM and many libraries are designed to take a full advantage of all available resources. • JVM ergonomics • Libraries ergonomics These ergonomics work well in VM but not in a container
3. Docker is a leaking abstraction • Docker uses Linux namespaces and cgroups • In a docker instance, an application can access a host hardware configuration • Java doesn't know about Docker and cgroups (before April 2017) • Java reads host configuration from /proc not from /sys/fs/cgroup • Demo (htop shows java process)
5. Naive Scenario • Docker instance doesn’t have limits • Java without heap limit (no -Xmx option) • I hope nobody does this in production. But it works well in some circumstances. • According to current server JVM ergonomic, JVM will take ¼ of RAM for a heap
6. How much memory do I need for my Java application? 1. Test an application with a production scenario 2. Capture memory usage a. Easy way. docker stats b. Another way. jcmd <pid> VM.native_memory
7. Memory in Java JVM allocates memory for • Heap • Metaspace • Code • GC • Threads • Native objects Start Java with -XX:NativeMemoryTracking=summary jcmd <pid> VM.native_memory
8. Crash scenario • Docker instance has memory limits • Java without memory limit • Demo (dmesg)
9. Normal scenario • Docker instance have memory limits (2 GB) • Java with -Xmx (1GB) • Regular scenario
10. Swap scenario • Docker instance have memory limits (600 MB) • Java with -Xmx (512 MB) • The most difficult to address scenario. From time to time a request latency is not stable
11. Swap and Crash scenario • Docker instance have memory and swap limit (700 MB) • Java with -Xmx (512 MB) • Crashes sometimes, hard to reproduce
12. CPU --cpuset-cpus as for now is the only way to tell JVM (8u131+) how many real cores you allow to use But this approach is very bad for resource utilization. It’s better • Use --cpus or --cpu-shares and don’t deploy CPU heavy services on the same host. • Configure JVM and an application according to provisioned resources
13. Useful JVM configuration options -XX:+UseSerialGC Uses only 1 GC thread. -XX:MaxRAM=n Sets the maximum amount of memory used by the JVM -XX:+UseCGroupMemoryLimitForHeap This flag present in the more recent builds of JDK8 tells the JVM to use the information in /sys/fs/cgroup/memory/memory.limit_in_bytes to calculate memory defaults. -XX:ParallelGCThreads=n Set the number of parallel GC threads to n. This is helpful if you are trying to limit the cpu usage of your container.
14. Useful tools • dstat • dmesg • htop • jcmd
15. Q & A