Looking for Smoking Guns in a Haystack: JVM Heap Analysis with Neo4J

We describe how and why we used the Neo4J graph database to analyse runtime behaviour of a proprietary embedded JVM deployed in one of the most widely used consumer products in the UK. We will demonstrate (via live coding) some queries that uncovered surprising aspects of our code, the runtime platform and Oracle’s Java compiler. We hope this talk will convince that graph databases can be easily used to build powerful software development and analysis tools for specific applications or niche platforms and languages.

Transcript

1. Image CC BY 2.0 Alex Indigo 2008 Looking for Smoking Guns in a Haystack JVM Heap Analysis with Neo4J Nat Pryce James Richardson
2. Sky PVR
3. PVR Platform Stack Electronic Programme Guide Java Platform Adaptors Middleware Linux Clean-Room JVM MIPS or ARM Java C
4. Platform Limitations Top-end PVR has: • 400 MHz MIPS CPU • 15 MB Java Heap • Limited flash memory for native code, Java bytecode & resources Older boxes are slower and smaller.
5. JVM for Embedded Systems
6. Embedded in a C Process JVM provided as a C library JVM runtime classes defined by the library, so cannot be inspected in the IDE
7. No JIT To save memory, we don’t use the vendor’s JIT Every bytecode counts, performance-wise E.g. invokevirtual faster than if statements
8. Slow Garbage Collection Especially full GC and compaction Have to control GC to ensure responsive UI
9. No Escape Analysis Common Java style creates lots of objects on the heap • For-each loop creates iterators • Auto-boxing
10. Limited Tooling No JVMTI No JMX and diagnostic MBeans Remote debugging requires valuable memory Proprietary heap-dump format No heap-dump analysis tools
11. Problem: heap fragmentation
12. A Fragmented Heap
13. Heaps are graphs! Image CC BY-SA 2.0 Kreg Steppe 2008
14. Relational Database
15. Graph Database
16. Import Process Python script Heap Dump Cypher neo4j-shell neo4j server
17. Much Too Slow Image CC BY 2.0 Peter Megyeri 2007
18. Import Process Java Program Heap Dump BatchInserter
19. Very Fast! Image CC BY 2.0 Flickr user poorboy1225 2011
20. Q: How to document the schema? Image CC BY 2.0 William Warby, 2010
21. Explore the Fine Graph?
22. Maths?
23. UML?
24. A: Our Own Graphical Notation
25. Example Queries and some surprising discoveries
26. Which classes use large Strings? MATCH (t:Type) -- (o) -[f:FIELD]-> (s:String) RETURN t.name as owner_type, f.name as owner_field, s.length as length, s.value as value ORDER BY length DESCENDING
27. How many descriptions? How big? MATCH (s:String) <-[f:FIELD]- () WHERE f.name =~ '.*\\.description' WITH DISTINCT s AS d RETURN COUNT(d) AS count, SUM(d.length)*2 AS total_bytes
28. What about arrays? MATCH (o) -[f:FIELD]-> (a:Array), (o) -- (ot:Type), (a) --> (at:Type) RETURN ot.name as owner_type, f.name as owner_field, a.length as array_length, at.name as array_type ORDER BY array_length DESCENDING, owner_type, owner_field ASCENDING
29. A Memory Hogging JSON Parser package org.json.simple.parser; class Yylex { private static final int ZZ_BUFFERSIZE = 16384; private static final int[] ZZ_LEXSTATE; private static final java.lang.String ZZ_CMAP_PACKED = "\t\u0000\u0001\u0007... private static final char[] ZZ_CMAP; private static final int[] ZZ_ACTION; private static final java.lang.String ZZ_ACTION_PACKED_0 = "\u0002\u0000... private static final int[] ZZ_ROWMAP; private static final java.lang.String ZZ_ROWMAP_PACKED_0 = "\u0000\u0000... private static final int[] ZZ_TRANS; private static final java.lang.String[] ZZ_ERROR_MSG; private static final int[] ZZ_ATTRIBUTE; private static final java.lang.String ZZ_ATTRIBUTE_PACKED_0 = "\u0002\u0000...; ...
30. What about the huge BitSets? MATCH (o:Object) -[f:FIELD]-> (bs:Object), (bs) -[:TYPE]-> (bst:Type), (o) -[:TYPE]-> (ot:Type), (bs) -[:FIELD]-> (bits:Array) WHERE bst.name = 'java/util/BitSet' RETURN ot.name as owner_type, f.name as owner_field, bits.length as bitset_size ORDER BY bitset_size DESCENDING
31. Bloated Bitsets enum EPGInfoBit { MonauralSoundType, SimpleStereoSoundType, SurroundSoundSoundType, DigitalSurroundSsoundType ... public final int asInt; private EPGInfoBit() { asInt = 1 << ordinal(); } } BitSet epgInfoBitsMask = new BitSet(); BitSet epgInfoBits = new BitSet(); void addBit(EPGInfoBit bit, boolean set) { epgInfoBitsMask.set(bit.asInt); if (set) { epgInfoBits.set(bit.asInt); } }
32. What are those SwitchMap arrays? MATCH (o) -[f:FIELD]-> (a:Array), (o) -- (ot:Type), (a) --> (at:Type) WHERE f.name =~ '.*SwitchMap.*' RETURN a.length as array_length, ot.name as owner_type, f.name as owner_field ORDER BY array_length DESCENDING
33. Java Language Specification, SE7 Chapter 13. Binary Compatibility … 13.4.26. Evolution of Enums Adding or reordering constants in an enum type will not break compatibility with pre-existing binaries. Bytecode cannot switch directly on enum ordinals.
34. Switching on Enums is Expensive Javac generates an additional synthetic class for every class that switches on an enum. The synthetic class contains a "switch map" int array • Maps enum ordinals to jump table indices. • Length is the number of enum elements • Initialised when the class is loaded But we don't change enums and client code independently
35. Solution Sponsor development of enum optimisations in Proguard
36. Take-Away Lessons
37. Take-Away Lessons Line-oriented data format Parse & Insert Graph Database (Niche) Development Environment Explore, Visualise, Analyse, Transform
38. sky.com/jobs
39. Nat Pryce natpryce.com info@natpryce.com @natpryce Questions? James Richardson time4tea.net info@time4tea.net @richajam sky.com/jobs