2016.5.22 数村 憲治 Python + GDB = Javaデバッガ Copyright 2016 FUJITSU LIMITED JJUG CCC 2016 Spring M-5

アジェンダ Copyright 2016 FUJITSU LIMITED やりたいこと GDB Pythonインタフェース VMStruct backtraceの実装 TODO 1

やりたいこと Copyright 2016 FUJITSU LIMITED coreファイルをデバッグ coreからJava VM(C++)とJavaコードを同時に操作 なぜcoreなのか？ 透過的なスタックトレース Javaオブジェクトのダンプ Javaならjdb、JVMならGDB。Java+JVMは？ 2

jstack - man Copyright 2016 FUJITSU LIMITED jstack(1) Troubleshooting Tools jstack(1) NAME jstack - Prints Java thread stack traces for a Java process, core file, or remote debug server. This command is experimental and unsupported. SYNOPSIS jstack [ options ] pid jstack [ options ] executable core jstack [ options ] [ server-id@ ] remote-hostname-or-IP options The command-line options. See Options. pid The process ID for which the stack trace is printed. The process must be a Java process. To get a list of Java processes running on a machine, use the jps(1) command. executable The Java executable from which the core dump was produced. core The core file for which the stack trace is to be printed. 3

jstack - man Copyright 2016 FUJITSU LIMITED DESCRIPTION The jstack command prints Java stack traces of Java threads for a specified Java process, core file, or remote debug server. For each Java frame, the full class name, method name, byte code index (BCI), and line number, when available, are printed. With the -m option, the jstack command prints both Java and native frames of all threads with the program counter (PC). OPTIONS -F Force a stack dump when jstack [-l] pid does not respond. -l Long listing. Prints additional information about locks such as a list of owned java.util.concurrent ownable synchronizers. See the AbstractOwnableSynchronizer class description at http://docs.oracle.com/javase/8/docs/api/java/util/concurrent/locks/AbstractOwnableSynchronizer.html -m Prints a mixed mode stack trace that has both Java and native C/C++ frames. 4

jstack -m Copyright 2016 FUJITSU LIMITED % /v3/java/jdk1.8.0_66/bin/jstack -m /v3/java/jdk1.8.0_66/bin/java core.2017 Attaching to core core.2017 from executable /v3/java/jdk1.8.0_66/bin/java, please wait... Debugger attached successfully. Server compiler detected. JVM version is 25.66-b17 Deadlock Detection: No deadlocks found. ----------------- 2018 ----------------- 0x00007f3492d03149 __pthread_cond_timedwait + 0x129 0x00007f3491e54ff3 _ZN2os5sleepEP6Threadlb + 0x283 0x00007f3491c56f02 JVM_Sleep + 0x3b2 0x00007f348bc15994 * java.lang.Thread.sleep(long) bci:0 (Interpreted frame) 0x00007f348bc07c4d * a.xxx3() bci:11 line:15 (Interpreted frame) 0x00007f348bc07c4d * a.xxx2() bci:0 line:10 (Interpreted frame) 0x00007f348bc07c4d * a.xxx1() bci:0 line:7 (Interpreted frame) 0x00007f348bc07c4d * a.main(java.lang.String[]) bci:0 line:4 (Interpreted frame) 0x00007f348bc007a7 0x00007f3491bc3c46 _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x1056 0x00007f3491c05262 _ZL17jni_invoke_staticP7JNIEnv_P9JavaValueP8_jobject11JNICallTypeP10_jmethodIDP18JNI_ArgumentPushe 0x00007f3491c21c6a jni_CallStaticVoidMethod + 0x17a 0x00007f3492ae7bcc JavaMain + 0x80c ----------------- 2020 ----------------- 0x00007f3492d02da0 __pthread_cond_wait + 0xc0 0x00007f3491e0ef07 _ZN7Monitor5IWaitEP6Threadl + 0xf7 0x00007f3491e0f826 _ZN7Monitor4waitEblb + 0x256 0x00007f3491b0ad93 _ZN13GCTaskManager8get_taskEj + 0x43 0x00007f3491b0bc28 _ZN12GCTaskThread3runEv + 0x188 5

jstackの課題 Copyright 2016 FUJITSU LIMITED スタックトレースしか出ない コアを他のマシンに移すと動作しない 拡張性に乏しい 修正が大変 6

GDB - backtrace Copyright 2016 FUJITSU LIMITED (gdb) bt #0 0x00007f3492d03149 in pthread_cond_timedwait@@GLIBC_2.3 at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:23 #1 0x00007f3491e5361f in os::PlatformEvent::park(long) () from /v3/java/jdk1.8.0_66/jre/lib/amd64/server/libjvm.so #2 0x00007f3491e54ff3 in os::sleep(Thread*, long, bool) () from /v3/java/jdk1.8.0_66/jre/lib/amd64/server/libjvm.so #3 0x00007f3491c56f02 in JVM_Sleep () from /v3/java/jdk1.8.0_66/jre/lib/amd64/server/libjvm.so #4 0x00007f348bc15994 in ?? () #5 0x00007f348c009800 in ?? () #6 0x00007f348bc156e2 in ?? () ….. #10 0x00007f348b3b3480 in ?? () #11 0x0000000000000000 in ?? () 7

stack unwinding Copyright 2016 FUJITSU LIMITED 呼び出し元fp pc (戻りアドレス) スタック 呼び出し元fp pc (戻りアドレス) 呼び出し先 フレーム 呼び出し元 フレーム fp(フレームポインタ) sp(スタックポインタ) デバッガは このアドレスから 該当シンボルを探す 8

コーリングコンベンション Copyright 2016 FUJITSU LIMITED GDBは各言語のコーリングコンベンションを意識して stack unwindingしている Javaは難しい？ インタプリタ・JIT・ネイティブのミックスフレーム このフレームがインタプリタフレームかどうか、 どうやって判断する？ コードが動的に生成される テンプレートインタプリタ・JIT 9

Javaコーリングコンベンション Copyright 2016 FUJITSU LIMITED A frame represents a physical stack frame (an activation). Frames can be C or Java frames, and the Java frames can be interpreted or compiled. In contrast, vframes represent source-level activations, so that one physical frame can correspond to multiple source level frames because of inlining. A frame is comprised of {pc, fp, sp} frame_x86.hpp frame interpreted compiled vframe vframe vframe 10

Javaコーリングコンベンション Copyright 2016 FUJITSU LIMITED // Layout of asm interpreter frame: // [expression stack ] * <- sp // [monitors ] ¥ // ... | monitor block size // [monitors ] / // [monitor block size ] // [byte code index/pointr] = bcx() bcx_offset // [pointer to locals ] = locals() locals_offset // [constant pool cache ] = cache() cache_offset // [methodData ] = mdp() mdx_offset // [Method* ] = method() method_offset // [last sp ] = last_sp() last_sp_offset // [old stack pointer ] (sender_sp) sender_sp_offset // [old frame pointer ] <- fp = link() return_addr_offset // [return pc ] // [oop temp ] (only for native calls) // [locals and parameters ] // <- sender sp 11

これまでのアプローチ Copyright 2016 FUJITSU LIMITED GDB自身の改造 OS毎にGDBをビルド 最新のGDBがすぐに使えない 修正変更の管理 HotSpotのコーリングコンベンションの追加 12

他のアプローチ Copyright 2016 FUJITSU LIMITED GDB JIT Compilation Interface https://sourceware.org/gdb/onlinedocs/gdb/JIT-Interface.html#JIT-Interface JITの修正が必要。インタプリタはダメ。 GDB script ifとかwhileはあるが。。。 13

アジェンダ Copyright 2016 FUJITSU LIMITED やりたいこと GDB Pythonインタフェース VMStruct backtraceの実装 TODO 14

Pythonインタフェース Copyright 2016 FUJITSU LIMITED https://sourceware.org/gdb/onlinedocs/gdb/Extending-GDB.html#Extending-GDB GDBでPythonスクリプトが使える PythonプログラムをGDBでデバッグすることではない configure時に--with-pythonを指定する % gdb -configuration This GDB was configured as follows: configure --host=x86_64-linux-gnu --target=x86_64-linux-gnu --with-auto-load-dir=$debugdir:$datadir/auto-load …….. --with-jit-reader-dir=/usr/lib/gdb (relocatable) --without-libunwind-ia64 --with-lzma --with-python=/usr (relocatable) 15

Inferiorオブジェクト Copyright 2016 FUJITSU LIMITED メモリアクセス — Function: Inferior.read_memory (address, length) Read length addressable memory units from the inferior, starting at address. Returns a buffer object, which behaves much like an array or a string. It can be modified and given to the Inferior.write_memory function. In Python 3, the return value is a memoryview object 16

メモリアクセス関数 Copyright 2016 FUJITSU LIMITED def readNbyteAsLittle(addr, size): try: buf = gdb.selected_inferior().read_memory(addr, size) except: return None i = size - 1 n = 0 while i >= 0: n = n << 8 m = ord(buf[i]) n += m i -= 1 return n 17

関数選択 Copyright 2016 FUJITSU LIMITED if (target.isSparc): readNbyte = readNbyteAsBig sizeof_pointer = 4 elif (target.isX86): readNbyte = readNbyteAsLittle sizeof_pointer = 4 elif (target.isX64): readNbyte = readNbyteAsLittle sizeof_pointer = 8 動作環境によって、関数を自動的に切り替え 18

アジェンダ Copyright 2016 FUJITSU LIMITED やりたいこと GDB Pythonインタフェース VMStruct backtraceの実装 TODO 19

VMStruct Copyright 2016 FUJITSU LIMITED JVM自身のデバッグ情報 JVM内部にテーブルとして保持 VMStructのアドバンテージ (vmStruct.hpp) クラス、フィールド、サイズ等 従来、DWARF等で定義されていたもの プラットフォーム・コンパイラ非依存 デバッグ用ビルドを作らなくてよい Javaのサイズを保持 (「jlong」 v.s. 「long long」) 20

class VMStruct (vmStruct.hpp) Copyright 2016 FUJITSU LIMITED class VMStructs { public: // The last entry is identified over in the serviceability agent by // the fact that it has a NULL fieldName static VMStructEntry localHotSpotVMStructs[]; // The last entry is identified over in the serviceability agent by // the fact that it has a NULL typeName static VMTypeEntry localHotSpotVMTypes[]; // Table of integer constants required by the serviceability agent. // The last entry is identified over in the serviceability agent by // the fact that it has a NULL typeName static VMIntConstantEntry localHotSpotVMIntConstants[]; ……. 21

VMStructEntry Copyright 2016 FUJITSU LIMITED typedef struct { const char* typeName; // The type name containing the given field (example: "Klass") const char* fieldName; // The field name within the type (example: "_name") const char* typeString; // Quoted name of the type of this field (example: "Symbol*"; int32_t isStatic; // Indicates whether following field is an offset or an address uint64_t offset; // Offset of field within structure; only used for nonstatic fields void* address; // Address of field; only used for static fields } VMStructEntry; Hotspot VMを構成する(C++の)クラス・フィールドを表現 22

VMStructEntry – 対応例 Copyright 2016 FUJITSU LIMITED typedef struct { const char* typeName; const char* fieldName; const char* typeString; int32_t isStatic; uint64_t offset; void* address; } class Klass : public Metadata { friend class VMStructs; … Symbol* _name; vmStruct.hpp klass.hpp _nameの(Klassクラスの先 頭からの)オフセット 使わない (staticのみ) 0 23

vmstructコマンド Copyright 2016 FUJITSU LIMITED class VMStruct (gdb.Command): def __init__ (self): super (VMStruct, self).__init__ ("info vmstruct", gdb.COMMAND self.gHotSpot = memory.getSymbolAddr("gHotSpotVMStructs") self.typeNameOffset = memory.readUint64FromSymbol( ¥ "gHotSpotVMStructEntryTypeNameOffset") self.fieldNameOffset = memory.readUint64FromSymbol( ¥ "gHotSpotVMStructEntryFieldNameOffset") self.typeStringOffset = memory.readUint64FromSymbol( ¥ "gHotSpotVMStructEntryTypeStringOffset") self.isStaticOffset = memory.readUint64FromSymbol( ¥ "gHotSpotVMStructEntryIsStaticOffset") 24

vmstructコマンド Copyright 2016 FUJITSU LIMITED src = memory.readPointer(self.gHotSpot) self.vmstruct = [] while (True) : entry = self.readVMStructEntry(src) if (entry == None): break self.vmstruct.append(entry) src += VMStructEntry.entrySize __init__の続き 25

vmstructコマンド Copyright 2016 FUJITSU LIMITED def readVMStructEntry(self, addr): #TypeName target = memory.readPointer(addr + self.typeNameOffset) if (target == 0): return None else: typeName = memory.readString(target) if (typeName == ""): return None #FieldName target = memory.readPointer(addr + self.fieldNameOffset) if (target == 0): fieldName= "" else: fieldName = memory.readString(target) 26

vmstructコマンド Copyright 2016 FUJITSU LIMITED def invoke(self, arg, from_tty): args = arg.split() argNum = len(args) if (argNum == 0): for entry in self.vmstruct: entry.printEntry() elif (argNum == 1): typeName = args[0] for entry in self.vmstruct: if entry.typeName.startswith(arg): entry.printEntry() elif (argNum == 2): … else: print ("usage : info vmstruct") print (" : info vmstruct ") 27

No content

アジェンダ Copyright 2016 FUJITSU LIMITED やりたいこと GDB Pythonインタフェース VMStruct backtraceの実装 TODO 29

backtraceの実現ステップ Copyright 2016 FUJITSU LIMITED 現在のフレームが、Javaのフレームかどうか判断する 現在のフレームを実行しているメソッド名を求める 前のフレーム(fp等)をセットする unwinderの登録 frame filterの登録 30

unwinder Copyright 2016 FUJITSU LIMITED フレームをたどる処理 GDBにunwinderを登録しておくと、 フレーム処理のたびに呼び出される unwinderでは、 自分が処理すべきフレームであればgdb.UnwindInfoを、 そうでなければNoneを返却。 31

unwinder登録 Copyright 2016 FUJITSU LIMITED if (target.isX64): jvmunwinder = JVMUnwinderX64() gdb.unwinder.register_unwinder(None, jvmunwinder) class JVMUnwinderX64(AbstractJVMUnwinder): name = "JVMUnwinderX64" enabled = True def __init__(self): super(JVMUnwinderX64, self).__init__() 32

unwinder呼び出し Copyright 2016 FUJITSU LIMITED def __call__(self, pending_frame): pc = pending_frame.read_register('rip') sp = pending_frame.read_register('rsp') bp = pending_frame.read_register('rbp') 33

unwinder呼び出し(続き) Copyright 2016 FUJITSU LIMITED if self.isInterpretedFrame(pc) : sp = bp + sender_sp_offset * memory.sizeof_pointer pc = gdb.Value(memory.readPointer(bp + ¥ return_addr_offset * memory.sizeof_pointer)) bp = gdb.Value(memory.readPointer(bp)) uinfo = pending_frame.create_unwind_info(FrameId(sp)) uinfo.add_saved_register(6, bp) uinfo.add_saved_register(7, sp) uinfo.add_saved_register(16, pc) return uinfo elif self.iscCompiledFrame(pc) : … else: return None 34

Javaコーリングコンベンション(再掲) Copyright 2016 FUJITSU LIMITED // Layout of asm interpreter frame: // [expression stack ] * <- sp // [monitors ] ¥ // ... | monitor block size // [monitors ] / // [monitor block size ] // [byte code index/pointr] = bcx() bcx_offset // [pointer to locals ] = locals() locals_offset // [constant pool cache ] = cache() cache_offset // [methodData ] = mdp() mdx_offset // [Method* ] = method() method_offset // [last sp ] = last_sp() last_sp_offset // [old stack pointer ] (sender_sp) sender_sp_offset // [old frame pointer ] <- fp = link() return_addr_offset // [return pc ] // [oop temp ] (only for native calls) // [locals and parameters ] // <- sender sp 35

isInterpretedFrame Copyright 2016 FUJITSU LIMITED class AbstractJVMUnwinder: __ai = memory.readPointer( ¥ vmstruct.getValue("AbstractInterpreter", "_code")) __sq_sb = memory.readPointer( __ai + vmstruct.getValue("StubQueue", "_stub_buffer")) __sq_bl= memory.readNbyte( __ai + vmstruct.getValue("StubQueue", "_buffer_limit"), ¥ vmtype.getValue("int")) def isInterpretedFrame(self, addr): return ((addr >= self.__sq_sb) and ¥ (addr < self.__sq_sb + self.__sq_bl)) 36

AbstractInterpreter Copyright 2016 FUJITSU LIMITED class AbstractInterpreter: AllStatic { friend class VMStructs; protected: static StubQueue* _code; // the interpreter code (codelets) abstractInterpreter.hpp stubs.hpp class StubQueue: public CHeapObj { friend class VMStructs; private: StubInterface* _stub_interface; // the interface prototype address _stub_buffer; // where all stubs are stored int _buffer_size; // the buffer size in bytes 37

frame filter登録 Copyright 2016 FUJITSU LIMITED class JavaFilter(): def __init__(self): self.name = "JavaFilter" self.priority = 100 self.enabled = True gdb.frame_filters[self.name] = self def filter(self, frame_iter): frame_iter = map(JavaDecorator, frame_iter) return frame_iter 38

decorator登録 Copyright 2016 FUJITSU LIMITED class JavaDecorator(gdb.FrameDecorator.FrameDecorator): def __init__(self, fobj): super(JavaDecorator, self).__init__(fobj) def function(self): frame = self.inferior_frame() pc = frame.pc() if jvmunwinder.isInterpretedFrame(pc) : methodOop = jvmunwinder.getMethodOop(frame) name = oop.getMethodName(methodOop) return ("j " + name) elif jvmunwinder.isEntryFrame(pc) : return " <>" return str(frame.name()) 39

backtrace Copyright 2016 FUJITSU LIMITED (gdb) bt #0 0x00007f3492d03149 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f3491e5361f in os::PlatformEvent::park(long) () at /v3/java/jdk1.8.0_66/jre/lib/amd64/server/libjvm.so #2 0x00007f3491e54ff3 in os::sleep(Thread*, long, bool) () at /v3/java/jdk1.8.0_66/jre/lib/amd64/server/libjvm.so #3 0x00007f3491c56f02 in JVM_Sleep () at /v3/java/jdk1.8.0_66/jre/lib/amd64/server/libjvm.so #4 0x00007f348bc15994 in j java/lang/Thread#sleep(J)V () #5 0x00007f348bc07c4d in j a#xxx3()V () #6 0x00007f348bc07c4d in j a#xxx2()V () #7 0x00007f348bc07c4d in j a#xxx1()V () #8 0x00007f348bc07c4d in j a#main([Ljava/lang/String;)V () #9 0x00007f348bc007a7 in <> () #10 0x00007f3491bc3c46 in JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, T at /v3/java/jdk1.8.0_66/jre/lib/amd64/server/libjvm.so #11 0x00007f3491c05262 in jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID #12 0x00007f3491c21c6a in jni_CallStaticVoidMethod () at /v3/java/jdk1.8.0_66/jre/lib/amd64/server/ #13 0x00007f3492ae7bcc in JavaMain () at /v3/java/jdk1.8.0_66/bin/../lib/amd64/jli/libjli.so #14 0x00007f3492cfd6aa in start_thread (arg=0x7f349311b700) at pthread_create.c:333 #15 0x00007f3492618eed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 40

No content

アジェンダ Copyright 2016 FUJITSU LIMITED やりたいこと GDB Pythonインタフェース VMStruct backtraceの実装 TODO 42

TODO Copyright 2016 FUJITSU LIMITED 各種プラットフォーム展開 コマンド充実 oopコマンド ライブデバッグ JDK9 43

oopコマンド Copyright 2016 FUJITSU LIMITED (instanceOop*) 0xefe983d0 : oop is java.net.SocksSocketImpl instance. +00 markOop _mark = 0x3ce7e9f8 +04 klassOop _klass = 0xcd649f38 === instance value === *** java.net.SocketImpl *** +08 Ljava.net.Socket; socket = 0xefe97db8 +0c Ljava.net.ServerSocket; serverSocket = 0x00000000 +10 protected Ljava.io.FileDescriptor; fd = 0xefe98478 +14 protected Ljava.net.InetAddress; address = 0xdf0cb5b0 +18 protected I port = 0x00009726 +1c protected I localport = 0x0000a5b1 *** java.net.PlainSocketImpl *** +20 private Ljava.net.SocketInputStream; socketInputStream = 0x00000000 +24 private Ljava.lang.Object; fdLock = 0xefe98468 +28 private Ljava.lang.Object; resetLock = 0xefe98470 +2c private Ljava.io.FileDescriptor; fd1 = 0x00000000 +30 private Ljava.net.InetAddress; anyLocalBoundAddr = 0xd8e4b968 44

Q/A Copyright 2016 FUJITSU LIMITED 45

Copyright 2010 FUJITSU LIMITED