Python + GDB = Javaデバッガ

2016.5.22
数村憲治
Python + GDB = Javaデバッガ
Copyright 2016 FUJITSU LIMITED
JJUG CCC 2016 Spring
M-5

View Slide

アジェンダ
やりたいこと
GDB Pythonインタフェース
VMStruct
backtraceの実装
TODO
1

View Slide

やりたいこと
coreファイルをデバッグ
coreからJava VM(C++)とJavaコードを同時に操作
なぜcoreなのか？
透過的なスタックトレース
Javaオブジェクトのダンプ
Javaならjdb、JVMならGDB。Java+JVMは？
2

View Slide

jstack - man
jstack(1) Troubleshooting Tools jstack(1)
NAME
jstack - Prints Java thread stack traces for a Java process, core file, or remote debug
server. This command is experimental and unsupported.
SYNOPSIS
jstack [ options ] pid
jstack [ options ] executable core
jstack [ options ] [ server-id@ ] remote-hostname-or-IP
options
The command-line options. See Options.
pid The process ID for which the stack trace is printed. The process must be a Java
process. To get a list of Java processes running on a machine, use the jps(1) command.
executable
The Java executable from which the core dump was produced.
core The core file for which the stack trace is to be printed.
3

View Slide

jstack - man
DESCRIPTION
The jstack command prints Java stack traces of Java threads for a specified Java process, core
file, or remote debug server. For each Java frame, the full class name, method name, byte code
index (BCI), and line number, when available, are printed.
With the -m option,
the jstack command prints both Java and native frames
of all threads with the program counter (PC).
OPTIONS
-F
Force a stack dump when jstack [-l] pid does not respond.
-l
Long listing. Prints additional information about locks such as a list of owned
java.util.concurrent ownable synchronizers. See the AbstractOwnableSynchronizer class
description at
http://docs.oracle.com/javase/8/docs/api/java/util/concurrent/locks/AbstractOwnableSynchronizer.html
-m
Prints a mixed mode stack trace that has both Java and
native C/C++ frames.
4

View Slide

jstack -m
% /v3/java/jdk1.8.0_66/bin/jstack -m /v3/java/jdk1.8.0_66/bin/java core.2017
Attaching to core core.2017 from executable /v3/java/jdk1.8.0_66/bin/java, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.66-b17
Deadlock Detection:
No deadlocks found.
----------------- 2018 -----------------
0x00007f3492d03149 __pthread_cond_timedwait + 0x129
0x00007f3491e54ff3 _ZN2os5sleepEP6Threadlb + 0x283
0x00007f3491c56f02 JVM_Sleep + 0x3b2
0x00007f348bc15994 * java.lang.Thread.sleep(long) bci:0 (Interpreted frame)
0x00007f348bc07c4d * a.xxx3() bci:11 line:15 (Interpreted frame)
0x00007f348bc07c4d * a.main(java.lang.String[]) bci:0 line:4 (Interpreted frame)
0x00007f348bc007a7
0x00007f3491bc3c46 _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x1056
0x00007f3491c05262 _ZL17jni_invoke_staticP7JNIEnv_P9JavaValueP8_jobject11JNICallTypeP10_jmethodIDP18JNI_ArgumentPushe
0x00007f3491c21c6a jni_CallStaticVoidMethod + 0x17a
0x00007f3492ae7bcc JavaMain + 0x80c
----------------- 2020 -----------------
0x00007f3492d02da0 __pthread_cond_wait + 0xc0
0x00007f3491e0ef07 _ZN7Monitor5IWaitEP6Threadl + 0xf7
0x00007f3491e0f826 _ZN7Monitor4waitEblb + 0x256
0x00007f3491b0ad93 _ZN13GCTaskManager8get_taskEj + 0x43
0x00007f3491b0bc28 _ZN12GCTaskThread3runEv + 0x188
5

View Slide

jstackの課題
スタックトレースしか出ない
コアを他のマシンに移すと動作しない
拡張性に乏しい
修正が大変
6

View Slide

GDB - backtrace
(gdb) bt
#0 0x00007f3492d03149 in pthread_cond_timedwait@@GLIBC_2.3
at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:23
#1 0x00007f3491e5361f in os::PlatformEvent::park(long) ()
from /v3/java/jdk1.8.0_66/jre/lib/amd64/server/libjvm.so
#2 0x00007f3491e54ff3 in os::sleep(Thread*, long, bool) ()
#3 0x00007f3491c56f02 in JVM_Sleep ()
#4 0x00007f348bc15994 in ?? ()
#5 0x00007f348c009800 in ?? ()
#6 0x00007f348bc156e2 in ?? ()
…..
#10 0x00007f348b3b3480 in ?? ()
#11 0x0000000000000000 in ?? ()
7

View Slide

stack unwinding
呼び出し元fp
pc (戻りアドレス)
スタック
呼び出し元fp
pc (戻りアドレス)
呼び出し先
フレーム
呼び出し元
フレーム
fp(フレームポインタ)
sp(スタックポインタ)
デバッガは
このアドレスから
該当シンボルを探す
8

View Slide

コーリングコンベンション
GDBは各言語のコーリングコンベンションを意識して
stack unwindingしている
Javaは難しい？
インタプリタ・JIT・ネイティブのミックスフレーム
このフレームがインタプリタフレームかどうか、
どうやって判断する？
コードが動的に生成される
テンプレートインタプリタ・JIT
9

View Slide

Javaコーリングコンベンション
A frame represents a physical stack frame (an activation).
Frames can be C or Java frames,
and the Java frames can be interpreted or compiled.
In contrast, vframes represent source-level activations,
so that one physical frame can correspond to
multiple source level frames because of inlining.
A frame is comprised of {pc, fp, sp}
frame_x86.hpp
frame
interpreted
compiled vframe
vframe
vframe
10

View Slide

Javaコーリングコンベンション
// Layout of asm interpreter frame:
// [expression stack ] * <- sp
// [monitors ] ¥
// ... | monitor block size
// [monitors ] /
// [monitor block size ]
// [byte code index/pointr] = bcx() bcx_offset
// [pointer to locals ] = locals() locals_offset
// [constant pool cache ] = cache() cache_offset
// [methodData ] = mdp() mdx_offset
// [Method* ] = method() method_offset
// [last sp ] = last_sp() last_sp_offset
// [old stack pointer ] (sender_sp) sender_sp_offset
// [old frame pointer ] <- fp = link() return_addr_offset
// [return pc ]
// [oop temp ] (only for native calls)
// [locals and parameters ]
// <- sender sp
11

View Slide

これまでのアプローチ
GDB自身の改造
OS毎にGDBをビルド
最新のGDBがすぐに使えない
修正変更の管理
HotSpotのコーリングコンベンションの追加
12

View Slide

他のアプローチ
GDB JIT Compilation Interface
https://sourceware.org/gdb/onlinedocs/gdb/JIT-Interface.html#JIT-Interface
JITの修正が必要。インタプリタはダメ。
GDB script
ifとかwhileはあるが。。。
13

View Slide

アジェンダ
やりたいこと
VMStruct
backtraceの実装
TODO
14

View Slide

Pythonインタフェース
https://sourceware.org/gdb/onlinedocs/gdb/Extending-GDB.html#Extending-GDB
GDBでPythonスクリプトが使える
PythonプログラムをGDBでデバッグすることではない
configure時に--with-pythonを指定する
% gdb -configuration
This GDB was configured as follows:
configure --host=x86_64-linux-gnu --target=x86_64-linux-gnu
--with-auto-load-dir=$debugdir:$datadir/auto-load
……..
--with-jit-reader-dir=/usr/lib/gdb (relocatable)
--without-libunwind-ia64
--with-lzma
--with-python=/usr (relocatable)
15

View Slide

Inferiorオブジェクト
メモリアクセス
— Function: Inferior.read_memory (address, length)
Read length addressable memory units from the
inferior, starting at address. Returns a buffer object,
which behaves much like an array or a string. It can be
modified and given to the Inferior.write_memory
function. In Python 3, the return value is a
memoryview object
16

View Slide

メモリアクセス関数
def readNbyteAsLittle(addr, size):
try:
buf = gdb.selected_inferior().read_memory(addr, size)
except:
return None
i = size - 1
n = 0
while i >= 0:
n = n << 8
m = ord(buf[i])
n += m
i -= 1
return n
17

View Slide

関数選択
if (target.isSparc):
readNbyte = readNbyteAsBig
sizeof_pointer = 4
elif (target.isX86):
readNbyte = readNbyteAsLittle
sizeof_pointer = 4
elif (target.isX64):
readNbyte = readNbyteAsLittle
sizeof_pointer = 8
動作環境によって、関数を自動的に切り替え
18

View Slide

アジェンダ
やりたいこと
VMStruct
backtraceの実装
TODO
19

View Slide

VMStruct
JVM自身のデバッグ情報
JVM内部にテーブルとして保持
VMStructのアドバンテージ (vmStruct.hpp)
クラス、フィールド、サイズ等
従来、DWARF等で定義されていたもの
プラットフォーム・コンパイラ非依存
デバッグ用ビルドを作らなくてよい
Javaのサイズを保持 (「jlong」 v.s. 「long long」)
20

View Slide

class VMStruct (vmStruct.hpp)
class VMStructs {
public:
// The last entry is identified over in the serviceability agent by
// the fact that it has a NULL fieldName
static VMStructEntry localHotSpotVMStructs[];
// the fact that it has a NULL typeName
static VMTypeEntry localHotSpotVMTypes[];
// Table of integer constants required by the serviceability agent.
// the fact that it has a NULL typeName
static VMIntConstantEntry localHotSpotVMIntConstants[];
…….
21

View Slide

VMStructEntry
typedef struct {
const char* typeName;
// The type name containing the given field (example: "Klass")
const char* fieldName;
// The field name within the type (example: "_name")
const char* typeString;
// Quoted name of the type of this field (example: "Symbol*";
int32_t isStatic;
// Indicates whether following field is an offset or an address
uint64_t offset;
// Offset of field within structure; only used for nonstatic fields
void* address;
// Address of field; only used for static fields
} VMStructEntry;
Hotspot VMを構成する(C++の)クラス・フィールドを表現
22

View Slide

VMStructEntry – 対応例
typedef struct {
const char* typeName;
const char* fieldName;
const char* typeString;
int32_t isStatic;
uint64_t offset;
void* address;
}
class Klass : public Metadata {
friend class VMStructs;
…
Symbol* _name;
vmStruct.hpp klass.hpp
_nameの(Klassクラスの先
頭からの)オフセット
使わない
(staticのみ)
0
23

View Slide

vmstructコマンド
class VMStruct (gdb.Command):
def __init__ (self):
super (VMStruct, self).__init__ ("info vmstruct", gdb.COMMAND
self.gHotSpot = memory.getSymbolAddr("gHotSpotVMStructs")
self.typeNameOffset = memory.readUint64FromSymbol( ¥
"gHotSpotVMStructEntryTypeNameOffset")
self.fieldNameOffset = memory.readUint64FromSymbol( ¥
"gHotSpotVMStructEntryFieldNameOffset")
self.typeStringOffset = memory.readUint64FromSymbol( ¥
"gHotSpotVMStructEntryTypeStringOffset")
self.isStaticOffset = memory.readUint64FromSymbol( ¥
"gHotSpotVMStructEntryIsStaticOffset")
24

View Slide

src = memory.readPointer(self.gHotSpot)
self.vmstruct = []
while (True) :
entry = self.readVMStructEntry(src)
if (entry == None):
break
self.vmstruct.append(entry)
src += VMStructEntry.entrySize
__init__の続き
25

View Slide

def readVMStructEntry(self, addr):
#TypeName
target = memory.readPointer(addr + self.typeNameOffset)
if (target == 0):
return None
else:
typeName = memory.readString(target)
if (typeName == ""):
return None
#FieldName
target = memory.readPointer(addr + self.fieldNameOffset)
if (target == 0):
fieldName= ""
else:
fieldName = memory.readString(target)
26

View Slide

def invoke(self, arg, from_tty):
args = arg.split()
argNum = len(args)
if (argNum == 0):
for entry in self.vmstruct:
entry.printEntry()
elif (argNum == 1):
typeName = args[0]
for entry in self.vmstruct:
if entry.typeName.startswith(arg):
entry.printEntry()
elif (argNum == 2):
…
else:
print ("usage : info vmstruct")
print (" : info vmstruct ")
27

View Slide

View Slide

アジェンダ
やりたいこと
VMStruct
backtraceの実装
TODO
29

View Slide

backtraceの実現ステップ
現在のフレームが、Javaのフレームかどうか判断する
現在のフレームを実行しているメソッド名を求める
前のフレーム(fp等)をセットする
unwinderの登録
frame filterの登録
30

View Slide

unwinder
フレームをたどる処理
GDBにunwinderを登録しておくと、
フレーム処理のたびに呼び出される
unwinderでは、
自分が処理すべきフレームであればgdb.UnwindInfoを、
そうでなければNoneを返却。
31

View Slide

unwinder登録
if (target.isX64):
jvmunwinder = JVMUnwinderX64()
gdb.unwinder.register_unwinder(None, jvmunwinder)
class JVMUnwinderX64(AbstractJVMUnwinder):
name = "JVMUnwinderX64"
enabled = True
def __init__(self):
super(JVMUnwinderX64, self).__init__()
32

View Slide

unwinder呼び出し
def __call__(self, pending_frame):
pc = pending_frame.read_register('rip')
sp = pending_frame.read_register('rsp')
bp = pending_frame.read_register('rbp')
33

View Slide

unwinder呼び出し(続き)
if self.isInterpretedFrame(pc) :
sp = bp + sender_sp_offset * memory.sizeof_pointer
pc = gdb.Value(memory.readPointer(bp + ¥
return_addr_offset * memory.sizeof_pointer))
bp = gdb.Value(memory.readPointer(bp))
uinfo = pending_frame.create_unwind_info(FrameId(sp))
uinfo.add_saved_register(6, bp)
uinfo.add_saved_register(7, sp)
uinfo.add_saved_register(16, pc)
return uinfo
elif self.iscCompiledFrame(pc) :
…
else:
return None
34

View Slide

Javaコーリングコンベンション(再掲)
// Layout of asm interpreter frame:
// [expression stack ] * <- sp
// [monitors ] ¥
// ... | monitor block size
// [monitors ] /
// [monitor block size ]
// [byte code index/pointr] = bcx() bcx_offset
// [pointer to locals ] = locals() locals_offset
// [constant pool cache ] = cache() cache_offset
// [methodData ] = mdp() mdx_offset
// [Method* ] = method() method_offset
// [last sp ] = last_sp() last_sp_offset
// [old stack pointer ] (sender_sp) sender_sp_offset
// [old frame pointer ] <- fp = link() return_addr_offset
// [return pc ]
// [oop temp ] (only for native calls)
// [locals and parameters ]
// <- sender sp
35

View Slide

isInterpretedFrame
class AbstractJVMUnwinder:
__ai = memory.readPointer( ¥
vmstruct.getValue("AbstractInterpreter", "_code"))
__sq_sb = memory.readPointer(
__ai + vmstruct.getValue("StubQueue", "_stub_buffer"))
__sq_bl= memory.readNbyte(
__ai + vmstruct.getValue("StubQueue", "_buffer_limit"), ¥
vmtype.getValue("int"))
def isInterpretedFrame(self, addr):
return ((addr >= self.__sq_sb) and ¥
(addr < self.__sq_sb + self.__sq_bl))
36

View Slide

AbstractInterpreter
class AbstractInterpreter: AllStatic {
protected:
static StubQueue* _code; // the interpreter code (codelets)
abstractInterpreter.hpp
stubs.hpp
class StubQueue: public CHeapObj {
private:
StubInterface* _stub_interface; // the interface prototype
address _stub_buffer; // where all stubs are stored
int _buffer_size; // the buffer size in bytes
37

View Slide

frame filter登録
class JavaFilter():
def __init__(self):
self.name = "JavaFilter"
self.priority = 100
self.enabled = True
gdb.frame_filters[self.name] = self
def filter(self, frame_iter):
frame_iter = map(JavaDecorator, frame_iter)
return frame_iter
38

View Slide

decorator登録
class JavaDecorator(gdb.FrameDecorator.FrameDecorator):
def __init__(self, fobj):
super(JavaDecorator, self).__init__(fobj)
def function(self):
frame = self.inferior_frame()
pc = frame.pc()
if jvmunwinder.isInterpretedFrame(pc) :
methodOop = jvmunwinder.getMethodOop(frame)
name = oop.getMethodName(methodOop)
return ("j " + name)
elif jvmunwinder.isEntryFrame(pc) :
return " <>"
return str(frame.name())
39

View Slide

backtrace
(gdb) bt
#0 0x00007f3492d03149 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1 0x00007f3491e5361f in os::PlatformEvent::park(long) ()
at /v3/java/jdk1.8.0_66/jre/lib/amd64/server/libjvm.so
#2 0x00007f3491e54ff3 in os::sleep(Thread*, long, bool) ()
#3 0x00007f3491c56f02 in JVM_Sleep () at /v3/java/jdk1.8.0_66/jre/lib/amd64/server/libjvm.so
#4 0x00007f348bc15994 in j java/lang/Thread#sleep(J)V ()
#5 0x00007f348bc07c4d in j a#xxx3()V ()
#6 0x00007f348bc07c4d in j a#xxx2()V ()
#7 0x00007f348bc07c4d in j a#xxx1()V ()
#8 0x00007f348bc07c4d in j a#main([Ljava/lang/String;)V ()
#9 0x00007f348bc007a7 in <> ()
#10 0x00007f3491bc3c46 in JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, T
#11 0x00007f3491c05262 in jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID
#12 0x00007f3491c21c6a in jni_CallStaticVoidMethod () at /v3/java/jdk1.8.0_66/jre/lib/amd64/server/
#13 0x00007f3492ae7bcc in JavaMain () at /v3/java/jdk1.8.0_66/bin/../lib/amd64/jli/libjli.so
#14 0x00007f3492cfd6aa in start_thread (arg=0x7f349311b700) at pthread_create.c:333
#15 0x00007f3492618eed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
40

View Slide

View Slide

アジェンダ
やりたいこと
VMStruct
backtraceの実装
TODO
42

View Slide

TODO
各種プラットフォーム展開
コマンド充実
oopコマンド
ライブデバッグ
JDK9
43

View Slide

oopコマンド
(instanceOop*) 0xefe983d0 : oop is java.net.SocksSocketImpl instance.
+00 markOop _mark = 0x3ce7e9f8
+04 klassOop _klass = 0xcd649f38
=== instance value ===
*** java.net.SocketImpl ***
+08 Ljava.net.Socket; socket = 0xefe97db8
+0c Ljava.net.ServerSocket; serverSocket = 0x00000000
+10 protected Ljava.io.FileDescriptor; fd = 0xefe98478
+14 protected Ljava.net.InetAddress; address = 0xdf0cb5b0
+18 protected I port = 0x00009726
+1c protected I localport = 0x0000a5b1
*** java.net.PlainSocketImpl ***
+20 private Ljava.net.SocketInputStream; socketInputStream = 0x00000000
+24 private Ljava.lang.Object; fdLock = 0xefe98468
+28 private Ljava.lang.Object; resetLock = 0xefe98470
+2c private Ljava.io.FileDescriptor; fd1 = 0x00000000
+30 private Ljava.net.InetAddress; anyLocalBoundAddr = 0xd8e4b968
44

View Slide

Q/A
45

View Slide


View Slide

Python + GDB = Javaデバッガ

Python + GDB = Javaデバッガ

Kenji Kazumura

More Decks by Kenji Kazumura

Other Decks in Programming

Featured

Transcript