Slide 1
Slide 1 text
GoConference’19
Linux middleware
development by Go
ʙݰքṗͰͱΕͨଠڕʹകιʔεΛఴ͑ͯʙ
Slide 2
Slide 2 text
GoConference’19
ࢁԼ @pyama86
GMOϖύϘ ٕज़ج൫νʔϜ
γχΞɾϓϦϯγύϧ
ten-snapon.com
pyama.fun
stns.jp
2
Slide 3
Slide 3 text
GoConference’19
झຯ: ࠗ׆
3
Slide 4
Slide 4 text
GoConference’19
4
ϗεςΟϯάࣄۀ &$ࢧԉࣄۀ ϋϯυϝΠυɾͦͷଞࣄۀ
Slide 5
Slide 5 text
45/4
-JOVY/444FSWFS
TUOTKQ
Slide 6
Slide 6 text
GoConference’19
Linux middleware
development by Go
6
Slide 7
Slide 7 text
GoConference’19
͜ͷηογϣϯͰ͢͜ͱ
w͠ख͕ૺ۰ͨ͠ϛυϧΣΞ։ൃ࣌ͷτϥϒϧʹ͍ͭͯ
wͦΕΒΛͲ͏ରԠ͔ͨ͠
wਓ͢Δ
7
Slide 8
Slide 8 text
GoConference’19
GoϛυϧΣΞ։ൃͱ૬ੑ͕ྑ͍
w(Pγϯά
ϧόΠφϦͰ؆୯ʹ͕Մೳ
w$(0Λར༻ͯ͠ɺطଘͷ$ݴޠͷࢿ࢈Λར༻͢Δ͜ͱ͕Ͱ͖Δ
wγεςϜϓϩάϥϛϯάʹඞཁͳػೳ͕΄΅Ұ௨Γଗ͍ͬͯΔ
wHPSPVUJOFΛར༻͢Δͱ؆୯ʹϚϧνεϨουϓϩάϥϛϯά͕Մೳ
8
Slide 9
Slide 9 text
GoConference’19
9
STNS
MJOVYTFSWFS
MJCOTTTUOTTP 45/4
SPPU!VCVOUVYFOJBMdJEQZBNB
VJE QZBNB
HJE QFQBCP
IUUQSFRVFTU
OFUIUUQ
#BDLFOET
50.-4FUDE
%ZOBNP%#
$(0
Slide 10
Slide 10 text
GoConference’19
10
libnss-stns.so
MJCOTTTUOTTPʹ͓͍ͯɺλʔϛφϧͰλϒิ͠
Α͏ͱ͢ΔͱϑϦʔζ͢Δ
MTd
Slide 11
Slide 11 text
GoConference’19
11
Slide 12
Slide 12 text
GoConference’19
12
what is libnss
w -JOVYͷ໊લղܾͷΈͰ͋ΔOTT͔Βಛఆͷ໋໊نʹج͍ͮͨTPΛಡ
ΈࠐΈɺҙͷํ๏Ͱ໊લղܾΛߦ͏͜ͱ͕Ͱ͖Δ
CBTI
SPPU!VCVOUVYFOJBMdTUSBDFUUGJEQZBNBcHSFQMJCOTT
PQFO MJCY@MJOVYHOVMJCOTT@DPNQBUTP
0@3%0/-:c0@$-0&9&$
PQFO MJCY@MJOVYHOVMJCOTT@OJTTP
0@3%0/-:c0@$-0&9&$
PQFO MJCY@MJOVYHOVMJCOTT@pMFTTP
0@3%0/-:c0@$-0&9&$
FUDQBTTXE
MJCOTT@pMFTTP
Slide 13
Slide 13 text
GoConference’19
13
libnss-stns 2016
w (P͔Βఏڙ͞ΕͨCVJMENPEFDTIBSFEͰTP TIBSFEPCKFDU
Λ࡞
SPPU!VCVOUVYFOJBMd(0"3$)BNEHPCVJMEYPCJOBSZMJCOTTTUOTTPa
CVJMENPEFDTIBSFENBJOHPSFTPVSDFHPQBTTXEHPTIBEPXHPHSPVQHP
Slide 14
Slide 14 text
GoConference’19
14
࣌ͷௐࠪ
ิલ
SPPU!VCVOUVYFOJBMdQTFGcHSFQ?ZBNB
QZBNBQUTTV
ิ࣮ߦத
SPPU!VCVOUVYFOJBMdQTFGcHSFQ?ZBNB
QZBNBQUTTV
QZBNBQUTTV
-JOVYͰλʔϛφϧิΛར༻͢Δͱ͖ɺ
എޙͰGPSL͕࣮ߦ͞Ε͍ͯΔ
Slide 15
Slide 15 text
GoConference’19
15
libnss
MJOVYTFSWFS
MJCOTTTUOTTP 45/4
GVODHFU SFTPVSDF@UZQF
DPMVNO
WBMVFTUSJOH
BUUSJCVUF6TFS(SPVQT
FSSPS
\
MPH1SJOU UFTU
SFR
FSSSFRVFTU/FX3FRVFTU SFTPVSDF@UZQF
DPMVNO
WBMVF
MPH1SJOU UFTU
JGFSSOJM\
SFUVSOOJM
FSS
^
SFTPVSDF
FSSSFR(FU
˒͕͜͜ࢮ͵
MPH1SJOU UFTU
IUUQSFRVFTU
Slide 16
Slide 16 text
GoConference’19
16
(PͰσʔϞϯ͕ੵۃతʹαϙʔτ͞Ε͍ͯͳ͍ͷͰ
ͳΜ͔౿ΜͰΔͱࢥ͍ͬͯͨ
Slide 17
Slide 17 text
GoConference’19
17
–dominikh
https://github.com/golang/go/issues/12734
l5PFYQBOE5IFJTTVFSFBMMZMJFTXJUIVTJOHDHP XIJDIOFUVTFT
JHOPSJOHUIFEFUBJMT
8IFOVTJOHDHP
UIF(PEFBEMPDLEFUFDUJPO
DBOOPUGVODUJPOQSPQFSMZ
CFDBVTF$XPSMENJHIUDBMM(PGVODUJPOTBU
BOZUJNF
TPJOUIFPSZOPEFBEMPDLFYJTUTXFNJHIUKVTUCFXBJUJOHGPS
BOFYUFSOBMGVODUJPODBMMJOEFpOJUFMZz
Slide 18
Slide 18 text
GoConference’19
࣌ͷԶ·ΔͰ
ԿΘ͔͍ͬͯͳ͍
18
Slide 19
Slide 19 text
GoConference’19
19
rIUUQTHJUIVCDPN45/4MJCOTT@TUOTQVMM
Slide 20
Slide 20 text
GoConference’19
20
Three years have passed since then.
Slide 21
Slide 21 text
GoConference’19
21
(gdb) bt
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1 0x00007f2271b2c4db in runtime.futexsleep (addr=0xc000032148, val=0, ns=-1) at /usr/local/go/src/
runtime/os_linux.go:46
#2 0x00007f2271b0cb95 in runtime.notesleep (n=0xc000032148) at /usr/local/go/src/runtime/
lock_futex.go:151
#3 0x00007f2271b34ad0 in runtime.stoplockedm () at /usr/local/go/src/runtime/proc.go:2076
#4 0x00007f2271b3638e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2477
#5 0x00007f2271b36517 in runtime.park_m (gp=0xc000000600) at /usr/local/go/src/runtime/proc.go:2605
#6 0x00007f2271b58443 in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299
...
#10 0x00007f2271b59d4e in runtime.cgocallback () at /usr/local/go/src/runtime/asm_amd64.s:684
#11 0x00007f2271d663d7 in _cgoexp_be88b77bdbf1__nss_stns_setpwent (a=0x7ffff24a6777, n=0, ctxt=0)
at _cgo_gotypes.go:129
#12 0x00007f2271c2ef47 in crosscall2 () at /usr/local/go/src/runtime/cgo/asm_amd64.s:59
#13 0x00007f2271d68046 in _nss_stns_setpwent () at _cgo_export.c:82
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
·ͣ$ͷ͕ؔίʔϧ͞ΕΔ
ΈΜͳେ͖gdb
Slide 22
Slide 22 text
GoConference’19
22
(gdb) bt
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1 0x00007f2271b2c4db in runtime.futexsleep (addr=0xc000032148, val=0, ns=-1) at /usr/local/go/src/
runtime/os_linux.go:46
#2 0x00007f2271b0cb95 in runtime.notesleep (n=0xc000032148) at /usr/local/go/src/runtime/
lock_futex.go:151
#3 0x00007f2271b34ad0 in runtime.stoplockedm () at /usr/local/go/src/runtime/proc.go:2076
#4 0x00007f2271b3638e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2477
#5 0x00007f2271b36517 in runtime.park_m (gp=0xc000000600) at /usr/local/go/src/runtime/proc.go:2605
#6 0x00007f2271b58443 in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299
...
#10 0x00007f2271b59d4e in runtime.cgocallback () at /usr/local/go/src/runtime/asm_amd64.s:684
#11 0x00007f2271d663d7 in _cgoexp_be88b77bdbf1__nss_stns_setpwent (a=0x7ffff24a6777, n=0,
ctxt=0) at _cgo_gotypes.go:129
#12 0x00007f2271c2ef47 in crosscall2 () at /usr/local/go/src/runtime/cgo/asm_amd64.s:59
#13 0x00007f2271d68046 in _nss_stns_setpwent () at _cgo_export.c:82
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
$ͷίϯςΩετ͔Β(PͷؔΛίʔϧ
ΈΜͳେ͖gdb
Slide 23
Slide 23 text
GoConference’19
23
ΈΜͳେ͖gdb
(gdb) bt
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1 0x00007f2271b2c4db in runtime.futexsleep (addr=0xc000032148, val=0, ns=-1) at /usr/local/go/src/
runtime/os_linux.go:46
#2 0x00007f2271b0cb95 in runtime.notesleep (n=0xc000032148) at /usr/local/go/src/runtime/
lock_futex.go:151
#3 0x00007f2271b34ad0 in runtime.stoplockedm () at /usr/local/go/src/runtime/proc.go:2076
#4 0x00007f2271b3638e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2477
#5 0x00007f2271b36517 in runtime.park_m (gp=0xc000000600) at /usr/local/go/src/runtime/proc.go:2605
#6 0x00007f2271b58443 in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299
...
#10 0x00007f2271b59d4e in runtime.cgocallback () at /usr/local/go/src/runtime/asm_amd64.s:684
#11 0x00007f2271d663d7 in _cgoexp_be88b77bdbf1__nss_stns_setpwent (a=0x7ffff24a6777, n=0, ctxt=0)
at _cgo_gotypes.go:129
#12 0x00007f2271c2ef47 in crosscall2 () at /usr/local/go/src/runtime/cgo/asm_amd64.s:59
#13 0x00007f2271d68046 in _nss_stns_setpwent () at _cgo_export.c:82
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
࣮ߦՄೳͳHPSPVUJOFΛ֬อ͠ɺ࣮ߦ͢Δ
Slide 24
Slide 24 text
GoConference’19
24
(gdb) bt
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1 0x00007f2271b2c4db in runtime.futexsleep (addr=0xc000032148, val=0, ns=-1) at /usr/local/go/src/
runtime/os_linux.go:46
#2 0x00007f2271b0cb95 in runtime.notesleep (n=0xc000032148) at /usr/local/go/src/runtime/
lock_futex.go:151
#3 0x00007f2271b34ad0 in runtime.stoplockedm () at /usr/local/go/src/runtime/proc.go:2076
#4 0x00007f2271b3638e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2477
#5 0x00007f2271b36517 in runtime.park_m (gp=0xc000000600) at /usr/local/go/src/runtime/proc.go:2605
#6 0x00007f2271b58443 in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299
...
#10 0x00007f2271b59d4e in runtime.cgocallback () at /usr/local/go/src/runtime/asm_amd64.s:684
#11 0x00007f2271d663d7 in _cgoexp_be88b77bdbf1__nss_stns_setpwent (a=0x7ffff24a6777, n=0, ctxt=0)
at _cgo_gotypes.go:129
#12 0x00007f2271c2ef47 in crosscall2 () at /usr/local/go/src/runtime/cgo/asm_amd64.s:59
#13 0x00007f2271d68046 in _nss_stns_setpwent () at _cgo_export.c:82
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
GVUFYͰ@'65&9@8"*5@13*7"5&ʹͯγάφϧΛ͕ͭ
ಧ͔ͣʹແݶʹͪଓ͚Δ
ΈΜͳେ͖gdb
Slide 25
Slide 25 text
GoConference’19
ͬ͘͟Γgoroutine
scheduler
goroutine 1
goroutine 2
goroutine 3
OS Thread 1
OS Thread 2
OS Thread3
HPSPVUJOFεϨου্ʹεέδϡʔϦϯά͞ΕΔ
Slide 26
Slide 26 text
GoConference’19
Ծઆ
fork࣌ʹmain thread͚ͩ
ίϐʔ͞ΕΔ͔Β͓͔͘͠ͳ͍ͬͯΔʁ
26
IUUQTMJOVYKNPTEOKQIUNM-%1@NBOQBHFTNBOGPSLIUNM
Slide 27
Slide 27 text
GoConference’19
fork?
scheduler
goroutine 1
goroutine 2
goroutine 3
OS Thread 1
OS Thread 2
OS Thread3
GPSL࣌ʹҰͭͷεϨου͔͠ίϐʔ͞Ε
ͳ͍͔Β͏·͘ಈ͍ͯͳ͍ʁ
Slide 28
Slide 28 text
GoConference’19
൱ఆ:idίϚϯυforkͯ͠Δ
28
root@ubuntu-xenial:~# strace -tt -f id pyama 2>&1 | grep pid |awk -F'] ' '{ print $1}' | sort | uniq
[pid 2460
[pid 2461
[pid 2462
[pid 2463
[pid 2464
[pid 2465
[pid 2466
[pid 2467
[pid 2468
Slide 29
Slide 29 text
GoConference’19
ͪΌΜͱॳظԽ͞Ε͍ͯΔ͔ʁ
29
func init() {
logger, err := syslog.New(syslog.LOG_ERR|syslog.LOG_USER, "stns")
if err != nil {
// syslog not found
fmt.Print(err)
} else {
log.SetOutput(logger)
}
log.Println("call init")
}
Slide 30
Slide 30 text
GoConference’19
ϑϦʔζ࣌init͕࣮ߦ͞Εͳ͍
30
pyama@ubuntu-xenial:~$ id pyama
uid=10301(pyama) gid=2000(pepabo) groups=2000(pepabo)
# syslog
Jul 10 04:16:18 ubuntu-xenial stns[16278]: 2019/07/10 04:16:18 call init
pyama@ubuntu-xenial:~$ ls ~[tab]
# syslog
ग़ྗͳ͠
Slide 31
Slide 31 text
GoConference’19
_cgo_wait_runtime_init_done()
31
CGO_NO_SANITIZE_THREAD GoInt _nss_stns_getspnam_r(char* p0, struct spwd* p1, char*
p2, size_t p3, struct spwd** p4)
{
__SIZE_TYPE__ _cgo_ctxt = _cgo_wait_runtime_init_done();
...
}
IUUQTRJJUBDPNZVHVJJUFNTFEECEB
DHPΛͬͨ$ͱ(PͷϦϯΫͷཪଆ
-JOVY͔ΒݺΕΔ$ͷؔ
(PͷSVOUJNFͷॳظԽ͕ྃ͢Δ·Ͱͭ
Slide 32
Slide 32 text
GoConference’19
_cgo_wait_runtime_init_done()
32
35 _cgo_wait_runtime_init_done(void) {
36 void (*pfn)(struct context_arg*);
37
38 pthread_mutex_lock(&runtime_init_mu);
39 while (runtime_init_done == 0) {
(gdb) p runtime_init_done
$1 = 1
ϑϦʔζͨ͠ͱ͖ͷϑϥάΛݟΔͱॳظԽࡁΈ
Slide 33
Slide 33 text
GoConference’19
Ͳ͜Ͱϑϥάཱ͕ͭͷ͔
33
root@ubuntu-xenial:~# gdb id
(gdb) set follow-fork-mode child
(gdb) b _cgo_wait_runtime_init_done
(gdb) run pyama
Starting program: /usr/bin/id pyama
(gdb) watch runtime_init_done
Hardware watchpoint 2: runtime_init_done
(gdb) c
Continuing.
....
Thread 2 "id" hit Hardware watchpoint 2: runtime_init_done
Old value = 0
New value = 1
x_cgo_notify_runtime_init_done (dummy=) at gcc_libinit.c:69
69 pthread_cond_broadcast(&runtime_init_cond);
มΛ8BUDIͯ͠มߋ͕͋Εࢭ·Δ
Y@DHP@OPUJGZ@SVOUJNF@JOJU@EPOF
Ͱมߋ͞Ε͍ͯΔ
Slide 34
Slide 34 text
GoConference’19
x_cgo_notify_runtime_init_doneͲ͔͜ΒݺΕΔʁ
34
(gdb) bt
#0 x_cgo_notify_runtime_init_done (dummy=) at gcc_libinit.c:69
#1 0x00007ffff61acf58 in runtime.asmcgocall () at /usr/local/go/src/runtime/asm_amd64.s:635
#2 0x00007ffff61a987e in runtime.(*mheap).alloc.func1 () at /usr/local/go/src/runtime/
mheap.go:1048
#3 0x00007ffff61ab763 in runtime.systemstack () at /usr/local/go/src/runtime/asm_amd64.s:
351
#4 0x00007ffff6186140 in ?? () at /usr/local/go/src/runtime/proc.go:1082 from /usr/lib/x86_64-
linux-gnu/libnss_stns.so.2
#5 0x00007ffff61ab5fd in runtime.rt0_go () at /usr/local/go/src/runtime/asm_amd64.s:201
#6 0x0000000000000000 in ?? ()
SVOUJNFSU@HPͷΞηϯϒϥΛḷΔͱɺେݩ
@SU@BNE@MJC@SU@BNE@MJC@HPSVOUJNFuSU@HP
ͷΑ͏ʹݺͼग़͞Ε͍ͯΔ
Slide 35
Slide 35 text
GoConference’19
c-sharedͷ߹runtimeͷॳظԽॲཧ͕ಡΈࠐΈ࣌ͷΈߦΘΕΔ
35
// _rt0_amd64_lib is common startup code for most amd64 systems when
// using -buildmode=c-archive or -buildmode=c-shared. The linker will
// arrange to invoke this function as a global constructor (for
// c-archive) or when the shared library is loaded (for c-shared).
// We expect argc and argv to be passed in the usual C ABI registers
// DI and SI.
TEXT _rt0_amd64_lib(SB),NOSPLIT,$0x50
// Align stack per ELF ABI requirements.
MOVQ SP, AX
ANDQ $~15, SP
HPMBOHHPTSDSVOUJNFBTN@BNET
Slide 36
Slide 36 text
GoConference’19
ൺֱ͢Δͱlibnss-stns͕طʹಡΈࠐ·Ε͍ͯͨ
36
pyama@ubuntu-xenial:~$ lsof -p $$ |grep libnss
bash 3810 pyama mem REG 8,1 9163712 57959 /usr/lib/x86_64-linux-gnu/libnss_stns.so
bash 3810 pyama mem REG 8,1 47600 2011 /lib/x86_64-linux-gnu/libnss_files-2.23.so
bash 3810 pyama mem REG 8,1 47648 2017 /lib/x86_64-linux-gnu/libnss_nis-2.23.so
bash 3810 pyama mem REG 8,1 35688 2006 /lib/x86_64-linux-gnu/libnss_compat-2.23.so
root@ubuntu-xenial:~# lsof -p $$ |grep libnss
bash 2022 root mem REG 8,1 47600 2011 /lib/x86_64-linux-gnu/libnss_files-2.23.so
bash 2022 root mem REG 8,1 47648 2017 /lib/x86_64-linux-gnu/libnss_nis-2.23.so
bash 2022 root mem REG 8,1 35688 2006 /lib/x86_64-linux-gnu/libnss_compat-2.23.so
MPDBMVTFS
TUOTVTFS
Slide 37
Slide 37 text
GoConference’19
ࣄͷཧ
CBTI͕ىಈ͠ɺMJCOTTTUOTTP͕ಡΈࠐ·Εɺಉ࣌ʹ(PͷSVOUJNF͕ॳظ
Խ͞ΕΔ
MTdΛ࣮ߦ͢ΔͱɺCBTI͕GPSL͞Εɺىಈͨ͠εϨουͷΈ͕GPSLઌ
ͷϓϩηεʹίϐʔ͞ΕΔ
OFUIUUQύοέʔδͷIUUQ$MJFOU%Pͷ݁Ռ͕ฦ٫͞Εͳ͍
37
Slide 38
Slide 38 text
GoConference’19
ࣄ͔ΒಘΒΕΔ݁
wGPSLޙͷϓϩηεʹ͓͍ͯɺ(PͷSVOUJNF͕ॳظԽ͞Εͳ͍ͨΊɺ
GPSLલʹॳظԽ͞ΕͨใͰ(PͷSVOUJNF͕ಈ͔͘Βਖ਼ৗʹಈ࡞͠
ͳ͍
38
Slide 39
Slide 39 text
GoConference’19
ֶͼɺͦͯ͠·ͱΊ
w$(0ΛDTIBSFEͰར༻͢ΔࡍʹɺجຊతʹGPSL͢ΔΑ͏ͳ༻
్ʹΘͳ͍΄͏͕ྑ͍
w͠ར༻͢ΔͳΒGPSLઌͷϓϩηεͰ໌ࣔతʹॳظԽ͢Δඞཁ
͕͋Δ
wਓֶؒͼɺલʹ;ΘͬͱҋύονͰย͚ͨਂ͘ཧ
ղ͠ղܾ͢Δ͜ͱ͕Ͱ͖ΔΑ͏ʹͳΔ
39
Slide 40
Slide 40 text
GoConference’19
ਂֶ͘ͼɺͰ͖Δڥ͕200mઌʹ͋Γ·͢
!QC@SFDSVJU
40