GoConference’19 Linux middleware development by Go ʙݰքṗͰͱΕͨଠ౛ڕʹക೑ιʔεΛఴ͑ͯʙ

GoConference’19 ࢁԼ ࿨඙ @pyama86 GMOϖύϘ ٕज़ج൫νʔϜ γχΞɾϓϦϯγύϧ ten-snapon.com pyama.fun stns.jp 2

GoConference’19 झຯ: ࠗ׆ 3

GoConference’19 4 ϗεςΟϯάࣄۀ &$ࢧԉࣄۀ ϋϯυϝΠυɾͦͷଞࣄۀ

45/4 -JOVY/444FSWFS TUOTKQ

GoConference’19 Linux middleware development by Go 6

GoConference’19 ͜ͷηογϣϯͰ࿩͢͜ͱ w࿩͠ख͕ૺ۰ͨ͠ϛυϧ΢ΣΞ։ൃ࣌ͷτϥϒϧʹ͍ͭͯ wͦΕΒΛͲ͏ରԠ͔ͨ͠ wਓ͸੒௕͢Δ 7

GoConference’19 Go͸ϛυϧ΢ΣΞ։ൃͱ૬ੑ͕ྑ͍ w(P͸γϯά ϧόΠφϦͰ؆୯ʹ഑෍͕Մೳ w$(0Λར༻ͯ͠ɺطଘͷ$ݴޠͷࢿ࢈Λར༻͢Δ͜ͱ͕Ͱ͖Δ wγεςϜϓϩάϥϛϯάʹඞཁͳػೳ͕΄΅Ұ௨Γଗ͍ͬͯΔ wHPSPVUJOFΛར༻͢Δͱ؆୯ʹϚϧνεϨουϓϩάϥϛϯά͕Մೳ 8

GoConference’19 9 STNS MJOVYTFSWFS MJCOTTTUOTTP 45/4 SPPU!VCVOUVYFOJBMdJEQZBNB VJE QZBNB HJE QFQBCP IUUQSFRVFTU OFUIUUQ #BDLFOET 50.-4FUDE %ZOBNP%# $(0

GoConference’19 10 libnss-stns.so MJCOTTTUOTTPʹ͓͍ͯɺλʔϛφϧͰλϒิ׬͠ Α͏ͱ͢ΔͱϑϦʔζ͢Δ MTd

GoConference’19 11

GoConference’19 12 what is libnss w -JOVYͷ໊લղܾͷ࢓૊ΈͰ͋ΔOTT͔Βಛఆͷ໋໊ن໿ʹج͍ͮͨTPΛಡ ΈࠐΈɺ೚ҙͷํ๏Ͱ໊લղܾΛߦ͏͜ͱ͕Ͱ͖Δ CBTI SPPU!VCVOUVYFOJBMdTUSBDFUUGJEQZBNBcHSFQMJCOTT PQFO MJCY@MJOVYHOVMJCOTT@DPNQBUTP 0@3%0/-:c0@$-0&9&$ PQFO MJCY@MJOVYHOVMJCOTT@OJTTP 0@3%0/-:c0@$-0&9&$ PQFO MJCY@MJOVYHOVMJCOTT@pMFTTP 0@3%0/-:c0@$-0&9&$ FUDQBTTXE MJCOTT@pMFTTP

GoConference’19 13 libnss-stns 2016 w (P͔Βఏڙ͞ΕͨCVJMENPEFDTIBSFEͰTP TIBSFEPCKFDU Λ࡞੒ SPPU!VCVOUVYFOJBMd(0"3$)BNEHPCVJMEYPCJOBSZMJCOTTTUOTTPa CVJMENPEFDTIBSFENBJOHPSFTPVSDFHPQBTTXEHPTIBEPXHPHSPVQHP

GoConference’19 14 ౰࣌ͷௐࠪ ิ׬લ SPPU!VCVOUVYFOJBMdQTFGcHSFQ?ZBNB QZBNBQUTTV ิ׬࣮ߦத SPPU!VCVOUVYFOJBMdQTFGcHSFQ?ZBNB QZBNBQUTTV QZBNBQUTTV -JOVYͰλʔϛφϧิ׬Λར༻͢Δͱ͖ɺ എޙͰ͸GPSL͕࣮ߦ͞Ε͍ͯΔ

GoConference’19 15 libnss MJOVYTFSWFS MJCOTTTUOTTP 45/4 GVODHFU SFTPVSDF@UZQF DPMVNO WBMVFTUSJOH BUUSJCVUF6TFS(SPVQT FSSPS \ MPH1SJOU UFTU SFR FSSSFRVFTU/FX3FRVFTU SFTPVSDF@UZQF DPMVNO WBMVF MPH1SJOU UFTU JGFSSOJM\ SFUVSOOJM FSS ^ SFTPVSDF FSSSFR(FU ˒͕͜͜ࢮ͵ MPH1SJOU UFTU IUUQSFRVFTU

GoConference’19 16 (PͰ͸σʔϞϯ͕ੵۃతʹαϙʔτ͞Ε͍ͯͳ͍ͷͰ ͳΜ͔౿ΜͰΔͱࢥ͍ͬͯͨ

GoConference’19 17 –dominikh https://github.com/golang/go/issues/12734 l5PFYQBOE5IFJTTVFSFBMMZMJFTXJUIVTJOHDHP XIJDIOFUVTFT JHOPSJOHUIFEFUBJMT 8IFOVTJOHDHP UIF(PEFBEMPDLEFUFDUJPO DBOOPUGVODUJPOQSPQFSMZ CFDBVTF$XPSMENJHIUDBMM(PGVODUJPOTBU BOZUJNF TPJOUIFPSZOPEFBEMPDLFYJTUTXFNJHIUKVTUCFXBJUJOHGPS BOFYUFSOBMGVODUJPODBMMJOEFpOJUFMZz

GoConference’19 ౰࣌ͷԶ͸·ΔͰ Կ΋Θ͔͍ͬͯͳ͍ 18

GoConference’19 19 rIUUQTHJUIVCDPN45/4MJCOTT@TUOTQVMM

GoConference’19 20 Three years have passed since then.

GoConference’19 21 (gdb) bt #0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536 #1 0x00007f2271b2c4db in runtime.futexsleep (addr=0xc000032148, val=0, ns=-1) at /usr/local/go/src/ runtime/os_linux.go:46 #2 0x00007f2271b0cb95 in runtime.notesleep (n=0xc000032148) at /usr/local/go/src/runtime/ lock_futex.go:151 #3 0x00007f2271b34ad0 in runtime.stoplockedm () at /usr/local/go/src/runtime/proc.go:2076 #4 0x00007f2271b3638e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2477 #5 0x00007f2271b36517 in runtime.park_m (gp=0xc000000600) at /usr/local/go/src/runtime/proc.go:2605 #6 0x00007f2271b58443 in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299 ... #10 0x00007f2271b59d4e in runtime.cgocallback () at /usr/local/go/src/runtime/asm_amd64.s:684 #11 0x00007f2271d663d7 in _cgoexp_be88b77bdbf1__nss_stns_setpwent (a=0x7ffff24a6777, n=0, ctxt=0) at _cgo_gotypes.go:129 #12 0x00007f2271c2ef47 in crosscall2 () at /usr/local/go/src/runtime/cgo/asm_amd64.s:59 #13 0x00007f2271d68046 in _nss_stns_setpwent () at _cgo_export.c:82 Backtrace stopped: previous frame inner to this frame (corrupt stack?) ·ͣ͸$ͷؔ਺͕ίʔϧ͞ΕΔ ΈΜͳେ޷͖gdb

GoConference’19 22 (gdb) bt #0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536 #1 0x00007f2271b2c4db in runtime.futexsleep (addr=0xc000032148, val=0, ns=-1) at /usr/local/go/src/ runtime/os_linux.go:46 #2 0x00007f2271b0cb95 in runtime.notesleep (n=0xc000032148) at /usr/local/go/src/runtime/ lock_futex.go:151 #3 0x00007f2271b34ad0 in runtime.stoplockedm () at /usr/local/go/src/runtime/proc.go:2076 #4 0x00007f2271b3638e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2477 #5 0x00007f2271b36517 in runtime.park_m (gp=0xc000000600) at /usr/local/go/src/runtime/proc.go:2605 #6 0x00007f2271b58443 in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299 ... #10 0x00007f2271b59d4e in runtime.cgocallback () at /usr/local/go/src/runtime/asm_amd64.s:684 #11 0x00007f2271d663d7 in _cgoexp_be88b77bdbf1__nss_stns_setpwent (a=0x7ffff24a6777, n=0, ctxt=0) at _cgo_gotypes.go:129 #12 0x00007f2271c2ef47 in crosscall2 () at /usr/local/go/src/runtime/cgo/asm_amd64.s:59 #13 0x00007f2271d68046 in _nss_stns_setpwent () at _cgo_export.c:82 Backtrace stopped: previous frame inner to this frame (corrupt stack?) $ͷίϯςΩετ͔Β(Pͷؔ਺Λίʔϧ ΈΜͳେ޷͖gdb

GoConference’19 23 ΈΜͳେ޷͖gdb (gdb) bt #0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536 #1 0x00007f2271b2c4db in runtime.futexsleep (addr=0xc000032148, val=0, ns=-1) at /usr/local/go/src/ runtime/os_linux.go:46 #2 0x00007f2271b0cb95 in runtime.notesleep (n=0xc000032148) at /usr/local/go/src/runtime/ lock_futex.go:151 #3 0x00007f2271b34ad0 in runtime.stoplockedm () at /usr/local/go/src/runtime/proc.go:2076 #4 0x00007f2271b3638e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2477 #5 0x00007f2271b36517 in runtime.park_m (gp=0xc000000600) at /usr/local/go/src/runtime/proc.go:2605 #6 0x00007f2271b58443 in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299 ... #10 0x00007f2271b59d4e in runtime.cgocallback () at /usr/local/go/src/runtime/asm_amd64.s:684 #11 0x00007f2271d663d7 in _cgoexp_be88b77bdbf1__nss_stns_setpwent (a=0x7ffff24a6777, n=0, ctxt=0) at _cgo_gotypes.go:129 #12 0x00007f2271c2ef47 in crosscall2 () at /usr/local/go/src/runtime/cgo/asm_amd64.s:59 #13 0x00007f2271d68046 in _nss_stns_setpwent () at _cgo_export.c:82 Backtrace stopped: previous frame inner to this frame (corrupt stack?) ࣮ߦՄೳͳHPSPVUJOFΛ֬อ͠ɺ࣮ߦ͢Δ

GoConference’19 24 (gdb) bt #0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536 #1 0x00007f2271b2c4db in runtime.futexsleep (addr=0xc000032148, val=0, ns=-1) at /usr/local/go/src/ runtime/os_linux.go:46 #2 0x00007f2271b0cb95 in runtime.notesleep (n=0xc000032148) at /usr/local/go/src/runtime/ lock_futex.go:151 #3 0x00007f2271b34ad0 in runtime.stoplockedm () at /usr/local/go/src/runtime/proc.go:2076 #4 0x00007f2271b3638e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2477 #5 0x00007f2271b36517 in runtime.park_m (gp=0xc000000600) at /usr/local/go/src/runtime/proc.go:2605 #6 0x00007f2271b58443 in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299 ... #10 0x00007f2271b59d4e in runtime.cgocallback () at /usr/local/go/src/runtime/asm_amd64.s:684 #11 0x00007f2271d663d7 in _cgoexp_be88b77bdbf1__nss_stns_setpwent (a=0x7ffff24a6777, n=0, ctxt=0) at _cgo_gotypes.go:129 #12 0x00007f2271c2ef47 in crosscall2 () at /usr/local/go/src/runtime/cgo/asm_amd64.s:59 #13 0x00007f2271d68046 in _nss_stns_setpwent () at _cgo_export.c:82 Backtrace stopped: previous frame inner to this frame (corrupt stack?) GVUFYͰ@'65&9@8"*5@13*7"5&ʹͯγάφϧΛ଴͕ͭ ಧ͔ͣʹແݶʹ଴ͪଓ͚Δ ΈΜͳେ޷͖gdb

GoConference’19 ͬ͘͟Γgoroutine scheduler goroutine 1 goroutine 2 goroutine 3 OS Thread 1 OS Thread 2 OS Thread3 HPSPVUJOF͸εϨου্ʹεέδϡʔϦϯά͞ΕΔ

GoConference’19 Ծઆ fork࣌ʹmain thread͚ͩ ίϐʔ͞ΕΔ͔Β͓͔͘͠ͳ͍ͬͯΔʁ 26 IUUQTMJOVYKNPTEOKQIUNM-%1@NBOQBHFTNBOGPSLIUNM

GoConference’19 fork? scheduler goroutine 1 goroutine 2 goroutine 3 OS Thread 1 OS Thread 2 OS Thread3 GPSL࣌ʹҰͭͷεϨου͔͠ίϐʔ͞Ε ͳ͍͔Β͏·͘ಈ͍ͯͳ͍ʁ

GoConference’19 ൱ఆ:idίϚϯυ΋forkͯ͠Δ 28 root@ubuntu-xenial:~# strace -tt -f id pyama 2>&1 | grep pid |awk -F'] ' '{ print $1}' | sort | uniq [pid 2460 [pid 2461 [pid 2462 [pid 2463 [pid 2464 [pid 2465 [pid 2466 [pid 2467 [pid 2468

GoConference’19 ͪΌΜͱॳظԽ͸͞Ε͍ͯΔ͔ʁ 29 func init() { logger, err := syslog.New(syslog.LOG_ERR|syslog.LOG_USER, "stns") if err != nil { // syslog not found fmt.Print(err) } else { log.SetOutput(logger) } log.Println("call init") }

GoConference’19 ϑϦʔζ࣌͸init͕࣮ߦ͞Εͳ͍ 30 pyama@ubuntu-xenial:~$ id pyama uid=10301(pyama) gid=2000(pepabo) groups=2000(pepabo) # syslog Jul 10 04:16:18 ubuntu-xenial stns[16278]: 2019/07/10 04:16:18 call init pyama@ubuntu-xenial:~$ ls ~[tab] # syslog ग़ྗͳ͠

GoConference’19 _cgo_wait_runtime_init_done() 31 CGO_NO_SANITIZE_THREAD GoInt _nss_stns_getspnam_r(char* p0, struct spwd* p1, char* p2, size_t p3, struct spwd** p4) { __SIZE_TYPE__ _cgo_ctxt = _cgo_wait_runtime_init_done(); ... } IUUQTRJJUBDPNZVHVJJUFNTFEECEB DHPΛ࢖ͬͨ$ͱ(PͷϦϯΫͷཪଆ -JOVY͔Βݺ͹ΕΔ$ͷؔ਺͸
 (PͷSVOUJNFͷॳظԽ͕׬ྃ͢Δ·Ͱ଴ͭ

GoConference’19 _cgo_wait_runtime_init_done() 32 35 _cgo_wait_runtime_init_done(void) { 36 void (*pfn)(struct context_arg*); 37 38 pthread_mutex_lock(&runtime_init_mu); 39 while (runtime_init_done == 0) { (gdb) p runtime_init_done $1 = 1 ϑϦʔζͨ͠ͱ͖ͷϑϥάΛݟΔͱॳظԽࡁΈ

GoConference’19 Ͳ͜Ͱϑϥάཱ͕ͭͷ͔ 33 root@ubuntu-xenial:~# gdb id (gdb) set follow-fork-mode child (gdb) b _cgo_wait_runtime_init_done (gdb) run pyama Starting program: /usr/bin/id pyama (gdb) watch runtime_init_done Hardware watchpoint 2: runtime_init_done (gdb) c Continuing. .... Thread 2 "id" hit Hardware watchpoint 2: runtime_init_done Old value = 0 New value = 1 x_cgo_notify_runtime_init_done (dummy=) at gcc_libinit.c:69 69 pthread_cond_broadcast(&runtime_init_cond); ม਺Λ8BUDIͯ͠มߋ͕͋Ε͹ࢭ·Δ Y@DHP@OPUJGZ@SVOUJNF@JOJU@EPOF ಺Ͱมߋ͞Ε͍ͯΔ

GoConference’19 x_cgo_notify_runtime_init_done͸Ͳ͔͜Βݺ͹ΕΔʁ 34 (gdb) bt #0 x_cgo_notify_runtime_init_done (dummy=) at gcc_libinit.c:69 #1 0x00007ffff61acf58 in runtime.asmcgocall () at /usr/local/go/src/runtime/asm_amd64.s:635 #2 0x00007ffff61a987e in runtime.(*mheap).alloc.func1 () at /usr/local/go/src/runtime/ mheap.go:1048 #3 0x00007ffff61ab763 in runtime.systemstack () at /usr/local/go/src/runtime/asm_amd64.s: 351 #4 0x00007ffff6186140 in ?? () at /usr/local/go/src/runtime/proc.go:1082 from /usr/lib/x86_64- linux-gnu/libnss_stns.so.2 #5 0x00007ffff61ab5fd in runtime.rt0_go () at /usr/local/go/src/runtime/asm_amd64.s:201 #6 0x0000000000000000 in ?? () SVOUJNFSU@HPͷΞηϯϒϥΛḷΔͱɺେݩ͸ @SU@BNE@MJC@SU@BNE@MJC@HPSVOUJNFuSU@HP ͷΑ͏ʹݺͼग़͞Ε͍ͯΔ

GoConference’19 c-sharedͷ৔߹͸runtimeͷॳظԽॲཧ͕ಡΈࠐΈ࣌ͷΈߦΘΕΔ 35 // _rt0_amd64_lib is common startup code for most amd64 systems when // using -buildmode=c-archive or -buildmode=c-shared. The linker will // arrange to invoke this function as a global constructor (for // c-archive) or when the shared library is loaded (for c-shared). // We expect argc and argv to be passed in the usual C ABI registers // DI and SI. TEXT _rt0_amd64_lib(SB),NOSPLIT,$0x50 // Align stack per ELF ABI requirements. MOVQ SP, AX ANDQ $~15, SP HPMBOHHPTSDSVOUJNFBTN@BNET

GoConference’19 ൺֱ͢Δͱlibnss-stns͕طʹಡΈࠐ·Ε͍ͯͨ 36 pyama@ubuntu-xenial:~$ lsof -p $$ |grep libnss bash 3810 pyama mem REG 8,1 9163712 57959 /usr/lib/x86_64-linux-gnu/libnss_stns.so bash 3810 pyama mem REG 8,1 47600 2011 /lib/x86_64-linux-gnu/libnss_files-2.23.so bash 3810 pyama mem REG 8,1 47648 2017 /lib/x86_64-linux-gnu/libnss_nis-2.23.so bash 3810 pyama mem REG 8,1 35688 2006 /lib/x86_64-linux-gnu/libnss_compat-2.23.so root@ubuntu-xenial:~# lsof -p $$ |grep libnss bash 2022 root mem REG 8,1 47600 2011 /lib/x86_64-linux-gnu/libnss_files-2.23.so bash 2022 root mem REG 8,1 47648 2017 /lib/x86_64-linux-gnu/libnss_nis-2.23.so bash 2022 root mem REG 8,1 35688 2006 /lib/x86_64-linux-gnu/libnss_compat-2.23.so MPDBMVTFS TUOTVTFS

GoConference’19 ࣄ৅ͷ੔ཧ CBTI͕ىಈ͠ɺMJCOTTTUOTTP͕ಡΈࠐ·Εɺಉ࣌ʹ(PͷSVOUJNF͕ॳظ Խ͞ΕΔ MTdΛ࣮ߦ͢ΔͱɺCBTI͕GPSL͞Εɺىಈͨ͠εϨουͷΈ͕GPSLઌ ͷϓϩηεʹίϐʔ͞ΕΔ OFUIUUQύοέʔδͷIUUQ$MJFOU%Pͷ݁Ռ͕ฦ٫͞Εͳ͍ 37

GoConference’19 ࣄ৅͔ΒಘΒΕΔ݁࿦ wGPSLޙͷϓϩηεʹ͓͍ͯɺ(PͷSVOUJNF͕ॳظԽ͞Εͳ͍ͨΊɺ GPSLલʹॳظԽ͞Εͨ৘ใͰ(PͷSVOUJNF͕ಈ͔͘Βਖ਼ৗʹಈ࡞͠ ͳ͍ 38

GoConference’19 ֶͼɺͦͯ͠·ͱΊ w$(0ΛDTIBSFEͰར༻͢Δࡍʹ͸ɺجຊతʹGPSL͢ΔΑ͏ͳ༻ ్ʹ͸࢖Θͳ͍΄͏͕ྑ͍ w΋͠ར༻͢ΔͳΒGPSLઌͷϓϩηεͰ໌ࣔతʹॳظԽ͢Δඞཁ ͕͋Δ wਓؒ͸ֶͼɺ೥લʹ;ΘͬͱҋύονͰย෇͚ͨ໰୊΋ਂ͘ཧ ղ͠ղܾ͢Δ͜ͱ͕Ͱ͖ΔΑ͏ʹͳΔ 39

GoConference’19 ਂֶ͘ͼɺ੒௕Ͱ͖Δ؀ڥ͕200mઌʹ͋Γ·͢ !QC@SFDSVJU 40