linux-middleware-development-by-go.pdf

Go言語を用いたLinuxサーバミドルウェア開発における失敗談を含むノウハウや、ベストプラクティスをトークします。Go言語はLinuxの各種ライブラリとの連携が容易であることや、CGOが利用できることで、従来の言語と比較しLinuxミドルウェアを作りやすい言語です。
一方で、言語として想定していないプロセス、メモリ状態もあり、そういった点に注意を払わないと想定外の挙動をしてしまいます。話し手がLinuxサーバミドルウェア開発を続ける中で実際に遭遇した動作、仕様を実例を元にトークします。

Transcript

1. GoConference’19 Linux middleware development by Go ʙݰքṗͰͱΕͨଠ౛ڕʹക೑ιʔεΛఴ͑ͯʙ
2. GoConference’19 ࢁԼ ࿨඙ @pyama86 GMOϖύϘ ٕज़ج൫νʔϜ γχΞɾϓϦϯγύϧ ten-snapon.com pyama.fun stns.jp 2
3. GoConference’19 झຯ: ࠗ׆ 3
4. GoConference’19 4 ϗεςΟϯάࣄۀ &$ࢧԉࣄۀ ϋϯυϝΠυɾͦͷଞࣄۀ
5. 45/4
   -JOVY
   /44
   4FSWFS
   TUOTKQ
6. GoConference’19 Linux middleware development by Go 6
7. GoConference’19 ͜ͷηογϣϯͰ࿩͢͜ͱ w࿩͠ख͕ૺ۰ͨ͠ϛυϧ΢ΣΞ։ൃ࣌ͷτϥϒϧʹ͍ͭͯ
   wͦΕΒΛͲ͏ରԠ͔ͨ͠
   wਓ͸੒௕͢Δ 7
8. GoConference’19 Go͸ϛυϧ΢ΣΞ։ൃͱ૬ੑ͕ྑ͍ w(P͸γϯά ϧόΠφϦͰ؆୯ʹ഑෍͕Մೳ
   w$(0Λར༻ͯ͠ɺطଘͷ$ݴޠͷࢿ࢈Λར༻͢Δ͜ͱ͕Ͱ͖Δ
   wγεςϜϓϩάϥϛϯάʹඞཁͳػೳ͕΄΅Ұ௨Γଗ͍ͬͯΔ
   wHPSPVUJOFΛར༻͢Δͱ؆୯ʹϚϧνεϨουϓϩάϥϛϯά͕Մೳ 8
9. GoConference’19 9 STNS MJOVY
   TFSWFS MJCOTTTUOTTP 45/4 SPPU!VCVOUVYFOJBMd
   JE
   QZBNB
   VJE QZBNB
   HJE QFQBCP
   IUUQ
   SFRVFTU OFUIUUQ #BDLFOET
   50.-4FUDE
   %ZOBNP%# $(0
10. GoConference’19 10 libnss-stns.so MJCOTTTUOTTPʹ͓͍ͯɺλʔϛφϧͰλϒิ׬͠ Α͏ͱ͢ΔͱϑϦʔζ͢Δ 
    MT
    d<UBC>
11. GoConference’19 11
12. GoConference’19 12 what is libnss w -JOVYͷ໊લղܾͷ࢓૊ΈͰ͋ΔOTT͔Βಛఆͷ໋໊ن໿ʹج͍ͮͨTPΛಡ ΈࠐΈɺ೚ҙͷํ๏Ͱ໊લղܾΛߦ͏͜ͱ͕Ͱ͖Δ CBTI SPPU!VCVOUVYFOJBMd
    TUSBDF
    UU
    G
    JE
    QZBNB
    
    c
    HSFQ
    MJCOTT
    
    PQFO MJCY@MJOVYHOVMJCOTT@DPNQBUTP
    0@3%0/-:c0@$-0&9&$
    
    
    
    PQFO MJCY@MJOVYHOVMJCOTT@OJTTP
    0@3%0/-:c0@$-0&9&$
    
    
    
    PQFO MJCY@MJOVYHOVMJCOTT@pMFTTP
    0@3%0/-:c0@$-0&9&$
    
     FUDQBTTXE MJCOTT@pMFTTP
13. GoConference’19 13 libnss-stns 2016 w (P
    ͔Βఏڙ͞ΕͨCVJMENPEF
    DTIBSFEͰTP TIBSFE
    PCKFDU Λ࡞੒ SPPU!VCVOUVYFOJBMd
    (0"3$)BNE
    HP
    CVJME
    Y
    P
    CJOBSZMJCOTTTUOTTP
    a
    
    CVJMENPEFDTIBSFE
    NBJOHP
    SFTPVSDFHP
    QBTTXEHP
    TIBEPXHP
    HSPVQHP
14. GoConference’19 14 ౰࣌ͷௐࠪ 
    ิ׬લ
    SPPU!VCVOUVYFOJBMd
    QT
    FG
    cHSFQ
    ?<Q>ZBNB
    QZBNB
    
    
    
    
    
    
    
    
    
    
    QUT
    
    
    
    
    TV
    
    ิ׬࣮ߦத
    SPPU!VCVOUVYFOJBMd
    QT
    FG
    cHSFQ
    ?<Q>ZBNB
    QZBNB
    
    
    
    
    
    
    
    
    
    
    QUT
    
    
    
    
    TV
    QZBNB
    
    
    
    
    
    
    
    
    
    
    QUT
    
    
    
    
    TV -JOVYͰλʔϛφϧิ׬Λར༻͢Δͱ͖ɺ
    എޙͰ͸GPSL͕࣮ߦ͞Ε͍ͯΔ
15. GoConference’19 15 libnss MJOVY
    TFSWFS MJCOTTTUOTTP 45/4 GVOD
    HFU SFTPVSDF@UZQF
    DPMVNO
    WBMVF
    TUSJOH
    BUUSJCVUF6TFS(SPVQT
    FSSPS
    \
    
    MPH1SJOU UFTU
    
    SFR
    FSS
    
    SFRVFTU/FX3FRVFTU SFTPVSDF@UZQF
    DPMVNO
    WBMVF
    
    MPH1SJOU UFTU
    
    JG
    FSS
    
    OJM
    \
    
    
    SFUVSO
    OJM
    FSS
    
    ^
    
    SFTPVSDF
    FSS
    
    SFR(FU
    ˒
    ͕͜͜ࢮ͵
    
    MPH1SJOU UFTU
    IUUQ
    SFRVFTU
16. GoConference’19 16 (PͰ͸σʔϞϯ͕ੵۃతʹαϙʔτ͞Ε͍ͯͳ͍ͷͰ
    ͳΜ͔౿ΜͰΔͱࢥ͍ͬͯͨ
17. GoConference’19 17 –dominikh https://github.com/golang/go/issues/12734 l5P
    FYQBOE
    5IF
    JTTVF
    SFBMMZ
    MJFT
    XJUI
    VTJOH
    DHP
    XIJDI
    OFU
    VTFT
    JHOPSJOH
    UIF
    EFUBJMT 
    8IFO
    VTJOH
    DHP
    UIF
    (P
    EFBEMPDL
    EFUFDUJPO
    DBOOPU
    GVODUJPO
    QSPQFSMZ
    CFDBVTF
    $
    XPSME
    NJHIU
    DBMM
    (P
    GVODUJPOT
    BU
    BOZ
    UJNF
    TP
    JO
    UIFPSZ
    OP
    EFBEMPDL
    FYJTUT
    XF
    NJHIU
    KVTU
    CF
    XBJUJOH
    GPS
    BO
    FYUFSOBM
    GVODUJPO
    DBMM
    JOEFpOJUFMZz
    
18. GoConference’19 ౰࣌ͷԶ͸·ΔͰ Կ΋Θ͔͍ͬͯͳ͍ 18
19. GoConference’19 19 rIUUQTHJUIVCDPN45/4MJCOTT@TUOTQVMM
20. GoConference’19 20 Three years have passed since then.
21. GoConference’19 21 (gdb) bt #0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536 #1 0x00007f2271b2c4db in runtime.futexsleep (addr=0xc000032148, val=0, ns=-1) at /usr/local/go/src/ runtime/os_linux.go:46 #2 0x00007f2271b0cb95 in runtime.notesleep (n=0xc000032148) at /usr/local/go/src/runtime/ lock_futex.go:151 #3 0x00007f2271b34ad0 in runtime.stoplockedm () at /usr/local/go/src/runtime/proc.go:2076 #4 0x00007f2271b3638e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2477 #5 0x00007f2271b36517 in runtime.park_m (gp=0xc000000600) at /usr/local/go/src/runtime/proc.go:2605 #6 0x00007f2271b58443 in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299 ... #10 0x00007f2271b59d4e in runtime.cgocallback () at /usr/local/go/src/runtime/asm_amd64.s:684 #11 0x00007f2271d663d7 in _cgoexp_be88b77bdbf1__nss_stns_setpwent (a=0x7ffff24a6777, n=0, ctxt=0) at _cgo_gotypes.go:129 #12 0x00007f2271c2ef47 in crosscall2 () at /usr/local/go/src/runtime/cgo/asm_amd64.s:59 #13 0x00007f2271d68046 in _nss_stns_setpwent () at _cgo_export.c:82 Backtrace stopped: previous frame inner to this frame (corrupt stack?) ·ͣ͸$ͷؔ਺͕ίʔϧ͞ΕΔ ΈΜͳେ޷͖gdb
22. GoConference’19 22 (gdb) bt #0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536 #1 0x00007f2271b2c4db in runtime.futexsleep (addr=0xc000032148, val=0, ns=-1) at /usr/local/go/src/ runtime/os_linux.go:46 #2 0x00007f2271b0cb95 in runtime.notesleep (n=0xc000032148) at /usr/local/go/src/runtime/ lock_futex.go:151 #3 0x00007f2271b34ad0 in runtime.stoplockedm () at /usr/local/go/src/runtime/proc.go:2076 #4 0x00007f2271b3638e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2477 #5 0x00007f2271b36517 in runtime.park_m (gp=0xc000000600) at /usr/local/go/src/runtime/proc.go:2605 #6 0x00007f2271b58443 in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299 ... #10 0x00007f2271b59d4e in runtime.cgocallback () at /usr/local/go/src/runtime/asm_amd64.s:684 #11 0x00007f2271d663d7 in _cgoexp_be88b77bdbf1__nss_stns_setpwent (a=0x7ffff24a6777, n=0, ctxt=0) at _cgo_gotypes.go:129 #12 0x00007f2271c2ef47 in crosscall2 () at /usr/local/go/src/runtime/cgo/asm_amd64.s:59 #13 0x00007f2271d68046 in _nss_stns_setpwent () at _cgo_export.c:82 Backtrace stopped: previous frame inner to this frame (corrupt stack?) $ͷίϯςΩετ͔Β(Pͷؔ਺Λίʔϧ ΈΜͳେ޷͖gdb
23. GoConference’19 23 ΈΜͳେ޷͖gdb (gdb) bt #0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536 #1 0x00007f2271b2c4db in runtime.futexsleep (addr=0xc000032148, val=0, ns=-1) at /usr/local/go/src/ runtime/os_linux.go:46 #2 0x00007f2271b0cb95 in runtime.notesleep (n=0xc000032148) at /usr/local/go/src/runtime/ lock_futex.go:151 #3 0x00007f2271b34ad0 in runtime.stoplockedm () at /usr/local/go/src/runtime/proc.go:2076 #4 0x00007f2271b3638e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2477 #5 0x00007f2271b36517 in runtime.park_m (gp=0xc000000600) at /usr/local/go/src/runtime/proc.go:2605 #6 0x00007f2271b58443 in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299 ... #10 0x00007f2271b59d4e in runtime.cgocallback () at /usr/local/go/src/runtime/asm_amd64.s:684 #11 0x00007f2271d663d7 in _cgoexp_be88b77bdbf1__nss_stns_setpwent (a=0x7ffff24a6777, n=0, ctxt=0) at _cgo_gotypes.go:129 #12 0x00007f2271c2ef47 in crosscall2 () at /usr/local/go/src/runtime/cgo/asm_amd64.s:59 #13 0x00007f2271d68046 in _nss_stns_setpwent () at _cgo_export.c:82 Backtrace stopped: previous frame inner to this frame (corrupt stack?) ࣮ߦՄೳͳHPSPVUJOFΛ֬อ͠ɺ࣮ߦ͢Δ
24. GoConference’19 24 (gdb) bt #0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536 #1 0x00007f2271b2c4db in runtime.futexsleep (addr=0xc000032148, val=0, ns=-1) at /usr/local/go/src/ runtime/os_linux.go:46 #2 0x00007f2271b0cb95 in runtime.notesleep (n=0xc000032148) at /usr/local/go/src/runtime/ lock_futex.go:151 #3 0x00007f2271b34ad0 in runtime.stoplockedm () at /usr/local/go/src/runtime/proc.go:2076 #4 0x00007f2271b3638e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2477 #5 0x00007f2271b36517 in runtime.park_m (gp=0xc000000600) at /usr/local/go/src/runtime/proc.go:2605 #6 0x00007f2271b58443 in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299 ... #10 0x00007f2271b59d4e in runtime.cgocallback () at /usr/local/go/src/runtime/asm_amd64.s:684 #11 0x00007f2271d663d7 in _cgoexp_be88b77bdbf1__nss_stns_setpwent (a=0x7ffff24a6777, n=0, ctxt=0) at _cgo_gotypes.go:129 #12 0x00007f2271c2ef47 in crosscall2 () at /usr/local/go/src/runtime/cgo/asm_amd64.s:59 #13 0x00007f2271d68046 in _nss_stns_setpwent () at _cgo_export.c:82 Backtrace stopped: previous frame inner to this frame (corrupt stack?) GVUFYͰ@'65&9@8"*5@13*7"5&ʹͯγάφϧΛ଴͕ͭ
    ಧ͔ͣʹແݶʹ଴ͪଓ͚Δ ΈΜͳେ޷͖gdb
25. GoConference’19 ͬ͘͟Γgoroutine scheduler goroutine 1 goroutine 2 goroutine 3 OS Thread 1 OS Thread 2 OS Thread3 HPSPVUJOF͸εϨου্ʹεέδϡʔϦϯά͞ΕΔ
26. GoConference’19 Ծઆ fork࣌ʹmain thread͚ͩ ίϐʔ͞ΕΔ͔Β͓͔͘͠ͳ͍ͬͯΔʁ 26 IUUQTMJOVYKNPTEOKQIUNM-%1@NBOQBHFTNBOGPSLIUNM
27. GoConference’19 fork? scheduler goroutine 1 goroutine 2 goroutine 3 OS Thread 1 OS Thread 2 OS Thread3 GPSL࣌ʹҰͭͷεϨου͔͠ίϐʔ͞Ε ͳ͍͔Β͏·͘ಈ͍ͯͳ͍ʁ
28. GoConference’19 ൱ఆ:idίϚϯυ΋forkͯ͠Δ 28 root@ubuntu-xenial:~# strace -tt -f id pyama 2>&1 | grep pid |awk -F'] ' '{ print $1}' | sort | uniq [pid 2460 [pid 2461 [pid 2462 [pid 2463 [pid 2464 [pid 2465 [pid 2466 [pid 2467 [pid 2468
29. GoConference’19 ͪΌΜͱॳظԽ͸͞Ε͍ͯΔ͔ʁ 29 func init() { logger, err := syslog.New(syslog.LOG_ERR|syslog.LOG_USER, "stns") if err != nil { // syslog not found fmt.Print(err) } else { log.SetOutput(logger) } log.Println("call init") }
30. GoConference’19 ϑϦʔζ࣌͸init͕࣮ߦ͞Εͳ͍ 30 pyama@ubuntu-xenial:~$ id pyama uid=10301(pyama) gid=2000(pepabo) groups=2000(pepabo) # syslog Jul 10 04:16:18 ubuntu-xenial stns[16278]: 2019/07/10 04:16:18 call init pyama@ubuntu-xenial:~$ ls ~[tab] # syslog ग़ྗͳ͠
31. GoConference’19 _cgo_wait_runtime_init_done() 31 CGO_NO_SANITIZE_THREAD GoInt _nss_stns_getspnam_r(char* p0, struct spwd* p1, char* p2, size_t p3, struct spwd** p4) { __SIZE_TYPE__ _cgo_ctxt = _cgo_wait_runtime_init_done(); ... } IUUQTRJJUBDPNZVHVJJUFNTFEECEB DHPΛ࢖ͬͨ$ͱ(PͷϦϯΫͷཪଆ
    
    -JOVY͔Βݺ͹ΕΔ$ͷؔ਺͸
 (PͷSVOUJNFͷॳظԽ͕׬ྃ͢Δ·Ͱ଴ͭ
32. GoConference’19 _cgo_wait_runtime_init_done() 32 35 _cgo_wait_runtime_init_done(void) { 36 void (*pfn)(struct context_arg*); 37 38 pthread_mutex_lock(&runtime_init_mu); 39 while (runtime_init_done == 0) { (gdb) p runtime_init_done $1 = 1 ϑϦʔζͨ͠ͱ͖ͷϑϥάΛݟΔͱॳظԽࡁΈ
33. GoConference’19 Ͳ͜Ͱϑϥάཱ͕ͭͷ͔ 33 root@ubuntu-xenial:~# gdb id (gdb) set follow-fork-mode child (gdb) b _cgo_wait_runtime_init_done (gdb) run pyama Starting program: /usr/bin/id pyama (gdb) watch runtime_init_done Hardware watchpoint 2: runtime_init_done (gdb) c Continuing. .... Thread 2 "id" hit Hardware watchpoint 2: runtime_init_done Old value = 0 New value = 1 x_cgo_notify_runtime_init_done (dummy=<optimized out>) at gcc_libinit.c:69 69 pthread_cond_broadcast(&runtime_init_cond); ม਺Λ8BUDIͯ͠มߋ͕͋Ε͹ࢭ·Δ Y@DHP@OPUJGZ@SVOUJNF@JOJU@EPOF
    ಺Ͱมߋ͞Ε͍ͯΔ
34. GoConference’19 x_cgo_notify_runtime_init_done͸Ͳ͔͜Βݺ͹ΕΔʁ 34 (gdb) bt #0 x_cgo_notify_runtime_init_done (dummy=<optimized out>) at gcc_libinit.c:69 #1 0x00007ffff61acf58 in runtime.asmcgocall () at /usr/local/go/src/runtime/asm_amd64.s:635 #2 0x00007ffff61a987e in runtime.(*mheap).alloc.func1 () at /usr/local/go/src/runtime/ mheap.go:1048 #3 0x00007ffff61ab763 in runtime.systemstack () at /usr/local/go/src/runtime/asm_amd64.s: 351 #4 0x00007ffff6186140 in ?? () at /usr/local/go/src/runtime/proc.go:1082 from /usr/lib/x86_64- linux-gnu/libnss_stns.so.2 #5 0x00007ffff61ab5fd in runtime.rt0_go () at /usr/local/go/src/runtime/asm_amd64.s:201 #6 0x0000000000000000 in ?? () SVOUJNFSU@HPͷΞηϯϒϥΛḷΔͱɺେݩ͸
    
    @SU@BNE@MJC
    
    @SU@BNE@MJC@HP
    
    SVOUJNFuSU@HP
    ͷΑ͏ʹݺͼग़͞Ε͍ͯΔ
35. GoConference’19 c-sharedͷ৔߹͸runtimeͷॳظԽॲཧ͕ಡΈࠐΈ࣌ͷΈߦΘΕΔ 35 // _rt0_amd64_lib is common startup code for most amd64 systems when // using -buildmode=c-archive or -buildmode=c-shared. The linker will // arrange to invoke this function as a global constructor (for // c-archive) or when the shared library is loaded (for c-shared). // We expect argc and argv to be passed in the usual C ABI registers // DI and SI. TEXT _rt0_amd64_lib(SB),NOSPLIT,$0x50 // Align stack per ELF ABI requirements. MOVQ SP, AX ANDQ $~15, SP HPMBOHHPTSDSVOUJNFBTN@BNET
36. GoConference’19 ൺֱ͢Δͱlibnss-stns͕طʹಡΈࠐ·Ε͍ͯͨ 36 pyama@ubuntu-xenial:~$ lsof -p $$ |grep libnss bash 3810 pyama mem REG 8,1 9163712 57959 /usr/lib/x86_64-linux-gnu/libnss_stns.so bash 3810 pyama mem REG 8,1 47600 2011 /lib/x86_64-linux-gnu/libnss_files-2.23.so bash 3810 pyama mem REG 8,1 47648 2017 /lib/x86_64-linux-gnu/libnss_nis-2.23.so bash 3810 pyama mem REG 8,1 35688 2006 /lib/x86_64-linux-gnu/libnss_compat-2.23.so root@ubuntu-xenial:~# lsof -p $$ |grep libnss bash 2022 root mem REG 8,1 47600 2011 /lib/x86_64-linux-gnu/libnss_files-2.23.so bash 2022 root mem REG 8,1 47648 2017 /lib/x86_64-linux-gnu/libnss_nis-2.23.so bash 2022 root mem REG 8,1 35688 2006 /lib/x86_64-linux-gnu/libnss_compat-2.23.so MPDBM
    VTFS
    TUOT
    VTFS
37. GoConference’19 ࣄ৅ͷ੔ཧ  CBTI͕ىಈ͠ɺMJCOTTTUOTTP͕ಡΈࠐ·Εɺಉ࣌ʹ(PͷSVOUJNF͕ॳظ Խ͞ΕΔ
     MT
    d<UBC>Λ࣮ߦ͢ΔͱɺCBTI͕GPSL͞Εɺىಈͨ͠εϨουͷΈ͕GPSLઌ ͷϓϩηεʹίϐʔ͞ΕΔ
     OFUIUUQύοέʔδͷIUUQ$MJFOU%Pͷ݁Ռ͕ฦ٫͞Εͳ͍ 37
38. GoConference’19 ࣄ৅͔ΒಘΒΕΔ݁࿦ wGPSLޙͷϓϩηεʹ͓͍ͯɺ(PͷSVOUJNF͕ॳظԽ͞Εͳ͍ͨΊɺ GPSLલʹॳظԽ͞Εͨ৘ใͰ(PͷSVOUJNF͕ಈ͔͘Βਖ਼ৗʹಈ࡞͠ ͳ͍ 38
39. GoConference’19 ֶͼɺͦͯ͠·ͱΊ w$(0ΛDTIBSFEͰར༻͢Δࡍʹ͸ɺجຊతʹGPSL͢ΔΑ͏ͳ༻ ్ʹ͸࢖Θͳ͍΄͏͕ྑ͍
    w΋͠ར༻͢ΔͳΒGPSLઌͷϓϩηεͰ໌ࣔతʹॳظԽ͢Δඞཁ ͕͋Δ
    wਓؒ͸ֶͼɺ೥લʹ;ΘͬͱҋύονͰย෇͚ͨ໰୊΋ਂ͘ཧ ղ͠ղܾ͢Δ͜ͱ͕Ͱ͖ΔΑ͏ʹͳΔ 39
40. GoConference’19 ਂֶ͘ͼɺ੒௕Ͱ͖Δ؀ڥ͕200mઌʹ͋Γ·͢ !QC@SFDSVJU 40