linux-middleware-development-by-go.pdf

 linux-middleware-development-by-go.pdf

Go言語を用いたLinuxサーバミドルウェア開発における失敗談を含むノウハウや、ベストプラクティスをトークします。Go言語はLinuxの各種ライブラリとの連携が容易であることや、CGOが利用できることで、従来の言語と比較しLinuxミドルウェアを作りやすい言語です。
一方で、言語として想定していないプロセス、メモリ状態もあり、そういった点に注意を払わないと想定外の挙動をしてしまいます。話し手がLinuxサーバミドルウェア開発を続ける中で実際に遭遇した動作、仕様を実例を元にトークします。

1b838da2065660793d5b26f2cdc32de7?s=128

Kazuhiko Yamashita

July 13, 2019
Tweet

Transcript

  1. 12.

    GoConference’19 12 what is libnss w -JOVYͷ໊લղܾͷ࢓૊ΈͰ͋ΔOTT͔Βಛఆͷ໋໊ن໿ʹج͍ͮͨTPΛಡ ΈࠐΈɺ೚ҙͷํ๏Ͱ໊લղܾΛߦ͏͜ͱ͕Ͱ͖Δ CBTI SPPU!VCVOUVYFOJBMdTUSBDFUUGJEQZBNBcHSFQMJCOTT

    PQFO MJCY@MJOVYHOVMJCOTT@DPNQBUTP 0@3%0/-:c0@$-0&9&$  PQFO MJCY@MJOVYHOVMJCOTT@OJTTP 0@3%0/-:c0@$-0&9&$  PQFO MJCY@MJOVYHOVMJCOTT@pMFTTP 0@3%0/-:c0@$-0&9&$  FUDQBTTXE MJCOTT@pMFTTP
  2. 15.

    GoConference’19 15 libnss MJOVYTFSWFS MJCOTTTUOTTP 45/4 GVODHFU SFTPVSDF@UZQF DPMVNO WBMVFTUSJOH

     BUUSJCVUF6TFS(SPVQT FSSPS \  MPH1SJOU UFTU   SFR FSSSFRVFTU/FX3FRVFTU SFTPVSDF@UZQF DPMVNO WBMVF   MPH1SJOU UFTU   JGFSSOJM\   SFUVSOOJM FSS  ^  SFTPVSDF FSSSFR(FU ˒͕͜͜ࢮ͵  MPH1SJOU UFTU  IUUQSFRVFTU
  3. 17.

    GoConference’19 17 –dominikh https://github.com/golang/go/issues/12734 l5PFYQBOE5IFJTTVFSFBMMZMJFTXJUIVTJOHDHP XIJDIOFUVTFT  JHOPSJOHUIFEFUBJMT 8IFOVTJOHDHP UIF(PEFBEMPDLEFUFDUJPO

    DBOOPUGVODUJPOQSPQFSMZ CFDBVTF$XPSMENJHIUDBMM(PGVODUJPOTBU BOZUJNF TPJOUIFPSZOPEFBEMPDLFYJTUTXFNJHIUKVTUCFXBJUJOHGPS BOFYUFSOBMGVODUJPODBMMJOEFpOJUFMZz
  4. 21.

    GoConference’19 21 (gdb) bt #0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536 #1

    0x00007f2271b2c4db in runtime.futexsleep (addr=0xc000032148, val=0, ns=-1) at /usr/local/go/src/ runtime/os_linux.go:46 #2 0x00007f2271b0cb95 in runtime.notesleep (n=0xc000032148) at /usr/local/go/src/runtime/ lock_futex.go:151 #3 0x00007f2271b34ad0 in runtime.stoplockedm () at /usr/local/go/src/runtime/proc.go:2076 #4 0x00007f2271b3638e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2477 #5 0x00007f2271b36517 in runtime.park_m (gp=0xc000000600) at /usr/local/go/src/runtime/proc.go:2605 #6 0x00007f2271b58443 in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299 ... #10 0x00007f2271b59d4e in runtime.cgocallback () at /usr/local/go/src/runtime/asm_amd64.s:684 #11 0x00007f2271d663d7 in _cgoexp_be88b77bdbf1__nss_stns_setpwent (a=0x7ffff24a6777, n=0, ctxt=0) at _cgo_gotypes.go:129 #12 0x00007f2271c2ef47 in crosscall2 () at /usr/local/go/src/runtime/cgo/asm_amd64.s:59 #13 0x00007f2271d68046 in _nss_stns_setpwent () at _cgo_export.c:82 Backtrace stopped: previous frame inner to this frame (corrupt stack?) ·ͣ͸$ͷؔ਺͕ίʔϧ͞ΕΔ ΈΜͳେ޷͖gdb
  5. 22.

    GoConference’19 22 (gdb) bt #0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536 #1

    0x00007f2271b2c4db in runtime.futexsleep (addr=0xc000032148, val=0, ns=-1) at /usr/local/go/src/ runtime/os_linux.go:46 #2 0x00007f2271b0cb95 in runtime.notesleep (n=0xc000032148) at /usr/local/go/src/runtime/ lock_futex.go:151 #3 0x00007f2271b34ad0 in runtime.stoplockedm () at /usr/local/go/src/runtime/proc.go:2076 #4 0x00007f2271b3638e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2477 #5 0x00007f2271b36517 in runtime.park_m (gp=0xc000000600) at /usr/local/go/src/runtime/proc.go:2605 #6 0x00007f2271b58443 in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299 ... #10 0x00007f2271b59d4e in runtime.cgocallback () at /usr/local/go/src/runtime/asm_amd64.s:684 #11 0x00007f2271d663d7 in _cgoexp_be88b77bdbf1__nss_stns_setpwent (a=0x7ffff24a6777, n=0, ctxt=0) at _cgo_gotypes.go:129 #12 0x00007f2271c2ef47 in crosscall2 () at /usr/local/go/src/runtime/cgo/asm_amd64.s:59 #13 0x00007f2271d68046 in _nss_stns_setpwent () at _cgo_export.c:82 Backtrace stopped: previous frame inner to this frame (corrupt stack?) $ͷίϯςΩετ͔Β(Pͷؔ਺Λίʔϧ ΈΜͳେ޷͖gdb
  6. 23.

    GoConference’19 23 ΈΜͳେ޷͖gdb (gdb) bt #0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536

    #1 0x00007f2271b2c4db in runtime.futexsleep (addr=0xc000032148, val=0, ns=-1) at /usr/local/go/src/ runtime/os_linux.go:46 #2 0x00007f2271b0cb95 in runtime.notesleep (n=0xc000032148) at /usr/local/go/src/runtime/ lock_futex.go:151 #3 0x00007f2271b34ad0 in runtime.stoplockedm () at /usr/local/go/src/runtime/proc.go:2076 #4 0x00007f2271b3638e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2477 #5 0x00007f2271b36517 in runtime.park_m (gp=0xc000000600) at /usr/local/go/src/runtime/proc.go:2605 #6 0x00007f2271b58443 in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299 ... #10 0x00007f2271b59d4e in runtime.cgocallback () at /usr/local/go/src/runtime/asm_amd64.s:684 #11 0x00007f2271d663d7 in _cgoexp_be88b77bdbf1__nss_stns_setpwent (a=0x7ffff24a6777, n=0, ctxt=0) at _cgo_gotypes.go:129 #12 0x00007f2271c2ef47 in crosscall2 () at /usr/local/go/src/runtime/cgo/asm_amd64.s:59 #13 0x00007f2271d68046 in _nss_stns_setpwent () at _cgo_export.c:82 Backtrace stopped: previous frame inner to this frame (corrupt stack?) ࣮ߦՄೳͳHPSPVUJOFΛ֬อ͠ɺ࣮ߦ͢Δ
  7. 24.

    GoConference’19 24 (gdb) bt #0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536 #1

    0x00007f2271b2c4db in runtime.futexsleep (addr=0xc000032148, val=0, ns=-1) at /usr/local/go/src/ runtime/os_linux.go:46 #2 0x00007f2271b0cb95 in runtime.notesleep (n=0xc000032148) at /usr/local/go/src/runtime/ lock_futex.go:151 #3 0x00007f2271b34ad0 in runtime.stoplockedm () at /usr/local/go/src/runtime/proc.go:2076 #4 0x00007f2271b3638e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2477 #5 0x00007f2271b36517 in runtime.park_m (gp=0xc000000600) at /usr/local/go/src/runtime/proc.go:2605 #6 0x00007f2271b58443 in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299 ... #10 0x00007f2271b59d4e in runtime.cgocallback () at /usr/local/go/src/runtime/asm_amd64.s:684 #11 0x00007f2271d663d7 in _cgoexp_be88b77bdbf1__nss_stns_setpwent (a=0x7ffff24a6777, n=0, ctxt=0) at _cgo_gotypes.go:129 #12 0x00007f2271c2ef47 in crosscall2 () at /usr/local/go/src/runtime/cgo/asm_amd64.s:59 #13 0x00007f2271d68046 in _nss_stns_setpwent () at _cgo_export.c:82 Backtrace stopped: previous frame inner to this frame (corrupt stack?) GVUFYͰ@'65&9@8"*5@13*7"5&ʹͯγάφϧΛ଴͕ͭ ಧ͔ͣʹແݶʹ଴ͪଓ͚Δ ΈΜͳେ޷͖gdb
  8. 25.

    GoConference’19 ͬ͘͟Γgoroutine scheduler goroutine 1 goroutine 2 goroutine 3 OS

    Thread 1 OS Thread 2 OS Thread3 HPSPVUJOF͸εϨου্ʹεέδϡʔϦϯά͞ΕΔ
  9. 27.

    GoConference’19 fork? scheduler goroutine 1 goroutine 2 goroutine 3 OS

    Thread 1 OS Thread 2 OS Thread3 GPSL࣌ʹҰͭͷεϨου͔͠ίϐʔ͞Ε ͳ͍͔Β͏·͘ಈ͍ͯͳ͍ʁ
  10. 28.

    GoConference’19 ൱ఆ:idίϚϯυ΋forkͯ͠Δ 28 root@ubuntu-xenial:~# strace -tt -f id pyama 2>&1

    | grep pid |awk -F'] ' '{ print $1}' | sort | uniq [pid 2460 [pid 2461 [pid 2462 [pid 2463 [pid 2464 [pid 2465 [pid 2466 [pid 2467 [pid 2468
  11. 29.

    GoConference’19 ͪΌΜͱॳظԽ͸͞Ε͍ͯΔ͔ʁ 29 func init() { logger, err := syslog.New(syslog.LOG_ERR|syslog.LOG_USER,

    "stns") if err != nil { // syslog not found fmt.Print(err) } else { log.SetOutput(logger) } log.Println("call init") }
  12. 30.

    GoConference’19 ϑϦʔζ࣌͸init͕࣮ߦ͞Εͳ͍ 30 pyama@ubuntu-xenial:~$ id pyama uid=10301(pyama) gid=2000(pepabo) groups=2000(pepabo) #

    syslog Jul 10 04:16:18 ubuntu-xenial stns[16278]: 2019/07/10 04:16:18 call init pyama@ubuntu-xenial:~$ ls ~[tab] # syslog ग़ྗͳ͠
  13. 31.

    GoConference’19 _cgo_wait_runtime_init_done() 31 CGO_NO_SANITIZE_THREAD GoInt _nss_stns_getspnam_r(char* p0, struct spwd* p1,

    char* p2, size_t p3, struct spwd** p4) { __SIZE_TYPE__ _cgo_ctxt = _cgo_wait_runtime_init_done(); ... } IUUQTRJJUBDPNZVHVJJUFNTFEECEB DHPΛ࢖ͬͨ$ͱ(PͷϦϯΫͷཪଆ   -JOVY͔Βݺ͹ΕΔ$ͷؔ਺͸
 (PͷSVOUJNFͷॳظԽ͕׬ྃ͢Δ·Ͱ଴ͭ
  14. 32.

    GoConference’19 _cgo_wait_runtime_init_done() 32 35 _cgo_wait_runtime_init_done(void) { 36 void (*pfn)(struct context_arg*);

    37 38 pthread_mutex_lock(&runtime_init_mu); 39 while (runtime_init_done == 0) { (gdb) p runtime_init_done $1 = 1 ϑϦʔζͨ͠ͱ͖ͷϑϥάΛݟΔͱॳظԽࡁΈ
  15. 33.

    GoConference’19 Ͳ͜Ͱϑϥάཱ͕ͭͷ͔ 33 root@ubuntu-xenial:~# gdb id (gdb) set follow-fork-mode child

    (gdb) b _cgo_wait_runtime_init_done (gdb) run pyama Starting program: /usr/bin/id pyama (gdb) watch runtime_init_done Hardware watchpoint 2: runtime_init_done (gdb) c Continuing. .... Thread 2 "id" hit Hardware watchpoint 2: runtime_init_done Old value = 0 New value = 1 x_cgo_notify_runtime_init_done (dummy=<optimized out>) at gcc_libinit.c:69 69 pthread_cond_broadcast(&runtime_init_cond); ม਺Λ8BUDIͯ͠มߋ͕͋Ε͹ࢭ·Δ Y@DHP@OPUJGZ@SVOUJNF@JOJU@EPOF ಺Ͱมߋ͞Ε͍ͯΔ
  16. 34.

    GoConference’19 x_cgo_notify_runtime_init_done͸Ͳ͔͜Βݺ͹ΕΔʁ 34 (gdb) bt #0 x_cgo_notify_runtime_init_done (dummy=<optimized out>) at

    gcc_libinit.c:69 #1 0x00007ffff61acf58 in runtime.asmcgocall () at /usr/local/go/src/runtime/asm_amd64.s:635 #2 0x00007ffff61a987e in runtime.(*mheap).alloc.func1 () at /usr/local/go/src/runtime/ mheap.go:1048 #3 0x00007ffff61ab763 in runtime.systemstack () at /usr/local/go/src/runtime/asm_amd64.s: 351 #4 0x00007ffff6186140 in ?? () at /usr/local/go/src/runtime/proc.go:1082 from /usr/lib/x86_64- linux-gnu/libnss_stns.so.2 #5 0x00007ffff61ab5fd in runtime.rt0_go () at /usr/local/go/src/runtime/asm_amd64.s:201 #6 0x0000000000000000 in ?? () SVOUJNFSU@HPͷΞηϯϒϥΛḷΔͱɺେݩ͸ @SU@BNE@MJC@SU@BNE@MJC@HPSVOUJNFuSU@HP ͷΑ͏ʹݺͼग़͞Ε͍ͯΔ
  17. 35.

    GoConference’19 c-sharedͷ৔߹͸runtimeͷॳظԽॲཧ͕ಡΈࠐΈ࣌ͷΈߦΘΕΔ 35 // _rt0_amd64_lib is common startup code for

    most amd64 systems when // using -buildmode=c-archive or -buildmode=c-shared. The linker will // arrange to invoke this function as a global constructor (for // c-archive) or when the shared library is loaded (for c-shared). // We expect argc and argv to be passed in the usual C ABI registers // DI and SI. TEXT _rt0_amd64_lib(SB),NOSPLIT,$0x50 // Align stack per ELF ABI requirements. MOVQ SP, AX ANDQ $~15, SP HPMBOHHPTSDSVOUJNFBTN@BNET
  18. 36.

    GoConference’19 ൺֱ͢Δͱlibnss-stns͕طʹಡΈࠐ·Ε͍ͯͨ 36 pyama@ubuntu-xenial:~$ lsof -p $$ |grep libnss bash

    3810 pyama mem REG 8,1 9163712 57959 /usr/lib/x86_64-linux-gnu/libnss_stns.so bash 3810 pyama mem REG 8,1 47600 2011 /lib/x86_64-linux-gnu/libnss_files-2.23.so bash 3810 pyama mem REG 8,1 47648 2017 /lib/x86_64-linux-gnu/libnss_nis-2.23.so bash 3810 pyama mem REG 8,1 35688 2006 /lib/x86_64-linux-gnu/libnss_compat-2.23.so root@ubuntu-xenial:~# lsof -p $$ |grep libnss bash 2022 root mem REG 8,1 47600 2011 /lib/x86_64-linux-gnu/libnss_files-2.23.so bash 2022 root mem REG 8,1 47648 2017 /lib/x86_64-linux-gnu/libnss_nis-2.23.so bash 2022 root mem REG 8,1 35688 2006 /lib/x86_64-linux-gnu/libnss_compat-2.23.so MPDBMVTFS TUOTVTFS