Upgrade to Pro — share decks privately, control downloads, hide ads and more …

linux-middleware-development-by-go.pdf

 linux-middleware-development-by-go.pdf

Go言語を用いたLinuxサーバミドルウェア開発における失敗談を含むノウハウや、ベストプラクティスをトークします。Go言語はLinuxの各種ライブラリとの連携が容易であることや、CGOが利用できることで、従来の言語と比較しLinuxミドルウェアを作りやすい言語です。
一方で、言語として想定していないプロセス、メモリ状態もあり、そういった点に注意を払わないと想定外の挙動をしてしまいます。話し手がLinuxサーバミドルウェア開発を続ける中で実際に遭遇した動作、仕様を実例を元にトークします。

Kazuhiko Yamashita

July 13, 2019
Tweet

More Decks by Kazuhiko Yamashita

Other Decks in Programming

Transcript

  1. GoConference’19
    Linux middleware
    development by Go
    ʙݰքṗͰͱΕͨଠ౛ڕʹക೑ιʔεΛఴ͑ͯʙ

    View Slide

  2. GoConference’19
    ࢁԼ ࿨඙ @pyama86
    GMOϖύϘ ٕज़ج൫νʔϜ
    γχΞɾϓϦϯγύϧ
    ten-snapon.com
    pyama.fun
    stns.jp
    2

    View Slide

  3. GoConference’19
    झຯ: ࠗ׆
    3

    View Slide

  4. GoConference’19
    4
    ϗεςΟϯάࣄۀ &$ࢧԉࣄۀ ϋϯυϝΠυɾͦͷଞࣄۀ

    View Slide

  5. 45/4
    -JOVY/444FSWFS
    TUOTKQ

    View Slide

  6. GoConference’19
    Linux middleware
    development by Go
    6

    View Slide

  7. GoConference’19
    ͜ͷηογϣϯͰ࿩͢͜ͱ
    w࿩͠ख͕ૺ۰ͨ͠ϛυϧ΢ΣΞ։ൃ࣌ͷτϥϒϧʹ͍ͭͯ
    wͦΕΒΛͲ͏ରԠ͔ͨ͠
    wਓ͸੒௕͢Δ
    7

    View Slide

  8. GoConference’19
    Go͸ϛυϧ΢ΣΞ։ൃͱ૬ੑ͕ྑ͍
    w(P͸γϯά
    ϧόΠφϦͰ؆୯ʹ഑෍͕Մೳ
    w$(0Λར༻ͯ͠ɺطଘͷ$ݴޠͷࢿ࢈Λར༻͢Δ͜ͱ͕Ͱ͖Δ
    wγεςϜϓϩάϥϛϯάʹඞཁͳػೳ͕΄΅Ұ௨Γଗ͍ͬͯΔ
    wHPSPVUJOFΛར༻͢Δͱ؆୯ʹϚϧνεϨουϓϩάϥϛϯά͕Մೳ
    8

    View Slide

  9. GoConference’19
    9
    STNS
    MJOVYTFSWFS
    MJCOTTTUOTTP 45/4
    SPPU!VCVOUVYFOJBMdJEQZBNB
    VJE QZBNB
    HJE QFQBCP

    IUUQSFRVFTU
    OFUIUUQ
    #BDLFOET
    50.-4FUDE
    %ZOBNP%#
    $(0

    View Slide

  10. GoConference’19
    10
    libnss-stns.so
    MJCOTTTUOTTPʹ͓͍ͯɺλʔϛφϧͰλϒิ׬͠
    Α͏ͱ͢ΔͱϑϦʔζ͢Δ
    MTd

    View Slide

  11. GoConference’19
    11

    View Slide

  12. GoConference’19
    12
    what is libnss
    w -JOVYͷ໊લղܾͷ࢓૊ΈͰ͋ΔOTT͔Βಛఆͷ໋໊ن໿ʹج͍ͮͨTPΛಡ
    ΈࠐΈɺ೚ҙͷํ๏Ͱ໊લղܾΛߦ͏͜ͱ͕Ͱ͖Δ
    CBTI
    SPPU!VCVOUVYFOJBMdTUSBDFUUGJEQZBNBcHSFQMJCOTT
    PQFO MJCY@MJOVYHOVMJCOTT@DPNQBUTP 0@3%0/-:c0@$-0&9&$

    PQFO MJCY@MJOVYHOVMJCOTT@OJTTP 0@3%0/-:c0@$-0&9&$

    PQFO MJCY@MJOVYHOVMJCOTT@pMFTTP 0@3%0/-:c0@$-0&9&$

    FUDQBTTXE
    MJCOTT@pMFTTP

    View Slide

  13. GoConference’19
    13
    libnss-stns 2016
    w (P͔Βఏڙ͞ΕͨCVJMENPEFDTIBSFEͰTP TIBSFEPCKFDU
    Λ࡞੒
    SPPU!VCVOUVYFOJBMd(0"3$)BNEHPCVJMEYPCJOBSZMJCOTTTUOTTPa
    CVJMENPEFDTIBSFENBJOHPSFTPVSDFHPQBTTXEHPTIBEPXHPHSPVQHP

    View Slide

  14. GoConference’19
    14
    ౰࣌ͷௐࠪ
    ิ׬લ
    SPPU!VCVOUVYFOJBMdQTFGcHSFQ?ZBNB
    QZBNBQUTTV
    ิ׬࣮ߦத
    SPPU!VCVOUVYFOJBMdQTFGcHSFQ?ZBNB
    QZBNBQUTTV
    QZBNBQUTTV
    -JOVYͰλʔϛφϧิ׬Λར༻͢Δͱ͖ɺ
    എޙͰ͸GPSL͕࣮ߦ͞Ε͍ͯΔ

    View Slide

  15. GoConference’19
    15
    libnss
    MJOVYTFSWFS
    MJCOTTTUOTTP 45/4
    GVODHFU SFTPVSDF@UZQF DPMVNO WBMVFTUSJOH
    BUUSJCVUF6TFS(SPVQT FSSPS
    \
    MPH1SJOU UFTU

    SFR FSSSFRVFTU/FX3FRVFTU SFTPVSDF@UZQF DPMVNO WBMVF

    MPH1SJOU UFTU

    JGFSSOJM\
    SFUVSOOJM FSS
    ^
    SFTPVSDF FSSSFR(FU
    ˒͕͜͜ࢮ͵
    MPH1SJOU UFTU

    IUUQSFRVFTU

    View Slide

  16. GoConference’19
    16
    (PͰ͸σʔϞϯ͕ੵۃతʹαϙʔτ͞Ε͍ͯͳ͍ͷͰ
    ͳΜ͔౿ΜͰΔͱࢥ͍ͬͯͨ

    View Slide

  17. GoConference’19
    17
    –dominikh
    https://github.com/golang/go/issues/12734
    l5PFYQBOE5IFJTTVFSFBMMZMJFTXJUIVTJOHDHP XIJDIOFUVTFT
    JHOPSJOHUIFEFUBJMT
    8IFOVTJOHDHP UIF(PEFBEMPDLEFUFDUJPO
    DBOOPUGVODUJPOQSPQFSMZ CFDBVTF$XPSMENJHIUDBMM(PGVODUJPOTBU
    BOZUJNF TPJOUIFPSZOPEFBEMPDLFYJTUTXFNJHIUKVTUCFXBJUJOHGPS
    BOFYUFSOBMGVODUJPODBMMJOEFpOJUFMZz

    View Slide

  18. GoConference’19
    ౰࣌ͷԶ͸·ΔͰ
    Կ΋Θ͔͍ͬͯͳ͍
    18

    View Slide

  19. GoConference’19
    19
    rIUUQTHJUIVCDPN45/4MJCOTT@TUOTQVMM

    View Slide

  20. GoConference’19
    20
    Three years have passed since then.

    View Slide

  21. GoConference’19
    21
    (gdb) bt
    #0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
    #1 0x00007f2271b2c4db in runtime.futexsleep (addr=0xc000032148, val=0, ns=-1) at /usr/local/go/src/
    runtime/os_linux.go:46
    #2 0x00007f2271b0cb95 in runtime.notesleep (n=0xc000032148) at /usr/local/go/src/runtime/
    lock_futex.go:151
    #3 0x00007f2271b34ad0 in runtime.stoplockedm () at /usr/local/go/src/runtime/proc.go:2076
    #4 0x00007f2271b3638e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2477
    #5 0x00007f2271b36517 in runtime.park_m (gp=0xc000000600) at /usr/local/go/src/runtime/proc.go:2605
    #6 0x00007f2271b58443 in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299
    ...
    #10 0x00007f2271b59d4e in runtime.cgocallback () at /usr/local/go/src/runtime/asm_amd64.s:684
    #11 0x00007f2271d663d7 in _cgoexp_be88b77bdbf1__nss_stns_setpwent (a=0x7ffff24a6777, n=0, ctxt=0)
    at _cgo_gotypes.go:129
    #12 0x00007f2271c2ef47 in crosscall2 () at /usr/local/go/src/runtime/cgo/asm_amd64.s:59
    #13 0x00007f2271d68046 in _nss_stns_setpwent () at _cgo_export.c:82
    Backtrace stopped: previous frame inner to this frame (corrupt stack?)
    ·ͣ͸$ͷؔ਺͕ίʔϧ͞ΕΔ
    ΈΜͳେ޷͖gdb

    View Slide

  22. GoConference’19
    22
    (gdb) bt
    #0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
    #1 0x00007f2271b2c4db in runtime.futexsleep (addr=0xc000032148, val=0, ns=-1) at /usr/local/go/src/
    runtime/os_linux.go:46
    #2 0x00007f2271b0cb95 in runtime.notesleep (n=0xc000032148) at /usr/local/go/src/runtime/
    lock_futex.go:151
    #3 0x00007f2271b34ad0 in runtime.stoplockedm () at /usr/local/go/src/runtime/proc.go:2076
    #4 0x00007f2271b3638e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2477
    #5 0x00007f2271b36517 in runtime.park_m (gp=0xc000000600) at /usr/local/go/src/runtime/proc.go:2605
    #6 0x00007f2271b58443 in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299
    ...
    #10 0x00007f2271b59d4e in runtime.cgocallback () at /usr/local/go/src/runtime/asm_amd64.s:684
    #11 0x00007f2271d663d7 in _cgoexp_be88b77bdbf1__nss_stns_setpwent (a=0x7ffff24a6777, n=0,
    ctxt=0) at _cgo_gotypes.go:129
    #12 0x00007f2271c2ef47 in crosscall2 () at /usr/local/go/src/runtime/cgo/asm_amd64.s:59
    #13 0x00007f2271d68046 in _nss_stns_setpwent () at _cgo_export.c:82
    Backtrace stopped: previous frame inner to this frame (corrupt stack?)
    $ͷίϯςΩετ͔Β(Pͷؔ਺Λίʔϧ
    ΈΜͳେ޷͖gdb

    View Slide

  23. GoConference’19
    23
    ΈΜͳେ޷͖gdb
    (gdb) bt
    #0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
    #1 0x00007f2271b2c4db in runtime.futexsleep (addr=0xc000032148, val=0, ns=-1) at /usr/local/go/src/
    runtime/os_linux.go:46
    #2 0x00007f2271b0cb95 in runtime.notesleep (n=0xc000032148) at /usr/local/go/src/runtime/
    lock_futex.go:151
    #3 0x00007f2271b34ad0 in runtime.stoplockedm () at /usr/local/go/src/runtime/proc.go:2076
    #4 0x00007f2271b3638e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2477
    #5 0x00007f2271b36517 in runtime.park_m (gp=0xc000000600) at /usr/local/go/src/runtime/proc.go:2605
    #6 0x00007f2271b58443 in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299
    ...
    #10 0x00007f2271b59d4e in runtime.cgocallback () at /usr/local/go/src/runtime/asm_amd64.s:684
    #11 0x00007f2271d663d7 in _cgoexp_be88b77bdbf1__nss_stns_setpwent (a=0x7ffff24a6777, n=0, ctxt=0)
    at _cgo_gotypes.go:129
    #12 0x00007f2271c2ef47 in crosscall2 () at /usr/local/go/src/runtime/cgo/asm_amd64.s:59
    #13 0x00007f2271d68046 in _nss_stns_setpwent () at _cgo_export.c:82
    Backtrace stopped: previous frame inner to this frame (corrupt stack?)
    ࣮ߦՄೳͳHPSPVUJOFΛ֬อ͠ɺ࣮ߦ͢Δ

    View Slide

  24. GoConference’19
    24
    (gdb) bt
    #0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
    #1 0x00007f2271b2c4db in runtime.futexsleep (addr=0xc000032148, val=0, ns=-1) at /usr/local/go/src/
    runtime/os_linux.go:46
    #2 0x00007f2271b0cb95 in runtime.notesleep (n=0xc000032148) at /usr/local/go/src/runtime/
    lock_futex.go:151
    #3 0x00007f2271b34ad0 in runtime.stoplockedm () at /usr/local/go/src/runtime/proc.go:2076
    #4 0x00007f2271b3638e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2477
    #5 0x00007f2271b36517 in runtime.park_m (gp=0xc000000600) at /usr/local/go/src/runtime/proc.go:2605
    #6 0x00007f2271b58443 in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299
    ...
    #10 0x00007f2271b59d4e in runtime.cgocallback () at /usr/local/go/src/runtime/asm_amd64.s:684
    #11 0x00007f2271d663d7 in _cgoexp_be88b77bdbf1__nss_stns_setpwent (a=0x7ffff24a6777, n=0, ctxt=0)
    at _cgo_gotypes.go:129
    #12 0x00007f2271c2ef47 in crosscall2 () at /usr/local/go/src/runtime/cgo/asm_amd64.s:59
    #13 0x00007f2271d68046 in _nss_stns_setpwent () at _cgo_export.c:82
    Backtrace stopped: previous frame inner to this frame (corrupt stack?)
    GVUFYͰ@'65&9@8"*5@13*7"5&ʹͯγάφϧΛ଴͕ͭ
    ಧ͔ͣʹແݶʹ଴ͪଓ͚Δ
    ΈΜͳେ޷͖gdb

    View Slide

  25. GoConference’19
    ͬ͘͟Γgoroutine
    scheduler
    goroutine 1
    goroutine 2
    goroutine 3
    OS Thread 1
    OS Thread 2
    OS Thread3
    HPSPVUJOF͸εϨου্ʹεέδϡʔϦϯά͞ΕΔ

    View Slide

  26. GoConference’19
    Ծઆ
    fork࣌ʹmain thread͚ͩ
    ίϐʔ͞ΕΔ͔Β͓͔͘͠ͳ͍ͬͯΔʁ
    26
    IUUQTMJOVYKNPTEOKQIUNM-%1@NBOQBHFTNBOGPSLIUNM

    View Slide

  27. GoConference’19
    fork?
    scheduler
    goroutine 1
    goroutine 2
    goroutine 3
    OS Thread 1
    OS Thread 2
    OS Thread3
    GPSL࣌ʹҰͭͷεϨου͔͠ίϐʔ͞Ε
    ͳ͍͔Β͏·͘ಈ͍ͯͳ͍ʁ

    View Slide

  28. GoConference’19
    ൱ఆ:idίϚϯυ΋forkͯ͠Δ
    28
    root@ubuntu-xenial:~# strace -tt -f id pyama 2>&1 | grep pid |awk -F'] ' '{ print $1}' | sort | uniq
    [pid 2460
    [pid 2461
    [pid 2462
    [pid 2463
    [pid 2464
    [pid 2465
    [pid 2466
    [pid 2467
    [pid 2468

    View Slide

  29. GoConference’19
    ͪΌΜͱॳظԽ͸͞Ε͍ͯΔ͔ʁ
    29
    func init() {
    logger, err := syslog.New(syslog.LOG_ERR|syslog.LOG_USER, "stns")
    if err != nil {
    // syslog not found
    fmt.Print(err)
    } else {
    log.SetOutput(logger)
    }
    log.Println("call init")
    }

    View Slide

  30. GoConference’19
    ϑϦʔζ࣌͸init͕࣮ߦ͞Εͳ͍
    30
    pyama@ubuntu-xenial:~$ id pyama
    uid=10301(pyama) gid=2000(pepabo) groups=2000(pepabo)
    # syslog
    Jul 10 04:16:18 ubuntu-xenial stns[16278]: 2019/07/10 04:16:18 call init
    pyama@ubuntu-xenial:~$ ls ~[tab]
    # syslog
    ग़ྗͳ͠

    View Slide

  31. GoConference’19
    _cgo_wait_runtime_init_done()
    31
    CGO_NO_SANITIZE_THREAD GoInt _nss_stns_getspnam_r(char* p0, struct spwd* p1, char*
    p2, size_t p3, struct spwd** p4)
    {
    __SIZE_TYPE__ _cgo_ctxt = _cgo_wait_runtime_init_done();
    ...
    }
    IUUQTRJJUBDPNZVHVJJUFNTFEECEB
    DHPΛ࢖ͬͨ$ͱ(PͷϦϯΫͷཪଆ

    -JOVY͔Βݺ͹ΕΔ$ͷؔ਺͸

    (PͷSVOUJNFͷॳظԽ͕׬ྃ͢Δ·Ͱ଴ͭ

    View Slide

  32. GoConference’19
    _cgo_wait_runtime_init_done()
    32
    35 _cgo_wait_runtime_init_done(void) {
    36 void (*pfn)(struct context_arg*);
    37
    38 pthread_mutex_lock(&runtime_init_mu);
    39 while (runtime_init_done == 0) {
    (gdb) p runtime_init_done
    $1 = 1
    ϑϦʔζͨ͠ͱ͖ͷϑϥάΛݟΔͱॳظԽࡁΈ

    View Slide

  33. GoConference’19
    Ͳ͜Ͱϑϥάཱ͕ͭͷ͔
    33
    root@ubuntu-xenial:~# gdb id
    (gdb) set follow-fork-mode child
    (gdb) b _cgo_wait_runtime_init_done
    (gdb) run pyama
    Starting program: /usr/bin/id pyama
    (gdb) watch runtime_init_done
    Hardware watchpoint 2: runtime_init_done
    (gdb) c
    Continuing.
    ....
    Thread 2 "id" hit Hardware watchpoint 2: runtime_init_done
    Old value = 0
    New value = 1
    x_cgo_notify_runtime_init_done (dummy=) at gcc_libinit.c:69
    69 pthread_cond_broadcast(&runtime_init_cond);
    ม਺Λ8BUDIͯ͠มߋ͕͋Ε͹ࢭ·Δ
    Y@DHP@OPUJGZ@SVOUJNF@JOJU@EPOF
    ಺Ͱมߋ͞Ε͍ͯΔ

    View Slide

  34. GoConference’19
    x_cgo_notify_runtime_init_done͸Ͳ͔͜Βݺ͹ΕΔʁ
    34
    (gdb) bt
    #0 x_cgo_notify_runtime_init_done (dummy=) at gcc_libinit.c:69
    #1 0x00007ffff61acf58 in runtime.asmcgocall () at /usr/local/go/src/runtime/asm_amd64.s:635
    #2 0x00007ffff61a987e in runtime.(*mheap).alloc.func1 () at /usr/local/go/src/runtime/
    mheap.go:1048
    #3 0x00007ffff61ab763 in runtime.systemstack () at /usr/local/go/src/runtime/asm_amd64.s:
    351
    #4 0x00007ffff6186140 in ?? () at /usr/local/go/src/runtime/proc.go:1082 from /usr/lib/x86_64-
    linux-gnu/libnss_stns.so.2
    #5 0x00007ffff61ab5fd in runtime.rt0_go () at /usr/local/go/src/runtime/asm_amd64.s:201
    #6 0x0000000000000000 in ?? ()
    SVOUJNFSU@HPͷΞηϯϒϥΛḷΔͱɺେݩ͸
    @SU@BNE@MJC@SU@BNE@MJC@HPSVOUJNFuSU@HP
    ͷΑ͏ʹݺͼग़͞Ε͍ͯΔ

    View Slide

  35. GoConference’19
    c-sharedͷ৔߹͸runtimeͷॳظԽॲཧ͕ಡΈࠐΈ࣌ͷΈߦΘΕΔ
    35
    // _rt0_amd64_lib is common startup code for most amd64 systems when
    // using -buildmode=c-archive or -buildmode=c-shared. The linker will
    // arrange to invoke this function as a global constructor (for
    // c-archive) or when the shared library is loaded (for c-shared).
    // We expect argc and argv to be passed in the usual C ABI registers
    // DI and SI.
    TEXT _rt0_amd64_lib(SB),NOSPLIT,$0x50
    // Align stack per ELF ABI requirements.
    MOVQ SP, AX
    ANDQ $~15, SP
    HPMBOHHPTSDSVOUJNFBTN@BNET

    View Slide

  36. GoConference’19
    ൺֱ͢Δͱlibnss-stns͕طʹಡΈࠐ·Ε͍ͯͨ
    36
    pyama@ubuntu-xenial:~$ lsof -p $$ |grep libnss
    bash 3810 pyama mem REG 8,1 9163712 57959 /usr/lib/x86_64-linux-gnu/libnss_stns.so
    bash 3810 pyama mem REG 8,1 47600 2011 /lib/x86_64-linux-gnu/libnss_files-2.23.so
    bash 3810 pyama mem REG 8,1 47648 2017 /lib/x86_64-linux-gnu/libnss_nis-2.23.so
    bash 3810 pyama mem REG 8,1 35688 2006 /lib/x86_64-linux-gnu/libnss_compat-2.23.so
    root@ubuntu-xenial:~# lsof -p $$ |grep libnss
    bash 2022 root mem REG 8,1 47600 2011 /lib/x86_64-linux-gnu/libnss_files-2.23.so
    bash 2022 root mem REG 8,1 47648 2017 /lib/x86_64-linux-gnu/libnss_nis-2.23.so
    bash 2022 root mem REG 8,1 35688 2006 /lib/x86_64-linux-gnu/libnss_compat-2.23.so
    MPDBMVTFS
    TUOTVTFS

    View Slide

  37. GoConference’19
    ࣄ৅ͷ੔ཧ
    CBTI͕ىಈ͠ɺMJCOTTTUOTTP͕ಡΈࠐ·Εɺಉ࣌ʹ(PͷSVOUJNF͕ॳظ
    Խ͞ΕΔ
    MTdΛ࣮ߦ͢ΔͱɺCBTI͕GPSL͞Εɺىಈͨ͠εϨουͷΈ͕GPSLઌ
    ͷϓϩηεʹίϐʔ͞ΕΔ
    OFUIUUQύοέʔδͷIUUQ$MJFOU%Pͷ݁Ռ͕ฦ٫͞Εͳ͍
    37

    View Slide

  38. GoConference’19
    ࣄ৅͔ΒಘΒΕΔ݁࿦
    wGPSLޙͷϓϩηεʹ͓͍ͯɺ(PͷSVOUJNF͕ॳظԽ͞Εͳ͍ͨΊɺ
    GPSLલʹॳظԽ͞Εͨ৘ใͰ(PͷSVOUJNF͕ಈ͔͘Βਖ਼ৗʹಈ࡞͠
    ͳ͍
    38

    View Slide

  39. GoConference’19
    ֶͼɺͦͯ͠·ͱΊ
    w$(0ΛDTIBSFEͰར༻͢Δࡍʹ͸ɺجຊతʹGPSL͢ΔΑ͏ͳ༻
    ్ʹ͸࢖Θͳ͍΄͏͕ྑ͍
    w΋͠ར༻͢ΔͳΒGPSLઌͷϓϩηεͰ໌ࣔతʹॳظԽ͢Δඞཁ
    ͕͋Δ
    wਓؒ͸ֶͼɺ೥લʹ;ΘͬͱҋύονͰย෇͚ͨ໰୊΋ਂ͘ཧ
    ղ͠ղܾ͢Δ͜ͱ͕Ͱ͖ΔΑ͏ʹͳΔ
    39

    View Slide

  40. GoConference’19
    ਂֶ͘ͼɺ੒௕Ͱ͖Δ؀ڥ͕200mઌʹ͋Γ·͢
    !QC@SFDSVJU
    40

    View Slide