Docker Swarm 기반 Redis SaaS 플랫폼 구축 사례

#Redis #Swarm

카카오에는 Redis/Redis Cluster를 설치부터 관리까지 운영을 전문으로 하는 조직이 있습니다. 3000개가 넘는 레디스 운영 경험을 바탕으로 Docker Swarm을 기반 구축한 Redis SaaS 플랫폼 구축사례를 공유합니다.

발표자 : silvia.lee
카카오 시스템 엔지니어와 레디스 플랫폼 엔지니어를 맡고있는 실비아입니다.

Transcript

1. $PQZSJHIU
    ,BLBP $PSQ
   "MM
   SJHIUT
   SFTFSWFE
   3FEJTUSJCVUJPO
   PS
   QVCMJD
   EJTQMBZ
   JT
   OPU
   QFSNJUUFE
   XJUIPVU
   XSJUUFO
   QFSNJTTJPO
   GSPN
   ,BLBP %PDLFS
   4XBSN ӝ߈ 3FEJT
   4BB4
   ೒ۖಬ ҳ୷ ࢎ۹ JG LBLBP  ஠஠য় ੉਷޷ 4JMWJBMFF
2. ࣗѐ *O.FNPSZ5' ੋ೐ۄ ৉ೡ ஠஠য়੄ 3FEJT3FEJT
   $MVTUFS
   ࢲߡ ࢸ஖ ߂ ҙܻ ׸׼ 3FEJT3FEJT
   $MVTUFS
   ನೣ ࣻୌѐ੄ 3FEJT
   ਍৔
3. ֙ _ ࣗѐ *O.FNPSZ5'
4. ҳ୷ ݾ੸ दझమ ҳઑ )" )JHI
   BWBJMBCJMJUZ 4XBSN
   بੑ ࢎਬ ҳ୷ ബҗ
5. ҳ୷ ݾ੸
6. ҳ୷ ݾ੸  ಞܻೠ ࢎਊ ࢎਊ੗ח ߹ب੄ ੘স੉ա ࢸ஖হ੉ ѐߊ ױ҅ 1IBTF ী ೙ਃೠ 3FEJTܳ ૊द ࢤࢿ೧ ࢎਊೡ ࣻ ੓ب۾ ૑ਗ
7. ҳ୷ ݾ੸  ಞܻೠ ࢎਊ ࢎਊ੗ח ߹ب੄ ੘স੉ա ࢸ஖হ੉ ѐߊ ױ҅ 1IBTF ী ೙ਃೠ 3FEJTܳ ૊द ࢤࢿ೧ ࢎਊೡ ࣻ ੓ب۾ ૑ਗ 1SPE %FW
8. ҳ୷ ݾ੸  ಞܻೠ ࢎਊ ࢎਊ੗ח ߹ب੄ ੘স੉ա ࢸ஖হ੉ ѐߊ ױ҅ 1IBTF ী ೙ਃೠ 3FEJTܳ ૊द ࢤࢿ೧ ࢎਊೡ ࣻ ੓ب۾ ૑ਗ VTFS BENJO 3FEJT ࢤࢿ ਃ୒ 1SPE %FW
9. ҳ୷ ݾ੸  ಞܻೠ ࢎਊ ࢎਊ੗ח ߹ب੄ ੘স੉ա ࢸ஖হ੉ ѐߊ ױ҅ 1IBTF ী ೙ਃೠ 3FEJTܳ ૊द ࢤࢿ೧ ࢎਊೡ ࣻ ੓ب۾ ૑ਗ 3FEJT ࢤࢿ ਃ୒ 1SPE %FW VTFS BENJO
10. ҳ୷ ݾ੸  ಞܻೠ ࢎਊ ࢎਊ੗ח ߹ب੄ ੘স੉ա ࢸ஖হ੉ ѐߊ ױ҅ 1IBTF ী ೙ਃೠ 3FEJTܳ ૊द ࢤࢿ೧ ࢎਊೡ ࣻ ੓ب۾ ૑ਗ 3FEJT ࢤࢿ ਃ୒ SFEJTFYBNQMFEPNBJODPN 1SPE %FW VTFS BENJO
11. ҳ୷ ݾ੸  ಞܻೠ ࢎਊ ࢎਊ੗ח ߹ب੄ ੘স੉ա ࢸ஖হ੉ ѐߊ ױ҅ 1IBTF ী ೙ਃೠ 3FEJTܳ ૊द ࢤࢿ೧ ࢎਊೡ ࣻ ੓ب۾ ૑ਗ 3FEJT ࢤࢿ ਃ୒ SFEJTFYBNQMFEPNBJODPN ೞܖ _ ୭؀ ࣻѐਘ 1SPE %FW VTFS BENJO
12. ҳ୷ ݾ੸  ಞܻೠ ࢎਊ ࢎਊ੗ח ߹ب੄ ੘স੉ա ࢸ஖হ੉ ѐߊ ױ҅ 1IBTF ী ೙ਃೠ 3FEJTܳ ૊द ࢤࢿ೧ ࢎਊೡ ࣻ ੓ب۾ ૑ਗ (6* 1SPE %FW VTFS
13. ҳ୷ ݾ੸  ಞܻೠ ࢎਊ ࢎਊ੗ח ߹ب੄ ੘স੉ա ࢸ஖হ੉ ѐߊ ױ҅ 1IBTF ী ೙ਃೠ 3FEJTܳ ૊द ࢤࢿ೧ ࢎਊೡ ࣻ ੓ب۾ ૑ਗ (6* 1SPE %FW VTFS
14. ҳ୷ ݾ੸  ಞܻೠ ࢎਊ ࢎਊ੗ח ߹ب੄ ੘স੉ա ࢸ஖হ੉ ѐߊ ױ҅ 1IBTF ী ೙ਃೠ 3FEJTܳ ૊द ࢤࢿ೧ ࢎਊೡ ࣻ ੓ب۾ ૑ਗ (6* SFEJTFYBNQMFEPNBJODPN 1SPE %FW VTFS
15. ҳ୷ ݾ੸  ಞܻೠ ࢎਊ ࢎਊ੗ח ߹ب੄ ੘স੉ա ࢸ஖হ੉ ѐߊ ױ҅ 1IBTF ী ೙ਃೠ 3FEJTܳ ૊द ࢤࢿ೧ ࢎਊೡ ࣻ ੓ب۾ ૑ਗ (6* SFEJTFYBNQMFEPNBJODPN ࠙ 1SPE %FW VTFS
16. ҳ୷ ݾ੸  1. ­ 3FEJT ¬ 1.߹ ୭؀ ѐ੄ 3FEJTܳ ߓನೞৈ ૘੸ب ೱ࢚
17. ҳ୷ ݾ੸  1. ­ 3FEJT ¬ 1.߹ ୭؀ ѐ੄ 3FEJTܳ ߓನೞৈ ૘੸ب ೱ࢚
18. ҳ୷ ݾ੸  1. ­ 3FEJT ¬ 1.߹ ୭؀ ѐ੄ 3FEJTܳ ߓನೞৈ ૘੸ب ೱ࢚
19. दझమ ҳઑ
20. VTFS 8&# दझమ ҳઑ
21. 8PSLFS 8PSLFS 8PSLFS 8PSLFS 8PSLFS "1* VTFS दझమ ҳઑ .BOBHFS - 8&# .BOBHFS .BOBHFS
22. दझమ ҳઑ "1* VTFS 8&# 8PSLFS 8PSLFS 8PSLFS 8PSLFS 8PSLFS .BOBHFS - .BOBHFS .BOBHFS
23. %# SFEJT FOEQPJOU दझమ ҳઑ "1* VTFS 8&# .BOBHFS - .BOBHFS .BOBHFS 8PSLFS 8PSLFS 8PSLFS 8PSLFS 8PSLFS %/4 SFEJTFYBNQMFEPNBJODPN SFEJTFYBNQMFEPNBJODPN SFEJTFYBNQMFEPNBJODPN j j
24. %# दझమ ҳઑ "1* 8&# .BOBHFS - .BOBHFS .BOBHFS 8PSLFS 8PSLFS 8PSLFS 8PSLFS 8PSLFS %/4 &MBTUJD4FBSDI 1SPNFUIFVT
25. %# दझమ ҳઑ ex) changes(redis_db_keys{instance=“redis1.example.domain.com:6379”}[7d]) max_over_time(redis_connected_clients{instance=“redis1.example.domain.com:6379”}[7d]) *EMF
    3FTPVSDF 1SPNFUIFVT .BOBHFS - .BOBHFS .BOBHFS 8PSLFS 8PSLFS 8PSLFS 8PSLFS 8PSLFS %/4 "1*
26. "1* %# दझమ ҳઑ VTFS .BOBHFS - .BOBHFS .BOBHFS 8PSLFS 8PSLFS 8PSLFS 8PSLFS 8PSLFS %/4 1SPNFUIFVT
27. %/4 %# दझమ ҳઑ BENJO 8PSLFS 8PSLFS .BOBHFS - .BOBHFS .BOBHFS 8PSLFS 8PSLFS 8PSLFS 8PSLFS 8PSLFS "1* 1SPNFUIFVT
28. )" )JHI
    BWBJMBCJMJUZ
29. )" )JHI
    BWBJMBCJMJUZ $MJFOU
    "QQ IPTU
    OFUXPSL      SFEJTFYBNQMFEPNBJODPN "
    
    
    
    
    
    
    
    
    
     .BOBHFS .BOBHFS 8PSLFS 8PSLFS 8PSLFS 8PSLFS 8PSLFS .BOBHFS -
30. )" )JHI
    BWBJMBCJMJUZ .BOBHFS - .BOBHFS .BOBHFS      8PSLFS 8PSLFS 8PSLFS 8PSLFS 8PSLFS SFEJTFYBNQMFEPNBJODPN "
    
    
    
    
    
    
    
    
    
     $MJFOU
    "QQ
31. .POJUPSJOH 3FEJTNPO )" )JHI
    BWBJMBCJMJUZ .BOBHFS - .BOBHFS .BOBHFS      8PSLFS 8PSLFS 8PSLFS 8PSLFS 8PSLFS SFEJTFYBNQMFEPNBJODPN SFEJTFYBNQMFEPNBJODPN SFEJTFYBNQMFEPNBJODPN j j
32. SFEJTFYBNQMFEPNBJODPN "
    
    
    
    
    
    
    
    
    
     )" )JHI
    BWBJMBCJMJUZ .BOBHFS - .BOBHFS .BOBHFS      8PSLFS 8PSLFS 8PSLFS 8PSLFS 8PSLFS 3FEJTNPO $MJFOU
    "QQ
33. SFEJTFYBNQMFEPNBJODPN "
    
    
    
    
    
    
    
    
    
     )" )JHI
    BWBJMBCJMJUZ .BOBHFS - .BOBHFS .BOBHFS      8PSLFS 8PSLFS 8PSLFS 8PSLFS 8PSLFS 3FEJTNPO $MJFOU
    "QQ
34. SFEJTFYBNQMFEPNBJODPN "
    
    
    
    
    
    
    
    
    
     "1* 6QEBUF )" )JHI
    BWBJMBCJMJUZ .BOBHFS - .BOBHFS .BOBHFS      8PSLFS 8PSLFS 8PSLFS 8PSLFS 8PSLFS 3FEJTNPO $MJFOU
    "QQ
35. SFEJTFYBNQMFEPNBJODPN "
    
    
    
    
    
    
    
    
    
     "1* )" )JHI
    BWBJMBCJMJUZ .BOBHFS - .BOBHFS .BOBHFS      8PSLFS 8PSLFS 8PSLFS 8PSLFS 8PSLFS 3FEJTNPO $MJFOU
    "QQ
36. SFEJTFYBNQMFEPNBJODPN "
    
    
    
    
    
    
    
    
    
     SFEJTFYBNQMFEPNBJODPN "
    
    
    
    
    
    
    
    
    
     )" )JHI
    BWBJMBCJMJUZ .BOBHFS - .BOBHFS .BOBHFS      8PSLFS 8PSLFS 8PSLFS 8PSLFS 8PSLFS 3FEJTNPO $MJFOU
    "QQ %# "1* %/4
37. )" )JHI
    BWBJMBCJMJUZ .BOBHFS - .BOBHFS .BOBHFS     8PSLFS 8PSLFS 8PSLFS 8PSLFS 3FEJTNPO $MJFOU
    "QQ SFEJTFYBNQMFEPNBJODPN "
    
    
    
    
    
    
    
    
    
     55-
    
38. "1* %/4 .POJUPSJOH 6QEBUF 6QEBUF )" )JHI
    BWBJMBCJMJUZ .BOBHFS - .BOBHFS .BOBHFS     8PSLFS 8PSLFS 8PSLFS 8PSLFS 3FEJTNPO $MJFOU
    "QQ SFEJTFYBNQMFEPNBJODPN "
    
    
    
    
    
    
    
    
    
     55-
    
39. VTFS BENJO )" )JHI
    BWBJMBCJMJUZ .BOBHFS - .BOBHFS .BOBHFS     8PSLFS 8PSLFS 8PSLFS 8PSLFS 3FEJTNPO $MJFOU
    "QQ SFEJTFYBNQMFEPNBJODPN "
    
    
    
    
    
    
    
    
    
     55-
    
    
    
    
40.  3FEJTNPO ࣗѐ  1ZUIPO
    ӝ߈ 3FEJT
    ݽפఠ݂ ؘހ  ѐ੄ ؘހীࢲ ୡ׼ ড ѐ੄ 3FEJT
    ੋझఢझ ݽפఠ݂ ࣻ೯ оמ  TDBMFPVU
    оמ  1PSUPQFSBUJPO
    فо૑ ఋੑ੄ ݽפఠ݂ ߑध ૑ਗ  ೾झ୓௼ पಁ ಕੌয়ߡ ੉߮౟ ੹࣠ )" )JHI
    BWBJMBCJMJUZ
41. 3FEJTNPO ࣗѐ 3FEJTNPO
42. 3FEJTNPO ࣗѐ  ߓ஖ ੘সਵ۽ ੋೠ ੌद੸ ౟ې೗ ૐо ߂ PQFSBUJPO
    IFBMUI
    DIFDL
    पಁ
43.  ߓ஖ ੘সਵ۽ ੋೠ ੌद੸ ౟ې೗ ૐо ߂ PQFSBUJPO
    IFBMUI
    DIFDL
    पಁ 3FEJTNPO ࣗѐ  ݽפఠ݂ ఋੑਸ QPSU
    DIFDL
    ߑधਵ۽ ߸҃ೞৈ ࠛ೙ਃೠ ಕੌয়ߡ ߑ૑
44. .BTUFS 3FQMJDB .BTUFS 3FQMJDB /PO3FQMJDB $MJFOU
    "QQ $MJFOU
    "QQ
45. .BTUFS 3FQMJDB .BTUFS ؘ੉ఠ ਬप ୭ࣗച .BTUFS )" ؘ੉ఠ ਬप ೲਊ .BTUFS 1SPNPUJPO 3FQMJDB /PO3FQMJDB $MJFOU
    "QQ $MJFOU
    "QQ
46. .BTUFS 3FQMJDB .BTUFS ؘ੉ఠ ਬप ୭ࣗച .BTUFS )" ؘ੉ఠ ਬप ೲਊ .BTUFS 3FQMJDB /PO3FQMJDB $MJFOU
    "QQ $MJFOU
    "QQ 1SPNPUJPO
47. 3FEJTNPO ࣗѐ 3FEJTNPO
48. • ௿ۄ੉঱౟ ਕ௼۽٘߹ ୭੸੄ ݽפఠ݂ਸ ઁҕ೧ ࠛ೙ਃೠ ಕੌয়ߡ ߑ૑ • /PO3FQMJDB
    ҳࢿীࢲب )" ૑ਗ 3FEJTNPO 3FEJTNPO ബҗ
49. %PDLFS
    4XBSN
    بੑ ࢎਬ
50.  ਃҳ ࢎ೦  ࡅܲ ߓನ  ֫਷ ૘੸ب  ੗ਗ ҙܻ 4DIFEVMJOH  ੗ਗ ࢚క ҙܻ 4UBUF  о߶਍ о࢚ച ӝࣿ QFSGPSNBODF %PDLFS
    4XBSN
    بੑ ࢎਬ
51.  ਃҳ ࢎ೦  ࡅܲ ߓನ  ֫਷ ૘੸ب  ੗ਗ ҙܻ 4DIFEVMJOH  ੗ਗ ࢚క ҙܻ 4UBUF  о߶਍ о࢚ച ӝࣿ QFSGPSNBODF %PDLFS
    4XBSN
    بੑ ࢎਬ ஶప੉ց য়ாझ౟ۨ੉࣌
52. %PDLFS
    4XBSN ,VCFSOFUFT *OTUBMMBUJPO
    
    4FUVQ 2VJDL
    BOE
    FBTZ $PNQMFY -FBSOJOH
    DVSWF -JHIUXFJHIU
    CVU
    MJNJUFE
    GVODUJPOBMJUZ )JHI
    CVU
    IBT
    NPSF
    GFBUVSFT (6* 9
    SFRVJSF
    UIJSEQBSUZ
    UPPMT /BUJWF
    EBTICPBSET 4DBMBCJMJUZ 0
    0 )JHI
    "WBJMBCJMJUZ 0 0 /FUXPSLJOH PWFSMBZ
    JOHSFTT
    IPTUCSJEHF *OHSFTT
    'MBOOFM
    $BMJDP
    BOE
    NPSF $PNNVOJUZ 4NBMMFS )VHF
    BOE
    BDUJWF 0QUJNBM
    VTF
    DBTF 2VJDL
    UP
    EFQMPZ
    BOE
    FBTZ
    UP
    NBOBHF )JHIEFNBOE
    BQQT
    BOE SFRVJSF
    B
    DPNQMFUF
    QBDLBHF %PDLFS
    4XBSN
    WT
    ,VCFSOFUFT
53. %PDLFS
    4XBSN ,VCFSOFUFT *OTUBMMBUJPO
    
    4FUVQ 2VJDL
    BOE
    FBTZ $PNQMFY -FBSOJOH
    DVSWF -JHIUXFJHIU
    CVU
    MJNJUFE
    GVODUJPOBMJUZ )JHI
    CVU
    IBT
    NPSF
    GFBUVSFT (6* 9
    SFRVJSF
    UIJSEQBSUZ
    UPPMT /BUJWF
    EBTICPBSET 4DBMBCJMJUZ 0
    0 )JHI
    "WBJMBCJMJUZ 0 0 /FUXPSLJOH PWFSMBZ
    JOHSFTT
    IPTUCSJEHF *OHSFTT
    'MBOOFM
    $BMJDP
    BOE
    NPSF $PNNVOJUZ 4NBMMFS )VHF
    BOE
    BDUJWF 0QUJNBM
    VTF
    DBTF 2VJDL
    UP
    EFQMPZ
    BOE
    FBTZ
    UP
    NBOBHF )JHIEFNBOE
    BQQT
    BOE SFRVJSF
    B
    DPNQMFUF
    QBDLBHF %PDLFS
    4XBSN
    WT
    ,VCFSOFUFT
54. %PDLFS
    4XBSN ,VCFSOFUFT *OTUBMMBUJPO
    
    4FUVQ 2VJDL
    BOE
    FBTZ $PNQMFY -FBSOJOH
    DVSWF -JHIUXFJHIU
    CVU
    MJNJUFE
    GVODUJPOBMJUZ )JHI
    CVU
    IBT
    NPSF
    GFBUVSFT (6* 9
    SFRVJSF
    UIJSEQBSUZ
    UPPMT /BUJWF
    EBTICPBSET 4DBMBCJMJUZ 0
    0 )JHI
    "WBJMBCJMJUZ 0 0 /FUXPSLJOH PWFSMBZ
    JOHSFTT
    IPTUCSJEHF *OHSFTT
    'MBOOFM
    $BMJDP
    BOE
    NPSF $PNNVOJUZ 4NBMMFS )VHF
    BOE
    BDUJWF 0QUJNBM
    VTF
    DBTF 2VJDL
    UP
    EFQMPZ
    BOE
    FBTZ
    UP
    NBOBHF )JHIEFNBOE
    BQQT
    BOE SFRVJSF
    B
    DPNQMFUF
    QBDLBHF %PDLFS
    4XBSN
    WT
    ,VCFSOFUFT
55. %PDLFS
    4XBSN ,VCFSOFUFT *OTUBMMBUJPO
    
    4FUVQ 2VJDL
    BOE
    FBTZ $PNQMFY -FBSOJOH
    DVSWF -JHIUXFJHIU
    CVU
    MJNJUFE
    GVODUJPOBMJUZ )JHI
    CVU
    IBT
    NPSF
    GFBUVSFT (6* 9
    SFRVJSF
    UIJSEQBSUZ
    UPPMT /BUJWF
    EBTICPBSET 4DBMBCJMJUZ 0 0 )JHI
    "WBJMBCJMJUZ 0 0
    /FUXPSLJOH PWFSMBZ
    JOHSFTT
    IPTUCSJEHF *OHSFTT
    'MBOOFM
    $BMJDP
    BOE
    NPSF $PNNVOJUZ 4NBMMFS )VHF
    BOE
    BDUJWF 0QUJNBM
    VTF
    DBTF 2VJDL
    UP
    EFQMPZ
    BOE
    FBTZ
    UP
    NBOBHF )JHIEFNBOE
    BQQT
    BOE SFRVJSF
    B
    DPNQMFUF
    QBDLBHF %PDLFS
    4XBSN
    WT
    ,VCFSOFUFT
56. %PDLFS
    4XBSN
    WT
    ,VCFSOFUFT 4DBMBCJMJUZ  ࢎਊ੗ ਃ୒ী ੄ೠ ࣻز ഛ੢  )1" )PSJ[POUBM
    1PE
    "VUPTDBMFS ܳ ాೠ ౟ې೗ ӝ߈੄ ੗ز ഛ੢
57. %PDLFS
    4XBSN
    WT
    ,VCFSOFUFT 4DBMBCJMJUZ  ࢎਊ੗ ਃ୒ী ੄ೠ ࣻز ഛ੢ IFMMPXPSME 
    SFQMJDBT TFSWJDF 4XBSN
    NBOBHFS IFMMPXPSME 4XBSN
    XPSLFS UBTL DPOUBJOFS
58. %PDLFS
    4XBSN
    WT
    ,VCFSOFUFT 4DBMBCJMJUZ  ࢎਊ੗ ਃ୒ী ੄ೠ ࣻز ഛ੢ $ docker service scale IFMMPXPSME=2 helloworld scaled to 2 IFMMPXPSME 
    SFQMJDBT TFSWJDF 4XBSN
    NBOBHFS IFMMPXPSME 4XBSN
    XPSLFS UBTL DPOUBJOFS
59. %PDLFS
    4XBSN
    WT
    ,VCFSOFUFT 4DBMBCJMJUZ  ࢎਊ੗ ਃ୒ী ੄ೠ ࣻز ഛ੢ $ docker service scale IFMMPXPSME=2 helloworld scaled to 2 IFMMPXPSME  SFQMJDBT TFSWJDF 4XBSN
    NBOBHFS IFMMPXPSME 4XBSN
    XPSLFS UBTL DPOUBJOFS IFMMPXPSME 4XBSN
    XPSLFS
60. .BTUFS
     .BTUFS
     .BTUFS
     $MJFOU
    "QQ LFZ LFZ LFZ 3FEJTT
    4DBMBCJMJUZ
61. .BTUFS
     .BTUFS
     .BTUFS
     LFZ LFZ LFZ 1SPYZ 3FEJTT
    4DBMBCJMJUZ $MJFOU
    "QQ
62. 3FEJTT
    4DBMBCJMJUZ OFX OFX
63. %PDLFS
    4XBSN
    WT
    ,VCFSOFUFT )JHI
    "WBJMBCJMJUZ  ݒפ੷ ֢٘੄ ҙܻܳ ాೠ оਊࢿ ഛࠁ  SFQMJDBTFUਸ ాೠ QPE
    ࢚క хद ߂ BVUPIFBMJOH
64. %PDLFS
    4XBSN
    WT
    ,VCFSOFUFT )JHI
    "WBJMBCJMJUZ  ݒפ੷ ֢٘੄ ҙܻܳ ాೠ оਊࢿ ഛࠁ IFMMPXPSME  SFQMJDBT 4XBSN
    NBOBHFS IFMMPXPSME 4XBSN
    XPSLFS IFMMPXPSME 4XBSN
    XPSLFS
65. %PDLFS
    4XBSN
    WT
    ,VCFSOFUFT )JHI
    "WBJMBCJMJUZ  ݒפ੷ ֢٘੄ ҙܻܳ ాೠ оਊࢿ ഛࠁ IFMMPXPSME  SFQMJDBT 4XBSN
    NBOBHFS IFMMPXPSME 4XBSN
    XPSLFS IFMMPXPSME 4XBSN
    XPSLFS
66. %PDLFS
    4XBSN
    WT
    ,VCFSOFUFT )JHI
    "WBJMBCJMJUZ  ݒפ੷ ֢٘੄ ҙܻܳ ాೠ оਊࢿ ഛࠁ IFMMPXPSME  SFQMJDBT 4XBSN
    NBOBHFS IFMMPXPSME 4XBSN
    XPSLFS IFMMPXPSME 4XBSN
    XPSLFS IFMMPXPSME 4XBSN
    XPSLFS
67. %PDLFS
    4XBSN ,VCFSOFUFT *OTUBMMBUJPO
    
    4FUVQ 2VJDL
    BOE
    FBTZ $PNQMFY -FBSOJOH
    DVSWF -JHIUXFJHIU
    CVU
    MJNJUFE
    GVODUJPOBMJUZ )JHI
    CVU
    IBT
    NPSF
    GFBUVSFT (6* 9
    SFRVJSF
    UIJSEQBSUZ
    UPPMT /BUJWF
    EBTICPBSET 4DBMBCJMJUZ 0 0 )JHI
    "WBJMBCJMJUZ 0 0
    /FUXPSLJOH PWFSMBZ
    JOHSFTT
    IPTUCSJEHF *OHSFTT
    'MBOOFM
    $BMJDP
    BOE
    NPSF $PNNVOJUZ 4NBMMFS )VHF
    BOE
    BDUJWF 0QUJNBM
    VTF
    DBTF 2VJDL
    UP
    EFQMPZ
    BOE
    FBTZ
    UP
    NBOBHF )JHIEFNBOE
    BQQT
    BOE SFRVJSF
    B
    DPNQMFUF
    QBDLBHF %PDLFS
    4XBSN
    WT
    ,VCFSOFUFT
68. %PDLFS
    4XBSN ,VCFSOFUFT *OTUBMMBUJPO
    
    4FUVQ 2VJDL
    BOE
    FBTZ $PNQMFY -FBSOJOH
    DVSWF -JHIUXFJHIU
    CVU
    MJNJUFE
    GVODUJPOBMJUZ )JHI
    CVU
    IBT
    NPSF
    GFBUVSFT (6* 9
    SFRVJSF
    UIJSEQBSUZ
    UPPMT /BUJWF
    EBTICPBSET 4DBMBCJMJUZ 0
    0 )JHI
    "WBJMBCJMJUZ 0 0 /FUXPSLJOH PWFSMBZ
    JOHSFTT
    IPTUCSJEHF *OHSFTT
    'MBOOFM
    $BMJDP
    BOE
    NPSF $PNNVOJUZ 4NBMMFS )VHF
    BOE
    BDUJWF 0QUJNBM
    VTF
    DBTF 2VJDL
    UP
    EFQMPZ
    BOE
    FBTZ
    UP
    NBOBHF )JHIEFNBOE
    BQQT
    BOE SFRVJSF
    B DPNQMFUF
    QBDLBHF %PDLFS
    4XBSN
    WT
    ,VCFSOFUFT
69. $MJFOU
    "QQ IPTU
    OFUXPSL %PDLFS
    4XBSN
    WT
    ,VCFSOFUFT /FUXPSLJOH      8PSLFS 8PSLFS 8PSLFS 8PSLFS 8PSLFS .BOBHFS - .BOBHFS .BOBHFS
70. %PDLFS
    4XBSN ,VCFSOFUFT *OTUBMMBUJPO
    
    4FUVQ 2VJDL
    BOE
    FBTZ $PNQMFY -FBSOJOH
    DVSWF -JHIUXFJHIU
    CVU
    MJNJUFE
    GVODUJPOBMJUZ )JHI
    CVU
    IBT
    NPSF
    GFBUVSFT (6* 9
    SFRVJSF
    UIJSEQBSUZ
    UPPMT /BUJWF
    EBTICPBSET 4DBMBCJMJUZ 0
    0 )JHI
    "WBJMBCJMJUZ 0 0 /FUXPSLJOH PWFSMBZ
    JOHSFTT
    IPTUCSJEHF *OHSFTT
    'MBOOFM
    $BMJDP
    BOE
    NPSF $PNNVOJUZ 4NBMMFS )VHF
    BOE
    BDUJWF 0QUJNBM
    VTF
    DBTF 2VJDL
    UP
    EFQMPZ
    BOE
    FBTZ
    UP
    NBOBHF )JHIEFNBOE
    BQQT
    BOE SFRVJSF
    B DPNQMFUF
    QBDLBHF %PDLFS
    4XBSN
    WT
    ,VCFSOFUFT
71. %PDLFS
    4XBSN ,VCFSOFUFT *OTUBMMBUJPO
    
    4FUVQ 2VJDL
    BOE
    FBTZ $PNQMFY -FBSOJOH
    DVSWF -JHIUXFJHIU
    CVU
    MJNJUFE
    GVODUJPOBMJUZ )JHI
    CVU
    IBT
    NPSF
    GFBUVSFT (6* 9
    SFRVJSF
    UIJSEQBSUZ
    UPPMT /BUJWF
    EBTICPBSET 4DBMBCJMJUZ 0
    0 )JHI
    "WBJMBCJMJUZ 0 0 /FUXPSLJOH PWFSMBZ
    JOHSFTT
    IPTUCSJEHF *OHSFTT
    'MBOOFM
    $BMJDP
    BOE
    NPSF $PNNVOJUZ 4NBMMFS )VHF
    BOE
    BDUJWF 0QUJNBM
    VTF
    DBTF 2VJDL
    UP
    EFQMPZ
    BOE
    FBTZ
    UP
    NBOBHF )JHIEFNBOE
    BQQT
    BOE SFRVJSF
    B DPNQMFUF
    QBDLBHF %PDLFS
    4XBSN
    WT
    ,VCFSOFUFT
72. %PDLFS
    4XBSN ,VCFSOFUFT *OTUBMMBUJPO
    
    4FUVQ 2VJDL
    BOE
    FBTZ $PNQMFY -FBSOJOH
    DVSWF -JHIUXFJHIU
    CVU
    MJNJUFE
    GVODUJPOBMJUZ )JHI
    CVU
    IBT
    NPSF
    GFBUVSFT (6* 9
    SFRVJSF
    UIJSEQBSUZ
    UPPMT /BUJWF
    EBTICPBSET 4DBMBCJMJUZ 0
    0 )JHI
    "WBJMBCJMJUZ 0 0 /FUXPSLJOH PWFSMBZ
    JOHSFTT
    IPTUCSJEHF *OHSFTT
    'MBOOFM
    $BMJDP
    BOE
    NPSF $PNNVOJUZ 4NBMMFS )VHF
    BOE
    BDUJWF 0QUJNBM
    VTF
    DBTF 2VJDL
    UP
    EFQMPZ
    BOE
    FBTZ
    UP
    NBOBHF )JHIEFNBOE
    BQQT
    BOE SFRVJSF
    B DPNQMFUF
    QBDLBHF %PDLFS
    4XBSN
    WT
    ,VCFSOFUFT
73. %PDLFS
    4XBSN ,VCFSOFUFT *OTUBMMBUJPO
    
    4FUVQ 2VJDL
    BOE
    FBTZ $PNQMFY -FBSOJOH
    DVSWF -JHIUXFJHIU
    CVU
    MJNJUFE
    GVODUJPOBMJUZ )JHI
    CVU
    IBT
    NPSF
    GFBUVSFT (6* 9
    SFRVJSF
    UIJSEQBSUZ
    UPPMT /BUJWF
    EBTICPBSET 4DBMBCJMJUZ 0
    0 )JHI
    "WBJMBCJMJUZ 0 0 /FUXPSLJOH PWFSMBZ
    JOHSFTT
    IPTUCSJEHF *OHSFTT
    'MBOOFM
    $BMJDP
    BOE
    NPSF $PNNVOJUZ 4NBMMFS )VHF
    BOE
    BDUJWF 0QUJNBM
    VTF
    DBTF 2VJDL
    UP
    EFQMPZ
    BOE
    FBTZ
    UP
    NBOBHF )JHIEFNBOE
    BQQT
    BOE SFRVJSF
    B DPNQMFUF
    QBDLBHF %PDLFS
    4XBSN
    WT
    ,VCFSOFUFT
74. %PDLFS
    4XBSN ,VCFSOFUFT *OTUBMMBUJPO
    
    4FUVQ 2VJDL
    BOE
    FBTZ $PNQMFY -FBSOJOH
    DVSWF -JHIUXFJHIU
    CVU
    MJNJUFE
    GVODUJPOBMJUZ )JHI
    CVU
    IBT
    NPSF
    GFBUVSFT (6* 9
    SFRVJSF
    UIJSEQBSUZ
    UPPMT /BUJWF
    EBTICPBSET 4DBMBCJMJUZ 0
    0 )JHI
    "WBJMBCJMJUZ 0 0 /FUXPSLJOH PWFSMBZ
    JOHSFTT
    IPTUCSJEHF *OHSFTT
    'MBOOFM
    $BMJDP
    BOE
    NPSF $PNNVOJUZ 4NBMMFS )VHF
    BOE
    BDUJWF 0QUJNBM
    VTF
    DBTF 2VJDL
    UP
    EFQMPZ
    BOE
    FBTZ
    UP
    NBOBHF )JHIEFNBOE
    BQQT
    BOE SFRVJSF
    B DPNQMFUF
    QBDLBHF %PDLFS
    4XBSN
    WT
    ,VCFSOFUFT
75. %PDLFS
    4XBSN ബҗ • %PDLFS
    4XBSN੉ ஶప੉ց झா઴݂ ߂ ࢚క ҙܻܳ ೧ષਵ۽ॄ 1.
    ૘੸ب ೱ࢚ • ௿۞झఠ੄ ए਍ ࢸ஖ ߂ ࡅܲ ҳࢿ оמ • ࠺Ү੸ ծ਷ ۞׬ழ࠳৬ ब೒ೠ ௿۞झఠ ҳઑ۽ ҙܻ੗੄ ਍৔ ࠗ׸ хࣗ
76. ҳ୷ ബҗ
77. ҳ୷ ബҗ • ౭௄ ߬੉झ੄ 3FEJT ࢤࢿ ਃ୒ ߑध ؀न ਢਸ ాೠ ࢤࢿ ӝמ ઁҕਵ۽ BENJOҗ ࢎਊ੗р ழޭפா੉࣌ ࠺ਊ хࣗ VTFS BENJO 3FEJT ࢤࢿ ਃ୒ (6*
78. • ೞܖ ੉࢚ ࣗਃغ؍ 3FEJT
    FOEQPJOU
    ࢤࢿ दрਸ ࠙ਵ۽ ױ୷ BENJO 1SPE %FW ҳ୷ ബҗ
79. • ࢎਊ੗ ಞ੄ ೱ࢚ਵ۽ 3FEJT
    ੋझఢझо ࡅܰѱ ૐоೞח ೒ۖಬਵ۽ ࢿ੢ ҳ୷ ബҗ
80. &0%