Upgrade to Pro — share decks privately, control downloads, hide ads and more …

OpenShift 4のオンプレ導入の手引き/Install OCP4 on-premises

OpenShift 4のオンプレ導入の手引き/Install OCP4 on-premises

2020/4/27に行われた「OpenShift Meetup Tokyo #8(夕方開催!特別編) インストール&レジストリ」で使用した資料です。
https://openshift.connpass.com/event/174128/

オンプレ環境にOCP4を導入することを前提としたUPIインストールについて解説します。
制限環境でのインストールやその他の構成方法、OKD4についても記載しています。

[アジェンダ]
-Why オンプレ?
-OCP4 UPIインストール
-制限ネットワーク環境下におけるインストール
-その他の構成方法
-OKD4
-まとめ
-参考リンク

Shion Tanaka

April 27, 2020
Tweet

More Decks by Shion Tanaka

Other Decks in Technology

Transcript

  1. PQFOTIJGUKQ
    0QFO4IJGUͷ
    ΦϯϓϨಋೖͷखҾ͖
    "QS W

    0QFO4IJGU.FFUVQ5PLZP
    4IJPO5BOBLB !UOLPO

    1

    View Slide

  2. PQFOTIJGUKQ
    ΞδΣϯμ
    • 8IZΦϯϓϨʁ
    • 0$161*Πϯετʔϧ
    • ੍ݶωοτϫʔΫ؀ڥԼʹ͓͚ΔΠϯετʔϧ
    • ͦͷଞͷߏ੒ํ๏
    • 0,%
    • ·ͱΊ
    • ࢀߟϦϯΫ
    2

    View Slide

  3. PQFOTIJGUKQ
    ͓ئ͍ࣄ߲
    • هࡌ಺༰͸ݸਓͷݟղͰ͋Γɺॴଐ͢Δ૊৫ͷެࣜݟղͰ͸͋
    Γ·ͤΜ
    • ଞࣾ੡඼ʹ͍ͭͯͷݴٴ΋͋Γ·͕͢ɺಈ࡞Λอূ͢Δ΋ͷͰ
    ͸͋Γ·ͤΜɻࢀߟ৘ใͱͯ͠͝ཧղ͍ͩ͘͞ɻ
    • αϙʔτ΁ͷ໰͍߹Θͤ࣌͸ɺެࣜ৘ใΛࢀরͷ͏͑໰͍߹Θ
    ͍ͤͩ͘͞ɻهࡌ಺༰͸ඞͣ͠΋αϙʔτΛอূ͢Δ΋ͷͰ͸
    ͋Γ·ͤΜɻ
    • ͜͏ͨ͠Β͍͍Αʂվળ৘ใ͸Ͳ͠Ͳ͠πΠʔτɺίϝϯτ͘
    ͍ͩ͞
    3

    View Slide

  4. PQFOTIJGUKQ
    ࣗݾ঺հ
    • ໊લɿాத ࢘ԸʢλφΧ γΦϯʣ!UOLPO
    • ॴଐɿϨουϋοτגࣜձࣾ
    Ø ιϦϡʔγϣϯΞʔΩςΫτʢ෱Ԭࡏॅ۝भɾதࠃ஍ํ୲౰ʣ
    • ܦྺɿେֶ৬һઐ໳ֶߍߨࢣˠେֶ৘γεˠ೔ܥاۀˠݱ৬
    Ø લ৬ɿओʹԾ૝Խ *5ΠϯϑϥͷϓϦηʔϧε4&
    • ࠷ۙͷ׆ಈɿ
    Ø 0QFO4IJGUͷݕূɺϒϩάࣥච
    Ø 0QFO4IJGUϫʔΫγϣοϓ։࠵
    Ø ίϛϡχςΟ׆ಈࢀՃʢ0QFO4IJGU.FFUVQɺ0QFO4IJGUSVOɺ+BQBO3PPL.FFUVQɺͳͲʣ
    ೔໷ɺࣗ୐αʔόʔͰ0QFO4IJGUͷݕূதʜ ˠ ϥΠϒ഑৴ɺԻ੠ͷݕূதʜ
    4

    View Slide

  5. PQFOTIJGUKQ
    8IZΦϯϓϨʁ
    5

    View Slide

  6. PQFOTIJGUKQ
    8IZΦϯϓϨʁ
    • ΦϯϓϨʹͩ͜ΘΔཧ༝
    Øίετ
    ØύϑΥʔϚϯε
    ØηΩϡϦςΟ
    Øςετ໨త
    ØΤοδίϯϐϡʔςΟϯά
    ϝϦοτɿϢʔβʔʹ͍ۙڑ཭Ͱ࢖͏͜ͱ͕Ͱ͖Δ
    6

    View Slide

  7. PQFOTIJGUKQ
    8IZΦϯϓϨʁ
    • ΤϯδχΤΞతཉٻ
    ØΠϯϑϥ໨ઢͰ஌Γ͍ͨ
    ØͲ͏͍͏࢓૊Έͳͷ͔
    Øਓ೚ͤʹ͠ͳ͍
    Øഽײ֮Λେࣄʹ͢Δ
    7

    View Slide

  8. PQFOTIJGUKQ
    0QFO4IJGUͱΦϯϓϨ
    • ೔ຊͷΤϯλʔϓϥΠζاۀͰ͸ΦϯϓϨͰͷԾ૝ج൫ར༻͕
    ଟ͍
    • ৽͍͠γεςϜͷಋೖͷલʹɺ·ͣ͸ΦϯϓϨͰݕূɺߏஙΛ
    ࣮ࢪ͢Δέʔεʢखݩ͔ΒεϞʔϧελʔτʣ
    ίϯςφ΍,VCFSOFUFTӠʑͷલʹɺࣗલͷԾ૝ج൫Ͱ
    0QFO4IJGU͕ಈ͘ͷ͔Ͳ͏͔ʁͱ͍͏ٙ໰ɺٞ࿦͕ઌʹɻ
    ˠલʹਐΉͨΊʹ·ͣ͸ɺ΍ͬͯΈͯͱʹ͔͘ಈ͔ͯ͠ΈΔɻ
    8

    View Slide

  9. PQFOTIJGUKQ
    0$161*Πϯετʔϧ
    ֓ཁͱࣄલ४උ
    9

    View Slide

  10. PQFOTIJGUKQ
    ରԠϓϥοτϑΥʔϜ
    10
    ˞࣌఺
    Full Stack Automation (IPI) Pre-existing Infrastructure (UPI)
    Bare Metal
    *
    Developer Preview

    View Slide

  11. PQFOTIJGUKQ
    Πϯετʔϧํ๏
    • *1* *OTUBMMFSQSPWJTJPOFEJOGSBTUSVDUVSF

    11
    https://access.redhat.com/documentation/ja-jp/openshift_container_platform/4.3/html/installing/installation-configuration#installation-types
    AWS Azure GCP OpenStack Bare metal vSphere IBM Z
    カスタム X X X X
    Network Operator X X
    ネットワークが制限された
    インストール
    X X X
    AWS Azure GCP OpenStack Bare metal vSphere IBM Z
    デフォルト X X X
    カスタム X X X X
    Network Operator X X X
    プライベートクラスター X X X
    既存の仮想プライベートネ
    ットワーク
    X X X
    ˞࣌఺
    • 61* 6TFSQSPWJTJPOFEJOGSBTUSVDUVSF
    ࠓճ঺հ͢Δ಺༰͸ίϨ

    View Slide

  12. PQFOTIJGUKQ
    ϕΞϝλϧϝιου
    • ϕΞϝλϧϝιου͸෺ཧαʔόʔͷΈʹରԠʁ
    ØϝιουͳͷͰ࣮ࡍʹ͸Ծ૝؀ڥͰ΋Մೳ
    • ϕΞϝλϧϝιουͷར఺
    ØҰ൪γϯϓϧͳߏ੒ํ๏
    Ø0QFO4IJGUͷΠϯετʔϧํ๏΍ඞཁͳ؀ڥͷཧղɺֶशʹ޲͍ͯ
    ͍Δ
    ØΠϯϑϥΤϯδχΞతʹ͸Ұ൪औͬ෇͖΍͍͢ʢͱࢥ͏ʣ
    0QFO4IJGUͷΠϯετʔϧͱ͍͏໨తʹूதͰ͖Δ
    12

    View Slide

  13. PQFOTIJGUKQ
    ཁ݅ΠϯλʔωοτΞΫηε
    • 3FE)BU0QFO4IJGU$MVTUFS.BOBHFS ΁ΞΫηε
    ØΠϯετʔϧʹඞཁͳϓϩάϥϜͷμ΢ϯϩʔυ
    ØαϒεΫϦϓγϣϯͷ؅ཧʢΠϯετʔϧޙʣ
    • ΫϥελʔͷΠϯετʔϧɺߋ৽࣌ͷ2VBZJP΁ͷΞΫηε
    ØΫϥελʔ͔ΒΠϯλʔωοτ΁௚઀ΞΫηεͰ͖ͳ͍৔߹͸ɺϓϩ
    Ωγ͓Αͼඇ઀ଓ؀ڥͷΦϓγϣϯ༗Γ
    13

    View Slide

  14. PQFOTIJGUKQ
    ཁ݅Ϛγϯཁ݅
    • Ϋϥελʔͷߏ੒ʹඞཁͳ࠷খཁ݅
    14
    Ϛγϯ ୆਺ 04 W$16 3". ετϨʔδ උߟ
    #PPUTUSBQ/PEF 3)$04 ˞
    (# (#
    Πϯετʔϧ࣌
    ͷΈඞཁ
    .BTUFS/PEF 3)$04 (# (# ୆ඞਢ
    8PSLFS/PEF
    3)$04ɺ·ͨ
    3)&-
    (# (# ୆Ҏ্ ˞

    ˞3)$04͸ 3FE)BU&OUFSQSJTF-JOVYΛϕʔεͱ͓ͯ͠Γɺͦͷϋʔυ΢ΣΞೝఆ͓Αͼཁ͕݅ܧঝ͞
    ΕΔ͜ͱʹ஫ҙ
    ˞ຊ൪؀ڥͰͷՄ༻ੑΛߟྀ͠୆Ҏ্Λਪ঑
    ˞OPEFΫϥελʔͷ৔߹͸ɺ.BTUFS/PEF͕݉༻͢ΔͨΊ8PSLFS/PEF͸ෆཁ

    View Slide

  15. PQFOTIJGUKQ
    ཁ݅Πϯϑϥཁ݅
    • Ϋϥελʔ֎ͰඞཁͳΠϯϑϥ؀ڥ
    Ø%)$1αʔόʔ
    Ø%/4αʔόʔ
    Øϩʔυόϥϯαʔ
    Ø8FCαʔόʔ
    ͜ΕΒΛ·ͱΊͯʮ౿Έ୆αʔόʔʯͱͯ͠ߏங͢Δํ๏Λ঺
    հ
    15

    View Slide

  16. PQFOTIJGUKQ
    ཁ݅%)$1
    • 3)$04Ϛγϯ͸ॳճىಈ࣌ʹ%)$1ʹΑΔ*1ΞυϨεͷׂΓ౰͕ͯඞཁ
    Øʢਪ঑ʣ."$ΞυϨεࢦఆͰϗετ໊ɺ*1ΞυϨεΛݻఆׂ౰
    ʢઃఆྫʣ
    16
    マシン ホスト名 IPアドレス サブネットマスク ゲートウェイ DNSサーバー
    Bootstrap Node bootstrap.test.example.local 172.16.0.100 255.255.255.0 172.16.0.1 172.16.0.1
    Master Node
    master-0.test.example.local
    master-1.test.example.local
    master-2.test.example.local
    172.16.0.101
    172.16.0.102
    172.16.0.103
    255.255.255.0 172.16.0.1 172.16.0.1
    Worker Node
    worker-0.test.example.local
    worker-1.test.example.local
    172.16.0.104
    172.16.0.105
    255.255.255.0 172.16.0.1 172.16.0.1
    ࣮ࡍͷ؀ڥͰ͸࡞ۀܰݮͷͨΊ19&#PPUͷ࢖༻Λਪ঑

    View Slide

  17. PQFOTIJGUKQ
    ཁ݅%/4
    • ໊લղܾͷཁ݅
    17
    ίϯϙʔωϯτ Ϩίʔυ ର৅ϊʔυ
    ,VCFSOFUFT"1*
    [email protected]@EPNBJO
    [email protected]@EPNBJO
    #PPUTUSBQ .BTUFS
    3PVUFT [email protected]@EPNBJO 8PSLFS
    FUDE
    [email protected]@EPNBJO
    @[email protected]@[email protected]
    .BTUFS

    View Slide

  18. PQFOTIJGUKQ
    ཁ݅%/4ʢઃఆྫʣ
    18
    ϗετ໊ *1ΞυϨε උߟ
    BQJUFTUFYBNQMFMPDBM -#ͷ֎ଆΞυϨεΛࢦఆ
    BQJJOUUFTUFYBNQMFMPDBM -#ͷ಺ଆΞυϨεΛࢦఆ
    BQQTUFTUFYBNQMFMPDBM -#ͷ಺ଆΞυϨεΛࢦఆɻϫΠϧυΧʔυ%/4Ϩίʔυɻ
    FUDEUFTUFYBNQMFMPDBM
    γεςϜ಺෦ͰFUDE Ͱݻఆ໊Ͱ໊લղܾΛߦ͏ͷͰɺ
    .BTUFS/PEFͷϗετ໊ͱ͸ผͰొ࿥ඞཁ
    FUDEUFTUFYBNQMFMPDBM ಉ্
    FUDEUFTUFYBNQMFMPDBM ಉ্
    Ϩίʔυ ༏ઌ౓ ॏΈ ϙʔτ %/4໊
    @[email protected] FUDEUFTUFYBNQMFMPDBM
    @[email protected] FUDEUFTUFYBNQMFMPDBM
    @[email protected] FUDEUFTUFYBNQMFMPDBM

    View Slide

  19. PQFOTIJGUKQ
    ཁ݅ϩʔυόϥϯαʔ
    • Լهͷϙʔτ൪߸ͷϩʔυόϥϯαʔ͕ඞཁ
    19
    ༻్ ϙʔτ൪߸ ର৅ϊʔυ
    ,VCFSOFUFT"1*4FSWFS #PPUTUSBQ .BTUFS
    Ϛγϯઃఆαʔόʔ #PPUTUSBQ .BTUFS
    JOHSFTTϧʔλʔͷIUUQΞΫηε༻ 8PSLFS
    JOHSFTTϧʔλʔͷIUUQTΞΫηε༻ 8PSLFS

    View Slide

  20. PQFOTIJGUKQ
    ཁ݅ϩʔυόϥϯαʔʢઃఆྫʣ
    • ઃఆྫ
    20
    ϙʔτ൪߸ 44- όοΫΤϯυ උߟ
    :FT
    CPPUTUSBQUFTUFYBNQMFMPDBM
    NBTUFSUFTUFYBNQMFMPDBM
    NBTUFSUFTUFYBNQMFMPDBM
    NBTUFSUFTUFYBNQMFMPDBM
    ϒʔτετϥοϓϓϩηεऴྃޙ
    ͸#PPUTUSBQ/PEFͷઃఆ͸ෆཁ
    ͳͷͰɺ-PBE#BMBODFSͷొ࿥͔
    Β࡟আՄೳ
    :FT
    CPPUTUSBQUFTUFYBNQMFMPDBM
    NBTUFSUFTUFYBNQMFMPDBM
    NBTUFSUFTUFYBNQMFMPDBM
    NBTUFSUFTUFYBNQMFMPDBM

    XPSLFSUFTUFYBNQMFMPDBM
    XPSLFSUFTUFYBNQMFMPDBM
    :FT
    XPSLFSUFTUFYBNQMFMPDBM
    XPSLFSUFTUFYBNQMFMPDBM

    View Slide

  21. PQFOTIJGUKQ
    ཁ݅8FCαʔόʔ
    • υΩϡϝϯτʹ͸ৄࡉͳهࡌ͸ͳ͍͕ɺΠϯετʔϧ࡞ۀʹඞ
    ཁͳϑΝΠϧΛ)551αʔόʔʹ഑ஔ͢Δඞཁ͕͋Δɻ
    • ݕূ؀ڥͰ͸OHJOYͰ8FC4FSWFSΛՔಈ͠ɺ8FCެ։σΟϨ
    ΫτϦʹඞཁͳϑΝΠϧͷ഑ஔΛߦ͏
    • ͳ͓ɺOHJOY ͷϙʔτ൪߸͸-PBE#BMBODFSͷઃఆͱඃΒͳ͍
    Α͏ʹɺݕূ؀ڥͰ͸5$1 ˠ΁มߋ͍ͯ͠Δ ˞

    21
    ˞4&-JOVYͰ[email protected]@Uʹొ࿥͞Ε͍ͯΔ൪߸Λ࢖༻

    View Slide

  22. PQFOTIJGUKQ
    ཁ݅8FCαʔόʔ
    • 8FCެ։σΟϨΫτϦͷαϯϓϧ
    22
    /usr/share/nginx/html/
    ├── ocp
    └── rhcos
    ├── ignitions
    │ ├── bootstrap.ign
    │ ├── master.ign
    │ └── worker.ign
    └── images
    ├── latest
    │ ├── bios.raw.gz -> ../release/410/rhcos-4.1.0-x86_64-metal-bios.raw.gz
    └── release
    └── 410
    └── rhcos-4.1.0-x86_64-metal-bios.raw.gz
    ü JHOJUJPOTҎԼͷJHOϑΝΠϧ͸ɺ͜ͷޙͷΠϯετʔϧ࡞ۀͰ࡞੒ͨ͠΋ͷΛ഑ஔ͠·͢
    ü $PSF04*NBHFͷݩͷϑΝΠϧ໊͸௕͍ͷͰɺϦϯΫͰ୹ॖ໊ʹ͍ͯ͠·͢ɻϦωʔϜͯ͠഑ஔͯ͠΋໰୊
    ͋Γ·ͤΜɻ

    View Slide

  23. PQFOTIJGUKQ
    23
    操作端末:
    SSH Client
    Webブラウザ
    ルーター/FW 等
    踏み台サーバー:
    ルーティング(ip_forword)
    Firewall(firewalld:NAT)
    DNS(dnsmasq:TCP/UDP 53)
    DHCP(dnsmasq :UDP 67)
    Web(nginx:TCP 8008)
    LoadBalancer(HAproxy:TCP 80,443,22623,6443)
    oc client
    openshift-install
    Master Node:#1
    HOST: master-0
    IP: 172.16.0.101
    Worker Node:#1
    HOST: worker-0
    IP: 172.16.0.104
    Bootstrap Node:
    HOST: bootstrap
    IP: 172.16.0.100
    Master Node:#2
    HOST: master-1
    IP: 172.16.0.102
    Master Node:#3
    HOST: master-2
    IP: 172.16.0.103
    Worker Node:#2
    HOST: worker-1
    IP: 172.16.0.105
    社内ネットワーク:192.168.1.0/24
    検証用ネットワーク:172.16.0.0/24
    192.168.1.21
    172.16.0.1
    192.168.1.1
    cloud.redhat.com
    quay.io
    ߏ੒ਤʢશମ֓ཁʣ
    クラスターネットワーク:10.128.0.0/14
    サービスネットワーク:172.30.0.0/16
    ドメイン名:example.local
    クラスタ名:test

    View Slide

  24. PQFOTIJGUKQ
    24
    操作端末:
    SSH Client
    Webブラウザ
    ルーター/FW 等
    踏み台サーバー:
    ルーティング(ip_forword)
    Firewall(firewalld:NAT)
    DNS(dnsmasq:TCP/UDP 53)
    DHCP(dnsmasq :UDP 67)
    Web(nginx:TCP 8008)
    LoadBalancer(HAproxy:TCP 80,443,22623,6443)
    oc client
    openshift-install
    Master Node:#1
    HOST: master-0
    IP: 172.16.0.101
    Worker Node:#1
    HOST: worker-0
    IP: 172.16.0.104
    Bootstrap Node:
    HOST: bootstrap
    IP: 172.16.0.100
    Master Node:#2
    HOST: master-1
    IP: 172.16.0.102
    Master Node:#3
    HOST: master-2
    IP: 172.16.0.103
    Worker Node:#2
    HOST: worker-1
    IP: 172.16.0.105
    社内ネットワーク:192.168.1.0/24
    検証用ネットワーク:172.16.0.0/24
    192.168.1.21 172.16.0.1
    192.168.1.1 cloud.redhat.com
    quay.io
    ߏ੒ਤʢԾ૝Ϛγϯߏ੒ʣ
    vSwitch1
    vSwitch0
    &49J
    クラスターネットワーク:10.128.0.0/14
    サービスネットワーク:172.30.0.0/16
    ドメイン名:example.local
    クラスタ名:test

    View Slide

  25. PQFOTIJGUKQ
    ʢࢀߟʣࣗ୐αʔόʔεϖοΫ
    • $16".%3Z[FO DPSFUISFBE

    • ϝϞϦ(# %%3.I[1$(#Y

    • σΟεΫ
    Ø/7.F4BNTVOH&701MVT5#
    Ø4"5"4&"("5&45%-5#
    • ϚβʔϘʔυ"43PDL#.1SP #*041

    • (16/7*%*"(5ϑΝϯϨε (161BTTUISPVHI

    • /*$ΦϯϘʔυ (JHBCJU$5%FTLUPQ"EBQUFS&91*$5
    • ిݯ$PSTBJS3.Y
    • έʔε"OUFD4PMP8IJUF
    25

    View Slide

  26. PQFOTIJGUKQ
    ࣄલ४උ
    • ॏཁͳ͜ͱ͸େ൒͕ࣄલ४උ
    Øཁ݅Λຬͨ͢Πϯϑϥͷߏங
    ØωοτϫʔΫૄ௨ͷ֬ೝ
    • ϊʔυͷ४උ
    Ø෺ཧ PSԾ૝Ϛγϯ
    • ΠϯετʔϥʔɺΠϝʔδɺϓϧγʔΫϨοτͷೖख
    • ౿Έ୆αʔόʔͷߏங
    • ૢ࡞୺຤
    26

    View Slide

  27. PQFOTIJGUKQ
    ࣄલ४උΠϯετʔϥʔɺΠϝʔδ
    • ࢀߟྫɿ0$1W
    27
    名前 インストールor配置先 実際のファイル名(参考: v4.1.3) 備考
    oc client 踏み台サーバー*8 openshift-client-linux-4.1.3.tar.gz linux,mac,windows版有り
    openshift-install 踏み台サーバー*9 openshift-install-linux-4.1.3.tar.gz linux,mac版有り
    RHCOS ISOイメージ Bootstrap,Master,Worker Node rhcos-4.1.0-x86_64-installer.iso
    OpenShiftのリリースバージョ
    ンと
    必ずしも⼀致するものではあり
    ません。*10
    BIOSファイル
    踏み台サーバー(Web公開ディ
    レクトリ)
    rhcos-4.1.0-x86_64-metal-
    bios.raw.gz
    BIOS,UEFI版有り
    バージョンについてはRHCOSと
    同様

    View Slide

  28. PQFOTIJGUKQ
    ࣄલ४උΠϯετʔϥʔɺΠϝʔδ
    28

    View Slide

  29. PQFOTIJGUKQ
    ࣄલ४උ౿Έ୆αʔόʔͷߏங
    • 3)&-ͷαʔόʔΛ୆༻ҙ͠ɺωοτϫʔΫཁ݅Ͱඞཁͳα
    ʔϏεΛશͯ͜͜ͰՔಈ
    • ౿Έ୆αʔόʔͱͯ͠උ͑Δ΂͖ɺඞཁͳཁ݅͸Լهͷ௨Γ
    ØͭͷωοτϫʔΫΧʔυʢࣾ಺ωοτϫʔΫɺݕূ༻ωοτϫʔΫ
    ʹ઀ଓʣ
    Øࣾ಺ωοτϫʔΫ͔ΒΠϯλʔωοτ΁ΞΫηε͕Մೳͳݻఆ*1ΛҰ
    ͭ
    ØΠϯϑϥཁ݅Ͱ্͛ͨαʔϏε͕ՔಈͰ͖ΔϦιʔεΛ࣋ͭϚγϯ
    29
    ৄࡉͳߏஙํ๏͸ɺϒϩάهࣄ಺ͷʮࢀߟ౿Έ୆αʔόʔͷߏஙʯΛࢀর

    View Slide

  30. PQFOTIJGUKQ
    ࣄલ४උૢ࡞୺຤
    • ࣾ಺ωοτϫʔΫ͔Β౿Έ୆αʔόʔΛܦ༝ͯ͠ɺݕূ༻ωο
    τϫʔΫʹΞΫηε͢ΔͨΊʹ࢖༻
    • ࣾ಺ωοτϫʔΫͷϧʔλʔͳͲʹݕূ༻ωοτϫʔΫ΁ͷ
    ϧʔςΟϯάͷ௥Ճ͕೉͍͠৔߹͸ɺૢ࡞୺຤ʹ੩తϧʔτΛ
    ௥Ճͯ͠ରԠ͢Δ͜ͱ
    • ૢ࡞໨త
    Ø44)$MJFOUɿ౿Έ୆αʔόʔʹϩάΠϯͯ͠PQFOTIJGUJOTUBMMίϚϯ
    υͷ࣮ߦɺଞɻ౿Έ୆αʔόʔ͔Β֤ϊʔυʹ44)઀ଓͯ͠σόοά
    ૢ࡞ɺϩά֬ೝͳͲɻ
    Ø8FCϒϥ΢β ɿ৽نΫϥελ࡞੒ޙɺ8FC$POTPMF΁ͷ઀ଓ༻ɻ
    30

    View Slide

  31. PQFOTIJGUKQ
    ࣄલ४උૢ࡞୺຤ʢ44)ΩʔϖΞ࡞੒ʣ
    • ֤ϊʔυʹରͯ͠44)ެ։ݤೝূͰΞΫηε͢ΔͨΊʹඞཁͳ
    44)ΩʔϖΞͷ࡞੒Λߦ͏
    • ࢦఆͨ͠44)ެ։ݤ͕ɺ֤ϊʔυ ˞
    ͷDPSFϢʔβʔઃ
    ఆ dTTIBVUIPSJ[[email protected] ʹ௥Ճ͞Ε·͢ɻ
    ü υΩϡϝϯτͰ͸Ωʔͷੜ੒ޙʹTTIBHFOUͷىಈͷखॱ͕͋Γ·͕͢ɺ61*ΠϯετʔϧΛ
    ͢Δ໨తʹ͓͍ͯ͸ඞਢͰ͸͋Γ·ͤΜͷͰɺඈ͹ͯ͠΋େৎ෉Ͱ͢ɻ
    31
    $ ssh-keygen -t rsa -b 4096 -N '' -f ~/.ssh/new_rsa
    ˞#PPUTUSBQɺ.BTUFSɺ8PSLFS

    View Slide

  32. PQFOTIJGUKQ
    ࣄલ४උૢ࡞୺຤
    • ࣾ಺ωοτϫʔΫ͔Β౿Έ୆αʔόʔΛܦ༝ͯ͠ɺݕূ༻ωο
    τϫʔΫʹΞΫηε͢ΔͨΊʹ࢖༻
    • ࣾ಺ωοτϫʔΫͷϧʔλʔͳͲʹݕূ༻ωοτϫʔΫ΁ͷ
    ϧʔςΟϯάͷ௥Ճ͕೉͍͠৔߹͸ɺૢ࡞୺຤ʹ੩తϧʔτΛ
    ௥Ճͯ͠ରԠ͢Δ͜ͱ
    • ૢ࡞໨త
    Ø44)$MJFOUɿ౿Έ୆αʔόʔʹϩάΠϯͯ͠PQFOTIJGUJOTUBMMίϚϯ
    υͷ࣮ߦɺଞɻ౿Έ୆αʔόʔ͔Β֤ϊʔυʹ44)઀ଓͯ͠σόοά
    ૢ࡞ɺϩά֬ೝͳͲɻ
    Ø8FCϒϥ΢β ɿ৽نΫϥελ࡞੒ޙɺ8FC$POTPMF΁ͷ઀ଓ༻ɻ
    32

    View Slide

  33. PQFOTIJGUKQ
    0$161*Πϯετʔϧ
    Πϯετʔϧ࡞ۀৄࡉ
    33

    View Slide

  34. PQFOTIJGUKQ
    61*Πϯετʔϧͷϑϩʔ
    • ΠϯετʔϧίϯϑΟάϑΝΠϧͷ࡞੒
    • *HOJUJPOϑΝΠϧͷ࡞੒
    • 3)$04ͷϚγϯͷ࡞੒ʢϊʔυͷىಈʣ
    Øखಈ·ͨ͸19&#PPU
    • Ϋϥελʔͷ࡞੒
    • Ϋϥελʔ΁ͷϩάΠϯ
    • Ϛγϯͷ$43ͷঝೝ
    • $MVTUFS0QFSBUPSͷॳظઃఆ
    • Πϯετʔϧͷ׬ྃ
    34

    View Slide

  35. PQFOTIJGUKQ
    ΠϯετʔϧίϯϑΟάϑΝΠϧͷ࡞੒
    • ΠϯετʔϧσΟϨΫτϦͷ࡞੒ͱɺΠϯετʔϧઃఆϑΝΠ
    ϧ JOTUBMMDPOGJHZBNMΛ࡞੒͢Δ
    ΠϯετʔϧσΟϨΫτϦCBSFNFUBMͷྫ
    35
    $ mkdir bare-metal
    $ vi install-config.yaml
    $ cp install-config.yaml bare-metal/

    View Slide

  36. PQFOTIJGUKQ
    ΠϯετʔϧίϯϑΟάϑΝΠϧͷ࡞੒
    • JOTUBMMDPOGJHZBNM
    ØυϝΠϯ໊FYBNQMFMPDBM
    ØΫϥελ໊UFTU
    • QVMM4FDSFUࣄલʹೖखͨ͠಺༰Λ
    షΓ෇͚
    • TTI,FZࣄલ४උͰ࡞੒ͨ͠44)
    ΩʔϖΞʢ[email protected]
    ʣΛషΓ෇͚
    36
    apiVersion: v1
    baseDomain: example.local
    compute:
    - hyperthreading: Enabled
    name: worker
    replicas: 0
    controlPlane:
    hyperthreading: Enabled
    name: master
    replicas: 3
    metadata:
    name: test
    networking:
    clusterNetworks:
    - cidr: 10.128.0.0/14
    hostPrefix: 23
    networkType: OpenShiftSDN
    serviceNetwork:
    - 172.30.0.0/16
    platform:
    none: {}
    pullSecret: '{"auths":{"cloud.openshift.com":{"auth":
    ....
    ==","email":"[email protected]"}}}'
    sshKey: 'ssh-rsa AAAA...'

    View Slide

  37. PQFOTIJGUKQ
    *HOJUJPOϑΝΠϧͷ࡞੒
    • Πϯετʔϧʹඞཁͳ*HOJUJPOઃఆϑΝΠϧͳͲΛ࡞੒
    Ø͜ͷίϚϯυͷ࣮ߦޙʹɺΠϯετʔϧσΟϨΫτϦ಺ͷJOTUBMM
    DPOGJHZBNM͸࡟আ͞ΕΔ ˞

    37
    $ openshift-install create ignition-configs --dir=bare-metal
    bare-metal
    ├── auth
    │ ├── kubeadmin-password
    │ └── kubeconfig
    ├── bootstrap.ign
    ├── master.ign
    ├── metadata.json
    └── worker.ign
    ˞ࣄલʹJOTUBMMDPOGJHZBNMΛΠϯετʔϧσΟϨΫτϦҎ֎ʹอଘ͓ͯ͘͜͠ͱ
    ࡞੒֤ͨ͠JHOJUJPOϑΝΠϧ͸8FC4FSWFSͷ
    ެ։σΟϨΫτϦʹίϐʔ͢Δ

    View Slide

  38. PQFOTIJGUKQ
    3)$04Ϛγϯͷ࡞੒
    • *40·ͨ͸19&CPPUͰىಈ
    Ø*40Πϯετʔϧʹඞཁͳ৘ใΛखೖྗ
    Øʢਪ঑ʣ19&CPPUిݯΦϯͰࣗಈతʹΠϯετʔϧ࣮ߦ
    • ύϥϝʔλʔ
    ØΠϯετʔϧઌͷϒϩοΫσόΠεྫ
    TEB
    Ø3"8Πϝʔδͷ63-8FCαʔόʔͷϑΝΠϧ഑ஔ৔ॴΛࢦఆ
    Ø*HOJUJPOϑΝΠϧͷ63-8FCαʔόʔϑΝΠϧ഑ஔ৔ॴΛࢦఆ
    Ø Φϓγϣϯ
    *1ΞυϨε%)$1PS੩త*1ΞυϨε
    • ىಈ࣌ʹࢦఆͨ͠*HOJUJPOϑΝΠϧͰɺϊʔυͷछྨ͕ܾ·Δ
    • WΑΓ੩త*1ΞυϨεͷࢦఆ͕Մೳɻαϙʔλϒϧʹɻ
    38

    View Slide

  39. PQFOTIJGUKQ
    Ϋϥελʔͷ࡞੒
    • ϒʔτετϥοϓϓϩηεͷਐߦͱ׬ྃΛϞχλϦϯά
    • ԼهͷΑ͏ͳදࣔʹͳΔͱϒʔτετϥοϓϓϩηεͷ׬ྃ
    39
    $ openshift-install --dir=bare-metal wait-for bootstrap-complete
    openshift-install --dir=bare-metal wait-for bootstrap-complete
    INFO Waiting up to 30m0s for the Kubernetes API at
    https://api.test.example.local:6443...
    INFO API v1.13.4+d4417a7 up
    INFO Waiting up to 30m0s for bootstrapping to complete...
    INFO It is now safe to remove the bootstrap resources
    ˞෼Ҏ্ֻ͔ͬͯ΋׬ྃ͠ͳ͍৔߹͸ɺࣄલ४උ͕ෆ଍͔ؒҧ͍ͬͯΔՄೳੑ͕ߴ͍Ͱ͢ɻ
    ˞్தͰࢭΊͯ΍Γ௚͢৔߹͸ɺ3)$04ͷىಈ͔Β΍Γ௚͠·͢

    View Slide

  40. PQFOTIJGUKQ
    Ϋϥελʔ΁ͷϩάΠϯ
    • LVCFBENJOೝূ৘ใΛΤΫεϙʔτ͠ɺσϑΥϧτγεςϜ
    ϢʔβʔͰϩάΠϯͰ͖Δ͜ͱΛ֬ೝ
    40
    $ export KUBECONFIG=bare-metal/auth/kubeconfig
    $ oc whoami
    system:admin
    PDXIPBNJͷ݁ՌɺTZTUFNBENJO͕ฦͬͯ͘Ε͹׬ྃͰ͢ɻ ϒʔτετϥοϓϓϩηεͷ׬
    ྃ௚ޙ͙ͩ͢ͱɺΤϥʔ͕ฦͬͯ͘Δ͜ͱ͕͋Γ·͢ɻͦͷ৔߹͸ɺগ࣌ؒ͠Λஔ͍ͯ࠶౓࣮ߦ
    ͠·͢ɻ

    View Slide

  41. PQFOTIJGUKQ
    Ϛγϯͷ$43ͷঝೝ
    • ֤ϊʔυ͕Ϋϥελʹ௥Ճ͞Εͨࡍʹ࡞੒͞ΕΔূ໌ॻॺ໊ཁ
    ٻ͕ɺશͯঝೝ͞Ε͍ͯΔ͔֬ೝ
    41
    $ oc get nodes
    NAME STATUS ROLES AGE VERSION
    master-0 Ready master 63m v1.13.4+b626c2fe1
    master-1 Ready master 63m v1.13.4+b626c2fe1
    master-2 Ready master 64m v1.13.4+b626c2fe1
    worker-0 NotReady worker 76s v1.13.4+b626c2fe1
    worker-1 NotReady worker 70s v1.13.4+b626c2fe1

    View Slide

  42. PQFOTIJGUKQ
    Ϛγϯͷ$43ͷঝೝ
    • ূ໌ॻॺ໊ཁٻͷঝೝঢ়ଶΛ֬ೝ͠·͢ɻ
    • ࣮ߦ݁ՌྫʢOPEFʣ
    42
    $ oc get csr
    NAME AGE REQUESTOR CONDITION
    csr-c7pnq 10m system:node:master-1 Approved,Issued
    csr-cfrqj 9m56s system:node:master-0 Approved,Issued
    csr-cfxh6 10m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
    csr-fdx6l 10m system:node:master-2 Approved,Issued
    csr-p468w 10m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
    csr-zdzqr 10m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued

    View Slide

  43. PQFOTIJGUKQ
    $MVTUFS0QFSBUPSͷॳظઃఆ
    • શͯͷ0QFSBUPS͕ਖ਼ৗʹՔಈ͍ͯ͠Δ͜ͱ֬ೝ͠·͢ɻ
    ࠷ऴతʹશͯͷ0QFSBUPSͷεςʔλεʹ͓͍ͯ"7"*-"#-&͕
    536&ɺ%&(3"%&%͕'"-4&ʹͳΔ͜ͱ͕ΰʔϧ
    43
    $ oc get clusteroperators

    View Slide

  44. PQFOTIJGUKQ
    Πϯετʔϧͷ׬ྃ
    • શͯͷ0QFSBUPS͕ਖ਼ৗʹՔಈ͍ͯ͠Δ͜ͱ֬ೝޙɺԼهͷίϚϯυΛ࣮
    ߦ
    • ࠷ޙʹදࣔ͞ΕͨVTFSLVCFBENJO QBTTXPSE9999999999
    9999999999Λ࢖༻ͯ͠ɺ8FC$POTPMF΁ϩάΠϯ͠·͢ɻ
    44
    $ openshift-install --dir=bare-metal wait-for install-complete
    INFO Waiting up to 30m0s for the cluster at https://api.test.example.local:6443 to initialize...
    INFO Waiting up to 10m0s for the openshift-console route to be created...
    INFO Install complete!
    INFO To access the cluster as the system:admin user when using 'oc', run 'export
    KUBECONFIG=/root/OCP/bare-metal/auth/kubeconfig'
    INFO Access the OpenShift web-console here: https://console-openshift-console.apps.test.example.local
    INFO Login to the console with user: kubeadmin, password: XXXXX-XXXXX-XXXXX-XXXXX

    View Slide

  45. PQFOTIJGUKQ
    σϞ
    45

    View Slide

  46. PQFOTIJGUKQ
    ʢࢀߟʣ0$1ϩάΠϯը໘
    46

    View Slide

  47. PQFOTIJGUKQ
    ʢࢀߟʣ0$1ίϯιʔϧը໘
    47

    View Slide

  48. PQFOTIJGUKQ
    ʢࢀߟʣ$MVTUFS.BOBHFSͷը໘
    48

    View Slide

  49. PQFOTIJGUKQ
    ωοτϫʔΫ੍ݶ؀ڥʹ͓͚
    ΔΠϯετʔϧ
    49

    View Slide

  50. PQFOTIJGUKQ
    1SPYZ؀ڥԼͰͷΠϯετʔϧ
    • 0QFO4IJGU͔Β͸৽ͨʹ1SPYZͷαϙʔτ͕௥Ճ
    • Πϯετʔϧ͓࣌ΑͼΠϯετʔϧޙͷΫϥελʔ͔Βɺ
    1SPYZܦ༝ͰͷΠϯλʔωοτ௨৴͕ՄೳͱͳΓ·ͨ͠
    • ࢀߟ0QFO4IJGUʹ͓͚Δ1SPYZ؀ڥԼͰͷΠϯετʔϧ
    50

    View Slide

  51. PQFOTIJGUKQ
    ඇ઀ଓ؀ڥʹ͓͚ΔΠϯετʔϧ
    • Ϋϥελʔ͕Πϯλʔωοτ΁Ұ੾઀ଓͰ͖ͳ͍؀ڥͰͷΠϯετʔϧ
    ํ๏
    • ଟ͘ͷ࡞ۀྔ΍ߟྀࣄ߲ɺΤϥʔϋϯυϦϯά͕ඞཁͳͨΊڧ͘ඇਪ঑
    • ࢀߟ
    Ø0QFO4IJGUʹ͓͚ΔωοτϫʔΫ੍ݶ؀ڥԼͰͷΠϯετʔϧ
    Ø0QFO4IJGUʹ͓͚ΔωοτϫʔΫ੍ݶ؀ڥԼͰͷΠϯετʔϧʢ4BNQMFT
    0QFSBUPSઃఆฤʣ
    51

    View Slide

  52. PQFOTIJGUKQ
    ͦͷଞͷߏ੒ํ๏
    52

    View Slide

  53. PQFOTIJGUKQ
    OPEFΫϥελʔ
    • 8PSLFS/PEFͳ͠Ͱɺ.BTUFS/PEF୆ͰΫϥ
    ελʔΛߏ੒
    Ø.BTUFS/PEF͕8PSLFS/PEFΛ݉༻͠ɺશͯͷ1PE
    ͕.BTUFS/PEF্ͰՔಇ
    • W͔Β5FDI1SFWJFXѻ͍
    Øຊ൪؀ڥͰͷ࢖༻͸ඇਪ঑
    • ։ൃɺݕূ؀ڥͳͲͷ༻్Ͱ͸࠷௿ඞཁϊʔυ਺
    ͕ݮΔͷͰಋೖ͕͠қ͍
    • ࢀߟɿ 0QFO4IJGUʹ͓͚ΔϊʔυϕΞϝλϧσϓϩΠ
    53
    3ノードクラスター

    View Slide

  54. PQFOTIJGUKQ
    "MMJO0OFߏ੒
    • .BTUFS/PEF୆ͷΈͰ0$1Λ࣮ߦ͢Δߏ੒
    • ඇαϙʔτߏ੒ʢ51Ͱ΋ͳ͍ʣɻࣗݾ੹೚Ͱɻ
    • 3FWBNQFE0QFO4IJGU"MMJO0OF "*0
    GPS-BCTBOE'VO
    Ø্هͷCMPH͸৘ใͷൈ͚མ͕ͪ͋ΔͷͰɺݩιʔεͷ(JU)VCΛࢀর
    ü0$1"MM*O0OF 61*NPEF

    • ঢ়ଶͱͯ͠͸$PEF3FBEZ$POUBJOFSTͱಉ͡
    Ø֎෦͔ΒͷΞΫηε͕Մೳͳ఺͕ҟͳΔ
    54
    All-in-One

    View Slide

  55. PQFOTIJGUKQ
    0,%
    55

    View Slide

  56. PQFOTIJGUKQ
    8IZ0,%ʁ
    • 0,%5IF0SJHJO$PNNVOJUZ%JTUSJCVUJPOPG
    ,VCFSOFUFT
    • 0,%͸,VCFSOFUFTͷσΟετϦϏϡʔγϣϯͷͭɻ044ͷ
    ίϛϡχςΟϓϩδΣΫτɻ
    • 3FE)BU0QFO4IJGUʹ৮ΕΔબ୒ࢶͷҰͭ
    • 0QFO4IJGUͷಋೖલʹࣗ෼Ͱ࢖͑Δ؀ڥʹ0,%ΛೖΕͯΈΔ
    ØΦϯϓϨͷ؀ڥʹಋೖ
    56
    https://www.okd.io/#v3

    View Slide

  57. PQFOTIJGUKQ
    0$1SFMBUJPOTIJQUP0,%
    57

    View Slide

  58. PQFOTIJGUKQ
    0,%ͱ0QFO4IJGUͷҧ͍
    • 0,%ɿ
    Øϕʔε04ɿ'FEPSB$PSF04
    Ø৽ػೳɺόάमਖ਼ɺ͓ΑͼηΩϡϦςΟߋ৽͕ΞΫςΟϒͳ։ൃϒϥϯν
    ʹ૊Έࠐ·Εɺ࠷৽൛ͱͯ͠ϦϦʔε
    ØίϛϡχςΟαϙʔτ
    • 0QFO4IJGUɿ
    Øϕʔε04ɿ3FE)BU&OUFSQSJTF-JOVY$PSF04
    Ø௕ظؒͷϝϯςφϯεͱαϙʔτɺޙํޓ׵ੑɺόάϑΟοΫεͱηΩϡ
    ϦςΟύονͷఏڙɻηΩϡϦςΟରԠ΍ػೳͷόοΫϙʔτ
    Ø3FE)BU͕ࣾαϙʔτ
    58

    View Slide

  59. PQFOTIJGUKQ
    'FEPSB$PSF04 '$04

    • 0,%ͷϕʔεͱͳΔ04
    • Πϛϡʔλϒϧͳ04
    • *HOJUJPOϑΝΠϧͰ؀ڥઃఆ
    • σόοά໨తҎ֎ɺϊʔυ΁ͷ௚઀ΞΫηε͸ෆཁ
    • ֤छઃఆมߋ͸0,%Ϋϥελʔ্͔Β௚઀ߦ͏
    • ύοέʔδ؅ཧ͸SQNPTUSFFΛ࢖༻
    59
    https://getfedora.org/en/coreos/

    View Slide

  60. PQFOTIJGUKQ
    0,%ͷ։ൃঢ়گ
    • ։ൃ৔ॴ
    ØެࣜαΠτɿIUUQTXXXPLEJP
    Ø(JU)VCɿIUUQTHJUIVCDPNPQFOTIJGUPLE
    Ø4MBDLɿ,VCFSFOFUFT4MBDLPOPQFOTIJGUEFW
    • 0QFO4IJGUͷ("͔Β͔ͳΓ஗Ε͍ͯΔঢ়گɻ
    Ø0QFO4IJGUͷ("͸೥݄
    • υΩϡϝϯτ͸0,%-BUFTU͕0,%ʹରԠ
    Ø0$1ͱಉ౳ͷ಺༰ʹଗ͍ͭͭ͋Δ
    • 0,%ͷCFUB൛͕ެ։த
    Ø࠷৽͸WCFUB
    60
    ˞࣌఺

    View Slide

  61. PQFOTIJGUKQ
    ʢࢀߟʣ0,%ίϯιʔϧը໘
    61

    View Slide

  62. PQFOTIJGUKQ
    ʢࢀߟʣ0,%ϊʔυৄࡉ
    62

    View Slide

  63. PQFOTIJGUKQ
    ·ͱΊ
    63

    View Slide

  64. PQFOTIJGUKQ
    ·ͱΊ
    • ΦϯϓϨͰ0$1Λಋೖ͢ΔͨΊͷɺ61*Πϯετʔϧʹ͍ͭ
    ͯ঺հ
    • ෳࡶͳલఏ৚݅ΛΫϦΞ͢ΔͨΊʹɺ౿Έ୆αʔόʔΛߏங͢
    Δͱಋೖ͕άοͱָʹͳΔ
    • ωοτϫʔΫ੍ݶ؀ڥԼʹ͓͍ͯ΋0$1ͷಋೖ͸Մೳ
    • 0QFO4IJGUΛؾ݉Ͷͳ͘ࢼͯ͠Έ͍ͨํ͸0,%ʹτϥΠʂ
    ΦϯϓϨͰͷ0QFO4IJGUͷಋೖʹ௅ઓͯ͠Έ·ͤΜ͔ʁ
    64

    View Slide

  65. PQFOTIJGUKQ
    ࢀߟϦϯΫ
    65

    View Slide

  66. PQFOTIJGUKQ
    0QFO4IJGUͷ৘ใ
    • ੡඼υΩϡϝϯτ
    o 3FE)BU੡඼υΩϡϝϯτ
    o 0QFO4IJGU%PDVNFOUBUJPO
    66

    View Slide

  67. PQFOTIJGUKQ
    0QFO4IJGUͷ৘ใ
    • 0QFO4IJGUͷബ͍ຊ
    o λΠτϧɿ3FE)BU0QFO4IJGUೖ໳
    &OUFSQSJTF,VCFSOFUFT΁ͷϑΝʔετεςοϑ㿇
    ØϨουϋοτࣾһ༗ࢤͰ࡞੒
    Øͪ͜ΒͰແྉ഑෍த
    • ੺๧ΤϯδχΞϒϩά
    o Ϩουϋοτͷࣾһ͕༗ࢤͰॻ͍͍ͯΔϒϩά
    ØIUUQTSIFCIBUFOBCMPHDPN
    Øλάɿ3FE)BU0QFO4IJGUɺ3FE)BU0QFO4IJGU
    $POUBJOFS1MBUGPSN
    o ຊࢿྉͰ঺հͨ͠಺༰ͷݩωλ͸ͪ͜Βͷهࣄ
    67

    View Slide

  68. PQFOTIJGUKQ
    ΦϯϓϨΠϯετʔϧࣄྫ঺հ
    • 0$1PO/6$
    o 0QFO4IJGUΛ*OUFMͷ௒খܕ1$ʮ/6$ʯʹΠϯετʔϧ͢Δ
    2JJUB
    • 61*GPSW4QIFSF
    o 7.XBSFW4QIFSF؀ڥʹSIDPTPWBΛ࢖ͬͯ0QFO4IJGU؀ڥΛ࡞ͬ
    ͯΈΔ c೔ৗܥΤϯδχΞͷ5FDI#MPH
    • 0,%
    o 0,%PO'FEPSB$PSF04 '$04
    ΛϕΞϝλϧ61*Πϯετʔϧ
    ͯ͠0QFO4IJGUϚϧνϊʔυΫϥελΛࢼͯ͠Έͨ [BLJXPSLMPH
    68

    View Slide

  69. PQFOTIJGUKQ
    5IBOLZPV
    69

    View Slide