Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up
for free
コンテナ基盤を支えるHashiCorpソフトウェア / HashiCorp Softwares on Container Base
linyows
September 11, 2018
Technology
6
4k
コンテナ基盤を支えるHashiCorpソフトウェア / HashiCorp Softwares on Container Base
2018.09.11 DevOpsを支える今話題のHashiCorpツール群について(HashiCorp Meetup 3rd)でお話しした資料です
linyows
September 11, 2018
Tweet
Share
More Decks by linyows
See All by linyows
linyows
2
180
linyows
1
230
linyows
0
110
linyows
0
330
linyows
0
490
linyows
1
800
linyows
7
1.4k
linyows
1
3.4k
linyows
0
110
Other Decks in Technology
See All in Technology
sansandsoc
0
490
viva_tweet_x
1
450
kahara33
0
130
kaniza
0
120
hayatan
0
210
akitok_
2
800
ido_kara_deru
1
150
nitya
0
340
takumanakagame
1
320
watanabeyu
0
160
khrd
1
690
hmatsu47
0
110
Featured
See All Featured
geoffreycrofte
21
940
mthomps
38
2.3k
paulrobertlloyd
71
3.6k
michaelherold
225
8.5k
zenorocha
296
40k
chrislema
231
16k
trishagee
24
2.6k
addyosmani
311
21k
scottboms
251
11k
chrislema
173
14k
jasonvnalue
81
8.1k
hursman
106
9.3k
Transcript
খాԝ(.01FQBCP *OD %FW0QTΛࢧ͑Δࠓͷ)BTIJ$PSQπʔϧ܈ʹ͍ͭͯ )BTIJ$PSQ.FFUVQSE ίϯςφج൫Λࢧ͑Δ )BTIJ$PSQιϑτΣΞ
)BTIJ$PSQ.FFUVQSE (.0ϖύϘ ϓϦϯγύϧΤϯδχΞ !MJOZPXT CMPHUPNPIJTBPEBDPN
)BTIJ$PSQ.FFUVQSE ࠷ۙͷ͓ࣄ ϖύϘݚڀॴͱभେֶ͕ڞಉݚڀ
)BTIJ$PSQ.FFUVQSE দຊ྄հʹΑΔจʢ%*$0.0༧ߘʣਫ਼៛ʹ੍ޚՄೳͳ߃ৗੑͷ͋ΔߴूੵϚϧνΞΧϯτܕͷϝʔϧج൫ IUUQTSBOEQFQBCPDPNQBQFSTEJDPNPQSPDFFEJOHNBUTVNPUPSZQEG ࠷ۙͷ͓ࣄ ओʹ'BTU$POUBJOFSʹΑΔϝʔϧج൫ݚڀ։ൃɺ࠷࣮ۙྫΛ(JU)VCͰެ։ IUUQTHJUIVCDPN'BTU$POUBJOFS
)BTIJ$PSQ.FFUVQSE ࠷ۙͷ͓ࣄ 7BVMUͷ8PSLTIPQΛ͚ࣾʹ։࠵ɻҎԼͦͷհهࣄ IUUQTUFDIQFQBCPDPNWBVMUXPSLTIPQ
)BTIJ$PSQ.FFUVQSE 8&# %#13&44WPM )BTIJ$PSQ7BVMUͷهࣄدߘ Ԭͷ(PMBOHίϛϡχςΟ 'VLVPLBHPͷओ࠵ͷਓ MJOVYϢʔβͷ໊લղܾΛ (JU)VC͔ΒϚοϐϯά͢Δ ιϑτΣΞͷ։ൃ
)BTIJ$PSQ.FFUVQSE ίϯςφج൫Λࢧ͑Δ)BTIJ$PSQιϑτΣΞ
)BTIJ$PSQ.FFUVQSE ๏ Ұൠతͳ,VCFSOFUFT%PDLFS͍ͬͯ·ͤΜ ๏ -9%ͷΑ͏ͳγεςϜίϯςφͰ͋Γ·ͤΜ ๏ ಠࣗίϯςφڥΛఏڙ͢ΔଆͷͰ͢ ๏ 0PIPSJ'BTU$POUBJOFSɺ)BDPOJXBΛ͍ͬͯ·͢ લఏίϯςφج൫ͱ͍ͬͯʜ
)BTIJ$PSQ.FFUVQSE 0PIPSJ'BTU$POUBJOFS)BDPOJXB ☺
)BTIJ$PSQ.FFUVQSE 0PIPSJ'BTU$POUBJOFS)BDPOJXB ☺☺☺☺☺ ͜ΕΒಠࣗ։ൃͨ͠ͷͰ͢ ΞʔΩςΫνϟ ίϯςφϥϯλΠϜ ΦʔέετϨʔλʔ Կʹʁ
)BTIJ$PSQ.FFUVQSE ϩϦϙοϓʂϚωʔδυΫϥυ
)BTIJ$PSQ.FFUVQSE ๏ ίϯςφϕʔεͷ1BB4 ๏ ӡ༻͖ͷΫϥυͰΫϥυدΓͷϨϯλϧαʔό ๏ ίΞػೳͷΦʔτεέʔϧίϯςφෛՙʹԠͯ͡εέʔϧΞτ͠ෛՙ ܰݮΑΓεέʔϧΠϯ ๏ ఆ֎ͷ༻ྔϝʔϧ௨ར༻੍ݶͳͲͷઃఆ͕Մೳ
ϩϦϙοϓʂϚωʔδυΫϥυ
)BTIJ$PSQ.FFUVQSE ๏ ҆Ձͳίϯςφڥͷఏڙʹίϯςφ͕ߴूੵͰ͋Δඞཁ͕͋ΔʢϨϯ αόϩϯάςʔϧతʣ ๏ Ϣʔβཧͷίϯςφ͕ܧଓతʹ҆શͰ͋Δඞཁ͕͋ΔʢϛυϧΣΞ ґଘϥΠϒϥϦ͕ఆظతʹ࠷৽ʣ ๏ ίϯςφϦιʔεݖݶʹରͯ͠ॊೈͳઃఆ͕ՄೳͰ͋Δ͜ͱͱɺͦΕ Β͕ೳಈతͰ͋Δඞཁ͕͋Δʢίϯςφ͕ࣗಈతʹϦιʔεมߋʣ
ͳͥಠࣗ։ൃͯ͠͏ͷ͔
)BTIJ$PSQ.FFUVQSE ཁٻΛຬͨͨ͢Ίʹඞཁͩͬͨ ֤ٕज़ৄࡉʹ͍ͭͯݕࡧͯ͠Έ͍ͯͩ͘͞
)BTIJ$PSQ.FFUVQSE ίϯςφج൫Λࢧ͑Δ ϩϦϙοϓʂϚωʔδυΫϥυΛࢧ͑Δ )BTIJ$PSQιϑτΣΞ
)BTIJ$PSQ.FFUVQSE ·ͣγεςϜશମ૾
)BTIJ$PSQ.FFUVQSE $BDIF 1SPYZ "1* 4ZTUFN0WFSWJFX 4FDSFU.BOBHFS 4FSWJDF.BOBHFS .POJUPS "$.& #FIBWJPS5FTUFS
4.51 .FUSJDT 4DIFEVMFS 8FC "1* %# $BDIF +PC -# 1SPYZ $PNQVUF %JTQBUDIFS -# 4UPSBHF %# 4UBSUFS 1SPYZ &OW1SPEVDUJPO 4UBHJOH %FWFMPQNFOU "MFSU.BOBHFS )PTUJOH #VTJOFTT #BTUJPO %# 0QFO4UBDL #BSFNFUBM $.%#
)BTIJ$PSQ.FFUVQSE ϛυϧΣΞΛՃͯ͠ΈΔ
)BTIJ$PSQ.FFUVQSE $BDIF 1SPYZ "1* 4FDSFU.BOBHFS 4FSWJDF.BOBHFS .POJUPS "$.& #FIBWJPS5FTUFS 4.51
.FUSJDT 4DIFEVMFS 8FC "1* %# $BDIF +PC -# 1SPYZ $PNQVUF %JTQBUDIFS -# 4UPSBHF %# 4UBSUFS 1SPYZ "MFSU.BOBHFS )PTUJOH #VTJOFTT #BTUJPO %# 0QFO4UBDL #BSFNFUBM $.%# 4ZTUFN0WFSWJFX &OW1SPEVDUJPO 4UBHJOH %FWFMPQNFOU
)BTIJ$PSQ.FFUVQSE ࣍ʹσϓϩΠϑϩʔ
)BTIJ$PSQ.FFUVQSE 0QFO4UBDL #BSFNFUBM .""4 ,OJGF;FSP $* %FQMPZ'MPX
)BTIJ$PSQ.FFUVQSE ๏ 0QFO4UBDLͱ#BSFNFUBMͷϋΠϒϦουڥ ๏ 1BDLFSͰ࡞ΔΠϝʔδ࠷খݶͰશϩʔϧڞ௨Խͯ͠༻ ๏ 5FSSBGPSNͷ1SPWJTJPOFS༻ͤͣ,OJGF;FSPΛ͏ ๏ ։ൃڥ7BHSBOUʢϩʔϧ͕ଟ͍ͷͰZNMཧͰ͖ΔQMVHJOΛ༻ʣ ๏
සൟʹൃੜ͢Δେ͖ͳ༷มߋͱεςʔτϑϧͳϩʔϧଟ͍͜ͱ͔Β *NNVUBCMF*OGSBΛࣺͯΔઓུ શମతಛ
)BTIJ$PSQ.FFUVQSE ๏ 7BHSBOU ๏ 1BDLFS ๏ 5FSSBGPSN ๏ $POTVM ๏
7BVMU ๏ /PNBEʢͷͪʹKSBMMJTPOHPXPSLFSTͱബ͍"1*ʹมߋʣ ར༻͍ͯ͠Δ)BTIJ$PSQιϑτΣΞ
)BTIJ$PSQ.FFUVQSE αʔϏεܧଓʹͳͯ͘ͳΒͳ͍ ѹతײँ
)BTIJ$PSQ.FFUVQSE 5FSSBGPSN༻ͷಛ
)BTIJ$PSQ.FFUVQSE ๏ ՄೳͳݶΓNPEVMFΛ࠶ར༻Ͱ͖ΔΑ͏ʹ͍ͯ͠Δ ๏ ॳUGTUBUFΛHJUཧ͍ͯͯ͠ෳਓͰ࡞ۀ͢Δͱҋ͔͠ͳ͙͘͢ʹ4ʹมߋ ๏ 8PSLTQBDF1SPEVDUJPOͱ4UBHJOHͰ͍ͬͯΔ ๏ DMPVEJOJUͰ4MBDL௨ɺར༻ऀґଘ෦Λೖ͍ͯ͠Δ ๏
MJGFDZDMFͷઃఆࣄނࢭʹඞਢ ๏ $*ͰGNU͢ΔΑ͏ʹ͍ͯ͠Δ 5FSSBGPSN
)BTIJ$PSQ.FFUVQSE module "reserved_vip" { source = "../reserved_vip" count = "${var.int_vip_count}"
name = "${var.role}" network = "${var.network}" } module "pairaddress_port" { source = "../pairaddress_port" count = "${var.count}" network = "${var.network}" security_group_ids = ["${values(var.security_groups)}"] use_floating_ip = false allowed_ip_address = "${data.openstack_networking_subnet_v2.subnet.cidr}" role = "${var.role}" } resource "openstack_compute_instance_v2" "instance" { lifecycle { ignore_changes = ["user_data", "key_pair", "image_name", "availability_zone"] } count = "${var.count}" name = "${terraform.env != "staging" ? "" : "staging-"}${var.role}-${count.index + var.count_offset + 1}.${var.domain}" image_name = "${var.image_name}" flavor_name = "${var.flavor_name}" key_pair = "${var.key_pair}" availability_zone = "${var.availability_zones[(count.index + var.count_offset) % length(var.availability_zones)]}" security_groups = ["${keys(var.security_groups)}"] user_data = "${data.template_file.init.rendered}" network { port = "${element(module.pairaddress_port.ids, count.index)}" modules/ ├── instance │ ├── main.tf │ ├── outputs.tf │ └── variables.tf ├── instance_with_extvip │ ├── main.tf │ ├── outputs.tf │ └── variables.tf ├── instance_with_intvip │ ├── main.tf │ ├── outputs.tf │ └── variables.tf ├── pairaddress_port │ ├── main.tf │ ├── outputs.tf │ └── varaibales.tf ├── reserved_vip │ ├── main.tf │ ├── outputs.tf │ └── variables.tf └── volume ├── main.tf └── variables.tf 5FSSBGPSNࡉ͔۠ͬͨ͘NPEVMFͷྫ
)BTIJ$PSQ.FFUVQSE 5FSSBGPSNࡉ͔۠ͬͨ͘NPEVMFͷྫ module "api" { count = "${terraform.env == "staging"
? 3 : var.api_count}" source = "./modules/instance" role = "api" flavor_name = "c1.large" network_id = "${openstack_networking_network_v2.lan.id}" security_groups = { "${openstack_networking_secgroup_v2.base.name}" = "${openstack_networking_secgroup_v2.base.id}" "${openstack_networking_secgroup_v2.api.name}" = "${openstack_networking_secgroup_v2.api.id}" } } module "secretmanager" { count = "${terraform.env == "staging" ? 2 : var.vault_count}" source = "./modules/instance_with_intvip" role = "secretmanager" flavor_name = "c1.medium" network = "${var.nyah["tenant_name"]}-lan" security_groups = { "${openstack_networking_secgroup_v2.base.name}" = "${openstack_networking_secgroup_v2.base.id}" "${openstack_networking_secgroup_v2.secretmanager.name}" = "${openstack_networking_secgroup_v2.secretmanager.id}" } } ڞ௨ԽʹΑΓ*OTUBODFͷఆ͕ٛγϯϓϧʹ
)BTIJ$PSQ.FFUVQSE $POTVM༻ͷಛ
)BTIJ$PSQ.FFUVQSE ๏ ϝΠϯαʔϏεσ ΟεΧόϦʢͪΖΜࢹʣ ๏ .BDLFSFMͱׂ୲֎ܗࢹ͔αʔϏεࢹ͔ ๏ ϗετͷ໊લղܾ$POTVM%/4ͱ6OCPVOEΛ༻ ๏ 7BVMUͷετϨʔδόοΫΤϯυͱͯ͠ར༻
๏ શϊʔυʹ$POTVM"HFOUͱαʔϏεࢹͰ༻͢Δ1SPNFUIFVTͷ DPOTVM OPEF CMBDLCPYFYQPSUFS͕ೖ͍ͬͯΔ $POTVM
)BTIJ$PSQ.FFUVQSE $POTVM%/4ͷศར༻๏ $ cat ~/.ssh/config … Host bastion-1.ohr HostName xxx.xxx.xxx.xxx
User linyows Host *.ohr !bastion-1.ohr !bastion-2.ohr !staging-*.ohr !*baremetal.ohr ProxyCommand ssh -W "$(basename "$(sed -E "s/.ohr/.node.consul/"<<<"%h")")":%p bastion-1.ohr User linyows $ ssh app-1.ohr __ ___ __ _____ _/ / / _ \/ // / _ `/ _ \ /_//_/\_, /\_,_/_//_/ /___/ ubuntu https://github.pepabo.com/tech/packer-templates (c) GMO Pepabo, Inc. linyows@app-1:~$ ౿ΈαʔόͰ໊લղܾ͢Δ͜ͱͰଟஈ44)Λศརʹ
)BTIJ$PSQ.FFUVQSE $POTVM%/4ͷศར༻๏ 1SPYZͷ6QTUSFBNΛ$POTVM%/4ͰϥϯυϩϏϯ hosts: "foo.service.consul:443": listen: port: 443 ssl: certificate-file:
/etc/h2o/tls.crt key-file: /etc/h2o/tls.key paths: "/": proxy.reverse.url: "https://foo.service.consul:443/"
)BTIJ$PSQ.FFUVQSE $POTVM5FNQMBUFͷ༻ ,FFQBMJWFEͷDPOGʹDPOTVMUFNQMBUFΛ༻͢Δ͜ͱͰ1SPYZͷ$POTVMKPJOͰαʔϏεΠϯ͢Δ virtual_server <%= @vip %> 443 { delay_loop
10 lvs_sched rr lvs_method NAT protocol TCP {{range service "proxy|passing"}} real_server {{.Address}} 443 { weight 1 TCP_CHECK { connect_port 443 connect_timeout 30 } }{{end}} }
)BTIJ$PSQ.FFUVQSE ͍ΖΜͳϨΠϠʔ͕ͳΊΒ͔ʹϦϦʔε %/4 -# -# 1SPYZ 1SPYZ 1SPYZ /FX1SPYZ 8FC
8FC 8FC /FX8FC 8FC 8FC XFCTFSWJDFDPOTVM DPOTVMUFNQMBUF DPOTVMEOT YYYYYYYYYYY YYYYYYYYYYY YYYYYYYYYYY YYYYYYYYYYY
)BTIJ$PSQ.FFUVQSE 7BVMU༻ͷಛ
)BTIJ$PSQ.FFUVQSE 7BVMU ๏ 1,*ͱ5SBOTJUγʔΫϨοτΛར༻ʢ͞ΒʹՃ༧ఆʣ ๏ %#ʹอ࣋͢Δൿີใશͯ7BVMUͰ҉߸Խ ๏ ൃߦͨ͠ൿີใ$IFGͰʢSPPU$"DPOTVMUFNQMBUFͷUPLFOʣ ๏ 5PLFOSFOFXʢ55-ͷԆʣ͠ͳ͕Β༻
๏ NBYMFBTFUUMͷظݶͰࣦޮ͢Δʢ͘SFOFX͢Δͱཁҙʣৄ͘͠ޙड़
)BTIJ$PSQ.FFUVQSE ๏ $POTVMΛετϨʔδͱͯ͠ར༻͢ΔͱΞΫςΟϒͳ7BVMUʹରͯ͠ WBVMUTFSWJDFDPOTVM͕ࣗಈతʹઃఆ͞ΕΔ ๏ 7BVMUΛೝূہͱͯ͠ઃఆ͠αʔόূ໌ॻΛࣗͰൃߦ͢Δ ๏ 7BVMU࠶ىಈ͢Δͱ4FBM͞ΕΔ ๏ -FU`T&ODSZQUΛͬͯαʔόূ໌ॻൃߦ͢Δʁ
1,*ͷ3PPU$" 7BVMUαʔόʹ5-4ଓ͢Δ߹Ͳ͏ͨ͠Βྑ͍͔
)BTIJ$PSQ.FFUVQSE ๏ 7BVMUʹ4*()61γάφϧͰαʔόূ໌ॻͷ࠶ಡΈࠐΈΛ͢Δ ๏ 4*()61Ͱ4FBM͞Εͳ͍ ๏ "VEJUMPHͷMPHSPUBUFʹ͑Δ ๏ 7BVMU͕ൃߦͨ͠3PPU$"$IFGͰ $POTVMUF5FNQMBUFͷ༻
αʔόূ໌ॻʹDPOTVMUFNQMBUFΛ༻͢Δ͜ͱܧଓతͳূ໌ॻൃߦΛࣗಈԽ͢Δ
)BTIJ$PSQ.FFUVQSE vault { address = "https://127.0.0.1:8200" token = "<%= node['vault']['token']
%>" renew_token = true grace = "5m" ssl { enabled = true verify = false } } template { contents = "{{ with secret \"pki/issue/service-consul\" \"common_name=vault.service.consul\" }}{{ .Data.issuing_ca }}{{ end }}" destination = "/usr/share/ca-certificates/extra/Vault_Root_CA.crt" command = "sudo /usr/local/sbin/update_ca_certs" } template { contents = "{{ with secret \"pki/issue/service-consul\" \"common_name=vault.service.consul\" }}{{ .Data.certificate }}{{ end }}" destination = "/etc/vault.d/vault.service.consul.crt" command = "sudo /usr/local/sbin/reload_vault" } template { contents = "{{ with secret \"pki/issue/service-consul\" \"common_name=vault.service.consul\" }}{{ .Data.private_key }}{{ end }}" destination = "/etc/vault.d/vault.service.consul.key" command = "sudo /usr/local/sbin/reload_vault" } TVEPFSTͰڐՄ͢ΔͨΊʹ֤εΫϦϓτʹ ͍ͷͰΠϯϥΠϯͰهड़ 5PLFOͷ55-ΛԆͯ͠༻ $POTVMUF5FNQMBUFͷ༻
)BTIJ$PSQ.FFUVQSE ๏ "QQMJDBUJPO͕7BVMUʹରͯ͠ೝূ͢ΔBQQSPMF ๏ ೝূ͢Δͱࢦఆͷ55-͕ઃఆ͞Εͨ5PLFO͕ൃߦ͞ΕΔ ๏ "QQMJDBUJPOͦͷ5PLFOΛͬͯ7BVMUͱΓऔΓΛ͢Δ ๏ "QQMJDBUJPO͕BQQSPMFͷSPMF@JEͱTFDSFU@JEΛ͍࣋ͬͯͯҙຯ͕ͳ͍ ๏
"QQMJDBUJPOϓϩηεͷ֎Ͱ5PLFOΛൃߦ͢Δ "QQMJDBUJPO5PLFOͷ
)BTIJ$PSQ.FFUVQSE ๏ "QQMJDBUJPOͷσϓϩΠ࣌ʹBQQSPMFBVUIΛ͢ΔίϚϯυΛ࣮ߦ ๏ ίϚϯυೝূͷ18Ͱ͋ΔTFDSFU@JEͷ55-ԆΛ1045 ๏ ଓ͚ͯೝূΛ࣮ߦ͠5PLFOΛऔಘ͢Δ ๏ औಘͨ͠5PLFOΛ"QQMJDBUJPO͕ಡΊΔύεʹஔ "QQMJDBUJPO5PLFOͷղܾ๏
7BVMU $PNNBOE SPMF@JE TFDSFU@JE UPLFO
)BTIJ$PSQ.FFUVQSE 7BVMUUPLFO͕ࣦޮͯ͠ো
)BTIJ$PSQ.FFUVQSE ๏ "QQSPMFͷTFDSFU@JEΛSFOFX͢ΔͨΊʹDVTUPNTFDSFU@JEͱͯ͠ઃఆͯ͠ ͍Δ ๏ DVTUPNTFDSFU@JEʹઃఆ͢ΔUPLFOBVUIUPLFOͰൃߦ ๏ Ϛϯτͨ͠TFDSFU͝ͱʹNBYMFBTFUUMͱ͍͏ઃఆ͕͋ΓɺγεςϜશମ ʹNBYUUM͕ଘࡏ͢Δ ๏
྆ํ͕ະઃఆͷ߹ɺγεςϜͷNBYUUMͰ͋ΔEBZT্͕ݶ 7BVMUUPLFO͕ࣦޮͯ͠ো
)BTIJ$PSQ.FFUVQSE IUUQTXXXWBVMUQSPKFDUJPEPDTDPODFQUTUPLFOTIUNMUIFHFOFSBMDBTF ίϯηϓτʹॻ͍ͯ͋ͬͨ
)BTIJ$PSQ.FFUVQSE .BY55-ઃఆɺ֤Ϛϯτ͞ΕͨγʔΫϨοτ͝ͱʹઃఆ͢ ΔͷͰɺຊ൪ӡ༻࣌ʹඞͣߟྀ͠·͠ΐ͏ɻ ͱͯॏཁͰ͢
)BTIJ$PSQ.FFUVQSE ๏ 5PLFO͕ࣦޮͨ͠ΒBVEJUMPHʹΤϥʔ ͕සൃ͍ͯͨ͠ ๏ 7BVMUTFSWFSͷ$POTVMDIFDLTͰBVEJUMPHͷ ࢹΛՃ ๏ ݕͨ͠ͷ$POTVM"MFSUͰ4MBDL௨ Αͦ͠ΕͳΒࢹՃͩ
{ "name": "vault-audit-log", "tags": ["vault", "audit"], "checks": [ { "script": "sudo /usr/local/sbin/check_audit", "interval": "60s" } ] } #!/bin/bash check-log --file /var/log/vault_audit.log \ --pattern '\"error\":\".+\"' \ —exclude='invalid request|unsupported path|unsupported operation' .BDLFSFMͷDIFDLDPNNBOE
)BTIJ$PSQ.FFUVQSE $POTVMؾܰʹࢹՃͰ͖ͯศརʂ
)BTIJ$PSQ.FFUVQSE ๏ ϩϦϙοϓʂϚωʔδυΫϥυͰͷ)BTIJ$PSQιϑτΣΞͷ׆༻ํ๏ ΛഎܠͱಛΛ౿·͑ղઆ ๏ )BTIJ$PSQιϑτΣΞ͍͜ͳ͢͜ͱͰγεςϜ͕͍͍ײ͡ʹͳΔ ͷͰυΩϡϝϯτΛख़ಡ͢͠ ๏ )BTIJ$PSQͷ֤ιϑτΣΞʹྲྀΕ͕͋Γɺซ༻͢Δ͜ͱͰศར͕͞ ૿͢ʂʂʂ
$PODMVTJPO
)BTIJ$PSQ.FFUVQSE 5IBOLZPV 8FSFIJSJOH