Monitoring Kubernetes - Speaker Deck

Slide 1

Slide 1 text

Monitoring Kubernetes gcp ja night #29 Hatena Co., Ltd. @stanaka ! 2014/12/12

Slide 2

Slide 2 text

[PR] Mackerel https://mackerel.io/

Slide 3

Slide 3 text

Kubernetes • Docker ΛΫϥελ؅ཧ! • ͍Ζ͍Ζͳ؀ڥͰಈ͔͍ͨ͠ • MackerelͰϦιʔε؅ཧ͍ͨ͠

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Kubernetes Monitoring  ʹؔ࿈͢Δཁૉ • kubelet • cAdvisor • heapster

Slide 6

Slide 6 text

kubelet • ֤ϊʔυ(minion)Ͱಈ࡞ • kubernetes masterͱڠௐಈ࡞

Slide 7

Slide 7 text

cAdvisor • Dockerͷ౷ܭ৘ใΛऔಘ • ϗετશମͱίϯςφຖ • DockerҎ֎ͷίϯςφ΋ରԠ HPPHMFDBEWJTPS

Slide 8

Slide 8 text

heapster • Ϧιʔε৘ใऩूπʔϧ • kubarnetesઐ༻ • InﬂuxDB΁Πϯϙʔτ (PPHMF$MPVE1MBUGPSNIFBQTUFS

Slide 9

Slide 9 text

kubarnetesඪ४ϞχλϦϯά

Slide 10

Slide 10 text

kubarnetesඪ४ϞχλϦϯά • heapster + cAdvisor   → InfuxDB & Grafana   • Ϋϥελ಺ʹ͋ΔͷͰɺΫϥελΛ མ͢ͱམͪΔ

Slide 11

Slide 11 text

Minionͷ౷ܭ৘ใऔಘ • http://#{ipaddr}/stats/ • kubeletͷAPI • ཪͰcAdvisor͕ಈ͘ $ curl http://10.240.153.49:10250/stats/ | jq 'keys' [ "name", “spec", # εϖοΫ “stats", # ౷ܭ৘ใ 1ඵ͝ͱ 60ඵؒ෼ "subcontainers" ]

Slide 12

Slide 12 text

statsͷத਎ $ curl http://10.240.56.195:10250/stats/90559c9f-‐8123-‐11e4-‐ a0ec-‐42010af08f91/php-‐redis | jq ".stats | .[] | .timestamp" "2014-‐12-‐12T02:41:55.52640117Z" "2014-‐12-‐12T02:41:56.526265405Z" "2014-‐12-‐12T02:41:57.52632129Z" "2014-‐12-‐12T02:41:58.526385146Z" "2014-‐12-‐12T02:41:59.526285874Z" "2014-‐12-‐12T02:42:00.526267704Z" "2014-‐12-‐12T02:42:01.526227391Z" "2014-‐12-‐12T02:42:02.526312936Z" "2014-‐12-‐12T02:42:03.526225182Z" “2014-‐12-‐12T02:42:04.526287033Z" … (ҎԼɺ60ඵ෼)

Slide 13

Slide 13 text

statsͷத਎ #2 # curl http://10.240.153.49:10250/stats/ | jq '.stats | .[0] | keys' [ "cpu", "diskio", "filesystem", "memory", "network", "timestamp" ]

Slide 14

Slide 14 text

statsͷத਎ৄࡉ UJNFTUBNQ DQV NFNPSZ OFUXPSL \SY UY^@\CZUFT QBDLFUT FSSPST ESPQQFE^ pMFTZTUFN EFWJDF<> DBQBDJUZ VTBHF JP@UJNF EJTLJP JP@NFSHFE JP@RVFVFE JP@TFSWJDF@CZUFT NBKPS NJOPS TUBUT "TZOD 3FBE 4ZOD 5PUBM 8SJUF JP@TFSWJDF@UJNF JP@TFSWJDFE JP@UJNF JP@XBJU@UJNF TFDUPST

Slide 15

Slide 15 text

ϞχλϦϯάख๏ • Ϋϥελ্Ͱಈ͍͍ͯΔ΋ͷΛ೺Ѳ • kubecfg list pods • kubecfg get pods/#{ID} • Munion্ͷίϯςφͷঢ়ଶऔಘ • /stats/#{ID}/#{container}

Slide 16

Slide 16 text

kubecfg list pods • cluster/kubecfg.sh -json list pods \ JUFNT< \ JEGDFBFDBGG IPTULVCFSOFUFTNJOJPODCVPZBOU[PEJBDJOUFSOBM EFTJSFE4UBUF\ NBOJGFTU\ DPOUBJOFST< \ JNBHFCSFOEBOCVSOTQIQSFEJT OBNFQIQSFEJT

Slide 17

Slide 17 text

kubecfg get pods • cluster/kubecfg.sh -json get pods/ 73f777c6-8123-11e4-a0ec-42010af08f91 • Ϩεϙϯε͸ list pods ͱಉ͡

Slide 18

Slide 18 text

Munion্ͷίϯςφͷঢ়ଶऔಘ • curl http://10.240.56.195:10250/ stats/90559c9f-8123-11e4- a0ec-42010af08f91/php-redis | jq ".stats | .[] | .timestamp” • Ϩεϙϯε͸/stats/ͱಉ͡

Slide 19

Slide 19 text

Φʔτεέʔϧ͍ͤͨ͞ • Ϧιʔε৘ใऔಘ • αΠζมߋ൑அ • αΠζมߋ࣮ߦ(ඞཁ͕͋Ε͹)

Slide 20

Slide 20 text

podsͷಈతͳαΠζมߋ • resizeͰมߋ cluster/kubecfg.sh resize frontendController 4 $ cluster/kubecfg.sh list replicationControllers Name Image(s) Selector Replicas -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐ -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐ -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐ -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐ frontendController brendanburns/php-‐redis name=frontend 4

Slide 21

Slide 21 text

Kubenetesҹ৅ • ͍Ζ͍Ζศརʹ࢖͑ͦ͏Ͱເ͸͋Δ • ϓϩμΫγϣϯʹಥͬࠐΉͷ͸·ͩ… • དྷ೥Ն͙Β͍ʹ͸͜ͳΕ͖ͯͦ͏?