CIのモニタリングにstackdirver使ってみてる話

GCPUG Tokyo Stackdriver Day January 2019 にてLTで発表した資料です。
まだMetricsのみなので頑張っていきたい

Transcript

1. $*ͷϞχλϦϯάʹ TUBDLEJSWFS࢖ͬͯΈͯΔ࿩ GCPUG Tokyo Stackdriver Day January 2019 LT
2. XIPBNJ • ͍Μͱ !JOU@UU ◦ ৽ଔ̎೥໨ ΠϯϑϥΤϯδχΞ • ΍ͬͯΔ͜ͱ ◦ ΦϯϓϨ͔ΒΫϥ΢υ·Ͱ ◦ ͓࢓ࣄͰ͸Ϋϥ΢υͱίϯςφपΓ ◦ ࠷ۙ͸044׆ಈ DPOUSJCVUJPO ΋ঃʑʹग़དྷΔΑ͏ʹͳָ͖͍ͬͯͯ͠ • ޷͖! " # $ % & • 4UBDLESJWFSྺҰϲ݄ !ྺ͸೥͘Β͍
3. ໨࣍ • Ϟνϕʔγϣϯ • 1SPNFUIFVT
   UP
   TUBDLESJWFSʹ͍ͭͯ • ΍ͬͯΈͰͲ͏͔ͩͬͨ
4. Ϟνϕʔγϣϯ • ৽ن։ൃҊ݅ʹͯ($1Λར༻ • ,VCFSOFUFTར༻ (,& • $PODPVSTF$* 1JWPUBM੡ ͷ$*Λར༻ ◦ ͜Ε΋(,&্Ͱར༻ ◦ 13͝ͱͷࣗಈςετ΍ࣗಈσϓϩΠ౳େ׆༂த • ͕ͬͭΓ࢖͍ͬͯΔͷͰɺ$*͕஗͍ͱશͯͷ࡞ۀ͕஗͘ͳͬͯ͠·͏ ◦ ͪΌΜͱϞχλϦϯά͍ͨ͠ • ͔ͤͬ͘($1࢖ͬͯΔͷͰTUBDLESJWFSͰϞχλϦϯάͰ͖Δͱ࠷ߴ
5. Ϟνϕʔγϣϯ • ཉ͍͠৘ใ ◦ ֤8PSLFSͷ.FUSJDT $16 .FNPSZ *0ͳͲ ▪ (,&ͷϞχλϦϯάΛ0/ʹ͢Δͱ͙͢ݟ͑Δ ࠷ߴʂʂʂʂ ◦ ֤8PSLFS಺ͷίϯςφ਺΍QJQFMJOFͷϝτϦΫε ▪ σϑΥϧτͩͱݟ͑ͳ͍
6. Concourse
   Metrics
    • https://concourse-ci.org/metrics.html
7. Concourse
   Metrics
    • https://concourse-ci.org/metrics.html
8. Stackdriver… • ͔ͤͬ͘4UBDLESJWFS࢖ͬͯΔͷʹ%BUBEPH΍/FX3FMJDΛ࢖͏ͷ͸ʜ ◦ ҰͭͷαʔϏεͰϞχλϦϯάπʔϧ͕ࢄཚͯͯ͠΋͍͍͜ͱແ͍ • 1SPNFUIFVT΋ݐͯΔͷԯ߷ ◦ ͳΔ΂͘.POJUPSJOH৘ใͷू໿ઌ͸ϑϧϚωʔδυ͕خ͍͠
9. 1SPNFUIFVTUPTE • IUUQTHJUIVCDPN(PPHMF$MPVE1MBUGPSNLTTUBDLESJWFSUSFFNBTUFSQSPNFUIFVTUPTE • 1SPNFUIFVTͷFYQPSUFS͔ΒTUBDLESJWFSͷDVTUPN
   NFUSJDTͰѻ͑Δܗʹม ׵ͯ͘͠ΕΔ • (,&্ͷGMVFOUEͷϝτϦΫε 1SPNFUIFVT ΛૹΔͨΊʹ΋಺෦Ͱར༻ͯ͠ ͍Δ • ($&
   PS
   (,&্ͷΈಈ࡞ ެࣜΑΓ
10. ࢖͍ํ • ͜Μͳײ͡ͷZBNMΛॻ͍ͯTJEFDBSͰ෇͚ͯEFQMPZ͢Δ͚ͩ - name: prometheus-to-sd image: gcr.io/google-containers/prometheus-to-sd:v0.3.2 ports: - name: profiler containerPort: 6060 command: - /monitor - --stackdriver-prefix=custom.googleapis.com - --source=concourse:http://localhost:9391 - --pod-id=$(POD_NAME) - --namespace-id=$(POD_NAMESPACE) env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace 1PE Web container prometheus-to-sd locahost:9391/metric stackdirver
11. ͜Μͳײ͡ʹͳΔ # HELP concourse_builds_finished_total Total number of Concourse builds finished. # TYPE concourse_builds_finished_total counter concourse_builds_finished_total 174
12. ม׵ͷఆٛ • AQSPNFUIFVT UFYU
    GPSNBUA͔ΒTUBDLESJWFS΁ͷ֤छύϥϝʔλม׵͸ҎԼͷ௨Γ • .FUJSD5ZQF ◦ $06/5&3
    ͔ )*45(3".
    
    $6.6-"5*7& ◦ ͦΕҎ֎ 
    ("6(& func extractMetricKind(mType dto.MetricType) string { if mType == dto.MetricType_COUNTER || mType == dto.MetricType_HISTOGRAM { return "CUMULATIVE" } return "GAUGE" } https://github.com/GoogleCloudPlatform/k8s-stackdriver/blob/3d2aa1aacdd55c78cb2bfb74799ecf9d17fa0801/prometheus-to-sd/translator/translator.go#L379-L384
13. ม׵ͷఆٛ • 7BMVF5ZQF ◦ ͢ͰʹDVTUPN.FUSJDT͕ఆٛ͞Ε͍ͯΔ৔߹͸ఆٛ͞Ε͍ͯΔํʹै͏ ◦ .FUSJDT5ZQF͕)*450(3"
    
    
    %*453*#65*0/ ◦ ͦΕҎ֎ 
    */5 func extractValueType(mType dto.MetricType, originalDescriptor *v3.MetricDescriptor) string { if originalDescriptor != nil { return originalDescriptor.ValueType } if mType == dto.MetricType_HISTOGRAM { return "DISTRIBUTION" } return "INT64" } https://github.com/GoogleCloudPlatform/k8s-stackdriver/blob/3d2aa1aacdd55c78cb2bfb74799ecf9d17fa0801/prometheus-to-sd/translator/translator.go#L386-L397
14. ΋͏গ͠ৄ͘͠ • IUUQJOUUUIBUFOBCMPHKQFOUSZ • ίίʹॻ͖·ͨ͠
15. ࢖ͬͯΈͯ • ࠓͷͱ͜Ζ͸ಛʹ໰୊ͳ͘࢖͍͑ͯΔ • ެ͕ࣜDPOWFSUFSΛ४උͯ͘͠ΕͯΔͷ͸خ͍͠ • ($1ͰऔಘͰ͖ΔϝτϦΫεͱ$*ͷΞϓϦέʔγϣϯϝτϦΫεͷ͓͔͛Ͱ ͍͔ͭ͘ϘτϧωοΫΛՄࢹԽग़དྷͨɻ ◦ ಛఆͷQJQFMJOF͕૸ͬͯΔͱ͖ʹ(,&ͷOPEFʹͯEJTLͷ5ISPUUMF͕ൃੜ͍ͯ͠Δ͜ͱ͕൑໌ ◦ IUUQTDMPVEHPPHMFDPNDPNQVUFEPDTEJTLTQFSGPSNBODF • $VTUPN
    NFUSJDT͸༗ྉͳͷͰ͓஋ஈʹ͸஫ҙ • ΋͏ͪΐͬͱӡ༻ͨ͠ΓɺଞTUBDLESJWFSͷαʔϏεͱ࿈ܞͯ͠΋ͬͱ৭Μ ͳ৘ใΛूΊͯΈ͍ͨɻ