Tweet
Tweet

Slide 1

Slide 1 text


 Community Driven DevOps Get Together, Go Faster

Slide 2

Slide 2 text

Bryan Liu • 現任 LINE TW DevOps TF Lead & DevGov Facilitator • 經歷 RD, PM, AE 等職位 • ⾃ 2013 開始研究持續整合到嘗試導入 DevOps 等更有效率的開發⽅式 演講經歷 • LINE TechPulse 2023，以 DevOps 思維實施開發治理 • LINE TechPulse 2020，通過測試隔離改進⾃動化驗收測試 • DevOpsDays Taipei 2017，百倍數交付談主幹開發 • DevOps Summit Taipei 2016，Test Automation

Slide 3

Slide 3 text

Agenda

Slide 4

Slide 4 text

About LINE TW • TW 在地的服務 ➡ (Very biz Driven) • RD established ≈ 2015 (Good Eng. culture) • Back to 2018 when I joined: • All run Scrum w/ similar work fl ow • Di ff erent languages, stacks, release process • It’s hard to retrieve SW Px metrics

Slide 5

Slide 5 text

Results Demo

Slide 6

Slide 6 text

You Got to Believe in Something 
 Faith

Slide 7

Slide 7 text

解決⾝邊沒效率的問題 • ⼀個 hot fi x 到底要測幾次？ • Bug 重複出現 • Merge hell • Long integration & code freeze • QA 沒時間執⾏探索測試

Slide 8

Slide 8 text

• 質量反饋滯後性 (⾃動化測試與⽣產程式碼不同步) • 每天早上花 1 ~ 2 hr 檢查為何失敗 • New feature code checked in • Code bugs • Test automation fl aky • 數百、千隻的 test automations • Duplicated test coverage • RD 從來也不會來看測了什麼 • Pipeline con fi dence 完全沒幫助 解決⾝邊不合理的流程

Slide 9

Slide 9 text

DevOps Evangelist & Faith (Goals) (Realities)

Slide 10

Slide 10 text

DevOps Evangelist & Faith (Goals) (Realities)

Slide 11

Slide 11 text

Digital Transformation? Why Change is So Hard …

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

快速交付是種「能⼒」！ 現代企業⽣存的關鍵

Slide 14

Slide 14 text

Source: The Art of Corporate Endurance, Harvard Business Review、DevOps Enterprise Journal、cloud.google.com S P 
 E 
 E 
 D

Slide 15

Slide 15 text

不進則退...

Slide 16

Slide 16 text

實際情況 … • Biz 說「不需要這麼快啦」 • 我們團隊做 platform 的，重要的是要穩定！ • PV、廣告營收不要掉，穩穩地比較重要 ... Source: Not Wrong Long - is perfect software necessary in the world of continuous delivery 但別⼈可不是這麼想！

Slide 17

Slide 17 text

Source: Insights of Agile at Tesla with Joe Justice; Is continuous testing the key to SpaceX software success 具備「快」的「能⼒」 “Reduce cycle time to enable FAST experiments and learning! ”

Slide 18

Slide 18 text

Trunk-based 
 With 
 Feature Toggling vs Git fl ow with Alpha, Beta, Staging, Production Envs. “抓⼤、放⼩、 縮短 cycle time”

Slide 19

Slide 19 text

De fi nition Matters Essential Rules and Practices for CI: • Don’t check in on a broken build • Always run all commit tests locally before committing • Wait for commit tests to pass before moving on • Never go home on a broken build • Always be prepared to revert to the previous revision • Time-boxed fixing before reverting • Don’t comment out failed tests • Test-Driven Development • Take responsibility for all breakages that result from your changes

Slide 20

Slide 20 text

Controversial Still? “The data from State of DevOps Report is clear: trunk based development predicts higher throughput and better stability, and even higher job satisfaction and lower rate of burn-out”

Slide 21

Slide 21 text

快速反饋中的測試驅動的開發 Source: SbE in DevOpsNotes (Blog)

Slide 22

Slide 22 text

Test Automation Traditional Automation Pipeline Driven Automation collaborative model RD: UT &| AE: IT & E2E Discuss “Acceptance Criteria” TOGETHER KPI Automation coverage rate KEY examples (scenario) coverage Pipeline # Multiple pipelines One pipeline to production De fi nitive Verdict Blue / Yellow / Red, human verdict Success / Fail Automation Strategy What ever works in a 2-week cycle Modularized, stub, test isolation, testContext, … Traits Alpha, beta, staging, production CUJ coverage, Release strategies, O11y Branching Strategy Feature Branches Trunk-based development + feature toggle Continuous Improvement Get “Green” before scheduled release Continuously improve pipeline cycle-time

Slide 23

Slide 23 text

Automation Strategies Source: 什麼是⾃動化測試的標的？ • Three levels of test isolation • TestContext,
 TestContainers &
 ConsumerDrivenContr actTest (CDCT) • Other test practices for stabilities

Slide 24

Slide 24 text

「沒有嚴謹的準則、沒有固定⼯ 法，任何問題都會是值得討論，很 多⽅式看起來都是對的，但幾年後 你就會發現這些作法對 DevOps Elite 的⽬標毫無幫助，反⽽⼜會 變成要處理的 legacy。」 「別把 Weekly Release 當成團隊 ⽬標，它頂多是個 Milestone！因 為要練習的技能完全不⼀樣。」 「邁向 Elite 之路，終點與⼀路上 的風景，都跟別⼈很不⼀樣。」

Slide 25

Slide 25 text

Community Driven

Slide 26

Slide 26 text

• Unit test and CI Pipelines • Cloud native & Microservices • CI/CD & GitOps • Observability • Dev Governance History of DevOps TF • Platform Engineering • Pipeline optimization TF • Engineering operations TF • RUM & frontend tracing • Golden paths & Backstage 2018.06 2019.03 2020.08 2021.08 2022.08 2023.03 2023.06 • Task-Force: 為長期持續在某個領域的學習與導入 • Side-Project: 為中短期專注於所需的平台或⼯具的開發 • 團員皆為⾃願或團隊指派參與，約使⽤ 10 ~ 20% 時間

Slide 27

Slide 27 text

DevOps TF Illustration

Slide 28

Slide 28 text

Drive Change w/ Metrics

Slide 29

Slide 29 text

• 不設定覆蓋率的統⼀標準，但要⼀次比⼀次好（童軍守則） • 分階段：先求有、再求好、再來要求每個 PR 的 new code coverage • Workshops and more workshops • 公開的儀表板（Dashboard） • 找出表現好的團隊及發現好的作法 • 同儕比較的壓⼒ • 展現決⼼（commitment） • 可能是以季、年來規劃 • 固定週期拿出來檢視 • 隱惡揚善 Observe, Learn & Adapt

Slide 30

Slide 30 text

Team Autonomy (Test Coverage) : A circle means an average of code coverages of all repos in a project / service (Core components alone have much higher code coverage rate)

Slide 31

Slide 31 text

Together & Faster • Cloud native & GitOps • Observability • RUM & full trace (WIP) • A lot more …

Slide 32

Slide 32 text

Observe, Learn & Adapt • Infra team mindsets vs DevOps + SRE • Communication, communication & communication • Get together, Go fast • Fragmentation (down) • Cost of Maintenance (down) • Support E ff i ciency • Automatically upgrade • Dx satisfaction (up) “Only 10% of elite respondents indicated that their teams have fully implemented every SRE practice we investigated.” Source: state of DevOps 2021

Slide 33

Slide 33 text

Development Governance › Regulations: LINE開發者必須遵守的 8 條基本規則 › Guidelines: 描述實施法規的資訊和程序的指南，由 15 項組成 Development Operation

Slide 34

Slide 34 text

Observe, Learn & Adapt • 如何減少做表⾯功夫（常⾒於 Top-down） • 很多是需要持續改善（10,000 hrs） • 該做好的事就及早開始（Do it early） • 成果顯著（36% of best practice submission） • 開發治理有助於推⾏ DevOps practices

Slide 35

Slide 35 text

Platform Engineering TaskForce When platform, tools, SDKs are ready Life Finds a Way!

Slide 36

Slide 36 text

• 跨團隊（Cross-cutting） 都有的問題，似乎很適合以 “Guild” 社群⽅式來解決 • 平台開發者（RD）就是未來的使⽤者，很多問題都能及早發現並解決 • 持續深化 community driven - 以 Backstage 平台為載體 • 提⾼透明度，展現社群⽬標與進展 • Golden path 以 SW Template ⾃動化 • Tech Radar 活化架構選型彈性 • 社群適合組織特定⽬標、流程、⼯法、PoC 等等討論與驗證 • 運維（operational tasks）與後續開發還是需要固定⼈⼒ Observe, Learn & Adapt

Slide 37

Slide 37 text

Takeaway

Slide 38

Slide 38 text

“看症狀來開藥單... 其實很多問題都是⾃⼰創造出來的。”

Slide 39

Slide 39 text

• 更重要的是培養「快」的能⼒ • De-couple code deployment and release • Staged rollout, progressive release • Scalable A/B testing and reduce each experiment costs • Don’t forget the core abilities (UT, code review, …) • Develop “abilities” probably more important than “methodologies” • But “methodologies” provide “de fi nitions” which do matters! Trunk-based or NOT

Slide 40

Slide 40 text

• It’s all about “Solving problems” over “Building solutions” • Aim high, start low and make changes • (Leadership) Increase visibility & in fl uence outside of ur project • (Leadership) Authority comes from an ability to in fl uence others • Consensus and endorsement from mgmt (bonus, OKR also) • One man’s opinion v.s. team consensus • Top-down：短時間要有⼀定成果，對「持續改善」可能有負⾯效果 • Community：進度緩慢、團隊向⼼⼒（team moral ）不易維持 Community in a Digital World

Slide 41

Slide 41 text

“沒有⼀家公司是完美的，抱怨永遠不會結束， 但很多公司卻會結束。” Dragon boat festival, this is culture! （⽂化沒有對與錯，可能也不是 DevOps 能否導入的關鍵。） For Evangelists “如何幫助公司、團隊，學習到更有效 率的開發⽅式。” “想想如何說服⼤家，相信⼯程師⼤部分都是理性的。” “⾝為⼀位佈道者，你還是得相信點什麼， 然後學會不要臉！”

Slide 42

Slide 42 text

《從 A 到 A ＋》書的結尾說到：『做到卓越不⾒得比做到優秀困難， 反⽽可以少吃點苦頭，累積的動能會注入更多的能量，持續維持在平庸 將不斷耗掉能量，新注入的能量卻很少。』 Good v.s. Excellent • Ex: Trunk-based development Engineering Culture Very very few use trunk- based development Elite companies scale well with it New unicorns advertise it Best-selling books promote it Statistic data prove it

Slide 43

Slide 43 text

Source: https://cudl.lib.cam.ac.uk/