Make CI/CD More Comfortable Before/After Code Testing

Talked at https://testnight.connpass.com/event/117578/

Not about testing methods but focusing on CI/CD

Transcript

1. MAKE CI/CD MORE COMFORTABLE BEFORE/AFTER CODE TESTING 7TH MAR. ANDROID TEST NIGHT #6
2. Self Introduction Jumpei Matsuda @red_fat_daruma ▸ GitHub: jmatsu ▸ Software Architect @DeployGate ▸ A member of DroidKaigi ▸ Consists of craft beer (especially Belgium) and Japanese sake. 2
3. DroidKaigi 2019 Thank You for Your Work and Those Who Attended! ▸ About 1000 people joined, over 90 speakers talked. ▸ We have already published session videos! ▸ Check it out! https://bit.ly/2N1DLIb 3 https://droidkaigi.jp/2019
4. Agenda What I Will Talk About ▸ Motivation ▸ Definition of *comfortable* ▸ Tips to support your testing ▸ How to organize non-fragile CI ▸ Save/load intermediate results of the current/previous build ▸ Get advantages of DeployGate 4
5. Agenda What I Will NOT Talk About ▸ Technique for test code ▸ Testing method 5
6. Agenda What I Will Talk About ▸ Motivation ▸ Definition of *comfortable* ▸ Tips to support your testing ▸ How to organize non-fragile CI ▸ Save/load intermediate results of the current/previous build ▸ Get advantages of DeployGate 6
7. Motivation Before/After Code Testing? ▸ Testing is an important stage of dev. process ▸ There are many technique for testing and we are continuing to look for better practices ▸ However, ▸ It's a part of the whole process, and *code* is just one of testing factors ▸ Things around code testing are also important, and should be comfortable 7
8. Agenda What I Will Talk About ▸ Motivation ▸ Definition of *comfortable* ▸ Tips to support your testing ▸ How to organize non-fragile CI ▸ Save/load intermediate results of the current/previous build ▸ Get advantages of DeployGate 8
9. Definition What's *Comfortable* in This Talk? ▸ In short, it's to accelerate our development ▸ Not fragile, be stable ▸ Nobody likes CI/CD which fail frequently ▸ Be help for other development stages ▸ Finding/fixing bugs, QA, etc. 9
10. Agenda What I Will Talk About ▸ Motivation ▸ Definition of *comfortable* ▸ Tips to support your testing ▸ How to organize non-fragile CI ▸ Save/load intermediate results of the current/previous build ▸ Get advantages of DeployGate 10
11. How to organize non-fragile CI Not Fragile (Stable) Realized by Flexible ▸ No failure is allowed 11
12. Not fragile (stable) does not mean *robust* Flexible ⊂ Not Fragile (Stable) ▸ No failure is allowed ▸ Non-flexible with failures is an evil ▸ BAD: Fail CI anyway if any task fails ▸ BAD: Do not run a Test task if lint fails ▸ How to make it flexible? 12
13. Not fragile (stable) does not mean *robust* Flexible ⊂ Not Fragile (Stable) ▸ No failure is allowed ▸ Non-flexible with failures is an evil ▸ BAD: Fail CI anyway if any task fails ▸ BAD: Do not run a Test task if lint fails ▸ How to make it flexible? ▸ Check command status and make CI/CD fail if needed ▸ Separate jobs if possible 13
14. Not fragile (stable) does not mean *robust* Flexible ⊂ Not Fragile (Stable) ▸ No failure is allowed ▸ Non-flexible with failures is an evil ▸ BAD: Fail CI anyway if any task fails ▸ BAD: Do not run a Test task if lint fails ▸ How to make it flexible? ▸ Check command status and make CI/CD fail if needed ▸ Separate jobs if possible 14
15. Check command status and make CI/CD fail if needed Know Rules of Command Set To Be Executed ▸ Basically you have multiple roles in your dep. pipeline 15 Prepare dependencies Lints Reports for test-related res. Reports for others TESTING Assembling Deployment
16. Check command status and make CI/CD fail if needed Consider *Required* Dependencies Between Command Set ▸ You can notice some of commands looks independent 16 Prepare dependencies Lints Reports for test-related res. Reports for others TESTING Assembling Deployment
17. Check command status and make CI/CD fail if needed Check What Will Be Assured by Each Roles ▸ If you wanna assure nothing, you can eject CI/CD :p 17 Prepare dependencies Lints Reports for test-related res. Reports for others TESTING Assembling Deployment Code Quality Product Quality - Dev. experience - Product Quality through QA
18. Check command status and make CI/CD fail if needed So, Which Command Should Fail CI/CD Immediately? ▸ Please note that this slide and your though may differ 18 Prepare dependencies Lints Reports for test-related res. Reports for others TESTING Assembling Deployment No. Basically, yes. As for assembling, yes. Conditional yes for dep. Yes!
19. Examples Other Examples ▸ Use *continue* option when running Gradle tasks ▸ HSBEMFX
    MJOU%FCVH
    UFTU%FCVH6OJU5FTU
    DPOUJOVF
    P⒐JOF
    ▸ Use retry-logic to execute unstable commands ▸ For example, 500 error from maven repo. should be retried ▸ Execute reporting system like Danger anyway ▸ Basically we want to know test failures in a failure case 19
20. Not fragile (stable) does not mean *robust* Flexible ⊂ Not Fragile (Stable) ▸ No failure is allowed ▸ Non-flexible with failures is an evil ▸ BAD: Fail CI anyway if any task fails ▸ BAD: Do not run a Test task if lint fails ▸ How to make it flexible? ▸ Check command status and make CI/CD fail if needed ▸ Separate jobs if possible 20
21. Separate jobs if possible Built *Required* Graph Will Be Your Help ▸ Basically, jobs depend on execution time and CI services 21 Prepare dependencies Lints Reports for test-related res. Reports for others TESTING Assembling Deployment Vital job Test job Deployment job
22. Agenda What I Will Talk About ▸ Motivation ▸ Definition of *comfortable* ▸ Tips to support your testing ▸ How to organize non-fragile CI ▸ Save/load intermediate results of the current/previous build ▸ Get advantages of DeployGate 22
23. Save/load intermediate results of the current/previous build How To Treat Intermediate Results of CI ▸ Major CI services have cache systems ▸ Key-value style cache, repo/branch-based cache ▸ They are useful for build intermediate results like artifact dependencies' cache ▸ But, how about finer-grained cache...? 23
24. Save/load intermediate results of the current/previous build Cache During a Build ▸ Create a new file right after executing a command ▸ If the file exists, the command exited with SUCCESS ▸ Otherwise, the command failed 24
25. Save/load intermediate results of the current/previous build Cache Between Builds on a Single PR ▸ Use the CI's cache system? 25
26. Save/load intermediate results of the current/previous build Cache Between Builds on a Single PR ▸ Use the CI's cache system? ▸ NO. ▸ You can use GitHub issues and HTML ▸ Fortunately(?), GitHub issues and comments can parse HTML ▸ It means you can use HTML comment syntax as well 26 *No issue found by ktlint!* <!-- { "is_successful": true, "last_build_num": 48 } -->
27. Save/load intermediate results of the current/previous build Cache Between Builds on Multiple PRs ▸ Please let me know if there is a smart solution 27
28. Agenda What I Will Talk About ▸ Motivation ▸ Definition of *comfortable* ▸ Tips to support your testing ▸ How to organize non-fragile CI ▸ Save/load intermediate results of the current/previous build ▸ Get advantages of DeployGate 28
29. Get advantages of DeployGate DeployGate? ▸ A mobile app distribution service like Fabric Beta ▸ Annual BBQ planner in US ▸ https://deploygate.com 29
30. Get advantages of DeployGate Advantages of DeployGate (Require Paid Plans) ▸ Faster download through CDN ▸ Distribution tracks 30
31. Get advantages of DeployGate How To Have a Faster Download Experience ▸ CDN produces this advantage ▸ Therefore, first download from the signed URL is not fast, unfortunately 31
32. Get advantages of DeployGate How To Have a Faster Download Experience ▸ CDN produces this advantage ▸ Therefore, first download from the signed URL is not fast, unfortunately ▸ Download your artifact first on CI right after deployment ▸ Note that the URL will be expired, so this tip does not work if new URLs are published 32 artifact_url=$(<response> | jq -r ".results.file") curl -SLs $artifact_url >/dev/null
33. Get advantages of DeployGate Distribution Tracks ▸ Download pages for specific versions ▸ Like named internal track on Google Play Store 33
34. Get advantages of DeployGate Distribution Tracks ▸ Download pages for specific versions ▸ Like named internal track on Google Play Store ▸ Deploy release ver., latest develop ver., feature branch ver. ▸ GitHub:jmatsu/dpg will be your help ▸ GitHub:DroidKaigi/conference-app-2019 is a good practice 34
35. Matome dayo Be Comfortable Before/After Code Testing as Well ▸ How to organize non-fragile CI ▸ Reorganize jobs and handle exit statuses manually ▸ Save/load intermediate results of the current/previous build ▸ Use new files and/or html comments to save/load metadata ▸ Get advantages of DeployGate ▸ Download a first artifact on CI, use distribution tracks 35
36. Thank you for listening! Don't Hesitate To Ask Me any Questions ▸ Jumpei Matsuda ▸ @red_fat_daruma ▸ Software Architect @DeployGate ▸ A member of @DroidKaigi 36