Building CI from scratch

At the end of 2017, our team started to work on a completely new mobile SDK. We started from scratch and we wanted to fix issues that we witnessed during our work on the existing SDK. And CI was one of the major topics.

This talk is about our approach to building new CI from scratch. What we tried, what didn't work, what types of issues we faced.

Keywords for this talk: Jenkins, AWS, Serverless, Docker, Mac mini, git, repo, Gerrit, Java, Go

Transcript

1. Artem Nikitin Heisenbug | May, 2018 New CI from scratch
2. These slides are already available! https://bit.ly/2rOa0Ah © 2018 HERE | Public Heisenbug | May, 2018
3. About SDK © 2018 HERE | Public Heisenbug | May, 2018 https://developer.here.com/develop/mobile-sdks
4. Specifics of our existing SDK • Huge codebase in C++/Java/ObjC • 40 min checkout sources from scratch • 6 GB size of source code after checkout • 11M+ LOC (including dependencies) © 2018 HERE | Public Heisenbug | May, 2018
5. © 2018 HERE | Public Heisenbug | May, 2018
6. Problems of existed SDK • Build on top of huge internal monolithic core • Big size of SDK • No modularity in SDK • Low developer productivity © 2018 HERE | Public Heisenbug | May, 2018
7. Problems of existed CI • Big Jenkins setup https://www.youtube.com/watch?v=cT8GjzM1DXk https://conferences.oreilly.com/velocity/vl-eu-2017/public/schedule/detail/61951 • Highly opinionated CI setup • Hard to customize • Apple support is third class citizen © 2018 HERE | Public Heisenbug | May, 2018
8. © 2018 HERE | Public Heisenbug | May, 2018
9. 01 Solving architecture problems first
10. Huge and complex codebase • Huge internal codebase (10M+ LOC) • Very modular (1000+ Git repos) • Our project depends on almost 100 internal dependencies © 2018 HERE | Public Heisenbug | May, 2018
11. Solution • Using `repo` for working with codebase https://source.android.com/setup/develop/ • Gerrit for code reviews https://news.ycombinator.com/item?id=8605293 • Block changes to our dependencies if they are failing our builds/tests © 2018 HERE | Public Heisenbug | May, 2018
12. Lots of boilerplate code • Complex codebase structure with C++/Java/ObjC • Code written by different people from different teams • No consistency in implementation • Lots of manually written boilerplate code © 2018 HERE | Public Heisenbug | May, 2018
13. Solution • Using IDL to describe public API • Using code generation to generate all the boilerplate code based on IDL description • We are using Djinni from Dropbox https://github.com/dropbox/djinni © 2018 HERE | Public Heisenbug | May, 2018
14. Djinni, IDL © 2018 HERE | Public Heisenbug | May, 2018
15. Djinni, generated C++ © 2018 HERE | Public Heisenbug | May, 2018
16. Djinni, generated Java © 2018 HERE | Public Heisenbug | May, 2018
17. Djinni, generated ObjC © 2018 HERE | Public Heisenbug | May, 2018
18. Complex build system • Project setup is complicated and unfriendly • Separate build setups for local development and for CI © 2018 HERE | Public Heisenbug | May, 2018
19. Solution • Universal build system/setup for local development and CI • Make project setup as friendly as possible © 2018 HERE | Public Heisenbug | May, 2018
20. Script © 2018 HERE | Public Heisenbug | May, 2018
21. Differences in test coverage • Platform level tests written by different people from different teams • As a result, sometimes test coverage differ between platforms © 2018 HERE | Public Heisenbug | May, 2018
22. Solution • Using BDD as a tool • Specify test scenarios with Gherkin • It forces to have exactly the same tests on every platform © 2018 HERE | Public Heisenbug | May, 2018
23. 02 New CI
24. © 2018 HERE | Public Heisenbug | May, 2018
25. Foundation of our new CI • Jenkins (TeamCity? GitLab? Travis CI? AWS CodeBuild?) • AWS (GCP? Azure? Virtualization? Own hardware? Kubernetes? Mesos?) • Docker (Configuration management?) • Mac (Own hardware? Managed hosting? Virtualization?) © 2018 HERE | Public Heisenbug | May, 2018
26. CI Overview © 2018 HERE | Public Heisenbug | May, 2018 EC2 + EFS Mac nodes Jenkins Artifactory Gerrit S3 Device Farm
27. Workflow: Blocking change before merging © 2018 HERE | Public Heisenbug | May, 2018 Developer Gerrit Chang e
28. Workflow: Verify after merging © 2018 HERE | Public Heisenbug | May, 2018 Gerrit Developer S3
29. Workflow: Release © 2018 HERE | Public Heisenbug | May, 2018 Gerrit Artifactory User User
30. Jenkins © 2018 HERE | Public Heisenbug | May, 2018 EC2 + EFS Mac nodes Jenkins Artifactory Gerrit S3 Device Farm
31. Jenkins © 2018 HERE | Public Heisenbug | May, 2018 Freestyle jobs Declarative pipeline Easy to use + - Maintenance - + Configuration as Code -* + Shared lib/code -* + Reaction on job status -* + Complex scenarios +/- + WTF per minute min a lot
32. AWS © 2018 HERE | Public Heisenbug | May, 2018 EC2 + EFS Mac nodes Jenkins Artifactory Gerrit S3 Device Farm
33. AWS • Spot Instances - key element for scalable CI • A way to get resources with discount up to 90% • But no guarantee of availability! • We paid around 65 USD for 500+ hours in April • 0.12 USD per hour -> x3 cheaper then On-Demand price of the same instance! © 2018 HERE | Public Heisenbug | May, 2018
34. AWS © 2018 HERE | Public Heisenbug | May, 2018 EC2 + EFS Mac nodes Jenkins Artifactory Gerrit S3 Device Farm
35. AWS • EFS - it’s a NAS (Network-attached storage) • Very tricky throughput management https://docs.aws.amazon.com/efs/latest/ug/performance.html • You should monitor BurstCreditBalance metric • Around 25 USD per ~100GB of cache in April © 2018 HERE | Public Heisenbug | May, 2018
36. AWS Use cases: • We are using it for sharing ccache cache to speed up C++ compilation • You can use it to share something between EC2 instances © 2018 HERE | Public Heisenbug | May, 2018
37. AWS © 2018 HERE | Public Heisenbug | May, 2018 EC2 + EFS Mac nodes Jenkins Artifactory Gerrit S3 Device Farm
38. AWS • S3 – object storage • S3 isn’t a file system! • Not just for storing files • 2.52 USD in April for 1M+ of requests and 400GB+ of output traffic © 2018 HERE | Public Heisenbug | May, 2018
39. AWS Use cases: • We are using it to store temporary build artifacts (demo apps, reports, logs, screenshots, etc…) • We are delivering releases to public using S3 with CloudFront • I’m hosting my personal blog on S3 • Once I used it like a DB for a hackathon project, because it was easy to use and cheap J © 2018 HERE | Public Heisenbug | May, 2018
40. AWS © 2018 HERE | Public Heisenbug | May, 2018 EC2 + EFS Mac nodes Jenkins Artifactory Gerrit S3 Device Farm
41. AWS © 2018 HERE | Public Heisenbug | May, 2018
42. AWS • Lambda – new paradigm • Run code on events without thinking about infrastructure • Pay for amount of resources and time • Lambda’s from Java to Go -> 2x money savings • 250K+ requests and 180K seconds in April for 3.16 USD* © 2018 HERE | Public Heisenbug | May, 2018
43. AWS Use cases: • Posting review feedback to code changes in Gerrit • Killing unused EC2 instances • Updating AMI (like Packer but with our specifics) • Collecting metrics from test apps • In general, it’s a great glue for AWS services © 2018 HERE | Public Heisenbug | May, 2018
44. AWS © 2018 HERE | Public Heisenbug | May, 2018
45. AWS © 2018 HERE | Public Heisenbug | May, 2018 EC2 + EFS Mac nodes Jenkins Artifactory Gerrit S3 Device Farm
46. AWS • Device Farm – mobile devices in cloud • Supports fuzzing (random input events), mobile web, android and iOS testing • Supports Appium, Calabash and “native” tools • Remote access to devices • https://github.com/artemnikitin/devicefarm-ci-tool © 2018 HERE | Public Heisenbug | May, 2018
47. AWS Must Do in the Cloud: • Immutable infrastructure • Infrastructure as Code • Prepare for failures • Monitor your spending's © 2018 HERE | Public Heisenbug | May, 2018
48. AWS https://github.com/open-guides/og-aws https://www.expeditedssl.com/aws-in-plain-English https://read.acloud.guru/ https://www.slideshare.net/hornsby/10-lessons-from-10-years-of-aws https://seleniumcamp.com/talk/scaling-execution-of-protractorjs-on-aws- lambda-with-selenoid/ https://aws.amazon.com/blogs/devops/ui-testing-at-scale-with-aws- lambda/ https://www.youtube.com/watch?v=EDZBYbEwhm8 https://www.youtube.com/watch?v=73-G2zQ9sHU https://www.youtube.com/watch?v=ZgxZCXouBkY https://www.youtube.com/watch?v=w7X4gAQTk2E © 2018 HERE | Public Heisenbug | May, 2018
49. Docker © 2018 HERE | Public Heisenbug | May, 2018 EC2 + EFS
50. Docker • Docker – tool for operating-system-level virtualization, aka containerization • We are using it to encapsulate environment for builds • We are ”backing” images in AMI • It creates a problem. We need to keep AMI up to date © 2018 HERE | Public Heisenbug | May, 2018
51. Docker © 2018 HERE | Public Heisenbug | May, 2018
52. Mac nodes © 2018 HERE | Public Heisenbug | May, 2018 EC2 + EFS Mac nodes Jenkins Artifactory Gerrit S3 Device Farm
53. Mac nodes • It’s ok to manage 1 or 2 node(s) manually • It’s become a nightmare on a higher scale • Not everything can be automated easily © 2018 HERE | Public Heisenbug | May, 2018
54. Mac nodes • Ansible – tool for configuration management • Works for Linux, MacOS, Windows • What was done: managing operations via CLI • What is still missing: how to manage system stuff? (MacOS updates, Xcode updates, etc…) © 2018 HERE | Public Heisenbug | May, 2018
55. Mac nodes © 2018 HERE | Public Heisenbug | May, 2018
56. Mac nodes © 2018 HERE | Public Heisenbug | May, 2018
57. In the end… • We have our own CI setup what fits our needs • CI is relatively cheap • It can scale until we have money • We found some ways of managing Mac nodes automatically © 2018 HERE | Public Heisenbug | May, 2018
58. Thank you Contact Artem Nikitin hi@artemnikitin.com artemnikitin artemnikitin