Continuous Integration for the Rest of Us

Every project benefits from a good Continuous Integration & Delivery system. While there's countless articles on the latest Swift or SwiftUI tricks, there's surprisingly little talk about strategies around CI & CD. You will find case studies by industry giants like Shopify or Uber that talk about their solutions, however their CI/CD team in many cases is larger than some middle-sized companies.

At PSPDFKit, we don't have a separate team for CI/CD, our engineering team also owns testing and delivery. In this talk I'll give you an overview of the options today, and will explore what path we took for fast, automated testing in a multi-platform environment. We'll also talk about money, because running cost is an important topic for bootstrapped companies. Lastly, we'll look at options around configuration management tools to automate machine setups.

Transcript

1. Continuous Integration for the Rest of Us Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
2. Structure of this Talk — Requirements — CI Orchestrators — Virtualization vs Bare Metal — Configuration Management Tools Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
3. whoami — Peter Steinberger — Founder @ PSPDFKit — @steipete on Twitter — https://steipete.com/speaking — https://pspdfkit.com/blog/ 2020/continuous-integration- for-small-ios-macos-teams Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
4. Our Requirements — 50 people — 1,5 MLOC codebase — 25m compile (6-core MacMini) — 10 Concurrent Builds — Multi-Platform Support — Integration with GitHub PRs — First-class container/Docker support — Pipeline support (parallel steps, artifact support) — Great UI Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
5. Types of CI Solutions — Full-Featured, closed (Manage Pipelines & Run Builds) — Full-Featured + optional self-hosted — Orchestrators (Manage Pipelines & self-hosted runners) Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
6. Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
7. Jenkins Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
8. Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
9. Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
10. Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
11. TeamCity Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
12. Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
13. Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
14. Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
15. Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
16. Virtualized Hardware — macOS must run on Mac hardware — virtualization on PC works but is against Apple EULA — Apple doesn't sell server hardware Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
17. Virtualizing macOS — VMware vSphere (VMware Hypervisor) — Orka (MacStadium, KVM Hypervisor) — Anka (Veertu, Apple Hypervisor) Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
18. Virtualizing macOS — Performance — Bugs — Compatibility Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
19. Virtualizing macOS: Pricing — Anka: $600/core/year. 10 Mac minis * 6 cores = $36k/year — Orka: starting at $18k/year. Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
20. Bare Metal Mac Hardware Intel Mac mini (6-core 3.2 GHz, 32 GB RAM, 512 GB SSD) Retails at $1,899 Geekbench Score: 1100/5465 — MacWeb: $1,680/year — MacStadium: $2,148/year — Flow: $5,400/year — Amazon AWS: $9,487/year Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
21. Bare Metal Apple Silicon Apple M1 Mac mini (16 GB RAM, 1TB SSD) Retails at $1,299 Geekbench Score: 1705/7379 — MacWeb: $1,680/year — MacStadium: $1,788/year Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
22. Apple Silicon woes — iOS 14 Simulator is arm64; iOS 13 and below uses Intel (via Rosetta 2) — Simulator leaks resources, machines need to be rebooted from time to time — WebKit code crashes on iOS < 14 in bmalloc::HeapConstants::HeapConstants2 — We found architecture specific bugs in our codebase! 2 https://steipete.com/posts/apple-silicon-mac-mini-for-ci Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
23. Detect Rose!a 2 let NATIVE_EXECUTION = Int32(0) let EMULATED_EXECUTION = Int32(1) let UNKNOWN_EXECUTION = -Int32(1) /// Test if the process runs natively or under Rosetta private func processIsTranslated() -> Int32 { let key = "sysctl.proc_translated" var ret = Int32(0) var size: Int = 0 sysctlbyname(key, nil, &size, nil, 0) let result = sysctlbyname(key, &ret, &size, nil, 0) if result == -1 { if errno == ENOENT { return 0 } return -1 } return ret } Terminal: sysctl -in sysctl.proc_translated Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
24. What about the Mac Pro? 2.5GHz 28-core Intel Xeon. 192GB RAM. 8TB storage. Retails $19,099 Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
25. Automatic Machine Setup Requirement: Nodes need to have the same configuration Solution: Configuration Management Tools Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
26. Configuration Management Tools Script Name Script Store Scripting Language Ansible Playbooks Ansibe Galaxy YAML Chef Cookbooks Chef Supermarket Ruby Puppet Manifests Puppet Forge Puppet DSL (Ruby based) Salt Formulas Salt Stack Formulas YAML Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
27. Chef and Open Source Licensing Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
28. Microsoft’s Chef Cookbook: https://github.com/microsoft/macos-cookbook
29. Spotlight provides :spotlight default_action :set property :volume, String, name_property: true property :indexed, [true, false], default: true property :searchable, [true, false], default: true action_class do def state new_resource.indexed ? 'on' : 'off' end def search new_resource.searchable ? '' : '-d' end def volume_path(volume) volume == '/' ? volume : ::File.join('/Volumes', volume) end def target_volume volume_path(new_resource.volume) end def mdutil ['/usr/bin/mdutil'] end def desired_spotlight_state [state, target_volume, search] end end action :set do macosx_service 'metadata server' do service_name 'com.apple.metadata.mds' plist '/System/Library/LaunchDaemons/com.apple.metadata.mds.plist' action [:enable, :start] end execute "turn Spotlight indexing #{state} for #{target_volume}" do command mdutil + desired_spotlight_state.insert(0, '-i') not_if { MetadataUtil.new(target_volume).status_flags == desired_spotlight_state } end end Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
30. Spotlight // attributes/defaults.rb default['pspdfkit-ci-macos']['apfs_volume']['name'] = 'CI' default['pspdfkit-ci-macos']['apfs_volume']['mount_point'] = '/Volumes/CI' // recipes/default.rb spotlight apfs_volume['name'] do indexed false searchable false end Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
31. Disk Space // attributes/defaults.rb default['pspdfkit-ci-macos']['applications']['unneeded'] = %w( GarageBand iMovie Keynote Numbers Pages ) // recipes/default.rb node['pspdfkit-ci-macos']['applications']['unneeded'].each do |application| directory "delete #{application}" do path "/Applications/#{application}.app" recursive true action :delete end end Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
32. Screensaver execute 'disable login screensaver' do command "sudo defaults write /Library/Preferences/com.apple.screensaver loginWindowIdleTime 0" end execute 'disable ci user screensaver' do command "sudo defaults write /Users/#{admin_user_name}/Library/Preferences/com.apple.screensaver idleTime 0" end execute 'disable admin user screensaver' do command "sudo defaults write /Users/#{ci_user_name}/Library/Preferences/com.apple.screensaver idleTime 0" end Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
33. Xcode default['pspdfkit-ci-macos']['xcode-install']['version'] = '2.6.8' default['pspdfkit-ci-macos']['xcode']['versions'] = [ { version: '12.3', build_version: '12C33', }, ] default['pspdfkit-ci-macos']['xcode']['simulators'] = %w( iOS\ 12.0 iOS\ 12.1 iOS\ 12.2 iOS\ 12.4 iOS\ 13.0 iOS\ 13.1 iOS\ 13.2 iOS\ 13.3 iOS\ 13.4 iOS\ 13.5 iOS\ 13.6 iOS\ 13.7 iOS\ 14.0 iOS\ 14.1 iOS\ 14.2 ) Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
34. Chef Server vs Knife Zero Plugin: https://knife-zero.github.io bundle exec knife zero converge "name:macos-macstadium{number}" Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
35. Run Command on all Agents: bundle exec knife ssh "name:macos-*" "xcodebuild -version" Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
36. Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
37. Summary — Evaluated CI orchestrators & Buildkite — Managed vs Self-Hosted Runners — Virtualization vs Bare Metal — Configuration Management Tools & Chef Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete
38. Thanks! Peter Steinberger @steipete Continuous Integration for the Rest of Us — iOS Conf SG 2021 | Peter Steinberger — @steipete