Speed up your build with remote servers

Talk where I explain how you can use a remote server to speed up your build using Mainframer script and/or Mirakle plugin

Transcript

1. Speeding up your build with remote build servers
2. Enrique Ramirez Android dev @kikermo
3. Situation
4. Solution Buy a more expensive laptop
5. Alternative REMOTE BUILDS SOURCE BUILD FILES compiling
6. Requirements
7. Server side SDK >_ssh server no GUI
8. Client side (*) >_ssh
9. First approach Copy entire codebase to remote server build and run remotely adb rsync over tcp
10. $ rsync <local_project_path> <user>@<remote_server_ip>://<remote_project_path> $ rsync ./ kikermo@192.168.2.112://home/kikermo/workspace/todoApp/ $ ssh <user>@<remote_server_ip> $ ssh kikermo@192.168.2.112 $ ./gradlew <gradlew_task> $ ./gradlew assembleDebug $ adb install -r <path_to_apk> $ adb install -r app/build/output/apk/debug/myapp.apk $ adb shell am start -n <application_id>/<launch_activity> $ adb shell am start -n org.myapp/org.myapp.LaunchActivity
11. mainframer.sh Mainframer script (v2) adb rsync src rsync build
12. $ ./mainframer.sh ./gradlew <gradlew_task> $ ./mainframer.sh ./gradlew assembleDebug $ adb install -r <path_to_apk> $ adb install -r app/build/output/apk/debug/myapp.apk $ adb shell am start -n <application_id>/<launch_activity> $ adb shell am start -n org.myapp/org.myapp.LaunchActivity
14. Mirakle plugin Mirakle plugin adb rsync src rsync build
16. Configuration
17. Server side 1. Choose a linux distro and install it. 2. Install Java. 3. Install android sdk. 4. Make sure you have an ssh server installed. 5. Create a user for the each dev (useradd). 6. Add sdk path to /etc/environment /etc/environments export ANDOROID_HOME=/usr/lib/android-sdk export ANDROID_SDK_ROOT=/usr/lib/android-sdk
18. Client side (common) 1. Generate rsa key (if you don’t have one already). 2. Copy public key to remote server. 3. Create an ssh config file. 4. Check you can ssh into the remote machine without using a password.
19. ~/.ssh/config Host * UseKeychain yes AddKeysToAgent yes IdentityFile ~/.ssh/id_rsa Host mainframer User kikermo HostName 192.168.2.112 Port 22 IdentityFile ~/.ssh/id_rsa.pub PreferredAuthentications publickey ControlMaster auto ControlPath /tmp/%r@%h:%p ControlPersist 1 $ ssh-keygen $ ssh-copy-id <user>@<remote_machine> $ vim ~/.ssh/config $ ssh-add $ ssh mainframer
20. Client side (mainframer) 1. Download mainframer v2 script 2. Put it in your project 3. Configure ignore files for the sync a. Config b. Common ignore c. Local -> Remote ignore d. Remote -> Local ignore 4. Aliases (Optional)
21. $ wget https://raw.githubusercontent.com/buildfoundation/mainframer/v2.1.0/mainframer.sh <project_dir>/.mainframer/config remote_machine=mainframer local_compression_level=1 remote_compression_level=5 <project_dir>/.mainframer/ignore .idea/* *.iml *.DS_Store local.properties <project_dir>/.mainframer/localignore build <project_dir>/.mainframer/remoteignore src
22. Aliases ~/.bash_profile or ~/.bashrc … alias mf=’./mainframer.sh’ alias installDebug=’adb install -r app/outputs/apk/<your_build_variant_dir>’ alias runDebug=’adb shell am start -n <application_id>/<launch_activity>’ alias mfBuild=’./mainframer.sh ./gradlew assmeble<YourVariant> && installDebug && runDebug’ alias mfCleanBuild=’./mainframer.sh ./gradlew clean assmeble<YourVariant> && installDebug && runDebug’ alias mfTest=’./mainframer.sh ./gradlew <your_gradle_task_for_tests>’ alias mfSA=’./mainframer.sh ./gradlew <your_gradle_task_for_static_analysis>’
23. Client side (mirakle) 1. Create mirakle config file. ~/.gradle/init.d/mirakle_init.gradle initscript { repositories { jcenter() } dependencies { classpath "com.instamotor:mirakle:1.3.2" } } apply plugin: Mirakle rootProject { mirakle { host "mainframer" } }
24. The Windows Problem
25. The Windows Problem Although there is a native implementation of SSH, there are some required UNIX based commands that are not implemented the OS such as rsync. Possible solutions • Get a macbook • MinGW • Cygwin • Docker • WSL (Windows Subsystem for Linux)
26. WSL (Windows Subsystem for Linux) Admin PowerShell Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux
27. Useful tricks
28. MDNS/DNS-SD/Bonjour/Zeroconf • Allows you to discover your remote machine without having to use an IP address. • Only work on local networks, no VPN. • AVAHI server, common open source solution. $ ping 192.168.2.112 without $ ping myserver.local with
29. Aliases (extended) ~/.bash_profile or ~/.bashrc … ## Mirakle aliases alias mirakleOff=’mv ~/.gradle/init.d/mirakle_init.gradle ~/.gradle/init.d/mirakle_init.gradle.conf’ alias mirakleOn=’mv ~/.gradle/init.d/mirakle_init.gradle.conf ~/.gradle/init.d/mirakle_init.gradle’ ## Mainframer aliases alias speedUp=’echo -e “build\n!app/build/outputs”>> .mainframer/remoteignore’ alias slowDown=’git checkout .mainframer/remoteignore’ ## discard changes alias build=’speedUp && ./mainframer.sh ./gradlew assembleDebug && installDebug && runDebug && slowDown && ./mainframer.sh’
30. Dynamic user creation on server • Compatible with AD/LDAP. • User creation on the fly. • Kerberos. • Needs admin authorization.
31. Bash script for new users • Helps onboarding new devs. • Avoids unnecessary one time learning curve • Less troubleshooting needed
32. SSH Tunneling remote build server reachable server ssh
33. Advantages • Laptop doesn’t heat too much & battery last longer. • IDE doesn’t become unresponsive when building. • You can run test or static analysis in parallel while you code. • Slow networks might make the build syncing slower than building locally. • Requires configuration for every new developer. • Changing branches could generate a strange issues, needing to delete the remote workspace. • Requires a bit of maintenance. Disadvantage
34. Links • @MarkAllison (Styling Android) https://blog.stylingandroid.com/mainframer-mirakle/ • @Balachandarkm https://medium.com/@balachandarkm/speed-up-android-builds -with-mainframer-docker-container-in-a-powerful-remote-ma chine-ce7b9e122a49 • Mainframer repo https://github.com/buildfoundation/mainframer • Mirake rempo https://github.com/Instamotor-Labs/mirakle
35. Links • Active Directory dynamic user creation https://help.ubuntu.com/lts/serverguide/sssd-ad.html • SSH tunneling https://help.ubuntu.com/community/SSH/OpenSSH/PortForward ing
36. Configuration Links • /etc/environmetns • ~/.ssh/config • <project_dir>/.mainframer/config • <project_dir>/.mainframer/ignore • <project_dir>/.mainframer/localignore • <project_dir>/.mainframer/remoteignore • Aliases ~/.bash_profile or ~/.bashrc • ~/.gradle/init.d/mirakle_init.gradle
37. THANKS Q&A