Speed up your build with remote servers

Talk where I explain how you can use a remote server to speed up your build using Mainframer script and/or Mirakle plugin

Transcript

1. Speeding up your build with remote build servers
2. Enrique Ramirez Android dev @kikermo
3. Situation
4. Solution Buy a more expensive laptop
5. Alternative REMOTE BUILDS SOURCE BUILD FILES compiling
6. Requirements
7. Server side SDK >_ssh server no GUI
8. Client side (*) >_ssh
9. First approach Copy entire codebase to remote server build and run remotely adb rsync over tcp
10. $ rsync <local_project_path> <user>@<remote_server_ip>://<remote_project_path> $ rsync ./ kikermo@192.168.2.112://home/kikermo/workspace/todoApp/ $ ssh <user>@<remote_server_ip> $ ssh kikermo@192.168.2.112 $ ./gradlew <gradlew_task> $ ./gradlew assembleDebug $ adb install -r <path_to_apk> $ adb install -r app/build/output/apk/debug/myapp.apk $ adb shell am start -n <application_id>/<launch_activity> $ adb shell am start -n org.myapp/org.myapp.LaunchActivity
11. mainframer.sh Mainframer script (v2) adb rsync src rsync build
12. $ ./mainframer.sh ./gradlew <gradlew_task> $ ./mainframer.sh ./gradlew assembleDebug $ adb install -r <path_to_apk> $ adb install -r app/build/output/apk/debug/myapp.apk $ adb shell am start -n <application_id>/<launch_activity> $ adb shell am start -n org.myapp/org.myapp.LaunchActivity
14. Mirakle plugin Mirakle plugin adb rsync src rsync build
16. Configuration
17. Server side 1. Choose a linux distro and install it. 2. Install Java. 3. Install android sdk. 4. Make sure you have an ssh server installed. 5. Create a user for the each dev (useradd). 6. Add sdk path to /etc/environment /etc/environments export ANDOROID_HOME=/usr/lib/android-sdk export ANDROID_SDK_ROOT=/usr/lib/android-sdk
18. Client side (common) 1. Generate rsa key (if you don’t have one already). 2. Copy public key to remote server. 3. Create an ssh config file. 4. Check you can ssh into the remote machine without using a password.
19. ~/.ssh/config Host * UseKeychain yes AddKeysToAgent yes IdentityFile ~/.ssh/id_rsa Host mainframer User kikermo HostName 192.168.2.112 Port 22 IdentityFile ~/.ssh/id_rsa.pub PreferredAuthentications publickey ControlMaster auto ControlPath /tmp/%r@%h:%p ControlPersist 1 $ ssh-keygen $ ssh-copy-id <user>@<remote_machine> $ vim ~/.ssh/config $ ssh-add $ ssh mainframer
20. Client side (mainframer) 1. Download mainframer v2 script 2. Put it in your project 3. Configure ignore files for the sync a. Config b. Common ignore c. Local -> Remote ignore d. Remote -> Local ignore 4. Aliases (Optional)
21. $ wget https://raw.githubusercontent.com/buildfoundation/mainframer/v2.1.0/mainframer.sh <project_dir>/.mainframer/config remote_machine=mainframer local_compression_level=1 remote_compression_level=5 <project_dir>/.mainframer/ignore .idea/* *.iml *.DS_Store local.properties <project_dir>/.mainframer/localignore build <project_dir>/.mainframer/remoteignore src
22. Aliases ~/.bash_profile or ~/.bashrc … alias mf=’./mainframer.sh’ alias installDebug=’adb install -r app/outputs/apk/<your_build_variant_dir>’ alias runDebug=’adb shell am start -n <application_id>/<launch_activity>’ alias mfBuild=’./mainframer.sh ./gradlew assmeble<YourVariant> && installDebug && runDebug’ alias mfCleanBuild=’./mainframer.sh ./gradlew clean assmeble<YourVariant> && installDebug && runDebug’ alias mfTest=’./mainframer.sh ./gradlew <your_gradle_task_for_tests>’ alias mfSA=’./mainframer.sh ./gradlew <your_gradle_task_for_static_analysis>’
23. Client side (mirakle) 1. Create mirakle config file. ~/.gradle/init.d/mirakle_init.gradle initscript { repositories { jcenter() } dependencies { classpath "com.instamotor:mirakle:1.3.2" } } apply plugin: Mirakle rootProject { mirakle { host "mainframer" } }
24. The Windows Problem
25. The Windows Problem Although there is a native implementation of SSH, there are some required UNIX based commands that are not implemented the OS such as rsync. Possible solutions • Get a macbook • MinGW • Cygwin • Docker • WSL (Windows Subsystem for Linux)
26. WSL (Windows Subsystem for Linux) Admin PowerShell Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux
27. Useful tricks • MDNS/DNS-SD/Bonjour/Zeroconf (only local) • Extended use of aliases ◦ Enable/Disable mirakle ◦ Do syncing in multiple steps • Bash script for new starters • Dynamic user creation on server (AD/LDAP/Kerberos) • SSH tunelling (management) • Cronjobs ◦ Restart services periodically ◦ Create little reports of usage
28. Advantages • Laptop doesn’t heat too much & battery last longer. • IDE doesn’t become unresponsive when building. • You can run test or static analysis in parallel while you code. • Slow networks might make the build syncing slower than building locally. • Requires configuration for every new developer. • Changing branches could generate a strange issues, needing to delete the remote workspace. • Requires a bit of maintenance. Disadvantage
29. Links • @MarkAllison (Styling Android) https://blog.stylingandroid.com/mainframer-mirakle/ • @Balachandarkm https://medium.com/@balachandarkm/speed-up-android-builds -with-mainframer-docker-container-in-a-powerful-remote-ma chine-ce7b9e122a49 • Mainframer repo https://github.com/buildfoundation/mainframer • Mirake rempo https://github.com/Instamotor-Labs/mirakle
30. Links • Active Directory dynamic user creation https://help.ubuntu.com/lts/serverguide/sssd-ad.html • SSH tunneling https://help.ubuntu.com/community/SSH/OpenSSH/PortForward ing
31. Configuration Links • /etc/environmetns • ~/.ssh/config • <project_dir>/.mainframer/config • <project_dir>/.mainframer/ignore • <project_dir>/.mainframer/localignore • <project_dir>/.mainframer/remoteignore • Aliases ~/.bash_profile or ~/.bashrc • ~/.gradle/init.d/mirakle_init.gradle
32. THANKS Q&A