Everything-as-code. A polyglot journey.

As modern, agile developers we love to master several different languages all at once to be 100% productive. We define our development environments using Gradle. We implement our software in Java, Kotlin or another JVM based language. We use Groovy or Scala to test our code at different layers. We construct the build pipelines for our software using a Groovy DSL or JSON. We use YAML and Python to describe the infrastructure and the deployment for our applications. We document our architectures using AsciiDoc and JRuby. Welcome to Babel! Making the right choice in this multitude of available languages is not easy. This code intense, polyglot session is an opinionated journey into the modern era of software industrialization. The session has been presented at the #JavaOne 2016 conference. (c) @LeanderReimer @qaware

Transcript

1. Everything-as-code A polyglot journey. | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 1
2. About me Mario-Leander Reimer Chief Technologist, QAware GmbH mario-leander.reimer@qaware.de twitter://@LeanderReimer http://github.com/lreimer http://speakerdeck.com/lreimer http://www.qaware.de | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 2
3. | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 3
4. Which language or technology do real programmers use? | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 4
5. My #FirstSevenLanguages • Pascal • Basic • C / C++ • Assembler • PHP • Java • C# | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 5
6. My #LastSevenLanguages • Java • Groovy • TypeScript • Ruby • Kotlin • Scala • Python | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 6
7. | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 7
8. | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 8
9. | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 9
10. There is no unanimous opinion ... • http://spectrum.ieee.org/computing/software/ the-2015-top-ten-programming-languages • http://spectrum.ieee.org/computing/software/ the-2016-top-programming-languages • https://www.sitepoint.com/whats-best- programming-language-learn-2015/ | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 10
11. There is no best language! Every language is strong in a specific domain. | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 11
12. Real programmers are polyglot! | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 12
13. The IDE is our workbench. | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 13
14. open fun everythingAsCode() : Boolean { everytingIsMadeFromCode() && everythingIsMadeByCode() } val softwareIndustrialization = everythingAsCode() | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 14
15. Definition of Software Industrialization • This has nothing to do with cheap labor! • Automation of repetitive and laborious tasks • Better software quality through standardized, streamlined tool chain • Well integrated tool chain leads to a higher productivity and happiness of your team • Better cost efficiency and competitiveness | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 15
16. Quest for a polyglot project archetype • Which languages are used for specific domains in our projects? • Which tools are used for Setup, Build, Code, Test, CI, Infrastructure, Documentation? • What are the dos and don'ts of using a specific language or technology? + some wishful greenfield thinking! | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 16
17. The polyglot journey begins. | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 17
18. SEU-as-code | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 18
19. Lightweight Developer Provisioning • Use a build tool for the automated creation and update of a software development environment • Software packages are expressed as dependencies • Gradle tasks and Groovy are used instead of shell scripting • Everything is version controlled just like ordinary source code | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 19
20. plugins { id 'de.qaware.seu.as.code.base' version '2.4.0' } import static de.qaware.seu.as.code.plugins.base.Platform.isMac seuAsCode { seuHome = { if (isMac()) '/Volumes/Everything-as-code' else 'Y:' } projectName = 'Everything-as-code' } dependencies { // list of software dependencies ... software 'org.groovy-lang:groovy:2.4.7' software 'org.scala-lang:scala:2.11.8' software 'org.jruby:jruby:9.1.4.0' } | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 20
21. Build-as-code | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 21
22. Why Gradle beats Maven. • Very flexible. Gradle can build everything. • Polyglot builds are supported easily. • Succinct build scripts. Default conventions over configuration. • Incremental builds, reduced build times. • New features: Kotlin support, Composite Builds, ... • Frequent releases. Mature and stable. | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 22
23. apply plugin: 'application' apply plugin: 'war' apply plugin: 'kotlin' apply plugin: 'groovy' repositories { jcenter() } dependencies { providedCompile 'fish.payara.extras:payara-micro:4.1.1.163' // and many more ... } task everythingAsCode() << { println 'Everything-as-code @JavaOne2016.' } | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 23
24. Main-as-code | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 24
25. There is nothing wrong with Java as primary language! | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 25
26. But Kotlin is a serious alternative worth considering as primary language. | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 26
27. But why Kotlin? And not Scala, Clojure, ... • Easy to learn for a Java developer. • Null Safety! • Loads of other useful language features. • Small library size. • Good IDE support. • Wishful greenfield thinking. | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 27
28. @JsonIgnoreProperties(ignoreUnknown = true) data class Book(val title: String, val isbn: String, val author: String) { } @ApplicationScoped open class Bookshelf { private val books = listOf(Book("The Hitchhiker's Guide to the Galaxy", "0345391802")) open fun byIsbn(isbn: String): Book? = books.find { it.isbn == isbn } } @Path("books") @Produces(MediaType.APPLICATION_JSON) open class BookResource @Inject constructor(private val bookshelf: Bookshelf) { @GET @Path("/{isbn}") open fun byIsbn(@PathParam("isbn") isbn: String): Response { val book = bookshelf.byIsbn(isbn) return if (book != null) Response.ok(book).build() else Response.status(Status.NOT_FOUND).build() } } @ApplicationPath("api") class BookstoreAPI : Application() { override fun getClasses() = hashSetOf(JacksonFeature::class.java, BookResource::class.java) } | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 28
29. Test-as-code | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 29
30. Groovy and Spock for general testing class BookshelfSpec extends Specification { @Subject def bookshelf = new Bookshelf() @Unroll def "Find book #title by ISBN #isbn"() { when: 'we search a book by ISBN' def book = bookshelf.byIsbn(isbn) then: 'the title and author are correct' book?.title == title book?.author == author where: isbn || title | author "0345391802" || "The Hitchhiker's Guide to the Galaxy" | "Douglas Adams" "0345391829" || "Life, the Universe and Everything" | "Douglas Adams" } } | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 30
31. Scala and Gatling for load testing class BooksPerformanceTest extends Simulation { val conf = http.baseURL("http://localhost:18080").acceptHeader("application/json") val feeder = csv("books.csv").random val scn = scenario("Book Search") .exec(http("Get all books").get("/api/books")) .during(30 seconds) { feed(feeder) .exec(http("Get book by title ${Title}").get("/api/books?title=${Title}")) .pause(1 second) .exec(http("Get book with ISBN ${ISBN}").get("/api/books/${ISBN}")) } setUp(scn.inject(atOnceUsers(10), rampUsers(50) over (30 seconds))) .assertions(global.responseTime.max.lessThan(5000)) .protocols(conf) } | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 31
32. Pipeline-as-code | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 32
33. Use Gradle and Docker to start Jenkins task createJenkinsHome() { mkdir("${System.getProperty('user.home')}/Jenkins") } task createJenkins(type: Exec, group: 'jenkinsci', dependsOn: createJenkinsHome, description: 'Create Jenkins') { commandLine 'docker', 'run', '--name', 'jenkinsci', '-p', '8080:8080', '-p', '50000:50000', '-v', "${System.getProperty('user.home')}/Jenkins:/var/jenkins_home", 'jenkinsci/jenkins' } task startJenkins(type: Exec, group: 'jenkinsci', description: 'Start Jenkins') { commandLine 'docker', 'start', 'jenkinsci' } task stopJenkins(type: Exec, group: 'jenkinsci', description: 'Stop Jenkins') { commandLine 'docker', 'stop', 'jenkinsci' } | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 33
34. Add Jenkinsfile and define pipeline #!/usr/bin/env groovy node { stage 'Checkout SCM' checkout scm stage 'Build/Analyse/Test' sh './gradlew clean build' archiveUnitTestResults() archiveDistributions() stage 'Dockerize' sh './gradlew buildDockerImage' stage 'Generate Documentation' sh './gradlew asciidoctor' } | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 34
35. Infrastructure-as-code | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 35
36. Docker, Docker, Docker, ... FROM qaware-oss-docker-registry.bintray.io/base/debian8-jre8 MAINTAINER M.-Leander Reimer <mario-leander.reimer@qaware.de> RUN mkdir -p /app ADD build/distributions/everything-as-code-1.0.0.tar /app WORKDIR /app/everything-as-code-1.0.0 RUN chmod 755 bin/everything-as-code EXPOSE 18080 CMD ./bin/everything-as-code | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 36
37. Vagrant and Ruby for local VM setup require 'yaml' $setup = <<SCRIPT sudo apt-add-repository ppa:ansible/ansible sudo apt-get update sudo apt-get install -y ansible sshpass SCRIPT Vagrant.configure("2") do |config| config.vm.box = "ubuntu/trusty32" settings = YAML.load_file 'src/vagrant/vagrant.yml' config.vm.provider "virtualbox" do |vb| vb.name = settings['vm']['name'] vb.gui = false vb.memory = "512" end config.vm.provision "shell", inline: $setup end | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 37
38. Provisioning with Ansible (and Python) --- # file: jenkinsci.yml - hosts: jenkinsci remote_user: root tasks: - debug: msg="Creating a Jenkins pipeline job on {{ inventory_hostname }}" - jenkins_job: name: Everything-as-code Pipeline config: "{{ lookup('file', 'templates/pipeline-job.xml') }}" url: "http://{{ inventory_hostname }}" user: admin password: admin | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 38
39. Cluster orchestration with K8S --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: everything-as-code spec: replicas: 1 template: metadata: labels: tier: backend spec: containers: - name: everything-as-code image: "qaware-oss-docker-registry.bintray.io/lreimer/everything-as-code:1.0.0" ports: - containerPort: 18080 env: - name: PORT value: 18080 | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 39
40. Cluster orchestration with DC/OS { "id": "everything-as-code", "instances": 1, "cpus": 0.25, "mem": 256, "container": { "type": "DOCKER", "docker": { "image": "qaware-oss-docker-registry.bintray.io/lreimer/everything-as-code:1.0.0", "network": "BRIDGE", "portMappings": [ { "hostPort": 18080, "containerPort": 18080, "protocol": "tcp", "servicePort": 18080 } ] } }, "env": { "PORT": 18080 } } | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 40
41. Documentation-as-code | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 41
42. Yes, we need documentation! • And no, the source code is not enough. • Writing technical docs with Word is ! " # • Documentation should be located next to the source code: change code, change docs. • It should be easy, quick and fun to write. • Support for code, images, UML diagrams, ... | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 42
43. AsciidoctorJ and Gradle to the rescue plugins { id "org.asciidoctor.convert" version "1.5.3" } asciidoctorj { version = '1.5.4.1' } asciidoctor { sourceDir 'src/docs/architecture' resources { from('src/docs/architecture') { include 'images/**/*.png' include 'images/**/*.jpg' } } backends 'html5' options doctype: 'article' attributes 'source-highlighter': 'coderay' } | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 43
44. // Example architecture documentation using arc42 (https://arc42.github.io) :imagesdir: ./images = image:qaware-logo.png[QAware GmbH,2016] Everything-as-code :toc-title: Table of Contents :toc: [[section-introduction-and-goals]] == Introduction and Goals The introduction to the architecture documentation should list the driving forces that software architects must consider in their decisions. === Requirements Overview === Quality Goals === Stakeholders <<<< include::02_architecture_constraints.adoc[] // further includes for the remaining sections | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 44
45. Presentation-as-code | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 45
46. These slides are written in Markdown. --- ## [fit] These slides are written in Markdown. - This is for real programmers! :smiley: - Several open source projects available - Use HTML and JavaScript alternatively. --- | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 46
47. What's the takeaway from this journey? | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 47
48. Beware of the abstractions! | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 48
49. Everyone needs to do his homework. • Developers: be polyglot, keep learning! • Architects: choose the right language or tool for the job! • Project Managers: give your techies freedom! • Universities: teach polyglotism! | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 49
50. Have some fun and create your own polyglot project archetype. | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 50
51. Q & A | JavaOne 2016 | Everything-as-code -> { created with ❤ and ☕ by @LeanderReimer } 51