1 Pere Urbon-Bayes Software Engineer Elasticsearch Let's validate your releases with JRuby, RSpec and SSH 

2 $>whoami • Pere Urbon-Bayes (software engineer). • Have always worked with databases, data and analytics. • Graphdevroom@FOSDEM • When not coding I enjoy my time with my kid and wife. I also enjoy movies, tv series and running too. Basically, I’m doing everything I can to enjoy life. 

3 About testing? •How many of you test in regular basis? TDD? •How many do integration testing? •Acceptance testing? •User acceptance testing? •Other levels of validation? Gossiping around 

4 Testing your releases with JRuby, SSH and Vagrant 

5 Logstash 

6 Released as How logstash is delivered to the public 

7 The operating system matrix where it should run for sure. 

8 The plugins Why everything is so diverse… 

9 BOOM! WayTooManyOptionsException! 

10 Where everything started •Logstash 1.5 took long time to release (nearly a year). •https://www.elastic.co/blog/logstash-1-5-0-ga-released •Releases cadence need to be speed up. •Confidence with shipping new features and refactoring. •Manual testing never scales!. •A few times bugs discovered after release that would have been possible to caught with proper testing. •https://github.com/elastic/logstash/issues/4981 there is always a reason for it 

11 What do we want to need? •Automatic test with many different operating system’s. •Test should be OS independent. we should run on windows •Possible to be run locally, but also in CI environment without much hassle. •Test should be as uniform as possible with other elastic products. for ops 

12 What do we want to need? •Bootstrap machines to test specific situations on demand. •Have confidence with releases. •Test complex situations quick and easy. •Tests are easy to write and wire up. for developers 

13 

14 Vagrant OR Docker hosting the moving components 

15 Vagrant OR Docker •Quick to setup for integration purposes, started with zabbix, end it up with elasticsearch. •Maintenance for multiple configurations make it hardest. (new img’s) •We need test repeatable, we found this difficult for different persons (environment) and different external services versions. •Networking was a bit of an issue… •(elasticsearch cluster, VM’s, using mac’s to develop) •Docker API changes very frequently, it’s not really trusty. •The toolset had hard times to keep up with api changes. Why not docker? 

16 Docker Or Vagrant Vagrant, but? https://github.com/mitchellh/vagrant/issues/391 vagrant ssh only possible after restart network really? 

17 OR bare metal? why not? 

18 OpenSSH code once, execute many 

19 Test executors How to write your test code •RSpec http://rspec.info/ •Bacon https://github.com/chneukirchen/bacon •Test::Unit https://github.com/test-unit/test-unit •Minitest https://github.com/seattlerb/minitest Lots of options to write your test in Ruby, so we had to choose 

20 And the winner is? RSpec…. 

21 Options to run our integration test with RSpec •ServerSpec/ServerInfra v2 To recicle is good, right? With Serverspec, you can write RSpec tests for checking your servers are configured correctly. Serverspec tests your servers' actual state by executing command locally, via SSH, via WinRM, via Docker API and so on. So you don't need to install any agent softwares on your servers and can use any configuration management tools, Puppet, Ansible, CFEngine, Itamae and so on. But the true aim of Serverspec is to help refactoring infrastructure code. 

22 ServerSpec / ServerInfra v2 •Lots features and capabilities, from networking to X509 certificates. •Extensive list of OS supported. •Focused on checking infrastructure, not acceptance testing. •Overcomplicated syntax for a testing framework. The solution to test your infrastructure describe service('org.apache.httpd'), :if => os[:family] == 'darwin' do it { should be_enabled } it { should be_running } end 

23 Options to run our integration test with RSpec •Test Kitchen or Kitchen-CI Not good, but … Test Kitchen is a test harness tool to execute your configured code on one or more platforms in isolation. A driver plugin architecture is used which lets you run your code on various cloud providers and virtualization technologies such as Amazon EC2, Blue Box, CloudStack, Digital Ocean, Rackspace, OpenStack, Vagrant, Docker, LXC containers, and more. Many testing frameworks are already supported out of the box including Bats, shUnit2, RSpec, Serverspec, with others being created weekly. 

24 Kitchen CI •Lots of versatility, extensive list of supported providers and operating system’s. •Designed to test infra, but versatile enough to run QA testing. •Provision using chef. •RSpec test written using serverspec, but could also use bats (bash based) and other frameworks. •In our experience it was really slow due to all the necessary bootstrap involved. An intension to provide a complete automated solution for testing It will really be an awesome project in the feature 

25 Build your own 

26 Our solution to the problem Test Code + Communications + Machine providers RSpec Layer VM 0 (ubuntu-1404) VM 1 (debian-8) VM n (windows?) ssh ssh ssh / win RPC command runner serial / parallel ssh, and other protocols local execution provisioning * scripts ansible chef …. Machines Tests Communications x 

27 The Test layer 

28 The Communications layer how things get send to their final destination 

29 The Machinery provision 

30 The Machinery provision 

31 And we run this.. on jenkins, but there is no restriction 

32 

33 Certainly others are using other tools •Bats scripts run over vagrant ssh abstraction (elasticsearch) •Ansible as test executor over docker (beats) •…. There is a solution for everyones taste Are you doing anything similar? 

34 Brown bag lunch The elastic way…. 

35 Pere Urbon-Bayes Software Engineer Elasticsearch Thank you very much, questions?