2 $>whoami • Pere Urbon-Bayes (software engineer). • Have always worked with databases, data and analytics. • Graphdevroom@FOSDEM • When not coding I enjoy my time with my kid and wife. I also enjoy movies, tv series and running too. Basically, I’m doing everything I can to enjoy life.
3 About testing? •How many of you test in regular basis? TDD? •How many do integration testing? •Acceptance testing? •User acceptance testing? •Other levels of validation? Gossiping around
10 Where everything started •Logstash 1.5 took long time to release (nearly a year). •https://www.elastic.co/blog/logstash-1-5-0-ga-released •Releases cadence need to be speed up. •Confidence with shipping new features and refactoring. •Manual testing never scales!. •A few times bugs discovered after release that would have been possible to caught with proper testing. •https://github.com/elastic/logstash/issues/4981 there is always a reason for it
11 What do we want to need? •Automatic test with many different operating system’s. •Test should be OS independent. we should run on windows •Possible to be run locally, but also in CI environment without much hassle. •Test should be as uniform as possible with other elastic products. for ops
12 What do we want to need? •Bootstrap machines to test specific situations on demand. •Have confidence with releases. •Test complex situations quick and easy. •Tests are easy to write and wire up. for developers
15 Vagrant OR Docker •Quick to setup for integration purposes, started with zabbix, end it up with elasticsearch. •Maintenance for multiple configurations make it hardest. (new img’s) •We need test repeatable, we found this difficult for different persons (environment) and different external services versions. •Networking was a bit of an issue… •(elasticsearch cluster, VM’s, using mac’s to develop) •Docker API changes very frequently, it’s not really trusty. •The toolset had hard times to keep up with api changes. Why not docker?
19 Test executors How to write your test code •RSpec http://rspec.info/ •Bacon https://github.com/chneukirchen/bacon •Test::Unit https://github.com/test-unit/test-unit •Minitest https://github.com/seattlerb/minitest Lots of options to write your test in Ruby, so we had to choose
21 Options to run our integration test with RSpec •ServerSpec/ServerInfra v2 To recicle is good, right? With Serverspec, you can write RSpec tests for checking your servers are configured correctly. Serverspec tests your servers' actual state by executing command locally, via SSH, via WinRM, via Docker API and so on. So you don't need to install any agent softwares on your servers and can use any configuration management tools, Puppet, Ansible, CFEngine, Itamae and so on. But the true aim of Serverspec is to help refactoring infrastructure code.
22 ServerSpec / ServerInfra v2 •Lots features and capabilities, from networking to X509 certificates. •Extensive list of OS supported. •Focused on checking infrastructure, not acceptance testing. •Overcomplicated syntax for a testing framework. The solution to test your infrastructure describe service('org.apache.httpd'), :if => os[:family] == 'darwin' do it { should be_enabled } it { should be_running } end
23 Options to run our integration test with RSpec •Test Kitchen or Kitchen-CI Not good, but … Test Kitchen is a test harness tool to execute your configured code on one or more platforms in isolation. A driver plugin architecture is used which lets you run your code on various cloud providers and virtualization technologies such as Amazon EC2, Blue Box, CloudStack, Digital Ocean, Rackspace, OpenStack, Vagrant, Docker, LXC containers, and more. Many testing frameworks are already supported out of the box including Bats, shUnit2, RSpec, Serverspec, with others being created weekly.
24 Kitchen CI •Lots of versatility, extensive list of supported providers and operating system’s. •Designed to test infra, but versatile enough to run QA testing. •Provision using chef. •RSpec test written using serverspec, but could also use bats (bash based) and other frameworks. •In our experience it was really slow due to all the necessary bootstrap involved. An intension to provide a complete automated solution for testing It will really be an awesome project in the feature
26 Our solution to the problem Test Code + Communications + Machine providers RSpec Layer VM 0 (ubuntu-1404) VM 1 (debian-8) VM n (windows?) ssh ssh ssh / win RPC command runner serial / parallel ssh, and other protocols local execution provisioning * scripts ansible chef …. Machines Tests Communications x
33 Certainly others are using other tools •Bats scripts run over vagrant ssh abstraction (elasticsearch) •Ansible as test executor over docker (beats) •…. There is a solution for everyones taste Are you doing anything similar?
purbon
kashira
bengo4com
hagevvashi
linedevth
moriyuya
shimacos
tatsuru
miyashino
nishiuma
lmt_swallow
soracom
culvert
marktimemedia
paigeccino
philhawksworth
danielanewman
addyosmani
3n
afnizarnur
smashingmag
tenderlove
eileencodes