VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch Cloud Native Computing Switzerland 11.5.2017 CNCF Project: Fluentd Tobias Brunner VSHN AG 

VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch Agenda ● About /me ● What is it? ● How does it work? ● Use case ● Demo? 

VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch /home/tobru Tobias Brunner, Head of DevOps VSHN @tobruzh, [email protected] VSHN AG Since 2014, 18 people in Zürich Running web applications on-premises and in the clouds making both visitors and developers happy https://vshn.ch @vshn_ch 

VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch What is Fluentd? 

VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch What is Fluentd? Fluentd moves logs around: ● Collect logs from various sources ● Ship logs to various sinks 500+ Plugins, small core ● Plugins are distributed – no central repo http://www.fluentd.org/plugins/all 

VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch What is Fluentd? Fluent Bit: http://fluentbit.io/ ● Lightweight Forwarder: Specifically designed for data forwarding ● Written in C 

VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch How does it work? ● Declarative Syntax ● A Fluentd event consists of a tag, time and record: ● tag: Where an event comes from. For message routing ● time: When an event happens. Epoch time ● record: Actual log content. JSON object 

VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch How does it work? Input -> Filter -> Output @type tail path /var/log/nginx/access.log pos_file /var/log/nginx/pos tag nginx.access format nginx @type record_transformer hostname "#{Socket.gethostname}" @type copy @type elasticsearch host elasticsearch port 9200 include_tag_key true tag_key @log_name logstash_format true flush_interval 10s @type file path /tmp/nginxlogs time_slice_format %Y%m%d time_slice_wait 10m time_format %Y%m%dT%H%M%S%z compress gzip 

VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch Use case OpenShift: Central logging / EFK Stack ● Collects logs from containers ● Ships to Elasticsearch ● UI with Kibana ● Runs as DaemonSet on all nodes ● Mounts needed log directories from host ● Parses JSON Logs into fields 

VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch Demo ● All running in Docker: Elasticsearch, Fluentd, Kibana 

VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch End / Questions