Cloud Native Computing - Introduction to Fluentd

A short introduction into the Cloud Native Computing Foundation project Fluentd - Held at the first Cloud Native Computing Meetup in Zurich at 2017-05-11

Transcript

1. VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch Cloud Native Computing Switzerland 11.5.2017 CNCF Project: Fluentd Tobias Brunner VSHN AG
2. VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch Agenda • About /me • What is it? • How does it work? • Use case • Demo?
3. VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch /home/tobru Tobias Brunner, Head of DevOps VSHN @tobruzh, tobias.brunner@vshn.ch VSHN AG Since 2014, 18 people in Zürich Running web applications on-premises and in the clouds making both visitors and developers happy https://vshn.ch @vshn_ch
4. VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch What is Fluentd?
5. VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch What is Fluentd? Fluentd moves logs around: • Collect logs from various sources • Ship logs to various sinks 500+ Plugins, small core • Plugins are distributed – no central repo http://www.fluentd.org/plugins/all
6. VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch What is Fluentd? Fluent Bit: http://fluentbit.io/ • Lightweight Forwarder: Specifically designed for data forwarding • Written in C
7. VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch How does it work? • Declarative Syntax • A Fluentd event consists of a tag, time and record: • tag: Where an event comes from. For message routing • time: When an event happens. Epoch time • record: Actual log content. JSON object
8. VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch How does it work? Input -> Filter -> Output <source> @type tail path /var/log/nginx/access.log pos_file /var/log/nginx/pos tag nginx.access format nginx </source> <filter nginx.access> @type record_transformer <record> hostname "#{Socket.gethostname}" </record> </filter> <match nginx.access> @type copy <store> @type elasticsearch host elasticsearch port 9200 include_tag_key true tag_key @log_name logstash_format true flush_interval 10s </store> <store> @type file path /tmp/nginxlogs time_slice_format %Y%m%d time_slice_wait 10m time_format %Y%m%dT%H%M%S%z compress gzip </store> </match>
9. VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch Use case OpenShift: Central logging / EFK Stack • Collects logs from containers • Ships to Elasticsearch • UI with Kibana • Runs as DaemonSet on all nodes • Mounts needed log directories from host • Parses JSON Logs into fields
10. VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch Demo • All running in Docker: Elasticsearch, Fluentd, Kibana
11. VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch End / Questions