Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cloud Native Computing - Introduction to Fluentd

Cloud Native Computing - Introduction to Fluentd

A short introduction into the Cloud Native Computing Foundation project Fluentd - Held at the first Cloud Native Computing Meetup in Zurich at 2017-05-11

Tobias Brunner

May 11, 2017
Tweet

More Decks by Tobias Brunner

Other Decks in Technology

Transcript

  1. VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch
    Cloud Native Computing Switzerland
    11.5.2017
    CNCF Project:
    Fluentd
    Tobias Brunner
    VSHN AG

    View full-size slide

  2. VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch
    Agenda

    About /me

    What is it?

    How does it work?

    Use case

    Demo?

    View full-size slide

  3. VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch
    /home/tobru
    Tobias Brunner, Head of DevOps VSHN
    @tobruzh, [email protected]
    VSHN AG
    Since 2014, 18 people in Zürich
    Running web applications on-premises
    and in the clouds making both visitors
    and developers happy
    https://vshn.ch @vshn_ch

    View full-size slide

  4. VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch
    What is Fluentd?

    View full-size slide

  5. VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch
    What is Fluentd?
    Fluentd moves logs around:

    Collect logs from various sources

    Ship logs to various sinks
    500+ Plugins, small core

    Plugins are distributed – no central
    repo
    http://www.fluentd.org/plugins/all

    View full-size slide

  6. VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch
    What is Fluentd?
    Fluent Bit: http://fluentbit.io/

    Lightweight Forwarder: Specifically
    designed for data forwarding

    Written in C

    View full-size slide

  7. VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch
    How does it work?

    Declarative Syntax

    A Fluentd event consists of a tag,
    time and record:

    tag: Where an event comes from. For
    message routing

    time: When an event happens. Epoch
    time

    record: Actual log content. JSON
    object

    View full-size slide

  8. VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch
    How does it work?
    Input -> Filter -> Output

    @type tail
    path /var/log/nginx/access.log
    pos_file /var/log/nginx/pos
    tag nginx.access
    format nginx


    @type record_transformer

    hostname "#{Socket.gethostname}"



    @type copy

    @type elasticsearch
    host elasticsearch
    port 9200
    include_tag_key true
    tag_key @log_name
    logstash_format true
    flush_interval 10s


    @type file
    path /tmp/nginxlogs
    time_slice_format %Y%m%d
    time_slice_wait 10m
    time_format %Y%m%dT%H%M%S%z
    compress gzip


    View full-size slide

  9. VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch
    Use case
    OpenShift: Central logging / EFK
    Stack

    Collects logs from containers

    Ships to Elasticsearch

    UI with Kibana

    Runs as DaemonSet on all nodes

    Mounts needed log directories from
    host

    Parses JSON Logs into fields

    View full-size slide

  10. VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch
    Demo

    All running in Docker: Elasticsearch,
    Fluentd, Kibana

    View full-size slide

  11. VSHN AG I Neugasse 10 I 8005 Zürich I T 044 545 53 00 www.vshn.ch
    End / Questions

    View full-size slide