Fluentd - CNCF Paris

Talk on Fluentd with introduction on what it is, how it works, and some real life feedback on its usage. This was presented at the Cloud Native Paris Meetup on 15th February 2018 : https://www.meetup.com/Cloud-Native-Computing-Paris/events/247273583/

Transcript

1. Fluentd Dynamic configuration in an immutable infrastructure context Fluentd
2. Fluentd Who am I ▼ Alexis “Horgix” Chotard ▼ Systems Engineer & Consultant @ Xebia ▼ Love to automate things ▼ Striving to do everything cleanly 2 Horgix Horgix
3. Fluentd What I’ll present you today ▼ Theory ▽ Introduction to Fluentd ▽ How Fluentd works ▼ Experience feedback ▽ Context @ Photobox ▽ Photobox logs lifecycle ▼ Conclusion 3
4. Fluentd Introduction to Fluentd 4
5. Fluentd WTF is Fluentd “Fluentd? It’s Logstash but done right” 5
6. Fluentd Problem it wants to solve 6
7. Fluentd How it wants to solve it 7
8. Fluentd How it’s made ▼ Written in Ruby, but with performance sensitive parts in C ▼ Uses MessagePack ▼ 650+ plugins (data sources / data output) ▼ “Regular PC box can handle 18,000 messages/second with a single process” 8 https://msgpack.org/
9. Fluentd And a bit of history... Joins the CNCF in November 9 https://www.cncf.io/blog/2016/11/09/fluentd-joins-cloud-native-computing-foundation/ 2011 2016 ▼ Conceived by Treasure Data ▼ Open-sourced in October
10. Fluentd Distribution Fluentd VS td-agent: ▼ td-agent is a stable distribution package of Fluentd ▼ Kind of the same thing than Moby/Docker ▼ Community Support vs Treasure Data, Inc maintainers ▼ Gem vs rpm/deb/dmg packages ▼ ... 10
11. Fluentd Distribution Others : ▼ Maintained Docker image available ▼ Official Kubernetes Daemonset ▼ Fluent Bit / td-agent-bit : lightweight forwarder only 11 https://hub.docker.com/r/fluent/fluentd https://docs.fluentd.org/v0.12/articles/kubernetes-fluent
12. Fluentd How Fluentd works 12
13. Fluentd Log lifecycle ▼ Fluentd get logs from inputs ▼ ... and associate a (single) tag to it ▼ Matches the tag against outputs ▼ Then send the log accordingly ▼ Can also re-emit logs after modification with a new tag : filtering ! Input → filter 1 → ... → filter N → Output 13
14. Fluentd Log lifecycle <source> @type http port 9880 </source> <filter myapp.access> @type record_transformer <record> hello "Cloud Native Paris" </record> </filter> <match myapp.access> @type file path /var/log/fluent/access.log </match> 14 http://the.fluentd.host.example:9880/myapp.access?json={"event":"data"}
15. Fluentd Demo Docker log driver :) 15
16. Fluentd Logging driver for Kubernetes “Kubernetes doesn’t specify a logging agent, but two optional logging agents are packaged with the Kubernetes release: Stackdriver Logging for use with Google Cloud Platform, and Elasticsearch. You can find more information and instructions in the dedicated documents. Both use fluentd with custom configuration as an agent on the node.” 16 https://kubernetes.io/docs/concepts/cluster-administration/logging/
17. Fluentd Context @ 17
18. Fluentd Technical context Immutable infrastructure : 1. Build an image 2. Deploy this image, without changing it across environments 18
19. Fluentd Goals ▼ Do things cleanly, as Cloud Native as possible ▼ Ship logs! ▽ Logstash? Nope, decision already made ▽ Let's give a try to fluentd (td-agent actually) ▼ Have configuration that is re-usable across projects 19
20. Fluentd Ingestion in Elasticsearch 20 Forwarder instances & Elasticsearch cluster managed by an internal team
21. Fluentd What we have to do ▼ Embed the agent inside images (AMI) ▼ Configure it ▼ Make it take logs from: ▽ Application ▽ Nginx access logs (went out of ELB ones) ▼ Send it to the fluentd forwarders 21
22. Fluentd Logging destination - 12 factor “A twelve-factor app never concerns itself with routing or storage of its output stream. It should not attempt to write to or manage logfiles. Instead, each running process writes its event stream, unbuffered, to stdout.” 22 https://12factor.net/logs
23. Fluentd Parsing logs ▼ Complex log formats are a pain to parse: ▽ Multiline tracebacks ▽ Varying formats ▽ Strong coupling between the log format of the app and the parsed format ▼ Solution: ▽ Make nginx log in JSON ▽ Make the application log in JSON too! 23
24. Fluentd JSON logging - Nginx log_format json '{ ' {% for elt in nginx_log_elements %} '"{{ elt }}": "${{ elt }}"{{ ',' if not loop.last else '' }} ' {% endfor %} '}'; access_log syslog:server=unix:/dev/log,nohostname json; Result : log_format json '{ ' "http_referer": "$http_referer", "request_uri": "$request_uri", "status": "$status" '}'; access_log syslog:server=unix:/dev/log,nohostname json; 24
25. Fluentd JSON logging - Gunicorn (app) [loggers] keys=root, gunicorn.error [handlers] keys=console [formatters] keys=json 25 [logger_root] level=INFO handlers=console [logger_gunicorn.error] level=ERROR handlers=console propagate=0 qualname=gunicorn.error [handler_console] class=StreamHandler formatter=json args=(sys.stdout, ) [formatter_json] class=jsonlogging.JSONFormatter
26. Fluentd Photobox logs lifecycle 26
27. Fluentd Lifecycle - High level 27
28. Fluentd Getting logs from journald 28 https://github.com/reevoo/fluent-plugin-systemd
29. Fluentd Getting logs from journald <source> @type systemd tag hellocncf path /run/log/journal filters [{ "_SYSTEMD_UNIT": "myapp.service" }] read_from_head true <storage> @type local persistent true path /run/td-agent/myapp.pos </storage> </source> 29
30. Fluentd Parsing JSON <match hellocncf> @type parser tag myapp.parsed key_name full_message reserve_data yes format json inject_key_prefix parsed_ </match> 30
31. Fluentd Log enrichment with AWS metadata 31 https://github.com/takus/fluent-plugin-ec2-metadata
32. Fluentd Log enrichment with AWS metadata <match foo.**> type ec2_metadata aws_key_id YOUR_AWS_KEY_ID aws_sec_key YOUR_AWS_SECRET/KEY metadata_refresh_seconds 300 # Optional, default 300 seconds output_tag ${instance_id}.${tag} <record> instance_id ${instance_id} instance_type ${instance_type} az ${availability_zone} private_ip ${private_ip} vpc_id ${vpc_id} ami_id ${image_id} account_id ${account_id} project ${tagset_project} </record> </match> 32
33. Fluentd Lifecycle - Details 33
34. Fluentd Label - preconfigured forward Labels: like namespaces for Fluentd rules Relabel-ing: <match {{ app_name }}.app.**> @type relabel @label @PRECONFIGURED_FORWARD </match> 34 <label @PRECONFIGURED_FORWARD> <match **> @type forward <server> name tdagent-forwarder host my.tdagent.forwarder.example.org port 5150 </server> dns_round_robin true </match> </label>
35. Fluentd Conclusion 35
36. Fluentd Final result 36
37. Fluentd Why Fluentd is cool ▼ Many plugins for everything you can think of ▼ Really simple configurations ▼ Tags flow is straightforward 37
38. Fluentd Could be better ▼ Ruby :( ▼ Could be a pain for complex logging format (no JSON, multiline ...) ▼ Documentation 38
39. Fluentd Events 39
40. contact@dataxday.fr
41. A TECHNICAL CONFERENCE FOR DEVELOPERS AND TECH LEADS ABOUT DATA Data science: the next chapter Streaming: data in motion Big data architecture essentials dataxday.fr contact@dataxday.fr Pan Piper: 2-4 Impasse Lamier, 75011 Paris 17 May, 2018 SAVE THE DATE !
42. TROISIÈME ÉDITION 26 JUIN 2018 paris-container-day.fr
43. Le Paris Container Day est la conférence pionnière en France dédiée à l’ écosystème des conteneurs et de ses bonnes pratiques. Le thème de cette année: “Vivre avec l'Orchestration” Save the date! 26 juin 2018 au Cap Event Center, 3 Quai de Grenelle, 75015 Paris paris-container-day.fr / contact@paris-container-day.fr
44. Fluentd THE END Questions? 44