データルーター？Vector/Getting Started with Vector

Transcript

1. Getting Started with Vector Cloud native meetup tokyo #9 This

   document includes the work that is distributed in the Apache License 2.0

2. profile: name: Wataru Matsui org: [ Z Lab, 3bi.tech ]

   twitter: @watawuwu

3. • What’s Vector? • Usage • VS ... • Roadmap

   • Conclusions Agenda

4. What’s Vector？ https://vector.dev

5. Logs, Metrics & Events Router Is like Fluentd?

6. Developed by Timber.io https://timber.io

7. Feature • Log, Metrics, or Events • Agent Or Service

   • Fast • Correct • Clear Guarantee • Vendor Neutral • Easy To Deploy • Hot Reload

8. • Fluentd • Fluent Bit • Filebeat • Logstash Similar

   tool

9. 　Summary ©timber.io

10. ©timber.io

11. Topologies: Distributed ©timber.io

12. Topologies: Centralized ©timber.io

13. Topologies: Stream-Based ©timber.io

14. How to use Vector

15. Source types • file • statsd • syslog • tcp

    • vector • stdin(debug)

16. [sources.my_file_source_id]
 
 # REQUIRED - General 
 type = "file"

    # must be: "file"
 include = ["/var/log/nginx/*.log"]
 exclude = [""]
 Source config

17. [sources.my_tcp_source_id]
 
 # REQUIRED - General 
 type = "tcp"

    # must be: "tcp"
 address = ["0.0.0.0:9000"]
 Source config

18. Sink types • aws ◦ cloudwatch_logs ◦ kinesis_streams ◦ s3

    • elasticsearch • http • kafka • prometheus • splunk_hec • tcp • vector • console • blackhole(/dev/null)

19. [sinks.my_tcp_sink_id]
 
 # REQUIRED - General 
 type = "tcp"

    # must be: "tcp"
 input = ["my_tcp_source_id"]
 address = ["92.12.333.224:5000"]
 
 # OPTIONAL - Requests 
 encoding = "json" # default, enum: "json", "text" 
 
 Sinks config

20. [sinks.my_s3_sink_id]
 
 # REQUIRED - General 
 type = "s3"

    # must be: "s3"
 input = ["my_file_source_id"]
 bucket = "my-bucket"
 region = "ap-northeast-1"
 encoding = "ndjson" # enum: "ndjson", "text" 
 
 # OPTIONAL - Requests
 key_prefix = "date=%F/" # default Sinks config

21. [sinks.my_prometheus_sink_id]
 
 # REQUIRED - General 
 type = "prometheus"

    # must be: "prometheus" 
 input = ["my_log2metrics_source_id"]
 address = "0.0.0.0:9598" Sinks config

22. Transform types • Fileld ◦ add_fields ◦ remove_filed ◦ filed_filter

    • Paser ◦ grok_parser ◦ json_parser ◦ regex_parser ◦ tokenizer • log_to_metric • sampler • lua • vector • console • blackhole(/dev/null)

23. [transforms.my_regex_trans_id]
 
 # REQUIRED - General
 type = "regex_parser" #

    must be: "regex_parser" 
 inputs = ["my_file_source_id"]
 regex = "^(?P<host>[\\w\\.]+) - (?P<user>[\\w]+) (?P<bytes_in>[\\d]+) \\[(?P<timestamp>.*)\\] \"(? P<method>[\\w]+) (?P<path>.*)\" (?P<status>[\\d]+) (?P<bytes_out>[\\d]+)$"
 
 # OPTIONAL - Types
 [transforms.my_regex_trans_id.types]
 status = "int"
 method = "string"
 bytes_in = "int"
 bytes_out = "int" Transform config

24. [transforms.my_prometheus_trans_id]
 
 # REQUIRED - General
 type = "log_to_metric" #

    must be: "log_to_metric"
 inputs = ["my_file_source_id"]
 
 # OPTIONAL - Types
 [[transforms.my_regex_trans_id.metrics]]
 type = "counter" # enum: "counter", "gauge"
 field = "duration" 
 increment_by_value = false
 name = "duration_total"
 labels = {host = "${HOSTNAME}", region = "us-east-1"} 
 
 Transform config

25. [sources.logs] 
 type = 'file'
 include = ['/var/log/*.log']
 
 [transforms.tokenizer]

    
 inputs = ['logs']
 type = 'tokenizer'
 field_names = ["timestamp", "level", "message"]
 
 [transforms.sampler] 
 inputs = ['tokenizer']
 type = 'sampler'
 hash_field = 'request_id'
 rate = 10
 [sinks.search] 
 inputs = ['sampler']
 type = 'elasticsearch'
 host = '123.123.123.123:5000'
 
 [sinks.backup] 
 inputs = ['tokenizer']
 type = 's3'
 region = 'ap-northeast-1'
 bucket = 'log-backup'
 key_prefix = 'date=%F'
 Vector config

26. VS

27. 
 Vector
 FluentBit
 FluentD
 File to TCP
 76.7MiB/s
 35MiB/s
 26.1MiB/s


    Regex Parsing
 13.2MiB/s
 20.5MiB/s
 2.6MiB/s
 TCP to HTTP
 26.7MiB/s
 19.6MiB/s
 <1MiB/s
 Performance report by Timber.io

28. 
 Vector
 FluentBit
 FluentD
 Memory
 188.1MiB
 370MiB
 890MiB
 CPU
 1.51

    1m avg
 0.56 1m avg
 0.57 1m avg
 Performance report by Timber.io

29. Don't trust the reports. Measure, Measure, Measure!

30. Measure using GKE • Kubernetes: v1.13.7 • Node x4 ◦

    4 CPU ◦ 3.6 GB Memory ◦ 100 GB Storage(Standard) • Manifests ◦ https://github.com/watawuwu/vector-test

31. Memory Usage Mem usage is low Why fluent-bit uses memory?

    Vector
 26 MiB/s
 Fluent Bit
 1.091 GiB/s
 Fluentd
 92 MiB/s


32. CPU Usage CPU usage is high Vector
 1.84 core
 Fluent

    Bit
 0.26 core
 Fluentd
 1.25 core


33. IO Throughput Vector Fluentd Fluentd Bit Throughput is low Error

    in the test method？ Vector
 9.39 MiB/s
 Fluent Bit
 8.26 MiB/s
 Fluentd
 13.64 MiB/s


34. Roadmap

35. Roadmap • v0.4 Schemas(current) • v0.5 Stream Consumers • v0.6

    Columnar Writing • v0.7 CLI • v0.8 Wire Level Tailing • v1.0 Stable => 2019/12 Release!!

36. Conclusions

37. ADAPT
 TRIAL
 ASSESS
 HOLD
 watawuwu’s TECH RADAR

38. Thanks! Kubernetes, Cloud Native zlab.co.jp