Lock in $30 Savings on PRO—Offer Ends Soon! ⏳

データルーター?Vector/Getting Started with Vector

watawuwu
August 07, 2019

データルーター?Vector/Getting Started with Vector

watawuwu

August 07, 2019
Tweet

More Decks by watawuwu

Other Decks in Technology

Transcript

  1. Getting Started with Vector Cloud native meetup tokyo #9 This

    document includes the work that is distributed in the Apache License 2.0
  2. Feature • Log, Metrics, or Events • Agent Or Service

    • Fast • Correct • Clear Guarantee • Vendor Neutral • Easy To Deploy • Hot Reload
  3. [sources.my_file_source_id]
 
 # REQUIRED - General 
 type = "file"

    # must be: "file"
 include = ["/var/log/nginx/*.log"]
 exclude = [""]
 Source config
  4. [sources.my_tcp_source_id]
 
 # REQUIRED - General 
 type = "tcp"

    # must be: "tcp"
 address = ["0.0.0.0:9000"]
 Source config
  5. Sink types • aws ◦ cloudwatch_logs ◦ kinesis_streams ◦ s3

    • elasticsearch • http • kafka • prometheus • splunk_hec • tcp • vector • console • blackhole(/dev/null)
  6. [sinks.my_tcp_sink_id]
 
 # REQUIRED - General 
 type = "tcp"

    # must be: "tcp"
 input = ["my_tcp_source_id"]
 address = ["92.12.333.224:5000"]
 
 # OPTIONAL - Requests 
 encoding = "json" # default, enum: "json", "text" 
 
 Sinks config
  7. [sinks.my_s3_sink_id]
 
 # REQUIRED - General 
 type = "s3"

    # must be: "s3"
 input = ["my_file_source_id"]
 bucket = "my-bucket"
 region = "ap-northeast-1"
 encoding = "ndjson" # enum: "ndjson", "text" 
 
 # OPTIONAL - Requests
 key_prefix = "date=%F/" # default Sinks config
  8. [sinks.my_prometheus_sink_id]
 
 # REQUIRED - General 
 type = "prometheus"

    # must be: "prometheus" 
 input = ["my_log2metrics_source_id"]
 address = "0.0.0.0:9598" Sinks config
  9. Transform types • Fileld ◦ add_fields ◦ remove_filed ◦ filed_filter

    • Paser ◦ grok_parser ◦ json_parser ◦ regex_parser ◦ tokenizer • log_to_metric • sampler • lua • vector • console • blackhole(/dev/null)
  10. [transforms.my_regex_trans_id]
 
 # REQUIRED - General
 type = "regex_parser" #

    must be: "regex_parser" 
 inputs = ["my_file_source_id"]
 regex = "^(?P<host>[\\w\\.]+) - (?P<user>[\\w]+) (?P<bytes_in>[\\d]+) \\[(?P<timestamp>.*)\\] \"(? P<method>[\\w]+) (?P<path>.*)\" (?P<status>[\\d]+) (?P<bytes_out>[\\d]+)$"
 
 # OPTIONAL - Types
 [transforms.my_regex_trans_id.types]
 status = "int"
 method = "string"
 bytes_in = "int"
 bytes_out = "int" Transform config
  11. [transforms.my_prometheus_trans_id]
 
 # REQUIRED - General
 type = "log_to_metric" #

    must be: "log_to_metric"
 inputs = ["my_file_source_id"]
 
 # OPTIONAL - Types
 [[transforms.my_regex_trans_id.metrics]]
 type = "counter" # enum: "counter", "gauge"
 field = "duration" 
 increment_by_value = false
 name = "duration_total"
 labels = {host = "${HOSTNAME}", region = "us-east-1"} 
 
 Transform config
  12. [sources.logs] 
 type = 'file'
 include = ['/var/log/*.log']
 
 [transforms.tokenizer]

    
 inputs = ['logs']
 type = 'tokenizer'
 field_names = ["timestamp", "level", "message"]
 
 [transforms.sampler] 
 inputs = ['tokenizer']
 type = 'sampler'
 hash_field = 'request_id'
 rate = 10
 [sinks.search] 
 inputs = ['sampler']
 type = 'elasticsearch'
 host = '123.123.123.123:5000'
 
 [sinks.backup] 
 inputs = ['tokenizer']
 type = 's3'
 region = 'ap-northeast-1'
 bucket = 'log-backup'
 key_prefix = 'date=%F'
 Vector config
  13. VS

  14. 
 Vector
 FluentBit
 FluentD
 File to TCP
 76.7MiB/s
 35MiB/s
 26.1MiB/s


    Regex Parsing
 13.2MiB/s
 20.5MiB/s
 2.6MiB/s
 TCP to HTTP
 26.7MiB/s
 19.6MiB/s
 <1MiB/s
 Performance report by Timber.io
  15. 
 Vector
 FluentBit
 FluentD
 Memory
 188.1MiB
 370MiB
 890MiB
 CPU
 1.51

    1m avg
 0.56 1m avg
 0.57 1m avg
 Performance report by Timber.io
  16. Measure using GKE • Kubernetes: v1.13.7 • Node x4 ◦

    4 CPU ◦ 3.6 GB Memory ◦ 100 GB Storage(Standard) • Manifests ◦ https://github.com/watawuwu/vector-test
  17. Memory Usage Mem usage is low Why fluent-bit uses memory?

    Vector
 26 MiB/s
 Fluent Bit
 1.091 GiB/s
 Fluentd
 92 MiB/s

  18. CPU Usage CPU usage is high Vector
 1.84 core
 Fluent

    Bit
 0.26 core
 Fluentd
 1.25 core

  19. IO Throughput Vector Fluentd Fluentd Bit Throughput is low Error

    in the test method? Vector
 9.39 MiB/s
 Fluent Bit
 8.26 MiB/s
 Fluentd
 13.64 MiB/s

  20. Roadmap • v0.4 Schemas(current) • v0.5 Stream Consumers • v0.6

    Columnar Writing • v0.7 CLI • v0.8 Wire Level Tailing • v1.0 Stable => 2019/12 Release!!