solutions can be considered • Warming up topic data if it was small enough • Setting smaller log segment size to prevent inode lock contention in xfs during reading topic data • https://speakerdeck.com/line_developers/investigating-kafka-performance-issue- caused-by-lock-contention-in-xfs • For proceeding them, we have to know the topic names • If disk read metrics had fi le name, we can use that
each fi les • Expose following informations as the prometheus metrics • Read bytes as the value • File name as the label • No performance impact against Kafka broker
Packet Filter) • The feature is provided by Linux kernel • It makes able to hook kernel events without modifying kernel code • bcc (BPF compiler collection) • Toolkit to compile and run eBPF program with Python/Lua
bi_sector; /* device address in 512 byte sectors */ struct bio *bi_next; /* request queue link */ struct block_device *bi_bdev; unsigned long bi_flags; /* status, command, etc */ unsigned long bi_rw; /* bottom bits READ/WRITE, * top bits priority */ unsigned short bi_vcnt; /* how many bio_vec's */ unsigned short bi_idx; /* current index into bvl_vec */ /* Number of segments in this BIO after * physical address coalescing is performed. */ unsigned int bi_phys_segments; unsigned int bi_size; /* residual I/O count */ ... }
be used as read bytes • But read fi le can’t be extracted directly from this argument • Need to get read fi le from somewhere • Another kernel function in upper layer?
available now, then let’s just expose the metrics def record_event(cpu, data, size): event = b["events"].event(data) # Accumulate received data from eBPF program and expose as prometheus metrics b = BPF(text=bpf_text) b["events"].open_perf_buffer(record_event) while True: try: b.perf_buffer_poll() except KeyboardInterrupt: exit()
processing • per- fi le disk read stats help to identify the topic caused disk read • eBPF provides the way to observe system layer • And it’s not so hard