Research Paper Introduction #10 “Tales of the Tail:
 Hardware, OS, and Application-level Sources of Tail Latency” @cafenero_777 2020/05/12

• ॕʂ10ճ໨ʂʢࢲͷ෼Ͱ׵ࢉʣ

$ which • Tales of the Tail: Hardware, OS, and Application-level Sources of Tail Latency • Jialin Li, Naveen Kr. Sharma, Dan R. K. Ports, and Steven D. Gribble • University of Washington • SOCC ’14 • ACM Symposium on Cloud Computing • https://sites.google.com/site/2014socc/home/program

Agenda • ֓ཁͱಡ΋͏ͱͨ͠ཧ༝ • Introduction • Queuing Models and Predicted Latency • Measurement Method • Sources of Tail Latency • Related Work • Discussion • Conclusion

֓ཁͱಡ΋͏ͱͨ͠ཧ༝ • ֓ཁ • ϚϧνίΞ্ͷHW/OS/AppϨΠϠʔ͔ΒlatencyΛௐࠪ • ϞσϧԽͯ͠RPC/Memcached/NginxͰଌఆ͠ɺݪҼͱτϨʔυΦϑΛௐࠪ • ಡ΋͏ͱͨ͠ཧ༝ • Tail latencyͷݟํΛ஌Γ͔͔ͨͬͨΒɻ • େن໛෼ࢄγεςϜ͸Tail latency͕౰ͨΓલͷੈքʢΒ͍͠ʣͷͰɻ • Podcastܦ༝ • https://misreading.chat/2019/03/27/episode-54-tales-of-the-tail/

Introduction • ωοτϫʔΫӽ͠ʹ΍ΓऔΓ͢Δ෼ࢄγεςϜ • େن໛؀ڥͩͱதԝ஋͕ܻҧ͍ʹେ͖͘ͳΔʢ=99%ileͰ΋֬཰తʹे෼େ͖͍ʣ • ਺ઍ୆ͷmemcached@facebook, ̍ສ୆ͷindexαʔό@MS Bing • null-RPC, Memcached, Nginx (web-server)Ͱݕূ • ཧ࿦ϞσϧΑΓѱ͍݁Ռʹͳͬͨ • ݪҼΛௐ΂ͯtail-latencyΛվળ • ྫɿMemcached 99.9%ile latency: 14ms -> 32us • ྫɿthroughputͱlatencyͷτϨʔυΦϑ

Queuing Models and Predicted Latency (1/3) • ϕʔεϥΠϯʢཧ૝ͷ஗Ԇ෼෍ʣ͸Կ͔ʁ • ϞσϧԽ • γϯάϧΩϡʔ͕cݸͷworker (core, thread, process, etc)ͰFIFOͤ͞Δ • A/S/c queue (Kendallදه) • Arrival distributionʢ౸ண෼෍ʣ, Service time distributionʢαʔϏε࣌ؒ෼ ෍ʣ, ಠཱͨ͠cݸͷϫʔΧʔ • ॲཧʹݻఆ͔͔࣌ؒΔωοτϫʔΫӽ͠ͷαʔϏεΛ૝ఆ

Queuing Models and Predicted Latency (2/3) • Arrival distributions • ྫɿϦΫΤετॲཧ͕50usͰ׬ྃ͢ΔFIFOαʔϏε • 50usҎ಺ʹϥϯμϜϦΫΤετ͕དྷΔͱ”଴ͪ”͕ൃੜ • -> tail-latencyൃੜ • Utilization • ϫʔΧʔ਺1ݻఆͷ··ɺฏۉϦΫΤετ਺Λ૿΍͢=utilΛ্͛Δ • ಉ࣌ʹϦΫΤετ͕དྷΔ֬཰͕૿͑Δ • -> tail-latencyൃੜ • ͔͠΋util 50%->95%Ͱ99%ile latency͕10ഒ૿͑Δʂ • ϚΠΫϩόʔετ໰୊ • ҰճͰ΋ϦΫΤετ͕”͔ͿΔ”ͱɺ஗ԆΛҾ͖ى͜͢ΩϡʔΛ࡞ͬͯ͠·͏ ܭࢉ஋ ܭࢉ஋

Queuing Models and Predicted Latency (3/3) • Parallel servers feeding from one queue • ϫʔΧʔc૿΍ͤ͹1/cͰlatencyݮΔ • ڞ༗ΩϡʔΛ࢖͍ͬͯΔ৔߹ (ϑΥʔΫฒͼ) • ݸผΩϡʔͩͱlatency͸มΘΒͣɺthroughput͕૿͑Δ • Queuing discipline • Random worker: ݸผFIFOΩϡʔΛ࣋ͬͯΔ֤ϫʔΧʔ΁ׂΓ౰ͯ • Random request: ڞ༗Ωϡʔ಺ͷϦΫΤετΛϥϯμϜʹબ୒ׂ͠Γ౰ͯʢ౸ண࣌ؒؔ܎ͳ͠ʣ • Ωϡʔ͔ΒҾ͖ग़͢ํࣜʹΑͬͯlatency͕มΘΔ • medianͱ99%ileͰlatencyٯస͢Δ৔߹͋Γ • FIFO V.S. LIFO (stack) • FIFO V.S. Random request ܭࢉ஋ ܭࢉ஋

Measurement Method • Null RPC server • TCPͰϦΫΤετ128byteΛड͚ͯ128byteϨεϙϯεฦ͢ • ΞΫηϓτεϨου->ϫʔΧʔੜ੒-> read/write system call • OSґଘɿTCP, εϨουεέδϡʔϥ • Memcached • O(1)ͳhash-tableΛ࣋ͭin-memory KVSΞϓϦέʔγϣϯɻϫʔΧʔ਺͸ίΞ਺ʹൺྫ • UDPϞʔυɿ֤ϫʔΧʔεϨου͕FIFO • TCPϞʔυɿTCPίωΫγϣϯຖʹϫʔΧʔ͕ܾ·͍ͬͯΔʢׂ౰ॲཧ͸1-2usఔ౓ʣ • Nginx • ඇಉظI/O system callΛଟ༻ • ϫʔΧʔຖʢίΞຖʣʹΫϥΠΞϯτΛׂΓ౰ͯ • 85byte http request -> 849byte http response, ੩తϑΝΠϧΛฦͨ͢ΊɺόοϑΝΩϟογϡʹ౰ͨΔʢετϨʔδӨڹ͸ແࢹͰ͖Δʣ • epoll systemcallΛ࢖͍ͬͯΔ=४උ͕Ͱ͖ͨॱʹϑΝΠϧσΟεΫϦϓλΛฦ͢ -> FIFO • ֤ΞϓϦͰCPU100%࢖༻཰ͳঢ়ଶʹͯ͠ɺεϧʔϓοτΛଌఆ͠ɺϦΫΤετॲཧ࣌ؒΛݟੵ΋Δ

Sources of Tail Latency (Background Processes) • 1CPU, 1core, HT disabled • εέδϡʔϥ͕linuxσʔϞϯʹ΋ׂ࣌ؒ౰->ϦΫΤετ͕ͨ·Δ->tail-latency૿Ճʂ • nice஋Ͱεέδϡʔϥͷ༏ઌ౓ʢׂ౰࣌ؒʣΛௐ੔ɻׂΓ౰ͯΒΕͳ͍ͱ଴ͪɻ • ϦΞϧλΠϜεέδϡʔϥɿϦΞϧλΠϜϓϩηεͱͯ͠ࢦఆ͢Δͱ”ׂ࣌ؒΓࠐΈ”͕Ͱ͖Δ • ઐ༻ίΞɿεέδϡʔϥ଴͕ͪͳ͍ͷͰ౰વ଎͍ɻίϯςΩετεΠον΋ແ͠

Sources of Tail Latency (Non-FIFO Scheduling) • CFS (Completely Fair Scheduler) -> ॱংΑΓެฏੑॏࢹɺඇFIFO • ϚϧνεϨουΞϓϦ: ͲͷεϨουʹ࣌ؒΛׂΓ౰ͯΔ͔͸OS࣍ୈ • ૣ͘ऴΘ͔ͬͨɺͰ͸ͳ͍ • ϦΞϧλΠϜεέδϡʔϥʹ͢ΔͱɺFIFO͔ͭόοΫάϥ΢ϯυׯবݮ ଌఆ஋ ܭࢉ஋

Sources of Tail Latency (Multicore) • ಉҰNUMA্Ͱ1~4core࢖͏ • Null RPC server͸վળ • γϯάϧΩϡʔ • ଞ2ͭ͸1coreͱมΘΒͣ • ϦΫΤετ͕TCPίωΫγϣϯ࢖͍·Θ͠ • TCP͕ಛఆϫʔΧʔʹׂΓ౰ͯͷͨΊɺϫʔΧʔ͕ภΔ • Memcached • UDPͰγϯάϧΩϡʔʹͳΔ->վળ • Nginx • TCP (http)Λ్தͰcloseͯ͠ɺ࠶౓ͭͳ͗௚͠ɺͰվળ • workload࣍ୈɻɻ

Sources of Tail Latency (Interrupt Processing) • packetड৴ͰΧʔωϧׂࠐൃੜ -> irqbalance͕શcoreʹ͜ΕΛ෼ࢄ • ׂࠐൃੜ͸༧ظͰ͖ͳ͍ʢ=ॲཧ͕࣌ؒҰఆͰͳ͘ͳΔʣ • ڞ༗ΩϡʔͷFIFOͰ͸ͳ͍ • ઐ༻ίΞͳΒ͜ΕΛճආ • load͕௿͍ͱແବʢεϧʔϓοτ͕௿͍ʣ • େن໛ϚϧνίΞCPUͩͱઐ༻ίΞར༻ʁ

Sources of Tail Latency (NUMA Effects) • 8coreΛ2CPUʹ෼ࢄ • σϑΥϧτͰ͸ϝϞϦׂΓ౰͕ͯnode0͔Βɻ • memcachedεϨουͷ൒෼͸ϝϞϦΞΫηε͕NUMAΛ·͙ͨ • -> latency૿Ճ • null RPC/Nginx͸ϝϞϦ࢖༻ྔ͕গͳ͔ͬͨͷͰӨڹͳ͔ͬͨ • numactlͰcore/memory nodeΛࢦఆ • վળʂ

Sources of Tail Latency (Power Saving Optimizations) • CPU࢖༻཰10%Ͱଌఆ • CPU stateɿ C-state͔ΒcoreΛ”ى͜͢”͕͔͔࣌ؒΔ -> tail-latencyʹͳΔ • C3-state͔Βͷwakeup͸200usɺ͜ΕΛଌఆ • प೾਺ͷ௿Լɿ࢖ͬͯͳ͍ͱCPUΫϩοΫप೾਺ΛݮΒ͢ • Nginx͸CPUෛՙ͕ߴ͍ͨΊɺएׯվળ

Sources of Tail Latency: Summary • nice஋͚ͩͰ͸ෆे෼ɻϦΞϧλΠϜεέδϡʔϥ͸༗ޮ • ϚϧνεϨουΞϓϦέʔγϣϯ͸FIFOεέδϡʔϥͳΒ༗ޮ • ϚϧνίΞ͸༗ޮ͕ͩɺҰൠతʹ͸ʢTCPͳͲಛఆίωΫγϣϯΛಛఆ ϫʔΧʹׂΓ౰ͯΔΞʔΩςΫνϟͩͱʣޮՌ͕ऑ͍ • NUMA͸εϨουͱϝϞϦׂΓ౰ͯnodeΛ߹ΘͤΔ • ిྗͱtail latency͸τϨʔυΦϑ

Related Work • MapReduce/Spark • Ϩεϙϯε͕஗͍ͱผϗετʹ࠶ϦΫΤετൃߦ • શϨϓϦΧʹಉ࣌ʹ౤͛໭͖ͬͯͨ΋ͷΛ࠾༻ʢεϧʔϓοτͷແବݣ͍ʣ • ෆ׬શͳ݁ՌΛڐ༰͢Δ • Ϛϧνςφϯτ؀ڥ • latency sensitive VMͱCPU sensitive VMͰϗετΛ෼͚Δ • DCNWͷεΠονͷΩϡʔᷓΕ • DCTCPతͳΞϓϩʔν • ిྗ໰୊ • LBͰ௨৴دͤΔɻ࢖ͬͯͳ͍αʔό͸௿ফඅిྗঢ়ଶ

Discussion • Ϧιʔε֬อͷ࢓ํ • ࣌ؒతʢCFSʣ V.S. ۭؒతʢCPUίΞઐ༗ʣ • εϨου V.S. Πϕϯτ • εϨου+FIFO • Πϕϯτ+ϦΫΤετͷϫʔΧʔׂΓ౰ͯͷ࠷దԽ

Conclusion • Tail latencyͷݪҼΛϚϧνίΞHW, OS, ΞϓϦέʔγϣϯϨϕϧͰௐࠪ • ཧ࿦ͱൺֱ • όοΫάϥ΢ϯυϓϩηεͷׯব • ΩϡʔΠϯάͷํ๏ͱεέδϡʔϥ • Χʔωϧׂࠐ΍NUMAɺCPUলిྗػೳ • ࠷దԽ͢Δͱ99.9%ileΛେ෯ʹ࡟ݮՄೳ

EoP

༧උεϥΠυ

Queuing Models and Predicted Latency (1/3) • ϕʔεϥΠϯʢཧ૝ͷ஗Ԇ෼෍ʣ͸Կ͔ʁ • શͯͷϦΫΤετʹಉ͡Ԡ౴࣌ؒͰॲཧ͢Δ -> ࣮ࡍ͸͋Γ͑ͳ͍ • ϦΫΤετ͕དྷΔλΠϛϯά͕όϥόϥʢ=ಉ࣌ʹདྷΔͱ͖΋͋Γʣ-> ϚΠΫϩόʔετ͸஗Ԇൃੜ • ϦΫΤετॲཧ͕࣌ؒಉ͡Ͱ΋શମͷlatency͕ҧ͏͜ͱ͕͋Δ • ߴ͍loadͷͱ͖͸latency΋ߴ͍ʁ • ϚϧνίΞԽͰlatencyվળʁ • ΩϡʔΠϯά͸FIFO͕࠷଎ʁ • ϞσϧԽ • γϯάϧΩϡʔ͕cݸͷworker (core, thread, process, etc)ͰFIFOͤ͞Δ • A/S/c queue (Kendallදه) • Arrival distributionʢ౸ண෼෍ʣ, Service time distributionʢαʔϏε࣌ؒ෼෍ʣ, ಠཱͨ͠cݸͷϫʔΧʔ • ॲཧʹݻఆ͔͔࣌ؒΔωοτϫʔΫӽ͠ͷαʔϏεΛ૝ఆ

Measurement Method (Timestamping) • NICͰड͚ͯɺNIC͔Βग़͍ͯ͘·Ͱͷ࣌ࠁT1 ~ T6 • ΧʔωϧɺNWυϥΠόɺL7ϓϩτίϧΛमਖ਼ͯ͠ϦΫΤετύέοτʹ30byte௥Ճ • NTP disabled • T1: NWυϥΠό͕packetॲཧՄೳͱ௨஌ͨ࣌͠ • T2: TCP/UDPॲཧޙɺΞϓϦॲཧલ • T3: ΞϓϦ͕ίΞʹεέδϡʔϦϯά͞Εͨޙ • T4: ΞϓϦ͕read system callൃߦޙ=Ϣʔβϥϯυʹσʔλ͕ίϐʔ͞Εͨޙ • T5: ΞϓϦ͕write system callൃߦޙ • T6: packetΛૹ৴͢Δ࣌