3 state-of-the-art technologies in Linux and future of the containers #SECKUN

2021.02.21 「新しいセキュリティビジネスキャリア」シンポジウム

1. ཁૉٕज़ͷstate of the art͔Βߟ͑Δ ۙ౻Ӊஐ࿕ / GMO Pepabo, Inc. 2021.02.21 ʮ৽͍͠ηΩϡϦςΟϏδωεΩϟϦΞʯγϯϙδ΢Ϝ Linuxίϯςφͷະདྷ
2. GMOϖύϘגࣜձࣾ γχΞϓϦϯγύϧ ٕज़෦ٕज़ج൫νʔϜॴଐ ۙ౻Ӊஐ࿕ (@udzura)
3. ۙ౻Ӊஐ࿕ ུྺ • ࡾՏᅳͷਓɻچ٢ాൡߍ࣌शؗߴߍΛଔۀɺ౦ژେֶจֶ෦೔ຊޠ೔ຊจֶઐम՝ఔ ͷֶ࢜ଔʢ2007ʣɻ • Ϛείϛͷࣾ಺SEɺECαΠτ։ൃɺΦϯϥΠϯήʔϜ ։ൃͳͲΛܦͯ2013೥ΑΓݱ৬ɺಉ೥ʹ෱ԬҠॅɻ • RubyɺίϯςφɺΫϥ΢υωΠςΟϒٕज़ͳͲͷίϛϡχςΟͰ ׆ಈɻஶॻʹʮWebͰ࢖͑ΔmrubyγεςϜϓϩάϥϛϯάೖ໳ʯʢC&Rݚڀॴʣ • ޷͖ͳγεςϜίʔϧ͸ʢ࠷ۙ͸ʣ socketpair(2) ɻ
4. ࠓ೔ͷ͓࿩ •ίϯςφͷཁૉٕज़ʹ͍ͭͯɺࢲͷߨٛͰʢͬ͘͟Γʣཧղ͞Εͨํ ޲͚ͷ಺༰Ͱ͢ɻཁૉٕज़ͷղઆ͸ࢀߟࢿྉΛͲ͏ͧɻ •ࢀߟ1: https://container-security.dev/ •ࢀߟ2: ʰίϯςφܕԾ૝Խ֓࿦ʱʢ઒ޱ, ΧοτγεςϜʣ •௚ۙͷΧʔωϧʹؚ·ΕΔ৽ٕज़ͷ͏ͪɺίϯςφʹؔ܎͢Δ΋ͷΛ ঺հ͠·͢ɻ •͕࣌ؒ͋Ε͹ɺͦͷ্ͰίϯςφͷະདྷΛߟ͑·͢ɻ
5. cgroup v2
6. cgroup ͷ͓͞Β͍ •Linux Kernelͷجຊٕज़ͷҰͭɻ •ϓϩηεΛάϧʔϐϯά͠ɺͦͷάϧʔϓ୯ҐͰϦιʔεར༻ͷ੍ݶ Λ͔͚Δٕज़ɻCPUɺϝϞϦɺIOɺϓϩηε਺... IUUQTHJIZPKQBENJOTFSJBMMJOVY@DPOUBJOFST
   
7. cgroup v1ͷྫ •cgroupfs ͱ͍͏ϑΝΠϧγεςϜʹmkdir(2) read(2) write(2)ͳͲΛ࣮ ߦ͠ɺૢ࡞Λߦ͏
8. cgroup v2 •v1 ͷ͍͔ͭ͘ͷܽ఺ - ओʹ੍ޚର৅ʢίϯτϩʔϥʣ͝ͱʹσΟϨΫ τϦΛ෼͚ͳ͚Ε͹͍͚ͳ͍࢓༷ - Λࠀ෰͢΂͘։ൃ͞Εͨ •େ͖ͳҧ͍ͱͯ͠ɺ v1 Ͱ͸ίϯτϩʔϥผʹσΟϨΫτϦ ΛϚ΢ϯ τɺݸผʹάϧʔϓʹॴଐͰ͖ͨͷʹର͠ɺv2Ͱ͸શίϯτϩʔϥΛ ·ͱΊͨҰͭͷσΟϨΫτϦͷΈΛϚ΢ϯτ͠ɺ·ͱΊͯάϧʔϓΛ ࡞੒͢ΔڍಈʹͳΔɻ •ίϯςφͱͯ͠͸ͪ͜Βͷํ͕౎߹͕͍͍ɻ
9. Unified hierarchy /sys/fs/cgroup /sys/fs/cgroup /group-a /group-b /cpu.* /memory.* /io.* ... /cpu.* /memory.* /io.* ... /cpu /memory /blkio /group-a /group-b /group-a /group-c
10. ίϯςφϥϯλΠϜͰͷcgroupͷར༻ •ʢOCIܥͷʣϥϯλΠϜͰ͸ҎԼͷ2ͭͷઃఆ߲໨͕͋Δ •Cgroup Driver: ίϯςφʹׂΓ౰ͯΔcgroupΛͲ͏ίϯτϩʔϧ͢Δ͔ •cgroupfs: cgroupfs΁ͷ௚઀ͷϑΝΠϧૢ࡞ •systemd: systemdʹΑΔ؅ཧ •Cgroup Version: Ϧιʔε੍ݶʹ v1/v2 ͲͪΒΛར༻͢Δ͔ •/sys/fs/cgroup ʹͲͷϑΝΠϧγεςϜ͕Ϛ΢ϯτ͞ΕͯΔ͔Ͱ൑ఆ
11. v2 ͷ৽ػೳ •Unified Hierarchy •PSI(Pressure Stall Information) •eBPFͰcgoup IDͷऔಘ͕Մೳʹ •nsdelegate (ඇಛݖίϯςφʹ͸ॏཁ) •clone3(2) Ͱಛఆͷcgroup಺෦ʹ௚઀ϓϩηε࡞੒͕Մೳʹ •ͳͲͳͲ...
12. e.g. PSI(Pressure Stall Information) •γεςϜશମɺ·ͨ͸cgroup୯ҐͰར༻Ͱ͖Δෛՙͷࢦඪ •CPU, ϝϞϦ, IO Ͱ stall ͨ͠୯Ґ࣌ؒͰͷׂ߹ ΛܭଌͰ͖Δ •e.g. 1෼ؒͰ45ඵؒɺάϧʔϓͷ ͋Δϓϩηε͕CPUىҼͰ ஗Ԇͨ͠৔߹ɺcpu some: 75.00
13. e.g. eBPFͰͷτϥοΩϯά৘ใ •bpf_get_current_cgroup_id(void) ϔϧύʔ •eBPFͷΠϕϯτ͕ى͖ͨλεΫ͕Ͳͷcgroup(v2)ʹॴଐ͍ͯ͠Δ͔ɺ ͦͷIDΛฦ͢ɻ
14. How cgroup-v2 and PSI Impacts Cloud Native? Uchio Kondo / GMO Pepabo, Inc. 2019.07.23 CloudNative Days Tokyo 2019 Image from pixabay: https://pixabay.com/images/id-3193865/
15. eBPF per containers
16. eBPFٕज़ͱ͸ •ϢʔβۭؒͰ࡞ͬͨϓϩάϥϜΛΧʔωϧͰಈ͔ٕ͢ज़ͷͻͱͭ •2012೥ʹseccomp΁ͷಋೖɺ2013೥ʹLinuxͷSDNͰͷԠ༻͕࣮૷͞ ΕɺͦΕҎ߱੒ख़͢Δ •ϑΟϧλϦϯά͕ಘҙʢtcpdump, seccomp, bpftraceʣ •Χʔωϧͷ৘ใʹΞΫηεͰ͖Δ͕ɺةݥͳίʔυ͸ಈ͔ͳ͍ͳͲ ҆શੑ͕͋Δఔ౓୲อ͞Ε͍ͯΔ
17. eBPFͷԠ༻ྫ
18. ίϯςφͷeBPFτϨʔεઓུ •ઓུ͕͍͔ͭ͋͘Δ •Linux Namespace·ͨ͸cgroup (v2)ͷ৘ใ͕ར༻Ͱ͖Δ
19. ྫ1: task_struct ͷ৘ใΛḷΔ •task_struct→nsproxy ͔Β namespaceͷ৘ใΛ औಘͯ͠ϑΟϧλ͢Δ ʢcxrayʣ IUUQTHJUIVCDPNNSUDDYSBZCMPCNBTUFSQLHUSBDFSPQFOPQFOHP--
    
20. ྫ2: NS಺/ϗετͰͷPIDΛൺֱ •BPFϓϩάϥϜͰऔಘͰ͖ͨ tidͱɺϗετͰͷtidΛ ൺֱ͠ɺҰக͠ͳ͚Ε͹ ίϯςφͱ൑ఆ͢Δ ʢTraceeʣ • task_structґଘ IUUQTHJUIVCDPNBRVBTFDVSJUZUSBDFFCMPCNBJOUSBDFFUSBDFFCQGD-ɹ
21. ྫ3: cgroup helperΛར༻ IUUQTHJUIVCDPNVE[VSBDPQFODMPTFCMPCNBTUFSTSDCQGDPQFODMPTFCQGD
    
22. ࣮૷ྫ •copenclose(8) •ۙ౻ͷPoC (BPF+Rust) •ϑϥάͰtask_struct/ cgroup v2 ID Λ੾Γସ͑
23. bpf_get_current_cgroup_id(void) を添えて Uchio Kondo / Container Runtime Meetup #3 ランタイムとcgroupの
 xxxな関係 * Photo by Fukuoka City
24. seccomp
25. seccompͷ͓͞Β͍ •ϓϩάϥϜʹ͓͚ΔγεςϜίʔϧݺͼग़͠ΛϑΟϧλϦϯά͢Δ •γεςϜίʔϧͷҾ਺ͷ৚݅ʹΑͬͯࢦఆΛม͑ΒΕΔ •blacklist(denylist), whitelist(allowlist) ͳͲΛ࣮૷Ͱ͖Δ •ϑϥά͕ࡉ͔͘ଘࡏ͠ɺྫ͑͹γεςϜίʔϧͷauditϩάͷΈɺ೚ҙ ͷerrnoΛฦͤ͞ΔɺͳͲͷࢦఆ͕Ͱ͖Δ
26. seccompͷར༻(mruby)
27. User space notification •seccompʹΑΓγεςϜίʔϧݺͼग़͠Λݕ஌͠ɺͦͷڍಈΛϢʔβ ϥϯυͷϓϩάϥϜʹҕͶΔ͜ͱ͕Ͱ͖Δٕज़ •Linux 5.0 (2019/3) ͔Βͷಋೖ •൑அ͢Δ·ͰɺͦͷγεςϜίʔϧ͸ϒϩοΫ͢Δ •e.g. LXCͰͷσόΠεΞΫηεͷ੍ޚ IUUQTHJIZPKQBENJOTFSJBMMJOVY@DPOUBJOFSTɹ
28. User space notification IUUQTHJIZPKQBENJOTFSJBMMJOVY@DPOUBJOFSTɹ • LXCͰֶͿίϯςφೖ໳ ୈ47ճɹඇಛݖίϯςφͷՄೳੑΛ޿͛Δseccomp notifyػೳ ΑΓ
29. ࣮૷ྫʢmrubyར༻ʣ •ҎԼͷΑ͏ͳ acceptor.rbΛ ༻ҙ͢Δ
30. ࣮૷ྫʢmrubyར༻ʣ •ҎԼͷinvokerΛܦ༝ͯ͠ϓϩάϥϜΛ ىಈɺ listen(3) ΛݺͿ
31. listen(2) ͷىಈݕ஌ •acceptor.rb ଆͷίϯιʔϧͰڐՄ/ېࢭΛ੍ޚՄೳɻ •ېࢭͨ͠Βͦͷ··ىಈࣦഊͯ͠invokerϓϩηε͕མͪΔ •ڐՄͨ͠ΒԿ΋ͳ͔͔ͬͨͷΑ͏ʹɺىಈΛܧଓͯ͠Ϧοεϯɻ
32. listen(2) ͷىಈݕ஌ ېࢭ࣌ͷग़ྗ ڐՄ࣌ͷग़ྗ
33. Ԡ༻ʁ •ʮ೚ҙͷϥΠϒϥϦؔ਺ݺͼग़͠ʯͰϓϩηεΛఀࢭɺCRIU(*)ʹΑΓ ϓϩηεμϯϓΛ࡞੒͢Δ࣮ݧΛߦͬͨɻ •LD_PRELOAD + ϥούؔ਺ + ʮԿ΋͠ͳ͍ʯsyscall + seccomp IUUQTVE[VSBIBUFOBCMPHKQFOUSZ
    
    $IFDLQPJOU
    BOE
    3FTUPSF
    *O
    6TFSTQBDF
    ϓϩηεͷঢ়ଶΛอଘɺ͔ͦ͜Β࠶ੜ͢Δٕज़
    IUUQTDSJVPSH.BJO@1BHF
    
34. ߟ࡯
35. ৽ٕज़ʹΑΓͰ͖Δ͜ͱ͸૿͑Δ͕... •৽ٕज़ͷʮग़ݱʯͱʮීٴʯͷλΠϛϯά͸ζϨΔ •ͨͱ͑͹ cgroup v2ͷॳग़͸2013೥ɻ •2019 ~ 2020 ೥ʹϥϯλΠϜͰͷରԠ͕ਐΜͩଆ໘ •पลͷπʔϧ͕ग़ݱ͢Δͷ͸΋ͬͱઌͰ͋Ζ͏ •ग़ݱظʹ୯ମͰٕज़Λݕূ͠ɺʢηΩϡϦςΟؚΊʣͲ͏͍͏໰୊͕ ͋Δ͔ɺͲ͏͍͏Մೳੑ͕͋Δ͔ݕূ͢Δҙٛ͸େ͖͍
36. eBPF ͸Linuxͷجຊٕज़ʹͳΓͭͭ͋Δ •ద༻ൣғ͕ͲΜͲΜ޿·͍ͬͯΔ •τϨʔγϯάɺଳҬ੍ޚ΍ωοτϫʔΩϯάͷ΄͔ɺcgroup(v2) deviceͷ൑ఆɺLSM BPF programͳͲͳͲ... •ηΩϡϦςΟͷจ຺Ͱ͸τϨʔεɺ؂ࠪɺҟৗݕ஌ͱ͔ܽͤͳ͍ٕज़ ʹͳΔ͜ͱ͕૝૾͞ΕΔ •Ұͭͷprog typeʹ৮͓͚ͬͯͩ͘Ͱ΋ײ֮͸෼͔Γͦ͏