FORKര஄ ີண24࣌ @ୈ28ճγΣϧܳษڧձ େࡕαςϥΠτ (2017.04.22) MSR (@msr386)

໨࣍ ➤ ࣗݾ঺հ ➤ forkര஄ͱ͸ ➤ forkര஄ ࣮ԋ ➤ forkര஄ൃੜ࣌ͷΫϥογϡμϯϓΛऔΓɺղੳ ➤ OOM Killerൃಈ࣌ ➤ 24࣌ؒख़੒ ➤ ରࡦ ➤ ·ͱΊ

ࣗݾ঺հ ➤ ϋϯυϧωʔϜ: MSR (Twitter ID: @msr386) ➤ ͓ͻͱΓ༷Mastodon࢝Ί·ͨ͠
 https://mastodon.msr-pc.com/@msr386 ➤ C++ϓϩάϥϛϯάͱ(ฏ࿨ͳ)αʔόʔӡ༻͕ಘҙ ➤ Chromium Contributor … Ͱͨ͠ ➤ “Tabbed Shell”ΛମݧͰ͖ΔWebϒϥ΢βͷ࡞ऀ
 https://app.tungsten-start.net/

ຊLTͷ໨త

FORKര஄ͷةݥੑͷ
 ࠶֬ೝ

FORKര஄ͱ͸

:(){ :|:& };: ※Illustrated by ͍Β͢ͱ΍, composed by @nmrmsys

FORKര஄ͱ͸ ➤ ୹࣌ؒʹେྔͷࢠϓϩηεΛੜ੒͢Δ͜ͱʹΑΓɺϦιʔεΛރׇͤ͞Δ
 ةݥͳര஄ ➤ forkγεςϜίʔϧ͕ޠݯ (※ಠࣗݚڀ) ➤ ةݥγΣϧܳʹΑΔforkര஄
 
 ͕γΣϧܳք۾Ͱ༗໊ ➤ forkγεςϜίʔϧ͕ݺͼग़͠ՄೳͳΒ͹ɺݴޠ͸໰Θͳ͍
 
 ͳͲ $ :(){ :|:& };: $ perl -e 'while(1){fork();}'

ඃ֐ใࠂ ➤ ʮίϚϯυ౤ೖͨ͠ॠؒʹࢮΜͩᵎ(^o^ )ᵊࡾʯ
 (https://twitter.com/mutz0623/status/502072141869629441) ➤ ʮ#ةݥγΣϧܳ ͷ:(){: | :};:࣮ߦͯ͠ΈͨΒϋϯά͚ͨ͠Ͳશ͘ཧ༝෼͔ΒΜɻ୭͔ ڭ͑ͯʯ
 (https://twitter.com/blackenedgold/status/502079602068422658) ➤ ʮ#ةݥγΣϧܳ docker্Ͱ࣮ߦͨ͠Βϗετࣄ੦͖·ͨ͠ʯ
 (https://twitter.com/nnikirom/status/502093471109230593) ➤ ɾɾɾͳͲଟ਺

FORKര஄ͷٙ໰ ➤ forkര஄Λ࣮ߦ͢ΔͱͲ͏ͳΔͷ͔ʁ ➤ ͳͥforkര஄͸ةݥ͔ʁ ➤ γΣϧͷԠ౴͕ͳ͘ͳͬͨͱ͖ʹԿ͕ى͍ͬͯ͜Δ͔ʁ ͜ΕΒͷٙ໰Λղফ͢ΔͨΊ
 GPSLര஄Λരൃͤͯ͞ɺരൃதͷঢ়ଶΛௐ΂Δ͜ͱʹͨ͠

ඃ֐ऀͦͷ̍ɹ࣮ԋ؀ڥ ➤ Parallels DesktopʹͯԾ૝ϚγϯΛ࡞੒ ➤ OS: CentOS 7.3 (x86_64, GUIΠϯετʔϧ) ➤ CPUίΞ਺: 2 (࣮ػ͸2ίΞ4εϨου) ➤ ϝϞϦ: 4GB ➤ σΟεΫ: SSD

࣮ԋ

࣮ߦ࣌ͷ༷ࢠʢҰྫʣ ➤ ࣮ߦޙ͙͢ʹϝϞϦෆ଍͕ൃੜ͢Δ͕ɺ·ͩ͜ͷ࣌఺Ͱ͸Ԡ౴͕͋Δ৔߹͕͋Δ ➤ ͠͹Β͘͢ΔͱSSH͕ɺͦͷޙίϯιʔϧͷԠ౴΋ͳ͘ͳΔ ➤ ϝϞϦෆ଍Ҏ֎ʹϓϩηεID͕ރׇ͢Δ৔߹΋͋Δ ➤ Ctrl+C͸ແྗ ➤ γΣϧԠ౴͕͋Ε͹pkill, killallͰԠٸॲஔՄೳ(Ͱ͖Δ࣌ؒ͸΄΅ͳ͍) ➤ ࠷ऴతʹ͸෺ཧతʹిݯΛམͱ͔͢Ϧηοτ͢Δ͔͠ͳ͍ ➡ ୹࣌ؒͰϦιʔε͕ރׇ͠ɺఀࢭ͕ࠔ೉͔ͩΒةݥ

FORKര஄ͷٙ໰ ➤ forkര஄Λ࣮ߦ͢ΔͱͲ͏ͳΔͷ͔ʁ
 ˠϝϞϦෆ଍ʹͳΓɺγΣϧͷԠ౴͕ͳ͘ͳΔ ➤ ͳͥforkര஄͸ةݥ͔ʁ
 ˠ୹࣌ؒͰϦιʔε͕ރׇ͠ɺఀࢭ͕ࠔ೉ʹͳΔͨΊ ➤ γΣϧͷԠ౴͕ͳ͘ͳͬͨͱ͖ʹԿ͕ى͍ͬͯ͜Δ͔ʁ
 ˠ???

ΫϥογϡμϯϓΛऔΔ
 (ϝϞϦෆ଍ൃੜ࣌)

લ४උ 1. (VMware࢖༻࣌ͷΈ)ىಈ࣌ʹ.vmemϑΝΠϧΛੜ੒ͤ͞ͳ͍Α͏ʹ͢Δ
 .vmxʹ mainMem.useNamedFile = "FALSE" Λ௥Ճ 2. εϫοϓΛ੾Δ
 
 ˞forkര஄ʹΑΔϝϞϦෆ଍࣌ʹσΟεΫI/OͷաෛՙΛ๷͙ 3. ϓϩηε਺্ݶΛ૿΍͢ or ແ੍ݶʹ # swapoff -a # ulimit -u unlimited

લ४උ 5. Ϋϥογϡμϯϓऔಘػೳ kdumpΛ༗ޮʹ͢Δ
 
 
 
 ˞࠷ޙͷίϚϯυͷ࣮ߦ݁Ռ͕ "kdump is operational" ͱͳΔ͜ͱΛ֬ೝ 6. OOM Killerൃಈ࣌ʹΧʔωϧύχοΫΛҾ͖ى͜͢Α͏ʹڍಈΛมߋ͢Δ
 # systemctl enable kdump.service # systemctl restart kdump # systemctl status kdump # sysctl vm.panic_on_oom=1

લ४උ 7. ΫϥογϡμϯϓղੳʹඞཁͳίϯϙʔωϯτΛΠϯετʔϧ
 
 
 
 
 
 
 
 
 # uname -v 3.10.0-514.6.1.el7.x86_64 # yum install kexec-tools # wget http://debuginfo.centos.org/7/x86_64/kernel- debuginfo-3.10.0-514.6.1.el7.x86_64.rpm # wget http://debuginfo.centos.org/7/x86_64/kernel-debuginfo-common- x86_64-3.10.0-514.6.1.el7.x86_64.rpm # rpm -ivh kernel-debuginfo*.rpm ↓ ࣮ߦதͷΧʔωϧόʔδϣϯΛ߹ΘͤΔඞཁ͕͋Δ

࣮ԋ

Ϋϥογϡμϯϓղੳ ➤ crashίϚϯυΛ࢖༻͢Δ(ཁrootϢʔβʔ) ➤ crashίϚϯυର࿩தͷ୅දతͳίϚϯυ(※ύΠϓɺϦμΠϨΫτ࢖༻Մ) ➤ γεςϜঢ়ଶΛදࣔ(ىಈ௚ޙ΋ಉ༷ͷ಺༰͕දࣔ͞ΕΔ)
 ɹcrash> sys ➤ dmesg಺༰Λදࣔ
 ɹcrash> log ➤ ΧʔωϧϝϞϦঢ়ଶΛදࣔ
 ɹcrash> kmem -i ➤ ϓϩηεҰཡΛදࣔ
 ɹcrash> ps

Ϋϥογϡμϯϓղੳ # crash /usr/lib/debug/lib/modules/3.10.0-514.6.1.el7.x86_64/vmlinux/
 var/crash/127.0.0.1-2017-02-10-21:09:00/vmcore
 .. (ུ) ..
 KERNEL: /usr/lib/debug/lib/modules/3.10.0-514.6.1.el7.x86_64/vmlinux
 DUMPFILE: vmcore [PARTIAL DUMP]
 CPUS: 2
 DATE: Wed Feb 8 17:01:10 2017
 UPTIME: 00:10:07
 LOAD AVERAGE: 5225.57, 1249.51, 414.67
 TASKS: 25887
 NODENAME: localhost.localdomain
 RELEASE: 3.10.0-514.6.1.el7.x86_64
 VERSION: #1 SMP Wed Jan 18 13:06:36 UTC 2017
 MACHINE: x86_64 (3999 Mhz)
 MEMORY: 4 GB
 PANIC: "Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled"
 PID: 24241
 COMMAND: "bash"
 TASK: ffff8800231ebec0 [THREAD_INFO: ffff8800232c0000]
 CPU: 0
 STATE: TASK_RUNNING (PANIC)
 
 crash>

Ϋϥογϡμϯϓղੳ crash> log
 (ུ)
 [ 607.651832] [30662] 0 30662 28846 108 11 0 0 bash
 [ 607.651833] Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled
 (ུ)
 [ 607.652230] CPU: 0 PID: 24241 Comm: bash Not tainted 3.10.0-514.6.1.el7.x86_64 #1
 [ 607.652444] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
 [ 607.652904] ffffffff818db658 000000003e9e51e2 ffff8800232c3a08 ffffffff816862ac
 [ 607.653139] ffff8800232c3a88 ffffffff8167f6b3 ffffffff00000010 ffff8800232c3a98
 [ 607.653389] ffff8800232c3a38 000000003e9e51e2 000000000000006c ffffffff818df587
 [ 607.653650] Call Trace:
 [ 607.653905] [] dump_stack+0x19/0x1b
 [ 607.654173] [] panic+0xe3/0x1f2
 [ 607.654435] [] check_panic_on_oom+0x55/0x60
 [ 607.654683] [] out_of_memory+0x23b/0x4f0
 (ུ)
 crash>

Ϋϥογϡμϯϓղੳ crash> kmem -i
 PAGES TOTAL PERCENTAGE
 TOTAL MEM 966388 3.7 GB ----
 FREE 21550 84.2 MB 2% of TOTAL MEM
 USED 944838 3.6 GB 97% of TOTAL MEM
 SHARED 60308 235.6 MB 6% of TOTAL MEM
 BUFFERS 0 0 0% of TOTAL MEM
 CACHED 2209 8.6 MB 0% of TOTAL MEM
 SLAB 196732 768.5 MB 20% of TOTAL MEM
 
 TOTAL SWAP 0 0 ----
 SWAP USED 0 0 100% of TOTAL SWAP
 SWAP FREE 0 0 0% of TOTAL SWAP
 
 COMMIT LIMIT 483194 1.8 GB ----
 COMMITTED 2219010 8.5 GB 459% of TOTAL LIMIT
 crash>

Ϋϥογϡμϯϓղੳ crash> ps
 PID PPID CPU TASK ST %MEM VSZ RSS COMM
 0 0 0 ffffffff819c1460 RU 0.0 0 0 [swapper/0]
 0 0 1 ffff880137511f60 RU 0.0 0 0 [swapper/1]
 1 0 1 ffff880137400000 UN 0.1 193628 2772 systemd
 (ུ)
 crash> ps | grep bash | grep -v ZO | wc -l
 14212
 crash> ps | grep bash | grep ZO | wc -l
 11561
 crash> ps | wc -l
 25888
 crash>

ΫϥογϡμϯϓͰ൑໌͢Δ͜ͱ ➤ load average͕ͱͯͭ΋ͳ͘େ͖͍ ➤ bashϓϩηε͕େྔൃੜ͍ͯ͠Δ ➤ bashͷκϯϏϓϩηε͕൒਺

FORKര஄ 24࣌ؒख़੒

̎̐࣌ؒख़੒ͷલ४උ ➤ ͍ͭͰ΋ΧʔωϧύχοΫͰ͖ΔΑ͏ʹɺϚδοΫSysRqΛ༗ޮʹ͢Δ
 
 
 ϚδοΫSysRqΛ༗ޮʹͨ͠؀ڥͰ͸ɺҎԼ͕༗ޮʹͳΔ ★ Alt + SysRq + C ͰΧʔωϧύχοΫ (ίϯιʔϧ্ͷΈ) ★ γΣϧͰ΍Γ͍ͨ৔߹͸/proc/sysrq-triggerʹCΛॻ͖ࠐΉ
 ➤ forkര஄ʹΑΔແବͳిؾ୅ΛݮΒͨ͢ΊCPUΛμ΢ϯΫϩοΫ ➤ ͋ͱ͸forkര஄Λ24࣮࣌ؒߦ͠ଓ͚Δ͚ͩ! # sysctl kernel.sysrq=1
 # sysctl kernel.panic_on_oops=1 $ echo C > /proc/sysrq-trigger

ඃ֐ऀͦͷ̎ɹ̎̐࣌ؒख़੒؀ڥ ➤ VMware PlayerʹͯԾ૝ϚγϯΛ࡞੒ ➤ OS: CentOS 7.3 (x86_64, ࠷খΠϯετʔϧ) ➤ CPUίΞ਺: 3 (࣮ػ͸4ίΞ8εϨου) ➤ ϝϞϦ: 16GB ➤ σΟεΫ: SSD ➤ forkര஄ͷ࢓ࠐΈ͸24࣌ؒલ

࣮ԋ

࣮ߦ݁Ռ crash> sys
 KERNEL: /usr/lib/debug/lib/modules/3.10.0-514.16.1.el7.x86_64/ vmlinux
 DUMPFILE: vmcore [PARTIAL DUMP]
 CPUS: 3
 DATE: Sun Apr 23 17:03:17 2017
 UPTIME: 1 days, 00:09:34
 LOAD AVERAGE: 66363.51, 66362.75, 66362.18
 TASKS: 113062
 NODENAME: localhost.localdomain
 RELEASE: 3.10.0-514.16.1.el7.x86_64
 VERSION: #1 SMP Wed Apr 12 15:04:24 UTC 2017
 MACHINE: x86_64 (3996 Mhz)
 MEMORY: 16 GB
 PANIC: "SysRq : Trigger a crash"
 crash>

࣮ߦ݁Ռ crash> kmem -i
 PAGES TOTAL PERCENTAGE
 TOTAL MEM 4062963 15.5 GB ----
 FREE 33982 132.7 MB 0% of TOTAL MEM
 USED 4028981 15.4 GB 99% of TOTAL MEM
 SHARED 407319 1.6 GB 10% of TOTAL MEM
 BUFFERS 0 0 0% of TOTAL MEM
 CACHED 2918 11.4 MB 0% of TOTAL MEM
 SLAB 900008 3.4 GB 22% of TOTAL MEM
 
 TOTAL SWAP 0 0 ----
 SWAP USED 0 0 100% of TOTAL SWAP
 SWAP FREE 0 0 0% of TOTAL SWAP
 
 COMMIT LIMIT 2031481 7.7 GB ----
 COMMITTED 12510251 47.7 GB 615% of TOTAL LIMIT
 crash>

࣮ߦ݁Ռ crash> ps | grep bash | wc -l
 112940
 crash> ps | grep bash | grep -v ZO | wc -l
 66341
 crash> ps | grep bash | grep ZO | wc -l
 46599
 crash> ←࣮ߦத/଴ػதͷbashϓϩηε ←κϯϏঢ়ଶͷbashϓϩηε

̎̐࣌ؒख़੒݁Ռ ➤ ϝϞϦɺϓϩηεঢ়ଶͷ͍ͣΕ΋ϝϞϦෆ଍௚ޙͱมΘΒͳ͔ͬͨ
 ˞࣮ߦϓϩηε਺͕ଟ͍ͷ͸౥ࡌϝϞϦ͕ଟ͍ͨΊ ➤ ͨͩCPU࣌ؒΛ࿘අ͢Δ͚ͩͩͬͨɾɾɾ

༧๷ ➤ 1Ϣʔβʔ͕ੜ੒Ͱ͖Δϓϩηε਺ʹ্ݶΛ͚ͭΔ
 ˞ઃఆํ๏͸σΟετϦϏϡʔγϣϯʹΑΓҟͳΔ
 ɹulimit, setrlimit, /etc/security/limits.confͷnproc
 ※CPUϦιʔεΛ઎༗͢ΔͷͰɺ׬શʹ͸๷͛ͳ͍͜ͱʹ஫ҙ ➤ forkര஄ݕग़ΧʔωϧϞδϡʔϧrexFBDΛಋೖ͢Δ
 ˞2000೥Ҏདྷϝϯςφϯε͞Ε͍ͯͳ͍ ➤ ୭͕forkര஄Λ࣮ߦͨ͠ͷ͔͕Θ͔ΔηΩϡϦςΟରࡦύον(grsecurity)΋ଘࡏ͢ Δ

ΫϥογϡμϯϓͰ͸Θ͔Βͳ͍՝୊ ➤ Ϋϥογϡμϯϓ͸εφοϓγϣοτ
 มԽΛଊ͑ΒΕͳ͍
 ྫ1) രൃதͷbashϓϩηεͷҰੜΛ௥͏
 ྫ2) bashϓϩηε͕κϯϏϓϩηεʹͳΔաఔ
 ྫ3) ࣌ؒ͝ͱͷϝϞϦফඅྔͷભҠ
 ͳͲ

·ͱΊ ➤ forkര஄͸୹࣌ؒʹେྔͷࢠϓϩηεΛੜ੒͢Δ௒ةݥͳര஄ ➤ forkര஄͕ةݥͳཧ༝ ➤ ϦιʔεଈރׇͰγΣϧԠ౴ෆೳ ➤ ఀࢭࠔ೉ ➤ forkര஄࣮ߦதͷϓϩηεঢ়ଶΛΫϥογϡμϯϓͰ؍࡯ ➤ େྔͷbashϓϩηεʹΑΔϦιʔεރׇ͕໌֬ʹ ➤ 24࣌ؒ์ஔͯ͠΋ಛஈͷมԽͳ͠ ➤ ରࡦ͸ੜ੒ϓϩηε਺੍ݶ

ࢀߟ ➤ ࣌ؒࠩϑΥʔΫര஄ͷఏҊͱ࣮ߦ #ةݥγΣϧܳ
 https://blog.ueda.asia/?p=7081 ➤ େྔϓϩηεੜ੒ʹΑΔOOM KillerΛ༻͍ͨ߈ܸ΁ͷରࡦ
 http://www.ipsj.or.jp/sig/os/index.php? plugin=attach&refer=ComSys2012%2Fposter&openfile=12-613-1.pdf ➤ kdumpͷઃఆͱcoreͷղੳํ๏
 http://nekoi.net/linux/kdump-kexec-tools-kernel-debug ➤ QA/Sysrq - FedoraProject
 https://fedoraproject.org/wiki/QA/Sysrq ➤ rexFBD
 http://rexgrep.tripod.com/rexfbdmain.htm