logrotate殺プロセス事件 YAPC::Okinawa 2018 前夜祭 LT/Mystery of logrotate's death

YAPC::Okinawa 2018 ONNASON 前夜祭 LT

Transcript

1. logrotate
   ࡴɹࣄ݅ @kazeburo ϓϩ
   ηε
2. Someday 3:30 AM [CRITICAL] Fluentd’s position file has not been updated for several minutes. Alert!! AppXXX
3. Jan 24 03:24:18 appXXX logrotate: ALERT exited abnormally with [1] We found these error msg in /var/log/message.
4. • The files appear to be rotated. • But `postrotate` was not executed. • The daemon kept writing logs to the old file. • Fluentd read a new file, but no logs are written. /var/log/httpd/*log { compress daily delaycompress missingok ifempty rotate 7 dateext sharedscripts su root root postrotate /sbin/service httpd graceful > \ /dev/null 2>/dev/null || true endscript
5. How works logrotate • Find log files to rotate. • Rename old log files. • Create new logfiles with `open(name, O_CREAT |O_EXCL)`. • exec `postrotate`. DIED HERE NOT EXCUTED
6. O_CREAT|O_EXCL • O_CREAT • ϑΝΠϧ͕ଘࡏ͠ͳ͔ͬͨ৔߹͸࡞੒ (create) ͢Δɻ • O_EXCL • ͜ͷݺͼग़͠ͰϑΝΠϧ͕࡞੒͞ΕΔ͜ͱΛอূ͢Δɻ͜ͷϑϥά ͕ O_CREAT ͱ Ұॹʹࢦఆ͞Εɺ pathname ͷϑΝΠϧ͕طʹଘࡏͨ͠৔ ߹ɺ open() ͸ࣦഊ ͢Δɻ
7. ɾɾɾͭ·Γ
8. Someone created a file during the moment between rename and create at 3:30 AM!!!
9. ͳɾɾɾͳΜ (ུ
11. • PHP͸ϦΫΤετॲཧ͕ऴΘΔͱجຊશͯͷϝϞϦΛղ์͢Δ • File descriptorΛϦΫΤετΛ௒͑ͯอ࣋͢Δ͜ͱ͕Ͱ͖ͳ͍ • Monologͱ͍͏loggerϥΠϒϥϦͰ͸౎౓O_CREATΛ͚ͭͯϑΝΠϧ։͘ • ͭ·Γrename௚ޙʹloggerΛ࢖͏ͱɺlogrotate͕࡞੒͢ΔΑΓ΋ૣ͘ϑΝΠϧ Λ࡞ͬͯ͠·͏ɻ3࣌൒ʹ
13. • logrotate͕ऴΘΔ·ͰϦΫΤετΛࢭΊΔ • logrotate supports `prerotate` & `lastaction`. • logrotateʹpatchΛ͋ͯͯO_EXCLΛ࡟আ • logroateͷڍಈΛ֎͔Βมߋͯ͠O_EXCLΛ࡟আ ࠾༻
14. LD_PRELOAD ؀ڥม਺ LD_PRELOAD Λ࢖͏ͱϓϩάϥϜͷ࣮ߦલʹ ೚ҙͷڞ༗ΦϒδΣΫτ(ϥΠϒϥϦ)ΛಡΈࠐΈɺ ϓϩάϥϜͷڍಈʹมߋΛՃ͑Δ͜ͱ͕Ͱ͖Δ
15. https://github.com/kazeburo/no_excl_open int open(const char *pathname, int flags, ...) { static int (*func_open)(const char *, int, mode_t); va_list ap; mode_t mode; int fd; if (!func_open) func_open = dlsym (RTLD_NEXT, "open"); va_start(ap, flags); mode = va_arg(ap, int); va_end(ap); // O_RDWR and ! O_TRUNC if ( (flags & O_CREAT) != 0 && (flags & O_RDWR) != 0 && (flags & O_TRUNC) == 0 ) { flags = flags & ~O_EXCL; // ͜͜ͰO_EXCLΛআڈ } fd = func_open(pathname, flags, mode); return fd; }
16. /etc/cron.daily/logrotate #!/bin/sh LD_PRELOAD=/path/to/open_no_excl.so /usr/sbin/logrotate \ -s /var/lib/logrotate/logrotate.status /etc/logrotate.conf EXITVALUE=$? if [ $EXITVALUE != 0 ]; then /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]" fi exit 0
17. ✌ ഊ๺Λ஌Γ͍ͨ