Pro Yearly is on sale from $80 to $50! »

logrotate殺プロセス事件 YAPC::Okinawa 2018 前夜祭 LT/Mystery of logrotate's death

700669515ee872152d8b9403c2a0cf8c?s=47 kazeburo
March 03, 2018

logrotate殺プロセス事件 YAPC::Okinawa 2018 前夜祭 LT/Mystery of logrotate's death

YAPC::Okinawa 2018 ONNASON 前夜祭 LT

700669515ee872152d8b9403c2a0cf8c?s=128

kazeburo

March 03, 2018
Tweet

Transcript

  1. logrotateࡴɹࣄ݅ @kazeburo ϓϩ ηε

  2. Someday 3:30 AM [CRITICAL] Fluentd’s position file has not been

    updated for several minutes. Alert!! AppXXX
  3. Jan 24 03:24:18 appXXX logrotate: ALERT exited abnormally with [1]

    We found these error msg in /var/log/message.
  4. • The files appear to be rotated. • But `postrotate`

    was not executed. • The daemon kept writing logs to the old file. • Fluentd read a new file, but no logs are written. /var/log/httpd/*log { compress daily delaycompress missingok ifempty rotate 7 dateext sharedscripts su root root postrotate /sbin/service httpd graceful > \ /dev/null 2>/dev/null || true endscript
  5. How works logrotate • Find log files to rotate. •

    Rename old log files. • Create new logfiles with `open(name, O_CREAT |O_EXCL)`. • exec `postrotate`. DIED HERE NOT EXCUTED
  6. O_CREAT|O_EXCL • O_CREAT • ϑΝΠϧ͕ଘࡏ͠ͳ͔ͬͨ৔߹͸࡞੒ (create) ͢Δɻ • O_EXCL •

    ͜ͷݺͼग़͠ͰϑΝΠϧ͕࡞੒͞ΕΔ͜ͱΛอূ͢Δɻ͜ͷϑϥά ͕ O_CREAT ͱ Ұॹʹࢦఆ͞Εɺ pathname ͷϑΝΠϧ͕طʹଘࡏͨ͠৔ ߹ɺ open() ͸ࣦഊ ͢Δɻ
  7. ɾɾɾͭ·Γ

  8. Someone created a file during the moment between rename and

    create at 3:30 AM!!!
  9. ͳɾɾɾͳΜ (ུ

  10. None
  11. • PHP͸ϦΫΤετॲཧ͕ऴΘΔͱجຊશͯͷϝϞϦΛղ์͢Δ • File descriptorΛϦΫΤετΛ௒͑ͯอ࣋͢Δ͜ͱ͕Ͱ͖ͳ͍ • Monologͱ͍͏loggerϥΠϒϥϦͰ͸౎౓O_CREATΛ͚ͭͯϑΝΠϧ։͘ • ͭ·Γrename௚ޙʹloggerΛ࢖͏ͱɺlogrotate͕࡞੒͢ΔΑΓ΋ૣ͘ϑΝΠϧ Λ࡞ͬͯ͠·͏ɻ3࣌൒ʹ

  12. None
  13. • logrotate͕ऴΘΔ·ͰϦΫΤετΛࢭΊΔ • logrotate supports `prerotate` & `lastaction`. • logrotateʹpatchΛ͋ͯͯO_EXCLΛ࡟আ

    • logroateͷڍಈΛ֎͔Βมߋͯ͠O_EXCLΛ࡟আ ࠾༻
  14. LD_PRELOAD ؀ڥม਺ LD_PRELOAD Λ࢖͏ͱϓϩάϥϜͷ࣮ߦલʹ ೚ҙͷڞ༗ΦϒδΣΫτ(ϥΠϒϥϦ)ΛಡΈࠐΈɺ ϓϩάϥϜͷڍಈʹมߋΛՃ͑Δ͜ͱ͕Ͱ͖Δ

  15. https://github.com/kazeburo/no_excl_open int open(const char *pathname, int flags, ...) { static

    int (*func_open)(const char *, int, mode_t); va_list ap; mode_t mode; int fd; if (!func_open) func_open = dlsym (RTLD_NEXT, "open"); va_start(ap, flags); mode = va_arg(ap, int); va_end(ap); // O_RDWR and ! O_TRUNC if ( (flags & O_CREAT) != 0 && (flags & O_RDWR) != 0 && (flags & O_TRUNC) == 0 ) { flags = flags & ~O_EXCL; // ͜͜ͰO_EXCLΛআڈ } fd = func_open(pathname, flags, mode); return fd; }
  16. /etc/cron.daily/logrotate #!/bin/sh LD_PRELOAD=/path/to/open_no_excl.so /usr/sbin/logrotate \ -s /var/lib/logrotate/logrotate.status /etc/logrotate.conf EXITVALUE=$? if

    [ $EXITVALUE != 0 ]; then /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]" fi exit 0
  17. ✌ ഊ๺Λ஌Γ͍ͨ