Upgrade to Pro — share decks privately, control downloads, hide ads and more …

logrotate殺プロセス事件 YAPC::Okinawa 2018 前夜祭 LT/Mystery of logrotate's death

kazeburo
March 03, 2018

logrotate殺プロセス事件 YAPC::Okinawa 2018 前夜祭 LT/Mystery of logrotate's death

YAPC::Okinawa 2018 ONNASON 前夜祭 LT

kazeburo

March 03, 2018
Tweet

More Decks by kazeburo

Other Decks in Technology

Transcript

  1. Someday 3:30 AM [CRITICAL] Fluentd’s position file has not been

    updated for several minutes. Alert!! AppXXX
  2. Jan 24 03:24:18 appXXX logrotate: ALERT exited abnormally with [1]

    We found these error msg in /var/log/message.
  3. • The files appear to be rotated. • But `postrotate`

    was not executed. • The daemon kept writing logs to the old file. • Fluentd read a new file, but no logs are written. /var/log/httpd/*log { compress daily delaycompress missingok ifempty rotate 7 dateext sharedscripts su root root postrotate /sbin/service httpd graceful > \ /dev/null 2>/dev/null || true endscript
  4. How works logrotate • Find log files to rotate. •

    Rename old log files. • Create new logfiles with `open(name, O_CREAT |O_EXCL)`. • exec `postrotate`. DIED HERE NOT EXCUTED
  5. O_CREAT|O_EXCL • O_CREAT • ϑΝΠϧ͕ଘࡏ͠ͳ͔ͬͨ৔߹͸࡞੒ (create) ͢Δɻ • O_EXCL •

    ͜ͷݺͼग़͠ͰϑΝΠϧ͕࡞੒͞ΕΔ͜ͱΛอূ͢Δɻ͜ͷϑϥά ͕ O_CREAT ͱ Ұॹʹࢦఆ͞Εɺ pathname ͷϑΝΠϧ͕طʹଘࡏͨ͠৔ ߹ɺ open() ͸ࣦഊ ͢Δɻ
  6. https://github.com/kazeburo/no_excl_open int open(const char *pathname, int flags, ...) { static

    int (*func_open)(const char *, int, mode_t); va_list ap; mode_t mode; int fd; if (!func_open) func_open = dlsym (RTLD_NEXT, "open"); va_start(ap, flags); mode = va_arg(ap, int); va_end(ap); // O_RDWR and ! O_TRUNC if ( (flags & O_CREAT) != 0 && (flags & O_RDWR) != 0 && (flags & O_TRUNC) == 0 ) { flags = flags & ~O_EXCL; // ͜͜ͰO_EXCLΛআڈ } fd = func_open(pathname, flags, mode); return fd; }