$30 off During Our Annual Pro Sale. View Details »

fluentdで大量のログを Google BigQueryに送ってみた #fluentdmeetup /send_a_log_to_bigquery_with_fluentd

fluentdで大量のログを Google BigQueryに送ってみた #fluentdmeetup /send_a_log_to_bigquery_with_fluentd

Fluentd Meetup 2015 夏の発表資料

Kaneko Tatsuya

June 01, 2015
Tweet

More Decks by Kaneko Tatsuya

Other Decks in Programming

Transcript

  1. qVFOUEͰେྔͷϩάΛ
    (PPHMF#JH2VFSZʹૹͬͯΈͨ
    ϐΫγϒΠϯϑϥDBUBUTVZ

    View Slide

  2. ࣗݾ঺հ
    w DBUBUTVZͱ͍͏*%ͰΠϯλʔωοτ্Ͱ׆ಈ
    w ΧλπΠͱ͔ͬͯݺ͹Ε͍ͯΔ
    w ϐΫγϒגࣜձࣾͰΠϯϑϥΛ୲౰
    w ޿ࠂαʔόʔ͕ଟ͍͕ଞʹ΋৭ʑ
    w ޿ࠂαʔόʔͷ࿩ʹ͍ͭͯ͸ࠓ౓ͷ(P$POͰɺएखΠϯϑϥͱͯ͠ͷ࿩ʹͭ
    ͍ͯ+5'ͷ࠲ஊձͰ࿩͢༧ఆ
    w ৽ॻͰʰQJYJWΤϯδχΞ͕ڭ͑Δϓϩάϥϛϯάೖ໳ʱͱ͍͏ຊΛग़͠·ͨ͠
    w ϐΫγϒࣾ಺ͷඇΤϯδχΞ޲͚ʹ࡞ͬͨݚम಺༰ͷॻ੶Խ
    w ࠓճ͸ࣾ಺ͷ਺ଟ͍qVFOUEࣄྫͷͭΛ঺հ

    View Slide

  3. #JH2VFSZΛಋೖ͍ͨ͠ཧ༝
    w ϩάΛอଘ͢ΔετϨʔδαʔόʔΛ༻ҙ͢Δͷ͸େม
    w #JH2VFSZͷετϨʔδྉۚ͸QFS(# QFSNPOUI
    w #JH2VFSZʹૹΕ͹؆୯ʹϩάσʔλΛղੳͰ͖Δ
    w 42--JLFͷߏจͰ৘ใΛऔಘͰ͖Δ
    w +0*/ͳͲͰଞͷςʔϒϧͱͷ৘ใͱඥ෇͚Δ͜ͱ΋؆୯
    w େྔͰ΋͙͢ʹ݁Ռ͕ฦͬͯ͘Δ
    w ࣾ಺Ͱ΋ղੳج൫ͱͯ͠ಋೖࣄྫ͕૿͖͍͑ͯͯΔ
    w "1*΋͋Δ

    View Slide

  4. େྔͷϩάΛ#JH2VFSZʹૹΓ͍ͨ
    w qVFOUEͰϩάΛू໿αʔόʔʹूΊΔج൫͸Ͱ͖͍ͯͨͷͰू໿αʔόʔ
    ͔ΒૹΔΑ͏ʹ͍ͨ͠
    w qVFOUQMVHJOCJHRVFSZΛ࢖ͬͯ4USFBNJOH*OTFSUTͰॻ͖ࠐΉ͜ͱʹ
    w 'MVFOUEͰ(PPHMF#JH2VFSZʹϩάΛૠೖͯ͠ΫΤϦΛ࣮ߦ͢Δ2JJUB
    IUUQRJJUBDPNIBSVLBTBOJUFNTFEBDFD
    • buffer_type fileʹͯ͠buffer_chunk_limitΛ#JH2VFSZͷ੍ݶ
    ͷ.#ΑΓখ͘͞ɺbuffer_queue_limitΛେ͖͘
    • buffer_queue_limitͷ෼ϑΝΠϧσΟεΫϦϓλΛେ͖͘
    w #JH2VFSZ͕௕࣌ؒো֐ʹͳΒͳ͍ݶΓ͸҆ఆͯ͠ૹΕΔ

    View Slide

  5. #JH2VFSZͷྉۚ
    w 4USFBNJOH*OTFSUTʹ͸݄·Ͱ͸QFS SPXT͔͔Δ
    w ݄ʹྉ͕ۚมΘΔ
    w 4UPSBHFʹ͸QFS(# QFSNPOUI
    w େྔͷϩάΛ4USFBNJOH*OTFSUTͰૹΔͱૹΔ͚͓͕͔͔ͩۚΔ
    w IUUQTDMPVEHPPHMFDPNCJHRVFSZQSJDJOH
    w όονॲཧͰૹΔ෼ʹ͸ྉۚ͸͔͔Βͳ͍
    w ͨͩ͠όονͰૹΔ৔߹͸ VQEBUFTQFSUBCMFQFSEBZͷ੍ݶ͕͋Δ
    w IUUQTDMPVEHPPHMFDPNCJHRVFSZRVPUBQPMJDZ

    View Slide

  6. #JH2VFSZʹόονॲཧͰ҆ఆతʹૹΔ
    w ྉۚΛ཈͑ΔͨΊʹόονͰૹΓ͍ͨ
    w qVFOUQMVHJOpMFBMUFSOBUJWFͰH[JQѹॖͨ͠ϑΝΠϧΛอଘ͢Δ͜ͱ͕Ͱ͖Δ
    • buffer_type fileʹͯ͠buffer_chunk_limit/flush_intervalΛద੾ʹઃఆ
    ͢Δͱ҆ఆͯ͠ϑΝΠϧΛॻ͖ࠐΊΔ
    • ݱࡏbuffer_chunk_limit 512m/flush_interval 5m
    w খ͞ͳH[JQϑΝΠϧ͕ͨ͘͞ΜͰ͖ΔͨΊ݁߹ͯ͠େ͖ͳH[JQϑΝΠϧΛ࡞Δ
    w +FOLJOTͰ࣌ؒʹճ࣮ߦ͍ͯ͠Δ
    w +FOLJOTͰCRίϚϯυΛ࢖ͬͯH[JQϑΝΠϧΛ௚઀ૹΔ
    w ϐʔΫ࣌ؒଳલޙ͸࣌ؒͰૹΓ͖Εͳ͍͕+FOLJOT͕ΩϡʔΠϯάͯ͘͠ΕΔ
    w ࣦഊͨ͠Β࣍ͷϏϧυͰෳ਺ͷϩάϑΝΠϧΛૹΔ

    View Slide

  7. ·ͱΊ
    w #JH2VFSZʹେྔʹϩάΛૹΔͨΊʹqVFOUEΛ࢖༻ͨ͠
    w 4USFBNJOH*OTFSUTͰૹΔ৔߹͸buffer_type fileʹͯ͠
    buffer_chunk_limitͱbuffer_queue_limitΛௐ੔͢Δ
    w Ұ౓ϑΝΠϧʹอଘ͍ͨ͠৔߹͸H[JQѹॖͯ͠
    buffer_chunk_limit/flush_intervalΛௐ੔͢Δ
    w όονͰૹͬͨํ͕͍҆ͷͰྔ͕ଟ͍ͱϝϦοτ͕େ͖͍
    w όονͰૹΔ৔߹͸ࣦഊͨ͠ͱ͖ͷ͜ͱΛͪΌΜͱߟ͑Δ

    View Slide