Upgrade to Pro — share decks privately, control downloads, hide ads and more …

fluentdで大量のログを Google BigQueryに送ってみた #fluentdmeetup /send_a_log_to_bigquery_with_fluentd

fluentdで大量のログを Google BigQueryに送ってみた #fluentdmeetup /send_a_log_to_bigquery_with_fluentd

Fluentd Meetup 2015 夏の発表資料

C0ac34dd3f585ed25b2f79857e5b174d?s=128

Kaneko Tatsuya

June 01, 2015
Tweet

Transcript

  1. qVFOUEͰେྔͷϩάΛ (PPHMF#JH2VFSZʹૹͬͯΈͨ ϐΫγϒΠϯϑϥDBUBUTVZ

  2. ࣗݾ঺հ w DBUBUTVZͱ͍͏*%ͰΠϯλʔωοτ্Ͱ׆ಈ w ΧλπΠͱ͔ͬͯݺ͹Ε͍ͯΔ w ϐΫγϒגࣜձࣾͰΠϯϑϥΛ୲౰ w ޿ࠂαʔόʔ͕ଟ͍͕ଞʹ΋৭ʑ w

    ޿ࠂαʔόʔͷ࿩ʹ͍ͭͯ͸ࠓ౓ͷ(P$POͰɺएखΠϯϑϥͱͯ͠ͷ࿩ʹͭ ͍ͯ+5'ͷ࠲ஊձͰ࿩͢༧ఆ w ৽ॻͰʰQJYJWΤϯδχΞ͕ڭ͑Δϓϩάϥϛϯάೖ໳ʱͱ͍͏ຊΛग़͠·ͨ͠ w ϐΫγϒࣾ಺ͷඇΤϯδχΞ޲͚ʹ࡞ͬͨݚम಺༰ͷॻ੶Խ w ࠓճ͸ࣾ಺ͷ਺ଟ͍qVFOUEࣄྫͷͭΛ঺հ
  3. #JH2VFSZΛಋೖ͍ͨ͠ཧ༝ w ϩάΛอଘ͢ΔετϨʔδαʔόʔΛ༻ҙ͢Δͷ͸େม w #JH2VFSZͷετϨʔδྉۚ͸QFS(# QFSNPOUI w #JH2VFSZʹૹΕ͹؆୯ʹϩάσʔλΛղੳͰ͖Δ w 42--JLFͷߏจͰ৘ใΛऔಘͰ͖Δ

    w +0*/ͳͲͰଞͷςʔϒϧͱͷ৘ใͱඥ෇͚Δ͜ͱ΋؆୯ w େྔͰ΋͙͢ʹ݁Ռ͕ฦͬͯ͘Δ w ࣾ಺Ͱ΋ղੳج൫ͱͯ͠ಋೖࣄྫ͕૿͖͍͑ͯͯΔ w "1*΋͋Δ
  4. େྔͷϩάΛ#JH2VFSZʹૹΓ͍ͨ w qVFOUEͰϩάΛू໿αʔόʔʹूΊΔج൫͸Ͱ͖͍ͯͨͷͰू໿αʔόʔ ͔ΒૹΔΑ͏ʹ͍ͨ͠ w qVFOUQMVHJOCJHRVFSZΛ࢖ͬͯ4USFBNJOH*OTFSUTͰॻ͖ࠐΉ͜ͱʹ w 'MVFOUEͰ(PPHMF#JH2VFSZʹϩάΛૠೖͯ͠ΫΤϦΛ࣮ߦ͢Δ2JJUB IUUQRJJUBDPNIBSVLBTBOJUFNTFEBDFD •

    buffer_type fileʹͯ͠buffer_chunk_limitΛ#JH2VFSZͷ੍ݶ ͷ.#ΑΓখ͘͞ɺbuffer_queue_limitΛେ͖͘ • buffer_queue_limitͷ෼ϑΝΠϧσΟεΫϦϓλΛେ͖͘ w #JH2VFSZ͕௕࣌ؒো֐ʹͳΒͳ͍ݶΓ͸҆ఆͯ͠ૹΕΔ
  5. #JH2VFSZͷྉۚ w 4USFBNJOH*OTFSUTʹ͸݄·Ͱ͸QFS SPXT͔͔Δ w ݄ʹྉ͕ۚมΘΔ w 4UPSBHFʹ͸QFS(# QFSNPOUI w

    େྔͷϩάΛ4USFBNJOH*OTFSUTͰૹΔͱૹΔ͚͓͕͔͔ͩۚΔ w IUUQTDMPVEHPPHMFDPNCJHRVFSZQSJDJOH w όονॲཧͰૹΔ෼ʹ͸ྉۚ͸͔͔Βͳ͍ w ͨͩ͠όονͰૹΔ৔߹͸ VQEBUFTQFSUBCMFQFSEBZͷ੍ݶ͕͋Δ w IUUQTDMPVEHPPHMFDPNCJHRVFSZRVPUBQPMJDZ
  6. #JH2VFSZʹόονॲཧͰ҆ఆతʹૹΔ w ྉۚΛ཈͑ΔͨΊʹόονͰૹΓ͍ͨ w qVFOUQMVHJOpMFBMUFSOBUJWFͰH[JQѹॖͨ͠ϑΝΠϧΛอଘ͢Δ͜ͱ͕Ͱ͖Δ • buffer_type fileʹͯ͠buffer_chunk_limit/flush_intervalΛద੾ʹઃఆ ͢Δͱ҆ఆͯ͠ϑΝΠϧΛॻ͖ࠐΊΔ •

    ݱࡏbuffer_chunk_limit 512m/flush_interval 5m w খ͞ͳH[JQϑΝΠϧ͕ͨ͘͞ΜͰ͖ΔͨΊ݁߹ͯ͠େ͖ͳH[JQϑΝΠϧΛ࡞Δ w +FOLJOTͰ࣌ؒʹճ࣮ߦ͍ͯ͠Δ w +FOLJOTͰCRίϚϯυΛ࢖ͬͯH[JQϑΝΠϧΛ௚઀ૹΔ w ϐʔΫ࣌ؒଳલޙ͸࣌ؒͰૹΓ͖Εͳ͍͕+FOLJOT͕ΩϡʔΠϯάͯ͘͠ΕΔ w ࣦഊͨ͠Β࣍ͷϏϧυͰෳ਺ͷϩάϑΝΠϧΛૹΔ
  7. ·ͱΊ w #JH2VFSZʹେྔʹϩάΛૹΔͨΊʹqVFOUEΛ࢖༻ͨ͠ w 4USFBNJOH*OTFSUTͰૹΔ৔߹͸buffer_type fileʹͯ͠ buffer_chunk_limitͱbuffer_queue_limitΛௐ੔͢Δ w Ұ౓ϑΝΠϧʹอଘ͍ͨ͠৔߹͸H[JQѹॖͯ͠ buffer_chunk_limit/flush_intervalΛௐ੔͢Δ

    w όονͰૹͬͨํ͕͍҆ͷͰྔ͕ଟ͍ͱϝϦοτ͕େ͖͍ w όονͰૹΔ৔߹͸ࣦഊͨ͠ͱ͖ͷ͜ͱΛͪΌΜͱߟ͑Δ