fluentdで大量のログを Google BigQueryに送ってみた #fluentdmeetup /send_a_log_to_bigquery_with_fluentd

Fluentd Meetup 2015 夏の発表資料

Transcript

1. qVFOUEͰେྔͷϩάΛ
   (PPHMF#JH2VFSZʹૹͬͯΈͨ
   ϐΫγϒΠϯϑϥDBUBUTVZ
   

   View Slide

2. ࣗݾ঺հ
   w DBUBUTVZͱ͍͏*%ͰΠϯλʔωοτ্Ͱ׆ಈ
   w ΧλπΠͱ͔ͬͯݺ͹Ε͍ͯΔ
   w ϐΫγϒגࣜձࣾͰΠϯϑϥΛ୲౰
   w ޿ࠂαʔόʔ͕ଟ͍͕ଞʹ΋৭ʑ
   w ޿ࠂαʔόʔͷ࿩ʹ͍ͭͯ͸ࠓ౓ͷ(P$POͰɺएखΠϯϑϥͱͯ͠ͷ࿩ʹͭ
   ͍ͯ+5'ͷ࠲ஊձͰ࿩͢༧ఆ
   w ৽ॻͰʰQJYJWΤϯδχΞ͕ڭ͑Δϓϩάϥϛϯάೖ໳ʱͱ͍͏ຊΛग़͠·ͨ͠
   w ϐΫγϒࣾ಺ͷඇΤϯδχΞ޲͚ʹ࡞ͬͨݚम಺༰ͷॻ੶Խ
   w ࠓճ͸ࣾ಺ͷ਺ଟ͍qVFOUEࣄྫͷͭΛ঺հ
   

   View Slide

3. #JH2VFSZΛಋೖ͍ͨ͠ཧ༝
   w ϩάΛอଘ͢ΔετϨʔδαʔόʔΛ༻ҙ͢Δͷ͸େม
   w #JH2VFSZͷετϨʔδྉۚ͸QFS(# QFSNPOUI
   w #JH2VFSZʹૹΕ͹؆୯ʹϩάσʔλΛղੳͰ͖Δ
   w 42--JLFͷߏจͰ৘ใΛऔಘͰ͖Δ
   w +0*/ͳͲͰଞͷςʔϒϧͱͷ৘ใͱඥ෇͚Δ͜ͱ΋؆୯
   w େྔͰ΋͙͢ʹ݁Ռ͕ฦͬͯ͘Δ
   w ࣾ಺Ͱ΋ղੳج൫ͱͯ͠ಋೖࣄྫ͕૿͖͍͑ͯͯΔ
   w "1*΋͋Δ
   

   View Slide

4. େྔͷϩάΛ#JH2VFSZʹૹΓ͍ͨ
   w qVFOUEͰϩάΛू໿αʔόʔʹूΊΔج൫͸Ͱ͖͍ͯͨͷͰू໿αʔόʔ
   ͔ΒૹΔΑ͏ʹ͍ͨ͠
   w qVFOUQMVHJOCJHRVFSZΛ࢖ͬͯ4USFBNJOH*OTFSUTͰॻ͖ࠐΉ͜ͱʹ
   w 'MVFOUEͰ(PPHMF#JH2VFSZʹϩάΛૠೖͯ͠ΫΤϦΛ࣮ߦ͢Δ2JJUB
   IUUQRJJUBDPNIBSVLBTBOJUFNTFEBDFD
   • buffer_type fileʹͯ͠buffer_chunk_limitΛ#JH2VFSZͷ੍ݶ
   ͷ.#ΑΓখ͘͞ɺbuffer_queue_limitΛେ͖͘
   • buffer_queue_limitͷ෼ϑΝΠϧσΟεΫϦϓλΛେ͖͘
   w #JH2VFSZ͕௕࣌ؒো֐ʹͳΒͳ͍ݶΓ͸҆ఆͯ͠ૹΕΔ
   

   View Slide

5. #JH2VFSZͷྉۚ
   w 4USFBNJOH*OTFSUTʹ͸݄·Ͱ͸QFS SPXT͔͔Δ
   w ݄ʹྉ͕ۚมΘΔ
   w 4UPSBHFʹ͸QFS(# QFSNPOUI
   w େྔͷϩάΛ4USFBNJOH*OTFSUTͰૹΔͱૹΔ͚͓͕͔͔ͩۚΔ
   w IUUQTDMPVEHPPHMFDPNCJHRVFSZQSJDJOH
   w όονॲཧͰૹΔ෼ʹ͸ྉۚ͸͔͔Βͳ͍
   w ͨͩ͠όονͰૹΔ৔߹͸ VQEBUFTQFSUBCMFQFSEBZͷ੍ݶ͕͋Δ
   w IUUQTDMPVEHPPHMFDPNCJHRVFSZRVPUBQPMJDZ
   

   View Slide

6. #JH2VFSZʹόονॲཧͰ҆ఆతʹૹΔ
   w ྉۚΛ཈͑ΔͨΊʹόονͰૹΓ͍ͨ
   w qVFOUQMVHJOpMFBMUFSOBUJWFͰH[JQѹॖͨ͠ϑΝΠϧΛอଘ͢Δ͜ͱ͕Ͱ͖Δ
   • buffer_type fileʹͯ͠buffer_chunk_limit/flush_intervalΛద੾ʹઃఆ
   ͢Δͱ҆ఆͯ͠ϑΝΠϧΛॻ͖ࠐΊΔ
   • ݱࡏbuffer_chunk_limit 512m/flush_interval 5m
   w খ͞ͳH[JQϑΝΠϧ͕ͨ͘͞ΜͰ͖ΔͨΊ݁߹ͯ͠େ͖ͳH[JQϑΝΠϧΛ࡞Δ
   w +FOLJOTͰ࣌ؒʹճ࣮ߦ͍ͯ͠Δ
   w +FOLJOTͰCRίϚϯυΛ࢖ͬͯH[JQϑΝΠϧΛ௚઀ૹΔ
   w ϐʔΫ࣌ؒଳલޙ͸࣌ؒͰૹΓ͖Εͳ͍͕+FOLJOT͕ΩϡʔΠϯάͯ͘͠ΕΔ
   w ࣦഊͨ͠Β࣍ͷϏϧυͰෳ਺ͷϩάϑΝΠϧΛૹΔ
   

   View Slide

7. ·ͱΊ
   w #JH2VFSZʹେྔʹϩάΛૹΔͨΊʹqVFOUEΛ࢖༻ͨ͠
   w 4USFBNJOH*OTFSUTͰૹΔ৔߹͸buffer_type fileʹͯ͠
   buffer_chunk_limitͱbuffer_queue_limitΛௐ੔͢Δ
   w Ұ౓ϑΝΠϧʹอଘ͍ͨ͠৔߹͸H[JQѹॖͯ͠
   buffer_chunk_limit/flush_intervalΛௐ੔͢Δ
   w όονͰૹͬͨํ͕͍҆ͷͰྔ͕ଟ͍ͱϝϦοτ͕େ͖͍
   w όονͰૹΔ৔߹͸ࣦഊͨ͠ͱ͖ͷ͜ͱΛͪΌΜͱߟ͑Δ
   

   View Slide