fluentdで大量のログを Google BigQueryに送ってみた #fluentdmeetup /send_a_log_to_bigquery_with_fluentd

Fluentd Meetup 2015 夏の発表資料

Transcript

1. qVFOUEͰେྔͷϩάΛ
   (PPHMF
   #JH2VFSZʹૹͬͯΈͨ ϐΫγϒ
   Πϯϑϥ
   DBUBUTVZ
2. ࣗݾ঺հ w DBUBUTVZͱ͍͏*%ͰΠϯλʔωοτ্Ͱ׆ಈ
   w ΧλπΠͱ͔ͬͯݺ͹Ε͍ͯΔ
   w ϐΫγϒגࣜձࣾͰΠϯϑϥΛ୲౰
   w ޿ࠂαʔόʔ͕ଟ͍͕ଞʹ΋৭ʑ
   w ޿ࠂαʔόʔͷ࿩ʹ͍ͭͯ͸ࠓ౓ͷ(P$POͰɺएखΠϯϑϥͱͯ͠ͷ࿩ʹͭ ͍ͯ+5'ͷ࠲ஊձͰ࿩͢༧ఆ
   w ৽ॻͰʰQJYJWΤϯδχΞ͕ڭ͑Δϓϩάϥϛϯάೖ໳ʱͱ͍͏ຊΛग़͠·ͨ͠
   w ϐΫγϒࣾ಺ͷඇΤϯδχΞ޲͚ʹ࡞ͬͨݚम಺༰ͷॻ੶Խ
   w ࠓճ͸ࣾ಺ͷ਺ଟ͍qVFOUEࣄྫͷͭΛ঺հ
3. #JH2VFSZΛಋೖ͍ͨ͠ཧ༝ w ϩάΛอଘ͢ΔετϨʔδαʔόʔΛ༻ҙ͢Δͷ͸େม
   w #JH2VFSZͷετϨʔδྉۚ͸
   QFS
   (#
   QFS
   NPOUI
   w #JH2VFSZʹૹΕ͹؆୯ʹϩάσʔλΛղੳͰ͖Δ
   w 42-
   -JLFͷߏจͰ৘ใΛऔಘͰ͖Δ
   w +0*/ͳͲͰଞͷςʔϒϧͱͷ৘ใͱඥ෇͚Δ͜ͱ΋؆୯
   w େྔͰ΋͙͢ʹ݁Ռ͕ฦͬͯ͘Δ
   w ࣾ಺Ͱ΋ղੳج൫ͱͯ͠ಋೖࣄྫ͕૿͖͍͑ͯͯΔ
   w "1*΋͋Δ
4. େྔͷϩάΛ#JH2VFSZʹૹΓ͍ͨ w qVFOUEͰϩάΛू໿αʔόʔʹूΊΔج൫͸Ͱ͖͍ͯͨͷͰू໿αʔόʔ ͔ΒૹΔΑ͏ʹ͍ͨ͠
   w qVFOUQMVHJOCJHRVFSZΛ࢖ͬͯ4USFBNJOH
   *OTFSUTͰॻ͖ࠐΉ͜ͱʹ
   w 'MVFOUEͰ(PPHMF
   #JH2VFSZʹϩάΛૠೖͯ͠ΫΤϦΛ࣮ߦ͢Δ
   
   2JJUB
   IUUQRJJUBDPNIBSVLBTBOJUFNTFEBDFD
   • buffer_type fileʹͯ͠buffer_chunk_limitΛ#JH2VFSZͷ੍ݶ ͷ.#ΑΓখ͘͞ɺbuffer_queue_limitΛେ͖͘ • buffer_queue_limitͷ෼ϑΝΠϧσΟεΫϦϓλΛେ͖͘ w #JH2VFSZ͕௕࣌ؒো֐ʹͳΒͳ͍ݶΓ͸҆ఆͯ͠ૹΕΔ
5. #JH2VFSZͷྉۚ w 4USFBNJOH
   *OTFSUTʹ͸݄·Ͱ͸
   QFS
    
   SPXT͔͔Δ
   w ݄ʹྉ͕ۚมΘΔ
   w 4UPSBHFʹ͸
   QFS
   (#
   QFS
   NPOUI
   w େྔͷϩάΛ4USFBNJOH
   *OTFSUTͰૹΔͱૹΔ͚͓͕͔͔ͩۚΔ
   w IUUQTDMPVEHPPHMFDPNCJHRVFSZQSJDJOH
   w όονॲཧͰૹΔ෼ʹ͸ྉۚ͸͔͔Βͳ͍
   w ͨͩ͠όονͰૹΔ৔߹͸ 
   VQEBUFT
   QFS
   UBCMF
   QFS
   EBZͷ੍ݶ͕͋Δ
   w IUUQTDMPVEHPPHMFDPNCJHRVFSZRVPUBQPMJDZ
6. #JH2VFSZʹόονॲཧͰ҆ఆతʹૹΔ w ྉۚΛ཈͑ΔͨΊʹόονͰૹΓ͍ͨ
   w qVFOUQMVHJOpMFBMUFSOBUJWFͰH[JQѹॖͨ͠ϑΝΠϧΛอଘ͢Δ͜ͱ͕Ͱ͖Δ
   • buffer_type fileʹͯ͠buffer_chunk_limit/flush_intervalΛద੾ʹઃఆ ͢Δͱ҆ఆͯ͠ϑΝΠϧΛॻ͖ࠐΊΔ • ݱࡏbuffer_chunk_limit 512m/flush_interval 5m w খ͞ͳH[JQϑΝΠϧ͕ͨ͘͞ΜͰ͖ΔͨΊ݁߹ͯ͠େ͖ͳH[JQϑΝΠϧΛ࡞Δ
   w +FOLJOTͰ࣌ؒʹճ࣮ߦ͍ͯ͠Δ
   w +FOLJOTͰCRίϚϯυΛ࢖ͬͯH[JQϑΝΠϧΛ௚઀ૹΔ
   w ϐʔΫ࣌ؒଳલޙ͸࣌ؒͰૹΓ͖Εͳ͍͕+FOLJOT͕ΩϡʔΠϯάͯ͘͠ΕΔ
   w ࣦഊͨ͠Β࣍ͷϏϧυͰෳ਺ͷϩάϑΝΠϧΛૹΔ
7. ·ͱΊ w #JH2VFSZʹେྔʹϩάΛૹΔͨΊʹqVFOUEΛ࢖༻ͨ͠
   w 4USFBNJOH
   *OTFSUTͰૹΔ৔߹͸buffer_type fileʹͯ͠ buffer_chunk_limitͱbuffer_queue_limitΛௐ੔͢Δ
   w Ұ౓ϑΝΠϧʹอଘ͍ͨ͠৔߹͸H[JQѹॖͯ͠ buffer_chunk_limit/flush_intervalΛௐ੔͢Δ
   w όονͰૹͬͨํ͕͍҆ͷͰྔ͕ଟ͍ͱϝϦοτ͕େ͖͍
   w όονͰૹΔ৔߹͸ࣦഊͨ͠ͱ͖ͷ͜ͱΛͪΌΜͱߟ͑Δ