AWSでクローラを構築するときのベストプラクティスを思いつきたい人生だった

雑兵MeetUp #3
http://zohyo.connpass.com/event/24420/

Transcript

1. AWSͰΫϩʔϥΛߏங͢Δ ͱ͖ͷϕετϓϥΫςΟεΛ ࢥ͍͖͍ͭͨਓੜͩͬͨ masaaki.takeuchi @ise_daisuke
2. ࣗݾ঺հ ✓ webͷΤϯδχΞͰ͢ ✓ ҩྍܥʁϔϧεέΞܥʁwebاۀͰಇ͍͍ͯ·͢ ✓ લճ͸ʮࡏ୐ۈ຿Λࢧ͑Δʢ͔΋͠Εͳ͍ʣٕज़ʯͱ͍͏ ൃදΛ͠·ͨ͠
3. ࠓ೔࿩͢͜ͱ ✓ ࢓ࣄͰ৮Δ͜ͱʹͳͬͨΫϩʔϥʹ͍ͭͯ ✓ ͜ͷۀքʹ10೥͍ۙ͘·͕͢·ͱ΋ʹΫϩʔϥʹ৮ͬͨ͜ͱ͸ͳ͔ͬͨɺɺ ✓ AWSʹ͓͚ΔΫϩʔϥΞʔΩςΫνϟʹ͍ͭͯ ✓ Ϋϩʔϥͷ࣮૷ͷ࿩͸΄΅͠·ͤΜ
4. զΒ͕Ϋϩʔϥʹ͍ͭͯ ✓ 47౎ಓ෎ݝ෼ͷͳʹ͔ΛΫϩʔϧ͢Δ΋ͷ ✓ ձࣾͷͦͷลʹஔ͍ͯ͋ΔαʔόͰɺखಈ࣮ߦͯͨ͠ ✓ ͜ΕΛఆظతʹࣗಈతʹ࣮ߦ͢Δʂˡϛογϣϯ
5. Ϋϩʔϥ ✓ ͳΜͱͳ͘ͷఆٛʢͱ͍͏͔ࢲͷೝࣝʣ ✓ web͔ΒσʔλΛऔಘ͠ߏ଄Խ͢Δ ✓ ͦͷಈ࡞͸ओʹ3ͭͷϑΣʔζʹ෼ׂͰ͖Δ ✓ Ϋϩʔϧ/μ΢ϯϩʔυ/ύʔε +------------------------------------------------------------+ | | | | | +-----------+ +------------+ +----------+ | +----+ | | | url | | html | | | | | | | crawler | +----> | downloader | +----> | parser | +---> | DB | | | | | | | | | | | | +-----------+ +------------+ +----------+ | +----+ | | | | +------------------------------------------------------------+
6. AWS্ʹΫϩʔϥΛߏங ✓ ElasticBeanstalkͷWorker ✓ http://docs.aws.amazon.com/ja_jp/elasticbeanstalk/latest/dg/ using-features-managing-env-tiers.html
7. ElasticBeanstalkΛར༻ͨ͠ΫϩʔϥΞʔΩςΫνϟ SQS ElastiCache S3 +-----------+ +----------+ +---------------+ | | | | | | | job queue | | lock | | shared data | | | | | | | +----+------+ +----+-----+ +---+-----------+ ^ ^ ^ | | | | | | +-----------+------------------+----------------+------------+ | | Lambda | worker1 worker2 worker3 | EC2 +---------+ | +-----------+ +------------+ +----------+ | +-----------+ | | | | | url | | html | | | | | | Trigger | +---> | | crawler | +----> | downloader | +----> | parser | +---> | mongoDB | | | | | | | | | | | | | +---------+ | +-----------+ +------------+ +----------+ | +-----------+ | | | | +------------------------------------------------------------+ Elastic Beanstalk
8. ͳΜ͔͖ͯ͢
9. ͖ͯ͢ͳͱ͜Ζ ✓ crawl/downloadͱparseΛฒߦͯ͠ಈ࡞Ͱ͖Δ ✓ ਈ࢜తʹcrawlˍdownloadɺparse͸ฒྻ࣮ߦ͠·͘Γ ✓ ʢcrawlerͱdownloader͸෼͚ͳͯ͘΋͍͍͔΋͠Εͳ͍ʣ ✓ ॲཧ͕٧·Γग़ͨ͠Βworker૿΍ͤ͹͍͍ʂ ✓ ཷ·͍ͬͯΔδϣϒͷ਺ʹ߹Θͤͯauto scaleʂ ✓ ͜Ε͕ϕετ͡ΌͶʢࡨཚ
10. ͔͠͠໰୊͕
11. ໰୊ ✓ Ϋϩʔϥͷ໰୊ ✓ crawler/downloader/parser͕͚ͬ͜͏ີʹ݁߹͍ͯ͠Δ ✓ Ϋϩʔϧର৅ʢαΠτଆʣͷ໰୊ ✓ 47௨ΓͷϩδοΫ͕ඞཁ ✓ post͡Όͳ͍ͱϖʔδભҠͰ͖ͳ͍ͱ͔ɺɺ ✓ nϖʔδ໨දࣔঢ়ଶ͔Βn+10ϖʔδ໨·Ͱ͔͠ϖʔδભҠͰ͖ͳ͍ͱ͔ɺɺ ✓ ʢؔ܎ͳ͍͚Ͳɺͳ͔ͥ߳઒ͱ௕໺ͱ࡛ۄͰΫϩʔϧࣦഊ͢Δ͜ͱ͕ଟ͍ͱ͔ɺɺʣ ✓ ͭ·Γ ✓ ·͋·͋࡞Γ௚͕͠ඞཁ͕ͩɺਅਖ਼໘͔Β޲͖߹͍ͨ͘ͳ͍ɺɺ
12. ͖ͬ͞ͷΞʔΩςΫνϟʹ߹Θͤͯ࡞Γ௚͢Ջͳ͍ɺɺ ʢͱ͍͏్͔த·Ͱ࡞͚ͬͨͲɺʮ͋ɺ͜Εؒʹ߹ΘΜ΍ͭ ΍ʯͬͯͳΓ·ͨ͠ʣ
13. ΋͏crawler/downloader/parserΛ෼཭ͤͣɺ ౎ಓ෎ݝ͝ͱʢαΠτʣ͝ͱʹ෼཭͢Ε͹͍͍ͷͰ͸ɺɺ
14. ෼཭͠ͳ͍ύλʔϯ +-----------------------------------------------------+ | | | +----------+ +----------+ +----------+ | | | | | | | | | | | hokkaido | | aomori | ...... | okinawa | | | | | | | | | | | +----------+ +----------+ +----------+ | | X | | X | +-------X---------------------------------------------+ X X +---------X-------------------------+ | | | crawler -> downloader -> parser | | | +-----------------------------------+
15. ͳΜ͔ ✓ 1पճͬͯૉ๿ͳΞʔΩςΫνϟʹ໭͖ͬͯͨײ͋Δ ✓ ͱ͸ݴ͑ɺ͖ͬ͞ͷΞʔΩςΫνϟΑΓ΋ಈ࡞͸γϯϓϧʹͳΔ͸ͣ ✓ ۃྗΞϓϦέʔγϣϯଆʹ৮ΒͣʹࡁΈͦ͏ ✓ ී௨ʹEC2্Ͱcronಈ͔ͯ͠΋͍͍͚Ͳɺɺ ✓ ͍·ίϯςφӡ༻ΛਐΊ͍ͯΔ ✓ EC2 Container ServiceʢECSʣͷTaskͰόον࣮ߦͰ͖Δ ✓ ECS ✓ AWS্ͰDockerίϯςφͷ࣮ߦɺ؅ཧΛߦ͏ͨΊͷϓϥοτϑΥʔϜ ✓ λεΫ ✓ ECS্ͷΞϓϦέʔγϣϯͷ୯Ґ ✓ λεΫఆٛͱݺ͹ΕΔίϯςφઃఆΛ࡞੒࣮͠ߦ͢ΔͱɺECS্Ͱίϯςφ্ཱ͕͕ͪΔ
16. AWS্ʹΫϩʔϥΛߏங̎ EC2 Container Service +-----------------------------------------------------+ | | Lambda | container1 container2 container47 | +---------+ | +----------+ +----------+ +----------+ | | | | | | | | | | | | Trigger | +---> | | hokkaido | | aomori | ...... | okinawa | | | | | | | | | | | | +---------+ | +----------+ +----------+ +----------+ | | X | | X | +-------X---------------------------------------------+ X X +---------X-------------------------+ | | | crawler -> downloader -> parser | | | +-----------------------------------+ ✓ ✓ ݩʹͳΔλεΫΛ࡞੒ ✓ Lambda͔Β౎ಓ෎ݝ͝ͱʹλεΫ࣮ߦ ✓ ࣮ߦ࣌ʹίϚϯυ্ॻ͖ ✓ php exec.php hokkaido Έ͍ͨͳײ͡
17. ॴײ ✓ ແཧ໼ཧײ͋Δ͚Ͳɺ͍͍ ✓ ౎ಓ෎ݝ͝ͱʹ෼཭͞Εɺݩʑߟ͍͑ͯͨΞʔΩςΫνϟΑΓ΋ಈ࡞͕γϯϓϧʹͳͬͨ ✓ ҰԠɺεέʔϧΞ΢τ͸Ͱ͖Δ ✓ ౎ಓ෎ݝ͕࠷খ࣮ߦ୯ҐͱͳΔ͕ ✓ ϩά΋ઃఆ࣍ୈͰऔΕΔ ✓ ίϯςφ಺ͷϩά͸ίϯςφऴྃͱಉ࣌ʹফ͑Δ ✓ ϗετଆʹϚ΢ϯτͨ͠ύεʹอଘ ✓ ಉҰΠϯελϯεʹfluentd༻taskΛ࣮ߦ͓͖ͯ͠ɺcloudwatch΁ྲྀ͢
18. ՝୊ ✓ ECS͸λεΫ࣮ߦϦΫΤετΛड͚औͬͨͱ͖Ϧιʔεʹۭ͖͕ແ͚Ε͹ɺλ εΫΛࣺͯΔͬΆ͍ ✓ auto scaleͰϦιʔεΛ࢖͍੾ΔલʹεέʔϧΞ΢τ͔ ✓ ಉҰ౎ಓ෎ݝʢαΠτʣʹରͯ͠ɺฒߦͯ͠Ϋϩʔϧͯ͠͠·͏Մೳੑ͕࢒ͬ ͍ͯΔ ✓ ϩοΫ͢Δ࢓૊Έ͸ඞཁ ✓ ౎ಓ෎ݝʢαΠτʣ୯Ґͷߴ଎Խ͕ͮ͠Β͍ߏ੒ ✓ parseͱͦΕҎ֎ͷฒྻԽΛݕ౼͍ͨ͠ͱ͜Ζ
19. ·ͱΊ ✓ AWS্ͰΫϩʔϥΛߏங͢Δͱ͖ͷϕετϓϥΫςΟεΛࢥ͍͍ͭ ͨເΛݟͨ ✓ ೲظΛकΓͭͭɺΞϓϦέʔγϣϯʹ߹ΘͤͨΞʔΩςΫνϟΛݕ ౼ͨ͠ ✓ dockerˍECS৮ָ͍ͬͯͯ͠
20. ͋Γ͕ͱ͏͍͟͝·ͨ͠