Upgrade to Pro — share decks privately, control downloads, hide ads and more …

1人月で50万件以上のデータをクロールして検索可能にした話

 1人月で50万件以上のデータをクロールして検索可能にした話

2018年6月1日に行われたAWS主催の「Startup Archiitecture of the Year 2018」にてMESON COOの小林が発表した自社アーキテクチャに関する発表内容です。
#AWS #AWSSummit #Fargate

AR_Ojisan

June 01, 2018
Tweet

More Decks by AR_Ojisan

Other Decks in Technology

Transcript

  1. 1ਓ݄Ͱ50ສ݅Ҏ্ͷσʔλΛ
    Ϋϩʔϧͯ͠ݕࡧՄೳʹͨ͠࿩
    MESON COO
    খྛ༎थ

    View full-size slide

  2. খྛ༎थʢAR͓͡͞Μʣ
    גࣜձࣾMESON COO
    ޷͖ͳαʔϏεɿFargate
    @AR_Ojisan

    View full-size slide

  3. heymesh: Search engine for 3D models
    13αΠτ, 50ສ݅Λ௒͑Δ3DϞσϧσʔλΛԣஅݕࡧ

    View full-size slide

  4. heymesh: Search engine for 3D models
    ϥΠηϯε΍Ձ֨ɺεϖοΫͰͷϑΟϧλϦϯάɹ

    View full-size slide

  5. heymeshΛࢧ͑ΔAWSαʔϏε
    Fargate
    Elasticsearch
    Service
    DynamoDB NLB Lambda

    View full-size slide

  6. git push
    ΤϯδχΞ
    Ϣʔβʔ
    heymesh όοΫΤϯυΞʔΩςΫνϟʔ

    View full-size slide

  7. git push
    ΤϯδχΞ
    Ϣʔβʔ
    σʔλऩू

    View full-size slide

  8. Scrapy Cloud
    git push
    ΤϯδχΞ
    Ϣʔβʔ
    ݕࡧΞϓϦέʔγϣϯ

    View full-size slide

  9. 1ਓͰ̍ϲ݄Ͱ࡞Γ͖ΔͨΊʹ
    ҙࣝͨ̏ͭ͠ͷϙΠϯτ

    View full-size slide

  10. 1. పఈతʹϦιʔε؅ཧ͠ͳ͍

    View full-size slide

  11. git push
    ΤϯδχΞ
    Ϣʔβʔ
    ϚωʔδυˍαʔόʔϨεαʔϏε

    View full-size slide

  12. ΤϯδχΞ࠾༻͕
    ϓϩμΫτ։ൃ࠷େͷ೉ؔ

    View full-size slide

  13. ਓ͸ޏ͑ͳ͍
    ͚Ͳɺૣ͘։ൃ͠ͳ͖Ό͍͚ͳ͍ɾɾɾ

    View full-size slide

  14. ϚωʔδυˍαʔόʔϨεαʔϏεΛ׆༻
    ຊ౰ʹ։ൃ͕ඞཁͳ෦෼ʹ͚ͩूதͰ͖Δ

    View full-size slide

  15. 2. AWSαʔϏε୯ҐͰͷ
    ॲཧͷ෼ׂ

    View full-size slide

  16. git push
    ΤϯδχΞ
    Ϣʔβʔ
    Ϋϩʔϧॲཧ
    εΫϨΠϐϯάॲཧ
    σʔλ੔ܗॲཧ
    ಠࣗ఺਺ܭࢉॲཧ
    SQSΛ࢖ͬͨύΠϓϥΠϯॲཧ

    View full-size slide

  17. ϓϩμΫτͷ։ൃॳظ͸
    ໰୊༧ଌ͕ͮ͠Β͍

    View full-size slide

  18. git push
    ΤϯδχΞ
    Ϣʔβʔ
    ࣮ࡍʹى͖ͨࣄ৅
    ॳظ૝ఆΑΓσʔλ੔ܗॲཧ͕ෳࡶԽ
    ϘτϧωοΫʹͳͬͯ͠·ͬͨ

    View full-size slide

  19. git push
    ΤϯδχΞ
    Ϣʔβʔ
    ॳظ૝ఆΑΓσʔλ੔ܗॲཧ͕ෳࡶԽ
    ϘτϧωοΫʹͳͬͯ͠·ͬͨ
    ΑΓ௕͍ॲཧʹॊೈʹରԠͰ͖Δ
    FargateʹॲཧΛҠߦ
    ࣮ࡍʹى͖ͨࣄ৅

    View full-size slide

  20. AWSαʔϏε୯ҐͰͷૄ݁߹ͳॲཧ෼ׂ
    ֤AWSαʔϏεͷڧΈΛ࠷େݶ׆༻

    View full-size slide

  21. AWSαʔϏε୯ҐͰͷૄ݁߹ͳॲཧ෼ׂ
    ໰୊ൃੜ࣌ʹ΋มߋൣғΛ໌͔֬ͭ࠷খݶʹ

    View full-size slide

  22. 3. άϩʔόϧج४ͷٕज़બఆ

    View full-size slide

  23. heymesh͸શੈքͷϢʔβʔ͕λʔήοτ

    View full-size slide

  24. ٕज़બఆ΋άϩʔόϧલఏͰͳ͍ͱ
    ੈքͷελʔτΞοϓͱઓ͑ͳ͍

    View full-size slide

  25. git push
    ΤϯδχΞ
    Ϣʔβʔ
    ଟ਺ͷՕॴͰFargateΛར༻

    View full-size slide

  26. git push
    ΤϯδχΞ
    Ϣʔβʔ
    ଟ਺ͷՕॴͰFargateΛར༻
    ΞʔΩςΫνϟ͸શͯ
    όʔδχΞϦʔδϣϯͰߏங

    View full-size slide

  27. άϩʔόϧج४ͷٕज़બఆ
    ศརͳ৽αʔϏεΛҰૣ͘ར༻Մೳ

    View full-size slide

  28. άϩʔόϧج४ͷٕज़બఆ
    άϩʔόϧͰ΋ઓ͑ΔελʔτΞοϓʹ

    View full-size slide

  29. ελʔτΞοϓ͸ͱʹ͔͘ૣ͘
    αʔϏεΛϦϦʔε͢Δ͜ͱ͕େࣄ

    View full-size slide

  30. ̏ͭͷϙΠϯτΛҙࣝ͢Δ͜ͱͰ
    গͳ͍։ൃϦιʔεͰਝ଎ͳ։ൃ͕Մೳ

    View full-size slide

  31. ͝੩ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ

    View full-size slide