Upgrade to Pro — share decks privately, control downloads, hide ads and more …

IQON Crawler BackEnd ~AWS Managed Service makes...

IQON Crawler BackEnd ~AWS Managed Service makes women beautiful~

Tatsuro Mitsuno

January 25, 2018
Tweet

More Decks by Tatsuro Mitsuno

Other Decks in Technology

Transcript

  1. Google Play ϕετΞϓϦ J04ΞϓϦ ɾ"QQ4UPSF#&45 ɾ&TTFOUJBMೝఆ "OESPJEΞϓϦ ɾ೥(PPHMF1MBZϕετΞϓϦ ɾ೥(PPHMF1MBZϕετΞϓϦ ɾ೥(PPHMF1MBZϕετΞϓϦ


    ϕετΠϊϕʔςΟϒΞϓϦେ৆ ɾ೥(PPHMF1MBZϕετΞϓϦ   σΠϦʔϔϧύʔ෦໳ೖ৆ ɾτοϓσϕϩούʔೝఆ ೥࿈ଓϕετΞϓϦड৆͸ ੈքͰ7"4*-:͚ͩ © 2018 VASILY,Inc.
  2. © 2018 VASILY,Inc. online-shot: Icons made by Vectors Market from

    www.flaticon.com is licensed by CC 3.0 BY dress: Icons made by Freepik from www.flaticon.com is licensed by CC 3.0 BY more than 300 EC sites per day more than 1 million fashion items per day Scale of crawl 
  3. © 2018 VASILY,Inc. ʙ2016 Self managed XPSLFS # XPSLFS "

    XPSLFS $ XPSLFS % XPSLFS " enq/deq lock/unlock Parallel Distributed Processing Application ( called `worker`) Reserved EC2 Instances x N sets RDS, DynamoDB, S3
  4. © 2018 VASILY,Inc. ౰࣌ͷ՝୊ ▸ ϛυϧ΢ΣΞ΁ͷෆ҆ ▸ )"ߏ੒͸ਖ਼͘͠ػೳ͢Δ͔ʁ ▸ FORVFVFEFRVFVFόϥϯε͕ཚΕΔͱ3FEJT͕NFNPSZGVMMͰམͪΔ

    ▸ อकͷෛՙ ▸ Ϩγϐʹ৐ͤΒΕͳ͍ϊ΢ϋ΢ͷଘࡏʢίϚϯυΛͲ͏ଧ͔ͭɺͳͲʣ ▸ ೔࣍ɾि࣍ɾقઅ͝ͱʹมಈ͢ΔඞཁϦιʔε΁ͷ௥ैʢϐʔΫʹ͋ΘͤΔͱແବ͕ଟ͍ʣ
  5. © 2018 VASILY,Inc. 2017ʙ AWS managed worker CFn OpsWorks RDS

    EC2 SpotFleet CloudWatch / Events Lambda SQS ElastiCache S3 DynamoDB
  6. © 2018 VASILY,Inc. 2017ʙ AWS managed worker CFn OpsWorks RDS

    EC2 SpotFleet CloudWatch / Events Lambda SQS ElastiCache S3 DynamoDB Phase.1 Phase.2 Phase.3
  7. © 2018 VASILY,Inc. ஈ֊తʹҠߦ ▸ 1IBTFΩϡʔΠϯάͷվળ   ▸ 424

    &MBTUJ$BDIF ▸ 1IBTFϦιʔε؅ཧͷվળ ʙ  ▸ &$4QPU'MFFU $MPVE8BUDI&WFOUT -BNCEB ▸ 1IBTFߏ੒؅ཧͷվળ ʙ  ▸ $MPVE'PSNBUJPO 0QT8PSLT
  8. © 2018 VASILY,Inc. 1IBTFΩϡʔΠϯάͷվળ worker CFn OpsWorks RDS EC2 SpotFleet

    CloudWatch / Events Lambda S3 DynamoDB SQS ElastiCache © 2018 VASILY,Inc.
  9. © 2018 VASILY,Inc. 1IBTFΩϡʔΠϯάͷվળ worker CFn OpsWorks RDS EC2 SpotFleet

    CloudWatch / Events Lambda S3 DynamoDB SQS ElastiCache ▸ ΫϩʔϥʔͷϩοΫͱΩϡʔΛ෼ׂ ▸ &MBTUJ$BDIF෼ࢄॲཧͷϩοΫػߏ ▸ ৴པੑͷߴ͍)"ߏ੒ ▸ ϝοηʔδΩϡʔͷ໾ׂΛ෼཭͠ɺඞཁͳ༰ྔΛݟੵ΋Γқ͘ ▸ 424ϝοηʔδɾΩϡʔ ▸ ༰ྔ໰୊͔Βղ์ ▸ ϝοηʔδ͕ෳ਺ճEFR͞Εͯ͠·͏ͱ͍͏੍໿΋ΫϩʔϧͰ͸Өڹۃখ ▸ ݒ೦͍ͯͨ͠FOREFRͷύϑΥʔϚϯεྼԽ΋໰୊ͳ͠ © 2018 VASILY,Inc.
  10. © 2018 VASILY,Inc. 1IBTFϦιʔε؅ཧͷվળ worker CFn OpsWorks RDS SQS ElastiCache

    S3 DynamoDB EC2 SpotFleet CloudWatch / Events Lambda © 2018 VASILY,Inc.
  11. © 2018 VASILY,Inc. 1IBTFϦιʔε؅ཧͷվળ worker CFn OpsWorks RDS SQS ElastiCache

    S3 DynamoDB EC2 SpotFleet CloudWatch / Events Lambda ▸ ॊೈͰ҆Ձͳܭࢉࢿݯͱͦͷ؅ཧΛಋೖ ▸ &$4QPU'MFFU ▸ ඞཁͳ࣌ʹඞཁͳ͚ͩ֬อ ▸ ऴΘͬͨΒམͱ͢ ▸ $MPVE8BUDI&WFOUT ▸ 4QPU'MFFUͷঢ়ଶΛ؂ࢹ ▸ डಈ؂ࢹͱೳಈ؂ࢹͷ྆ํ ▸ -BNCEB ▸ $MPVE8BUDI͔ΒൃՐ ▸ ΫϩʔϥʔγεςϜͷίϯτϩʔϧ © 2018 VASILY,Inc.
  12. © 2018 VASILY,Inc. 1IBTFϦιʔε؅ཧͷվળʢิ଍ʣ CFn OpsWorks RDS SQS ElastiCache S3

    DynamoDB EC2 SpotFleet CloudWatch / Events Lambda worker ▸ "QBDIF.FTPTͰ4QPU'MFFUΛந৅Խ ▸ ΞϓϦέʔγϣϯ͸%PDLFSίϯςφԽ ▸ ".*͸ࣄલʹ࡞ͬͨΧελϜ".* ▸ ͦΕͱ6TFS4DSJQU ▸ ίʔυ͸ίϯςφىಈ࣌ʹ஫ೖ © 2018 VASILY,Inc.
  13. © 2018 VASILY,Inc. 1IBTFߏ੒؅ཧͷվળ worker RDS EC2 SpotFleet CloudWatch /

    Events Lambda SQS ElastiCache S3 DynamoDB CFn OpsWorks © 2018 VASILY,Inc.
  14. © 2018 VASILY,Inc. 1IBTFߏ੒؅ཧͷվળ worker RDS EC2 SpotFleet CloudWatch /

    Events Lambda SQS ElastiCache S3 DynamoDB CFn OpsWorks ▸ ߏ੒؅ཧͷίʔυԽͱࣗಈԽ ▸ $'O $MPVE'PSNBUJPO  ▸ 0QT8PSLT ޙड़ ͱ4QPU'MFFUͷઃఆΛ:".-Ͱఆٛ ▸ 0QT8PSLT ▸ &$4QPU'MFFUͷߏ੒؅ཧ ▸ Πϯελϯε͸6TFS4DSJQUͰىಈ࣌ʹొ࿥$IFG࣮ߦ ▸ ΰʔϧσϯΠϝʔδ͕ෆཁʹ © 2018 VASILY,Inc.
  15. © 2018 VASILY,Inc. 2017ʙ AWS managed worker CFn OpsWorks RDS

    EC2 SpotFleet CloudWatch / Events Lambda SQS ElastiCache S3 DynamoDB
  16. © 2018 VASILY,Inc. ·ͱΊ ▸ *20/Ϋϩʔϥʔ͸"84ϚωʔδυɾαʔϏεΛϑϧʹ࢖ͬͯߏங͞Ε͍ͯΔ ▸ "84ϚωʔδυɾαʔϏεͷྑ͍ͱ͜Ζ ▸ ϛυϧ΢ΣΞࣗ਎ͷӡ༻͔Βղ์ʢখن໛ͳ૊৫ʹ͸ͱͯ΋ॿ͔Δʣ

    ▸ ߴ͍Մ༻ੑɾ֦ுੑ ▸ *20/͕৽઱ͳϑΝογϣϯΞΠςϜΛΫϩʔϧͰ͖Δͷ͸
 "84ϚωʔδυɾαʔϏεͷྗ͋ͬͯͦ͜ ▸ "84ϚωʔδυɾαʔϏε͸ঁੑΛΩϨΠʹ͢Δ2&%