IQON Crawler BackEnd ~AWS Managed Service makes women beautiful~

Transcript

1. © 2018 VASILY,Inc.
   *20/Ϋϩʔϥʔͷ෣୆ཪɻ
   ʙঁੑΛΩϨΠʹ͢Δ"84ϚωʔδυɾαʔϏεʙ
   X-Tech JAWS #2
   2018/01/25 Thu
   Tatsuro MITSUNO
   

   View Slide

2. © 2018 VASILY,Inc.
   ࣗݾ঺հ
   ޫ໺ୡ࿕
   גࣜձࣾ7"4*-:ΠϯϑϥΤϯδχΞ
   tatsuro.mitsuno
   ϠϑʔגࣜձࣾʹͯόοΫΤϯυΤϯδχΞͱͯ͠
   ϠϑΦΫɾ޿ࠂͷ։ൃʹܞΘΔɻ೥͔Β͸7"4*-:
   ͰΠϯϑϥΤϯδχΞͱͯ͠αʔϏεͷ৴པੑ޲্ʹ຿Ί
   Δɻ΢ΟεΩʔͱ༿ר͕޷͖ɻ
   kotatsu360
   Icon illustrated by YOSHI 

   https://ja-jp.facebook.com/yoshi.yone.7
   

   View Slide

3. ໊ࣾגࣜձࣾ7"4*-: ϰΝγϦʔ
   7"4*-: *OD
   ઃཱ೥݄
   ॴࡏ஍౦ژ౎ौ୩۠ਆٶલஸ໨੨ࢁΦʔόϧɾϏϧ'
   ैۀһ໊
   ࢿຊۚԯ

   ୅දऔక໾$&0ۚࢁ༟थ

   औక໾$50ࠓଜխ޾
   

   גओגࣜձࣾελʔττΡσΠ
   
   © 2018 VASILY,Inc.
   

   View Slide

4. © 2018 VASILY,Inc.
   Ҏ্ͷϑΝογϣϯ&$αΠτ͔Βͷ΂ສ఺Λ௒͑Δ঎඼Λܝࡌ
   ݄ؒສਓҎ্͕ར༻͢Δ೔ຊ࠷େڃͷϑΝογϣϯαΠτ
   

   View Slide

5. Google Play
   ϕετΞϓϦ
   J04ΞϓϦ
   ɾ"QQ4UPSF#&45
   ɾ&TTFOUJBMೝఆ
   "OESPJEΞϓϦ
   ɾ೥(PPHMF1MBZϕετΞϓϦ
   ɾ೥(PPHMF1MBZϕετΞϓϦ
   ɾ೥(PPHMF1MBZϕετΞϓϦ

   ϕετΠϊϕʔςΟϒΞϓϦେ৆
   ɾ೥(PPHMF1MBZϕετΞϓϦ
   σΠϦʔϔϧύʔ෦໳ೖ৆
   ɾτοϓσϕϩούʔೝఆ
   ೥࿈ଓϕετΞϓϦड৆͸
   ੈքͰ7"4*-:͚ͩ
   © 2018 VASILY,Inc.
   

   View Slide

6. © 2018 VASILY,Inc.
   ػցֶश΍ਂ૚ֶशΛ׆༻ͨ͠ϑΝογϣϯʹؔ͢Δݚڀ։ൃ
   ը૾͔ΒΞΠςϜݕग़ྨࣅը૾ݕࡧ 7"&ͱ("/Λར༻ͨ͠ϑΝογϣϯΞΠςϜݕࡧ *#*4
   
   ࠃ಺ͷ*#*4΍.*36ͳͲͷֶձʹ΋ੵۃతʹࢀՃ
   

   View Slide

7. © 2018 VASILY,Inc.
   ೥݄;0;0508/΍8&"3ΛӡӦ͢Δ
   ελʔττΡσΠάϧʔϓʹࢀը ࢠձࣾԽ
   
   

   View Slide

8. © 2018 VASILY,Inc.
   *20/Ϋϩʔϥʔͷ෣୆ཪɻ
   

   View Slide

9. © 2018 VASILY,Inc.
   Ҏ্ͷϑΝογϣϯ&$αΠτ͔Βͷ΂ສ఺Λ௒͑Δ঎඼Λܝࡌ
   ݄ؒສਓҎ্͕ར༻͢Δ೔ຊ࠷େڃͷϑΝογϣϯαΠτ
   

   View Slide

10. © 2018 VASILY,Inc.
    IQON Overview
    EC sites
    
    core part of IQON
    IQON
    Crawler
    
    

    View Slide

11. © 2018 VASILY,Inc.
    online-shot: Icons made by Vectors Market from www.flaticon.com is licensed by CC 3.0 BY
    dress: Icons made by Freepik from www.flaticon.com is licensed by CC 3.0 BY
    more than
    300 EC sites
    per day
    more than
    1 million fashion items
    per day
    Scale of crawl
    
    

    View Slide

12. © 2018 VASILY,Inc.
    ʙ2016 Self managed
    
    XPSLFS
    #
    XPSLFS
    "
    XPSLFS
    $
    XPSLFS
    %
    XPSLFS
    "
    enq/deq
    lock/unlock
    Parallel Distributed Processing Application ( called `worker`)
    Reserved
    EC2 Instances
    x N sets
    RDS, DynamoDB, S3
    

    View Slide

13. © 2018 VASILY,Inc.
    ౰࣌ͷ՝୊
    ▸ ϛυϧ΢ΣΞ΁ͷෆ҆
    ▸ )"ߏ੒͸ਖ਼͘͠ػೳ͢Δ͔ʁ
    ▸ FORVFVFEFRVFVFόϥϯε͕ཚΕΔͱ3FEJT͕NFNPSZGVMMͰམͪΔ
    ▸ อकͷෛՙ
    ▸ Ϩγϐʹ৐ͤΒΕͳ͍ϊ΢ϋ΢ͷଘࡏʢίϚϯυΛͲ͏ଧ͔ͭɺͳͲʣ
    ▸ ೔࣍ɾि࣍ɾقઅ͝ͱʹมಈ͢ΔඞཁϦιʔε΁ͷ௥ैʢϐʔΫʹ͋ΘͤΔͱແବ͕ଟ͍ʣ
    

    View Slide

14. © 2018 VASILY,Inc.
    Ϋϩʔϥʔ͕ಈ࡞͠ͳ͍
    
    ϑΝογϣϯΞΠςϜ͕ू·Βͳ͍
    
    Ϣʔβʢঁੑʣ͕ཉ͍͠෺Λݟ͚ͭΒΕͳ͍ɾߪೖͰ͖ͳ͍
    
    ঁੑ͕ΩϨΠʹͳΔΩοΧέ͕ݮΔ
    

    View Slide

15. © 2018 VASILY,Inc.
    Ϋϩʔϥʔ͕͍͍ײ͡ʹಈ͘
    
    ϑΝογϣϯΞΠςϜ͕ू·Δ
    
    Ϣʔβʢঁੑʣ͸ཉ͍͠෺͕ݟ͔ͭΔɾߪೖͰ͖Δ
    
    ੈͷঁੑ͸΋ͬͱΩϨΠʹͳΔ
    "84ϚωʔδυαʔϏεΛར༻࣮ͯ͠ݱ
    

    View Slide

16. © 2018 VASILY,Inc.
    2017ʙ AWS managed
    
    worker
    CFn OpsWorks
    RDS
    EC2 SpotFleet
    CloudWatch / Events
    Lambda
    SQS
    ElastiCache
    S3
    DynamoDB
    

    View Slide

17. © 2018 VASILY,Inc.
    2017ʙ AWS managed
    
    worker
    CFn OpsWorks
    RDS
    EC2 SpotFleet
    CloudWatch / Events
    Lambda
    SQS
    ElastiCache
    S3
    DynamoDB
    Phase.1
    Phase.2
    Phase.3
    

    View Slide

18. © 2018 VASILY,Inc.
    ஈ֊తʹҠߦ
    ▸ 1IBTFΩϡʔΠϯάͷվળ
    
    ▸ 424 &MBTUJ$BDIF
    ▸ 1IBTFϦιʔε؅ཧͷվળ ʙ
    
    ▸ &$4QPU'MFFU $MPVE8BUDI&WFOUT -BNCEB
    ▸ 1IBTFߏ੒؅ཧͷվળ ʙ
    
    ▸ $MPVE'PSNBUJPO 0QT8PSLT
    

    View Slide

19. © 2018 VASILY,Inc.
    1IBTFΩϡʔΠϯάͷվળ
    
    worker
    CFn OpsWorks
    RDS
    EC2 SpotFleet
    CloudWatch / Events
    Lambda
    S3
    DynamoDB
    SQS
    ElastiCache
    © 2018 VASILY,Inc.
    

    View Slide

20. © 2018 VASILY,Inc.
    1IBTFΩϡʔΠϯάͷվળ
    
    worker
    CFn OpsWorks
    RDS
    EC2 SpotFleet
    CloudWatch / Events
    Lambda
    S3
    DynamoDB
    SQS
    ElastiCache
    ▸ ΫϩʔϥʔͷϩοΫͱΩϡʔΛ෼ׂ
    ▸ &MBTUJ$BDIF෼ࢄॲཧͷϩοΫػߏ
    ▸ ৴པੑͷߴ͍)"ߏ੒
    ▸ ϝοηʔδΩϡʔͷ໾ׂΛ෼཭͠ɺඞཁͳ༰ྔΛݟੵ΋Γқ͘
    ▸ 424ϝοηʔδɾΩϡʔ
    ▸ ༰ྔ໰୊͔Βղ์
    ▸ ϝοηʔδ͕ෳ਺ճEFR͞Εͯ͠·͏ͱ͍͏੍໿΋ΫϩʔϧͰ͸Өڹۃখ
    ▸ ݒ೦͍ͯͨ͠FOREFRͷύϑΥʔϚϯεྼԽ΋໰୊ͳ͠
    © 2018 VASILY,Inc.
    

    View Slide

21. © 2018 VASILY,Inc.
    1IBTFϦιʔε؅ཧͷվળ
    
    worker
    CFn OpsWorks
    RDS
    SQS
    ElastiCache
    S3
    DynamoDB
    EC2 SpotFleet
    CloudWatch / Events
    Lambda
    © 2018 VASILY,Inc.
    

    View Slide

22. © 2018 VASILY,Inc.
    1IBTFϦιʔε؅ཧͷվળ
    
    worker
    CFn OpsWorks
    RDS
    SQS
    ElastiCache
    S3
    DynamoDB
    EC2 SpotFleet
    CloudWatch / Events
    Lambda
    ▸ ॊೈͰ҆Ձͳܭࢉࢿݯͱͦͷ؅ཧΛಋೖ
    ▸ &$4QPU'MFFU
    ▸ ඞཁͳ࣌ʹඞཁͳ͚ͩ֬อ
    ▸ ऴΘͬͨΒམͱ͢
    ▸ $MPVE8BUDI&WFOUT
    ▸ 4QPU'MFFUͷঢ়ଶΛ؂ࢹ
    ▸ डಈ؂ࢹͱೳಈ؂ࢹͷ྆ํ
    ▸ -BNCEB
    ▸ $MPVE8BUDI͔ΒൃՐ
    ▸ ΫϩʔϥʔγεςϜͷίϯτϩʔϧ
    © 2018 VASILY,Inc.
    

    View Slide

23. © 2018 VASILY,Inc.
    1IBTFϦιʔε؅ཧͷվળʢิ଍ʣ
    
    CFn OpsWorks
    RDS
    SQS
    ElastiCache
    S3
    DynamoDB
    EC2 SpotFleet
    CloudWatch / Events
    Lambda
    worker
    ▸ "QBDIF.FTPTͰ4QPU'MFFUΛந৅Խ
    ▸ ΞϓϦέʔγϣϯ͸%PDLFSίϯςφԽ
    ▸ ".*͸ࣄલʹ࡞ͬͨΧελϜ".*
    ▸ ͦΕͱ6TFS4DSJQU
    ▸ ίʔυ͸ίϯςφىಈ࣌ʹ஫ೖ
    © 2018 VASILY,Inc.
    

    View Slide

24. © 2018 VASILY,Inc.
    1IBTFߏ੒؅ཧͷվળ
    
    worker
    RDS
    EC2 SpotFleet
    CloudWatch / Events
    Lambda
    SQS
    ElastiCache
    S3
    DynamoDB
    CFn OpsWorks
    © 2018 VASILY,Inc.
    

    View Slide

25. © 2018 VASILY,Inc.
    1IBTFߏ੒؅ཧͷվળ
    
    worker
    RDS
    EC2 SpotFleet
    CloudWatch / Events
    Lambda
    SQS
    ElastiCache
    S3
    DynamoDB
    CFn OpsWorks
    ▸ ߏ੒؅ཧͷίʔυԽͱࣗಈԽ
    ▸ $'O $MPVE'PSNBUJPO
    
    ▸ 0QT8PSLT ޙड़
    ͱ4QPU'MFFUͷઃఆΛ:".-Ͱఆٛ
    ▸ 0QT8PSLT
    ▸ &$4QPU'MFFUͷߏ੒؅ཧ
    ▸ Πϯελϯε͸6TFS4DSJQUͰىಈ࣌ʹొ࿥$IFG࣮ߦ
    ▸ ΰʔϧσϯΠϝʔδ͕ෆཁʹ
    © 2018 VASILY,Inc.
    

    View Slide

26. © 2018 VASILY,Inc.
    2017ʙ AWS managed
    
    worker
    CFn OpsWorks
    RDS
    EC2 SpotFleet
    CloudWatch / Events
    Lambda
    SQS
    ElastiCache
    S3
    DynamoDB
    

    View Slide

27. © 2018 VASILY,Inc.
    ·ͱΊ
    ▸ *20/Ϋϩʔϥʔ͸"84ϚωʔδυɾαʔϏεΛϑϧʹ࢖ͬͯߏங͞Ε͍ͯΔ
    ▸ "84ϚωʔδυɾαʔϏεͷྑ͍ͱ͜Ζ
    ▸ ϛυϧ΢ΣΞࣗ਎ͷӡ༻͔Βղ์ʢখن໛ͳ૊৫ʹ͸ͱͯ΋ॿ͔Δʣ
    ▸ ߴ͍Մ༻ੑɾ֦ுੑ
    ▸ *20/͕৽઱ͳϑΝογϣϯΞΠςϜΛΫϩʔϧͰ͖Δͷ͸

    "84ϚωʔδυɾαʔϏεͷྗ͋ͬͯͦ͜
    ▸ "84ϚωʔδυɾαʔϏε͸ঁੑΛΩϨΠʹ͢Δ2&%
    

    View Slide

28. © 2018 VASILY,Inc.
    We are Hiring
    https://www.wantedly.com/companies/vasily
    

    View Slide