Upgrade to Pro — share decks privately, control downloads, hide ads and more …

日々大量の素材を扱うために改善したこと/webgenba-meetup-03

 日々大量の素材を扱うために改善したこと/webgenba-meetup-03

2018/03/07 Web現場Meetup #3

hamuyuuki

March 08, 2018
Tweet

More Decks by hamuyuuki

Other Decks in Technology

Transcript

  1. ೔ʑେྔͷૉࡐΛѻ͏ͨΊ ʹվળͨ͜͠ͱ 2018/03/07 Webݱ৔Meetup ϐΫελגࣜձࣾɹҏ౻ ༟ر mits / 33114786

  2. ࣗݾ঺հ • ҏ౻ ༟ر • 2015/08 ʙ ϐΫελגࣜձࣾ • 2017/01

    ʙ PIXTA VIETNAM ʹग़޲ • 2018/01 ʙ ٕज़ਪਐάϧʔϓ 
  3. ϐΫελגࣜձࣾͷ͝঺հ 

  4. ձࣾ֓ཁ • ઃཱ೔: 2005/08/25 • ཧ೦: ࠽ೳΛͭͳ͗ɺੈքΛϙδςΟϒʹ͢Δ • ࣄۀ಺༰: PIXTA,

    fotowa, Snapmart ͷӡӦ • ڌ఺: ೔ຊ, ୆࿷, λΠ, γϯΨϙʔϧ, ϕτφϜ, ؖࠃ 
  5. https://pixta.jp ೔ຊޠΛؚΉ6ݴޠͰల։த 

  6. ࠓ೔࿩͢͜ͱ • എܠ • ࣮ࢪ಺༰ • ࠓޙͷ՝୊ • ·ͱΊ 

  7. എܠ 

  8. ૉࡐ͕ PIXTA Ͱൢച͞ΕΔ·Ͱ 1. Ξοϓϩʔυ 2. ਃ੥ʢૉࡐʹλΠτϧɾλά౳Λඥ෇͚ʣ 3. ৹ࠪʢૉࡐͷΫΦϦςΟ౳Λ൑அʣ 4.

    ൢച։࢝ʂ 
  9. ૉࡐ͕ PIXTA Ͱൢച͞ΕΔ·Ͱ 1. Ξοϓϩʔυ 2. ਃ੥ʢૉࡐʹλΠτϧɾλά౳Λඥ෇͚ʣ 3. ৹ࠪʢૉࡐͷΫΦϦςΟ౳Λ൑அʣ 4.

    ൢച։࢝ʂ 
  10. Ξοϓϩʔυ • ϚΠϖʔδܦ༝ͷΞοϓϩʔυ • গྔͷૉࡐΛ PIXTA Ͱൢച͍ͨ͠ࡍʹ࢖༻ • FTPαʔόʔܦ༝ͷΞοϓϩʔυ •

    େྔͷૉࡐΛ PIXTA Ͱൢച͍ͨ͠ࡍʹ࢖༻ 
  11. Ξοϓϩʔυ • ϚΠϖʔδܦ༝ͷΞοϓϩʔυ • গྔͷૉࡐΛ PIXTA Ͱൢച͍ͨ͠ࡍʹ࢖༻ • FTPαʔόʔܦ༝ͷΞοϓϩʔυ •

    େྔͷૉࡐΛ PIXTA Ͱൢച͍ͨ͠ࡍʹ࢖༻ 
  12. FTPαʔόʔܦ༝ͷΞοϓϩʔυ • ετοΫϑΥτۀքͰ͸Ұൠతͳํ๏ • େྔͷૉࡐΛอ͍࣋ͯ͠Δ ΫϦΤΠλʔ ͕ޮ ཰Α͘ΞοϓϩʔυͰ͖ΔͨΊʹଘࡏ͢Δ  FTP

    Server ΫϦΤΠλʔ Ξοϓϩʔυ
  13. چFTPΞοϓϩʔυγεςϜ  PIXTA (Batch) S3 FTP Server ΫϦΤΠλʔ ᶃΞοϓϩʔυ PIXTA

    (Admin) ϐΫελ ελοϑ ᶄ࿈བྷʢϚΠϖʔδ্ʣ ᶅFTP=>S3ಉظґཔ ᶅFTP=>S3ಉظґཔ ᶆS3ಉظ ᶇ݁ՌΛ௨஌ ᶈαϜωΠϧੜ੒ ᶉૉࡐ৘ใΛอଘ Worker
  14. چFTPΞοϓϩʔυγεςϜ  PIXTA (Batch) S3 FTP Server ΫϦΤΠλʔ ᶃΞοϓϩʔυ PIXTA

    (Admin) ϐΫελ ελοϑ ᶄ࿈བྷʢϚΠϖʔδ্ʣ ᶅFTP=>S3ಉظґཔ ᶅFTP=>S3ಉظґཔ ᶆS3ಉظ ᶇ݁ՌΛ௨஌ ᶈαϜωΠϧੜ੒ ᶉૉࡐ৘ใΛอଘ Worker
  15. Ξοϓϩʔυ݅਺ͷਪҠ  ,  ,  ,  , 

    , ೥ ೥ ೥ ೥ ೥ ೥
  16. چFTPΞοϓϩʔυγεςϜ  PIXTA (Batch) S3 FTP Server ΫϦΤΠλʔ ᶃΞοϓϩʔυ PIXTA

    (Admin) ϐΫελ ελοϑ ᶄ࿈བྷʢϚΠϖʔδ্ʣ ᶅFTP=>S3ಉظґཔ ᶅFTP=>S3ಉظґཔ ᶆS3ಉظ ᶇ݁ՌΛ௨஌ ᶈαϜωΠϧੜ੒ ᶉૉࡐ৘ใΛอଘ Worker
  17. ໰୊఺ • ॲཧϑϩʔͷෳࡶ͞ʹΑΔӨڹ • Ξοϓϩʔυ݅਺૿Ճ܏޲ʹΑΔɺ͞ΒͳΔ γεςϜͷεέʔϧԽ 

  18. ࣮ࢪ಺༰ 

  19. ໰୊఺ • ॲཧϑϩʔͷෳࡶ͞ʹΑΔӨڹ • Ξοϓϩʔυ݅਺૿Ճ܏޲ʹΑΔɺ͞ΒͳΔ γεςϜͷεέʔϧԽ 

  20. ৽FTPΞοϓϩʔυγεςϜ  PIXTA (API) S3 FTP Server Lambda ΫϦΤΠλʔ ᶃΞοϓϩʔυ

    ᶄS3ʹಉظ ᶅΠϕϯττϦΨ ᶆαϜωΠϧੜ੒ʢಈըʣ ᶆαϜωΠϧอଘʢը૾ʣ ᶆૉࡐ৘ใΛอଘ S3 Worker
  21. ໰୊఺ • ॲཧϑϩʔͷෳࡶ͞ʹΑΔӨڹ • Ξοϓϩʔυ݅਺૿Ճ܏޲ʹΑΔɺ͞ΒͳΔ γεςϜͷεέʔϧԽ 

  22. چFTPΞοϓϩʔυγεςϜ  PIXTA (Batch) S3 FTP Server ΫϦΤΠλʔ ᶃΞοϓϩʔυ PIXTA

    (Admin) ϐΫελ ελοϑ ᶄ࿈བྷʢϚΠϖʔδ্ʣ ᶅFTP=>S3ಉظґཔ ᶅFTP=>S3ಉظґཔ ᶆS3ಉظ ᶇ݁ՌΛ௨஌ ᶈαϜωΠϧੜ੒ ᶉૉࡐ৘ใΛอଘ Worker
  23. ৽FTPΞοϓϩʔυγεςϜ  PIXTA (API) S3 FTP Server Lambda ΫϦΤΠλʔ ᶃΞοϓϩʔυ

    ᶄS3ʹಉظ ᶅΠϕϯττϦΨ ᶆαϜωΠϧੜ੒ʢಈըʣ ᶆαϜωΠϧอଘʢը૾ʣ ᶆૉࡐ৘ใΛอଘ S3 Worker
  24. ໰୊఺ • ॲཧϑϩʔͷෳࡶ͞ʹΑΔӨڹ • Ξοϓϩʔυ݅਺૿Ճ܏޲ʹΑΔɺ͞ΒͳΔ γεςϜͷεέʔϧԽ 

  25. چFTPΞοϓϩʔυγεςϜ  PIXTA (Batch) S3 FTP Server ΫϦΤΠλʔ ᶃΞοϓϩʔυ PIXTA

    (Admin) ϐΫελ ελοϑ ᶄ࿈བྷʢϚΠϖʔδ্ʣ ᶅFTP=>S3ಉظґཔ ᶅFTP=>S3ಉظґཔ ᶆS3ಉظ ᶇ݁ՌΛ௨஌ ᶈαϜωΠϧੜ੒ ᶉૉࡐ৘ใΛอଘ Worker
  26. ৽FTPΞοϓϩʔυγεςϜ  PIXTA (API) S3 FTP Server Lambda ΫϦΤΠλʔ ᶃΞοϓϩʔυ

    ᶄS3ʹಉظ ᶅΠϕϯττϦΨ ᶆαϜωΠϧੜ੒ʢಈըʣ ᶆαϜωΠϧอଘʢը૾ʣ ᶆૉࡐ৘ใΛอଘ S3 Worker
  27. AWS Lambda ͱ͸ 

  28. ৽FTPΞοϓϩʔυγεςϜ  PIXTA (API) S3 FTP Server Lambda ΫϦΤΠλʔ ᶃΞοϓϩʔυ

    ᶄS3ʹಉظ ᶅΠϕϯττϦΨ ᶆαϜωΠϧੜ੒ʢಈըʣ ᶆαϜωΠϧอଘʢը૾ʣ ᶆૉࡐ৘ใΛอଘ S3 Worker
  29. • Inotify + Ruby ͰϑΝΠϧγεςϜΛ؂ࢹ • S3ʹϑΝΠϧΛಉظ ᶄS3ʹಉظ 

  30. ৽FTPΞοϓϩʔυγεςϜ  PIXTA (API) S3 FTP Server Lambda ΫϦΤΠλʔ ᶃΞοϓϩʔυ

    ᶄS3ʹಉظ ᶅΠϕϯττϦΨ ᶆαϜωΠϧੜ੒ʢಈըʣ ᶆαϜωΠϧอଘʢը૾ʣ ᶆૉࡐ৘ใΛอଘ S3 Worker
  31. ᶅΠϕϯττϦΨ • ΠϕϯττϦΨʹ S3 Λ௥Ճ • Suffix ʹର৅ͱͳΔϑΝΠϧΛࢦఆ • ը૾ɺಈըͦΕͧΕͷ֦ுࢠ

    • Event types ʹର৅ͱͳΔΠϕϯτΛࢦఆ • ObjectCreatedͰΦϒδΣΫτ࡞੒Λର৅ 
  32. ৽FTPΞοϓϩʔυγεςϜ  PIXTA (API) S3 FTP Server Lambda ΫϦΤΠλʔ ᶃΞοϓϩʔυ

    ᶄS3ʹಉظ ᶅΠϕϯττϦΨ ᶆαϜωΠϧੜ੒ʢಈըʣ ᶆαϜωΠϧอଘʢը૾ʣ ᶆૉࡐ৘ใΛอଘ S3 Worker
  33. ᶆLambda • ը૾ϑΝΠϧʹ͍ͭͯ͸ Lambda ಺ͰαϜω ΠϧΛੜ੒ͯ͠ɺS3 ʹอଘ • ಈըϑΝΠϧʹ͍ͭͯ͸ αϜωΠϧੜ੒Λ

    SQSܦ༝Ͱґཔ • ૉࡐ৘ใΛ APIܦ༝Ͱอଘ 
  34. ᶆLambda(ը૾αϜωΠϧੜ੒) import GraphicsMagick from 'gm'; export default class JpgProcessor {

    constructor({ awsRegion, srcBucket, srcKey }) { . this.gm = GraphicsMagick.subClass({ imageMagick: true }); . } processImage() { return this._resizeThumbnail(); } _resizeThumbnail() { return this.gm(this.imageDataBody) .resize(THUMBNAIL_SIZE, THUMBNAIL_SIZE) .toBuffer(this.imageType, (err, buffer) => { . . }; } }
  35. ݁Ռ • ॲཧϑϩʔͷෳࡶ͞ʹΑΔӨڹ • ૉࡐΞοϓϩʔυΛτϦΨʹ͢Δ͜ͱͰɺॲཧϑϩʔ ΛҰํํ޲ʹͯ͠ෳࡶ͞Λͳͨ͘͠ • Ξοϓϩʔυ݅਺૿Ճ܏޲ʹΑΔɺ͞ΒͳΔγεςϜͷ εέʔϧԽ •

    ϐΫελελοϑ΁ͷ࿈བྷΛͳ͘͠ɺLambdaΛ࢖༻ ͢Δ͜ͱͰεέʔϧԽΛਤͬͨ 
  36. ࠓޙͷ՝୊ 

  37. Amazon EBS ͷϚ΢ϯτ • ౰ॳɺFTPαʔόʔ্Ͱ s3fs Λ࢖ͬͯ s3 ΛϚ΢ ϯτ͍͕ͯͨ͠ɺύϑΥʔϚϯε͕ग़ͳ͔ͬͨ

    • FTPαʔόʔ Ͱ͸ɺ1000GBͷEBSΛϚ΢ϯτ ͍ͯ͠Δ • 5୆ಈ͍͍ͯͯίετ͕ 500$ ~ 1000$ 
  38. Amazon EBS ͷϚ΢ϯτ 

  39. ·ͱΊ 

  40. ·ͱΊ • ॲཧϑϩʔͷෳࡶ͞ʹΑΔӨڹ • ૉࡐΞοϓϩʔυΛτϦΨʹ͢Δ͜ͱͰɺॲཧϑϩʔ ΛҰํํ޲ʹͯ͠ෳࡶ͞Λͳͨ͘͠ • Ξοϓϩʔυ݅਺૿Ճ܏޲ʹΑΔɺ͞ΒͳΔγεςϜͷ εέʔϧԽ •

    ϐΫελελοϑ΁ͷ࿈བྷΛͳ͘͠ɺLambdaΛ࢖༻ ͢Δ͜ͱͰεέʔϧԽΛਤͬͨ 
  41. We are hiring! 

  42. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ  mits / 33114786