日々大量の素材を扱うために改善したこと/webgenba-meetup-03

by hamuyuuki

Slide 1

Slide 1 text

೔ʑେྔͷૉࡐΛѻ͏ͨΊ ʹվળͨ͜͠ͱ 2018/03/07 Webݱ৔Meetup ϐΫελגࣜձࣾɹҏ౻ ༟ر mits / 33114786

Slide 2

Slide 2 text

ࣗݾ঺հ • ҏ౻ ༟ر • 2015/08 ʙ ϐΫελגࣜձࣾ • 2017/01 ʙ PIXTA VIETNAM ʹग़޲ • 2018/01 ʙ ٕज़ਪਐάϧʔϓ

Slide 3

Slide 3 text

ϐΫελגࣜձࣾͷ͝঺հ

Slide 4

Slide 4 text

ձࣾ֓ཁ • ઃཱ೔: 2005/08/25 • ཧ೦: ࠽ೳΛͭͳ͗ɺੈքΛϙδςΟϒʹ͢Δ • ࣄۀ಺༰: PIXTA, fotowa, Snapmart ͷӡӦ • ڌ఺: ೔ຊ, ୆࿷, λΠ, γϯΨϙʔϧ, ϕτφϜ, ؖࠃ

Slide 5

Slide 5 text

https://pixta.jp ೔ຊޠΛؚΉ6ݴޠͰల։த

Slide 6

Slide 6 text

ࠓ೔࿩͢͜ͱ • എܠ • ࣮ࢪ಺༰ • ࠓޙͷ՝୊ • ·ͱΊ

Slide 7

Slide 7 text

എܠ

Slide 8

Slide 8 text

ૉࡐ͕ PIXTA Ͱൢച͞ΕΔ·Ͱ 1. Ξοϓϩʔυ 2. ਃ੥ʢૉࡐʹλΠτϧɾλά౳Λඥ෇͚ʣ 3. ৹ࠪʢૉࡐͷΫΦϦςΟ౳Λ൑அʣ 4. ൢച։࢝ʂ

Slide 9

Slide 9 text

ૉࡐ͕ PIXTA Ͱൢച͞ΕΔ·Ͱ 1. Ξοϓϩʔυ 2. ਃ੥ʢૉࡐʹλΠτϧɾλά౳Λඥ෇͚ʣ 3. ৹ࠪʢૉࡐͷΫΦϦςΟ౳Λ൑அʣ 4. ൢച։࢝ʂ

Slide 10

Slide 10 text

Ξοϓϩʔυ • ϚΠϖʔδܦ༝ͷΞοϓϩʔυ • গྔͷૉࡐΛ PIXTA Ͱൢച͍ͨ͠ࡍʹ࢖༻ • FTPαʔόʔܦ༝ͷΞοϓϩʔυ • େྔͷૉࡐΛ PIXTA Ͱൢച͍ͨ͠ࡍʹ࢖༻

Slide 11

Slide 11 text

Ξοϓϩʔυ • ϚΠϖʔδܦ༝ͷΞοϓϩʔυ • গྔͷૉࡐΛ PIXTA Ͱൢച͍ͨ͠ࡍʹ࢖༻ • FTPαʔόʔܦ༝ͷΞοϓϩʔυ • େྔͷૉࡐΛ PIXTA Ͱൢച͍ͨ͠ࡍʹ࢖༻

Slide 12

Slide 12 text

FTPαʔόʔܦ༝ͷΞοϓϩʔυ • ετοΫϑΥτۀքͰ͸Ұൠతͳํ๏ • େྔͷૉࡐΛอ͍࣋ͯ͠Δ ΫϦΤΠλʔ ͕ޮ ཰Α͘ΞοϓϩʔυͰ͖ΔͨΊʹଘࡏ͢Δ FTP Server ΫϦΤΠλʔ Ξοϓϩʔυ

Slide 13

Slide 13 text

چFTPΞοϓϩʔυγεςϜ PIXTA (Batch) S3 FTP Server ΫϦΤΠλʔ ᶃΞοϓϩʔυ PIXTA (Admin) ϐΫελ ελοϑ ᶄ࿈བྷʢϚΠϖʔδ্ʣ ᶅFTP=>S3ಉظґཔ ᶅFTP=>S3ಉظґཔ ᶆS3ಉظ ᶇ݁ՌΛ௨஌ ᶈαϜωΠϧੜ੒ ᶉૉࡐ৘ใΛอଘ Worker

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Ξοϓϩʔυ݅਺ͷਪҠ , , , , , ೥ ೥ ೥ ೥ ೥ ೥

Slide 16

Slide 16 text

Slide 17

Slide 17 text

໰୊఺ • ॲཧϑϩʔͷෳࡶ͞ʹΑΔӨڹ • Ξοϓϩʔυ݅਺૿Ճ܏޲ʹΑΔɺ͞ΒͳΔ γεςϜͷεέʔϧԽ

Slide 18

Slide 18 text

࣮ࢪ಺༰

Slide 19

Slide 19 text

໰୊఺ • ॲཧϑϩʔͷෳࡶ͞ʹΑΔӨڹ • Ξοϓϩʔυ݅਺૿Ճ܏޲ʹΑΔɺ͞ΒͳΔ γεςϜͷεέʔϧԽ

Slide 20

Slide 20 text

৽FTPΞοϓϩʔυγεςϜ PIXTA (API) S3 FTP Server Lambda ΫϦΤΠλʔ ᶃΞοϓϩʔυ ᶄS3ʹಉظ ᶅΠϕϯττϦΨ ᶆαϜωΠϧੜ੒ʢಈըʣ ᶆαϜωΠϧอଘʢը૾ʣ ᶆૉࡐ৘ใΛอଘ S3 Worker

Slide 21

Slide 21 text

໰୊఺ • ॲཧϑϩʔͷෳࡶ͞ʹΑΔӨڹ • Ξοϓϩʔυ݅਺૿Ճ܏޲ʹΑΔɺ͞ΒͳΔ γεςϜͷεέʔϧԽ

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Slide 24

Slide 24 text

໰୊఺ • ॲཧϑϩʔͷෳࡶ͞ʹΑΔӨڹ • Ξοϓϩʔυ݅਺૿Ճ܏޲ʹΑΔɺ͞ΒͳΔ γεςϜͷεέʔϧԽ

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Slide 27

Slide 27 text

AWS Lambda ͱ͸

Slide 28

Slide 28 text

Slide 29

Slide 29 text

• Inotify + Ruby ͰϑΝΠϧγεςϜΛ؂ࢹ • S3ʹϑΝΠϧΛಉظ ᶄS3ʹಉظ

Slide 30

Slide 30 text

Slide 31

Slide 31 text

ᶅΠϕϯττϦΨ • ΠϕϯττϦΨʹ S3 Λ௥Ճ • Sufﬁx ʹର৅ͱͳΔϑΝΠϧΛࢦఆ • ը૾ɺಈըͦΕͧΕͷ֦ுࢠ • Event types ʹର৅ͱͳΔΠϕϯτΛࢦఆ • ObjectCreatedͰΦϒδΣΫτ࡞੒Λର৅

Slide 32

Slide 32 text

Slide 33

Slide 33 text

ᶆLambda • ը૾ϑΝΠϧʹ͍ͭͯ͸ Lambda ಺ͰαϜω ΠϧΛੜ੒ͯ͠ɺS3 ʹอଘ • ಈըϑΝΠϧʹ͍ͭͯ͸ αϜωΠϧੜ੒Λ SQSܦ༝Ͱґཔ • ૉࡐ৘ใΛ APIܦ༝Ͱอଘ

Slide 34

Slide 34 text

ᶆLambda(ը૾αϜωΠϧੜ੒) import GraphicsMagick from 'gm'; export default class JpgProcessor { constructor({ awsRegion, srcBucket, srcKey }) { . this.gm = GraphicsMagick.subClass({ imageMagick: true }); . } processImage() { return this._resizeThumbnail(); } _resizeThumbnail() { return this.gm(this.imageDataBody) .resize(THUMBNAIL_SIZE, THUMBNAIL_SIZE) .toBuffer(this.imageType, (err, buffer) => { . . }; } }

Slide 35

Slide 35 text

݁Ռ • ॲཧϑϩʔͷෳࡶ͞ʹΑΔӨڹ • ૉࡐΞοϓϩʔυΛτϦΨʹ͢Δ͜ͱͰɺॲཧϑϩʔ ΛҰํํ޲ʹͯ͠ෳࡶ͞Λͳͨ͘͠ • Ξοϓϩʔυ݅਺૿Ճ܏޲ʹΑΔɺ͞ΒͳΔγεςϜͷ εέʔϧԽ • ϐΫελελοϑ΁ͷ࿈བྷΛͳ͘͠ɺLambdaΛ࢖༻ ͢Δ͜ͱͰεέʔϧԽΛਤͬͨ

Slide 36

Slide 36 text

ࠓޙͷ՝୊

Slide 37

Slide 37 text

Amazon EBS ͷϚ΢ϯτ • ౰ॳɺFTPαʔόʔ্Ͱ s3fs Λ࢖ͬͯ s3 ΛϚ΢ ϯτ͍͕ͯͨ͠ɺύϑΥʔϚϯε͕ग़ͳ͔ͬͨ • FTPαʔόʔ Ͱ͸ɺ1000GBͷEBSΛϚ΢ϯτ ͍ͯ͠Δ • 5୆ಈ͍͍ͯͯίετ͕ 500$ ~ 1000$

Slide 38

Slide 38 text

Amazon EBS ͷϚ΢ϯτ

Slide 39

Slide 39 text

·ͱΊ

Slide 40

Slide 40 text

·ͱΊ • ॲཧϑϩʔͷෳࡶ͞ʹΑΔӨڹ • ૉࡐΞοϓϩʔυΛτϦΨʹ͢Δ͜ͱͰɺॲཧϑϩʔ ΛҰํํ޲ʹͯ͠ෳࡶ͞Λͳͨ͘͠ • Ξοϓϩʔυ݅਺૿Ճ܏޲ʹΑΔɺ͞ΒͳΔγεςϜͷ εέʔϧԽ • ϐΫελελοϑ΁ͷ࿈བྷΛͳ͘͠ɺLambdaΛ࢖༻ ͢Δ͜ͱͰεέʔϧԽΛਤͬͨ