On-demand image server with Python

Slide 1

Slide 1 text

On-demand image server with Python MinJae Kwon (@mingrammer) 2017.05.27 Python Seminar

Slide 2

Slide 2 text

Name ӂ޹੤ (MinJae Kwon) Nickname @mingrammer Email [email protected] Who ই઱؀೟Ү ࣗ೐౟ਝয೟җ 3೟֙ / Backend Developer Blog https://mingrammer.com Facebook https://facebook.com/mingrammer Github https://github.com/mingrammer Eng Blog https://medium.com/@mingrammer

Slide 3

Slide 3 text

2. ѐߊ زӝ 5. दաܻয় 3. рױೠ ইఃఫ୊ Contents 6. Next & Future 1. ৡ٣ݔ٘ ੉޷૑ ࢲߡ? 4. ݽٕ ҳഅ୓

Slide 4

Slide 4 text

ৡ٣ݔ٘ ੉޷૑ ࢲߡ?

Slide 5

Slide 5 text

ৡ٣ݔ٘ ੉޷૑ ࢲߡ? Image Server image resized thumbnail rotated

Slide 6

Slide 6 text

ৡ٣ݔ٘ ੉޷૑ ࢲߡ? Image Server image resized thumbnail rotated User Service desired parameters new image

Slide 7

Slide 7 text

ৡ٣ݔ٘ ੉޷૑ ࢲߡ? ੉޷૑ܳ ޷ܻ ࢤࢿ೧ف૑ ঋইب ؽ ਗೡٸ ׮নೠ ഋక੄ ੉޷૑ܳ ৡ٣ݔ٘۽ ࢤࢿ оמ ੢੼

Slide 8

Slide 8 text

ৡ٣ݔ٘ ੉޷૑ ࢲߡ? ੉޷૑ܳ ࢎਊೞח Ѣ੄ ݽٚҔ ׮নೠ ࢎ੉ૉ੄ ੉޷૑о ೙ਃೠ ݽ߄ੌ জ ੉޷૑ ӝ߈ ࢲ࠺झ ੸ਊ оמೠ ࢲ࠺झ

Slide 9

Slide 9 text

ѐߊ زӝ

Slide 10

Slide 10 text

ѐߊ زӝ 1. ഥࢎীࢲ ٣੗੉ցо ੉޷ ٜ݅য֬਷ ੉޷૑ ࢎ੉ૉ੄ ߸҃੉ ೙ਃ 2. ੉޷૑ ਗࠄਸ о૓ ٣੗੉ցীѱ ࢎ੉ૉ ߸҃ ਃ୒੉ ೙ਃ 3. рױೠ ੘সੋ Ѫী ࠺೧ ࣗਊغח ࠺ਊ੉ ֫਺ 4. ٜ݅੗!

Slide 11

Slide 11 text

ѐߊ زӝ ߈ա੺݅ী ٜ݅Ҋ ׮਺ ಁ஖ٸ ੸ਊ ੉޷૑ ܻࢎ੉૚ ੗زച۽ ੋ೧ ࢕੉ ખ ؊ աই૗

Slide 12

Slide 12 text

ѐߊ زӝ ׼द ഥࢎীࢲ ೙ਃೠ ੉޷૑ ܻࢎ੉૚ ӝמ਷ ੉޷૑ ࠺ਯ݅ ਬ૑೧ࢲ “width”݅ ઑ੺ೞҊ நयೞח ӝמ ٯ ೞա But

Slide 13

Slide 13 text

ѐߊ زӝ ખ ؊ ਬোೠ য়೑ࣗझ ৡ٣ݔ٘ ੉޷૑ ࢲߡܳ ٜ݅যࠁ੗ https://github.com/mingrammer/ﬁtter

Slide 14

Slide 14 text

рױೠ ইఃఫ୊

Slide 15

Slide 15 text

рױೠ ইఃఫ୊ API Server API server for ﬁtter

Slide 16

Slide 16 text

рױೠ ইఃఫ୊ API Server Configuration Parser Fitter Configuration Parse the customizable configuration

Slide 17

Slide 17 text

рױೠ ইఃఫ୊ API Server Conﬁguration Parser Fitter Conﬁguration Storage Manage storage

Slide 18

Slide 18 text

рױೠ ইఃఫ୊ API Server Conﬁguration Parser Fitter Conﬁguration Storage Supports source/store storages Source Store

Slide 19

Slide 19 text

рױೠ ইఃఫ୊ API Server Configuration Parser Fitter Configuration Storage Supports file system and s3 Source Store s3 fs

Slide 20

Slide 20 text

рױೠ ইఃఫ୊ API Server Conﬁguration Parser Fitter Conﬁguration Storage Source Store s3 fs Engine Processing the transforming

Slide 21

Slide 21 text

рױೠ ইఃఫ୊ API Server Conﬁguration Parser Fitter Conﬁguration Storage Source Store s3 fs Engine Thumbnail Flip Resize Rotate Various mode

Slide 22

Slide 22 text

рױೠ ইఃఫ୊ API Server Conﬁguration Parser Fitter Conﬁguration Storage Source Store s3 fs Engine Thumbnail Flip Resize Rotate Cache Server Caching the generated image

Slide 23

Slide 23 text

рױೠ ইఃఫ୊ API Server Conﬁguration Parser Fitter Conﬁguration Storage Source Store s3 fs Engine Thumbnail Flip Resize Rotate Cache Server Cache Store Manage the cache store

Slide 24

Slide 24 text

рױೠ ইఃఫ୊ API Server Conﬁguration Parser Fitter Conﬁguration Storage Source Store s3 fs Engine Thumbnail Flip Resize Rotate Cache Server Cache Store in-memory redis Supports in-memory and redis

Slide 25

Slide 25 text

рױೠ ইఃఫ୊ API Server Conﬁguration Parser Fitter Conﬁguration Storage Source Store s3 fs Engine Thumbnail Flip Resize Rotate Cache Server Cache Store in-memory redis Paramter Validator Validate the parameters

Slide 26

Slide 26 text

рױೠ ইఃఫ୊ API Server Conﬁguration Parser Storage Engine Cache Server Cache Store Paramter Validator

Slide 27

Slide 27 text

рױೠ ইఃఫ୊ API Server ﬂask Conﬁguration Parser pyyaml Storage boto3 requests Engine wand Cache Server Cache Store redis-py Paramter Validator Python Fullstack

Slide 28

Slide 28 text

ݽٕ ҳഅ୓

Slide 29

Slide 29 text

ݽٕ ҳഅ୓ ۨನ૑షܻ ҳࢿ

Slide 30

Slide 30 text

ݽٕ ҳഅ୓ API Server Optional URL for Flask? Pre-registered Not pre-registered

Slide 31

Slide 31 text

ݽٕ ҳഅ୓ API Server Optional URL for Flask?

Slide 32

Slide 32 text

ݽٕ ҳഅ୓ Conﬁg parsing - Using common classes

Slide 33

Slide 33 text

ݽٕ ҳഅ୓ Conﬁg parsing Load Conﬁg Parsing & Validating Check Cache Show errors invalid or failed valid

Slide 34

Slide 34 text

ݽٕ ҳഅ୓ Conﬁg parsing - Load and validate the required ﬁelds

Slide 35

Slide 35 text

ݽٕ ҳഅ୓ Config parsing - Load and validate the required fields and set the configs

Slide 36

Slide 36 text

ݽٕ ҳഅ୓ Conﬁg parsing - Load and validate the required ﬁelds

Slide 37

Slide 37 text

ݽٕ ҳഅ୓ Params validator Request Validate Params Handling Show errors invalid valid

Slide 38

Slide 38 text

ݽٕ ҳഅ୓ Params validator - Validate all params and restriction of each mode

Slide 39

Slide 39 text

ݽٕ ҳഅ୓ Params validator - Validate the all params before view handling using decorator

Slide 40

Slide 40 text

Double caching? ݽٕ ҳഅ୓ Check Cache Server Check Store Storage Generate Use cached Use cached exists exists not not

Slide 41

Slide 41 text

Double caching? - Use decorators!! Decorator is love ݽٕ ҳഅ୓

Slide 42

Slide 42 text

Double caching? - Use decorators!! Decorator is love ݽٕ ҳഅ୓

Slide 43

Slide 43 text

How to use different classes on same function? ݽٕ ҳഅ୓ Stroage S3 File System Cache Store Source Store Source Store In memory Redis class … class … class … class … class … class …

Slide 44

Slide 44 text

How to use different classes on same function? - Use Interface-like class ݽٕ ҳഅ୓

Slide 45

Slide 45 text

How to use different classes on same function? - Use Interface-like class ݽٕ ҳഅ୓

Slide 46

Slide 46 text

How to use different classes on same function? - Use Interface-like class ݽٕ ҳഅ୓ Two types are totally different but they have same interfaces!

Slide 47

Slide 47 text

There is very-simple engine - Just wrapping the each mode. Common interface :D ݽٕ ҳഅ୓

Slide 48

Slide 48 text

We are the world - Combine these modules! ݽٕ ҳഅ୓ Validate Params Cache Interface Storage Interface Storage Interface Parsing Conﬁg Transform Interface Check Cache Store Check Storage Transform Engine Handle Storage

Slide 49

Slide 49 text

दաܻয়

Slide 50

Slide 50 text

दաܻয় 1. ۽ஸ ӝ߈ ੉޷૑ ߸ജ ో 2. ੉޷૑ ߸ജ ߂ ௿ۄ਋٘ ߔস ో 3. ߔূ٘ ৡ٣ݔ٘ ੉޷૑ ࢲߡ

Slide 51

Slide 51 text

दաܻয় 1. ۽ஸ ӝ߈ ੉޷૑ ߸ജ ో 2. ੉޷૑ ߸ജ ߂ ௿ۄ਋٘ ߔস ో 3. ߔূ٘ ৡ٣ݔ٘ ੉޷૑ ࢲߡ

Slide 52

Slide 52 text

दաܻয় நद ࢲߡ ࢸ੿ झషܻ૑ ࢸ੿ ২࣌ ࢸ੿

Slide 53

Slide 53 text

दաܻয় ﬁtter ࢲߡ ҳز

Slide 54

Slide 54 text

दաܻয় ੉޷૑ ߸ജ ਃ୒ Wallpaper/squares_widew.png ੉޷૑ܳ 200 x 300ਵ۽ ߸ജ resize

Slide 55

Slide 55 text

दաܻয় What is ‘get’? show : Retrieve the image information asynchronously (not yet)  get : Retrieve the image information synchronously redirect : Redirect to url of transformed image

Slide 56

Slide 56 text

दաܻয় ਃ୒ റ ߈ജч

Slide 57

Slide 57 text

दաܻয় நद ࢲߡ (Redis) ഛੋ

Slide 58

Slide 58 text

दաܻয় S3 ߡఉ நद ٣۩షܻ ഛੋ

Slide 59

Slide 59 text

दաܻয় ੉޷૑ ߸ജ Ѿҗ ഛੋ 2880 x 1800 200 x 300

Slide 60

Slide 60 text

दաܻয় IT WORKS GOOD THING

Slide 61

Slide 61 text

Next & Future

Slide 62

Slide 62 text

Next & Future 1. In memory நद झషয ૑ਗ 2. ࠺زӝ ୊ܻ ૑ਗ (Celery) 3. স۽٘ ߂ ৻ࠗ ੉޷૑ ୊ܻ ૑ਗ 4. ୶о ২࣌ (Size, Preﬁx, Shard ١) ૑ਗ 5. ੉޷૑ ୊ܻ ঘ࣌ ୶о (Crop ١) 6. ࢸ੿/प೯ਊ ﬁtter CLI ઁҕ 7. ۽ஸ ߸ജోਊ ௑ࣛ ઁҕ 8. పझ౟ ௏٘ ੘ࢿ 9. ޙࢲച 10. ٣೒۽੉ ೐۽ࣁझ ઁҕ (Pluggable Service)

Slide 63

Slide 63 text

хࢎ೤פ׮ MinJae Kwon (@mingrammer) 2017.05.27 Python Seminar On-demand image server with Python