$30 off During Our Annual Pro Sale. View Details »

On-demand image server with Python

On-demand image server with Python

This is about on-demand image server, the "fitter'

mingrammer

May 27, 2017
Tweet

More Decks by mingrammer

Other Decks in Programming

Transcript

  1. On-demand image server with Python
    MinJae Kwon (@mingrammer)
    2017.05.27 Python Seminar

    View Slide

  2. Name
    ӂ޹੤ (MinJae Kwon)
    Nickname
    @mingrammer
    Email
    [email protected]
    Who
    ই઱؀೟Ү ࣗ೐౟ਝয೟җ 3೟֙ / Backend Developer
    Blog
    https://mingrammer.com
    Facebook
    https://facebook.com/mingrammer
    Github
    https://github.com/mingrammer
    Eng Blog
    https://medium.com/@mingrammer

    View Slide

  3. 2. ѐߊ زӝ
    5. दաܻয়
    3. рױೠ ইఃఫ୊
    Contents
    6. Next & Future
    1. ৡ٣ݔ٘ ੉޷૑ ࢲߡ?
    4. ݽٕ ҳഅ୓

    View Slide

  4. ৡ٣ݔ٘ ੉޷૑ ࢲߡ?

    View Slide

  5. ৡ٣ݔ٘ ੉޷૑ ࢲߡ?
    Image Server
    image resized
    thumbnail
    rotated

    View Slide

  6. ৡ٣ݔ٘ ੉޷૑ ࢲߡ?
    Image
    Server
    image resized
    thumbnail
    rotated
    User Service
    desired parameters
    new
    image

    View Slide

  7. ৡ٣ݔ٘ ੉޷૑ ࢲߡ?
    ੉޷૑ܳ ޷ܻ ࢤࢿ೧ف૑ ঋইب ؽ
    ਗೡٸ ׮নೠ ഋక੄ ੉޷૑ܳ ৡ٣ݔ٘۽ ࢤࢿ оמ
    ੢੼

    View Slide

  8. ৡ٣ݔ٘ ੉޷૑ ࢲߡ?
    ੉޷૑ܳ ࢎਊೞח Ѣ੄ ݽٚҔ
    ׮নೠ ࢎ੉ૉ੄ ੉޷૑о ೙ਃೠ ݽ߄ੌ জ
    ੉޷૑ ӝ߈ ࢲ࠺झ
    ੸ਊ оמೠ ࢲ࠺झ

    View Slide

  9. ѐߊ زӝ

    View Slide

  10. ѐߊ زӝ
    1. ഥࢎীࢲ ٣੗੉ցо ੉޷ ٜ݅য֬਷ ੉޷૑ ࢎ੉ૉ੄ ߸҃੉ ೙ਃ
    2. ੉޷૑ ਗࠄਸ о૓ ٣੗੉ցীѱ ࢎ੉ૉ ߸҃ ਃ୒੉ ೙ਃ
    3. рױೠ ੘সੋ Ѫী ࠺೧ ࣗਊغח ࠺ਊ੉ ֫਺
    4. ٜ݅੗!

    View Slide

  11. ѐߊ زӝ
    ߈ա੺݅ী ٜ݅Ҋ ׮਺ ಁ஖ٸ ੸ਊ
    ੉޷૑ ܻࢎ੉૚ ੗زച۽ ੋ೧ ࢕੉ ખ ؊ աই૗

    View Slide

  12. ѐߊ زӝ
    ׼द ഥࢎীࢲ ೙ਃೠ ੉޷૑ ܻࢎ੉૚ ӝמ਷
    ੉޷૑ ࠺ਯ݅ ਬ૑೧ࢲ “width”݅ ઑ੺ೞҊ நयೞח ӝמ ٯ ೞա
    But

    View Slide

  13. ѐߊ زӝ
    ખ ؊ ਬোೠ য়೑ࣗझ ৡ٣ݔ٘ ੉޷૑ ࢲߡܳ ٜ݅যࠁ੗
    https://github.com/mingrammer/fitter

    View Slide

  14. рױೠ ইఃఫ୊

    View Slide

  15. рױೠ ইఃఫ୊
    API Server
    API server for fitter

    View Slide

  16. рױೠ ইఃఫ୊
    API Server
    Configuration Parser
    Fitter Configuration
    Parse the customizable configuration

    View Slide

  17. рױೠ ইఃఫ୊
    API Server
    Configuration Parser
    Fitter Configuration
    Storage
    Manage storage

    View Slide

  18. рױೠ ইఃఫ୊
    API Server
    Configuration Parser
    Fitter Configuration
    Storage
    Supports source/store storages
    Source
    Store

    View Slide

  19. рױೠ ইఃఫ୊
    API Server
    Configuration Parser
    Fitter Configuration
    Storage
    Supports file system and s3
    Source
    Store
    s3 fs

    View Slide

  20. рױೠ ইఃఫ୊
    API Server
    Configuration Parser
    Fitter Configuration
    Storage
    Source
    Store
    s3 fs
    Engine Processing the transforming

    View Slide

  21. рױೠ ইఃఫ୊
    API Server
    Configuration Parser
    Fitter Configuration
    Storage
    Source
    Store
    s3 fs
    Engine
    Thumbnail
    Flip
    Resize
    Rotate
    Various mode

    View Slide

  22. рױೠ ইఃఫ୊
    API Server
    Configuration Parser
    Fitter Configuration
    Storage
    Source
    Store
    s3 fs
    Engine
    Thumbnail
    Flip
    Resize
    Rotate
    Cache Server
    Caching the generated image

    View Slide

  23. рױೠ ইఃఫ୊
    API Server
    Configuration Parser
    Fitter Configuration
    Storage
    Source
    Store
    s3 fs
    Engine
    Thumbnail
    Flip
    Resize
    Rotate
    Cache Server
    Cache Store
    Manage the cache store

    View Slide

  24. рױೠ ইఃఫ୊
    API Server
    Configuration Parser
    Fitter Configuration
    Storage
    Source
    Store
    s3 fs
    Engine
    Thumbnail
    Flip
    Resize
    Rotate
    Cache Server
    Cache Store
    in-memory redis
    Supports in-memory and redis

    View Slide

  25. рױೠ ইఃఫ୊
    API Server
    Configuration Parser
    Fitter Configuration
    Storage
    Source
    Store
    s3 fs
    Engine
    Thumbnail
    Flip
    Resize
    Rotate
    Cache Server
    Cache Store
    in-memory redis
    Paramter
    Validator
    Validate the parameters

    View Slide

  26. рױೠ ইఃఫ୊
    API Server
    Configuration Parser
    Storage
    Engine
    Cache Server
    Cache Store
    Paramter
    Validator

    View Slide

  27. рױೠ ইఃఫ୊
    API Server
    flask
    Configuration Parser
    pyyaml
    Storage
    boto3
    requests
    Engine
    wand
    Cache Server
    Cache Store
    redis-py
    Paramter
    Validator
    Python Fullstack

    View Slide

  28. ݽٕ ҳഅ୓

    View Slide

  29. ݽٕ ҳഅ୓
    ۨನ૑షܻ ҳࢿ

    View Slide

  30. ݽٕ ҳഅ୓
    API Server
    Optional URL for Flask?
    Pre-registered Not pre-registered

    View Slide

  31. ݽٕ ҳഅ୓
    API Server
    Optional URL for Flask?

    View Slide

  32. ݽٕ ҳഅ୓
    Config parsing - Using common classes

    View Slide

  33. ݽٕ ҳഅ୓
    Config parsing
    Load Config
    Parsing
    &
    Validating
    Check Cache
    Show errors
    invalid
    or
    failed
    valid

    View Slide

  34. ݽٕ ҳഅ୓
    Config parsing - Load and validate the required fields

    View Slide

  35. ݽٕ ҳഅ୓
    Config parsing - Load and validate the required fields and set the configs

    View Slide

  36. ݽٕ ҳഅ୓
    Config parsing - Load and validate the required fields

    View Slide

  37. ݽٕ ҳഅ୓
    Params validator
    Request Validate Params Handling
    Show errors
    invalid
    valid

    View Slide

  38. ݽٕ ҳഅ୓
    Params validator - Validate all params and restriction of each mode

    View Slide

  39. ݽٕ ҳഅ୓
    Params validator - Validate the all params before view handling using decorator

    View Slide

  40. Double caching?
    ݽٕ ҳഅ୓
    Check
    Cache Server
    Check
    Store Storage
    Generate
    Use cached Use cached
    exists
    exists
    not not

    View Slide

  41. Double caching? - Use decorators!! Decorator is love
    ݽٕ ҳഅ୓

    View Slide

  42. Double caching? - Use decorators!! Decorator is love
    ݽٕ ҳഅ୓

    View Slide

  43. How to use different classes on same function?
    ݽٕ ҳഅ୓
    Stroage
    S3 File System
    Cache Store
    Source Store Source Store
    In memory Redis
    class … class …
    class … class … class … class …

    View Slide

  44. How to use different classes on same function? - Use Interface-like class
    ݽٕ ҳഅ୓

    View Slide

  45. How to use different classes on same function? - Use Interface-like class
    ݽٕ ҳഅ୓

    View Slide

  46. How to use different classes on same function? - Use Interface-like class
    ݽٕ ҳഅ୓
    Two types are totally different but they have same interfaces!

    View Slide

  47. There is very-simple engine - Just wrapping the each mode. Common interface :D
    ݽٕ ҳഅ୓

    View Slide

  48. We are the world - Combine these modules!
    ݽٕ ҳഅ୓
    Validate
    Params
    Cache
    Interface
    Storage
    Interface
    Storage
    Interface
    Parsing
    Config
    Transform
    Interface
    Check
    Cache Store
    Check
    Storage
    Transform
    Engine
    Handle
    Storage

    View Slide

  49. दաܻয়

    View Slide

  50. दաܻয়
    1. ۽ஸ ӝ߈ ੉޷૑ ߸ജ ో
    2. ੉޷૑ ߸ജ ߂ ௿ۄ਋٘ ߔস ో
    3. ߔূ٘ ৡ٣ݔ٘ ੉޷૑ ࢲߡ

    View Slide

  51. दաܻয়
    1. ۽ஸ ӝ߈ ੉޷૑ ߸ജ ో
    2. ੉޷૑ ߸ജ ߂ ௿ۄ਋٘ ߔস ో
    3. ߔূ٘ ৡ٣ݔ٘ ੉޷૑ ࢲߡ

    View Slide

  52. दաܻয়
    நद ࢲߡ ࢸ੿
    झషܻ૑ ࢸ੿
    ২࣌ ࢸ੿

    View Slide

  53. दաܻয়
    fitter ࢲߡ ҳز

    View Slide

  54. दաܻয়
    ੉޷૑ ߸ജ ਃ୒
    Wallpaper/squares_widew.png ੉޷૑ܳ 200 x 300ਵ۽ ߸ജ resize

    View Slide

  55. दաܻয়
    What is ‘get’?
    show : Retrieve the image information asynchronously (not yet)

    get : Retrieve the image information synchronously
    redirect : Redirect to url of transformed image

    View Slide

  56. दաܻয়
    ਃ୒ റ ߈ജч

    View Slide

  57. दաܻয়
    நद ࢲߡ (Redis) ഛੋ

    View Slide

  58. दաܻয়
    S3 ߡఉ நद ٣۩షܻ ഛੋ

    View Slide

  59. दաܻয়
    ੉޷૑ ߸ജ Ѿҗ ഛੋ
    2880 x 1800 200 x 300

    View Slide

  60. दաܻয়
    IT WORKS
    GOOD THING

    View Slide

  61. Next & Future

    View Slide

  62. Next & Future
    1. In memory நद झషয ૑ਗ
    2. ࠺زӝ ୊ܻ ૑ਗ (Celery)
    3. স۽٘ ߂ ৻ࠗ ੉޷૑ ୊ܻ ૑ਗ
    4. ୶о ২࣌ (Size, Prefix, Shard ١) ૑ਗ
    5. ੉޷૑ ୊ܻ ঘ࣌ ୶о (Crop ١)
    6. ࢸ੿/प೯ਊ fitter CLI ઁҕ
    7. ۽ஸ ߸ജోਊ ௑ࣛ ઁҕ
    8. పझ౟ ௏٘ ੘ࢿ
    9. ޙࢲച
    10. ٣೒۽੉ ೐۽ࣁझ ઁҕ (Pluggable Service)

    View Slide

  63. хࢎ೤פ׮
    MinJae Kwon (@mingrammer)
    2017.05.27 Python Seminar
    On-demand image server with Python

    View Slide