On-demand image server with Python

This is about on-demand image server, the "fitter'

Transcript

1. On-demand image server with Python
   MinJae Kwon (@mingrammer)
   2017.05.27 Python Seminar
   

   View Slide

2. Name
   ӂ޹੤ (MinJae Kwon)
   Nickname
   @mingrammer
   Email
   k239507@gmail.com
   Who
   ই઱؀೟Ү ࣗ೐౟ਝয೟җ 3೟֙ / Backend Developer
   Blog
   https://mingrammer.com
   Facebook
   https://facebook.com/mingrammer
   Github
   https://github.com/mingrammer
   Eng Blog
   https://medium.com/@mingrammer
   

   View Slide

3. 2. ѐߊ زӝ
   5. दաܻয়
   3. рױೠ ইఃఫ୊
   Contents
   6. Next & Future
   1. ৡ٣ݔ٘ ੉޷૑ ࢲߡ?
   4. ݽٕ ҳഅ୓
   

   View Slide

4. ৡ٣ݔ٘ ੉޷૑ ࢲߡ?
   

   View Slide

5. ৡ٣ݔ٘ ੉޷૑ ࢲߡ?
   Image Server
   image resized
   thumbnail
   rotated
   

   View Slide

6. ৡ٣ݔ٘ ੉޷૑ ࢲߡ?
   Image
   Server
   image resized
   thumbnail
   rotated
   User Service
   desired parameters
   new
   image
   

   View Slide

7. ৡ٣ݔ٘ ੉޷૑ ࢲߡ?
   ੉޷૑ܳ ޷ܻ ࢤࢿ೧ف૑ ঋইب ؽ
   ਗೡٸ ׮নೠ ഋక੄ ੉޷૑ܳ ৡ٣ݔ٘۽ ࢤࢿ оמ
   ੢੼
   

   View Slide

8. ৡ٣ݔ٘ ੉޷૑ ࢲߡ?
   ੉޷૑ܳ ࢎਊೞח Ѣ੄ ݽٚҔ
   ׮নೠ ࢎ੉ૉ੄ ੉޷૑о ೙ਃೠ ݽ߄ੌ জ
   ੉޷૑ ӝ߈ ࢲ࠺झ
   ੸ਊ оמೠ ࢲ࠺झ
   

   View Slide

9. ѐߊ زӝ
   

   View Slide

10. ѐߊ زӝ
    1. ഥࢎীࢲ ٣੗੉ցо ੉޷ ٜ݅য֬਷ ੉޷૑ ࢎ੉ૉ੄ ߸҃੉ ೙ਃ
    2. ੉޷૑ ਗࠄਸ о૓ ٣੗੉ցীѱ ࢎ੉ૉ ߸҃ ਃ୒੉ ೙ਃ
    3. рױೠ ੘সੋ Ѫী ࠺೧ ࣗਊغח ࠺ਊ੉ ֫਺
    4. ٜ݅੗!
    

    View Slide

11. ѐߊ زӝ
    ߈ա੺݅ী ٜ݅Ҋ ׮਺ ಁ஖ٸ ੸ਊ
    ੉޷૑ ܻࢎ੉૚ ੗زച۽ ੋ೧ ࢕੉ ખ ؊ աই૗
    

    View Slide

12. ѐߊ زӝ
    ׼द ഥࢎীࢲ ೙ਃೠ ੉޷૑ ܻࢎ੉૚ ӝמ਷
    ੉޷૑ ࠺ਯ݅ ਬ૑೧ࢲ “width”݅ ઑ੺ೞҊ நयೞח ӝמ ٯ ೞա
    But
    

    View Slide

13. ѐߊ زӝ
    ખ ؊ ਬোೠ য়೑ࣗझ ৡ٣ݔ٘ ੉޷૑ ࢲߡܳ ٜ݅যࠁ੗
    https://github.com/mingrammer/fitter
    

    View Slide

14. рױೠ ইఃఫ୊
    

    View Slide

15. рױೠ ইఃఫ୊
    API Server
    API server for fitter
    

    View Slide

16. рױೠ ইఃఫ୊
    API Server
    Configuration Parser
    Fitter Configuration
    Parse the customizable configuration
    

    View Slide

17. рױೠ ইఃఫ୊
    API Server
    Configuration Parser
    Fitter Configuration
    Storage
    Manage storage
    

    View Slide

18. рױೠ ইఃఫ୊
    API Server
    Configuration Parser
    Fitter Configuration
    Storage
    Supports source/store storages
    Source
    Store
    

    View Slide

19. рױೠ ইఃఫ୊
    API Server
    Configuration Parser
    Fitter Configuration
    Storage
    Supports file system and s3
    Source
    Store
    s3 fs
    

    View Slide

20. рױೠ ইఃఫ୊
    API Server
    Configuration Parser
    Fitter Configuration
    Storage
    Source
    Store
    s3 fs
    Engine Processing the transforming
    

    View Slide

21. рױೠ ইఃఫ୊
    API Server
    Configuration Parser
    Fitter Configuration
    Storage
    Source
    Store
    s3 fs
    Engine
    Thumbnail
    Flip
    Resize
    Rotate
    Various mode
    

    View Slide

22. рױೠ ইఃఫ୊
    API Server
    Configuration Parser
    Fitter Configuration
    Storage
    Source
    Store
    s3 fs
    Engine
    Thumbnail
    Flip
    Resize
    Rotate
    Cache Server
    Caching the generated image
    

    View Slide

23. рױೠ ইఃఫ୊
    API Server
    Configuration Parser
    Fitter Configuration
    Storage
    Source
    Store
    s3 fs
    Engine
    Thumbnail
    Flip
    Resize
    Rotate
    Cache Server
    Cache Store
    Manage the cache store
    

    View Slide

24. рױೠ ইఃఫ୊
    API Server
    Configuration Parser
    Fitter Configuration
    Storage
    Source
    Store
    s3 fs
    Engine
    Thumbnail
    Flip
    Resize
    Rotate
    Cache Server
    Cache Store
    in-memory redis
    Supports in-memory and redis
    

    View Slide

25. рױೠ ইఃఫ୊
    API Server
    Configuration Parser
    Fitter Configuration
    Storage
    Source
    Store
    s3 fs
    Engine
    Thumbnail
    Flip
    Resize
    Rotate
    Cache Server
    Cache Store
    in-memory redis
    Paramter
    Validator
    Validate the parameters
    

    View Slide

26. рױೠ ইఃఫ୊
    API Server
    Configuration Parser
    Storage
    Engine
    Cache Server
    Cache Store
    Paramter
    Validator
    

    View Slide

27. рױೠ ইఃఫ୊
    API Server
    flask
    Configuration Parser
    pyyaml
    Storage
    boto3
    requests
    Engine
    wand
    Cache Server
    Cache Store
    redis-py
    Paramter
    Validator
    Python Fullstack
    

    View Slide

28. ݽٕ ҳഅ୓
    

    View Slide

29. ݽٕ ҳഅ୓
    ۨನ૑షܻ ҳࢿ
    

    View Slide

30. ݽٕ ҳഅ୓
    API Server
    Optional URL for Flask?
    Pre-registered Not pre-registered
    

    View Slide

31. ݽٕ ҳഅ୓
    API Server
    Optional URL for Flask?
    

    View Slide

32. ݽٕ ҳഅ୓
    Config parsing - Using common classes
    

    View Slide

33. ݽٕ ҳഅ୓
    Config parsing
    Load Config
    Parsing
    &
    Validating
    Check Cache
    Show errors
    invalid
    or
    failed
    valid
    

    View Slide

34. ݽٕ ҳഅ୓
    Config parsing - Load and validate the required fields
    

    View Slide

35. ݽٕ ҳഅ୓
    Config parsing - Load and validate the required fields and set the configs
    

    View Slide

36. ݽٕ ҳഅ୓
    Config parsing - Load and validate the required fields
    

    View Slide

37. ݽٕ ҳഅ୓
    Params validator
    Request Validate Params Handling
    Show errors
    invalid
    valid
    

    View Slide

38. ݽٕ ҳഅ୓
    Params validator - Validate all params and restriction of each mode
    

    View Slide

39. ݽٕ ҳഅ୓
    Params validator - Validate the all params before view handling using decorator
    

    View Slide

40. Double caching?
    ݽٕ ҳഅ୓
    Check
    Cache Server
    Check
    Store Storage
    Generate
    Use cached Use cached
    exists
    exists
    not not
    

    View Slide

41. Double caching? - Use decorators!! Decorator is love
    ݽٕ ҳഅ୓
    

    View Slide

42. Double caching? - Use decorators!! Decorator is love
    ݽٕ ҳഅ୓
    

    View Slide

43. How to use different classes on same function?
    ݽٕ ҳഅ୓
    Stroage
    S3 File System
    Cache Store
    Source Store Source Store
    In memory Redis
    class … class …
    class … class … class … class …
    

    View Slide

44. How to use different classes on same function? - Use Interface-like class
    ݽٕ ҳഅ୓
    

    View Slide

45. How to use different classes on same function? - Use Interface-like class
    ݽٕ ҳഅ୓
    

    View Slide

46. How to use different classes on same function? - Use Interface-like class
    ݽٕ ҳഅ୓
    Two types are totally different but they have same interfaces!
    

    View Slide

47. There is very-simple engine - Just wrapping the each mode. Common interface :D
    ݽٕ ҳഅ୓
    

    View Slide

48. We are the world - Combine these modules!
    ݽٕ ҳഅ୓
    Validate
    Params
    Cache
    Interface
    Storage
    Interface
    Storage
    Interface
    Parsing
    Config
    Transform
    Interface
    Check
    Cache Store
    Check
    Storage
    Transform
    Engine
    Handle
    Storage
    

    View Slide

49. दաܻয়
    

    View Slide

50. दաܻয়
    1. ۽ஸ ӝ߈ ੉޷૑ ߸ജ ో
    2. ੉޷૑ ߸ജ ߂ ௿ۄ਋٘ ߔস ో
    3. ߔূ٘ ৡ٣ݔ٘ ੉޷૑ ࢲߡ
    

    View Slide

51. दաܻয়
    1. ۽ஸ ӝ߈ ੉޷૑ ߸ജ ో
    2. ੉޷૑ ߸ജ ߂ ௿ۄ਋٘ ߔস ో
    3. ߔূ٘ ৡ٣ݔ٘ ੉޷૑ ࢲߡ
    

    View Slide

52. दաܻয়
    நद ࢲߡ ࢸ੿
    झషܻ૑ ࢸ੿
    ২࣌ ࢸ੿
    

    View Slide

53. दաܻয়
    fitter ࢲߡ ҳز
    

    View Slide

54. दաܻয়
    ੉޷૑ ߸ജ ਃ୒
    Wallpaper/squares_widew.png ੉޷૑ܳ 200 x 300ਵ۽ ߸ജ resize
    

    View Slide

55. दաܻয়
    What is ‘get’?
    show : Retrieve the image information asynchronously (not yet)

    get : Retrieve the image information synchronously
    redirect : Redirect to url of transformed image
    

    View Slide

56. दաܻয়
    ਃ୒ റ ߈ജч
    

    View Slide

57. दաܻয়
    நद ࢲߡ (Redis) ഛੋ
    

    View Slide

58. दաܻয়
    S3 ߡఉ நद ٣۩షܻ ഛੋ
    

    View Slide

59. दաܻয়
    ੉޷૑ ߸ജ Ѿҗ ഛੋ
    2880 x 1800 200 x 300
    

    View Slide

60. दաܻয়
    IT WORKS
    GOOD THING
    

    View Slide

61. Next & Future
    

    View Slide

62. Next & Future
    1. In memory நद झషয ૑ਗ
    2. ࠺زӝ ୊ܻ ૑ਗ (Celery)
    3. স۽٘ ߂ ৻ࠗ ੉޷૑ ୊ܻ ૑ਗ
    4. ୶о ২࣌ (Size, Prefix, Shard ١) ૑ਗ
    5. ੉޷૑ ୊ܻ ঘ࣌ ୶о (Crop ١)
    6. ࢸ੿/प೯ਊ fitter CLI ઁҕ
    7. ۽ஸ ߸ജోਊ ௑ࣛ ઁҕ
    8. పझ౟ ௏٘ ੘ࢿ
    9. ޙࢲച
    10. ٣೒۽੉ ೐۽ࣁझ ઁҕ (Pluggable Service)
    

    View Slide

63. хࢎ೤פ׮
    MinJae Kwon (@mingrammer)
    2017.05.27 Python Seminar
    On-demand image server with Python
    

    View Slide