Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A story about distributed computing before the cloud arrived - Rogier Lommers - bol.com

GoDays
January 23, 2020

A story about distributed computing before the cloud arrived - Rogier Lommers - bol.com

What do you do when you need a lot of hardware for a short period, but you are unable to get it the normal way? Well, think out-of-the-box... During this presentation I will tell you about how we applied distributed processing in a very unconventional way: by making use of our colleagues laptops and a little bit of Go's cross-platform compiling.

GoDays

January 23, 2020
Tweet

More Decks by GoDays

Other Decks in Technology

Transcript

  1. A story about
    distributed computing
    before the cloud arrived
    Re-design for
    the cloud
    Rogier Lommers
    software
    architect

    View Slide

  2. who is bol.com?
    2004-2007 2008-2009 2010 2011 2012 2013 2014 2015
    1999-2003 2016 2017
    Selling secondhand
    Books Entertainment Electronics
    Small household
    items
    Baby Baby & Pregnancy
    clothing
    Toys
    Animal
    Jewelry, Watches & Bags
    Businesses selling through
    bol.com
    Reading digitally
    Special, right?
    Started in a
    port-a-cabin
    Active in Belgium Takeover by Ahold
    Logistics through
    bol.com
    bol.com app
    Unlimited
    reading
    Cooking & Dining
    Garden & DIY
    Big household
    items
    Beauty & Health Save
    Constantly
    2018
    Alcohol
    Shoes
    Health
    Fusion Ahold
    Delhaize
    Expansion bol.com
    fulfilment center
    2019
    Living &
    Sleeping
    Sport &
    Leisure
    Opening offices
    Belgium
    Audio books
    Sustainable
    packaging
    First steps
    sustainable products
    Introduction

    View Slide

  3. View Slide

  4. View Slide

  5. View Slide

  6. View Slide

  7. 102 million in
    database
    12 million incoming
    URLs, each day
    Assets

    View Slide

  8. Mobile website / Mobile weblog / Mobile marketing
    eBook apps External apps
    Multi-channel approach

    View Slide

  9. View Slide

  10. the asset pipeline
    fetcher
    event-queue
    asset processing part
    renderer
    matcher
    serving part
    original
    assets
    renditions
    {
    "height": 1056,
    "width": 342,
    "fit": "fill",
    "fill_color": "white",
    "trim": "auto",
    "output_format": "image/jpeg”
    }
    synchronous

    View Slide

  11. size does matter…

    View Slide

  12. google
    introduced
    guetzli
    - a simple, standalone CLI tool, available for
    linux, macOS and Windows
    - results are fantastic: 20% / 45% reduction in
    jpeg file sizes
    - no loss in perceived quality
    a new open source jpeg
    encoder

    View Slide

  13. 87 kb

    View Slide

  14. View Slide

  15. 87 kb 64 kb
    BEFORE GUETZLI AFTER GUETZLI

    View Slide

  16. the asset pipeline
    fetcher
    event-queue
    asset processing part
    renderer
    matcher
    serving part
    original
    assets
    renditions
    {
    "height": 1056,
    "width": 342,
    "fit": "fill",
    "fill_color": "white",
    "trim": "auto",
    "output_format": "image/jpeg”
    }
    synchronous
    guetzli
    queue

    View Slide

  17. View Slide

  18. View Slide

  19. View Slide

  20. the asset pipeline
    fetcher
    asset processing part
    renderer
    event-queue
    matcher
    serving part
    original
    assets
    renditions
    {
    "height": 1056,
    "width": 342,
    "fit": "fill",
    "fill_color": "white",
    "trim": "auto",
    "output_format": "image/jpeg”
    }
    synchronous
    guetzli
    queue
    colleague 1 colleague n
    GET /work
    POST /work
    colleague 1 colleague n
    colleague 1 colleague n
    colleague 1 colleague n

    View Slide

  21. the “miracle”-client
    - written in Go
    - CLI (linux, mac and windows)
    - one binary, easy installation (opt-in)
    - guetzli binary embedded (github.com/jteeuwen/go-bindata)
    - auto-update (github.com/inconshreveable/go-update)
    - opt-in for colleagues
    - company software portal was able to push the client to the
    opted-in-people

    View Slide

  22. how it works
    colleague
    renderer
    GET /work
    guetzli
    queue renditions
    POST /work
    GUETZLI

    View Slide

  23. the game is on

    View Slide

  24. View Slide

  25. conclusion

    View Slide

  26. thanks!
    Rogier Lommers
    [email protected]

    View Slide