A story about distributed computing before the cloud arrived - Rogier Lommers - bol.com

What do you do when you need a lot of hardware for a short period, but you are unable to get it the normal way? Well, think out-of-the-box... During this presentation I will tell you about how we applied distributed processing in a very unconventional way: by making use of our colleagues laptops and a little bit of Go's cross-platform compiling.

Transcript

1. A story about
   distributed computing
   before the cloud arrived
   Re-design for
   the cloud
   Rogier Lommers
   software
   architect
   

   View Slide

2. who is bol.com?
   2004-2007 2008-2009 2010 2011 2012 2013 2014 2015
   1999-2003 2016 2017
   Selling secondhand
   Books Entertainment Electronics
   Small household
   items
   Baby Baby & Pregnancy
   clothing
   Toys
   Animal
   Jewelry, Watches & Bags
   Businesses selling through
   bol.com
   Reading digitally
   Special, right?
   Started in a
   port-a-cabin
   Active in Belgium Takeover by Ahold
   Logistics through
   bol.com
   bol.com app
   Unlimited
   reading
   Cooking & Dining
   Garden & DIY
   Big household
   items
   Beauty & Health Save
   Constantly
   2018
   Alcohol
   Shoes
   Health
   Fusion Ahold
   Delhaize
   Expansion bol.com
   fulfilment center
   2019
   Living &
   Sleeping
   Sport &
   Leisure
   Opening offices
   Belgium
   Audio books
   Sustainable
   packaging
   First steps
   sustainable products
   Introduction
   

   View Slide

3. View Slide

4. View Slide

5. View Slide

6. View Slide

7. 102 million in
   database
   12 million incoming
   URLs, each day
   Assets
   

   View Slide

8. Mobile website / Mobile weblog / Mobile marketing
   eBook apps External apps
   Multi-channel approach
   

   View Slide

9. View Slide

10. the asset pipeline
    fetcher
    event-queue
    asset processing part
    renderer
    matcher
    serving part
    original
    assets
    renditions
    {
    "height": 1056,
    "width": 342,
    "fit": "fill",
    "fill_color": "white",
    "trim": "auto",
    "output_format": "image/jpeg”
    }
    synchronous
    

    View Slide

11. size does matter…
    

    View Slide

12. google
    introduced
    guetzli
    - a simple, standalone CLI tool, available for
    linux, macOS and Windows
    - results are fantastic: 20% / 45% reduction in
    jpeg file sizes
    - no loss in perceived quality
    a new open source jpeg
    encoder
    

    View Slide

13. 87 kb
    

    View Slide

14. View Slide

15. 87 kb 64 kb
    BEFORE GUETZLI AFTER GUETZLI
    

    View Slide

16. the asset pipeline
    fetcher
    event-queue
    asset processing part
    renderer
    matcher
    serving part
    original
    assets
    renditions
    {
    "height": 1056,
    "width": 342,
    "fit": "fill",
    "fill_color": "white",
    "trim": "auto",
    "output_format": "image/jpeg”
    }
    synchronous
    guetzli
    queue
    

    View Slide

17. View Slide

18. View Slide

19. View Slide

20. the asset pipeline
    fetcher
    asset processing part
    renderer
    event-queue
    matcher
    serving part
    original
    assets
    renditions
    {
    "height": 1056,
    "width": 342,
    "fit": "fill",
    "fill_color": "white",
    "trim": "auto",
    "output_format": "image/jpeg”
    }
    synchronous
    guetzli
    queue
    colleague 1 colleague n
    GET /work
    POST /work
    colleague 1 colleague n
    colleague 1 colleague n
    colleague 1 colleague n
    

    View Slide

21. the “miracle”-client
    - written in Go
    - CLI (linux, mac and windows)
    - one binary, easy installation (opt-in)
    - guetzli binary embedded (github.com/jteeuwen/go-bindata)
    - auto-update (github.com/inconshreveable/go-update)
    - opt-in for colleagues
    - company software portal was able to push the client to the
    opted-in-people
    

    View Slide

22. how it works
    colleague
    renderer
    GET /work
    guetzli
    queue renditions
    POST /work
    GUETZLI
    

    View Slide

23. the game is on
    

    View Slide

24. View Slide

25. conclusion
    

    View Slide

26. thanks!
    Rogier Lommers
    rlommers@bol.com
    

    View Slide