A story about distributed computing before the cloud arrived - Rogier Lommers - bol.com

What do you do when you need a lot of hardware for a short period, but you are unable to get it the normal way? Well, think out-of-the-box... During this presentation I will tell you about how we applied distributed processing in a very unconventional way: by making use of our colleagues laptops and a little bit of Go's cross-platform compiling.

Transcript

1. A story about distributed computing before the cloud arrived Re-design for the cloud Rogier Lommers software architect
2. who is bol.com? 2004-2007 2008-2009 2010 2011 2012 2013 2014 2015 1999-2003 2016 2017 Selling secondhand Books Entertainment Electronics Small household items Baby Baby & Pregnancy clothing Toys Animal Jewelry, Watches & Bags Businesses selling through bol.com Reading digitally Special, right? Started in a port-a-cabin Active in Belgium Takeover by Ahold Logistics through bol.com bol.com app Unlimited reading Cooking & Dining Garden & DIY Big household items Beauty & Health Save Constantly 2018 Alcohol Shoes Health Fusion Ahold Delhaize Expansion bol.com fulfilment center 2019 Living & Sleeping Sport & Leisure Opening offices Belgium Audio books Sustainable packaging First steps sustainable products Introduction
7. 102 million in database 12 million incoming URLs, each day Assets
8. Mobile website / Mobile weblog / Mobile marketing eBook apps External apps Multi-channel approach
10. the asset pipeline fetcher event-queue asset processing part renderer matcher serving part original assets renditions { "height": 1056, "width": 342, "fit": "fill", "fill_color": "white", "trim": "auto", "output_format": "image/jpeg” } synchronous
11. size does matter…
12. google introduced guetzli - a simple, standalone CLI tool, available for linux, macOS and Windows - results are fantastic: 20% / 45% reduction in jpeg file sizes - no loss in perceived quality a new open source jpeg encoder
13. 87 kb
15. 87 kb 64 kb BEFORE GUETZLI AFTER GUETZLI
16. the asset pipeline fetcher event-queue asset processing part renderer matcher serving part original assets renditions { "height": 1056, "width": 342, "fit": "fill", "fill_color": "white", "trim": "auto", "output_format": "image/jpeg” } synchronous guetzli queue
20. the asset pipeline fetcher asset processing part renderer event-queue matcher serving part original assets renditions { "height": 1056, "width": 342, "fit": "fill", "fill_color": "white", "trim": "auto", "output_format": "image/jpeg” } synchronous guetzli queue colleague 1 colleague n GET /work POST /work colleague 1 colleague n colleague 1 colleague n colleague 1 colleague n
21. the “miracle”-client - written in Go - CLI (linux, mac and windows) - one binary, easy installation (opt-in) - guetzli binary embedded (github.com/jteeuwen/go-bindata) - auto-update (github.com/inconshreveable/go-update) - opt-in for colleagues - company software portal was able to push the client to the opted-in-people
22. how it works colleague renderer GET /work guetzli queue renditions POST /work GUETZLI
23. the game is on
25. conclusion
26. thanks! Rogier Lommers rlommers@bol.com