R *is* Production Safe

For those of us within the R Community, it almost seems like a silly question but, based on my experiences working with those outside the community, it can be a very real fear. In a sense, this talk is more aimed at those for whom R is a strange and exotic beast to be feared, but hopefully you’ll get something from it too ;)

Transcript

1. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com R is Production-Safe Mark Sellors Head of Data Engineering - Mango Solutions
2. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com About Me • Head of Data Engineering • Run Mango Data Labs • Architecture • Automation • Industrialisation • DevOps/DataOps • @sellorm dsradar.com
3. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com What is production? • Many different things to many people • Could be: • A web-scale, public facing server environment • Could just be a simple script • In general it’s just something that gets run more than once, and is important to your organisation
4. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com “You can’t use R in Production” -- Anonymous
5. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com “You can’t use R in Production” -- Anonymous
6. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com Things We Know About R… • Mature • Well documented • Well understood • Commercial versions and/or support • Dynamic – fast moving ecosystem • Amazing community
7. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com Things other people think about R… • Weird syntax • Not many users • Only used by statisticians and Data Scientists • Performance problems • Not very versatile • No good for large data sets
8. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com Some myths about R… • Weird syntax • Not many users • Only used by statisticians and Data Scientists • Performance problems • Not very versatile • No good for large data sets
9. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com Disconnect, what disconnect? • We all know that R is great • ”All the people I follow on twitter are using R in production” • There are conferences, like this one (!), where even more people are using R
10. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com Why the disconnect? • Great communities can also create powerful feedback loops and echo chambers • R’s domain specific nature makes it somewhat unapproachable for many outside the community • Fear – A thing that I don’t understand must surely be a bad thing
11. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com
12. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com How do we combat fear? • Education • Outreach • Collaboration • Empathy
13. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com
14. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com “Nobody else is using R in Production” -- Anonymous
15. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com “Nobody else is using R in Production” -- Anonymous
16. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com Allianz • What: Insurance Claim scoring • How: API’s in R • Using: Rserve, Java
17. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com Worldpay • What: Call Centre CRM tool and x-sell pipeline • How: Web app written in R • Using: Shiny
18. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com ONS • What: Migrate from SAS • How: Migrate SAS based services to R • Using: Plumber & command line
19. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com Hedge Fund • What: Batch based scoring system • How: Sophisticated command line application • Using: R and various databases
20. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com How are they doing it?
21. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com It’s not that hard! • Collaboration • Thinking about what they’re doing • Implementing more formal methodologies • Rigour around processes • Well managed release strategy • Isolated environments
22. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com Requirements Design Development Testing Production Example Methodology
23. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com Dev Test Prod Example Environments
24. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com
25. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com How much rigour do I need? This all sounds like a lot of effort
26. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com Rigour investment checklist • What’s the impact of what you’re doing? • Who’s affected if it’s not working? • What is the cost of an outage to the business? • Why is the new solution even needed? • Where will the code be run? • When will it be run?
27. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com Think about what you’re doing!
28. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com Why does any of this matter? • Reduce time to insight • Lower costs • Improved quality
29. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com “I’ll never be able to do any of this in my organisation”
30. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com Don’t despair, there is hope • Find a sympathetic person on the ops side • Learn to do it yourself • Start small • Amaze people!
31. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com > summary(talk) •You can use R in production environments • If you think about what you’re doing • and approach the task with care • Find sympathetic collaborators • If you are doing it already, tell people about it • If you aren’t, what are you waiting for?
32. Mark Sellors – Head of Data Engineering msellors@mango-solutions.com Any Questions?