Security Change Through Feedback

YouTube Link :: https://www.youtube.com/watch?v=hzpOYPhSxwc

Like many security teams, Riot has been challenged by new paradigms that came with the move to the cloud. We discuss how our security team has developed a security culture based on feedback and self-service to best thrive in the cloud. We detail how the team assessed the security gaps and challenges in our move into AWS, then describe how the team works within Riot’s unique feedback culture. Walk away with a better understanding of securing projects within AWS without blocking development teams. Learn how we use the internal RFC process, the built-in features of AWS that help provide better security by default, our approach to developer education, and tools we developed, and those from the community, to provide visibility into the security posture of AWS.

Transcript

1. Security Change through Feedback @ Riot
2. AGENDA Level Up Change Challenges Who
3. 17 years in Networking & Security across many industries GSE #44, HackEire CTF creator & founding member of Ireland’s first CSIRT @Riot I combine two of my passions along with some in LA Who Am I? @markofu
4. 100 MILLION MONTHLY ACTIVE PLAYERS MORE THAN 27 MILLION DAILY ACTIVE PLAYERS MORE THAN 7.5 MILLION PEAK CONCURRENT PLAYERS
5. Aspire Who Are We?
6. Teamwork
7. AGENDA Who Level Up Challenges Change
8. !! # of VPNs VPC VPC VPC Connecting AWS
9. Tragedy of the commons Service/resource limits No isolation Shared Accounts
10. Boil The Ocean
11. Secrets
12. Firefighting
13. AGENDA Challenges Who Change Level Up
15. Change by Tools AWS :: KMS, IAM, ACM, STS, CloudTrail, CloudWatch, VPC Flowlogs RIOT-DEVELOPED :: AWSKey (Temporal Keys), Cloud Inquisitor EXTERNAL :: Security Monkey, Terraform, Packer, Elasticsearch/Kibana
16. Minimize the use of local, long-lived AWS IAM Keys Provides temporary AWS API tokens (via STS) & activity monitoring Reduce impact of an API Key Compromise Temporal Goals
18. Problem Statement While AWS is a great place to rapidly iterate and test new features, the vast number of accounts, instances and usage has no easy way of attributing a running instance back to an owner or feature. Ownership
19. Why :: Incident Response is hard when you don’t know who owns what Why :: If you don’t need it, why is it running? What :: Tagging is incredibly easy to use to identify ownership What’s missing?
20. RFCs=Tech Design
21. RFC Feedback Not an approval process, it’s about receiving advice! Becomes a standard through adoption @ scopes Received comments & iterate through the draft
22. AWS Security RFCs that we’ve written: o AWS Standards and Best Practices o Securing AWS environments and their Applications o Securing AWS data at rest o Minimising local AWS accounts o AWS Ownership Attribution
24. Shrink the change => No decision paralysis Feedback & moved to the adoption stage Standard across Riot Solution
25. Let’s go write some code Cloud Inquisitor moves from ideation to implementation Cloud Inquisitor
26. Required Tags :: Name, Owner & Accounting Non-compliant Tagging => Notification 4 weeks => Shutdown ; 12 weeks => Terminate Implementation Details
27. MurderBot Cloud Inquisitor moved from notification into shutdownmurdering mode
28. Feels bad & yes, we received a lot of feedback But we still work at Riot Open & transparent Root Cause Analysis (RCA) So, what next?
30. Engineering “By doing a RCA, the team has truly showed themselves to be part of Engineering. We all make mistakes - this is how we learn and improve. /fistbump ” Cam Dunn (Tech Director), Dec. 2016
31. Our communications & planning sucked Confusion around RFC Adoption & lack of clarity on aspects of the RFC Our notification code had bugs Learnings
32. Improved UX, error-handling & new functionality Manual checking of instances for important products Over-Indexing on communication & lots of checks on alignment Implementation
33. 2nd Adoption, Yay! bcc Engineering “Thanks for everyone's input and consideration for RFC0026, aka MurderBot,over the last several weeks. This is now adopted at Riot scope.” Mike Seavers (Director of Engineering), Feb. 2017
34. Removes incorrectly tagged & un-owned AWS objects Checks that security features are turned on throughout our AWS Infra DNS hijacking & IAM policy management Cinq Features
36. Back-End :: Modular framework (Py 3.5+), Flask , SQL Alchemy & MySQL Front-End :: AngularJS on Nginx Deployment :: Packer & Docker (dev only) Cinq Tech
42. Configuration as code Able to ship quickly
43. Problem :: Left in the corner Solution :: Build relationships, get alignment & iterate Problem :: Silver bullet? Solution :: Best for the job, i.e. solve the specific problem Problem :: Boil The Ocean Solution :: Shrink the change – biggest impact, lowest effort possible Takeaways
44. AGENDA Challenges Who Level Up Change
45. All TaggableAWS Objects (revised RFCs) EBS Snapshots, Amazon S3 & AWS Organizations compatibility Futures
46. Repo :: https://github.com/RiotGames/cloud-inquisitor Collaboration :: Slack, GitHub Issues Contributions Welcome :: Roadmap, Ideas & Pull Requests OSS
47. Thank You