17 years in Networking & Security across many industries
GSE #44, HackEire CTF creator & founding member of Ireland’s first CSIRT
@Riot I combine two of my passions along with some in LA
Who Am I?
@markofu
Slide 4
Slide 4 text
100 MILLION
MONTHLY ACTIVE
PLAYERS
MORE THAN
27 MILLION
DAILY ACTIVE
PLAYERS
MORE THAN
7.5 MILLION
PEAK CONCURRENT
PLAYERS
Slide 5
Slide 5 text
Aspire
Who Are We?
Slide 6
Slide 6 text
Teamwork
Slide 7
Slide 7 text
AGENDA
Who
Level Up
Challenges
Change
Slide 8
Slide 8 text
!!
# of VPNs
VPC
VPC
VPC
Connecting AWS
Slide 9
Slide 9 text
Tragedy of the commons
Service/resource limits
No isolation
Shared Accounts
Minimize the use of local, long-lived AWS IAM Keys
Provides temporary AWS API tokens (via STS) & activity monitoring
Reduce impact of an API Key Compromise
Temporal Goals
Slide 17
Slide 17 text
No content
Slide 18
Slide 18 text
Problem Statement
While AWS is a great place to rapidly iterate and test
new features, the vast number of accounts, instances
and usage has no easy way of attributing a running
instance back to an owner or feature.
Ownership
Slide 19
Slide 19 text
Why :: Incident Response is hard when you don’t know who owns what
Why :: If you don’t need it, why is it running?
What :: Tagging is incredibly easy to use to identify ownership
What’s missing?
Slide 20
Slide 20 text
RFCs=Tech Design
Slide 21
Slide 21 text
RFC Feedback
Not an approval process, it’s about receiving advice!
Becomes a standard through adoption @ scopes
Received comments & iterate through the draft
Slide 22
Slide 22 text
AWS Security
RFCs that we’ve written:
o AWS Standards and Best Practices
o Securing AWS environments and their
Applications
o Securing AWS data at rest
o Minimising local AWS accounts
o AWS Ownership Attribution
Slide 23
Slide 23 text
No content
Slide 24
Slide 24 text
Shrink the change => No decision paralysis
Feedback & moved to the adoption stage
Standard across Riot
Solution
Slide 25
Slide 25 text
Let’s go write some code
Cloud Inquisitor moves from ideation to implementation
Cloud Inquisitor
MurderBot
Cloud Inquisitor moved from notification into shutdownmurdering mode
Slide 28
Slide 28 text
Feels bad & yes, we received a lot of feedback
But we still work at Riot
Open & transparent Root Cause Analysis (RCA)
So, what next?
Slide 29
Slide 29 text
No content
Slide 30
Slide 30 text
Engineering
“By doing a RCA, the team has truly showed
themselves to be part of Engineering. We all make
mistakes - this is how we learn and improve.
/fistbump ”
Cam Dunn (Tech Director), Dec. 2016
Slide 31
Slide 31 text
Our communications & planning sucked
Confusion around RFC Adoption & lack of clarity on aspects of the RFC
Our notification code had bugs
Learnings
Slide 32
Slide 32 text
Improved UX, error-handling & new functionality
Manual checking of instances for important products
Over-Indexing on communication & lots of checks on alignment
Implementation
Slide 33
Slide 33 text
2nd Adoption, Yay!
bcc Engineering
“Thanks for everyone's input and consideration for
RFC0026, aka MurderBot,over the last several
weeks. This is now adopted at Riot scope.”
Mike Seavers (Director of Engineering), Feb. 2017
Slide 34
Slide 34 text
Removes incorrectly tagged & un-owned AWS objects
Checks that security features are turned on throughout our AWS Infra
DNS hijacking & IAM policy management
Cinq Features
Slide 35
Slide 35 text
No content
Slide 36
Slide 36 text
Back-End :: Modular framework (Py 3.5+), Flask , SQL Alchemy & MySQL
Front-End :: AngularJS on Nginx
Deployment :: Packer & Docker (dev only)
Cinq Tech
Slide 37
Slide 37 text
No content
Slide 38
Slide 38 text
No content
Slide 39
Slide 39 text
No content
Slide 40
Slide 40 text
No content
Slide 41
Slide 41 text
No content
Slide 42
Slide 42 text
Configuration as code Able to ship quickly
Slide 43
Slide 43 text
Problem :: Left in the corner
Solution :: Build relationships, get alignment & iterate
Problem :: Silver bullet?
Solution :: Best for the job, i.e. solve the specific problem
Problem :: Boil The Ocean
Solution :: Shrink the change – biggest impact, lowest effort possible
Takeaways