P S E C , B U T F O R S T A R T U P S TWUBHUBOOK Hello all, We’re here to tell you about a ﬁctitious startup from the future. You may be wondering about the watermark in this slide. Well it’s our logo, the narwhal. Any good startup logo has to indicate growth and that’s why its horn goes “up and to the right”. Today I’m wearing an alternate version of our logo, the narwhal spelled gnarwhal. We’re going to cover a lot of topics today so I want to encourage thought provoking questions. We don’t have time to explain everything in detail so I want to encourage people to ask questions that may help other people understanding better. As a rule of thumb ask “will this help my understanding or is this is an important question to address for everyone’s clarity”. As a reward, I have bags of gold: 100% kona private reserve coﬀee. Questions are welcome at any time, but please do not interrupt. Think of it like a bug bounty for this presentation!
the fun things LA has to offer while he longs for the days when he can move back home. Finally has a twitter account. a.k.a. @brentjo on GitHub @gsmbj on twitter Brent Johnson TROJAN. BOUNTY TRIAGE EXPERT. BUSINESS LOGIC FLAWS ARE HIS FRIEND. Hi everyone, I’m Brent and I’m ﬁnishing up my last semester of undergrad right across town at USC. I worked for GitHub over the last half of 2016, where I did lots of bug bounty triage, code review, and some development on their automated security scanner. I’m excited to ﬁnish up my degree and move back to the bay area where I’ll begin working for GitHub full time.
RANGER. DOES NOT LIKE COMPUTERS. Hello everyone, I’m Neil. I have a long history with OWASP, Los Angeles, and appsec California. I’d like to thank the organizers for giving me the opportunity to expense a trip back here.
B H U B B O O K Greenfield In many disciplines a greenfield project is one that lacks constraints imposed by prior work. The analogy is to that of construction on greenfield land where there is no need to work within the constraints of existing buildings or infrastructure - Wikipedia Let's start with arbitrary classiﬁcations of startup maturity levels. Greenﬁeld projects are the ideal scenario since by deﬁnition you have not accrued any technical debt that could make your life a living hell later on. Features haven’t been ﬂeshed out, technologies haven’t been chosen. Security is almost never involved at this point.
B H U B B O O K C O N T ’ D Young application Think pre-pre-pre-pre-pre-IPO. Young applications are the class of apps that may have some serious amounts of code written, but maybe haven’t “hit it big” or made its mark yet. It’s also pretty rare for security to be involved in this phase.
N T H E J O B Oddly, the mannequin challenge is still even in 2025. (B) Setup ﬁctitious tbook scenario • We’ll be following the story of a ﬁctitious company called Twubhubbook. • Twubhubbook is a very young app just past the Greenﬁeld stage. • Luckily the founders have made securing data a core business practice. • In this scenario, two contractors are hired. One to manage application security and a separate contractor to manage ops. Both are part time and both report to directly the VP of engineering. • Tbook also has an existing engineering team that is capable and on board with doing the right thing • While engineering and security have their disagreements, this relationship works out well enough that it wouldn’t be described as “horribly broken
Review architecture | Code review culture (neil) • Always be upgrading • This may be the hardest to accomplish, I’ve only ever seen this happen once at a company of about 15 engineers • Getting stuck on older versions of libraries often means you cannot leverage the best of what is available and you have to deal with code rot • Vulnerable APIs don’t get patched. • Developers may have more familiarity with newer, patched versions and make misguided assumptions about the safety of an API • You get to take advantage of newer integrations like Sprockets + SRI. • Unsupported code greatly increases the likelihood that you’ll be left backporting the patch. • E.g. per-action CSRF tokens in rails. • more work on the appsec person’s plate
I N E S S O F P R E V E N T I O N SECURITY DOES NOT HAVE TIME TO FIX OR FIND BUGS (neil) • Security doesn’t have time to ﬁx bugs and they don’t have much time to ﬁnd bugs, so prevention is the best option. • Design with “secure by default” in mind • Opt out of security, not in • Make opting out of security a very conscious decision • BypassAllAuthorizationBecauseIKnowThisIsSafe • OptOutOfAllProtection • Realize this will only have moderate adoption at this time. Secure by default not fully embraced by engineers. • One debatable point, when auto escaping templates are not used, is how aggressive do you escape data in templates? Numbers? Constants? All the things? • Numbers can become strings. • Constants can become dynamic. • Another note on escaping. Contextual encoding is all the rage, and has had success, but I consider it an anti pattern. Only use on context: the html context. Encoding for anything else is a smell and should only be used in legacy applications. Come at me. • Rollout all security headers • Do your best to support fewer browsers. Don’t support antiquated browsers, but do be sensitive to the needs of your users. GitHub basically only cares about current versions of chrome and that makes our lives so much easier. • Launch bug bounty • Bug bounties show patterns of issues that can be turned into learning opportunities and are often overlooked. • “The best indicator of the next bug is the last bug” - Alex Smolen
so far? • We’ve instantly scaled the security team via the bug bounty • We have some level of assurance that code has been checked for security defects by humans and robots • In theory, the product accrues less technical debt • Through our Content Security Policy, we have some control over what 3rd party integrations are added. At the very least more thought and discussion have to take place. • We have a plan for triaging and remediating issues through experience with the bug bounty • We have faith that we don’t need to re architect our app for security reasons for a very long time • We’ve also eliminated or limited classes of bugs with headers and cookie settings • We’ve limited cross site scripting via our Content Security Policy. • Our script-src directive is pretty much as strict as possible per spec • We’ve set object-src to ‘none’ to disable the use of ﬂash/java. • We’ve enabled subresource integrity which ensures sources from our CDN match precomputed integrity values • This means that a compromise of your content delivery network is limited. • Eliminated browser traﬃc sniﬃng over plaintext, • We’re on the HSTS preload lists so HTTPS is baked into the browser for your site. • No never want any http plaintext connections to out hosts • Only including HTTPS links in our Content security policy hosts ensures this. • We’ve limited browser https MITM via public key pinning.
N T H E J O B (B) • Don’t underestimate the importance of building a happy healthy culture. We want to establish a “vulns are just bugs” culture. They should be triaged, prioritized, and tested just like other bugs. • Security bugs shouldn’t be grounds for shaming. • Don’t punish the person who wrote the code to ﬁx the issue or do any investigation. This person is already under enough stress. • After all, everybody has written vulnerable code in the real world. • When analyzing the bug we should ask questions such as: • Can we delete this code? • Does the framework need hardening? • Is there a pattern of mistakes that we can account for? • Is this actually security by default? • How can we detect this earlier? • How can we socialize this issue to bring awareness? • People also shouldn’t be thinking in terms of “us” or “them”, it should be a “we” • You could take the initiative and join meetings • Or provide engineering support for non-security related coding eﬀorts. • Always congratulate people on shipping code • For a team that’s involved in almost every project, we don’t congratulate people nearly enough • You don't want to fall into the mindset of treating every new project or feature as something that’s going to make your life more diﬃcult, treat it as something that's good for everyone. Unless it’s actually just terrible.
a milestone • Tbook is referred to as a unicorn. • Tbook is growing faster than it can manage. • It’s time to hire a team. • Contractors help manage the transition, possibly staying on as an advisor or joining the team fully. • The team plans splitting into security ops and appsec. • Appsec becomes it’s own entity, roles are more clearly deﬁned, people outside of the security org have a better idea of who might be able to help out on a speciﬁc task.
SCALE CULTURE MUST STAY STRONG FLEXIBILITY IS IMPORTANT (neil) • Culture is fading, quality drops, bugs surface • Illustrate falling quality as a culture problem and all of a sudden it’s hard to argue against • People don’t fail, processes do. Bad processes are a result of a fading culture. Fading culture can be addressed with better processes. • Conversations with security are not happening as frequently • This does not mean any exclusion is intentional • Hive knowledge might not be comprehensive • Risk assessments may be in high disagreement • Native code is frequently used to enhance critical paths. Security team outsources native code assessments since it’s not their thing and they haven’t hired for it yet.
incident” • Tbook has its ﬁrst major incident • Good things come out of incidents • Talk about Tweetdeck incident • If emoji, then don’t escape • Content security policy eﬀort had stalled out months before • Incident meant it was implemented (IIRC fairly quickly. Jim?) • Create an issue, chat room, open video conference • The investigation is not about the blame game, but to ﬁnd where things went wrong • “People don’t fail, processes do” again? • Talk about the bug that caused the private repo disclosure, nobody could have seen that one coming • Fix the problem • Diagnose impact • Notify the aﬀected • Write up a detailed public account of what happened • People appreciate transparency and details • Prevent the same mistake from happening again
We’ve been through our ﬁrst incident. We now know where the gaps in our planning are. • Through the post mortem, process failures are identiﬁed and remedied. • The company had a rude awakening and interest in securing all the things is at an all time high. • But throughout the whole process we’ve been very sensitive not to enact security theater • Engineers, sympathetic to the appsec cause push for change. Many make direct contributions during this push for “catching up” with our technical debt.
O F ? ? ? Mommy, wow! I’m a big kid now. (B) The security program grows up • Day to day tasks are well deﬁned • We’ve crossed the “year of Content security policy” and • All internal and external apps have Content security policy • Other critical security roles such as the incident response team (SIRT), government/risk management/compliance team (GRC), network engineering, IT, are doin’ their thing
STANDARDS BODIES | SHARED RESPONSIBILITY (neil) • Tbook security program reaches a new level of maturity • Framework hardening is a common task • Code and data are separated :hooray: • Logicless templates were a success • (front end) HTML is not be generated via string concatenation (theoretical) • Interpolated SQL will not run (theoretical) • Work with standards bodies to augment and enhance specs is done in a strategic fashion. E.g. • Require-sri-for • origin-when-cross-origin referrer policy • Content security policy hashes • Nonces didn’t work for Twitter’s use case (html fragment that was cached) • Content security policy hashes for external sources (actually a thing of the future!) • Appsec is fully integrated into the SDLC • Arch reviews • Code review • Automated review • Non-appsec peer review • Beta tested with employees and bounty researchers • Shared responsibility amongst the team is clear
• The bug bounty starts to provide less quality reports • Low hanging fruit is gone • Appsec team announces bounty promotional program • Going into this idea, not all of us were convinced we should do this • We considered scenarios where the low traﬃc continued and the winner would get 12k for a referrer leak • We considered scenarios where one person has dozens of bugs but only feels like reporting 3 • We consider scenarios where one person won all awards • But in every single case, it was hard to argue that it isn’t a net win for GitHub and that’s really all that mattered • I can’t reveal any details at the moment, but I can say it has been an overwhelming success and it will be repeated.
(neil) • Tbook prepares to IPO • Company doubles in size year over year. • Security role starts to get a little more... compliancy • Security team grows at a slower pace. • Having run out of time to implement key strategic technologies, acquisitions become more common • Cultures suﬀers again • Security team comes up with a diligence plan for acquisitions • Do they even security? If not, your job just got harder. • Security team? • Bug bounty? • Content security policy? • Password storage? • Network segmentation? • Datacenter redundancy? • How are credentials managed? • Operational maturity? • Incident response? • Is this an acquihire or a product augmentation? • Is the service going to continue to run?
COLLEGE — CAN YOU REVIEW THIS MIPS CODE OUR CORE BUSINESS OPS NOW DEPEND ON? Stack Diversifies (neil) • Tbook’s development stack diversiﬁes through acquisitions and need to be redesigned • Tbook appsec team cries, then accepts, then disregards the new stacks while crossing ﬁngers.
THE FRAMEWORK, DESIGN SERVICES SECURELY. • TBook’s development stack consolidates • The time for SOA is here • The security team works very closely with the team making the decision. • We probably won’t get much say, but we can ﬁgure out how to secure it. • Be hardasses on the “secure by default” thing here • Seriously, this is your last chance to change things for the better. • Harden the shit out of the framework • Headers by default • e.g. Content security policy strict mode, no exceptions • Design services… securely • Make scary APIs even more scary. Make it hard for your engineers to do the wrong thing. • Have a plan for automation • Designing APIs securely means you can stop relying on static analysis and use GREP instead.
Tbook eventually moves beyond basic appsec and into business logic. • The Appsec team is implementing specs designed earlier, essentially acting as beta testers of the internet • Unit tests for all authorization checks are required and negative test cases are added • Diﬀerent roles are added for users because not everyone has the same account needs • Billing managers, marketing, PR, etc don’t need full access • Risk-based rights are applied to user accounts • A “read only” mode is introduced for those suspected of being compromised • High proﬁle accounts receive extra scrutiny • Automated analysis of user activity is done to expose patterns of abuse • Log correlation and machine learning help classify activity • Alerts and dashboards around identiﬁed activities keep incident command people informed • It’s important to understand how people are abusing your site or where they are focused • Then combine this with risk analysis and you’ll know where you need to prioritize your eﬀorts. • Lastly, at this point, account recovery mechanisms are secure and usable