P S E C , B U T F O R S T A R T U P S TWUBHUBOOK Hello all, We’re here to tell you about a fictitious startup from the future. You may be wondering about the watermark in this slide. Well it’s our logo, the narwhal. Any good startup logo has to indicate growth and that’s why its horn goes “up and to the right”. Today I’m wearing an alternate version of our logo, the narwhal spelled gnarwhal. We’re going to cover a lot of topics today so I want to encourage thought provoking questions. We don’t have time to explain everything in detail so I want to encourage people to ask questions that may help other people understanding better. As a rule of thumb ask “will this help my understanding or is this is an important question to address for everyone’s clarity”. As a reward, I have bags of gold: 100% kona private reserve coffee. Questions are welcome at any time, but please do not interrupt. Think of it like a bug bounty for this presentation!
the fun things LA has to offer while he longs for the days when he can move back home. Finally has a twitter account. a.k.a. @brentjo on GitHub @gsmbj on twitter Brent Johnson TROJAN. BOUNTY TRIAGE EXPERT. BUSINESS LOGIC FLAWS ARE HIS FRIEND. Hi everyone, I’m Brent and I’m finishing up my last semester of undergrad right across town at USC. I worked for GitHub over the last half of 2016, where I did lots of bug bounty triage, code review, and some development on their automated security scanner. I’m excited to finish up my degree and move back to the bay area where I’ll begin working for GitHub full time.
RANGER. DOES NOT LIKE COMPUTERS. Hello everyone, I’m Neil. I have a long history with OWASP, Los Angeles, and appsec California. I’d like to thank the organizers for giving me the opportunity to expense a trip back here.
B H U B B O O K Greenfield In many disciplines a greenfield project is one that lacks constraints imposed by prior work. The analogy is to that of construction on greenfield land where there is no need to work within the constraints of existing buildings or infrastructure - Wikipedia Let's start with arbitrary classifications of startup maturity levels. Greenfield projects are the ideal scenario since by definition you have not accrued any technical debt that could make your life a living hell later on. Features haven’t been fleshed out, technologies haven’t been chosen. Security is almost never involved at this point.
B H U B B O O K C O N T ’ D Young application Think pre-pre-pre-pre-pre-IPO. Young applications are the class of apps that may have some serious amounts of code written, but maybe haven’t “hit it big” or made its mark yet. It’s also pretty rare for security to be involved in this phase.
B H U B B O O K C O N T ’ D Mature application (Neil) Most of us will join a startup at this phase. The app is established, technology may be settled upon, and there are enough users where uptime, reliability, and feature development may override security decisions. To state the obvious, • Security is trivial to build in, excruciatingly painful to bolt on. • For the “young” applications, I’ve advised an appsec program on 1-3 hours a week. Beyond an initial flurry of work in the beginning, their security is in maintenance mode. It’s no where near perfect (hello unsafe eval), but the gains are becoming less important, and plans to massage away blockers are in place. Putting the practices we will discuss in place is very cheap when you’re a young company. • One example of problems that can arise when bolting on security is you may have front end javascript frameworks that just straight up are not compatible with strict content security policy. • Culture and architectural decisions can really cost you and put you in difficult spots. • The earlier you think about security, the better prepared you are for every stage of startup life. • No firedrills to become compliant. • And A decent security program is something that pretty much has to be in place pre-IPO.
N T H E J O B Oddly, the mannequin challenge is still even in 2025. (B) Setup fictitious tbook scenario • We’ll be following the story of a fictitious company called Twubhubbook. • Twubhubbook is a very young app just past the Greenfield stage. • Luckily the founders have made securing data a core business practice. • In this scenario, two contractors are hired. One to manage application security and a separate contractor to manage ops. Both are part time and both report to directly the VP of engineering. • Tbook also has an existing engineering team that is capable and on board with doing the right thing • While engineering and security have their disagreements, this relationship works out well enough that it wouldn’t be described as “horribly broken
current | Review architecture | Code review culture (neil) In no particular order, here are 4 very important things to think about early on. • Establish a code review culture. • Ask “what could go wrong” with every pull request • So many problems can be avoided by asking the right questions and this one does not get asked enough. • Ensure that all code receives human eyes for improvements or assurance, nobody is allowed to ship code without someone else's approval. • Let people know on day 1 that they should not hesitate to ping security if something seems fishy, whether it’s their own code or someone else’s • Integrate automation with code review, raise a flag if anything suspicious is detected. Often a regex will get you very far. Watching files for changes is also trivial to implement and very powerful. Static analysis is even better, but is sometimes unnecessary. • file watches • Critical files/code is touched (auth’n/z) • regular expressions • File operations • Inline javascript • Shelling out • Static analysis • Interpolated SQL with user input leading to sqli • Unescaped output leading to xss
| Review architecture | Code review culture (neil) • Review or change architecture • I’m going to over generalize and say there are two architectures • The traditional App where html is generated on the server, often with separate mobile apps • And an API-based app where rendering is done on the client • API client-based apps have benefits traditional apps do not. • You can focus on building a secure API and then securing each client • No separate authorization framework needed • Typically an SPA app with an API will use localStorage for auth creds • This reduces CSRF risk but the threat of XSS is greater since it must be accessible by javascript. • Because cookies are not used, Content security policy is a must here • Good news, it’s just one page! • Move the admin panel and all non-consumer facing end points behind VPN • This gives you a layer of protection and can make it much harder to find bugs in the non-consumer facing endpoints • Think about how are the origins setup and how they interact • E.g. github-pages-site.github.io • raw.githubusercontent.com • Consider logic less templates • e.g. mustache
| Review architecture | Code review culture (neil) • Decide on an allowed/banned technology list and include this lesson as part of on boarding • Flash • Flash has had many critical security flaws. The industry is moving away from flash. • JSONP • JSONP is a hack that has been solved by using CORS. JSONP has been used for many XSS and content-sniffing attacks. JSONP is a scar on the internet. • Inline javascript • First of all, it’s ugly and hard to manage • Content security policy is a lot easier to apply when you don’t have any • eval (front end and backend) • To put it one way, using eval is like shipping code that has never been peer reviewed. • eval is not faster than JSON.parse, even if it is I don’t care • Dynamic javascript • Do not have code that writes code. Dynamic javascript is inherently dangerous. • This is analogous to using eval in a really roundabout way • all of the previously mentioned things are detectable via regex that can be caught in code review by a bot
Review architecture | Code review culture (neil) • Always be upgrading • This may be the hardest to accomplish, I’ve only ever seen this happen once at a company of about 15 engineers • Getting stuck on older versions of libraries often means you cannot leverage the best of what is available and you have to deal with code rot • Vulnerable APIs don’t get patched. • Developers may have more familiarity with newer, patched versions and make misguided assumptions about the safety of an API • You get to take advantage of newer integrations like Sprockets + SRI. • Unsupported code greatly increases the likelihood that you’ll be left backporting the patch. • E.g. per-action CSRF tokens in rails. • more work on the appsec person’s plate
I N E S S O F P R E V E N T I O N SECURITY DOES NOT HAVE TIME TO FIX OR FIND BUGS (neil) • Security doesn’t have time to fix bugs and they don’t have much time to find bugs, so prevention is the best option. • Design with “secure by default” in mind • Opt out of security, not in • Make opting out of security a very conscious decision • BypassAllAuthorizationBecauseIKnowThisIsSafe • OptOutOfAllProtection • Realize this will only have moderate adoption at this time. Secure by default not fully embraced by engineers. • One debatable point, when auto escaping templates are not used, is how aggressive do you escape data in templates? Numbers? Constants? All the things? • Numbers can become strings. • Constants can become dynamic. • Another note on escaping. Contextual encoding is all the rage, and has had success, but I consider it an anti pattern. Only use on context: the html context. Encoding for anything else is a smell and should only be used in legacy applications. Come at me. • Rollout all security headers • Do your best to support fewer browsers. Don’t support antiquated browsers, but do be sensitive to the needs of your users. GitHub basically only cares about current versions of chrome and that makes our lives so much easier. • Launch bug bounty • Bug bounties show patterns of issues that can be turned into learning opportunities and are often overlooked. • “The best indicator of the next bug is the last bug” - Alex Smolen
so far? • We’ve instantly scaled the security team via the bug bounty • We have some level of assurance that code has been checked for security defects by humans and robots • In theory, the product accrues less technical debt • Through our Content Security Policy, we have some control over what 3rd party integrations are added. At the very least more thought and discussion have to take place. • We have a plan for triaging and remediating issues through experience with the bug bounty • We have faith that we don’t need to re architect our app for security reasons for a very long time • We’ve also eliminated or limited classes of bugs with headers and cookie settings • We’ve limited cross site scripting via our Content Security Policy. • Our script-src directive is pretty much as strict as possible per spec • We’ve set object-src to ‘none’ to disable the use of flash/java. • We’ve enabled subresource integrity which ensures sources from our CDN match precomputed integrity values • This means that a compromise of your content delivery network is limited. • Eliminated browser traffic sniffing over plaintext, • We’re on the HSTS preload lists so HTTPS is baked into the browser for your site. • No never want any http plaintext connections to out hosts • Only including HTTPS links in our Content security policy hosts ensures this. • We’ve limited browser https MITM via public key pinning.
N T H E J O B (B) • Don’t underestimate the importance of building a happy healthy culture. We want to establish a “vulns are just bugs” culture. They should be triaged, prioritized, and tested just like other bugs. • Security bugs shouldn’t be grounds for shaming. • Don’t punish the person who wrote the code to fix the issue or do any investigation. This person is already under enough stress. • After all, everybody has written vulnerable code in the real world. • When analyzing the bug we should ask questions such as: • Can we delete this code? • Does the framework need hardening? • Is there a pattern of mistakes that we can account for? • Is this actually security by default? • How can we detect this earlier? • How can we socialize this issue to bring awareness? • People also shouldn’t be thinking in terms of “us” or “them”, it should be a “we” • You could take the initiative and join meetings • Or provide engineering support for non-security related coding efforts. • Always congratulate people on shipping code • For a team that’s involved in almost every project, we don’t congratulate people nearly enough • You don't want to fall into the mindset of treating every new project or feature as something that’s going to make your life more difficult, treat it as something that's good for everyone. Unless it’s actually just terrible.
O S A Y “ B E C A R E F U L ” SECURITY IS NOT HERE TO SAY “NO” (B) The over-arching theme of your culture should be that security is not here to say no, we’re here to say be careful.
a milestone • Tbook is referred to as a unicorn. • Tbook is growing faster than it can manage. • It’s time to hire a team. • Contractors help manage the transition, possibly staying on as an advisor or joining the team fully. • The team plans splitting into security ops and appsec. • Appsec becomes it’s own entity, roles are more clearly defined, people outside of the security org have a better idea of who might be able to help out on a specific task.
SCALE CULTURE MUST STAY STRONG FLEXIBILITY IS IMPORTANT (neil) • Culture is fading, quality drops, bugs surface • Illustrate falling quality as a culture problem and all of a sudden it’s hard to argue against • People don’t fail, processes do. Bad processes are a result of a fading culture. Fading culture can be addressed with better processes. • Conversations with security are not happening as frequently • This does not mean any exclusion is intentional • Hive knowledge might not be comprehensive • Risk assessments may be in high disagreement • Native code is frequently used to enhance critical paths. Security team outsources native code assessments since it’s not their thing and they haven’t hired for it yet.
incident” • Tbook has its first major incident • Good things come out of incidents • Talk about Tweetdeck incident • If emoji, then don’t escape • Content security policy effort had stalled out months before • Incident meant it was implemented (IIRC fairly quickly. Jim?) • Create an issue, chat room, open video conference • The investigation is not about the blame game, but to find where things went wrong • “People don’t fail, processes do” again? • Talk about the bug that caused the private repo disclosure, nobody could have seen that one coming • Fix the problem • Diagnose impact • Notify the affected • Write up a detailed public account of what happened • People appreciate transparency and details • Prevent the same mistake from happening again
We’ve been through our first incident. We now know where the gaps in our planning are. • Through the post mortem, process failures are identified and remedied. • The company had a rude awakening and interest in securing all the things is at an all time high. • But throughout the whole process we’ve been very sensitive not to enact security theater • Engineers, sympathetic to the appsec cause push for change. Many make direct contributions during this push for “catching up” with our technical debt.
O F ? ? ? Mommy, wow! I’m a big kid now. (B) The security program grows up • Day to day tasks are well defined • We’ve crossed the “year of Content security policy” and • All internal and external apps have Content security policy • Other critical security roles such as the incident response team (SIRT), government/risk management/compliance team (GRC), network engineering, IT, are doin’ their thing
STANDARDS BODIES | SHARED RESPONSIBILITY (neil) • Tbook security program reaches a new level of maturity • Framework hardening is a common task • Code and data are separated :hooray: • Logicless templates were a success • (front end) HTML is not be generated via string concatenation (theoretical) • Interpolated SQL will not run (theoretical) • Work with standards bodies to augment and enhance specs is done in a strategic fashion. E.g. • Require-sri-for • origin-when-cross-origin referrer policy • Content security policy hashes • Nonces didn’t work for Twitter’s use case (html fragment that was cached) • Content security policy hashes for external sources (actually a thing of the future!) • Appsec is fully integrated into the SDLC • Arch reviews • Code review • Automated review • Non-appsec peer review • Beta tested with employees and bounty researchers • Shared responsibility amongst the team is clear
• The bug bounty starts to provide less quality reports • Low hanging fruit is gone • Appsec team announces bounty promotional program • Going into this idea, not all of us were convinced we should do this • We considered scenarios where the low traffic continued and the winner would get 12k for a referrer leak • We considered scenarios where one person has dozens of bugs but only feels like reporting 3 • We consider scenarios where one person won all awards • But in every single case, it was hard to argue that it isn’t a net win for GitHub and that’s really all that mattered • I can’t reveal any details at the moment, but I can say it has been an overwhelming success and it will be repeated.
(neil) • Tbook prepares to IPO • Company doubles in size year over year. • Security role starts to get a little more... compliancy • Security team grows at a slower pace. • Having run out of time to implement key strategic technologies, acquisitions become more common • Cultures suffers again • Security team comes up with a diligence plan for acquisitions • Do they even security? If not, your job just got harder. • Security team? • Bug bounty? • Content security policy? • Password storage? • Network segmentation? • Datacenter redundancy? • How are credentials managed? • Operational maturity? • Incident response? • Is this an acquihire or a product augmentation? • Is the service going to continue to run?
COLLEGE — CAN YOU REVIEW THIS MIPS CODE OUR CORE BUSINESS OPS NOW DEPEND ON? Stack Diversifies (neil) • Tbook’s development stack diversifies through acquisitions and need to be redesigned • Tbook appsec team cries, then accepts, then disregards the new stacks while crossing fingers.
THE FRAMEWORK, DESIGN SERVICES SECURELY. • TBook’s development stack consolidates • The time for SOA is here • The security team works very closely with the team making the decision. • We probably won’t get much say, but we can figure out how to secure it. • Be hardasses on the “secure by default” thing here • Seriously, this is your last chance to change things for the better. • Harden the shit out of the framework • Headers by default • e.g. Content security policy strict mode, no exceptions • Design services… securely • Make scary APIs even more scary. Make it hard for your engineers to do the wrong thing. • Have a plan for automation • Designing APIs securely means you can stop relying on static analysis and use GREP instead.
Tbook eventually moves beyond basic appsec and into business logic. • The Appsec team is implementing specs designed earlier, essentially acting as beta testers of the internet • Unit tests for all authorization checks are required and negative test cases are added • Different roles are added for users because not everyone has the same account needs • Billing managers, marketing, PR, etc don’t need full access • Risk-based rights are applied to user accounts • A “read only” mode is introduced for those suspected of being compromised • High profile accounts receive extra scrutiny • Automated analysis of user activity is done to expose patterns of abuse • Log correlation and machine learning help classify activity • Alerts and dashboards around identified activities keep incident command people informed • It’s important to understand how people are abusing your site or where they are focused • Then combine this with risk analysis and you’ll know where you need to prioritize your efforts. • Lastly, at this point, account recovery mechanisms are secure and usable
oreoshake
hatena
cmtamai
utam0k
kawaguti
ishiayaya
asei
matsuihidetoshi
you
cmsuzu
sogaoh
moongift
sstephenson
brad_frost
brianwarren
reverentgeek
jasonvnalue
dougneiner
jponch
ufuk
lynnandtonic
productmarketing
aarron
roundedbygravity
