DevOps Inferno: A Journey through the 9 circles of AppSec

This talk describes specific practices, examples and steps that can be taken by operations, development and security teams to reinforce their improvement and learning cycles. By encouraging the growth of the cross-team collaboration embed in the DevOps sentiment and reflecting upon the benefits the audience will be able to reflect how to mature and continue improving practices beyond what is standard towards more specific and robust practices that enhance the security stances our products, workplaces, and communities. At the end of the slides you can find some resources that can assist you to build and achieve that awareness and context.

Transcript

1. DevOps Inferno A Journey through the 9 circles of AppSec BSides San Diego 2020 Luis Villalobos @Villaworld Photo by JR Korpa on Unsplash
2. Beware! This is not a Rant! This is a collection of developer perspectives and suggestions to overcome the bridges that prevent or hinder our collaboration regarding security efforts under a DevOps mindset. With a little pinch of drama to make it fun.
3. The Landscape • An organization where security teams are separate with almost no interaction with development teams • Teams that are being built up, and maturing as time passes
4. Quest Go to the Buildmaster Enter the black forest of secure software development Believe in the party Overcome the 9 circles of AppSec Bring back ancient knowledge
5. The Party The Developers Security Team The Ops Team The Support Team
6. Buildmaster Counsel
7. Learn to wash your hands! Photo by Dane Deaner on Unsplash
8. The Washing hands • Not Testing – Not Washing your hands • Testing – Washing your hands • Testing including baseline security – Washing your hands like WHO recommends • Context aware testing – Wash you hands as your profession/product needs.
9. The importance lies in the confidence developed But it is not the definitive answer
10. The black forest of secure software development Photo by Melissa Van Gogh on Unsplash
11. Our three beasts Security as a second class concern Give and forget awareness sessions Lack of context between teams Photo by Luke Tanis on Unsplash Photo by Kachi Awaji on Unsplash Photo by Michael LaRosa on Unsplash
12. The Black gate of Security reviews Photo by Edan Cohen on Unsplash • You deliver your first candidate to the assessment • Multiple findings
13. Limbo First Circle At the border of insight Photo by Markus Spiske on Unsplash
14. Static Analysis $ docker pull owasp/sonarqube […] $ docker run -d -p 9000:9000 -p 9092:9092 owasp/sonarqube Dynamic Analysis $ docker pull owasp/zap2docker-stable […] $ docker run -t owasp/zap2docker-stable zap-baseline.py -t https://www.example.com $ brew update && brew install dependency-check $ dependency-check -h $ dependency-check --project Testing --out . –scan ./target Dependency Check
15. Plan Develop Test Deploy We concentrate our three tools here, without the context related awareness
16. Second Circle No rest for the wicked Photo by Mārtiņš Zemlickis on Unsplash
17. The perils of blind faith Tools become checkpoints, they are perceived as taxes instead of investments Bad timing. if after the fact, findings could be left behind (exceptions) You add up to your technical debt and to that list of “technical sins”
18. Enter the value guardian
19. A code for interaction • You request a ticket with an SLA for your $BUNDLE to be tested • Boilerplates are given to kickstart an app • Thresholds are established to see if your creations are up to the test Photo by Artur Tumasjan on Unsplash
20. The road to defects is paved with good intentions (policies)
21. Plan Develop Test Deploy The test phase expands and grows into a long period of inactivity
22. Third Circle The gaze of a watchdog Photo by Geran de Klerk on Unsplash
23. The Ticket Distance phenomenon Security is calculated as an overhead to a feature in all estimations If requesting the ticket is not straightforward we promote less requests, making the cycle vicious. No interaction outside the ticket or request, we never know if it is enough
24. Possible hints to a ticket distance effect • Ivory templates lead to forgotten or misunderstood security decisions • How often and for which purpose do Dev and Security teams interact? • Are you collecting any metric, that could be used for security insight? Photo by Olivier Depaep on Unsplash
25. Embedding security into our lifecycle cannot be reduced to a checkpoint, it must be the consequence of conscious acts of risk evaluation.
26. Is this all we need?
27. A conundrum Leaning towards Security • Impact to delivery rates? • We Push to pay our tech debt Leaning towards Development • Delivery of insecure features? • We deprioritize paying the tech debt
28. Fourth Circle Pulling heavy weigths Photo by Thiago Teles on Unsplash
29. Level up cycle Awareness and training as a feature value multiplier Security practices in our day to day tasks Finding Security in our Context Look for feedback and knowledge
30. Interest is also a limited resource • Not all party members believe the same • The Project related peers might not understand • The business contact might not have sufficient information • Management might believe another metric is more important • The priority might not be what you think.
31. …Then you are thunderstruck • Thunder comes as a bug • As a new vulnerability or risk • A new unexpected compliance element • Change in regulation
32. Fifth Circle A Mosh Pit Everyone fighting each other Photo by Joel Muniz on Unsplash
33. It is not a fight. We are just stepping on each others toes.
34. Recovery should be rehearsed It builds up like a muscle.
35. Rehearsing Chaos • Certainty and understanding for supporting roles • Like a Kata, but for a synchronized team of artists • Record your findings even if they are just for yourself
36. Why should we rehearse? • Are you sure your backups are working? • Postmortems
37. Communicating Chaos • Timely, certain, profesional and proportionate communications: Due diligence and due care. • PR skills are indeed required: Even if your audience is just in- company.
38. Without numbers we won’t get support
39. Sixth Circle The Bolgias of Uncollaboration Security impact of metrics darkness. Photo by Alex Radelich on Unsplash
40. Suddenly, something happened Again…
41. Plan Develop Test Deploy What happened? After fixing the “Thunder” everyone realizes that joint monitoring and alerting could have helped understanding the context. “The Thunder”
42. Attone for the sin of having telemetry, but not using it. Awareness, Good monitoring and a rehearsed recovery plan decrease data breach cost impact
43. IBM Data Breach Report Aside from Incident Response teams • Security monitoring • Encryption • Automation • Testing IR plans • Leverage of SIEMs Not sharing our insights could have an economic impact to the businesses we support 100 110 120 130 140 150 160 170 2016 2017 2018 2019 Per record cost impact of certain practices Cost Per record (Cost - Mitigation)
44. Is my Working Feature secure? Is someone using my debugging? Is a hidden feature being used in the wild? Is someone poking around? Too many log in attempts? Are we handling error correctly? Is this the expected consumption rate? Did this happen after a deployment? Can I tell if the product is not working?
45. A Dashboard? Is that all?
46. Where to start? This is too much at once.
47. Build what you need first • Take a look into your relation with your security team • Take a look into your value stream • Lead times • Start with the first blocker in the stream • Waste • Take a look into the risks and bugs • Take a look at your knowledge
48. Seventh Circle The implementation stone ditch We will take the matter into our own hands Photo by Mariana Proença on Unsplash
49. Implementation TDS – Julien Vehent Securing the Pipeline • Define a Baseline • Drive testing from the DevOps pipeline • Never deploy sub standard code • Empower DevOps Teams to fix issues • Save time to work on the complex stuff
50. Implementation TDD – Day to day. Over the baseline • The baseline gets you started, do we really know what it is not covering? Could the baseline be enough? • What about context specific testing for your product/feature?
51. Implementation Principles • SOLID (2000) vs SDS(1974) • Testing boundaries • The guidelines are there Boundary Boundary
52. Design Architecture • DDD and ORM situation § What place does security has in Architectural practices? § A cross cutting concern
53. But, we don’t have enough time for all of this
54. Trust and Interest • Time allocation for maintenance and focused improvement becomes increasingly important • The effect of having at least a liasion for security concerns • Secure Code reviews
55. Trust and Interest • Share and be sure your sec-dev counterpart knows the value stream and context of the product: They understand your hurdles, practices and habits • Build the team awareness-knowledge cycle, Look for someone who wants to know § Managers we are looking at you
56. Trust and Interest 2.Storming 1.Forming 3.Norming 4.Performing
57. Eight Circle The release of all evil The encounter between two worlds Photo by Mazhar Zandsalimi on Unsplash
58. We don’t get to choose the priority • What do we tackle first? • How can we know if the threats are prioritized? • Do we have time for security?
59. “I don’t know” An Anonymous Business Entity
60. “This vulnerability is critical” A security assesment report
61. “It does not impact the feature” An Anonymous Business Entity
62. “We have a new finding” An Anonymous Security Entity
63. “Shouldn’t you know that?” An Anonymous Business Entity
64. “You should fix this ASAP” An Anonymous Security Entity
65. “Can’t the security team handle that?” An Anonymous Business Entity
66. Translation
67. 006-analyst.png Getting security in the sight of business • When requirements are designed, discussed or desired, include Evil Personas or misuse cases • Security Transition from NFR to feature component
68. • Security evaluations should be added to the readiness of a feature, in addition to considering the whole product without caring for just its parts • OWASP Top 10 is just the tip of the iceberg, we should be moving more into things like ASVS Getting security in the sight of business
69. 021-planning.png Getting business in the sight of security Communication goes two ways, so, it is important to include participation of the operations and security teams during planning and acceptance processes
70. Getting the right time for security • Combining the context with the vulnerabilities a feature can embed is time well spent § Continuous assessment of thresholds and threats foster the prevention of inadvertent tech debt
71. The last shape Everybody plays an important part. If we don’t collaborate the cycle will bring our problems back until we fix them. So where do we put the defense now?
72. The whole party should acknowledge priorities based on an active collaboration Awareness is not just for Devs
73. Ninth Circle Oblivion of peer and community needs The forgotten ones Photo by Dominik Dancs on Unsplash
74. How did we get here?
75. The forgotten ones • 27 Job Descriptions, 30 min search, 6 prominent software companies • Jobs span over multiple targets: Web, Mobile, OS, DB engs, SRE’s • From entry to senior level • 5 mention security tangentially • Less than 50% have the word security or secure
76. • No security relationship, expectation, path or evolution • No mention in the screening questions • Absence of Compliance and regulation impact on the job The forgotten ones
77. • If we are not faced with such questions we should ask them • It is not a gate The forgotten ones
78. Security should be embedded with the makers But we don’t have enough makers
79. Enlightment The never ending story Photo by Sergey Pesterev on Unsplash
80. Where do we find it? • It is built. Its awareness also affects managers and human resources • In proactive engagement, not after the fact • In automating after collaborating • In every day practices, not in gates • In all roles, not for the privileged few
81. What do we gain? • Improved awareness regarding what our product needs • The Security Community has a lot to teach us regarding Incidents, Risks and Threats • Less Inadvertent and Reckless Tech debt • Better plans, better execution, better products
82. The grain of hope The never ending story of human collaboration
83. Bonus Circle Leveraging communities to build knowledge Photo by Javier Allegue Barros on Unsplash
84. Building collective knowledge • Its power comes from the process of enhancing awareness into greater knowledge through a rinse and repeat effect: Collaboration at its best • At a regular cadence it shapes the awareness longer than a masterclass It can be built around motivated individuals instead of just “on the job” education or training • Be sure to measure tool usage, awareness and practices.
85. Questions? @Villaworld
86. The Written Words • The following books contain knowledge that is useful for multiple roles in organizations • The DevOps Handbook • Agile Application Security • Developing Data Intensive applications • Securing DevOps • Domain Driven Design • Clean Architecture • Test Driven Development By example • Chaos Engineering
87. Talks • These talk illustrate more focused examples of efforts that can be championed for several stages and situations in the SDLC • Variant Analysis - https://youtu.be/6WwP7eUY52Y • Building an AppSec Program with a Budget of $0: Beyond the OWASP Top 10 - https://youtu.be/5RmHQKeXgk4 • Attack Driven Development: Getting Started in Application Security - https://youtu.be/dOm_hWk1OFU • Threat modeling at speed & scale - https://youtu.be/5jyL-CHib54 • AppSec: From the OWASP Top Ten(s) to the OWASP ASVS - https://youtu.be/nvzMN5Z8DJI • Data Driven Post Mortems - https://www.youtube.com/watch?v=46 HbRGAxCwE • Test Driven Security - https://www.youtube.com/watch?v=1N lbf7XXn7s
88. Resources • These are resources and tools that could be used to improve our security stance • https://github.com/OWASP/sonarqube • https://www.zaproxy.org/ • https://owasp.org/www-project-dependency-check/- • https://github.com/OWASP/user-security-stories • https://www.checkmarx.com/2017/04/20/six-steps-secure- software-development-agile-era/ • https://snyk.io/blog/sql-injection-orm-vulnerabilities/ • https://www.oracle.com/technetwork/java/seccodeguide- 139067.html#0 • https://www.owasp.org/images/5/53/OWASP_Code_Review_Gu ide_v2.pdf • https://blog.rapid7.com/2020/01/14/how-to-define-and- communicate-vulnerability-risk-across-your-company/ • https://coreruleset.org/ • https://martinfowler.com/bliki/TestPyramid.html • https://portswigger.net/web-security • https://www.ibm.com/security/data-breach • https://www.martinfowler.com/bliki/TechnicalDebtQuadrant.ht ml • https://github.com/OWASP/www-project-proactive- controls/blob/master/v3/OWASP_Top_10_Proactive_Contr ols_V3.docx • https://about.gitlab.com/blog/2017/02/01/gitlab-dot-com- database-incident/ • https://github.com/threatspec/threatspec • https://www.cs.virginia.edu/~evans/cs551/saltzer/
89. Special Thanks • For the icons: • Internet Security Icon pack made by Freepik from Flaticon https://www.flaticon.com/packs/internet-security-2 • Agile Methodology Icon pack made by Eucalyp from Flaticon https://www.flaticon.com/packs/agile-methodology