Real World Security Testing

Testing for security is an increasingly important and visible part of software delivery. The classic formula for security testing isn’t working and security analysis tools can’t reliably find certain basic problems. Matt introduces concrete things testing teams can do to contribute to the security of a system and identifies opportunities to include more advanced manual testing of scenarios that are often overlooked. Process improvements that range from articulating security acceptance criteria to checklists are discussed. In some cases, test automation can be used to raise the security of delivered software. In all cases, the villain persona and negative testing scenarios are foundational to effective security testing. In addition to specific actionable security testing strategies, one goal of the talk will be to provide some high level context for thinking about security testing and how to integrate it into the software development lifecycle effectively. Take the security of your applications to the next level and be more confident.

Transcript

1. Real World
   Security Testing
   Matt Konda
   @mkonda
   

   View Slide

2. Introduction
   1997 2006 2014
   Consultant
   Engineer
   Software
   Architect
   Director of
   Engineering
   Rabble Rouser:
   Perl
   Java Applet
   C++
   J2EE
   J2EE

   Spring
   Analytics
   Certificate Authority
   Vulnerability Scanner
   Penetration Test Manager
   Pricing
   Retail
   Banking
   Manufacturing
   Pharma
   Healthcare
   Research
   Ruby
   Rails
   Chicago BSides 2011, 2012
   Defcon Skytalk
   OWASP Chicago, MSP 2013
   AppSec USA 2012, 2013
   ChicagoRuby 2013
   Secure 360
   Lone Star Ruby 2013
   WindyCityRails 2013
   Chicago JUG 2014
   RailsConf 2014
   Converge 2014
   Chicago Coder Conference 2015
   MS in CS
   Founder
   Consultant
   Agile
   Clojure
   Graph Database
   Trying to hack a
   business model that
   succeeds while
   helping developers.
   Domains:
   Projects:
   DevOps / Automation
   Training
   Coaching
   Code Review
   Plugged in to SDLC
   Consulting
   Assessments
   @mkonda
   mkonda@jemurai.com
   DevOps
   Growing
   

   View Slide

3. View Slide

4. View Slide

5. View Slide

6. View Slide

7. View Slide

8. Why are we here?
   

   View Slide

9. View Slide

10. You don’t want
    
    to be on the list.
    

    View Slide

11. ARE
    STAKEHOLDERS
    ASKING FOR
    SECURITY?
    

    View Slide

12. In many cases, stakeholders
    are not in good positions to
    define security requirements.
    

    View Slide

13. In many cases, testing teams
    are in a good position to see
    mismatched expectations.
    

    View Slide

14. And let’s be honest.
    

    View Slide

15. We don’t know how to
    stop it.
    

    View Slide

16. But we know it costs
    

    View Slide

17. And might cost some
    people their
    

    View Slide

18. But its also because
    there is no other place to
    find these security issues.
    

    View Slide

19. Adversary?
    

    View Slide

20. Villain
    

    View Slide

21. View Slide

22. Agenda
    • Intro (We just did this)
    • Talking about Testing
    • Talk about scenarios from tester perspectives
    • Understanding security tools (and limitations)
    • Building Security Into Your SDLC
    • Wrap up
    

    View Slide

23. Testers
    

    View Slide

24. Manual testers
    

    View Slide

25. View Slide

26. View Slide

27. Exploratory testing
    

    View Slide

28. http://testobsessed.com/wp-content/uploads/2011/04/testheuristicscheatsheetv1.pdf
    

    View Slide

29. http://www.jemurai.com/images/SecurityTestingCheatSheet.pdf
    

    View Slide

30. Villain
    

    View Slide

31. View Slide

32. The takeaway is people
    trained and manually looking
    for specific things is powerful.
    

    View Slide

33. View Slide

34. View Slide

35. View Slide

36. View Slide

37. View Slide

38. View Slide

39. View Slide

40. View Slide

41. View Slide

42. View Slide

43. Now take automated
    testing and developer
    testing.
    

    View Slide

44. The tools make it easier
    to get better coverage for
    certain things.
    

    View Slide

45. View Slide

46. We still see mostly
    happy path testing.
    

    View Slide

47. The challenge is the same:
    education and time
    

    View Slide

48. So given both:
    Manual testing
    Automated testing
    

    View Slide

49. Let’s step back and think
    of two specific cases…
    

    View Slide

50. 1. Injection
    2. Fraud
    

    View Slide

51. Injection
    • Attacker supplies unexpected input that gets
    executed at some point.
    • Can result in system commands running, data
    being dumped, and code running in places it
    shouldn’t.
    

    View Slide

52. Injection
    • Can often be detected by looking at code.
    • Can also be detected by testing different inputs
    and seeing what happens in relatively predictable
    ways.
    • In short, tools (both security and test automation)
    can often find this.
    

    View Slide

53. Fraud
    

    View Slide

54. Fraud
    • Of course there are many many different paths
    through which fraud can take place.
    • Consider: An attacker uses someone else’s
    account info that was leaked in a public breach to
    make a transaction and send the purchase to a
    new address.
    

    View Slide

55. Fraud
    • In very few cases can fraud be detected by
    security tools. At least not without big custom
    implementation work. (Machine learning +
    behavioral analysis)
    • A tester could try adding a purchase to a new
    shipping address and verify that the application
    prompts for the user’s credit card. (What Amazon
    does)
    

    View Slide

56. The interesting thing to think about here is:
    who is better situated to see fraud use
    cases than testers?
    

    View Slide

57. View Slide

58. Villain
    

    View Slide

59. These are just brief
    examples to illustrate a
    point.
    

    View Slide

60. Next we’ll go deeper.
    

    View Slide

61. Scenarios
    

    View Slide

62. Cross Site Scripting (XSS)
    

    View Slide

63. View Slide

64. XSS is code injection
    

    View Slide

65. Malicious user input gets treated like
    code by the browser.
    

    View Slide

66. Comes in different flavors
    

    View Slide

67. Reflected XSS
    • User input is displayed back to the logged in user.
    • A classic example is a search box.
    • You search for XYZ but we couldn’t find it.
    OR
    • Here are your results for XYZ
    

    View Slide

68. Demo with simple Node.js handlebars
    application.
    

    View Slide

69. Persistent XSS
    • User input is saved for later in a DB or
    storage.
    • When later shown it is treated like code.
    

    View Slide

70. The correct way to prevent XSS is to
    ensure all output is ENCODED.
    

    View Slide

71. Output Encoding
    < <
    > >
    

    View Slide

72. What is encoding?
    alert(‘hi’);
    Becomes:
    <script>alert(‘hi’);</script>
    It is the difference between
    rendering and evaluating.
    

    View Slide

73. You might ask about input validation...
    

    View Slide

74. It can help, but with data coming from
    many different places the correct way
    to prevent XSS is with output encoding.
    

    View Slide

75. Why do we care?
    

    View Slide

76. View Slide

77. XSS targets our users AND potentially
    bypasses network controls.
    

    View Slide

78. It also renders other protections
    useless, including CSRF and SSL.
    

    View Slide

79. It can also result in session leakage.
    

    View Slide

80. cookie=”+ document.cookie />
    

    View Slide

81. And phishing…
    http://ow.ly/QuCEv
    

    View Slide

82. So let’s just assume you want to
    eradicate XSS.
    

    View Slide

83. Manual testing strategies
    • Test in Firefox.
    • Try adding different scripts to form
    inputs:
    • alert(3)
    •
    •
    

    View Slide

84. Manual testing strategies
    • Still testing in Firefox.
    • See if you can manipulate the DOM and
    break CODE with user input:
    • “/>
    • '';!--"=&{()}
    

    View Slide

85. Manual testing strategies
    • Read code...
    • In js:
    • {{{
    • innerHtml
    • In Rails:
    • .raw
    • .html_safe
    

    View Slide

86. Automated testing strategies
    • We use tools to help us find these.
    • Developer tools.
    • Burp.
    

    View Slide

87. Demo using js and webdriver.
    

    View Slide

88. http://localhost:3001/?search=alert(1)</<br/>

    View Slide

89. View Slide

90. What do we really expect?
    

    View Slide

91. Incremental improvement where
    possible.
    

    View Slide

92. At a minimum, testing fields manually
    for obvious XSS patterns.
    

    View Slide

93. Villain
    

    View Slide

94. View Slide

95. Authorization
    

    View Slide

96. Insecure Direct
    Object Reference
    

    View Slide

97. Insecure Direct Object Reference
    • Enumerate and access resources that are not referenced
    by the application or intended to be accessible to the user
    Dennis Jim
    Salary Record Salary Record
    ?
    

    View Slide

98. Insecure Direct Object Reference
    • In many apps, it is often easy to predict URL patterns.
    (Because of REST)
    • It is also extremely easy to REPLAY or resend a request
    that has been tampered with.
    • Therefore, it is always important to check that the
    logged in user has permissions to perform the action
    requested.
    

    View Slide

99. Insecure Direct Object Reference
    • If you’re like me, you might be doing this:
    • Listing methods
    • Delete methods
    • Update methods
    • Business Operation Methods
    

    View Slide

100. Instance Based
     Security
     

     View Slide

101. Insecure Direct Object Reference
     • Typically requires custom logic to
     implement rules.
     • Beware of the product that says it
     can find this automatically!
     

     View Slide

102. DEMO
     cucumber --name "person is restricted
     from accessing project they do not
     own"
     

     View Slide

103. Given(/^a new project created by a user$/) do
     uuid = SecureRandom.uuid
     @user1 = "fb_user_1_#{uuid}@jemurai.com"
     register_as_user(@user1, "password")
     new_project("Insecure Direct Object Reference #{uuid}",
     "Forceful Browsing Desc")
     @url = current_url
     end
     When(/^a different person attempts to access the project$/) do
     logout(@user1)
     uuid = SecureRandom.uuid
     @user2 = "fb_user_2_#{uuid}@jemurai.com"
     register_as_user(@user2, "password")
     end
     Then(/^the system should prevent access$/) do
     visit @url
     expect(page).not_to have_content "Forceful Browsing Desc"
     end
     

     View Slide

104. Function level
     access control
     

     View Slide

105. Example: Can a regular
     user perform admin
     functions?
     

     View Slide

106. Example: Can a regular
     timesheet user approve
     timesheets?
     

     View Slide

107. Testing authorization requires
     knowledge of application and
     business logic.
     

     View Slide

108. Authorization is a great example
     of a place where testing can
     provide HUGE benefits.
     

     View Slide

109. Villain
     

     View Slide

110. View Slide

111. Note that when I say this I
     mean both MANUAL and
     AUTOMATED.
     

     View Slide

112. Security Tools
     

     View Slide

113. Static Analysis
     

     View Slide

114. Dynamic Analysis
     

     View Slide

115. App / Network Scanning
     

     View Slide

116. View Slide

117. Speaking of tools…
     

     View Slide

118. View Slide

119. View Slide

120. B+
     C
     B+
     F
     D
     D
     F
     B
     A-
     C
     The items on
     the left are
     the OWASP
     Top 10.
     The grades
     are mine and
     are arbitrary.
     https://www.owasp.org/index.php/Top_10_2013-Top_10
     

     View Slide

121. EVIL
     False
     Positives
     Are
     a Necessary
     

     View Slide

122. View Slide

123. View Slide

124. Security in the SDLC
     

     View Slide

125. Lots of security issues exist because of
     process and our prioritization.
     

     View Slide

126. Security in the SDLC
     • Building software is a process.
     • The best way to make software secure is to make security
     part of the process.
     • There are many ways to do this - none is perfect.
     • Find a way to make the security fit your process.
     

     View Slide

127. Requirements
     Design
     Code
     Test
     Maintenance
     Classic Waterfall Delivery
     

     View Slide

128. Requirements
     Design
     Code
     Test
     Maintenance
     Classic Waterfall Delivery
     Security
     

     View Slide

129. View Slide

130. Story
     Continuous Delivery: The Unit of work is a story
     Requirements Design
     Code
     Test
     

     View Slide

131. Story
     Continuous Delivery: The Unit of Work is a Story
     Requirements Design
     Code
     Test
     Security Requirements
     Security Unit Tests
     Exploratory Testing
     Static Analysis on Commit
     Code Review
     Threat model / attack
     surface
     Checklists
     Understand Dependencies
     

     View Slide

132. continuous delivery
     

     View Slide

133. Classic security sees this
     and wants to …
     

     View Slide

134. continuous delivery
     

     View Slide

135. Jenkins
     

     View Slide

136. View Slide

137. View Slide

138. View Slide

139. View Slide

140. View Slide

141. View Slide

142. View Slide

143. View Slide

144. View Slide

145. View Slide

146. Villain
     

     View Slide

147. View Slide

148. Baseline Security
     Requirements
     

     View Slide

149. Story Points
     

     View Slide

150. Estimates to Include
     Security Considerations
     

     View Slide

151. Here’s why:
     

     View Slide

152. Agile metrics
     Credit: rallydev.com
     

     View Slide

153. Story Review
     

     View Slide

154. Incremental Code Review
     

     View Slide

155. Continuous Integration
     

     View Slide

156. Checklists
     

     View Slide

157. View Slide

158. View Slide

159. View Slide

160. View Slide

161. Agile metrics
     Credit: rallydev.com
     

     View Slide

162. Testing
     

     View Slide

163. View Slide

164. Spend time on scenarios
     for negative testing
     

     View Slide

165. If you like all this stuff,
     maybe you should be a
     security champion
     

     View Slide

166. Standup
     

     View Slide

167. Architecture review
     

     View Slide

168. Operationalize
     

     View Slide

169. View Slide

170. View Slide

171. DevOps
     

     View Slide

172. DevOps Stats
     •High performers
     
     •2x success rate.
     
     •12x faster MTTR.
     
     •30x more deployments
     
     •8000x faster lead times
     
     •2x more like to exceed profitability,
     market share, productivity goals
     
     •50% market cap growth
     

     View Slide

173. Automation
     

     View Slide

174. Credit: Matt Tesauro at AppSecEU 2015
     http://www.slideshare.net/mtesauro/mtesauro-keynote-appseceu
     

     View Slide

175. Wrap Up
     

     View Slide

176. Agenda
     • Intro
     • Talking about Testing
     • Talk about scenarios from tester perspectives
     • Understanding security tools (and limitations)
     • Building Security Into Your SDLC
     • Wrap up (We’re doing this)
     

     View Slide

177. Takeaways
     • Channel a villain
     
     • Ask security questions in requirements
     
     • Test for XSS, Injection, Authorization
     
     • Leverage automation where possible
     
     • Know some of the limits of security tools
     
     • Visit owasp.org to find more resources!
     

     View Slide

178. Villain
     

     View Slide

179. View Slide

180. Thank you.
     

     View Slide