No todo es DNS: enumeración de software en tiempos modernos

Transcript

1. No todo es DNS: enumeración de software en tiempos modernos Claudio Salazar
2. Whois • CEO alertot (SUP18) • Founder spect.cl ★ Web Security Workshop, UTFSM (2013) ★ Pinguinux’s member
3. Agenda 1. Enumeration techniques 2. Software’s enumeration state of the art 3. detectem, the project 4. Long-term challenges
4. Enumeration techniques [ ref: https://www.slideshare.net/bugcrowd/ekoparty-2017-the-bug-hunters-methodology/ ]
5. Enumeration techniques - OSINT jhaddix/domain aboul3la/Sublist3r
6. Enumeration techniques - Port scanning nmap masscan
7. Enumeration techniques - Port scanning
8. Enumeration techniques - Web software urbanadventurer/WhatWeb AliasIO/Wappalyzer builtwith.com
9. Enumeration techniques - Web software
10. Software’s enumeration state of the art
11. State of the art - Status
12. State of the art - Little development
13. State of the art - No Javascript support
14. State of the art - False positives <p> Use: https://code.jquery.com/jquery-3.2.1.js </p>
15. State of the art - False positives
16. State of the art - False positives
17. State of the art - Lack of testing suite
18. detectem, the project
19. Detectem - Goals • Reliable version extraction • Avoid false positives as much as possible • Work on sites with intensive use of Javascript • Have Test Driven Development as rule
20. Detectem - Features • Passive detection with a browser (Splash) • Works on the list of requests/responses (HAR) • Javascript support through Splash • Based on a plugin system • Command-line executable & Web service
21. detectem - Architecture
22. detectem - demo
23. Detectem - Plugin system • Idea taken from WhatWeb • Provide enough flexibility for corner cases • Plugins contain metadata • Plugins contain matchers, indicators and hints
24. Detectem - Matchers • Unit in charge of version extraction • Could be simple regular expressions or functions • Adapt to the context to avoid false positives • Main strongness in the framework
25. How it works - URL matcher https://code.jquery.com/jquery-3.1.1.js
26. How it looks - URL matcher class JqueryPlugin(Plugin): name = 'jquery' homepage = 'https://jquery.com/' matchers = [ {'url': ‘/jquery-(?P<version>[0-9\.]+)\.js'}, ]
27. How it works - URL matcher scope Every request/response url except the first one
28. How it works - Body matcher https://code.jquery.com/jquery.js /*! * jQuery JavaScript Library v3.1.1
29. How it looks - Body matcher class JqueryPlugin(Plugin): name = 'jquery' homepage = 'https://jquery.com/' matchers = [ {'body': '/\*\! jQuery v(?P<version>[0-9\.]+)'}, ]
30. How it works - Body matcher scope Every response except the first one
31. How it works - Header matcher Server: Apache/2.4.18 (Ubuntu)
32. How it looks - Header matcher class Apache(Plugin): name = 'apache' homepage = 'http://httpd.apache.org/' matchers = [{ 'header': ( 'Server', 'Apache/(?P<version>[0-9\.]+)' ) }]
33. How it works - Header matcher scope Only first response
34. How it works - XPath matcher <meta name="generator" content="Ghost 0.11" >
35. How it looks - XPath matcher class GhostPlugin(Plugin): name = 'ghost' homepage = 'https://www.ghost.org/' matchers = [{ ‘xpath': ( meta_generator('Ghost'), '(?P<version>[0-9\.]+)' ) }]
36. How it works - XPath matcher scope Only first response
37. How it works - DOM matcher > d3 <- {version: "3.5.17", ascending: ƒ, ...} > d3.version <- "3.5.17"
38. How it looks - DOM matcher class D3JSPlugin(Plugin): name = 'd3.js' homepage = 'https://d3js.org' js_matchers = [{ 'check': 'window.d3', 'version': 'window.d3.version' }]
39. How it works - DOM matcher scope Page DOM
40. detectem - Additional information Indicators Hints
41. detectem - Indicator mission Indicates the presence of a software but not its version
42. detectem - Indicator look class NewsShowProGk5Plugin(Plugin): name = 'news-show-pro-gk5' homepage = 'https://gavick.com/news-show-pro' indicators = [ {'body': '\* @package News Show Pro GK5\n'}, ]
43. detectem - Indicator return value { 'from_url': ‘http://d.tld/mod_news_pro_gk5/sty.css', 'homepage': 'https://www.gavick.com/news-show-pro', 'name': 'news-show-pro-gk5', 'type': 'indicator' }
44. detectem - Hint mission A software is hinted in presence of a software directly related to it.
45. detectem - Hint look class NewsShowProGk5Plugin(Plugin): name = 'news-show-pro-gk5' homepage = 'https://gavick.com/news-show-pro' hints = ["joomla"]
46. detectem - Hint return value { 'from_url': 'https://domain.tld', 'homepage': 'https://www.joomla.org/', 'name': 'joomla', 'type': 'hint' }
47. Long-term challenges
48. Challenges - Web is Wild • Multiple versions in the same page • Version pollution • There’s no standard software
49. Long-term challenges • Increase number of supported softwares • Improvement from NLP/ML/AI • Add concurrency • Improve performance at scale
50. Challenges - Some news • Migrate Wappalyzer DB • Add bundle splitting feature • Add DOM hash matcher • Tests with Chrome headless
51. github.com/alertot/detectem