No todo es DNS: enumeración de software en tiempos modernos

Transcript

1. No todo es DNS:
   enumeración de software
   en tiempos modernos
   Claudio Salazar
   

   View Slide

2. Whois
   ● CEO alertot (SUP18)
   ● Founder spect.cl
   ★ Web Security Workshop, UTFSM (2013)
   ★ Pinguinux’s member
   

   View Slide

3. Agenda
   1. Enumeration techniques
   2. Software’s enumeration state of the art
   3. detectem, the project
   4. Long-term challenges
   

   View Slide

4. Enumeration techniques
   [ ref: https://www.slideshare.net/bugcrowd/ekoparty-2017-the-bug-hunters-methodology/ ]
   

   View Slide

5. Enumeration techniques - OSINT
   jhaddix/domain
   aboul3la/Sublist3r
   

   View Slide

6. Enumeration techniques - Port scanning
   nmap
   masscan
   

   View Slide

7. Enumeration techniques - Port scanning
   

   View Slide

8. Enumeration techniques - Web software
   urbanadventurer/WhatWeb
   AliasIO/Wappalyzer
   builtwith.com
   

   View Slide

9. Enumeration techniques - Web software
   

   View Slide

10. Software’s enumeration state
    of the art
    

    View Slide

11. State of the art - Status
    

    View Slide

12. State of the art - Little development
    

    View Slide

13. State of the art - No Javascript support
    

    View Slide

14. State of the art - False positives
    
    Use: https://code.jquery.com/jquery-3.2.1.js
    
    

    View Slide

15. State of the art - False positives
    

    View Slide

16. State of the art - False positives
    

    View Slide

17. State of the art - Lack of testing suite
    

    View Slide

18. detectem, the project
    

    View Slide

19. Detectem - Goals
    ● Reliable version extraction
    ● Avoid false positives as much as possible
    ● Work on sites with intensive use of
    Javascript
    ● Have Test Driven Development as rule
    

    View Slide

20. Detectem - Features
    ● Passive detection with a browser (Splash)
    ● Works on the list of requests/responses (HAR)
    ● Javascript support through Splash
    ● Based on a plugin system
    ● Command-line executable & Web service
    

    View Slide

21. detectem - Architecture
    

    View Slide

22. detectem - demo
    

    View Slide

23. Detectem - Plugin system
    ● Idea taken from WhatWeb
    ● Provide enough flexibility for corner cases
    ● Plugins contain metadata
    ● Plugins contain matchers, indicators and
    hints
    

    View Slide

24. Detectem - Matchers
    ● Unit in charge of version extraction
    ● Could be simple regular expressions or functions
    ● Adapt to the context to avoid false positives
    ● Main strongness in the framework
    

    View Slide

25. How it works - URL matcher
    https://code.jquery.com/jquery-3.1.1.js
    

    View Slide

26. How it looks - URL matcher
    class JqueryPlugin(Plugin):
    name = 'jquery'
    homepage = 'https://jquery.com/'
    matchers = [
    {'url': ‘/jquery-(?P[0-9\.]+)\.js'},
    ]
    

    View Slide

27. How it works - URL matcher scope
    Every request/response url
    except the first one
    

    View Slide

28. How it works - Body matcher
    https://code.jquery.com/jquery.js
    /*! * jQuery JavaScript Library v3.1.1
    

    View Slide

29. How it looks - Body matcher
    class JqueryPlugin(Plugin):
    name = 'jquery'
    homepage = 'https://jquery.com/'
    matchers = [
    {'body': '/\*\! jQuery v(?P[0-9\.]+)'},
    ]
    

    View Slide

30. How it works - Body matcher scope
    Every response
    except the first one
    

    View Slide

31. How it works - Header matcher
    Server: Apache/2.4.18 (Ubuntu)
    

    View Slide

32. How it looks - Header matcher
    class Apache(Plugin):
    name = 'apache'
    homepage = 'http://httpd.apache.org/'
    matchers = [{
    'header': (
    'Server',
    'Apache/(?P[0-9\.]+)'
    )
    }]
    

    View Slide

33. How it works - Header matcher scope
    Only first response
    

    View Slide

34. How it works - XPath matcher
    name="generator"
    content="Ghost 0.11"
    >
    

    View Slide

35. How it looks - XPath matcher
    class GhostPlugin(Plugin):
    name = 'ghost'
    homepage = 'https://www.ghost.org/'
    matchers = [{
    ‘xpath': (
    meta_generator('Ghost'),
    '(?P[0-9\.]+)'
    )
    }]
    

    View Slide

36. How it works - XPath matcher scope
    Only first response
    

    View Slide

37. How it works - DOM matcher
    > d3
    <- {version: "3.5.17", ascending: ƒ, ...}
    > d3.version
    <- "3.5.17"
    

    View Slide

38. How it looks - DOM matcher
    class D3JSPlugin(Plugin):
    name = 'd3.js'
    homepage = 'https://d3js.org'
    js_matchers = [{
    'check': 'window.d3',
    'version': 'window.d3.version'
    }]
    

    View Slide

39. How it works - DOM matcher scope
    Page DOM
    

    View Slide

40. detectem - Additional information
    Indicators
    Hints
    

    View Slide

41. detectem - Indicator mission
    Indicates the presence of a software
    but not its version
    

    View Slide

42. detectem - Indicator look
    class NewsShowProGk5Plugin(Plugin):
    name = 'news-show-pro-gk5'
    homepage = 'https://gavick.com/news-show-pro'
    indicators = [
    {'body': '\* @package News Show Pro GK5\n'},
    ]
    

    View Slide

43. detectem - Indicator return value
    {
    'from_url': ‘http://d.tld/mod_news_pro_gk5/sty.css',
    'homepage': 'https://www.gavick.com/news-show-pro',
    'name': 'news-show-pro-gk5',
    'type': 'indicator'
    }
    

    View Slide

44. detectem - Hint mission
    A software is hinted
    in presence of a software
    directly related to it.
    

    View Slide

45. detectem - Hint look
    class NewsShowProGk5Plugin(Plugin):
    name = 'news-show-pro-gk5'
    homepage = 'https://gavick.com/news-show-pro'
    hints = ["joomla"]
    

    View Slide

46. detectem - Hint return value
    {
    'from_url': 'https://domain.tld',
    'homepage': 'https://www.joomla.org/',
    'name': 'joomla',
    'type': 'hint'
    }
    

    View Slide

47. Long-term challenges
    

    View Slide

48. Challenges - Web is Wild
    ● Multiple versions in the same page
    ● Version pollution
    ● There’s no standard software
    

    View Slide

49. Long-term challenges
    ● Increase number of supported softwares
    ● Improvement from NLP/ML/AI
    ● Add concurrency
    ● Improve performance at scale
    

    View Slide

50. Challenges - Some news
    ● Migrate Wappalyzer DB
    ● Add bundle splitting feature
    ● Add DOM hash matcher
    ● Tests with Chrome headless
    

    View Slide

51. github.com/alertot/detectem
    

    View Slide