OSINT tools for security auditing

OSINT tools for security auditing

Transcript

1. José Manuel Ortega
   @jmortegac
   PYCONES 7-9 OCT 2016
   

   View Slide

2. View Slide

3. https://github.com/jmortega/osint_tools_security_auditing
   

   View Slide

4. ▪ OSINT introduction
   ▪ Server information(Censys,Shodan)
   ▪ OSINT tools developed with python
   ▪ Geolocation,Metadata
   ▪ Twitter,Footprinting,FullContact
   

   View Slide

5. ▪ Define a specific target and data you wish to obtain
   ▪ Technical-Accounts,servers,services,software
   ▪ Social-Social Media,Email,Photos
   ▪ Physical-Address,Home IP address,Footprinting
   ▪ Logical-Network,Operational intelligence
   

   View Slide

6. ▪ GeoLocation
   ▪ IP address
   ▪ Email address
   ▪ Telephone Number
   ▪ Usernames in social network profiles
   ▪ Metadata information from images
   ▪ Server information & vulnerabilities
   

   View Slide

7. View Slide

8. ▪ https://www.censys.io/api/v1/view/ipv4/address>
   ▪ https://www.censys.io/api/v1/view/websites/
   

   View Slide

9. View Slide

10. View Slide

11. View Slide

12. View Slide

13. ▪ Checking data with ip address
    ▪ https://www.shodan.io/host/144.76.246.116
    

    View Slide

14. View Slide

15. https://developer.shodan.io/api
    

    View Slide

16. ▪ https://bitbucket.org/LaNMaSteR53/recon-ng
    ▪ Open Source OSINT toolkit written in python
    ▪ Actively maintained
    ▪ Uses modules and saves all recollected
    information in databases
    

    View Slide

17. ▪ dnspython - http://www.dnspython.org/
    ▪ dicttoxml - https://github.com/quandyfactory/dicttoxml/
    ▪ jsonrpclib - https://github.com/joshmarshall/jsonrpclib/
    ▪ lxml - http://lxml.de/
    ▪ slowaes - https://code.google.com/p/slowaes/
    ▪ XlsxWriter - https://github.com/jmcnamara/XlsxWriter/
    ▪ Mechanize
    ▪ PyPDF2
    ▪ sqlite3
    

    View Slide

18. View Slide

19. View Slide

20. View Slide

21. View Slide

22. View Slide

23. View Slide

24. View Slide

25. ▪ https://github.com/laramies/theHarvester
    

    View Slide

26. View Slide

27. ▪ httplib
    ▪ socket
    ▪ requests
    ▪ shodan
    

    View Slide

28. View Slide

29. ▪ pip install osrframework
    ▪ Developed in python 2.7
    ▪ Integrates with maltego transforms
    ▪ https://pypi.python.org/pypi/osrframework/0.13.2
    ▪ https://github.com/i3visio/osrframework
    

    View Slide

30. ▪ BeautifulSoup
    ▪ Requests
    ▪ Mechanize
    ▪ pyDNSresolving name servers
    ▪ python-whoisto recover the whois info from a domain
    ▪ tweepyfor connecting with Twitter API
    ▪ Skype4Py for connecting with Skype API
    ▪ Python-emailahoyfor checking email address
    ▪ Multiprocessingimport Process, Queue, Pool
    

    View Slide

31. View Slide

32. View Slide

33. View Slide

34. View Slide

35. View Slide

36. View Slide

37. View Slide

38. View Slide

39. Source Location Notes
    abuse.ch http://www.abuse.ch Various malware trackers.
    AdBlock https://easylist-
    downloads.adblockplus.org/easylist.txt
    AdBlock pattern matches
    AlienVault https://reputation.alienvault.com AlienVault’s IP reputation database.
    Autoshun.org http://www.autoshun.org Blacklists.
    AVG Site Safety Report http://www.avgthreatlabas.com Site safety checker.
    Bing http://www.bing.com Scraping but future version to also use API.
    Blocklist.de http://lists.blocklist.de Blacklists.
    Checkusernames.com http://www.checkusernames.com Look up username availability on popular sites.
    DNS Your configured DNS server. Defaults to your local DNS but can be configured to
    whatever IP address you supply SpiderFoot.
    DomainTools http://www.domaintools.com
    DroneBL http://www.dronebl.org
    Facebook http://www.facebook.com Scraping but future version to also use API.
    FreeGeoIP http://freegeoip.net
    Google http://www.google.com Scraping but future version to also use API.
    Google+ http://plus.google.com Scraping but future version to also use API.
    Google Safe Browsing http://www.google.com/safebrowsing Site safety checker.
    LinkedIn http://www.linkedin.com Scraping but future version to also use API.
    malc0de.com http://malc0de.com Blacklists.
    malwaredomainlist.com http://www.malwaredomainlist.com Blacklists.
    

    View Slide

40. Source Location Notes
    malwaredomains.com http://www.malwaredomains.com Blacklists.
    McAfee SiteAdvisor http://www.siteadvisor.com Site safety checker.
    NameDroppers http://www.namedroppers.org
    Nothink.org http://www.nothink.org Blacklists.
    OpenBL http://www.openbl.org Blacklists.
    PasteBin http://www.pastebin.com Achieved through Google scraping.
    PGP Servers http://pgp.mit.edu/pks/ PGP public keys.
    PhishTank http://www.phishtank.org Identified phishing sites.
    Project Honeypot http://www.projecthoneypot.org Blacklists. API key needed.
    RIPE/ARIN http://stat.ripe.net/
    Robtex http://www.robtex.com
    SANS ISC http://isc.sans.edu Internet Storm Center IP reputation database.
    SHODAN http://www.shodanhq.com API key needed.
    SORBS http://www.sorbs.net Blacklists.
    SpamHaus http://www.spamhaus.org Blacklists.
    ThreatExpert http://www.threatexpert.com Blacklists.
    TOR Node List http://torstatus.blutmagie.de
    TotalHash.com http://www.totalhash.com Domains/IPs used by malware.
    UCEPROTECT http://www.uceprotect.net Blacklists.
    VirusTotal http://www.virustotal.com Domains/IPs used by malware. API key needed.
    Whois Various Whois servers for different TLDs.
    Yahoo http://www.yahoo.com Scraping but future version to also use API.
    Zone-H http://www.zone-h.org Easy to get black-listed. Log onto the site in a
    browser from the IP you’re scanning from first and
    enter the CAPTCHA, then it should be fine.
    

    View Slide

41. ▪ Python 2.7
    ▪ BeautifulSoup
    ▪ DNSPython
    ▪ Socks
    ▪ Socket
    ▪ SSL
    ▪ CherryPy
    ▪ M2MCrypto
    ▪ Netaddr
    ▪ pyPDF
    

    View Slide

42. from bs4 import BeautifulSoup, SoupStrainer
    

    View Slide

43. View Slide

44. View Slide

45. View Slide

46. ▪ PDFPyPDF2,PDFMiner
    ▪ ImagesPillow,pyexiv2(python 2.7),gexiv2(python 3)
    

    View Slide

47. import geoip2
    import geoip2.database
    http://dev.maxmind.com/geoip/geoip2/geolite2/
    

    View Slide

48. ▪ Orb(Python 2.x)
    • https://github.com/epsylon/orb
    • python-whois - Python module for retrieving WHOIS information
    • python-dnspython - DNS toolkit for Python
    • python-nmap - Python interface to the Nmap port scanner
    • InstaRecon(Python 2.x)
    • https://github.com/vergl4s/instarecon
    • dnspython,ipaddress
    • ipwhois,python-whois
    • requests,shodan
    

    View Slide

49. View Slide

50. View Slide

51. ▪ BeautifulSoup for parsing web information
    ▪ Requests,urllib3 for synchronous requests
    ▪ Asyncio,aiohttp for asynchronous requests
    ▪ Robobrowser,Scrapy for web crawling
    ▪ PyGeoIP,geoip2,geojson for GeoLocation
    ▪ python-twitter,tweepy for connecting with twitter
    ▪ Shodan for obtain information for servers
    ▪ DNSPython,netaddr for resolving ip address
    

    View Slide

52. View Slide

53. python tinfoleak.py pycones -i -s --sdate 2016/01/01 --hashtags --mentions --meta --media [d] --geo
    GEOFILE --top 10 -o report.html
    

    View Slide

54. View Slide

55. ▪ import tweepyTwitter API library for Python
    ▪ from PIL import Image, ExifTags, ImageCmsmetadata from
    images
    ▪ import pyexiv2metadata from images
    ▪ import urllib2requests
    ▪ from OpenSSL import SSL
    ▪ from jinja2 import Template, Environment,
    FileSystemLoaderreport
    

    View Slide

56. View Slide

57. View Slide

58. View Slide

59. View Slide

60. View Slide

61. ▪ We know we have a valid email address
    ▪ What other profiles are associated with this
    address?
    ▪ Go to fullcontact.com for an API key…..
    

    View Slide

62. View Slide

63. View Slide

64. View Slide

65. View Slide

66. View Slide

67. ▪ https://sourceforge.net/projects/spiderfoot
    ▪ http://www.edge-security.com/theharvester.php
    ▪ https://developer.shodan.io/api
    ▪ http://www.clips.ua.ac.be/pattern
    ▪ http://www.pentest-
    standard.org/index.php/PTES_Technical_Guidelines#OSINT
    ▪ http://www.vicenteaguileradiaz.com/tools
    ▪ https://github.com/automatingosint/osint_public
    ▪ http://www.automatingosint.com/blog/
    

    View Slide

68. View Slide

69. Thanks!
    @jmortegac
    AMSTERDAM 9-12 MAY 2016
    

    View Slide