osint + python: extracting information from tor network and darkweb

7c4b1ae16723b56facc7a8a8f95aa6ce?s=47 jmortegac
August 29, 2019

osint + python: extracting information from tor network and darkweb

The talk will start explaining how Tor project can help us to the research and development of tools for online anonymity and privacy of its users while surfing the Internet, by establishing virtual circuits between the different nodes that make up the Tor network. Later, we will review main tools for discover hidden services in tor network with osint tools. Finally we will use python for extracting information from tor network with specific modules like stem https://stem.torproject.org/

These could be the main points of the talk:

- Introduction to Tor project and hidden services
- Discovering hidden services with osint tools
- Extracting information from tor network with python



August 29, 2019


  1. www.sti-innsbruck.at @jmortegac BSIDES MANCHESTER, 2019 OSINT + PYTHON: Extracting information

    from TOR network and Darkweb
  2. About me 2 http://jmortega.github.io/

  3. About me 3

  4. About me 4 AGENDA

  5. About me 5 AGENDA

  6. Agenda • Introduction to Tor project and discover hidden services

    • Modules and packages we can use in python for connecting with Tor network • Tools that allow search hidden services and atomate the crawling process in Tor network • OSINT TOOLS for discovering hidden services 6
  7. Surface vs Deep vs Dark Web 7

  8. What is Tor? 8 • Tor is a free tool

    that allows people to use the internet anonymously. • Tor anonymizes the origin of your traffic
  9. What is Tor? 9

  10. What is Tor? 10

  11. Onion Routing 11 Tor is based on Onion Routing, a

    technique for anonymous communication over a computer network.
  12. 12 Onion Routing

  13. 13 User's software or client incrementally builds a circuit of

    encrypted connections through relays on the network. Establish TOR circuit
  14. 14 When we connect to the TOR network, we do

    it through a circuit formed by 3 repeaters, where the encrypted packet sent from the client is passing. Each time the packet goes through a repeater, an encryption layer is added. Establish TOR circuit
  15. 15 User's software or client incrementally builds a circuit of

    encrypted connections through relays on the network. Hidden services
  16. 16 Directory server

  17. Hidden services 17 https://metrics.torproject.org/hidserv-dir-onions-seen.html

  18. Tor NODE List 18

  19. Tor NODE List 19 https://www.dan.me.uk/tornodes http://torstatus.blutmagie.de

  20. Tor NODE List 20 https://onionite.now.sh

  21. Exonera TOR 21 https://metrics.torproject.org/exonerator.html

  22. Relay search 22 https://metrics.torproject.org/rs.html#simple

  23. Relay search 23 https://metrics.torproject.org/rs.html#simple

  24. Relay search 24 https://metrics.torproject.org/rs.html#simple

  25. Discover hidden services 25 HiddenWiki:http://wikitjerrta4qgz4.onion/ Dark Links: http://wiki5kauuihowqi5.onion Tor Links:

    http://torlinkbgs6aabns.onion Dark Web Links: http://jdpskjmgy6kk4urv.onion/links.html HDWiki: http://hdwikicorldcisiy.onion OnionDir: http://dirnxxdraygbifgc.onion DeepLink: http://deeplinkdeatbml7.onion Ahmia: http://msydqstlz2kzerdg.onion
  26. Tor onnion services 26

  27. Tor onnion services 27 https://en.wikipedia.org/wiki/List_of_Tor_onion_ services https://en.wikipedia.org/wiki/The_Hidden_Wiki

  28. TOR2web 28 https://www.onion.to

  29. TOR browser 29 https://www.torproject.org/download/

  30. 30 Onion Routing

  31. Installing TOR 31 sudo apt-get update sudo apt-get install tor

    sudo /etc/init.d/tor restart
  32. TORrc 32

  33. Running TOR 33 $ tor --SocksPort 9050 --ControlPort 9051

  34. Running TOR 34

  35. Tor service 35 service tor start/restart service tor status

  36. Connecting with TOR 36 Stem https://stem.torproject.org/ TorRequest https://github.com/erdiaker/torrequest Requests +

  37. Stem 37 pip install stem

  38. TOR descriptors 38 Server descriptor: Complete information about a repeater

    ExtraInfo descriptor: Extra information about the repeater Micro descriptor: Contains only the information necessary for TOR clients to communicate with the repeater Consensus (Network status): File issued by the authoritative entities of the network and made up of multiple entries of information on repeaters (router status entry) Router status entry: Information about a repeater in the network, each of these elements is included in the consensus file generated by the authoritative entities.
  39. TOR spec 39

  40. Stem 40 from stem import Signal from stem.control import Controller

    with Controller.from_port(port = 9051) as controller: controller.authenticate(password='your password set for tor controller port in torrc') print("Success!") controller.signal(Signal.NEWNYM) print("New Tor connection processed")
  41. Periodic Tor IP Rotation 41 import time from stem import

    Signal from stem.control import Controller def main(): while True: time.sleep(20) print ("Rotating IP") with Controller.from_port(port = 9051) as controller: controller.authenticate() controller.signal(Signal.NEWNYM) #gets new identity if __name__ == '__main__': main()
  42. Stem.Circuit status 42 from stem.control import Controller controller = Controller.from_port(port=9051)

    controller.authenticate() print(controller.get_info('circuit-status'))
  43. Stem.Network status 43 from stem.control import Controller controller = Controller.from_port(port=9051)

    controller.authenticate(password) entries = controller.get_network_statuses() for routerEntry in entries: print(routerEntry)
  44. Stem.circuits 44

  45. Stem.circuits 45

  46. Server descriptors 46

  47. Introduction points 47

  48. Tor nyx 48 https://nyx.torproject.org/

  49. Tor nyx 49

  50. Tor nyx 50

  51. Tor nyx 51

  52. VIDEO 52

  53. TorRequest 53 from torrequest import TorRequest with TorRequest() as tr:

    response = tr.get('http://ipecho.net/plain') print(response.text) # not your IP address tr.reset_identity() response = tr.get('http://ipecho.net/plain') print(response.text) # another IP address
  54. Request 54 import requests def get_tor_session(): session = requests.session() #

    Tor uses the 9050 port as the default socks port session.proxies = {'http': 'socks5h://', 'https': 'socks5h://'} return session # Following prints your normal public IP print(requests.get("http://httpbin.org/ip").text) # Make a request through the Tor connection # Should print an IP different than your public IP session = get_tor_session() print(session.get("http://httpbin.org/ip").text) r = session.get('https://www.facebookcorewwwi.onion/') print(r.headers)
  55. Analyze hidden services 55 1) Queries to the data sources.

    2) Filter adresses that are active. 3) Testing against each active address and analysis of the response. 4) Store URLs from websites. 5) Perform a crawling process against each service 6) Apply patterns and regular expressions to detect specific content(for example mail addresses)
  56. OSINT 56

  57. Ahmia search engine 57 https://ahmia.fi/

  58. Torch search engine 58 http://xmh57jrzrnw6insl.onion

  59. UnderDir Search engine 59

  60. Hidden services 60

  61. Search Hidden services 61

  62. 62 Search Hidden services

  63. 63 Search Hidden services

  64. Other tools 64 POOPAK - TOR Hidden Service Crawler https://github.com/teal33t/poopak

    Tor spider https://github.com/absingh31/Tor_Spider Tor router https://gitlab.com/edu4rdshl/tor-router
  65. DarkSeach 65 https://darksearch.io/

  66. DarkSeach vs Ahmia 66 • Both offers results directly accessible

    on the inernet thanks to Tor2Web with connecting tor network. • DarkSeach provide a free API to automate searches (with some limitations to avoid the DDOS) • DarkSeach indexes almost half million .onion addresses.Ahmia indexes almost 5.000 sites. • Finally, both search engines not keep logs of searches done.
  67. DarkSeach API 67 https://darksearch.io/apidoc

  68. DarkSeach API 68 https://darksearch.io/api/search?query=bsides

  69. DarkSeach API 69 https://darksearch.io/api/search?query=python

  70. Onion investigator 70 https://oi.ctrlbox.com/

  71. Onion investigator 71 https://oi.ctrlbox.com/index.php?search=apps:N ginx

  72. Inspect onion address 72 https://github.com/k4m4/onioff

  73. Inspect onion address 73 https://github.com/k4m4/onioff

  74. Crawling onion address 74 https://github.com/DedSecInside/TorBot

  75. Crawling onion address 75 https://github.com/DedSecInside/TorBot

  76. Crawling onion address 76 https://github.com/MikeMeliz/TorCrawl.py

  77. Crawling onion address 77 https://github.com/dirtyfilthy/freshonions-torscr aper

  78. docker-onion-nmap 78 https://github.com/milesrichardson/docker-onio n-nmap

  79. Onion scan 79 https://github.com/s-rah/onionscan

  80. Dark Web map 80 https://www.hyperiongray.com/dark-web-map/

  81. GitHub repositories https://github.com/serfer2/python-deepweb 81

  82. GitHub repositories https://github.com/jmortega/python_dark_web 82