Darkweb + Python: discover, analyze and extract information from hidden services

Darkweb + Python: discover, analyze and extract information from hidden services

The talk will start explaining how Tor project can help us to the research and development of tools for online anonymity and privacy of its users while surfing the Internet, by establishing virtual circuits between the different nodes that make up the Tor network. In addition, we will review how Tor works from anonymity point of view, preventing websites from tracking you. Python help us to automate the process to search an discover hidden services thanks to packages like requests,requesocks and sockspy,At this point we will review the crawling process and show tools in python ecosystem available for this task(https://github.com/jmortega/python_dark_web)

These could be the talking points:

-Introduction to Tor project and hidden services
-Discovering hidden services.
-Modules and packages we can use in python for connecting with Tor network
-Tools that allow search hidden services and atomate the crawling process in Tor network

7c4b1ae16723b56facc7a8a8f95aa6ce?s=128

jmortegac

May 04, 2019
Tweet

Transcript

  1. www.sti-innsbruck.at @jmortegac May,2019 DARKWEB + PYTHON: DISCOVER, ANALYZE AND EXTRACT

    INFORMATION FROM HIDDEN SERVICES
  2. About me 2 http://jmortega.github.io/

  3. About me 3

  4. Agenda • Introduction to Tor project and hidden services •

    Discovering hidden services • Modules and packages we can use in python for connecting with Tor network • Tools that allow search hidden services and atomate the crawling process in Tor network 4
  5. Surface vs Deep vs Dark Web 5

  6. What is Tor? 6 • Tor is a free tool

    that allows people to use the internet anonymously. • Tor anonymizes the origin of your traffic
  7. What is Tor? 7

  8. What is Tor? 8

  9. Onion Routing 9 Tor is based on Onion Routing, a

    technique for anonymous communication over a computer network.
  10. 10 Onion Routing

  11. 11 User's software or client incrementally builds a circuit of

    encrypted connections through relays on the network. Establish TOR circuit
  12. 12 When we connect to the TOR network, we do

    it through a circuit formed by 3 repeaters, where the encrypted packet sent from the client is passing. Each time the packet goes through a repeater, an encryption layer is added. Establish TOR circuit
  13. 13 User's software or client incrementally builds a circuit of

    encrypted connections through relays on the network. Hidden services
  14. Hidden services 14 https://metrics.torproject.org/hidserv-dir-onions-seen.html

  15. Tor NODE List 15

  16. Tor NODE List 16 https://www.dan.me.uk/tornodes http://torstatus.blutmagie.de

  17. Tor NODE List 17 https://onionite.now.sh

  18. Exonera TOR 18 https://metrics.torproject.org/exonerator.html

  19. Relay search 19 https://metrics.torproject.org/rs.html#simple

  20. Relay search 20 https://metrics.torproject.org/rs.html#simple

  21. Relay search 21 https://metrics.torproject.org/rs.html#simple

  22. Discover hidden services 22 HiddenWiki:http://wikitjerrta4qgz4.onion/ Dark Links: http://wiki5kauuihowqi5.onion Tor Links:

    http://torlinkbgs6aabns.onion Dark Web Links: http://jdpskjmgy6kk4urv.onion/links.html HDWiki: http://hdwikicorldcisiy.onion OnionDir: http://dirnxxdraygbifgc.onion DeepLink: http://deeplinkdeatbml7.onion Ahmia: http://msydqstlz2kzerdg.onion
  23. Tor onnion services 23

  24. Tor onnion services 24 https://en.wikipedia.org/wiki/List_of_Tor_onion_ services https://en.wikipedia.org/wiki/The_Hidden_Wiki

  25. TOR2web 25 https://www.onion.to

  26. TOR browser 26 https://www.torproject.org/download/

  27. 27 Onion Routing

  28. Installing TOR 28 sudo apt-get update sudo apt-get install tor

    sudo /etc/init.d/tor restart
  29. TORrc 29

  30. Running TOR 30 $ tor --SocksPort 9050 --ControlPort 9051

  31. Running TOR 31

  32. Tor service 32 service tor start/restart service tor status

  33. Connecting with TOR 33 Stem https://stem.torproject.org/ TorRequest https://github.com/erdiaker/torrequest Requests +

    socks5
  34. Stem 34 pip install stem

  35. TOR descriptors 35 Server descriptor: Complete information about a repeater

    ExtraInfo descriptor: Extra information about the repeater Micro descriptor: Contains only the information necessary for TOR clients to communicate with the repeater Consensus (Network status): File issued by the authoritative entities of the network and made up of multiple entries of information on repeaters (router status entry) Router status entry: Information about a repeater in the network, each of these elements is included in the consensus file generated by the authoritative entities.
  36. TOR spec 36

  37. Stem 37 from stem import Signal from stem.control import Controller

    with Controller.from_port(port = 9051) as controller: controller.authenticate(password='your password set for tor controller port in torrc') print("Success!") controller.signal(Signal.NEWNYM) print("New Tor connection processed")
  38. Periodic Tor IP Rotation 38 import time from stem import

    Signal from stem.control import Controller def main(): while True: time.sleep(20) print ("Rotating IP") with Controller.from_port(port = 9051) as controller: controller.authenticate() controller.signal(Signal.NEWNYM) #gets new identity if __name__ == '__main__': main()
  39. Stem.Circuit status 39 from stem.control import Controller controller = Controller.from_port(port=9051)

    controller.authenticate() print(controller.get_info('circuit-status'))
  40. Stem.Network status 40 from stem.control import Controller controller = Controller.from_port(port=9051)

    controller.authenticate(password) entries = controller.get_network_statuses() for routerEntry in entries: print(routerEntry)
  41. Stem.circuits 41

  42. Stem.circuits 42

  43. Server descriptors 43

  44. Introduction points 44

  45. Tor nyx 45 https://nyx.torproject.org/

  46. Tor nyx 46

  47. Tor nyx 47

  48. Tor nyx 48

  49. TorRequest 49 from torrequest import TorRequest with TorRequest() as tr:

    response = tr.get('http://ipecho.net/plain') print(response.text) # not your IP address tr.reset_identity() response = tr.get('http://ipecho.net/plain') print(response.text) # another IP address
  50. Request 50 import requests def get_tor_session(): session = requests.session() #

    Tor uses the 9050 port as the default socks port session.proxies = {'http': 'socks5h://127.0.0.1:9050', 'https': 'socks5h://127.0.0.1:9050'} return session # Following prints your normal public IP print(requests.get("http://httpbin.org/ip").text) # Make a request through the Tor connection # Should print an IP different than your public IP session = get_tor_session() print(session.get("http://httpbin.org/ip").text) r = session.get('https://www.facebookcorewwwi.onion/') print(r.headers)
  51. Analyze hidden services 51 1) Queries to the data sources.

    2) Filter adresses that are active. 3) Testing against each active address and analysis of the response. 4) Store URLs from websites. 5) Perform a crawling process against each service 6) Apply patterns and regular expressions to detect specific content(for example mail addresses)
  52. Ahmia search engine 52 https://ahmia.fi/

  53. Torch search engine 53 http://xmh57jrzrnw6insl.onion

  54. UnderDir Search engine 54

  55. Hidden services 55

  56. Search Hidden services 56

  57. Other tools 57 POOPAK - TOR Hidden Service Crawler https://github.com/teal33t/poopak

    Tor spider https://github.com/absingh31/Tor_Spider Tor router https://gitlab.com/edu4rdshl/tor-router
  58. Onnion scan 58 https://github.com/s-rah/onionscan

  59. Dark Web map 59 https://www.hyperiongray.com/dark-web-map/

  60. GitHub repositories https://github.com/serfer2/python-deepweb https://github.com/jmortega/python_dark_web 60