Darkweb + Python: discover, analyze and extract information from hidden services

The talk will start explaining how Tor project can help us to the research and development of tools for online anonymity and privacy of its users while surfing the Internet, by establishing virtual circuits between the different nodes that make up the Tor network. In addition, we will review how Tor works from anonymity point of view, preventing websites from tracking you. Python help us to automate the process to search an discover hidden services thanks to packages like requests,requesocks and sockspy,At this point we will review the crawling process and show tools in python ecosystem available for this task(https://github.com/jmortega/python_dark_web)

These could be the talking points:

-Introduction to Tor project and hidden services
-Discovering hidden services.
-Modules and packages we can use in python for connecting with Tor network
-Tools that allow search hidden services and atomate the crawling process in Tor network

Transcript

1. www.sti-innsbruck.at @jmortegac May,2019 DARKWEB + PYTHON: DISCOVER, ANALYZE AND EXTRACT INFORMATION FROM HIDDEN SERVICES
2. About me 2 http://jmortega.github.io/
3. About me 3
4. Agenda • Introduction to Tor project and hidden services • Discovering hidden services • Modules and packages we can use in python for connecting with Tor network • Tools that allow search hidden services and atomate the crawling process in Tor network 4
5. Surface vs Deep vs Dark Web 5
6. What is Tor? 6 • Tor is a free tool that allows people to use the internet anonymously. • Tor anonymizes the origin of your traffic
7. What is Tor? 7
8. What is Tor? 8
9. Onion Routing 9 Tor is based on Onion Routing, a technique for anonymous communication over a computer network.
10. 10 Onion Routing
11. 11 User's software or client incrementally builds a circuit of encrypted connections through relays on the network. Establish TOR circuit
12. 12 When we connect to the TOR network, we do it through a circuit formed by 3 repeaters, where the encrypted packet sent from the client is passing. Each time the packet goes through a repeater, an encryption layer is added. Establish TOR circuit
13. 13 User's software or client incrementally builds a circuit of encrypted connections through relays on the network. Hidden services
14. Hidden services 14 https://metrics.torproject.org/hidserv-dir-onions-seen.html
15. Tor NODE List 15
16. Tor NODE List 16 https://www.dan.me.uk/tornodes http://torstatus.blutmagie.de
17. Tor NODE List 17 https://onionite.now.sh
18. Exonera TOR 18 https://metrics.torproject.org/exonerator.html
19. Relay search 19 https://metrics.torproject.org/rs.html#simple
20. Relay search 20 https://metrics.torproject.org/rs.html#simple
21. Relay search 21 https://metrics.torproject.org/rs.html#simple
22. Discover hidden services 22 HiddenWiki:http://wikitjerrta4qgz4.onion/ Dark Links: http://wiki5kauuihowqi5.onion Tor Links: http://torlinkbgs6aabns.onion Dark Web Links: http://jdpskjmgy6kk4urv.onion/links.html HDWiki: http://hdwikicorldcisiy.onion OnionDir: http://dirnxxdraygbifgc.onion DeepLink: http://deeplinkdeatbml7.onion Ahmia: http://msydqstlz2kzerdg.onion
23. Tor onnion services 23
24. Tor onnion services 24 https://en.wikipedia.org/wiki/List_of_Tor_onion_ services https://en.wikipedia.org/wiki/The_Hidden_Wiki
25. TOR2web 25 https://www.onion.to
26. TOR browser 26 https://www.torproject.org/download/
27. 27 Onion Routing
28. Installing TOR 28 sudo apt-get update sudo apt-get install tor sudo /etc/init.d/tor restart
29. TORrc 29
30. Running TOR 30 $ tor --SocksPort 9050 --ControlPort 9051
31. Running TOR 31
32. Tor service 32 service tor start/restart service tor status
33. Connecting with TOR 33 Stem https://stem.torproject.org/ TorRequest https://github.com/erdiaker/torrequest Requests + socks5
34. Stem 34 pip install stem
35. TOR descriptors 35 Server descriptor: Complete information about a repeater ExtraInfo descriptor: Extra information about the repeater Micro descriptor: Contains only the information necessary for TOR clients to communicate with the repeater Consensus (Network status): File issued by the authoritative entities of the network and made up of multiple entries of information on repeaters (router status entry) Router status entry: Information about a repeater in the network, each of these elements is included in the consensus file generated by the authoritative entities.
36. TOR spec 36
37. Stem 37 from stem import Signal from stem.control import Controller with Controller.from_port(port = 9051) as controller: controller.authenticate(password='your password set for tor controller port in torrc') print("Success!") controller.signal(Signal.NEWNYM) print("New Tor connection processed")
38. Periodic Tor IP Rotation 38 import time from stem import Signal from stem.control import Controller def main(): while True: time.sleep(20) print ("Rotating IP") with Controller.from_port(port = 9051) as controller: controller.authenticate() controller.signal(Signal.NEWNYM) #gets new identity if __name__ == '__main__': main()
39. Stem.Circuit status 39 from stem.control import Controller controller = Controller.from_port(port=9051) controller.authenticate() print(controller.get_info('circuit-status'))
40. Stem.Network status 40 from stem.control import Controller controller = Controller.from_port(port=9051) controller.authenticate(password) entries = controller.get_network_statuses() for routerEntry in entries: print(routerEntry)
41. Stem.circuits 41
42. Stem.circuits 42
43. Server descriptors 43
44. Introduction points 44
45. Tor nyx 45 https://nyx.torproject.org/
46. Tor nyx 46
47. Tor nyx 47
48. Tor nyx 48
49. TorRequest 49 from torrequest import TorRequest with TorRequest() as tr: response = tr.get('http://ipecho.net/plain') print(response.text) # not your IP address tr.reset_identity() response = tr.get('http://ipecho.net/plain') print(response.text) # another IP address
50. Request 50 import requests def get_tor_session(): session = requests.session() # Tor uses the 9050 port as the default socks port session.proxies = {'http': 'socks5h://127.0.0.1:9050', 'https': 'socks5h://127.0.0.1:9050'} return session # Following prints your normal public IP print(requests.get("http://httpbin.org/ip").text) # Make a request through the Tor connection # Should print an IP different than your public IP session = get_tor_session() print(session.get("http://httpbin.org/ip").text) r = session.get('https://www.facebookcorewwwi.onion/') print(r.headers)
51. Analyze hidden services 51 1) Queries to the data sources. 2) Filter adresses that are active. 3) Testing against each active address and analysis of the response. 4) Store URLs from websites. 5) Perform a crawling process against each service 6) Apply patterns and regular expressions to detect specific content(for example mail addresses)
52. Ahmia search engine 52 https://ahmia.fi/
53. Torch search engine 53 http://xmh57jrzrnw6insl.onion
54. UnderDir Search engine 54
55. Hidden services 55
56. Search Hidden services 56
57. Other tools 57 POOPAK - TOR Hidden Service Crawler https://github.com/teal33t/poopak Tor spider https://github.com/absingh31/Tor_Spider Tor router https://gitlab.com/edu4rdshl/tor-router
58. Onnion scan 58 https://github.com/s-rah/onionscan
59. Dark Web map 59 https://www.hyperiongray.com/dark-web-map/
60. GitHub repositories https://github.com/serfer2/python-deepweb https://github.com/jmortega/python_dark_web 60