osint + python: extracting information from tor network and darkweb

The talk will start explaining how Tor project can help us to the research and development of tools for online anonymity and privacy of its users while surfing the Internet, by establishing virtual circuits between the different nodes that make up the Tor network. Later, we will review main tools for discover hidden services in tor network with osint tools. Finally we will use python for extracting information from tor network with specific modules like stem https://stem.torproject.org/

These could be the main points of the talk:

- Introduction to Tor project and hidden services
- Discovering hidden services with osint tools
- Extracting information from tor network with python

Transcript

1. www.sti-innsbruck.at @jmortegac BSIDES MANCHESTER, 2019 OSINT + PYTHON: Extracting information from TOR network and Darkweb
2. About me 2 http://jmortega.github.io/
3. About me 3
4. About me 4 AGENDA
5. About me 5 AGENDA
6. Agenda • Introduction to Tor project and discover hidden services • Modules and packages we can use in python for connecting with Tor network • Tools that allow search hidden services and atomate the crawling process in Tor network • OSINT TOOLS for discovering hidden services 6
7. Surface vs Deep vs Dark Web 7
8. What is Tor? 8 • Tor is a free tool that allows people to use the internet anonymously. • Tor anonymizes the origin of your traffic
9. What is Tor? 9
10. What is Tor? 10
11. Onion Routing 11 Tor is based on Onion Routing, a technique for anonymous communication over a computer network.
12. 12 Onion Routing
13. 13 User's software or client incrementally builds a circuit of encrypted connections through relays on the network. Establish TOR circuit
14. 14 When we connect to the TOR network, we do it through a circuit formed by 3 repeaters, where the encrypted packet sent from the client is passing. Each time the packet goes through a repeater, an encryption layer is added. Establish TOR circuit
15. 15 User's software or client incrementally builds a circuit of encrypted connections through relays on the network. Hidden services
16. 16 Directory server
17. Hidden services 17 https://metrics.torproject.org/hidserv-dir-onions-seen.html
18. Tor NODE List 18
19. Tor NODE List 19 https://www.dan.me.uk/tornodes http://torstatus.blutmagie.de
20. Tor NODE List 20 https://onionite.now.sh
21. Exonera TOR 21 https://metrics.torproject.org/exonerator.html
22. Relay search 22 https://metrics.torproject.org/rs.html#simple
23. Relay search 23 https://metrics.torproject.org/rs.html#simple
24. Relay search 24 https://metrics.torproject.org/rs.html#simple
25. Discover hidden services 25 HiddenWiki:http://wikitjerrta4qgz4.onion/ Dark Links: http://wiki5kauuihowqi5.onion Tor Links: http://torlinkbgs6aabns.onion Dark Web Links: http://jdpskjmgy6kk4urv.onion/links.html HDWiki: http://hdwikicorldcisiy.onion OnionDir: http://dirnxxdraygbifgc.onion DeepLink: http://deeplinkdeatbml7.onion Ahmia: http://msydqstlz2kzerdg.onion
26. Tor onnion services 26
27. Tor onnion services 27 https://en.wikipedia.org/wiki/List_of_Tor_onion_ services https://en.wikipedia.org/wiki/The_Hidden_Wiki
28. TOR2web 28 https://www.onion.to
29. TOR browser 29 https://www.torproject.org/download/
30. 30 Onion Routing
31. Installing TOR 31 sudo apt-get update sudo apt-get install tor sudo /etc/init.d/tor restart
32. TORrc 32
33. Running TOR 33 $ tor --SocksPort 9050 --ControlPort 9051
34. Running TOR 34
35. Tor service 35 service tor start/restart service tor status
36. Connecting with TOR 36 Stem https://stem.torproject.org/ TorRequest https://github.com/erdiaker/torrequest Requests + socks5
37. Stem 37 pip install stem
38. TOR descriptors 38 Server descriptor: Complete information about a repeater ExtraInfo descriptor: Extra information about the repeater Micro descriptor: Contains only the information necessary for TOR clients to communicate with the repeater Consensus (Network status): File issued by the authoritative entities of the network and made up of multiple entries of information on repeaters (router status entry) Router status entry: Information about a repeater in the network, each of these elements is included in the consensus file generated by the authoritative entities.
39. TOR spec 39
40. Stem 40 from stem import Signal from stem.control import Controller with Controller.from_port(port = 9051) as controller: controller.authenticate(password='your password set for tor controller port in torrc') print("Success!") controller.signal(Signal.NEWNYM) print("New Tor connection processed")
41. Periodic Tor IP Rotation 41 import time from stem import Signal from stem.control import Controller def main(): while True: time.sleep(20) print ("Rotating IP") with Controller.from_port(port = 9051) as controller: controller.authenticate() controller.signal(Signal.NEWNYM) #gets new identity if __name__ == '__main__': main()
42. Stem.Circuit status 42 from stem.control import Controller controller = Controller.from_port(port=9051) controller.authenticate() print(controller.get_info('circuit-status'))
43. Stem.Network status 43 from stem.control import Controller controller = Controller.from_port(port=9051) controller.authenticate(password) entries = controller.get_network_statuses() for routerEntry in entries: print(routerEntry)
44. Stem.circuits 44
45. Stem.circuits 45
46. Server descriptors 46
47. Introduction points 47
48. Tor nyx 48 https://nyx.torproject.org/
49. Tor nyx 49
50. Tor nyx 50
51. Tor nyx 51
52. VIDEO 52
53. TorRequest 53 from torrequest import TorRequest with TorRequest() as tr: response = tr.get('http://ipecho.net/plain') print(response.text) # not your IP address tr.reset_identity() response = tr.get('http://ipecho.net/plain') print(response.text) # another IP address
54. Request 54 import requests def get_tor_session(): session = requests.session() # Tor uses the 9050 port as the default socks port session.proxies = {'http': 'socks5h://127.0.0.1:9050', 'https': 'socks5h://127.0.0.1:9050'} return session # Following prints your normal public IP print(requests.get("http://httpbin.org/ip").text) # Make a request through the Tor connection # Should print an IP different than your public IP session = get_tor_session() print(session.get("http://httpbin.org/ip").text) r = session.get('https://www.facebookcorewwwi.onion/') print(r.headers)
55. Analyze hidden services 55 1) Queries to the data sources. 2) Filter adresses that are active. 3) Testing against each active address and analysis of the response. 4) Store URLs from websites. 5) Perform a crawling process against each service 6) Apply patterns and regular expressions to detect specific content(for example mail addresses)
56. OSINT 56
57. Ahmia search engine 57 https://ahmia.fi/
58. Torch search engine 58 http://xmh57jrzrnw6insl.onion
59. UnderDir Search engine 59
60. Hidden services 60
61. Search Hidden services 61
62. 62 Search Hidden services
63. 63 Search Hidden services
64. Other tools 64 POOPAK - TOR Hidden Service Crawler https://github.com/teal33t/poopak Tor spider https://github.com/absingh31/Tor_Spider Tor router https://gitlab.com/edu4rdshl/tor-router
65. DarkSeach 65 https://darksearch.io/
66. DarkSeach vs Ahmia 66 • Both offers results directly accessible on the inernet thanks to Tor2Web with connecting tor network. • DarkSeach provide a free API to automate searches (with some limitations to avoid the DDOS) • DarkSeach indexes almost half million .onion addresses.Ahmia indexes almost 5.000 sites. • Finally, both search engines not keep logs of searches done.
67. DarkSeach API 67 https://darksearch.io/apidoc
68. DarkSeach API 68 https://darksearch.io/api/search?query=bsides
69. DarkSeach API 69 https://darksearch.io/api/search?query=python
70. Onion investigator 70 https://oi.ctrlbox.com/
71. Onion investigator 71 https://oi.ctrlbox.com/index.php?search=apps:N ginx
72. Inspect onion address 72 https://github.com/k4m4/onioff
73. Inspect onion address 73 https://github.com/k4m4/onioff
74. Crawling onion address 74 https://github.com/DedSecInside/TorBot
75. Crawling onion address 75 https://github.com/DedSecInside/TorBot
76. Crawling onion address 76 https://github.com/MikeMeliz/TorCrawl.py
77. Crawling onion address 77 https://github.com/dirtyfilthy/freshonions-torscr aper
78. docker-onion-nmap 78 https://github.com/milesrichardson/docker-onio n-nmap
79. Onion scan 79 https://github.com/s-rah/onionscan
80. Dark Web map 80 https://www.hyperiongray.com/dark-web-map/
81. GitHub repositories https://github.com/serfer2/python-deepweb 81
82. GitHub repositories https://github.com/jmortega/python_dark_web 82