osint + python: extracting information from tor network and darkweb

Slide 1

Slide 1 text

www.sti-innsbruck.at @jmortegac BSIDES MANCHESTER, 2019 OSINT + PYTHON: Extracting information from TOR network and Darkweb

Slide 2

Slide 2 text

About me 2 http://jmortega.github.io/

Slide 3

Slide 3 text

About me 3

Slide 4

Slide 4 text

About me 4 AGENDA

Slide 5

Slide 5 text

About me 5 AGENDA

Slide 6

Slide 6 text

Agenda • Introduction to Tor project and discover hidden services • Modules and packages we can use in python for connecting with Tor network • Tools that allow search hidden services and atomate the crawling process in Tor network • OSINT TOOLS for discovering hidden services 6

Slide 7

Slide 7 text

Surface vs Deep vs Dark Web 7

Slide 8

Slide 8 text

What is Tor? 8 • Tor is a free tool that allows people to use the internet anonymously. • Tor anonymizes the origin of your traffic

Slide 9

Slide 9 text

What is Tor? 9

Slide 10

Slide 10 text

What is Tor? 10

Slide 11

Slide 11 text

Onion Routing 11 Tor is based on Onion Routing, a technique for anonymous communication over a computer network.

Slide 12

Slide 12 text

12 Onion Routing

Slide 13

Slide 13 text

13 User's software or client incrementally builds a circuit of encrypted connections through relays on the network. Establish TOR circuit

Slide 14

Slide 14 text

14 When we connect to the TOR network, we do it through a circuit formed by 3 repeaters, where the encrypted packet sent from the client is passing. Each time the packet goes through a repeater, an encryption layer is added. Establish TOR circuit

Slide 15

Slide 15 text

15 User's software or client incrementally builds a circuit of encrypted connections through relays on the network. Hidden services

Slide 16

Slide 16 text

16 Directory server

Slide 17

Slide 17 text

Hidden services 17 https://metrics.torproject.org/hidserv-dir-onions-seen.html

Slide 18

Slide 18 text

Tor NODE List 18

Slide 19

Slide 19 text

Tor NODE List 19 https://www.dan.me.uk/tornodes http://torstatus.blutmagie.de

Slide 20

Slide 20 text

Tor NODE List 20 https://onionite.now.sh

Slide 21

Slide 21 text

Exonera TOR 21 https://metrics.torproject.org/exonerator.html

Slide 22

Slide 22 text

Relay search 22 https://metrics.torproject.org/rs.html#simple

Slide 23

Slide 23 text

Relay search 23 https://metrics.torproject.org/rs.html#simple

Slide 24

Slide 24 text

Relay search 24 https://metrics.torproject.org/rs.html#simple

Slide 25

Slide 25 text

Discover hidden services 25 HiddenWiki:http://wikitjerrta4qgz4.onion/ Dark Links: http://wiki5kauuihowqi5.onion Tor Links: http://torlinkbgs6aabns.onion Dark Web Links: http://jdpskjmgy6kk4urv.onion/links.html HDWiki: http://hdwikicorldcisiy.onion OnionDir: http://dirnxxdraygbifgc.onion DeepLink: http://deeplinkdeatbml7.onion Ahmia: http://msydqstlz2kzerdg.onion

Slide 26

Slide 26 text

Tor onnion services 26

Slide 27

Slide 27 text

Tor onnion services 27 https://en.wikipedia.org/wiki/List_of_Tor_onion_ services https://en.wikipedia.org/wiki/The_Hidden_Wiki

Slide 28

Slide 28 text

TOR2web 28 https://www.onion.to

Slide 29

Slide 29 text

TOR browser 29 https://www.torproject.org/download/

Slide 30

Slide 30 text

30 Onion Routing

Slide 31

Slide 31 text

Installing TOR 31 sudo apt-get update sudo apt-get install tor sudo /etc/init.d/tor restart

Slide 32

Slide 32 text

TORrc 32

Slide 33

Slide 33 text

Running TOR 33 $ tor --SocksPort 9050 --ControlPort 9051

Slide 34

Slide 34 text

Running TOR 34

Slide 35

Slide 35 text

Tor service 35 service tor start/restart service tor status

Slide 36

Slide 36 text

Connecting with TOR 36 Stem https://stem.torproject.org/ TorRequest https://github.com/erdiaker/torrequest Requests + socks5

Slide 37

Slide 37 text

Stem 37 pip install stem

Slide 38

Slide 38 text

TOR descriptors 38 Server descriptor: Complete information about a repeater ExtraInfo descriptor: Extra information about the repeater Micro descriptor: Contains only the information necessary for TOR clients to communicate with the repeater Consensus (Network status): File issued by the authoritative entities of the network and made up of multiple entries of information on repeaters (router status entry) Router status entry: Information about a repeater in the network, each of these elements is included in the consensus file generated by the authoritative entities.

Slide 39

Slide 39 text

TOR spec 39

Slide 40

Slide 40 text

Stem 40 from stem import Signal from stem.control import Controller with Controller.from_port(port = 9051) as controller: controller.authenticate(password='your password set for tor controller port in torrc') print("Success!") controller.signal(Signal.NEWNYM) print("New Tor connection processed")

Slide 41

Slide 41 text

Periodic Tor IP Rotation 41 import time from stem import Signal from stem.control import Controller def main(): while True: time.sleep(20) print ("Rotating IP") with Controller.from_port(port = 9051) as controller: controller.authenticate() controller.signal(Signal.NEWNYM) #gets new identity if __name__ == '__main__': main()

Slide 42

Slide 42 text

Stem.Circuit status 42 from stem.control import Controller controller = Controller.from_port(port=9051) controller.authenticate() print(controller.get_info('circuit-status'))

Slide 43

Slide 43 text

Stem.Network status 43 from stem.control import Controller controller = Controller.from_port(port=9051) controller.authenticate(password) entries = controller.get_network_statuses() for routerEntry in entries: print(routerEntry)

Slide 44

Slide 44 text

Stem.circuits 44

Slide 45

Slide 45 text

Stem.circuits 45

Slide 46

Slide 46 text

Server descriptors 46

Slide 47

Slide 47 text

Introduction points 47

Slide 48

Slide 48 text

Tor nyx 48 https://nyx.torproject.org/

Slide 49

Slide 49 text

Tor nyx 49

Slide 50

Slide 50 text

Tor nyx 50

Slide 51

Slide 51 text

Tor nyx 51

Slide 52

Slide 52 text

VIDEO 52

Slide 53

Slide 53 text

TorRequest 53 from torrequest import TorRequest with TorRequest() as tr: response = tr.get('http://ipecho.net/plain') print(response.text) # not your IP address tr.reset_identity() response = tr.get('http://ipecho.net/plain') print(response.text) # another IP address

Slide 54

Slide 54 text

Request 54 import requests def get_tor_session(): session = requests.session() # Tor uses the 9050 port as the default socks port session.proxies = {'http': 'socks5h://127.0.0.1:9050', 'https': 'socks5h://127.0.0.1:9050'} return session # Following prints your normal public IP print(requests.get("http://httpbin.org/ip").text) # Make a request through the Tor connection # Should print an IP different than your public IP session = get_tor_session() print(session.get("http://httpbin.org/ip").text) r = session.get('https://www.facebookcorewwwi.onion/') print(r.headers)

Slide 55

Slide 55 text

Analyze hidden services 55 1) Queries to the data sources. 2) Filter adresses that are active. 3) Testing against each active address and analysis of the response. 4) Store URLs from websites. 5) Perform a crawling process against each service 6) Apply patterns and regular expressions to detect specific content(for example mail addresses)

Slide 56

Slide 56 text

OSINT 56

Slide 57

Slide 57 text

Ahmia search engine 57 https://ahmia.fi/

Slide 58

Slide 58 text

Torch search engine 58 http://xmh57jrzrnw6insl.onion

Slide 59

Slide 59 text

UnderDir Search engine 59

Slide 60

Slide 60 text

Hidden services 60

Slide 61

Slide 61 text

Search Hidden services 61

Slide 62

Slide 62 text

62 Search Hidden services

Slide 63

Slide 63 text

63 Search Hidden services

Slide 64

Slide 64 text

Other tools 64 POOPAK - TOR Hidden Service Crawler https://github.com/teal33t/poopak Tor spider https://github.com/absingh31/Tor_Spider Tor router https://gitlab.com/edu4rdshl/tor-router

Slide 65

Slide 65 text

DarkSeach 65 https://darksearch.io/

Slide 66

Slide 66 text

DarkSeach vs Ahmia 66 ● Both offers results directly accessible on the inernet thanks to Tor2Web with connecting tor network. ● DarkSeach provide a free API to automate searches (with some limitations to avoid the DDOS) ● DarkSeach indexes almost half million .onion addresses.Ahmia indexes almost 5.000 sites. ● Finally, both search engines not keep logs of searches done.

Slide 67

Slide 67 text

DarkSeach API 67 https://darksearch.io/apidoc

Slide 68

Slide 68 text

DarkSeach API 68 https://darksearch.io/api/search?query=bsides

Slide 69

Slide 69 text

DarkSeach API 69 https://darksearch.io/api/search?query=python

Slide 70

Slide 70 text

Onion investigator 70 https://oi.ctrlbox.com/

Slide 71

Slide 71 text

Onion investigator 71 https://oi.ctrlbox.com/index.php?search=apps:N ginx

Slide 72

Slide 72 text

Inspect onion address 72 https://github.com/k4m4/onioff

Slide 73

Slide 73 text

Inspect onion address 73 https://github.com/k4m4/onioff

Slide 74

Slide 74 text

Crawling onion address 74 https://github.com/DedSecInside/TorBot

Slide 75

Slide 75 text

Crawling onion address 75 https://github.com/DedSecInside/TorBot

Slide 76

Slide 76 text

Crawling onion address 76 https://github.com/MikeMeliz/TorCrawl.py

Slide 77

Slide 77 text

Crawling onion address 77 https://github.com/dirtyfilthy/freshonions-torscr aper