Darkweb + Python: discover, analyze and extract information from hidden services

Slide 1

Slide 1 text

www.sti-innsbruck.at @jmortegac May,2019 DARKWEB + PYTHON: DISCOVER, ANALYZE AND EXTRACT INFORMATION FROM HIDDEN SERVICES

Slide 2

Slide 2 text

About me 2 http://jmortega.github.io/

Slide 3

Slide 3 text

About me 3

Slide 4

Slide 4 text

Agenda • Introduction to Tor project and hidden services • Discovering hidden services • Modules and packages we can use in python for connecting with Tor network • Tools that allow search hidden services and atomate the crawling process in Tor network 4

Slide 5

Slide 5 text

Surface vs Deep vs Dark Web 5

Slide 6

Slide 6 text

What is Tor? 6 • Tor is a free tool that allows people to use the internet anonymously. • Tor anonymizes the origin of your traffic

Slide 7

Slide 7 text

What is Tor? 7

Slide 8

Slide 8 text

What is Tor? 8

Slide 9

Slide 9 text

Onion Routing 9 Tor is based on Onion Routing, a technique for anonymous communication over a computer network.

Slide 10

Slide 10 text

10 Onion Routing

Slide 11

Slide 11 text

11 User's software or client incrementally builds a circuit of encrypted connections through relays on the network. Establish TOR circuit

Slide 12

Slide 12 text

12 When we connect to the TOR network, we do it through a circuit formed by 3 repeaters, where the encrypted packet sent from the client is passing. Each time the packet goes through a repeater, an encryption layer is added. Establish TOR circuit

Slide 13

Slide 13 text

13 User's software or client incrementally builds a circuit of encrypted connections through relays on the network. Hidden services

Slide 14

Slide 14 text

Hidden services 14 https://metrics.torproject.org/hidserv-dir-onions-seen.html

Slide 15

Slide 15 text

Tor NODE List 15

Slide 16

Slide 16 text

Tor NODE List 16 https://www.dan.me.uk/tornodes http://torstatus.blutmagie.de

Slide 17

Slide 17 text

Tor NODE List 17 https://onionite.now.sh

Slide 18

Slide 18 text

Exonera TOR 18 https://metrics.torproject.org/exonerator.html

Slide 19

Slide 19 text

Relay search 19 https://metrics.torproject.org/rs.html#simple

Slide 20

Slide 20 text

Relay search 20 https://metrics.torproject.org/rs.html#simple

Slide 21

Slide 21 text

Relay search 21 https://metrics.torproject.org/rs.html#simple

Slide 22

Slide 22 text

Discover hidden services 22 HiddenWiki:http://wikitjerrta4qgz4.onion/ Dark Links: http://wiki5kauuihowqi5.onion Tor Links: http://torlinkbgs6aabns.onion Dark Web Links: http://jdpskjmgy6kk4urv.onion/links.html HDWiki: http://hdwikicorldcisiy.onion OnionDir: http://dirnxxdraygbifgc.onion DeepLink: http://deeplinkdeatbml7.onion Ahmia: http://msydqstlz2kzerdg.onion

Slide 23

Slide 23 text

Tor onnion services 23

Slide 24

Slide 24 text

Tor onnion services 24 https://en.wikipedia.org/wiki/List_of_Tor_onion_ services https://en.wikipedia.org/wiki/The_Hidden_Wiki

Slide 25

Slide 25 text

TOR2web 25 https://www.onion.to

Slide 26

Slide 26 text

TOR browser 26 https://www.torproject.org/download/

Slide 27

Slide 27 text

27 Onion Routing

Slide 28

Slide 28 text

Installing TOR 28 sudo apt-get update sudo apt-get install tor sudo /etc/init.d/tor restart

Slide 29

Slide 29 text

TORrc 29

Slide 30

Slide 30 text

Running TOR 30 $ tor --SocksPort 9050 --ControlPort 9051

Slide 31

Slide 31 text

Running TOR 31

Slide 32

Slide 32 text

Tor service 32 service tor start/restart service tor status

Slide 33

Slide 33 text

Connecting with TOR 33 Stem https://stem.torproject.org/ TorRequest https://github.com/erdiaker/torrequest Requests + socks5

Slide 34

Slide 34 text

Stem 34 pip install stem

Slide 35

Slide 35 text

TOR descriptors 35 Server descriptor: Complete information about a repeater ExtraInfo descriptor: Extra information about the repeater Micro descriptor: Contains only the information necessary for TOR clients to communicate with the repeater Consensus (Network status): File issued by the authoritative entities of the network and made up of multiple entries of information on repeaters (router status entry) Router status entry: Information about a repeater in the network, each of these elements is included in the consensus file generated by the authoritative entities.

Slide 36

Slide 36 text

TOR spec 36

Slide 37

Slide 37 text

Stem 37 from stem import Signal from stem.control import Controller with Controller.from_port(port = 9051) as controller: controller.authenticate(password='your password set for tor controller port in torrc') print("Success!") controller.signal(Signal.NEWNYM) print("New Tor connection processed")

Slide 38

Slide 38 text

Periodic Tor IP Rotation 38 import time from stem import Signal from stem.control import Controller def main(): while True: time.sleep(20) print ("Rotating IP") with Controller.from_port(port = 9051) as controller: controller.authenticate() controller.signal(Signal.NEWNYM) #gets new identity if __name__ == '__main__': main()

Slide 39

Slide 39 text

Stem.Circuit status 39 from stem.control import Controller controller = Controller.from_port(port=9051) controller.authenticate() print(controller.get_info('circuit-status'))

Slide 40

Slide 40 text

Stem.Network status 40 from stem.control import Controller controller = Controller.from_port(port=9051) controller.authenticate(password) entries = controller.get_network_statuses() for routerEntry in entries: print(routerEntry)

Slide 41

Slide 41 text

Stem.circuits 41

Slide 42

Slide 42 text

Stem.circuits 42

Slide 43

Slide 43 text

Server descriptors 43

Slide 44

Slide 44 text

Introduction points 44

Slide 45

Slide 45 text

Tor nyx 45 https://nyx.torproject.org/

Slide 46

Slide 46 text

Tor nyx 46

Slide 47

Slide 47 text

Tor nyx 47

Slide 48

Slide 48 text

Tor nyx 48

Slide 49

Slide 49 text

TorRequest 49 from torrequest import TorRequest with TorRequest() as tr: response = tr.get('http://ipecho.net/plain') print(response.text) # not your IP address tr.reset_identity() response = tr.get('http://ipecho.net/plain') print(response.text) # another IP address

Slide 50

Slide 50 text

Request 50 import requests def get_tor_session(): session = requests.session() # Tor uses the 9050 port as the default socks port session.proxies = {'http': 'socks5h://127.0.0.1:9050', 'https': 'socks5h://127.0.0.1:9050'} return session # Following prints your normal public IP print(requests.get("http://httpbin.org/ip").text) # Make a request through the Tor connection # Should print an IP different than your public IP session = get_tor_session() print(session.get("http://httpbin.org/ip").text) r = session.get('https://www.facebookcorewwwi.onion/') print(r.headers)

Slide 51

Slide 51 text

Analyze hidden services 51 1) Queries to the data sources. 2) Filter adresses that are active. 3) Testing against each active address and analysis of the response. 4) Store URLs from websites. 5) Perform a crawling process against each service 6) Apply patterns and regular expressions to detect specific content(for example mail addresses)

Slide 52

Slide 52 text

Ahmia search engine 52 https://ahmia.fi/

Slide 53

Slide 53 text

Torch search engine 53 http://xmh57jrzrnw6insl.onion

Slide 54

Slide 54 text

UnderDir Search engine 54

Slide 55

Slide 55 text

Hidden services 55

Slide 56

Slide 56 text

Search Hidden services 56

Slide 57

Slide 57 text

Other tools 57 POOPAK - TOR Hidden Service Crawler https://github.com/teal33t/poopak Tor spider https://github.com/absingh31/Tor_Spider Tor router https://gitlab.com/edu4rdshl/tor-router

Slide 58

Slide 58 text

Onnion scan 58 https://github.com/s-rah/onionscan

Slide 59

Slide 59 text

Dark Web map 59 https://www.hyperiongray.com/dark-web-map/

Slide 60

Slide 60 text

GitHub repositories https://github.com/serfer2/python-deepweb https://github.com/jmortega/python_dark_web 60