Darkweb + Python: discover, analyze and extract information from hidden services

The talk will start explaining how Tor project can help us to the research and development of tools for online anonymity and privacy of its users while surfing the Internet, by establishing virtual circuits between the different nodes that make up the Tor network. In addition, we will review how Tor works from anonymity point of view, preventing websites from tracking you. Python help us to automate the process to search an discover hidden services thanks to packages like requests,requesocks and sockspy,At this point we will review the crawling process and show tools in python ecosystem available for this task(https://github.com/jmortega/python_dark_web)

These could be the talking points:

-Introduction to Tor project and hidden services
-Discovering hidden services.
-Modules and packages we can use in python for connecting with Tor network
-Tools that allow search hidden services and atomate the crawling process in Tor network

Transcript

1. www.sti-innsbruck.at
   @jmortegac May,2019
   DARKWEB + PYTHON: DISCOVER,
   ANALYZE AND EXTRACT INFORMATION
   FROM HIDDEN SERVICES
   

   View Slide

2. About me
   2
   http://jmortega.github.io/
   

   View Slide

3. About me
   3
   

   View Slide

4. Agenda
   • Introduction to Tor project and hidden
   services
   • Discovering hidden services
   • Modules and packages we can use in
   python for connecting with Tor network
   • Tools that allow search hidden services
   and atomate the crawling process in Tor
   network
   4
   

   View Slide

5. Surface vs Deep vs Dark Web
   5
   

   View Slide

6. What is Tor?
   6
   • Tor is a free tool that allows people to use the
   internet anonymously.
   • Tor anonymizes the origin of your traffic
   

   View Slide

7. What is Tor?
   7
   

   View Slide

8. What is Tor?
   8
   

   View Slide

9. Onion Routing
   9
   Tor is based on Onion Routing, a technique for
   anonymous communication over a computer network.
   

   View Slide

10. 10
    Onion Routing
    

    View Slide

11. 11
    User's software or client
    incrementally builds a
    circuit of encrypted
    connections through
    relays on the network.
    Establish TOR circuit
    

    View Slide

12. 12
    When we connect to
    the TOR network, we
    do it through a circuit
    formed by 3
    repeaters, where the
    encrypted packet sent
    from the client is
    passing. Each time
    the packet goes
    through a repeater, an
    encryption layer is
    added.
    Establish TOR circuit
    

    View Slide

13. 13
    User's software or client
    incrementally builds a
    circuit of encrypted
    connections through
    relays on the network.
    Hidden services
    

    View Slide

14. Hidden services
    14
    https://metrics.torproject.org/hidserv-dir-onions-seen.html
    

    View Slide

15. Tor NODE List
    15
    

    View Slide

16. Tor NODE List
    16
    https://www.dan.me.uk/tornodes
    http://torstatus.blutmagie.de
    

    View Slide

17. Tor NODE List
    17
    https://onionite.now.sh
    

    View Slide

18. Exonera TOR
    18
    https://metrics.torproject.org/exonerator.html
    

    View Slide

19. Relay search
    19
    https://metrics.torproject.org/rs.html#simple
    

    View Slide

20. Relay search
    20
    https://metrics.torproject.org/rs.html#simple
    

    View Slide

21. Relay search
    21
    https://metrics.torproject.org/rs.html#simple
    

    View Slide

22. Discover hidden services
    22
    HiddenWiki:http://wikitjerrta4qgz4.onion/
    Dark Links: http://wiki5kauuihowqi5.onion
    Tor Links: http://torlinkbgs6aabns.onion
    Dark Web Links:
    http://jdpskjmgy6kk4urv.onion/links.html
    HDWiki: http://hdwikicorldcisiy.onion
    OnionDir: http://dirnxxdraygbifgc.onion
    DeepLink: http://deeplinkdeatbml7.onion
    Ahmia: http://msydqstlz2kzerdg.onion
    

    View Slide

23. Tor onnion services
    23
    

    View Slide

24. Tor onnion services
    24
    https://en.wikipedia.org/wiki/List_of_Tor_onion_
    services
    https://en.wikipedia.org/wiki/The_Hidden_Wiki
    

    View Slide

25. TOR2web
    25
    https://www.onion.to
    

    View Slide

26. TOR browser
    26
    https://www.torproject.org/download/
    

    View Slide

27. 27
    Onion Routing
    

    View Slide

28. Installing TOR
    28
    sudo apt-get update
    sudo apt-get install tor
    sudo /etc/init.d/tor restart
    

    View Slide

29. TORrc
    29
    

    View Slide

30. Running TOR
    30
    $ tor --SocksPort 9050 --ControlPort 9051
    

    View Slide

31. Running TOR
    31
    

    View Slide

32. Tor service
    32
    service tor start/restart
    service tor status
    

    View Slide

33. Connecting with TOR
    33
    Stem
    https://stem.torproject.org/
    TorRequest
    https://github.com/erdiaker/torrequest
    Requests + socks5
    

    View Slide

34. Stem
    34
    pip install stem
    

    View Slide

35. TOR descriptors
    35
    Server descriptor: Complete information about a repeater
    ExtraInfo descriptor: Extra information about the repeater
    Micro descriptor: Contains only the information necessary for
    TOR clients to communicate with the repeater
    Consensus (Network status): File issued by the authoritative
    entities of the network and made up of multiple entries of
    information on repeaters (router status entry)
    Router status entry: Information about a repeater in the
    network, each of these elements is included in the consensus
    file generated by the authoritative entities.
    

    View Slide

36. TOR spec
    36
    

    View Slide

37. Stem
    37
    from stem import Signal
    from stem.control import Controller
    with Controller.from_port(port = 9051) as
    controller:
    controller.authenticate(password='your
    password set for tor controller port in torrc')
    print("Success!")
    controller.signal(Signal.NEWNYM)
    print("New Tor connection processed")
    

    View Slide

38. Periodic Tor IP Rotation
    38
    import time
    from stem import Signal
    from stem.control import Controller
    def main():
    while True:
    time.sleep(20)
    print ("Rotating IP")
    with Controller.from_port(port = 9051) as controller:
    controller.authenticate()
    controller.signal(Signal.NEWNYM) #gets new identity
    if __name__ == '__main__':
    main()
    

    View Slide

39. Stem.Circuit status
    39
    from stem.control import Controller
    controller = Controller.from_port(port=9051)
    controller.authenticate()
    print(controller.get_info('circuit-status'))
    

    View Slide

40. Stem.Network status
    40
    from stem.control import Controller
    controller = Controller.from_port(port=9051)
    controller.authenticate(password)
    entries = controller.get_network_statuses()
    for routerEntry in entries:
    print(routerEntry)
    

    View Slide

41. Stem.circuits
    41
    

    View Slide

42. Stem.circuits
    42
    

    View Slide

43. Server descriptors
    43
    

    View Slide

44. Introduction points
    44
    

    View Slide

45. Tor nyx
    45
    https://nyx.torproject.org/
    

    View Slide

46. Tor nyx
    46
    

    View Slide

47. Tor nyx
    47
    

    View Slide

48. Tor nyx
    48
    

    View Slide

49. TorRequest
    49
    from torrequest import TorRequest
    with TorRequest() as tr:
    response = tr.get('http://ipecho.net/plain')
    print(response.text) # not your IP address
    tr.reset_identity()
    response = tr.get('http://ipecho.net/plain')
    print(response.text) # another IP address
    

    View Slide

50. Request
    50
    import requests
    def get_tor_session():
    session = requests.session()
    # Tor uses the 9050 port as the default socks port
    session.proxies = {'http': 'socks5h://127.0.0.1:9050',
    'https': 'socks5h://127.0.0.1:9050'}
    return session
    # Following prints your normal public IP
    print(requests.get("http://httpbin.org/ip").text)
    # Make a request through the Tor connection
    # Should print an IP different than your public IP
    session = get_tor_session()
    print(session.get("http://httpbin.org/ip").text)
    r = session.get('https://www.facebookcorewwwi.onion/')
    print(r.headers)
    

    View Slide

51. Analyze hidden services
    51
    1) Queries to the data sources.
    2) Filter adresses that are active.
    3) Testing against each active address and
    analysis of the response.
    4) Store URLs from websites.
    5) Perform a crawling process against each
    service
    6) Apply patterns and regular expressions to
    detect specific content(for example mail
    addresses)
    

    View Slide

52. Ahmia search engine
    52
    https://ahmia.fi/
    

    View Slide

53. Torch search engine
    53
    http://xmh57jrzrnw6insl.onion
    

    View Slide

54. UnderDir Search engine
    54
    

    View Slide

55. Hidden services
    55
    

    View Slide

56. Search Hidden services
    56
    

    View Slide

57. Other tools
    57
    POOPAK - TOR Hidden Service Crawler
    https://github.com/teal33t/poopak
    Tor spider
    https://github.com/absingh31/Tor_Spider
    Tor router
    https://gitlab.com/edu4rdshl/tor-router
    

    View Slide

58. Onnion scan
    58
    https://github.com/s-rah/onionscan
    

    View Slide

59. Dark Web map
    59
    https://www.hyperiongray.com/dark-web-map/
    

    View Slide

60. GitHub repositories
    https://github.com/serfer2/python-deepweb
    https://github.com/jmortega/python_dark_web
    60
    

    View Slide