osint + python: extracting information from tor network and darkweb

The talk will start explaining how Tor project can help us to the research and development of tools for online anonymity and privacy of its users while surfing the Internet, by establishing virtual circuits between the different nodes that make up the Tor network. Later, we will review main tools for discover hidden services in tor network with osint tools. Finally we will use python for extracting information from tor network with specific modules like stem https://stem.torproject.org/

These could be the main points of the talk:

- Introduction to Tor project and hidden services
- Discovering hidden services with osint tools
- Extracting information from tor network with python

Transcript

1. www.sti-innsbruck.at
   @jmortegac
   BSIDES MANCHESTER, 2019
   OSINT + PYTHON:
   Extracting information from TOR
   network and Darkweb
   

   View Slide

2. About me
   2
   http://jmortega.github.io/
   

   View Slide

3. About me
   3
   

   View Slide

4. About me
   4
   AGENDA
   

   View Slide

5. About me
   5
   AGENDA
   

   View Slide

6. Agenda
   • Introduction to Tor project and discover
   hidden services
   • Modules and packages we can use in
   python for connecting with Tor network
   • Tools that allow search hidden services
   and atomate the crawling process in Tor
   network
   • OSINT TOOLS for discovering hidden
   services
   6
   

   View Slide

7. Surface vs Deep vs Dark Web
   7
   

   View Slide

8. What is Tor?
   8
   • Tor is a free tool that allows people to use the
   internet anonymously.
   • Tor anonymizes the origin of your traffic
   

   View Slide

9. What is Tor?
   9
   

   View Slide

10. What is Tor?
    10
    

    View Slide

11. Onion Routing
    11
    Tor is based on Onion Routing, a technique for
    anonymous communication over a computer network.
    

    View Slide

12. 12
    Onion Routing
    

    View Slide

13. 13
    User's software or client
    incrementally builds a
    circuit of encrypted
    connections through
    relays on the network.
    Establish TOR circuit
    

    View Slide

14. 14
    When we connect to
    the TOR network, we
    do it through a circuit
    formed by 3
    repeaters, where the
    encrypted packet sent
    from the client is
    passing. Each time
    the packet goes
    through a repeater, an
    encryption layer is
    added.
    Establish TOR circuit
    

    View Slide

15. 15
    User's software or client
    incrementally builds a
    circuit of encrypted
    connections through
    relays on the network.
    Hidden services
    

    View Slide

16. 16
    Directory server
    

    View Slide

17. Hidden services
    17
    https://metrics.torproject.org/hidserv-dir-onions-seen.html
    

    View Slide

18. Tor NODE List
    18
    

    View Slide

19. Tor NODE List
    19
    https://www.dan.me.uk/tornodes
    http://torstatus.blutmagie.de
    

    View Slide

20. Tor NODE List
    20
    https://onionite.now.sh
    

    View Slide

21. Exonera TOR
    21
    https://metrics.torproject.org/exonerator.html
    

    View Slide

22. Relay search
    22
    https://metrics.torproject.org/rs.html#simple
    

    View Slide

23. Relay search
    23
    https://metrics.torproject.org/rs.html#simple
    

    View Slide

24. Relay search
    24
    https://metrics.torproject.org/rs.html#simple
    

    View Slide

25. Discover hidden services
    25
    HiddenWiki:http://wikitjerrta4qgz4.onion/
    Dark Links: http://wiki5kauuihowqi5.onion
    Tor Links: http://torlinkbgs6aabns.onion
    Dark Web Links:
    http://jdpskjmgy6kk4urv.onion/links.html
    HDWiki: http://hdwikicorldcisiy.onion
    OnionDir: http://dirnxxdraygbifgc.onion
    DeepLink: http://deeplinkdeatbml7.onion
    Ahmia: http://msydqstlz2kzerdg.onion
    

    View Slide

26. Tor onnion services
    26
    

    View Slide

27. Tor onnion services
    27
    https://en.wikipedia.org/wiki/List_of_Tor_onion_
    services
    https://en.wikipedia.org/wiki/The_Hidden_Wiki
    

    View Slide

28. TOR2web
    28
    https://www.onion.to
    

    View Slide

29. TOR browser
    29
    https://www.torproject.org/download/
    

    View Slide

30. 30
    Onion Routing
    

    View Slide

31. Installing TOR
    31
    sudo apt-get update
    sudo apt-get install tor
    sudo /etc/init.d/tor restart
    

    View Slide

32. TORrc
    32
    

    View Slide

33. Running TOR
    33
    $ tor --SocksPort 9050 --ControlPort 9051
    

    View Slide

34. Running TOR
    34
    

    View Slide

35. Tor service
    35
    service tor start/restart
    service tor status
    

    View Slide

36. Connecting with TOR
    36
    Stem
    https://stem.torproject.org/
    TorRequest
    https://github.com/erdiaker/torrequest
    Requests + socks5
    

    View Slide

37. Stem
    37
    pip install stem
    

    View Slide

38. TOR descriptors
    38
    Server descriptor: Complete information about a repeater
    ExtraInfo descriptor: Extra information about the repeater
    Micro descriptor: Contains only the information necessary for
    TOR clients to communicate with the repeater
    Consensus (Network status): File issued by the authoritative
    entities of the network and made up of multiple entries of
    information on repeaters (router status entry)
    Router status entry: Information about a repeater in the
    network, each of these elements is included in the consensus
    file generated by the authoritative entities.
    

    View Slide

39. TOR spec
    39
    

    View Slide

40. Stem
    40
    from stem import Signal
    from stem.control import Controller
    with Controller.from_port(port = 9051) as
    controller:
    controller.authenticate(password='your
    password set for tor controller port in torrc')
    print("Success!")
    controller.signal(Signal.NEWNYM)
    print("New Tor connection processed")
    

    View Slide

41. Periodic Tor IP Rotation
    41
    import time
    from stem import Signal
    from stem.control import Controller
    def main():
    while True:
    time.sleep(20)
    print ("Rotating IP")
    with Controller.from_port(port = 9051) as controller:
    controller.authenticate()
    controller.signal(Signal.NEWNYM) #gets new identity
    if __name__ == '__main__':
    main()
    

    View Slide

42. Stem.Circuit status
    42
    from stem.control import Controller
    controller = Controller.from_port(port=9051)
    controller.authenticate()
    print(controller.get_info('circuit-status'))
    

    View Slide

43. Stem.Network status
    43
    from stem.control import Controller
    controller = Controller.from_port(port=9051)
    controller.authenticate(password)
    entries = controller.get_network_statuses()
    for routerEntry in entries:
    print(routerEntry)
    

    View Slide

44. Stem.circuits
    44
    

    View Slide

45. Stem.circuits
    45
    

    View Slide

46. Server descriptors
    46
    

    View Slide

47. Introduction points
    47
    

    View Slide

48. Tor nyx
    48
    https://nyx.torproject.org/
    

    View Slide

49. Tor nyx
    49
    

    View Slide

50. Tor nyx
    50
    

    View Slide

51. Tor nyx
    51
    

    View Slide

52. VIDEO
    52
    

    View Slide

53. TorRequest
    53
    from torrequest import TorRequest
    with TorRequest() as tr:
    response = tr.get('http://ipecho.net/plain')
    print(response.text) # not your IP address
    tr.reset_identity()
    response = tr.get('http://ipecho.net/plain')
    print(response.text) # another IP address
    

    View Slide

54. Request
    54
    import requests
    def get_tor_session():
    session = requests.session()
    # Tor uses the 9050 port as the default socks port
    session.proxies = {'http': 'socks5h://127.0.0.1:9050',
    'https': 'socks5h://127.0.0.1:9050'}
    return session
    # Following prints your normal public IP
    print(requests.get("http://httpbin.org/ip").text)
    # Make a request through the Tor connection
    # Should print an IP different than your public IP
    session = get_tor_session()
    print(session.get("http://httpbin.org/ip").text)
    r = session.get('https://www.facebookcorewwwi.onion/')
    print(r.headers)
    

    View Slide

55. Analyze hidden services
    55
    1) Queries to the data sources.
    2) Filter adresses that are active.
    3) Testing against each active address and
    analysis of the response.
    4) Store URLs from websites.
    5) Perform a crawling process against each
    service
    6) Apply patterns and regular expressions to
    detect specific content(for example mail
    addresses)
    

    View Slide

56. OSINT
    56
    

    View Slide

57. Ahmia search engine
    57
    https://ahmia.fi/
    

    View Slide

58. Torch search engine
    58
    http://xmh57jrzrnw6insl.onion
    

    View Slide

59. UnderDir Search engine
    59
    

    View Slide

60. Hidden services
    60
    

    View Slide

61. Search Hidden services
    61
    

    View Slide

62. 62
    Search Hidden services
    

    View Slide

63. 63
    Search Hidden services
    

    View Slide

64. Other tools
    64
    POOPAK - TOR Hidden Service Crawler
    https://github.com/teal33t/poopak
    Tor spider
    https://github.com/absingh31/Tor_Spider
    Tor router
    https://gitlab.com/edu4rdshl/tor-router
    

    View Slide

65. DarkSeach
    65
    https://darksearch.io/
    

    View Slide

66. DarkSeach vs Ahmia
    66
    ● Both offers results directly accessible on the
    inernet thanks to Tor2Web with connecting tor
    network.
    ● DarkSeach provide a free API to automate
    searches (with some limitations to avoid the DDOS)
    ● DarkSeach indexes almost half million .onion
    addresses.Ahmia indexes almost 5.000 sites.
    ● Finally, both search engines not keep logs of
    searches done.
    

    View Slide

67. DarkSeach API
    67
    https://darksearch.io/apidoc
    

    View Slide

68. DarkSeach API
    68
    https://darksearch.io/api/search?query=bsides
    

    View Slide

69. DarkSeach API
    69
    https://darksearch.io/api/search?query=python
    

    View Slide

70. Onion investigator
    70
    https://oi.ctrlbox.com/
    

    View Slide

71. Onion investigator
    71
    https://oi.ctrlbox.com/index.php?search=apps:N
    ginx
    

    View Slide

72. Inspect onion address
    72
    https://github.com/k4m4/onioff
    

    View Slide

73. Inspect onion address
    73
    https://github.com/k4m4/onioff
    

    View Slide

74. Crawling onion address
    74
    https://github.com/DedSecInside/TorBot
    

    View Slide

75. Crawling onion address
    75
    https://github.com/DedSecInside/TorBot
    

    View Slide

76. Crawling onion address
    76
    https://github.com/MikeMeliz/TorCrawl.py
    

    View Slide

77. Crawling onion address
    77
    https://github.com/dirtyfilthy/freshonions-torscr
    aper
    

    View Slide

78. docker-onion-nmap
    78
    https://github.com/milesrichardson/docker-onio
    n-nmap
    

    View Slide

79. Onion scan
    79
    https://github.com/s-rah/onionscan
    

    View Slide

80. Dark Web map
    80
    https://www.hyperiongray.com/dark-web-map/
    

    View Slide

81. GitHub repositories
    https://github.com/serfer2/python-deepweb
    81
    

    View Slide

82. GitHub repositories
    https://github.com/jmortega/python_dark_web
    82
    

    View Slide