Upgrade to Pro — share decks privately, control downloads, hide ads and more …

BoT2013 海量資料時代的網路分析

51b26506f600ed92d091ce6e2dfdcc1f?s=47 Allen Own
September 12, 2013

BoT2013 海量資料時代的網路分析


Allen Own

September 12, 2013

More Decks by Allen Own

Other Decks in Technology


  1. 海量資料時代的網路分析 BoT2013ୋʞ̨֣ᝄਜBotnetਈ಻ၾԣطҦஔ޼ীึ ॽख͍ Allen Own allenown@devco.re

  2. Who Am I ॽख͍ (Allen Own) allenown@devco.re DEVCORE Ꮦ˃੒ဧੂБڗ CHROOT

    ϓࡰ HITCON ̨ᝄᎡ܄ϋึਓᐼ̜ NISRA ༟τྠඟ௴፬ɛ ༟τҦঐږ޷ᆤᘩᒄϋڿࠏ
  3. ආБִ݁ఊЗeΆุഃသீ಻༊ਖ਼ࣩ ΂Άุeኪஔʿִ݁ఊЗᑺࢪʿᚥਪ ༟ৃτΌӻ୕޼Ӻʿක೯ ၣ༩τΌ஝ྌܔໄ Ꭱ܄ҸᏘ˓جeᎡ܄৛ᔳ ҸᏘ೻όeܝژ೻όක೯ၾ޼Ӻ EC-Council Certified Ethical Hacker

    Computer Hacking Forensic Investigator
  4. 2013/07/19 (五) ~ 20 (六) 中央研究院 ⼈人⽂文社會科學館

  5. None
  6. What is Big Data?

  7. None
  8. None
  9. None
  10. Big Data Big data[1][2] is the term for a collection

    of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, [3] search, sharing, transfer, analysis,[4] and visualization. The trend to larger data sets is due to the additional information derivable from analysis of a single large set of related data, as compared to separate smaller sets with the same total amount of data, allowing correlations to be found to "spot business trends, determine quality of research, prevent diseases, link legal citations, combat crime, and determine real-time roadway traffic conditions."[5][6][7]
  11. How Big Data fights back against APTs and Malware? http://www.seculert.com/blog/2013/05/how-big-

    data-fights-back-against-apts-and-malware.html http://info.umbrella.com/infographic-using-big-data- for-malware-protection.html
  12. What is Big Data?

  13. What is Big Data? BIG

  14. How about... 9TB?

  15. How about... 9TB?

  16. Internet Census 2012 http://internetcensus2012.bitbucket.org/ While playing around with the Nmap

    Scripting Engine (NSE) we discovered an amazing number of open embedded devices on the Internet. Many of them are based on Linux and allow login to standard BusyBox with empty or default credentials. We used these devices to build a distributed port scanner to scan all IPv4 addresses. These scans include service probes for the most common ports, ICMP ping, reverse DNS and SYN scans. We analyzed some of the data to get an estimation of the IP address usage.
  17. Internet Census 2012 http://internetcensus2012.bitbucket.org/download/ internet_census_2012.torrent Decompressing all data results in

    9TB of raw logfiles, but this code can also be used to recompress the data into gzip files. The gziped dataset should be ~1.5TB.
  18. None
  19. None
  20. Hilbert Browser http://internetcensus2012.bitbucket.org/hilbert/ index.html

  21. We prepared 26TB. Special thanks to GD.

  22. None
  23. We spent 2 months to decompress parts of zpaq files.

  24. None
  25. How to use?

  26. How to use? ANS: Use your force.

  27. None
  28. grep -e "Apache/2.2.3" *

  29. grep -e "Apache/2.2.3" * 1MBJOUFYU3VMFT

  30. But... it took 15 minutes... (one single file)

  31. Search Faster!

  32. None
  33. elasticsearch http://www.elasticsearch.org/ flexible and powerful PQFOTPVSDF, distributed real- time search

    and analytics engine for the cloud. Case Study: Fog Creek, Stack Overflow, SoundCloud, StumbleUpon, Github, foursquare, Wordpress, salesforce
  34. None
  35. You Know, for Search.

  36. I'm too lazy to code.

  37. None
  38. logstash http://logstash.net logstash is a tool for managing events and

    logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, logstash comes with a web interface for searching and drilling into all of your logs. *UJTGVMMZGSFFBOEGVMMZPQFOTPVSDF
  39. None
  40. None
  41. It works!

  42. I need stronger UI/UX!

  43. None
  44. Kibana http://kibana.org Kibana is an PQFOTPVSDF (MIT License), browser based

    interface to Logstash and ElasticSearch. Once you have those in place, Kibana is a breeze to install and configure (really, I swear). And as you'll see below, none too hard to operate. Check out the screenshots for an idea of what Kibana is all about.
  45. None
  46. None
  47. None
  48. None
  49. None


  52. 2"

  53. 5IBOLT 翁浩正 Allen Own allenown@devco.re