Threat Detection Using Time Series Analysis of Darknet Probes and OSSEC Reports

Transcript

1. Threat Detection
   Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 1 / 83
   

   View Slide

2. Threat Detection Using Time Series Analysis and
   Summary Statistics of Darknet Probes and OSSEC
   Reports
   Clay Wells
   School of Arts and Sciences
   University of Pennsylvania
   clayw@sas.upenn.edu
   April 19, 2016
   

   View Slide

3. About
   Setting The Stage
   Definitions
   Software Used
   Data Sources
   Our Approach
   Overview of Our Approach
   Darknet Setup
   OSSEC
   Data Preparation
   Darknet Data
   OSSEC Report Data
   Using R
   Summary
   Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 1 / 83
   

   View Slide

4. Overview
   About
   Setting The Stage
   Definitions
   Software Used
   Data Sources
   Our Approach
   Overview of Our Approach
   Darknet Setup
   OSSEC
   Data Preparation
   Darknet Data
   OSSEC Report Data
   Using R
   Summary
   Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 2 / 83
   

   View Slide

5. About - Me
   • ≈ 9 years at UPenn
   • ≈11 months in current position
   • ≈ 8 years department of biostatistics & epidemiology
   • 10 years at UF (programmer, sysadmin)
   • Red Hat 4.0 Colgate, based on 2.0.18 kernel (not RHEL 4)
   Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 3 / 83
   

   View Slide

6. About - UPenn, By The Numbers
   Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 4 / 83
   

   View Slide

7. About - UPenn, By The Numbers
   Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 5 / 83
   

   View Slide

8. About - UPenn, By The Numbers
   Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 6 / 83
   

   View Slide

9. About - UPenn, By The Numbers
   Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 7 / 83
   

   View Slide

10. About - Inspiration
    • Lots of data from a various sources
    Darknet sensor ⇒ OSSEC-HIDS ⇒ HECTOR
    • What can we learn from our data?
    • How can we use our data for threat detection?
    • We need to do something, need to start somewhere!
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 8 / 83
    

    View Slide

11. About - Inspiration, Tons of Data
    Report ’OSSEC Report: Successful Auths’ completed.
    ------------------------------------------------
    ->Processed alerts: 687351
    ->Post-filtering alerts: 4085
    ->First alert: 2016 Jan 25 00:00:01
    ->Last alert: 2016 Jan 25 23:59:39
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 9 / 83
    

    View Slide

12. About - Inspiration, Tons of Data
    Top entries for ’Source ip’:
    ------------------------------------------------
    12X.XX.XXX.XX |860
    10.0.X.XX |663
    10.0.X.XX |413
    12X.XX.XXX.X8 |288
    13X.XX.XXX.X3 |52
    12X.XX.XXX.XX |51
    98.115.235.181 |24
    76.99.36.224 |13
    12X.XX.XXX.XXX |11
    13X.XX.XX.XXX |10
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 10 / 83
    

    View Slide

13. About - Inspiration, Tons of Data
    Top entries for ’Username’:
    ------------------------------------------------
    root |1278
    nag |518
    nagmon |384
    mavvel |62
    m_nagios |51
    nag_mon |49
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 11 / 83
    

    View Slide

14. About - Inspiration, Tons of Data
    Top entries for ’Location’:
    ------------------------------------------------
    (host1.xxx.X.edu) 128.XX.XXX.XX-.. |1119
    (host2.xxx.X.edu) 128.XX.XXX.XX->/va.. |585
    (host3.xxx.X.edu) 128.XX.XXX.XX->/var/log/..|406
    (host4.xxx.X.edu) 128.XX.XX.XX->/var/l.. |343
    (host4.xxx.X.edu) 128.XX.XX.XX->/var/l.. |343
    (host5.xxx.X.edu) 128.XX.XXX.XXX->/va.. |278
    (host6.xxx.X.edu) 128.XX.XXX.XXX->/var/l.. |188
    (host6.xxx.X.edu) 128.XX.XXX.XXX->/var/l.. |188
    (host7.xxx.X.edu) 128.XX.XXX.XXX->/var/.. |148
    (host7.xxx.X.edu) 128.XX.XXX.XXX>/var/.. |148
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 12 / 83
    

    View Slide

15. About - Inspiration
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 13 / 83
    

    View Slide

16. About - Inspiration
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 14 / 83
    

    View Slide

17. About - Inspiration
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 15 / 83
    

    View Slide

18. About - Inspiration, Spaghetti Plot
    http://www.nicebread.de/visually-weighted-regression-in-r-a-la-solomon-hsiang/
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 16 / 83
    

    View Slide

19. About - Inspiration, Watercolor Plot
    http://www.nicebread.de/visually-weighted-regression-in-r-a-la-solomon-hsiang/
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 17 / 83
    

    View Slide

20. Thank you, Internet
    { data, dropped packets, threats, blackhole,
    ip header, blurry hands, internet, ... }
    for all the lovely images
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 18 / 83
    

    View Slide

21. About - Future Work
    Green star, lower right corner
    (*)
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 19 / 83
    

    View Slide

22. Overview
    About
    Setting The Stage
    Definitions
    Software Used
    Data Sources
    Our Approach
    Overview of Our Approach
    Darknet Setup
    OSSEC
    Data Preparation
    Darknet Data
    OSSEC Report Data
    Using R
    Summary
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 20 / 83
    

    View Slide

23. Setting The Stage - Evidence-Based Decision Making
    Good displays of data help to reveal knowledge relevant to understanding
    mechanism, process and dynamics, cause and effect.
    - Edward Tufte
    Visual Explanations: Images and Quantities, Evidence and Narrative.
    (*)
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 21 / 83
    

    View Slide

24. Definitions - Threats
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 22 / 83
    

    View Slide

25. Setting the Stage - Threats
    Source: European Union Agency for Network and Information Security - Threat Landscape 2015
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 23 / 83
    

    View Slide

26. Definitions - Data Sources
    • Darknet sensor ⇐ Blackhole, dead address
    • Dropped packets ⇐ Unsolicited traffic sent to our sensor
    • OSSEC ⇐ Host-based Intrusion Detection System
    (*)
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 24 / 83
    

    View Slide

27. Definitions - Summary Statistics
    • Quartile
    • Interquartile Range (IQR)
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 25 / 83
    

    View Slide

28. Summary Statistics - Example
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 26 / 83
    

    View Slide

29. Software Used
    • iptables
    • OSSEC
    • HECTOR (SEIM - Justin Klein Keane)
    • Python
    • RStudio IDE for R - Why not Python?
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 27 / 83
    

    View Slide

30. Data Sources
    • Darknet sensor ⇒ OSSEC ⇒ HECTOR
    • Fantastic foundation
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 28 / 83
    

    View Slide

31. Data Sources - Darknet Probe Reports
    Port Number Hit Count
    -------------------------
    25/tcp 644
    0/udp 128
    23/tcp 64
    6000/tcp 27
    21320/tcp 21
    3389/tcp 20
    (*)
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 29 / 83
    

    View Slide

32. Data Sources - Analyzing OSSEC Alerts
    Typical OSSEC alert
    OSSEC HIDS Notification.
    2015 Nov 16 07:16:16
    Received From: (xxxx.upenn.edu) 128.xx.xx.xx -> /var/log/secure
    Rule: 5701 fired (level 8) -> "Possible attack on the ssh server (or version gathering)."
    Portion of the log(s):
    Nov 16 07:16:15 writing sshd[8284]: Bad protocol version identification ’SSH-2.0’ from UNKNOWN
    OSSEC HIDS Notification.
    2015 Nov 16 07:16:18
    Received From: (xxxx.upenn.edu) 128.xx.xx.xx -> /var/log/secure
    Rule: 5701 fired (level 8) -> "Possible attack on the ssh server (or version gathering)."
    Portion of the log(s):
    Nov 16 07:16:17 writing sshd[8284]: Bad protocol version identification ’SSH-2.0’ from UNKNOWN
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 30 / 83
    

    View Slide

33. Setting the Stage - HECTOR
    • Interally developed SEIM
    • Justin Klein Keane
    • https://github.com/madirish/hector
    (*)
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 31 / 83
    

    View Slide

34. Setting the Stage - HECTOR
    • https://github.com/madirish/hector
    (*)
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 32 / 83
    

    View Slide

35. Setting the Stage - Start Asking Questions
    • Visibility into active attacks/threats?
    • What’s being targeted?
    • What can/should we measure?
    • What more can we learn?
    (*)
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 33 / 83
    

    View Slide

36. Setting the Stage - OSSEC Reports, Digging Deeper
    Created OSSEC reports (daily, intra-day) by group, location, all locations
    • Attack alerts
    • Drupal attacks
    • Successful logins
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 34 / 83
    

    View Slide

37. Goals - Insights, Questions, Clarity
    We’re not setting out to find answers.
    • What type of packets are sensors receiving?
    • What new data might we be interested in capturing?
    • Establish and track baselines over time?
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 35 / 83
    

    View Slide

38. Overview
    About
    Setting The Stage
    Definitions
    Software Used
    Data Sources
    Our Approach
    Overview of Our Approach
    Darknet Setup
    OSSEC
    Data Preparation
    Darknet Data
    OSSEC Report Data
    Using R
    Summary
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 36 / 83
    

    View Slide

39. Overview of Our Approach - Data Mining
    • Data
    • Create scripts to generate datasets (Target Data)
    • Clean/prepare data (Preprocessed Data)
    • Create smaller datasets (Transformed Data)
    • Create visualizations (Patterns)
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 37 / 83
    

    View Slide

40. Darknet Setup - Log Dropped Packets
    iptables -A INPUT -d 128.XXX.XXX.XXX/32 -m state \
    --state NEW -m comment --comment "Log dropped packets" \
    -j LOG --log-prefix "iptables "
    --log-ip-options --log-tcp-options
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 38 / 83
    

    View Slide

41. Darknet Setup - Log Dropped Packets
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 39 / 83
    

    View Slide

42. Darknet Setup - Logged to Syslog
    Dropped packet entry in /var/log/message
    Apr 3 14:14:02 host01 kernel: [1052872.333883] iptables
    IN=eth0 OUT= MAC=b8:xx SRC=46.161.40.120 DST=xxx.xx.xx.xx
    LEN=40 TOS=0x08 PREC=0x40 TTL=237 ID=52036 PROTO=TCP
    SPT=41427 DPT=3389 WINDOW=1024 RES=0x00 SYN URGP=0
    (*)
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 40 / 83
    

    View Slide

43. Darknet Setup - Custom OSSEC rule
    
    4100
    iptables
    
    Darknet sensor detection for HECTOR.
    
    
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 41 / 83
    

    View Slide

44. Darknet Setup - Search OSSEC rule ID
    $ sudo find /var/ossec/rules -type f -exec grep -H 4100 {} \;
    /../rules/local_rules.xml:
    /../rules/local_rules.xml: 4100
    /../rules/racoon_rules.xml:
    /../rules/racoon_rules.xml: 14100
    /../rules/firewall_rules.xml:
    /../rules/firewall_rules.xml: 4100
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 42 / 83
    

    View Slide

45. Darknet Setup - OSSEC firewall rule
    
    
    firewall
    Firewall rules grouped.
    
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 43 / 83
    

    View Slide

46. Darknet Setup - Darknet SQL Table
    -- Darknet sensor table
    CREATE TABLE IF NOT EXISTS ‘darknet‘ (
    ‘id‘ INT NOT NULL AUTO_INCREMENT,
    ‘src_ip‘ INT UNSIGNED NOT NULL,
    ‘dst_ip‘ INT UNSIGNED NOT NULL,
    ‘src_port‘ INT UNSIGNED NOT NULL,
    ‘dst_port‘ INT UNSIGNED NOT NULL,
    ‘proto‘ ENUM(’tcp’,’udp’,’icmp’),
    ‘country_code‘ VARCHAR(2),
    ‘received_at‘ TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (‘id‘),
    INDEX USING HASH (‘src_ip‘)
    ) ENGINE = INNODB;
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 44 / 83
    

    View Slide

47. OSSEC - Overview
    Host based Intrusion Detection System
    • Integrated log analysis
    • Windows registry monitoring
    • Rootkit detection
    • Active response
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 45 / 83
    

    View Slide

48. OSSEC - Rules
    
    104250
    admin/
    Drupal access denied to admin screen.
    
    drupal
    
    
    104260
    
    Multiple Drupal access denied to admin
    screens.
    drupal
    
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 46 / 83
    

    View Slide

49. OSSEC - Reports
    cat /var/ossec/logs/alerts/alerts.log | \
    /var/ossec/bin/ossec-reportd -n ’Drupal Alert Report’ \
    -f group drupal -r location rule -r rule location
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 47 / 83
    

    View Slide

50. OSSEC - Report Results
    Report ’Drupal Alert Report’ completed.
    ------------------------------------------------
    ->Processed alerts: 818949
    ->Post-filtering alerts: 114023
    ->First alert: 2016 Apr 03 00:00:10
    ->Last alert: 2016 Apr 03 16:08:19
    Top entries for ’Source ip’:
    ------------------------------------------------
    10.0.0.4 |64074
    127.0.0.1 |8851
    157.55.39.224 |996
    207.46.13.160 |867
    68.180.228.230 |814
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 48 / 83
    

    View Slide

51. OSSEC - Report Results
    Top entries for ’Level’:
    ------------------------------------------------
    Severity 3 |110525
    Severity 6 |3279
    Severity 10 |114
    Severity 11 |92
    Severity 12 |8
    Severity 9 |5
    Top entries for ’Group’:
    ------------------------------------------------
    drupal |114023
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 49 / 83
    

    View Slide

52. OSSEC - Report Results
    (neutron.x.edu) 128.x.x.x->/var/log/.. |63586
    rule: ’104250’
    rule: ’104220’
    rule: ’104260’
    rule: ’104262’
    104220 - Drupal failed login! |1322
    location: ’(neutron.x.edu) 128.x.x.x->/var/log/messages’
    location: ’(quasar.x.edu) 128.x.x.x->/var/log/messages’
    location: ’(ah-web1) 54.x.x.x->/var/log/messages’
    location: ’(ah-web2) 54.x.x.x->/var/log/messages’
    104262 - Multiple Drupal access denied to ad.. |8
    location: ’(quasar.x.edu) 128.x.x.x->/var/log/messages’
    location: ’(neutron.x.edu) 128.x.x.x->/var/log/messages’
    (*)
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 50 / 83
    

    View Slide

53. OSSEC - Darknet Report Results
    Top entries for ’Location’:
    ------------------------------------------------
    (host02) 16x.x.x.x->/var/log/messages |802 |
    (host01) 16x.x.x.x->/var/log/messages |516 |
    (host03) 128.x.x.x->/var/log/messages |454 |
    Top entries for ’Rule’:
    ------------------------------------------------
    200001 - Darknet sensor detection for HECTOR. |1772 |
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 51 / 83
    

    View Slide

54. OSSEC - Darknet Report Results
    Related entries for ’Source ip’:
    ------------------------------------------------
    128.x.x.x |122 |
    location: ’(host02) 16x.x.x.7->/var/log/messages’
    location: ’(host01) 16x.x.x.11->/var/log/messages’
    58.213.133.130 |84 |
    location: ’(host03) 128.x.x.x->/var/log/messages’
    80.82.78.38 |25 |
    location: ’(host02) 16x.x.x.7->/var/log/messages’
    location: ’(host01) 16x.x.x.11->/var/log/messages’
    location: ’(host03) 128.x.x.x->/var/log/messages’
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 52 / 83
    

    View Slide

55. OSSEC - Alerts SQL Table
    -- OSSEC alerts from clients
    CREATE TABLE IF NOT EXISTS ‘ossec_alert‘ (
    ‘alert_id‘ INT NOT NULL AUTO_INCREMENT,
    ‘alert_date‘ TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    ‘host_id‘ INT NOT NULL,
    ‘alert_log‘ VARCHAR(255) DEFAULT NULL,
    ‘rule_id‘ INT NOT NULL,
    ‘rule_src_ip‘ VARCHAR(15) DEFAULT NULL,
    ‘rule_src_ip_numeric‘ INT UNSIGNED,
    ‘rule_user‘ VARCHAR(20) DEFAULT NULL,
    ‘rule_log‘ TEXT DEFAULT NULL,
    ‘alert_ossec_id‘ VARCHAR(50) NOT NULL,
    PRIMARY KEY (‘alert_id‘),
    INDEX (‘host_id‘), INDEX (‘rule_id‘),
    INDEX USING HASH (‘rule_src_ip_numeric‘),
    INDEX USING HASH (‘rule_id‘),
    INDEX USING HASH (‘host_id‘),
    INDEX USING BTREE (‘alert_date‘)
    ) ENGINE = INNODB;
    (*)
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 53 / 83
    

    View Slide

56. Data We’re Looking At
    • Darknet Probes, 2012-2016 (2014)
    • Attack severity levels,
    • Attack count over 14 days, intraday
    (*)
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 54 / 83
    

    View Slide

57. Pause for Questions
    Questions?
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 55 / 83
    

    View Slide

58. Overview
    About
    Setting The Stage
    Definitions
    Software Used
    Data Sources
    Our Approach
    Overview of Our Approach
    Darknet Setup
    OSSEC
    Data Preparation
    Darknet Data
    OSSEC Report Data
    Using R
    Summary
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 56 / 83
    

    View Slide

59. Data Preparation - Darknet Data
    • Query the DB
    • Clean/prepare the data for R
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 57 / 83
    

    View Slide

60. Data Preparation - Darknet Query Script
    report-darknet.py ⇒ data-darknet-all.csv
    2016-04-03 21:01:39, 2222, tcp, 2222/tcp, 52.37.175.240, US
    2016-04-03 21:01:45, 2222, tcp, 2222/tcp, 52.37.175.240, US
    2016-04-03 21:04:27, 2222, tcp, 2222/tcp, 85.25.200.140, DE
    2016-04-03 21:09:23, 2222, tcp, 2222/tcp, 59.45.79.103, CN
    2016-04-03 21:09:51, 2222, tcp, 2222/tcp, 59.45.79.103, CN
    2016-04-03 21:18:39, 2222, tcp, 2222/tcp, 182.126.161.124, CN
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 58 / 83
    

    View Slide

61. Data Preparation - OSSEC Report Data
    • Parse OSSEC reports (archived logs)
    • Clean/prepare the data
    (*)
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 59 / 83
    

    View Slide

62. Data Preparation - OSSEC Report Data
    get-severity.sh ⇒ severity-2015-mon−d.csv
    Top entries for ’Level’:
    ------------------------------------------------
    Severity 6 |718
    Severity 10 |14
    Severity 8 |4
    Severity 12 |1
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 60 / 83
    

    View Slide

63. Data Preparation - OSSEC Report Results
    get-severity.sh ⇒ severity-2015-mon−d.csv
    2015-Oct-30, Severity 6, 147
    2015-Oct-30, Severity 10, 12
    2015-Oct-30, Severity 8, 2
    2015-Oct-30, Severity 12, 1
    2015-Oct-30, Severity 3, 1
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 61 / 83
    

    View Slide

64. Overview
    About
    Setting The Stage
    Definitions
    Software Used
    Data Sources
    Our Approach
    Overview of Our Approach
    Darknet Setup
    OSSEC
    Data Preparation
    Darknet Data
    OSSEC Report Data
    Using R
    Summary
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 62 / 83
    

    View Slide

65. Using R - Dataframes and Summary Statistics
    • Dataframe for each port
    • Dataframe by year
    • Frequency dataframe by month
    • Compute summary statistics
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 63 / 83
    

    View Slide

66. Using R - Importing Data, dataframes
    data.darknet.all <- read.csv("data-darknet-all-clean.csv")
    data.Timestamp <- as.POSIXlt(data.darknet.all$Timestamp)
    data.darknet.all.0 <-
    data.darknet.all[data.darknet.all$Port %in% c(0),]
    data.darknet.0.2014 <-
    data.darknet.all.0[data.darknet.all.0$Year==2014,]
    port0.2014 <- rowsum(data.darknet.0.2014$Count,
    group=data.darknet.0.2014$Month)
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 64 / 83
    

    View Slide

67. R Functions, Summary Statistics
    port0.2014.median <- median(port0.2014)
    port0.2014.avg <- mean(port0.2014)
    port0.2014.top <- quantile(port0.2014, 3/4)
    port0.2014.low <- quantile(port0.2014, 1/4)
    port0.2014.max <- max(port0.2014)
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 65 / 83
    

    View Slide

68. Using R - Darknet Visualizations
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 66 / 83
    

    View Slide

69. Using R - Darknet Visualizations, Outliers
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 67 / 83
    

    View Slide

70. Using R - Darknet Visualizations, Modified
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 68 / 83
    

    View Slide

71. Using R - Darknet Visualizations
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 69 / 83
    

    View Slide

72. Using R - Darknet Visualizations
    port0.all dataframe
    201205 10, 201309 780, 201310 1025, 201311 914
    201312 815, 201401 457, 201402 800, 201403 1050
    201404 1075, 201405 670, 201406 678, 201407 951
    201408 1076, 201409 1551, 201410 1979, 201411 1478
    201412 2387, 201501 2340, 201502 2512, 201503 1205
    201509 2086, 201510 2104, 201511 3129, 201512 2518
    201601 1480, 201602 6832
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 70 / 83
    

    View Slide

73. R Plotting Functions
    plot(port0.all, type=’b’, xlab=’Month’,
    ylab=’Freq’, ylim=c(10,port0.all.max),
    main=’Port 0/UDP Probes, 2012-2016’)
    abline(h=port0.all.median, v=NULL, col="blue", lty="dashed")
    abline(h=port0.all.avg, v=NULL, col="orange", lty="dashed")
    abline(h=port0.all.top, v=NULL, col="grey75", lty="dotted")
    abline(h=port0.all.low, v=NULL, col="grey75", lty="dotted")
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 71 / 83
    

    View Slide

74. Using R - Darknet Visualizations
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 72 / 83
    

    View Slide

75. Using R - Darknet Visualizations
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 73 / 83
    

    View Slide

76. 2014, Heartbleed
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 74 / 83
    

    View Slide

77. 2014 Port 443 Probes - Events?
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 75 / 83
    

    View Slide

78. 2014, POODLE
    http://arstechnica.com/security/2014/10/ssl-broken-again-in-poodle-attack/
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 76 / 83
    

    View Slide

79. Data Visualization - Attack Alerts
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 77 / 83
    

    View Slide

80. Using R - Demo
    RScript Demo
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 78 / 83
    

    View Slide

81. Overview
    About
    Setting The Stage
    Definitions
    Software Used
    Data Sources
    Our Approach
    Overview of Our Approach
    Darknet Setup
    OSSEC
    Data Preparation
    Darknet Data
    OSSEC Report Data
    Using R
    Summary
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 79 / 83
    

    View Slide

82. Summary
    • Dig into existing data, start asking questions
    • Start creating simple visualizations
    • Start compiling summary statistics
    • Continue adding logs, logs, and more logs!
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 80 / 83
    

    View Slide

83. Resources
    • iptables - http://netfilter.org/projects/iptables/
    • OSSEC - https://ossec.github.io/
    • HECTOR - https://github.com/madirish/hector
    • R - https://www.r-project.org/
    • RStudio - urlhttps://www.rstudio.com/
    • Lattice: Multivariate Data Visualization with R by Sarkar, Deepayan
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 81 / 83
    

    View Slide

84. Statistics & Data Visualization, Learning More
    Books
    • Statistics, 4th Edition by Freedman, Pisani, Purves
    • The Visual Display of Quantitative Information by Edward Tufte
    Data Visualization Video Playlist, Ted Talks
    • https://www.youtube.com/playlist?\list=
    PL31HXzEkTvxDFiKSZa-Ylw33bQ_NeHyGk
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 82 / 83
    

    View Slide

85. Thank You!
    { You, EDUCAUSE, Christine Brisson, Warren Petrofsky,
    Justin Klein Keane, InfoSec Reading Group,
    UPenn ISC Security Team, John Mulhern III, ... }
    Questions?
    Clay Wells (clayw@sas.upenn.edu) Threat Detection April 19, 2016 83 / 83
    

    View Slide