System audit log Web server log File sharing log Firewall log IDS log VPN log Challenge and Opportunity Security Log Analysis Log Explosion Features of security logs in the modern era: Large amount of logs Vast variety of logs Distributed log generation
No DFS in clouds by default Design for immutable files MapReduce w/ Distributed File System Lightweight design User specific data flow Distributed log importing Streaming log analysis Our Approach Our Approach Overview Workflow App Example Our easy-to-use framework for parallel log analysis Existing general purpose parallel computing framework
6 Security Event Occurrence Counter (IP counter) o Denial-of-service attack detection o Botnet detection o User pattern analysis / anomaly detection We use a three-layer hashmap counter to realize the application
Sanjay Ghemawat. "MapReduce: simplified data processing on large clusters." Communications of the ACM 51.1 (2008): 107-113. Logothetis, Dionysios, et al. "In-situ mapreduce for log processing." 2011 USENIX Annual Technical Conference (USENIX ATC’11). 2011. Yang, Shun-Fa, Wei-Yu Chen, and Yao-Tsung Wang. "ICAS: An inter-VM IDS Log Cloud Analysis System." Cloud Computing and Intelligence Systems (CCIS), 2011 IEEE International Conference on. IEEE, 2011. Francois, Jerome, et al. "BotCloud: Detecting botnets using MapReduce." Information Forensics and Security (WIFS), 2011 IEEE International Workshop on. IEEE, 2011. Feng, Junqiu, et al. "Elastic stream cloud (ESC): A stream-oriented cloud computing platform for Rich Internet Application." High Performance Computing and Simulation (HPCS), 2010 International Conference on. IEEE, 2010. Andreolini, Mauro, Michele Colajanni, and Stefania Tosi. "A software architecture for the analysis of large sets of data streams in cloud infrastructures." Computer and Information Technology (CIT), 2011 IEEE 11th International Conference on. IEEE, 2011.