Processing Terabytes of data every day … and sleeping at night (infiniteConf 2019)

Processing Terabytes of data every day … and sleeping at night (infiniteConf 2019)

This is the story of how we built a highly available data pipeline that processes terabytes of network data every day, making it available to security researchers for security assessment and threat hunting.

Building this kind of stuff in the cloud is not that complicated, but if you have to make it near real-time, fault tolerant and 24/7 available, well... that's another story. In this talk, Luciano and Domagoj will tell you how they achieved this ambitious goal and how they missed a few good nights of sleep while trying to do that!

Spoiler alert: contains AWS, lambda, elastic search, monitoring, alerting & more!

F3a6662b3cd161c3c2f13604965ed0f2?s=128

Luciano Mammino

July 04, 2019
Tweet