Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Wiff: The Wayfair Network Sniffer

B5cc90c7a3545123656605fc87e35723?s=47 Dan
May 06, 2014

Wiff: The Wayfair Network Sniffer

Talk given at Monitorama Portland

B5cc90c7a3545123656605fc87e35723?s=128

Dan

May 06, 2014
Tweet

Transcript

  1. Wiff: The Wayfair Sniffer Dan Rowe wayfair.com May 6, 2014

  2. ‹#› Who am I? ! ! Dan Rowe Wayfair.com @draco2002

    http://github/draco2003 ! Lead the Infrastructure Tools Teams - InternalTools : Customers are Employees - DevTools : Customers are Engineers ! Next Monitorama in New England? Boston 2015? I’ll bring reptiles :) !
  3. ‹#› Even Cats like Tegus

  4. Primary Engineers: Shawn Nichols and Nishan Subedi http://github.com/shnichols http://github.com/nishansubedi

  5. ‹#› Who is Wayfair? • Online retailer of home goods.

    • Offers more than 7 million products. • More than 16 million site visitors per month. • In the past year the company grew 55%. • 2013 sales reached $915 million.
  6. ‹#› Setting the Wayfair Environment Stage High level pieces of

    the puzzle: * Active / Active DC * Primarily Load-balancer -> PHP WebFarm * Everything else is a Heterogeneous Environment (PHP, Python, .NET, Java, Appliances running on Linux and Windows)
  7. ‹#› Logging Overview Syslog Commits Network Traffic App Logs (gelf)

    Unique Request ID Customer ID Files Involved Traffic Involved
  8. ‹#› Monitoring / Alerting Overview Syslog App Logs (gelf) Commits

    Network Traffic Ad Hoc Query Alerts HUD Dashboards
  9. So how about Wiff already??

  10. ‹#› What is Wiff? • Out of band network traffic

    sniffer and analyzer. • Kind of like Wireshark as a service. • Currently in production as Beta. ! ! ! ! ! ! ! ! ! ! • Essentially it is a Packet processing pipeline.
  11. ‹#› Super High Level Overview • Feed Packets in •

    Process Packets • Feed Data out • Report/Analyze • $$$ Profit $$$
  12. ‹#› Feed Packets in • Packets can be fed into

    Wiff in multiple ways. • Network interface • pcap file (or ring buffer of tcpdump files.) • RabbitMQ • egress or ingress traffic, if they are packets, we'll take'em all.
  13. ‹#› Process Packets • Based on protocol and processors enabled,

    it sifts through the packets. • Currently HTTP, HTTPS* and basic TCP are supported.
  14. ‹#› HTTPS • Requires keys to the kingdom • Need

    to map IP to key file in config • Not all SSL ciphers supported, but most are easy to add. • We don't store request or response bodies, but you can… • This is alpha as we improve performance at full volume.
  15. ‹#› Our typical HTTP Processing Flow • Packets are fed

    in ! • Wiff keeps track of connections ! • Orders the packets by sequence number ! • Stitches the payloads ! • Decrypting if needed. ! • The stream is then parsed into a response / request pair and sent to Elasticsearch
  16. ‹#› Feed Data Out • Reporters are used to send

    the processed data somewhere. • Our primary usage is send to Elasticsearch (via RabbitMQ) • Parse the stitched tcpstream into JSON Object of request / response pair. ! • Example reporter for sending to Elasticsearch for Windows/Low volume usage. !
  17. ‹#› Reporting / Analyzing / Alerting • Wiff is only

    the beginning of the pipeline. ! • Kibana friendly data format • Example/Pre-configured dashboards coming soon. ! • It’s in Elasticsearch, analyze to your hearts content. ! • Alert: • Tattle for Elasticsearch? (that's another talk ;) ) • Whatever you use now for alerting from ES queries.
  18. ‹#› Yeah great, whatever… • Webserver X can log this

    data • Application Y can log this data • Wiff is a companion tool • Not a replacement for logging at lower levels.
  19. ‹#› Where does it go? • You tell me •

    Fits where you need it. • Different configuration scenarios. • Choose your own adventure.
  20. Configuration: In front of the Load Balancer

  21. ‹#› Benfit: See what others can't • Who sees the

    errors or logs if the load balancer is mis- configured or erroring? (Other than the customer) • Web servers can only log the requests they see. • Web servers can only log the requests they complete. • Apache / Nginx don't write log line on segfault, etc.. • Application can only log requests they complete. • Logging not up high enough when needed? set-cookie anyone?
  22. ‹#› Benefit: Realtime traffic monitoring • Gives realtime visibility into

    all traffic. • Without slowing anything down • Without the need to change any other systems
  23. ‹#› Benefit: Out of band MOAWSL • Some environments have

    • a farm of web servers handling requests. • multiple types of web servers handling requests. • appliances handling some portion of requests. • lots of different log formats. ! • Single Pane of glass/Single format of data.
  24. Configuration: Outbound traffic watcher

  25. ‹#› Benefits : On the box reporting / Monitoring •

    Runs on Windows boxes to watch proprietary software. • Third Party Appliance / External api call latency • Packet RTT • Frequency of requests • Tracking / Investigating desktop traffic.
  26. ‹#› Demo (screenshots)

  27. ‹#› Demo (screenshots)

  28. ‹#› Demo (screenshots)

  29. ‹#› Demo (screenshots)

  30. ‹#› ToDos: • Improve SSL Decryption Performance. • Roll out

    and test distributed processing for scaling. • Add additional Protocol Parsers (SMTP, FTP, DNS, etc…) • Add additional Reporters
  31. ‹#› Notes: • Monitor dropped packets to reduce un-stichable requests.

    • HTTP Parser does not currently support SPDY or Websockets. • Your mileage may vary, pull requests for your environment welcome.
  32. ‹#› Thanks to all the creators of the Images used

    in the presentation: https://www.flickr.com/photos/intvgene/370973576 http://commons.wikimedia.org/wiki/File:Sausage_making-H-4.JPG https://www.flickr.com/photos/mevs/4607680584/ https://www.etsy.com/listing/175624772/ http://commons.wikimedia.org/wiki/File:Master_lock.JPG http://en.wikipedia.org/wiki/File:Colombo.Express.wmt.jpg http://en.wikipedia.org/wiki/File:Report-edit.svg http://malc50.blogspot.com/2011/12/whatever.html https://www.flickr.com/photos/tt2times/2568645910/ http://en.wikipedia.org/wiki/File:I-80_Eastshore_Fwy.jpg http://www.thecatsite.com/t/195403/bastian-and-the-tegu http://www.flickr.com/photos/streamishmc/4793978336/ http://hikethegiant.blogspot.com/2010/08/round-top-mountain-kennebec- highlands.html ! !
  33. Yes we are hiring: http://wayfair.com/careers Tell them DRowe sent you!

    Checkout the repo at: https://github.com/wayfair/wiff ! If you don’t want to build it yourself, we’ve tagged a release so you can grab the jar https://github.com/wayfair/wiff/releases/