Norikra defends web service

Norikra meetup #2

Transcript

1. Norikra defends web
   service
   Norikra meetup #2
   2015.06.03 @fujiwara
   

   View Slide

2. Fujiwara Shunichiro (@fujiwara)
   KAYAC Inc.
   

   View Slide

3. Norikra in a web service
   A community web service.
   • Realtime log analysis
   • HTTP status code, response time
   • worker logs
   • Switched from fluent-plugin-(datacounter|numeric-monitor)
   • Peak 10,000~ msgs/sec
   

   View Slide

4. Norikra in a web service
   

   View Slide

5. Norikra in a web service
   EC2 : r3.xlarge (4core 30GB mem)
   norikra start \
   -Xmx23012m \
   --small \
   --dump-stat-interval=300 \
   --stats=${DATA_DIR}/norikra.stats.json \
   --pidfile=${DATA_DIR}/norikra.pid \
   --gc-log=${DATA_DIR}/gc.log \
   -XX:+UseGCLogFileRotation \
   -XX:NumberOfGCLogFiles=10 \
   -XX:GCLogFileSize=1048576 \
   

   View Slide

6. Spammers has come
   Smartphone apps only at first.
   Web version (for browsers) released!!
   But, it gives many benefits for ATTACKERS...
   • Easy to analyze API protocols
   • Easy to create MANY accounts (email, twitter...)
   → Spam floods!!!
   

   View Slide

7. Norikra detects spam POSTs
   SELECT 'spam-detect' as tag,
   count(*) as calls,
   min(time) as time_min,
   max(time) as time_max,
   N as threshold,
   host
   FROM accesslog.win:time_batch(1 min)
   WHERE uri LIKE '/api/%' and method = 'POST'
   GROUP BY host
   HAVING count(*) >= N
   

   View Slide

8. Norikra detects spam account creation
   SELECT 'spam-detect' as tag,
   count(*) as calls,
   min(time) as time_min,
   max(time) as time_max,
   N as threshold,
   host
   FROM accesslog.win:time_batch(1 min)
   WHERE uri LIKE '/api/signup/twitter%'
   GROUP BY host
   HAVING count(*) >= N
   

   View Slide

9. fluent-plugin-spam-reactor
   In-house plugin
   
   type norikra
   norikra log-analyzer.service.consul:26571
   
   method sweep
   tag field tag
   tag_prefix norikra
   interval 3s
   
   
   
   type spam_reactor
   whitelist ["127.0.0.1", "192.168.0.0/16"]
   
   

   View Slide

10. fluent-plugin-spam-reactor
    • Norikra detects spammers IP address from access log
    • Fluentd fetches Norikra outputs as records
    

    View Slide

11. fluent-plugin-spam-reactor
    • Calc : count / (time_max - time_min) >= threshold
    • Set the IP address to memcached (exclude whitelist)
    memd.set("banned-host:#{host}", 1, expires, false)
    memd.set("banned-host-count:#{host}", count, 86400, false)
    • Emit a report message
    router.emit("spam-report", time, {"message" => message})
    WebApp BLOCKS accesses from IP address in memcached!
    

    View Slide

12. Merits of using memcached
    1. Flexible expires.
    • spam-reactor extends expires by exponential backoff.
    2. Accessibility between middlewares.
    • WebApp (Perl, Go)
    • Fluentd (Ruby)
    • Nginx (module, lua)
    

    View Slide

13. Conclusion
    • Norikra can detect spam accesses.
    • Easy to customize logic by queries
    • Fluentd custom plugins fit to your applications.
    

    View Slide