Slide 1

Slide 1 text

Slackboard - Slack proxy server in Go Tatsuhiko Kubo@cubicdaiya Shibuya.go#2 2016/03/22

Slide 2

Slide 2 text

@cubicdaiya / Tatsuhiko Kubo Principal Engineer, SRE @ Mercari, Inc. ngx_small_light, ngx_dynamic_upstream, nginx-build, slackboard, cachectl, gaurun, widebullet, etc…

Slide 3

Slide 3 text

nginxຊΛࣥච͠·ͨ͠ʢ2016/01/16ൃചʣ

Slide 4

Slide 4 text

Slackboard

Slide 5

Slide 5 text

Slackboard • Slack proxy server in Go • https://github.com/cubicdaiya/slackboard • Provides 3 commands • slackboard - proxy server daemon for Slack • slackboard-cli - client for slackboard • slackboard-log - client like cronlog for slackboard

Slide 6

Slide 6 text

Install Slackboard go get -u github.com/cubicdaiya/slackboard/…

Slide 7

Slide 7 text

Start Slackboard slackboard -c slackboard.toml

Slide 8

Slide 8 text

Configuration for slackboard [core] # listen port port = “29800” # URL for Incoming Webhooks slack_url = “https://hooks.slack.com/services/…” [log] # access log path or redirector access_log = “stdout” # error log path or redirector error_log = “stderr” # log level level = “error” [ui] # document root for Web UI root = “/var/www/slackboard/ui”

Slide 9

Slide 9 text

slackboard-cli $ echo mercari | \ slackboard-cli \ -c tech-test \ -s slackboard-server:29800 TMBDLCPBSE POST /notify-directly Post to Slack

Slide 10

Slide 10 text

slackboard-log $ ls $ slackboard-log \ -c tech-test \ -s slackboard-server:29800 -- ls not found TMBDLCPBSE POST /notify-directly Post to Slack

Slide 11

Slide 11 text

Slackboard APIs • POST /notify • POST /notify-directly • GET /stat/go • GET /config/app See slackboard/SPEC.md about details

Slide 12

Slide 12 text

POST /notify { “tag”: “error-s3”, “title”: “failed: aws s3 cp …” “text”: “s3://bucket/path/to/xxx.log.gz”, “level”: “warn”, “sync”: false } [[tags]] tag = “error-s3” channel = “#alert-sre” user = “package” ■ slackboard.toml ■ request body ■ posted message

Slide 13

Slide 13 text

POST /notify-directly { “payload”: { “channel”: “random”, “username”: “slackboard”, “icon_emoji”: “:cloud:”, “text”: “:bow:”, “attachments”: […] }, “sync”: false } ■ request body ■ posted message

Slide 14

Slide 14 text

Difference between /notify and /notify-directly • POST /notify • receives original payload • tag-based notification • POST /notify-directly • proxies payload to Slack directly • channel-based notification

Slide 15

Slide 15 text

Asynchronous or synchronousʁ • slackboard proxies message to slack asynchronous by default • “sync”: false • slackboard returns 200 OK to client at once • “sync”: true • slackboard return 200 OK after posted message to Slack

Slide 16

Slide 16 text

Attachments • Slack provides mechanism for rich-formatted message with Attachments • coloring, tiltling, authoring • embed image • long text • etc…

Slide 17

Slide 17 text

Colorize message by POST /notify echo information | slackboard-cli -t test -s slackboard-server:29800 -l info echo warning | slackboard-cli -t test -s slackboard-server:29800 -l warn echo critical | slackboard-cli -t test -s slackboard-server:29800 -l crit TMBDLCPBSE POST /notify Post to Slack

Slide 18

Slide 18 text

Colorize message by POST /notify-directly echo good | slackboard-cli -c random -s slackboard-server:29800 -C good echo bad | slackboard-cli -c random -s slackboard-server:29800 -C bad echo warning | slackboard-cli -c random -s slackboard-server:29800 -C warning TMBDLCPBSE POST /notify-directly Post to Slack

Slide 19

Slide 19 text

Alerting to Slack

Slide 20

Slide 20 text

Alerting to Slack with slackboard-cli # copy log file to Google Cloud Storage gsutil cp 2015-03-05T08.log.gz gs://${bucket}/ result=`echo $?` if [ $result -ne 0 ]; then echo "@channel: failed to copy log to GCS(2015-03-05T08)” | \ slackboard-cli -s slackboard-host:29800 -t error-gcs fi

Slide 21

Slide 21 text

Alerting to Slack with slackboard-log # copy log file to Google Cloud Storage slackboard-log -s slackboard-host:29800 -t error-gcs -- \ gsutil cp 20150305T08.log.gz gs://${bucket}/

Slide 22

Slide 22 text

fluent-plugin-slackboard https://github.com/cubicdaiya/fluent-plugin-slackboard gem install fluent-plugin-slackboard Alerting to Slack with Fluentd

Slide 23

Slide 23 text

fluent-plugin-slackboard # required type slackboard host 127.0.0.1 port 29800 channel random fetch_key message # optional username slackboard icon_emoji :clipboard: parse full sync false echo ‘{“message”: “bokko”}’ | \ fluent-cat slackboard TMBDLCPBSE qVFOUE

Slide 24

Slide 24 text

By the way,

Slide 25

Slide 25 text

Why is proxy server for Slack required? • Messages are posted from various servers • Where are messages posted from? • Messages are posted from various programs • Perl?Python?PHP?Ruby? or… Which library? • Different by engineer and team

Slide 26

Slide 26 text

Slackboard’s approach • Aggregate messages to Slack and logs by • slackboard • Unify notification method by • slackboard-cli • slackboard-log

Slide 27

Slide 27 text

In the case @

Slide 28

Slide 28 text

Before

Slide 29

Slide 29 text

It’s painfulʂ • Incoming Webhooks URL is staggled in various servers • What if URL is changed? • Each client program uses different language and libraries • (ϊ ʄДʄ)ϊ~~~ᵲᵲ

Slide 30

Slide 30 text

Now

Slide 31

Slide 31 text

Summary • Slackboard provides • Proxy server daemon for Slack • Client for slackboard • The goal of Slackboard • Aggregate messages to Slack • Unifies method for posting message to Slack

Slide 32

Slide 32 text

We are hiring Go Engineer! https://www.mercari.com/jp/jobs/