Manage cloud server with open source tools

Slide 1

Slide 1 text

#JCConf .BOBHF4FSWFST0O5IF $MPVEXJUI0QFO4PVSDF5PPMT 何永琳 @ Fliptop Director of Engineering ⼯工頭

Slide 2

Slide 2 text

陾玑 • 捀➊랂䧮⦛꨽銳湋䱽ծ庠ꆀ • 剣➊랂䭷垥꨽銳湋䱽ծ庠ꆀ • 剣➊랂䊨Ⱘ〳⟄䍲⸔䧮⦛ • NFUSJDTCZDPEBIBMF • 33%5PPMT

Slide 3

Slide 3 text

锞铇㻜涸㔐瘸鸏⦐㉏겗

Slide 4

Slide 4 text

⡹♧㣔漗䎙⦐㼭儘 ref: http://xkcd.com/386/

Slide 5

Slide 5 text

⡹♧㣔漗䎙⦐㼭儘 ref: http://xkcd.com/386/

Slide 6

Slide 6 text

⡹♧㣔漗䎙⦐㼭儘 ref: https://jawbone.com/up

Slide 7

Slide 7 text

⡹♧㣔漗䎙⦐㼭儘 ref: https://jawbone.com/up

Slide 8

Slide 8 text

植㻜䖃䖃嫲⡹䟝⫹涸婫ꃏ

Slide 9

Slide 9 text

铇㻜涸꬗㼩荈䧮僽佖㊥涸Ꟛ㨥

Slide 10

Slide 10 text

㥶卓䧮⦛㼩荈魨涸钢濼鿪剤铐䊴 齡䧮⦛㻨涸鮿넓ヤ

Slide 11

Slide 11 text

4PGUXBSF-JGFDZDMF

Slide 12

Slide 12 text

斊麌♳涸朜屣 • 㣐ꆀ崨ꆀ弈Ⰶ • ♳涰㣔涸麋遤 • 顺俱䏨⪓⟧ • 㢪鿈剪⹢ 40" .JDSP4FSWJDF ㉏겗 • 䳕倝炽넒

Slide 13

Slide 13 text

斊麌♳涸朜屣 • 㣐ꆀ崨ꆀ弈Ⰶ • ♳涰㣔涸麋遤 • 顺俱䏨⪓⟧ • 㢪鿈剪⹢㉏겗 • 䳕倝炽넒 • 5ISPVHIQVU • 3FTQPOTFUJNF • XPSTUDBTF35

Slide 14

Slide 14 text

斊麌♳涸朜屣 • 㣐ꆀ崨ꆀ弈Ⰶ • ♳涰㣔涸麋遤 • 顺俱䏨⪓⟧ • 㢪鿈剪⹢㉏겗 • 䳕倝炽넒 • .FNPSZ-FBL • $BDIF6TBHF )JU .JTT • +7.)FBQTJ[F

Slide 15

Slide 15 text

斊麌♳涸朜屣 • 㣐ꆀ崨ꆀ弈Ⰶ • ♳涰㣔涸麋遤 • 顺俱䏨⪓⟧ • 㢪鿈剪⹢㉏겗 • 䳕倝炽넒 • QFSGPSNBODF

Slide 16

Slide 16 text

斊麌♳涸朜屣 • 㣐ꆀ崨ꆀ弈Ⰶ • ♳涰㣔涸麋遤 • 顺俱䏨⪓⟧ • 㢪鿈剪⹢㉏겗 • 䳕倝炽넒 • "WBJMBCJMJUZ • 3FMJBCJMJUZ • 3FTQPOTF5JNF • %PXOUJNF • .BJOUFOBODF 4DIFEVMF

Slide 17

Slide 17 text

斊麌♳涸朜屣 • 㣐ꆀ崨ꆀ弈Ⰶ • ♳涰㣔涸麋遤 • 顺俱䏨⪓⟧ • 㢪鿈剪⹢㉏겗 • 䳕倝炽넒 • 鐱⠮倝炽넒꨽宠 • 倝涸禹窠⿮侷

Slide 18

Slide 18 text

.FUSJDT#Z$PEBIBMF Mind The Gap!

Slide 19

Slide 19 text

(BVHFT • 鎹ꏖ殹♴涸⧩ Metrics.newGauge(QueueManager.class, "pending-jobs",   new Gauge() { @Override public Integer value() { return queue.size(); } });

Slide 20

Slide 20 text

$PVOUFS private final Counter pendingJobs = Metrics.newCounter(QueueManager.class, "pending-jobs"); public void addJob(Job job) { pendingJobs.inc(); queue.offer(job); } public Job takeJob() { pendingJobs.dec(); return queue.take(); }

Slide 21

Slide 21 text

.FUFS • 鎹ꏖ㶸》涸鸟桧 private final Meter requests = Metrics.newMeter(RequestHandler.class, "requests", “requests", TimeUnit.SECONDS); public void handleRequest(Request request, Response response) { requests.mark(); // etc }

Slide 22

Slide 22 text

)JTUPHSBNT • 鎹ꏖ顺俱涸窠鎙ⴔ區ծⴔ⡑ minimum maximum mean standard deviation 75 percentile 95 percentile 98 percentile 99 percentile 99.9 percentile

Slide 23

Slide 23 text

)JTUPHSBNT private final Timer responses = Metrics.newTimer(  RequestHandler.class, "responses", TimeUnit.MILLISECONDS, TimeUnit.SECONDS); public String handleRequest(Request request, Response response) { final TimerContext context = responses.time(); try { // etc; return "OK"; } finally { context.stop(); } }

Slide 24

Slide 24 text

)JTUPHSBNT暶殯⸆腋

Slide 25

Slide 25 text

)JTUPHSBNT暶殯⸆腋 Data Points: 1,000 req/sec * 60 * 60 * 24 = 691,200,000 or 5 GB

Slide 26

Slide 26 text

)JTUPHSBNT暶殯⸆腋 • 䱰垺䖃❀ⴔꗻⰊ涸✲⟝⨊俷 ref: http://codahale.com/codeconf-2011-04-09-metrics-metrics-everywhere.pdf

Slide 27

Slide 27 text

剤✫鸏❉须俲䖕䧮⦛銴䙦랃欽ヤ

Slide 28

Slide 28 text

+.9JOUFSGBDF

Slide 29

Slide 29 text

KTPOJOUFSGBDF "jvm" : { "vm" : { "name" : "Java HotSpot(TM) 64-Bit Server VM", "version" : "1.6.0_45-b06" }, "memory" : { "totalInit" : 8.5761344E7, "totalUsed" : 8.4700376E7, "totalMax" : 1.087111168E9, "totalCommitted" : 1.16129792E8, "heapInit" : 6.1447488E7, "heapUsed" : 2.876592E7, "heapMax" : 9.50796288E8, "heapCommitted" : 5.9834368E7, "heap_usage" : 0.030254556483922664, "non_heap_usage" : 0.4103368905874399, "memory_pool_usages" : { "Code Cache" : 0.058741251627604164, "Eden Space" : 0.0375141103764524, "Perm Gen" : 0.6161499953851467, "Survivor Space" : 0.003552734375, "Tenured Gen" : 0.028688572989019088 }

Slide 30

Slide 30 text

KTPOJOUFSGBDF "daemon_thread_count" : 17, "thread_count" : 74, "current_time" : 1415698443730, "uptime" : 20471620, "fd_usage" : 0.0013427734375, "thread-states" : { "timed_waiting" : 0.7837837837837838, "blocked" : 0.013513513513513514, "runnable" : 0.16216216216216217, "waiting" : 0.04054054054054054, "new" : 0.0, "terminated" : 0.0 }, "garbage-collectors" : { "MarkSweepCompact" : { "runs" : 5, "time" : 946 }, "Copy" : { "runs" : 268889, "time" : 498572 } } "application.info" : { "version" : { "type" : "gauge", "value" : "1.2.0-20140319-SNAPSHOT" } },

Slide 31

Slide 31 text

(BVHF "net.sf.ehcache.Cache.http-service-cache" : { "accuracy" : { "type" : "gauge", "value" : "None" }, "eviction-count" : { "type" : "gauge", "value" : 2521997 }, "in-memory-hits" : { "type" : "gauge", "value" : 1213691 }, "in-memory-misses" : { "type" : "gauge", "value" : 3531312 }, "in-memory-objects" : { "type" : "gauge", "value" : 3587 },

Slide 32

Slide 32 text

.FUFS "org.eclipse.jetty.servlet.ServletContextHandler" : { "2xx-responses" : { "type" : "meter", "event_type" : "responses", "unit" : "seconds", "count" : 733665, "mean" : 2.014399884583874, "m1" : 0.18145603566649257, "m5" : 0.6958598626391137, "m15" : 0.9097181383462438 }, "4xx-responses" : { "type" : "meter", "event_type" : "responses", "unit" : "seconds", "count" : 2006, "mean" : 0.00550780829135732, "m1" : 3.0263603123472702E-58, "m5" : 4.125713901938956E-14, "m15" : 4.134678963142411E-7 },

Slide 33

Slide 33 text

)JTUPHSBN "com.fliptop.spendscore.backend.craweler" : { "requests" : { "type" : "timer", "duration" : { "unit" : "milliseconds", "min" : 0.076781, "max" : 43874.994542, "mean" : 406.840418123641, "std_dev" : 583.3339808219433, "median" : 554.004911, "p75" : 603.6283882500001, "p95" : 708.6854972499999, "p98" : 849.4521964999997, "p99" : 1024.8766623300019, "p999" : 2524.4020420530032 }, "rate" : { "unit" : "seconds", "count" : 150290, "mean" : 0.41264477142585376, "m1" : 0.026800109705232115, "m5" : 0.09772776121229207, "m15" : 0.13448567778710924 } }

Slide 34

Slide 34 text

殥㕮

Slide 35

Slide 35 text

䬘陪㜡 ref:http://www.nagios.com/products/nagiosxi/screenshots

Slide 36

Slide 36 text

⚮-PBE#BMBODFS ref:http://www.nagios.com/products/nagiosxi/screenshots public class MyMongoDBHealthCheck{ public MyMongoDBHealthCheck(MongoDB db) { this.db = db; super(“mongodb”); } @Override public Result check() { try { // run query return Result.healthy(); } catch (Exception e) { return Result.unhealthy("Cannot connect to Mongodb”); } } } HealthChecks.register(new MarketoHealthCheck(marketo));

Slide 37

Slide 37 text

⚮-PBE#BMBODFS ref:http://www.nagios.com/products/nagiosxi/screenshots HealthChecks.register(new MarketoHealthCheck(marketo)); // in pre-processing filter Map health = HealthChecks.runHealthChecks() // proceed only if all health checks return OK.  // return exception if any check fails.