Manage cloud server with open source tools

2e147e13e3cf585a86626062d710a205?s=47 yunglin
November 14, 2014

Manage cloud server with open source tools

2e147e13e3cf585a86626062d710a205?s=128

yunglin

November 14, 2014
Tweet

Transcript

  1. #JCConf .BOBHF4FSWFST0O5IF $MPVEXJUI0QFO4PVSDF5PPMT 何永琳 @ Fliptop Director of Engineering ⼯工頭

  2. 陾玑 • 捀➊랂䧮⦛꨽銳湋䱽ծ庠ꆀ • 剣➊랂䭷垥꨽銳湋䱽ծ庠ꆀ • 剣➊랂䊨Ⱘ〳⟄䍲⸔䧮⦛ • NFUSJDTCZDPEBIBMF •

    33%5PPMT
  3. 锞铇㻜涸㔐瘸鸏⦐㉏겗

  4. ⡹♧㣔漗䎙⦐㼭儘 ref: http://xkcd.com/386/

  5. ⡹♧㣔漗䎙⦐㼭儘 ref: http://xkcd.com/386/

  6. ⡹♧㣔漗䎙⦐㼭儘 ref: https://jawbone.com/up

  7. ⡹♧㣔漗䎙⦐㼭儘 ref: https://jawbone.com/up

  8. 植㻜䖃䖃嫲⡹䟝⫹涸婫ꃏ

  9. 铇㻜涸꬗㼩荈䧮僽佖㊥涸Ꟛ㨥

  10. 㥶卓䧮⦛㼩荈魨涸钢濼鿪剤铐䊴 齡䧮⦛㻨涸鮿넓ヤ

  11. 4PGUXBSF-JGFDZDMF

  12. 斊麌♳涸朜屣 • 㣐ꆀ崨ꆀ弈Ⰶ • ♳涰㣔涸麋遤 • 顺俱䏨⪓⟧ • 㢪鿈剪⹢ 40"

    .JDSP4FSWJDF ㉏겗 • 䳕倝炽넒
  13. 斊麌♳涸朜屣 • 㣐ꆀ崨ꆀ弈Ⰶ • ♳涰㣔涸麋遤 • 顺俱䏨⪓⟧ • 㢪鿈剪⹢㉏겗 •

    䳕倝炽넒 • 5ISPVHIQVU • 3FTQPOTFUJNF • XPSTUDBTF35
  14. 斊麌♳涸朜屣 • 㣐ꆀ崨ꆀ弈Ⰶ • ♳涰㣔涸麋遤 • 顺俱䏨⪓⟧ • 㢪鿈剪⹢㉏겗 •

    䳕倝炽넒 • .FNPSZ-FBL • $BDIF6TBHF )JU .JTT  • +7.)FBQTJ[F
  15. 斊麌♳涸朜屣 • 㣐ꆀ崨ꆀ弈Ⰶ • ♳涰㣔涸麋遤 • 顺俱䏨⪓⟧ • 㢪鿈剪⹢㉏겗 •

    䳕倝炽넒 • QFSGPSNBODF
  16. 斊麌♳涸朜屣 • 㣐ꆀ崨ꆀ弈Ⰶ • ♳涰㣔涸麋遤 • 顺俱䏨⪓⟧ • 㢪鿈剪⹢㉏겗 •

    䳕倝炽넒 • "WBJMBCJMJUZ • 3FMJBCJMJUZ • 3FTQPOTF5JNF • %PXOUJNF • .BJOUFOBODF 4DIFEVMF
  17. 斊麌♳涸朜屣 • 㣐ꆀ崨ꆀ弈Ⰶ • ♳涰㣔涸麋遤 • 顺俱䏨⪓⟧ • 㢪鿈剪⹢㉏겗 •

    䳕倝炽넒 • 鐱⠮倝炽넒꨽宠 • 倝涸禹窠⿮侷
  18. .FUSJDT#Z$PEBIBMF Mind The Gap!

  19. (BVHFT • 鎹ꏖ殹♴涸⧩ Metrics.newGauge(QueueManager.class, "pending-jobs", 
 new Gauge<Integer>() { @Override

    public Integer value() { return queue.size(); } });
  20. $PVOUFS private final Counter pendingJobs = Metrics.newCounter(QueueManager.class, "pending-jobs"); public void

    addJob(Job job) { pendingJobs.inc(); queue.offer(job); } public Job takeJob() { pendingJobs.dec(); return queue.take(); }
  21. .FUFS • 鎹ꏖ㶸》涸鸟桧 private final Meter requests = Metrics.newMeter(RequestHandler.class, "requests",

    “requests", TimeUnit.SECONDS); public void handleRequest(Request request, Response response) { requests.mark(); // etc }
  22. )JTUPHSBNT • 鎹ꏖ顺俱涸窠鎙ⴔ區ծⴔ⡑ minimum maximum mean standard deviation 75 percentile

    95 percentile 98 percentile 99 percentile 99.9 percentile
  23. )JTUPHSBNT private final Timer responses = Metrics.newTimer(
 RequestHandler.class, "responses", TimeUnit.MILLISECONDS,

    TimeUnit.SECONDS); public String handleRequest(Request request, Response response) { final TimerContext context = responses.time(); try { // etc; return "OK"; } finally { context.stop(); } }
  24. )JTUPHSBNT暶殯⸆腋

  25. )JTUPHSBNT暶殯⸆腋 Data Points: 1,000 req/sec * 60 * 60 *

    24 = 691,200,000 or 5 GB
  26. )JTUPHSBNT暶殯⸆腋 • 䱰垺䖃❀ⴔꗻⰊ涸✲⟝⨊俷 ref: http://codahale.com/codeconf-2011-04-09-metrics-metrics-everywhere.pdf

  27. 剤✫鸏❉须俲䖕䧮⦛銴䙦랃欽ヤ

  28. +.9JOUFSGBDF

  29. KTPOJOUFSGBDF "jvm" : { "vm" : { "name" : "Java

    HotSpot(TM) 64-Bit Server VM", "version" : "1.6.0_45-b06" }, "memory" : { "totalInit" : 8.5761344E7, "totalUsed" : 8.4700376E7, "totalMax" : 1.087111168E9, "totalCommitted" : 1.16129792E8, "heapInit" : 6.1447488E7, "heapUsed" : 2.876592E7, "heapMax" : 9.50796288E8, "heapCommitted" : 5.9834368E7, "heap_usage" : 0.030254556483922664, "non_heap_usage" : 0.4103368905874399, "memory_pool_usages" : { "Code Cache" : 0.058741251627604164, "Eden Space" : 0.0375141103764524, "Perm Gen" : 0.6161499953851467, "Survivor Space" : 0.003552734375, "Tenured Gen" : 0.028688572989019088 }
  30. KTPOJOUFSGBDF "daemon_thread_count" : 17, "thread_count" : 74, "current_time" : 1415698443730,

    "uptime" : 20471620, "fd_usage" : 0.0013427734375, "thread-states" : { "timed_waiting" : 0.7837837837837838, "blocked" : 0.013513513513513514, "runnable" : 0.16216216216216217, "waiting" : 0.04054054054054054, "new" : 0.0, "terminated" : 0.0 }, "garbage-collectors" : { "MarkSweepCompact" : { "runs" : 5, "time" : 946 }, "Copy" : { "runs" : 268889, "time" : 498572 } } "application.info" : { "version" : { "type" : "gauge", "value" : "1.2.0-20140319-SNAPSHOT" } },
  31. (BVHF "net.sf.ehcache.Cache.http-service-cache" : { "accuracy" : { "type" : "gauge",

    "value" : "None" }, "eviction-count" : { "type" : "gauge", "value" : 2521997 }, "in-memory-hits" : { "type" : "gauge", "value" : 1213691 }, "in-memory-misses" : { "type" : "gauge", "value" : 3531312 }, "in-memory-objects" : { "type" : "gauge", "value" : 3587 },
  32. .FUFS "org.eclipse.jetty.servlet.ServletContextHandler" : { "2xx-responses" : { "type" : "meter",

    "event_type" : "responses", "unit" : "seconds", "count" : 733665, "mean" : 2.014399884583874, "m1" : 0.18145603566649257, "m5" : 0.6958598626391137, "m15" : 0.9097181383462438 }, "4xx-responses" : { "type" : "meter", "event_type" : "responses", "unit" : "seconds", "count" : 2006, "mean" : 0.00550780829135732, "m1" : 3.0263603123472702E-58, "m5" : 4.125713901938956E-14, "m15" : 4.134678963142411E-7 },
  33. )JTUPHSBN "com.fliptop.spendscore.backend.craweler" : { "requests" : { "type" : "timer",

    "duration" : { "unit" : "milliseconds", "min" : 0.076781, "max" : 43874.994542, "mean" : 406.840418123641, "std_dev" : 583.3339808219433, "median" : 554.004911, "p75" : 603.6283882500001, "p95" : 708.6854972499999, "p98" : 849.4521964999997, "p99" : 1024.8766623300019, "p999" : 2524.4020420530032 }, "rate" : { "unit" : "seconds", "count" : 150290, "mean" : 0.41264477142585376, "m1" : 0.026800109705232115, "m5" : 0.09772776121229207, "m15" : 0.13448567778710924 } }
  34. 殥㕮

  35. 䬘陪㜡 ref:http://www.nagios.com/products/nagiosxi/screenshots

  36. ⚮-PBE#BMBODFS ref:http://www.nagios.com/products/nagiosxi/screenshots public class MyMongoDBHealthCheck{ public MyMongoDBHealthCheck(MongoDB db) { this.db

    = db; super(“mongodb”); } @Override public Result check() { try { // run query return Result.healthy(); } catch (Exception e) { return Result.unhealthy("Cannot connect to Mongodb”); } } } HealthChecks.register(new MarketoHealthCheck(marketo));
  37. ⚮-PBE#BMBODFS ref:http://www.nagios.com/products/nagiosxi/screenshots HealthChecks.register(new MarketoHealthCheck(marketo)); // in pre-processing filter Map<String, Result>

    health = HealthChecks.runHealthChecks() // proceed only if all health checks return OK.
 // return exception if any check fails.
  38. 2"