nothing Old Master fail Run Redis fail over nothing master.foo-redis.service.dc1.consul 54 master.foo-redis.service.dc1.consul 54 192.0.2.11 check ok check ng failover nothing nothing
def is_alive(self): """Determine if connection to server is still alive. Returns bool - True = alive, False = error or cannot connect. """ res = True try: if self.db_conn is None: res = False else: # ping and is_connected only work partially, try exec_query # to make sure connection is really alive retval = self.db_conn.is_connected() if retval: self.exec_query("SHOW DATABASES") else: res = False except: res = False return res Health Check時に MySQLに接続し SHOW DATABASESを実行 111
def _reconnect_master(self, pingtime=3): """Tries to reconnect to the master This method tries to reconnect to the mast after 3 attemps, returns False. """ if self.master and self.master.is_alive(): return True is_connected = False i = 0 while i < 3: try: self.master.connect() is_connected = True break except: pass time.sleep(pingtime) i += 1 return is_connected 失敗した場合は ping(pingtime) x3 試す x3はハードコードされている 112
Redis http://redis.io/ Redis Sentinel http://redis.io/topics/sentinel MySQL https://www-jp.mysql.com/ MySQL Utilities https://dev.mysql.com/downloads/utilities/ Consul by HashiCorp https://www.consul.io/ consul-template https://github.com/hashicorp/consul-template Dnsmasq - network services for small networks. - Simon Kelley http://www.thekelleys.org.uk/dnsmasq/doc.html Consulと自作OSSを活用した100台規模のWebサービス運用 by FUJIWARA Shunichiro https://speakerdeck.com/fujiwara3/consultozi-zuo-osswohuo-yong-sita100tai-gui-mo- falsewebsabisuyun-yong 参考URL 155
「迷ったら健全な方」 Being healthy dev and ops in Cookpad by Issei Naruta / Speaker Deck https://speakerdeck.com/mirakui/being-healthy-dev-and-ops-in-cookpad 「仕事道具に対しても責任を持ちたい」 監視ツールの話 by @kazeburo / slideshare http://www.slideshare.net/kazeburo/ss-13361002 参考URL 157 ImmutableServer / Martin Fowler http://martinfowler.com/bliki/ImmutableServer.html BlueGreenDeployment / Martin Fowler http://martinfowler.com/bliki/BlueGreenDeployment.html Trash Your Servers and Burn Your Code: Immutable Infrastructure and Disposable Components / Chad Fowler http://chadfowler.com/blog/2013/06/23/immutable-deployments/ インフラ系技術の流れ / Gosuke Miyashita http://mizzy.org/blog/2013/10/29/1/ Rebuild.fm 25: Immutable Infrastructure (Naoya Ito, Gosuke Miyashita). http://rebuild.fm/25/