20 Jan 2017 / This is Mercari, This is a SRE - Mercari DAY

by Kenichi Sasaki

Slide 1

Slide 1 text

This is Mercari, This is an SRE Mercari DAY 2017 @AcademyHills Kenichi Sasaki

Slide 2

Slide 2 text

Introduce myself • Kenichi Sasaki (@siroken3) • Mercari, Inc (JOIN at 2014/7)  Site Reliability Engineering (SRE) Team • αʔόϓϩϏδϣχϯάɾσϓϩΠͷ੔උ • ։ൃ؀ڥͳͲͷ੔උ

Slide 3

Slide 3 text

Agenda • Site Reliability Engineering (SRE) ͱ͸ • MercariͱMercariͷSRE • Mercari SREͷࣄྫ • ·ͱΊ

Slide 4

Slide 4 text

SRE ͱ͸

Slide 5

Slide 5 text

4JUF3FMJBCJMJUZ&OHJOFFSJOH w (PPHMF͕ఏএ w (PPHMFͷ༷ʑͳϓϩμΫτɾ αʔϏεΛԣஅͯ͠ɺαΠτ ͷ৴པੑΛ޲্ͤ͞Δ 4PGUXBSF&OHJOFFS5FBN

Slide 6

Slide 6 text

SRE ͷ࠾༻(US) Facebook, Twitter, Apple, eBay, Microsoft.. 3,662

Slide 7

Slide 7 text

SRE in JP • 2015/11 Mercari Tech Blog ʹͯ঺հͯ͠Ҏ߱஫ ໨͕ू·Δ • “ΠϯϑϥνʔϜվΊ Site Reliability Engineering (SRE) νʔϜʹͳΓ·ͨ͠”  http://tech.mercari.com/entry/2015/11/18/153421 • 2016/06 SRE Tech Talk#1 • https://connpass.com/event/34825/

Slide 8

Slide 8 text

SRE Tech Talk#2! • https://eventdots.jp/event/609456 • 2017/01/23 12:00 క੾

Slide 9

Slide 9 text

SREͷ࠾༻(JP) 40!

Slide 10

Slide 10 text

Mercari & Mercari SRE

Slide 11

Slide 11 text

Infrastructure UK ४උத JP US ͘͞ΒΠϯλʔωοτ  ੴङDC AWS GCP

Slide 12

Slide 12 text

Architecture US nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. Client Multimedia Corporate data center Traditional server Mobile Client IAM Add-on Example: IAM Add-on ence ) Assignment/ Task Requester Workers DNS-RR App App App App App App MySQL MySQL memcached memcached nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia C d Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific DNS-RR App App App App App App MySQL MySQL memcached memcached EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 JP Ϋϥ΢υͰ΋ ಉ͡ߏ੒ util util cloud cloud util util EC2 EC2

Slide 13

Slide 13 text

Mercari SRE Team

Slide 14

Slide 14 text

Mercari SRE • 2015/11 ͔Β SRE ʹมߋ  ͦΕ·Ͱ͸ΠϯϑϥνʔϜ • ϝϯόʔ͸5ਓ • શһ͕౦ژΦϑΟεۈ຿

Slide 15

Slide 15 text

SREʹมߋͨ͠ཧ༝ • ʮΠϯϑϥʯ͔ΒΠϝʔδ͢Δۀ຿ͷ࿮Λ௒͑Δ • ϝϧΧϦΛ5೥ɾ10೥ͱ௕͘࢖͍ͬͯͨͩͨ͘Ίͷʮ৴པੑʯ • εϚϗͰ͸24࣌ؒ޷͖ͳ࣌ʹɺ޷͖ͳ৔ॴ͔ΒΞΫηε͕དྷ Δɻ͍ͭͰ΋շదʹΞΫηε͕Ͱ͖ΔʮՄ༻ੑʯͱ  ʮύϑΥʔϚϯεʯ • ιϑτ΢ΣΞΤϯδχΞϦϯάͰ͜ΕΒΛղܾ͍ͯ͘͠νʔϜ • (ւ֎/USͰ΋௨͡Δ໊শ)

Slide 16

Slide 16 text

Mercari SREͷۀ຿ • APIαʔόɺϛυϧ΢ΣΞͷՄ༻ੑɺύϑΥʔϚϯεͷ  ҡ࣋ɾ޲্ • OnCall (ো֐ରԠͷ౰൪) • ϩάऩूɾ෼ੳج൫ͷߏஙɺӡ༻ • αʔόϓϩϏδϣχϯάɾσϓϩΠͷ੔උ • ηΩϡϦςΟͷ୲อ

Slide 17

Slide 17 text

Mercari SREࣄྫ • Gaurun - push notiﬁcation server • ngx_luaʹΑΔBalancer֦ு • PHP daemon class preload • URL Shorten Service • ChatOpsˍZero DownTime Deploy • etc…

Slide 18

Slide 18 text

ChatOps Deploy

Slide 19

Slide 19 text

ChatOps Deploy • ϝϧΧϦ • SCMͱͯ͠GitHubΛ࠾༻ • IssueTrackerͰͷ࢓༷ݕ౼ • GitHub PullRequestͰͷϐΞϨϏϡʔ

Slide 20

Slide 20 text

Google Calendar + ChatOps

Slide 21

Slide 21 text

Inside ChatOps @jp-go-bold: yes check PullRequest check Issue Tracker exec Ansible merge PR git clone git clone composer install deploy application preprocessing ϐΞϨϏϡʔࡁ Ϛωʔδϟঝೝࡁ

Slide 22

Slide 22 text

·ͱΊ

Slide 23

Slide 23 text

·ͱΊ • ϝϧΧϦ͸C2CϚʔέοτϓϨΠεɺ৴པੑॏཁ • SRE͸৴པੑΛγεςϜ໘͔Βࢧ͑Δ • ۀ຿͸Πϯϑϥʹཹ·Βͳ͍ɻ৴པੑΛ௥ٻ