Inside of Blog; Light and shadow of the service matured for 15 years and challenge chaos and legacy

by LINE DevDay 2019

Slide 1

Slide 1 text

2019 DevDay Inside of Blog; Light and Shadow of the Service Matured for 15 Years and Challenge Chaos and Legacy > Takahiro Omori > LINE Development Team B

Slide 2

Slide 2 text

>Takahiro Ohmori • Lead of the Blog development team • Joined Blog development in 2015 • Server Side Engineer • Writing code in Perl Speaker

Slide 3

Slide 3 text

livedoor Blog / LINE BLOG LINE BLOG livedoor Blog > Launch 2003 > 16 Years Ago > Launch 2014 > 5 Years Ago

Slide 4

Slide 4 text

livedoor Blog

Slide 5

Slide 5 text

> LAMP • Linux + Apache + MySQL + Perl • Web App Framework: Sledge > One of the largest blog service in Japan • Many users, many contents • High performance, high availability livedoor Blog

Slide 6

Slide 6 text

System Overview: Blog Core load balancer CDN main DB cache cache pool storage image store www image worker batch etc portal www app CMS www app user blog www app cluster DB

Slide 7

Slide 7 text

System Overview: ALL adv  admin feed  service user information blog auth user profile traffic report feed  api feed  fetcher news  sender talk  notifier image  resize image  fetcher auth  service blog  admin service  admin search anti  spam mobile  page image  storage file  storage email  sender traffic  analyzer access  trade page  render snippet  generator blog  reader internal  feed api Blog Core

Slide 8

Slide 8 text

410,000+ 750+ 3,800+ 43,500+ 550+ 70+ livedoor Blog in Numbers

Slide 9

Slide 9 text

410,000+ 750+ 3,800+ 43,500+ 550+ 70+ livedoor Blog in Numbers Developers

Slide 10

Slide 10 text

410,000+ 750+ 3,800+ 43,500+ 550+ 70+ livedoor Blog in Numbers Developers Servers

Slide 11

Slide 11 text

410,000+ 750+ 3,800+ 43,500+ 550+ 70+ livedoor Blog in Numbers Developers Servers DB Tables

Slide 12

Slide 12 text

410,000+ 750+ 3,800+ 43,500+ 550+ 70+ livedoor Blog in Numbers Developers Servers DB Tables Files

Slide 13

Slide 13 text

410,000+ 750+ 3,800+ 43,500+ 550+ 70+ livedoor Blog in Numbers Developers Servers DB Tables Files Program Files

Slide 14

Slide 14 text

410,000+ 750+ 3,800+ 43,500+ 550+ 70+ livedoor Blog in Numbers Developers Servers DB Tables Files Program Files Lines of Code

Slide 15

Slide 15 text

Ongoing Projects Server Migration 2019 2018 2017

Slide 16

Slide 16 text

Server Migration Ongoing Projects 2019 2018 2017 Always-On HTTPS (SSL)

Slide 17

Slide 17 text

Case Studies

Slide 18

Slide 18 text

CASE #01  Document Not Found

Slide 19

Slide 19 text

Document Not Found

Slide 20

Slide 20 text

CASE #02 Development Server Not Found

Slide 21

Slide 21 text

Development Environment Difficulty Level Dev server  exists Functions properly Matches with  production server Does not use production data Used to have dev server Easy START!! No No No No No Yes Yes Yes Yes Yes NIGHTMARE Normal HARD

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Slide 24

Slide 24 text

Slide 25

Slide 25 text

CASE #03 Low Code Coverage

Slide 26

Slide 26 text

CASE #03 Low Code Coverage Test Not Found

Slide 27

Slide 27 text

CASE #04 Too Many DNS Records

Slide 28

Slide 28 text

Too Many DNS Records Over 300 records

Slide 29

Slide 29 text

>Total 300 / Deleted 230 Garbage DNS Records 77% 23% Used Not used

Slide 30

Slide 30 text

CASE #05  Too Many Functions

Slide 31

Slide 31 text

> Old functions • Feature phone page • Trackback • Old CMS • FTP • moblog • etc... Remove Old Functions

Slide 32

Slide 32 text

CASE #06  Perl

Slide 33

Slide 33 text

> Very old version > No change since service launch LEGACY: Perl 5.8 2002 ver 5.8 <====== 2007 ver 5.10 2010 ver 5.12 2011 ver 5.14 2012 ver 5.16 2013 ver 5.18 ... 2019 ver 5.30 Perl Release

Slide 34

Slide 34 text

• Apache module • mod_perl ver 1.3 • + Apache ver 1.3 LEGACY: mod_perl 1.3 2009 mod_perl ver 1.31 2010 Apache ver 1.3.42 Latest Update

Slide 35

Slide 35 text

> 2 versions of Perl > Dual management of dependent modules > Maintain code compatibility > Since 2013 Perl 5.8 + Perl 5.16

Slide 36

Slide 36 text

> 2 versions of Perl > Dual management of dependent modules > Maintain code compatibility Perl 5.8 + Perl 5.16

Slide 37

Slide 37 text

CASE #07  MySQL

Slide 38

Slide 38 text

CASE #07  MySQL 4.0

Slide 39

Slide 39 text

LEGACY: MySQL 4.0 2003 ver 4.0 <====== 2004 ver 4.1 2005 ver 5.0 2008 ver 5.1 2010 ver 5.5 2013 ver 5.6 2015 ver 5.7 2018 ver 8.0 MySQL Release > Very old version > No change since service launch

Slide 40

Slide 40 text

> Online DDL v5.6 (Not Available) > Fast Index Creation v5.1 (Not Available) > Trigger v5.0 (Not Available) • Percona Toolkit: pt-online-schema-change v5.0 (Not Supported) ALTER TABLE

Slide 41

Slide 41 text

> Online DDL v5.6 (not available) > Fast Index Creation v5.1 (not available) > Trigger v5.0 (not available) • Percona Toolkit: pt-online-schema-change v5.0 (not supported) ALTER TABLE (Virtually impossible)

Slide 42

Slide 42 text

Workaround TABLE A Original App

Slide 43

Slide 43 text

Workaround 1. Copy blank table TABLE A Original TABLE B New App

Slide 44

Slide 44 text

Workaround 2. Alter table 1. Copy blank table TABLE A Original TABLE B New App

Slide 45

Slide 45 text

Workaround 3. Double write App 2. Alter table 1. Copy blank table TABLE A Original TABLE B New

Slide 46

Slide 46 text

Workaround Batch 4. Row copy 3. Double write App 2. Alter table 1. Copy blank table TABLE A Original TABLE B New

Slide 47

Slide 47 text

Workaround App TABLE A Original TABLE B New

Slide 48

Slide 48 text

Workaround 1. Copy blank table TABLE A Original TABLE B New > CREATE TABLE new LIKE orig; v4.1 (not available)

Slide 49

Slide 49 text

Side Effects

Slide 50

Slide 50 text

Data Bloat > innodb_file_per_table v4.1 (not available) > InnoDB Table Compression v5.1 (not available) > Transparent Page Compression v5.7 (not available)

Slide 51

Slide 51 text

MySQL 4.0: Server Migration MySQL 4.0

Slide 52

Slide 52 text

MySQL 4.0: Server Migration DB server MySQL 5.7 MySQL 4.0

Slide 53

Slide 53 text

MySQL 4.0: Server Migration MySQL 4.0 MySQL 5.7 DB server

Slide 54

Slide 54 text

MySQL 4.0: Server Migration DB server MySQL 4.0 MySQL 4.0

Slide 55

Slide 55 text

MySQL 4.0: Server Migration DB server MySQL 4.0 MySQL 4.0

Slide 56

Slide 56 text

MySQL 4.0: Server Migration App server DB server MySQL 4.0 MySQL 4.0 MySQL 4.0

Slide 57

Slide 57 text

MySQL 4.0: Server Migration App server DB server MySQL 4.0 is not database... MySQL 4.0 MySQL 4.0 MySQL 4.0

Slide 58

Slide 58 text

Why Can't Migrate? utf-8 utf-8 euc-jp euc-jp TABLE Mixed character set

Slide 59

Slide 59 text

Why Can't Migrate? utf-8 utf-8 euc-jp euc-jp TABLE 4.0 4.1 5.0 5.1 Mixed character set

Slide 60

Slide 60 text

> libmysqlclient 5.7.5 • --skip-secure-auth REMOVED MySQL 4.0: DEADLINE MySQL 4.0  server MySQL client v5.6 MySQL client v5.7

Slide 61

Slide 61 text

> libmysqlclient 5.7.5 • --skip-secure-auth (removed) MySQL 4.0: DEADLINE MySQL 4.0  server MySQL client v5.6 MySQL client v5.7

Slide 62

Slide 62 text

livedoor Blog Roadmap

Slide 63

Slide 63 text

Roadmap: livedoor Blog 2019 1H 2019 2H 2020 1H 2020 2H 2021 1H Server Migration HTTPS Perl Update UTF-8 MySQL Update

Slide 64

Slide 64 text

LINE BLOG

Slide 65

Slide 65 text

> Blog Service of LINE Brand • Authentication with LINE account • Launched 2014 LINE BLOG

Slide 66

Slide 66 text

History of LINE BLOG

Slide 67

Slide 67 text

Service History 2014.11 LINE BLOG Launch

Slide 68

Slide 68 text

Service History 2016.11 2014.11 LINE BLOG Launch LINE BLOG Renewal Launch

Slide 69

Slide 69 text

Service History 2014.11 livedoor Blog ASP LINE BLOG Launch

Slide 70

Slide 70 text

Service History 2014.11 2016.04 livedoor Blog ASP LINE BLOG Launch Begin renewal  development 

Slide 71

Slide 71 text

Service History 2016.11 2014.11 2016.04 livedoor Blog ASP LINE BLOG Launch LINE BLOG Renewal Launch Begin renewal  development 

Slide 72

Slide 72 text

Forked from livedoor Blog 2016.04

Slide 73

Slide 73 text

> MySQL 4.0 > Perl 5.8 > mod_perl > Mixed character set > Apache > Low code coverage > Legacy subsystems Forked from livedoor Blog

Slide 74

Slide 74 text

Forked from livedoor Blog > MySQL 4.0 > Perl 5.8 > mod_perl > Mixed character set > Apache > Low code coverage > Legacy subsystems

Slide 75

Slide 75 text

> MySQL 4.0 > Perl 5.8 > mod_perl > Mixed character set > Apache > Low code coverage > Legacy subsystems > HTTP Breaking Away from Chaos and Legacy > MySQL 5.6 > Perl 5.16 > PSGI > utf-8 > nginx > High code coverage > LINE platform > HTTPS

Slide 76

Slide 76 text

Server  Migration HTTPS 2017 LINE BLOG  Renewal LINE BLOG's Contribution > Verification • Server migration • Database migration • Middleware updates > Test cases • Data charset conversion • Always-On HTTPS (SSL) > etc...