Inside of Blog; Light and shadow of the service matured for 15 years and challenge chaos and legacy

Takahiro Omori
LINE Development Team B
https://linedevday.linecorp.com/jp/2019/sessions/C1-2

1. 2019 DevDay Inside of Blog; Light and Shadow of the Service Matured for 15 Years and Challenge Chaos and Legacy > Takahiro Omori > LINE Development Team B
2. >Takahiro Ohmori • Lead of the Blog development team • Joined Blog development in 2015 • Server Side Engineer • Writing code in Perl Speaker
3. livedoor Blog / LINE BLOG LINE BLOG livedoor Blog > Launch 2003 > 16 Years Ago > Launch 2014 > 5 Years Ago
4. livedoor Blog
5. > LAMP • Linux + Apache + MySQL + Perl • Web App Framework: Sledge > One of the largest blog service in Japan • Many users, many contents • High performance, high availability livedoor Blog
6. System Overview: Blog Core load balancer CDN main DB cache cache pool storage image store www image worker batch etc portal www app CMS www app user blog www app cluster DB
7. System Overview: ALL adv
 admin feed
 service user information blog auth user profile traffic report feed
 api feed
 fetcher news
 sender talk
 notifier image
 resize image
 fetcher auth
 service blog
 admin service
 admin search anti
 spam mobile
 page image
 storage file
 storage email
 sender traffic
 analyzer access
 trade page
 render snippet
 generator blog
 reader internal
 feed api Blog Core
8. 410,000+ 750+ 3,800+ 43,500+ 550+ 70+ livedoor Blog in Numbers
9. 410,000+ 750+ 3,800+ 43,500+ 550+ 70+ livedoor Blog in Numbers Developers
10. 410,000+ 750+ 3,800+ 43,500+ 550+ 70+ livedoor Blog in Numbers Developers Servers
11. 410,000+ 750+ 3,800+ 43,500+ 550+ 70+ livedoor Blog in Numbers Developers Servers DB Tables
12. 410,000+ 750+ 3,800+ 43,500+ 550+ 70+ livedoor Blog in Numbers Developers Servers DB Tables Files
13. 410,000+ 750+ 3,800+ 43,500+ 550+ 70+ livedoor Blog in Numbers Developers Servers DB Tables Files Program Files
14. 410,000+ 750+ 3,800+ 43,500+ 550+ 70+ livedoor Blog in Numbers Developers Servers DB Tables Files Program Files Lines of Code
15. Ongoing Projects Server Migration 2019 2018 2017
16. Server Migration Ongoing Projects 2019 2018 2017 Always-On HTTPS (SSL)
17. Case Studies
18. CASE #01
 Document Not Found
19. Document Not Found
20. CASE #02 Development Server Not Found
21. Development Environment Difficulty Level Dev server
 exists Functions properly Matches with
 production server Does not use production data Used to have dev server Easy START!! No No No No No Yes Yes Yes Yes Yes NIGHTMARE Normal HARD
22. Development Environment Difficulty Level Dev server
 exists Functions properly Matches with
 production server Does not use production data Used to have dev server Easy START!! No No No No No Yes Yes Yes Yes Yes NIGHTMARE Normal HARD
23. Development Environment Difficulty Level Dev server
 exists Functions properly Matches with
 production server Does not use production data Used to have dev server Easy START!! No No No No No Yes Yes Yes Yes Yes NIGHTMARE Normal Hard
24. Development Environment Difficulty Level Dev server
 exists Functions properly Matches with
 production server Does not use production data Used to have dev server Easy START!! No No No No No Yes Yes Yes Yes Yes Nightmare Normal Hard
25. CASE #03 Low Code Coverage
26. CASE #03 Low Code Coverage Test Not Found
27. CASE #04 Too Many DNS Records
28. Too Many DNS Records Over 300 records
29. >Total 300 / Deleted 230 Garbage DNS Records 77% 23% Used Not used
30. CASE #05
 Too Many Functions
31. > Old functions • Feature phone page • Trackback • Old CMS • FTP • moblog • etc... Remove Old Functions
32. CASE #06
 Perl
33. > Very old version > No change since service launch LEGACY: Perl 5.8 2002 ver 5.8 <====== 2007 ver 5.10 2010 ver 5.12 2011 ver 5.14 2012 ver 5.16 2013 ver 5.18 ... 2019 ver 5.30 Perl Release
34. • Apache module • mod_perl ver 1.3 • + Apache ver 1.3 LEGACY: mod_perl 1.3 2009 mod_perl ver 1.31 2010 Apache ver 1.3.42 Latest Update
35. > 2 versions of Perl > Dual management of dependent modules > Maintain code compatibility > Since 2013 Perl 5.8 + Perl 5.16
36. > 2 versions of Perl > Dual management of dependent modules > Maintain code compatibility Perl 5.8 + Perl 5.16
37. CASE #07
 MySQL
38. CASE #07
 MySQL 4.0
39. LEGACY: MySQL 4.0 2003 ver 4.0 <====== 2004 ver 4.1 2005 ver 5.0 2008 ver 5.1 2010 ver 5.5 2013 ver 5.6 2015 ver 5.7 2018 ver 8.0 MySQL Release > Very old version > No change since service launch
40. > Online DDL v5.6 (Not Available) > Fast Index Creation v5.1 (Not Available) > Trigger v5.0 (Not Available) • Percona Toolkit: pt-online-schema-change v5.0 (Not Supported) ALTER TABLE
41. > Online DDL v5.6 (not available) > Fast Index Creation v5.1 (not available) > Trigger v5.0 (not available) • Percona Toolkit: pt-online-schema-change v5.0 (not supported) ALTER TABLE (Virtually impossible)
42. Workaround TABLE A Original App
43. Workaround 1. Copy blank table TABLE A Original TABLE B New App
44. Workaround 2. Alter table 1. Copy blank table TABLE A Original TABLE B New App
45. Workaround 3. Double write App 2. Alter table 1. Copy blank table TABLE A Original TABLE B New
46. Workaround Batch 4. Row copy 3. Double write App 2. Alter table 1. Copy blank table TABLE A Original TABLE B New
47. Workaround App TABLE A Original TABLE B New
48. Workaround 1. Copy blank table TABLE A Original TABLE B New > CREATE TABLE new LIKE orig; v4.1 (not available)
49. Side Effects
50. Data Bloat > innodb_file_per_table v4.1 (not available) > InnoDB Table Compression v5.1 (not available) > Transparent Page Compression v5.7 (not available)
51. MySQL 4.0: Server Migration MySQL 4.0
52. MySQL 4.0: Server Migration DB server MySQL 5.7 MySQL 4.0
53. MySQL 4.0: Server Migration MySQL 4.0 MySQL 5.7 DB server
54. MySQL 4.0: Server Migration DB server MySQL 4.0 MySQL 4.0
55. MySQL 4.0: Server Migration DB server MySQL 4.0 MySQL 4.0
56. MySQL 4.0: Server Migration App server DB server MySQL 4.0 MySQL 4.0 MySQL 4.0
57. MySQL 4.0: Server Migration App server DB server MySQL 4.0 is not database... MySQL 4.0 MySQL 4.0 MySQL 4.0
58. Why Can't Migrate? utf-8 utf-8 euc-jp euc-jp TABLE Mixed character set
59. Why Can't Migrate? utf-8 utf-8 euc-jp euc-jp TABLE 4.0 4.1 5.0 5.1 Mixed character set
60. > libmysqlclient 5.7.5 • --skip-secure-auth REMOVED MySQL 4.0: DEADLINE MySQL 4.0
 server MySQL client v5.6 MySQL client v5.7
61. > libmysqlclient 5.7.5 • --skip-secure-auth (removed) MySQL 4.0: DEADLINE MySQL 4.0
 server MySQL client v5.6 MySQL client v5.7
62. livedoor Blog Roadmap
63. Roadmap: livedoor Blog 2019 1H 2019 2H 2020 1H 2020 2H 2021 1H Server Migration HTTPS Perl Update UTF-8 MySQL Update
64. LINE BLOG
65. > Blog Service of LINE Brand • Authentication with LINE account • Launched 2014 LINE BLOG
66. History of LINE BLOG
67. Service History 2014.11 LINE BLOG Launch
68. Service History 2016.11 2014.11 LINE BLOG Launch LINE BLOG Renewal Launch
69. Service History 2014.11 livedoor Blog ASP LINE BLOG Launch
70. Service History 2014.11 2016.04 livedoor Blog ASP LINE BLOG Launch Begin renewal
 development

71. Service History 2016.11 2014.11 2016.04 livedoor Blog ASP LINE BLOG Launch LINE BLOG Renewal Launch Begin renewal
 development

72. Forked from livedoor Blog 2016.04
73. > MySQL 4.0 > Perl 5.8 > mod_perl > Mixed character set > Apache > Low code coverage > Legacy subsystems Forked from livedoor Blog
74. Forked from livedoor Blog > MySQL 4.0 > Perl 5.8 > mod_perl > Mixed character set > Apache > Low code coverage > Legacy subsystems
75. > MySQL 4.0 > Perl 5.8 > mod_perl > Mixed character set > Apache > Low code coverage > Legacy subsystems > HTTP Breaking Away from Chaos and Legacy > MySQL 5.6 > Perl 5.16 > PSGI > utf-8 > nginx > High code coverage > LINE platform > HTTPS
76. Server
 Migration HTTPS 2017 LINE BLOG
 Renewal LINE BLOG's Contribution > Verification • Server migration • Database migration • Middleware updates > Test cases • Data charset conversion • Always-On HTTPS (SSL) > etc...
77. Next 15 Years
78. Start Small
79. Now
80. Join Us in This Battle!
81. Thank You