Inside of Blog; Light and shadow of the service matured for 15 years and challenge chaos and legacy

Inside of Blog; Light and shadow of the service matured for 15 years and challenge chaos and legacy

Takahiro Omori
LINE Development Team B
https://linedevday.linecorp.com/jp/2019/sessions/C1-2

Be4518b119b8eb017625e0ead20f8fe7?s=128

LINE DevDay 2019

November 20, 2019
Tweet

Transcript

  1. 2019 DevDay Inside of Blog; Light and Shadow of the

    Service Matured for 15 Years and Challenge Chaos and Legacy > Takahiro Omori > LINE Development Team B
  2. >Takahiro Ohmori • Lead of the Blog development team •

    Joined Blog development in 2015 • Server Side Engineer • Writing code in Perl Speaker
  3. livedoor Blog / LINE BLOG LINE BLOG livedoor Blog >

    Launch 2003 > 16 Years Ago > Launch 2014 > 5 Years Ago
  4. livedoor Blog

  5. > LAMP • Linux + Apache + MySQL + Perl

    • Web App Framework: Sledge > One of the largest blog service in Japan • Many users, many contents • High performance, high availability livedoor Blog
  6. System Overview: Blog Core load balancer CDN main DB cache

    cache pool storage image store www image worker batch etc portal www app CMS www app user blog www app cluster DB
  7. System Overview: ALL adv
 admin feed
 service user information blog

    auth user profile traffic report feed
 api feed
 fetcher news
 sender talk
 notifier image
 resize image
 fetcher auth
 service blog
 admin service
 admin search anti
 spam mobile
 page image
 storage file
 storage email
 sender traffic
 analyzer access
 trade page
 render snippet
 generator blog
 reader internal
 feed api Blog Core
  8. 410,000+ 750+ 3,800+ 43,500+ 550+ 70+ livedoor Blog in Numbers

  9. 410,000+ 750+ 3,800+ 43,500+ 550+ 70+ livedoor Blog in Numbers

    Developers
  10. 410,000+ 750+ 3,800+ 43,500+ 550+ 70+ livedoor Blog in Numbers

    Developers Servers
  11. 410,000+ 750+ 3,800+ 43,500+ 550+ 70+ livedoor Blog in Numbers

    Developers Servers DB Tables
  12. 410,000+ 750+ 3,800+ 43,500+ 550+ 70+ livedoor Blog in Numbers

    Developers Servers DB Tables Files
  13. 410,000+ 750+ 3,800+ 43,500+ 550+ 70+ livedoor Blog in Numbers

    Developers Servers DB Tables Files Program Files
  14. 410,000+ 750+ 3,800+ 43,500+ 550+ 70+ livedoor Blog in Numbers

    Developers Servers DB Tables Files Program Files Lines of Code
  15. Ongoing Projects Server Migration 2019 2018 2017

  16. Server Migration Ongoing Projects 2019 2018 2017 Always-On HTTPS (SSL)

  17. Case Studies

  18. CASE #01
 Document Not Found

  19. Document Not Found

  20. CASE #02 Development Server Not Found

  21. Development Environment Difficulty Level Dev server
 exists Functions properly Matches

    with
 production server Does not use production data Used to have dev server Easy START!! No No No No No Yes Yes Yes Yes Yes NIGHTMARE Normal HARD
  22. Development Environment Difficulty Level Dev server
 exists Functions properly Matches

    with
 production server Does not use production data Used to have dev server Easy START!! No No No No No Yes Yes Yes Yes Yes NIGHTMARE Normal HARD
  23. Development Environment Difficulty Level Dev server
 exists Functions properly Matches

    with
 production server Does not use production data Used to have dev server Easy START!! No No No No No Yes Yes Yes Yes Yes NIGHTMARE Normal Hard
  24. Development Environment Difficulty Level Dev server
 exists Functions properly Matches

    with
 production server Does not use production data Used to have dev server Easy START!! No No No No No Yes Yes Yes Yes Yes Nightmare Normal Hard
  25. CASE #03 Low Code Coverage

  26. CASE #03 Low Code Coverage Test Not Found

  27. CASE #04 Too Many DNS Records

  28. Too Many DNS Records Over 300 records

  29. >Total 300 / Deleted 230 Garbage DNS Records 77% 23%

    Used Not used
  30. CASE #05
 Too Many Functions

  31. > Old functions • Feature phone page • Trackback •

    Old CMS • FTP • moblog • etc... Remove Old Functions
  32. CASE #06
 Perl

  33. > Very old version > No change since service launch

    LEGACY: Perl 5.8 2002 ver 5.8 <====== 2007 ver 5.10 2010 ver 5.12 2011 ver 5.14 2012 ver 5.16 2013 ver 5.18 ... 2019 ver 5.30 Perl Release
  34. • Apache module • mod_perl ver 1.3 • + Apache

    ver 1.3 LEGACY: mod_perl 1.3 2009 mod_perl ver 1.31 2010 Apache ver 1.3.42 Latest Update
  35. > 2 versions of Perl > Dual management of dependent

    modules > Maintain code compatibility > Since 2013 Perl 5.8 + Perl 5.16
  36. > 2 versions of Perl > Dual management of dependent

    modules > Maintain code compatibility Perl 5.8 + Perl 5.16
  37. CASE #07
 MySQL

  38. CASE #07
 MySQL 4.0

  39. LEGACY: MySQL 4.0 2003 ver 4.0 <====== 2004 ver 4.1

    2005 ver 5.0 2008 ver 5.1 2010 ver 5.5 2013 ver 5.6 2015 ver 5.7 2018 ver 8.0 MySQL Release > Very old version > No change since service launch
  40. > Online DDL v5.6 (Not Available) > Fast Index Creation

    v5.1 (Not Available) > Trigger v5.0 (Not Available) • Percona Toolkit: pt-online-schema-change v5.0 (Not Supported) ALTER TABLE
  41. > Online DDL v5.6 (not available) > Fast Index Creation

    v5.1 (not available) > Trigger v5.0 (not available) • Percona Toolkit: pt-online-schema-change v5.0 (not supported) ALTER TABLE (Virtually impossible)
  42. Workaround TABLE A Original App

  43. Workaround 1. Copy blank table TABLE A Original TABLE B

    New App
  44. Workaround 2. Alter table 1. Copy blank table TABLE A

    Original TABLE B New App
  45. Workaround 3. Double write App 2. Alter table 1. Copy

    blank table TABLE A Original TABLE B New
  46. Workaround Batch 4. Row copy 3. Double write App 2.

    Alter table 1. Copy blank table TABLE A Original TABLE B New
  47. Workaround App TABLE A Original TABLE B New

  48. Workaround 1. Copy blank table TABLE A Original TABLE B

    New > CREATE TABLE new LIKE orig; v4.1 (not available)
  49. Side Effects

  50. Data Bloat > innodb_file_per_table v4.1 (not available) > InnoDB Table

    Compression v5.1 (not available) > Transparent Page Compression v5.7 (not available)
  51. MySQL 4.0: Server Migration MySQL 4.0

  52. MySQL 4.0: Server Migration DB server MySQL 5.7 MySQL 4.0

  53. MySQL 4.0: Server Migration MySQL 4.0 MySQL 5.7 DB server

  54. MySQL 4.0: Server Migration DB server MySQL 4.0 MySQL 4.0

  55. MySQL 4.0: Server Migration DB server MySQL 4.0 MySQL 4.0

  56. MySQL 4.0: Server Migration App server DB server MySQL 4.0

    MySQL 4.0 MySQL 4.0
  57. MySQL 4.0: Server Migration App server DB server MySQL 4.0

    is not database... MySQL 4.0 MySQL 4.0 MySQL 4.0
  58. Why Can't Migrate? utf-8 utf-8 euc-jp euc-jp TABLE Mixed character

    set
  59. Why Can't Migrate? utf-8 utf-8 euc-jp euc-jp TABLE 4.0 4.1

    5.0 5.1 Mixed character set
  60. > libmysqlclient 5.7.5 • --skip-secure-auth REMOVED MySQL 4.0: DEADLINE MySQL

    4.0
 server MySQL client v5.6 MySQL client v5.7
  61. > libmysqlclient 5.7.5 • --skip-secure-auth (removed) MySQL 4.0: DEADLINE MySQL

    4.0
 server MySQL client v5.6 MySQL client v5.7
  62. livedoor Blog Roadmap

  63. Roadmap: livedoor Blog 2019 1H 2019 2H 2020 1H 2020

    2H 2021 1H Server Migration HTTPS Perl Update UTF-8 MySQL Update
  64. LINE BLOG

  65. > Blog Service of LINE Brand • Authentication with LINE

    account • Launched 2014 LINE BLOG
  66. History of LINE BLOG

  67. Service History 2014.11 LINE BLOG Launch

  68. Service History 2016.11 2014.11 LINE BLOG Launch LINE BLOG Renewal

    Launch
  69. Service History 2014.11 livedoor Blog ASP LINE BLOG Launch

  70. Service History 2014.11 2016.04 livedoor Blog ASP LINE BLOG Launch

    Begin renewal
 development

  71. Service History 2016.11 2014.11 2016.04 livedoor Blog ASP LINE BLOG

    Launch LINE BLOG Renewal Launch Begin renewal
 development

  72. Forked from livedoor Blog 2016.04

  73. > MySQL 4.0 > Perl 5.8 > mod_perl > Mixed

    character set > Apache > Low code coverage > Legacy subsystems Forked from livedoor Blog
  74. Forked from livedoor Blog > MySQL 4.0 > Perl 5.8

    > mod_perl > Mixed character set > Apache > Low code coverage > Legacy subsystems
  75. > MySQL 4.0 > Perl 5.8 > mod_perl > Mixed

    character set > Apache > Low code coverage > Legacy subsystems > HTTP Breaking Away from Chaos and Legacy > MySQL 5.6 > Perl 5.16 > PSGI > utf-8 > nginx > High code coverage > LINE platform > HTTPS
  76. Server
 Migration HTTPS 2017 LINE BLOG
 Renewal LINE BLOG's Contribution

    > Verification • Server migration • Database migration • Middleware updates > Test cases • Data charset conversion • Always-On HTTPS (SSL) > etc...
  77. Next 15 Years

  78. Start Small

  79. Now

  80. Join Us in This Battle!

  81. Thank You

  82. None
  83. None