Ops Tool made by Perl Beginner #yapcasia 2012

You can talk using this slide without any permission:)

Transcript

1. Ops Tool
   made by
   Perl Beginner
   @riywo
   Sep, 28, 2012
   YAPC::Asia 2012
   

   View Slide

2. @riywo
   

   View Slide

3. http://www.publicdomainpictures.net/view-image.php?image=4314&picture=t-shirt-image
   

   View Slide

4. View Slide

5. Perl
   Beginner
   

   View Slide

6. I Love
   Cats
   

   View Slide

7. DataBase
   Engineer
   MySQL
   

   View Slide

8. Operation
   Engineer
   DevOps
   

   View Slide

9. Operation
   Engineer?
   

   View Slide

10. Trouble Shooting
    Design System
    Setup Server
    On Calling
    Isu nage
    Sync Info
    Operation
    Inspection
    Develop Tools
    Capacity Plan
    

    View Slide

11. Trouble Shooting
    Design System
    Setup Server
    On Calling
    Isu nage?
    Sync Info
    Operation
    Inspection
    Develop Tools
    Capacity Plan
    

    View Slide

12. =Throwing
    Chair :-p
    

    View Slide

13. F*cking Query
    SELECT SQL_CALC_FOUD_ROWS *
    FROM table
    WHERE c1 > 1000 OR c2 != 100
    ORDER BY c3
    LIMIT 10 OFFSET 100000;
    

    View Slide

14. http://www.flickr.com/photos/andrew_freese/2200774154/
    Operation
    Engineer
    

    View Slide

15. dev> I did nothing..., but....
    

    View Slide

16. ops> What did you do?
    dev> I did nothing..., but....
    

    View Slide

17. http://www.flickr.com/photos/jaydoubleyougee/2259260845/
    Operation
    Engineer
    

    View Slide

18. I Love
    Cats
    

    View Slide

19. But...., do they only happen
    in Dev side?
    

    View Slide

20. Operation
    Error!!!!!!!!!!!!!
    

    View Slide

21. Operation Mistake
    # rm -fr /tmp/*
    # rm -fr / tmp/*
    

    View Slide

22. Capacity Over
    Sorry,
    we have no more servers....
    

    View Slide

23. ...and... Setup Error
    

    View Slide

24. server_id = 1212
    default_character_set=sjis
    innodb_buffer_pool_size=2G
    server_id = 1213
    default_character_set=utf8
    innodb_buffer_pool_size=3G
    

    View Slide

25. Can’t copy the same file to
    all servers
    => edit manually
    

    View Slide

26. Can’t copy the same file to
    all servers
    => edit manually
    M
    istake!!!
    

    View Slide

27. Current servers’
    sysctl changed
    But setup operation
    did not follow it...
    

    View Slide

28. Current servers’
    sysctl changed
    But setup operation
    did not follow it...
    M
    istake!!!
    

    View Slide

29. http://www.flickr.com/photos/clickflashphotos/3402287993/
    

    View Slide

30. Why?
    Are you a Professional?
    

    View Slide

31. Someone has
    “Setup Procedure Manual”
    1. OS install
    1.1 IP address
    1.2 nameserver
    1.3 OS user
    2. SSH Configuration
    2.1 ...
    3. Install Perl
    3.1 ...
    

    View Slide

32. Someone have
    “Setup Procedure Manual”
    1. OS install
    1.1 IP address
    1.2 nameserver
    1.3 OS user
    2. SSH Configuration
    2.1 ...
    3. Install Perl
    3.1 ...
    Operate
    Manually
    Check
    Visually
    

    View Slide

33. # ssh $TGT cat /etc/my.cnf | diff -u /etc/my.cnf -
    - server_id = 2323
    + server_id = 2324
    - log_slave_updaets
    Yes, this difference is OK
    If $TGT is Master DB
    this difference is NG
    

    View Slide

34. # ssh $TGT cat /etc/my.cnf | diff -u /etc/my.cnf -
    - server_id = 2323
    + server_id = 2324
    - log_slave_updaets
    Yes, this difference is OK
    If $TGT is Master DB
    this difference is NG
    Impossible
    

    View Slide

35. OK,
    Setup scripts help us!
    

    View Slide

36. OK,
    Setup scripts help us!
    ✗
    

    View Slide

37. Patch, Patch, Patch...
    patch for a new component many times,
    setup script become a spaghetti code...
    

    View Slide

38. Script Bug
    OpsEngi usually not good at programing,
    the script will have many many bugs...
    

    View Slide

39. Unfollow current change
    For trouble, servers’ config were changed,
    but setup script isn’t changed...
    

    View Slide

40. No Test!!!! F*ck!!!
    usually setup script is simple shell script,
    only listed exec commands
    no test
    

    View Slide

41. No Test!!!! F*ck!!!
    usually setup script is simple shell script,
    only listed exec commands
    no test
    

    View Slide

42. Check return code
    Check status of servers
    ✗
    

    View Slide

43. Describe
    

    View Slide

44. Server Correct Status
    /etc/sysconfig/ /etc/my.cnf /etc/sysctrl.conf
    /etc/ssh/ssh_config / /etc/cron.d
    /etc/resolv.conf /etc/yum.repos.d/
    /service/app/run chkconfig ulimit httpd.conf
    rpms
    

    View Slide

45. If you describe the correct status
    you can also test current servers!
    

    View Slide

46. Correct
    Status
    New Server
    Server
    Server
    Server
    Server
    Server
    

    View Slide

47. Correct
    Status
    New Server
    Server
    Server
    Server
    Server
    Server
    Test
    Setup
    

    View Slide

48. Break Time :)
    Remain 10min?
    

    View Slide

49. Correct
    Status
    New Server
    Server
    Server
    Server
    Server
    Server
    Test
    Setup
    

    View Slide

50. http://www.flickr.com/photos/stocktoc/4539506889/
    Touryo
    ౩ྊNBTUFS
    DBSQFOUFSPGTFSWFST
    

    View Slide

51. Describe
    

    View Slide

52. Orchestration
    Configuration
    Bootstrapping
    Server Setup
    

    View Slide

53. Orchestration
    Configuration
    Bootstrapping
    Server Setup
    Data sync, LB configuration
    OS install
    

    View Slide

54. Orchestration
    Configuration
    Bootstrapping
    Server Setup
    Configure, Install Middleware
    

    View Slide

55. Describe test/setup procedures
    DSL = blueprint.pl
    (cron,httpd,mysql,...)
    test {
    my $c = shift;
    ok ...
    };
    run {
    my $c = shift;
    $c->ssh(...)
    };
    

    View Slide

56. server_id =
    default_character_set=
    Server’s my.cnf
    Text::Xslate::render
    Test example
    diff
    

    View Slide

57. server_id =
    default_character_set=
    Server’s my.cnf
    Text::Xslate::render
    Run example
    SSH install
    

    View Slide

58. server1 => mbga:web
    server2 => mbga:db:master
    server3 => game1:web
    Describe role of nodes
    

    View Slide

59. Describe blueprints of roles
    mbga:web => [cron, httpd]
    mbga:db:master => [cron, mysql]
    game1:web => [cron, httpd]
    

    View Slide

60. Describe attribute of roles
    mbga:web => {
    cron_hour => 1,
    proc_num => 50,
    }
    mbga:db:master => {...}
    game1:web => {...}
    

    View Slide

61. How to
    develop?
    

    View Slide

62. Make time
    

    View Slide

63. Research
    

    View Slide

64. Small snippet
    

    View Slide

65. Ask Devs!
    

    View Slide

66. Lessons for
    Manage Tool
    Development
    

    View Slide

67. Maintenance
    Can you hand over its maintenance?
    

    View Slide

68. Touryo blueprint are written
    by other staffs :)
    

    View Slide

69. Simple
    The UNIX philosophy
    

    View Slide

70. Touryo only care about
    server configuration.
    Combine other tools
    ex. AdminTool(node-role)
    

    View Slide

71. Replace
    Is it able to replace easily?
    

    View Slide

72. By Touryo,
    undocumented procedures
    have been described.
    Easier to replace!
    

    View Slide

73. Any
    Questions?
    

    View Slide

74. Thanks!
    

    View Slide

75. @riywo
    blog.riywo.com
    weblog.riywo.com
    

    View Slide